In this presentation, we consider data $x,1\dots,x_n$ generated from a normal distribution $\mathcal{N}(0,\Sigma)$, where the covariance matrix is $\Sigma=A+OBO^\ast+I_p$. Here, $A$ and $B$ are diagonal matrices of small rank $r,s<p$, and $O$ is a Haar orthogonal or unitary matrix. We study the fluctuations of the eigenvalues of the empirical covariance matrix $\hat{\Sigma}=\frac{1}{n}\sum_{k=1}^nx_kx_k^\ast$ when $n$ and $p$ go to infinity. Interestingly, because of the randomness coming from both the sampling and the Haar matrix, these fluctuations are very different depending on the scaling regimes. We look specifically at the cases where (i) $p/n\rightarrow c\in(0,\infty)$, (ii) $p/n\rightarrow 0$ with $p^2/n\not\rightarrow 0$, and (iii) $p^2/n\rightarrow 0$. We show that the convergence rate changes between $p$ and $\sqrt{n}$. Moreover, the limit law is not always the usual Gaussian one (like the well-known limit from Bai and Yao). When $p^2/n$ does not go to infinity, the limit takes unusual forms, such as convolutions of Gaussian and exponential distributions. As a consequence, the classical spectral tests that rely on the standard Gaussian limit may be misspecified if the ratios $p/n$ and $p^2/n$ are in these problematic regimes. To solve this issue, we propose spectral tests that are well-specified. We show how they can be used for concrete applications, such as signal processing with interference (in moderate dimension) and genomic data with batch effects (in high dimension).