Optical rotation detection system (ORDS) utilizing quantum nondemolition (QND) measurement methods is widely applied in the field of quantum metrology and quantum information. However, the sensitivity of ORDS is limited by the uncertainty from optical-couple noise during the measurement of the atomic spin ensemble. In this study, we specifically analyze the mechanism of optical-couple noise caused by the fluctuations of probe light’s polarization in the modulated ORDS with a new model established to describe atomic spin precession in this particular condition. It is discovered that transverse electron-spin polarization errors are generated by the residual probe photon spin polarization in the ORDS, which results in extra coupling magnetic noise. In order to suppress this noise, a novel in situ method is proposed that the resultant electron-spin errors are reduced by a specifically designed closed-loop system. The results are verified through the ORDS in a co-magnetometer. After zeroing the extra probe photon spin polarization, an angular sensitivity better than

$1 \times 10 ^{\mathrm {-8}}$

rad/Hz

$^{\mathrm {1/2}}$

is achieved for frequencies higher than 5 Hz, demonstrating a probe background noise of 0.26 fT/Hz

$^{\mathrm {1/2}}$

@14.5 Hz, approaching electronic noise and photon shot noise (PSN). With closed-loop control, the optical rotation bias instability is promoted by 4.2 times (from 107.9 to

$25.7 \mu $

rad/h), and the angular noise of the ORDS is reduced by 2.4 times at 1–100 Hz. The measurement uncertainty of the realized ORDS nears the standard quantum limit, paving the road for long-term ultrasensitive measurements for new physics explorations.

	Year	Citations

About