Abstract: To solve the problem that the linearized measurement and relative dynamics models cause the angles-only relative navigation system to be unobservable. First, the precise relative dynamics and nonlinear measurement equations of the chaser spacecraft and space target are established based on spherical coordinate system. The first-order and second-order Taylor series expansion are performed on this nonlinear measurement equatio. Then, the direction of the angles-only initial relative orbit determination (IROD) solution is determined by the the measurement equation obtained from first-order Taylor series expansion. Subsequently, based on the orthogonal right singular vector, the approximate angles-only IROD solution is determined by the measurement equation obtained from the second-order Taylor series expansion. An iterative optimization method is proposed to improve the accuracy of the approximate angles-only IROD solution. Then, an angles-only relative navigation filter based on spherical coordinate system is designed using Adaptive Extended Kalman filter (A-EKF). Finally, performance testing is conducted via a built semi-physical simulation system. The experimental test results show that the method proposed provides complete observability for angles-only relative navigation system, with the relative orbit determination error of the proposed method can reach about 0.65% of the measured range.