Abstract: Horn-Schunch(HS) optical flow algorithm can detect the whole moving object in the dynamic background.However,the complexity of the algorithm is great and it can not detect the optical flow in the region where the texture is smooth in the image.Based on the analysis on HS algorithm,this paper proposed an algorithm combined with the pyramid Lucas-Kanade(pyrLK) optical flow and the HS optical flow to detect the moving object in dynamic scene.In this paper,we firstly calculate the sparse optical flow of the image using the pyrLK optical flow algorithm.Then we eliminate the moving vector of the moving object and the mismatched points in terms of the moving direction and the amplitude of the sparse optical flow.At last,we take the moving vector extracted from background as the initial value for HS optical flow calculation to detect the whole moving object.The experiment result shows that our algorithm improved the speed of the detection for moving object and it fits the smart surveillance with the Pan/Tilt camera.