Abstract: Individual identity differences such as race and gender always exist in the process of facial expression recognition, which will reduce the classification performance of the system. Therefore, this chapter proposes anDE-Gabor feature to enhance the robustness of identity.Firstly, a double subsampling strategy is proposed to reduce the dimension of Gabor feature to obtain a more compact E-Gabor feature. Then, according to the interference of identity information, the E-Gabor features were sparsely represented on neutral feature dictionary and expression feature dictionary respectively, and the virtual neutral feature and virtual expression feature of the sample were reconstructed, and the two differential coding were used to obtain independent identity features. Finally, the SVM model was trained based on DE-Gabor features to classify facial expressions.In addition, the DE-Gabor featurewere applied to data sets of different races and genders to explore the relationship between identity interference and facial expression recognition under different cultural backgrounds. Experimental results on BU3DFE data set show that the classification performance of DE-Gabor features is better than other methods.