Objectives The implementation of ecological compensation represents a pivotal policy instrument for addressing the conflict between ecological preservation and economic development. Investigating the impact of ecological compensation mechanisms on the conservation behavior of farmers in natural reserves holds significant importance in enhancing current ecological compensation policy.

Methods Based on the theory of planned behavior and the theoretical analysis framework of "the mechanism of ecological compensation on the ecological construction protection behavior of farmers in nature reserves", this paper proposed a research hypothesis, and based on the micro-survey data of 378 farmers in 3 national nature reserves in Yunnan and Hunan provinces. The structural equation model was used to analyze the factors affecting the ecological protection behavior of farmers in nature reserves.

Results The results demonstrated that: (1) Ecological compensation can significantly improve the enthusiasm of farmers for ecological protection; (2) The psychological variables of farmers (behavioral attitude, subjective norms, perceived behavioral control) not only exerted a significant positive and direct impact on their ecological protection intentions, but also had an indirect impact on their ecological protection behaviors through the ecological protection intentions variables; (3) Farmers' ecological protection intention significantly positively affects their ecological protection behavior. (4) Education level, income level, and planting scale exerted a significant influence on the various hypothesis paths. Among them, farmers with higher education and income were more willing to implement ecological protection behavior, while farmers with lower income and larger planting scales were more concerned about ecological compensation policies and relied on subsidized income. (5) There were certain differences in the impact of different protected areas in different hypothetical paths, which are mainly related to the geographical location of natural protected areas and the level of economic development.

Conclusions The study put forward suggestions from three aspects: the establishment of a precise and efficient ecological compensation policy system, the improvement of the psychological expectations of farmers' participation in ecological protection, and the exploration of a new mechanism of community co-management in nature reserves, which will help to further clarify the logical relationship behind farmers' participation in ecological protection in nature reserves. The study had important theoretical significance and practical value for guiding farmers to participate in ecological protection, promoting the vigorous development of biodiversity, and improving the ecological compensation mechanism.