In terrestrial ecosystems, ecological processes and patterns within focal patches frequently depend on their matrix. Crop fields (focal patches) are often surrounded by a mosaic of other land-use types (matrix), which may act as habitats for organisms and differ in terms of the immigration activities of organisms to the fields. We examined whether matrix quality affects wild pollinator abundance in crop fields, given that the species (Apis cerana) generally nest in the cavities of natural trees. We examined fields of a pollination-dependent crop (Fagopyrum esculentum) surrounded by plantations and natural forests, which comprised the matrix. Our analysis revealed a clear positive effect of the natural forest on the pollinator abundance, but the plantation forest had little effects. These indicate that agricultural patches are influenced by their matrix quality and the resulting crop pollinator abundance, suggesting the importance of matrix management initiatives such as forest restoration surrounding agricultural fields to improve crop production.