A growing body of field studies has shown that phenotype of an individual plant depends not only on its own genotype but also on those of neighboring plants. Yet, this fact has not been considered in genome-wide association studies (GWAS). Based on the Ising model of ferromagnetism, here we proposed a new methodology named “neighbor GWAS” that incorporated neighbor genotypic identity into a regression model. Our simulations showed that phenotypic variation explained by neighbor effects approached to a plateau when an effective spatial scale became narrow; thus, the effective scale of neighbor effects could be estimated by patterns of the phenotypic variation explained. The power to detect causal variants of neighbor effects were moderate to strong when a trait was governed by a few major genes. Then, we applied the neighbor GWAS to field herbivory data on 200 accessions of A. thaliana, and found that the neighbor effects more largely contributed to the observed damage variation than self-genotype effects did. These findings highlight the overlooked but significant role of plant neighborhood effects in shaping phenotypic variation in field environments.