Understanding the relative importance of selection and stochastic factors in population divergence of adaptive traits is a classical topic in evolutionary biology. In the damselfly Ischnura senegalensis, females exhibit heritable color dimorphism (andromorph and gynomorph). Morph frequencies changed with latitude in Japan, ranging from 0% (south) to 100% (north) for andromorphs, while they were relatively stable at a local scale. Here, we evaluated the role of selection and drift on population divergence of morph frequencies by comparing the population divergences of the color locus and neutral (microsatellite) loci. The divergence of color locus among populations was larger than that of neutral loci at a large geographic scale, suggesting that the geographic pattern of color locus is mainly determined by divergent selection. Contrarily, the divergence of color locus among adjacent populations tended to be smaller than neutral expectation. The effect of balancing selection may overcome the effects of drift and divergent selection at a small geographic scale. In both small and large geographic scales, no phylogenetic effect on morph frequency was found. We discuss the relative importance of divergent selection, balancing selection and stochastic processes at two different geographic settings.