When there is gene flow between populations adapting to contrasting environments, recombination breaks up a combination of alleles beneficial for local adaptation. Therefore, recombination can prevent adaptive divergence in the presence of gene flow. Chromosomal rearrangement, such as inversion, can reduce recombination and therefore contribute to divergence with gene flow. Although theoretical studies support this hypothesis, few empirical studies have yet investigated how prevalent chromosomal rearrangements are and whether they contribute to divergence between parapatric and sympatric populations. The presence of multiple marine and freshwater populations in both parapatry and allopatry makes the stickleback fish an excellent model for testing the hypothesis. Furthermore, recent technological advances in genome assembly enable us to comprehensively investigate chromosomal rearrangement. Here, we newly assembled and compared genomes of multiple marine and freshwater stickleback populations. We successfully detected many inversions between populations. Regions with inversions between sympatric or parapatric population pairs showed significantly higher genetic differentiation compared to collinear regions, suggesting that inversion acts as barriers of gene flow. No such a trend was observed between allopatric pairs. Some inversions were shared among geographically isolated freshwater populations, suggesting that these exist as standing genetic variations in this species and possibly contribute to freshwater adaptation.