The recurrent evolution of self-fertilization (selfing) from outcrossing in flowering plants has been a major scientific topic since Darwin. In many outcrossing species, selfing is prevented by the self-incompatibility (SI) system, which consists of male and female components determining specific recognition between pollen and pistils, and SI modifier genes. From both theoretical and empirical points of view, much attention has been paid to clarify which component degraded first, thereby leading to the loss of SI and the origin of selfing. While theories have predicted that mutations disabling pollen specificity have a higher probability of leading to successful mating and are thus more likely to spread than those disabling pistil specificity, empirical reports supporting this prediction were lacking. Using multiple selfing species of the Brassicaceae, we demonstrate that the female components and the modifier genes of the SI system are still functional in many accessions, suggesting that the degradation of male components was responsible for the loss of SI. Furthermore, our compilation and statistical analysis of recent studies suggest that this male-driven loss of SI is indeed common in wild species of the Brassicaceae, consistent with theoretical predictions regarding the evolutionary advantage of mutations in male components.