This study aims to evaluate the effect of long-term grazing and timing of protection on the soil seed banks. The soil seed banks and the coverage of aboveground understory vegetation were surveyed in 2019 in 4 valleys: an early-exclosure valley was built in 2006 (about 10 years after overgrazing), a late-exclosure valley was built in 2017 (about 20 years after), and 2 valleys without exclosure. We compared the data to the corresponding data obtained in 2010 by Honda. The results demonstrated that due to the long-term deer overgrazing, soil seed banks tended to be depleted outside the exclosures and did not recover in the late-exclosure although the deer density has been rather low. Therefore, it requires a long time for the vegetation to recover. Besides, in the early-exclosure valley, the soil seed bank was abundant and the species diversity was high, and the succession proceeded to change both understory vegetation and soil seed bank towards shrub and tree dominant composition. More than 10 years of deer overgrazing will deplete soil seed banks to the level where recovery of vegetation is hard to expect. Conservation of seed banks by the early protection is necessary for the vegetation recovery in forest ecosystems.