Forest fragmentation does not only remove carbon stocks of logged forests, but also erode substantial biomass from remnant forests through elevated tree mortality. Some experimental studies have documented such initial response of tree communities to fragmentation, though longer-term dynamics is still largely unknown. We studied tree communities and biomass across nine stands of Japanese temperate forests, fragmented anthropogenically around 1940s, along a gradient of fragment size (0.3–6.3 ha). In 1999, aboveground tree biomass tended to be greater in larger fragments, suggesting greater loss of biomass in smaller fragments following forest fragmentation. Analysis of size structure and tree-ring series infers that the loss of biomass could be attributed to deaths of large trees soon after the fragmentation events. Afterwards the biomass recovered more rapidly with increasing severity of fragmentation; during the period of 1999-2011, smaller fragments gained greater biomass partly because of the proliferation of disturbance-dependent species, so that no relationship between fragment area and biomass was found in 2011. These results suggest that the time required for tree biomass to recover is extended for decades and certainly over half a century.