If the targeted land is an illegal target by the time Roiling Terrain resolves, the entire spell doesn’t resolve. No damage will be dealt.

The effects of the spell happen in sequence. By the time Roiling Terrain checks how many land cards are in the affected player’s graveyard, the land Roiling Terrain destroyed is (most likely) already in that graveyard.

If the targeted land regenerates or has indestructible, Roiling Terrain still deals damage to the targeted land’s controller.