Once you announce that you’re casting Wizard’s Retort, no player may take other actions until the spell’s been paid for. Notably, players can’t try to raise the spell’s cost by removing your Wizards.