If you pay no life, you shuffle the rest of your library and then put the top five cards back on top in any order.

In other words: — Step 1: Look at the top five cards of your library. — Step 2: If you like them, proceed to step 3a. If you don't like them, proceed to step 3b. — Step 3a: Shuffle the rest of your library, then put those five cards back on top of your library in the order you want. The spell has finished resolving. — Step 3b: Put those five cards on the bottom of your library in the order you want. Pay 1 life. Return to step 1.

If you have five or fewer cards in your library, you can pay 1 life as many times as you like, but you'll keep seeing the same cards.