The reservation would come into play to ensure that the VMs in RP1 would get 10 GHz.
Now if a core on the servers were 1 GHz, then all those VMs could consume would be 10 GHz leaving another 10 GHz for the VMs in RP2 and the 10 non-RP VMs. Those 20 VMs would split the 10 GHz to run at 500 MHz each.
If the CPU cores were running at more than 1 GHz, then by my understanding it would get a bit more complicated. Let's say a core is running at 1500 MHz. All 30 VMs are trying to max out their CPUs.
The 10 VMs in RP1 would get at least 10 GHz (as per the reservation). Then remaining 10 GHz would be split between the 30 VMs using shares. RP1 would get 3333 MHz as would RP2. Each non-RP VM would get 333 MHz. So the 10 GHz would be split using 30,000 shares.
RP1 VM - 1333 MHz
RP2 VM - 333 MHz
non-RP VM - 333 MHz.
See page 19 of this presentation for some background on how remaining resources are split after a reservation is applied - download3.vmware.com/vmworld/2012/top10/vsp1683.pdf. You can find the audio for the presentation on YouTube.
I wouldn't suggest keeping VMs outside of a resource pool. If you were to end up with additional VMs in the RPs, those VMs would be splitting 10,000 shares between an increasing number of VMs resulting in a smaller split of the overall clusters resources. Going back to 1 GHz cores, if you end up with 20 VMs in RP2, then those VMs and the 10 non-RP VMs would still be sharing 10 GHz, but the VMs in RP2 would split 5 GHz 20 ways (250 MHz per VM) while the non-RP VMs would stil end up with 500 MHz each.
