TIP : change the Communication Policy to "Check In" every 8 hours (you may want to increase this if you have a lot of clients).
This way when the server is upgraded agents get rolled out in a slow but steady pace.
Also if you run into problems - you have a few hours to fix them. If you still have problems you can turn off communication completely.
We set all our agent install and upgrade policies to a schedule vs. the default of "Once". Our upgrade form 7.6 HF 2 to 4 required no intervention in the agent upgrade process.
After 2 days we returned to our normal 2 hour "Check In".
Per your question - the agents only get upgraded after the server.