Terry Bu makes a valid point - if the NIC is only running at 10 Mbps or some other "slow" permutation under WinPE, then the restore is going to take a lot longer. Sometimes the connection speed can be checked easily by looking at the colour of the connection light on the switch.
The default speed for the NIC is often set in the INF file which is part of the driver fileset. You can open the INF file and have a look at the definitions therein. There is usually a section where the different NIC speeds are listed as description strings, and another section where a default setting points to one of these. The default may point to an "Auto" setting, but if you find that the connection speed is incorrect you could try editing the INF file to explicitly default to the lan speed you want to connect at.
Drivers can be loaded dynamically from the WinPE command prompt after WinPE has booted, which means you can experiment with different INF file configurations without having to rebuild the WinPE image each time. If you make a bootable USB key, or point your bootable WinPE media at a USB key which had the INF SYS and CAT files on it, then all you need to do at the command prompt is type
DRVLOAD <full path and filename of driver INF file>