May I introduce to you YUMI (Your Universal Multiboot Installer) from Pendrivelinux.com. I use this to set up my USB thumbdrive to have a menu of different ISO's that I can boot from. I have a bunch of utilities and tools including a PE 4.0 boot disk. This would require you to pick the iso it wouldn't have it auto choose base on arhitecture but would be better than 2 usb disks.
In your use case you could create a 64bit ISO and a 32bit ISO and then use YUMI to build a menu to choose what ISO you want to boot from. You then boot up choose your ISO it will then mount and boot from the ISO.
http://www.pendrivelinux.com/yumi-multiboot-usb-creator/
Step 1 pick your usb drive (the first ISO you add will you also want to format the drive)
Step 2 choose your distribution (for pe boot iso's choose Windows Vista/7/8 Installer)
step 3 choose your .iso
Step 4 repeate for each additional ISO. (but don't formate the drive after the first ISO is added)
Hope that helps.