Vxdisksetup and encapsulating a disk via vxdiskadm are two very different things.

vxdisksetup:
Vxdisksetup configures the disk for use by Veritas Volume Manager (VxVM). vxdisksetup is called by the vxdiskadd command as part of initial disk configuration. It is not typically necessary to call this command directly.

I believe you are sellecting the root disk encapsulate & mirror in vxdiskadm.  Is this for rootdg?
See the guide, page 114:
http://sfdoccentral.symantec.com/sf/5.0/solaris/pd...

From the  vxencap man page: 
"In the case of partitions that contain file systems, the entries in /etc/vfstab are modified to use the new volume devices.  ... A reboot may be necessary to complete the process of encapsulation. "

Encapsulation is only disk/volumes where you are saving data.  There are challenges to growing and shrinking encapsulated data. (until 5.0mp3).   So, if it's not the boot disks, it's probably better  to create a new volume & filesystem, copy the data, umount the old, remount the new in it's place,  and change the mount device in /etc/vfstab.

As mentioned above, these are two different things..

1. Error while running vxdisksetup,
      -- Is this drive visible in format ?
      -- Does it contain a valid VTOC ? can you see a VTOC table when you run prtvtoc <raw_dev> ?

2. Regarding encapsulation
    -- Yes, encapsulation requires reboot, as explained above, it is separate process for separate purpose.

Hope this helps.

Gaurav

while encapsulation is for a disk which already contains data.
Reboot is only needed for encapsulation of root disk.

Encapsulation you can use vxencap (would require a reboot)
vxdiskadm - high level command menu driven command.

Incase you still require clarification, do let know.

Cheers,
prax