Adding a Managed Host (MH) to the Storage Foundation Management Server (SFMS) causes previously added Managed Hosts (MHs) to disappear from the SFMS console and SFMS domain (as shown in the output of vxdomstatus)

Symptoms

After adding a new managed host or legacy managed host (MH/LMH) to an SFMS environment it is seen in the SFMS console. In addition the output of "vxdomstatus" reports all Provider Access Layer (PAL) agents running on the MH as part of the SFMS domain.  
 
When another MH is added to the SFMS environment, however, the original MH disappears from both the SFMS console and SFMS domain, i.e. only the new MH will be shown in the SFMS console and PAL agents running on the original MH are no longer shown in the output of a "vxdomstatus"

Cause

The issue occurs when multiple MHs/LMHs client software is installed from a single software image (i.e. via JumpStart, Ignite, NIM e.t.c.)
 
To uniquely identify each MH within the SFMS domain, a unique identifier (HOSTGUID) is generated on each MH/LMH during SFMS client software installation. This HOSTGUID is used in the central SFMS database as the identifier for the MH within the SFMS domain. If a MH being registered with SFMS has the same HOSTGUID as a previously registered MH, the new MH will replace the old MH in the central SFMS database (rather than creating two separate entries for each MH). As such, only the most recently registered MH with a given HOSTGUID will be registered in the SFMS domain causing the previous MH to disappear from the SFMS console and output of "vxdomstatus".  Other issues are raised because of the single software image. For example, the PERSISTENT_ID of vxsvc and the Veritas Enterprise Administrator (VEA) agents are the same. This will confuse VEA domain server.
 
To display a MHs HOSTGUID the following command should be run as root on the MH:
 
# /etc/vx/uuid/bin/osuuid list
 
For example:

{ab491f72-1dd1-11b2-88e2-0003ba2476c7}

To display the vxsvc and VEA agents' PERSISTENT ID:

For example:

                [REG_SZ] "PERSISTENT_ID" = "{6011c8cc-1dd2-11b2-8028-0003bac09999}";

                [REG_SZ] "PERSISTENT_ID" = "{601acf44-1dd2-11b2-8028-0003bac09999}";

                [REG_SZ] "PERSISTENT_ID" = "{6024f582-1dd2-11b2-8028-0003bac09999}";

                [REG_SZ] "PERSISTENT_ID" = "{600790d2-1dd2-11b2-8028-0003bac09999}";

Resolution

To resolve the situation, we need to migrate the MHs/LMHs to standalone mode. Any MHs/LMHs with non unique HOSTGUID identifiers and the PERSISTENT_ID should have their HOSTGUID and PERSISTENT_ID identifiers regenerated. And finally migrate the MHs/LMHs to central mode. This can be done as follows:

# /opt/VRTSmh/bin/fix_cloned_mh -d MS0.exampledomain.com -p vea

An example of above procedure is shown below.  In this example our MHs are "MH0" and "MH1", our Storage Foundation Management Server is "MS0.exampledomain.com" and our VEA password is "vea":

Fix the MHs with conflicting HOSTGUIDs and PERSISTENT_ID:

      root@MH0 # /opt/VRTSmh/bin/fix_cloned_mh -d MS0.exampledomain.com -p vea
      root@MH1 # /opt/VRTSmh/bin/fix_cloned_mh -d MS0.exampledomain.com -p vea


Once complete all MHs will display correctly in the SFMS console and each will show their PAL agents as running in the output of "vxdomstatus":

root@MS0 # /opt/VRTScs/bin/vxdomstatus
*******************************************************************************
Domain Controller fame

=============================================================
Agent Host Status
=============================================================
Root fame UP
StorageAgent MH0.exampledomain.com UP
StorageAgent MH1.exampledomain.com UP
actionagent MS0.exampledomain.com UP
actionagent MH0.exampledomain.com DOWN
actionagent MH1.exampledomain.com DOWN
gridcentral MS0.exampledomain.com UP
gridnode MS0.exampledomain.com UP
gridnode MH0.exampledomain.com UP
gridnode MH1.exampledomain.com UP
*******************************************************************************

Best Practice

It is recommended migrating MH/LMH to standalone mode before cloning the MH/LMH image. This will reduce the runtime of fix_cloned_mh.sh by skipping the migration of the MH/LMH to standalone mode.

1. Install MH/LMH
2. Upgrade MH/LMH to 1.1
3. Migrate the MH/LMH to standalone mode

For example:

/opt/VRTSmh/bin/migrate standalone_mode -d MS0.exampledomain.com -p vea

4. Clone the MH/LMH image
5. Install the new host with the cloned image, change the IP and the hostname.
6. Run /opt/VRTSmh/bin/fix_cloned_mh on the original MH and the cloned host.