How to troubleshoot MSMQ message flow in clustered environment for Enterprise Vault
|Article:HOWTO75029|||||Created: 2012-04-01|||||Updated: 2012-09-04|||||Article URL http://www.symantec.com/docs/HOWTO75029|
Enterprise Vault (EV) enables an organisation to store messaging and file system data automatically in centrally held archives. Archiving allows messages and documents to be retained for policy and compliance reasons, and reduces the amount of data stored on primary disk space. Some items It can archive include:
- Items in Microsoft Exchange user mailboxes
- Items in Microsoft Exchange Public Folders
- Items in Domino mail files
- Files from network file servers
EV organises the archives in entities called vault stores, and each vault store has its own database. The vault store database holds information about the archives in the vault store and all the items stored in each archive.
EV runs a number of tasks, such as the Exchange Mailbox Archiving Tasks that archives items from user mailboxes. The task is assigned a target that has a policy assigned to it that defines where the item is to be archived and when it is to be archived. The archiving tasks collects the item to be archived and passes them to the Storage Service. When the item is archived the original is deleted and a shortcut to the archived item is created. On instruction from the Storage Service the Indexing Service will index items as they are archived, and it is the Index Service that searches indexes and returns information during searches.
How MSMQ works with EV
Archive jobs can be either submitted by a user or part of a regular schedule that runs . These jobs then appear in the EV queues for processing:
- User submitted jobs appear in "MSMQ...<source>... a2" queue
- Scheduled jobs to archive when started appear in the "MSMQ...<source>... a5" queue
- Scheduled jobs to archive that are "run now" are "MSMQ ...<source>... a3" queue
where the source will be the Exchange server or some other messaging or file server application.
The server where the archive task runs may not be the server where the archive store is. When configured an EV instance points to a particular Exchange instance (for example). Thus EV will create a vault store location for that user. When EV runs archive jobs, then these are submitted to the local message queue for processing, the archive message is processed by the local EV Storage Archive Task / Storage Service. The message flow then will be:
- archive job queue a5
- vault store processing by Storage Service
From time to time an organisation may move exchange users to a different exchange server, however the user's archive does not move to a new Vault Store. If this is the case then when the scheduled archive jobs run the message to be archived will have to be transferred back to the EV instance where the users vault store resides. In order to do this the message will go to the MSMQ Outbound Queue where it is transferred to the appropriate remote server with EV. On the remote server the message arrives at the EV Storage Archive Queue and is processed by the EV Storage Archive Task process.
Consider the diagram below, the user is initially on Exchange Server A. When the user is moved to Exchange Server B, their vault store remains with the Enterprise Vault Service Group 1 vault store. The message flow becomes:
- EV mailbox scanning on Exchange Server 2 identifies messages in the users mailbox per the archive policy to be archived and these messages are placed in a queue "MSMQ..Exchange Server B.. a5"
- The vault store is located on the EV instance on Cluster Node 1, the message must be moved to that node for archiving
- The message is moved to the MSMQ outbound queue on Cluster Node 2 for Cluster Node 1
- The message is moved by MSMQ to Cluster Node 1
- The message is moved to the user's vault store by the Storage Service on Cluster Node 1
- For remote processing MSMQ must show an outgoing connection IP to the remote server where the target vault store is located
- Journaling can be enabled to track message flow
- To view the clustered MSMQ it is required to use the ClusterCompMgmt.bat to launch the Server Manager in the virtual context of the EV instance (by default in c:\ program files\Enterprise Vault\)
How to check MSMQ is listening
If the MSMQ driver is not listening on port 1801 then it will not receive messages. In particular the clustered MSMQ driver (the virtual instance) must be listening for message flow that may be necessary between clustered MSMQ. On the cluster nodes where EV is online with the MSMQ, check that the clustered instance of the service is listening by the following steps:
netstat –abno | findstr 1801,
C:\>netstat -abno |findstr 1801
TCP 10.251.59.54:1801 0.0.0.0:0 LISTENING 4148
TCP 10.251.59.118:1801 0.0.0.0:0 LISTENING 102556
TCP 10.251.59.117:1801 0.0.0.0:0 LISTENING 75272
This lists all instances of MSMQ binding to local IP addresses. It could be the clustered MSMQ for EV, or it could be the local MSMQ service. In this case there are three different process ID so there are three differnet message queue drivers running. To confirm that one of these is the virtual driver in use for EV execute "tasklist /svc | findstr <PID>", where the EV cluster group IP address is 10.251.59.117, the corresponding Process ID is 75272
C:\>tasklist /svc | findstr 75272
mqsvc.exe 75272 MSMQ$EVSG-MSMQ
This indicates that the virtual message queue driver has bound to the port that MSMQ is expected to run at. MSMQ$EVSG-MSMQ is the name of the clustered driver in the EV service group. If this is not present then EV will not be able to send / receive messages.
Tracking message Flow with End2End
To see if MSMQ is sending messages between queues, enable End2End logging. Refer to the following Microsoft Technet article:
The following steps show how to enable End2End logging and then use it to see if messages are flowing between queues. For example, if EV is not processing messages from queues, it might be appropriate to use End2End logging to see if the messages are leaving the source instance and getting to the destination queue. Finding if messages are flowing or not can narrow down the problem scope.
Enable End2End tracing Log
To enable end-to-end tracing
1. In Event Viewer, open Applications and Services Logs.
Go to Computer Management/Event Viewer/Applications and Services Logs
2. In the console tree, double-click Microsoft, double-click Windows, and then double-click MSMQ.
3. In the details pane, right-click End2End, and then click Properties.
4. On the General tab, select the Enable logging check box.
5. Click OK.
Review End2End tracing Log
- In Event Viewer, open Applications and Services Logs.Where?
· Computer Management/Event Viewer/Applications and Services Logs
- In the console tree, double-click Microsoft, double-click Windows, double-click MSMQ, and then click End2End.
- In the details pane, view the end-to-end tracing event.
- You should see 2 messages logged event ID 4 and event ID 1 in End2End log on the receiving server’s after a message from the private queue.
Note: Events will not contain contents of message or the description, so best practice is to clear the End2End logs completely first on both nodes before sending any messages.
5. You will see event ID 5 and event ID 3 on server sending the message in End2End log. Once again these events do not show message contents or description.
Testing Message flow with third party utilities to populate messsage queues
To obtain more information about the problem of failing to send messages between clustered MSMQ instances there are a number of third party tools that allow sending of messages from queues manually created under the virtual MSMQ. This removes the need to instigate message flow from EV and enables testing of the clustered MSMQ's ability to send and receive messages.
To ensure testing of the clustered instance (as opposed to the local MSMQ), then when using Veritas Cluster Server, open the Application Manager to connect to the virtual server where the virtual MSMQ instance is online. It is necessary to create a private queue on each node.
If this scenario fails, then it may be appropriate to repeat the testing here using the local MSMQ. The steps will be the same however when indicated to connect to the virutal MSMQ use the local MSMQ instead.
1. Create private queues on each cluster node
a) First open Application Manager
b) Select the msmq service group currently on that server and select Manage
This cluster has two virtual computers with MSMQ
- MSMQ1 = Virtual name of Messaging Queue Service Group on Node1
- MSMQ3 = Virtual name of Messaging Queue Service Group on Node3
c) Computer Management for that particular msmq service group will open.
Expand Message Queuing and create a new Private Queue. No need to select transactional checkbox.
For the purposes of this example, the following queue names are created
- priv_queue_msmq1 = Private Queue Created on MSMQ1
- priv_queue_msmq3 = Private Queue Created on MSMQ3
2. Using the third party tool send messages between Node1 and Node3
a. Start with Node1, connect to the virtual MSMQ virtual name
b. select the private queue, priv_queue_msmq1
c. create a message in this queue and mark the destination as priv_queue_msmq3
d. On Node3, connect to virutal MSMQ and check priv_queue_msmq3 for the message
3. Reverse the steps in Step 2 to confirm message transfer from priv_queue_msmq3 to priv_queue_msmq1
Article URL http://www.symantec.com/docs/HOWTO75029