Video Screencast Help
Symantec to Separate Into Two Focused, Industry-Leading Technology Companies. Learn more.

DTRACE - Basics

Created: 18 May 2011 • Updated: 18 May 2011 | 6 comments
Language Translations
Rob.Wilcox's picture
+4 4 Votes
Login to vote

 

Introduction

This article explains what DTRACE is; how to view DTRACE log files at a basic level; where you can run DTRACE from; how to setup your own .ini file and some basic information on setting up filtering of DTRACE in order to try to capture information which is more relevant to a problem you might be investigating.

DTRACE is a diagnostic tracing utility for Enterprise Vault, and covers almost all of its component parts.  It covers tracing at a number of levels giving progressively more detail, ie brief, medium and verbose.  It also covers tracing on non-Enterprise Vault servers, such as an FSA file server target or a SharePoint server, which have Enterprise Vault components installed.  It is used to help investigate a problem - it's not something that should be running "normally" on an Enterprise Vault server; there can be some performance implications of running it.

One of the notable exceptions to the rule is that DTRACE isn’t a client based utility, so, it’s not something that exists in the Enterprise Vault Outlook Add-in.  That component has its own tracing capabilities.

Example of running DTRACE

There are two ways of running DTRACE on an Enterprise Vault server, there is the “old fashioned” command line method which I’ll describe here, and there is the option to run DTRACE directly from the VAC – that’s described in a subsequent section.

To run DTRACE on the Enterprise Vault server, and give it a quick test-drive, perform the following steps :

  1. From the Enterprise Vault server, open a command prompt.
  2. Navigate, in the command prompt, to the location where Enterprise Vault is installed
  3. Type Dtrace followed by the Return key. You will see the following :

 

  1. For the purposes of this example type set archivetask v followed by the Return key
  2. Assuming c:\temp exists, on the next line type log c:\temp\archivetask.log followed by the Return key
  3. Open the Vault Admin Console, locate your mailbox archiving task, and do a Run Now for one particular user (just so it’s quick to run). Wait a few minutes, so that you’re sure the task will have finished running.
  4. Go back to the DTRACE window, and type log followed by the Return key. Note: As a snippet from the more advanced DTRACE article, you can at this point type O, followed by the Return key and DTRACE will list the files that it has logged to, you can then open this file using the number associated with it.
  5. Type Exit. Your window should look similar to this :

 

In Windows Explorer locate thec:\temp\archivetask.log file.  Just before opening it, you can get a very high level view of whether or not you got anything of any use by just observing the file size.  Is it tiny?  Is it a few hundred Kb, or a Mb?  If it’s very small – it might be that nothing was traced at all.  We’ll come back to why that might be. 

For now let’s open the log file.  It’ll look similar to this :

 

There are two things to look for at this time :

The "header":

This includes things like the server name, time, version of Windows, UTC offset and so on.  It also includes the trace options which were enabled, and filters (see later)

The "footer":

The footer comes right at the bottom of the trace file (of course) and shows you the registry of the Enterprise Vault server, and the version numbers for all the binaries :

 

From the Vault Administration Console

As described in the previous section you can run DTRACE from a command line, and you can also run it from the Vault Administration Console.   In order to do this perform these steps :

  1. Make the Trace node visible. By default the trace node isn't visible in the VAC. 

 

With the VAC open, go to Tools -> Advanced Features and select it to enable it

  1. Right click and choose refresh on the servers node, and you will now see the Trace node 

 

  1. Right click on Traces, and choose New. A wizard will start which will run you through creating a trace file.
  2. The first major choice is the type of trace you want to collect, there are many available, in a drop down list :

 

For the purposes of this test, pick Mailbox Archiving Issues (Exchange). You can then give a title and description to the issue you are trying to investigate

  1. We then go to the size / run time dialog :

 

Once you've confirmed all of the information the VAC will start the trace straight away and you'll see this :

 

Unfortunately the VAC doesn't auto refresh, so you will need to refresh it a few times, but eventually it will complete as shown below :

 

Once finished you can right click on the trace title, and that will show you the following options:

 

Going to properties can be useful so that you can relocate where the log file(s) were created if you need to them take them off the server for further analysis.

 

Looking for Tilde’s ~

One of the first things that you should do after checking the basics of whether something was actually logged or not, and the “header” and “footer” of the DTRACE log file described previously, is that you can search for Errors, Warnings, and Exception.  To do that perform the following steps:

  1. With the DTRACE open in Notepad (or any other text editor), launch the editor search option and search for ~E (ie tilde E). If there are some errors you'll see a line like this:

32418 21:30:00.507  [10108] (w3wp) <12996> EV~E Event ID: 6979 AutoStorageOnline error. |Reason: Access denied      [0xc0041801] |Reference: [UAIM/1615E9AEE1DF12EEB8EDAA8C7988E04EF1110000EVAULT1] |

  1. You would then look "around" that error for other interesting lines of trace... and sometimes it can be very obvious, and sometimes NOT very obvious. Warnings are logged in a similar manner to errors, and if you search for ~W you will see :

567 21:24:46.245  [12484] (ArchiveTask) <10888> EV~W Event ID: 3379 There were too few archivable items available to take mailbox '/o=First Org/ou=T1/cn=Recipients/cn=John1' below the quota-free level of 40.| |Quota limit: 80000KB| |Quota used: 52108KB (65 of quota limit)| |Previously archived (pending): 0KB| |Required to archive: 4108KB (adjusted 4100KB)| |Amount archived (this run): 8KB| | You may need to review your mailbox archiving policies to allow more items to be archived if you regularly see this warning for mailboxes. |

You can do ~I too.

 

  1. The last thing you might search for is the word Exception.  Here is an example:

223 21:24:43.510  [11304] (StorageOnlineOpns) <14236> EV:H CSaveset2::get_DocumentType _com_error exception. hr=%1 could not be found.  [0x80030002]

It is worth noting that there are some common ones (this being one of them) which are completely normal.  So it's not got a good success rate, and is something you're likely to do with the assistance of Symantec Support. Of course this can be quite cumbersome in Notepad, but there are a number of other alternatives, such as TextPad.  This and other tools will be covered in a future article.

 

Filtering

Filtering is a powerful way of trying to trim down what you’re looking for, if you want to track something over time, or, you’re looking for what happens on a particular part of a task as it repeats over each item that it touches or each mailbox/FSA target, etc.

It takes a little time and patience to put together the desired filtering, and requires some practice.  The basis, of course, is the original “big” DTRACE file that you would have already generated by now.  The problem with the big file is just that, it’s too big for handling easily.

In this simple example I will explain how to set up a filter to limit the trace file to contain information relating to policy settings on particular folders in a box.  For this we want to:

  1. Launch DTRACE from the command line
  2. Type set arvchivetask v followed by the Return key
  3. Type filter followed by the Return key
  4. Type Clear Includes followed by the Return key (or C I for short)
  5. Type + "Folder Policy Settings"  followed by return
  6. Type exit to return to the main DTRACE settings
  7. Type log c:\temp\fps.log
  8. Now run the archiving task for a single user
  9. Type log
  10. Open the log file which was generated and you will see entries like this :

3,155 16:36:05.104 [3,972] (ArchiveTask) <10096> EV:M EFP - Folder Policy Settings for [Inbox]

  Filter Type = Using users default policy

  Archive Settings Locked = TRUE

  Override Locks = FALSE

  Delete Original = TRUE [LOCKED]

  Create Shortcut = TRUE [LOCKED]

  Archive unread items = TRUE [LOCKED]

  Archiving based on : Age [LOCKED]

  Age Based Period 0 Days

  Not using Size Priority Settings

3,249 16:36:05.323 [3,972] (ArchiveTask) <10096> EV:M EFP - Folder Policy Settings for [f1]

  Filter Type = Using users default policy

  Archive Settings Locked = TRUE

  Override Locks = FALSE

  Delete Original = TRUE [LOCKED]

  Create Shortcut = TRUE [LOCKED]

  Archive unread items = TRUE [LOCKED]

  Archiving based on : Age [LOCKED]

  Age Based Period 0 Days

  Not using Size Priority Settings

 

.. and so on for each folder in the mailbox.

As mentioned above filtering is very powerful.  The above is just a simple example.  Filtering is usually something you head towards once you know what sorts of information you want to capture - and that's an art as much as a science!  There are also triggers which can be used.  At a high level these allow tracing to start when a particular entry is encountered in the tracing.  Filtering and Tracing will be expanded on in a future article covering more advanced DTRACE techniques.

 

Build your own INI file

As you can see in the VAC there a number of predefined trace files that you can run, but you can also write your own, and store them for using in the future.  This is how you do it:

  1. In Windows Explorer navigate to the subfolder of the Enterprise Vault program folder called Scripts for DTrace
  2. Pick an existing file and copy it, and then name it something helpful - for example name it : FPS Investigation Trace
  3. You then edit this file to include the steps that you would normally run manually.  In this example the end result will look like this :

[Description]

Use this script for looking at Folder Policy Settings

 

[Commands]

set ArchiveTask v

filter

c i

+ "Folder Policy Settings"

exit

log ?

  1. In the VAC right click on the Trace node under the server, and choose to do a New trace file
  2. From the drop down list of types of trace you can generate choose the one you just created, and walk through the rest of the wizard. When this trace file runs, a filtered trace will be created to help investigate our example issue.

As this is a complex topic, further examples will be given in a future article.

 

Conclusion

As this introduction will have shown you, DTRACE is a powerful diagnostic tool that you'll be able to use usually in conjunction with Symantec Support.  It is often necessary to run the same tracing multiple times, and for this purpose the settings that you put in DTRACE can be stored in a file, and re-run from the VAC as required.  The files you get can be big, and depending on the problem being investigated it might be necessary to engage in creating filters to refine what DTRACE reports on.  Running DTRACE can have a negative effect on performance at times, so it isn't something that should be left running long time.

Comments 6 CommentsJump to latest comment

Bruce Crankshaw2's picture

Nice one Rob

Very interesting

0
Login to vote
Rob.Wilcox's picture

Thanks - Part Two is in the works.

0
Login to vote
CBailey's picture

Rob - Both articles on DTRACE are full of great information. Your knowledge and the transfer of it to us is greatly appreciated!

0
Login to vote
Rob.Wilcox's picture
0
Login to vote
SymGuy-IT's picture

Rob rocks!!!

0
Login to vote
Rob.Wilcox's picture

Thanks - my fan base is slowly increasing :)

0
Login to vote