Workflow - Using Syslog, Delimited Text, Monitoring Projects, and Web Services to Avoid Late Night Phone Calls

Created: 09 Mar 2009
Login to vote
+13 13 Votes

There's a whole slew of network and security management products that can send events to a syslog server. But, what person in their right mind wants to stare at a scrolling log file all day? What's worse, most syslog applications stop being useful if you want them to do something about the problem. You should expect that they will filter the incoming log entries, write them to a file or database, and maybe send an e-mail if they match a pattern. By using Workflow as a processing engine, you can do something intelligent with those events. For example:

  • Schedule a job through the Altiris Task server to fix the problem
  • Forward specific events to a network management console
  • Assign a human task to fix the problem
  • Fill out a service desk ticket after looking up the server's owner and inventory in the Altiris CMDB
  • Update that ticket with additional events from the same server
  • Compare the current event against other recent events to gain perspective (correlation)
  • Traditional syslog server activities (Write to a log file, database, etc)
  • Or here's a wacky idea, check to see if that server is in the middle of a maintenance window before getting someone out of bed at 2am!

By incorporating the free version of Kiwi Syslog Server for Windows, we can achieve our integration goal. The remainder of this article focuses on a way to get syslog events into a Workflow process. Because this method relies upon reading lines from a single log file, it is sufficient for lighter syslog operations, but it would be insufficient for a large environment with thousands of incoming messages. In that situation, it would be appropriate to use the premium version of Kiwi, which can insert each incoming message into a database table. Workflow would then retrieve them from the table (instead of the log file). I've intentionally separated the input module from the processing module to make that transition possible.

Even if you are a Workflow expert, there's a good chance you have never used a Monitoring project.  In this article we will also use one project to invoke another (through webservices), parse a delimited text file (csv), and perform multiple file operations. This article assumes you have created a few Workflow projects in the past .  Let's get started!

 

Architecture

Below is an illustration of what we are building.  We will simulate an external system sending a syslog message to the Kiwi Syslog Daemon. A Workflow monitoring project will read the resulting Kiwi log file, then invoke a second Workflow project's web service with the contents of each new message.  The second Workflow project will then analyze the message and send an email if it meets a certain criteria.

 View Inline Image

Setup the Syslog Server

  1. Download and install the Kiwi Syslog Daemon onto the Workflow Server (or a dedicated server).  Note:  Under current licensing (Workflow 6), you should keep Kiwi on your Workflow Server.  If you run Workflow Designer on a separate computer, then you will need to map a drive to this server when developing your workflow code. I'll start referring to this program simply as Kiwi.
  2. Open the Kiwi Service Manager > File > Setup
  3. In the Setup window, open > Rules > Default > Actions
  4. By default, all incoming messages are written to C:\Program Files\Syslogd\Logs\SyslogCatchAll.txt. We need dedicated access to our own log file, but we don't want to tamper with this primary log file. Instead, we will create a parallel log file action.
    Right-click on "Log to File" and choose "Copy Action"
  5. Right-click on the Actions branch and chose "Paste Action"
  6. Right-click on the new "Log to file" action and choose "Rename Action". Set to "Log to file for Workflow" and press Enter
  7. Change the log file name to C:\Program Files\Syslogd\Logs\SyslogForWorkflow.txt
    View Inline Image
  8. Click the OK button to save the new settings
  9. To verify, Select File > Send test message to localhost
  10. In the log directory, there should now be two files, each with the same new test message similar to:
    View Inline Image
    2009-01-23 19:26:31    Local7.Debug    127.0.0.1    Kiwi Syslog Daemon - Test message number 0001

Create an Integration project to parse delimited text

The first thing we need is a custom component that can parse the log file from Kiwi and create a variable for each row.

  1. From Workflow Designer, create a new Integration project "Kiwi_Log_integration"
  2. Choose the Flat File > Separated Values Generator
  3. In step 1 of the wizard
    • Set the file to the location of SyslogForWorkflow.txt 
    • Uncheck the "First row is field names" box 
    • Change the separator to (TAB)
    • Set Default Encoding to US-ASCII
      View Inline Image
  4. For steps 2 and 3, click next
  5. For step 4
    • Update the Namespace to something that matches your company "Symantec.Community.SyslogText"
    • Update the TypeName to "SyslogMessage"
    • Accept the new names for the Read and Write Components
    • Set the Components Category to something that matches your company and the purpose of the components (I like to use the same value from Namespace) "Symantec.Community.SyslogText"
      View Inline Image
  6. For Step 5 set the fields as depicted below (matches the Kiwi Documentation, not necessarily the actual syslog specification):
    View Inline Image
  7. Click Finish to close the wizard
  8. Click the Compile and Close button

Create a Monitoring project to read the log file

Next, we need a background service that runs on the syslog server.  It will check every few minutes for new log entries, parse each row, then invoke another workflow project for each message that we find interesting.

  1. From Workflow Designer, create a new Monitoring project named "Syslog_Logfile_monitor"
  2. Add a new Project property "InputSyslogFilePath" and set the value to "C:\Program Files\Syslogd\Logs\SyslogForWorkflow.txt"
  3. On the canvas, delete the Create Notification Server Credentials component
  4. Add the FileExists component after the Start component
    • Setup tab > Path for Check = [[ProjectProperties].InputSyslogFilePath]
    • Connect "Fatal Exception" and "File not found" to the End component
  5. Add the Read File component after the "File found" path from the previous component
    • Configuration tab > File Path type = Full Path
    • Configuration tab > Path and FIle Name = [[ProjectProperties].InputSyslogFilePath]
    • Configuration tab > Output File Variable Name = syslogFile
  6. Add the Delete File component to the chain.  We do this to avoid re-reading the same messages next time. Because we configured Kiwi to create a duplicate log file, there is no true loss of data. However, this is a weakness in our design.  In a very busy environment, it's possible to overlook  an unread message that was inserted immediately after we read the file into memory, but before we deleted the old log file.  For reference, Kiwi will create a new log file when it receives the next message.  This is why the database option is more reliable.
    • Output tab > File Name = [[ProjectProperties].InputSyslogFilePath]
  7. Import Components button > Custom library tab > Add the "Kiwi_log_integration.dll"
  8. Add the SyslogMessageRead component to the chain
    • Output tab > Exceptions Variable Name = SyslogReadExceptions
    • Output tab > Results Variable Name = SyslogMessages
    • Setup tab > File Data Type Variable Name = syslogFile
    • Connect the "Conversion Failed" path to the End component
    • Temporarily connect the "Conversion done" path to the End component
  9. Ensure your model looks like the following:
    View Inline Image
  1. Launch the debugger and try running the project. 
    • If you receive an error referencing a mismatch on the number of csv fields, then you probably forgot to set the encoding type to "US-ASCII" back in the delimited text integration project.  Correct the integration project, then reload this one.
    • On the Execute Log tab, scroll down to the bottom.  You should see a SyslogMessage array similar to the following:
      Data Name: SyslogMessages
           Values:
               [0] Type: SyslogMessage
      EventTime: 1/23/2009 11:46:47 PM
      Priority: Local7.Debug
      ServerName: 127.0.0.1
      Message: Kiwi Syslog Daemon - Test message number 0009
      
  2. We are done with this project for the moment.  Before we can finish it, we have to create a couple of more projects, then connect them together.

 

Create a Workflow project to process incoming messages

We are now ready to implement a message processing engine.  Unlike most Workflow projects, this one won't start with a human initiated action.  The goal is to have this project passively wait for incoming messages from other sources.  To facilitate this, we will expose its web service so that external systems can pass messages directly to our engine.  For example, our monitoring project or a Monitor Solution 7 action (Call Web Service on Server task) .

  1. From Workflow Designer, create a new Monitoring project named "Syslog_Message_Engine_workflow"
  2. On the canvas, delete the Create Notification Server Credentials component
  3. Select the Input Data section (under the Model: Primary branch in the left pane)
  4. Add the following input parameters as depicted (Click thumbnail to enlarge). Setting Default Values will make our lives easier when running the debugger repeatedly. Notice that these Default Values are intended to make it obvious that they aren't real data. As with any programming language, this is a best practice to avoid mixing real and fictitious data so thoroughly that it can't be spotted and remediated if it accidentally shows up in production.

    View Inline Image

    Tip: When exchanging data between projects or models, anything beyond 4 parameters will quickly become cumbersome and error-prone to code. A better solution would be to use the Generator to create a custom data-type that represents a single message. That custom data-type would have four properties, corresponding to each parameter. The input parameter for the primary model would then specify only that custom data-type. This would also allow you to receive an array of messages per invocation, as opposed to the single message that we are doing for simplicity.

  5. The Priority portion of each message contains a severity code (Local7.Debug).  For demonstration purposes, we will extract this value and use it to help determine how to handle the message.

    Add the Split Text Into Collection component after the Start component

    • Configuration tab > Text to Split = [Priority]  (Process Variable)
    • Configuration tab > Delimiter = Other
    • Configuration tab > Delimiter Text = .  (That's a period)
    • Configuration tab > Output Variable Name = priorityText
    • Rename the component to "Split Priority Text Into Collection"
       
  6. Add the Matches Rule component to the chain
    • Evaluation tab > Compare to List as depicted below:
      View Inline Image  
    • Evaluation tab > Compare to Variable > priorityText[last]   (This is the last member of the priorityText collection that we created with the prior component)
    • Rename the component to "Severity Level"
       
  7. Add a Send Email component after the Debug path of the previous component
    • Email Settings > SMTPServer Name = yourSMTPServer
    • Email Contents > From Address = syslog@yourdomain.com
    • Email Contents > To Address = you@yourdomain.com
    • Email Contents > Subject = Syslog Debug message
    • Email Contents > Send Html Content = checked
    • Email Contents > Html Content = Drag all four input parameters into the email and design as you see fit. For example:
      View Inline Image Tip: In production code, you should never hard code the SMTP server or email addresses. Instead make them project properties to avoid future headaches when your mail server is replaced.
       
  8. Connect all remaining output paths of the Severity Level component to the End component. Your model should look similar to this:
    View Inline Image
  9. It's now time to test this project.  Launch the debugger and double-click the Execute method.  You should be prompted for the input parameters.  Since we already provided default values, all you have to do is click OK.
    View Inline Image  
  10. The execution tracker should follow the Debug path and trigger the debug email.
    View Inline Image

    View Inline Image  

  11. Now that we know the project is working, it's time to publish it to our Workflow Server.  This must be done so we can build a webservice integration project.  One the project is published, it will provide the WSDL that tells the generator how to build a custom component.

    From Workflow Designer select File > Publish Project > Publish Application to Server

    • Virtual Directory = SyslogMessageEngine
    • Deploy to = Check the box for your server
      View Inline Image  
    • On the Application Properties Editor, make sure the Base URL to Project is using the actual Workflow Server's dns name (FQDN) and port(if you aren't running a dedicated WF server).  =  http://myWorkflowServer:9001/SyslogMessageEngine
    • Click Save
    • Click Yes to open the published project.  Note: If you get a 404 error, it doesn't prove that application failed to publish, only that Worfklow Designer doesn't know the root URL configured.  Also, click No to skip publishing to the Altiris NS as a ServiceWorkflowItem
    • Because our web service only uses primitive data types (int, strings, datetime, etc), ASP.NET provides an interface that can be used from your web browser to manually invoke the web service.  Click the Execute link on the .asmx page to see this interface.  Feel free to try submitting your own fictitious syslog message.
    • Verify that you can retrieve the WSDL of the published application by appending the WSDL parameter to the application's URL in your web browser (or click the Service Description link on the asmx page):
      http://myWorkflowServer:9001/SyslogMessageEngine/MyService.asmx?WSDL

      The response should look similar to this (click thumbnail to enlarge):
      View Inline Image
  12. That's it.  Remember the WSDL URL for the next project.
     

Create an Integration project to invoke the web service

We now need a custom component that can invoke the Syslog Message Engine's web service.

  1. From Workflow Designer, create a new Integration project "Syslog_Message_Engine_WS_integration"
  2. Choose the Enterprise Resources > Web Service Caller Generator
    Tip: This is a perfect opportunity to point out the Multiple Generator Container generator.  For more complex projects, using this special generator would allow us to reduce the number of dlls we need to track.  By putting all our integration libraries inside one big library file, we can cut down on version confusion, and all the related projects can share the same custom generated components and data-types.  However, a tutorial usage is a topic for another article.  Feel free to play around with it, it's relatively straightforward.
  3. Choose URL for the Source Type, then paste the URL from the previous project into the Url section, For example:
    http://myWorkflowServer:9001/SyslogMessageEngine/M...

  4. Click the Add button, then Next
  5. On Step 3 (Namespaces and Categories)
    • Update the Namespace to something that matches your company "Symantec.Community.SyslogMessageEngine"
    • Set the Default Category to "Symantec.Community.SyslogMessageEngine.WS"
    • Change the Name pattern to = <methodname>
    • Default URL Generation = Use and populate Project property
    • Project Property name = SyslogMessageEngine_WebserviceURL
  6. On Step 4, uncheck everything with the exception of "Execute"
  7. Click "Compile and close"

Update the Monitoring project to send each message

It's now time to glue our monitoring and engine projects together.

 

  1. From Workflow Designer, open the Monitoring project "Syslog_Logfile_monitor"
  2. Add the For Each Element in Collection component
    1. Connect it to the output path "Conversion Done" on the Syslog Message Read component
    2. Connect the Finished path to the End component
    3. Rename to "For Each Syslog Message"
    4. Configuration tab > Array Variable Type = SyslogMessage (It's in the Kiwi_Log_Int library)
    5. Configuration tab > Array Variable Name = SyslogMessages (existing process variable)
    6. Configuration tab > Item Output Variable Name = currentMessage
  3. Click the "Import Components" button in the left pane
    1. Choose the Custom Libraries tab
    2. Select the Syslog_Message_Engine_WS_integration.dll
    3. Click the Add button
  4. Add the new component Symantec.Community.Syslogmessageengine.WS.Execute to the "next element" path of the previous component
    1. Rename to "Invoke SME Webservice"
    2. Connect the output path of this component back into the previous component (forming a loop)
    3. Params Usage tab > Set all parameters to required
    4. Configuration tab > Event Time = [currentMessage.EventTime]
    5. Configuration tab > Message = [currentMessage.Message]
    6. Configuration tab > Priority = [currentMessage.Priority]
    7. Configuration tab > Server Name = [currentMessage.ServerName]
  5. Your completed model should look similar to the following:
  6. Try out the updated project in the debugger.  You should see it run through the loop for each row in your log file. You should receive an e-mail for each Debug level priority message that is processed by the monitoring project.

    If you need more sample log data, open up the Kiwi manager, and press CTRL-T several times.  After a slight delay, a new syslog row will be added to the target log file for each key press (It will also recreate the file if it was deleted by our monitoring project). 

  7. We now need to prepare the monitoring project for deployment. Go to the Publishing tab of the project. 
    1. Set the Deployment Type to Windows Service
    2. Set Windows Service name to "Syslog Monitor"
    3. Set the Schedule to "Every Day Pattern" > Edit > Interval to 00:01:00 (Every minute, vs the default of every hour)
  8. From within the Monitoring project, select File > Publish Project > Publish Application to Server
    1. Virtual Directory = Syslog_Logfile_monitor
    2. Deploy to = Check the box for your server
    3. Click OK
       
  9. The last step is to fix the file permissions.  When you ran the project in the debugger, it used your credentials to access the Syslog log file.  When the project is deployed, by default, it will be running under the "Network Service" account.  The Network Service account is heavily restricted, and won't have access.  You might notice some errors in the Workflow Logfile (Start > Altiris > Workflow Designer > Tools > Configuration and Logging Tool).  To fix, use the following steps:
    1. Open Windows Explorer to C:\Program Files\Syslogd directory
    2. Right-click the Logs directory > Sharing and Security > Security Tab
    3. Click Add button > Type "Network Service" > Click OK button
    4. Ensure that the Full Control > Allow box is checked.
    5. Click OK to close the security window
       
  10. You should now have a functioning Syslog processing engine.  As a final test, use the CTRL-T shortcut on the Kiwi window to generate more messages, then check the designated email account for the corresponding notifications. If you encounter problems, restart the Workflow Server Extensions service and check the Workflow Logging tool for error messages.

 

Wrapping up

What's left to do?  Make this project into something useful by modifying the Syslog_Message_Engine workflow project to meet your needs.  Instead of just sending an email, parse the Message string for interesting items, then assign a Workflow Dialog Task, create a Service Desk ticket, and or write out the message to a database table for comparision against future syslog messages.

 

Reference Material

The BSD syslog Protocol
http://www.ietf.org/rfc/rfc3164.txt

4.1 syslog Message Parts

   The full format of a syslog message seen on the wire has three
   discernable parts.  The first part is called the PRI, the second part
   is the HEADER, and the third part is the MSG.  The total length of
   the packet MUST be 1024 bytes or less.

...

Numerical Code    Severity
          0       Emergency: system is unusable
          1       Alert: action must be taken immediately
          2       Critical: critical conditions
          3       Error: error conditions
          4       Warning: warning conditions
          5       Notice: normal but significant condition
          6       Informational: informational messages
          7       Debug: debug-level messages


Syslog protocol
http://www.networkdictionary.com/protocols/SyslogP...

No assumption is made upon the formatting or contents of the syslog messages. The syslog packet size is limited to 1024 bytes and carries the following information:

 

  • Facility – integers indicate the categories of sources that generate the syslog messages. These sources can be the operating system, the process, or an application.
  • Severity – single digit integers indicate the severity of the message,
  • Hostname - The hostname field consists of the host name (as configured on the host itself) or the IP address. In devices such as routers or firewalls, which use multiple interfaces, syslog uses the IP address of the interface from which the message is transmitted.
  • Timestamp: The timestamp is the local time, in MMM DD HH:MM:SS format, of the device when the message was generated.
  • Message - This is the text of the syslog message, along with some additional information about the process that generated the message.

 

Filed Under

Tags: