by Hal Flynn
When considering the security of a system we need to be concerned not only with which features and tools we use to implement the access restrictions, but also with what logging of access we do.
Logging is important for two main reasons: regular analysis of our logs gives us an early warning of suspicious activity and, if stored securely it can provide the evidence required to find out what went wrong when a breach in the security policy occurs. There are other areas where logging helps as well, such as analysis of our security policies for correct implementation, as well as debugging auditing that can report pertinent information to our security model. Solaris provides a rich logging system available as part of the core OS in the form of SunSHIELD BSM Auditing. This is one of the most powerful security features that Solaris provides out of the box, yet it is probably the least understood and least used.
This article will give an overview of what Solaris BSM auditing can do and will give some examples of implementing common auditing policies. In a future article we will cover how to use auditing as an application and kernel developer.
So, what is BSM auditing? First, it is not syslog. The audit trail is written to binary files on the local system (or NFS mounts). The system provides two utilities for filtering (auditreduce) and viewing (praudit). Audit records are initiated from two distinct places in Solaris: privileged user land programs (such as login) and the Solaris kernel. All security sensitive kernel system calls will generate an audit record when BSM auditing is enabled.
The following user land programs in Solaris can write audit records:
/bin/login (rlogind, rshd, telnetd) /usr/bin/su /usr/bin/newgrp /usr/dt/bin/dtlogin /usr/sbin/in.ftpd /usr/sbin/rexd /usr/sbin/in.uucpd /usr/bin/passwd /usr/sbin/allocate /usr/sbin/deallocate /usr/sbin/mountd /usr/sbin/crond /usr/sbin/init /usr/sbin/halt /usr/sbin/uadmin
Adminsuite v2.3 and v3.0 also do auditing via BSM for account and host maintenance.
The first step in using BSM auditing is enabling the kernel support and ensuring auditd is started at boot time. To do this you need to run /etc/security/bsmconv (either as root or a user that has been given the Audit Control RBAC profile in Solaris 8) since auditing is not enabled in the default Solaris installation. Running bsmconv not only enables auditing but also sets up device allocation (which disables vold(1M)) and disables Stop-A. It disables Stop-A by putting "set abort_enable = 0" into /etc/system. If you don't wish Stop-A to be disabled or if you have already done this by updating /etc/default/kbd (Solaris 7 and above), you can remove this line from /etc/system. If you don't want to use device allocation and want vold(1M) to continue to run then move /etc/security/audit/spool/S92volmgt back to /etc/rc2.d/.
After bsmconv has been run the system needs to be rebooted so that the c2audit module is properly loaded and the internal audit settings and structures are setup. Before rebooting it is wise to setup the /etc/security/audit_control file to say what auditing we want - this file can be updated without a reboot but it is good practice to set it now before the first reboot.
The main configuration file for auditing is /etc/security/audit_control, in this file we set which classes of events we want to generate audit records and where we want those records to go.
Example: "Login" Events
To record the "login" events for all users add the class `lo` to the "flags:" line of /etc/security/audit_control - don't worry about the other lines in there just now we will come back to those later. The login events are created by login (telnet, rsh, rlogin), dtlogin, in.ftpd, su, rexd, in.uucpd. For example:
dir:/var/audit flags: lo minfree: 20 naflags: lo
An example successful event for a remote login from hepcat:
header,81,2,login - rlogin,,Wed Aug 27 09:46:53 1997, + 511485295 msec subject,darrenm,darrenm,techies,darrenm,techies,10100,10100,24 5 hepcat text,successful login
An example failed login event when coming in via ftp from netwon:
header,77,2,ftp access,,Wed Sep 03 16:56:30 1997, + 712178483 msec subject,darrenm,darrenm,techies,darrenm,techies,1200,1200,0 20 newton text,bad password return,failure,1
Including `lo` on the flags line will log events regardless of whether it was a success or a failure, if we only want to log failures then like all classes we put a - in front of the class name.
It is good practice to have both success and failure login events in the audit trail regardless of what we audit, as this will help to provide context to us humans for everything else we look at in the audit trail.
Example: Logging All Commands
Some sites require as part of their security policy that all of the commands run by a user are logged. There are many insecure solutions touted for this requirement that involve using the shell .history file and putting it in a place the user can't see etc. The only sure way to do this is to intercept the execve(2) call and log at that point, this is what BSM auditing does when we turn on the class `ex`. The events get logged by the kernel implementation of execve(2) so no changing of LD_LIBRARY_PATH or other user configurables can bypass this.
An example auditrecord for the class `ex`:
header,103,2,execve(2),,Thu Jun 25 11:39:32 1998, + 52420844 msec path,/usr/bin/ls attribute,100555,bin,bin,8388608,0,0 subject,darrenm,root,other,root,other,8722,408,0 0 braveheart return,success,0
This shows that the user darrenm run /usr/bin/ls as root on the host braveheart on 25 Jun 1998.
By default only the command is logged to the BSM audit trail. If you wish to have the command arguments logged as well then we need to change a policy in the audit system. To do this run:
# auditconfig -setpolicy +argv
This instructs auditing to log the arguments to commands. This takes effect immediately but to ensure that it is set on each system boot add the same line to /etc/security/audit_startup. The environment variables in effect at the time can also be logged by adding the +arge policy (see auditconfig(1M) for more details on audit policies, though this is less useful.) Note that this is the command line as seen by the execve(2) system and may not reflect exactly what the user typed on the command line because of shell matching and globing.
Often sites wish to log all of the commands that the root user runs believing that this gives them a trail of everything that went on. There is a big caveat here: if you have root access to a host, you can turn off auditing - doing so does generate an event but the logs can always be modified/destroyed. If this is an important part of your site security policy, you should look into using write-only media for storing the audit log files. You should also consider having a warning system external to Solaris that detects when the write-only media is disconnected from the OS.
The /etc/security/audit_event file and the audit_control(4) man page describe the other audit classes available. Turning on classes such as fr for file reads will generate a lot of audit data even on an system with low usage. It is not possible to audit access-only to specific files in Solaris but auditreduce can filter the audit trail to show only the files you are interested in.
A recommended minimum set of classes is: lo, ad, na. Which includes login/out events (lo), admin events (ad) such as filesystem mounts creation of users and non-attributable events (na). These include Stop-A, which we can't be sure was done by any particular user.
Configuring Auditing On a Per User Basis
So far we have dealt with auditing at the system level so all users are audited equally. For reasons of disk space or organization policy it is sometimes necessary to have a different policy for particular users. Setting flags in the audit_control file applies them for all users on the system. To set audit flags for selected users we use the /etc/security/audit_user file. The file has the following format:
username:always audit flags:never audit flags
As of Solaris 8, audit_user(4) can be stored in the nameservice - note that audit_user is normally not world-readable so, storing it in a nameservice may reveal important system policy information that is not available when using files. The audit_user source is not listed explicitly in the nsswitch.conf but follows the same search order as used for the passwd entry.
Audit trail analysis
To convert the binary audit trail into human readable ASCII, the praudit(1M) command is used. praudit uses the information from the audit_event and audit_class files together with data from the nameservice (via getXbyY() calls) to turn raw events, uids, IP addresses etc into text. It is important to remember that the binary audit trail stores raw uid, gid and ip addresses so if uids or IP addresses are used later for different names some further context from your own administration change system may be required to identify the correct user or host. In general, I recommend never to reuse uids or gids. There is plenty of space for unique values for everyone who ever comes through your organization.
praudit has a few basic options that determine single or multi-line display and delimiters but provides no mechanism for choosing which events get displayed. Choosing the events is done by using auditreduce(1M). The auditreduce(1M) command is often thought to be overloaded to performing two functions 1) audit record selection 2) audit trail managment. In actuality, these are one in the same. auditreduce takes binary audit trail(s) as its input and generates a new binary audit trail as the output.
If we are using auditreduce to get a selection of audit records, such as the commands run by a user during a given time period, and we want to display those records, then we would probably use the output of auditreduce and pipe it directly to praudit. For example, to find all of the login events for user alice in October 2000:
# auditreduce -a 20001001 -b +31d -u alice -c lo | praudit
This says to auditreduce, start processing records from 1st October 2000 and before 31 days (so until the end of 31st October) this forms a range. We then specify the username using -u and finally we say the audit class of login events `lo`. If we didn't pipe this to praudit we would get a binary audit trail as standard output or in the file specified using the -O flag.
So where did the input data come from? Unless instructed otherwise, auditreduce will read all of the audit files under /etc/security/audit and process each one looking for records that match the required selection criteria.
Using auditreduce multiple times to drill down can be a very effective tool: instead of translating the output using praudit, the binary file is preserved on each search. Rather than give lots of further examples, I suggest reading over the auditreduce(1M) man pages to get a feel for what you can filter on. The important thing to understand at this point is that auditreduce conducts selection of records and praudit simply displays them in a human readable form.
Managing the Audit Trails
Audit records are actually written to files by the kernel rather than directly by the process being audited. There is a userland daemon, auditd(1M) that tells the kernel which file to write to and does basic management to ensure that there is space to write the audit records.
The name of the current audit file is in /etc/security/audit_data. This file has two fields ":" separated, the first is the PID of auditd, the second is the full path name of the active audit file. For example:
# cat /etc/security/audit_data 431:/etc/security/audit/talisker.0/files/20001031192753.not_terminated.talisker
The location of the files is determined by the "dir" entries in /etc/security/audit_control. For example:
dir:/etc/security/audit/talisker.0/files dir:/etc/security/audit/talisker.1/files minfree: 20 flags: lo,-ex,-ad naflags: lo,ex,ad
We mentioned in the previous section that auditreduce(1M) looks for audit trail files under /etc/security/audit if no files are given on the command line. For this reason it is recommended to have all filesystems that are used for audit files mounted under /etc/security/audit. To use the name of the host followed by a number is the normal practice, but anything that means something to you can be used. Note that it is possible to have NFS-mounted directories but they will have to be shared with root access to the client doing the auditing. Since auditd(1M) runs as uid 0 - this is different to the audit system in SunOS 4.x where auditd ran as the audit user, the reason for this change was to support having Secure NFS-mounted filesystems used for auditing (there is no way to ensure that the key for the audit user would be in keyserv at startup so root is used instead).
On startup, or when instructed to start a new audit file by running audit -n, auditd(1M) will create the file in the first listed directory with at least minfree percentage space available. Timestamps are of the form %Y%m%d%H%M%S (as defined in date(1)). The phrase "not_terminated" in place of an end timestamp means that auditd has not closed this file. In normal operation, there is only one "not_terminated" file per host, but if the machine should panic or lose power it is unlikely that auditd would have a chance to close and rename the file.
The following example shows the audit records for the host talisker between October 30th and November 19th. Note that the timestamps are in GMT not local time, so you may see "future" dates for the time zone you live in.
20001030225810.20001031192753.talisker 20001031192753.20001119043210.talisker 20001119043210.not_terminated.talisker
Best practice dictates that each directory listed in the audit_control file should be a separate filesystem and should be used only for audit records. Since it is a normal UFS filesystem, you can use logical volume management software to mirror the filesystems and protect your audit data and your normal backup software to preserve and archive the data.
Once a directory has less than minfree percentage space remaining, auditd will start a new audit file in the next directory with enough space (ie less than minfree). On doing this it runs the /etc/security/audit_warn script which sends an email to the members of the audit_warn alias. When each filesystem listed has reached minfree it will then start back at the first and fill it until no space is available - a further warning will be sent by audit_warn saying that the hard limit has been reached.
Once all filesystems have been filled another audit policy comes into place. At this point in time, one of two things should be done: the audit records that can't be written should be dropped, but a count should be kept and when space is available a entry saying how many lost audit records there were should be written. For some sites this is not acceptable and it is better to stop the system from functioning until space is available. To change this policy from the default of count to suspending processing run:
# auditconfig -setpolicy -cnt
To make this the default, remove the line in /etc/security/audit_startup that has -setpolicy +cnt.
Note that auditconfig(1M) says that suspending is the default. This is true; however, the configuration setup by running bsmconv adds an entry to audit_startup that sets the count policy so the default for Solaris is actually to count rather than suspend.
Summary
This article has discussed the basic setup of SunSHIELD BSM auditing and basic analysis and management of the audit trail. The configuration of BSM, details of a working configuration, and management of the configuration were covered. And finally, links were provided to further the knowledge and sharpen the learning curve of the reader.
This article originally appeared on SecurityFocus.com -- reproduction in whole or in part is not allowed without expressed written consent.