Endpoint Protection

Background

Even with the protection of a security perimeter, the fact remains that firewalls aren't foolproof and that potential attackers are hard at work, 24 hours a day. Furthermore, some organizations must allow remote logins to machines from sites outside their perimeter, meaning they must maintain a certain number of semi-exposed hosts that are vulnerable to attack. It is not enough to know that a system has been exploited. If an attack is detected, how can it be determine that a system has actually been compromised and important files removed or altered? One answer is to always run Tripwire, or one of its free-ware cousins, in order to detect changes in critical system files.

Unfortunately, this is not always an efficient solution. Back in the early Nineties, I worked for a well-known research institution in the western U.S.A. We experienced a series of hacks and mishaps stemming from incorrect machine configuration changes that led to internal and external embarrassment. These problems led systems us to try academic Tripwire. Unfortunately, it quickly became clear that Tripwire was a labor time-sink!

Given that running Tripwire may be a security necessity, it would be beneficial to do so as efficiently as possible. This paper will focus on ways to reduce the time and labor required to effectively operate run the Tripwire security and configuration monitoring tool. The information presented here is applicable to the version of Tripwire sold by Tripwire Security Systems Inc as well as the earlier "academic" (Purdue University) versions of Tripwire.

Tripwire: Boon or Bust?

Tripwire is much like the fabled elephant and the blind men: how you feel about it depends on the perspective from which you approach it. A person who has successfully used Tripwire to detect cracked binaries and/or system misconfigurations will have nothing but praise for it. On the other hand, someone who has been "stuck in the trenches" reading through endless reports in an attempt to find problems, will think that it's a labor-intensive waste of time. Minimizing the labor required dictates that reports be as brief, and as infrequent, as they possibly can be made. Using Tripwire on a day-to-day basis can be an uncreative and essentially boring activity. On the other hand, if one can reduce the torrent of data that Tripwire provides, and make it simpler to use than it is "out of the box", then using it can become bearable (if not necessarily palatable.) Fortunately, it is possible to reduce the time and effort required to administer Tripwire, as the next section of this discussion will illustrate.

The Time Sinks

More than a year experience in running Tripwire has made it pretty clear where the inefficiencies within the product lie, as of commercial version 2.2.1 (Linux) or 2.2.1A (Solaris).

History Mechanism

The single most important time efficiency issue with Tripwire is the lack of a report history mechanism, which would drastically reduce the number of reports. For instance, a dozen systems being checked three times per day can result in over 1000 reports per month, any one of which could contain the critical information the tool is supposed to detect. Even the most careful tuning cannot prevent this; for instance, the installation or modification of a large software package may suddenly result in a large report that will continue until the administrator has time to do a database update. This is especially true if an administrator cannot make policy tuning and database update his first priority.

Policy Tuning

Tripwire does not have the capability to automatically tailor rules in the policy file (to minimize report volume) by watching a system over a period of time in order to recommend rule changes. At install time, the distributed policy file per processor/OS type is just a static file that comes with the distribution. The install procedure does not customize the file based on simple criteria, such as the presence of certain files or filesystem structures revealed by the mount table. It also excludes some important files (e.g. the root crontab). Since the same template policy file per processor/OS type results upon install, a lengthy tuning process is required with each machine in order to minimize useless report output. Currently, there is no way around this problem other than to try to make the tuning process as quick as possible until most noise is tuned out. In the meantime, a history mechanism would significantly lessen the urgency of tuning.

Report Formats

Although the commercial Tripwire product has five report formats, none of them offers a maximally-abbreviated single-line format that provides violation type, filename, and changed attribute keys in a single line. (Tripwire's closest format is level 1, which provides only violation type and filename.) The default (level 3) format is a multiple-line format that is time-consuming to use. In many cases, the signature associated with a changed attribute isn't even of interest (for example, who wants to stare at an MD5 signature?) and, in many cases, the signature associated with a file attribute could change frequently and defeat history. So it is ultimately up to the system administrator to inspect the attribute or file in question before doing database or policy update. We parse the default Tripwire format into a single-line-per-file, 3-field format which looks like:

  	            a /etc/inet/core 		m /etc/passwd smCM 		r /local/etc/named.foobar  

Although file attribute values are currently not included in the report, file content signatures may be added at a later time if automated diffs are implemented (see "Automated File Diffs" below).

Ease of Maintenance

Tripwire database and policy file maintenance are made easier if the Tripwire admin does not have to remember argument switches and long filenames. Maintenance time is also a good time to enforce security issues that the harried or newbie administrator might forget, such as making sure Tripwire passwords are protected by doing an update on the system console or via a secure shell link.

Lack of Regular Expressions

The Tripwire policy file allows complete exclusion or lower security policies on directory trees. However, it does not allow exclusion or changed security policies on regular expressions for filenames. Directories in which pseudo-random filenames are created by utilities that are not always under the Tripwire administrator's control - other than places like /tmp where they are expected - create "noise" in report output.

Another aspect of the regular expression problems is that it is not possible to instruct Tripwire to report on the addition of filenames that have historically been associated with cracking activity, for example: "/tmp/\..*".

E-Mail Report Minimization

Tripwire now allows e-mail reporting to go to different addresses for different portions of a machine's filesystems. Unfortunately, smaller organizations don't have the staff to assign responsibility for a system to several people. The best approach for such small organizations is to have no more than one report per Tripwire check run, and to have one primary person responsible per system.

Reducing the Time Sinks of Tripwire

Despite the numerous shortcomings listed above, it is possible to reduce the labor and anxiety required to use Tripwire by taking these steps:

• Implement two wrappers, one for routine checking, and one for database or policy update;
• Enforce security in the database and policy update wrapper, e.g. require ssh or console use, and provide reminders about ro/rw disk use. Allow checking to be done as a side-effect of database update so that a new report baseline can be established in a single step;
• Provide labor-saving convenience features in the update wrapper. For instance, policy update (which is CPU intensive) can be easily re-entered so previous changes aren't lost if the administrator has to interrupt out;
• In the checking wrapper, parse the default report format into a single-line-per-file compact format at time of checking;
• In the checking wrapper, implement a simple history mechanism and file exclusions to reduce duplicate reporting. A simple (diff-based) mechanism proved to be as good as an earlier, more complex mechanism that considered more than two previous reports. We have chosen to disregard the complexity that would be introduced by using the capability to specify severity level or rulename at check time, which we haven't found useful in our environment.

Tripwire Security Systems Inc. makes the claim that the encrypted database and policy files reduce the major vulnerability in earlier versions of the product because a database cannot be altered by an intruder. This statement is true, and is, at the time this article was written, one of Tripwire's advantages over free-ware equivalents. However, the fact remains that a determined attacker who gains root can simply replace the Tripwire binary with a Trojan, and the effect will be the same. Once a Trojan is in place, it won't matter what is in the database, or even if it is present.

In deciding how to deal with vulnerabilities, we had to decide which risks seemed important, and which did not. The goal that was deemed worthwhile within the context of our manpower is to: make it possible to determine what changes were made, by manual means, if a break-in should occur on our most vulnerable (exposed) hosts. This means protection of the tripwire binary, the wrapper that calls it for checking, and any interpreter on which the wrapper is dependent. (The Tripwire binary itself protects against deletion of database and policy files.)

We chose to meet this goal on semi-exposed hosts that are not behind our security perimeter in the simplest possible way: use a switchable read-write/read-only disk that requires physical access to these hosts. This method absolutely protects the Tripwire binary/database/policy software tree. It does not protect reports, which must be on r/w disk and could therefore be deleted, but loss of these would mainly affect history. On hosts that are behind the security perimeter, this goal cannot be met without considerable uncertainty. Obviously, one must fall back on a software monitor that attempts to win a race with a cracker by notifying system administration personnel of changes made by the cracker before the tool itself is terminated by the cracker. The creation of a truly comprehensive Tripwire integrity monitor (discussed below) is beyond the scope of this project. On most Tripwire-monitored systems that are behind the security perimeter, we do minimal checking of the Tripwire system by using existing system monitor tools to independently check the signature of key Tripwire files on a much more frequent interval than Tripwire runs; ultimately the integrity of Tripwire on these systems depends on automatic backups.

Further Improvements

As has been stated throughout this article, the major issue with Tripwire is the reduction of labor. While the simple steps discussed so far will reduce the labor required to use Tripwire, a further reduction of labor is possible within the structure/limitations imposed by Tripwire architecture. It is also apparent that further gains in efficiency are necessary before Tripwire is quick and easy enough to use that people will not mind using it.

Protecting Tripwire

Protection of Tripwire, such that any attempt to change its configuration by a cracker would be detected, remains a problem, even when a Tripwire itself is protected by a read-only disk. Every protection one can imagine for Tripwire can, of course, be defeated. However, the fact remains that from the time that a cracker gets root on a system, until he shuts down cron or otherwise defeats Tripwire, there is a race in time. This means that a small, specialized system monitoring tool dedicated purely to Tripwire protection which is non-cron-dependent, runs frequently, and is camouflaged to the greatest extent possible, is probably the best bet for winning that race-in-time with a cracker.

The minimum requirements that a comprehensive Tripwire monitor would need, are:

• It should be written in a compiled language to make details of what it does difficult for a cracker to ascertain.
• It should run as a at a frequent interval as a daemon that is not dependent on cron.
• At startup, it should register with a daemon on a secure host, via an encrypted channel, and check in with that host everytime it runs, so that an alarm can be raised if it disappears.
• It must verify the signatures of important system files on which Tripwire depends (e.g. cron and sendmail binaries, and config files which they require), as well as signatures of files in the Tripwire binary directory.
• It must verify that important system daemons (e.g. cron and sendmail) are running.
• It must verify that a switchable rw/ro disk, if present, was not accidentally left in rw mode by a careless administrator.
• It must detect network outages or route loss that would prevent it from reporting via the default network method.
• In cases where normal network communication appears to have been lost, it must have a device that is close to write-only from the standpoint of the system CPU (e.g. split-screen system console, raw communication device which cannot be unconfigured, local printer, logging of the console serial line to a remote console manager) in which to log an alarm.

The following additional work is being currently being done. (The comprehensive monitor is excluded from this list.)

Abbreviated Reporting - On machines used for software development, update of a software package (located in an area subject to Tripwire monitoring) that resides under its own root location can result in a large e-mail report. In this case, very little information is provided by listing every changed file. This is being dealt with by generalizing the exclusions file to a check-time configuration file that contains exclusions and abbreviations. Abbreviations are filename wild-cards that cause a head directory name to be reported just once with a count of the number of files affected, if abbreviation mode is turned on at check time.

Read-Only Disk on Redhat Linux Hosts - Use of a switchable read-write/read-only SCSI disk requires that a SCSI reset be done when the protection switch is changed. Part of our development effort included writing a "remount" command that uses the SCSI reset capability available in user-level as of Solaris-7. (This feature had previously been used on IRIX-6 machines; IRIX-6 provides a canned user-level command for SCSI reset.) We are currently investigating to what extent a ro/rw disk can function under Linux.

Speedier Database Update - Database update and re-check requires excessive elapsed time because the default update method requires an interactive editing step. We are currently investigating approaches that can place all of this activity in the background without unacceptable security risk, e.g. without Tripwire passwords visible in the process table.

Automated File Diffs. A change to an ascii text file, e.g. "/etc/passwd", creates an obligation on the part of the Tripwire administrator to determine if changes are benign or not. The obvious way to do this is to diff the file against a copy of the same file taken from a backup prior to the file's last mod date. Unfortunately, this procedure is very time-consuming and so administrators tend to skip it except in unusual cases. Since we have an automated backup system from which files can be retrieved by name, we are looking at development of a procedure that can produce the desired diffs for the administrator automatically, as an option to the checking procedure. If this proves to be feasible, it will probably be implemented only for files specified in the check-time configuration file, and file content signatures may be added to the abbreviated report format to support this.