If you want to monitor all actions on your servers, even those from root, there’s a pretty straightforward and easy solution on RHEL6. Let’s first analyze the different approaches:
- You can use pam_tty_audit to send all keystrokes to /var/log/audit/audit.log but this also logs passwords. The problem is that the kernel can not differentiate between normal and password text because that happens in userspace. There is a flag that can be set to hide characters on the TTY but if the kernel was to discard that data, then any admin could potentially hide itself using that flag so that’s not an ideal solution.
- You can use sudo to make sure admin actions are logged (the system auditor logs all sudo actions by default) but this does not prevent a user from running bash and then anything that gets done in that shell isn’t logged. If you know in advance the commands that the admin users can run (usually the case for operators), then you can create a fairly big and complex sudo configuration that meets your requirements. But for admins that are supposed to be able to launch any command, this is not a good solution.
- With the auditing subsystem and the auditd daemon, there’s a much simpler solution. By default, the auditd daemon audits all sudo and login events on the system, as well as SELinux AVCs. The results are logged into /var/log/audit/audit.log. Just by adding a few lines to the auditd configuration, you can audit all actions on the system, even those by root. This is the option we’re going to detail in this blog post.
On the system to be audited, make sure that the file /etc/audit/audit.rules contains the following lines:
# Buffer size
# Audit modifications to the audit configuration
-w /etc/audit/ -p wa -k auditconfig
-w /etc/libaudit.conf -p wa -k auditconfig
-w /etc/audisp/ -p wa -k auditconfig
# Log all process forks (to be filtered by /usr/bin/audisp-simplify)
-a exit,always -F arch=b64 -S execve -k logall
-a exit,always -F arch=b32 -S execve -k logall
Let’s explain the different configuration items. First, we tell the system to audit all changes to the audit configuration. This is because we don’t want root to modify the audit configuration to disable the logging of all actions. At least we’ll have an event that shows that the configuration has been tampered with. Modification of the audit configuration can be disabled by forcing the system to reboot in order to change configuration (see documentation). The three lines « -w path -p wa -k auditconfig » tell auditd to monitor those three paths for any modification. The -k parameter tells the system to associate the « auditconfig » keyword to those events. This is just to facilitate grepping the logs.
Then, the two last lines tell auditd to log an event every time the execve system call is called. This is the syscall that will necessarily be executed whenever a process forks. This way, you can trace every program execution on the system (with arguments). The reason why there are two lines for each architecture is because the syscall numbers are not the same for 32 and 64 bits architectures. Therefore, if you don’t specify the architecture, there’s a possibility that the kernel gets the syscall number wrong when looking it up. Again, the logall parameter is simply a keyword that will be logged with the auditing event and allows for easy grepping afterwards.
Download the utility audisp-simplify from http://sourceforge.net/projects/audispsimplifyplugin/ and save it to /usr/bin/audisp-simplify. This is a nifty little audisp plugin that will simplify the output of auditd logs, which are by default very verbose and difficult to read.
Make sure that the file /etc/audisp/plugins.d/simplify.conf contains the following lines:
direction = out
path = /usr/bin/audisp-simplify
type = always
format = string
This will make sure that process fork audit events will be filtered through the audisp-simplify plugin in order for the logs to take up less space and to go to /var/log/audisp-simplify instead of /var/log/audit/audit.log.
Create the empty file /etc/audisp/simplify.ignores. This file will contain rules to discard certain audit events. for example, if a high process fork rate application such as Nagios is running, then adding the following line to this file will discard all process fork events from processes that belong to the user nagios:
See the audisp-simplify script for other examples.
Restart the auditd service. You might need to yum install perl if it is not installed.
create 0600 root root
/bin/kill -HUP `cat /var/run/auditd.pid 2> /dev/null` || true
This will ensure that the file /var/log/audisp-simplify gets rotated correctly every Sunday and that it keeps access rights for root only since it contains sensitive information.
The system should now be logging all actions of all users, including root, to the file /var/log/audisp-simplify.
A few notes:
- If SELinux is activated, you will need to create a local policy. Please read the end of the /usr/bin/audisp-simplify script for detailed descriptions on how to do that.
- Performance considerations: a countinuous forking of processes at a rate of around 15 forks per second for 1.5 hours generates a VM CPU usage of around 4% (1% userspace, 3% kernelspace) and a log file of around 35MB on our standard hardware. Hence, performance should not be affected for standard servers. It could be beneficial to perform additional comparative benchmarks in case of a server with a build-type load (continuous integration / compilation of files, etc.) or Nagios-type load which has a high fork rate. In that case, taking advantage of the simplify.ignores file can help to lower the load.
- If /var/log/messages displays warnings like « audispd: queue is full – dropping events », increase the value for q_depth in /etc/audisp/audispd.conf. See https://access.redhat.com/solutions/25190 for more information
In order to make sure that an admin does not erase audit events, the output from auditd should be sent to a central logging server. We’re going to use the rsyslog imfile module to read the /var/log/audisp-simplify logfile and generate syslog events for each of the audit events. This way, they can be sent to the central logging server along with all the other syslog events.
Create the file /etc/rsyslog.d/centrallogging.conf:
This will create a listener that will transform each new line from /var/log/audisp-simplify into a syslog event of facility local6 and priority debug. We use debug priority because a higher priority will send these events to the /var/log/messages logfile, which would duplicate the information. We use local6 because it is unused. If it is already used by another application, another facility should be chosen.
On the central server, add the following lines to /etc/rsyslog.conf:
:syslogtag, contains, "audisp-simplify" -?tmplaudispsimplify;auditFormat
This will dump into the %HOSTNAME%/audisp-simplify file the messages from syslog, making sure that the format to dump is simply the message itself without any other syslog information. This way, we recover the original audisp-simplify logfile.