A few months ago I discovered rsyslog, a drop-in replacement for the standard Unix syslog daemon. Rsyslog can do a lot more than I will talk about, but the major appeal to me was built in mail sending.
Since rsyslog is completely compatible with syslog.conf, I started by cp /etc/syslog.conf /etc/rsyslog.conf. Then to make a few specific changes. First, we should know that you can split your config directives out to multiple files and read them in independently. You can include multiple files or whole directories.
$IncludeConfig /etc/rsyslog.d/
I commented that line out, since all my config will be in one file: rsyslog.conf
Next, let’s allow some remote connections to use this daemon for logging:
$AllowedSender UDP, 127.0.0.1, 192.168.1.0/24
$AllowedSender TCP, 127.0.0.1, 192.168.1.0/24
$UDPServerRun 514
This allows local loopback connections and the LAN.
Now, we get into the heart of rsyslog, modules and templates. First, we will load a module allowing the import and logging from klogd, the Linux kernel logging daemon:
$ModLoad /usr/lib/rsyslog/imklog.so
And then, a module to allow rsyslog to communicate to SMTP servers directly:
$ModLoad /usr/lib/rsyslog/ommail.so
Since these are the only modules I load, let’s go ahead and configure the directives for ommail:
$ActionMailSMTPServer airwaterunix.org
$ActionMailSMTPPort 25
$ActionExecOnlyOnceEveryInterval 1
$ActionMailEnableBody on
$ActionMailFrom alert
The options are pretty straightforward. The third option there can be increased to keep rsyslog from slamming your mail server, and the fourth option is incase you only want a subject line sent (in the case of SMS). The “alert” in the last option is the name of a specific mail alias on the server that points to root.
Now it’s time to set up some templates:
$template TraditionalFormat,”%timegenerated% %HOSTNAME% %syslogtag%%msg:::drop-last-lf%\n”
This one won’t apply to mail. It’s just the standard template to use for the messages that appear in the log files.
Next:
$template mailSubject,”%programname% on %hostname%”
$template mailBodyShort,”\r\n\r %timegenerated% \r\n\r\n %msg%”
$template mailBody,”\r\n\r %timegenerated% \r\n\r\n %hostname% %programname% %msg%”
A generic template for mail Subject, consisting of messages that will appear like “ssh on airwaterunix,” and two types of mail Body templates. One truncated style for SMS, and one longer one for email. I could have multiple Subject templates also, but it’s not really necessary at this time.
Now that we have defined the templates, we have to enable a Subject:
$ActionMailSubject mailSubject
Now come the conditionals. For every 'if...then' style line we use, we must define the mail recipient beforehand, as that directive is reset after the loop closes. So for our first recipient and conditional:
$ActionMailTo alert if ($fromhost == ‘airwaterunix’) and ($msg contains ‘Failed password’) or ($msg contains ‘eth0: link up’) then :ommail:;mailBodyShort
Simply, the logic works like: if
A
and
(B or C)
then
ACTION
So for this to be met, it must be happening on host airwaterunix, and the message must either be “Failed password” or “eth0: link up”
These two messages have nothing in common, except they are two messages for which I would like to trigger an email to user ‘alert’ (which happens to be an alias not only to root, but my SMS number as well). And since these are going to a phone, we use the mailBodyShort template so everything fits in one message.
Next, a slightly less alarming notifier:
$ActionMailTo admin if $msg contains ‘Authentication failed’ then :ommail:;mailBody
In this case, ‘admin’ is an alias to root and nothing else. This string will only pass through rsyslog if an incorrect password was entered via the ’su’ command. Since I typo this one from time to time, I don’t want an SMS every time I do it. An email will suffice.
Let’s add one more similar one, but this time to handle failed logins from the console:
$ActionMailTo admin if ($programname contains ‘login’) and ($msg contains ‘invalid password’) then :ommail:;mailBody
Next up, handling specific messages from smartd, the SMART disk monitoring daemon:
$ActionMailTo admin
if ($programname == ’smartd’) and
($msg contains ‘RawReadError_Rate’) or
($msg contains ‘ReallocatedSectorCt’) or
($msg contains ‘SeekErrorRate’) or
($msg contains ‘SpinRetryCount’) or
($msg contains ‘Reported_Uncorrect’) or
($msg contains ‘AirflowTemperatureCel’) or
($msg contains ‘Temperature_Celsius’) or
($msg contains ‘HardwareECCRecovered’) or
($msg contains ‘Offline_Uncorrectable’) or
($msg contains ‘UDMACRCError_Count’) or
($msg contains ‘MultiZoneError_Rate’) or
($msg contains ‘TAIncreaseCount’)
then :ommail:;mailBody
Note: Even though I split this up for legibility, rsyslog will not work unless you put your entire 'if...then' conditional on one single line.
Finally, we’ll add one to handle failed ssh logins on a secure remote server:
$ActionMailTo ssh-alert if ($fromhost == ’seraphim’ ) and ($msg contains ‘Failed password’) then :ommail:;mailBodyShort
In this case ’ssh-alert’ is an alias of several email addresses and SMS numbers that will all be notified in the event that an ssh login failure occurs on host “seraphim.”
The rest of rsyslog.conf is lines like:
.info;.warn;authpriv.none;cron.none;mail.none;news.none -/var/log/messages;TraditionalFormat
to handle the actual text file logging of messages, specified by a previously defined template.
