Background

Event driven network monitoring systems rely on acknowledgment for multiple reasons. These reasons include:

• To let other Zenoss administrators and users know that someone is aware of the problem and taking action.
• To suppress escalation of alerts.

To acknowledge an alert, a responder typically:

• Clicks the acknowledge link in the body of the default Zenoss alert email.
• Logs into the Zenoss server, navigates to the alert on the device or in the event manager, and acknowledges the alert.

This process works well when you're in the office or your operations are business hours only. Typically, this is not the case for most users of Zenoss. Most users have critical systems that require 24x7 monitoring and alerting. Additionally, managers and executives require escalation of messages from the on-call personnel through management and sometimes to executive management.

The acknowledgment model of Zenoss, in its current state, is limited. An alert will never escalate, as long as it is acknowledged. In a 24x7 monitoring and alerting model, an alert can and will come during non-business hours. The first responder (on-call personnel) is usually mobile and does not always have immediate access to a computer. To compound this, many Zenoss servers live on private networks that require VPN authentication to access them. There are times where the on-call personnel can not get into the system quick enough to prevent escalation from occuring. We all know how upset managers get when the on-call grunt doesn't do her job properly.

One possible solution to this issue is email based event acknowledgment. The on-call responder replies to the alert email on her PDA. The alert email is routed back to the Zenoss server for processing with Procmail. The body of the email is piped into a python script, via Procmail. The device and event id are extracted and the alert is acknowleged through Zenoss API calls. An email is sent back to the responder reporting whether or not the acknowledgment was successful.

The biggest caveat, for this model, is that incoming mail needs to be routed to your Zenoss server. As such, this will require knowledge of mail relay administration or you'll need to rope in your mail relay administrator to help you out. Since every company's mail setup is unique (Spam filters, Exchange Servers, Postfix Servers, et. al) I will leave this as an exercise for the user of this how-to. The second caveat is that your mail relay becomes a single point of failure for delivery. If your mail relay is down or in schedule maintenance, you won't be able to acknowledge events.

** This will work with plain-text, base64 encoded and multipart email types (Iphone, Blackberry, Exchange). I have tested this on Zenoss Core 2.4.5 and 2.5.0

Installation

Postfix Installation

Install Postfix from RPM or source.

Procmail Installation

Install Procmail from RPM or source and configure per the README.

Configuration

Postfix Configuration

Edit your configuration file to use procmail. This is typically located at:

vi /etc/postfix/main.cf

Edit mailbox_command to reflect the path of your Procmail installation.

mailbox_command = /usr/bin/procmail

Restart Postfix

/etc/init.d/postfix restart

Procmail Configuration

We will be filtering email at the user level, not the system level.

As the zenoss user:

vi ~/.procmailrc
chmod 600 ~/.procmailrc

Create a procmail directory to log to.

mkdir ~/procmail

Copy and paste these two sections into your .procmailrc. There can be space between them. The first section exports basic environment variables.

SHELL=/bin/bash
PMDIR=$HOME/procmail
LOGFILE=$PMDIR/pmlog
ZENHOME=/usr/local/zenoss
PYTHONPATH=$ZENHOME/lib/python
PATH=$ZENHOME/bin:$PATH

The next section is the Procmail formula. This says for any message that contains '[zenoss]' in the subject, forward the header and body (as STDIN) to /home/zenoss/zenemailack.py and redirect STDOUT and STDERR to $ZENHOME/log/zenemailack.log. This forumla also locks and waits until the script has finished. You can relocate zenemailack.py anywhere on your system, just update the path in the formula accordingly.

:0
* ^Subject:.*\[zenoss\].*
{
  :0 hbW:
  | /home/zenoss/zenemailack.py >> $ZENHOME/log/zenemailack.log 2>&1
}

Zenoss UI Configuration

Under the 'settings' tab, from the 'settings' link on the main navigation menu (left navbar), change this value to reflect the valid email address to your zenoss user.

From Address For Emails        zenoss@server.mydomain.com

* You must be able to receive emails at this email address on your Zenoss server.

Zenoss Server Configuration

Download the attached script below (zenemailack.py) and place it in /home/zenoss. You can install the file to any location of your choice. If you use a different location, make sure to update the Procmail formula to reflect the new path of your installation.

Change permissions to make the script executable.

chmod 755 ~/zenemailack.py

Edit zenemailack.py and change these variables to suit your environment:

# Update for outgoing email
MAIL = "/usr/sbin/sendmail -t "
from_addr = "zenoss@server.mydomain.com"

cc_address = ""

Testing

General

The first step is to make sure you can receive emails at zenoss@server.mydomain.com. If you can't receive a basic email, resolve this or you're out of luck.

Procmail

Once you can successfully receive emails, make sure that procmail is working. You can create a test formula in your .procmailrc to test that it's working correctly.

vi ~/.procmailrc

Append this to your .procmailrc and save it.

:0
* ^Subject:.*test
/dev/null

As zenoss user, test Procmail functionality by sending yourself an email with 'test' as the subject.

echo "foo" | mailx -s "test" zenoss

Check the procmail log for success.

tail ~/procmail/pmlog
From zenoss@localhost  Wed Sep  3 16:11:26 2008
 Subject: test
  Folder: /dev/null                                595

If you have no entry for this test in your Procmail log, Procmail is not working properly. Check your installation, double check that Postfix is configured to use Procmail and that you re-started Postfix after you configured it to use Procmail.

Re-edit ~/.procmailrc and remove the test formula you added above. Your .procmailrc should look like this after.

SHELL=/bin/bash
PMDIR=$HOME/procmail
LOGFILE=$PMDIR/pmlog
ZENHOME=/usr/local/zenoss
PYTHONPATH=$ZENHOME/lib/python
PATH=$ZENHOME/bin:$PATH
:0
* ^Subject:.*\[zenoss\].*
{  
  :0 hbW:
  | /home/zenoss/zenemailack.py >> $ZENHOME/log/zenemailack.log 2>&1
}

Zenoss

Once you've verified that you can recieve emails and Procmail works, test this with an actual event.

If you're impatient, you can inject an event into Zenoss. To do this, you can follow the admin guide. Just make sure your event matches an alertable condition.

Admin Guide - 7.1.8.1 – Creating Events Through the User Interface

If you're not impatient, just wait around until you get an alert. Once you receive the alert, simply reply. There is no need to type anything in your reply.

Make sure that your mail editor includes the body of the alert when you reply. I don't know of any that don't, by default, but I'm just throwing that in there.

You should receive a verification email back saying that the alert has been acknowledged or not. You’ll only receive a verification reply if the alert is in an unacknowledged state or something crapped out during the attempted acknowledgment. If you don’t get a reply email, the alert most likely cleared itself and has been moved to history, by Zenoss.  Once an event has been acknowledged, any reply-to-all email threads will not generate email noise. They will, however, be logged.

For debugging or troubleshooting, you can view the log at:

$ZENHOME/log/zenemailack.log

Caveats

• This script parses the username form the email address. If the username in your email does not correlate to your username in Zenoss, this may lead to confusion. If you setup your server to use LDAP, this should never be an issue.
• This script assumes that you have not changed the format of your alert email. If you have, you'll need to at least include the device and Event ID URL in the message body.

Device: %(device)s
<a href="%(eventUrl)s">Event Detail</a>

• The ownerid will be updated when you view the details of the event. However, when you click on the log tab, the user will show as admin. To me, this isn't a show stopper. If I can clearly see who took ownership of the event, the user stamp on the log doesn't matter. If I could figure out how to properly update this field, I would.
• Your Procmail log may take up a lot of space depending on how many devices you have and how many alerts you generate. Logging is generally useful for troubleshooting issues with Procmail. You may want to add this directory to logrotate.

vi /etc/logrotate.d/procmail_log

Add:

/home/zenoss/procmail/* {
weekly
rotate 2
copytruncate
}

Optionally, once Procmail works, you could disable logging by commenting out this line in your .procmailrc

#LOGFILE=$PMDIR/pmlog

Legal

Please feel free to contact me directly if you wish to use this in publication or for other commercial reasons.

Scott Haskell
shaskell614@yahoo.com