Splunking Mitel Voicemail Logs

KyleLeave a comment

If you’ve spent any amount of time around Mitel (formerly Shoretel) you know that the system processes auto-attendants the same way it does voicemail. That’s just fine up to and until the point you have a department asking for metrics of AA menus. In my case it was accounting wanting to make a case for a temp employee for W-2 season. To pull last year’s data I had to turn to backups but decided for the future I wanted to be able to pump the logs in to Splunk so I could just query it there. The Splunk Forwarder was already installed on the Mitel server forwarding the Windows logs so it was just a matter of adding the Mitel logs.

Some general info about the Mitel logs if you’ve never dug in to them before. They’re stored in the folder the software was installed in. For this site it was:

\Shoreline Data\Logs\

The voicemail logs themselves come as Vmail-[date/timestamp].Log and look like this:

The timestamp is 2-digit year/month/day and then six digits of time. The initial log is created at midnight. All zeros. If the log file fills up it creates a second and stamps it with the 24-hour time. In the above example 6:15PM and 24 seconds.

To get Splunk to watch the file and forward updates a [monitor://] stanza needs to be added to inputs.conf file on the forwarder of the Mitel server. Should find it in where Splunk is installed. For this site it was:

\Program Data\SplunkUniversalForwarder\etc\system\local\

If there isn’t a file there with that name it’s OK to create one. Splunk just uses the default config files if there’s not yet been any customization. Create a stanza like this:

[monitor://E:\Shoreline Data\Logs\Vmail*.Log]
sourcetype=mitel_voicemail_logs
index=shoretel

Keep in mind that most of Splunk is case sensitive so pay attention. You’ve been warned.

sourcetype= can be whatever you’d like that makes sense to you. It just creates a reliable searchable field to get started with.

index= is optional. In this environment there was already a separate index that other phone-related logs were dumped in to. If you do too send them there. Or create a special index for it. Or just omit the line and the logs will land in the default index. No wrong answer depending on the environment.

Restart the Splunk service on the server and you’re up and running.

I noticed quickly though that I was getting multiple lines of events in each Splunk event. This could make queries troublesome in the future so I needed to find a way to force Splunk to ingest one line per event.

The answer turned out to be a props.conf file on the Splunk indexer. On that machine navigate to where Splunk is installed to edit the file. At this site it was:

\splunk\etc\system\local\

Again if there’s no props.conf file just create one. You need a stanza to identify the incoming logs you want the settings applied to. The easiest way is to match on the sourcetype that is attached by the inputs.conf on the forwarder on the Mitel server. In this case:

[mitel_voicemail_logs]
MAX_EVENTS=1
LINE_BREAKER=([\r\n]+)
SHOULD_LINEMERGE=false

MAX_EVENTS=1 sets the max number of lines per event.

LINE_BREAKER=([\r\n]+) is a regex for locating newlines within the file.

SHOULD_LINEMERGE=false guarantees other lines in the default props.conf file don’t interfere with the settings here and cause multiple-line-events unexpectedly.

After saving the file restart the Splunk service on the indexer and watch the magic happen! Should look a little like this:

Common Certificate Conversions

KyleLeave a comment

Every organization I work with use certificates to secure web pages or VPN’s or the like. Sometimes I request the certificates, sometimes someone else does. Certificates are one of those things no one gives much thought to until suddenly it’s expiring (or expired!) and everyone’s freaking out (rightfully so!) about the certificate warnings end users are seeing on the “secured” pages. PKI (Public Key Infrastructure; what makes certificates work) is one of the most fickle security measures to deal with. Every misstep is a land mine that explodes and immediately takes your webpage into the red zone. In the best cases a little massaging fixes the issue. In the worst cases a whole new cert has to be requested and that takes time. These are a couple of massages I gave to a site last week that I’ve given to others before and suspect I will give to others again later.

Password with quotes to no quotes

Ever had to install a cert you didn’t request yourself? Ever been in a mixed environment where most of the boxes are Windows Servers but there a smattering of Cisco ASA’s and maybe an Apache or two hidden in the weeds? Windows doesn’t care much what you throw in the passphrase field when selecting one for your .pfx file. Cisco does though. So does Apache. Anything Linux-based does so I suspect Apple does as well but I don’t know for sure.

I was moving quickly that morning (remember I hinted sometimes an organization thinks about certificates when they’ve already expired?) and started to load their cert onto an ASA that served as their VPN endpoint for users in the field. {insert landmine sound here}

Hastily I was copying and pasting things I would need so I could access them quickly. The passphrase went in to a notepad window without a second glance. There was a quotation mark just about halfway through it. Believe it or not the exact same thing had been done just a couple years prior. By the exact same person. I’d warned this would be an issue every time and therefore when the new cert was requested they should avoid quotes.

There are a couple ways to untangle this knot. Here are just two. :

Method 1 – you have the .cer or .pem or .crt of the cert, the unencrypted .key file, and OpenSSL

Launch your command-line and run the following:

openssl pkcs12 -export -out cert.pfx -inkey keyfile.key -in certificate.crt

So you know what you are doing with this command …

  • “openssl” – runs OpenSSL.
  • “pkcs12 -export -out cert.pfx” – tells OpenSSL we’ll use the PKCS12 module to output a file and call it cert.pfx when we’re done.
  • “-inkey keyfile.key” – this is the private key we want meshed with the certificate file we have.
  • “-in cert.crt” – this is the certificate we want meshed with the key. It may also end in .pem or .cer if that’s the format you have available.
*Notice you get to enter a new password here. Leave out the quotes this time.

Method 2 – You just have the .pfx and its password and OpenSSL:

This is a multi-step process. What you’ll do is extract the private key and certificate(s) from the .pfx file, separate the encrypted private key, decrypt the private key, then repackage it with the certs using a password without quotes in it.

From your command-line:

openssl pkcs12 -out keys_out.txt -in cert.pfx
  • “openssl” – runs the program.
  • “pkcs12 ” – loads the PKCS12 module and tells it we want the output of this command in keys_out.txt.
  • “-in cert.pfx” – tells the program what PKCS12 file we are about to crack open.
The “Import Password” you are prompted for should be the password you used when you created the .pfx file. Most of the time it’s the same as the key file passphrase but your mileage may vary. The “PEM pass phrase” is the key file pass phrase. Not sure why it’s called something else to be honest.

So now there’s a text file with the certificate and an encrypted key file meshed together. To split it apart open it with notepad++ or whatever your favorite editor is and copy everything from —–BEGIN ENCRYPTED PRIVATE KEY—– to —–END ENCRYPTED PRIVATE KEY—–. Make sure you grab that header and footer and their dashes. Paste it in a new file and save it with a name that’s memorable (I used keyfile.key for this example) because we need to decrypt it now.

The body of your new file should look like this when complete. There may be other info ahead of or following the key begin/end tags. Just ignore it. This is the only ingredient you need.

From your command-line:

openssl rsa -in keyfile.key -out keyfile_decrypted.key
  • “openssl” – runs the program
  • “rsa -in keyfile.key” – loads the rsa module and tells it to decrypt the key file.
  • “-out keyfile_decrypted.key” – tells the rsa module where to dump the decrypted key file.
Now there’s a decrypted key file available. Let’s make ourselves a .crt file to combine with it.

Open the keys_out.txt (or whatever you’ve named it) with your editor again. Copy everything from —–BEGIN CERTIFICATE—– to —–END CERTIFICATE—– (including that header and footer again, ignore the extra stuff) and paste it in a new file. Save that new file with a .crt extension.

Now that you have a .crt file with your certificate’s info inside and an unencrypted key file you may take them and follow the steps above for Method #1. Following those steps you’ll combine the two and create a .pfx file with the password of your choosing. Sans quotes.

Converting .pfx to .pem

Pretty frequently I run in to devices that require certificates in a different format than what I’ve downloaded. Avocent Cyclades KVM’s come to mind. Or are they Emerson? Vertiv? I can never keep it straight. Especially since the branding seems to change every few firmware updates. Conversion is a quick command-and-password-entry away. Just substitute the path to your certificate and the certificate name as-needed in the following. 

openssl pkcs12 -in cert.pfx -out cert.pem
  • “openssl” – runs OpenSSL.
  • “pkcs12 -in cert.pfx” – loads the pkcs12 module and opens the .pfx to manipulate.
  • “-out cert.pem” – saves the file in the .pem format.

The 5W1H of Secure-Trajectory

KyleLeave a comment

Who: I’m Kyle, the man behind secure-trajectory. I’m hold a bachelor’s degree in CIS, a couple certifications I keep current (have had several I let expire; who hasn’t?), and have a little over a decade in the professional IT space. I started out on a help-desk in a call-center and worked my way up into designing, engineering, administering, and auditing computer networks and security. I am also a PCI-ISA.

What: Secure-Trajectory for now is just a blog. A side-project that lets me dabble some things like web development, blogging, learning the ins and outs of web-hosting, and so on. It’s going to serve as a sort of compliment to my collection of notebooks I’ve built over the years as a network admin/engineer.

When: When time allows. Like I mentioned earlier this is a side project but I aim to add something every month or so. We will have to wait and see what the tempo ends up being.

Where: It’s the interwebs so of course that affords me the ability to work on this wherever I happen to be as time allows.

Why: I suspect the internet is here to stay … by putting some of my work here it makes it easier for me to search my notes in the future. Maybe, possibly, it will help someone else further down the road if they run into the same or a similar issue.

How: Since I expect this to be more technical in nature in order to replace my notebooks and to help others it’s mostly going to be content with descriptions of issues, research I completed, code, and screen-grabs from my labs or some other sanitized output.