While this post is titled “Thoughts of Virtualizaiton”, the applications described can be easily applied to non-VM type installations.

Virtualization is a wonderful tool, it enables rapid growth and rapid deployment of new servers and services. However, just like any other platform that tends to grow across the time line, it poses the same annoying issue of managing a large system, especially when dealing with Asterisk based installations.

Let us imagine the following scenario: A Calling Card company while utilizes 8 different Asterisk application servers, are utilizing a single Database servers cluster and are receiving inbound calls from various sources and load balanced across all Asterisk application servers. What I’ve described above is more or less the practice most (if not) all calling card operators deploy. No matter if the usage is A2Billing, MOR, ASTBILL – the methodology is more or less the same.

One of the bigger issues with such an installation is debugging of a running session, more over, the ability to debug a session after it is finished. This situation is caused by a simple, yet annoying issue, we are operating within a “zero-knowledge” system, where we have no precognition of where a specific call will be handled utilizing our cluster. Now, if you are an experienced sysadmin, you would most probably do the following:

1. SSH to all your Asterisk servers.
2. Tile all consoles on your desktop.
3. Start the test – and hope your eyes are fast enough to capture the right gateway.

Well, this is the normal practice with most people – but I have to admit it’s kind of annoying. Now, let’s imagine that we are now building our system from scratch, we’re not using A2Billing or any of the other Open Source products, we simply build our own application framework. So, what do we need to do in order to keep track of our system correctly?

Step 1: Consolidate

Consolidate the messages coming from each of your gateways to a single logging facility. The best track would be to utilize some form of Syslog facility. For example, all the scripts and network services that I develop utilize a clear and concise interface to syslog. I usually re-direct the syslog facility that I use to an external server, thus, I get all the logs on a single syslog file system.

If you are worried about I/O issues on the syslog server, you can always create a “syslog-proxy” using tools such as memcached or others.

Step 2: Identify

Your syslog write function should always include a prefix indicating the name of the generating Asterisk server. For example, have something like the following prefix your syslog entry:

Dec 14 21:51:32 pbx [PBX01/6d6d6423a2244aa71980e5a5b437919e/check_pincode[22537]: agiParameters: check_pincode

While the syslog facility will include your generating hostname, when duplication VM’s, this would be a really good practice.

Step 3: Analyse

Once your logs are consolidated to a single environment, it should be fairly simple for you to go about and analyse these in a pre-defined routine. There is little to gain from analysing the logs on-the-fly, but analysing it every 5/10/15 minutes will prove worthwhile.

Step 4: Audit

Auditing is good – as long as you keep a clear view of what you audit and what you don’t. Audit key points in your application to a database can save you a whole lot of time of debugging – just make sure your audit is clear.

Keep the above in mind and you should be just fine creating any scale of platform.