Well, it’s been almost a month since I’ve started writing about the humbug project. Now, it’s time to actually get you people involved, at least in the initial levels. We are looking to add 10 additional members into the humbug call analytics suite. Currently available analytics during the alpha testing is inbound call analytics.

Our aim is to gather as much information as we can and as much user requests as we can, humbug is a community oriented project, thus it relies on community oriented input and feature requests. Participating members will  be granted access to the humbug analytics portal, allowing them to gather statistical information regarding their inbound call hits and their top ten DID numbers – we are working on additional statistics. As new stats will become available, we’ll role those out into the service as soon as possible.

In order to participate in the closed alpha testing, please send an email to alphatest at humbuglabs.org, and we’ll send you a short piece of dialplan code to insert into your Asterisk server. Technically speaking, we’ll send you a short AGI command that looks like this:

exten => _X.,n,AGI(agi://somehost/DataReceiver,some_unique_ident)

The above line needs to be inserted into any place you would like to generate call analytics from. We’ll also enclose configuration steps for FreePBX (and other FreePBX compatible distributions). We are hard at work for creating a FreePBX integrated module, so you can do a one-click install.

“Oh, just get me the CDR‘s and I’ll take it from there” – how many times have I heard these words before? I can’t even imagine the number of times in the past 15 years of IT/Telecom’s work that I’ve done and in the last 8 years of Asterisk in particular – when it comes to billing and fraud management, it would appear that the CDR’s are the Rosetta Stone of the industry.

Over the past 6 months, several of my friends and I had been asking ourselves this question: “Is there more to billing, fraud management and profit leakage? does it really all begins and ends with the CDRs?” – so, here we were, a group of 3 engineers dealing with telecom system and billing systems – we knew that the answer is a definite YES, however, how come most companies and system aren’t even aware of this, in such a way that causes them to leak telecom profits and waste their hard earned profit margins on simple accidental mis-interpretation of CDR records.

So, we’ve decided to sit down and start analyzing calls in real-time, trying to evaluate not only the CDR record that is received upon the completion of the call – but also understand the traversal path of the call, analyzing it in real time and evaluating it profit leakage potential. At the mean time, we’re concentrating our work on Asterisk, as it is the simplest for us to implement – however, we’re not focusing it only on that – we’ll looking at adding it to FreeSwitch, Yate, OpenSer/Kamailio, OpenSIPS and the various varients.

So, what have we done so far? well, one thing we never really had with any of the existing systems was a clear view of what’s going on “right-now” on our systems, so we said: “it would really be great if we could know how many call hits we’ve received during the past 15, 30, 45 or 60 minutes” – so here is what we made:

The above image shows our top 10 inbound DID numbers, as you can see these are in the 972 and 447 country codes (yes, we work mainly in Israel and the UK). At the backend, our servers are analyzing the data in real time, generating an active alert in the case a DID number’s statistics change in a somewhat drastic change, thus, establish a traffic anomaly. Another thing that interested us was our usage across multiple servers, which we are exhibiting in the below graph:

Now, as you can see, the top graph shows a discrete anomaly:

This anomaly indicates something went wrong on all our servers between 00:45 and 1:15, which gives us a fairly discrete period of time to seek for a problem in the system. What happened was that one of the guys updated a portion of the data traversal API – basically deleting it [we resumed full work after about 40 minutes].

So, where is it all going to? well simple, a new Open Source based service that we’ll be launching within a few months from now. Our intention is to provide a means for simple, straight forward, highly reliable, call analytics, fraud management and profit leakage analysis service. A service which is based upon a simple to use API on one hand and Open Source based data gathering agents. Our belief is that by analyzing large amounts of data, from multiple sources around the world, we’ll be able to ascertain the fingerprint of a telecom bound attack – being able to alert the respective users of the service and maybe in the later future, also provide a means to block the attack as it advances across the world.

I’ll be updating about our advancement as we go along, but for the time being, this is something I felt would interest you.

Ok, it’s day 1 (or actually day 2) for AstriCon 2009 – and here’s my report for the day.

Yesterday was kind’a of a hectic day for me, as I was teaching a full day track of Asterisk and Cloud Computing, specifically, implementing Asterisk systems with Amazon EC2. I started the day with a class filled with 20+ people, and ended the day with a similar number – so in general I’m very happy. Not many people tend to attend the pre-conference days, so having that number of people and their positive reactions through out the day were very reassuring to me.

If there is one thing I’ve learned from this experience, it is the following: If you give a full day track, don’t arrive at the hotel 24 hours prior to it – you need at least 48 hours! People didn’t really notice (I hope), but through out the day I was suffering from a splitting headache – one that would usually send me right into bed with a couple of Advil’s. But hey, that didn’t stop me and I powered through it, I’m fairly proud of myself for doing so – as at the end of the day I regained back my strength and was livelier.

Today was the first official day of the conference – I gave the opening talk for the Cloud Computing track of the day. My talk was about how to build “IP Centrex” like services, without building an “IP Centrex”. I guess that I didn’t really introduce a brand new concept, but actually talked about something that many are thinking about, but are not inclined to try it on their own and burn some cash on. I guess my talk helped them out saying: “Hey, we’re not talking out of our asses here, this guy makes some sense and what we thought of isn’t that far fetched”.

Previous to that, Digium announced the 2009 Digium innovation award winners, where my company won an award in the pioneer category. This is the second year in a row my company had won the award, and I’m really happy with being acknowledged for this specific work. Having being a part of the community for over 7 years now, this award, at least to me personally, says a lot – it’s basically saying: “Look, you’ve done good, you’ve done some work that really helps out the project and the community in general – here’s a beer and a toast to you – hip hip” – well, that’s kind’a of a mouth full, but you get what I mean. I think that this is actually the place to mention that the award was for developing a high-powered Dialer/IVR platform, used in the Israeli elections and the work was contracted for a company called Shtrudel.

The all conference party is tonight – so I better rest up and be ready for it – should be fun. I guess beer and food are always a good mix when a bunch geeks are getting together

For those of you who know personally, you probably already know that last month I became a father. I guess the transition is something that I was more or less ready for, at least on the technical terms of the transition. However, the thing that completely surprised me was the mental transition – which isn’t even related to the somewhat lack of sleep here and there.

So, here I am, about a month and half away into the final touches to our new home, spending the weekend deploying over 100 meters of CAT-5 cabling in the house – yes the house is network rigged to the teeth. I’m sitting in my daughters room, clamping away the wall sockets for the network, thinking to myself: “hmmmm…. will Nitzan need a single network connection? or should I put at least two for future usage? … hmmmm…. well, I guess time would tell”. In any case, so there I was, spending most of my weekend being my own geeky self, thinking about wiring, networking, wireless exposures, access points, etc. I then go back home, and suddenly, all that disappears the minute I put Nitzan on my shoulder to burp her. It’s really funny, but with her on my shoulder, I guess everything goes away for a few minutes. My brother-in-law informed the house that he caught me burping Nitzan, while sitting at my computer answering emails with the other hand – Ok, so I can’t stop being a geek all together.

In any case, here I am juggling the various aspects of being a father to a new born baby, attending to the various tasks required to final touches of the house (painters, cleaners, air cons, dry walls, etc) and of course, attending to my customers – some of which are completely ambivalent to the fact that I’m under a constant lack of sleep in the past month. Well, I guess in a couple of months Nitzan will start sleeping better, and would make life easier for both me and my wife; in the mean while, we take comfort with the sleep periods my wife gets during the day, so that I can work and cater to my customers, while she caters to Nitzan during the nights – and I have to be honest about this, when it comes to the baby, my wife is the closest thing to a Jedi Knight, her ability to stay focused and clam even when the Nitzan is screaming is amazing – I can’t always do it.

Ok, enough about the house and Nitzan, let’s go back to been geeks for a bit. As you can see on the right hand side of the blog, I’ll be speaking at the up-coming Astricon. I’ll be giving a talk about how to build “IP-Centrex” like installations, utilizing Asterisk and tools like VMWARE, XEN and OpenVZ. However, while my talk may be interesting to you (I hope), my pre-conference tutorial will be much more interesting. I’ll be giving a full day tutorial, teaching people how to install Asterisk in a clouded environment (cloud computing), mainly the Amazon EC2 cloud computing infrastructure. For those of you reading this blog, you may have noticed that I’ve developed a distinct interest in the Amazon EC2 cloud, which I’ve written about several times and also lectured about at Amoocon. While my Amoocon presentation was mainly informational, at Astricon I’ll be primarily teaching you how to do what I did. Well, I won’t be teaching you the inner workings of the GreenfieldTech IVR API framework, although, if you’re gonna ask questions I will answer (especially if you ask the question 3 times, I can’t stand it when people ask the same question 3 times – I just have to answer it – nudge nudge).

Ok, back to fatherhood and Nitzan stuff. The mental transition that I was referring to before is something that I felt last night in its most force. My wife and I decided to go to one of the malls, not far away from our house. So, we entrusted Nitzan with my in-laws and drove to the mall. The mall I’m referring to is called “The seven stars mall” and we like it. It’s not a big mall, but its got this shop called DOMO, that carries these high class cooking ware (my wife and I really like to cook – my chilli con-carne is well known). So, here we were walking the mall, after I ordered a pair of shoes that I needed. So, my wife comments: “You know something, let’s see if there is some sale at Super-Pharm.” – and then we ended up purchasing baby formula, pacifiers and baby wipes. I then asked my wife if she maybe wants to walk into DOMO, but we both didn’t really think about it – suddenly, something that was like a default prior to Nitzan is no longer a default – interesting isn’t it?

In any case, this is how my life looks like at this point in time – and I have to admit that I kind’a like it. Sure, I don’t get as much sleep as I got before, but hey, I’m happy with it – so I just keep on smiling and go on forward.

Well, it’s finally out – my new book that is

Some of you already know, but over the past year I’ve been busy writing a new book. This time it’s a book for Asterisk developers, especially tailored to PHP developers wishing to utilizing the PHPAGI framework. The book is out from Packt Publishing (Like my old AsteriskNOW book) and is updated with all the recent changes in Asterisk – including version 1.6.X and DAHDI.

Asterisk Developers Book

If you like my work with Asterisk and would like to read more of my work, go ahead and get an electronic version of this book. I know it’s a little self promoting, by hey, it never hurts does it?

I’ve included a chapter on how to build a complete project from scratch, detailing the various analysis steps and various paradigms required to develop a fully functional Asterisk based application. I believe that even experienced Asterisk developers will benefit from this book.

As some of you may know, I’ll be attending the ITExpo in Miami Beach, Florida. The subject I’ll be lecturing about is “Virtualizing Asterisk”. However, I have to be honest, I really need to change the subject to be called “Asterisk in the Cloud“.

Ever since the introduction of Amazon EC2, people had been trying to get Asterisk to run properly inside an EC2 instance. While installing a vanilla Asterisk on any of the Fedora/RedHat variant instances in EC2 isn’t much of a hassle, getting the funky stuff to work is a little more tricky.

One of these tricky bits (which I hadn’t yet found a solution for) is the issue of supplying a timer for Asterisk’s MeetMe application. In the old days (prior to Asterisk 1.6), Asterisk required the utilization of a virtual timer driver, provided by Zaptel in the past and now the DAHDI framework. The problem is, that while you are fully capable of compiling and installing DAHDI on an Amazon EC2 instance – the problem starts once you want to use it.

A few words about Amazon EC2

For those not familiar with Amazon EC2, its general infrastructure is based upon the XEN virtualization project. XEN is a para-virtualization framework, meaning that is performs some of the work utilizing the underlying Operating System kernel and some of the work performed with a special Kernel in the virtualized Operating System instance. This poses an interesting issue with every type of application that relies on hardware resources and their emulation.

To learn more about the XEN project, go to http://www.xen.org.

So, where’s the big deal?

So, if you can compile your code and run it in an instance, as long as you have the kernel headers and kernel source packages – you should be just fine – right? WRONG!

Amazon EC2 deploys its own Kernel binary image upon bootstrap, causing what ever compilation you may have done to the Kernel to go away (unless you’re creating a machine from real scratch). Another issue is a version skew between the installed Operating System kernel modules, the actual kernel and the installed compiler. For example, the instance that I was using had the XEN capable kernel compiled with gcc version 4.0.X, while the installed operating system was gcc version 4.1.X – so, no matter what I did to compile my kernel modules or binary kernel, I would always end up in a situation where loading the newly compiled kernel modules will generate an error.

Did I manage to solve it? – NOT YET. I’m still working on it, and I have to admit, that considering the fact that I have over 10 years of Linux experience and had compiled kernels from scratch many times, this one has gotten me a little baffled – I guess I’ll just need a few more nights and a case of Red-Bull to crack this one open.

So, what can we do with EC2?

In my view, EC2 + Asterisk is the ultimate IN/NGN services environment – and I have proof of that. A recent lab test that I did with one of my customers showed a viable commercial alternative to Sigvalue when using Asterisk and EC2 structures. The main reason for our belief in using EC2 was the following graph:

IN/NGN usage over 24 hours

What we’ve noticed was that while our IN/NGN system was generating traffic, it’s general usage showed peak usage for a period of 2.5 hours, with a gradial increase and decrease over a period of almost 10 hours. Immediately that led us to a question: “Can we use Amazon EC2 to provide an automatd scaling facility for the IN/NGN system, allowing the system to reduce its size as required?”

To do this, we’ve devised the following IN/NGN system:

Amazon EC2 Enabled IN/NGN Platform

Our softswitch would have a static definition of routing calls to all our Asterisk servers, including our EC2 instances which had static Elastic IP numbers assigned to these. The EC2 Controller server was incharge of initiating the EC2 instances at the pre-defined times, mainly, 30 minutes prior to the projected increase in traffic. Once the controller reaches its due timer, it will automatically launch the EC2 instances required to sustain the inbound traffic.

For our tests, we’ve initiated 5 AMI instances, using the EC2 c1.medium instance. This instance basically includes 2 cores of an AMD opteron, about 8GB of RAM and about 160GB of Hard drive – more than enough. Initially, we’ve started spreading the load evenly across the servers, reaching about 80 concurrent channels per instance, and all was working just fine. We managed to reach a point where we were able to sustain a total of about 110 concurrent channels per instance, including the media handling – which is not too bad, considering that we are running inside a XEN instance. The one thing that made the entire environment extremely light weight is the GTx Suite of APIs for Asterisk. Thanks to the GTx Suite of APIs, scalability is fairly simple, as all application-layer logic is controlled from a central business logic engine, serving the Asterisk servers via an XML-RPC based web service. Thanks to Amazon, practically infinite, bandwidth allocation – the connections from the Asterisk servers to the US based central business logic was set at a whopping 25mSec, thus, there was no visible delay to the end user.

It is clear that the utilization of Asterisk and EC2 operational constructs can allow a carrier to establish their own IN/NGN environment. However, how these are designed, implemented and operated are at the hands of the carrier – and not a specific vendor. If the carriers around the world will take to this approach, time will tell. As a recent survey stated that 18% of the US PBX market is currently dominated by Open Source solution, having Digium dominate 85% of these 18% (~15%), I’m confident that we will see this combination of solutions in the near future.

As some of may know, I spend most of my time working as an Asterisk Developer and Consultant in my own company – called Greenfield Technologies Ltd. - named GreenfieldTech for short. Since the day I started my company I knew a few facts and truths:

1. GreenfieldTech starts as a one man operation – thus, spending more than 15% of my day to day tasks on support issues will surely stress me out and drive my business to the ground.
2. I don’t really want to deal with PBX installations and various office telephony aspects of using Asterisk – simply because it requires to much of item 1.
3. I need a way to be able to provide a fast response to my customers, maintain a low support over head and make sure that what ever I do, it can be accounted for.

Hence the above, I understood that GreenfieldTech‘s products are not really the various services it provides, but actually the products vary completely from what a product looks and feels like. I understood that in order to provide this fast development turn-around I needed my own development framework. Surely PHPAGI, Adhearsion and Asterisk-JAVA can easily provide for a development framework – but it still doesn’t help me one bit – I needed something different.

I then realized that GreenfieldTech‘s products are the one product line that all developers and vendor steer away from: Programmatical API frameworks. The programmatic framework should provide the most basic feature set, allowing it to be extended and continued onwards at ease. Hence, the suite of GTx API frameworks were born. A set of 3 different APIs provide all the facilities required for building any of the below:

• Dynamic IVR structures – driven via web services
• Recording systems
• Pre-Paid/Post-paid solutions
• Inteligent Network (IN) services
• Automatic Dialers – Predictive, Power, Progressive, Preview
• Broadcasting systems
• and more …

The entire API suite is based upon XML-RPC based web services. When I introduced XML-RPC to one of my customers, he immediately indicated: “But, XML-RPC doesn’t really provide for session oriented persistency. Why are you using XML-RPC?” – The reason is simple – we don’t need session persistency in the API – Asterisk can provide all of that internally. Thanks to Asterisk’s channel oriented architecture, we can regard each of Channel thread in Asterisk as a seperated data container, capable of holding all your session information and persistency information at ease.

Recorder API – http://www.greenfieldtech.net/products/gtrapi
IVR API – http://www.greenfieldtech.net/products/gtvapi
Dialer API – http://www.greenfieldtech.net/products/gtdapi

The best thing about the GTx API suite is this – they are all interconnected. For example, call being handled by the Dialer API can be then handed over to the IVR API, operating within the same Asterisk server or a remotely located Asterisk server (via SIP/IAX2).

In the time to come, I’ll be showing you various XML-RPC structures and example of how to use the GTx API suite, and do all sorts of interesting IVR and Dialer structures.

Ok, it’s been almost 24 hours since I started my serious playing around with Amazon EC2, and I can honestly say that I’m tired – however, I’m very pleased with my results. Like any other experiment, this one started with a requirement. The requirement was to install and operate one of the dialer frameworks I’ve written in the past year on an EC2 based instance. In order to evaluate the process, let’s start with our baseline installation, meaning, what am I using in the real-world:

Hardware and Software Specification

The original machine answered to the following specification: Quad Core CPU, 2GB of RAM, 250GB of Hard Drive. My original machine was running CentOS 5.2 with an x86_64 kernel installed. In terms of software installed, we had Asterisk 1.4.22.1, MySQL 5.X, PHP, FreePBX, Apache and my dialer framework.

Introducing Amazon EC2 AMI

An EC2 AMI is basically an image of a computer, contained with a single installation manifest on the Amazon cloud computing system. AMI’s provide for the simplest manner to start using EC2, as these usually include a pre-defined server installation, that usually has some stuff already installed.

Choosing your AMI

Amazon provides a multitude of AMI’s to work with, unfortunately, most of these are either out-dated and the sheer number of these makes the choice somewhat overwhelming. I decided to start working with a working AMI image of Fedora Core 8, that already had the LAMP stack installed – the one I used was:

The YUM Repository issue

For some strange reason, the AMI images contained in the Amazon repository are all unable to perform any updates to their installed RPM packages. It took me a while to understand what’s wrong, but in general, the fedora project had simply removed the old releases from their repository, so I had to go in and manually modify the /etc/yum.repos.d/ configuration files. For you convenience, here is the repos list that I’m using at this point:

YUM Repositories for Amazon EC2 Fedora images <- click this to download the file

[development]
name=Fedora Core $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/development/$basearch/
enabled=1
gpgcheck=0

[extras-development]
name=Fedora Extras $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/development/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=0

[extras]
name=Fedora Extras $releasever - $basearch
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/$releasever/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

[updates-testing]
name=Fedora Core $releasever - $basearch - Test Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/testing/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-test

[updates-released]
name=Fedora Core $releasever - $basearch - Released Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/$releasever/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

If you are trying to install an instance, you may use the above for any Fedora based AMI on EC2.

Creating my own AMI

While going about and building a single server is fun, I needed a way to create my own installation AMI once I’ve completed my modifications. The Amazon EC2 resources website gives out a whole lot of information, which can be somewhat confusing for the first time reader. So, I searched for a better way to construct my own AMI image. I found the following web page, which was really really helpful: Azeez’s Notes.

Azeez’s blog gives a very concise and to the point, step by step, guide to building your own AMI image – it got me up and running in less than 10 minute – WAY TO GO AZEEZ.

So far, so good …

So far my installed instances are working just fine and I’m currently operating a cluster of a few of these dialer systems. I’m in the process of checking what kind of mileage I’ll get from the EC2 instances, in comparison to a real hardware server – which is really interesting.

My main objective here is to be able to build a dialer-on-demand framework, which will enable my customers to increase their capacity utilizing a simple web interface to initiate my AMI instances. I’ll report back with additional information once I have it.

While this post is titled “Thoughts of Virtualizaiton”, the applications described can be easily applied to non-VM type installations.

Virtualization is a wonderful tool, it enables rapid growth and rapid deployment of new servers and services. However, just like any other platform that tends to grow across the time line, it poses the same annoying issue of managing a large system, especially when dealing with Asterisk based installations.

Let us imagine the following scenario: A Calling Card company while utilizes 8 different Asterisk application servers, are utilizing a single Database servers cluster and are receiving inbound calls from various sources and load balanced across all Asterisk application servers. What I’ve described above is more or less the practice most (if not) all calling card operators deploy. No matter if the usage is A2Billing, MOR, ASTBILL – the methodology is more or less the same.

One of the bigger issues with such an installation is debugging of a running session, more over, the ability to debug a session after it is finished. This situation is caused by a simple, yet annoying issue, we are operating within a “zero-knowledge” system, where we have no precognition of where a specific call will be handled utilizing our cluster. Now, if you are an experienced sysadmin, you would most probably do the following:

1. SSH to all your Asterisk servers.
2. Tile all consoles on your desktop.
3. Start the test – and hope your eyes are fast enough to capture the right gateway.

Well, this is the normal practice with most people – but I have to admit it’s kind of annoying. Now, let’s imagine that we are now building our system from scratch, we’re not using A2Billing or any of the other Open Source products, we simply build our own application framework. So, what do we need to do in order to keep track of our system correctly?

Step 1: Consolidate

Consolidate the messages coming from each of your gateways to a single logging facility. The best track would be to utilize some form of Syslog facility. For example, all the scripts and network services that I develop utilize a clear and concise interface to syslog. I usually re-direct the syslog facility that I use to an external server, thus, I get all the logs on a single syslog file system.

If you are worried about I/O issues on the syslog server, you can always create a “syslog-proxy” using tools such as memcached or others.

Step 2: Identify

Your syslog write function should always include a prefix indicating the name of the generating Asterisk server. For example, have something like the following prefix your syslog entry:

Dec 14 21:51:32 pbx [PBX01/6d6d6423a2244aa71980e5a5b437919e/check_pincode[22537]: agiParameters: check_pincode

While the syslog facility will include your generating hostname, when duplication VM’s, this would be a really good practice.

Step 3: Analyse

Once your logs are consolidated to a single environment, it should be fairly simple for you to go about and analyse these in a pre-defined routine. There is little to gain from analysing the logs on-the-fly, but analysing it every 5/10/15 minutes will prove worthwhile.

Step 4: Audit

Auditing is good – as long as you keep a clear view of what you audit and what you don’t. Audit key points in your application to a database can save you a whole lot of time of debugging – just make sure your audit is clear.

Keep the above in mind and you should be just fine creating any scale of platform.

So, over the past 2 years, we’ve seen many Asterisk appliances – ranging from the Digium AADK on one side of the spectrum to the RoweTel IP04 appliance, it is clear that appliances are the hottest thing in the Asterisk sector to-date.

While most appliances take on a similar form, usually one of the following: an ADI BlackFin reference design or a PC-Engines Geode based appliance, one company decided to go completely the other way around – namely PIKA technologies. Now, don’t let the name PIKA fool you, these guys are no pokemons, they know their stuff.

While at AstriCon 2008, I’ve been approached by the people at PIKA technologies, asking me to test and evaluate their WARP Appliance product – a Linux based appliance framework, for building Linux based appliances, with a focus on telephony. They awarded me with an appliance to test, equipped with 5 FXS ports and 4 FXO ports, I was on my way to start my telephony appliance.

First thing’s first, computing power – while most appliances rely on fairly low scale computing power, the PIKA appliances boasts a massive PowerPC CPU, with a uCkernel based Linux 2.6.X kernel – while many other appliances still rely on 2.4.X. The on-board LCD display makes finding your appliance IP number a breeze, which is very important in my book. Opening the box is very easy, exposing its inner workings:

One thing nice about the LCD display is the fact that once you plug something in, the LCD will show the applicable LCD icon as connected – making for quick diagnostics for the lame sysadmin a breeze.

Now we’re getting to the slightly annoying part, the WARP appliance is not your average PBX appliance – in fact, it’s not a PBX at all – it’s a framework for building telephony appliances. So, if you want to get Asterisk + FreePBX to run on it, you’ll need to spend some time getting it up and running correctly. If you want to start working with the appliance, I suggest that you get to know cross-compiling and build tools, in order to utilize the buildtools provided with the appliance.

Over all, the appliance comes with a fairly stock Linux and Asterisk pre-compiled, with all the required modules to get a fully functional PBX running. For my initial tests, I’ve used the AsteriskGUI, which worked just fine with the system, making the configuration a breeze.

I’m currently evaluating their FreePBX build-tool, provided by Philippe from FreePBX, and we’ll see how that goes.

I keep a vigilant eye on Google Alerts. To be more exact, I’ve got a few alerts defined, in accordance to subjects that interest me. Two of these subjects are Asterisk and FreeSwitch. Recently, the following had been posted on a personal blog:

Our PBX/IVR web-based generator Telfa has been moved from Asterisk to Freeswitch. Why?

Asterisk just seems to come from a different world than what I am used to. Inflexible and problematic. Very long configuration files with ancient syntax. Now I’m far from pretending I’ve used Asterisk enough to understand it pros and cons well, but I have a decent software development experience and I can tell when something “smells.” I didn’t want to build our system (that I want to be flexible and scale well) on some old technology that is only living from its past.

And (most importantly) there are many people experienced with both Asterisk and Freeswitch favoring the latter: Anders Brownworth, Jonathan Palley (creator of Telegraph, a Rails plug-in that lets you talk to Freeswitch), or of course the creator of Freeswitch (and former Asterisk developer!) Anthony Minessale himself.

[Extract from: http://onruby.flempo.com/2008/10/28/chose-freeswitch-over-asterisk/#comment-58]

Reading the above simply flames me UP – WHAT A LOAD OF BULL!

The writer admits that: “Now I’m far from pretending I’ve used Asterisk enough to understand it pros and cons well, but I have a decent software development experience and I can tell when something “smells.”” – If he’s such a worthy developer, the above simply proves that he’s simply an RTFM!

Asterisk provides a multitude of methodologies for configuration management, dialplan logic programming, programmatic API structures – basically, all the tools you need to go about and create your application. While I admit that FreeSwitch is slightly better at the “Core Switching” environment, implementing a near true Soft Switching architecture – Asterisk is a totally different thing.

A while back I decided to develop a couple of my better systems on FreeSwitch, as a test of my abilities. To see if I can be as proficient with FreeSwitch as I am with Asterisk. I re-developed the applications on FreeSwitch from scratch, however, FreeSwitch’s somewhat annoying XML configuration structures made it a hell for me to develop something that I would ask my customers to put their money on.

I started working with Asterisk in 2002, and starting 2003 I tried to have people in Israel start using Asterisk. I can honestly say that the first ever true adaptation of Asterisk in Israel came about only late 2005, and even that wasn’t for mission critical applications. Asterisk had achieved a point in which it is now considered a valid replacement for classic IN and NGN applications and engines, such as SigValue. I suspect that FreeSwitch will become a more dominant contender in that part within a period of 18 to 24 months, as its developer community will grow – time will tell.

In my large installations, I like using Asterisk as my IVR/Application engine envrionment and using FreeSwitch as my routing and LCR environment, simply because it’s capable of handling slightly more SIP-to-SIP connections than Asterisk – but that is the only reason, all the rest is Asterisk.

Both projects are wonderful tools, each one with its own set of strengths and weaknesses. Learn how to balance the use of the two, utilize the best of both world – and stop being a bloody purist, that’s exactly what leads to poor technologies such as Microsoft and poor life practices, like wars and terror.

Ok, it goes without saying: “A2Billing is one of the most complete Calling Card systems in the Asterisk market today.” – on the other hand, it is also true that: “A2Billing is one of the most complex and convoluted pieces of code ever written!”.

The combination of the above makes for a fairly combustible mixture, especially if you have a big system. Now, I recently ran into an issue, where PHP was litterally eating up almost 512MB of ram, in order to run the A2Billing reports. In it self, that didn’t make much sense to me. However, after inspecting the code, and realizing that A2Billing uses GD in run-time to generate images out of thousands of CDR records, it made perfect sense that it may just be eating up memory.

So, increasing the memory on PHP to go up to 512MB of RAM helps, but creates an interesting probelm. Whenever Apache will invoke a script, it will automatically consume a shitload of RAM, and for each time I intiate a new query, it will spawn a new Apache instance, and consume the same amount of memory. That said, after 6 queries of 512MB, about 50% of the machines RAM was already eaten up – and Apache will not free it!

At this point, I had 2 choices:

1. Go into the A2Billing code, change the GD code to work right or simply change it completely to something else (maybe flash).
2. Work around the problem with a mix of proper IT practices.

I admit that I hate quite a lot of things (I won’t list these here); however; nothing ranks up the list as modifying someone elses code, when I know for fact that it will be unmaintainable in the future. So, I choose option number 2.

I’ve being playing alot with Lighttpd lately, and got some really nice performance from it. So, I said to myself, this would be a great test to see if Lighttpd+FastCGI can solve the problem here. I had to work my way around lighttpd to do what I wanted and verify that my FastCGI server in Lighttpd doesn’t consume all memory, however, here is what I got working with A2Billing, and really nice.

Step 1: Enable the required modules:

server.modules              = (

                               "mod_access",

                               "mod_auth",

                               "mod_status",

                               "mod_fastcgi",

                               "mod_accesslog" )

Step 2: Enable the FastCGI Server

fastcgi.server             = ( ".php" =>

                               ( "localhost" =>

                                 (

                                   "socket" => "/var/run/lighttpd/php-fastcgi.socket",

                                   "bin-path" => "/usr/bin/php-cgi",

                                   "idle-timeout" => 30,

                                   "max-procs" => 1,

                                   "min-procs" => 1

                                 )

                               )

                            )

Step 3: Modify user permissions (required if you are using FreePBX)

server.username            = "asterisk"

server.groupname           = "asterisk"

Step 4: Setup authentication and authorization (optional)

#### auth module

## read authentication.txt for more info

auth.backend               = "htpasswd"

auth.backend.htpasswd.userfile = "/var/www/.htpasswd"

auth.require               = ( "/" =>

                               (

                                 "method"  => "basic",

                                 "realm"   => "A2Billing Management",

                                 "require" => "valid-user"

                               )

                             )

The above configuration made the interface spwan a single FastCGI, insuring that memory usage is never over utilized. I still need 512MB of RAM to run the scripts, but at least now it’s limited to only 512MB of RAM, out of a machine that has 16GB of RAM.

Well, it’s quite common to get a flame here and there on any Internet technical maling list, but the following really caught my eye. Here’s an extract from the TrixBox forum:

———————- CUT HERE ———————-

Subject: SS7 on Sangoma a101D

Over the past 12 months, I’ve been heavily involved in the development of high-speed dialers. While many companies published a prolific number of automatic dialers (power, predictive, broadcast) – none of these companies ever announced there top speed dialing capability.

An so I ask myself: “What is the reason not to release these numbers publicly?” – interesting, isn’t it. So, I decided to experiment myself and see if I would release public numbers for my dialer. I’ve designed my dialer to be capable of generating upto 35 calls concurrently. My questions was this: “While the dialer is fully capable these numbers in a test scenario, will this number be reached in real life?”

So, I started experimenting with my code – doing a very simple test. I’ve installed my dialer on a Dual Quad Core system, 8GB RAM, and a RAID-1 array as the storage. Now, I’ve setup the system to generate up to 35 calls every second, allowing the server to sustain a total of 180 concurrent calls – 360 channels.

What happend was actually this: the dialer generated about 180 dials within a period of about 8 seconds, and then, simply waited till some of the calls completed in order to generate additional calls. Ok, that is understandable, however, what is the throughput of the dialer? So, I decided to do another test, I filled up my queue with a total of 50,000 records. In addition, I’ve arranged with my carrier to terminate the calls to on of their SigValue systems, to sustain the enormous number of inbounds. In addition, I asked them to perform a small measurement of the average call inits per second. I was shocked with the result!

While the dialer simply peaked upon startup, after 20 seconds of operations it started normalizing at around 6 call initiations per second. But how can that be? how can it be that a dialer capable of dialing 35 calls is slowed down to crawl? – the reason is simple, the latency imposed by the PSTN/VoIP network, the time it takes for the calls to terminate and the actual time the call is left on the air, while the dial happens. All these factors together had given me the notion that there is no practical top speed, as it is in direct relation to the demographics of the dialer operations.

Having said that, is it possible to devise a formula to calcualte this number on a per demographic basis? maybe on a per PSTN/VoIP carrier type? maybe a combination of the both? will a formula such as that will enable for the better creation of broadcast/predictive dialer? – no use dialing tons of number into a demographic that can’t handle it, while at the other side of the spectrum, what demographics will benifit from the utilization of an automated dialer?

For a while now I’ve been toying around with the idea of utilizing Lighttpd for various web based applications. One of these application is my Automatic Dialer framework, also known as the GTD-API. The main issue with the GTD-API (besides that it is highly reliant on a MySQL database), is the fact that all requests have to be processed via XML-RPC HTTP post requests.

The main issue that I had was this: in a production scenario, a dialer management system will generate over 100 requests to the XML-RPC server. While Apache is fully capable of rendering services at such a speed, its increasing size and boilerplate automatically introduce a management issue. In addition, as I was trying to build a dialer appliance that can be used in any enterprise, the ever expanding Apache wasn’t a good choice.

While I was looking at both NginX and Lighttpd, the latter captured my eye, thanks to a simple advantage – The integration of FastCGI based PHP was so easy, that it almost troubling that I used Apache all these years.

At this point, once I got Lighhtpd working with my Dialer, I said to myself: “It would be really cool to go about and send status reports back from the dialer, directly to the web client activating the call. In addition, I really don’t want to go about and perform these updates to the database, then query the database – that would, literally, kill the MySQL server.

So, I implemented a local session storage area for each call, which updated the call status as it traverses. The information was stored on the hard drive, which allowed a better response time than the ever indexing MySQL server. The status reports were picked up from the Lighttpd server via an Ajax client (which I didn’t write – I suck at JS) – and it works quite well.

I wonder, can Lighttpd completely replace Apache? … time will tell…