30 Jan 09 Asterisk 3rd Party API – and their importance

As some of may know, I spend most of my time working as an Asterisk Developer and Consultant in my own company – called Greenfield Technologies Ltd. - named GreenfieldTech for short. Since the day I started my company I knew a few facts and truths:

1. GreenfieldTech starts as a one man operation – thus, spending more than 15% of my day to day tasks on support issues will surely stress me out and drive my business to the ground.
2. I don’t really want to deal with PBX installations and various office telephony aspects of using Asterisk – simply because it requires to much of item 1.
3. I need a way to be able to provide a fast response to my customers, maintain a low support over head and make sure that what ever I do, it can be accounted for.

Hence the above, I understood that GreenfieldTech’s products are not really the various services it provides, but actually the products vary completely from what a product looks and feels like. I understood that in order to provide this fast development turn-around I needed my own development framework. Surely PHPAGI, Adhearsion and Asterisk-JAVA can easily provide for a development framework – but it still doesn’t help me one bit – I needed something different.

I then realized that GreenfieldTech’s products are the one product line that all developers and vendor steer away from: Programmatical API frameworks. The programmatic framework should provide the most basic feature set, allowing it to be extended and continued onwards at ease. Hence, the suite of GTx API frameworks were born. A set of 3 different APIs provide all the facilities required for building any of the below:

• Dynamic IVR structures – driven via web services
• Recording systems
• Pre-Paid/Post-paid solutions
• Inteligent Network (IN) services
• Automatic Dialers – Predictive, Power, Progressive, Preview
• Broadcasting systems
• and more …

The entire API suite is based upon XML-RPC based web services. When I introduced XML-RPC to one of my customers, he immediately indicated: “But, XML-RPC doesn’t really provide for session oriented persistency. Why are you using XML-RPC?” – The reason is simple – we don’t need session persistency in the API – Asterisk can provide all of that internally. Thanks to Asterisk’s channel oriented architecture, we can regard each of Channel thread in Asterisk as a seperated data container, capable of holding all your session information and persistency information at ease.

Recorder API – http://www.greenfieldtech.net/products/gtrapi
IVR API – http://www.greenfieldtech.net/products/gtvapi
Dialer API – http://www.greenfieldtech.net/products/gtdapi

The best thing about the GTx API suite is this – they are all interconnected. For example, call being handled by the Dialer API can be then handed over to the IVR API, operating within the same Asterisk server or a remotely located Asterisk server (via SIP/IAX2).

In the time to come, I’ll be showing you various XML-RPC structures and example of how to use the GTx API suite, and do all sorts of interesting IVR and Dialer structures.

Tags: Asterisk, dialer, digium, Greenfield Technologies, GreenfieldTech, Interactive Voice Response, IVR, Recorded

21 Jan 09 Thoughts of virtualization – Part III

Ok, it’s been almost 24 hours since I started my serious playing around with Amazon EC2, and I can honestly say that I’m tired – however, I’m very pleased with my results. Like any other experiment, this one started with a requirement. The requirement was to install and operate one of the dialer frameworks I’ve written in the past year on an EC2 based instance. In order to evaluate the process, let’s start with our baseline installation, meaning, what am I using in the real-world:

Hardware and Software Specification

The original machine answered to the following specification: Quad Core CPU, 2GB of RAM, 250GB of Hard Drive. My original machine was running CentOS 5.2 with an x86_64 kernel installed. In terms of software installed, we had Asterisk 1.4.22.1, MySQL 5.X, PHP, FreePBX, Apache and my dialer framework.

Introducing Amazon EC2 AMI

An EC2 AMI is basically an image of a computer, contained with a single installation manifest on the Amazon cloud computing system. AMI’s provide for the simplest manner to start using EC2, as these usually include a pre-defined server installation, that usually has some stuff already installed.

Choosing your AMI

Amazon provides a multitude of AMI’s to work with, unfortunately, most of these are either out-dated and the sheer number of these makes the choice somewhat overwhelming. I decided to start working with a working AMI image of Fedora Core 8, that already had the LAMP stack installed – the one I used was:

The YUM Repository issue

For some strange reason, the AMI images contained in the Amazon repository are all unable to perform any updates to their installed RPM packages. It took me a while to understand what’s wrong, but in general, the fedora project had simply removed the old releases from their repository, so I had to go in and manually modify the /etc/yum.repos.d/ configuration files. For you convenience, here is the repos list that I’m using at this point:

YUM Repositories for Amazon EC2 Fedora images <- click this to download the file

[development]
name=Fedora Core $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/development/$basearch/
enabled=1
gpgcheck=0

[extras-development]
name=Fedora Extras $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/development/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=0

[extras]
name=Fedora Extras $releasever - $basearch
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/$releasever/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

[updates-testing]
name=Fedora Core $releasever - $basearch - Test Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/testing/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-test

[updates-released]
name=Fedora Core $releasever - $basearch - Released Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/$releasever/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

If you are trying to install an instance, you may use the above for any Fedora based AMI on EC2.

Creating my own AMI

While going about and building a single server is fun, I needed a way to create my own installation AMI once I’ve completed my modifications. The Amazon EC2 resources website gives out a whole lot of information, which can be somewhat confusing for the first time reader. So, I searched for a better way to construct my own AMI image. I found the following web page, which was really really helpful: Azeez’s Notes.

Azeez’s blog gives a very concise and to the point, step by step, guide to building your own AMI image – it got me up and running in less than 10 minute – WAY TO GO AZEEZ.

So far, so good …

So far my installed instances are working just fine and I’m currently operating a cluster of a few of these dialer systems. I’m in the process of checking what kind of mileage I’ll get from the EC2 instances, in comparison to a real hardware server – which is really interesting.

My main objective here is to be able to build a dialer-on-demand framework, which will enable my customers to increase their capacity utilizing a simple web interface to initiate my AMI instances. I’ll report back with additional information once I have it.

Tags: Amazon, Asterisk, Cloud computing, dialer, EC2, Virtualization, VMWARE

15 Sep 08 Dialers – Myth or Truth?

Over the past 12 months, I’ve been heavily involved in the development of high-speed dialers. While many companies published a prolific number of automatic dialers (power, predictive, broadcast) – none of these companies ever announced there top speed dialing capability.

An so I ask myself: “What is the reason not to release these numbers publicly?” – interesting, isn’t it. So, I decided to experiment myself and see if I would release public numbers for my dialer. I’ve designed my dialer to be capable of generating upto 35 calls concurrently. My questions was this: “While the dialer is fully capable these numbers in a test scenario, will this number be reached in real life?”

So, I started experimenting with my code – doing a very simple test. I’ve installed my dialer on a Dual Quad Core system, 8GB RAM, and a RAID-1 array as the storage. Now, I’ve setup the system to generate up to 35 calls every second, allowing the server to sustain a total of 180 concurrent calls – 360 channels.

What happend was actually this: the dialer generated about 180 dials within a period of about 8 seconds, and then, simply waited till some of the calls completed in order to generate additional calls. Ok, that is understandable, however, what is the throughput of the dialer? So, I decided to do another test, I filled up my queue with a total of 50,000 records. In addition, I’ve arranged with my carrier to terminate the calls to on of their SigValue systems, to sustain the enormous number of inbounds. In addition, I asked them to perform a small measurement of the average call inits per second. I was shocked with the result!

While the dialer simply peaked upon startup, after 20 seconds of operations it started normalizing at around 6 call initiations per second. But how can that be? how can it be that a dialer capable of dialing 35 calls is slowed down to crawl? – the reason is simple, the latency imposed by the PSTN/VoIP network, the time it takes for the calls to terminate and the actual time the call is left on the air, while the dial happens. All these factors together had given me the notion that there is no practical top speed, as it is in direct relation to the demographics of the dialer operations.

Having said that, is it possible to devise a formula to calcualte this number on a per demographic basis? maybe on a per PSTN/VoIP carrier type? maybe a combination of the both? will a formula such as that will enable for the better creation of broadcast/predictive dialer? – no use dialing tons of number into a demographic that can’t handle it, while at the other side of the spectrum, what demographics will benifit from the utilization of an automated dialer?

Tags: Asterisk, demographic, dialer, predictive, PSTN, vicidial, VoIP

27 Aug 08 Apache vs. Lighttpd (AKA: Lighty)

For a while now I’ve been toying around with the idea of utilizing Lighttpd for various web based applications. One of these application is my Automatic Dialer framework, also known as the GTD-API. The main issue with the GTD-API (besides that it is highly reliant on a MySQL database), is the fact that all requests have to be processed via XML-RPC HTTP post requests.

The main issue that I had was this: in a production scenario, a dialer management system will generate over 100 requests to the XML-RPC server. While Apache is fully capable of rendering services at such a speed, its increasing size and boilerplate automatically introduce a management issue. In addition, as I was trying to build a dialer appliance that can be used in any enterprise, the ever expanding Apache wasn’t a good choice.

While I was looking at both NginX and Lighttpd, the latter captured my eye, thanks to a simple advantage – The integration of FastCGI based PHP was so easy, that it almost troubling that I used Apache all these years.

At this point, once I got Lighhtpd working with my Dialer, I said to myself: “It would be really cool to go about and send status reports back from the dialer, directly to the web client activating the call. In addition, I really don’t want to go about and perform these updates to the database, then query the database – that would, literally, kill the MySQL server.

So, I implemented a local session storage area for each call, which updated the call status as it traverses. The information was stored on the hard drive, which allowed a better response time than the ever indexing MySQL server. The status reports were picked up from the Lighttpd server via an Ajax client (which I didn’t write – I suck at JS) – and it works quite well.

I wonder, can Lighttpd completely replace Apache? … time will tell…

Tags: Ajaz, Apache, Asterisk, dialer, Lighttpd, mysql, php

02 Mar 08 The world’s fastest Asterisk based Dialer

As most of you already know, I’m heavily involved within the Asterisk Open PBX project. Over the course of the past 5 years of my dealing with Asterisk, Asterisk had always suffered a serious flaw, and that is, a single-threaded Manager interface – which usually led to serious dead-locks when writing a multi-threaded server that connects to it.

One of my long time challenges was to surpass the 4-5 originate requests to the Asterisk Manager interface, enabling me to automatically dial more than 4-5 calls at the same second. My initial work had began with the idea of increasing that by a factor of 50%, going up to around 7-8 calls per second – I had achieved that using a combination of smart synchronization between the manager interface and my originating server – and also enabling asynchronous originate requests – however, that methodology had proved to be problematic – in terms of reliability.

I understood that something else had to be devised, something that doesn’t rely completely on the manager interface, and that will allow me to originate calls freely, without clogging up the manager interface. So, I decided to move my interest from the Manager interface, and concentrate on understanding Asterisk’s channel handling, especially, how do calls originating from the manager interface are handled by the Asterisk spooler and the Asterisk channel drivers.

more will follow…

Tags: advertizing, AGI, ami, Asterisk, dialer, digium, freeswitch, manager, openpbx, php, power, sangoma, VoIP