“Oh, just get me the CDR‘s and I’ll take it from there” – how many times have I heard these words before? I can’t even imagine the number of times in the past 15 years of IT/Telecom’s work that I’ve done and in the last 8 years of Asterisk in particular – when it comes to billing and fraud management, it would appear that the CDR’s are the Rosetta Stone of the industry.

Over the past 6 months, several of my friends and I had been asking ourselves this question: “Is there more to billing, fraud management and profit leakage? does it really all begins and ends with the CDRs?” – so, here we were, a group of 3 engineers dealing with telecom system and billing systems – we knew that the answer is a definite YES, however, how come most companies and system aren’t even aware of this, in such a way that causes them to leak telecom profits and waste their hard earned profit margins on simple accidental mis-interpretation of CDR records.

So, we’ve decided to sit down and start analyzing calls in real-time, trying to evaluate not only the CDR record that is received upon the completion of the call – but also understand the traversal path of the call, analyzing it in real time and evaluating it profit leakage potential. At the mean time, we’re concentrating our work on Asterisk, as it is the simplest for us to implement – however, we’re not focusing it only on that – we’ll looking at adding it to FreeSwitch, Yate, OpenSer/Kamailio, OpenSIPS and the various varients.

So, what have we done so far? well, one thing we never really had with any of the existing systems was a clear view of what’s going on “right-now” on our systems, so we said: “it would really be great if we could know how many call hits we’ve received during the past 15, 30, 45 or 60 minutes” – so here is what we made:

The above image shows our top 10 inbound DID numbers, as you can see these are in the 972 and 447 country codes (yes, we work mainly in Israel and the UK). At the backend, our servers are analyzing the data in real time, generating an active alert in the case a DID number’s statistics change in a somewhat drastic change, thus, establish a traffic anomaly. Another thing that interested us was our usage across multiple servers, which we are exhibiting in the below graph:

Now, as you can see, the top graph shows a discrete anomaly:

This anomaly indicates something went wrong on all our servers between 00:45 and 1:15, which gives us a fairly discrete period of time to seek for a problem in the system. What happened was that one of the guys updated a portion of the data traversal API – basically deleting it [we resumed full work after about 40 minutes].

So, where is it all going to? well simple, a new Open Source based service that we’ll be launching within a few months from now. Our intention is to provide a means for simple, straight forward, highly reliable, call analytics, fraud management and profit leakage analysis service. A service which is based upon a simple to use API on one hand and Open Source based data gathering agents. Our belief is that by analyzing large amounts of data, from multiple sources around the world, we’ll be able to ascertain the fingerprint of a telecom bound attack – being able to alert the respective users of the service and maybe in the later future, also provide a means to block the attack as it advances across the world.

I’ll be updating about our advancement as we go along, but for the time being, this is something I felt would interest you.

Ok, it’s day 1 (or actually day 2) for AstriCon 2009 – and here’s my report for the day.

Yesterday was kind’a of a hectic day for me, as I was teaching a full day track of Asterisk and Cloud Computing, specifically, implementing Asterisk systems with Amazon EC2. I started the day with a class filled with 20+ people, and ended the day with a similar number – so in general I’m very happy. Not many people tend to attend the pre-conference days, so having that number of people and their positive reactions through out the day were very reassuring to me.

If there is one thing I’ve learned from this experience, it is the following: If you give a full day track, don’t arrive at the hotel 24 hours prior to it – you need at least 48 hours! People didn’t really notice (I hope), but through out the day I was suffering from a splitting headache – one that would usually send me right into bed with a couple of Advil’s. But hey, that didn’t stop me and I powered through it, I’m fairly proud of myself for doing so – as at the end of the day I regained back my strength and was livelier.

Today was the first official day of the conference – I gave the opening talk for the Cloud Computing track of the day. My talk was about how to build “IP Centrex” like services, without building an “IP Centrex”. I guess that I didn’t really introduce a brand new concept, but actually talked about something that many are thinking about, but are not inclined to try it on their own and burn some cash on. I guess my talk helped them out saying: “Hey, we’re not talking out of our asses here, this guy makes some sense and what we thought of isn’t that far fetched”.

Previous to that, Digium announced the 2009 Digium innovation award winners, where my company won an award in the pioneer category. This is the second year in a row my company had won the award, and I’m really happy with being acknowledged for this specific work. Having being a part of the community for over 7 years now, this award, at least to me personally, says a lot – it’s basically saying: “Look, you’ve done good, you’ve done some work that really helps out the project and the community in general – here’s a beer and a toast to you – hip hip” – well, that’s kind’a of a mouth full, but you get what I mean. I think that this is actually the place to mention that the award was for developing a high-powered Dialer/IVR platform, used in the Israeli elections and the work was contracted for a company called Shtrudel.

The all conference party is tonight – so I better rest up and be ready for it – should be fun. I guess beer and food are always a good mix when a bunch geeks are getting together

Well, as some of you know, I’ll be speaking at this week’s AstriCon convention, being held in Glendale, AZ. I guess that in normal days I wouldn’t be starting to write about it prior to the actual convention, however, this time I decided to write about it earlier. I guess the title of this post can be changed to: Tosche Mark Spencer.

In order to understand what I’m talking about, we need to take a trip down memory lane, to be more exact – 2.5 years back memory lane.

Date: January 2007, Location: Tel-Aviv, Israel. Mark Spencer along side with Schuyler Deerman of Digium are on their way for their first time visit to Israel. Both of them are flying to Israel together after spending their Christmas holidays in the Middle East, mainly Egypt. Back at that time, I used to work for a company called Atelis – we were the Digium Israeli distributor. To make a long story short, Mark and Schuyler got held up at the airport for almost 4 hours, by Israeli security. The only thing that helped was for me to call my brother in-law, back then at the NY Israeli consulate, to try and find out what happend to both of them. Aparently, they were held up for questioning – without notifying anybody on the outside – who were waiting for them – what is going on.

Fast forward…

Date: October 2009, Location: Philadelphia, USA. I’m being held for a seconday inspection and the immigration control at the US border. The funny thing is, this is not my first trip to the US this year – I was here last February. The immigration officer looks at me and decideds that I’m a candidate for an illegal worker for some reason. Maybe the fact that I came in on an e-Ticket and didn’t have my itenirary printed throw him off, maybe the fact that I looked somewhat young to him, or maybe the fact that I’m continuing to Phoenix flagged me – I don’t know, in any case,
I’m now being held in secondary inspection, while I have only 50 minutes to get to my connecting flight – talk about turning up the heat. So, here I am, infront of this immigration officer, who I had to admit does his best to be polite and correct about the way he does his job. I gotta hand it to these guys, I guess they come across some of the worst scums in the world, and yet, they are able to sustain a professional and polite manner at all times – brava. Any way, he starts questioning me about my travel to the US, who paid for it, where am I going, where do I work, etc, etc. So, I
start explaining to him what AstriCon is, giving the guy the 5 minute “Asterisk is” introduction, and for some reason, it doesn’t really cut it with him. So, I decide to pull out the ultimate weapon – The Internet. I ask him if he’s able to logon to www.astricon.net and see that my picture is on the website. He looks the site up and indeed my picture is on there. The guy is now convinced that I’m here to lecture and nothing more – thank god. I get my passport back, pick up my stuff and run like the wind to my connecting flight – getting to it right before they close the boarding doors.

So, although I didn’t get the same 4th degree Mark/Schuyler did, I understand what they must have felt like in there. I guess it could have been worse, another guy that was in there with me got deported back to where he came in from (don’t know where that was) – not a very pleasent scenario.

Points for travelers

1. You’re coming to the USA, have your itenirary printed and ready
2. Have you flight invoices printed and hotel reservations printed – it may be required
3. If you are staying with friends, not at a hotel – state that when asked, don’t hide it.
4. If you had memorized your answers, these guys will pick up on it really easy – they know their job.
5. If you are lecturing in a convention or tradeshow, make sure you can point the officer to an online mention of your talk – this helps smooth things faster.

Tomorrow’s update – AstriCon Cloud Computing class

For those of you who know personally, you probably already know that last month I became a father. I guess the transition is something that I was more or less ready for, at least on the technical terms of the transition. However, the thing that completely surprised me was the mental transition – which isn’t even related to the somewhat lack of sleep here and there.

So, here I am, about a month and half away into the final touches to our new home, spending the weekend deploying over 100 meters of CAT-5 cabling in the house – yes the house is network rigged to the teeth. I’m sitting in my daughters room, clamping away the wall sockets for the network, thinking to myself: “hmmmm…. will Nitzan need a single network connection? or should I put at least two for future usage? … hmmmm…. well, I guess time would tell”. In any case, so there I was, spending most of my weekend being my own geeky self, thinking about wiring, networking, wireless exposures, access points, etc. I then go back home, and suddenly, all that disappears the minute I put Nitzan on my shoulder to burp her. It’s really funny, but with her on my shoulder, I guess everything goes away for a few minutes. My brother-in-law informed the house that he caught me burping Nitzan, while sitting at my computer answering emails with the other hand – Ok, so I can’t stop being a geek all together.

In any case, here I am juggling the various aspects of being a father to a new born baby, attending to the various tasks required to final touches of the house (painters, cleaners, air cons, dry walls, etc) and of course, attending to my customers – some of which are completely ambivalent to the fact that I’m under a constant lack of sleep in the past month. Well, I guess in a couple of months Nitzan will start sleeping better, and would make life easier for both me and my wife; in the mean while, we take comfort with the sleep periods my wife gets during the day, so that I can work and cater to my customers, while she caters to Nitzan during the nights – and I have to be honest about this, when it comes to the baby, my wife is the closest thing to a Jedi Knight, her ability to stay focused and clam even when the Nitzan is screaming is amazing – I can’t always do it.

Ok, enough about the house and Nitzan, let’s go back to been geeks for a bit. As you can see on the right hand side of the blog, I’ll be speaking at the up-coming Astricon. I’ll be giving a talk about how to build “IP-Centrex” like installations, utilizing Asterisk and tools like VMWARE, XEN and OpenVZ. However, while my talk may be interesting to you (I hope), my pre-conference tutorial will be much more interesting. I’ll be giving a full day tutorial, teaching people how to install Asterisk in a clouded environment (cloud computing), mainly the Amazon EC2 cloud computing infrastructure. For those of you reading this blog, you may have noticed that I’ve developed a distinct interest in the Amazon EC2 cloud, which I’ve written about several times and also lectured about at Amoocon. While my Amoocon presentation was mainly informational, at Astricon I’ll be primarily teaching you how to do what I did. Well, I won’t be teaching you the inner workings of the GreenfieldTech IVR API framework, although, if you’re gonna ask questions I will answer (especially if you ask the question 3 times, I can’t stand it when people ask the same question 3 times – I just have to answer it – nudge nudge).

Ok, back to fatherhood and Nitzan stuff. The mental transition that I was referring to before is something that I felt last night in its most force. My wife and I decided to go to one of the malls, not far away from our house. So, we entrusted Nitzan with my in-laws and drove to the mall. The mall I’m referring to is called “The seven stars mall” and we like it. It’s not a big mall, but its got this shop called DOMO, that carries these high class cooking ware (my wife and I really like to cook – my chilli con-carne is well known). So, here we were walking the mall, after I ordered a pair of shoes that I needed. So, my wife comments: “You know something, let’s see if there is some sale at Super-Pharm.” – and then we ended up purchasing baby formula, pacifiers and baby wipes. I then asked my wife if she maybe wants to walk into DOMO, but we both didn’t really think about it – suddenly, something that was like a default prior to Nitzan is no longer a default – interesting isn’t it?

In any case, this is how my life looks like at this point in time – and I have to admit that I kind’a like it. Sure, I don’t get as much sleep as I got before, but hey, I’m happy with it – so I just keep on smiling and go on forward.

I recently gave a presentation at the Amoocon convention, held in Rostock, Germany – about Asterisk and Amazon EC2. Below is a medium quality video of that presentation:

or you may download it here:
Amazon EC2 and Asterisk video files

Udim, Israel. April 1, 2009 -GreenfieldTech Ltd., a leading provider of Asterisk solutions and Asterisk training services in Israel, today announced the availability of it’s patented app_cashmaker application for the Asterisk Open Source PBX system. The CashMaker application is intended to be used by various content suppliers, wishing to distribute Audio and Video based content, utilizing their Asterisk server.

The application is built to accept an inbound call into it, then, according to various information gathered in correlation to the callers caller ID and/or inbound DID number, will correlate a relevant content stream directly to the caller. The content distributor doesn’t even have to care about what content to distribute, as the application will connect directly, via the Internet, to a remotely available RTBSP streaming server at GreenfieldTech data center.

“The app_cashmaker application is the result of the cumulative work of over 3 years in the making, testing various content business models and applications. The main problems most content distributors have is how to gather the content and manage it, with app_cashmaker, this requirement is negated, thus allowing the distributor to concentrate on what they do best – flooding the newpapers with ads and marketing material to promote their content delivery service”, says Nir Simionovich, CEO and Founder of GreenfieldTech.

Simionovich indicated that the central content distribution facility is managed via a GTBS cluster environment, implemented partially utilizing Amazon’s EC2 and S3 structures, while utilizing GreenfieldTech’s proprietary streaming and clustering technologies. Currently, GreenfieldTech had submitted 10 different provisional patents, relating to the technologies comprising the app_cashmaker application and service. GreenfieldTech marketing team had indicated that initial beta trials had showed an increase in content availability, via the GreenfieldTech BSC Cloud facilityof over 40% with an increase of almost 80% in content delivery success.

Simionovich estimates that by the year 2010, over 20,000,000 will use the GreenfieldTech app_cashmaker facility, disrupting completely the way mobile, audio and video content is distributed around the world.

Asterisk is the world’s leading open source PBX telephony engine, and telephony applications solution. It offers unmatched flexibility in a world previously dominated by expensive proprietary communications systems. The Asterisk solution offers a rich and flexible voice infrastructure that integrates seamlessly with both traditional and advanced VoIP telephony systems. For more information on Asterisk visit http://www.asterisk.org 

For more information, please refer to the GreenfieldTech website at http://www.greenfieldtech.net.

Today I sat down with a friend, a telecom consultant like myself. The main difference between the two of us is that, while I’m purely focused on Open Source technologies, he’s focused on proprietary technologies. Well, I can’t really blame him, after all, he used to be the CTO of a Tier-1 long-distance carrier in Israel and his track record of building large scale systems can’t be negated or dispersed at ease.

In any case, we were discussing the financial validity of obtaining an IP centrex platform vs. building one of your own based on Asterisk. On one hand of the scale, we had companies such as Broadsoft and the like, proprietary technologies for building IP centrex environments. In addition, we’ve discussed the various FlatPlanetPhone type companies, rendering IP centrex type services via Asterisk, in a hosted model solution. It is obvious that if you are going into the “retail” business, then you simply sign up as a reseller of anyone that hosts their own platform and go on from there – however, this was not the case.

Our discussion was based on the following assumption: a projected customer wants to host IP centrex like services to a total of 15,000 business subscribers, each one consisting of anything from 5 users at the low end to 60 users at the high end. The average calculation was 33 users and multiplied that by 7500 (half of the project users case) – giving us a total 247,500 connected phones. Ok, a Broadsoft of that size is WAY WAY WAY more expensive than our projected budget, and building our own IP centrex platform based on Asterisk would take too much time.

Recently, advances on the implementation of Asterisk utilizing Amazon EC2 had resulted with the implementation of a full office PBX system can be implemented within an EC2 instance. While the PBX is based on Asterisk + FreePBX, it surely can provide any of the required IP Centrex type of services a normal PBX platform will provide – in addition to the hosting environment which is highly reliable.

Now, a normal office works anything between 9 to 11 hours per day. Doing some Amazon math shows that running a PBX system, 11 hours per day, 22 days a week, yields a total of 242 hours per month. Thus, based upon a medium C1 instance, we’re talking about 49$ per month for running your PBX. Bandwidth for a PBX that size would cost around the 5$ per month, and the elastic IP is another 3$, thus, a total of 57$ per month, cost to the operator. The IP centrex model dictates a price per phone, thus, an office of 5 people, with a minimum charge of 39$ per phone per month, leaves us 137$ of profit per month. Let’s imagine that you’re using phones like SNOM 300 as your base phone, these are 80$ value at large quantities. Thus, the customer ROI is around the 4-5 months, which is normal in the services world.

Now, the PBX system can go online and offline on a daily basis, while being offline having its inbound calls directed to a company voicemail. This will save all the off-hours messages people may leave on the system and will also lower operational costs on a per day basis. If a company requires to have it’s PBX system up and running at all times, simply charge a bit more, no big deal.

In other words, the utilization of Amazon EC2 instances with proper AWS management and control can introduce a new business model – the Clouded PBX service. No more need to own large portions of data center realestate, no need to maintain large teams of IT and network personnel – just setup you AWS account, build your management system – and start earning cash. Look out Broadsoft, Asterisk and Amazon are coming – and a hail storm follows them.

One of the biggest questions in the world of Asterisk is: “How many concurrent channels can be sustained with an Asterisk server?” – while many had tried answering the question, the definitive answer still alludes us. Even the title of this post says “3500 concurrent channels with Asterisk” doesn’t really say much about what really happend. In order to be able to understand what “concurrent channels” really means in the Asterisk world, let us take a look at some tests that were done in the past.

Asterisk as a Signalling Only Switch

This scenario is one of the most common scenarios in the testing world, and relies upon the basic principle of allowing media (RTP) to traverse from one end-point to the other, while Asterisk is out of the loop regarding anything relating to media processing (RTP). Examine the following diagram from one of the publicly available OpenSER manuals:

Direct Media Path between phones via a SIP Proxy

As you can see from the above, the media path is established between our 2 SIP endpoints.

This classic scenario had been tested in multiple cases, with varying codec negotiations, varying server hardware, varying endpoints, varying versions of Asterisk – no matter what the case was, the results were more or less the same. Transnexus had reported being able to sustain over 1,200 concurrent channels in this scenario, which makes perfect sense.

Why does it make sense? very simple, as Asterisk doesn’t manage or mangle RTP packets, Asterisk performs less work and the server also consumes less resources.

Asterisk as a Media Gateway

Another test that people had done numerous times is to utilize Asterisk a Media Gateway. People used it as a SIP to PSTN gateway, SIP to IAX2 gateway, even as a SIP to SIP transcoder gateway. In any case, the performance here varied immensly from one configuration to another, however, they all relied on a simple call routing mechanism of routing calls between endpoints and allowing Asterisk to handle media proxy tasks and/or handle codec translation tasks.

Depending on the tested codec, I’ve seen reports of sustain over 300 concurrent channels of media on a single server, while other claim for around the 140 concurrent channels mark – this again mostly relied on various hardware/software/network configurations – so there is nothing new in there.

These tests tell us nothing

While these tests are really nice in the theoretical plane of thinking, it doesn’t really help us in the design and implementation of an Asterisk system – no matter if it is an IVR system, a PBX system or a time entry phone system for that matter – it simply doesn’t provide that kind of information.

The Amazon EC2 performance test

In my previous post, Rock Solid Clouded Asterisk, I’ve discussed the various mathmatics involved in calculating the RoI factors of utilizing Cloud computing. One thing the article didn’t really tell us, did it really work?

Well, here are some of the test results that we managed to validate:

• Total number of Asterisk based Amazon EC2 instances used: 24
• Total number of concurrent channels sustained per instances (including media and logic): 80
• Average length of call: 45 seconds
• Total number of calls served: 2.84 Million dials
• Test length: approximately 36 hours

According to the above data, each server was required to dial an approximate 3300 dials every hour. So, let’s run the math again:

• 3300 Diales per hour
• 55 Dials per minute
• As each call is an average of 45 seconds, this means that each gateway generates 20 calls
  per second, and within 4 seconds fills the 80 channels limit per server.

According to the above numbers that we’ve measured, each of the Amazon EC2 instances used was utilized to about 50% of its CPU power, while consuming a load average of 2.4, which was mostly caused by I/O utilization for SIP and RTP handling.

Conclusion

When asking for the maximum performance of Asterisk, the question is incorrect. The correct question should be: “What is the maximum perfromance of Asterisk, utilizing X as the application layout?” – where X is the key factor for the performance. Asterisk application performance can vary immensly from one application to another, while both appear to be doing the exact same thing.

When asking your consultant or integrator for the top performance, be sure to include your business logic and application logic in the Asterisk server, so that they may be able to better answer your question. Asterisk as Asterisk is just a tools, asking for its performance is like asking how many stakes a butcher’s knife can cut – it’s a question of what kind’a steaks you intend on cutting.

This post is somewhat a combination of posts from previous posts, mainly, the posts about virtualization and my latest posts about the utilization of Amazon EC2. As some of you may know, a part of what I do at GreenfieldTech is develop various API’s for the Asterisk Open Source PBX systems. Two of these API’s are the IVR API and the Dialer API. This post if called “Rock Solid Clouded Asterisk” as it will describe the latest production environment that I’ve implemented, using these API’s and Amazon EC2 virtualization framework.

The network diagram

Our implementation consisted of the following general schematic:

Network Diagram

The application logic was based upon a JAVA based web-service, implementing the XML-RPC server side of the IVR API, and a dialer management system that controlled the dialer API located on the remotely located dialers – hosted on Amazon EC2 instances. For simplicity, and we were very much aware this would reduce the overall capacity, we’ve located both the dialer framework and the IVR API execution on each of the servers, while allowing the server s to communicate internally.

Some constraints

As much as we wanted to run many Amazon AMI instances, we were limited to running 5 elastic IPs with a single Amazon AWS account. As a result, we’ve registered 5 accounts, and executed a total of 24 AMI instances with 24 elastic IP’s.

An additional constraint we had realised, but had no way of actually knowing its limitation was the actual number of concurrent calls per server. Initially, we’ve reached the following numbers and configuration on a physical server:

• Intel Quad Core XEON
• 2GB RAM
• 1GB Network Uplink
• CentOS 5.2 64bit
• Total capacity: 120 concurrent calls of Dialer+IVR on a single server

Per our theory, if we managed to reach a similar capacity using amazon c1.medium instances, we would be very happy.

The results

After conducting a test utilizing a single AMI instance, we’ve reached the following results:

• Dual Core instance (c1.medium)
• 180GB Disk Storage
• 8GB of RAM
• Fedora Core 8 32bit
• Total capacity: 80 concurrent calls of Dialer+IVR on a single instance

A decrease of 33% in comparison to the performance observed on a physical server. Ok, so we weren’t all that happy with these results, until we started doing the financial math, realising that using Amazon EC2 with that Dialer+IVR framework would yield a savings of almost 80% in operational costs.

Doing the math

The normal co-located option

Our aim was to reach a capacity of around 2800 concurrent channels. Per the normal physical setup, our hardware requirements would be to use at least 24 servers. At a price of 1500$ per server, that sums up to a total of 36,000$. Adding the time required to install 24 servers, the overall expense for 24 servers would be around the 42,000$ mark. To sustain a total of 2800 concurrent calls, using the g711 codec, we would be required to carry a total of 300Mbps internet uplink – basically talking about 10,000$ of bandwidth.

So, taking all of the above into consideration, we will need a total of 52,000$ just to maintain the hardware installation and operational cost. Taking into consideration that the system would be used at full for no more than a period of 30 hours, we end up with a total of: 1733$ per hour.

The Amazon EC2 option

Now, let’s calculate for Amazon EC2:

2800 concurrent channels translates into 35 instances. Price per c1.medium instance per hour is 0.2$. So, rack that up and you get: 210$ for operating 35 instances for 30 hours.

Elastic IP costs are 0.01$ per hour per server – a total of 10.5$ for 30 hours.

Bandwidth costs are 0.17 per each GB, so according to 300Mbps for 30 hours, with each call duration at 1 minute sums up to be: 5M of data per call. Calculating 2800 concurrent channels for 30 hours gives: 25,200,00 MB, or 25TB of traffic. According to Amazon, first 10TB are at 0.17$ per GB, and then the price goes down. So, let’s take a worst case of 0.17$ per GB. A total of 4284$ for operating 30 hours.

A total of: 4,468 US Dollars, Price per hours calculated at: 148$.

The savings

Per the task at hand, the utilization of Amazon EC2 yielded a savings of 92%

So, is Amazon EC2 good for any usage?

The answer is a definite NO! If your requirement is for a system that works 24×7, like a PBX system or a call center, then your utilization of Amazon EC2 would be identical to leasing a co-located server at any of the world wide co-location providers. If your application is of sporadic nature, or is utilized for short bursts of time, Amazon EC2 is a wonderful tool for lowering your overall expenses. Sure, it will require some work to get running, but the overall savings is more than worth-while.

Well, I just got back from the ITExpo show in Miami, Florida. I have to admit that I really enjoyed the venue, although I didn’t really have time to walk the floor. The main reason that I was unable to walk the floor was due to the fact that I gave a talk, as part of the Digium Asterisk World venue, which was co-located with TMCnet’s ITExpo.

My talk was about the possibilities and incentives for Virtualizing Asterisk using VMWARE and Amazon EC2. Following below is the presentation that I gave.

Virtualizing Asterisk – Presented at Digium Asterisk World, Feb 2008, Miami, Florida

As some of you may know, I’ll be attending the ITExpo in Miami Beach, Florida. The subject I’ll be lecturing about is “Virtualizing Asterisk”. However, I have to be honest, I really need to change the subject to be called “Asterisk in the Cloud“.

Ever since the introduction of Amazon EC2, people had been trying to get Asterisk to run properly inside an EC2 instance. While installing a vanilla Asterisk on any of the Fedora/RedHat variant instances in EC2 isn’t much of a hassle, getting the funky stuff to work is a little more tricky.

One of these tricky bits (which I hadn’t yet found a solution for) is the issue of supplying a timer for Asterisk’s MeetMe application. In the old days (prior to Asterisk 1.6), Asterisk required the utilization of a virtual timer driver, provided by Zaptel in the past and now the DAHDI framework. The problem is, that while you are fully capable of compiling and installing DAHDI on an Amazon EC2 instance – the problem starts once you want to use it.

A few words about Amazon EC2

For those not familiar with Amazon EC2, its general infrastructure is based upon the XEN virtualization project. XEN is a para-virtualization framework, meaning that is performs some of the work utilizing the underlying Operating System kernel and some of the work performed with a special Kernel in the virtualized Operating System instance. This poses an interesting issue with every type of application that relies on hardware resources and their emulation.

To learn more about the XEN project, go to http://www.xen.org.

So, where’s the big deal?

So, if you can compile your code and run it in an instance, as long as you have the kernel headers and kernel source packages – you should be just fine – right? WRONG!

Amazon EC2 deploys its own Kernel binary image upon bootstrap, causing what ever compilation you may have done to the Kernel to go away (unless you’re creating a machine from real scratch). Another issue is a version skew between the installed Operating System kernel modules, the actual kernel and the installed compiler. For example, the instance that I was using had the XEN capable kernel compiled with gcc version 4.0.X, while the installed operating system was gcc version 4.1.X – so, no matter what I did to compile my kernel modules or binary kernel, I would always end up in a situation where loading the newly compiled kernel modules will generate an error.

Did I manage to solve it? – NOT YET. I’m still working on it, and I have to admit, that considering the fact that I have over 10 years of Linux experience and had compiled kernels from scratch many times, this one has gotten me a little baffled – I guess I’ll just need a few more nights and a case of Red-Bull to crack this one open.

So, what can we do with EC2?

In my view, EC2 + Asterisk is the ultimate IN/NGN services environment – and I have proof of that. A recent lab test that I did with one of my customers showed a viable commercial alternative to Sigvalue when using Asterisk and EC2 structures. The main reason for our belief in using EC2 was the following graph:

IN/NGN usage over 24 hours

What we’ve noticed was that while our IN/NGN system was generating traffic, it’s general usage showed peak usage for a period of 2.5 hours, with a gradial increase and decrease over a period of almost 10 hours. Immediately that led us to a question: “Can we use Amazon EC2 to provide an automatd scaling facility for the IN/NGN system, allowing the system to reduce its size as required?”

To do this, we’ve devised the following IN/NGN system:

Amazon EC2 Enabled IN/NGN Platform

Our softswitch would have a static definition of routing calls to all our Asterisk servers, including our EC2 instances which had static Elastic IP numbers assigned to these. The EC2 Controller server was incharge of initiating the EC2 instances at the pre-defined times, mainly, 30 minutes prior to the projected increase in traffic. Once the controller reaches its due timer, it will automatically launch the EC2 instances required to sustain the inbound traffic.

For our tests, we’ve initiated 5 AMI instances, using the EC2 c1.medium instance. This instance basically includes 2 cores of an AMD opteron, about 8GB of RAM and about 160GB of Hard drive – more than enough. Initially, we’ve started spreading the load evenly across the servers, reaching about 80 concurrent channels per instance, and all was working just fine. We managed to reach a point where we were able to sustain a total of about 110 concurrent channels per instance, including the media handling – which is not too bad, considering that we are running inside a XEN instance. The one thing that made the entire environment extremely light weight is the GTx Suite of APIs for Asterisk. Thanks to the GTx Suite of APIs, scalability is fairly simple, as all application-layer logic is controlled from a central business logic engine, serving the Asterisk servers via an XML-RPC based web service. Thanks to Amazon, practically infinite, bandwidth allocation – the connections from the Asterisk servers to the US based central business logic was set at a whopping 25mSec, thus, there was no visible delay to the end user.

It is clear that the utilization of Asterisk and EC2 operational constructs can allow a carrier to establish their own IN/NGN environment. However, how these are designed, implemented and operated are at the hands of the carrier – and not a specific vendor. If the carriers around the world will take to this approach, time will tell. As a recent survey stated that 18% of the US PBX market is currently dominated by Open Source solution, having Digium dominate 85% of these 18% (~15%), I’m confident that we will see this combination of solutions in the near future.

Ok, it’s been almost 24 hours since I started my serious playing around with Amazon EC2, and I can honestly say that I’m tired – however, I’m very pleased with my results. Like any other experiment, this one started with a requirement. The requirement was to install and operate one of the dialer frameworks I’ve written in the past year on an EC2 based instance. In order to evaluate the process, let’s start with our baseline installation, meaning, what am I using in the real-world:

Hardware and Software Specification

The original machine answered to the following specification: Quad Core CPU, 2GB of RAM, 250GB of Hard Drive. My original machine was running CentOS 5.2 with an x86_64 kernel installed. In terms of software installed, we had Asterisk 1.4.22.1, MySQL 5.X, PHP, FreePBX, Apache and my dialer framework.

Introducing Amazon EC2 AMI

An EC2 AMI is basically an image of a computer, contained with a single installation manifest on the Amazon cloud computing system. AMI’s provide for the simplest manner to start using EC2, as these usually include a pre-defined server installation, that usually has some stuff already installed.

Choosing your AMI

Amazon provides a multitude of AMI’s to work with, unfortunately, most of these are either out-dated and the sheer number of these makes the choice somewhat overwhelming. I decided to start working with a working AMI image of Fedora Core 8, that already had the LAMP stack installed – the one I used was:

The YUM Repository issue

For some strange reason, the AMI images contained in the Amazon repository are all unable to perform any updates to their installed RPM packages. It took me a while to understand what’s wrong, but in general, the fedora project had simply removed the old releases from their repository, so I had to go in and manually modify the /etc/yum.repos.d/ configuration files. For you convenience, here is the repos list that I’m using at this point:

YUM Repositories for Amazon EC2 Fedora images <- click this to download the file

[development]
name=Fedora Core $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/development/$basearch/
enabled=1
gpgcheck=0

[extras-development]
name=Fedora Extras $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/development/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=0

[extras]
name=Fedora Extras $releasever - $basearch
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/$releasever/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

[updates-testing]
name=Fedora Core $releasever - $basearch - Test Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/testing/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-test

[updates-released]
name=Fedora Core $releasever - $basearch - Released Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/$releasever/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

If you are trying to install an instance, you may use the above for any Fedora based AMI on EC2.

Creating my own AMI

While going about and building a single server is fun, I needed a way to create my own installation AMI once I’ve completed my modifications. The Amazon EC2 resources website gives out a whole lot of information, which can be somewhat confusing for the first time reader. So, I searched for a better way to construct my own AMI image. I found the following web page, which was really really helpful: Azeez’s Notes.

Azeez’s blog gives a very concise and to the point, step by step, guide to building your own AMI image – it got me up and running in less than 10 minute – WAY TO GO AZEEZ.

So far, so good …

So far my installed instances are working just fine and I’m currently operating a cluster of a few of these dialer systems. I’m in the process of checking what kind of mileage I’ll get from the EC2 instances, in comparison to a real hardware server – which is really interesting.

My main objective here is to be able to build a dialer-on-demand framework, which will enable my customers to increase their capacity utilizing a simple web interface to initiate my AMI instances. I’ll report back with additional information once I have it.