15 Oct 09 Astricon 2009 – Glendale, AZ – Part II

Ok, it’s day 1 (or actually day 2) for AstriCon 2009 – and here’s my report for the day.

Yesterday was kind’a of a hectic day for me, as I was teaching a full day track of Asterisk and Cloud Computing, specifically, implementing Asterisk systems with Amazon EC2. I started the day with a class filled with 20+ people, and ended the day with a similar number – so in general I’m very happy. Not many people tend to attend the pre-conference days, so having that number of people and their positive reactions through out the day were very reassuring to me.

If there is one thing I’ve learned from this experience, it is the following: If you give a full day track, don’t arrive at the hotel 24 hours prior to it – you need at least 48 hours! People didn’t really notice (I hope), but through out the day I was suffering from a splitting headache – one that would usually send me right into bed with a couple of Advil’s. But hey, that didn’t stop me and I powered through it, I’m fairly proud of myself for doing so – as at the end of the day I regained back my strength and was livelier.

Today was the first official day of the conference – I gave the opening talk for the Cloud Computing track of the day. My talk was about how to build “IP Centrex” like services, without building an “IP Centrex”. I guess that I didn’t really introduce a brand new concept, but actually talked about something that many are thinking about, but are not inclined to try it on their own and burn some cash on. I guess my talk helped them out saying: “Hey, we’re not talking out of our asses here, this guy makes some sense and what we thought of isn’t that far fetched”.

Previous to that, Digium announced the 2009 Digium innovation award winners, where my company won an award in the pioneer category. This is the second year in a row my company had won the award, and I’m really happy with being acknowledged for this specific work. Having being a part of the community for over 7 years now, this award, at least to me personally, says a lot – it’s basically saying: “Look, you’ve done good, you’ve done some work that really helps out the project and the community in general – here’s a beer and a toast to you – hip hip” – well, that’s kind’a of a mouth full, but you get what I mean. I think that this is actually the place to mention that the award was for developing a high-powered Dialer/IVR platform, used in the Israeli elections and the work was contracted for a company called Shtrudel.

The all conference party is tonight – so I better rest up and be ready for it – should be fun. I guess beer and food are always a good mix when a bunch geeks are getting together

Tags: Amazon, Amazon EC2, Asterisk, Business, Cloud computing, digium, EC2, GreenfieldTech, Innovation Award, Virtualization, VMWARE

14 May 09 Asterisk and Amazon EC2 – Amoocon Presentation

I recently gave a presentation at the Amoocon convention, held in Rostock, Germany – about Asterisk and Amazon EC2. Below is a medium quality video of that presentation:

or you may download it here:
Amazon EC2 and Asterisk video files

Tags: AGI, Amazon, Amazon EC2, Asterisk, Cloud computing, Dialers, economy, GreenfieldTech, php, PHPAGI, Virtualization, VMWARE

02 Feb 09 So long SigValue – Hello Asterisk + EC2!

As some of you may know, I’ll be attending the ITExpo in Miami Beach, Florida. The subject I’ll be lecturing about is “Virtualizing Asterisk”. However, I have to be honest, I really need to change the subject to be called “Asterisk in the Cloud“.

Ever since the introduction of Amazon EC2, people had been trying to get Asterisk to run properly inside an EC2 instance. While installing a vanilla Asterisk on any of the Fedora/RedHat variant instances in EC2 isn’t much of a hassle, getting the funky stuff to work is a little more tricky.

One of these tricky bits (which I hadn’t yet found a solution for) is the issue of supplying a timer for Asterisk’s MeetMe application. In the old days (prior to Asterisk 1.6), Asterisk required the utilization of a virtual timer driver, provided by Zaptel in the past and now the DAHDI framework. The problem is, that while you are fully capable of compiling and installing DAHDI on an Amazon EC2 instance – the problem starts once you want to use it.

A few words about Amazon EC2

For those not familiar with Amazon EC2, its general infrastructure is based upon the XEN virtualization project. XEN is a para-virtualization framework, meaning that is performs some of the work utilizing the underlying Operating System kernel and some of the work performed with a special Kernel in the virtualized Operating System instance. This poses an interesting issue with every type of application that relies on hardware resources and their emulation.

To learn more about the XEN project, go to http://www.xen.org.

So, where’s the big deal?

So, if you can compile your code and run it in an instance, as long as you have the kernel headers and kernel source packages – you should be just fine – right? WRONG!

Amazon EC2 deploys its own Kernel binary image upon bootstrap, causing what ever compilation you may have done to the Kernel to go away (unless you’re creating a machine from real scratch). Another issue is a version skew between the installed Operating System kernel modules, the actual kernel and the installed compiler. For example, the instance that I was using had the XEN capable kernel compiled with gcc version 4.0.X, while the installed operating system was gcc version 4.1.X – so, no matter what I did to compile my kernel modules or binary kernel, I would always end up in a situation where loading the newly compiled kernel modules will generate an error.

Did I manage to solve it? – NOT YET. I’m still working on it, and I have to admit, that considering the fact that I have over 10 years of Linux experience and had compiled kernels from scratch many times, this one has gotten me a little baffled – I guess I’ll just need a few more nights and a case of Red-Bull to crack this one open.

So, what can we do with EC2?

In my view, EC2 + Asterisk is the ultimate IN/NGN services environment – and I have proof of that. A recent lab test that I did with one of my customers showed a viable commercial alternative to Sigvalue when using Asterisk and EC2 structures. The main reason for our belief in using EC2 was the following graph:

What we’ve noticed was that while our IN/NGN system was generating traffic, it’s general usage showed peak usage for a period of 2.5 hours, with a gradial increase and decrease over a period of almost 10 hours. Immediately that led us to a question: “Can we use Amazon EC2 to provide an automatd scaling facility for the IN/NGN system, allowing the system to reduce its size as required?”

To do this, we’ve devised the following IN/NGN system:

Our softswitch would have a static definition of routing calls to all our Asterisk servers, including our EC2 instances which had static Elastic IP numbers assigned to these. The EC2 Controller server was incharge of initiating the EC2 instances at the pre-defined times, mainly, 30 minutes prior to the projected increase in traffic. Once the controller reaches its due timer, it will automatically launch the EC2 instances required to sustain the inbound traffic.

For our tests, we’ve initiated 5 AMI instances, using the EC2 c1.medium instance. This instance basically includes 2 cores of an AMD opteron, about 8GB of RAM and about 160GB of Hard drive – more than enough. Initially, we’ve started spreading the load evenly across the servers, reaching about 80 concurrent channels per instance, and all was working just fine. We managed to reach a point where we were able to sustain a total of about 110 concurrent channels per instance, including the media handling – which is not too bad, considering that we are running inside a XEN instance. The one thing that made the entire environment extremely light weight is the GTx Suite of APIs for Asterisk. Thanks to the GTx Suite of APIs, scalability is fairly simple, as all application-layer logic is controlled from a central business logic engine, serving the Asterisk servers via an XML-RPC based web service. Thanks to Amazon, practically infinite, bandwidth allocation – the connections from the Asterisk servers to the US based central business logic was set at a whopping 25mSec, thus, there was no visible delay to the end user.

It is clear that the utilization of Asterisk and EC2 operational constructs can allow a carrier to establish their own IN/NGN environment. However, how these are designed, implemented and operated are at the hands of the carrier – and not a specific vendor. If the carriers around the world will take to this approach, time will tell. As a recent survey stated that 18% of the US PBX market is currently dominated by Open Source solution, having Digium dominate 85% of these 18% (~15%), I’m confident that we will see this combination of solutions in the near future.

Tags: Amazon, Asterisk, Broadsoft, digium, EC2, Nortel, Personeta, S3, SigValue, Sylantro, Virtualization

21 Jan 09 Thoughts of virtualization – Part III

Ok, it’s been almost 24 hours since I started my serious playing around with Amazon EC2, and I can honestly say that I’m tired – however, I’m very pleased with my results. Like any other experiment, this one started with a requirement. The requirement was to install and operate one of the dialer frameworks I’ve written in the past year on an EC2 based instance. In order to evaluate the process, let’s start with our baseline installation, meaning, what am I using in the real-world:

Hardware and Software Specification

The original machine answered to the following specification: Quad Core CPU, 2GB of RAM, 250GB of Hard Drive. My original machine was running CentOS 5.2 with an x86_64 kernel installed. In terms of software installed, we had Asterisk 1.4.22.1, MySQL 5.X, PHP, FreePBX, Apache and my dialer framework.

Introducing Amazon EC2 AMI

An EC2 AMI is basically an image of a computer, contained with a single installation manifest on the Amazon cloud computing system. AMI’s provide for the simplest manner to start using EC2, as these usually include a pre-defined server installation, that usually has some stuff already installed.

Choosing your AMI

Amazon provides a multitude of AMI’s to work with, unfortunately, most of these are either out-dated and the sheer number of these makes the choice somewhat overwhelming. I decided to start working with a working AMI image of Fedora Core 8, that already had the LAMP stack installed – the one I used was:

The YUM Repository issue

For some strange reason, the AMI images contained in the Amazon repository are all unable to perform any updates to their installed RPM packages. It took me a while to understand what’s wrong, but in general, the fedora project had simply removed the old releases from their repository, so I had to go in and manually modify the /etc/yum.repos.d/ configuration files. For you convenience, here is the repos list that I’m using at this point:

YUM Repositories for Amazon EC2 Fedora images <- click this to download the file

[development]
name=Fedora Core $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/development/$basearch/
enabled=1
gpgcheck=0

[extras-development]
name=Fedora Extras $releasever - Development Tree
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/development/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=0

[extras]
name=Fedora Extras $releasever - $basearch
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/extras/$releasever/$basearch/
enabled=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-extras
gpgcheck=1

[updates-testing]
name=Fedora Core $releasever - $basearch - Test Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/testing/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora-test

[updates-released]
name=Fedora Core $releasever - $basearch - Released Updates
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/updates/$releasever/$basearch/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

[base]
name=Fedora Core $releasever - $basearch - Base
baseurl=http://archives.fedoraproject.org/pub/archive/fedora/linux/core/$releasever/$basearch/os/
enabled=1
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-fedora

If you are trying to install an instance, you may use the above for any Fedora based AMI on EC2.

Creating my own AMI

While going about and building a single server is fun, I needed a way to create my own installation AMI once I’ve completed my modifications. The Amazon EC2 resources website gives out a whole lot of information, which can be somewhat confusing for the first time reader. So, I searched for a better way to construct my own AMI image. I found the following web page, which was really really helpful: Azeez’s Notes.

Azeez’s blog gives a very concise and to the point, step by step, guide to building your own AMI image – it got me up and running in less than 10 minute – WAY TO GO AZEEZ.

So far, so good …

So far my installed instances are working just fine and I’m currently operating a cluster of a few of these dialer systems. I’m in the process of checking what kind of mileage I’ll get from the EC2 instances, in comparison to a real hardware server – which is really interesting.

My main objective here is to be able to build a dialer-on-demand framework, which will enable my customers to increase their capacity utilizing a simple web interface to initiate my AMI instances. I’ll report back with additional information once I have it.

Tags: Amazon, Asterisk, Cloud computing, dialer, EC2, Virtualization, VMWARE

30 Nov 08 Thoughts of virtualization – Part II

I really like going to events at the GarageGeeks (www.garagegeeks.org) – apart from meeting with some friends, having a good laugh and enjoying the casual beer, the technical talks and the general subjects that are being attended at a gathering are always interesting.

This time, the Geeks had decided to focus on Cloud Computing, trying to introduce the somewhat highly evolving field of the IT/Computing industry to the visitors of Garage. Now, I have to admit that during the past few weeks I’ve been studying the various aspects of utilizing Amazon EC2 services, and I’ve come to the somewhat problematic conclusion: “Cloud computing is a wonderful solution, as long as you have clear and concise control over your computing cloud!”

The main problem with traditional cloud computing, as offered by Amazon and others, is that the physical boundaries of your system are somewhat unknown – not to you and in some cases, not to your provider either. Let us imagine a Tier based architecture platform, implemented utilizing Amazon EC2 technologies – our web front-end may be located in the US, the application server in the UK and the Database in Zimbabwe for all that matters – in accordance to EC2 and traditional cloud computing, that is fairly normal.While EC2 and some other providers had introduced the idea of a Zone, making sure that your virtual servers are all located in the same physical data center, other solutions are yet to follow.

The above design introduces an interesting “Critical Path” management problem, tied deep into the Tier based architecture design pattern. Companies such as GigaSpaces and others had made it their business to cash in on the problem, by providing a distributed, memory based, middle-ware – literally sweeping the “Critical Path” problem under the rug. However, as the platform will evolve, “Critical Path” issues have a tendency to re-appear, usually, a lot more serious than the original problem – due to new constraints and new operational paradigms within the operational construct of the cloud platform.

So, question be asked: “Is cloud computing good? Can you really build a full service based upon cloud computing services?” – The answer isn’t a straight forward yes or no. For example, while I have a couple of servers on EC2, serving me as development servers mainly, I’ve built my own little cloud environment on my own servers – used for production usage. I’ve used a mixture of OpenVZ servers tied together with my own provisional interface to create a seamless cloud, enabling me to replicate servers, control their operational structures at east and most important – expand my cloud rapidly by introducing new servers to my cloud.

Is my paradigm good for all purposes? I can’t really say – it works for me, it may not work for you. The thing is this, don’t expect cloud computing to come in and solve all your problems, you may run into new ones you are completely unfamiliar with.

Tags: Amazon EC2, Cloud computing, openVZ, Virtualization, VMWARE