The Asterisk scaleability Unicorn

Ok, the picture shows a donkey not a Unicorn – as you know, Unicorns are very hard to find. Asterisk Scaleability is somewhat of a unicorn – not because it doesn’t exist, it is a little tricky to do and get it right first time.

Over the years, there had been multiple approaches to building a scaleable Asterisk platform, most of them relied on the same principals: multiple Asterisk servers, singular point of entry with load balancing, single point of exit with LCR. Normally, when you talk Class-4 services (call routing, DID services, Calling Cards), this methodology would work just fine. When it comes to Class-5 (Centrex, Voicemail, Queues), things tend to get a bit more complex – but again, the basic methodology applies and remains. Over the years, we’ve seen contenders come and go, FreeSwitch, Kamailio, OpenSIPS, SER, OpenSEMS – they are all a means to an end, just get the number of concurrent calls and CPS ratio higher.

The question is this: “Is there a singular approach to Asterisk scaleability? is there a bullet proof recipe that we can use to achieve this Unicorn type configuration?” – the answer is: NO! – it is very much dependent on your application, your client side application, your general usage patterns and what kind of agility you are looking to expose to the end consumer.

Since the inception of Asterisk, and specifically since the inception of FreeSwitch, many people had been dissing Asterisk as being a monolithic environment. Many times, if you ask someone – “what does that mean?” – you would end up with a very googly eyed face, not really understanding what monolithic means. Yes, Asterisk is by definition a monolithic environment, which means, it was designed to work a self enclosed unit. If you think about it, how many PBX systems do you know that are not monolithic. The question in that case is: “If Asterisk is monolithic, how can we scale and expand it? how can we build something really big from something like Asterisk?”.

In martial arts you always learn to use your opponents strength as their weakness, as your weakness as your strength. If Asterisk’s monolithic nature is its weakness, let’s try and make it into its strength. How do we do that? we make sure that any decision (process, calculation, state handling, etc) that is cross platform is handled outside of Asterisk, while keeping call control and media handling at the monolithic layer. This yields two distinct advantages: we can develop our cross platform logic at ease, without impacting our Asterisk development process, we can develop our Asterisk logic as a singular unit and expand it as required, simply by adding more computation units horizontally. In network and platform design there is a simple rule of thumb – growing deep is complex, growing wide is simple. If the question of scaleability becomes a question of brute forcing additional computation resources, the issue is simple. If scaling out requires changes in the computational structure – you’ve done something wrong.

Over the years, we’ve developed several large scale Asterisk platforms. These had recently hit the combined user number of 15 Million, with over 850 Million minutes served on all platforms combined. Some of these are carrier oriented, some are social oriented – but in all of them the scaleabilty factor was important. In other words, the Unicorn is out there, we’ve actually managed to find it several times, each time somewhere else – but it was always grazing in similar locations. If you keep looking for the bulls, you will surely miss the Unicorn standing at the right of the road – right next to you.

Stanley is gone – Welcome PHPARI

In my previous post I’ve announced the bootstrapping of a new PHP project, called “Project Stanley”. Project Stanley was an attempt at creating a Asterisk ARI developer kit, based upon the PHP programming language (yes, I call it a programming language).

Shortly after initiating the development and reaching a point where our code was actually able to do something, we realized that we’ve gone the wrong way. The wire frame we’ve created relied heavily on the Ellislabs CodeIgniter MVC framework. Now, don’t get me wrong, I love CodeIgniter – but, it was the wrong choice. It was wrong, because we were locking our developers into an MVC structure, that truly isn’t needed for something like this.

So, we’ve stopped working on Project Stanley (you can still find it on github if you really want to) and we migrated the code into the PHPARI project. PHPARI is a cleaner approach to providing a simple, to the point, ARI developer kit using PHP. It relies on 2 PHP external libraries – PHPWS and PEST.

PHPWS is a WebSocket client implementation in PHP, while PEST is a REST client implemented in PHP. Both are actively maintained and had been tested by multiple projects as stable and battle tested. We’ve also enabled PHPARI in packagist, you can look it up for installation. Make sure you use the dev-master part of the package, not the dev-develop – it’s unstable and may actually contain broken code.

Asterisk ARI – What AGI/AMI should have been

Asterisk ARI – for a seasoned AGI/AMI developer like myself, ARI is a serious mind warp. Why is it a mind warp? simple, it’s all the things we wanted AGI to be, and the reliability we wanted AMI to have, minus all the work around we needed to do – in order to get similar functionality in the past.

So, is ARI truly a replacement for AGI/AMI? well… I think the true answer will be NO. Is a replacement for the Asterisk dialplan? well… I think the answer to that is NO as well. “Say, are you messed in the head? first you say “What AGI/AMI should have been”, and then you say it’s not a replacement? – are you mental?” – well, there are a few reasons why I claim it’s not a direct replacement, and I’ll detail these here.

In order to explain, I’ll give a few examples, using the “in-development” PHP ARI wireframe that I’m developing, called Stanley.

Synchronous vs. Asynchronous

ARI by definition is asynchronous. Keeping that in mind, in means that that any command you give it will get queued or spooled in some manner, and return back an immediate result. Just to illustrate it, let’s examine the following code segment:

$this->stasisLogger->notice("Stasis Start");
$lastResult = $this->channels->channel_playback($this->ari_endpoint, $messageData->channel->id, "sound:hello-world");
$this->stasisLogger->notice("Last result: " . $lastResult);
$lastResult = $this->channels->channel_playback($this->ari_endpoint, $messageData->channel->id, "sound:demo-congrats");
$this->stasisLogger->notice("Last result: " . $lastResult);

For all practical purposes, you should regard $this->stasisLogger as a simple logging object, and $this->channels as a model to initiate ARI Channel requests. If you use the above the code, and activate it from with a Stasis application, you would listen to the “hello-world” and “demo-congrats” segments. Now, let us examine the following code segment:

$this->stasisLogger->notice("Stasis Start");
$lastResult = $this->channels->channel_playback($this->ari_endpoint, $messageData->channel->id, "sound:hello-world");
$this->stasisLogger->notice("Last result: " . $lastResult);
$lastResult = $this->channels->channel_playback($this->ari_endpoint, $messageData->channel->id, "sound:demo-congrats");
$this->stasisLogger->notice("Last result: " . $lastResult);
$this->channels->channel_delete($this->ari_endpoint, $messageData->channel->id);

The only difference here is the last line. If you activate this code, you will hear the world “Hello”, immediately followed by a disconnect. “Wait a minute, what just happened? – wasn’t I supposed to hear everything?” – that’s exactly the point, the answer is NO! The asynch nature of ARI will simply queue the first 2 playback requests, while the hangup is performed almost immediately – the playback simply never get to be executed.

In other words, if you need something to be synchronous within the dialplan, you may need to work differently about it. If you are familiar with the Node.JS framework, you are fairly familiar with this issue.

ARI is for writing applications, not IVRs

When the Asterisk team created ARI, their idea was simple: “Don’t manage the queue application, simply write your own”. Same applies for managing multi party conference calls, call origination, etc. In 2009 I wrote a book about AGI programming, where I’ve explained the methodology for “Atomic AGI development“. The concept behind Atomic AGI was to contain small logic units in AGI scripts, and leave most of the heavy lifting to the dialplan. This methodology enables to create scaleable Asterisk platforms at fair ease, and introduce additional technologies, without going about and adding odd things into Asterisk.

ARI is meant to do something similar, in the form where you can go about and create your own logic, contain it into a singular application and activate when you require – for example, rewriting the queue application. One of the first applications that I’ve decided to re-write using ARI was a Radio broadcasting system that I’ve developed in 2006. The problem with that application was that I need to hold about 600 callers in a single queue, and attach them over to the broadcasting booth as required. Of course I needed to enable full call control, caller management, UI and more. Initially, I used MeetMe, MySQL, and AMI to do this. Later on it changed to MeetMe, Redis, AstManProxy and some other tools – but it never seemed to please me. The fact that I needed to maintain 2 MeetMe bridges, one for holding people and one for the actual broadcasting really bugged me. Yes, when Asterisk 1.8 came out I migrated to the Bridge application and yes, I updated bits and pieces here and there, but it was never what I wanted it to be.

When I started playing around with ARI, I said to myself – this is the perfect application to migrate to ARI. The only thing I needed was a simple Stasis application to read my state correctly, and that would be activated once the called is put into the waiting area – so in terms, I’ve developed a very simple queue application.

IVR heavy lifting was done using dialplan, but the actual service was done with ARI.

Blades and Bleeding Edge

Now, before you go about migrating all your existing code to ARI – you must remember this: If you walk on the bleeding edge, expect the blade to cut you here and there. Currently, I hadn’t yet seen any proper ARI wireframe available. I’ve seen some work done with Node.JS and Ruby, but I can’t say that I’ve taken a fancy to any of those. Honestly, my comfort zone is very much PHP and C/C++, what can I say, I’m old school.

When I started building the Stanley wireframe, it was fairly frustrating – simply because not everything was that much clear and clean. In addition, as Asterisk advances, ARI will change and advance as well. What ever you write, make sure it’s modular enough so you can change it as required.

 

Astricon 2014 – Start your engines!

Ok All, this is my official Astricon Countdown – start your engines, as Eric Klein and myself will be attending Astricon this year, Vegas here we come.

So, what are we going to talk about: Security and Cloud computing. Yes, over the past year, I’ve been returning to my old stomping ground, the various cloud infrastructure that is publicly available – and how to exploit it to the max. I will be talking about the various methods of speeding up your clouded Asterisk server, and most importantly, I’ll share some of the methodologies and logic behind building these instances, maintaining them and the various do’s and don’ts I’ve learned along the way.

I’m planning a few surprises and giveaways for my talk, so make sure you stay updated on this page 🙂

 

Building your Asterisk based Business – Part I

** Cross posted from: http://www.greenfieldtech.net/blog/2014/03/building-your-asterisk-based-business-part-i **

Since the inception of GreenfieldTech, back in 2007, we’ve assisted over 20 different VoIP companies to bootstrap their activities and launch their products. During that period, some of these companies had become a great success and some had disappeared from the face of the planet. This series of posts will bring the story of some of them – and we’ll try to analyse what made each company into a success or a failure.

Making the case for Asterisk based business

Let’s be honest and truthful with ourselves, we’re all capitalists. Yes, we are first Open Source evangelists and promoters, but at the end of the day, we do need to pay our bills and make ends meet. Thus, we monetize open source (in our case Asterisk) to the best of our abilities – there is no shame in that, and honestly, we take high pride in our ability to assist companies in monetizing open source tools and project in a productive and lucrative manner.

When people talk about Asterisk based businesses, normally they will consider one of the following tracks:

1. Asterisk PBX Integrator – Integrating Asterisk based PBX systems for companies of various sizes. Normally, this will include infrastructure installation, cabling, server support, hardware sales, etc.
2. Hosted PBX Service Provider – Providing VoIP PBX services, without any in-door server equipment, relying on the Internet or leased lines only. Normally this will include similar activities as the Asterisk PBX integrator.
3. Hosted IVR Service Provider – Providing Hosted Interactive Voice Response services to content providers and enterprises that can’t sustain their own IVR infrastructure.
4. Hosted Premium Services Provider – Identical to “Hosted IVR” with a focus on premium services content and adult content.
5. Telecom Carrier – Whether you are a mobile carrier or a landline CLEC, Asterisk applications can benefit a carrier of any size.
6. Value added services provider – Providing complimentary services to Asterisk and its derivatives.

The above is a very short list, as each item on that list can be expanded to 6 more sub-categories – however, these represent the major business types (not including consultants and developers, who fall under category number 1). There are similarities between these and significant differences as well. What may be true for one, maybe completely wrong for another – it all depends on your business goals and product development life cycle and life span. We will limit ourselves to discuss options 2 through 6.

Case I: Long Distance Calling Cards Operator in the US

June 2007, a representative of a calling cards operator in the US approached. The operator was back then using a hosted service from a company called Solegy (long gone) and was fairly unhappy with the service. Their main complaints were: lack of support responsiveness, lack of feature set, inability to expand existing feature set – and most importantly, inability to sustain a proper business model (unlimited calls), due to high termination and hosting prices.

At that time, the calling card operator had put the following restrictions as to creating the service:

1. Bootstrap pricing should not exceed the $10,000 USD.
2. All existing customers should be migrated to the new system.
3. All existing access numbers should be migrated to the new system.
4. New system should be based upon ready-made software – not customized development.
5. New system should enable additional service development and scalability.
6. New system should allow hosting on hosted servers, without any need for proprietary hardware.

The solution that was chosen (after all, it was 2007) was a mixture of OpenSER, coupled with Asterisk 1.4 and A2Billing. Calls will hit the OpenSER SIP proxy, which will then load balance to the various Asterisk servers. The solution met the various constraints listed above – and later on included a monthly support/maintenance cost that was sustainable for the business. The company continued on to provide direct DID numbers, VoIP termination services, Roaming SIM cards solutions, Hospitality Mobile phones and additional services. In 2010, the company had sold its operations to another company – which was a disaster. During the recent changes in the roaming market in the US and other regulations, the company had seized its operations and is no longer operating.

Was this company a success story? – Complicated Answer

Between the years 2007 and 2010, the company had grown from 1000 customers to a whopping 15,000 customers, paying a monthly service fee ranging from $15 USD to $59 USD. Roughly calculated, that’s an average of $550K USD per month turn-over. In deed, termination costs and operational costs rose up to around $450K USD a month, but considering the fact that the company had only 4 employees and two additional outsourced support resources, a monthly Net revenue of $100K USD is not bad – we can surely mark this as a success. The company realized that in order to sustain its business, they would require proper customer care and support services and they made sure these resources were clearly managed and sourced.

Following the company’s purchase in 2010, the new owner had decided that customer care and support structure aren’t really required, as the sales staff can handle customer care and support can be rendered by an outside resource on a per-incident base, with no binding SLA service.  Within less 12 months, the company customer base shrunk from 15,000 to around 3,000, bringing the entire operation to a stand-still. The new owner tried focusing on new business tracks, without preserving and maintaining the existing lucrative tracks.

So, what went wrong here?

The primary answer would be: failure of the new management to understand the business. Calling cards and VoIP services are customer oriented services. This applies to Mobile VoIP OTT services as well. Asterisk is a solid tool to use when building your business, it will take you a long way and make sure your ROI and TCO will remain as low as possible, while preserving your knowledge and experience in-house as much as possible. When a successful company is acquired by a financial outfit, that has no valid experience in the market sector, in many cases – it will fail. The lack of understanding of customer care structures, proper support, proper monitoring, proper NOC, ticketing handling, NOC liaising and proper technical escalation are the main attributes of a successful service and product in this industry.

In our next post

In our next post we’ll discuss the world of “ad-revenue financed calls” and the “call-back industry”, as it was in 2008, 2010 and what happened to it today.