Extreme Asterisk Cloud Performance – Part I

*** This post was originally posted at http://www.greenfieldtech.net

Here’s a challenging question for the Asterisk technical savvy of you… What is the top performance you can squeeze out of an Asterisk box, running on Amazon EC2 – or to that extent, a cloud infrastructure? If you scout the Internet, you may find various answers – however, most of them aren’t backed up by real numbers or real information,made accessible in a normal readable form.
Recently, we’ve become heavily involved in a project requiring massive usage of cloud based infrastructure. I won’t go into details as to what the project is or what we are doing, however, I felt that some interesting facts about Asterisk 11.0.1 and Cloud infrastructure can be shared with the rest of you.

Before we dig deep into the actual results, let’s talk about the various measurements usually associated with performance assessments of an Asterisk box, mainly, the machines load average. In order to continue, we must first understand what the Linux Load Average actually is. Most of you know load average as the below:

Load Average Example

Most people know the load average as those 3 numbers, ranging from 0 to anything higher, and if the numbers reach a certain level – it’s bad. But the question is: “What is a good number? and what makes a number bad?” First, let’s understand what the number represents. Load average is an exponential average of all your machines processes. Running processes, sleeping processes, waiting processes and on Linux, also processes currently waiting for I/O access. Now, these number are directly correlated to the number of processors/cores your server has. In general terms, a machine with a single core, any number higher than 1 is considered bad – where 1 represents 100% of the resources being consumed. So, if your machine has 4 cores, the number 4 is your top most number – and from there it’s linear. Now, can we calculate HyperThreading into the equation, multiple CPU pipelines, SSD access – in Linux, all these come into play into that equation. In other words, we’ll never know what is the actual top limit, but working with a rule of thumb based upon the number of cores is a good practice – specifically if your operational environment is a virtualized one.

Now, there are 3 numbers in there – a 1 minute average, a 5 minute average and a 15 minute average. Technically speaking, the 1 minute average isn’t really interesting – as it is highly affected by context switches and process bootstrapping, thus, there is a good chande that its number will be higher than the “advised” number. The numbers that are more interesting are the 5 minute and 15 minute average. Technically speaking, if your machine’s load average is considerably higher than the advised at these, something is definitely wrong.

Updates, Astricon, Allo.com and more

Ok, before all of you jump at my throat for not posting for a long time – I’m sorry. I can’t believe the last time I posted was around June, much has changed since then. So, let’s start with some updates… well, there are no big news updates. Since I left Humbug, I’ve been doing my best to keep busy with GreenfieldTech projects. We’ve successfully completed 10 different custom development projects since June and we’ve started a brand new services branch at GreenfieldTech – the VoIP Security Audits branch – but that’s a different post altogether.

So, I spoke at this years’ Astricon, which took place in Atlanta, Georgia!. This is my first time to the southern parts of the USA and might I add, Southern Hospitality is something you need to experience (I’ll write about that later). Much had changed at Digium in the past 2 years. Many of the long time project members had left to new grounds, Asterisk-SCF is now officially no longer in active development, in other words – this years’ Astricon was a fresh breeze of wind, specifically with new people at the driver seat and new ideas springing about.

So, let’s talk PRI cards. Once every often a company approaches me to evaluate their products. I do my best to be as impartial as I can – after all, Digium products are my favorite. However, I’m always happy to see a product that can compete nicely with Digium, simply because I believe it will make Digium products better and stronger. So, a couple of months ago, Allo.com approached me to evaluate their PRI card. I agreed, and they’ve sent me their PCI-e version of a quad span E1/T1 card.

Allo.Com PRI Card

So, let’s start with what I didn’t like about the card. People, it’s 2012, technology had progressed a great deal since the old Zaptel days, not to talk about the old Xilinx Spartan chips – comm’on, even the crappy Chinese boards don’t use that anymore – move on. Ok, let’s put aside the issue of the actual chips being used on the board – can someone PLEASE explain to me why I need to patch my DAHDI modules to support these cards? How shall I put it, patching DAHDI/ZAPTEL is so 2004. Make your card fully compatible with DAHDI, no patching, stream line the card with the DAHDI stock kernel module – OpenVOX did it, Yeastar did it (do an extent) – you want me to use your hardware, make it easy to install and simple to update and maintain.

OK, regardless of my somewhat reluctant feelings regarding drivers compatibility, I had the unit installed in a test gateway. It performed as I expected from a low-cost compatible. It held up nicely with normal traffic, but when I tried pushing 30 call initiations per second on the card, it heated up slightly and CPU spikes could be observed here and there. Now, in an office scenario – sure this card will do nicely – in a service provider scenario – I’ll think twice. Now, in the past I’ve received similar performance from other clone cards, so my estimate is that there is a group of engineers passing from one company to another, coming in with the know-how for a single design and they wrap that into a card.

Final word regarding the Allo.com card – not my favorite, but definitely a possibility for office environments. I have no idea how their other equipment holds up, but I hope that it holds as well, so that the office/smb market has a new option to choose from.

I will post my Astricon update later on.

The natural evolution… of people and startups

It has been a while since I’ve written, I guess I didn’t have much to write about. Ok, I admit it, I had a shit load of stuff to write about, however, I simply never got around to actually do it. A full time job, two babies – or as they call it – real life, has somewhat managed to creep up to me and take its toll on my writing time. Over the course of the past few months, I’ve been heavily thinking about the role of the founder in a startup. Many people regard the founder of a startup as the CEO or some other key role in the company, but it’s not always like that. The question that I asked myself, and I have been for quite some time, is: “as a founder of a startup, what is the most important thing I need to know?” – I recently realized that the most important thing, is also the hardest thing to do.

As I see it now, the most important thing a founder needs to know how to do is: “When to walk away!” – Yes, you heard me right, understanding that you as a founder have to walk away from your creation is the hardest thing you’ll ever do. In many cases, walking away may seem to you like bailing out or giving up, but this is not the case. Sometimes, walking away means: “I’m entrusting my creation with someone else, simply because I can’t progress it any more”. It is very easy to become infatuated with your startup, it’s just like a baby. Initially, it’s just an idea, then after a long incubation time, suddenly it takes form. You hire people, slowly afterwards, you have a proof of concept, then you start selling, then different things can happen – some good, some bad. At that point in time, that’s a critical path, this is the time where you might find yourself at an impasse with your partners, investors, customers, VC’s, etc. What should you do? stay and fight? or just pack up and leave? – well, it all depends on your situation. I won’t detail the situation when you need to fight and stand your ground, as those are fairly easy. The harder ones relate to when to leave.

I would say, that if any of the following are upon you, it’s time to leave:

  • Your investors are pushing toward an illogical track, although, they have no idea how the market nor the niche works.
  • You ran into financial issues and leaving is a logical step in order to continue the life of the company.
  • You are at an impasse with the CEO or other members of management, the issue can’t be bridged.
  • You have lost trust with your partners.

At any of those, you must pack up and leave. The reasons are simple enough, all of those relate to direct trust between people. When trust is no longer there, it’s better to pack it up and move forward, simply because it won’t go to good places.

Can you trust your integrator with Fraud Analysis?

As some of you know, over the past 9 months, I’ve been heavily involved in the establishment of Humbug. For those who may not know, Humbug is a Call Analytics and Fraud Analysis SAAS. Now, differing from many of the current telephony SAAS projects, we are not based on Amazon EC2 or some other public cloud infrastructure, we build our own cloud environment. Why do we build our own cloud? simple, we need to keep your data secured and confidential. At Humbug, we see ourselves as a cross between Google Analytics – in our ability to analyze and handle data and Verisign – in our security and confidentiality requirements and methodologies.

Question be asked, why do people trust Verisign to provide SSL certificates around the world. What makes Verisign’s CA better than a privately owned CA – the answer is simple, it’s a third party 2 entities can entrust at the same time. Humbug aims to provide the same lever of trust, simply because we regard your data as sacred and valuable.

Since about 2 months ago, we’ve been contacting various Asterisk integrators around the world, inviting them to evaluate Humbug services. Now, while some integrators and vendors were somewhat reluctant, others were more than happy to join. We now have over 250 monitored systems around the world, with system being monitored and analyzed in Israel, USA, UK, Brazil and more.

The thing that amazed me in regards to some of the integrators who decided not to participate was that they claimed: “we provide our customers our own brew of fraud analysis service, we don’t require your SAAS”. Now, while I can accept the fact that an integrator would offer such a SAAS as an in-house service, I can’t see why a customer would rely on these services. In my view, relying on your integrator to provide fraud analysis services is like relying on the integrator of your alarm system to provide hired guard services – it just doesn’t make any sense to me. Why doesn’t it make sense? in Hebrew we say: “Go prove that you have a sister”. Imagine that your PBX integrator offer you such a service, then, in some obscure manner, your PBX gets hijacked and you get slammed with 50K$ worth of phone calls to Somalia. Now, your integrator would say: “Hmmmmm… that’s odd, we didn’t even get those CDR events to our system… you really got hacked bad…” – sure, if you only rely on CDR records to do your analysis (which is what 99.9% of integrators do). There is much much much much more to fraud analysis than just CDR analysis – if it all began and finished with CDR analysis, then by far Cvidya, Verint, NICE and many others would have been made redundant.

Allowing your integrator to provide you with fraud analysis SAAS is like putting the fox to guard the hen house, when things louse up (and they may), he’s the first one to bail out saying: “It’s not my fault”.

Humbug takes a totally different approach to fraud analysis, specifically, in the way we regards the various PBX systems and integrators. We are vendor agnostic and integrator agnostic – we will provide you with the clear and concise information you require in order to make an educated decision as to how you were de-frauded (if de-frauded) and provide you a faster alerting and response time. Our recent adventures had lowered our fraud alert response time from 60 minutes, down to 14 minutes in some cases. Most fraud analysis system carry a 24-36 hour turn around time, by that time, you can be out of 50K$ – our aim is to lower that number to no more than a 100$ in the worst case. Ambitious? yes, down right crazy? probably so, but we always say: “Aim for the moon, you’ll land on a star!” – so we know we’ll get there.

Good bye Symbian, Hello Android

A Nokia E90 (open).

Image via Wikipedia

For those who had been reading this blog for some time now, you may have stumbled across my blog post from 2008, regarding me buying a Nokia E90 - http://www.simionovich.com/2008/06/06/i-finally-purchased-a-nokia-e90.

Well, it’s a fact, since the year 1998, I’ve been an avid Nokia fan. I think I’ve ranged from the old Nokia 51XX, through the 6XXX up to the E61, E62 and E90 – if it was some funky Nokia phone that gave me some new feature, I most probably had it. I guess that the time I spent at m-Wise, working closely with various mobile content technologies had put its toll on me – and I became a Nokia Cell Phone addict. For many years I couldn’t imagine myself digressing from the Nokia clan. Even when my friends moved from their Nokia/Motorola/Ericsson phones to a star spangled iPhone – I remained faithful to my old habits – and remained with my trusty Nokia.

Nokia 5800 XpressMusic showing Wikipedia's mai...

Image via Wikipedia

About two years ago I promised myself this: “If you ever decide to move to a touch screen phone, don’t go ala iPhone, stay for a Nokia phone” – so I waited. The initial Nokia touch phones came out. The first Nokia touch phone that came out, I believe shortly after the iPhone was the Nokia 5800, also known as the Nokia XpressMusic.

I’ve got one thing to say about this phone – “What the hell were you thinking???” – it’s a phone, not a bloody MP3 player – if I wanted an MP3 player, I would have bought an iPod. Apart from being the slowest phones I’ve ever encountered, its touch interface was annoying and disruptive.

So, I didn’t buy the Nokia 5800 – I simply had no use for it. At that point I decided to wait a bit more, and see what Nokia cooks up. Shortly after seeing the 5800 in dis-action, I met a new member of the Nokia clan: the Nokia 700.

The Nokia 700 was a totally new thing, not really a phone, not really a PDA – somewhat of a cross between the two. It was big and bulky, and I couldn’t imagine myself walking around with one of these – however, it showed some promise. Sure, it was big, bulky, slow and anything bad you can say about a device – however, it had one thing – it showed potential – something to look for. At that time, I decided that I needed a proper smart phone and purchased the Nokia E90 – and I was fairly happy with it till 8 months ago.

You are probably asking, why would an avid Nokia fan become displeased with his trusty E90 phone – the answer is simple – the plastics. The plastics are of such low quality, that after 18 months of usage, the paint job starts to peel away from the phone. As you run more and more applications, or store more data, the phone becomes sluggish and slow – to the point where you have to reset it.

So, 2 months ago I gave up, I said to myself: “that’s it, time to move forward and leave the Nokia clan” – but I still didn’t want to put myself with the iPhone clan – or to be more exact, the iPhone cult movement. While at the Amoocon convention, I came across some people who were using HTC phones, specifically the HTC Evo. Well, I was somewhat taken with this snazy piece of hardware. It was fast, it was fluid and for some funky reason, I felt at home with it. Could it be, have I found a new clan for my mobile needs? I returned back home starting to examine my options. The HTC Evo isn’t available in Israel, the next best thing is the HTC Desire.

The HTC Desire is also known as the Google Nexus-1, basically it’s the same phone. I tried using the Nexus-1, but I didn’t like it. Specifically, I didn’t like the fact that the four keys are touch based – on the HTC these are real keys, making my life much easier.

So now, I’m equipped with the HTC Desire, and apart from the occasional Android crash (not too often to be honest) – it is one of the best phones I ever had. It’s fast, syncs my life into a manageable construct and most importantly, it’s become a second nature to me. The only disadvantage of owning such a phone is that you need a massive Data plan with your carrier – this little machine can gobble up ten’s of megabytes on a daily basis. My old Nokia E90 was using 25MB of data per month, with the Desire, I consume that much in less than a day – that is an amazing number.

In order to get better into Android development, I’ve ordered an Eken M002 device. This is a 7″, Android based tablet PC. I’ll be posting information about that once it arrives.

Enhanced by Zemanta