Technology? Religion? or just pure Ego?

Open Source – What really drives it? is the desire to change and create something new? is it a firm belief in the idea that knowledge wants to be free and that software should roam the world? or when you boil down – is it just plain Ego?

I’ve been an Open Source advocate and evangelist for the better part of the last 20 years. I’ve started my days with Slackware Linux, moved to RedHat, then to Mandrake, then over to CentOS – which is now my choice of OS for the desktop and server. During these 20 years, I’ve seen various project come and go, companies rise and fall, technologies adopted and abandoned. A recent post on facebook from Dovid Bender got me thinking about this issue again:

Now, let’s put aside the grand discussion on the way the OpenSIPS project came about, their domain hijacking tactics, their overall confusion in the initial stages in regards to the difference between OpenSIP and OpenSER/Kamailio – let’s just put these apart for a second. Honestly, I can’t really tell the two apart, they use the same general configuration syntax and in most cases (over 95%), you can use the same configuration on both and it would work exactly the same. So, what does it boils down to? it boils down to Ego. Do I want to be considered traditional and stable and work with Kamailio, or would I like to be perceived as cutting-edge and work with OpenSIPS (although that isn’t true at all).

The same issue can be attributed to the ever growing battle between Asterisk and FreeSWITCH. Now, each one was built for a totally different class of operation (although, Asterisk 12 does introduce new functionality that makes it shine much harder than FreeSWITCH). People repeat the old “You’re melting our switches” FreeSWITCH urban myth, but again, I still hadn’t seen one installation that truly did everything with FreeSWITCH and is truly focused on using FreeSWITCH to leverage something else. So, if FreeSWITCH is only used as a media/application server, then I see no difference between it and Asterisk in that regard. More than that, if the added value of using FreeSWITCH is just a mere 5-10% increase in performance, it just isn’t worth my time to do so. Now, I’ve used FreeSWITCH in the past, don’t get me wrong – it’s a wonderful tool in that respect, and for Class-4 switching it is a massive tool. But when it comes to Class-5 and high-level services, sorry to say, Asterisk will always be my choice – not because it is better, not because it’s support and community is far more experienced, not because it out-performs FreeSWITCH – it will always be due to one simple reason – it is the one I know will require the less amount of ongoing support and maintenance and will bring me to my target much faster than FreeSWITCH.

A few weeks ago, I put the following status on facebook:

Now, the two have direct correlation – When a CTO/VP R&D isn’t a telecom’s guy – and he takes decisions for development of the platform – simply based upon the writings of others on the net – which is purely influenced by a religious war – he is incapable of making the right decision. Take Jajah for example, when Roman and Daniel started Jajah, they only tool they used back then with Asterisk@Home – because that’s what they had. When the company grew, they could have easily moved to new grounds – FreeSWITCH was already around. Why didn’t they? Why did Jajah remain with Asterisk – adding OpenSER/Kamailio into the mix later on? Why didn’t they move to a new platform? was it because they have loads of code developed? companies throw away code like dirty socks every other day – they had the resources. Fact remains, the service was alive for a long time, the company was bought out by Telefonica Digital at a price of $215 Million.

On the other hand, let’s take a company like Truphone (and pardon me James, I know you’re gonna kick my ass next time we meet). Truphone had changed technologies over the course of times many times. Each time, abandoning the previous tech and going for a new one. So did companies like Rebtel, Spikko, Skuku and others. Amazingly enough, none of them could be considered a massive success. Word on the market currently says that Truphone is looking for additional investors, as their existing ones aren’t willing to put in more cash. Spikko’s original model is totally gone and the company literally caved-in on itself – and same applies to many others.

So, what does it boil down to? is Asterisk better? is OpenSIPS better? – these are the wrong questions. The questions should be:

  1. Is your R&D lead actually knows the arena he’s treading in?
  2. Are your decisions based on actual investigation or just by whim?
  3. Are you completely aware of the various obstacles and challenges you’ll meet?
  4. Are you building your development and product on rapidly changing technology?
  5. Who is backing your choice? a proper business entity? or a mere group of people with an idea?

When it comes to choosing between Asterisk and FreeSWITCH, here are my reasons for choosing Asterisk over FreeSwtich any day:

  1. The ability to rapidly prototype any application is 5 times faster and 2 times more economical than FreeSWITCH
  2. The installation path for FreeSWITCH is much more complex and convoluted than Asterisk, making future maintenance a nightmare
  3. Digium is indeed a young company, but it sticks by its products and makes all efforts to make it the best it can – I always have someone to talk to
  4. Barracuda Networks is a well established company in the Storage/Security market – if you go to their website, their support for FreeSWITCH (CudaTel) isn’t there at all – does that mean something?
  5. Asterisk is a very reliable, dependable, predictable piece of code – it is something I can put my money on and know exactly what I’ll get, FreeSWITCH still isn’t

Extreme Asterisk Cloud Performance – Part I

*** This post was originally posted at

Here’s a challenging question for the Asterisk technical savvy of you… What is the top performance you can squeeze out of an Asterisk box, running on Amazon EC2 – or to that extent, a cloud infrastructure? If you scout the Internet, you may find various answers – however, most of them aren’t backed up by real numbers or real information,made accessible in a normal readable form.
Recently, we’ve become heavily involved in a project requiring massive usage of cloud based infrastructure. I won’t go into details as to what the project is or what we are doing, however, I felt that some interesting facts about Asterisk 11.0.1 and Cloud infrastructure can be shared with the rest of you.

Before we dig deep into the actual results, let’s talk about the various measurements usually associated with performance assessments of an Asterisk box, mainly, the machines load average. In order to continue, we must first understand what the Linux Load Average actually is. Most of you know load average as the below:

Load Average Example

Most people know the load average as those 3 numbers, ranging from 0 to anything higher, and if the numbers reach a certain level – it’s bad. But the question is: “What is a good number? and what makes a number bad?” First, let’s understand what the number represents. Load average is an exponential average of all your machines processes. Running processes, sleeping processes, waiting processes and on Linux, also processes currently waiting for I/O access. Now, these number are directly correlated to the number of processors/cores your server has. In general terms, a machine with a single core, any number higher than 1 is considered bad – where 1 represents 100% of the resources being consumed. So, if your machine has 4 cores, the number 4 is your top most number – and from there it’s linear. Now, can we calculate HyperThreading into the equation, multiple CPU pipelines, SSD access – in Linux, all these come into play into that equation. In other words, we’ll never know what is the actual top limit, but working with a rule of thumb based upon the number of cores is a good practice – specifically if your operational environment is a virtualized one.

Now, there are 3 numbers in there – a 1 minute average, a 5 minute average and a 15 minute average. Technically speaking, the 1 minute average isn’t really interesting – as it is highly affected by context switches and process bootstrapping, thus, there is a good chande that its number will be higher than the “advised” number. The numbers that are more interesting are the 5 minute and 15 minute average. Technically speaking, if your machine’s load average is considerably higher than the advised at these, something is definitely wrong.

Updates, Astricon, and more

Ok, before all of you jump at my throat for not posting for a long time – I’m sorry. I can’t believe the last time I posted was around June, much has changed since then. So, let’s start with some updates… well, there are no big news updates. Since I left Humbug, I’ve been doing my best to keep busy with GreenfieldTech projects. We’ve successfully completed 10 different custom development projects since June and we’ve started a brand new services branch at GreenfieldTech – the VoIP Security Audits branch – but that’s a different post altogether.

So, I spoke at this years’ Astricon, which took place in Atlanta, Georgia!. This is my first time to the southern parts of the USA and might I add, Southern Hospitality is something you need to experience (I’ll write about that later). Much had changed at Digium in the past 2 years. Many of the long time project members had left to new grounds, Asterisk-SCF is now officially no longer in active development, in other words – this years’ Astricon was a fresh breeze of wind, specifically with new people at the driver seat and new ideas springing about.

So, let’s talk PRI cards. Once every often a company approaches me to evaluate their products. I do my best to be as impartial as I can – after all, Digium products are my favorite. However, I’m always happy to see a product that can compete nicely with Digium, simply because I believe it will make Digium products better and stronger. So, a couple of months ago, approached me to evaluate their PRI card. I agreed, and they’ve sent me their PCI-e version of a quad span E1/T1 card.

Allo.Com PRI Card

So, let’s start with what I didn’t like about the card. People, it’s 2012, technology had progressed a great deal since the old Zaptel days, not to talk about the old Xilinx Spartan chips – comm’on, even the crappy Chinese boards don’t use that anymore – move on. Ok, let’s put aside the issue of the actual chips being used on the board – can someone PLEASE explain to me why I need to patch my DAHDI modules to support these cards? How shall I put it, patching DAHDI/ZAPTEL is so 2004. Make your card fully compatible with DAHDI, no patching, stream line the card with the DAHDI stock kernel module – OpenVOX did it, Yeastar did it (do an extent) – you want me to use your hardware, make it easy to install and simple to update and maintain.

OK, regardless of my somewhat reluctant feelings regarding drivers compatibility, I had the unit installed in a test gateway. It performed as I expected from a low-cost compatible. It held up nicely with normal traffic, but when I tried pushing 30 call initiations per second on the card, it heated up slightly and CPU spikes could be observed here and there. Now, in an office scenario – sure this card will do nicely – in a service provider scenario – I’ll think twice. Now, in the past I’ve received similar performance from other clone cards, so my estimate is that there is a group of engineers passing from one company to another, coming in with the know-how for a single design and they wrap that into a card.

Final word regarding the card – not my favorite, but definitely a possibility for office environments. I have no idea how their other equipment holds up, but I hope that it holds as well, so that the office/smb market has a new option to choose from.

I will post my Astricon update later on.

Can you trust your integrator with Fraud Analysis?

As some of you know, over the past 9 months, I’ve been heavily involved in the establishment of Humbug. For those who may not know, Humbug is a Call Analytics and Fraud Analysis SAAS. Now, differing from many of the current telephony SAAS projects, we are not based on Amazon EC2 or some other public cloud infrastructure, we build our own cloud environment. Why do we build our own cloud? simple, we need to keep your data secured and confidential. At Humbug, we see ourselves as a cross between Google Analytics – in our ability to analyze and handle data and Verisign – in our security and confidentiality requirements and methodologies.

Question be asked, why do people trust Verisign to provide SSL certificates around the world. What makes Verisign’s CA better than a privately owned CA – the answer is simple, it’s a third party 2 entities can entrust at the same time. Humbug aims to provide the same lever of trust, simply because we regard your data as sacred and valuable.

Since about 2 months ago, we’ve been contacting various Asterisk integrators around the world, inviting them to evaluate Humbug services. Now, while some integrators and vendors were somewhat reluctant, others were more than happy to join. We now have over 250 monitored systems around the world, with system being monitored and analyzed in Israel, USA, UK, Brazil and more.

The thing that amazed me in regards to some of the integrators who decided not to participate was that they claimed: “we provide our customers our own brew of fraud analysis service, we don’t require your SAAS”. Now, while I can accept the fact that an integrator would offer such a SAAS as an in-house service, I can’t see why a customer would rely on these services. In my view, relying on your integrator to provide fraud analysis services is like relying on the integrator of your alarm system to provide hired guard services – it just doesn’t make any sense to me. Why doesn’t it make sense? in Hebrew we say: “Go prove that you have a sister”. Imagine that your PBX integrator offer you such a service, then, in some obscure manner, your PBX gets hijacked and you get slammed with 50K$ worth of phone calls to Somalia. Now, your integrator would say: “Hmmmmm… that’s odd, we didn’t even get those CDR events to our system… you really got hacked bad…” – sure, if you only rely on CDR records to do your analysis (which is what 99.9% of integrators do). There is much much much much more to fraud analysis than just CDR analysis – if it all began and finished with CDR analysis, then by far Cvidya, Verint, NICE and many others would have been made redundant.

Allowing your integrator to provide you with fraud analysis SAAS is like putting the fox to guard the hen house, when things louse up (and they may), he’s the first one to bail out saying: “It’s not my fault”.

Humbug takes a totally different approach to fraud analysis, specifically, in the way we regards the various PBX systems and integrators. We are vendor agnostic and integrator agnostic – we will provide you with the clear and concise information you require in order to make an educated decision as to how you were de-frauded (if de-frauded) and provide you a faster alerting and response time. Our recent adventures had lowered our fraud alert response time from 60 minutes, down to 14 minutes in some cases. Most fraud analysis system carry a 24-36 hour turn around time, by that time, you can be out of 50K$ – our aim is to lower that number to no more than a 100$ in the worst case. Ambitious? yes, down right crazy? probably so, but we always say: “Aim for the moon, you’ll land on a star!” – so we know we’ll get there.