Technology? Religion? or just pure Ego?

Open Source – What really drives it? is the desire to change and create something new? is it a firm belief in the idea that knowledge wants to be free and that software should roam the world? or when you boil down – is it just plain Ego?

I’ve been an Open Source advocate and evangelist for the better part of the last 20 years. I’ve started my days with Slackware Linux, moved to RedHat, then to Mandrake, then over to CentOS – which is now my choice of OS for the desktop and server. During these 20 years, I’ve seen various project come and go, companies rise and fall, technologies adopted and abandoned. A recent post on facebook from Dovid Bender got me thinking about this issue again:

Now, let’s put aside the grand discussion on the way the OpenSIPS project came about, their domain hijacking tactics, their overall confusion in the initial stages in regards to the difference between OpenSIP and OpenSER/Kamailio – let’s just put these apart for a second. Honestly, I can’t really tell the two apart, they use the same general configuration syntax and in most cases (over 95%), you can use the same configuration on both and it would work exactly the same. So, what does it boils down to? it boils down to Ego. Do I want to be considered traditional and stable and work with Kamailio, or would I like to be perceived as cutting-edge and work with OpenSIPS (although that isn’t true at all).

The same issue can be attributed to the ever growing battle between Asterisk and FreeSWITCH. Now, each one was built for a totally different class of operation (although, Asterisk 12 does introduce new functionality that makes it shine much harder than FreeSWITCH). People repeat the old “You’re melting our switches” FreeSWITCH urban myth, but again, I still hadn’t seen one installation that truly did everything with FreeSWITCH and is truly focused on using FreeSWITCH to leverage something else. So, if FreeSWITCH is only used as a media/application server, then I see no difference between it and Asterisk in that regard. More than that, if the added value of using FreeSWITCH is just a mere 5-10% increase in performance, it just isn’t worth my time to do so. Now, I’ve used FreeSWITCH in the past, don’t get me wrong – it’s a wonderful tool in that respect, and for Class-4 switching it is a massive tool. But when it comes to Class-5 and high-level services, sorry to say, Asterisk will always be my choice – not because it is better, not because it’s support and community is far more experienced, not because it out-performs FreeSWITCH – it will always be due to one simple reason – it is the one I know will require the less amount of ongoing support and maintenance and will bring me to my target much faster than FreeSWITCH.

A few weeks ago, I put the following status on facebook:

Now, the two have direct correlation – When a CTO/VP R&D isn’t a telecom’s guy – and he takes decisions for development of the platform – simply based upon the writings of others on the net – which is purely influenced by a religious war – he is incapable of making the right decision. Take Jajah for example, when Roman and Daniel started Jajah, they only tool they used back then with Asterisk@Home – because that’s what they had. When the company grew, they could have easily moved to new grounds – FreeSWITCH was already around. Why didn’t they? Why did Jajah remain with Asterisk – adding OpenSER/Kamailio into the mix later on? Why didn’t they move to a new platform? was it because they have loads of code developed? companies throw away code like dirty socks every other day – they had the resources. Fact remains, the service was alive for a long time, the company was bought out by Telefonica Digital at a price of $215 Million.

On the other hand, let’s take a company like Truphone (and pardon me James, I know you’re gonna kick my ass next time we meet). Truphone had changed technologies over the course of times many times. Each time, abandoning the previous tech and going for a new one. So did companies like Rebtel, Spikko, Skuku and others. Amazingly enough, none of them could be considered a massive success. Word on the market currently says that Truphone is looking for additional investors, as their existing ones aren’t willing to put in more cash. Spikko’s original model is totally gone and the company literally caved-in on itself – and same applies to many others.

So, what does it boil down to? is Asterisk better? is OpenSIPS better? – these are the wrong questions. The questions should be:

  1. Is your R&D lead actually knows the arena he’s treading in?
  2. Are your decisions based on actual investigation or just by whim?
  3. Are you completely aware of the various obstacles and challenges you’ll meet?
  4. Are you building your development and product on rapidly changing technology?
  5. Who is backing your choice? a proper business entity? or a mere group of people with an idea?

When it comes to choosing between Asterisk and FreeSWITCH, here are my reasons for choosing Asterisk over FreeSwtich any day:

  1. The ability to rapidly prototype any application is 5 times faster and 2 times more economical than FreeSWITCH
  2. The installation path for FreeSWITCH is much more complex and convoluted than Asterisk, making future maintenance a nightmare
  3. Digium is indeed a young company, but it sticks by its products and makes all efforts to make it the best it can – I always have someone to talk to
  4. Barracuda Networks is a well established company in the Storage/Security market – if you go to their website, their support for FreeSWITCH (CudaTel) isn’t there at all – does that mean something?
  5. Asterisk is a very reliable, dependable, predictable piece of code – it is something I can put my money on and know exactly what I’ll get, FreeSWITCH still isn’t

Extreme Asterisk Cloud Performance – Part I

*** This post was originally posted at

Here’s a challenging question for the Asterisk technical savvy of you… What is the top performance you can squeeze out of an Asterisk box, running on Amazon EC2 – or to that extent, a cloud infrastructure? If you scout the Internet, you may find various answers – however, most of them aren’t backed up by real numbers or real information,made accessible in a normal readable form.
Recently, we’ve become heavily involved in a project requiring massive usage of cloud based infrastructure. I won’t go into details as to what the project is or what we are doing, however, I felt that some interesting facts about Asterisk 11.0.1 and Cloud infrastructure can be shared with the rest of you.

Before we dig deep into the actual results, let’s talk about the various measurements usually associated with performance assessments of an Asterisk box, mainly, the machines load average. In order to continue, we must first understand what the Linux Load Average actually is. Most of you know load average as the below:

Load Average Example

Most people know the load average as those 3 numbers, ranging from 0 to anything higher, and if the numbers reach a certain level – it’s bad. But the question is: “What is a good number? and what makes a number bad?” First, let’s understand what the number represents. Load average is an exponential average of all your machines processes. Running processes, sleeping processes, waiting processes and on Linux, also processes currently waiting for I/O access. Now, these number are directly correlated to the number of processors/cores your server has. In general terms, a machine with a single core, any number higher than 1 is considered bad – where 1 represents 100% of the resources being consumed. So, if your machine has 4 cores, the number 4 is your top most number – and from there it’s linear. Now, can we calculate HyperThreading into the equation, multiple CPU pipelines, SSD access – in Linux, all these come into play into that equation. In other words, we’ll never know what is the actual top limit, but working with a rule of thumb based upon the number of cores is a good practice – specifically if your operational environment is a virtualized one.

Now, there are 3 numbers in there – a 1 minute average, a 5 minute average and a 15 minute average. Technically speaking, the 1 minute average isn’t really interesting – as it is highly affected by context switches and process bootstrapping, thus, there is a good chande that its number will be higher than the “advised” number. The numbers that are more interesting are the 5 minute and 15 minute average. Technically speaking, if your machine’s load average is considerably higher than the advised at these, something is definitely wrong.

Updates, Astricon, and more

Ok, before all of you jump at my throat for not posting for a long time – I’m sorry. I can’t believe the last time I posted was around June, much has changed since then. So, let’s start with some updates… well, there are no big news updates. Since I left Humbug, I’ve been doing my best to keep busy with GreenfieldTech projects. We’ve successfully completed 10 different custom development projects since June and we’ve started a brand new services branch at GreenfieldTech – the VoIP Security Audits branch – but that’s a different post altogether.

So, I spoke at this years’ Astricon, which took place in Atlanta, Georgia!. This is my first time to the southern parts of the USA and might I add, Southern Hospitality is something you need to experience (I’ll write about that later). Much had changed at Digium in the past 2 years. Many of the long time project members had left to new grounds, Asterisk-SCF is now officially no longer in active development, in other words – this years’ Astricon was a fresh breeze of wind, specifically with new people at the driver seat and new ideas springing about.

So, let’s talk PRI cards. Once every often a company approaches me to evaluate their products. I do my best to be as impartial as I can – after all, Digium products are my favorite. However, I’m always happy to see a product that can compete nicely with Digium, simply because I believe it will make Digium products better and stronger. So, a couple of months ago, approached me to evaluate their PRI card. I agreed, and they’ve sent me their PCI-e version of a quad span E1/T1 card.

Allo.Com PRI Card

So, let’s start with what I didn’t like about the card. People, it’s 2012, technology had progressed a great deal since the old Zaptel days, not to talk about the old Xilinx Spartan chips – comm’on, even the crappy Chinese boards don’t use that anymore – move on. Ok, let’s put aside the issue of the actual chips being used on the board – can someone PLEASE explain to me why I need to patch my DAHDI modules to support these cards? How shall I put it, patching DAHDI/ZAPTEL is so 2004. Make your card fully compatible with DAHDI, no patching, stream line the card with the DAHDI stock kernel module – OpenVOX did it, Yeastar did it (do an extent) – you want me to use your hardware, make it easy to install and simple to update and maintain.

OK, regardless of my somewhat reluctant feelings regarding drivers compatibility, I had the unit installed in a test gateway. It performed as I expected from a low-cost compatible. It held up nicely with normal traffic, but when I tried pushing 30 call initiations per second on the card, it heated up slightly and CPU spikes could be observed here and there. Now, in an office scenario – sure this card will do nicely – in a service provider scenario – I’ll think twice. Now, in the past I’ve received similar performance from other clone cards, so my estimate is that there is a group of engineers passing from one company to another, coming in with the know-how for a single design and they wrap that into a card.

Final word regarding the card – not my favorite, but definitely a possibility for office environments. I have no idea how their other equipment holds up, but I hope that it holds as well, so that the office/smb market has a new option to choose from.

I will post my Astricon update later on.

Can you trust your integrator with Fraud Analysis?

As some of you know, over the past 9 months, I’ve been heavily involved in the establishment of Humbug. For those who may not know, Humbug is a Call Analytics and Fraud Analysis SAAS. Now, differing from many of the current telephony SAAS projects, we are not based on Amazon EC2 or some other public cloud infrastructure, we build our own cloud environment. Why do we build our own cloud? simple, we need to keep your data secured and confidential. At Humbug, we see ourselves as a cross between Google Analytics – in our ability to analyze and handle data and Verisign – in our security and confidentiality requirements and methodologies.

Question be asked, why do people trust Verisign to provide SSL certificates around the world. What makes Verisign’s CA better than a privately owned CA – the answer is simple, it’s a third party 2 entities can entrust at the same time. Humbug aims to provide the same lever of trust, simply because we regard your data as sacred and valuable.

Since about 2 months ago, we’ve been contacting various Asterisk integrators around the world, inviting them to evaluate Humbug services. Now, while some integrators and vendors were somewhat reluctant, others were more than happy to join. We now have over 250 monitored systems around the world, with system being monitored and analyzed in Israel, USA, UK, Brazil and more.

The thing that amazed me in regards to some of the integrators who decided not to participate was that they claimed: “we provide our customers our own brew of fraud analysis service, we don’t require your SAAS”. Now, while I can accept the fact that an integrator would offer such a SAAS as an in-house service, I can’t see why a customer would rely on these services. In my view, relying on your integrator to provide fraud analysis services is like relying on the integrator of your alarm system to provide hired guard services – it just doesn’t make any sense to me. Why doesn’t it make sense? in Hebrew we say: “Go prove that you have a sister”. Imagine that your PBX integrator offer you such a service, then, in some obscure manner, your PBX gets hijacked and you get slammed with 50K$ worth of phone calls to Somalia. Now, your integrator would say: “Hmmmmm… that’s odd, we didn’t even get those CDR events to our system… you really got hacked bad…” – sure, if you only rely on CDR records to do your analysis (which is what 99.9% of integrators do). There is much much much much more to fraud analysis than just CDR analysis – if it all began and finished with CDR analysis, then by far Cvidya, Verint, NICE and many others would have been made redundant.

Allowing your integrator to provide you with fraud analysis SAAS is like putting the fox to guard the hen house, when things louse up (and they may), he’s the first one to bail out saying: “It’s not my fault”.

Humbug takes a totally different approach to fraud analysis, specifically, in the way we regards the various PBX systems and integrators. We are vendor agnostic and integrator agnostic – we will provide you with the clear and concise information you require in order to make an educated decision as to how you were de-frauded (if de-frauded) and provide you a faster alerting and response time. Our recent adventures had lowered our fraud alert response time from 60 minutes, down to 14 minutes in some cases. Most fraud analysis system carry a 24-36 hour turn around time, by that time, you can be out of 50K$ – our aim is to lower that number to no more than a 100$ in the worst case. Ambitious? yes, down right crazy? probably so, but we always say: “Aim for the moon, you’ll land on a star!” – so we know we’ll get there.

Business 2.0 – Taking the leap forward…

The following post doesn’t really fit in line with the normal spirit of the blog, simply because it’s not funny nor directly related to technology. It’s called Business 2.0, as it relates to the ever problematic question any business owner has: “When should I grow and how?”.

As you may know, I’ve been a freelance Asterisk Platform developer since early 2003, turned to freelance development (Penguin for hire) around April 2007. Since that time, I’ve built systems and platforms for some of the better known brands around the world. Be it working directly with the customer or through a 3rd party (as a sub contractor) – I can easily say that I’ve completed over 120 different large scale projects within 3 years time. Now, when I refer to projects, I’m not referring to installing PBX systems, I don’t do that at all – I’m referring to highly complex application level development, creating some of the most innovative Asterisk based systems I’ve ever seen.

Image representing Jajah as depicted in CrunchBase

Image via CrunchBase


Image via Wikipedia

Putting aside everything, finalizing a rough estimate of 40 development projects on a yearly base, most of these performed solely by myself is a fairly challenging task. Sure, at times I’ll outsource some work to other freelancers like myself, specifically in fields where I’m not all that fluent (Database, Web Development, UI) – but yet, doing that means that I’m conducting 3 – 5 projects on a monthly basis.

After doing so for 3 years now, I can’t help but start thinking about expanding my business, taking it to the next level by hiring more people and building it up to a new level. Question remains for this: “How? What is the natural track of expanding your business?” – of course the simple answer would be: “Just hire another developer or two, and start doing more sales” – it’s not as simple as it sounds. After thinking about it for some time, I’ve concluded there are a few models of expansion:

Model 1: Organic Growth

Organic growth can be described as the simplest way of growth: “Hire a new guy and get more work in”. The problem with this model that it is fully reliant on your ability to sell more. However, as you concentrate on sales more, you take time from the development and delivery process – thus, the addition of the new developer is not a 100% addition, it’s actually 100% (developer) minus 40% (you) – so you are not at 200% capacity, you are 160% capacity. Surely 160 is 100, however, for the initial 6 months, till the guy learns the ropes, you are not at 160, you are actually at 80 – can you and your business sustain that?

Thus, the main issue with Organic growth is cash flow, can your business sustain the elevated expenses with less income for the period of transition? If the answer is NO, then you need a different method. If the answer is YES, then you are in the best place in the world, however, bear in mind that taking someone to work for you is a responsibility – people are not resources, they are human beings, with families and children – taking someone to work for you is like taking responsibility for their lives.

Model 2: The Partner

Panama Business and Investment
Image by thinkpanama via Flickr

A partnership with a person who is equally matched to you is always a good option. Technically speaking, it means that you are teaming with someone who generates as much work as you do and is capable of finalizing the work as fast and as good as you can. Yet, taking a partner doesn’t negate the requirement for a new employee or two. In this case, you may end up with too much sales with too little staff to deliver – that is a big problem.

Another issue with partners is the issue of trust. While most partners tend to rely on each other and trust each other, that trust can easily be broken (in most cases by stupid things). It’s enough for one partner to now carry its weight in sales/development to initiate a chain reaction, shortly ending in the partnership dissolving.

So, the partner is a good option, however, may prove to be problematic if the wrong partner is chosen – in addition, dissolving a partnership solely on these issues isn’t all that simple – and usually ends up in litigation and other judicial issues – YUCK!

Model 3: Un-intrusive Angel

Some people ragard Un-intrusive Angels as “Stupid Money” – an Angel investor that doesn’t interfere in your company business model and operations. In many cases, this is how start-up companies start – someone gives them a lump sum of money to start their business, signing off to own a portion of the new company.

An un-intrusive investor usually gives you the money and pays you a visit once every few months to see how his money is spent. Don’t expect to raise a whole lot from these people, usually you will get anything from 25K$ to around 250K$ – tops. If you are getting an investment from an Angel, make sure you plan your business carefully – and make sure your investor knows what he is getting into. The Angel is not a found piggy bank, he is a business man looking for profit – if you make sure his expectations of profits (time frame, amount, percentage, etc) are kept within the reason of your business – he will make an educated decision and invest accordingly. Promises like: “you’ll double your money in 3 years” are stupid – make sure it’s realistic and to the point. If you promise the moon, and reach a star – that’s a problem, if you promise the skies and hit a start – that’s wonderful.

Model 4: The Strategic-Intrusive Angel

Jeff pulver

Image by TheFemGeek via Flickr

A strategic angel is similar to the previous one in terms of funds, however, he is more capable in assisting your business meet its goals. Usually, it would be someone who is already a well established figure in your business sector, had made his money from previous companies and is now looking for new ideas and businesses. I call him an intrusive Angel, as sometimes he may have ideas as to where your business should go – and he will make sure you hear his ideas. You may regard it as annoying, but you should still listen to your Angel and pay him the respect he deserves.

Sometimes this Angel may invest in your business due to the fact that he has a hidden agenda. An agenda can be: The angel looks at your business and see a certain potential you are not planning, he’ll invest and try to re-direct your company to the agenda he sees. This is usually the case when your angel is invested into several endeavours that is either parallel to each other or may have orthogonal intersection points. These angels can be the builders of your business or the destroyers, it is up to you to make sure the latter doesn’t happen.


So, which model did I choose? – I didn’t choose yet, I’m still figuring it out myself. What ever the model may be, the choice isn’t simple nor straight forward. At best, whatever choice I’ll take will have a profound impact on my business and me – so I’ll need to weigh my options carefully. If you can think of an additional model, I’d love to hear about it – so just comment on this post.

Reblog this post [with Zemanta]