Don’t Replicate – Federate

For many years, the question of high availability had always circled the same old subject of replication – how do we replicate data across nodes? how do we replicate the configuration to stay unified across nodes? Is active-active truly better than active-passive? and most importantly, what happens beyond the two node scenario?

Since the inception of the Linux-HA project (and I do believe it’s been around for years now – over 15 years), it has been the pivotal tool for creating Linux based high-availability clusters. Heartbeat, Stonith and Mon will take care of floating the IP numbers and services across – no biggy there, making sure the data is consistent across the board, that’s something completely different. Recently, one of the better known Asterisk Commercial offerings had launched an Asterisk-HA solution – it’s been long due – it’s just a shame it’s a commercial offering without an Open Source derivative, after all, it is Open Source based (I hope).

However, being a high availability solution on one hand, doesn’t mean you are truly a clustered solution – it is an active-passive solution, with a major caveat (at least as I see it), that if your data sync fails for some reason, you end up with a split-brain issue – and your entire solution is now made moot. Don’t get me wrong here, I think that for now, the solution is the next best thing to sliced bread, simply because there is no other solution out there. However, the fact this is the only solution, doesn’t make it the right solution.

What does federating mean in this respect? it means that data doesn’t need to be replicated across the board, it is automatically trickled across the network, making sure all nodes in the network have clear visibility for it. If a node fails inside the cluster, client automatically redirect themselves to a new node, no need for floating IP numbers. Call routing is automatically determined upon request and are never preset for the entire platform. And most importantly, the amount of data traversed between the nodes is as minimal as possible, preventing excessive usage of network resources and I/O.

What would it mean to federate the configuration of a PBX system? first of all, make sure each unit is capable of working on its own, information should be trickled across the nodes via two methodologies: A multicast/broadcast mechanism (for local LAN connected nodes) and a Published/Subscriber relation (for externally connected nodes). When a change is made to any of the systems, that change is then replicated to all the systems. The configuration is never fully transmitted between nodes (apart from a new node joining the cluster). Routing decisions are dynamically made across the network, they are not predetermined or preconfigured. There is no need to keep the cluster nodes in perfect physical alignment, mixing hardware specifications should be considered the norm. External devices should be able to “speak” to the cluster, without being aware of its existence.

Once we achieve all of the above, we’ll truly get to a point where we’ve clustered Asterisk (or another open source project) the right way.

Federating Asterisk – truth or myth?

During this years’ Asterisk Developers’ Conference, one of the subjects I’ve raised an issue for Asterisk is: “Federating Multiple Asterisk Instances”. Now, for the seasoned Asterisk user/developer, the answer would be simple – use Kamailio/OpenSIPS for that scalability, and use Asterisk as a Media Gateway or application server.

But I ask the following: “What if we could federate Asterisk without the need for an external component? What if we could federate Asterisk in such a way where our users aren’t event aware of the federation process, and it’s fully autonomous? What would actually be required in order to do that?”

I’m normally confronted with these questions on a day to day basis, looking at the problem from different angles – thinking to myself: “Ok, I know the normal box here – but where are the outer limits? what can I do to make it more robust on one hand, without truly making a mess out of it.”

A federated database is defined as: “A federated database system is a type of meta-database management system (DBMS), which transparently maps multiple autonomous database systems into a single federated database. The constituent databases are interconnected via a computer network and may be geographically decentralized. Since the constituent database systems remain autonomous, a federated database system is a contrastable alternative to the (sometimes daunting) task of merging several disparate databases. A federated database, or virtual database, is a composite of all constituent databases in a federated database system. There is no actual data integration in the constituent disparate databases as a result of data federation.” –

So, we would like to virtually create a “map-reduce” functionality for Asterisk? can we truly create a map-reduce’ish functionality for Asterisk? should it be internal? should it be external?

In order to accomplish this, we are required to create a federator – a device capable of handling the information regarding each users, device, trunk, provider and other wise SIP/IAX2 entity connected to our system. The federator for all practical purposes is a data store, be it a key-value store, a database, a shared memory environment or some other form of data distribution layer.

Here are some key issues that true federation may be required to tackle:

  1. Geo-Position Agnostic – A truly federated system should render services identically across the board, regardless of where the user is located.
  2. Services Agnostic – A truly federated system doesn’t care if the user is connected to an Asterisk server version 12 or 13, it should behave identically.
  3. Version Agnostic – A truly federated infrastructure can leverage older version and even other software, without changing the underlying federation layer.
  4. Predictable Scalability – A truly federated infrastructure will allow for growth to be planned linearly, with discrete measure methods.

So, you want a tip on how to start federating your systems? here’s step number 1 – there is no central registry, there is no SIP proxy, there is only the cloud and the services it renders. Start thinking from this point and see where you go.

The Asterisk scaleability Unicorn

Ok, the picture shows a donkey not a Unicorn – as you know, Unicorns are very hard to find. Asterisk Scaleability is somewhat of a unicorn – not because it doesn’t exist, it is a little tricky to do and get it right first time.

Over the years, there had been multiple approaches to building a scaleable Asterisk platform, most of them relied on the same principals: multiple Asterisk servers, singular point of entry with load balancing, single point of exit with LCR. Normally, when you talk Class-4 services (call routing, DID services, Calling Cards), this methodology would work just fine. When it comes to Class-5 (Centrex, Voicemail, Queues), things tend to get a bit more complex – but again, the basic methodology applies and remains. Over the years, we’ve seen contenders come and go, FreeSwitch, Kamailio, OpenSIPS, SER, OpenSEMS – they are all a means to an end, just get the number of concurrent calls and CPS ratio higher.

The question is this: “Is there a singular approach to Asterisk scaleability? is there a bullet proof recipe that we can use to achieve this Unicorn type configuration?” – the answer is: NO! – it is very much dependent on your application, your client side application, your general usage patterns and what kind of agility you are looking to expose to the end consumer.

Since the inception of Asterisk, and specifically since the inception of FreeSwitch, many people had been dissing Asterisk as being a monolithic environment. Many times, if you ask someone – “what does that mean?” – you would end up with a very googly eyed face, not really understanding what monolithic means. Yes, Asterisk is by definition a monolithic environment, which means, it was designed to work a self enclosed unit. If you think about it, how many PBX systems do you know that are not monolithic. The question in that case is: “If Asterisk is monolithic, how can we scale and expand it? how can we build something really big from something like Asterisk?”.

In martial arts you always learn to use your opponents strength as their weakness, as your weakness as your strength. If Asterisk’s monolithic nature is its weakness, let’s try and make it into its strength. How do we do that? we make sure that any decision (process, calculation, state handling, etc) that is cross platform is handled outside of Asterisk, while keeping call control and media handling at the monolithic layer. This yields two distinct advantages: we can develop our cross platform logic at ease, without impacting our Asterisk development process, we can develop our Asterisk logic as a singular unit and expand it as required, simply by adding more computation units horizontally. In network and platform design there is a simple rule of thumb – growing deep is complex, growing wide is simple. If the question of scaleability becomes a question of brute forcing additional computation resources, the issue is simple. If scaling out requires changes in the computational structure – you’ve done something wrong.

Over the years, we’ve developed several large scale Asterisk platforms. These had recently hit the combined user number of 15 Million, with over 850 Million minutes served on all platforms combined. Some of these are carrier oriented, some are social oriented – but in all of them the scaleabilty factor was important. In other words, the Unicorn is out there, we’ve actually managed to find it several times, each time somewhere else – but it was always grazing in similar locations. If you keep looking for the bulls, you will surely miss the Unicorn standing at the right of the road – right next to you.

Astricon 2014 – Start your engines!

Ok All, this is my official Astricon Countdown – start your engines, as Eric Klein and myself will be attending Astricon this year, Vegas here we come.

So, what are we going to talk about: Security and Cloud computing. Yes, over the past year, I’ve been returning to my old stomping ground, the various cloud infrastructure that is publicly available – and how to exploit it to the max. I will be talking about the various methods of speeding up your clouded Asterisk server, and most importantly, I’ll share some of the methodologies and logic behind building these instances, maintaining them and the various do’s and don’ts I’ve learned along the way.

I’m planning a few surprises and giveaways for my talk, so make sure you stay updated on this page 🙂


Building your Asterisk based Business – Part I

** Cross posted from: **

Since the inception of GreenfieldTech, back in 2007, we’ve assisted over 20 different VoIP companies to bootstrap their activities and launch their products. During that period, some of these companies had become a great success and some had disappeared from the face of the planet. This series of posts will bring the story of some of them – and we’ll try to analyse what made each company into a success or a failure.

Making the case for Asterisk based business

Let’s be honest and truthful with ourselves, we’re all capitalists. Yes, we are first Open Source evangelists and promoters, but at the end of the day, we do need to pay our bills and make ends meet. Thus, we monetize open source (in our case Asterisk) to the best of our abilities – there is no shame in that, and honestly, we take high pride in our ability to assist companies in monetizing open source tools and project in a productive and lucrative manner.

When people talk about Asterisk based businesses, normally they will consider one of the following tracks:

1. Asterisk PBX Integrator – Integrating Asterisk based PBX systems for companies of various sizes. Normally, this will include infrastructure installation, cabling, server support, hardware sales, etc.
2. Hosted PBX Service Provider – Providing VoIP PBX services, without any in-door server equipment, relying on the Internet or leased lines only. Normally this will include similar activities as the Asterisk PBX integrator.
3. Hosted IVR Service Provider – Providing Hosted Interactive Voice Response services to content providers and enterprises that can’t sustain their own IVR infrastructure.
4. Hosted Premium Services Provider – Identical to “Hosted IVR” with a focus on premium services content and adult content.
5. Telecom Carrier – Whether you are a mobile carrier or a landline CLEC, Asterisk applications can benefit a carrier of any size.
6. Value added services provider – Providing complimentary services to Asterisk and its derivatives.

The above is a very short list, as each item on that list can be expanded to 6 more sub-categories – however, these represent the major business types (not including consultants and developers, who fall under category number 1). There are similarities between these and significant differences as well. What may be true for one, maybe completely wrong for another – it all depends on your business goals and product development life cycle and life span. We will limit ourselves to discuss options 2 through 6.

Case I: Long Distance Calling Cards Operator in the US

June 2007, a representative of a calling cards operator in the US approached. The operator was back then using a hosted service from a company called Solegy (long gone) and was fairly unhappy with the service. Their main complaints were: lack of support responsiveness, lack of feature set, inability to expand existing feature set – and most importantly, inability to sustain a proper business model (unlimited calls), due to high termination and hosting prices.

At that time, the calling card operator had put the following restrictions as to creating the service:

1. Bootstrap pricing should not exceed the $10,000 USD.
2. All existing customers should be migrated to the new system.
3. All existing access numbers should be migrated to the new system.
4. New system should be based upon ready-made software – not customized development.
5. New system should enable additional service development and scalability.
6. New system should allow hosting on hosted servers, without any need for proprietary hardware.

The solution that was chosen (after all, it was 2007) was a mixture of OpenSER, coupled with Asterisk 1.4 and A2Billing. Calls will hit the OpenSER SIP proxy, which will then load balance to the various Asterisk servers. The solution met the various constraints listed above – and later on included a monthly support/maintenance cost that was sustainable for the business. The company continued on to provide direct DID numbers, VoIP termination services, Roaming SIM cards solutions, Hospitality Mobile phones and additional services. In 2010, the company had sold its operations to another company – which was a disaster. During the recent changes in the roaming market in the US and other regulations, the company had seized its operations and is no longer operating.

Was this company a success story? – Complicated Answer

Between the years 2007 and 2010, the company had grown from 1000 customers to a whopping 15,000 customers, paying a monthly service fee ranging from $15 USD to $59 USD. Roughly calculated, that’s an average of $550K USD per month turn-over. In deed, termination costs and operational costs rose up to around $450K USD a month, but considering the fact that the company had only 4 employees and two additional outsourced support resources, a monthly Net revenue of $100K USD is not bad – we can surely mark this as a success. The company realized that in order to sustain its business, they would require proper customer care and support services and they made sure these resources were clearly managed and sourced.

Following the company’s purchase in 2010, the new owner had decided that customer care and support structure aren’t really required, as the sales staff can handle customer care and support can be rendered by an outside resource on a per-incident base, with no binding SLA service.  Within less 12 months, the company customer base shrunk from 15,000 to around 3,000, bringing the entire operation to a stand-still. The new owner tried focusing on new business tracks, without preserving and maintaining the existing lucrative tracks.

So, what went wrong here?

The primary answer would be: failure of the new management to understand the business. Calling cards and VoIP services are customer oriented services. This applies to Mobile VoIP OTT services as well. Asterisk is a solid tool to use when building your business, it will take you a long way and make sure your ROI and TCO will remain as low as possible, while preserving your knowledge and experience in-house as much as possible. When a successful company is acquired by a financial outfit, that has no valid experience in the market sector, in many cases – it will fail. The lack of understanding of customer care structures, proper support, proper monitoring, proper NOC, ticketing handling, NOC liaising and proper technical escalation are the main attributes of a successful service and product in this industry.

In our next post

In our next post we’ll discuss the world of “ad-revenue financed calls” and the “call-back industry”, as it was in 2008, 2010 and what happened to it today.