Being a successful Asterisk Consultant

A while back, John Todd from Digium, had posted an entry on the Digium blog web site, regarding how to be a successfull Asterisk consultant. While I completely agree with John’s views on the matter, from obtaining a dCAP certificate to the involvement with the community – there are a few points missing from that post, at least in my view. I will try to add some additional information here, in the hopes that it may help you build your business.

Point 1 – Stay Focused

Most of us Asterisk consultants come from diversified areas of expertise. Most of us are plain old IP sysadmins or network managers who got thrown into the Asterisk world due to a requirement – got hooked on it and simply continued onwards. Some of us are developers, some web oriented, some core oriented, but developers yet. The diversity of most Asterisk consultants skill set can easily side track them.

When I say side track, I don’t meant that they don’t know what they are doing, I mean – it’s easy to try and swallow more than they can chew at one time. For example, example a sysadmin turning into an Asterisk consultant, after installing over 200 Asterisk systems. Now, a customer comes to him and says: “Well, I’m gonna give you the work, but I want you to also take over the various IT management aspects of the system.” – If at this point you will say: “YES” you are more of less dooming your business. You are an Asterisk consultant, no matter how a talented IT sysadmin you are, going about and taking both roles on your self would render you in a situation where you, at some point, will be in a situation where you are handling an extreme IT condition at that customer, rendering completely incapable of rendering services to your other customers. Remember, stay focused on what you do, you won’t run into a situation where you will be forced to hurt a customer.

Point 2 – Earning more is sometimes loosing money

This point relates directly to the previous one. Let us imagine that I’m an Asterisk developer with a background of Web development. When confronted with a project that may include both Asterisk and Web Development – the most logical answer would be “YES” – however, web developers tend to forget that they are working autonomously. Most web developers are backed up by teams of graphic artists, database developers, database managers and IT managers. Thus, a web application is much more than the web logic involved with it. Are you an all encompassing developer, capable of cater to all aspects of a web development project and an Asterisk project? if you have your own in house DBA and other resources, you should be fine, however, if you don’t – at some point in the project – you will be forced to outsource the work to a 3rd party – thus, lowering your net income on the project. So, by taking such a project you believe you will be earning more money, while in fact, at the end of the project you may end up in debt to 3rd party sub-contractors you hired.

Point 3 – Be true with yourself

Always be true and honest with yourself and always ask yourself: “is this really a deal that will advance me? or may it actually set me back?” – failing to answer these two questions for every project you are about to take on will end up with some disappointment. Remember, you can fool all people some of the time, you can fool a few people all the time – can you can’t fool yourself! You are your own worse judge, jury and executioner. If you end up doing a project that doesn’t feel right for you, or something with the various aspects of the project troubles your no a moral ground, at some point in time, it will creep up on you and bite you back in the ass.

Point 4 – Use it, don’t abuse it

We all deal with various aspects of the Asterisk project, an Open Source project at its core. It’s very easy to become side tracked by large sums of money, in order to either violate a GPL code or doing something which is completely negated to the Open Source spirit or the Asterisk community. Sure, you will abuse Asterisk and/or other Open Source Asterisk related projects, however, at some point, it will be discovered and your name will be smudged. For example, if you integrate ViciDial to a customer, tell them it’s ViciDial and don’t change its logo to something else. Same applies to FreePBX, A2Billing or other Asterisk related packages – at some point your customer will find out you integrated Open Source – and you will be branded a cheat.

For example, 2 weeks ago I was at a call center, where one of Israel’s leading Asterisk integrator had built a dialer platform for the call center. The call center manager told me that they paid a sum of about 120,000 Israeli Shekels (approx 30,000$) for that dialer. I was really interested to see the product, while the only thing I saw was a “logo” modified “ViciDial” with a couple of hooks into FreePBX (that also had its logo changed to the company logo). The customer was sure he was getting a personalised job, while actually, the entire amount of work done can be amounted to about 12-16 hours of work. Ok, so the hardware costs about 8000USD – still, 22,000$ for installing and modifying two pages on ViciDial – you can’t say that is right – is it?

Conclusion

Always be true to yourself, to your customers and to the community – you’ll never loose.

Copyright Enforcement in Israel – you gott’a be kidding me…

A few weeks ago I had posted one of my usuall “Open Source License” rants, where I explained and ranted about the state of Open Source license enforcement in Israel. A recent study by the IIPA (International Intelectual Property Alliance) had positioned Israel as the number 1 copyright piracy country in the world!

When you think abuot it, it’s a little strange, as Israel is fairly small. However,  in relation to the number of Internet connected users in Israel, the number of downloads of pirated software or other copyrighted material in Israel is of the highest percentage in the world. Sure, we all download a movie or episode here and there, but, some people in Israel go about and completely utilize pirated material only. Sure, I like watching my weekly episode of Fringe, but what can I do that no network in Israel is broadcasting it. So, I download the episodes via Bittorrent and watch them as they are published. However, on the other hand, I do purchase Microsoft licenses for my PC’s (yes, I have a Windows XP and a Windows Vista box - running Windows and Office), I did purchase a Mandriva PowerPack package for my Linux destktop and notebook and yes, I did purchase my books about DOJO, PHP and AJAX – so, I can honestly say that my utilization of pirated material is that for things I can’t obtain in Israel at all.

One would argue that it is still piracy, well, there is a certain point in that – however, if there is no one to pirate from where you are located, how can you pirate something? according to the dictionary, the noun priate means:

  1. One who robs at sea or plunders the land from the sea without commission from a sovereign nation.
  2. A ship used for this purpose.
  3. One who preys on others; a plunderer.
  4. One who makes use of or reproduces the work of another without authorization.
  5. One that operates an unlicensed, illegal television or radio station.
Ok, let’s take a look at the above and examine:
  1. Considering the fact that I’m not at sea nor am I attacking from the sea, I don’t qualify for item 1.
  2. I won’t even consider number 2.
  3. I don’t prey on others to take something, the airing of a TV show in the US is well published. Hell, the TV stations even publish their content online – only available in the US however – according to item 3.
  4. Ok, I do make use personal use of another persons work without authorization, however, as there is no local representation for the show that I’m watching – that point is somewhat muted in my view – according to item 4.
  5. I don’t operate an illegal or other wise unlicensed TV or Radio station – according to item 5.
So, taking all of the above, I can be considering a small time pirate – I only pirate the shows that I like watching. What’s available here I watch on TV.
Nonetheless, I’m not arguing that copyrighted material piracy is OK – the simple reason is that people in Israel even pirate the things that don’t need to be pirated. For example, Open Source software is being exploited and resold in Israel as proprietary software. Actually, people in Israel have no idea what Open Source really means, thus, people can push whatever lame story to people.
For example, this week I went to a meeting at a small Contact Center. I went there to discuss the installation of a Recording System for the installed PBX system (I have developed one of the most robust CRM/ERP aware recording systems for Asterisk). In any case, I go to the meeting and sit down with the CEO and owner of the Contact Center. I start explaining that I’m using Asterisk, he suddenly stops me and says that he met with the CEO of a certain company, who claims that they developed Asterisk. Actually, he said that the CEO claimed that the initial idea for Asterisk was his. I was pissed off! I started explaining to the man that Asterisk is developed by Digium and it’s an Open Source product and basically, apart from Asterisk Business Edition, no-one, not even I, can sell Asterisk as is. We can create a product based on Asterisk, but we can’t sell Asterisk, nor claim it is ours. After showing the man some websites and various videos of Mark Spencer discussing Asterisk he asked me: “How can that man claim that he developed Asterisk, when it is clear that he didn’t?” – and I responded: “Because people in Israel don’t give a damn and remain ambivalent to the truth”.
I guess that is the same reason why Internet Piracy is so big in Israel. Much of the stuff we want isn’t available here in shops, so go ahead and pirate it. Once you’re used to pirating something, pirating anything simply becomes a second nature to you. I suggest that the IIPA do a better statistic and check the actual pirated content being downloaded, out of which, check how much content isn’t available in Israel in normal distribution channels – and then remove that information from the statistics. I’m confident that while the number will still be high, Israel will no longer be number 1 in the list.

Read my words – 3500 concurrent channels with Asterisk!

One of the biggest questions in the world of Asterisk is: “How many concurrent channels can be sustained with an Asterisk server?” – while many had tried answering the question, the definitive answer still alludes us. Even the title of this post says “3500 concurrent channels with Asterisk” doesn’t really say much about what really happend. In order to be able to understand what “concurrent channels” really means in the Asterisk world, let us take a look at some tests that were done in the past.

Asterisk as a Signalling Only Switch

This scenario is one of the most common scenarios in the testing world, and relies upon the basic principle of allowing media (RTP) to traverse from one end-point to the other, while Asterisk is out of the loop regarding anything relating to media processing (RTP). Examine the following diagram from one of the publicly available OpenSER manuals:

Direct Media Path between phones via a SIP Proxy

Direct Media Path between phones via a SIP Proxy

As you can see from the above, the media path is established between our 2 SIP endpoints.

This classic scenario had been tested in multiple cases, with varying codec negotiations, varying server hardware, varying endpoints, varying versions of Asterisk – no matter what the case was, the results were more or less the same. Transnexus had reported being able to sustain over 1,200 concurrent channels in this scenario, which makes perfect sense.

Why does it make sense? very simple, as Asterisk doesn’t manage or mangle RTP packets, Asterisk performs less work and the server also consumes less resources.

Asterisk as a Media Gateway

Another test that people had done numerous times is to utilize Asterisk a Media Gateway. People used it as a SIP to PSTN gateway, SIP to IAX2 gateway, even as a SIP to SIP transcoder gateway. In any case, the performance here varied immensly from one configuration to another, however, they all relied on a simple call routing mechanism of routing calls between endpoints and allowing Asterisk to handle media proxy tasks and/or handle codec translation tasks.

Depending on the tested codec, I’ve seen reports of sustain over 300 concurrent channels of media on a single server, while other claim for around the 140 concurrent channels mark – this again mostly relied on various hardware/software/network configurations – so there is nothing new in there.

These tests tell us nothing

While these tests are really nice in the theoretical plane of thinking, it doesn’t really help us in the design and implementation of an Asterisk system – no matter if it is an IVR system, a PBX system or a time entry phone system for that matter – it simply doesn’t provide that kind of information.

The Amazon EC2 performance test

In my previous post, Rock Solid Clouded Asterisk, I’ve discussed the various mathmatics involved in calculating the RoI factors of utilizing Cloud computing. One thing the article didn’t really tell us, did it really work?

Well, here are some of the test results that we managed to validate:

  • Total number of Asterisk based Amazon EC2 instances used: 24
  • Total number of concurrent channels sustained per instances (including media and logic): 80
  • Average length of call: 45 seconds
  • Total number of calls served: 2.84 Million dials
  • Test length: approximately 36 hours

According to the above data, each server was required to dial an approximate 3300 dials every hour. So, let’s run the math again:

  • 3300 Diales per hour
  • 55 Dials per minute
  • As each call is an average of 45 seconds, this means that each gateway generates 20 calls
    per second, and within 4 seconds fills the 80 channels limit per server.

According to the above numbers that we’ve measured, each of the Amazon EC2 instances used was utilized to about 50% of its CPU power, while consuming a load average of 2.4, which was mostly caused by I/O utilization for SIP and RTP handling.

Conclusion

When asking for the maximum performance of Asterisk, the question is incorrect. The correct question should be: “What is the maximum perfromance of Asterisk, utilizing X as the application layout?” – where X is the key factor for the performance. Asterisk application performance can vary immensly from one application to another, while both appear to be doing the exact same thing.

When asking your consultant or integrator for the top performance, be sure to include your business logic and application logic in the Asterisk server, so that they may be able to better answer your question. Asterisk as Asterisk is just a tools, asking for its performance is like asking how many stakes a butcher’s knife can cut – it’s a question of what kind’a steaks you intend on cutting.

Sangoma USBfxo: too little, too late…

Sangoma recently introduced a new FXO product, the USBfxo. The USBfxo is a dual FXO port device, connected to your Asterisk server via a USB connection. Now, while I do admire the way Sangoma keeps trying to kick it up a notch with new products, but isn’t Sangoma a little late to jump on the USB train?

Sangoma USBfxo Device

Sangoma USBfxo Device

Xorcom had been in this business for 4 years now and I see no reason why would the Sangoma product be any better than the Xorcom product. In addition, if Sangoma is targeting their product at the very low-end PBX systems, in my book, they actually missed the product line. In my view, if Sangoma wants to put a proper USB device on the market, it should have a minimum of 4 ports on it, 3 FXO and 1 FXS. You are probably wondering why I’m propsing such a weird combo, well, the reason is simple – Fax machines and they yet to be improved Asterisk FAX capabilities, and the fact that people still use FXS port of physical fax machines. I’m one of the biggest Asterisk and VoIP promoters I know, and even I use a physical fax machine at some points in time. True I used Hylafax and IAXmodem to receive most of my fax transmissions, but when it comes to sending faxes, nothing beats a physical machine.

So, as I started saying, Sorry Sangoma, too little, too late … better luck next time!

Rock Solid Clouded Asterisk

This post is somewhat a combination of posts from previous posts, mainly, the posts about virtualization and my latest posts about the utilization of Amazon EC2. As some of you may know, a part of what I do at GreenfieldTech is develop various API’s for the Asterisk Open Source PBX systems. Two of these API’s are the IVR API and the Dialer API. This post if called “Rock Solid Clouded Asterisk” as it will describe the latest production environment that I’ve implemented, using these API’s and Amazon EC2 virtualization framework.

The network diagram

Our implementation consisted of the following general schematic:

Network Diagram

Network Diagram

The application logic was based upon a JAVA based web-service, implementing the XML-RPC server side of the IVR API, and a dialer management system that controlled the dialer API located on the remotely located dialers – hosted on Amazon EC2 instances. For simplicity, and we were very much aware this would reduce the overall capacity, we’ve located both the dialer framework and the IVR API execution on each of the servers, while allowing the server s to communicate internally.

Some constraints

As much as we wanted to run many Amazon AMI instances, we were limited to running 5 elastic IPs with a single Amazon AWS account. As a result, we’ve registered 5 accounts, and executed a total of 24 AMI instances with 24 elastic IP’s.

An additional constraint we had realised, but had no way of actually knowing its limitation was the actual number of concurrent calls per server. Initially, we’ve reached the following numbers and configuration on a physical server:

  • Intel Quad Core XEON
  • 2GB RAM
  • 1GB Network Uplink
  • CentOS 5.2 64bit
  • Total capacity: 120 concurrent calls of Dialer+IVR on a single server

Per our theory, if we managed to reach a similar capacity using amazon c1.medium instances, we would be very happy.

The results

After conducting a test utilizing a single AMI instance, we’ve reached the following results:

  • Dual Core instance (c1.medium)
  • 180GB Disk Storage
  • 8GB of RAM
  • Fedora Core 8 32bit
  • Total capacity: 80 concurrent calls of Dialer+IVR on a single instance

A decrease of 33% in comparison to the performance observed on a physical server. Ok, so we weren’t all that happy with these results, until we started doing the financial math, realising that using Amazon EC2 with that Dialer+IVR framework would yield a savings of almost 80% in operational costs.

Doing the math

The normal co-located option

Our aim was to reach a capacity of around 2800 concurrent channels. Per the normal physical setup, our hardware requirements would be to use at least 24 servers. At a price of 1500$ per server, that sums up to a total of 36,000$. Adding the time required to install 24 servers, the overall expense for 24 servers would be around the 42,000$ mark. To sustain a total of 2800 concurrent calls, using the g711 codec, we would be required to carry a total of 300Mbps internet uplink – basically talking about 10,000$ of bandwidth.

So, taking all of the above into consideration, we will need a total of 52,000$ just to maintain the hardware installation and operational cost. Taking into consideration that the system would be used at full for no more than a period of 30 hours, we end up with a total of: 1733$ per hour.

The Amazon EC2 option

Now, let’s calculate for Amazon EC2:

2800 concurrent channels translates into 35 instances. Price per c1.medium instance per hour is 0.2$. So, rack that up and you get: 210$ for operating 35 instances for 30 hours.

Elastic IP costs are 0.01$ per hour per server – a total of 10.5$ for 30 hours.

Bandwidth costs are 0.17 per each GB, so according to 300Mbps for 30 hours, with each call duration at 1 minute sums up to be: 5M of data per call. Calculating 2800 concurrent channels for 30 hours gives: 25,200,00 MB, or 25TB of traffic. According to Amazon, first 10TB are at 0.17$ per GB, and then the price goes down. So, let’s take a worst case of 0.17$ per GB. A total of 4284$ for operating 30 hours.

A total of: 4,468 US Dollars, Price per hours calculated at: 148$.

The savings

Per the task at hand, the utilization of Amazon EC2 yielded a savings of 92%

So, is Amazon EC2 good for any usage?

The answer is a definite NO! If your requirement is for a system that works 24×7, like a PBX system or a call center, then your utilization of Amazon EC2 would be identical to leasing a co-located server at any of the world wide co-location providers. If your application is of sporadic nature, or is utilized for short bursts of time, Amazon EC2 is a wonderful tool for lowering your overall expenses. Sure, it will require some work to get running, but the overall savings is more than worth-while.