As some of you know, over the past 9 months, I’ve been heavily involved in the establishment of Humbug. For those who may not know, Humbug is a Call Analytics and Fraud Analysis SAAS. Now, differing from many of the current telephony SAAS projects, we are not based on Amazon EC2 or some other public cloud infrastructure, we build our own cloud environment. Why do we build our own cloud? simple, we need to keep your data secured and confidential. At Humbug, we see ourselves as a cross between Google Analytics – in our ability to analyze and handle data and Verisign – in our security and confidentiality requirements and methodologies.

Question be asked, why do people trust Verisign to provide SSL certificates around the world. What makes Verisign’s CA better than a privately owned CA – the answer is simple, it’s a third party 2 entities can entrust at the same time. Humbug aims to provide the same lever of trust, simply because we regard your data as sacred and valuable.

Since about 2 months ago, we’ve been contacting various Asterisk integrators around the world, inviting them to evaluate Humbug services. Now, while some integrators and vendors were somewhat reluctant, others were more than happy to join. We now have over 250 monitored systems around the world, with system being monitored and analyzed in Israel, USA, UK, Brazil and more.

The thing that amazed me in regards to some of the integrators who decided not to participate was that they claimed: “we provide our customers our own brew of fraud analysis service, we don’t require your SAAS”. Now, while I can accept the fact that an integrator would offer such a SAAS as an in-house service, I can’t see why a customer would rely on these services. In my view, relying on your integrator to provide fraud analysis services is like relying on the integrator of your alarm system to provide hired guard services – it just doesn’t make any sense to me. Why doesn’t it make sense? in Hebrew we say: “Go prove that you have a sister”. Imagine that your PBX integrator offer you such a service, then, in some obscure manner, your PBX gets hijacked and you get slammed with 50K$ worth of phone calls to Somalia. Now, your integrator would say: “Hmmmmm… that’s odd, we didn’t even get those CDR events to our system… you really got hacked bad…” – sure, if you only rely on CDR records to do your analysis (which is what 99.9% of integrators do). There is much much much much more to fraud analysis than just CDR analysis – if it all began and finished with CDR analysis, then by far Cvidya, Verint, NICE and many others would have been made redundant.

Allowing your integrator to provide you with fraud analysis SAAS is like putting the fox to guard the hen house, when things louse up (and they may), he’s the first one to bail out saying: “It’s not my fault”.

Humbug takes a totally different approach to fraud analysis, specifically, in the way we regards the various PBX systems and integrators. We are vendor agnostic and integrator agnostic – we will provide you with the clear and concise information you require in order to make an educated decision as to how you were de-frauded (if de-frauded) and provide you a faster alerting and response time. Our recent adventures had lowered our fraud alert response time from 60 minutes, down to 14 minutes in some cases. Most fraud analysis system carry a 24-36 hour turn around time, by that time, you can be out of 50K$ – our aim is to lower that number to no more than a 100$ in the worst case. Ambitious? yes, down right crazy? probably so, but we always say: “Aim for the moon, you’ll land on a star!” – so we know we’ll get there.

Ok, it’s day 1 (or actually day 2) for AstriCon 2009 – and here’s my report for the day.

Yesterday was kind’a of a hectic day for me, as I was teaching a full day track of Asterisk and Cloud Computing, specifically, implementing Asterisk systems with Amazon EC2. I started the day with a class filled with 20+ people, and ended the day with a similar number – so in general I’m very happy. Not many people tend to attend the pre-conference days, so having that number of people and their positive reactions through out the day were very reassuring to me.

If there is one thing I’ve learned from this experience, it is the following: If you give a full day track, don’t arrive at the hotel 24 hours prior to it – you need at least 48 hours! People didn’t really notice (I hope), but through out the day I was suffering from a splitting headache – one that would usually send me right into bed with a couple of Advil’s. But hey, that didn’t stop me and I powered through it, I’m fairly proud of myself for doing so – as at the end of the day I regained back my strength and was livelier.

Today was the first official day of the conference – I gave the opening talk for the Cloud Computing track of the day. My talk was about how to build “IP Centrex” like services, without building an “IP Centrex”. I guess that I didn’t really introduce a brand new concept, but actually talked about something that many are thinking about, but are not inclined to try it on their own and burn some cash on. I guess my talk helped them out saying: “Hey, we’re not talking out of our asses here, this guy makes some sense and what we thought of isn’t that far fetched”.

Previous to that, Digium announced the 2009 Digium innovation award winners, where my company won an award in the pioneer category. This is the second year in a row my company had won the award, and I’m really happy with being acknowledged for this specific work. Having being a part of the community for over 7 years now, this award, at least to me personally, says a lot – it’s basically saying: “Look, you’ve done good, you’ve done some work that really helps out the project and the community in general – here’s a beer and a toast to you – hip hip” – well, that’s kind’a of a mouth full, but you get what I mean. I think that this is actually the place to mention that the award was for developing a high-powered Dialer/IVR platform, used in the Israeli elections and the work was contracted for a company called Shtrudel.

The all conference party is tonight – so I better rest up and be ready for it – should be fun. I guess beer and food are always a good mix when a bunch geeks are getting together

For those of you who know personally, you probably already know that last month I became a father. I guess the transition is something that I was more or less ready for, at least on the technical terms of the transition. However, the thing that completely surprised me was the mental transition – which isn’t even related to the somewhat lack of sleep here and there.

So, here I am, about a month and half away into the final touches to our new home, spending the weekend deploying over 100 meters of CAT-5 cabling in the house – yes the house is network rigged to the teeth. I’m sitting in my daughters room, clamping away the wall sockets for the network, thinking to myself: “hmmmm…. will Nitzan need a single network connection? or should I put at least two for future usage? … hmmmm…. well, I guess time would tell”. In any case, so there I was, spending most of my weekend being my own geeky self, thinking about wiring, networking, wireless exposures, access points, etc. I then go back home, and suddenly, all that disappears the minute I put Nitzan on my shoulder to burp her. It’s really funny, but with her on my shoulder, I guess everything goes away for a few minutes. My brother-in-law informed the house that he caught me burping Nitzan, while sitting at my computer answering emails with the other hand – Ok, so I can’t stop being a geek all together.

In any case, here I am juggling the various aspects of being a father to a new born baby, attending to the various tasks required to final touches of the house (painters, cleaners, air cons, dry walls, etc) and of course, attending to my customers – some of which are completely ambivalent to the fact that I’m under a constant lack of sleep in the past month. Well, I guess in a couple of months Nitzan will start sleeping better, and would make life easier for both me and my wife; in the mean while, we take comfort with the sleep periods my wife gets during the day, so that I can work and cater to my customers, while she caters to Nitzan during the nights – and I have to be honest about this, when it comes to the baby, my wife is the closest thing to a Jedi Knight, her ability to stay focused and clam even when the Nitzan is screaming is amazing – I can’t always do it.

Ok, enough about the house and Nitzan, let’s go back to been geeks for a bit. As you can see on the right hand side of the blog, I’ll be speaking at the up-coming Astricon. I’ll be giving a talk about how to build “IP-Centrex” like installations, utilizing Asterisk and tools like VMWARE, XEN and OpenVZ. However, while my talk may be interesting to you (I hope), my pre-conference tutorial will be much more interesting. I’ll be giving a full day tutorial, teaching people how to install Asterisk in a clouded environment (cloud computing), mainly the Amazon EC2 cloud computing infrastructure. For those of you reading this blog, you may have noticed that I’ve developed a distinct interest in the Amazon EC2 cloud, which I’ve written about several times and also lectured about at Amoocon. While my Amoocon presentation was mainly informational, at Astricon I’ll be primarily teaching you how to do what I did. Well, I won’t be teaching you the inner workings of the GreenfieldTech IVR API framework, although, if you’re gonna ask questions I will answer (especially if you ask the question 3 times, I can’t stand it when people ask the same question 3 times – I just have to answer it – nudge nudge).

Ok, back to fatherhood and Nitzan stuff. The mental transition that I was referring to before is something that I felt last night in its most force. My wife and I decided to go to one of the malls, not far away from our house. So, we entrusted Nitzan with my in-laws and drove to the mall. The mall I’m referring to is called “The seven stars mall” and we like it. It’s not a big mall, but its got this shop called DOMO, that carries these high class cooking ware (my wife and I really like to cook – my chilli con-carne is well known). So, here we were walking the mall, after I ordered a pair of shoes that I needed. So, my wife comments: “You know something, let’s see if there is some sale at Super-Pharm.” – and then we ended up purchasing baby formula, pacifiers and baby wipes. I then asked my wife if she maybe wants to walk into DOMO, but we both didn’t really think about it – suddenly, something that was like a default prior to Nitzan is no longer a default – interesting isn’t it?

In any case, this is how my life looks like at this point in time – and I have to admit that I kind’a like it. Sure, I don’t get as much sleep as I got before, but hey, I’m happy with it – so I just keep on smiling and go on forward.

I recently gave a presentation at the Amoocon convention, held in Rostock, Germany – about Asterisk and Amazon EC2. Below is a medium quality video of that presentation:

or you may download it here:
Amazon EC2 and Asterisk video files

This post is somewhat a combination of posts from previous posts, mainly, the posts about virtualization and my latest posts about the utilization of Amazon EC2. As some of you may know, a part of what I do at GreenfieldTech is develop various API’s for the Asterisk Open Source PBX systems. Two of these API’s are the IVR API and the Dialer API. This post if called “Rock Solid Clouded Asterisk” as it will describe the latest production environment that I’ve implemented, using these API’s and Amazon EC2 virtualization framework.

The network diagram

Our implementation consisted of the following general schematic:

Network Diagram

The application logic was based upon a JAVA based web-service, implementing the XML-RPC server side of the IVR API, and a dialer management system that controlled the dialer API located on the remotely located dialers – hosted on Amazon EC2 instances. For simplicity, and we were very much aware this would reduce the overall capacity, we’ve located both the dialer framework and the IVR API execution on each of the servers, while allowing the server s to communicate internally.

Some constraints

As much as we wanted to run many Amazon AMI instances, we were limited to running 5 elastic IPs with a single Amazon AWS account. As a result, we’ve registered 5 accounts, and executed a total of 24 AMI instances with 24 elastic IP’s.

An additional constraint we had realised, but had no way of actually knowing its limitation was the actual number of concurrent calls per server. Initially, we’ve reached the following numbers and configuration on a physical server:

• Intel Quad Core XEON
• 2GB RAM
• 1GB Network Uplink
• CentOS 5.2 64bit
• Total capacity: 120 concurrent calls of Dialer+IVR on a single server

Per our theory, if we managed to reach a similar capacity using amazon c1.medium instances, we would be very happy.

The results

After conducting a test utilizing a single AMI instance, we’ve reached the following results:

• Dual Core instance (c1.medium)
• 180GB Disk Storage
• 8GB of RAM
• Fedora Core 8 32bit
• Total capacity: 80 concurrent calls of Dialer+IVR on a single instance

A decrease of 33% in comparison to the performance observed on a physical server. Ok, so we weren’t all that happy with these results, until we started doing the financial math, realising that using Amazon EC2 with that Dialer+IVR framework would yield a savings of almost 80% in operational costs.

Doing the math

The normal co-located option

Our aim was to reach a capacity of around 2800 concurrent channels. Per the normal physical setup, our hardware requirements would be to use at least 24 servers. At a price of 1500$ per server, that sums up to a total of 36,000$. Adding the time required to install 24 servers, the overall expense for 24 servers would be around the 42,000$ mark. To sustain a total of 2800 concurrent calls, using the g711 codec, we would be required to carry a total of 300Mbps internet uplink – basically talking about 10,000$ of bandwidth.

So, taking all of the above into consideration, we will need a total of 52,000$ just to maintain the hardware installation and operational cost. Taking into consideration that the system would be used at full for no more than a period of 30 hours, we end up with a total of: 1733$ per hour.

The Amazon EC2 option

Now, let’s calculate for Amazon EC2:

2800 concurrent channels translates into 35 instances. Price per c1.medium instance per hour is 0.2$. So, rack that up and you get: 210$ for operating 35 instances for 30 hours.

Elastic IP costs are 0.01$ per hour per server – a total of 10.5$ for 30 hours.

Bandwidth costs are 0.17 per each GB, so according to 300Mbps for 30 hours, with each call duration at 1 minute sums up to be: 5M of data per call. Calculating 2800 concurrent channels for 30 hours gives: 25,200,00 MB, or 25TB of traffic. According to Amazon, first 10TB are at 0.17$ per GB, and then the price goes down. So, let’s take a worst case of 0.17$ per GB. A total of 4284$ for operating 30 hours.

A total of: 4,468 US Dollars, Price per hours calculated at: 148$.

The savings

Per the task at hand, the utilization of Amazon EC2 yielded a savings of 92%

So, is Amazon EC2 good for any usage?

The answer is a definite NO! If your requirement is for a system that works 24×7, like a PBX system or a call center, then your utilization of Amazon EC2 would be identical to leasing a co-located server at any of the world wide co-location providers. If your application is of sporadic nature, or is utilized for short bursts of time, Amazon EC2 is a wonderful tool for lowering your overall expenses. Sure, it will require some work to get running, but the overall savings is more than worth-while.

Well, I just got back from the ITExpo show in Miami, Florida. I have to admit that I really enjoyed the venue, although I didn’t really have time to walk the floor. The main reason that I was unable to walk the floor was due to the fact that I gave a talk, as part of the Digium Asterisk World venue, which was co-located with TMCnet’s ITExpo.

My talk was about the possibilities and incentives for Virtualizing Asterisk using VMWARE and Amazon EC2. Following below is the presentation that I gave.

Virtualizing Asterisk – Presented at Digium Asterisk World, Feb 2008, Miami, Florida

I really like going to events at the GarageGeeks (www.garagegeeks.org) – apart from meeting with some friends, having a good laugh and enjoying the casual beer, the technical talks and the general subjects that are being attended at a gathering are always interesting.

This time, the Geeks had decided to focus on Cloud Computing, trying to introduce the somewhat highly evolving field of the IT/Computing industry to the visitors of Garage. Now, I have to admit that during the past few weeks I’ve been studying the various aspects of utilizing Amazon EC2 services, and I’ve come to the somewhat problematic conclusion: “Cloud computing is a wonderful solution, as long as you have clear and concise control over your computing cloud!”

The main problem with traditional cloud computing, as offered by Amazon and others, is that the physical boundaries of your system are somewhat unknown – not to you and in some cases, not to your provider either. Let us imagine a Tier based architecture platform, implemented utilizing Amazon EC2 technologies – our web front-end may be located in the US, the application server in the UK and the Database in Zimbabwe for all that matters – in accordance to EC2 and traditional cloud computing, that is fairly normal.While EC2 and some other providers had introduced the idea of a Zone, making sure that your virtual servers are all located in the same physical data center, other solutions are yet to follow.

The above design introduces an interesting “Critical Path” management problem, tied deep into the Tier based architecture design pattern. Companies such as GigaSpaces and others had made it their business to cash in on the problem, by providing a distributed, memory based, middle-ware – literally sweeping the “Critical Path” problem under the rug. However, as the platform will evolve, “Critical Path” issues have a tendency to re-appear, usually, a lot more serious than the original problem – due to new constraints and new operational paradigms within the operational construct of the cloud platform.

So, question be asked: “Is cloud computing good? Can you really build a full service based upon cloud computing services?” – The answer isn’t a straight forward yes or no. For example, while I have a couple of servers on EC2, serving me as development servers mainly, I’ve built my own little cloud environment on my own servers – used for production usage. I’ve used a mixture of OpenVZ servers tied together with my own provisional interface to create a seamless cloud, enabling me to replicate servers, control their operational structures at east and most important – expand my cloud rapidly by introducing new servers to my cloud.

Is my paradigm good for all purposes? I can’t really say – it works for me, it may not work for you. The thing is this, don’t expect cloud computing to come in and solve all your problems, you may run into new ones you are completely unfamiliar with.