22 Feb 10 Call Analytics – Beyond CDR analysis – Part I

“Oh, just get me the CDR’s and I’ll take it from there” – how many times have I heard these words before? I can’t even imagine the number of times in the past 15 years of IT/Telecom’s work that I’ve done and in the last 8 years of Asterisk in particular – when it comes to billing and fraud management, it would appear that the CDR’s are the Rosetta Stone of the industry.

Over the past 6 months, several of my friends and I had been asking ourselves this question: “Is there more to billing, fraud management and profit leakage? does it really all begins and ends with the CDRs?” – so, here we were, a group of 3 engineers dealing with telecom system and billing systems – we knew that the answer is a definite YES, however, how come most companies and system aren’t even aware of this, in such a way that causes them to leak telecom profits and waste their hard earned profit margins on simple accidental mis-interpretation of CDR records.

So, we’ve decided to sit down and start analyzing calls in real-time, trying to evaluate not only the CDR record that is received upon the completion of the call – but also understand the traversal path of the call, analyzing it in real time and evaluating it profit leakage potential. At the mean time, we’re concentrating our work on Asterisk, as it is the simplest for us to implement – however, we’re not focusing it only on that – we’ll looking at adding it to FreeSwitch, Yate, OpenSer/Kamailio, OpenSIPS and the various varients.

So, what have we done so far? well, one thing we never really had with any of the existing systems was a clear view of what’s going on “right-now” on our systems, so we said: “it would really be great if we could know how many call hits we’ve received during the past 15, 30, 45 or 60 minutes” – so here is what we made:

The above image shows our top 10 inbound DID numbers, as you can see these are in the 972 and 447 country codes (yes, we work mainly in Israel and the UK). At the backend, our servers are analyzing the data in real time, generating an active alert in the case a DID number’s statistics change in a somewhat drastic change, thus, establish a traffic anomaly. Another thing that interested us was our usage across multiple servers, which we are exhibiting in the below graph:

Now, as you can see, the top graph shows a discrete anomaly:

This anomaly indicates something went wrong on all our servers between 00:45 and 1:15, which gives us a fairly discrete period of time to seek for a problem in the system. What happened was that one of the guys updated a portion of the data traversal API – basically deleting it [we resumed full work after about 40 minutes].

So, where is it all going to? well simple, a new Open Source based service that we’ll be launching within a few months from now. Our intention is to provide a means for simple, straight forward, highly reliable, call analytics, fraud management and profit leakage analysis service. A service which is based upon a simple to use API on one hand and Open Source based data gathering agents. Our belief is that by analyzing large amounts of data, from multiple sources around the world, we’ll be able to ascertain the fingerprint of a telecom bound attack – being able to alert the respective users of the service and maybe in the later future, also provide a means to block the attack as it advances across the world.

I’ll be updating about our advancement as we go along, but for the time being, this is something I felt would interest you.

Tags: Application programming interface, Asterisk, Israel, opensource, Real-time computing, Rosetta Stone, Site Management, Statistics

15 Oct 09 Astricon 2009 – Glendale, AZ – Part II

Ok, it’s day 1 (or actually day 2) for AstriCon 2009 – and here’s my report for the day.

Yesterday was kind’a of a hectic day for me, as I was teaching a full day track of Asterisk and Cloud Computing, specifically, implementing Asterisk systems with Amazon EC2. I started the day with a class filled with 20+ people, and ended the day with a similar number – so in general I’m very happy. Not many people tend to attend the pre-conference days, so having that number of people and their positive reactions through out the day were very reassuring to me.

If there is one thing I’ve learned from this experience, it is the following: If you give a full day track, don’t arrive at the hotel 24 hours prior to it – you need at least 48 hours! People didn’t really notice (I hope), but through out the day I was suffering from a splitting headache – one that would usually send me right into bed with a couple of Advil’s. But hey, that didn’t stop me and I powered through it, I’m fairly proud of myself for doing so – as at the end of the day I regained back my strength and was livelier.

Today was the first official day of the conference – I gave the opening talk for the Cloud Computing track of the day. My talk was about how to build “IP Centrex” like services, without building an “IP Centrex”. I guess that I didn’t really introduce a brand new concept, but actually talked about something that many are thinking about, but are not inclined to try it on their own and burn some cash on. I guess my talk helped them out saying: “Hey, we’re not talking out of our asses here, this guy makes some sense and what we thought of isn’t that far fetched”.

Previous to that, Digium announced the 2009 Digium innovation award winners, where my company won an award in the pioneer category. This is the second year in a row my company had won the award, and I’m really happy with being acknowledged for this specific work. Having being a part of the community for over 7 years now, this award, at least to me personally, says a lot – it’s basically saying: “Look, you’ve done good, you’ve done some work that really helps out the project and the community in general – here’s a beer and a toast to you – hip hip” – well, that’s kind’a of a mouth full, but you get what I mean. I think that this is actually the place to mention that the award was for developing a high-powered Dialer/IVR platform, used in the Israeli elections and the work was contracted for a company called Shtrudel.

The all conference party is tonight – so I better rest up and be ready for it – should be fun. I guess beer and food are always a good mix when a bunch geeks are getting together

Tags: Amazon, Amazon EC2, Asterisk, Business, Cloud computing, digium, EC2, GreenfieldTech, Innovation Award, Virtualization, VMWARE

19 Jul 09 A baby, a house and a full time job

For those of you who know personally, you probably already know that last month I became a father. I guess the transition is something that I was more or less ready for, at least on the technical terms of the transition. However, the thing that completely surprised me was the mental transition – which isn’t even related to the somewhat lack of sleep here and there.

So, here I am, about a month and half away into the final touches to our new home, spending the weekend deploying over 100 meters of CAT-5 cabling in the house – yes the house is network rigged to the teeth. I’m sitting in my daughters room, clamping away the wall sockets for the network, thinking to myself: “hmmmm…. will Nitzan need a single network connection? or should I put at least two for future usage? … hmmmm…. well, I guess time would tell”. In any case, so there I was, spending most of my weekend being my own geeky self, thinking about wiring, networking, wireless exposures, access points, etc. I then go back home, and suddenly, all that disappears the minute I put Nitzan on my shoulder to burp her. It’s really funny, but with her on my shoulder, I guess everything goes away for a few minutes. My brother-in-law informed the house that he caught me burping Nitzan, while sitting at my computer answering emails with the other hand – Ok, so I can’t stop being a geek all together.

In any case, here I am juggling the various aspects of being a father to a new born baby, attending to the various tasks required to final touches of the house (painters, cleaners, air cons, dry walls, etc) and of course, attending to my customers – some of which are completely ambivalent to the fact that I’m under a constant lack of sleep in the past month. Well, I guess in a couple of months Nitzan will start sleeping better, and would make life easier for both me and my wife; in the mean while, we take comfort with the sleep periods my wife gets during the day, so that I can work and cater to my customers, while she caters to Nitzan during the nights – and I have to be honest about this, when it comes to the baby, my wife is the closest thing to a Jedi Knight, her ability to stay focused and clam even when the Nitzan is screaming is amazing – I can’t always do it.

Ok, enough about the house and Nitzan, let’s go back to been geeks for a bit. As you can see on the right hand side of the blog, I’ll be speaking at the up-coming Astricon. I’ll be giving a talk about how to build “IP-Centrex” like installations, utilizing Asterisk and tools like VMWARE, XEN and OpenVZ. However, while my talk may be interesting to you (I hope), my pre-conference tutorial will be much more interesting. I’ll be giving a full day tutorial, teaching people how to install Asterisk in a clouded environment (cloud computing), mainly the Amazon EC2 cloud computing infrastructure. For those of you reading this blog, you may have noticed that I’ve developed a distinct interest in the Amazon EC2 cloud, which I’ve written about several times and also lectured about at Amoocon. While my Amoocon presentation was mainly informational, at Astricon I’ll be primarily teaching you how to do what I did. Well, I won’t be teaching you the inner workings of the GreenfieldTech IVR API framework, although, if you’re gonna ask questions I will answer (especially if you ask the question 3 times, I can’t stand it when people ask the same question 3 times – I just have to answer it – nudge nudge).

Ok, back to fatherhood and Nitzan stuff. The mental transition that I was referring to before is something that I felt last night in its most force. My wife and I decided to go to one of the malls, not far away from our house. So, we entrusted Nitzan with my in-laws and drove to the mall. The mall I’m referring to is called “The seven stars mall” and we like it. It’s not a big mall, but its got this shop called DOMO, that carries these high class cooking ware (my wife and I really like to cook – my chilli con-carne is well known). So, here we were walking the mall, after I ordered a pair of shoes that I needed. So, my wife comments: “You know something, let’s see if there is some sale at Super-Pharm.” – and then we ended up purchasing baby formula, pacifiers and baby wipes. I then asked my wife if she maybe wants to walk into DOMO, but we both didn’t really think about it – suddenly, something that was like a default prior to Nitzan is no longer a default – interesting isn’t it?

In any case, this is how my life looks like at this point in time – and I have to admit that I kind’a like it. Sure, I don’t get as much sleep as I got before, but hey, I’m happy with it – so I just keep on smiling and go on forward.

Tags: Amazon EC2, Amazon Elastic Compute Cloud, Application programming interface, Cloud computing, File sharing, Google, Microsoft, Pirate Bay

18 Feb 09 Asterisk AGI Programming – New Book

Well, it’s finally out – my new book that is

Some of you already know, but over the past year I’ve been busy writing a new book. This time it’s a book for Asterisk developers, especially tailored to PHP developers wishing to utilizing the PHPAGI framework. The book is out from Packt Publishing (Like my old AsteriskNOW book) and is updated with all the recent changes in Asterisk – including version 1.6.X and DAHDI.

If you like my work with Asterisk and would like to read more of my work, go ahead and get an electronic version of this book. I know it’s a little self promoting, by hey, it never hurts does it?

I’ve included a chapter on how to build a complete project from scratch, detailing the various analysis steps and various paradigms required to develop a fully functional Asterisk based application. I believe that even experienced Asterisk developers will benefit from this book.

Tags: AGI, Asterisk, Asterisk Book, Asterisk Developers, Book, GPL, open source, Packt Publishing, php, PHPAGI

02 Feb 09 So long SigValue – Hello Asterisk + EC2!

As some of you may know, I’ll be attending the ITExpo in Miami Beach, Florida. The subject I’ll be lecturing about is “Virtualizing Asterisk”. However, I have to be honest, I really need to change the subject to be called “Asterisk in the Cloud“.

Ever since the introduction of Amazon EC2, people had been trying to get Asterisk to run properly inside an EC2 instance. While installing a vanilla Asterisk on any of the Fedora/RedHat variant instances in EC2 isn’t much of a hassle, getting the funky stuff to work is a little more tricky.

One of these tricky bits (which I hadn’t yet found a solution for) is the issue of supplying a timer for Asterisk’s MeetMe application. In the old days (prior to Asterisk 1.6), Asterisk required the utilization of a virtual timer driver, provided by Zaptel in the past and now the DAHDI framework. The problem is, that while you are fully capable of compiling and installing DAHDI on an Amazon EC2 instance – the problem starts once you want to use it.

A few words about Amazon EC2

For those not familiar with Amazon EC2, its general infrastructure is based upon the XEN virtualization project. XEN is a para-virtualization framework, meaning that is performs some of the work utilizing the underlying Operating System kernel and some of the work performed with a special Kernel in the virtualized Operating System instance. This poses an interesting issue with every type of application that relies on hardware resources and their emulation.

To learn more about the XEN project, go to http://www.xen.org.

So, where’s the big deal?

So, if you can compile your code and run it in an instance, as long as you have the kernel headers and kernel source packages – you should be just fine – right? WRONG!

Amazon EC2 deploys its own Kernel binary image upon bootstrap, causing what ever compilation you may have done to the Kernel to go away (unless you’re creating a machine from real scratch). Another issue is a version skew between the installed Operating System kernel modules, the actual kernel and the installed compiler. For example, the instance that I was using had the XEN capable kernel compiled with gcc version 4.0.X, while the installed operating system was gcc version 4.1.X – so, no matter what I did to compile my kernel modules or binary kernel, I would always end up in a situation where loading the newly compiled kernel modules will generate an error.

Did I manage to solve it? – NOT YET. I’m still working on it, and I have to admit, that considering the fact that I have over 10 years of Linux experience and had compiled kernels from scratch many times, this one has gotten me a little baffled – I guess I’ll just need a few more nights and a case of Red-Bull to crack this one open.

So, what can we do with EC2?

In my view, EC2 + Asterisk is the ultimate IN/NGN services environment – and I have proof of that. A recent lab test that I did with one of my customers showed a viable commercial alternative to Sigvalue when using Asterisk and EC2 structures. The main reason for our belief in using EC2 was the following graph:

What we’ve noticed was that while our IN/NGN system was generating traffic, it’s general usage showed peak usage for a period of 2.5 hours, with a gradial increase and decrease over a period of almost 10 hours. Immediately that led us to a question: “Can we use Amazon EC2 to provide an automatd scaling facility for the IN/NGN system, allowing the system to reduce its size as required?”

To do this, we’ve devised the following IN/NGN system:

Our softswitch would have a static definition of routing calls to all our Asterisk servers, including our EC2 instances which had static Elastic IP numbers assigned to these. The EC2 Controller server was incharge of initiating the EC2 instances at the pre-defined times, mainly, 30 minutes prior to the projected increase in traffic. Once the controller reaches its due timer, it will automatically launch the EC2 instances required to sustain the inbound traffic.

For our tests, we’ve initiated 5 AMI instances, using the EC2 c1.medium instance. This instance basically includes 2 cores of an AMD opteron, about 8GB of RAM and about 160GB of Hard drive – more than enough. Initially, we’ve started spreading the load evenly across the servers, reaching about 80 concurrent channels per instance, and all was working just fine. We managed to reach a point where we were able to sustain a total of about 110 concurrent channels per instance, including the media handling – which is not too bad, considering that we are running inside a XEN instance. The one thing that made the entire environment extremely light weight is the GTx Suite of APIs for Asterisk. Thanks to the GTx Suite of APIs, scalability is fairly simple, as all application-layer logic is controlled from a central business logic engine, serving the Asterisk servers via an XML-RPC based web service. Thanks to Amazon, practically infinite, bandwidth allocation – the connections from the Asterisk servers to the US based central business logic was set at a whopping 25mSec, thus, there was no visible delay to the end user.

It is clear that the utilization of Asterisk and EC2 operational constructs can allow a carrier to establish their own IN/NGN environment. However, how these are designed, implemented and operated are at the hands of the carrier – and not a specific vendor. If the carriers around the world will take to this approach, time will tell. As a recent survey stated that 18% of the US PBX market is currently dominated by Open Source solution, having Digium dominate 85% of these 18% (~15%), I’m confident that we will see this combination of solutions in the near future.

Tags: Amazon, Asterisk, Broadsoft, digium, EC2, Nortel, Personeta, S3, SigValue, Sylantro, Virtualization