22 Feb 10 Call Analytics – Beyond CDR analysis – Part I

“Oh, just get me the CDR’s and I’ll take it from there” – how many times have I heard these words before? I can’t even imagine the number of times in the past 15 years of IT/Telecom’s work that I’ve done and in the last 8 years of Asterisk in particular – when it comes to billing and fraud management, it would appear that the CDR’s are the Rosetta Stone of the industry.

Over the past 6 months, several of my friends and I had been asking ourselves this question: “Is there more to billing, fraud management and profit leakage? does it really all begins and ends with the CDRs?” – so, here we were, a group of 3 engineers dealing with telecom system and billing systems – we knew that the answer is a definite YES, however, how come most companies and system aren’t even aware of this, in such a way that causes them to leak telecom profits and waste their hard earned profit margins on simple accidental mis-interpretation of CDR records.

So, we’ve decided to sit down and start analyzing calls in real-time, trying to evaluate not only the CDR record that is received upon the completion of the call – but also understand the traversal path of the call, analyzing it in real time and evaluating it profit leakage potential. At the mean time, we’re concentrating our work on Asterisk, as it is the simplest for us to implement – however, we’re not focusing it only on that – we’ll looking at adding it to FreeSwitch, Yate, OpenSer/Kamailio, OpenSIPS and the various varients.

So, what have we done so far? well, one thing we never really had with any of the existing systems was a clear view of what’s going on “right-now” on our systems, so we said: “it would really be great if we could know how many call hits we’ve received during the past 15, 30, 45 or 60 minutes” – so here is what we made:

The above image shows our top 10 inbound DID numbers, as you can see these are in the 972 and 447 country codes (yes, we work mainly in Israel and the UK). At the backend, our servers are analyzing the data in real time, generating an active alert in the case a DID number’s statistics change in a somewhat drastic change, thus, establish a traffic anomaly. Another thing that interested us was our usage across multiple servers, which we are exhibiting in the below graph:

Now, as you can see, the top graph shows a discrete anomaly:

This anomaly indicates something went wrong on all our servers between 00:45 and 1:15, which gives us a fairly discrete period of time to seek for a problem in the system. What happened was that one of the guys updated a portion of the data traversal API – basically deleting it [we resumed full work after about 40 minutes].

So, where is it all going to? well simple, a new Open Source based service that we’ll be launching within a few months from now. Our intention is to provide a means for simple, straight forward, highly reliable, call analytics, fraud management and profit leakage analysis service. A service which is based upon a simple to use API on one hand and Open Source based data gathering agents. Our belief is that by analyzing large amounts of data, from multiple sources around the world, we’ll be able to ascertain the fingerprint of a telecom bound attack – being able to alert the respective users of the service and maybe in the later future, also provide a means to block the attack as it advances across the world.

I’ll be updating about our advancement as we go along, but for the time being, this is something I felt would interest you.

Tags: Application programming interface, Asterisk, Israel, opensource, Real-time computing, Rosetta Stone, Site Management, Statistics

12 Feb 10 Beyond the dialtone – PBX user experience revisited

When most of us think about PBX systems, we usually associate these with cumbersome usage, confusing dialing codes and in most cases – a PBX system is automatically associated with the annoying task of transferring a call from one handset to another. Lately, I’ve been thinking deeply about how people use PBX systems, is this really the only way to use a PBX system? is there something else to the mix? can we really enrich one of the oldest operational paradigms in the world? – and for that matter, can the public be re-educated to assimilate a new breed of PBX systems or services?

As to answering the question of re-educating the public, I guess I’ll have to leave that question to the head shrinks. As to answering the latter, enriching the PBX experience is both achievable and advisable. When I say enriching, I mainly talk about your ability to bring to the IP phone functionality usually not associated with it. Imagine to have the ability to receive a stock exchange RSS feed to your phones idle screen, notice that you stock is either rising or falling, and by the flick of a button – either sell or buy. We’ve all come accustomed to IP phones that look like the one of the right. A whole bunch of buttons, that in most cases have no direct use when our phone is utilized using a single account. However, these buttons can be externally re-assigned and re-programmed to achieve greater functionality – surpassing the normal behavior of just making phone calls.

The technology involved exists on almost every high-end IP phone on the market (well, at least those made by SNOM, Aastra, Cisco and Polycom – most of the Chinese makers don’t have this) – it’s called a Mini Browsers. Mini Browsers are exactly what they are called, these are simplified versions of your typical Internet browser. Some vendors had produced their own XML based Mini browser markup language (SNOM, Cisco, Aastra) while others had decided to provide a sub-set of XHTML (Polycom). The variations between the vendors are at the neck deep of the problems of using Mini Browsers, and that is that the formats are considerably different. Sure, SNOM had more or less adopted Cisco’s general structure, however, it still varies.

Through the utilization of this technology, it is possible to create phone based browser applications, that seem native to the phone user, as the general interface resembles the native phone interface. It is now the developers job to make the web interface displayed to the user as seamless and as native as possible, keeping in mind that the developer must remain agnostic to the information retrieval layer. Most companies leave their phone systems and these tasks to their system administrators and infrastructure team, however, this task is far beyond their capabilities and skill set. Creating an agnostic IP phone minibrowser dislplay layer, capable of utilizing multiple vendors and models, is a question of content management and content rendering, very must similar to the content transcoding problem that is common to the mobile content world – in other words, a sys-admin will create an ad-hoc solution, a programmer will create a proper, well structured, well designed solution that carry the enterprise beyond its initial needs and requirements.

A short example of how these interfaces work can be found here – on my company blog.

Tags: Aastra Technologies, Asterisk, Cisco, Cisco Systems, digium, Fonality, FreePBX, freeswitch, Markup language, Nortel, Polycom, sangoma, SIP, TrixBox, User interface, VoIP, XHTML, XML

02 Feb 10 Digium TE205P vs. OpenVox D210P

If there is one thing I like doing is testing hardware, specifically, testing new hardware that is related to Asterisk. I was more than pleased when OpenVox had approached me, asking to review one of their products – specifically after I once announced that I really dislike cheap clone cards. So, I got OpenVox’s D210P card, which is a fairly similar clone to the TE205/TE210 of Digium, and I decided to take a it for a test drive.

So, first off, lets take a look at Digium’s TE205 card:

Digium TE205P Card

The card is based upon two specific chips, the Xilinx Spartan FPGA and an Inifineon based Quad E1/T1/J1 framer chip. Technically speaking, the entire brain of the outfit is located in the Xilinx FPGA (naturally), which on the TE205P now enables remote firmware upgrades and some additional features. Digium had been using Xilinx based boards for over 8 years now, and they’ve been doing the job more than well.

Now, let’s take a look at the OpenVox clone board:

OpenVox D210 Card

The Test Scenario and Comparison

All of the above tests were conducted according to the following scenario:

Testing Lab Server

In general, I’ve connected 3 different IP phones to the testing server: A Polycom 650, a SNOM 370 and a Grandstream GXP2000. All IP phones include the latest firmwares and updates and were all working flawlessly with another similar setup, so I assumed they were all bug and issue free for the testing lab. The main reason I’m using 64Bit CentOS is simply due to the fact that all my servers are 64Bit capable (mainly E5410 and E5405).

Test 1: Normal Telephony

Well, in general, the card does exactly what it should – provides a connection to an E1 circuit (we only have E1 circuits in Israel). I’ve conducted normal telephony functions from all the above mentioned phones. In general, I’ve conduct from each phone a total of 40 calls, and repeated the test once for the Digium TE205P card and once for the OpenVox D210P card. The results were fairly similar with a slight advantage for Digium. In general, the OpenVox card had slipped about 4% of the calls, mainly to an IRQ miss that occurred for some reason. With the Digium card, the IRQ misses were not exhibited, allowing for all 120 calls to traverse normally.

Conclusion: In a normal office telephony scenario, the D210P is a fair choice – however, not my preference for a Call Center or a service provider.

Test 2: 3G based transmission (64kbps bearer capability)

I’ve been playing around with IVVR and Asterisk, mainly using the Fontventa H264 packages for Asterisk (that’s why I used 1.4 branch). With this test, the D210P provided less then medium results, specifically when trying to stream large 3gpp based video streams, while the TE205P had showed no specific issue with the transmission. Main issues exhibited were related to choppy video streams, causing jumps in the stream. The Digium card was fully capable of stream the video without a hitch. Now, I won’t hold this again OpenVox, as this usage is fairly advanced and is required by a very small portion of the market, but I believe they still have some work to do there. As they are using the same framer as Digium, I would deduce that their firmware is either an older import from Digium (reverse engineer) or some other firmware related issue.

Conclusion: Not a pick for 3G transmission with Asterisk.

Test 3: Dropped calls during high loads

No matter what test I did, with OpenVox I’ve always received a dropped call ratio of around 3-4% – when at high loads that went up to around 7%. When I mean high loads, I mean generating 30 outbound calls from Asterisk to one circuit, then receiving them on the second port (yes, a back-loop). I’ve conducted 100 runs of this test, at various speeds. It would appear that when generating calls with a 100ms interval between one initiation to another on the circuit, the OpenVox will drop a call here and there – at sporadic intervals. This may be actually related to the IRQ misses exhibited in Test 1.

Conclusion: If you have high load anticipated – OpenVox is not the choice for you.

Test 4: CPU Load/Spikes

It is a well known fact that all card that are used with Asterisk introduce load spikes of a sporadic nature. In the past, the masters of low spikes were Sangoma, however, with the introduction of Digium’s VoiceBus, that balance had tipped and Digium took the upper hand. In order to evaluate the spikes, I’ve monitor the machines’ load while having 30 calls traverse from one port to the other. The calls were playing back a static file of 5 minutes, and after disconnecting the calls would generate and additional one and continue from there. Both cards exhibited slight spikes when multiple calls either originate or disconnect, however, the CPU spikes that the OpenVox card had exhibited were about 40% higher than the ones exhibited by Digium and there were more spikes than with Digium.

Conclusion: If your system isn’t as beefy as mine, and you need full capacity – OpenVox isn’t the choice for you

Overall Operational Conclusion

The OpenVox card promises to be a low-cost alternative to the Digium card, and it surely delivers. Over all, if you have an office PBX system or a low scale IVR environment, the OpenVox alternative can be evaluated, although it’s not my personal favorite. Sure, in many cases I can say: “OpenVox would do the job” – but hey, I would always rather go with the original and not the clone. I believe that OpenVox are far ahead of its clone competitors (Atcom, Yeastar, Varion, PhonicEQ, etc), simply because it does a better job at building and designing a better card – however, they still have some way to go in order to be completely in-lined with Digium and Sangoma.

Tags: Asterisk, D210, digium, E1, OpenVOX, sangoma, T1, TE205, TE210

21 Jan 10 Chinese Domain Scam Alert!

Over the years I’ve seen many scams running on the net. Ranging from the ever annoying chain mails to the ever popular Nigerian Sting – Internet fraud is all around us. Lately, I’ve been hit by a new type of fraud attack, a domain registration fraud attack – mainly located in China and Hong-Kong.

As you may know, I’m the owner and CEO of a company called GreenfieldTech, dealing with Asterisk and VoIP application and platform development. Now, we operate world wide and render services to some of the world biggest brand in the telecom industry. So, we take our copyright and brand very seriously, when we receive an indication that someone is or may be infringing our copyright or brand, we take a stand for it.

So, today I’ve received this email:

Dear CEO, 

We are a domain name registrar centre in HongKong,and in charge of the registeration in
Asia, We have something important need to confirm through your company. 

We received a formal application from a company called "Hempus International Holdings
Ltd" applying to register 

Internet keyword :     greenfieldtech 

Domain names :

 greenfieldtech.asia    
 greenfieldtech.cn    
 greenfieldtech.com.cn    
 greenfieldtech.hk    
 greenfieldtech.in    
 greenfieldtech.mobi    
 greenfieldtech.net.cn    
 greenfieldtech.tw

In China and also in Asia on January 21 2010. During our auditing procedure we find out
that the alleged "Hempus International Holdings Ltd" has no trade mark,Intellectual
property, nor patent even similar to that word. As authorized anti-cybersquatting
organization we hereby suspect the alleged "Hempus International Holdings Ltd" to be a 
domain grabber. Hence we need you confirmation for two things:

First of all, whether this alleged "Hempus International Holdings Ltd" is your business
partner or distributor in China.

Secondly, Whether do you need to protect the intellectual property right which should have
belonged to you?. (The alleged "Hempus International Holdings Ltd" will be entitled to obtain
a domain not needed by  original trademark owner.)
If you are not in charge of this please transfer this email to appropriate dept.in order to
deal with this issue better, please let someone who is  responsible for trademark or domain
name contact me as soon as possible.
_____________________________________________________________________________________________
 Confidentiality Notice: This is a letter for confirmation. If the mentioned third party is
 your business partner or distributor in China please DO NOT reply.  We will automatically
 confirm application from your business partner after this audit procedure.we have to notify
 you,and our registration organization are  not responsible for any dispute questions about
 trade mark,intellectual property nor patent after they succeed in registration.hope you can
 understand.thank you.
 ____________________________________________________________________________________________

Sincerely,
  kaka.xu

Sponsoring Registrar:sk holdings company ltd 
 Web:www.sk-dns.org/www.asia-gov.com
 Tel:00852-95660489 / 00852-95660103 
 Fax:00852-30696940

Email:kaka.xu@skdns.org/

Address: 3A, Units 20/F, Far East Consortium Bldg, 121 Des Voeux Road, Central, Hong Kong

kaka.xu

2010.01.21

So, this is obviously a scam, as when I searched the alleged company, I couldn’t find anything. However, the term “International Holdings Ltd.” had produced many scam alerts and related information popped up everywhere. Now, bear in mind that this is the 10th time them past 2 months that I’m receiving such emails. So, I’ve formulated the following response to them, and you are welcome to use it:

Dear Kaka,

Thank you for contacting us in regards to this matter, to be completely frank with you,
we’ve received over the past 2 months a similar request/demand from various Asian registrars
in China/Hong-Kong. Through our contacts in the far-east, we’ve concluded that your
request/demand is fraudulent, and that the company you indicated doesn’t even exist.
Please note that your approach to us claiming that someone wants to infringe our copyright
and brand had been noted and passed to our legal department. In addition, we’ve forwarded your
email and general company information to various SPAM, Abuse and Security teams that are in
contact with us around the world (mainly, [Mention your really BIG business partners and
large customers here - also through in some ISPs in the far-east, specifically China). Should
your company register ANY of the below mentioned domain names or keywords, following this email,
we shall be forced to follow legal actions in accordance to the laws of the state of [Put your
country here] and other countries where our company has representatives or local business
engaged partners.

P.S.

[Always add a personal note - and refer to something in the mail they sent, for example]

On a personal note, when sending emails to anyone in Israel, I would suggest that you choose a
different name, other than Kaka. Kaka in Hebrew is directly related to the bodily function of
purging waste – also known as taking a dump in the toilet.

Tags: Asia, China, Domain name registrar, economy, GreenfieldTech, HongKong, Intellectual property, internet, Israel, Law, Microsoft, Trademark, violations, Voice over Internet Protocol

27 Dec 09 Winsows? Salsa? CheckPoint watching too much Seinfeld

Recently, I had to install the CheckPoint SecureClient on my notebook, which is currently running Windows 7 (ok, a linux guys running Windows 7 is something completely different, but let’s talk about that later). In any case, I’ve gone into the CheckPoint website, looking for SecureClient, and got a really funny Seinfeld flash-back:

This kinda reminded me of this:

Tags: CheckPoint, Linux, Operating system, SecureClient, Seinfeld, Windows 7