Managers! Project Managers, Sales Managers, Marketing Managers, Performance Managers – they are all obsessed! – Why are they obsessed, because we made them that way – it’s our fault! Since the invention of the electronic spreadsheet, managers relied on the same tools for making decisions – charts, tables, graphs – between us, I hate Excel or for that respect, any other “spreadsheet” product. Managers rely on charts to translate the ever complex world we live in, into calculable, simple to understand, dry and boring numbers.
Just to give a rough idea, I have a friend who’s a CEO of a high-tech company in the “social media” sector. He knows how to calculate how much every dollar he spent on Google adwords, is translated back into sales. He is able to tell me exactly how much a new customer cost him and he is very much capable of telling these number just like that. When we last met he asked me: “Say, how do you determine your performance on your network? is there proper and agreed upon metric you use?” – it got me thinking, I’ve been using ASR and ACD for years, but, have we been using it wrong?

So, the question is: What is the proper way of calculating your ASR and ACD? and is MoS a truly reputable measure for assessing your service quality.

Calculating ACD and Why is MoS so biased

ACD stands for Average Call Duration (in most cases), which means that it is the average call duration for answered calls. Normally, an ACD is a factor to determine if the quality of your termination is good – of course, in very much empirical manner only. Normally, if you ask anyone in the industry he will say the following: “If the ACD is over 3.5 minutes, your general quality is good. If your ACD is under 1 minute, your quality is degraded or just shitty. Anything in between, a little hard to say. So, in that respect, MoS comes to the rescue. MoS stands for Mean Opinion Score – in general terms in means, judging from one side of the call, how does that side see the general quality of the line. MoS is presented as a float number, ranging from 0 to 5. Where 0 is the absolute worst quality you can get (to be honest, I’ve never seen anything worse than 3.2) and 5 represents the best quality you can get (again, I’ve never seen anything go above 4.6).

So, this means that if our ACD is anything between 1 minute and 3.5 minutes, we should consult our MoS to see if the quality is ok or not. But here is a tricky question: “Where do you monitor the quality? – the client or the server? the connection into the network? or the connection going out of the network? in other words, too many factors, too many places to check, too much statistical data to analyse – in other words, many graphs, many charts – no real information provided.

If your statistical information isn’t able of providing you with concise information, like: “The ACD in the past 15 minutes to Canada had dropped 15 points and is currently at 1.8 minutes per call – get this sorted!”, then all the graphs you may have are pointless.

Calculating ASR and the Release Cause Forest

While ISDN (Q.931) made the question of understanding your release cause fairly simple, VoIP made the once fairly clear world into a mess. Why is that? Q.931 was very much preset for you at the network layer – SIP makes life easier for the admin to setup his own release causes. For example, I have a friend who says: “I translate all 500 errors from my providers to a 486 error to my customers” – Why would he do that? why in gods name would somebody deliberately make his customers see a falsified view of their termination quality – simple: SLA’s and commitments. If my commitment to a customer would be for a 90% success service level, I would make sure that my release causes to him won’t include 5XX errors that much. A SIP 486 isn’t an error or an issue, the subscriber is simply busy – what can you ask more than that?

As I see it, ASR should be calculated into 3 distinct numbers: SUCCESS, FAILURE and NOS (None Other Specified). NOS is very much similar to the old Q.931 release of “Normal, unspecified” – Release Cause 31. So what goes where exactly?

SUCCESS has only one value in to – ANSWER, or Q.931 Release cause 16 – Normal Call Clearing

FAILURE will include anything in the range of 5XX errors: “Server failure”, “Congestion”, etc.

NOS will include the following: “No Answer”, “Busy (486)”, “Cancel (487)”, “Number not found (404)”, etc

Each one of these should get a proper percentage number. You will be amazed at your results. We’ve implemented such a methodology for several of our customers, who were complaining that all their routes were performing badly. We were amazed to find out that their routes had 40% success, 15% failure and 45% NOS. Are we done? not even close.

The NOS Drill Down

Now, NOS should drilled down – but that analysis should not be part of the general ASR calculation. We should now re-calculate our NOS, according to the following grouping:

“BUSY GROUP” – Will include the number of busy release codes examined

“CANCEL GROUP” – Will include the number of cancelled calls examined

“NOT FOUND” – Will include any situation where the number wasn’t found (short number, ported, wrong dialing code, etc)

“ALL OTHERS” – Anything that doesn’t fall into the above categories

This drill down can rapidly show any of the below scenarios:

  • BUSY GROUP is not proportional – Normally will indicate a large amount of calls to similar destinations on your network. Normally, may indicate one of the following issues:
    • It’s holiday season and many people are on the phone – common
    • You have a large number of call center customers, targeting the same locations – common
    • One of your signalling gateway is being attacked – rare
    • One or more of your termination providers is return the wrong release code – common
  • CANCEL GROUP is not proportional – Normally will indicate a large number of calls are being canceled at the source, either a routed source of a direct source. Normally, may indicate one of the following issues:
    • You have severe latency issues in your network and your PDD (Pre Dial Delay) had increased – rare
    • Your network is under attack, causing a higher PDD – common
    • You have a customer originating the annoying “Missed Call” dialing methodology – common
    • One of your termination providers has False Answer Supervision due to usage of SIM gateways – common when dialing Africa
  • NOT FOUND GROUP is not proportional – Normally will indicate a large number of calls are being rejected by your carriers. Normally, may indicate one of he following issues:
    • One of your call center customers is using a shitty data list to generate calls – common
    • One of your call center customers is trying to phish numbers – common
    • One of your signalling gateways is under attack and you are currently being scanned – common
    • One of your upstream carriers is returning the wrong release code for error 503 – common

So, now the ball is in the hands of the tech teams to investigate the issue and understand the source. The most dangerous issues are the ones where your upstream carrier will change release causes, as these are the most problematic to analyse. If you do find a carrier that does this – just drop them completely, don’t complain, just pay them their dues and walk away. Don’t expect to get your money’s worth out of them, the chances are very slim for that.

 

 

As some of you know, over the past 9 months, I’ve been heavily involved in the establishment of Humbug. For those who may not know, Humbug is a Call Analytics and Fraud Analysis SAAS. Now, differing from many of the current telephony SAAS projects, we are not based on Amazon EC2 or some other public cloud infrastructure, we build our own cloud environment. Why do we build our own cloud? simple, we need to keep your data secured and confidential. At Humbug, we see ourselves as a cross between Google Analytics – in our ability to analyze and handle data and Verisign – in our security and confidentiality requirements and methodologies.

Question be asked, why do people trust Verisign to provide SSL certificates around the world. What makes Verisign’s CA better than a privately owned CA – the answer is simple, it’s a third party 2 entities can entrust at the same time. Humbug aims to provide the same lever of trust, simply because we regard your data as sacred and valuable.

Since about 2 months ago, we’ve been contacting various Asterisk integrators around the world, inviting them to evaluate Humbug services. Now, while some integrators and vendors were somewhat reluctant, others were more than happy to join. We now have over 250 monitored systems around the world, with system being monitored and analyzed in Israel, USA, UK, Brazil and more.

The thing that amazed me in regards to some of the integrators who decided not to participate was that they claimed: “we provide our customers our own brew of fraud analysis service, we don’t require your SAAS”. Now, while I can accept the fact that an integrator would offer such a SAAS as an in-house service, I can’t see why a customer would rely on these services. In my view, relying on your integrator to provide fraud analysis services is like relying on the integrator of your alarm system to provide hired guard services – it just doesn’t make any sense to me. Why doesn’t it make sense? in Hebrew we say: “Go prove that you have a sister”. Imagine that your PBX integrator offer you such a service, then, in some obscure manner, your PBX gets hijacked and you get slammed with 50K$ worth of phone calls to Somalia. Now, your integrator would say: “Hmmmmm… that’s odd, we didn’t even get those CDR events to our system… you really got hacked bad…” – sure, if you only rely on CDR records to do your analysis (which is what 99.9% of integrators do). There is much much much much more to fraud analysis than just CDR analysis – if it all began and finished with CDR analysis, then by far Cvidya, Verint, NICE and many others would have been made redundant.

Allowing your integrator to provide you with fraud analysis SAAS is like putting the fox to guard the hen house, when things louse up (and they may), he’s the first one to bail out saying: “It’s not my fault”.

Humbug takes a totally different approach to fraud analysis, specifically, in the way we regards the various PBX systems and integrators. We are vendor agnostic and integrator agnostic – we will provide you with the clear and concise information you require in order to make an educated decision as to how you were de-frauded (if de-frauded) and provide you a faster alerting and response time. Our recent adventures had lowered our fraud alert response time from 60 minutes, down to 14 minutes in some cases. Most fraud analysis system carry a 24-36 hour turn around time, by that time, you can be out of 50K$ – our aim is to lower that number to no more than a 100$ in the worst case. Ambitious? yes, down right crazy? probably so, but we always say: “Aim for the moon, you’ll land on a star!” – so we know we’ll get there.

The following post doesn’t really fit in line with the normal spirit of the blog, simply because it’s not funny nor directly related to technology. It’s called Business 2.0, as it relates to the ever problematic question any business owner has: “When should I grow and how?”.

As you may know, I’ve been a freelance Asterisk Platform developer since early 2003, turned to freelance development (Penguin for hire) around April 2007. Since that time, I’ve built systems and platforms for some of the better known brands around the world. Be it working directly with the customer or through a 3rd party (as a sub contractor) – I can easily say that I’ve completed over 120 different large scale projects within 3 years time. Now, when I refer to projects, I’m not referring to installing PBX systems, I don’t do that at all – I’m referring to highly complex application level development, creating some of the most innovative Asterisk based systems I’ve ever seen.

Image representing Jajah as depicted in CrunchBase

Image via CrunchBase

Vodafone

Image via Wikipedia

Putting aside everything, finalizing a rough estimate of 40 development projects on a yearly base, most of these performed solely by myself is a fairly challenging task. Sure, at times I’ll outsource some work to other freelancers like myself, specifically in fields where I’m not all that fluent (Database, Web Development, UI) – but yet, doing that means that I’m conducting 3 – 5 projects on a monthly basis.

After doing so for 3 years now, I can’t help but start thinking about expanding my business, taking it to the next level by hiring more people and building it up to a new level. Question remains for this: “How? What is the natural track of expanding your business?” – of course the simple answer would be: “Just hire another developer or two, and start doing more sales” – it’s not as simple as it sounds. After thinking about it for some time, I’ve concluded there are a few models of expansion:

Model 1: Organic Growth

Organic growth can be described as the simplest way of growth: “Hire a new guy and get more work in”. The problem with this model that it is fully reliant on your ability to sell more. However, as you concentrate on sales more, you take time from the development and delivery process – thus, the addition of the new developer is not a 100% addition, it’s actually 100% (developer) minus 40% (you) – so you are not at 200% capacity, you are 160% capacity. Surely 160 is 100, however, for the initial 6 months, till the guy learns the ropes, you are not at 160, you are actually at 80 – can you and your business sustain that?

Thus, the main issue with Organic growth is cash flow, can your business sustain the elevated expenses with less income for the period of transition? If the answer is NO, then you need a different method. If the answer is YES, then you are in the best place in the world, however, bear in mind that taking someone to work for you is a responsibility – people are not resources, they are human beings, with families and children – taking someone to work for you is like taking responsibility for their lives.

Model 2: The Partner

Panama Business and Investment
Image by thinkpanama via Flickr

A partnership with a person who is equally matched to you is always a good option. Technically speaking, it means that you are teaming with someone who generates as much work as you do and is capable of finalizing the work as fast and as good as you can. Yet, taking a partner doesn’t negate the requirement for a new employee or two. In this case, you may end up with too much sales with too little staff to deliver – that is a big problem.

Another issue with partners is the issue of trust. While most partners tend to rely on each other and trust each other, that trust can easily be broken (in most cases by stupid things). It’s enough for one partner to now carry its weight in sales/development to initiate a chain reaction, shortly ending in the partnership dissolving.

So, the partner is a good option, however, may prove to be problematic if the wrong partner is chosen – in addition, dissolving a partnership solely on these issues isn’t all that simple – and usually ends up in litigation and other judicial issues – YUCK!

Model 3: Un-intrusive Angel

Some people ragard Un-intrusive Angels as “Stupid Money” – an Angel investor that doesn’t interfere in your company business model and operations. In many cases, this is how start-up companies start – someone gives them a lump sum of money to start their business, signing off to own a portion of the new company.

An un-intrusive investor usually gives you the money and pays you a visit once every few months to see how his money is spent. Don’t expect to raise a whole lot from these people, usually you will get anything from 25K$ to around 250K$ – tops. If you are getting an investment from an Angel, make sure you plan your business carefully – and make sure your investor knows what he is getting into. The Angel is not a found piggy bank, he is a business man looking for profit – if you make sure his expectations of profits (time frame, amount, percentage, etc) are kept within the reason of your business – he will make an educated decision and invest accordingly. Promises like: “you’ll double your money in 3 years” are stupid – make sure it’s realistic and to the point. If you promise the moon, and reach a star – that’s a problem, if you promise the skies and hit a start – that’s wonderful.

Model 4: The Strategic-Intrusive Angel

Jeff pulver

Image by TheFemGeek via Flickr

A strategic angel is similar to the previous one in terms of funds, however, he is more capable in assisting your business meet its goals. Usually, it would be someone who is already a well established figure in your business sector, had made his money from previous companies and is now looking for new ideas and businesses. I call him an intrusive Angel, as sometimes he may have ideas as to where your business should go – and he will make sure you hear his ideas. You may regard it as annoying, but you should still listen to your Angel and pay him the respect he deserves.

Sometimes this Angel may invest in your business due to the fact that he has a hidden agenda. An agenda can be: The angel looks at your business and see a certain potential you are not planning, he’ll invest and try to re-direct your company to the agenda he sees. This is usually the case when your angel is invested into several endeavours that is either parallel to each other or may have orthogonal intersection points. These angels can be the builders of your business or the destroyers, it is up to you to make sure the latter doesn’t happen.

Prolog:

So, which model did I choose? – I didn’t choose yet, I’m still figuring it out myself. What ever the model may be, the choice isn’t simple nor straight forward. At best, whatever choice I’ll take will have a profound impact on my business and me – so I’ll need to weigh my options carefully. If you can think of an additional model, I’d love to hear about it – so just comment on this post.

Reblog this post [with Zemanta]

When I last reviewed the PIKA WARP Asterisk appliance, I  named the post “Pokemon Asterisk” – today I’ve decided to review the PIKA WARP Asterisk appliance again, only this time, with the newly released Asterisk GUI 2.0 release – our cuddly Pikachu is now a Raichu (relax, it took me about 30 minutes to find out what a Picachu evolves into).

The new PIKA appliance now boasts the new star fangled Digium Asterisk GUI 2.0, which takes the old Asterisk GUI (which was OK, but still had some miles to go) and more or less throws it into the waste bin. The new GUI is far more useful, far more usable and most importantly – makes life way easier for the integrator. While the previous version of the PIKA Warp appliance was targeted for developers, the new version of the WARP is aimed directly into the heart of the integration scene.

Asterisk GUI 2.0 on PIKA WARP

Asterisk GUI 2.0 on PIKA WARP

Now, I have to admit that after upgrading the system to the new PIKA WARP cuImage I had some issues logging into the system. So, what I did is more or less hack myself in via ‘single user mode’. Here’s a small guide on how to do that. Before we being, you will require a serial cable connection to the WARP appliance in order to do this, which means, this is more or less a hardcore procedure.

The PIKA WARP Serial Connector Port

When the system boots up, and you are confronted with a message saying “Hit any key to stop autoboot:” simply hit any key on your keyboard, and you’ll be fronted with the “=>” prompt, indicating that the boot loader is now waiting for information. Now, we need to tell the PIKA WARP appliance to boot into single user mode.

To do so, we need to modify the ‘ramargs’ environment variable of UBOOT, to indicate that we want to start single user mode. Enter the following command:

setenv ramargs setenv bootargs root=/dev/ram rw ramdisk_size=130000 single

This will indicate to the UBOOT loader to initiate a single user mode bootup. Once in single user mode, you can use the ‘passwd’ command to change the root password of the PIKA WARP appliance. This procedure can be used by an other PIKA WARP based appliance.

Once of the nice additions in the new Asterisk GUI 2.0 is the support for Class of Service, which doesn’t really exist in FreePBX. In many offices, managers like to restrict various extensions from accessing different parts of the telephony system – that is performed utilizing the Class of Service screen.

Class of Service management

Class of Service management

The “Class of Service” management enables you to create groups with access to specific trunks or PBX functions, thus, enabling you to seperate users and groups of users from specific PBX resources. For example, some users can be completely restricted from using outbound trunks, while others can be restricted to using a single FXO interface out of 4 connected FXO interfaces. In general, this is one of the best features in the GUI yet in my opinion.

I’m currently reviewing the new version, so once I have new information I’ll post my findings.

As I wrote a couple of days, I’m currently in the process of evaluating the PIKA WARP appliance. As I already said before, the PIKA WARP isn’t a real PBX, but actually a framework for building PBX type appliances.

For me, the entire evaluation process is more or less a process of trial and error, trying to figure out how the box is built, both from the software side and both from the build-tools provided with the appliance.

The process of flashing the new FreePBX based images wasn’t much of a hassle, actually, after more or less fucking up the box a couple of times, I became fairly handy with both the warploader and uboot tools, used to flash the onboard flash memory with a new boot image. After flashing the FreePBX about 3 times, I got it to come up right on my browser window, which produced the following screen:

PIKA FreePBX Screen

PIKA FreePBX Screen

Now, it was fairly clear to me that MySQL (a mandatory component of FreePBX) isn’t a shoe in for the WARP appliance, after all, for an appliance it would be a blowtware. So, FreePBX here comes with SQLite, which on the surface should provide for similar functionality. Having worked with SQLite in the past, I knew for fact that some portions of FreePBX will not work – who am I kidding? if they would work it would be the most amazing thing I’ve seen a long time.

Mainly, this is caused by variations between the SQL code and various table management that FreePBX imposes on the FreePBX database structure, when installing or upgrading new or existing modules. For example, trying to go about and install the FreePBX IVR module yields the following error:

FreePBX error on WARP appliance

FreePBX error on WARP appliance

Well, that’s understandable – basically means that the unit isn’t yet fully useable with FreePBX – but we are getting there I guess, after all, rome wasn’t built in a single day.