The GUI Game

A recent post on the Elastix community page yielded the following:

Guayaquil, Ecuador, May 4th 2015 – PaloSanto Solutions, the company behind the Elastix Project, has established today a GIT repository for the development of a brand and distro agnostic GUI for unified communications servers. Additionally it conceives multi-tenancy.

The project will be independent and will start off with the mutiltenant GUI from Elastix MT. Currently this development is focused on using Asterisk as the telephony manager at its core, however, the project seeks to establish the basis for the inclusion of other projects that currently exist in the industry like FreeSwitch.

This project will be initially hosted and moderated by PaloSanto Solutions; it is an idea together with the PBX in a Flash team.

“We believe that by using Elastix GUI to kick off the project will help in assimilating it faster”, said PBX in a Flash’s CEO, Ward Mundy. “PaloSanto Solutions’ decision to create a GIT repo for development cooperation is also important because it will attract new actors to the project that will enhance its functionality”, he added.

“Since the release of Elastix MT we believed that there is the possibility to continue revolutionizing unified communications, and to establish tools that had not been integrated before”. Said Edgar Landivar, CEO of Elastix. “With the establishment of this repo we hope to establish a standard in the industry that will replace the concept of VoIP PBX”, he concluded.

The project is available today at https://github.com/elastixmt/elastix-mt-gui, and rules are being set regarding contributions to the development so that all interested people can join immediately.

Well, I think that I can understand where Elastix/PIAF are going with this. For a while now, I’ve seen various communications and rants roaming the community, regarding the way the FreePBX GUI is dominating the “Distro Market” – so to speak.

In general, I see this as a good thing, as it means that people will start re-focusing on technology. On the other side, we will end up – again – with the normal Open Source/Commercial debates (should we do? why shouldn’t we do?).

I know that people are going to jump me for what I’m about to say right now, but since the introduction of Asterisk 13 and ARI, the gaps between Asterisk and FreeSWITCH are rapidly closing in my book. I’ve already deployed several systems using ARI – and I don’t use a GUI at all. FreePBX in my book had become a large set of code, assembled partially by people who know what they are doing and partially not. With the Sangoma buy-out, I suspect that we’ll see more and more “Sangoma Centric” modules and features – and that’s normal. Same will apply to the Palo Santo alternative, as it becomes more and more acceptable by the market.

Is it truly the destiny of an Open Source product? to become dominated by their creators? to become a mere vessel of their creators to market their solutions and services. A recent conversation I had with a prospective client went somewhat sour, with me at a loss of answers to a very simple question. The client asked me: “Is there a FreePBX high availability solution? – one that is off the shelf”, my responses were these:

  • You can use the Schmoozecom High Availability solution
  • You can use the Digium High Availability solution
  • You can use the Xorcom High Availability solution

He replied – “What? isn’t there an Open Source alternative? something that had been tested and verified?” – my answer was: “We can always use Linux-HA, Heartbeat, mond and other Open Source tools to create the solution. But it won’t be as slick and neat as the commercial solutions, simply because that requires time.

About 9 years ago, Digium had a product called “Asterisk Business Edition”. The business edition was a highly regressed version of Asterisk, with slightly less features, aimed at being an Enterprise Grade product to work against. Digium realized fairly fast that the product had no place and had abandoned it several years later. Open source is about choice, make your choice, aid others is making a choice – but don’t take our freedom for choice. In my view, the distributions should target themselves to be “Market Aggregators”, not “Market Shapers” or “Market Makers” – this is not the stock market and no wolves den. The Asterisk Exchange is a good idea, if Digium would turn that into something that is embedded into AsteriskNOW and allows people to buy and install, directly from the UI, that would be a winner.

The Apple Store and Google Play became a success not because Apple or Google promoted them, they were made into success by the developers and people who wrote new things. Asterisk, FreeSwitch, Kamailio, OpenSIPS, FreePBX, Asterisk GUI – at least as see it, should be regarded as platforms for innovation, the things that make us go: “Hmmm…. interesting”, the immortal “Facsinating” comment made Mr. Spock.

Just my 0.02$ on the issue…

Source code and individuality

Developers! We are the modern day artists, the masters of the keyboard and the sculptors of algorithms and ideas. We turn obscure thought and imagination into real life creations, capable of doing things previously not done (well, at least not in the same form). As such, we are individuals and unique – each one of us in our own way. Whether we develop a mobile app or a web application, our unique style, way of thought, organization and coding style will be reflected into our creation – we can’t help it, this is who we are.

About 2 years ago I’ve done a project for a start-up company in Israel, where I developed a full blown switching environment for them. I worked on that project for around 9 months and how shall I put it, my name was all over the place. Normally, when I take a piece of code from the OpenSource/PublicDomain, I will document where it came from within the code – then I will add a simple remark next to my modifications.

So, the other day I met one of the new developers working on the project – who didn’t know I was the original developer. And he told me about some issue that he was having with his project – so, in a very natural way, I pointed out to him that the original code wasn’t meant to work like that, specifically, he should into a specific function to resolve the issue and add some additional code to make it work as he wanted. The guy was shocked – “What the hell? are you psychic or something? how can you know that?” – I replied – “Well, I wrote the damn code, I should know”, which followed by me showing him the original source code on my computer. The guy said: “Yes, that is the source code, but all the remarks of the original source code are gone”. Seems that following my departure from the project, someone went into great length in order to remove the various comments I’ve put into the code, to make its origins as unclear as possible.

So, on one hand, I truly understood it – after all, the guy running the show doesn’t want the new people to call up the previous developers and exposing new stuff to them – even if by mistake. On the other hand – Dude, are you really that lame? are you really the afraid that your team will know who wrote the original code?. Source code is a living organism, it is an unique as the person who wrote it and will evolve and change as more people write more code. The Asterisk project still contains remarks that Mark Spencer put back in 2002 – and they are no longer relevant to the existing code, but only to an obscure part of the original code – but it’s still there.

So, to sum up, I never remove remarks that other people wrote from my code – it’s rude, it’s bad practice and worst of all – it’s ugly and disrespectful. Developer will join and leave a project, show your minimal level of respect by respecting their code and their remarks, leave them where they are – removing them is just like performing an act of murder.

Can you say – antitrust?

Version control! one of the most controversial subjects in the software industry. Whether you’re a Subversion fanatic, a Git hard core or a mercurial elitist – everybody has something to say about version control. While in the past we had put our trust in local CVS and SVN repositories, today, most of use utilize cloud based services such as Github, BitBucket or Gitlab.

After spending much time this week setting up our new gitlab repositories – mainly for finished projects that are no longer in active development, and can be removed from our quota at Github, I cam to realize that all these companies are somewhat at a position to be considered as “anti-trusted”. Imagine a hypothetical situation where github starts examining the code we submit to it, not only the public one, but also the private one. Imagine what kind of intellectual property assets they have access to.

In 2001, Tim Robbins portrayed a software giant CEO that is so driven by ambition and greed, that he is actually willing to have developers killed for their code. Where in 2001 developers were very much working in closed quarters and sharing their work via privatized means, today, almost all of us use the clouds in some form. Can they be trusted? What happens if one of them gets bought out by a software giant?

Let us imagine the follow scenario:

The GitGiantCloud (GGC) service has been recently acquired by MegaGreedySoftwareCorp (MGSC). MGSC announces that it will continue to run GGC as always, however, in the background they start analyzing the code within the privatized repositories – completely violating their EULA. Would anyone know about it? the answer is NO. Is it considered a breach? well, they can always excuse it as: “we identified a potential breach, we had to take these measures to investigate it”. In other words, even if they are reading your code – you’ll never know if it’s true or not.

Statisics, Analytics – stop whacking off

Managers! Project Managers, Sales Managers, Marketing Managers, Performance Managers – they are all obsessed! – Why are they obsessed, because we made them that way – it’s our fault! Since the invention of the electronic spreadsheet, managers relied on the same tools for making decisions – charts, tables, graphs – between us, I hate Excel or for that respect, any other “spreadsheet” product. Managers rely on charts to translate the ever complex world we live in, into calculable, simple to understand, dry and boring numbers.
Just to give a rough idea, I have a friend who’s a CEO of a high-tech company in the “social media” sector. He knows how to calculate how much every dollar he spent on Google adwords, is translated back into sales. He is able to tell me exactly how much a new customer cost him and he is very much capable of telling these number just like that. When we last met he asked me: “Say, how do you determine your performance on your network? is there proper and agreed upon metric you use?” – it got me thinking, I’ve been using ASR and ACD for years, but, have we been using it wrong?

So, the question is: What is the proper way of calculating your ASR and ACD? and is MoS a truly reputable measure for assessing your service quality.

Calculating ACD and Why is MoS so biased

ACD stands for Average Call Duration (in most cases), which means that it is the average call duration for answered calls. Normally, an ACD is a factor to determine if the quality of your termination is good – of course, in very much empirical manner only. Normally, if you ask anyone in the industry he will say the following: “If the ACD is over 3.5 minutes, your general quality is good. If your ACD is under 1 minute, your quality is degraded or just shitty. Anything in between, a little hard to say. So, in that respect, MoS comes to the rescue. MoS stands for Mean Opinion Score – in general terms in means, judging from one side of the call, how does that side see the general quality of the line. MoS is presented as a float number, ranging from 0 to 5. Where 0 is the absolute worst quality you can get (to be honest, I’ve never seen anything worse than 3.2) and 5 represents the best quality you can get (again, I’ve never seen anything go above 4.6).

So, this means that if our ACD is anything between 1 minute and 3.5 minutes, we should consult our MoS to see if the quality is ok or not. But here is a tricky question: “Where do you monitor the quality? – the client or the server? the connection into the network? or the connection going out of the network? in other words, too many factors, too many places to check, too much statistical data to analyse – in other words, many graphs, many charts – no real information provided.

If your statistical information isn’t able of providing you with concise information, like: “The ACD in the past 15 minutes to Canada had dropped 15 points and is currently at 1.8 minutes per call – get this sorted!”, then all the graphs you may have are pointless.

Calculating ASR and the Release Cause Forest

While ISDN (Q.931) made the question of understanding your release cause fairly simple, VoIP made the once fairly clear world into a mess. Why is that? Q.931 was very much preset for you at the network layer – SIP makes life easier for the admin to setup his own release causes. For example, I have a friend who says: “I translate all 500 errors from my providers to a 486 error to my customers” – Why would he do that? why in gods name would somebody deliberately make his customers see a falsified view of their termination quality – simple: SLA’s and commitments. If my commitment to a customer would be for a 90% success service level, I would make sure that my release causes to him won’t include 5XX errors that much. A SIP 486 isn’t an error or an issue, the subscriber is simply busy – what can you ask more than that?

As I see it, ASR should be calculated into 3 distinct numbers: SUCCESS, FAILURE and NOS (None Other Specified). NOS is very much similar to the old Q.931 release of “Normal, unspecified” – Release Cause 31. So what goes where exactly?

SUCCESS has only one value in to – ANSWER, or Q.931 Release cause 16 – Normal Call Clearing

FAILURE will include anything in the range of 5XX errors: “Server failure”, “Congestion”, etc.

NOS will include the following: “No Answer”, “Busy (486)”, “Cancel (487)”, “Number not found (404)”, etc

Each one of these should get a proper percentage number. You will be amazed at your results. We’ve implemented such a methodology for several of our customers, who were complaining that all their routes were performing badly. We were amazed to find out that their routes had 40% success, 15% failure and 45% NOS. Are we done? not even close.

The NOS Drill Down

Now, NOS should drilled down – but that analysis should not be part of the general ASR calculation. We should now re-calculate our NOS, according to the following grouping:

“BUSY GROUP” – Will include the number of busy release codes examined

“CANCEL GROUP” – Will include the number of cancelled calls examined

“NOT FOUND” – Will include any situation where the number wasn’t found (short number, ported, wrong dialing code, etc)

“ALL OTHERS” – Anything that doesn’t fall into the above categories

This drill down can rapidly show any of the below scenarios:

  • BUSY GROUP is not proportional – Normally will indicate a large amount of calls to similar destinations on your network. Normally, may indicate one of the following issues:
    • It’s holiday season and many people are on the phone – common
    • You have a large number of call center customers, targeting the same locations – common
    • One of your signalling gateway is being attacked – rare
    • One or more of your termination providers is return the wrong release code – common
  • CANCEL GROUP is not proportional – Normally will indicate a large number of calls are being canceled at the source, either a routed source of a direct source. Normally, may indicate one of the following issues:
    • You have severe latency issues in your network and your PDD (Pre Dial Delay) had increased – rare
    • Your network is under attack, causing a higher PDD – common
    • You have a customer originating the annoying “Missed Call” dialing methodology – common
    • One of your termination providers has False Answer Supervision due to usage of SIM gateways – common when dialing Africa
  • NOT FOUND GROUP is not proportional – Normally will indicate a large number of calls are being rejected by your carriers. Normally, may indicate one of he following issues:
    • One of your call center customers is using a shitty data list to generate calls – common
    • One of your call center customers is trying to phish numbers – common
    • One of your signalling gateways is under attack and you are currently being scanned – common
    • One of your upstream carriers is returning the wrong release code for error 503 – common

So, now the ball is in the hands of the tech teams to investigate the issue and understand the source. The most dangerous issues are the ones where your upstream carrier will change release causes, as these are the most problematic to analyse. If you do find a carrier that does this – just drop them completely, don’t complain, just pay them their dues and walk away. Don’t expect to get your money’s worth out of them, the chances are very slim for that.

 

 

Don’t Replicate – Federate

For many years, the question of high availability had always circled the same old subject of replication – how do we replicate data across nodes? how do we replicate the configuration to stay unified across nodes? Is active-active truly better than active-passive? and most importantly, what happens beyond the two node scenario?

Since the inception of the Linux-HA project (and I do believe it’s been around for years now – over 15 years), it has been the pivotal tool for creating Linux based high-availability clusters. Heartbeat, Stonith and Mon will take care of floating the IP numbers and services across – no biggy there, making sure the data is consistent across the board, that’s something completely different. Recently, one of the better known Asterisk Commercial offerings had launched an Asterisk-HA solution – it’s been long due – it’s just a shame it’s a commercial offering without an Open Source derivative, after all, it is Open Source based (I hope).

However, being a high availability solution on one hand, doesn’t mean you are truly a clustered solution – it is an active-passive solution, with a major caveat (at least as I see it), that if your data sync fails for some reason, you end up with a split-brain issue – and your entire solution is now made moot. Don’t get me wrong here, I think that for now, the solution is the next best thing to sliced bread, simply because there is no other solution out there. However, the fact this is the only solution, doesn’t make it the right solution.

What does federating mean in this respect? it means that data doesn’t need to be replicated across the board, it is automatically trickled across the network, making sure all nodes in the network have clear visibility for it. If a node fails inside the cluster, client automatically redirect themselves to a new node, no need for floating IP numbers. Call routing is automatically determined upon request and are never preset for the entire platform. And most importantly, the amount of data traversed between the nodes is as minimal as possible, preventing excessive usage of network resources and I/O.

What would it mean to federate the configuration of a PBX system? first of all, make sure each unit is capable of working on its own, information should be trickled across the nodes via two methodologies: A multicast/broadcast mechanism (for local LAN connected nodes) and a Published/Subscriber relation (for externally connected nodes). When a change is made to any of the systems, that change is then replicated to all the systems. The configuration is never fully transmitted between nodes (apart from a new node joining the cluster). Routing decisions are dynamically made across the network, they are not predetermined or preconfigured. There is no need to keep the cluster nodes in perfect physical alignment, mixing hardware specifications should be considered the norm. External devices should be able to “speak” to the cluster, without being aware of its existence.

Once we achieve all of the above, we’ll truly get to a point where we’ve clustered Asterisk (or another open source project) the right way.