Image via Wikipedia

When I started using Open Source software, it seemed like all Open Source projects are driven by philanthropic agendas. We were all focused on “sticking it to the man” – showing all these would be software vendors that community driven projects can do just as well – if not better.

"When I was a child I spoke as a child I
understood as a child I thought as a child;
but when I became a man I put away childish
things." - I Cor. xiii. 11.

Well, I’m not claiming that Open Source is childish – absolutely not, however, when you are a student you tend to look at things in one way, when you have a family to care for – you start looking at things differently. You remember these days in life when your dad said: “When you’ll have children you will understand” – well, now I do.

So, what am I rambling about exactly? I’ll tell you. The day before Passover I attended several meetings, which when I came back home had pissed me off immensely. I feel an urge to write all about these meetings, including who I met exactly, however – I won’t do that. However, I will give a rough idea of these.

Meeting 1 : A world recognized Mobile application player

I came into the meeting with this company, where the CTO of the company explained to me that they are looking to create an Asterisk based solution for their application’s users. My initial question was: how many users? what is your concurrency level? – The answer that I got was: “Oh, we don’t need something major, just a few lines of configurations in Asterisk config files in order to make this work”.

I left the meeting slightly pissed off, thinking to myself: “You bloody inconsiderate prick! You bring me to a meeting, spend my time – and then telling me that this is just a few lines of configuration. If it is that simple, why don’t you do it yourself? you have 20 developers in there, 4 IT people and god knows how many outsourced workers off-shore – if it was that simple, you would have done it already – so probably it isn’t – right?”

Meeting 2 : A well established IVR services vendor

The second meeting was with a well established IVR content vendor, this company runs around 16M minutes of inbound IVR traffic every month. They invited me in order to talk about expanding into new countries, wishing to get premium based access numbers in various countries. So, we started talking, and the guy indicates that he wants a certain kick-back payout, which I know is impossible – at least without charging the user more. Actually, the guy indicated that out of the interconnect fee, he wants to get almost 90% as a kick back.

Meeting 3 : A start up rendering IVR content

The third meeting was the most amazing one – these guys wanted to build an Asterisk system to server around 4000 concurrent channels – outsource the entire development to my company – and pay as a revenue share. When I asked for their business model, marketing plan, investors, profiles – I got a response of – we don’t yet have all of these, we only have an idea at this point that we want to implement.

Garage based companies are built by people who can do the work themselves, not the other way around.

At this point, you are probably asking yourself: “What does this have to do with the title?” – Well, all of these meetings had one thing in common. The people I met were under the impression that Open Source is some form of philanthropy. Or to be more exact, people who deal with the Open Source market are philanthropists. My question is this: “Why are we perceived as philanthropists? don’t we have families to care for? don’t we need to pay mortgages and bills just like everybody else?”. I guess when people read about the various Open Source entrepreneurs, such as Mark Shuttleworth – the immediately associate Open Source with Big Exists – this is not the case.

At some level, this is purely our fault – we educated people that Open Source is a highly economical methodology of solving technical challenges. No where along the way, had we educated the public that behind the model there are people, people who need to make a living.

If you are an Open Source consultant, developer, evangelist or just someone who may have an opinion on this, I’d love to read what you say.

Well, it’s been almost a month since I’ve started writing about the humbug project. Now, it’s time to actually get you people involved, at least in the initial levels. We are looking to add 10 additional members into the humbug call analytics suite. Currently available analytics during the alpha testing is inbound call analytics.

Our aim is to gather as much information as we can and as much user requests as we can, humbug is a community oriented project, thus it relies on community oriented input and feature requests. Participating members will  be granted access to the humbug analytics portal, allowing them to gather statistical information regarding their inbound call hits and their top ten DID numbers – we are working on additional statistics. As new stats will become available, we’ll role those out into the service as soon as possible.

In order to participate in the closed alpha testing, please send an email to alphatest at humbuglabs.org, and we’ll send you a short piece of dialplan code to insert into your Asterisk server. Technically speaking, we’ll send you a short AGI command that looks like this:

exten => _X.,n,AGI(agi://somehost/DataReceiver,some_unique_ident)

The above line needs to be inserted into any place you would like to generate call analytics from. We’ll also enclose configuration steps for FreePBX (and other FreePBX compatible distributions). We are hard at work for creating a FreePBX integrated module, so you can do a one-click install.

I recently gave a presentation at the Amoocon convention, held in Rostock, Germany – about Asterisk and Amazon EC2. Below is a medium quality video of that presentation:

or you may download it here:
Amazon EC2 and Asterisk video files

Udim, Israel. April 1, 2009 -GreenfieldTech Ltd., a leading provider of Asterisk solutions and Asterisk training services in Israel, today announced the availability of it’s patented app_cashmaker application for the Asterisk Open Source PBX system. The CashMaker application is intended to be used by various content suppliers, wishing to distribute Audio and Video based content, utilizing their Asterisk server.

The application is built to accept an inbound call into it, then, according to various information gathered in correlation to the callers caller ID and/or inbound DID number, will correlate a relevant content stream directly to the caller. The content distributor doesn’t even have to care about what content to distribute, as the application will connect directly, via the Internet, to a remotely available RTBSP streaming server at GreenfieldTech data center.

“The app_cashmaker application is the result of the cumulative work of over 3 years in the making, testing various content business models and applications. The main problems most content distributors have is how to gather the content and manage it, with app_cashmaker, this requirement is negated, thus allowing the distributor to concentrate on what they do best – flooding the newpapers with ads and marketing material to promote their content delivery service”, says Nir Simionovich, CEO and Founder of GreenfieldTech.

Simionovich indicated that the central content distribution facility is managed via a GTBS cluster environment, implemented partially utilizing Amazon’s EC2 and S3 structures, while utilizing GreenfieldTech’s proprietary streaming and clustering technologies. Currently, GreenfieldTech had submitted 10 different provisional patents, relating to the technologies comprising the app_cashmaker application and service. GreenfieldTech marketing team had indicated that initial beta trials had showed an increase in content availability, via the GreenfieldTech BSC Cloud facilityof over 40% with an increase of almost 80% in content delivery success.

Simionovich estimates that by the year 2010, over 20,000,000 will use the GreenfieldTech app_cashmaker facility, disrupting completely the way mobile, audio and video content is distributed around the world.

Asterisk is the world’s leading open source PBX telephony engine, and telephony applications solution. It offers unmatched flexibility in a world previously dominated by expensive proprietary communications systems. The Asterisk solution offers a rich and flexible voice infrastructure that integrates seamlessly with both traditional and advanced VoIP telephony systems. For more information on Asterisk visit http://www.asterisk.org 

For more information, please refer to the GreenfieldTech website at http://www.greenfieldtech.net.

Well, it’s finally out – my new book that is

Some of you already know, but over the past year I’ve been busy writing a new book. This time it’s a book for Asterisk developers, especially tailored to PHP developers wishing to utilizing the PHPAGI framework. The book is out from Packt Publishing (Like my old AsteriskNOW book) and is updated with all the recent changes in Asterisk – including version 1.6.X and DAHDI.

Asterisk Developers Book

If you like my work with Asterisk and would like to read more of my work, go ahead and get an electronic version of this book. I know it’s a little self promoting, by hey, it never hurts does it?

I’ve included a chapter on how to build a complete project from scratch, detailing the various analysis steps and various paradigms required to develop a fully functional Asterisk based application. I believe that even experienced Asterisk developers will benefit from this book.

While this post is titled “Thoughts of Virtualizaiton”, the applications described can be easily applied to non-VM type installations.

Virtualization is a wonderful tool, it enables rapid growth and rapid deployment of new servers and services. However, just like any other platform that tends to grow across the time line, it poses the same annoying issue of managing a large system, especially when dealing with Asterisk based installations.

Let us imagine the following scenario: A Calling Card company while utilizes 8 different Asterisk application servers, are utilizing a single Database servers cluster and are receiving inbound calls from various sources and load balanced across all Asterisk application servers. What I’ve described above is more or less the practice most (if not) all calling card operators deploy. No matter if the usage is A2Billing, MOR, ASTBILL – the methodology is more or less the same.

One of the bigger issues with such an installation is debugging of a running session, more over, the ability to debug a session after it is finished. This situation is caused by a simple, yet annoying issue, we are operating within a “zero-knowledge” system, where we have no precognition of where a specific call will be handled utilizing our cluster. Now, if you are an experienced sysadmin, you would most probably do the following:

1. SSH to all your Asterisk servers.
2. Tile all consoles on your desktop.
3. Start the test – and hope your eyes are fast enough to capture the right gateway.

Well, this is the normal practice with most people – but I have to admit it’s kind of annoying. Now, let’s imagine that we are now building our system from scratch, we’re not using A2Billing or any of the other Open Source products, we simply build our own application framework. So, what do we need to do in order to keep track of our system correctly?

Step 1: Consolidate

Consolidate the messages coming from each of your gateways to a single logging facility. The best track would be to utilize some form of Syslog facility. For example, all the scripts and network services that I develop utilize a clear and concise interface to syslog. I usually re-direct the syslog facility that I use to an external server, thus, I get all the logs on a single syslog file system.

If you are worried about I/O issues on the syslog server, you can always create a “syslog-proxy” using tools such as memcached or others.

Step 2: Identify

Your syslog write function should always include a prefix indicating the name of the generating Asterisk server. For example, have something like the following prefix your syslog entry:

Dec 14 21:51:32 pbx [PBX01/6d6d6423a2244aa71980e5a5b437919e/check_pincode[22537]: agiParameters: check_pincode

While the syslog facility will include your generating hostname, when duplication VM’s, this would be a really good practice.

Step 3: Analyse

Once your logs are consolidated to a single environment, it should be fairly simple for you to go about and analyse these in a pre-defined routine. There is little to gain from analysing the logs on-the-fly, but analysing it every 5/10/15 minutes will prove worthwhile.

Step 4: Audit

Auditing is good – as long as you keep a clear view of what you audit and what you don’t. Audit key points in your application to a database can save you a whole lot of time of debugging – just make sure your audit is clear.

Keep the above in mind and you should be just fine creating any scale of platform.

As most of you reading this blog, you must know that I’ve published an AsteriskNOW book early this year. I’m happy to say that I’ve received numerous emails since the publication of the book, all asking various questions about Asterisk in general and AsteriskNOW in particular.

Most of the questions that I received were related to the development of AGI and AMI scripts, and how to utilize Asterisk as an application layer platform. This inspired me to work on additonal title, to complete my previous book. I’m currently working on an Asterisk Developers guide, for AGI/CTI development with Asterisk. The book serves as a complete guide, summarizing the various aspects of developing AGI/CTI applications with Asterisk, while, enjoying various additions from my day-to-day experience with AGI/CTI development with Asterisk. The book is written as an eye-opener for experienced developers, wishing to make their transition to AGI/CTI development, and teaches them how to avoid the most common mistakes of early day AGI developers.

I’ll be be updating the blog with my progress. Currently, chapters 1 through 5 are complete, covering the basic aspects of dial-plan and AGI development. More information will be released soon.

Lately I’ve come to the realization, that we are to blame for our own inability to promote Open Source and the adaptation of Open Source proficiency. Being an Open Source evangelist and consultant, this is very weird to be said by one like myself, however, this is my realization – and I will explain.

In the early days of Open Source adaptations (late 90′s, early 2000), Open Source software was a somewhat magical solution that meant: pay nothing, get more. Software packages like Linux, Apache, mySQL, PostgreSQL and programming languages like PERL and PHP had lowered the bar on the adaptation of new technologies, and enabled a prolific number of solutions and services.

I still remember the early days, when a Windows based Mail Relay would cost anything between 800$ to 1200$, and I would come in with a Linux based solution that would do the same thing for FREE – amazing. As time progressed, so did the technology and the penetration of Open Source into new fields. CRM, ERP, Telecoms, management – all of these now enjoy a diverse number of Open Source solutions. However, the original concept of ‘Open Source = Magical FREE Solution’ had still remained in the minds of managers and business people.

Today we are confronted with ‘would-be’ Open Source solution experts, which adopt and develop upon Open Source products and project various applications. In example, let’s take a look at Asterisk. Asterisk has a multitude of Open Source solutions, ranging from PBX system, Prepaid calling cards, Wholesale routing platforms, Attendance system, Presence systems – and even a plant watering solution. The problem with this ever growing number of solutions is that Asterisk is immediately considered to be: “A magical solution” capable of solving any problem – when it’s not even remotely related to Asterisk. For example, a friend of mine had been asked to develop an Asterisk based solution, that would support a total of 250 concurrent call initiations and up-to 3000 concurrent calls on the system. Any Asterisk developer would take a look at this, and would immediately say: “Hmmm…. this requires several servers, but hey, what about the application itself? that would also have an impact”. Now, the customer of the project has a ‘would-be’ Asterisk tech in his company which said: “I was able to initiate 200 concurrent SIP invites to Asterisk via SIPP, no problem’ – HELLO! STUPID! where’s the application? where’s the database? where’s the user information flow? comm’on, are you listening to yourself speak? or simply are filled with the gasses coming out of your ass that are affecting your brain?

Now, once the customer learns that Asterisk is most probably not the right solution for the problem, he becomes angry. Why? because he now learns that he needs to spend about 10 times more money than he anticipated for the creation of this tool – well, that’s life when you have no idea what you are doing/saying, and you believe in magical solutions. However, we – “The Open Source Community – is the one to blame for this scenario, because we got the world accustomed to the idea that Open Source is like magic – flip the Linux magic wand, and the rest will solve itself.

I’d like to open the floor for discussion on this, as I believe most of you will have something to say about this.

As most of you already know, I’m heavily involved within the Asterisk Open PBX project. Over the course of the past 5 years of my dealing with Asterisk, Asterisk had always suffered a serious flaw, and that is, a single-threaded Manager interface – which usually led to serious dead-locks when writing a multi-threaded server that connects to it.

One of my long time challenges was to surpass the 4-5 originate requests to the Asterisk Manager interface, enabling me to automatically dial more than 4-5 calls at the same second. My initial work had began with the idea of increasing that by a factor of 50%, going up to around 7-8 calls per second – I had achieved that using a combination of smart synchronization between the manager interface and my originating server – and also enabling asynchronous originate requests – however, that methodology had proved to be problematic – in terms of reliability.

I understood that something else had to be devised, something that doesn’t rely completely on the manager interface, and that will allow me to originate calls freely, without clogging up the manager interface. So, I decided to move my interest from the Manager interface, and concentrate on understanding Asterisk’s channel handling, especially, how do calls originating from the manager interface are handled by the Asterisk spooler and the Asterisk channel drivers.

more will follow…

OK, if there is one thing I really hate, is that when something so little as a small configuration change from one version to another causes things not to function properly. It is one thing to make sure that your code is backward compatible, no one really expects that your scripting language will suddenly start behaving differently, just because you upgraded to a new minor version – right?

So, here is the case, over the course of the past 2 years, I’ve done some extensive AGI programming with the PHP programming language – hell, I think that if I’ll sit down and write a book about AGI programming with PHP, it would be one of the most complete ones. I remember that when I started writing AGI scripts, I used to simply write everything by myself, each script was a self contained unit, everything was working hunky-dory. Shortly later, I discovered the PHPAGI class, a nicely wrapped AGI class for AGI, FastAGI and Asterisk Manager interaction, that made my life really easy. When Asterisk evolved to version 1.2, most of my PHPAGI scripts kept on working, apart from a small part of the Manager interface, however, that was easily worked around – and we’re back hunky dory. Even Asterisk 1.4 and Asterisk TRUNK liked PHPAGI, no problem there – till a week ago.

So here I am, working on a new project, and I’m working on my new snazzy development kit, that is installed according to the following:

CentOS 5.0 x86_64 with Kernel 2.6.18-8.1.4.el5
Asterisk stable 1.4.14
php version 5.1.6 (RPM package: php-5.1.6-12.el5)

At this point, all hell broke loss! All of my old PHPAGI scripts had simply gone crazy on me, causing each of the AGI “GET VARIABLE” commands to fail, always returning variables which are not there at all. That was fairly annoying, I’m issuing a command, getting an error response, issue the same command again, get a proper response only with the wrong variable, issue the command again – get the proper value. How annoying! – All these scripts used to work perfectly in the past, what happened? So, I deploy an older code of mine, guess what, that code doesn’t work either – at this point, I start pulling hairs off my head (not that I have much to pull on, my hair is very short).

So, what does any good Open Source programmer do when confronted with such a problem? “Use the source Luke, use the source!” – lets dig into the Asterisk code. Adding some AST_LOG functions here and there, doing some more debugging using valgrind, everything within Asterisk looks nice. Just to make sure, lets downgrade the Asterisk version and run the test again – same results. Conclusion: Asterisk isn’t the problem, doesn’t really matter what version I use.

OK, lets check the PHP scripts – some PHPAGI class hacking, more debug here, mode debug there, a few more syslog calls – everything looks on the PHPAGI script as if the input from Asterisk is messed up. Conclusion: something in the middle is fucking up royally!

Now, being a good programmer I said to myself: “OK, so one said that PHPAGI is the only language out there, lets go Python – besides, I always wanted to learn Python”. So I spend a day learning some Python, just so I would be able to re-write my script into Python. Ran the script – holly shit! – it works, as if nothing happened. Conclusion: something in the PHP environment is screwing up the environment. So, I decide to modify my execution from using php-cli to php-cgi, for all practical matters (at least from the script side) – it should behave identically. Modified it to work with php-cgi, suddenly the $argv variable isn’t being passed to the script! – Hmmmm…. that’s odd, it was there a minute ago. Lets dig into php.ini and see what’s wrong:

; This directive tells PHP whether to declare the argv&argc variables (that
; would contain the GET information). If you don’t use these variables, you
; should turn it off for increased performance.
register_argc_argv = Off

Dear god, why would $argv and $argc be turned off? just for performance sake? how would I run the AGI scripts that I want to run, after all, I am using the $argv variable. Turn the parameter back to “On” – JOY! Everything is back working as it should. So the question that remains is: “What is the main difference between php-cli and php-cgi that had caused this issue?” According to the voip-info.org website, there is some form of difference, however, it doesn’t really say how it is exhibited in the operation of AGI scripts.

Well, at least I solved my problem for the mean while, I’ll work on getting the proper explanation of php-cli vs. php-cgi later on.