I recently gave a presentation at the Amoocon convention, held in Rostock, Germany – about Asterisk and Amazon EC2. Below is a medium quality video of that presentation:

or you may download it here:
Amazon EC2 and Asterisk video files

Well, it’s finally out – my new book that is

Some of you already know, but over the past year I’ve been busy writing a new book. This time it’s a book for Asterisk developers, especially tailored to PHP developers wishing to utilizing the PHPAGI framework. The book is out from Packt Publishing (Like my old AsteriskNOW book) and is updated with all the recent changes in Asterisk – including version 1.6.X and DAHDI.

Asterisk Developers Book

If you like my work with Asterisk and would like to read more of my work, go ahead and get an electronic version of this book. I know it’s a little self promoting, by hey, it never hurts does it?

I’ve included a chapter on how to build a complete project from scratch, detailing the various analysis steps and various paradigms required to develop a fully functional Asterisk based application. I believe that even experienced Asterisk developers will benefit from this book.

The Open Source movement had been in existence since the 60′s, and we can surely find its roots somewhere along the hippie culture and movement. While Free-Love had transcended to Free-Code, or to be more exact – Free-Knowledge, the question of the sources for your Open Source is still questionable. Comparing it with the Sixties, it’s easy to compare the various “Free-Love” movements with the various “Open Source Paradigms” of today. While GPL, BSD, MPL, ZPL and others preach for Open Source adaptation – each one took a different path.

While the paths differ, but the end result is more or less the same, all suffer from a serious lack – a bad reputation. While in the early 2000, Open Source usually meant – highly stable, state of the art technology, increased ROI, lowered TCO and most importantly for many – COOL. Coming 2008, Open Source is starting to get a bad rep, due to the ever increasing simplicity of entering the Open Source world.

I started using Linux somewhere around 1994. My first Linux distribution was a Slackware, with a kernel of 1.0.28 – I needed 99 floppy disks in order to install the system, and it took me a few hours to do so. However, I can’t forget my amazement at seeing the X-Windows environment booting up, and more than that, being completely overwhelmed with the fact that I have a fully functional UNIX environment in my house, just like the one I had in my Army office. Now, I basically had no one to teach me this new environment, so, I had to take my UNIX skills (Solaris and AIX) and adopt to Slackware Linux – it took me a few weeks to get around, but I got around and stuck to it ever since.

Now, let’s jump 14 years forward in time. The year is 2008, a graphic based environment for Linux is no longer a myth and it is getting better and better by the day. People are starting to adopt Linux beyond the academic and the ISP market sectors, slowly integrating Linux based distributions (Mandriva, Ubutnu) on to their desktops and notebooks. Linux is become simple and appealing to everybody.

When something becomes easy to use, people make good use of it – a good example is the Asterisk project. Projects such as TrixBox (AKA: AsteriskAtHome), PBXinaFlash, AsteriskNOW and others had made Asterisk into a simple installation product, that can be installed and managed by any half-decent sysadmin. Problem is, while a half-decent sysadmin will do a fair job of maintaining the system, a shitty sysadmin will crap everything to hell. But hell, that is true for almost anything related to computers or technology – there’s nothing new here! Well, there is nothing new and everything is now new. People who were more or less selling shoes 3 years, then 2 years decided to sell ISP routers, then a year ago started selling IP phones, are now selling Asterisk based systems – using these distibutions, while having no idea what they are selling or promoting. For these people, Asterisk is nothing more beyond FreePBX – once encountering deeper issues, will simply abandon the customer – leaving the Open Source product with a bad rap with the, now disappointed, customer.

I want to believe that other places in the world are different, I want to believe that Israel will reach a point in time when this doesn’t happen – however, I guess that only time will tell and I surely hope this will change in Israel.

Ok, it goes without saying: “A2Billing is one of the most complete Calling Card systems in the Asterisk market today.” – on the other hand, it is also true that: “A2Billing is one of the most complex and convoluted pieces of code ever written!”.

The combination of the above makes for a fairly combustible mixture, especially if you have a big system. Now, I recently ran into an issue, where PHP was litterally eating up almost 512MB of ram, in order to run the A2Billing reports. In it self, that didn’t make much sense to me. However, after inspecting the code, and realizing that A2Billing uses GD in run-time to generate images out of thousands of CDR records, it made perfect sense that it may just be eating up memory.

So, increasing the memory on PHP to go up to 512MB of RAM helps, but creates an interesting probelm. Whenever Apache will invoke a script, it will automatically consume a shitload of RAM, and for each time I intiate a new query, it will spawn a new Apache instance, and consume the same amount of memory. That said, after 6 queries of 512MB, about 50% of the machines RAM was already eaten up – and Apache will not free it!

At this point, I had 2 choices:

1. Go into the A2Billing code, change the GD code to work right or simply change it completely to something else (maybe flash).
2. Work around the problem with a mix of proper IT practices.

I admit that I hate quite a lot of things (I won’t list these here); however; nothing ranks up the list as modifying someone elses code, when I know for fact that it will be unmaintainable in the future. So, I choose option number 2.

I’ve being playing alot with Lighttpd lately, and got some really nice performance from it. So, I said to myself, this would be a great test to see if Lighttpd+FastCGI can solve the problem here. I had to work my way around lighttpd to do what I wanted and verify that my FastCGI server in Lighttpd doesn’t consume all memory, however, here is what I got working with A2Billing, and really nice.

Step 1: Enable the required modules:

server.modules              = (

                               "mod_access",

                               "mod_auth",

                               "mod_status",

                               "mod_fastcgi",

                               "mod_accesslog" )

Step 2: Enable the FastCGI Server

fastcgi.server             = ( ".php" =>

                               ( "localhost" =>

                                 (

                                   "socket" => "/var/run/lighttpd/php-fastcgi.socket",

                                   "bin-path" => "/usr/bin/php-cgi",

                                   "idle-timeout" => 30,

                                   "max-procs" => 1,

                                   "min-procs" => 1

                                 )

                               )

                            )

Step 3: Modify user permissions (required if you are using FreePBX)

server.username            = "asterisk"

server.groupname           = "asterisk"

Step 4: Setup authentication and authorization (optional)

#### auth module

## read authentication.txt for more info

auth.backend               = "htpasswd"

auth.backend.htpasswd.userfile = "/var/www/.htpasswd"

auth.require               = ( "/" =>

                               (

                                 "method"  => "basic",

                                 "realm"   => "A2Billing Management",

                                 "require" => "valid-user"

                               )

                             )

The above configuration made the interface spwan a single FastCGI, insuring that memory usage is never over utilized. I still need 512MB of RAM to run the scripts, but at least now it’s limited to only 512MB of RAM, out of a machine that has 16GB of RAM.

For a while now I’ve been toying around with the idea of utilizing Lighttpd for various web based applications. One of these application is my Automatic Dialer framework, also known as the GTD-API. The main issue with the GTD-API (besides that it is highly reliant on a MySQL database), is the fact that all requests have to be processed via XML-RPC HTTP post requests.

The main issue that I had was this: in a production scenario, a dialer management system will generate over 100 requests to the XML-RPC server. While Apache is fully capable of rendering services at such a speed, its increasing size and boilerplate automatically introduce a management issue. In addition, as I was trying to build a dialer appliance that can be used in any enterprise, the ever expanding Apache wasn’t a good choice.

While I was looking at both NginX and Lighttpd, the latter captured my eye, thanks to a simple advantage – The integration of FastCGI based PHP was so easy, that it almost troubling that I used Apache all these years.

At this point, once I got Lighhtpd working with my Dialer, I said to myself: “It would be really cool to go about and send status reports back from the dialer, directly to the web client activating the call. In addition, I really don’t want to go about and perform these updates to the database, then query the database – that would, literally, kill the MySQL server.

So, I implemented a local session storage area for each call, which updated the call status as it traverses. The information was stored on the hard drive, which allowed a better response time than the ever indexing MySQL server. The status reports were picked up from the Lighttpd server via an Ajax client (which I didn’t write – I suck at JS) – and it works quite well.

I wonder, can Lighttpd completely replace Apache? … time will tell…

Lets put it this way: I am no web developer, nor was I ever, nor will I ever be. While I do enjoy playing around with various web designs and web technologies, I’m no web developer. Think about it, this blog is based upon the WordPress system, which means, that while I could easily build my own blog system – I didn’t.As most developers are, I am a lazy person – when it comes to writing code, that is. This means that when I write an application, I really like spending my time working on the application logic, rather than wasting my time on GUI. I’ve always looked for better ways to improve my applications development track, especially, the ability to seperate the logic from the display in such a way that both become agnostic to one another. So, I started looking into various MVC (Model-View-Controller) frameworks – which had been springing up all over the past year.

While the most popular one seems to be Ruby-On-Rails, I don’t like Ruby and prefare PHP. Various options exist here, so I seeked one that was easy to integrate and that is backed by a company of some sort.

Zend Framework

Much like PHP, Zend Framework appears to be a mixture of functions, closely wrapped into a set of classes, to help you create an MVC environment. I’ve started learning it, and shortly came to a conclusion: much like PHP, Zend Framework enables you a feature rich environment, however, it isn’t always clear how to get to do something with it.

CakePHP

CakePHP appears to be a slightly more rigid PHP MVC environment, with a fairly vibrant and lively community backing it up. However, like any other young community backed project, it lacks one main element: proper documentation. The documents available on the CakePHP.org website are sketchy at best, which means, there is no ordered manner of getting started fast with CakePHP.

Code Igniter

Code Igniter is a PHP MVC environment, backed by Ellis Labs – the same people behing Expression Engine. So, from a technical point of view, this is a plus, as Ellis Labs had made it its business to make Code Igniter a valid product. The amount of information available on the Internet is satisfactory, and the documentation on the website is more than good – it’s down right GREAT! The addition of video tutorials with a very clear naration enables even a novice developer to go about and start working fast with Code Igniter. The one thing that Code Igniter lacks is a rigid framework, which means, that just like PHP – it’s easy to fuck about and mess things up.

My Choice

Currently, I use a mixture of Code Igniter + Prototype + XAJAX to build my web applications, which makes for a fairly rapid development environment – I’d love to hear what you have to say about these.

As most of you reading this blog, you must know that I’ve published an AsteriskNOW book early this year. I’m happy to say that I’ve received numerous emails since the publication of the book, all asking various questions about Asterisk in general and AsteriskNOW in particular.

Most of the questions that I received were related to the development of AGI and AMI scripts, and how to utilize Asterisk as an application layer platform. This inspired me to work on additonal title, to complete my previous book. I’m currently working on an Asterisk Developers guide, for AGI/CTI development with Asterisk. The book serves as a complete guide, summarizing the various aspects of developing AGI/CTI applications with Asterisk, while, enjoying various additions from my day-to-day experience with AGI/CTI development with Asterisk. The book is written as an eye-opener for experienced developers, wishing to make their transition to AGI/CTI development, and teaches them how to avoid the most common mistakes of early day AGI developers.

I’ll be be updating the blog with my progress. Currently, chapters 1 through 5 are complete, covering the basic aspects of dial-plan and AGI development. More information will be released soon.

As most of you already know, I’m heavily involved within the Asterisk Open PBX project. Over the course of the past 5 years of my dealing with Asterisk, Asterisk had always suffered a serious flaw, and that is, a single-threaded Manager interface – which usually led to serious dead-locks when writing a multi-threaded server that connects to it.

One of my long time challenges was to surpass the 4-5 originate requests to the Asterisk Manager interface, enabling me to automatically dial more than 4-5 calls at the same second. My initial work had began with the idea of increasing that by a factor of 50%, going up to around 7-8 calls per second – I had achieved that using a combination of smart synchronization between the manager interface and my originating server – and also enabling asynchronous originate requests – however, that methodology had proved to be problematic – in terms of reliability.

I understood that something else had to be devised, something that doesn’t rely completely on the manager interface, and that will allow me to originate calls freely, without clogging up the manager interface. So, I decided to move my interest from the Manager interface, and concentrate on understanding Asterisk’s channel handling, especially, how do calls originating from the manager interface are handled by the Asterisk spooler and the Asterisk channel drivers.

more will follow…

OK, if there is one thing I really hate, is that when something so little as a small configuration change from one version to another causes things not to function properly. It is one thing to make sure that your code is backward compatible, no one really expects that your scripting language will suddenly start behaving differently, just because you upgraded to a new minor version – right?

So, here is the case, over the course of the past 2 years, I’ve done some extensive AGI programming with the PHP programming language – hell, I think that if I’ll sit down and write a book about AGI programming with PHP, it would be one of the most complete ones. I remember that when I started writing AGI scripts, I used to simply write everything by myself, each script was a self contained unit, everything was working hunky-dory. Shortly later, I discovered the PHPAGI class, a nicely wrapped AGI class for AGI, FastAGI and Asterisk Manager interaction, that made my life really easy. When Asterisk evolved to version 1.2, most of my PHPAGI scripts kept on working, apart from a small part of the Manager interface, however, that was easily worked around – and we’re back hunky dory. Even Asterisk 1.4 and Asterisk TRUNK liked PHPAGI, no problem there – till a week ago.

So here I am, working on a new project, and I’m working on my new snazzy development kit, that is installed according to the following:

CentOS 5.0 x86_64 with Kernel 2.6.18-8.1.4.el5
Asterisk stable 1.4.14
php version 5.1.6 (RPM package: php-5.1.6-12.el5)

At this point, all hell broke loss! All of my old PHPAGI scripts had simply gone crazy on me, causing each of the AGI “GET VARIABLE” commands to fail, always returning variables which are not there at all. That was fairly annoying, I’m issuing a command, getting an error response, issue the same command again, get a proper response only with the wrong variable, issue the command again – get the proper value. How annoying! – All these scripts used to work perfectly in the past, what happened? So, I deploy an older code of mine, guess what, that code doesn’t work either – at this point, I start pulling hairs off my head (not that I have much to pull on, my hair is very short).

So, what does any good Open Source programmer do when confronted with such a problem? “Use the source Luke, use the source!” – lets dig into the Asterisk code. Adding some AST_LOG functions here and there, doing some more debugging using valgrind, everything within Asterisk looks nice. Just to make sure, lets downgrade the Asterisk version and run the test again – same results. Conclusion: Asterisk isn’t the problem, doesn’t really matter what version I use.

OK, lets check the PHP scripts – some PHPAGI class hacking, more debug here, mode debug there, a few more syslog calls – everything looks on the PHPAGI script as if the input from Asterisk is messed up. Conclusion: something in the middle is fucking up royally!

Now, being a good programmer I said to myself: “OK, so one said that PHPAGI is the only language out there, lets go Python – besides, I always wanted to learn Python”. So I spend a day learning some Python, just so I would be able to re-write my script into Python. Ran the script – holly shit! – it works, as if nothing happened. Conclusion: something in the PHP environment is screwing up the environment. So, I decide to modify my execution from using php-cli to php-cgi, for all practical matters (at least from the script side) – it should behave identically. Modified it to work with php-cgi, suddenly the $argv variable isn’t being passed to the script! – Hmmmm…. that’s odd, it was there a minute ago. Lets dig into php.ini and see what’s wrong:

; This directive tells PHP whether to declare the argv&argc variables (that
; would contain the GET information). If you don’t use these variables, you
; should turn it off for increased performance.
register_argc_argv = Off

Dear god, why would $argv and $argc be turned off? just for performance sake? how would I run the AGI scripts that I want to run, after all, I am using the $argv variable. Turn the parameter back to “On” – JOY! Everything is back working as it should. So the question that remains is: “What is the main difference between php-cli and php-cgi that had caused this issue?” According to the voip-info.org website, there is some form of difference, however, it doesn’t really say how it is exhibited in the operation of AGI scripts.

Well, at least I solved my problem for the mean while, I’ll work on getting the proper explanation of php-cli vs. php-cgi later on.

There is nothing more annoying than an incomplete specification design, brought to you as a design document. Why does every person that is capable of operating a word processor at a beginner level and operating Visio at a level of a child, believe that they are able to produce a proper design specification or ever a proper requirements document?

I had just received a so-called “specification document” from a prospective customer, that had specified the required system as “a pre-paid IVR system for premium services”, that was accompanied by a half page flow chart that had 4 blocks on it, without any description of what the system does. Now, I called the customer, explaining to him that the document isn’t complete, and I’d like to meet with him in order to write a proper document – the following conversation took place:

Customer: I don’t think a meeting is needed, if you are too stupid to understand what I want – then I don’t think I want to work with you!.

Me: I’m very sorry that you think that I’m not good enough to carry your project, you are welcome to take your business to someone else who is more capable than I am.

Customer: Well, the thing is that you are my last hope, everyone said you can develop anything. I sent this document to a few companies, they all returned price quotes that were too high.

Me: They returned a high price quote because they have no idea what you want, so they gave you a price quote so high, so you would back off and not call them again. I want to help you do the project, but in order to do it, I need you to define your requirements better.

Customer: Are you familiar with the functionality of system X?

Me: Yes, that is a fairly complex system – development time is around 6-8 weeks, at minimum.

Customer: No, I need it working within 10 days, I already lost 6 weeks of getting price quotes.

Me: I’m sorry, but in this case, I can’t really help you, your requirements and time considerations are impractical.

At this point, the customer had started screaming on the phone – which was a point when I had bid the customer farewell and had disconnected the phone call, as I never conduct my business in a manner when people are screaming without listening to what they are shouting.

Now you tell me, was I logical in my response? or maybe the customer is right? why do customers have the impression that building complex systems is a matter of simply installing the computer, slapping a few pieces of code on the machine, test it a few times – and walla, a running system is ready for production?

Call me old fashioned, but I believe that each development should be backed by a proper specification design, a design document, implementation design, regression testing documents and acceptance tests – if you don’t agree, I’d love to hear about it.

What would you call a platform that is made up of multiple elements, each one connected one to another by a very thin line of integration – having each one reliant on the other for proper functionality? – A MESS!

Recently, a customer of mine had asked me to review a platform that he has, that had been built by another integrator. According to the customer, the platform was built on top of an Asterisk + MySQL stack, with an Apache based front end. I had asked the customer to send me some code segments from the platform, so that I may have a better view of things. The code segments that were sent had suggested that the platform isn’t far from the description. Asterisk is indeed the main IVR core, MySQL is used as a database, and it would appear that the scripting language is PHP – all is good, looks fairly nice. So, next step was to logon to the actual platform and take a closed look…

At this point, all hell broke loose! Aparently, the web front end was based on a Python based engine called django. While django was an interesting piece of code in terms of learning it and maybe using it, the immediate meshing of django into the IVR application framework didn’t make any sense to me. In addition to that, I’ve come to learn that there is also an Oracle database server installed on the system, running some funky JAVA server, that was communicating with the IVR application, via a very flaky interface. In other words, the platform is a mixture of technologies, integrations that are done in a none standard way and worse of all, no documentation anywhere in the code about how it works.

Now, the customer wants me to take control over the platform and to assume responsibilities for it. My immediate answer was a big NO WAY! Imagine that you need to take responsibility for a child that is not yours, one that you know for fact that is prune to burn down your house, as he shows clear signs of pyromania – would you assume responsibility? I don’t think so. So, I tell the customer all the facts and the customer says: “I don’t care, I want it in working condition, ready for production in 7 days!” – in other words, and impossible mission, especially with all the decoupled technologies within the application framework.

What would you have done?