The Nir Simionovich blog

For a while now I’ve been toying around with the idea of utilizing Lighttpd for various web based applications. One of these application is my Automatic Dialer framework, also known as the GTD-API. The main issue with the GTD-API (besides that it is highly reliant on a MySQL database), is the fact that all requests have to be processed via XML-RPC HTTP post requests.

The main issue that I had was this: in a production scenario, a dialer management system will generate over 100 requests to the XML-RPC server. While Apache is fully capable of rendering services at such a speed, its increasing size and boilerplate automatically introduce a management issue. In addition, as I was trying to build a dialer appliance that can be used in any enterprise, the ever expanding Apache wasn’t a good choice.

While I was looking at both NginX and Lighttpd, the latter captured my eye, thanks to a simple advantage - The integration of FastCGI based PHP was so easy, that it almost troubling that I used Apache all these years.

At this point, once I got Lighhtpd working with my Dialer, I said to myself: “It would be really cool to go about and send status reports back from the dialer, directly to the web client activating the call. In addition, I really don’t want to go about and perform these updates to the database, then query the database - that would, literally, kill the MySQL server.

So, I implemented a local session storage area for each call, which updated the call status as it traverses. The information was stored on the hard drive, which allowed a better response time than the ever indexing MySQL server. The status reports were picked up from the Lighttpd server via an Ajax client (which I didn’t write - I suck at JS) - and it works quite well.

I wonder, can Lighttpd completely replace Apache? … time will tell…

Lets put it this way: I am no web developer, nor was I ever, nor will I ever be. While I do enjoy playing around with various web designs and web technologies, I’m no web developer. Think about it, this blog is based upon the WordPress system, which means, that while I could easily build my own blog system - I didn’t.As most developers are, I am a lazy person - when it comes to writing code, that is. This means that when I write an application, I really like spending my time working on the application logic, rather than wasting my time on GUI. I’ve always looked for better ways to improve my applications development track, especially, the ability to seperate the logic from the display in such a way that both become agnostic to one another. So, I started looking into various MVC (Model-View-Controller) frameworks - which had been springing up all over the past year.

While the most popular one seems to be Ruby-On-Rails, I don’t like Ruby and prefare PHP. Various options exist here, so I seeked one that was easy to integrate and that is backed by a company of some sort.

Zend Framework

Much like PHP, Zend Framework appears to be a mixture of functions, closely wrapped into a set of classes, to help you create an MVC environment. I’ve started learning it, and shortly came to a conclusion: much like PHP, Zend Framework enables you a feature rich environment, however, it isn’t always clear how to get to do something with it.

CakePHP

CakePHP appears to be a slightly more rigid PHP MVC environment, with a fairly vibrant and lively community backing it up. However, like any other young community backed project, it lacks one main element: proper documentation. The documents available on the CakePHP.org website are sketchy at best, which means, there is no ordered manner of getting started fast with CakePHP.

Code Igniter

Code Igniter is a PHP MVC environment, backed by Ellis Labs - the same people behing Expression Engine. So, from a technical point of view, this is a plus, as Ellis Labs had made it its business to make Code Igniter a valid product. The amount of information available on the Internet is satisfactory, and the documentation on the website is more than good - it’s down right GREAT! The addition of video tutorials with a very clear naration enables even a novice developer to go about and start working fast with Code Igniter. The one thing that Code Igniter lacks is a rigid framework, which means, that just like PHP - it’s easy to fuck about and mess things up.

My Choice

Currently, I use a mixture of Code Igniter + Prototype + XAJAX to build my web applications, which makes for a fairly rapid development environment - I’d love to hear what you have to say about these.

It always amazes me how fragile our body really is. For those of you who had met in person, you know that I’m a fairly bulky guy - which means that I’m slightly larger than most people. In any case, there is nothing more frustrating than throwing your back - it’s the most annoying thing ever.

Throwing your back is somewhat like having an invisible fly hovering next to your ear, you can hear it, but you can’t swat it! A thrown back is just the same, you are perfectly fine in every other aspect, but you can’t do anything because you are STUCK!

I’m writing this post from bed, and believe me, as much as that sounds inviting - IT ISN’T!

As most of you reading this blog, you must know that I’ve published an AsteriskNOW book early this year. I’m happy to say that I’ve received numerous emails since the publication of the book, all asking various questions about Asterisk in general and AsteriskNOW in particular.

Most of the questions that I received were related to the development of AGI and AMI scripts, and how to utilize Asterisk as an application layer platform. This inspired me to work on additonal title, to complete my previous book. I’m currently working on an Asterisk Developers guide, for AGI/CTI development with Asterisk. The book serves as a complete guide, summarizing the various aspects of developing AGI/CTI applications with Asterisk, while, enjoying various additions from my day-to-day experience with AGI/CTI development with Asterisk. The book is written as an eye-opener for experienced developers, wishing to make their transition to AGI/CTI development, and teaches them how to avoid the most common mistakes of early day AGI developers.

I’ll be be updating the blog with my progress. Currently, chapters 1 through 5 are complete, covering the basic aspects of dial-plan and AGI development. More information will be released soon.

Back in the year 1999, long before I started my Asterisk days, I spent most of my time as a security consultant and cyber forensics expert. I remember that in those days, most of the hacks were script kiddies exploiting some Windows IIS well known hole, and you would usually get the “Hacked by Chinese” black display on your website - how annoying!

In any case, I’ve recently replaced my co-location firewall. I’ve migrated from a Linux system running IPtables with a manual script, to a fully blown IPCOP installation. Ok, so IPCOP is nothing more than a fancy GUI for IPtables, but hey, it makes my life a whole lot easier on the management side - and it’s very stable - so who am I to complain?

I’ve decided to run a small experiment, I wanted to setup a Linux box, with a root password of 123456. My question was this, how much time will pass from the moment the machine was up, on a new IP address, till the machine gets hacked - and more importantly, from where and what got installed on the machine?

So, the machine fired up for the first time at Fri Jul 25 23:19, believe it or not, the machine got hacked at Sat Jul 26 00:50. A mere 90 minutes into the air, and the machine got hacked. The funny thing was that at Sat Jul 26 03:09 it got hacked again to the same account, then at Sat Jul 26 03:21, which also closed the root access via SSH at this point. Following below is the last log:

root     pts/0        77.127.137.52    Sat Jul 26 06:04   still logged in
reboot   system boot  2.6.18-53.1.14.e Sat Jul 26 06:02          (00:17)
root     pts/1        92.80.195.126    Sat Jul 26 03:21 - 03:24  (00:03)
root     pts/0        78.110.163.31    Sat Jul 26 03:09 - 05:20  (02:11)
root     pts/1        60.220.240.7     Sat Jul 26 00:50 - 00:50  (00:00)
root     pts/0        77.127.137.52    Fri Jul 25 23:24 - 01:39  (02:14)
root     tty1                          Fri Jul 25 23:22 - 23:24  (00:01)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 23:19          (07:00)
root     tty1                          Fri Jul 25 22:14 - down   (01:03)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 21:58          (01:19)

I admit it, putting a machine on the open net, with a root password of 123456 and open root access to SSH - that’s kind of a honey pot the size of the grand canyon. But what amazed me here was not the speed, but actually the locations of the hacks: 60.220.240.7, 78.110.163.31 and 92.80.195.126. One hacker is in China, the other in Romania and the third in the UK. What is this? a real hacker? maybe 3 different robots scanning? - I can’t really tell here. However, the traces they left were interesting enough - which lead me to believe we’re talking about robot hacking.

First off, a look at /var/log/audit/audit.log immediately showed the logins - the hacker didn’t even remove the log file - marking of a script kiddie running an automated script. So, what did they leave on my box, let’s take a look. Running ‘netstat -apn | less’ would show me open ports, unless netstat was replaced. However, lets start with this:

tcp        0      0 172.31.31.16:34183          195.47.220.2:6667           ESTABLISHED 2940/crond
tcp        0      1 172.31.31.16:57263          195.54.102.4:6667           SYN_SENT    2940/crond
tcp        0      1 172.31.31.16:46043          195.68.221.221:6667         SYN_SENT    2940/crond

Ok, so this is most probably an IRC bot waiting for instructions from the hacker - till now nothing special. The script tries to masquerade the bot with a legitimate process name: crond. Well, that may fool a beginner Linux Sysadmin, however, seeing crond connecting to 3 other hosts at TCP 6667 - ok, that’s kind’a lame - no?

I wonder where he hid the script? maybe he replaced crond?

root@pbx:~ $ find / -name "crond"
/usr/sbin/crond
/var/tmp/.www/crond
/var/lock/subsys/crond
/etc/sysconfig/crond
/etc/rc.d/init.d/crond
/etc/pam.d/crond
root@pbx:~ $

Hmm… /var/tmp/.www/crond looks promising, let’s see what’s in there:

root@pbx:~ $ ls -la /var/tmp/
total 24
drwxrwxrwt  4 root root 4096 Jul 26  2008 .
drwxr-xr-x 25 root root 4096 Jul 25  2008 ..
drwxr-xr-x  2 root root 4096 Jun 27 17:03 .spd
drwxr-xr-x  4  501  502 4096 Jul 26  2008 .www

Yummy! Let’s check it out:

root@pbx:/var/tmp $ ll .spd/

total 1316

-rwxr-xr-x 1 root root    265 Nov 19  2005 gen-pass.sh

-rwxr-xr-x 1 root root     72 Jun 26 19:43 pass_file

-rwxr-xr-x 1 root root  21407 Nov 19  2005 pscan2

-rwxr-xr-x 1 root root    218 Jun 27 16:59 s

-rwxr-xr-x 1 root root 453972 Nov 19  2005 ss

-rwxr-xr-x 1 root root 842736 Jun 26 19:20 ssh-scan

-rwxr-xr-x 1 root root    312 Jun 27 17:02 x

root@pbx:/var/tmp $ ll .www/

total 888

-rwxr-xr-x 1  501  502    353 Jul 26  2008 1.user

-rwxr-xr-x 1  501  502    349 Jul 26  2008 2.user

-rwxr-xr-x 1  501  502    353 Mar 14  2009 3.user

-rwxr-xr-x 1  501  502    317 Nov  6  2007 autorun

-rw-r--r-- 1 root root      0 Jul 26  2008 belgian.seen

-rwxr-xr-x 1  501  502    942 May 15  2003 checkmech

-rwxr-xr-x 1  501  502  23237 May 15  2003 configure

-rwxr-xr-x 1  501  502 492135 Mar  4  2005 crond

-rwxr-xr-x 1  501  502     48 Jul 26  2008 cron.d

-rwxr-xr-x 1  501  502    171 Jul 26  2008 cutitas

-rwxr-xr-x 1  501  502   4147 May 15  2003 genuser

-rwxr-xr-x 1  501  502    157 Jul 25 17:36 LinkEvents

-rwxr-xr-x 1  501  502      0 Oct 15  2007 lucifer.seen

-rwxr-xr-x 1  501  502   2154 May 15  2003 Makefile

-rwxr-xr-x 1  501  502     14 Jul 26  2008 m.dir

-rwxr-xr-x 1  501  502  22882 May 15  2003 m.help

-rwxr-xr-x 1  501  502    748 May 15  2003 mkindex

-rwxr-xr-x 1  501  502   1043 Jul 26  2008 m.lev

-rwxr-xr-x 1  501  502      5 Jul 25 17:35 m.pid

-rwxr-xr-x 1  501  502   1068 Jul 26  2008 m.ses

-rwxr-xr-x 1  501  502   1675 Mar 25  2009 m.set

-rwxr-xr-x 1  501  502 167964 Mar 16  2001 pico

-rwxr-xr-x 1  501  502  84476 Jun 23  2006 pico.tgz

drwxr-xr-x 2  501  502   4096 Jul 23 15:48 r

-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{700.seen

-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{800.seen

-rwxr-xr-x 1  501  502    715 Jul 12 22:00 shadow}{900.seen

drwxr-xr-x 2  501  502   4096 Jul 23 15:51 src

-rw-r--r-- 1 root root   1842 Jul 26  2008 zak.seen

Looks like .spd is the SSH scanner and the .www directory contains the actual bot binary - ok, I can respect that. The contents of the cron.d file suggested that the script utilizes crontab to verify that the bot is always up and running - and examination of its code assured me of that.

So, what have we learned from the above: just one thing! When installing a server for the first time, DON’T USE A SILLY PASSWORD LIKE 123456 - EVEN NOT FOR THE INSTALLATION PHASE! Scanning robots appear to be scanning the entire Internet over and over and over again, doing so in seconds - so by the time you install your server, set it up completely, there is a good chance it will already be compromised.

We’ve recently learned that Fonality had forked FreePBX into its own version, to better serve the TrixBox community. Judging from what we’ve learned about Fonality/TrixBox/KG over the past few months, it is my personal belief that this is just another from of spin on the “TrixBox calls home” feature, simply doing something to hide it better - most probably will be somewhere in the management code now.

However, it led me to an interesting discussion with a friend - “will Fonality fork Asterisk?”

It is fairly clear that Fonality is doing all in its power to go about and distinguish itself from the rest of the community and the Asterisk eco system, by simply creating a product that is completely seperated from Asterisk. The amount of patches and modifications going into the TrixBox distribution, makes the running Asterisk on TrixBox a completely different one than the one running on AsteriskNOW, Elastix or pbx-in-a-flash. Is it stable? that is a good question, I’d like to believe that it is. After all, if it wouldn’t have been stable, Fonality would have been out of business. Fonality also goes to great deal to make sure that their TrixBox resellers can’t replicate their appliance easily. For example, over the course of the past 12 months, Fonality had changed the insides of their TrixBox appliance a few times, each time with a different motherboard, a different set of distribution packages and so on.

In the same fashion, it is only common sense for Fonality to fork Asterisk to their own product. My assumption is that Fonality at some point will either fork Asterisk, migrate their code to FreeSwitch or more probably CallWeaver, take over one of these projects like they took over TrixBox/AAH and completely distinguish themselves from the Asterisk community and product line. Will it do good for them? time will tell - if it happens. Will it be good for Asterisk/Digium? - in general terms that answer will be yes, as it will make Fonality/TrixBox automatically distinguishable from Asterisk. Which when asked what is the different between TrixBox and Asterisk, the immediate answer would be: “These are two completely different products!”.

It is my belief that by Q4 2010 we are to see some major shifts in the Open Source Telecom arena. My projection is that by Q4 2010 Digium will be in a position for either an IPO or an M&A. While my personal belief is that Digium prefers an IPO, an M&A proposition from a major vendor (Cisco/Nortel/Avaya) will come before the IPO option. I also believe that by the Q4 2010 Fonality will either fork Asterisk, adopt FreeSwitch or CallWeaver and distinguish themselves.

It is also my belief that by Q4 2010 Sangoma will try to acquire an Open Source PBX/Switch project. Although they recently acquired a SIP PBX company, I believe that this acquiry is nothing more than a small publicity stunt to keep Sangoma in the investor’s mind, making sure that Sangoma progresses in some form. The recent news about Sangoma integrating their signalling stacks to FreeSwitch makes me believe that the most likely candidate will be FreeSwitch - the Asterisk nemesis.

In any case, stay tuned for 2009… here we come…

Lately I’ve come to the realization, that we are to blame for our own inability to promote Open Source and the adaptation of Open Source proficiency. Being an Open Source evangelist and consultant, this is very weird to be said by one like myself, however, this is my realization - and I will explain.

In the early days of Open Source adaptations (late 90’s, early 2000), Open Source software was a somewhat magical solution that meant: pay nothing, get more. Software packages like Linux, Apache, mySQL, PostgreSQL and programming languages like PERL and PHP had lowered the bar on the adaptation of new technologies, and enabled a prolific number of solutions and services.

I still remember the early days, when a Windows based Mail Relay would cost anything between 800$ to 1200$, and I would come in with a Linux based solution that would do the same thing for FREE - amazing. As time progressed, so did the technology and the penetration of Open Source into new fields. CRM, ERP, Telecoms, management - all of these now enjoy a diverse number of Open Source solutions. However, the original concept of ‘Open Source = Magical FREE Solution’ had still remained in the minds of managers and business people.

Today we are confronted with ‘would-be’ Open Source solution experts, which adopt and develop upon Open Source products and project various applications. In example, let’s take a look at Asterisk. Asterisk has a multitude of Open Source solutions, ranging from PBX system, Prepaid calling cards, Wholesale routing platforms, Attendance system, Presence systems - and even a plant watering solution. The problem with this ever growing number of solutions is that Asterisk is immediately considered to be: “A magical solution” capable of solving any problem - when it’s not even remotely related to Asterisk. For example, a friend of mine had been asked to develop an Asterisk based solution, that would support a total of 250 concurrent call initiations and up-to 3000 concurrent calls on the system. Any Asterisk developer would take a look at this, and would immediately say: “Hmmm…. this requires several servers, but hey, what about the application itself? that would also have an impact”. Now, the customer of the project has a ‘would-be’ Asterisk tech in his company which said: “I was able to initiate 200 concurrent SIP invites to Asterisk via SIPP, no problem’ - HELLO! STUPID! where’s the application? where’s the database? where’s the user information flow? comm’on, are you listening to yourself speak? or simply are filled with the gasses coming out of your ass that are affecting your brain?

Now, once the customer learns that Asterisk is most probably not the right solution for the problem, he becomes angry. Why? because he now learns that he needs to spend about 10 times more money than he anticipated for the creation of this tool - well, that’s life when you have no idea what you are doing/saying, and you believe in magical solutions. However, we - “The Open Source Community - is the one to blame for this scenario, because we got the world accustomed to the idea that Open Source is like magic - flip the Linux magic wand, and the rest will solve itself.

I’d like to open the floor for discussion on this, as I believe most of you will have something to say about this.

I guess web robots are funny creatures, especially when you write the weirdest stuff on your blog. For example, one of the my previous posts was about a movie I’ve seen lately, “You Don’t Mess With The Zohan”. Now, imagine my surprise to get into my inbox about 6 messages, from various web robots that had come across my post and tagged it along the Internet.

Now, this raises a very interesting question in my head: “Is it possible to create a logic bomb, that would render the entire web robots network into a frenzy?” - for example, let’s say I would like to sabotage the opening night of a movie I dislike. Technically, it would require me to open multiple websites that talk about this movie, then, make sure that the web robots find these pages - meaning, make the pages robot friendly, this will end up in a form of storm of robots on these pages, and most probably, a storm of robots on each other - interesting, isn’t it?

Here’s another question, let’s say I create 1000 web domains, all having the same web front page. All front pages simply include a list of links to a single website, listing all the available links on that website. If a robot encounters that page, will it start traversing the links to that website? and if there 1000 of these, will these automatically bring the Google page rank up? I’m no SEO expert, but my logic says yes.

Well, I guess it’s time to do some experimenting I guess…

Ok, I admit it, the topic sounds ultra geeky and nurdy - but I can’t help it, there is something about booting up your computer from a USB pen drive, having all your nicely wrapped tools in there and having fun with it.

In this case, my pen drive is actually the driving force behind an extremely powerful call recording system, based on the Asterisk Open Source PBX system. Essentially, the Cruzer boots up a CentOS 5.1 system, fully equipped with an Asterisk + Zaptel + LibPRI + FreePBX. The system is configured to utilize up to 12 E1 circuits, with auto sensing scripts that will automatically configure your system upon first boot-up. Once the system had booted up, it will start identifying your hardware hard drives, and will start cataloging to these hard drives all the recordings according to the pre-determined logic.

I currently use a MySQL database on the Pen Drive to store catalog information only, which is working nicely - but I need to figure out a better way to store more information - 2GB of MySQL storage may be enough for a short while, but serving a large contact center won’t be much of a good idea - I think.

The Pen Drive was created using tools from www.pendrivelinux.com, which contains wonderful information about how to create your own custom Linux based Pen Drive - Excellent!

I’ll start off with this: I Like Adam Sandler Movies, they always make me laugh. For some strange reason, Adam Sandler movies are among the few movie subject my wife and I agree on. While we watch many movies on our home theater system, Adam Sandler is one of the only things we’ll agree to see a real theater. In any case, tonight we’ve gone to see his latest one: “You Don’t Mess With the Zohan!”.

In one sentence that sums the entire movie: The movie is really funny and enjoyable. However, I’d like to address some the stereotypes and obvious misunderstanding of Israeli nature that are portrayed in the movie. First of all, the movie shows Zohan, walking around the beach of Tel-Aviv, and portraying a macho man on steroids, in our case, the ultimate Israeli soldier enjoying his vacation - while parading nude on the beach and flapping his salami all over.

Hello! We don’t do that - While it is true that Israeli men are considered somewhat rough around the edges, we don’t usually parade our personal parts on the beach. Actually, there are laws in Israel prohibiting that, no matter if you are Soldier or war hero.

Anti-Terrorist fighters are celebrities - How un-true! In Israel, anti-terrorist fighters are well kept secrets, you never see their faces. Even combat pilots are never shown on TV. According to the movie, Zohan, an anti-terrorist fighter is a well known figure in Israel and people regard him as a legend. Actually, he is so know, that even when coming to America, he can’t flee his fans who recognize him - a situation that can’t happen in real life.

In addition to the above, the Tel-Aviv beach portrayed in the movie, while truly being shot in Tel-Aviv, is not a hyper-drived version of Miami. I admit that during summer the Tel-Aviv beach is crowded and picturesque, but it is nothing like the movie portraits.

The movie continues to portray Zohan as an over dozed sex maniac, capable of performing sex with almost any female partner. In addition, this portrait is later on in the movie taken over by the character of Uri, another Israeli living in New-York - managing a would be called electronics shop. I’m very happy to say that most Israeli men don’t behave like the ones in the movie, it’s just plain silly.

Over the course of the movie, one of Israel’s national foods is being exploited all over - the Humus. For those of you who don’t know, Humus is a paste like dish, made of a mixture of Chickpeas, Sesame paste (also know as Tehina) and other herbs and spices. It is true, we love our Humus, but we don’t use it to brush our teeth, dip our glasses in it, put it in our tea or coffee - it is a good starter dish, but it doesn’t define us.

Another thing to say about the movie, is the cast. While Zohan (Adam Sandler) is the main character of the movie, other characters are portrayed by memorable actors. “The Phantom”, a Lebanese terrorist, is portrayed by John Turturro, a fairly interesting choice. While John’s comic talents had been observed over the years, his role as a terrorist, turned fast-food giant, turned show sales man is hysterical. Salim, a Lebanese farmer who’s goat was taken by Zohan is portrayed by Rob Schneider. Rob is well known for his role as Duece Bigalow, America/European Gigolo. Rob does an amazing job at portraying a fairly simpleton farmer, trying to rebuild his life in the US, while being confronted by an old nightmare - Zohan. His almost cartoonish attempts to harm Zohan and later on dialing the “Hizbella” terrorist hotline - is simply as funny as funny can get.

Well, other than the above, I highly urge you to go see the movie, a really funny flic and a nice 90 minute get away from your regular day hassle bussle. Btw, if you ever come to Israel - don’t look for the Fizzy Bubbly drink - IT DOESN’T EXIST - WE DRINK COKE!