Like most of the world, I’ve been following the recent market turmoil with a great burden on my shoulder. When you think about it, I’m not a stock broker, nor am I a multi-billionaire that has his funds invested in various stocks and bonds, that a single 0.1% shift in the NASDAQ translates to millions of dollars. I’m a software developer, a freelance one, dealing in the Open Source – and like anybody else, I’m worried about how this crisis immediately affects me.

Today, I came across two items, post on – Israel’s topmost Internet based financial/business daily. The two items dealt with how three of the better known VC’s in Israel had started instructing their investees to start cutting down costs – mainly, firing people. The three VC’s that I’m talking about are: Carmel Ventures, Benchmark Israel and Sequoia Capital. You are probably wondering why is this interesting? the VC’s in the item had directly instructed their investees to cut down people, costs, operational costs, loose dead weight – in other words, find ways to reduce your costs. Sequoia even out did Carmel and Benchmark, by inviting the investees to a meeting called: “RIP: Good Times”!

Shortly after I finished reading the two items, I got a phone call from a friend working at one of Sequoia’s companies (a well known one in Israel) asking me if he can come work for me. I was surprised, this is the first time I’ve ever read something in the news, and was directly affected by it. As far as I gathered, his company basically took a team of 8 people and reduced it to 2. Now, I completely understand tightening up, but running an operation on a 25% man power is stupid! Running at 50% is manageable, but 25% is down right crazy. For 2 people to do the work of 8, they would need to eat, drink, sleep, live, do everything within the office – I know, I’ve been there. During the year 2003, m-Wise was more or less in the shit. In the year 2002 I had a team that consisted of another SysAdmin and 3 more support techs. In 2003 I was left alone, and I basically did everything myself! – how crazy is that. But again, I decided that I’m not going to have a life for a certain period of time – that is all, not everybody is willing to make that sacrifice.

Now, this case goes hand in hand with my previous post – the migration to Open Source technologies is no longer a myth or a “nice-to-have” issue, it is a matter of business continuity and good expense management. Think about it, the company that fired 75% of their team, could have easily replaced part of their server infrastructure from Windows to Linux, migrate their Oracle database to PostgreSQL and save thousands and thousands of dollars a year, and maybe even save a job or two in the process.

Now, here’s what I think (and I know for fact I’m gonna get slammed here): Hey, VC’s, stop telling the companies to let go people. Sure, get rid of dead weight – no one needs those M$ based shitty, money grabbing, time consuming, hardware intensive environment. Wouldn’t it be better to not pay M$ a few ten’s of thousands of dollars a year, and maybe save a man’s job, or maybe even 2? M$ has enough money of their own, all you are doing is making sure they keep on making money, while the rest are fighting for their lives. Why don’t the VC’s hire Open Source consultants, to help them examine their investees and maybe, just maybe, they will find ways to invest their funds in a wiser way and help these companies to survive the current financial turmoil.

Back in the year 1999, long before I started my Asterisk days, I spent most of my time as a security consultant and cyber forensics expert. I remember that in those days, most of the hacks were script kiddies exploiting some Windows IIS well known hole, and you would usually get the “Hacked by Chinese” black display on your website – how annoying!

In any case, I’ve recently replaced my co-location firewall. I’ve migrated from a Linux system running IPtables with a manual script, to a fully blown IPCOP installation. Ok, so IPCOP is nothing more than a fancy GUI for IPtables, but hey, it makes my life a whole lot easier on the management side – and it’s very stable – so who am I to complain?

I’ve decided to run a small experiment, I wanted to setup a Linux box, with a root password of 123456. My question was this, how much time will pass from the moment the machine was up, on a new IP address, till the machine gets hacked – and more importantly, from where and what got installed on the machine?

So, the machine fired up for the first time at Fri Jul 25 23:19, believe it or not, the machine got hacked at Sat Jul 26 00:50. A mere 90 minutes into the air, and the machine got hacked. The funny thing was that at Sat Jul 26 03:09 it got hacked again to the same account, then at Sat Jul 26 03:21, which also closed the root access via SSH at this point. Following below is the last log:

root     pts/0    Sat Jul 26 06:04   still logged in
reboot   system boot  2.6.18-53.1.14.e Sat Jul 26 06:02          (00:17)
root     pts/1    Sat Jul 26 03:21 - 03:24  (00:03)
root     pts/0    Sat Jul 26 03:09 - 05:20  (02:11)
root     pts/1     Sat Jul 26 00:50 - 00:50  (00:00)
root     pts/0    Fri Jul 25 23:24 - 01:39  (02:14)
root     tty1                          Fri Jul 25 23:22 - 23:24  (00:01)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 23:19          (07:00)
root     tty1                          Fri Jul 25 22:14 - down   (01:03)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 21:58          (01:19)

I admit it, putting a machine on the open net, with a root password of 123456 and open root access to SSH – that’s kind of a honey pot the size of the grand canyon. But what amazed me here was not the speed, but actually the locations of the hacks:, and One hacker is in China, the other in Romania and the third in the UK. What is this? a real hacker? maybe 3 different robots scanning? – I can’t really tell here. However, the traces they left were interesting enough – which lead me to believe we’re talking about robot hacking.

First off, a look at /var/log/audit/audit.log immediately showed the logins – the hacker didn’t even remove the log file – marking of a script kiddie running an automated script. So, what did they leave on my box, let’s take a look. Running ‘netstat -apn | less’ would show me open ports, unless netstat was replaced. However, lets start with this:

tcp        0      0           ESTABLISHED 2940/crond
tcp        0      1           SYN_SENT    2940/crond
tcp        0      1         SYN_SENT    2940/crond

Ok, so this is most probably an IRC bot waiting for instructions from the hacker – till now nothing special. The script tries to masquerade the bot with a legitimate process name: crond. Well, that may fool a beginner Linux Sysadmin, however, seeing crond connecting to 3 other hosts at TCP 6667 – ok, that’s kind’a lame – no?

I wonder where he hid the script? maybe he replaced crond?

root@pbx:~ $ find / -name "crond"
root@pbx:~ $

Hmm… /var/tmp/.www/crond looks promising, let’s see what’s in there:

root@pbx:~ $ ls -la /var/tmp/
total 24
drwxrwxrwt  4 root root 4096 Jul 26  2008 .
drwxr-xr-x 25 root root 4096 Jul 25  2008 ..
drwxr-xr-x  2 root root 4096 Jun 27 17:03 .spd
drwxr-xr-x  4  501  502 4096 Jul 26  2008 .www

Yummy! Let’s check it out:

root@pbx:/var/tmp $ ll .spd/
total 1316
-rwxr-xr-x 1 root root    265 Nov 19  2005
-rwxr-xr-x 1 root root     72 Jun 26 19:43 pass_file
-rwxr-xr-x 1 root root  21407 Nov 19  2005 pscan2
-rwxr-xr-x 1 root root    218 Jun 27 16:59 s
-rwxr-xr-x 1 root root 453972 Nov 19  2005 ss
-rwxr-xr-x 1 root root 842736 Jun 26 19:20 ssh-scan
-rwxr-xr-x 1 root root    312 Jun 27 17:02 x
root@pbx:/var/tmp $ ll .www/
total 888
-rwxr-xr-x 1  501  502    353 Jul 26  2008 1.user
-rwxr-xr-x 1  501  502    349 Jul 26  2008 2.user
-rwxr-xr-x 1  501  502    353 Mar 14  2009 3.user
-rwxr-xr-x 1  501  502    317 Nov  6  2007 autorun
-rw-r--r-- 1 root root      0 Jul 26  2008 belgian.seen
-rwxr-xr-x 1  501  502    942 May 15  2003 checkmech
-rwxr-xr-x 1  501  502  23237 May 15  2003 configure
-rwxr-xr-x 1  501  502 492135 Mar  4  2005 crond
-rwxr-xr-x 1  501  502     48 Jul 26  2008 cron.d
-rwxr-xr-x 1  501  502    171 Jul 26  2008 cutitas
-rwxr-xr-x 1  501  502   4147 May 15  2003 genuser
-rwxr-xr-x 1  501  502    157 Jul 25 17:36 LinkEvents
-rwxr-xr-x 1  501  502      0 Oct 15  2007 lucifer.seen
-rwxr-xr-x 1  501  502   2154 May 15  2003 Makefile
-rwxr-xr-x 1  501  502     14 Jul 26  2008 m.dir
-rwxr-xr-x 1  501  502  22882 May 15  2003
-rwxr-xr-x 1  501  502    748 May 15  2003 mkindex
-rwxr-xr-x 1  501  502   1043 Jul 26  2008 m.lev
-rwxr-xr-x 1  501  502      5 Jul 25 17:35
-rwxr-xr-x 1  501  502   1068 Jul 26  2008
-rwxr-xr-x 1  501  502   1675 Mar 25  2009 m.set
-rwxr-xr-x 1  501  502 167964 Mar 16  2001 pico
-rwxr-xr-x 1  501  502  84476 Jun 23  2006 pico.tgz
drwxr-xr-x 2  501  502   4096 Jul 23 15:48 r
-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{700.seen
-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{800.seen
-rwxr-xr-x 1  501  502    715 Jul 12 22:00 shadow}{900.seen
drwxr-xr-x 2  501  502   4096 Jul 23 15:51 src
-rw-r--r-- 1 root root   1842 Jul 26  2008 zak.seen

Looks like .spd is the SSH scanner and the .www directory contains the actual bot binary – ok, I can respect that. The contents of the cron.d file suggested that the script utilizes crontab to verify that the bot is always up and running – and examination of its code assured me of that.

So, what have we learned from the above: just one thing! When installing a server for the first time, DON’T USE A SILLY PASSWORD LIKE 123456 – EVEN NOT FOR THE INSTALLATION PHASE! Scanning robots appear to be scanning the entire Internet over and over and over again, doing so in seconds – so by the time you install your server, set it up completely, there is a good chance it will already be compromised.

Lately I’ve come to the realization, that we are to blame for our own inability to promote Open Source and the adaptation of Open Source proficiency. Being an Open Source evangelist and consultant, this is very weird to be said by one like myself, however, this is my realization – and I will explain.

In the early days of Open Source adaptations (late 90’s, early 2000), Open Source software was a somewhat magical solution that meant: pay nothing, get more. Software packages like Linux, Apache, mySQL, PostgreSQL and programming languages like PERL and PHP had lowered the bar on the adaptation of new technologies, and enabled a prolific number of solutions and services.

I still remember the early days, when a Windows based Mail Relay would cost anything between 800$ to 1200$, and I would come in with a Linux based solution that would do the same thing for FREE – amazing. As time progressed, so did the technology and the penetration of Open Source into new fields. CRM, ERP, Telecoms, management – all of these now enjoy a diverse number of Open Source solutions. However, the original concept of ‘Open Source = Magical FREE Solution’ had still remained in the minds of managers and business people.

Today we are confronted with ‘would-be’ Open Source solution experts, which adopt and develop upon Open Source products and project various applications. In example, let’s take a look at Asterisk. Asterisk has a multitude of Open Source solutions, ranging from PBX system, Prepaid calling cards, Wholesale routing platforms, Attendance system, Presence systems – and even a plant watering solution. The problem with this ever growing number of solutions is that Asterisk is immediately considered to be: “A magical solution” capable of solving any problem – when it’s not even remotely related to Asterisk. For example, a friend of mine had been asked to develop an Asterisk based solution, that would support a total of 250 concurrent call initiations and up-to 3000 concurrent calls on the system. Any Asterisk developer would take a look at this, and would immediately say: “Hmmm…. this requires several servers, but hey, what about the application itself? that would also have an impact”. Now, the customer of the project has a ‘would-be’ Asterisk tech in his company which said: “I was able to initiate 200 concurrent SIP invites to Asterisk via SIPP, no problem’ – HELLO! STUPID! where’s the application? where’s the database? where’s the user information flow? comm’on, are you listening to yourself speak? or simply are filled with the gasses coming out of your ass that are affecting your brain?

Now, once the customer learns that Asterisk is most probably not the right solution for the problem, he becomes angry. Why? because he now learns that he needs to spend about 10 times more money than he anticipated for the creation of this tool – well, that’s life when you have no idea what you are doing/saying, and you believe in magical solutions. However, we – “The Open Source Community – is the one to blame for this scenario, because we got the world accustomed to the idea that Open Source is like magic – flip the Linux magic wand, and the rest will solve itself.

I’d like to open the floor for discussion on this, as I believe most of you will have something to say about this.

Ok, I admit it, the topic sounds ultra geeky and nurdy – but I can’t help it, there is something about booting up your computer from a USB pen drive, having all your nicely wrapped tools in there and having fun with it.

In this case, my pen drive is actually the driving force behind an extremely powerful call recording system, based on the Asterisk Open Source PBX system. Essentially, the Cruzer boots up a CentOS 5.1 system, fully equipped with an Asterisk + Zaptel + LibPRI + FreePBX. The system is configured to utilize up to 12 E1 circuits, with auto sensing scripts that will automatically configure your system upon first boot-up. Once the system had booted up, it will start identifying your hardware hard drives, and will start cataloging to these hard drives all the recordings according to the pre-determined logic.

I currently use a MySQL database on the Pen Drive to store catalog information only, which is working nicely – but I need to figure out a better way to store more information – 2GB of MySQL storage may be enough for a short while, but serving a large contact center won’t be much of a good idea – I think.

The Pen Drive was created using tools from, which contains wonderful information about how to create your own custom Linux based Pen Drive – Excellent!