A little security experiment…

Back in the year 1999, long before I started my Asterisk days, I spent most of my time as a security consultant and cyber forensics expert. I remember that in those days, most of the hacks were script kiddies exploiting some Windows IIS well known hole, and you would usually get the “Hacked by Chinese” black display on your website – how annoying!

In any case, I’ve recently replaced my co-location firewall. I’ve migrated from a Linux system running IPtables with a manual script, to a fully blown IPCOP installation. Ok, so IPCOP is nothing more than a fancy GUI for IPtables, but hey, it makes my life a whole lot easier on the management side – and it’s very stable – so who am I to complain?

I’ve decided to run a small experiment, I wanted to setup a Linux box, with a root password of 123456. My question was this, how much time will pass from the moment the machine was up, on a new IP address, till the machine gets hacked – and more importantly, from where and what got installed on the machine?

So, the machine fired up for the first time at Fri Jul 25 23:19, believe it or not, the machine got hacked at Sat Jul 26 00:50. A mere 90 minutes into the air, and the machine got hacked. The funny thing was that at Sat Jul 26 03:09 it got hacked again to the same account, then at Sat Jul 26 03:21, which also closed the root access via SSH at this point. Following below is the last log:

root     pts/0        77.127.137.52    Sat Jul 26 06:04   still logged in
reboot   system boot  2.6.18-53.1.14.e Sat Jul 26 06:02          (00:17)
root     pts/1        92.80.195.126    Sat Jul 26 03:21 - 03:24  (00:03)
root     pts/0        78.110.163.31    Sat Jul 26 03:09 - 05:20  (02:11)
root     pts/1        60.220.240.7     Sat Jul 26 00:50 - 00:50  (00:00)
root     pts/0        77.127.137.52    Fri Jul 25 23:24 - 01:39  (02:14)
root     tty1                          Fri Jul 25 23:22 - 23:24  (00:01)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 23:19          (07:00)
root     tty1                          Fri Jul 25 22:14 - down   (01:03)
reboot   system boot  2.6.18-53.1.14.e Fri Jul 25 21:58          (01:19)

I admit it, putting a machine on the open net, with a root password of 123456 and open root access to SSH – that’s kind of a honey pot the size of the grand canyon. But what amazed me here was not the speed, but actually the locations of the hacks: 60.220.240.7, 78.110.163.31 and 92.80.195.126. One hacker is in China, the other in Romania and the third in the UK. What is this? a real hacker? maybe 3 different robots scanning? – I can’t really tell here. However, the traces they left were interesting enough – which lead me to believe we’re talking about robot hacking.

First off, a look at /var/log/audit/audit.log immediately showed the logins – the hacker didn’t even remove the log file – marking of a script kiddie running an automated script. So, what did they leave on my box, let’s take a look. Running ‘netstat -apn | less’ would show me open ports, unless netstat was replaced. However, lets start with this:

tcp        0      0 172.31.31.16:34183          195.47.220.2:6667           ESTABLISHED 2940/crond
tcp        0      1 172.31.31.16:57263          195.54.102.4:6667           SYN_SENT    2940/crond
tcp        0      1 172.31.31.16:46043          195.68.221.221:6667         SYN_SENT    2940/crond

Ok, so this is most probably an IRC bot waiting for instructions from the hacker – till now nothing special. The script tries to masquerade the bot with a legitimate process name: crond. Well, that may fool a beginner Linux Sysadmin, however, seeing crond connecting to 3 other hosts at TCP 6667 – ok, that’s kind’a lame – no?

I wonder where he hid the script? maybe he replaced crond?

root@pbx:~ $ find / -name "crond"
/usr/sbin/crond
/var/tmp/.www/crond
/var/lock/subsys/crond
/etc/sysconfig/crond
/etc/rc.d/init.d/crond
/etc/pam.d/crond
root@pbx:~ $

Hmm… /var/tmp/.www/crond looks promising, let’s see what’s in there:

root@pbx:~ $ ls -la /var/tmp/
total 24
drwxrwxrwt  4 root root 4096 Jul 26  2008 .
drwxr-xr-x 25 root root 4096 Jul 25  2008 ..
drwxr-xr-x  2 root root 4096 Jun 27 17:03 .spd
drwxr-xr-x  4  501  502 4096 Jul 26  2008 .www

Yummy! Let’s check it out:

root@pbx:/var/tmp $ ll .spd/
total 1316
-rwxr-xr-x 1 root root    265 Nov 19  2005 gen-pass.sh
-rwxr-xr-x 1 root root     72 Jun 26 19:43 pass_file
-rwxr-xr-x 1 root root  21407 Nov 19  2005 pscan2
-rwxr-xr-x 1 root root    218 Jun 27 16:59 s
-rwxr-xr-x 1 root root 453972 Nov 19  2005 ss
-rwxr-xr-x 1 root root 842736 Jun 26 19:20 ssh-scan
-rwxr-xr-x 1 root root    312 Jun 27 17:02 x
root@pbx:/var/tmp $ ll .www/
total 888
-rwxr-xr-x 1  501  502    353 Jul 26  2008 1.user
-rwxr-xr-x 1  501  502    349 Jul 26  2008 2.user
-rwxr-xr-x 1  501  502    353 Mar 14  2009 3.user
-rwxr-xr-x 1  501  502    317 Nov  6  2007 autorun
-rw-r--r-- 1 root root      0 Jul 26  2008 belgian.seen
-rwxr-xr-x 1  501  502    942 May 15  2003 checkmech
-rwxr-xr-x 1  501  502  23237 May 15  2003 configure
-rwxr-xr-x 1  501  502 492135 Mar  4  2005 crond
-rwxr-xr-x 1  501  502     48 Jul 26  2008 cron.d
-rwxr-xr-x 1  501  502    171 Jul 26  2008 cutitas
-rwxr-xr-x 1  501  502   4147 May 15  2003 genuser
-rwxr-xr-x 1  501  502    157 Jul 25 17:36 LinkEvents
-rwxr-xr-x 1  501  502      0 Oct 15  2007 lucifer.seen
-rwxr-xr-x 1  501  502   2154 May 15  2003 Makefile
-rwxr-xr-x 1  501  502     14 Jul 26  2008 m.dir
-rwxr-xr-x 1  501  502  22882 May 15  2003 m.help
-rwxr-xr-x 1  501  502    748 May 15  2003 mkindex
-rwxr-xr-x 1  501  502   1043 Jul 26  2008 m.lev
-rwxr-xr-x 1  501  502      5 Jul 25 17:35 m.pid
-rwxr-xr-x 1  501  502   1068 Jul 26  2008 m.ses
-rwxr-xr-x 1  501  502   1675 Mar 25  2009 m.set
-rwxr-xr-x 1  501  502 167964 Mar 16  2001 pico
-rwxr-xr-x 1  501  502  84476 Jun 23  2006 pico.tgz
drwxr-xr-x 2  501  502   4096 Jul 23 15:48 r
-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{700.seen
-rwxr-xr-x 1  501  502    661 Jul 12 22:00 shadow}{800.seen
-rwxr-xr-x 1  501  502    715 Jul 12 22:00 shadow}{900.seen
drwxr-xr-x 2  501  502   4096 Jul 23 15:51 src
-rw-r--r-- 1 root root   1842 Jul 26  2008 zak.seen

Looks like .spd is the SSH scanner and the .www directory contains the actual bot binary – ok, I can respect that. The contents of the cron.d file suggested that the script utilizes crontab to verify that the bot is always up and running – and examination of its code assured me of that.

So, what have we learned from the above: just one thing! When installing a server for the first time, DON’T USE A SILLY PASSWORD LIKE 123456 – EVEN NOT FOR THE INSTALLATION PHASE! Scanning robots appear to be scanning the entire Internet over and over and over again, doing so in seconds – so by the time you install your server, set it up completely, there is a good chance it will already be compromised.

We are to blame…

Lately I’ve come to the realization, that we are to blame for our own inability to promote Open Source and the adaptation of Open Source proficiency. Being an Open Source evangelist and consultant, this is very weird to be said by one like myself, however, this is my realization – and I will explain.

In the early days of Open Source adaptations (late 90’s, early 2000), Open Source software was a somewhat magical solution that meant: pay nothing, get more. Software packages like Linux, Apache, mySQL, PostgreSQL and programming languages like PERL and PHP had lowered the bar on the adaptation of new technologies, and enabled a prolific number of solutions and services.

I still remember the early days, when a Windows based Mail Relay would cost anything between 800$ to 1200$, and I would come in with a Linux based solution that would do the same thing for FREE – amazing. As time progressed, so did the technology and the penetration of Open Source into new fields. CRM, ERP, Telecoms, management – all of these now enjoy a diverse number of Open Source solutions. However, the original concept of ‘Open Source = Magical FREE Solution’ had still remained in the minds of managers and business people.

Today we are confronted with ‘would-be’ Open Source solution experts, which adopt and develop upon Open Source products and project various applications. In example, let’s take a look at Asterisk. Asterisk has a multitude of Open Source solutions, ranging from PBX system, Prepaid calling cards, Wholesale routing platforms, Attendance system, Presence systems – and even a plant watering solution. The problem with this ever growing number of solutions is that Asterisk is immediately considered to be: “A magical solution” capable of solving any problem – when it’s not even remotely related to Asterisk. For example, a friend of mine had been asked to develop an Asterisk based solution, that would support a total of 250 concurrent call initiations and up-to 3000 concurrent calls on the system. Any Asterisk developer would take a look at this, and would immediately say: “Hmmm…. this requires several servers, but hey, what about the application itself? that would also have an impact”. Now, the customer of the project has a ‘would-be’ Asterisk tech in his company which said: “I was able to initiate 200 concurrent SIP invites to Asterisk via SIPP, no problem’ – HELLO! STUPID! where’s the application? where’s the database? where’s the user information flow? comm’on, are you listening to yourself speak? or simply are filled with the gasses coming out of your ass that are affecting your brain?

Now, once the customer learns that Asterisk is most probably not the right solution for the problem, he becomes angry. Why? because he now learns that he needs to spend about 10 times more money than he anticipated for the creation of this tool – well, that’s life when you have no idea what you are doing/saying, and you believe in magical solutions. However, we – “The Open Source Community – is the one to blame for this scenario, because we got the world accustomed to the idea that Open Source is like magic – flip the Linux magic wand, and the rest will solve itself.

I’d like to open the floor for discussion on this, as I believe most of you will have something to say about this.

SanDisk Cruzer + CentOS 5.1 Live = Let the good times roll

Ok, I admit it, the topic sounds ultra geeky and nurdy – but I can’t help it, there is something about booting up your computer from a USB pen drive, having all your nicely wrapped tools in there and having fun with it.

In this case, my pen drive is actually the driving force behind an extremely powerful call recording system, based on the Asterisk Open Source PBX system. Essentially, the Cruzer boots up a CentOS 5.1 system, fully equipped with an Asterisk + Zaptel + LibPRI + FreePBX. The system is configured to utilize up to 12 E1 circuits, with auto sensing scripts that will automatically configure your system upon first boot-up. Once the system had booted up, it will start identifying your hardware hard drives, and will start cataloging to these hard drives all the recordings according to the pre-determined logic.

I currently use a MySQL database on the Pen Drive to store catalog information only, which is working nicely – but I need to figure out a better way to store more information – 2GB of MySQL storage may be enough for a short while, but serving a large contact center won’t be much of a good idea – I think.

The Pen Drive was created using tools from www.pendrivelinux.com, which contains wonderful information about how to create your own custom Linux based Pen Drive – Excellent!

Open Source business sustainability

While Open Source projects around the world gather up the troops and become recognized for what they are: highly polished, highly effective, extremely economical products – the situation in Israel is fairly different. We’ve all heard about companies like Zimbra (recently acquired by Yahoo), MySQL (recently acquired by SUN) and others, which had struck BIG TIME. However, the situation in Israel differs immensely.

I’ve been invited to participate in a panel at the Garage Geeks, to discuss the various aspects of Open Source sustainability. I’ve made it my business to build a business completely surrounded by Open Source, devoted to the promotion and adaptation of Open Source – and when possible, the promotion of Open Source licensing models and the understanding of what they mean.

In one of my previous posts, I’ve indicated that Open Source projects are highly exploited in an illegal manner in Israel, thus, making Open Source business in Israel a high target for Open Crooks. The question immediately arises, how can an Open Source project become successful? In addition to that, what are the factors that make a good Open Source project a grand Open Source project.

Step 1: Features

For an Open Source project to become popular and frequently used, it should have an extensive range of features, which is constantly being upgraded and enhanced. Taking from my own personal favorite, let’s take a look at Asterisk – the Open Source PBX. Over the course of the past 5 years, Asterisk had evolved to include hundreds of features. Each new feature in an Open Source product expose it to a new market. With Asterisk, the introduction of an Answering machine detection tool had introduced it to the automatic dialer and contact center market. The introduction of LumenVox speech recognition had introduced it to the ASR market, and so on.

While features are important, it is also very important to make sure the features included are features that the community and users require. While it is really cool to have a mod_kitchensink for the Apache web server, no one really uses it.

Step 2: Community

In order for an Open Source product to become successful, it MUST have a vibrant and active community – better yet, more than one. While an active developer community is important for the advancement of the project, a set of auxiliary communities is required. A users community is a must, rendering support and usage ideas to its members. No less important is a business oriented community, one that speaks to the manager level people, those making the decisions in organizations. Tap into that level, and the Open Source project is now gaining followers from other side of the border.

Managers tend to be highly traditional in thinking, not inclined to utilize Open Source at first try. A vibrant business community of the Open Source project can do wonders to the project, especially with its promotion and adaptation into existing and new business structures.

Step 3: Funding and Sustainability

Funding an Open Source project doesn’t entirely mean – MONEY! Well, eventually it does mean money, but not in the normal way we think or work with money. Open Source developers don’t work primarily for the money, the driving force behind Open Source developers is different. Question be: “If Open Source developers aren’t motivated by money, why would you need funding?” – the reason is simple, the surroundings of an Open Source project require funding.

The surroundings of an Open Source project mainly include the following: public events, developer meetings, servers, hosting, travel fares, participating in trade shows and others. All of the above are generally associated with Marketing, however, marketing an Open Source project is sometimes as important as the project itself. If we are to examine the growth of the Linux community and user base in the world, we are mainly thankful to RedHat in its early days (1996-2001), closely followed by Debian with its recent off spring Ubuntu (2006-2008). Imagine, you can now go into an IBM dealer and ask to buy your notebook with Linux, how cool is that? how did that happen? did the world suddenly realise Linux is better than Windows? – the answer is NO! The marketing efforts of these companies had proven worth while, as the concept of using Linux as a desktop became common in recent years.

Step 4: Training and Certification

If your Open Source project is UberGeek targeted only, than you have a very slim chance of making it big. Lowering the bar on the requirements for the adaptation of an Open Source project is highly important and can be mostly achieved by training and certification. The training makes it possible for people to learn more about an Open Source project, while the certification makes the project seem more desirable and exclusive.

Why do people seek M$ and Ci$co certifications? simple, because they know these certifications mean something to manager level people and decision makers. The certification is a written (actually printed) proof that you know what you are talking about and that you are truly a professional working in the field of that Open Source project.

Conclusion:

If all of the above are met, you are surely on your way to create the next big Open Source project – and you are on your way to world fame and rock-star feeling.

Predictive dialers enhance contact center performance – truth or myth?

Since I’ve released my dialer framework demo about 2 months ago, I’ve been swamped with many requests from various contact centers around the world – to utilize my dialer framework for the development of a custom made predictive dialer.

For those of you who are not in the know, a predictive dialer is a tool that is capable of analyzing the performance of each agent in a contact center, accurately predicting when his current call will be completed, and thus, start calling outbound to ensure that the agent is utilized as much as possible.

Most contact center managers believe that if an agent is utilized 100% of the day (or at least a close enough number), they will maximize their profits and work will be done faster. This is not always the case, and there are some cases where predictive dialers will be nothing more than a “White Elephant”, sitting in your call center, doing nothing.

Considering the following scenario: We have a contact center selling computer insurance plans by phone. Each agent is trained to make a sale, that is: “Don’t get off the bloody phone without a credit card!”. One of the issues with such a contact center is that there is no-way of predicting how long a sale will take. Lets imagine that one call a sale happens in 15 minutes, while in the next, we start with the kid in the house, move to the older brother, move to the mother, move to the father, ending up making a sale after 35 minutes. In other words, we have no way of profiling an agent, as there is no proper profile to the customers.

So you can argue that by utilizing statistical models and proper targeting of potential customers, we can go about and perform more accurate predictions. However, these predictions will all go up in flames, the minute a deviation from the norm of the statistic happens. We then immediately create a form of ripple effect, that is then carried across the entire contact center.

In the book “The Goal“, by Eliyahu M. Goldratt, the author tells us a story about a group of boys walking in the woods. The group of boys constantly are unable to walk the path at the designated speed, due to various timing and synchronization issues. In theory, a predictive dialer is used to better synchronize the contact center intake (numbers to be dialed), with the contact center’s ability to perform (the ability to make a sale). However, this model fails when the sale constraint is unknown, thus, making the entire model fail.

In most cases, contact centers are better off using “Preview Dialers” and not “Predictive Dialers”, unless, the contact center is highly targeted with its campaigns and sale strategy. A “sell or die” contact center strategy immediately negates the possibility of accurately measuring the contact center performance and bottle necks, thus, having an automatic pace creator in such a scenario will become redundant and will most probably just cost funds.