(Article originally posted at InfoWorld Magazine)

The 5th annual Southern California Linux Expo (SCALE 5x) was hosted at the LAX Airport Westin this past weekend. It was a relatively small affair, nothing as extravagant as LinuxWorld in San Francisco. The show floor was small and there were no large, fancy booths from big companies. The classes were short one hour sessions that flew by while the speakers presented at an almost frantic pace sometimes. Yet this was a surprisingly good conference. I will absolutely attend again next year.

What made SCALE so special? For starters, they held a special Open Source Health Care Summit all day Friday. Health care is an industry where Open Source Software (OSS) has not yet made big inroads, but the industry would benefit greatly from adoption of Open Source software. We heard a presentation on what is wrong with the healthcare industry and how OSS can help to fix these problems. Fred Trotter told us about Open Source electronic medical record (EMR) applications. There were Open Source Software case studies from a major cancer center in the southeast, as well as a group of government funded clinics in California. And we even saw how a small Linux appliance can help disparate medical applications talk to each other and share data.

Also on Friday, SCALE held a special summit on women in Open Source. One set of statistics claims that while 25% of proprietary software developers are women, only 1.5% of Open Source developers are women! Several organizations are actively trying to make the OSS community more inviting to women.

As I said earlier, the booths were small. Even IBM, one of the gold sponsors of the event, had the largest booth – yet it was 10 feet deep and maybe 25 feet long. But appearances were deceptive here. There were companies present with some serious offerings and they meant business! And these big business players were standing side by side with small Open Source projects and community organizations. Everybody played nicely together as we’ve come to expect from the Open Source community at these events. The small size of this event gave me more face time with the big business tech experts as well as the Open Source community leaders and developers. I learned about new projects and products that impressed me beyond any expectations I might have had.

The one hour seminars/presentations went well and were included in the price of registration. The presentations were pretty fast and furious since they were only one hour long, but most of the presenters were well prepared with slides and notes so the classes were easy to follow and quite informative. All in all, the $70 that I paid for the Open Source Health Care Summit and two days of presentations and show floor access was money well spent. Seriously, this conference was a complete steal at $70. So I’m sorry to hear that you missed the event. I hope that you sign up early for next year’s show!

High Mobley
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

Have you ever received a word document in email, only to find that it cannot be opened or edited by your version of the MS Office? Or perhaps you want to open a paper you wrote in 1996, only to find that you wrote it with Nisus on Mac OS 7, and you are now running Windows XP? Well, you can say goodbye to those days. It’s time to take a look at the format of the future: Open Document Format (ODF).

ODF is a new ISO standard (ISO/IEC 26300:2006) that has already been adopted by several countries including Malaysia, Italy, and Belgium (and the state of Massachusetts). It is also backed by corporations such as IBM, Sun, Google, Red Hat, Novell, and Oracle. It is supported by applications such as IBM’s Workplace, Sun’s Star Office, the open source Open Office and KOffice suite, Mobile Office for your smart phones, and NeoOffice support for Mac is on the way. Heck, even Google Docs supports it, so when you receive a document as an email attachment, you can just open it up and edit it right there in your gmail! It would seem that everyone supports ODF, but the giant itself, Microsoft.

Until 4 days ago.

It was announced on 2007/02/02 (on sourceforge, no less), that the OpenXML Translator 1.0 is available for download. This enables users to use ODF in Office XP to Office 2007. There is also a proof-of-concept plugin for Microsoft Word 97 – 2007 that has been released by the Open Document Foundation.

ODF is not some distant, futuristic standard, it’s something you can start using right this moment. It not only makes document sharing a whole lot easier, it also ensures what you write today, will still be accessible ten years from now.

Josh Kuo
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

When I build a network or a system, I try my best to make sure that everything is as redundant as possible: redundant power supplies, RAID for the drives in case of a hard drive failure, backup routes in OSPF in case someone trips over the network cable… you get the idea. But what happens if the CPU overheats in one of the web servers and causes it to crash? Or what if someone yanks the network cable from your LDAP server? Or if someone flips the switch and accidentally turns off the accounting database server? If you incorporate High Availability (HA) into your system design, the answer is “nothing”. Your web site will still be running, your network users can still login, and the accounting department won’t notice any glitch. You don’t even have to leave your desk.

Open Source makes HA easy to implement, with offerings like heartbeat, keepalived, and CARP.

Let’s say you have mostly static web content for your web server and you can fit everything on a 2GB compact flash card. Then you can build two solid state machines using the Debian Router Project. Using simple solid state hardware means less moving parts and less likelihood of a hardware failure. Then you can use heartbeat to create your HA web server cluster. If you have content which changes more frequently, like the leases file for a DHCP server, a database, or a file server, then you should look into using DRBD to synchronize the two file systems.

Heartbeat requires you to setup a private link for the two machines (nodes) to communicate, so they know the other node is still alive. While you can just use a crossover cable to connect the two nodes I would strongly recommend that you install two network cards in each node and setup a private VLAN or network just for the heartbeat communication. This will give you a little more flexibility later. You will need 5 IP addresses total, two for the private heartbeat link, two public ips – one for each of the nodes (if you wish to manage them remotely), and one more public ip for a “virtual” IP address that is held up by the heartbeat software. This virtual ip address is the IP address your users visit. (By the way, heartbeat supports IPv6)

Once you have heartbeat configured on both nodes and have designated one of nodes to be the master, the two will start “pinging” each other over the private link. Now to see it in action: Start a ping to the “virtual” IP address, and unplug the network cable for the master node or just shut it down to simulate a disaster. You should lose a few pings, but in just a few seconds, the backup node will realize that the master is no longer responding, and will take over the virtual ip address and reply to your pings. This means, if one of the nodes failed in production environment, users will only experience seconds of outage, instead of minutes, or dare I say, hours.

Now, if you have followed my advice about putting the heartbeat link on its own VLAN instead of just hooking it up with a crossover cable, you have the flexibility to move the backup server to a different location in the building (or however far you VLAN will reach). Why? This protects you from a bigger scale of disaster, say, a power outage for the entire room, fire, or flood (hey, I’ve seen it happen). If you have the two heartbeat hosts separated physically, you stand a better chance of surviving the disaster. Plugging both machines into the same network switch creates another single point of failure, so it is highly recommended that your backup machine be connected to a different network switch, and preferably a different power grid.

keepalived uses VRRP (Virtual Router Redundancy Protocol), a widely supported protocol amongst routers. This means it can be integrated nicely into your existing network infrastructure. keepalived was originally designed to work for multiple routers, and it works pretty much the same way heartbeat does, except keepalived does not need a dedicated private link, and it is easier to setup more than two nodes. (It is unclear whether or not keepalived currently supports IPv6)

So far you’ve achieved automatic fail-over. But don’t you feel that all these back up nodes sitting around is a bit of a waste? Can you leverage all those idle computing power? You mean you want load balancing on top of your HA functionality? Open Source answers with CARP (Common Address Redundancy Protocol). The OpenBSD team released CARP in 2003 as a replacement and enhancement to VRRP, it features:

• Secure “heartbeat” communication
• No need for dedicated, private link for “heartbeat” communication
• Basic load balancing functionality
• Supports IPv6
• Available for OpenBSD, NetBSD, FreeBSD, and Linux (implemented as ucarp)

You can also combine CARP with pfsync (OpenBSD’s packet filter), and now you can build a cluster of firewalls/routers that are always online, load balances amongst each node, and in case of a failure, users do not lose any sessions or states.

In conclusion, heartbeat (along with DRBD) is the easiest to setup for a 2-node cluster, Keepalived integrates well into your VRRP environment, and CARP brings security and load balancing to the table. In case you are wondering how mature this technology is, heartbeat has been around for years, and has a list of success stories.

Josh Kuo
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

Security is a topic that is getting more and more attention these days, and encryption plays a large role in security. However, those of us who have played with encryption know that it consumes a significant amount of system resources. If you are doing your encryption in software, you are most likely playing a catch-up game to your network speed (when encryption network traffic) and storage volume (when encrypting file system).

The tradition approach is to get an encryption card and drop it into your PCI slot. But have you checked out encryption built directly into the CPU? This is not exactly news, since VIA Technologies has been making CPU’s with encryption built-in since 2004. VIA processors with PadlLock has SHA1-256 (Secure Hashing Algorithm), AES (Advanced Encryption Standard), and random number generator all built into the hardware.

So how fast is hardware encryption? In this benchmark, you can see that a 1.2GHz VIA processor can encrypt about 5 to 16 times faster than a Pentium IV 2.4GHz. And in this benchmark, where the author tests against encrypted file system and IPSec connections, there is almost no slow down when doing IPSec with PadLock, and you only lose about 10% of performance when writing to encrypted file system. Compare that to software encryption where you are looking at roughly 50% to 80% loss in performance.

Josh Kuo
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

Sure you have one. Everyone nowadays has at least one wireless router at home, be it Linksys, NetGear, D-Link, or Buffalo. With new wireless products being released nearly every month, I am willing to bet that some of you even have a couple of the older wireless routers collecting dust in your closet. Well, it’s time to take them out and put them to good use.

Check out the OpenWRT project. OpenWRT is a Linux distribution for embedded devices, and it brings a lot of exciting possibilities to your humble wireless router. Although still in its release candidate stage (currently at RC6), OpenWRT is very usable and feature-rich right out of the box. Be warned, you could void your manufacturer warranty by installing OpenWRT on your wireless routers.

So what can you do with an embedded Linux device running on limited RAM and very small storage? As it turns out, quite a lot actually. You can install asterisk, and have your personal, customizable PBX (private branch exchange). If you already have a SIP phone or some kind of VoIP phone interface (such as the Cisco ATA 186 adapter), you can have your very own VoIP system at home, all running out of your low power-consumption embedded hardware.

Put your router/firewall on steroids by installing packages like nmap (network security scanner), snort (intrusion detection), and tcpdump (packet sniffer). Together with iptables (which comes with the Linux kernel), you can turn your OpenWRT box into a powerful security tool. Install openvpn, and you have a very affordable VPN device. And if it strikes your fancy, you can install quagga and turn your dusty little Linksys into an OSPF and BGP-capable router.

Want to provide your own wireless hotspot? No problem. Install chillispot, and you are ready to go. You can even install FreeRADIUS on the OpenWRT for the authentication back-end, and WPA (wifi protected access) for the added security.

You can turn it into an all purpose office server by installing DHCP, cups (print server), lighthttpd (web server), NTP (time server) and OpenSSH or dropbear (secure remote administration). If your router has a USB port, you can also turn it into a file server by hooking it up with a USB hard drive and installing NFS.

And don’t forget that this is a wireless router. It has a wireless card, so take advantage of it! Install kismet on it, and you have a wireless sniffer. This can prove to be invaluable if you ever need to analyze the airwaves at a remote location, but don’t want to leave your expensive laptop on-site. Drop in place a $50 OpenWRT box loaded with kismet instead.

Here is one way to use your old wireless router: In the past, I had setup a few cheap Linksys WRT54g boxes with OpenWRT and vtun, and dropped one at each of our remote locations. This gave me the ability to have layer 2 tunnels to each of the remote sites. I kept one in my house, and if I ever needed to troubleshoot a remote network problem, I just setup the tunnel between the two OpenWRT boxes, connected my laptop or testing equipment to the OpenWRT sitting on my desk, and it was like being on the remote physical network! This saved me a number of times, being able to perform packet capturing on the remote network, observing the network traffic in real-time, requesting and obtaining DHCP addresses… essentially, I could experience exactly what the remote user was experiencing, all from the comfort of my own home.
This is just the beginning of what embedded Linux can do for you. To find out more what embedded Linux can do fo r your enterprise, check out Secure Linux Appliances in Your Enterprise. So dig up your old wireless router, check it against the hardware compatibility list, and see if your router is OpenWRT compatible, and open yourself up to a wrt of possibilities!

Josh Kuo
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

By now you’ve either seen them or read about them. Companies are selling all kinds of useful appliances based on embedded Linux. Some are for small tasks like wireless APs, mobile devices, or cell phones. Others are geared towards enterprise needs like load balancers, routers, and NAS (network attached storage) and SANs (storage attached network). They all run some version of Linux or BSD. You know you have a couple of Linux geeks working for you in the IT department. Why aren’t they coming up with some of these cool Linux appliances for your own company to use? The excellent Debian Router project by Vadim Berkgaut is the help that your Linux admins need to develop their very own Linux appliances.

At my company, q!Bang Solutions, we provide all types of IT solutions, but our strong suit is our solutions built upon Open Source software. Our employees have used the Debian Router Project (which we refer to as “DebRouter”) to build numerous solutions, including firewalls, OSPF and BGP routers, DNS servers, and even VoIP servers. DebRouter is a cornerstone of our technology solutions.

What’s great about DebRouter is that you get a fully functional Debian Linux installation. So you can add whatever software packages you want to extend the functionality of the DebRouter. This is implemented through the usual Debian package management utilities, which means that you can change a DebRouter’s functionality on the fly and in the field after it’s been deployed.

Another important feature of DebRouter is that it boots from a flash device like a compact flash card (via an IDE adapter) or a USB flash drive. So if there are any problems with changes you’ve made, a reboot takes you back to the previous known-good version of your running system. Does this mean that you lose changes you’ve made when power to the DebRouter goes out? No. DebRouter implements a “write to flash” function much like a hardware router or manageable switch. So you can install and configure new packages, test them out, and write your changes to the flash-based boot media if everything went well in testing. If your tests revealed there was a problem, then just reboot without writing the changes to flash and you will roll back to the same state of the filesystem that you had before your changes. This makes it extremely easy to test potentially unstable software and configuration changes. If things don’t work, just reboot, and voila! Your working system is back within seconds.

This also means that the machines are harder for crackers to abuse if they succeed in infiltrating the DebRouter. If you discover that your DebRouter has been compromised, you can reboot and be rid of the cracker. Then you check for security updates from Debian, install them, write your changes, and you’re back up and running. I can tell you from experience that eradicating a cracker’s presence from a normal machine with hard drives whose data persists across reboots is not this easy!

The boot process of the DebRouter provides another nice benefit. DebRouter boots from flash media, creates a RAM disk, copies the flash media’s filesystem to the RAM disk and then unmounts the flash media filesystem and runs from the RAM disk. RAM is fast – lot faster than any hard drive. So now your filesystem I/O speed is absurdly fast. So if you install the Apache web server and put up some HTML and image files, you now have one of the fastest web servers available – without the hassle of a special configuration to load your pages into a ramdisk. It can also run web scripts (such as PHP, Perl, Python, Ruby, etc.) as fast as your normal hard drive based servers do.

What can you build with a DebRouter? Here are a few ideas to get you started:

• Add the Quagga routing software package to make an OSPF/RIP/BGP router
• Install the Apache web server with Perl/PHP/Python/etc scripting environments
• Use the Asterisk software for a cheap VoIP server for a remote office
• NAT/Firewall
• Web content filtering via the Squid proxy package
• Make a captive portal system for wireless networks in cafes or other public access areas
• DNS server using the venerable and always popular BIND software
• Create a network sniffer with the tcpdump utility which writes data to a remote NAS or other storage device
• Combined with a NAS (Network Attached Storage) or an NFS server, a DebRouter can do most anything.

Since most enterprises will try to install all machines in racks, I checked a couple of online vendors to see how much it would cost to build a good 1RU DebRouter machine. I found that a 1RU machine far above the minimum specs can be had for $500, including shipping. This includes a 1RU case, motherboard with all essential functionality on board, a P4 2.8GHz CPU, 1GB ram, and a 512MB CF card and IDE-based CF reader.

So how about a $500 router that can do RIP/OSPF/BGP? Consider both the business and technology reasons that your company might want to use a DebRouter instead of a router from Cisco or one of the other routing big boys. The business side is easy. The hardware is cheap, even for a system with generous amounts of RAM and CPU. For the price of a typical router support contract, you can buy a couple of extra DebRouters to have sitting around as spares ready to jump into action if you have a hardware failure on your primary DebRouter. Subsequent years of support contracts you don’t need to buy equal money that remains in your coffers helping to fatten up your Christmas bonus next year. Of course, let’s not forget that most router vendors charge extra for the advanced software like OSPF or BGP routing, or encryption software so that you can use the more secure SSH instead of the gaping security hole called Telnet to remotely connect to your router. DebRouter has all that (and so much more) for free!

On the technology side, with the screaming fast processors available today, a DebRouter can pretty well hold its own against most of the major router vendors’ offerings. And it’s the versatility of the DebRouter that will likely interest your techies. Did I mention that Linux does 802.1q VLANs? How about an OSPF router that does double duty as a slave DNS server? Or perhaps an edge router that also acts as a VPN concentrator with strong encryption for hundreds of tunnels?

So walk on down to IT and find those two Linux guys tucked away in their cubicles and let them loose on a Debian Router project. They should be glad to have an interesting project to work on instead of trying to recover emails that Marge from Accounting accidentally deleted the other day, and you just might get some nifty devices from them that save you some cash on your bottom line. Your Linux admins are welcome to reach out to me if they need some help or just want to share their ideas on a new use for a Debian Router.

In the future, I’ll touch on embedded Linux in extremely cheap devices that are excellent for smaller tasks.
[My q!Bang Solutions co-owner Josh Kuo beat me to the punch. Read his article "Beef Up Your Wireless Router".]

High Mobley
Co-Owner of q!Bang Solutions

(Article originally posted at InfoWorld Magazine)

Have you checked out Google’s Analytics package yet? No? Why not? It’s a strong web analytics package and is offered for free from Google.

Let’s first address the definition of “web analytics.” Wikipedia offers the following explanation which fits the parameters of this article quite well:

Web analytics is the measurement of the behaviour of visitors to a website. In a commercial context, it especially refers to the measurement of which aspects of the website work towards the business objectives; for example, which landing pages encourage people to make a purchase.

Google Analytics is not a web log file analyzer – which is a good thing. Log file analyzers are dependent upon the web server to execute the analyzer scripts on a regular basis and can get a little resource intensive for a busy site. Plus, what happens if you lose those log files due to a disk error or filesystem corruption before they are analyzed and put into the web statistics database? And what good is your log file analyzer data when you’re moving to a new server platform? You would most likely have to start from scratch with your data collection.

Enter Google Analytics. It works based on small snippets of code embedded in your web pages which cause the user’s browser to call a script on Google’s servers which culls the pertinent information from the user’s browser. So web analytics doesn’t take place on your servers or use your bandwidth! There is nothing for the IT staff to monitor or maintain.

Just in case anyone is entertaining thoughts of massive Google conspiracy theories, don’t fret! The data which is being noted by Google Analytics is the same data that your web browser freely and happily gives up every time it hits any web site. This includes things like what type of web browser you’re using, which operating system your computer uses, etc. It’s pretty innocuous stuff, and every other web site that you visit gets the exact same information from your browser, so Google’s not doing anything nasty.

Don’t think that a free analytics package doesn’t come with serious features. In addition to the standard statistics you would expect from a good web log file analyzer, Google Analytics provides you with the ability to view trends over time with user-definable date ranges. For your marketing department, Google Analytics has user-defined goals which are reported separately. You can also define the “funnel” or chain of URLs that the user is expected to follow to reach the goal URL. This enables you to track the effectiveness of your marketing campaigns individually and see which ones are really paying off.

And if that feature sounds attractive, then you will like the fact that Google has integrated its AdWords advertising program with the Analytics program. Your AdWords keywords are automatically imported into your Analytics account. And from within the AdWords interface, you can see ROI and other metrics for each keyword you bought on AdWords. Google Analytics plays nice with the competition too. The keyword campaign comparison reports show all your keywords from all the search engines.

Like any good analytics package, Google Analytics will track a user’s navigation through your web site. However, Google’s package has an additional feature that I expect many people will like. You can view an overlay of your site. For each clickable link on your web page, you will see a small bar graph representation of how many clicks that particular link gets. The longer the bar, the more clicks that particular link got during the time period for which you are viewing results. Sure, it’s kind of eye candy, but some people work better with visual representations, and here they have it. Speaking of eye candy, I’m partial to the Geo Targeting feature which shows a world map and places colored dots based on where your web traffic is coming from. The dots get bigger for a region which has more traffic coming to your site.

Google Analytics has a lot to offer. It’s packed with useful features, and it’s free. Well… kind of free. You get up to 5 million page views per month. That’s a lot of page views though, and if your site will go over the 5 million views per month, then all you have to do is open an AdWords account to get unlimited page views for your Google Analytics. It’s still a darned cheap option. And if your site gets that much traffic, you could pay for the AdWords account by putting up Google’s AdSense advertisements on your busy site, but that’s another article…

High Mobley
Co-Owner of q!Bang Solutions

(Originally posted on InfoWorld Magazine)

It is often cited that the biggest issue in the fight against worms and viruses and other such malware is uneducated users. If a person doesn’t understand why it’s a bad thing to open email attachments from people that he doesn’t know, then you can bet that he will open every attachment which comes to him. Several email clients (not just MS Outlook!) will happily open and execute any Visual Basic or batch file that a user clicks on. Then wham! – You’ve got an infected machine that’s probably already calling home to the nasty individual who wrote the malware and now “owns” the user’s computer – which you as the IT department have to go and fix…

Of course the various network security and bug tracking sites are great about announcing the security flaws and exploits that are found, but arguably their audience is only people who are already pretty savvy about security issues. So I was pleased to see an article written more for public consumption at howstuffworks.com today, entitled “What’s the problem with Microsoft Word?”. The author, Julia Layton, does an excellent job of explaining some computer security jargon and bringing the layman up to speed with the MS Word zero-day flaws which were recently announced. I hope that this is a sign of a new trend of educating the end user in a comprehensible language.

When I was a full time sysadmin and helpdesk tech responsible for a few hundred users and 50 servers, I struggled to explain the same topics to the many end users individually. So instead, I sent out ocassional messages via email with some helpful tip on how to use their computer or a link to a web article that contained some useful information on a subject that I knew would tweak their interest. So I always had these sorts of articles bookmarked to send out to my users. They appreciated that I was trying to educate them and I appreciated that I had fewer infected machines to reformat and reinstall.

High Mobley
Co-Owner of q!Bang Solutions

High Mobley’s article Southern California Linux Expo (SCALE) 5x exhibitors: non-profits has been posted to the InfoWorld Magazine web site. The article runs down the non-profit exhibitors at the recent SCALE 5x event at the Los Angeles Ariport Westin Hotel. Comments are welcomed and appreciated.

Josh Kuo’s article Dell, the new (free) Linux provider? has been posted to the InfoWorld Magazine web site. The article examines Dell’s new look at pre-installed Linux. And it’s not the Linux that you’d expect! Comments are welcomed and appreciated.

High Mobley’s article Open Source Network Monitoring at SCALE 5x has been posted to the InfoWorld Magazine web site. In the article he talks about three major network montioring products that were being exhibited at the SCALE 5x event in Los Angeles last week. Comments are welcomed and appreciated.

High Mobley’s article Southern California Linux Expo 5x has been posted to the InfoWorld Magazine web site. Comments are welcomed and appreciated.