Reflecting on 14 years of free software

14 years ago last month, I created my first PGP key to sign up to be a Debian developer. I recall what brought me to that place. I had been trying to improve my skill-set for my resume and wanted to learn to program.

Considering Linux was free compared to development software on Windows (and it ran on my Pentium 90MHz CPU when BSD didn't), it was an easy choice. However, I had no idea what I was getting into.

At the time, I was on a steep learning curve. This command line thing was nothing like the Apple //e BASIC prompt I was used to from my youth, and not even close to Mac/Windows. I was literally reinstalling my Linux OS 2-5 times a week because I would dig around into things that I had no business checking into. I tried several distributions of the time including RedHat, Slackware and Debian. I settled on Debian because it had the largest software repository, and I wanted no limitations to my journey into the realm of a software developer.

Back then, configuring my network meant first configuring the serial port and modem and then PPP and related software, in addition to chat scripts (used to provide username/password). Luckily I worked as a web designer for a local ISP, so the *nix gurus there gave me plenty of help.

As happens with free software, it isn't too long before you start finding "bugs." These annoying little things that stand in the way of you and your Linux Adulthood. At first, you just kick it around, try to avoid irritating the little thing, but eventually, you find yourself on IRC or a bug tracking system trying to find help.

I immersed myself into providing feedback to hackers and coders to test what could be wrong with my system. Surely, I thought, this was not just a problem I was having but sat amazed at how intuitive these programmers were and how steadfast in wanting to help me fix the issue. Their tireless efforts inspired me to return as much as I could.

I decided to join this group of lively lads known as Debian Developers, and submitted my PGP key and new-maintainer request. I got a call from Ian Jackson, while at work, and verified information by FAXing a few identification-proving materials to him in London. This was an exhilarating experience. I had never talked to a Brit before, much less one that was in Britain (yes, I was a little sheltered and naive). Now I just needed a way to give back to this group of about 800 developers and it's thousands of users.

As luck would have it, I got quite familiar with the inner workings of Debian's package system (DPKG/DEB) and how it worked on UltraSPARC computers. Working at NASA, I had access to all sorts of SPARC hardware, and, at the time, Debian's SPARC port was a fledgling of hope, without any guidance. I began automatic builds of Debian's vast software repository on my UltraSPARC II desktop system at work. I'd come in in the morning, verify the builds, PGP sign them, and upload the lot to the repository. I was king of SPARC!

Yes, this did get to my head. I was young, eager and worst of all, blinded by the slightest recognition. I thrived on acknowledgement and was empowered by the adulation of my peers. I dove into Debian work like Michael Phelps at a community pool; head first and with no purpose. I spent all of my spare time working on SPARC build failures, taking over things like glibc, PAM and OpenLDAP maintenance. I was hooked and my ego took me to the next logical step, running for Debian Project Leader. However, my arrogant and harsh online persona left me with few supporters, the first time around...and the second time around too.

Two years later, wiser and tempered by humility, I ran for DPL again. This time with a clear vision of what I wanted to accomplish and a vague image of my future legacy. You can read the whole thing here. As I read it, after all these years, I'm reminded of how little I knew of the real world, but I'm aghast at my own confidence and ambitious attitude. Time has a way of dulling that drive. During this DPL election, I had a clear win, and so began my 15 minutes of fame.

My new found leadership was longing for things to "fix." I started with Debian's non-US archive. We wanted encryption to be mainline, but US export restrictions were a hassle. We took on a pro-bono lawyer to help us with the specifics and finally figured out how to abide by such restrictions without opening ourselves up to legal action. The cool part was that we had to email and snail-mail notifications for each and every upload of a package that fell under these restrictions. Each notification looked something like this.

If you know anything about Debian, you know packages get uploaded by the dozens a day, if not more. We were basically flooding the bureau (with the remote hope that they would realize how ridiculous this all was). The original mailing was 2 reams of paper, double sided in a single package. This occurred about once a month. It was sheer insanity, but it got us one step closer to what we wanted...world domination!

My next step was to build up our infrastructure. Debian is heavily reliant on donations -- equipment and money. We had a good chunk of money, but we never spent it. We had decent donated hardware and bandwidth, but the main donor at the time would whine and cry and make threats that left us wondering if he would yank it all away some day. Either way, we got new hardware with large disk space at my local ISP. Chucked down $5000 for a Sun RAID array with about 320Gigs of disk. Pretty damn expensive.

How I loved this time of my life. I was well known and in the headlines of Slashdot and Linux Gazette on a regular basis. I remember being able to pick up a book or magazine or two at Barnes and Noble that had my name and/or picture in it. I would be lying if I said I didn't miss that.

But deep down, I'm a developer. It didn't take long before I had that yearning to "do some real work" and by that I mean staying up all night in front of a shell prompt trying to figure out why that oops disappears when I add in some debug printk's or reverse engineer the endianness of an OHCI-1394 packet on sparc64. Anyways, on to better things I went and many more adventures awaited me.

As I moved through my career, I became more and more focused on Linux kernel work. From embedded to server, from network drivers to mpeg drivers, from MMUs to CPUs. I've never regretted a single step of the journey. As I sit here now, working from home at my newest job, I reflect, not with a sense of accomplishment, but with a sense of humility, knowing that there were many greater, smarter and harder working folks that traversed those same years making it all happen and enabling the opportunities that I've had.

So for anyone who stumbles upon this lonely blog entry, wondering what this whole free software thing is; take a seat, pour a cup of tea, and relax for a few minutes. It's probably the last time you will have that brief illusion of a normal life, but you wont miss it one bit.

Cheers

Follow-up: Power Architecture Related Tracks Proposed for UDS-r

A few weeks ago I posted about some tracks at UDS concerning PowerPC. Here are links to the session results.

• Virtualization support for Power architecture
• Power Architecture Kernel Development
• PowerPC Bootloader Options

I need to clean up the items. The main take away is that the PowerPC kernels will be maintained separately from the mainline kernels, which means we will be getting support for some new architectures. It will probably be a couple of weeks before I get this setup, but expect it nonetheless.

The other side is the boot loader. This is a tricky and complex implementation point. Details are in the session notes, but this may have implications based on relevant work being done on ARM as well.

That's the extent of it at this point. Looking forward to great things with Raring.

Servergy Announces New PowerPC Developer Board

DISCLAIMER: I work for Servergy, Inc.

This week, at UDS, Servergy has announced that it will be designing and selling a PowerPC based developer board like no other currently on the market. Typical Power dev kits are using out-dated and feature-poor CPUs. As a follow up, they made this formal announcement.

Servergy plans many needed features, including:

• Multi-core processor
• Hardware virtualization (via Linux kernel KVM)
• Gigabit ethernet
• HDMI video output
• Network offloading engines
• SATA controller
• Audio output
• USB Ports
• SD Card slot

Servergy has dubbed this board P-Cubed.

They are planning a wide range of software support including firmware/boot-loader source code and pre-built images for creating bootable SD cards. Support for major Linux distributions will include Debian, Ubuntu, Fedora and openSUSE.

The platform is geared toward making modern Power systems available to developers for a fraction of the cost of full fledged server systems (Servergy's primary market). While the board is aimed at increasing the ecosystem and community around Linux-on-Power, the pricing is sure to attract hobbyists and students as well.

While Servergy did not say the exact price, they are aiming at a sub-$200 system. Keep an eye on Servergy's website for news and pre-order form.

How to Setup an Alternative Linux Distro Chroot Under Ubuntu - Part 2

NOTE: This is part 2 of a 2 part series.

In my last post, we setup a chroot environment for our RPM based distribution in /srv/chroots/rhel6-ppc. Now we'll setup schroot so we can access this environment as if it were the booted system, and using a union type scheme (an schroot configuration setup) so that we will always have a pristine environment for builds and such.

So let's look at /etc/schroot/schroot.conf and add the following entry:

[rhel6]
type=directory
union-type=overlayfs
description=RedHat Enterprise Linux 6
groups=adm,root
directory=/srv/chroots/rhel6-ppc
profile=default

This is the basic setup. The key here is that it uses overlayfs for union mounting the original. This means that after you exit a newly created schroot for this entry, it will be purged and the original chroot will not be changed.

Also, the profile=default means it will use configurations from /etc/schroot/default/. Make sure to add yourself to the adm group or run schroot as root.

In order to try it out, use the following command:

schroot -c rhel6

From here, you can do whatever it is you like to do in your new environment!

How to Setup an Alternative Linux Distro Chroot Under Ubuntu - Part 1

NOTE: This is part 1 of a 2 part series.

Developing a new server product requires me to test all sorts of things, including multiple distributions. As an Ubuntu developer, my main platform is, of course, Ubuntu.

It's a PITA to run multiple distributions from one system (and not very productive doing it from multiple machines), so I decided to setup chroots for each one. My production system has three environments outside of the main Ubuntu install: RHEL5, RHEL6 and SLES11.

Fortunately, Ubuntu has a nice tool called schroot. I like it because it's based off the original tool called dchroot, which I wrote back in 1999 (wow). It was mainly to allow people to use the UltraSPARC developer systems with more than one release of Debian without me having to setup multiple machines.

Fast forward 13 years, and now we have schroot. This tool has come a long way, and even includes support for snap shot of file systems so you can always start with a pristine environment. This is useful to me because I want to make sure when I build a package, only the required dependencies are installed, and I don't want to worry about screwing up the original environment. Not to mention, I can start more than one session and they wont bother each other.

In addition to schroot, we will need the rinse package. For anyone familiar with debootstrap, it's basically the same thing, but for RPM based systems. It will download and bootstrap all the required RPMs needed for a particular distro, in a manner suitable for a chroot environment.

sudo apt-get install schroot rinse

Now, if you look in /etc/rinse/rinse.conf, you will see several already configured RPM distributions. If you are wanting to do one for RHEL, you will need to either use CentOS instead, or duplicate the matching CentOS entry and rename, being sure to change the mirror URL to match your location. For my RHEL distros, I have a local RPM repository, so I use this entry:

[rhel-6]
mirror       = file:///srv/rhel6-ppc/RPMS/media/Packages

You will also need to copy the matching .packages file in /etc/rinse/ naming it the same as your entry.

Now decide where you want your chroot to be located. I've decided to put mine in /srv/chroots/rhel6-ppc. Create this directory, and then you can run the rinse command as follows:

sudo rinse --arch ppc --directory /srv/chroots/rhel6-ppc --distribution rhel-6

.

NOTE: I have my rinse script hacked a little to allow ppc as an arch. You will probably use i386 or amd64. Also, the --distribution argument is the same as the entry name.

In my next post, we'll move on to configuring schroot with a union type backing.

Where to Obtain PowerPC Dev Kits

I was asked this today on #ubuntu-kernel. It's a good question, and one which I hear often. Most people can go with old Mac hardware, but those things are kind of obsolete and largish for many people. Not to mention it doesn't do much justice for the modern CPUs you see today (multi-core, hardware virtualization, SATA, DDR2/3, etc.). Unfortunately, you aren't going to find many cheap modern Power kits like you would expect in the ARM world, but here some some quick links that I was able to put together. I would stick with Freescale, but IBM has some dev kits too.

• Micetek - These are actually really nice
• Embedded Planet - Very low CPU speeds
• IBM 750FX - 64-bit but unsure about pricing
• Emerson - Unsure of pricing, but very broad list of CPUs for IBM and Freescale

If you know of more, please comment and I'll add them to this post.

Can Canonical Make Skunkwork Work?

This post is in regards to Mark's recent announcement that Ubuntu 13.04 will be using a skunkworks approach to some of its more wow-factor features.

I know some people are going to cry out from their basement, screaming "community, community, community!" Let's face it though, one thing Linux is missing is being able to release something and people say "holy shitballs, batman!"

Anyway, I totally applaud Mark and Canonical for this decision. If it all goes well, here's my vote for naming 13.10 the Snazzy Skunk.

The GOP/DEM Circlejerk

This is slightly off from my normal blog topics, but it is election season. As I stare off with an uncontrolled dazed look on my face, pondering which candidate to vote for, a deep depression has set in.

I could choose Obama. At least I'll know what I'm getting, even though I disagree with his wealth distribution policies and bailing out too-big-to-fail companies that are sending us in a spiral of debt.

I could choose Romney, since I'm Republican, but I honestly don't like him as a person, and I feel he will serve money hungry people more than a strict capitalist agenda.

I could vote Libertarian, but, let's be honest, that will just take votes away from one of the other candidates. At this point, a vote for anything other than Romney is like voting for Obama.

My depression begins when I consider that the two primary political parties are only interested in preserving themselves. Considering that Democratic and Republican platforms have morphed every election campaign into what they think will get them voted into power, or attempt to separate themselves from one another, I can never be sure that the party I choose on my beliefs really fits who I'm voting for.

Every election it's the same thing: "Your problems are because of <insert opposite party>." Neither party will say "our platform allowed these problems to happen" or "the bills we voted caused a downturn in the economy, we'll revert that and try something else."

Let's consider this. Say you work at a company. You and another co-worker handle a particular product or service together. You work well together, bouncing ideas and patting each other on the back. One day, your manager quits, and since your company promotes from with-in, you know that you or your co-worker are going to replace him. You'll get more money, access to the executive bathroom, more vacation days, etc.

So during the manager's last couple weeks, you work your ass off. You make sure that the higher ups responsible for the decision see your hard work. You are basically campaigning for this new position. Your co-worker, being prudent, does the same thing.

Now, instead of working closely together, you are both trying to make yourself look more important and more manager-like. During these few weeks, arguments ensue, and instead of trying to build a great product or service, you are positioning, spreading distrust against your co-worker and generally being unproductive, all for the power of that promotion.

In the end, your co-worker wins out. You blame it on him bringing the top-shelf bottle of liquor to the Christmas party and taking the boss out to golf with some no-name NBA player that his cousin is towel-boy for. It's definitely not because he's better.

Now you stew in your hatred toward him as he becomes YOUR boss. At first, you continue to do your job, until one day he tells you that he thinks the product would be better if you changed one aspect. This doesn't sit well with you. "Who does he think he is?" It wasn't your idea, so it must be crap. You toss all logic aside and just hate his suggestion, regardless of the fact that it will save tons of money and is generally better at solving a certain problem than your solution would have been.

You do as you're told, but you don't have it in you to put in 100% effort. You don't necessarily sabotage the product, but the solution he asked you to implement is definitely half-assed and you don't care.

Now the product goes into production, and sure enough, the implementation falls flat, the one that you worked on. However, it was his idea, so you don't care. When the firestorm comes down from above, you blame the problem on your back-stabbing co-worker that took yer job. "He wanted it implemented this way, but I wanted to do it different." Sure enough, he's canned and now you have the manager job.

You got what you wanted. The power courses through your veins like an espresso hopped up on Redbull. You are the man. Now, two people work for you, one of whom was there when you did the poor implementation, and overheard you complain about your manager at the water cooler. He knows what you did, and has a distaste for how you got your new job. He thinks you are incompetent and undeserving: He is just like you used to be.

You get my meaning here? How can we expect Congress or the President to do anything useful if their entire intent is to make their affiliate party look good, not to mention, self-serve their lust for power? How is the President supposed to get anything done when his counterparts in Congress are trying to set him up for failure so they have bullet points on the next presidential election?

I wish I had some sort of proposal to fix this, but alas, I'm not a political scientist, so all I can do is bring the bad news.

Power Architecture Related Tracks Proposed for UDS-r

On my ramp-up toward UDS-r, I've created some blueprints and pinged some related folks to get them into the proper tracks.

I'm hoping to get a lot of interest and discussion around there, so here they are:

• Virtualization support for Power architecture
• Power Architecture Kernel Development
• PowerPC Bootloader Options

So this covers a wide range of topics. The most in-depth one is the Virtualization blueprint. As of yet, I've not seen a lot of broad support for non-x86 in OpenStack and related software. While it works (I've set it up), it just doesn't do a lot to make me happy happy.

The boot loaders blueprint is basically an RFC. The idea of Power architecture on a non-embedded system not having OpenFirmware is about on par with Dell selling an Intel system without a BIOS. The Power systems do have u-Boot (Das Boot), but that's not as robust as it needs to be. I'm thinking something like grub2 being compiled agains the u-Boot API that u-Boot can load modularly or perhaps something like the kexec based loader that the ps3 used.

Finally, the kernel development is a host discussion that needs to be hammered out with the Canonical Kernel Team so we can all be happy and not step on the primary architectures, but still being able to spread some support for newer Power equipment.

Cheers and see you in Denmark!

EDIT: Updated link for boot loaders blueprint

The Rack Revolution

As I sit here in my cozy home on my comfy couch, I am bewildered and amazed at just how far things have come in the last decade.

Let's take a quick inventory of my immediate surroundings:

• ✔ Laptop
• ✔ WiFi
• ✔ Smart Phone
• ✔ HD TV
• ✔ High Speed Internet
• ✖ Server Farm

Hmm...that last one's a bit different from the old days. I used to have a nice collection of loudly humming, room-warming servers in my garage. As a telecommuter, I needed it. My blog was running on it, my email was running on it and my firewall was running on it.

What happened? Well, we all know the answer to that question: things consolidated into the "Cloud." Instead of under-the-table boxes running our local services, we now have providers doing the heavy (literally) lifting for us.

So what do they run on? Practically the same loudly humming room-warmers that we used to keep under our desks. However, in recent years, the move is being made to lower the operating costs of these rack farms into quiet, low-powered, self-cooling, maintainable animals.

As most places have tried to just tone down, or spread thin, some have been making the move to efficiency. Enter the reverse revolution of the CPU to something more applicable to today's computing needs. Instead of powering with high-wattage x86 chips, many are dipping their toes into the shallow end of the alternative-processor kiddy-pool.

And with that I introduce an amazing NEW and WILD CPU: PowerPC!

Oh, you've heard of it? It's legacy and old-hat, you say? I must be thinking of a different PowerPC CPU then. The company I've been gainfully employed with for the past 1.5 years seems to be using something quite different than your grandmother's Power chip. Not quite the behemoth of the IBM Power7 iron (in size nor noise), but not the wussy of your old PowerMac neither.

We're talking multiway SoCs with full floating-point running at a fraction of the wattage of just about anything else on the market. Add with it full hardware virtualization (via KVM), and you begin to see where in the market this is headed.

We've already been engaging multiple Linux and software vendors to give a complete and first rate experience on this new class of hardware. You'll have multiple choices when it comes to supporting and administrating, whether it's one system or a room full of racks filled.

So here's my not-so-humble way of introducing you to Servergy, Inc.. They've been around for 3 years, but expect to be hearing a lot more about us in the coming months. If you're going to be at a Linux or Cloud/Server related event in the near future, chances are you will run into one of us. I'll actually be at Ubuntu's UDS-r in Copenhagen at the end of October. I'm hoping to have a live demonstration while I'm there.

Cheers

NOTE: In this article I am speaking solely on my behalf. None of what I've said can be taken as a statement by the company I work for: Servergy, Inc.

The Community Conundrum: PowerPC

In my recent work, I've been dealing a lot with PowerPC. As an old Mac user, I've had a soft spot for PowerPC for ages. Like most people, until recently, I've considered PowerPC an aging and dying architecture. Even with IBM selling PowerLinux systems, the lack of cheap hardware for developers has left a hole not easily filled, no matter how man old PowerMacs you buy in eBay.

However, there are a lot of PowerPC platforms that do fill this gap left by PowerMac. Some are even 32-bit platforms that can compete in today's markets.

So why have you never heard of them? Why can't you download Fedora or Ubuntu to install on your PowerPC of today? Several reason:

• Distributions don't really support it.
  
• The "community" behind it is driven at the kernel and low-level, not at the distribution level (see last bullet item).
  

This circle of support appears to be the hold up. Convincing even community supported architectures like Ubuntu and Fedora to support these different kernel flavors is met with archaic skepticism, and is usually concluded with "there is no community" to which I usually respond "because there is no support."

Something has to give here. Linux and Open Source isn't where we want the chicken-and-egg scenario to happen. You can't walk up to a Linux distro with a community and say "Here we are, let's do this" in much the same way as you can't go to a community and say "Come over here with us. We don't support you yet, but we'd like you to prove that you're worth it."

So where to begin...