Saturday, August 23, 2008

Intrepid Ibex (8.10) Moves to 2.6.27

After considerable discussion on ubuntu-devel mailing list, and in the Ubuntu Kernel Team's last IRC meeting, we've made the move to 2.6.27 for Intrepid in the hopes that it will provide a more robust experience for our users.

The source package was just uploaded to the archive for building, so in about 24 hours, we should so it on mirrors.

Tuesday, August 19, 2008

The Linux Ecosystem...Changes Ahead

So I've been privy to, and sometimes involved in, many conversations about the Linux ecosystem. How it evolved, how it is now, and where it will go from here. The most important factor has been how Linux kernel development has been funded over the years and what needs to happen to ensure it remains funded.

Given that Linux is not owned by anyone (not even by us, the developers) it is hard to say who should and will fund its future. The tides of money are constantly shifting as companies involved in this ecosystem decide where they fit in, what they want from it, and ultimately, how much they are willing to spend and for how long.

So let's go through some history. I'd like to remind people that I am not an expert on Linux kernel history in this sense. This is all from my recollection over 10+ years of being involved.

In the beginning (the past)

Well, there wasn't much. Let's face it, in the start, it was a hobby for most everyone. No company took it seriously. The device vendors that did write drivers did so out of free will, and usually poorly. Volunteers still had to munge it and cram it into the main kernel tree. It was a much simpler time. Folks did things for vanity and sheer enjoyment.

We can relate this time to when things got done because some individual wanted it done. Corporations were still on the sidelines waiting to see what happened (if they were even looking at all).

At this point, people were working on the kernel in their free time. Lots of them were in college, which means no families to support, no mortgage, no worry about retirement. I'm sure a lot of people (including myself) thought "This looks good on a resume, plus I get to do things I like".

The corporations emerge (still the past)

So at some point, people decided they wanted to make a living off this thing called Linux. Everyone knew that for the hardwares vendors to care about Linux, it had to have a corporate entity to talk to and users demanding. The boom of venture-capital-backed Linux vendors emerged (aka distributions). We know they have come and gone over the years, with only a few emerging in the black.

These companies positioned themselves in many different ways. Some trying to become service oriented, while others relied on licensing to make their money. I wont delve into this topic much, but let's look at how these Linux vendors advanced.

Remember, we are just coming out of the previous stage. No corporations are yet seriously funding development in Linux. It, in itself, is still a long ways off from having all the features that users really want in an OS. How do these distributions get these features? Easy, they hire developers that have been doing this all along.

This is the initial way things get done. You pay someone to do it.

So how did this fall on the distributions? Simply because the corporate distributions required it in order to compete in the market. The OEM's and hardware vendors didn't care. Their stuff was selling on Windows and Unix platforms without problems. They had no financial requirement or user demand to worry about supporting Linux at this point. If their hardware was popular enough with Linux, someone would write a driver for it.

Enter the hardware vendors (sort of past and up to now)

Now that Linux is starting to be a commercial "thing", hardware vendors are taking notice. Not only because of the press around it, but because their own customers are starting to demand it. Big customers.

In addition, companies are starting to see a way for them to piggyback on all the hype and press coverage. If you're "Linux Friendly", you've got a whole bunch of geeks with purchasing leverage behind your company.

Large hardware vendors are starting to take notice. Companies devote whole groups of engineers at supporting Linux. And not just in some odd way, they are doing it our way. Open, and in the community. They work with distributions to get early adoption of drivers. They work with upstream to integrate these drivers and features into the kernel. They participate in steering the process, and drive a lot of what we do. OEM's can finally lay down the requirement "Must be supported in Linux" to their ODM's.

Where did these engineers come from? Right from the Linux kernel community. Most people hired to work on the Linux kernel by a company, cut their teeth for zero money in the community. Some of them have also been hired away from the distributions.

As a previous hiring manager for Ubuntu's kernel team, I can tell you personally, I generally skipped the CV and went straight to the kernel commit logs and linux-kernel mailing list to verify someone. The CV was just a backup.

This is one of the beauties of our process. On a CV, most people look good. Even their references (which they choose) are all likely to tell you what you want to hear. But nothing can tell about a persons personality like the thread from 6 months ago where this person tried to get a feature accepted upstream on lkml. You wouldn't have seen how this person either defended themselves valiantly, or wussed out just because Alan Cox had some harsh words. You wouldn't have known about how they worked for months to take their original idea and rework it to suit the issues brought up on submission, or whether they let the idea die because they couldn't handle the criticism.

Back to the topic though...I'll repeat, these hardware vendors are hiring kernel developers. But it isn't just the hardware manudacturers. You also have companies like Oracle, Google and VMWare hiring them. Some companies even have enough cashflow that they can hire a high level upstream kernel developer for pure bragging rights, or a consortium sponsored by many companies hiring people to just keep doing what they were doing for nothing.

This is definitely where the shift comes in. More and more, we see hardware vendors developing Linux drivers that are released at the same time the device goes public. This development is occurring in-house, and not out in the community. Sure, the community still integrates it, and goes through the code review process, but how many new drivers are coming from someone not associated with the vendors that made the device? Fewer and fewer.

The road ahead...(now and into the future)

So as things move ahead, there will be less for the distributions to do for hardware support. Most vendors will produce the driver, and community+distributions will play a big part in integrating these drivers. New subsystems will emerge to support new ranges of devices. It's not too hard to see vendors working together in the community to solidify features and API's that their drivers need (e.g. mac80211, GPU, other wireless technologies, multi-core features, memory management, etc).

Most developers will cut their teeth on helping to integrate and enhance these things from the vendors. The community will revolve around major restructuring of the kernel to ease development and improve stability.

So where does that leave the distributions? With the majority of the kernel work being handled by vendors, the distributions will fall into a level of consumption. Let's face it, distributions are best at integration (which is part development, so let's not get confused). Distributions are also good at noticing trends, which are fed upstream. Yes, they will still drive new ideas, and possibly even develop these ideas in-house, but they wont be the ones driving the the bulk of the work because they wont be the ones creating the new hardware that will require it.

The idea that distributions should ultimately be responsible for the kernel funding is not possible to sustain. In the current ecosystem, it is not required for a distro to invest heavily in upstream kernel work. Because Linux is open and free, there is nothing forcing them to do so. If this company instead invests heavily in integration and usability, they will produce a better product for the masses, beating the other distributions in the end, and leaving the kernel developers hired by those other distributions without jobs.

In the end...(entirely made up)

If a distribution is popular enough, the hardware vendors will want it to run on their goods. OEM's and hardware vendors who work together to help bring support for their hardware to the kernel will ultimately beat out competitors. The age where Linux is in-demand enough to create this ecosystem is close at hand, and in some ways, already exists.

Nothing is written in stone. No one can predict what will happen, we can only speculate. However, we can probably be assured that the funding to keep Linux around will come from many places, maybe even ones we haven't thought of yet.

I, for one, look forward to what's ahead.

Thursday, August 14, 2008

2.6.27-rc3 Kernel images for Hardy/8.04 and Intrepid/8.10

I've built some Ubuntu kernels based on 2.6.27-rc3. They are available here.

Please feel free to test. Report bugs directly to the kernel-team mailing list.

If you are running hardy, and want to use the iwlagn driver (4965 11n support), then make sure to get the linux-restricted-modules-common package from Intrepid as well.

Thursday, August 7, 2008

Ubuntu Kernel Next

Normally in Ubuntu's development cycle, we don't begin work on the kernel for a release until that release opens for development.

We are starting something new this time around. Now that 2.6.26 is released, and the kernel in Intrepid/8.10 (our current development cycle) is pretty stable, we have opened up a new git tree called ubuntu-next. Do not confuse this with linux-next, they are different concepts.

We are not spending a lot of time adding features to this tree. It is basically a rebase of all of our patches on top of the latest kernel in linux-2.6 upstream git. Our patches are consolidated and given some consistency (and a few pushed upstream).

At regular intervals, binary packages of this tree will be made available (usually at -rc milestones from upstream). In fact, the first installment of these are now available at:

I've built them for Hardy/8.04 and Intrepid/8.10. The only difference between Intrepid and Hardy is the compiler, which means your dkms and other third-party module recompiles should work :)

A few points to remember when using these:

  • They are not guaranteed stable, or even to be able to boot. Keep away from small children.

  • These images will not be supplied in a PPA or any other APT type repository. We need a barrier to prevent people from just adding these to their systems wholesale.

  • We will not provide respins of linux-restricted-modules or any other modules package in the Ubuntu archive with these. Headers are provided, so use them if you need to.

  • DO NOT (and I can't say this enough) file bugs on these packages. They will be ignored.

  • DO (and I can't say this enough either) feel free to reproduce your bugs against official kernels with these kernels. It's a good data point to know whether the bug has been fixed by a newer kernel

  • DO NOT (and I will make physical threats on this one) pump these packages into a public APT repo and tell the world so that it is "easier" to use bleeding edge stuff. If folks don't know how to install these packages, they surely don't need to be running them (we wont help them fix things after trying them).

  • DO NOT ask us to put this into Intrepid/8.10 3 weeks before RC just because it makes your mouse stop squeaking whenever a cat is near the computer. Honestly, this kernel isn't going to get the same testing, and we have deadlines for a reason.

Other than that, enjoy.

Sunday, August 3, 2008

Ubuntu Kernel Team IRC Meeting

Our next team meeting is August 5th (Tue) at 16:00 UTC. We invite any and all community members to participate. Read our agenda and see what we've done in the past. Quick note, we've been slacking on this important event for awhile now, and are trying to get things rolling again.