Tuesday, August 19, 2008

The Linux Ecosystem...Changes Ahead

So I've been privy to, and sometimes involved in, many conversations about the Linux ecosystem. How it evolved, how it is now, and where it will go from here. The most important factor has been how Linux kernel development has been funded over the years and what needs to happen to ensure it remains funded.

Given that Linux is not owned by anyone (not even by us, the developers) it is hard to say who should and will fund its future. The tides of money are constantly shifting as companies involved in this ecosystem decide where they fit in, what they want from it, and ultimately, how much they are willing to spend and for how long.

So let's go through some history. I'd like to remind people that I am not an expert on Linux kernel history in this sense. This is all from my recollection over 10+ years of being involved.

In the beginning (the past)

Well, there wasn't much. Let's face it, in the start, it was a hobby for most everyone. No company took it seriously. The device vendors that did write drivers did so out of free will, and usually poorly. Volunteers still had to munge it and cram it into the main kernel tree. It was a much simpler time. Folks did things for vanity and sheer enjoyment.

We can relate this time to when things got done because some individual wanted it done. Corporations were still on the sidelines waiting to see what happened (if they were even looking at all).

At this point, people were working on the kernel in their free time. Lots of them were in college, which means no families to support, no mortgage, no worry about retirement. I'm sure a lot of people (including myself) thought "This looks good on a resume, plus I get to do things I like".

The corporations emerge (still the past)

So at some point, people decided they wanted to make a living off this thing called Linux. Everyone knew that for the hardwares vendors to care about Linux, it had to have a corporate entity to talk to and users demanding. The boom of venture-capital-backed Linux vendors emerged (aka distributions). We know they have come and gone over the years, with only a few emerging in the black.

These companies positioned themselves in many different ways. Some trying to become service oriented, while others relied on licensing to make their money. I wont delve into this topic much, but let's look at how these Linux vendors advanced.

Remember, we are just coming out of the previous stage. No corporations are yet seriously funding development in Linux. It, in itself, is still a long ways off from having all the features that users really want in an OS. How do these distributions get these features? Easy, they hire developers that have been doing this all along.

This is the initial way things get done. You pay someone to do it.

So how did this fall on the distributions? Simply because the corporate distributions required it in order to compete in the market. The OEM's and hardware vendors didn't care. Their stuff was selling on Windows and Unix platforms without problems. They had no financial requirement or user demand to worry about supporting Linux at this point. If their hardware was popular enough with Linux, someone would write a driver for it.

Enter the hardware vendors (sort of past and up to now)

Now that Linux is starting to be a commercial "thing", hardware vendors are taking notice. Not only because of the press around it, but because their own customers are starting to demand it. Big customers.

In addition, companies are starting to see a way for them to piggyback on all the hype and press coverage. If you're "Linux Friendly", you've got a whole bunch of geeks with purchasing leverage behind your company.

Large hardware vendors are starting to take notice. Companies devote whole groups of engineers at supporting Linux. And not just in some odd way, they are doing it our way. Open, and in the community. They work with distributions to get early adoption of drivers. They work with upstream to integrate these drivers and features into the kernel. They participate in steering the process, and drive a lot of what we do. OEM's can finally lay down the requirement "Must be supported in Linux" to their ODM's.

Where did these engineers come from? Right from the Linux kernel community. Most people hired to work on the Linux kernel by a company, cut their teeth for zero money in the community. Some of them have also been hired away from the distributions.

As a previous hiring manager for Ubuntu's kernel team, I can tell you personally, I generally skipped the CV and went straight to the kernel commit logs and linux-kernel mailing list to verify someone. The CV was just a backup.

This is one of the beauties of our process. On a CV, most people look good. Even their references (which they choose) are all likely to tell you what you want to hear. But nothing can tell about a persons personality like the thread from 6 months ago where this person tried to get a feature accepted upstream on lkml. You wouldn't have seen how this person either defended themselves valiantly, or wussed out just because Alan Cox had some harsh words. You wouldn't have known about how they worked for months to take their original idea and rework it to suit the issues brought up on submission, or whether they let the idea die because they couldn't handle the criticism.

Back to the topic though...I'll repeat, these hardware vendors are hiring kernel developers. But it isn't just the hardware manudacturers. You also have companies like Oracle, Google and VMWare hiring them. Some companies even have enough cashflow that they can hire a high level upstream kernel developer for pure bragging rights, or a consortium sponsored by many companies hiring people to just keep doing what they were doing for nothing.

This is definitely where the shift comes in. More and more, we see hardware vendors developing Linux drivers that are released at the same time the device goes public. This development is occurring in-house, and not out in the community. Sure, the community still integrates it, and goes through the code review process, but how many new drivers are coming from someone not associated with the vendors that made the device? Fewer and fewer.

The road ahead...(now and into the future)

So as things move ahead, there will be less for the distributions to do for hardware support. Most vendors will produce the driver, and community+distributions will play a big part in integrating these drivers. New subsystems will emerge to support new ranges of devices. It's not too hard to see vendors working together in the community to solidify features and API's that their drivers need (e.g. mac80211, GPU, other wireless technologies, multi-core features, memory management, etc).

Most developers will cut their teeth on helping to integrate and enhance these things from the vendors. The community will revolve around major restructuring of the kernel to ease development and improve stability.

So where does that leave the distributions? With the majority of the kernel work being handled by vendors, the distributions will fall into a level of consumption. Let's face it, distributions are best at integration (which is part development, so let's not get confused). Distributions are also good at noticing trends, which are fed upstream. Yes, they will still drive new ideas, and possibly even develop these ideas in-house, but they wont be the ones driving the the bulk of the work because they wont be the ones creating the new hardware that will require it.

The idea that distributions should ultimately be responsible for the kernel funding is not possible to sustain. In the current ecosystem, it is not required for a distro to invest heavily in upstream kernel work. Because Linux is open and free, there is nothing forcing them to do so. If this company instead invests heavily in integration and usability, they will produce a better product for the masses, beating the other distributions in the end, and leaving the kernel developers hired by those other distributions without jobs.

In the end...(entirely made up)

If a distribution is popular enough, the hardware vendors will want it to run on their goods. OEM's and hardware vendors who work together to help bring support for their hardware to the kernel will ultimately beat out competitors. The age where Linux is in-demand enough to create this ecosystem is close at hand, and in some ways, already exists.

Nothing is written in stone. No one can predict what will happen, we can only speculate. However, we can probably be assured that the funding to keep Linux around will come from many places, maybe even ones we haven't thought of yet.

I, for one, look forward to what's ahead.


  1. Either you haven't seen very many vendor drivers, or you haven't bothered to look very closely at how they good they are, or aren't.

  2. daniels> I've seen lots of vendor drivers. While most are not good enough to get into the kernel as-is, they are producing these drivers more now than in the past. The trend will only continue, and as such, it's just a matter of time before they are written well from the start (most likely because they will hire real kernel developers and not just hope their Windows driver writers can port it).

    Since your comment is so vague, I can only assume this is what you are talking about.

  3. "New subsystems will emerge to support new ranges of devices. It's not too hard to see vendors working together in the community to solidify features and API's that their drivers need (e.g. mac80211, GPU, other wireless technologies, multi-core features, memory management, etc)."

    is it not?

    You don't think the fact that we distros employ a lot of subsystem maintainers primarily to avoid the case where one vendor blocks code from another vendor for some wierd advantage can't happen?

    Competeing vendors only usually work together when there is someone holding them to ransom with double barrel shotguns, otherwise we would just get 10 ATA RAID stacks, 25 network stacks etc.

    Do you really think Intel wanted to re-write their wireless drivers for devicescape?

    I know working for Red Hat we get access to a lot of upcoming hw and do a lot of work on drivers for it before releases, that we can't do it publically is annoying, but not many hw vendors try to release magic drivers without working with at least one distro on them.

    distros have a quite valid place in the hw development ladder, keeping vendors in line and forcing them to cooperate against what the forsee as their best interests.

  4. airlies> I understand your point, but don't you think the reason things are like that now is because these companies are coming from one ecosystem to another?

    Don't you expect that to change as Linux becomes the norm instead of the "new" thing?

    Distro's will, and should, always play a part in this, no doubt. I just think the burden on distros will become less over time.

  5. Hardware vendors are not kernel developers, and probably will never be. It's not their speciality, nor their primary concern.

    Their primary concerns are:
    a) get software working well enough that it proves the hardware works, for development;
    b) get it into good enough shape so they can ship it (for Windows, this is easy, for Linux this means convincing vendors, which seems to be remarkably easy for Ubuntu, Mandriva, and most of the third-rate distros).

    If you're expecting them to provide good code straight off the bat, then while I salute your optimism, I have to question your grip on reality.

  6. I believe competing companies will always find it hard to justify working together at some levels.

    Also you might just get one set of competitors working together when other parts of the same company have the same problem.

    HW vendors like having distros around and also upstream maintainers that aren't local, so they can justify to management in the companies why they are working with their competitors. They always have the excuse that maintainers won't accept their stuff. When it comes to the maintainers bonus being tied to a certain piece of hw working in the upstream kernel, you start to see a lot more conflict of interests, and I think the hw/distro split is at least good for separating this.

    I don't think we'll ever get rid of all the clueless management, they keep bringing in new ones.

  7. daniels> For Windows, you are over simplifying. There's still driver certification. And while you might not like the company doing the cert (Microsoft) it does add overhead.

    I'm not sure what your crack about "easy for Ubuntu" infers, but I'll assume you're a senseless troll and refrain from further comment.

  8. airlied> I can agree with that, but I have this sense that if we are still fighting with hw vendors in 10 years like we are now, we aren't making progress. Something has got to give...

  9. Ben> Yes, the barrier has got a lot higher recently, but people are still adjusting to that (and not without complaint). As for the 'senseless troll' bit, I wasn't inferring Ubuntu was a third-rate distro, merely noting that the barrier to entry for merging patches, particularly in the kernel, seems to be lower than in other distros, particularly when it comes to new drivers. Would you say that isn't the case?

  10. The thing is I don't fight with many hw vendors now, the hw vendors are giving, not the distros.

    The HW vendors realise they need serious Linux teams, but they also realise they need to deal with distros esp in the enterprise arena. The thing is we aren't changing what we do, the vendors are. I don't think they'll ever reach the pinnacle where we can entrust the whole driver ecosystem to them and I don't think they want that either.

    For Red Hat we view the distro's job to push forward the common ecosystem to allow the vendors to write drivers that perform well and usefully on our distros. That is why Red Hat puts so much effort into things like X.org and kernel maintainers, so the common platform can move forward so that the drivers can take advantage of the hw features. e.g. RH is funding DRI2 so vendors can provide DRI2 drivers so we do a proper composited desktop experience, same goes for kernel modesetting.

    HW vendors have not shown the ability in many areas to move the common platform forward, they get their driver to the same level as everyone else or like Nvidia hide the useful platform level features away.

    So I don't think distros can sit back and detach and say hw vendors will do all the work for us, at some point you either lead the charge or get trampled over.

  11. daniels> In the context of your original statement, I took it that you meant that Ubuntu was more likely to take crappy code than other distros.

    I don't think this is true. While we do include more third-party drivers, I say we introduce fewer non-mainstream patches to the kernel itself. I looked at the fedora kernel recently and it had > 13Megs of patches along with it. I'm not saying that's bad, but I'm using it as a defense that we take in more code in our tree than other distros (just remember not to include the debian/ directory in diffs of our tree since I didn't include the spec file and other scripts in the above diff either).

    Yes, we take in more drivers, but they are usually obsolete hardware that we don't want to regress on (things we've supported for 3-4 years now). Which means it doesn't affect anyone but those old systems.

    However, like other distros (even RH and Suse) we do take in updates to existing drivers to support newer hardware.

  12. airlied> I keep feeling you think I believe the distros wont be responsible for part of the ecosystem, and I have to repeat, I don't think so :)

    I believe things will shift, and I think the hw vendors will shift with it. "Lead or get trampled" applies to them as well. It's no different than distros having to play nice with each other in order to advance the ecosystem (which Linux distros seem to do very well, IMO).

    If you and I can carry on a decent conversation such as this...if Redhat/Novell can stand in the same room and talk about the future of the kernel, and agree, then hw vendors can do it too :)

  13. "I don't think this is true. While we do include more third-party drivers, I say we introduce fewer non-mainstream patches to the kernel itself. I looked at the fedora kernel recently and it had 13Megs of patches along with it."

    Fedora doesn't have out of kernel drivers. So we have to include all drivers in the distro kernel tarball not as a separate cupboard we hide stuff in. Now the other big difference is Red Hat employs groups of people to maintain and/or write the code for nearly all of those patches. I usually push my upstream queues for Linus into Fedora as does John Linville the wireless maintainer. I also have kernel modesetting previews, nouveau driver previews etc.

    I don't think Ubuntu can claim that they can support the random stuff that ends up in their kernel, but Fedora has the manpower to stand behind any large changes we put into the kernel before they are upstream. We also cleaned up quite a lot of drivers for upstreaming for webcams etc.

  14. airlied> We only did that for a few releases, and it wasn't to "hide" things. It was meant to separate the kernel from the third-party drivers.

    I had us go back to including our third-party drivers in the main kernel tree (under ubuntu/ so as not to be confused with stock kernel drivers...makes it easier to see what we've added...because we don't like confusion...).

    Note that we barely include any in-tree patches that came from us. We usually cherry-pick these singular fixes from upstream kernel. Fixes we do ourselves, we can surely support (since we had the manpower to write it, we have the manpower to support it). Note, fixes we do ourselves usually boil down to 1-5 liners. We aren't talking about large patches.

    All of the patches to our stock kernel (whether from us or upstream) generally go through at least three sets of eyeballs before being allowed into our tree (with only 5 kernel devs, that's a pretty high percentage), and we don't accept anything that causes a high bit of churn in the tree, increasing the chances of regression and problems (for the very reason you cited).

    The largest patch we have in place right now, is AppArmor. Aside from what anyone thinks about it, we talked with upstream (yes, Novell employees) about supporting our use of this patch. They've done a tremendous job, helping with VFS quirks and other issues. We've also provided them with lots of testing, and sent fixes back to them (I know Kees works closely with JJ concerning AppArmor).

    So we don't jump into huge updates beyond stock without first checking to make sure we can handle dealing with it. I'm not sure where you think we have boat loads of patches to the stock kernel, but in reality, we don't.

    Note, if you check hardy/2.6.24 kernel against stock 2.6.24, it may appear to have a huge delta, but minus debian/ files, and consider it is actually 2.6.24.y (I can't recall the minor, but it's up there) and includes cherry picks for CVE's, major bug fixes and such (from upstream), there is still very little code that we introduced that wasn't already upstream.

    We don't pull in the entire wireless-dev tree or dri2 tree. Why? Because we can't support it. We have requests to do so, and we refuse such requests.

    Now, in the past I have done such a thing. I talked to jgarzik at one point (around 2.6.17?) about including the ata for-linus tree in Ubuntu. He assured me it was stable. He was wrong, and I learned a lesson the hard way :)

    So the bottom line is, we are very cautious about what we include, and have even out right refused to include drivers and updates requested by our hw vendor partners (the ones paying us) because of the support and stability issues it could have caused.

  15. Okay so Ubuntu's kernel isn't as bad as it used to be, which is good, but really Fedora's kernel diff isn't a major problem, we are always pushing things upstream from our kernel diff, I mean always. So nothing goes in that we can't personally upstream.

    But back to the original discussion, hw vendors have a lot more to lose than distros when they collaborate. The patent/time to market/secret features/lawyers matter a lot more in their areas. I don't think this will change.

    I also don't think they'll agree on things like wireless stacks without herding, look how much effort it took to get down to one wireless stack. Now the distros should have been more proactive in that area, but if the hw vendors had been more proactive I don't think it would have worked out usefully, we would just have 4 stacks shipping in each kernel one for each vendors driver.

    I think distros will always have to herd hw vendors into line, but I think hw vendors have come a long way from where we were 5 years ago, and I think will be in a better place in 5 years.

    However they will never replace a good kernel distro team ever. You'll also notice as Canonical actually gets into doing some enterprise support that you can't support new hw for 4-5 years after OS release without some serious staffing. One of the big pushes we have with hw vendors is deciding who should backport the support into older distros. They don't want to do it, we'd rather not do it, but at some point you have to support hw your customer just bought on a 4 year old distro. When that happens you better hope you've hired a bunch of kernel/X.org hackers or just ship unsupportable binaries.

  16. airlied> It's already happened :) We've been exposed to a combination of the hw vendor having to backport things (to 2.6.15) and us having to do it. We've been able to manage thus far.

    The last point release for 6.06, which is a 2.6.15 kernel (I know it isn't even 3 years old yet, but the kernel in it is), contained quite a few backported drivers that I personally did the work for (a lot of my career before Canonical was backporting 2.4.x and 2.6.x drivers to 2.0.30 kernels for another company, including entire subsystems).

    Granted, you never know how much work backporting will take until you have to do it 3-5 years down the road, but we've been able to manage things thus far.

    I know that RH has a lot more experience in this as a company, but believe me that we have some pretty experienced people in the company who have been through these things before and understand what we can expect as a company.

    Canonical will definitely be hiring more kernel developers in the future. We have positions open now. I would be surprised if the current kernel team didn't double or more within the next year or so.

    However, right now, it's not financially smart for us to hire resources we don't need. We have specific positions to fill based on known work we _need_ to do. In addition, growing pains are being kept to manageable levels on purpose (our company is still quite small at ~180 employees). Hiring a dozen kernel devs in 6 months would slow our team to a crawl just on logistics alone.

    So while I can agree on some of your points, I hope you can agree that Canonical can't just turn into this magical upstream kernel force overnight. No company does that overnight.

    While RH may have ramped up to that quicker than Canonical, that's only because the state of Linux way back then demanded they do it, or fail.

    And while you and others might think that we are leeching off of your great work, we're merely taking our baby steps in the right order. Starting a commercial Linux distro now does not require heavy focus on kernel development. We have to build up to that.

    Oh, and I appreciate your praise for our kernel getting better as far as delta, but quite honestly, neither I nor Canonical can take credit. The huge amount of changes we used to apply to the kernel was driven by the need for stabilization and newer hardware support, and huge amounts of churn upstream. Our ability to tone down what we pull in is a direct result of the hard work from upstream kernel devs and hw vendors cooperating. Credit where credit is due...