Friday, August 21, 2015

Encryption is not for the bad guys

I've been reading a lot of articles about presidential candidates and their stances on encryption for online privacy. It befuddles me how ridiculous their arguments are. You hear things like "only evildoers use encryption" or "if you have nothing to hide, you shouldn't be against this."

These are the same arguments that gave us our miranda rights, protection against illegal search and seizure, etc. Just because a citizen exercises their right to privacy does not mean they are hiding something, nor can it be taken as probable cause to remove their rights.

Saying that law enforcement needs to remove encryption so they can find bad guys is like saying houses need to have open windows so they can see in to houses. It's like saying we should do away with home security systems because only people with illegal items use them.

Any candidate that wants to keep me from protecting my privacy and security on the grounds of protecting me from some would-be terrorist is on a power trip and does not stand for the same constitution that I do.

Monday, July 27, 2015

How to Alienate Your Customers And Drive Your Online Service Into the Ground

So you've spent months, maybe years, building an online service. You have lots of customers and people are starting to rely on your service for personal and/or business purposes. It has a great reputation and it's basically the only one of its kind. Now what?

If you're like one particular service that I started using a couple of months ago, you ruin all of your good will and hard work inside of a week. I'm not about to call out who this service is, but I will gladly tell you exactly how they went about this epic failure of standard practices. So let's get started.

Step One: Plan a maintenance window


You've got to keep these customers happy with new features, not to mention, you've got to make room for the unexpected growth you've seen.

So let's do it all at once:

  • Lots of new customer facing features? Check
  • Lots of backend features to handle unexpected growth? Check
  • Move to new hardware? Check
  • Database updates across billion-row tables? Check
  • Test plan using mirrored copy of current data set? Ain't nobody got time for dat.
  • Backout plan in case something goes wrong? Pshaw, we won't need one!
  • Timetable for how long this operation will take? Eh, can't take too long, right?

Here's the thing folks; never, and I mean never, run a maintenance window that involves multiple moving parts. If you can't perform these actions individually, then you've messed up somewhere in building this thing out from the start. If your upgrades require moving to new hardware, then do it separate from the rest.

Move to new hardware with the existing application and data. Don't mix features that aren't interdependent and make sure to test any database migrations on a full backup data set (or at least a good portion of it) before doing it on the live data set.

Next, always have a way to go back. If you're upgrading a database, have some way to revert back to the original in an instant. Whether it's a backup, a snapshot, or whatever. Don't depend on being able to back out the change you've made (i.e. running more SQL commands on the live data). You want a pristine place to draw from.

But now that you've screwed that up, let's continue on.

Step Two: Don't ever take the blame


Now that you've pushed this "New and Improved" version of your service in the most obscenely unprofessional way, you will definitely have something go wrong. It's not an if or when, it's just going to happen. First off, don't bother checking to make sure everything went well. Just go to bed and pat yourself on the back.

When you wake up in the morning and see things aren't working as you expect, don't bother replying to the customers' cries for help just yet. Let them know who's boss and who runs this joint. You, that's right. After awhile, give a little update. Remember what your english teacher taught you; less is more. Something like this will do just fine:

We're working hard to make our service better. Please bear with us while we continue to do so.

Some companies forget that their customers are not stupid. We know when something's wrong and in cases like this, we know you screwed up somehow. Don't make it worse by glossing it over. Gives us the straight poop, so to speak. The main word here is transparency. Most companies have this knee-jerk reaction of trying to make it look like "oh, we just had some bad luck, we didn't do anything wrong."

Believe me, even if you aren't transparent, the fact that you aren't is pretty transparent to us. Just like when my kids were 3 years old, I could tell when they were lying ("But dad, I swear, it was the dog that ate all the cookies mommy just baked"...as cookie crumbs fell from his face).

Now that you're busy ignoring the flames on the customer front, let's take care of this problem.

Step Three: Take your time


Who's in a hurry? Not this guy! Amiright? It's already broken, you have no capability to revert all of this crap, and you have to get these new features to the masses else what was it all for? Just keep pushing forward like an angry crowd at a music festival.

What you need to remember is that all of this work is for naught if your customers all leave. You have to be able to suck it up and back all of this out to get back to a stable base and come back at it later. The particular failure that I saw this past week may have been able to revert all of the mess they started. I don't know (they didn't talk much). I can only assume that a) they couldn't revert (bad planning) or b) they were so consumed with making this work, they decided to push forward.

It's hard to say which scenario is worse, but if you do find yourself in a position where an upgrade has broken things and you are able to revert back to a known good state, don't let your ego get the best of you. Just REVERT, go back to the drawing board, perform some postmortem, and start up again on a fresh day.

Lastly, above all else, talk to your customers regularly through the process. Waiting 4-15 hours between updates is a super bad idea. Making these status updates vague and absolving yourself of any responsibility is mistake number two.

Just remember, if you own a particularly new market, the only time a competitor will even think of jumping in is when you lose the trust of your customers. They will capitalize on your mistakes.

Thursday, March 7, 2013

Power Up

As a POWER architecture hardware vendor, we've definitely run into quite a few wish-list items for software we want to have on our platform. Whether it's for customers or just to have a feature complete set of packages in everyday distributions, we want to see things build everywhere, and run just as well as the x86 counterparts.

Starting soon, we are kicking off a PowerUp (my cool label) initiative in order to direct POWER developers toward software that needs a little love on our POWER platforms. Software targets range from the completely unsupported (e.g. Google's v8 Javascript engine, D Languages's phobos) to optimizing specifically for POWER (e.g. OpenJDK).

To collect these initiatives together, we will be starting a new PowerUp portal. For now, we have begun a GitHub Team where we have forked relevant repositories. Forums for discussion and participation will also follow. Feel free to clone and hack away. Email me if you have any questions (or wait until the forums and portal open).

NOTE: PowerUp is just my initial name. That may or may not change.

I'll update this blog post when more information is available.

Wednesday, March 6, 2013

Ubuntu Rolling Releases Vs. Hardware Companies

So I have to speak out on this whole issue. I work for Servergy, and for almost two years I've been working on Ubuntu's PowerPC port in order for our new hardware platform, the CTS-1000, to have an out-of-the-box solution for our customers. We've been hedging on Ubuntu, since it was able to provide us a known quantity for release dates and an open community that we could participate in (especially being able to take advantage of my core-developer status).

Now, after so much work, so much planning, we are worried about 13.04 never being properly released. This would leave us with no stable Linux distribution for our hardware, basically yanking the rug out from under all of our work. Having a stable release every two years also enlarges the support gap for our followup platforms. Now I realize most hardware vendors are x86-based, and their issues are likely limited to supporting peripherals, so this affects us more than most. The issue we face is supporting entirely new hardware platforms and SoCs with a completely new kernel (likely requiring lots of supporting patches). This is the type of thing that, historically, isn't allowed to be added to an LTS release.

So I have to wonder, if Ubuntu does adopt this rolling release schedule, how viable is it for us? I would still be happy if Ubuntu had one release per year, with every other release becoming an LTS. However, the two year window is just entirely too large to depend on for quick moving hardware bring up and release.

Wednesday, November 7, 2012

Reflecting on 14 years of free software

14 years ago last month, I created my first PGP key to sign up to be a Debian developer. I recall what brought me to that place. I had been trying to improve my skill-set for my resume and wanted to learn to program.

Considering Linux was free compared to development software on Windows (and it ran on my Pentium 90MHz CPU when BSD didn't), it was an easy choice. However, I had no idea what I was getting into.

At the time, I was on a steep learning curve. This command line thing was nothing like the Apple //e BASIC prompt I was used to from my youth, and not even close to Mac/Windows. I was literally reinstalling my Linux OS 2-5 times a week because I would dig around into things that I had no business checking into. I tried several distributions of the time including RedHat, Slackware and Debian. I settled on Debian because it had the largest software repository, and I wanted no limitations to my journey into the realm of a software developer.

Back then, configuring my network meant first configuring the serial port and modem and then PPP and related software, in addition to chat scripts (used to provide username/password). Luckily I worked as a web designer for a local ISP, so the *nix gurus there gave me plenty of help.

As happens with free software, it isn't too long before you start finding "bugs." These annoying little things that stand in the way of you and your Linux Adulthood. At first, you just kick it around, try to avoid irritating the little thing, but eventually, you find yourself on IRC or a bug tracking system trying to find help.

I immersed myself into providing feedback to hackers and coders to test what could be wrong with my system. Surely, I thought, this was not just a problem I was having but sat amazed at how intuitive these programmers were and how steadfast in wanting to help me fix the issue. Their tireless efforts inspired me to return as much as I could.

I decided to join this group of lively lads known as Debian Developers, and submitted my PGP key and new-maintainer request. I got a call from Ian Jackson, while at work, and verified information by FAXing a few identification-proving materials to him in London. This was an exhilarating experience. I had never talked to a Brit before, much less one that was in Britain (yes, I was a little sheltered and naive). Now I just needed a way to give back to this group of about 800 developers and it's thousands of users.

As luck would have it, I got quite familiar with the inner workings of Debian's package system (DPKG/DEB) and how it worked on UltraSPARC computers. Working at NASA, I had access to all sorts of SPARC hardware, and, at the time, Debian's SPARC port was a fledgling of hope, without any guidance. I began automatic builds of Debian's vast software repository on my UltraSPARC II desktop system at work. I'd come in in the morning, verify the builds, PGP sign them, and upload the lot to the repository. I was king of SPARC!

Yes, this did get to my head. I was young, eager and worst of all, blinded by the slightest recognition. I thrived on acknowledgement and was empowered by the adulation of my peers. I dove into Debian work like Michael Phelps at a community pool; head first and with no purpose. I spent all of my spare time working on SPARC build failures, taking over things like glibc, PAM and OpenLDAP maintenance. I was hooked and my ego took me to the next logical step, running for Debian Project Leader. However, my arrogant and harsh online persona left me with few supporters, the first time around...and the second time around too.

Two years later, wiser and tempered by humility, I ran for DPL again. This time with a clear vision of what I wanted to accomplish and a vague image of my future legacy. You can read the whole thing here. As I read it, after all these years, I'm reminded of how little I knew of the real world, but I'm aghast at my own confidence and ambitious attitude. Time has a way of dulling that drive. During this DPL election, I had a clear win, and so began my 15 minutes of fame.

My new found leadership was longing for things to "fix." I started with Debian's non-US archive. We wanted encryption to be mainline, but US export restrictions were a hassle. We took on a pro-bono lawyer to help us with the specifics and finally figured out how to abide by such restrictions without opening ourselves up to legal action. The cool part was that we had to email and snail-mail notifications for each and every upload of a package that fell under these restrictions. Each notification looked something like this.

If you know anything about Debian, you know packages get uploaded by the dozens a day, if not more. We were basically flooding the bureau (with the remote hope that they would realize how ridiculous this all was). The original mailing was 2 reams of paper, double sided in a single package. This occurred about once a month. It was sheer insanity, but it got us one step closer to what we wanted...world domination!

My next step was to build up our infrastructure. Debian is heavily reliant on donations -- equipment and money. We had a good chunk of money, but we never spent it. We had decent donated hardware and bandwidth, but the main donor at the time would whine and cry and make threats that left us wondering if he would yank it all away some day. Either way, we got new hardware with large disk space at my local ISP. Chucked down $5000 for a Sun RAID array with about 320Gigs of disk. Pretty damn expensive.

How I loved this time of my life. I was well known and in the headlines of Slashdot and Linux Gazette on a regular basis. I remember being able to pick up a book or magazine or two at Barnes and Noble that had my name and/or picture in it. I would be lying if I said I didn't miss that.

But deep down, I'm a developer. It didn't take long before I had that yearning to "do some real work" and by that I mean staying up all night in front of a shell prompt trying to figure out why that oops disappears when I add in some debug printk's or reverse engineer the endianness of an OHCI-1394 packet on sparc64. Anyways, on to better things I went and many more adventures awaited me.

As I moved through my career, I became more and more focused on Linux kernel work. From embedded to server, from network drivers to mpeg drivers, from MMUs to CPUs. I've never regretted a single step of the journey. As I sit here now, working from home at my newest job, I reflect, not with a sense of accomplishment, but with a sense of humility, knowing that there were many greater, smarter and harder working folks that traversed those same years making it all happen and enabling the opportunities that I've had.

So for anyone who stumbles upon this lonely blog entry, wondering what this whole free software thing is; take a seat, pour a cup of tea, and relax for a few minutes. It's probably the last time you will have that brief illusion of a normal life, but you wont miss it one bit.

Cheers