Friday, March 25, 2016

Easy Rake-based Deployment for Git-hosted Rails Apps

I searched a lot of places for an easy way to automate my deployments for OpenStrokes, a website I've been working on. Some things were just too complex (capistrano) and some were way too simple (StackOverflow answers that didn't do everything, or didn't check for errors).

So, as most people do, I wrote my own. Hopefully this short rake task can help you as well. This assumes that your application server has your app checked out as a clone of some git repo you push changes to and that you are running under passenger. When I want to deploy, I log in to my production server, cd to my app repo, and then run:

rake myapp:deploy

For just strictly view updates, it completes in 3 seconds or less. There are several things it does, in addition to checking for errors:

  • Checks to make sure the app's git checkout isn't dirty from any local edits.
  • Fetches the remote branch and checks if there are any new commits, exits if not.
  • Tags the current production code base before pulling the changes.
  • Does a git pull with fast-forward only (to avoid unexpected merging).
  • Checks if there are any new gems to install via bundle (checks for changes in Gemfile and Gemfile.lock).
  • Checks if there are any database migrations that need to be done (checks for changes to db/schema.db db/migrations/*).
  • Checks for possible changes to assets and precompiles if needed (checks Gemfile.lock and app/assets/*).
  • Restarts passenger to pick up the changes.
  • Does a HEAD request on / to make sure it gets an expected 200 showing the server is running without errors.

The script can also take a few arguments:

  • :branch Git branch, defaults to master
  • :remote Git remote, defaults to origin
  • :server_url URL for HEAD request to check server after completion

Note, if the task encounters an error, you have to manually complete the deploy. You should not rerun the task.

Any finally, here is the task itself. You can save this to lib/tasks/myapp.rb

# We can't use Rake::Task because it can fail when things are mid
# upgrade

require "net/http"

def do_at_exit(start_time)
  puts "Time: #{(Time.now - start_time).round(3)} secs"
end

def start_timer
  start_time = Time.now
  at_exit { do_at_exit(start_time) }
end

namespace :myapp do
  desc 'Deployment automation'
  task :deploy, [:branch, :remote, :server_url] do |t, args|
    start_timer

    # Arg supercedes env, which supercedes default
    branch = args[:branch] || ENV['DEPLOY_BRANCH'] || 'master'
    remote = args[:remote] || ENV['DEPLOY_REMOTE'] || 'origin'
    server_url = args[:server_url] || ENV['DEPLOY_SERVER_URL'] || 'http://localhost/'

    puts "II: Starting deployment..."

    # Check for dirty repo
    unless system("git diff --quiet")
      puts "WW: Refusing to deploy on a dirty repo, exiting."
      exit 1
    end

    # Update from remote so we can check for what to do
    system("git fetch -n #{remote}")

    # See if there's anything new at all
    if system("git diff --quiet HEAD..#{remote}/#{branch} --")
      puts "II: Nothing new, exiting"
      exit
    end

    # Tag this revision...
    tag = "prev-#{DateTime.now.strftime("%Y%m%dT%H%M%S")}"
    system("git tag -f #{tag}")

    # Pull in the changes
    if ! system("git pull --ff-only #{remote} #{branch}")
      puts "EE: Failed to fast-forward to #{branch}"
      exit 1
    end

    # Base command to check for differences
    cmd = "git diff --quiet #{tag}..HEAD"

    if system("#{cmd} Gemfile Gemfile.lock")
      puts "II: No updates to bundled gems"
    else
      puts "II: Running bundler..."
      Bundler.with_clean_env do
        if ! system("bundle install")
          puts "EE: Error running bundle install"
          exit 1
        end
      end
    end

    if system("#{cmd} db/schema.rb db/migrate/")
      puts "II: No db changes"
    else
      puts "II: Running db migrations..."
      # We run this as a sub process to avoid errors
      if ! system("rake db:migrate")
        puts "EE: Error running db migrations"
        exit 1
      end
    end

    if system("#{cmd} Gemfile.lock app/assets/")
      puts "II: No changes to assets"
    else
      puts "II: Running asset updates..."
      if ! system("rake assets:precompile")
        puts "EE: Error precompiling assets"
        exit 1
      end
      system("rake assets:clean")
    end

    puts "II: Restarting Passenger..."
    FileUtils.touch("tmp/restart.txt")

    puts "II: Checking HTTP response code..."

    uri = URI.parse(server_url)
    res = nil

    Net::HTTP.start(uri.host, uri.port, :use_ssl => uri.scheme == 'https') do |http|
      req = Net::HTTP::Head.new(uri, {'User-Agent' => 'deploy/net-check'})
      res = http.request req
    end

    if res.code != "200"
      puts "EE: Server returned #{res.code}!!!"
      exit 1
    else
      puts "II: Everything appears to be ok"
    end
  end
end

Here's an example of the command output:

$ rake myapp:deploy
II: Starting deployment...
remote: Counting objects: 15, done.
remote: Compressing objects: 100% (8/8), done.
remote: Total 8 (delta 6), reused 0 (delta 0)
Unpacking objects: 100% (8/8), done.
From /home/user/myapp
   efee45c..e5468c1  master     -> origin/master
From /home/user/myapp
 * branch            master     -> FETCH_HEAD
Updating efee45c..e5468c1
Fast-forward
 app/views/users/_display.html.erb     | 7 +++++--
 public/svg/badges/caretakers-club.svg | 1 -
 2 files changed, 5 insertions(+), 3 deletions(-)
 delete mode 100644 public/svg/badges/caretakers-club.svg
II: No updates to bundled gems
II: No db changes
II: No changes to assets
II: Restarting Passenger...
II: Checking HTTP response code...
II: Everything appears to be ok
Time: 3.031 secs

Friday, August 21, 2015

Encryption is not for the bad guys

I've been reading a lot of articles about presidential candidates and their stances on encryption for online privacy. It befuddles me how ridiculous their arguments are. You hear things like "only evildoers use encryption" or "if you have nothing to hide, you shouldn't be against this."

These are the same arguments that gave us our miranda rights, protection against illegal search and seizure, etc. Just because a citizen exercises their right to privacy does not mean they are hiding something, nor can it be taken as probable cause to remove their rights.

Saying that law enforcement needs to remove encryption so they can find bad guys is like saying houses need to have open windows so they can see in to houses. It's like saying we should do away with home security systems because only people with illegal items use them.

Any candidate that wants to keep me from protecting my privacy and security on the grounds of protecting me from some would-be terrorist is on a power trip and does not stand for the same constitution that I do.

Monday, July 27, 2015

How to Alienate Your Customers And Drive Your Online Service Into the Ground

So you've spent months, maybe years, building an online service. You have lots of customers and people are starting to rely on your service for personal and/or business purposes. It has a great reputation and it's basically the only one of its kind. Now what?

If you're like one particular service that I started using a couple of months ago, you ruin all of your good will and hard work inside of a week. I'm not about to call out who this service is, but I will gladly tell you exactly how they went about this epic failure of standard practices. So let's get started.

Step One: Plan a maintenance window


You've got to keep these customers happy with new features, not to mention, you've got to make room for the unexpected growth you've seen.

So let's do it all at once:

  • Lots of new customer facing features? Check
  • Lots of backend features to handle unexpected growth? Check
  • Move to new hardware? Check
  • Database updates across billion-row tables? Check
  • Test plan using mirrored copy of current data set? Ain't nobody got time for dat.
  • Backout plan in case something goes wrong? Pshaw, we won't need one!
  • Timetable for how long this operation will take? Eh, can't take too long, right?

Here's the thing folks; never, and I mean never, run a maintenance window that involves multiple moving parts. If you can't perform these actions individually, then you've messed up somewhere in building this thing out from the start. If your upgrades require moving to new hardware, then do it separate from the rest.

Move to new hardware with the existing application and data. Don't mix features that aren't interdependent and make sure to test any database migrations on a full backup data set (or at least a good portion of it) before doing it on the live data set.

Next, always have a way to go back. If you're upgrading a database, have some way to revert back to the original in an instant. Whether it's a backup, a snapshot, or whatever. Don't depend on being able to back out the change you've made (i.e. running more SQL commands on the live data). You want a pristine place to draw from.

But now that you've screwed that up, let's continue on.

Step Two: Don't ever take the blame


Now that you've pushed this "New and Improved" version of your service in the most obscenely unprofessional way, you will definitely have something go wrong. It's not an if or when, it's just going to happen. First off, don't bother checking to make sure everything went well. Just go to bed and pat yourself on the back.

When you wake up in the morning and see things aren't working as you expect, don't bother replying to the customers' cries for help just yet. Let them know who's boss and who runs this joint. You, that's right. After awhile, give a little update. Remember what your english teacher taught you; less is more. Something like this will do just fine:

We're working hard to make our service better. Please bear with us while we continue to do so.

Some companies forget that their customers are not stupid. We know when something's wrong and in cases like this, we know you screwed up somehow. Don't make it worse by glossing it over. Gives us the straight poop, so to speak. The main word here is transparency. Most companies have this knee-jerk reaction of trying to make it look like "oh, we just had some bad luck, we didn't do anything wrong."

Believe me, even if you aren't transparent, the fact that you aren't is pretty transparent to us. Just like when my kids were 3 years old, I could tell when they were lying ("But dad, I swear, it was the dog that ate all the cookies mommy just baked"...as cookie crumbs fell from his face).

Now that you're busy ignoring the flames on the customer front, let's take care of this problem.

Step Three: Take your time


Who's in a hurry? Not this guy! Amiright? It's already broken, you have no capability to revert all of this crap, and you have to get these new features to the masses else what was it all for? Just keep pushing forward like an angry crowd at a music festival.

What you need to remember is that all of this work is for naught if your customers all leave. You have to be able to suck it up and back all of this out to get back to a stable base and come back at it later. The particular failure that I saw this past week may have been able to revert all of the mess they started. I don't know (they didn't talk much). I can only assume that a) they couldn't revert (bad planning) or b) they were so consumed with making this work, they decided to push forward.

It's hard to say which scenario is worse, but if you do find yourself in a position where an upgrade has broken things and you are able to revert back to a known good state, don't let your ego get the best of you. Just REVERT, go back to the drawing board, perform some postmortem, and start up again on a fresh day.

Lastly, above all else, talk to your customers regularly through the process. Waiting 4-15 hours between updates is a super bad idea. Making these status updates vague and absolving yourself of any responsibility is mistake number two.

Just remember, if you own a particularly new market, the only time a competitor will even think of jumping in is when you lose the trust of your customers. They will capitalize on your mistakes.

Thursday, March 7, 2013

Power Up

As a POWER architecture hardware vendor, we've definitely run into quite a few wish-list items for software we want to have on our platform. Whether it's for customers or just to have a feature complete set of packages in everyday distributions, we want to see things build everywhere, and run just as well as the x86 counterparts.

Starting soon, we are kicking off a PowerUp (my cool label) initiative in order to direct POWER developers toward software that needs a little love on our POWER platforms. Software targets range from the completely unsupported (e.g. Google's v8 Javascript engine, D Languages's phobos) to optimizing specifically for POWER (e.g. OpenJDK).

To collect these initiatives together, we will be starting a new PowerUp portal. For now, we have begun a GitHub Team where we have forked relevant repositories. Forums for discussion and participation will also follow. Feel free to clone and hack away. Email me if you have any questions (or wait until the forums and portal open).

NOTE: PowerUp is just my initial name. That may or may not change.

I'll update this blog post when more information is available.

Wednesday, March 6, 2013

Ubuntu Rolling Releases Vs. Hardware Companies

So I have to speak out on this whole issue. I work for Servergy, and for almost two years I've been working on Ubuntu's PowerPC port in order for our new hardware platform, the CTS-1000, to have an out-of-the-box solution for our customers. We've been hedging on Ubuntu, since it was able to provide us a known quantity for release dates and an open community that we could participate in (especially being able to take advantage of my core-developer status).

Now, after so much work, so much planning, we are worried about 13.04 never being properly released. This would leave us with no stable Linux distribution for our hardware, basically yanking the rug out from under all of our work. Having a stable release every two years also enlarges the support gap for our followup platforms. Now I realize most hardware vendors are x86-based, and their issues are likely limited to supporting peripherals, so this affects us more than most. The issue we face is supporting entirely new hardware platforms and SoCs with a completely new kernel (likely requiring lots of supporting patches). This is the type of thing that, historically, isn't allowed to be added to an LTS release.

So I have to wonder, if Ubuntu does adopt this rolling release schedule, how viable is it for us? I would still be happy if Ubuntu had one release per year, with every other release becoming an LTS. However, the two year window is just entirely too large to depend on for quick moving hardware bring up and release.