Today I Learned

6 posts about #devops

Stop a docker-compose container without removal

Suppose you need to stop your docker-compose container but you don't want to remove the container. For example, this container runs your database, so removing the container would mean that you would also lose all your data.

Don't use docker-compose down. This will remove your container and you will have rebuild your database from scratch. Sad panda.

Instead, use docker-compose stop. You can then restart the container and have all the data back that you had before.

Resources:

Chef override attributes vs chef-client -j switch

When applying recipes via chef-client, you can override attributes via the --json-attributes (or -j) switch. However, the override_attributes option on roles, environments and recipes has higher precedence over the --json-attributes switch.

For more info about Chef attribute precedence, use this as a reference. The attributes set via --json-attributes are normal attribute types.

Fixing ZDT Deploys with Monitoring-Driven Develop.

We were debugging a bug in our zero downtime deploy script. We tested the deploy using Chef, Kitchen and Vagrant. The bug was causing an error page to show for less than 3 seconds at the very end of the deploy.

We used Monitoring Driven Development to consistently reproduce the bug (the red phase), and to validate our fix (the green phase). Here were the steps:

Preparation:

  1. kitchen login # ssh to the vagrant box
  2. sudo tail -fn100 /var/log/nginx/go.access.log # to see the response codes from the server. If all responses show 200 code during the deploy, the deploy is working.
  3. Open other terminal window
  4. kitchen login # ssh to the vagrant box
  5. watch -n 1 "curl localhost" # make a request every 1 second

Then we followed this feedback loop

On the host computer:

  1. Make changes to the deploy recipe
  2. kitchen converge # this will run our deploy
  3. Check the output of the logs to see the result of the changes
  4. Repeat the loop until bug is fixed

Turn AWS tags into a useful data structure with jq

The JSON responses from the AWS API contain tags in a data structure like this:

"Tags": [
    {
        "Value": "consul-test-jf",
        "Key": "Name"
    },
    {
        "Value": "test-jf",
        "Key": "consul-group"
    },
    {
        "Value": "server",
        "Key": "consul-role"
    }
]

This structure is awkward to query with jq, but you can map it into a normal object like this:

jq '<path to Tags> | map({"key": .Key, "value": .Value}) | from_entries'

Which returns an object that looks like this:

{
  "consul-role": "server",
  "consul-group": "test-jf",
  "Name": "consul-test-jf"
}

ZDT Column Rename in a Distributed System

In order to deploy code to a highly available distributed system any two sequential versions of the code can be running at the same time. Therefore they need to be compatible.

  1. Add the new column, keep the columns in sync when updating.
  2. Migrate the data, start using the new column however fallback to the old column if the new column is blank, continue keeping the columns in sync.
  3. Remove all dependencies on the old column, only use the new column, do not sync them anymore.
  4. Drop the column.

When in Rails, Step #3 requires some special care as the column needs to be marked for removal:

module MarkColumnsForRemoval
  def mark_columns_for_removal(*columns_marked_for_removal)
    @columns_marked_for_removal = columns_marked_for_removal.map(&:to_s)
  end

  ##
  # Overrides ActiveRecord's list of the database columns in order to hide a column which we intend to delete
  # This ensures that ActiveRecord does not try to read or write to the column
  #
  def columns
    cols = super
    cols.reject { |col| (@columns_marked_for_removal || []).include?(col.name.to_s) }
  end
end

class SomeModel < ActiveRecord::Base
  # Remove this as part of step 4 when dropping the old_column
  extend MarkColumnsForRemoval
  mark_columns_for_removal :old_column
end

Add a role to all EC2 servers on a Chef Server

knife exec -E 'nodes.find("ec2:*") { |n|
  n.run_list << "role[awsrole]" unless
  n.run_list.include?("role[awsrole]"); n.save }'