Paul Gross's Blog

My current project has a CruiseControl.rb build dedicated to testing our database rollback scripts. Since this does not seem to be a common practice, I thought that I would share our experience.

The setup:

We have one DDL file and one DML file per release. These are written using rails migration syntax, but we hand over only SQL to the client for releasing. The client wants a rollback script for each release to rollback the database schema and data (as much as possible) to the previous release.

The problem:

Rollback scripts are hand created and prone to bugs, so we wanted to test them.

Our solution:

We set up a cruise loop specifically designed to test the rollback script. Our criteria for determining if the rollback was successful is:

• The schema of the old version matches the schema of the rolled back database
• The content tables are identical to the old versions. These are tables full of static data that we show on the page. They change with each release as the web pages evolve.
• We can run the old version of the tests against the rolled back database.

Say the team is currently working on release 3. Here is what the cruise build loop does:

1. Builds the database to release 2 by running each version of the DDL and DML in succession (eg, DDL v1, DML v1, DDL v2, DML v2)
2. Dumps the database schema to a file using “rake db:schema:dump”
3. Copies the static data tables to backup tables, eg: content -> content_last_release
4. Builds the database to release 3 (DDL v3, DML v3)
5. Runs the rollback script
6. Dumps the database schema again and compares it with the saved file from step #2
7. Checks that the static tables match the backups (content and content_last_release are identical)
8. Runs the test suite from the release 2 branch on the current database

Notes

When comparing the content tables, it is tempting to write code that pulls the data from both tables as arrays and then does an assert_equal:

content = [..]
content_last_release = [..]
assert_equal content_last_release, content

However, when this assertion fails, the output will be two huge arrays that are mostly similar. It is very hard to look at the output and figure out what is different. We used code like this instead:

content = [..]
content_last_release = [..]
assert_equal [], content - content_last_release
assert_equal [], content_last_release - content

Now, only the differences are displayed when it fails.

The Results

The cruise loop has been incredibly helpful. We found numerous bugs in the rollback script. Additionally, we no longer forget to update it when we make database changes.

Obviously, there is always some data that cannot be rolled back. The rolled back database will not be identical to the database before the upgrade. However, this cruise loop ensures that it is as similar as possible.

I ran into a subtle problem yesterday that I feel is worth sharing. I was setting up SSH public keys so one box could ssh to another without a password prompt. (See OpenSSH Public Key Authentication for more info.) It was all set up correctly. The client had an id_rsa and id_rsa.pub, and the server had the id_rsa.pub in the authorized_keys file.

It just didn’t work, however, and I spent quite some time trying to figure it out. Turning on verbose output on the client didn’t help much, and I didn’t have the necessary access on the server to see logs.

Finally, I stumbled upon the solution listed at the OpenSSH FAQ. Apparently, if the authorized_keys file has permissions for users other than the owner, the authentication will fail. It is common for programs to fail because of too few permissions. But this is the first time I’ve seen a program fail because a file had too many permissions. Once I removed the extra access (chmod 600), everything worked fine.

I was playing with some code the other day, and I noticed that constants work strangely in class_eval blocks. Sometimes it can find the constant, and sometimes it cannot.

For example, say we have a class with a constant:

class Foo
  CONS = 'const'
end

Now, if we reopen the class and print the constant, it works as expected:

>> class Foo
>>   puts CONS
>> end
const

However, if we use a class_eval instead of reopening the class, ruby cannot find the constant:

>> Foo.class_eval { puts CONS }
NameError: uninitialized constant CONS
        from (irb):12
        from (irb):11:in `class_eval'
        from (irb):11

It gets even stranger. We can still see the constant in the list of constants:

>> Foo.class_eval { puts self.constants }
CONS

Furthermore, a const_get still works:

>> Foo.class_eval { puts const_get(:CONS) }
const

Finally, if we try the class_eval again with a string instead of a block, it works:

>> Foo.class_eval "puts CONS"
const

Apparently, constants have a problem with blocks inside of class_eval.

Update (2/20/09): Check out Useful unix tricks – part 2, Useful unix tricks – part 3 and Useful unix tricks – part 4

My current project uses macs for development and linux boxes for build and deployment. I also run linux on my personal computers, so I have learned the value of command line proficiency. Here are a few tricks that I think are worth sharing. I learned some of these on my own, and I picked up others while on this project.

Ctrl+r searches through history

Are you tired of using the up arrow to try to find old commands? Ctrl+r lets you search the history. Type Ctrl+r and start typing part of the old command. Pressing Ctrl+r again finds the next match. Once you find the one you like, just press enter to run it or the left or right arrows to modify it.

Use the history command plus !# to run old commands

Searching the history with Ctrl+r shows you old commands one at a time. Use the history command to see a list of them instead.

% history
    1  cd /tmp
    2  scp user@host:~/my_file .

Now, if you want to run one of those old commands, use !#. For example, to run the scp command again, type:

!2

Source command runs file in current shell

The source command is especially useful when shell config files have changed. For example:

% echo alias ls="ls -F" >> .zshrc
% source .zshrc

This will run the .zshrc file in the current shell. There is no need to open a new terminal to see the changes.

Launch a shell with sudo to run multiple commands

Update (2/16/08): Use “sudo -s” instead of “sudo bash” to start a shell as root.

If you have to run multiple commands with sudo, consider just running “sudo zsh” or “sudo bash” to start a shell as root. If you have sudo rights, but cannot sudo a shell, try this: Run “sudo vi” and then type :shell to open a shell within vi.

Shell expansion using {}

% echo a{a,b,c}
aa ab ac

The {} contain a comma separated list of expansions. One common use is to back up a file. For example, the command:

cp foo.sh foo.sh.old

can be rewritten as:

cp foo.sh{,.old}

zsh is a great shell

zsh is my favorite shell, and I don’t think most people understand how great it is. Here are a few things that it can do:

Tab completion is application specific. For example, type svn and then press tab:

% svn
add       cleanup   diff      info      merge     propedit  resolved  unlock
blame     commit    export    list      mkdir     propget   revert    update
cat       copy      help      lock      move      proplist  status
checkout  delete    import    log       propdel   propset   switch

zsh also tab completes options. For example, type mkdir – and then press tab:

% mkdir -
--context  -Z  -- set SELinux context
--help         -- display help information
--mode     -m  -- set permission mode
--parents  -p  -- make parent directories as needed
--verbose  -v  -- print message for each created directory
--version      -- display version information

These both work for many common programs. Just remember to put the following in your ~/.zshrc file:

autoload -U compinit
compinit

I also like changing my prompt to something that looks like

host:~/my/current/directory%

using the following lines in my .zshrc file:

export PS1="%m:%~%# "

zsh adds a -D option to history to display how long each command took. This is helpful since I don’t have to remember to time my commands up front:

% history -D
    1  0:00  cd myproject
    2  0:07  rake test

The filename completion is also a little better than bash. Repeatedly pressing TAB will cycle through all completions.

Tab competion even works across scp/rsync/etc. For example, typing the first line and pressing tab will show all of the files that I can scp:

% scp paul@host:/etc/apache2/
README                 envvars                mods-enabled/          ssl/
apache2.conf           httpd.conf             ports.conf             svn-auth-file
apache2.conf.dpkg-old  magic                  sites-available/       webdav-auth-file
conf.d/                mods-available/        sites-enabled/

However, set up ssh keys before trying this trick unless you want zsh prompting for a password on each completion attempt.