Matt Connolly's Blog

my brain dumps here…

A Ruby gem to backup and migrate data from a Rails app

Ever wanted to quickly duplicate your database in a rails app so you can do some development work on real data, but not risk breaking it?

Background:

Rails has some very good strategies for managing your schema with rake tasks such as db:migrate, db:schema:load, etc. Switching between production and development environments lets you use different databases.

Great, but how do you get the data from one environment to another?

With MySQL you can duplicate a database (easy with phpMyAdmin), and with Sqlite3 you can simply copy the database.db file.

But what if I want to go from MySQL to Sqlite3 just for working??

There are some database differences, and there’s a rake task that’s already managing the database structure. I need something to move the data as well.

In my particular case, I’m using redmine as an issue tracker, hacking on it, and also evaluating the Chiliproject fork. So I’d like to be able to move all my data: schema, data and files easily from one to another.

The Rails-Backup-Migrate Gem

So, I put together a gem to do this. It’s inspired by Tobias Lütke’s backup.rake file he posted a good 5 years ago: http://blog.leetsoft.com/2006/5/29/easy-migration-between-databases

It appears to be for an early version of rails. It also had some issues with tables belonging to “many-to-many” relationships, and was also slow because it instantiated every single ActiveRecord object as it went.

I’ve sped the process up by skipping the ActiveRecord object steps, and updated the table definitions for rails 2.3 (redmine runs on this) and it appears to be working for rails 3.0 as well.

This gem provides the following rake tasks to a rails app:

rake site:backup[backup_file]                          # Backup everything: schema, database to yml, and all files in 'files' directory.
rake site:backup:db[backup_file]                       # Dump schema and entire db in YML files to a backup file.
rake site:backup:files[backup_file]                    # Archive all files in the `files` directory into a backup file.
rake site:backup:schema[backup_file]                   # Dump schema to a backup file.
rake site:restore[backup_file]                         # Erase and reload db schema and data from backup files, and restore all files in the 'files' directory.
rake site:restore:db[backup_file]                      # Erase and reload entire db schema and data from backup file.
rake site:restore:files[backup_file]                   # Restore all files in the 'files' directory.
rake site:restore:schema[backup_file]                  # Erase and reload db schema from backup file.

The default backup_file is "site-backup.tgz" and is relative to rails app root directory.

Installing:

Installing manually:

$ gem install rails-backup-migrate

Installing with Bundler:

Add the following to your Gemfile:

gem rails-backup-migrate

And then

$ bundle install

Using rails-backup-migrate:

To use the gem in your rails app, simply add the following line to the end of your Rakefile:

require 'rails-backup-migrate'

Which will load the above tasks into your rails app’s task list. If you’re using bundler

$ # backup everything into the default file 'site-backup.tgz'
$ rake site:backup
$ # now, restore into another instance (eg: clone/checkout another copy of your project)
$ cd /path/to/another/instance/of/my/rails/app
$ rake site:restore[/path/to/original/instance/of/my/rails/app/site-backup.tgz]

Done.

The source code is available on github: https://github.com/mattconnolly/rails-backup-migrate

This is the first gem I’ve made and published, and I hope it makes life a little bit easier for someone!

Advertisements

8 responses to “A Ruby gem to backup and migrate data from a Rails app

  1. Todd 6 August, 2011 at 06:50

    This looks pretty handy. I tried it and got:

    rake aborted!
    “\xE2” from ASCII-8BIT to UTF-8

    I looked at my database.yml and they’re set to utf8 and the collation on the database tables is “utf8_unicode_ci.” Any thoughts?

  2. Kiire Kala 20 February, 2012 at 05:29

    Looks like your github link could use some updating:
    Is it
    https://github.com/mattconnolly/rails-backup-migrate
    perhaps?

  3. Siddhant Chothe (@sidnc86) 20 March, 2013 at 22:35

    looked quite nice first up when I read about this on github and then here. Looked like backup_migrate module in drupal. Tried it on windows and got this though

    rake aborted!
    cannot load such file — rails-backup-migrate

    What has worked so far:
    gem install rails-backup-migrate
    Have also installed tar command from
    http://gnuwin32.sourceforge.net/packages/gtar.htm

    Any help is appreciated

  4. mechonian5 10 March, 2016 at 17:38

    I have been using this gem with great success on mac so far, but it throws random error on windows, could you please shed some light on windows backup and restoration? am getting this error “rake : No such file or directory – tar -xzf”
    tar (child): Cannot connect to B: resolve failed

    gzip: stdin: unexpected end of file
    tar: Child returned status 128
    tar: Error is not recoverable: exiting now

    Any help is highly appreciated!

    • mattconnolly 11 March, 2016 at 00:59

      The gem relies on UNIX command line tools to create the archive. These don’t exist on Windows.

      I have no plans to add windows support but would accept a pull request if you were inclined to write one.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: