Wikheu - Managing Drupal Multisite Releases with SVN

Posted Jun 4, 2010 // 1 comments

Multisite, the ability to run several websites from [almost] the same code-base is one of the most powerful and unique features of Drupal. You can deploy a "platform": the Drupal core and common set of contributed modules (typically located under "sites/all") only once and use this common base to build multiple websites. The websites, however, can have unique look-and-feel implemented via site-specific themes and can even customize functionality, through the use of additional modules, specific to the site, installed under: sites/sitedomain.com/modules.

Multisite is nothing new for Drupal old-timers. A lot of organizations use it to great success. There is one "problem" when using multisite, however: since the code for all sites is under a subfolder of the same "sites" folder, how do you separate releases for each website? Usually each site on the platform should have an independent release cycle. Just because you need to release code for site1, does not mean that you are ready to also release code for site2. Similarly, you should not have to tie "platform" releases with the releases of the individual sites in a multisite setup. However, if you deploy code from a version control (usually a good idea, since you can keep history of changes) having code with different release cycles in one place, in version control, can cause problems.

There are many different ways to solve this problem. The most obvious one is probably to use branches and merging. It's not necessarily the easiest or the most reliable one, however, especially if you have large and increasing number of sites and a somewhat large group of developers working on the code. Managing various branches can quickly become a tedious job causing more problems than solving ones.

Another solution could be to keep code for each site in separate repos and build them together when you need to deploy. It's a good solution and building a production deployment is a good technique, but if you keep code in completely separate repos you occasionally will have to make builds for the development process as well (e.g. new site added and all developer workspaces need to get it), not just release process. Now that is not cool, since it introduces additional coordination, manual work and as any such thing is potentially error-prone.

Ideally you would do "builds" for releases, but not need to mess with the build scripts for just development. It would be even better if "production builds" were versioned in a version control, as well.

Recently we, at Phase2, had to deal with the release management in a multisite environment and decided to try build the "ideal" solution mentioned above, using Subversion. We were deploying our platform to the Acquia's cloud hosting, which uses subversion for release management, anyway, so it was a great match. Following is the description of a Drupal-specific solution that we ended-up with and the explanation of the set of scripts that manage the release process.

To accompany this blog post, we are publishing full source code of the scripts, used in the solution, to GitHub, under GPL license. We created a project code-named Wikheu. (in Lenape, Delaware Indian the verb means: to build) http://github.com/phase2/wikheu and whole-heartedly welcome any suggestions to improvements or code contributions.

Code Layout

In our case, we needed to have multiple instances of OpenPublish running off of the same platform (Pressflow core + OpenPublish + some additional platform modules under sites/all).

Before we even get into release management, some housekeeping notes are due. It's always a good idea to try minimize changes to Drupal core and make core upgrades as isolated as possible. For that reason, we typically extract "sites" folder one level up the folder tree and leave a symlink in the core instead. This way we don't need to ever touch any file under the core's folder tree, we install and develop our customizations in a completely separate folder. Drupal core updates become trivial: download new core, replace stub "sites" folder with the symlink to "our" sites folder - done. Typically we also create "docroot" (or whatever you call Apache vhost root) folder as symlink to a core. This way we can easily switch between different core versions if needed.

In case of a multisite setup of OpenPublish (which is a distribution) we also extract and symlink "profiles" folder, because we use OpenPublish installation profile. We use it to install new sites of the multisite, and therefore "profiles" folder has custom code (OP profile). All-in-all the final layout looks something like the following diagram:

Code Release Workflow

We want the release process to be transparent to the developers, so the "trunk" of the Subversion repo (i.e. the development branch) is structured in the same exact way as described above. Developers can just check the code out of "trunk" and create an instant workspace ready for development. No build process required for the development.

To release a website, our scripts perform following tasks:

1. We take a "snapshot" of the website folder on the trunk and place it in a special sub-repo (with an svn copy) tagged with a release tag. We use uniform and fully automated naming convention for generating release tags: our tags are the current date (to the seconds precision). Since we use dates as tags, obviously time needs to be synced between all places where you may be running build from. In reality, we just agree to always run builds from a single "build" server.

2. We copy the latest release branch into a special tag: "live". "Live" branch/tag for a website always points to its latest release. This helps us keep up-to-date pointers to the latest website releases from a platform release. Since SVN does not support the notion of "floating tags" (the only time we miss CVS! :chuckle:) in reality we are just removing the old "live" branch and copying latest snapshot to "live", again.

After multiple releases of a website, the resulting structure, in SVN, would look something like the following diagram:

We think it's a good idea to create "releases" folder on the same level as the "trunk", but it is not a requirement. The location of the releases folder is configurable. You do need to pre-create "websites" and "platform" folders, under the "releases" folder, however.

A typical SVN structure would look something like the following:

Configuration

The structure described works really well for us. However, we realize that people may have different preferences, so wikheu build scripts allow for some flexibility through a configuration file. The git repo contains config.info.default file, which you need to copy to config.info and customize for your environment. The configuration file allows to set four values:

1
2
3
4

releases_location=https://svn.example.com/projname/releases
trunk_location=https://svn.example.com/projname/trunk
core_symlink_name=docroot
sites_location=https://svn.example.com/projname/trunk/sites

1. releases_location - indicates the SVN URL of the branch where you want your releases to be generated. That location MUST have "websites" and "profile" sub-folders!

2. trunk_location - SVN URL of the brach that has drupal code. It must have web root folder (e.g. wwwroot or html or whatever name you use) as its subfolder. It is also a requirement that web root folder be a symlink to drupal/pressflow core.

3. core_symlink the name of the symlink to the core. This allows to name it something else than "wwwroot" if need be.

4. sites_location - the SVN URL of the sites folder under trunk (developmen branch). We like it to be on the same level as core and be symlinked, because it helps core updates, but if for some reason you have different setup, this configuration option allows you to accommodate for that.

Running the Script

Both website deployment and platform deployment are managed from the same master script:

Deploying a new release of the platform is as easy as running:

1

./build platform

(make sure the "build" file has execution permission for the required users). Platform build script will print-out new release SVN location, which may look something like:

1

START: releasing platform under releases/platform/2010-06-03-133833

You use this information to switch SVN branch accordingly in the production. For full disclosure: this step could be automated but we decided releasing platform update should not be an implicit step, because it would be risky.

Similarly, to build a new release of a website:

1

./build site openherald.com

where openherald.com is the domain of the website (and the name of the site under sites folder). You do not need to svn switch when releasing a website, because platform refers to the sites through an svn:external pointing to a constant location called "live". You do need to perform an "svn update", under the "sites" folder, however.

Please also note that if you release a new website, you will need to rebuild platform, so that platform gets a chance to "see" the new website, but if you just release a new version of an already-released website, you don't need to release a new version of the platform.