One-click website deployment using TeamCity, NAnt, Git and Powershell.

Problem

Until recently my organisation relied on FTP to push site updates. Whilst this certainly seems like a simple solution at first glance it soon becomes unmanageable. To name but a few of the difficulties:

• For large sites re-uploading every file takes a long time, so developers tend to try and push individual files relating to their set of changes. Nobody really knows which version of the build is live and files soon get out of sync and the site collapses.
• If something does go wrong things have to be restored from a backup, which again takes time.
• Tracking down bugs is difficult it’s nigh-on impossible to correlate the live site to a particular build source control.
• Web farms are particularly problematic because the files have to manually be pushed to multiple nodes.

Requirements

We needed a deployment solution to fit the following requirements:

• Bandwidth-friendly – deploys should only be transferring changes, not re-uploading the entire site.
• Quick to switch between builds to minimise downtime
• Easy to revert to previous builds
• Able to easily deploy to multiple servers (web farms)
• Platform-independent. We are primarily a Microsoft shop but didn’t want the deployment system to be tied to Visual Studio/ASP.NET.
• Easy to deploy. Ideally it should be as simple as clicking ‘Build’ in Teamcity.
• Ability to have multiple configurations, for example testing, staging and live.

Solution

Overview

The solution essentially boils down to the following:

• Use NAnt to build our solutions and apply any configuration-specific changes (for example, web.config changes which need to be made for staging/live).
• Use a version control system (git) to hold each successful build.
• Have each server in the web farm frequently pull and apply the latest changes from the build server.

Requirements

You will need the following tools installed:

On the build server:

• NAnt
• Git (My suggestion would be to install Git Extensions which includes everything you need plus a few nice GUI tools for windows users)

On each web server:

• Powershell
• Git (again, I would recommend installing Git Extensions)

Step 1 – Build configurations

Create a ‘DeploymentOverrides’ directory in the root of your solution and a subdirectory for each configuration (development, testing, staging, live etc.) The idea is that the contents of the relevant folder will be copied over the top of each successful build. In the case of ASP.NET for example you might want to put a different web.config in each folder.

Step 2 – Create Git Repositories for the build outputs

On the build server you’ll need to create git repositories to hold the successful outputs from each build. For example:

mkdir F:\GitRepositories\Testing\MyWebsite
mkdir F:\GitRepositories\Staging\MyWebsite
mkdir F:\GitRepositories\Live\MyWebsite

F:
cd F:\GitRepositories\Testing\MyWebsite
git init

cd F:\GitRepositories\Staging\MyWebsite
git init

cd F:\GitRepositories\Live\MyWebsite
git init

Step 3 – NAnt build script

NAnt is a great tool to automate builds. It uses an XML configuration to describe the different ‘targets’. The following example script holds a few different configuration options (Local, Testing, Staging and Live).

When NAnt is run the solution is compiled using MSBuild, the overrides are copied over from the relevant subdirectory of DeploymentOverrides, and the resultant output is committed its corresponding git repository.

By convention this file should be called default.build and placed in the root of your solution.

<?xml version="1.0"?>
<project name="YourProjectName" default="default">

    <!--
    Build configurations:
    Local        - For local builds/testing
    Testing    - Teamcity deploy to development server
    Staging - Teamcity deploy to staging server
    Live        - Teamcity deploy to live servers
    -->

  <property name="solutionFilename" value="${project::get-name()}.sln" />

    <!-- This is the source folder. i.e. set this to the folder containing the output of the build -->
  <property name="msBuildOutputFolder" value="Website" />

  <!-- manual targets to override above default -->
  <target name="local">
      <property name="deployTarget" value="local" />
  </target>
  <target name="testing">
    <property name="deployTarget" value="testing" />
  </target>
  <target name="staging">
      <property name="deployTarget" value="staging" />
  </target>
  <target name="live">
      <property name="deployTarget" value="live" />
  </target>

  <target name="setEnvironmentalProperties">
    <!-- Set deployment target to local if not explictly specified -->
    <if test="${not property::exists('deployTarget')}">
          <fail message="Must specify valid build target."/>
      </if>

    <call target="setEnvironmentalPropertiesLocal" if="${deployTarget=='local'}" />
  <call target="setEnvironmentalPropertiesTesting" if="${deployTarget=='testing'}" />
  <call target="setEnvironmentalPropertiesStaging" if="${deployTarget=='staging'}" />
  <call target="setEnvironmentalPropertiesLive" if="${deployTarget=='live'}" />

  </target>

  <target name="setEnvironmentalPropertiesLocal">
    <!-- Web config settings -->
    <property name="deploymentOverridesSource" value="DeploymentOverrides/Local" />

    <!-- This is the destination git repository used to hold successful builds -->
    <property name="gitRepository" value="SuccessfulBuilds/Local" />
  </target>

    <target name="setEnvironmentalPropertiesTesting">
    <!-- Web config settings -->
    <property name="deploymentOverridesSource" value="DeploymentOverrides/Testing" />

    <!-- This is the destination git repository used to hold successful builds -->
    <property name="gitRepository" value="F:/GitRepositories/Testing/MyWebsite" />
    </target>

    <target name="setEnvironmentalPropertiesStaging">
    <!-- Web config settings -->
    <property name="deploymentOverridesSource" value="DeploymentOverrides/Staging" />

    <!-- This is the destination git repository used to hold successful builds -->
    <property name="gitRepository" value="F:/GitRepositories/Staging/MyWebsite" />
  </target>

  <target name="setEnvironmentalPropertiesLive">
        <!-- Web config settings -->
    <property name="deploymentOverridesSource" value="DeploymentOverrides/Live" />

    <!-- This is the destination git repository used to hold successful builds -->
    <property name="gitRepository" value="F:/GitRepositories/Live/MyWebsite" />
  </target>

  <target name="default" depends="compile" />

  <target name="compile">
    <msbuild project="${solutionFilename}">
        <property name="Configuration" value="Release"/>
    </msbuild>
  </target>

  <target name="deploy" depends="compile, addToGit" />

  <target name="updateSettings" depends="setEnvironmentalProperties">
      <!-- Copy config files for this build configuration over the output from the build -->
      <copy todir="${gitRepository}" overwrite="true">
      <fileset defaultexcludes="false" basedir="${deploymentOverridesSource}">
          <include name="**/*" />
          <exclude name=".do-not-delete" />
    </fileset>
  </copy>
  </target>

    <!-- ************************************************************** -->
    <!-- *** Tasks to add successful build output to git repository *** -->
    <!-- ************************************************************** -->

  <target name="copyBuildOutputToGit" depends="setEnvironmentalProperties">

  <!-- Delete entire working folder from git leaving only the main .git folder behind -->
  <delete>
      <fileset defaultexcludes="false" basedir="${gitRepository}">
          <include name="**/*" />
          <exclude name=".git" />
          <exclude name=".git/**" />
    </fileset>
  </delete>

  <!-- Copy entire output of successful build into the git working folder -->
  <copy todir="${gitRepository}">
      <fileset defaultexcludes="false" basedir="${msBuildOutputFolder}">
          <include name="**/*" />
    </fileset>
  </copy>
  </target>

  <target name="addToGit" depends="setEnvironmentalProperties, copyBuildOutputToGit, updateSettings">
  <!-- Commit the contents of the working folder to the git repository -->

  <!-- Write timestamp of build into a file. Useful for reference but also ensures there is always
       a change to commit. This way if commit fails we know there was actually an error.
       (git commit fails if the index is empty)
   -->
  <tstamp />
  <echo message="NAnt build successful at ${tstamp.now}" file="${gitRepository}/build.log" append="false"/>

  <!-- This check is necessary so that we don't inadvertently end up checking in files
  to the source code repository if the build repository doesn't exist! -->
  <if test="${not directory::exists(gitRepository + '/.git')}" >
         <fail message="Git repository for build output is not initalised!"/>
  </if>

  <!-- Stage files to git index -->
  <exec append="true" workingdir="${gitRepository}" program="git">
      <arg value="add" />
      <arg path="${gitRepository}/." />
  </exec>

  <!-- Commit files -->
  <exec append="true" workingdir="${gitRepository}" program="git">
      <arg line="commit" />
      <arg value="-a" />
      <arg value="-m" />
      <arg value="Successful build: ${tstamp.now}" />
  </exec>

  <!-- Update server info (for HTTP repositories) -->
  <exec append="true" workingdir="${gitRepository}" program="git">
      <arg line="update-server-info" />
  </exec>
  </target>

</project>

Step 4 – Set up Teamcity (optional)

You will want to create a teamcity build for each different configuration (Testing, Staging, Live etc). Be sure to choose nant as the runner and set the target appropriately.

You can of course skip this step and simply call nant manually from the command line if you wish.

Step 5 – Expose the git repositories

There are various ways to do this, but since we are only going to be pulling from the repository then setting it up as HTTP is the simplest method and provides some simple security (basic authentication).

I simply pointed create a new IIS website with its root at F:\GitRepositories and enabled basic authentication. Depending on your security requirements etc you may want to contemplate using SSL or other means of exposing the repository to the web servers (VPN, SCP, Samba etc).

Step 6 – Clone the git repositories onto the web servers

On the web servers, you will need to clone the appropriate git repository from the build server into the root each of your IIS applications, for example:

git clone http://username:password@mybuildserver.com/Live/MyWebsite

Step 7 – Set up the servers to regularly pull the latest updates

For simplicity, and because we needed to be able to spin up servers on demand without adjusting the configuration,we chose to have the web servers continuously poll for updates from the build server. This means that the build server doesn’t need to know anything about the web servers.

A push-based system might be more efficient (less network chatter, no need for polling) but it is left as an exercise for the reader!

In order to pull updates we just need to tell git to fetch and merge the latest changes. The following powershell script will automate this process and automatically fetch updates for every git repository. As a bonus, it also writes status updates to the windows event log.

# Perform fetch on all git repositories immediately beneath F:\git-deploy-repos
F:
cd F:\git-deploy-repos
dir | %{
    if (test-path "$_\.git")
    {
        echo "Performing fetch on : $_"
        cd $_
        git fetch
        # Writing an event
        $EventLog = New-Object System.Diagnostics.EventLog('Application')
        $EventLog.MachineName = "."
        $EventLog.Source = "Fetch-Updates"
        if ($?)
        {
        echo "Fetch on $_ completed"
        $EventLog.WriteEntry("Successfully fetched updates for $_","Information", $EventID)
                echo "Applying changes to dev build: $_"
                git clean -f -d
                git reset --hard head
                git merge origin/master
                if ($?)
                {
                    echo "Changed succesfully applied to: $_"
                    $EventLog.WriteEntry("Successfully applied updates for build $_","Information", $EventID)
                }
                else
                {
                    echo "Failed to apply changes to: $_"
                    $EventLog.WriteEntry("Failed to apply updates for build $_","Error", $EventID)
                }
        }
        else
        {
        echo "Fetch on $_ failed"
        $EventLog.WriteEntry("Failed to fetch updates for $_","Error", $EventID)
        }
        cd ..
    }
}

Set up a windows scheduled task to run this powershell script every few minutes and you are done 🙂

Step 8 – Take it for a spin!

Now all the configuration is done deployment is as simple as clicking ‘build’ in teamcity. This will build the solution, copy in any configuration-specific overrides (web.config etc.), and add it to the local git repository. The next time the scheduled task runs on each of the web servers the changes will be pulled over the network and then applied to the live website.

If you need to revert to the previous build you can simply run the following command on each of the web servers.

git reset --hard HEAD^

Of course you can revert to any build you like! Git Extensions really comes into its own in this scenario because it means you can easily visualise the timeline and view changes between different builds. Switching between builds a couple of clicks!

See also

Rob Conery has also proposed a git-based solution for website deployments which is worth a read: http://blog.wekeroad.com/2009/11/23/deploying-a-web-application-with-git-and-ftp