For some time now I’ve been thinking about all the different approaches that I have seen and heard with the goal of zero-downtime deployment for a horizontally-scaled application that relies on a database or some form of persistent storage. On most of the projects that I have worked on this was not a requirement, although a speedy and automated deployment was preferred. However, other projects were for websites and applications that formed the backbone of a 24/7 company and as such the idea of even a couple of minutes downtime was unacceptable. We usually applied a variant of a rolling deployment, so how did we keep more than one version of an application serving customers in production?

The following is a list of ideas and approaches (in no particular order of importance) that I have either thought up or shamelessly borrowed from those who are way more clever than me:

  • Don’t deploy applications and schemas at the same time. Pick one, and be consistent.
    • Usually it is simpler to push applications out first so that they are ready for the updates to the schema. In addition, it is also easier to back-out a failed application deployment.
  • Try to decouple domain logic from database-backed objects - treat the database as another (potentially hostile) integration point and don’t let persistent objects into your domain. You may have a devil of a time digging them out.
  • Don’t integrate between systems at the database level. Please for the love of whomever you find important, wrap that stuff in a service. SOAP, REST, POX, JSON, whatever … just stop peeking into other applications’ data.
  • Applications should ignore new columns/attributes if they are not yet ready for them, and be lenient with missing columns/attributes. This may mean that you have to be clever in how you map from the database to your application.
    • ActiveRecord - actually quite useful when you have extra rows or missing rows, since you can interrogate the domain object about its capabilities.
    • iBatis/myBatis - can be quite flexible in which set of queries you use to retrieve or update objects.
    • Spring JdbcTemplate and query objects - relatively straight-forward to figure out if a particular column exists and then use the appropriate query/statement.
    • Hibernate - can be a bit of a bugger in getting it to behave, especially if columns are removed. You usually wind up creating common interfaces for persistent classes, with multiple versions of implementing classes. You can decide which set of classes to wire into Hibernate by using schema or result set metadata before initialising the SessionFactory. Worse issues exist for other persistence technologies in Java, especially the many flavours of JPA that like to instrument the bytecode of classes at startup.
    • Django - may need to do some clever switching in models.py based on the capabilities of the schema. Not quite as bad as Hibernate, but not as nice as ActiveRecord.
    • NOSQL databases are great, but you still have to be careful in how you bind data to objects or data structures. Most importantly applications should not remove attributes or bits of data that they do not yet understand.
  • Database schema updates aren’t simple any more either.
    • New table columns should be nullable.
    • Use SQL default values for any new rows, populate existing rows with sensible defaults if possible. Note that new developers may have the wrong idea about a default value for a column.
    • You may need to use triggers to populate values that are important for one deployed version of the application.
    • Renaming columns and tables is possible, but you may need views to support older versions of the application.
    • Referential integrity is important, but should not be enforced until all application instances are at a version where that makes sense. Sometimes indexes will just have to do.
    • Don’t delete columns until either nothing uses them, or can handle their absence.
    • Data-munging and conversion is tricky, you may need views or duplicate tables and/or columns.