0

I am currently maintaining a Review Board server (https://www.reviewboard.org/) and one of the big pain points is scheduling upgrades.

Currently the process is:

  1. Take the server down and send all users an email notifying them that all Review Board services will be down.
  2. Run the upgrade script provided by the Review Board maintainers which could take hours if there are many database evolutions as there are many users.
  3. Bring the server back up after significant down time (sometimes 5+ hours).

Are there better known solutions when upgrading 3rd party software than that above to minimize downtime?

My theory is that you can:

  1. Create a separate copy of the current installation of Review Board along with a dumped database from some time (X).
  2. Perform the upgrade on the duplicated installation.
  3. Take the server down (for a much shorter time than in the current process) and email users. Check the current time (Y) and see determine all additional changes to the database since time (X).
  4. Apply the database evolutions to just the diff of changes between time (X) and time (Y). Then, insert the missing database rows.

The point I am struggling with is implementing steps 3 and 4 as these are not provided by the Review Board maintainers. Are there already prebuilt tools to help with these issues?

Additional Details:

I am using a MySQL database to back Review Board written in Python.

2
  • 1
    This question appears to be off-topic because it is about application maintenance.
    – user118874
    Nov 19, 2014 at 16:50
  • basically, as the current answer suggests, what you're looking for is load balancing. If you balance the request load on multiple servers, then your app doesn't shut down when you take down one server for maintenance.
    – deltree
    Nov 19, 2014 at 17:45

1 Answer 1

-2

In general, multiple servers are used to ensure that downtime is close to zero. For instance, the poor man's two servers solution is:

  1. You start switching all users from server A to server B,

  2. Once all users are on server B, you make it impossible to access the server A from the outside,

  3. The server A is updated,

  4. Before bringing it back online, the server A is checked to ensure that it is working as expected; you don't want to discover that the update failed (or introduced regressions) from your users calling you to know why can't they access your service any longer.

  5. The server A becomes accessible from the outside,

  6. The users are switched from server B to server A,

  7. You repeat the same actions to update the server B.

At any moment do you send e-mails to the users, because the goal is to keep the service up and running during the whole upgrade process. A good example is Stack Exchange: they push updates to production very frequently (if I remember well from the podcasts, dozen of pushes every day), while remaining online. The only cases where they redirect the users to PEAK datacenter in Oregon is when they have to make changes in their infrastructure.

As for the applicability of this process to a specific product you are using, consider asking for support to the company developing the concerned product.

You must log in to answer this question.