So as I type this, my office (4 people) are upgrading our version of Sakai. I thought this would be a good moment to go through what that actually means as we wait for the servers to do their thang.
2 days ago we did a smaller scale 'deploy' on our test servers to see how everything would go and to get an idea for timing. That meant we:
- Make a new backup snapshot of the prod-db incase we need to rollback changes for the main deploy.
- Put up a fresh new snapshot of the prod data on our test database.
- Run the supplied *.sql update scripts on that database.
- Deploy the new sakai code to the testing servers.
- Let hibernate run do its thing and sync up the rest of the database.
The above two steps can take an hour. Our database is 100-200 gigs and there are a lot of scripts.
- Once sakai-test1 & sakai-test2 are back up and the tomcat server is running, do a few sanity checks to make sure everything is working.
Once we do all of that and everything is working fine, we're ready.
So it's the day of, you have your coffee and your sys-admin (Thanks Joe!) has made sure everything is working. We had a bunch of firmware upgrades to make to our iSCSI and HP Blade System box so we ran through those. Our database system is MariaDB so we upgraded our installation to 5.5.39. The newst version of 10 went GA only a few months ago so we are holding off on that until it can go through a few more weeks of general usage until we upgrade that.
We brought down the production web/database servers and the loadbalancers are telling everything Sakai is down, what do we do now?
...same as the day before
Since we already did the upgrade/merge/deploy on the test servers and it was succcesfull, we do the same on production. There are a few differences because it is our production server but it is essentially the same thing. We run the included update scripts on the prod database and then deploy the new code on one of the production machines. This allows us to easily back out and get a working version up incase something goes wrong. Once the one prod server is up and running and Hibernate has upgraded everything to the current working copy I will then be putting my computer volume all the way up and turning on Richard Wagner's Flight of the Valkyries as I hit the enter button to deploy it to the rest of the production servers!