Unplanned outage :: www.ovirt.org :: 2012-12-06

[Karsten 'quaid' Wade]

We just had an unplanned outage of the www.ovirt.org site. The site was
unavailable for about three minutes.

== Details ==

I got a new theme.css file from Garrett to make the topbar a bit more
obvious to logged-in users. But I didn't want to push it and restart the
app, which is the default OpenShift behavior.

OpenShift has a hot deploy feature, set by putting a marker (touching a
file) in .openshift/markers/hot_deploy. When the marker is in place
during a 'git push', OpenShift is not supposed to restart the service.
Ideally, just having the marker there is enough so that the very push
itself doesn't restart anything.

Unfortunately, things didn't work as expected. When I pushed the change,
the output indicated that it stopped the service despite the hot_deploy
marker, and then didn't restart the service because of the hot_deploy
marker. I had to remove the marker and do another push to get it to
start the service.

git push
Counting objects: 22, done.
Delta compression using up to 4 threads.
Compressing objects: 100% (14/14), done.
Writing objects: 100% (14/14), 1.80 KiB, done.
Total 14 (delta 10), reused 0 (delta 0)
remote: restart_on_add=false
remote: Waiting for stop to finish
remote: Done
remote: restart_on_add=false
remote: ~/git/wiki.git ~/git/wiki.git
remote: ~/git/wiki.git
remote: Running .openshift/action_hooks/pre_build
remote: Running .openshift/action_hooks/build
remote: Running .openshift/action_hooks/deploy
remote: Database found, skipping build
remote: hot_deploy_added=false
remote: App will not be started due to presence of hot_deploy marker
remote: Running .openshift/action_hooks/post_deploy
To
ssh://847edb45aea84198838f915be6faa066@wiki-ovirt.rhcloud.com/~/git/wiki.git/
   85d7d77..d9a1670  master -> master

== Affected services ===

* www.ovirt.org

=== Not-affected services ===

* lists.ovirt.org
* resources.ovirt.org

== Future plans ==

Make the hot deploy work during a planned outage window rather than
during the middle of the work day. When hot deploy is working, we should
have fewer occasions of restarting the website just because we want to
push a CSS, config, etc. change.

-- 
Karsten 'quaid' Wade, Sr. Analyst - Community Growth
http://TheOpenSourceWay.org  .^\  http://community.redhat.com
@quaid (identi.ca/twitter/IRC)  \v'  gpg: AD0E0C41

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 253 bytes
Desc: OpenPGP digital signature
URL: <http://lists.ovirt.org/pipermail/infra/attachments/20121206/5d837128/attachment.sig>