| jeblair | 80.3% | 01:53 |
|---|---|---|
| fungi | made it past 80% on phase 2 so far | 01:53 |
| fungi | heh | 01:53 |
| jeblair | 83.9% | 03:14 |
| fungi | looks like it finished while i slept... i'll get it remounted and the buffered logs rsync'd back in now | 10:52 |
| fungi | remounted and rsync underway. once this finishes i'll get the logs vhost back online, move the /etc/update-motd.d files back into place, reenable puppet and stand down the #status alert | 10:58 |
| fungi | ianw: is that your morse code in the screen session? | 11:04 |
| ianw | argh yes sorry | 11:04 |
| fungi | np | 11:04 |
| ianw | isn't it like stupid oclock for you? | 11:05 |
| fungi | nah. 7am local her | 11:05 |
| fungi | e | 11:05 |
| ianw | ahh, ok. i'm all logged out now, i won't mess anything else up :) | 11:06 |
| fungi | i left the fsck output alone and opened a new screen window for the rsync, but all fsck seemed to have to correct was the one file which tripped it up the first time | 11:07 |
| ianw | it's a lot of fscking for one fscking file | 11:08 |
| fungi | so true | 11:09 |
| mordred | fungi: that was the longest and most annoying incident I think we've ever had | 12:28 |
| fungi | well, technically it's not over _yet_ | 12:29 |
| fungi | rsync is still underway | 12:29 |
| fungi | though it's probably far enough along i can put the vhost back into service | 12:29 |
| fungi | doing that now | 12:29 |
| fungi | and done | 12:29 |
| fungi | looks like that worked fine | 12:30 |
| fungi | reenabling puppet now too, though i'll wait to lift the #status alert until the rsync completes | 12:31 |
| mordred | ++ | 12:35 |
| fungi | and done | 13:21 |
| *** ChanServ changes topic to "situation normal" | 13:26 | |
| -openstackstatus- NOTICE: The http://logs.openstack.org/ site is back in operation; previous logs as well as any uploaded during the outage should be available again; jobs which failed with POST_FAILURE can also be safely rechecked. | 13:26 | |
| fungi | status alert lifted, removing the old /srv/static/docs-draft/logs-tmp copy now and moving back to #openstack-infra | 13:26 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!