| *** bhunter71 has quit IRC | 00:50 | |
| *** bhunter71 has joined #openstack-infra-incident | 02:26 | |
| *** bhunter71 has quit IRC | 03:43 | |
| *** ig0r__ has joined #openstack-infra-incident | 05:52 | |
| *** ig0r_ has quit IRC | 05:55 | |
| *** ig0r__ has quit IRC | 12:17 | |
| *** ig0r_ has joined #openstack-infra-incident | 12:23 | |
| *** bhunter71 has joined #openstack-infra-incident | 13:32 | |
| *** bhunter71 has quit IRC | 13:47 | |
| *** bhunter71 has joined #openstack-infra-incident | 15:04 | |
| *** ig0r__ has joined #openstack-infra-incident | 15:19 | |
| *** ig0r_ has quit IRC | 15:22 | |
| *** ig0r__ has quit IRC | 19:19 | |
| *** ig0r_ has joined #openstack-infra-incident | 19:19 | |
| -openstackstatus- NOTICE: Gerrit will be unavailable from 22:00 to 22:30 UTC for project renames | 21:02 | |
| *** ig0r_ has quit IRC | 21:10 | |
| *** ChanServ changes topic to "Gerrit rename starting at 22:00" | 21:49 | |
| * jeblair dusts off the furniture | 21:49 | |
| pleia2 | o/ | 21:49 |
|---|---|---|
| * Clint sneezes. | 21:49 | |
| pleia2 | https://etherpad.openstack.org/p/repo-renaming-2015-07-10 | 21:49 |
| *** anteaya has joined #openstack-infra-incident | 21:50 | |
| jeblair | pleia2: have you disabled the cron entry on puppetmaster? | 21:52 |
| pleia2 | I followed fungi's advice and did step 2 a few minutes ago, so it's all stopped | 21:52 |
| pleia2 | jeblair: yep | 21:52 |
| jeblair | pleia2: cool, and it wasn't/isn't running (since it can re-enable itself)? | 21:52 |
| pleia2 | jeblair: it was still running, but I confirmed it didn't re-add itself to the crontab once it finished | 21:53 |
| jeblair | and it's done now? | 21:53 |
| pleia2 | it's done | 21:53 |
| jeblair | woot | 21:53 |
| jeblair | pleia2: are you on statusbot's list of friends? | 21:53 |
| pleia2 | I am! | 21:53 |
| anteaya | yay | 21:53 |
| pleia2 | we should add statusbot as a step, because syntax woo | 21:54 |
| jeblair | cool, so that step is yours too :) | 21:54 |
| jeblair | yep | 21:54 |
| pleia2 | thanks | 21:55 |
| jeblair | pleia2: last run took 3 minutes to complete, and it didn't have topic changes, so i think now is actually a good time to issue the command | 21:55 |
| pleia2 | ok | 21:56 |
| jeblair | pleia2: also, you can issue it in any channel, including this one :) | 21:56 |
| pleia2 | ah, good to know! | 21:56 |
| jeblair | pleia2: so at 2200 i'll stop zuul, and let you know when that's done, then you can proceed with stopping gerrit | 21:57 |
| pleia2 | great | 21:58 |
| -openstackstatus- NOTICE: Gerrit is unavailable from approximately 22:00 to 22:30 UTC for project renames | 21:58 | |
| *** ChanServ changes topic to "Gerrit is unavailable from approximately 22:00 to 22:30 UTC for project renames" | 21:58 | |
| jeblair | stopping zuul | 22:00 |
| jeblair | done | 22:00 |
| jeblair | pleia2: all yours | 22:00 |
| pleia2 | ok, stopping gerrit | 22:00 |
| pleia2 | done | 22:01 |
| jeblair | pleia2: want me to handle git* | 22:01 |
| jeblair | ? | 22:01 |
| pleia2 | jeblair: sure, I'll start on the mysql things on review.o.o | 22:01 |
| jeblair | pleia2: i think i see a typo | 22:03 |
| pleia2 | uh oh | 22:04 |
| jeblair | pleia2: i think it's only in my section | 22:04 |
| pleia2 | ok | 22:04 |
| jeblair | sudo mv /var/lib/git/{stackforge/networking-plumgrid,openstack/networking-plumgird}.git | 22:04 |
| jeblair | 'gird' at the end | 22:04 |
| jeblair | but it looks right in the other sections | 22:04 |
| pleia2 | plumgrid | 22:04 |
| pleia2 | I fixed | 22:04 |
| pleia2 | that'll teach me for manually adding an i | 22:05 |
| anteaya | control f is red for gird for me on the etherpad | 22:05 |
| anteaya | red meaning no occurances | 22:05 |
| pleia2 | moving on to step 6 to rename git things on review.o.o | 22:05 |
| jeblair | anteaya: ++ | 22:05 |
| anteaya | jeblair: can I mark step 3 as done? | 22:06 |
| jeblair | anteaya: sorry, i did | 22:06 |
| anteaya | ah okay thanks | 22:06 |
| pleia2 | I can do the lucerne bit if we're good | 22:07 |
| pleia2 | lucene too | 22:07 |
| pleia2 | lucerne makes milk :d | 22:07 |
| jeblair | pleia2: lucene is yours | 22:07 |
| pleia2 | k, on it | 22:07 |
| jeblair | pleia2: after you're done with review git... | 22:07 |
| jeblair | are you? | 22:08 |
| jeblair | pleia2: (step 5 and 6 are pre-reqs for 7) | 22:08 |
| pleia2 | yep | 22:08 |
| pleia2 | jeblair: want to do github things? | 22:08 |
| jeblair | pleia2: can do | 22:09 |
| jeblair | pleia2: while i'm here, what's your github id? | 22:09 |
| pleia2 | pleia2 | 22:09 |
| jeblair | i'll set you up so you can do that part next time :) | 22:09 |
| pleia2 | thanks | 22:09 |
| anteaya | ha ha ha | 22:09 |
| pleia2 | 33% of reindexing done | 22:11 |
| anteaya | woooo | 22:12 |
| jeblair | i'm taking step 16 too | 22:17 |
| pleia2 | thanks | 22:17 |
| pleia2 | 90% | 22:17 |
| pleia2 | got some errors not related to what we did today, will pastebin when this is done | 22:18 |
| jeblair | k | 22:19 |
| pleia2 | http://paste.openstack.org/show/363135/ | 22:19 |
| jeblair | oh interesting | 22:20 |
| jeblair | should be harmless i think | 22:21 |
| pleia2 | ok | 22:21 |
| pleia2 | maybe something to look at during our next window | 22:21 |
| pleia2 | or in preparation for it | 22:21 |
| jeblair | ya | 22:21 |
| pleia2 | shall I start gerrit? | 22:21 |
| jeblair | pleia2: go for it | 22:22 |
| * pleia2 does this as her own user this time | 22:22 | |
| pleia2 | should be up | 22:22 |
| jeblair | starting zuul | 22:22 |
| anteaya | my -2 is removed | 22:23 |
| anteaya | all clear on the patches | 22:23 |
| pleia2 | thanks | 22:23 |
| jeblair | pleia2: do you want to aprv the changes or shall i? | 22:23 |
| pleia2 | jeblair: you can go for it | 22:24 |
| jeblair | woo, gertty mass review :) | 22:24 |
| pleia2 | :) | 22:24 |
| anteaya | ha ha ha | 22:24 |
| jeblair | done | 22:24 |
| pleia2 | so do we wait on starting puppet again until they land? | 22:25 |
| jeblair | those are in the queue now, so i'll go ahead and restore zuul's queue | 22:25 |
| pleia2 | k | 22:25 |
| jeblair | pleia2: yep, they need to land before we can restart puppet | 22:25 |
| jeblair | drat https://jenkins06.openstack.org/job/gate-project-config-irc-access/103/ | 22:27 |
| jeblair | maybe we should avoid irc changes on renames | 22:27 |
| pleia2 | :\ | 22:27 |
| anteaya | that is the same random error: UnicodeDecodeError: 'utf8' codec can't decode byte 0xe5 in position 82: invalid continuation byte | 22:28 |
| anteaya | no idea what it means | 22:28 |
| jeblair | pleia2: you can 'status ok' now, since the outage is over for everyone except the renamed projects | 22:28 |
| anteaya | and it is always position 82 | 22:28 |
| pleia2 | ok | 22:29 |
| jeblair | anteaya: i suspect it's the name of science fiction author | 22:29 |
| pleia2 | #status ok | 22:29 |
| openstackstatus | pleia2: sending ok | 22:29 |
| anteaya | I'm not following | 22:29 |
| * anteaya suspects her knowledge of authors is less than expected here | 22:30 | |
| jeblair | pleia2: you can also put a message after ok that will appear in the notice | 22:30 |
| pleia2 | noted | 22:30 |
| jeblair | anteaya: i don't know which, just one with a non-roman character | 22:30 |
| *** ChanServ changes topic to "Gerrit rename starting at 22:00" | 22:30 | |
| anteaya | hmmmm | 22:30 |
| pleia2 | heh, still have the #notice out there | 22:31 |
| anteaya | perhaps it got into the channels file and only gets found sometimes? sometimes that job passes | 22:31 |
| openstackstatus | pleia2: finished sending ok | 22:31 |
| jeblair | pleia2: just on this channel -- i changed the topic before you sent the command | 22:32 |
| pleia2 | ah :) | 22:32 |
| pleia2 | I see, good good | 22:32 |
| anteaya | grepping gerritbot/channels.yaml doesn't turn up any non-ascii characters | 22:34 |
| anteaya | grep for non-ascii characters: grep -R --color='auto' -P -n "[\x80-\xFF]" ./* | 22:34 |
| anteaya | or grep --color='auto' -P -n "[\x80-\xFF]" filename.fileextension | 22:34 |
| jeblair | anteaya: i think it's coming from the irc server | 22:35 |
| anteaya | oh | 22:35 |
| anteaya | now I understand the authors comment | 22:36 |
| jeblair | oh, the change is to gerribot/channels.yaml, so i guess we have to have it in the rename; i thought this was the change adding a new channel or something | 22:37 |
| anteaya | no | 22:37 |
| jeblair | i don't think zuul is in a good state :/ | 22:38 |
| anteaya | it doesn't look happy | 22:38 |
| jeblair | 2015-07-10 22:38:31,829 INFO zuul.Gearman: Cancel build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> for job gate-project-config-pep8 | 22:38 |
| jeblair | 2015-07-10 22:38:31,829 DEBUG zuul.Gearman: Build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> has not started yet | 22:38 |
| jeblair | 2015-07-10 22:38:31,829 DEBUG zuul.Gearman: Looking for build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> in queue | 22:38 |
| jeblair | 2015-07-10 22:38:31,830 DEBUG zuul.Gearman: Response to cancel build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> request: ERR UNKNOWN_JOB | 22:38 |
| jeblair | 2015-07-10 22:38:32,831 DEBUG zuul.Gearman: Still unable to find build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> to cancel | 22:38 |
| jeblair | 2015-07-10 22:38:32,832 DEBUG zuul.Gearman: Unable to cancel build <Build 5b1d7211d4e74a9491a26b0c51be978b of gate-project-config-pep8 on <Worker Unknown>> | 22:38 |
| pleia2 | :\ | 22:40 |
| jeblair | restarting zuul | 22:40 |
| *** bhunter71 has quit IRC | 22:40 | |
| jeblair | reapproving changes | 22:42 |
| *** nibalizer has joined #openstack-infra-incident | 22:42 | |
| jeblair | and re-enqueing | 22:42 |
| pleia2 | thanks | 22:42 |
| * anteaya crosses her fingers | 22:42 | |
| jeblair | ugh, lots of not registered; we should have had nodes | 22:43 |
| anteaya | maybe we will get an irc server with an author's name spelled the way the test can handle | 22:43 |
| anteaya | :( | 22:43 |
| anteaya | I wish I had something useful to say | 22:43 |
| jeblair | i've stopped zuul again and am deleting all nodepool nodes | 22:43 |
| anteaya | mordred: have you anything useful and supportive to offer? | 22:44 |
| pleia2 | let me know if I can jump in anywhere | 22:44 |
| jeblair | mordred: /usr/local/lib/python2.7/dist-packages/os_client_config/config.py:226: UserWarning: /home/nodepool/.config/openstack/clouds.yaml use the keyword 'cloud' to reference a known vendor profile. This has been deprecated in favor of the 'profile' keyword. | 22:46 |
| jeblair | pleia2: will do -- just waiting for nodepool nodes to be ready, then will restart zuul again and re-enqueue | 22:47 |
| * pleia2 nods | 22:47 | |
| nibalizer | i also about if I can be helpful | 22:47 |
| jeblair | pleia2: i suspect nodepool's accounting was far enough off that it didn't think it needed to create nodes of some types, yet we had zero of them, so the jobs weren't registered | 22:48 |
| jeblair | k, bare and devstack-trusty nodes are online | 22:48 |
| *** clarkb has joined #openstack-infra-incident | 22:48 | |
| pleia2 | aha | 22:49 |
| anteaya | clarkb: hi | 22:49 |
| jeblair | bare-centos6 ready | 22:49 |
| clarkb | hello | 22:49 |
| anteaya | clarkb: the backscroll: http://eavesdrop.openstack.org/irclogs/%23openstack-infra-incident/%23openstack-infra-incident.2015-07-10.log | 22:49 |
| jeblair | bare-precise ready | 22:50 |
| jeblair | just waiting on a few more now | 22:52 |
| jeblair | close enough i think; the remaining are f21 | 22:53 |
| jeblair | starting zuul | 22:54 |
| jeblair | re-enqueing | 22:54 |
| anteaya | there are 3 patches that are running the irc access job | 22:55 |
| jeblair | if they don't pass this time, we will just force-merge them | 22:56 |
| anteaya | sounds good to me | 22:56 |
| jeblair | queues are reloaded | 22:58 |
| * anteaya stares at the status page | 23:02 | |
| pleia2 | chugging away :) | 23:02 |
| mordred | jeblair: that warning is fine | 23:02 |
| mordred | jeblair: I have a patch up to change to teh new words - but it's still the same functional | 23:02 |
| jeblair | mordred: thanks | 23:03 |
| jeblair | all the irc jobs passed | 23:05 |
| Clint | the problematic server is leguin.freenode.net | 23:05 |
| pleia2 | thanks freenode | 23:05 |
| jeblair | Clint: she doesn't have a non-roman char does she? | 23:05 |
| Clint | jeblair: it's spewing Latin-1 instead of UTF-8 | 23:06 |
| mordred | maybe it's £eguin.freenode.net | 23:06 |
| Clint | specifically in "Umeå" twice | 23:06 |
| jeblair | at least she didn't when she lived in berkeley.. maybe she picked one up in portland? | 23:06 |
| mordred | jeblair: portland will utf-8 you | 23:06 |
| jeblair | Clint: feel like letting the freenode ops know? | 23:08 |
| Clint | sure | 23:08 |
| anteaya | Clint: thanks | 23:08 |
| pleia2 | waiting on a precise node for the last one | 23:08 |
| pleia2 | there it goes | 23:09 |
| mordred | \o/ | 23:11 |
| pleia2 | jeblair: so once this merges, I can re-enable the puppet cron job? | 23:12 |
| jeblair | pleia2: yep | 23:12 |
| pleia2 | (cutting it close, but timing should work out nicely for our */15 job) | 23:12 |
| jeblair | pleia2: and once it finishes its run, the new jjb/zuul configs should be in place and then the outage is over for the renamed projects too | 23:13 |
| pleia2 | cool, so then I can do the .gitreview things | 23:13 |
| jeblair | yep | 23:13 |
| pleia2 | ok, merged | 23:13 |
| jeblair | yay! | 23:13 |
| pleia2 | cron job is re-enabled | 23:13 |
| pleia2 | and 23:15 run_all.sh is on its way | 23:15 |
| * mordred hands pleia2 an alpaca of cron-running | 23:16 | |
| pleia2 | thank you, sir! | 23:16 |
| pleia2 | fluffy alpacas | 23:16 |
| * mordred just realized he should start handing out new zealand possums | 23:16 | |
| pleia2 | I'm going to peru next month, alpacas for everyone | 23:16 |
| mordred | woot! | 23:17 |
| mordred | pleia2: how long are you there for and do you know all the things you're doing? | 23:17 |
| * mordred has a good friend who is a jazz trumpeter from peru who is going to move in to the spare room with his wife and baby next week ... | 23:17 | |
| mordred | s/the spare room/mordred's spare room/ | 23:18 |
| mordred | hrm. I should move chat to the other channel | 23:18 |
| pleia2 | :) | 23:18 |
| jeblair | i have to run, sorry, i'll be back later if anything else comes up | 23:21 |
| pleia2 | thanks jeblair | 23:21 |
| anteaya | thanks jeblair | 23:24 |
| anteaya | pleia2: what is left? | 23:24 |
| pleia2 | anteaya: puppet things chugging along, once finished the new projects will show up in their new space on git.o.o and I can update their .gitreview files | 23:25 |
| anteaya | k | 23:25 |
| anteaya | go you | 23:25 |
| anteaya | pleia2: I'm going to get another walk in, I'll be about an hour, will check when I return | 23:43 |
| pleia2 | everything is all prepped, I'll just wait until the end of the puppet run to be sure | 23:44 |
| pleia2 | anteaya: enjoy :) | 23:44 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!