| ianw | pabelanger: all that job split up ready for review? i'll look in on that too | 00:02 |
|---|---|---|
| pabelanger | ianw: for base-minimal? yah, ready for some eyes | 00:02 |
| *** hamzy has joined #openstack-infra | 00:09 | |
| *** tonyb has quit IRC | 00:16 | |
| *** dingyichen has joined #openstack-infra | 00:17 | |
| ianw | yay, all working | 00:24 |
| ianw | i wonder how many people noticed that rtd publishing was failing, and then noticed they could hook it up to github, and now do a dance gerrit->github->rtd ping. it bet it's !0 | 00:25 |
| *** gyee has quit IRC | 00:36 | |
| *** edmondsw has joined #openstack-infra | 00:37 | |
| *** edmondsw has quit IRC | 00:42 | |
| *** jcoufal has joined #openstack-infra | 00:49 | |
| *** mriedem_afk has quit IRC | 01:34 | |
| *** hongbin has joined #openstack-infra | 01:44 | |
| *** ramishra has joined #openstack-infra | 02:00 | |
| *** yamamoto has joined #openstack-infra | 02:01 | |
| *** jcoufal has quit IRC | 02:05 | |
| *** ramishra has quit IRC | 02:08 | |
| *** yamamoto has quit IRC | 02:12 | |
| *** yamamoto has joined #openstack-infra | 02:18 | |
| tristanC | corvus: clarkb: isn't the logic of os-loganalyze (e.g. linkable timestamp) going to be implemented in the zuul dashboard? | 02:21 |
| *** tonyb has joined #openstack-infra | 02:22 | |
| *** yamamoto has quit IRC | 02:23 | |
| *** edmondsw has joined #openstack-infra | 02:25 | |
| tristanC | oh i see, the first step seems to be doing static HTMLification | 02:29 |
| *** edmondsw has quit IRC | 02:30 | |
| tristanC | then either we add the reporting code to the htmlify role, either wait for the zuul dashboard enhancement | 02:30 |
| *** psachin has joined #openstack-infra | 02:34 | |
| *** dave-mccowan has quit IRC | 02:35 | |
| *** yamamoto has joined #openstack-infra | 03:03 | |
| jhesketh | tristanC: small query in 550978 if you have time :-) | 03:07 |
| *** yamamoto has quit IRC | 03:11 | |
| *** rlandy|bbl is now known as rlandy | 03:14 | |
| *** udesale has joined #openstack-infra | 03:31 | |
| *** yamamoto has joined #openstack-infra | 03:37 | |
| *** yamamoto has quit IRC | 03:41 | |
| *** yamamoto has joined #openstack-infra | 03:43 | |
| *** hongbin has quit IRC | 03:52 | |
| *** hwoarang has quit IRC | 03:52 | |
| *** yamamoto has quit IRC | 04:02 | |
| *** ramishra has joined #openstack-infra | 04:03 | |
| *** yamamoto has joined #openstack-infra | 04:03 | |
| *** mschuppert has joined #openstack-infra | 04:06 | |
| *** yamamoto has quit IRC | 04:14 | |
| *** yamamoto has joined #openstack-infra | 04:28 | |
| *** rlandy has quit IRC | 04:28 | |
| *** udesale has quit IRC | 04:35 | |
| *** viks_ has joined #openstack-infra | 04:37 | |
| *** yamamoto has quit IRC | 04:38 | |
| *** dklyle has quit IRC | 04:39 | |
| *** viks_ has quit IRC | 04:42 | |
| *** yamamoto has joined #openstack-infra | 04:43 | |
| *** yamamoto has quit IRC | 04:47 | |
| *** udesale has joined #openstack-infra | 04:55 | |
| *** yamamoto has joined #openstack-infra | 04:56 | |
| *** yamamoto has quit IRC | 04:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/job/{job_name} route https://review.openstack.org/550978 | 05:36 |
| *** quiquell has joined #openstack-infra | 05:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/projects and /{tenant}/project/{project} routes https://review.openstack.org/550979 | 05:40 |
| *** jaosorior has quit IRC | 05:40 | |
| *** jaosorior has joined #openstack-infra | 05:41 | |
| *** janki has joined #openstack-infra | 05:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines route https://review.openstack.org/541521 | 05:44 |
| *** jrist has quit IRC | 05:49 | |
| *** xarses has joined #openstack-infra | 05:55 | |
| *** apetrich has joined #openstack-infra | 05:57 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: scheduler: add job's parent name to the rpc job_list method https://review.openstack.org/573473 | 06:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/labels route https://review.openstack.org/553979 | 06:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/nodes route https://review.openstack.org/553998 | 06:01 |
| *** yamamoto has joined #openstack-infra | 06:03 | |
| *** jrist has joined #openstack-infra | 06:03 | |
| *** yamamoto_ has joined #openstack-infra | 06:07 | |
| *** yamamoto has quit IRC | 06:09 | |
| *** yamamoto_ has quit IRC | 06:10 | |
| *** jesusaur has quit IRC | 06:40 | |
| *** chason has joined #openstack-infra | 06:43 | |
| *** jesusaur has joined #openstack-infra | 06:45 | |
| *** ccamacho has joined #openstack-infra | 06:45 | |
| *** rcernin has quit IRC | 06:54 | |
| *** chason has quit IRC | 07:01 | |
| *** ginopc has joined #openstack-infra | 07:01 | |
| *** chason has joined #openstack-infra | 07:02 | |
| *** annp has quit IRC | 07:02 | |
| *** chason has quit IRC | 07:08 | |
| *** amoralej|off is now known as amoralej | 07:09 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/labels route https://review.openstack.org/553979 | 07:12 |
| *** tosky has joined #openstack-infra | 07:26 | |
| openstackgerrit | Slawek Kaplonski proposed openstack-infra/project-config master: Move ironic-tempest job for Neutron to "in tree" https://review.openstack.org/588181 | 07:28 |
| *** jtomasek_ has quit IRC | 07:29 | |
| *** jtomasek has joined #openstack-infra | 07:31 | |
| *** chason has joined #openstack-infra | 07:44 | |
| *** jpich has joined #openstack-infra | 07:53 | |
| *** Bhujay has joined #openstack-infra | 07:56 | |
| quiquell | Do you know if we can 'curl' review.o.o to get the CI results of a change ? | 07:58 |
| *** dtantsur|afk is now known as dtantsur | 08:01 | |
| *** tommylikehu is now known as tommylikehu2 | 08:02 | |
| *** tommylikehu2 is now known as tommylikehu | 08:03 | |
| *** derekh has joined #openstack-infra | 08:03 | |
| *** tommylikehu is now known as tommylikehu_afk | 08:04 | |
| *** Bhujay has quit IRC | 08:05 | |
| *** tommylikehu_afk is now known as tommylikehu | 08:09 | |
| *** jpena|off is now known as jpena | 08:20 | |
| *** vivsoni_ has quit IRC | 08:29 | |
| *** jiapei has joined #openstack-infra | 08:33 | |
| *** ginux has joined #openstack-infra | 08:34 | |
| *** ginux is now known as Guest10945 | 08:34 | |
| *** xarses has quit IRC | 08:37 | |
| *** ginopc has quit IRC | 08:38 | |
| *** electrofelix has joined #openstack-infra | 08:49 | |
| *** jaosorior has quit IRC | 09:03 | |
| *** chason has quit IRC | 09:10 | |
| *** gfidente has joined #openstack-infra | 09:13 | |
| *** pbourke has joined #openstack-infra | 09:26 | |
| *** chason has joined #openstack-infra | 09:28 | |
| *** vivsoni has joined #openstack-infra | 09:31 | |
| *** zoli is now known as zoli|lunch | 09:32 | |
| *** chason has quit IRC | 10:06 | |
| *** chason has joined #openstack-infra | 10:07 | |
| *** agopi has quit IRC | 10:08 | |
| *** chason has quit IRC | 10:12 | |
| *** bradm has joined #openstack-infra | 10:37 | |
| *** zoli|lunch is now known as zoli | 10:50 | |
| *** e0ne has joined #openstack-infra | 11:01 | |
| *** e0ne has quit IRC | 11:01 | |
| *** e0ne has joined #openstack-infra | 11:02 | |
| *** e0ne has quit IRC | 11:02 | |
| *** jpena is now known as jpena|lunch | 11:03 | |
| *** e0ne has joined #openstack-infra | 11:17 | |
| *** e0ne has quit IRC | 11:17 | |
| *** vivsoni has quit IRC | 11:17 | |
| *** rh-jelabarre has joined #openstack-infra | 11:26 | |
| *** auristor has quit IRC | 11:36 | |
| *** dave-mccowan has joined #openstack-infra | 11:43 | |
| *** boden has joined #openstack-infra | 11:43 | |
| *** jiapei has quit IRC | 11:47 | |
| *** tpsilva has joined #openstack-infra | 11:48 | |
| *** jpena|lunch is now known as jpena | 11:57 | |
| *** hemna_ has quit IRC | 12:04 | |
| *** rosmaita has joined #openstack-infra | 12:12 | |
| *** auristor has joined #openstack-infra | 12:14 | |
| *** hemna_ has joined #openstack-infra | 12:15 | |
| *** panda|rover is now known as panda|rover|off | 12:17 | |
| *** sthussey has joined #openstack-infra | 12:20 | |
| *** kgiusti has joined #openstack-infra | 12:21 | |
| evrardjp | can someone explain me what is the purpose of project-config's bindep-fallback file ? | 12:22 |
| evrardjp | AJaeger: you are the last one to touch this, maybe you know? ^ | 12:23 |
| pabelanger | evrardjp: it's a legacy file for when we used to have devstack based images. The goal is to eventually delete it, and is under freeze for the most part | 12:25 |
| evrardjp | pabelanger: well the thing is that it's used for bindep testing | 12:25 |
| evrardjp | and it's outdated | 12:26 |
| pabelanger | evrardjp: which job? | 12:26 |
| pabelanger | when we bring on newer versions of fedora, we just skipping adding a test for it | 12:26 |
| evrardjp | ok | 12:27 |
| pabelanger | which means, newer OSes are broken for bindep-fallback and is a way that projects should add a bindep.txt file to their repo | 12:27 |
| evrardjp | that sounds a bad idea | 12:27 |
| evrardjp | pabelanger: oh i see | 12:27 |
| evrardjp | so the idea is to REALLY push the bindep into projects' repo? | 12:27 |
| pabelanger | yup | 12:27 |
| pabelanger | last I checked a lot of projects already have bindep.txt file | 12:28 |
| evrardjp | proof is here that it doesn't work for SUSE tumbleweed and leap 15: https://review.openstack.org/#/c/588209/ | 12:28 |
| evrardjp | pabelanger: yeah. | 12:28 |
| evrardjp | so I will work on those projects across the board then | 12:28 |
| evrardjp | I will abandon my idea of changing the fallback | 12:28 |
| evrardjp | I should have asked earlier : p | 12:29 |
| pabelanger | yah, ideally all those jobs will go away, once we remove the file | 12:29 |
| evrardjp | pabelanger: if you could help on those reviews, I'd be happy: https://review.openstack.org/#/q/topic:loci-suseleap15+status:open+project:openstack-infra/bindep | 12:30 |
| evrardjp | ianw: as you were already a reviewer in some of those, I'd be happy to see a review too ^ | 12:31 |
| evrardjp | if you need anything reviewed that I can help, shoot :) | 12:31 |
| *** mriedem has joined #openstack-infra | 12:39 | |
| *** rlandy has joined #openstack-infra | 12:42 | |
| *** efried is now known as fried_rice | 12:48 | |
| *** rfolco|off is now known as rfolco|ruck | 12:52 | |
| *** nicolasbock has joined #openstack-infra | 13:00 | |
| *** edmondsw_ has joined #openstack-infra | 13:00 | |
| *** eharney has joined #openstack-infra | 13:07 | |
| *** amoralej is now known as amoralej|lunch | 13:07 | |
| *** agopi has joined #openstack-infra | 13:10 | |
| *** agopi_ has joined #openstack-infra | 13:11 | |
| *** quiquell is now known as quiquell|off | 13:13 | |
| *** agopi has quit IRC | 13:14 | |
| *** agopi_ is now known as agopi | 13:14 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Replace with_first_found with lookup first_found https://review.openstack.org/588546 | 13:16 |
| *** quiquell|off has quit IRC | 13:17 | |
| *** ramishra has quit IRC | 13:22 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove old inactive users https://review.openstack.org/588553 | 13:22 |
| *** jcoufal has joined #openstack-infra | 13:24 | |
| *** dave-mccowan has quit IRC | 13:25 | |
| *** mriedem is now known as mriedem_afk | 13:25 | |
| *** agopi_ has joined #openstack-infra | 13:25 | |
| *** lbragstad has quit IRC | 13:26 | |
| *** agopi has quit IRC | 13:28 | |
| *** stephenfin is now known as finucannot | 13:28 | |
| *** agopi_ is now known as agopi | 13:31 | |
| *** psachin has quit IRC | 13:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add sudoers file and groups https://review.openstack.org/587854 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add bridge.openstack.org to trusted ssh list https://review.openstack.org/587855 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add emacs and vim to base-server packages https://review.openstack.org/587983 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add pip and virtualenv to bridge.openstack.org https://review.openstack.org/587984 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install and configure ansible on bridge https://review.openstack.org/587985 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add an Ansible role to configure exim https://review.openstack.org/588089 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update install_modules to not need puppet https://review.openstack.org/588394 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Rename update_puppet to update-system-config https://review.openstack.org/588396 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Fix some little ansible issues https://review.openstack.org/588397 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Replace with_first_found with lookup first_found https://review.openstack.org/588546 | 13:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove old inactive users https://review.openstack.org/588553 | 13:51 |
| *** bnemec is now known as beekneemech | 13:51 | |
| *** Emine has joined #openstack-infra | 13:53 | |
| fungi | pabelanger: i didn't realize we were skipping testing the fallback file for some platforms. that job was serving as functional testing of bindep on those platforms, so without it we don't necessarily have any guarantees it'll run there do we? or was a different functional test job added to supplant it? | 13:55 |
| fungi | i expected we'd eventually move that file into the bindep repo as a test fixture once we no longer needed it for actual fallback | 13:57 |
| pabelanger | fungi: yah, moving a file in repo should be good if we want to keep testing on specific distro. | 13:57 |
| fungi | well, presumably we want to test against all the distros that we have | 13:57 |
| pabelanger | ideally, ya. Some have slipped though | 13:58 |
| * mordred looks forward to not needing the fallback file | 14:01 | |
| pabelanger | ++ | 14:01 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Install and configure ansible on bridge https://review.openstack.org/587985 | 14:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add an Ansible role to configure exim https://review.openstack.org/588089 | 14:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Update install_modules to not need puppet https://review.openstack.org/588394 | 14:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Rename update_puppet to update-system-config https://review.openstack.org/588396 | 14:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Fix some little ansible issues https://review.openstack.org/588397 | 14:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Replace with_first_found with lookup first_found https://review.openstack.org/588546 | 14:07 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Remove old inactive users https://review.openstack.org/588553 | 14:07 |
| *** alex_xu has quit IRC | 14:08 | |
| mordred | pabelanger: updated https://review.openstack.org/#/c/587854 to add a comment based on your review | 14:08 |
| *** edmondsw_ is now known as edmondsw | 14:08 | |
| mordred | clarkb, corvus, fungi, pabelanger: ^^ I updated the stack based on review comments and have re-tested on bridge.o.o | 14:09 |
| *** amoralej|lunch is now known as amoralej | 14:10 | |
| *** rpittau has quit IRC | 14:18 | |
| clarkb | fungi with bindeps switch to distro the in repo test suite has fixtures for all supported distro os-release files. Not a direct on going test of functionality on that distro but pretty close | 14:23 |
| *** janki has quit IRC | 14:28 | |
| fungi | yeah, i'd still feel more comfortable if we tested it on all the platforms where we might run it in other jobs, but its unit testing is fairly comprehensive | 14:33 |
| *** hongbin has joined #openstack-infra | 14:33 | |
| *** jcoufal_ has joined #openstack-infra | 14:36 | |
| *** jcoufal has quit IRC | 14:37 | |
| *** Emine has quit IRC | 14:39 | |
| *** nhicher has joined #openstack-infra | 14:40 | |
| *** Emine has joined #openstack-infra | 14:40 | |
| *** mriedem_afk is now known as mriedem | 14:43 | |
| clarkb | In this case it looks like the problem is with the actual contents of the file and not bindep itself | 14:47 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: WIP Use ansible for openstack_project::server https://review.openstack.org/585836 | 14:48 |
| mordred | clarkb: I have responded to and/or fixed all of your review comments from yesterday | 14:49 |
| clarkb | mordred: thanks, I'll go back through that list when project renames are done | 14:50 |
| clarkb | I have disabled puppet cron on the puppetmaster | 14:50 |
| mordred | clarkb: sweet | 14:50 |
| clarkb | mordred: when doing ^ I noticed that hte cloud launcher ansible might be something we want ot move to bridge early since it is all ansible | 14:50 |
| mordred | clarkb: agree | 14:50 |
| mordred | clarkb: also - we still have infra cloud things in place | 14:51 |
| *** janki has joined #openstack-infra | 14:51 | |
| clarkb | mordred: ya noticed that too | 14:51 |
| clarkb | mordred: I think it may noop because we removed nodes from inventory | 14:52 |
| clarkb | hrm notice that the -1 on the chef change isn't due to zuul config update being unhappy its because we have linting things on projects.yaml | 14:53 |
| clarkb | scas: fungi ^ fyi I'm going t oaddress the two items that were caught now | 14:54 |
| clarkb | http://logs.openstack.org/71/585471/3/check/project-config-gerrit/d3a56e4/job-output.txt.gz#_2018-07-24_20_49_55_568618 | 14:54 |
| *** rpioso|afk is now known as rpioso | 14:55 | |
| mnaser | is infra still maintaining openstack/ansible-role-cloud-launcher ? | 14:56 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Unretire the openstack-chef project https://review.openstack.org/585471 | 14:56 |
| clarkb | fungi: scas ^ that should make zuul happy. We might also want to relax those rules so that we can make project entries uniform across a larger project | 14:57 |
| clarkb | mnaser: in as much as we use it today and we fix the occasional bug. I don't think we've added features in a while | 14:57 |
| mnaser | clarkb: https://review.openstack.org/#/c/588332/ i proposed some test and i think the tests are failing :< | 14:58 |
| tosky | mordred: hi! Regarding sahara-tests and local changes, I did some tests but I probably did some mistake, because I get the wrong result | 14:59 |
| tosky | mordred: namely https://review.openstack.org/#/c/581781/ and https://review.openstack.org/#/c/588515/ (which should test it) | 14:59 |
| scas | clarkb: thanks. i wasn't 100% on the process, but wanted to get it on the schedule with haste | 14:59 |
| *** Emine has quit IRC | 15:00 | |
| *** Emine has joined #openstack-infra | 15:00 | |
| clarkb | mnaser: looks like it expects a devstack cloud to be running that it can talk to, but I don't see any devstack setup | 15:00 |
| evrardjp | pabelanger: fungi I can fix it for new distros if you want. | 15:01 |
| mnaser | clarkb: yep, that seems to be the issue. | 15:01 |
| mnaser | is there an easy way to get devstack in there using a different basejob or so | 15:01 |
| mordred | mnaser: yah - there is a base job | 15:01 |
| mnaser | brrr, guess i signed myself up for more work | 15:02 |
| mnaser | :< | 15:02 |
| fungi | clarkb: we're less than an hour to go time. should we proceed with stopping puppet? | 15:03 |
| mordred | mnaser: devstack-tox-functional-consumer will make a devstack in pre-run and is set upto run tox-based functional tests | 15:03 |
| clarkb | fungi: I already did :) | 15:03 |
| fungi | oh, excellent! | 15:03 |
| clarkb | fungi: please double check ;) | 15:03 |
| mnaser | mordred: wow, that's awesome | 15:03 |
| mordred | mnaser: behold the awesome power of zuul. bow down to its majesty. | 15:04 |
| fungi | i wasn't following the state of our ansible changes super closely yesterday so didn't know if the process for that has changed now | 15:04 |
| mnaser | https://docs.openstack.org/devstack/latest/zuul_jobs.html i didnt get through my reading to get there, thats sweet | 15:04 |
| mnaser | !! | 15:04 |
| openstack | mnaser: Error: "!" is not a valid command. | 15:04 |
| mordred | :) | 15:04 |
| clarkb | fungi: it shouldn't have changed. Only hiera data organization changed | 15:04 |
| fungi | okay, cool | 15:04 |
| clarkb | I still don't know if corvus has had a chance to read through the updated process | 15:05 |
| fungi | it's not too far off from what we did last time | 15:09 |
| fungi | honestly, if we just take a safety snapshot of the pipeline contents in zuul when we're starting, it's probably all the insurance policy we need | 15:09 |
| clarkb | ok, maybe add that as a step? | 15:10 |
| fungi | added | 15:12 |
| clarkb | thanks | 15:12 |
| mordred | clarkb, fungi: I haven't been following the renames super closely, but I am here to help should I be useful | 15:14 |
| clarkb | mordred: https://etherpad.openstack.org/p/project-renames-2018-08-03 is the tracking/process document if yo uwant to give that a read | 15:15 |
| *** jcoufal has joined #openstack-infra | 15:15 | |
| corvus | clarkb: i'll give it a once over now | 15:16 |
| *** jcoufal_ has quit IRC | 15:16 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Switch storyboard url to be by name https://review.openstack.org/588597 | 15:18 |
| corvus | hrm, we don't actually stop gerrit anymore do we? | 15:18 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Add cloud launcher role to infra. channel https://review.openstack.org/588598 | 15:19 |
| openstackgerrit | Mohammed Naser proposed openstack-infra/project-config master: Remove Ansible function jobs from cloud-launcher https://review.openstack.org/588599 | 15:19 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Switch storyboard url to be by name https://review.openstack.org/588600 | 15:20 |
| clarkb | corvus: the playbook stops it | 15:20 |
| fungi | corvus: not directly, but the playbook does | 15:20 |
| mordred | clarkb: thanks | 15:20 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Add netcat to redhat-common map-packages https://review.openstack.org/588601 | 15:20 |
| fungi | playbook is basically 1. stop gerrit, 2. update db, 3. move files, 4. start gerrit, 5. rename groups, 6. initiate online reindex | 15:21 |
| corvus | how long is the downtime? | 15:21 |
| fungi | (some of those middle steps may be in a slightly different order) | 15:21 |
| fungi | corvus: roughly equivalent to a normal gerrit restart | 15:21 |
| clarkb | corvus: I expect about 10 minutes max. I think it takes gerrit about 5 minutes to start now and that gives us 5 minutes to stop and update things (which should be plenty) | 15:21 |
| corvus | okay, since we're not restarting zuul, i guess the idea is we'll roll the dice on not many changes merging during the downtime? | 15:22 |
| clarkb | corvus: ya, to avoid needing to manually push updated project-config everywhere while gerrit spends its time replicating the world | 15:22 |
| corvus | (we used to be able to pause zuul, we should add that back) | 15:22 |
| fungi | a pause certainly would be convenient there | 15:23 |
| clarkb | mordred: one thing you can double check is that the groups/hosts used in the playbook map to the right hosts still. hosts: review for example | 15:23 |
| fungi | but worst case zuul fails to report some change(s) and those have to be requeued by their maintaniers | 15:23 |
| fungi | (i think?) | 15:24 |
| corvus | yeah. i expect that to happen some, but i think this is a reasonable procedure for today. | 15:24 |
| mordred | clarkb: yah - host: review == review01.openstack.org | 15:24 |
| clarkb | mordred: thanks | 15:24 |
| corvus | it'll continue to get better in the future :) | 15:24 |
| fungi | i wonder if the notedb situation will make it possible to do this without gerrit restarts, though i'm not holding out hope | 15:25 |
| corvus | though... we may end up with a bunch of merge failures if a change fails to report and zuul can't re-merge changes behind it. | 15:25 |
| mordred | fwiw: "ansible --list-hosts review" is the way to check that | 15:25 |
| *** janki has quit IRC | 15:29 | |
| *** zoli is now known as zoli|gone | 15:38 | |
| *** zoli|gone is now known as zoli | 15:38 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add pause/unpause support to scheduler https://review.openstack.org/588610 | 15:44 |
| corvus | ^ for later :) | 15:45 |
| clarkb | mordred: in the rename playbook is the comment # TODO: gerrit startup exceeds the timeout, so this task fails | 15:45 |
| clarkb | mordred: will the play after that run even if the task fails? | 15:45 |
| mordred | clarkb: no, it shouldn't | 15:46 |
| clarkb | hrm ok | 15:46 |
| clarkb | thats ok, its the end of the playbook and only group renames and reindexing left. can do that mnaully if gerrit doesn't start quickly neough | 15:46 |
| mordred | clarkb: I think maybe last time we thught about splitting it into two playbooks | 15:46 |
| mordred | clarkb: ++ | 15:46 |
| clarkb | I think we can ignore error there since subsequent play has you check gerrit is happy before hitting enter to coninue | 15:46 |
| clarkb | but I won't do that for this run, it is a straightforward manual set of steps if necessary | 15:47 |
| mordred | agree | 15:47 |
| fungi | sounds fine | 15:48 |
| *** ccamacho has quit IRC | 15:57 | |
| *** mriedem is now known as hansmoleman | 15:57 | |
| * fungi is on hand for the maintenance, but also in release team meeting which is just winding up | 15:58 | |
| fungi | was someone going to send a #status notice? have any wording drafted yet? | 15:58 |
| clarkb | fungi: I hadn't drafted anything yet but maybe "The infra team is renaming projects in Gerrit. There will be a short ~10 minute Gerrit downtime in a few minutes as a result. | 15:59 |
| clarkb | also can someone else review https://review.openstack.org/#/c/575478/ before we force merge? | 15:59 |
| fungi | sounds good. looking at 575478 now | 16:00 |
| clarkb | #status notice The infra team is renaming projects in Gerrit. There will be a short ~10 minute Gerrit downtime in a few minutes as a result. | 16:01 |
| openstackstatus | clarkb: sending notice | 16:01 |
| * clarkb wonders if the notice is actually sending | 16:02 | |
| -openstackstatus- NOTICE: The infra team is renaming projects in Gerrit. There will be a short ~10 minute Gerrit downtime in a few minutes as a result. | 16:02 | |
| clarkb | oh there it is, I'm just not in as many channels that get it early I guess | 16:02 |
| clarkb | shall I force merge the changes now? anyone willing to grab the zuul queues? | 16:03 |
| corvus | i'll grab zuul queues | 16:04 |
| clarkb | maybe wait for notice to finish sending | 16:04 |
| clarkb | corvus: thanks | 16:04 |
| openstackstatus | clarkb: finished sending notice | 16:04 |
| clarkb | I've added myself to the project bootstrappers group and am force merging those two changes now | 16:04 |
| *** Guest10945 has quit IRC | 16:05 | |
| corvus | gimme a second; i think there's a version mismatch between zuul-changes script and running server | 16:05 |
| clarkb | corvus: ok | 16:05 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-website master: WIP Add podcast.__init__ audio stream https://review.openstack.org/588615 | 16:05 |
| fungi | oh, that's fun | 16:06 |
| clarkb | (for those following along changes have not been force merged yet) | 16:06 |
| *** rlandy is now known as rlandy|mtg | 16:07 | |
| corvus | okay, queues saved | 16:07 |
| clarkb | corvus: ready for me to proceed? | 16:07 |
| corvus | patched script in ~root/zuul-changes.py | 16:07 |
| corvus | clarkb: yep | 16:07 |
| openstackgerrit | Merged openstack-infra/project-config master: Unretire the openstack-chef project https://review.openstack.org/585471 | 16:08 |
| openstackgerrit | Merged openstack-infra/project-config master: Rename the API-WG to API-SIG https://review.openstack.org/575478 | 16:08 |
| clarkb | done and I've rmeoved myself from bootstrappers | 16:08 |
| clarkb | can someone check if replication is complete for those changes and I will get ready to run the playbook | 16:09 |
| *** ccamacho has joined #openstack-infra | 16:09 | |
| clarkb | I'm starting a root screen on puppetmaster for that | 16:09 |
| fungi | i'm looking | 16:09 |
| *** gyee has joined #openstack-infra | 16:09 | |
| fungi | there's a backlog of github pushes hours old, but nothing for git.o.o in the queues | 16:10 |
| fungi | er, for gitNN.o.o that is | 16:11 |
| clarkb | https://git.openstack.org/cgit/openstack-infra/project-config/log/ shows the two commits too | 16:11 |
| clarkb | ready to proceed with playbook? | 16:11 |
| fungi | i think so | 16:11 |
| clarkb | ok will run it in root screen on puppetmaster | 16:11 |
| clarkb | it is running | 16:12 |
| fungi | i see the shutdown in the gerrit log now | 16:13 |
| *** melwitt is now known as jgwentworth | 16:14 | |
| clarkb | mysql failed | 16:14 |
| clarkb | couldn't connect | 16:14 |
| mordred | well that's not great | 16:14 |
| clarkb | seems like it was expecting a local connection | 16:15 |
| scas | wasn't me (probably was me) | 16:15 |
| mordred | clarkb: you are running as root? | 16:15 |
| clarkb | mordred: yes | 16:15 |
| fungi | in a screen session | 16:15 |
| clarkb | its in screen | 16:15 |
| clarkb | the issue is I think we don't have a mysql default conf for talking to the remote db | 16:16 |
| mordred | we used to | 16:16 |
| mordred | where did it go? | 16:16 |
| clarkb | mordred: we made a new server guessing that as done by hand? | 16:16 |
| fungi | it's ~root/.gerrit_db.cnf | 16:16 |
| clarkb | mordred: ^ is used for db backups but not default | 16:16 |
| mordred | ah. gotit. then yeah, we should update something to add that file | 16:16 |
| clarkb | mordred: in the interim can we copy that file to the default name and rerun the playbook? | 16:17 |
| fungi | you'll need a copy of the playbook with the gerrit stop removed probably | 16:17 |
| mordred | clarkb: yes. ~/.my.cnf | 16:17 |
| clarkb | ok first I'm copying ~root/.gerrit_db.cnf to ~root/.my.conf | 16:17 |
| clarkb | done | 16:18 |
| clarkb | now will make copy of the playbook and remove the gerrit stop | 16:18 |
| mordred | ++ | 16:18 |
| clarkb | fungi: more if following along in the screen that look good to go now? | 16:19 |
| *** fried_rice is now known as fried_rolls | 16:19 | |
| fungi | i can't see it all, but if you removed the gerrit stop that should be enough i think | 16:20 |
| *** udesale has quit IRC | 16:20 | |
| clarkb | ya I did | 16:20 |
| mordred | yah | 16:20 |
| fungi | lgtm | 16:20 |
| clarkb | running now | 16:20 |
| fungi | watching gerrit error_log now | 16:21 |
| clarkb | gerrit is starting back up now | 16:21 |
| fungi | there it goes | 16:21 |
| fungi | Gerrit Code Review 2.13.9-4-g2a605d5 ready | 16:22 |
| clarkb | oh wow it didn' tfail | 16:22 |
| clarkb | let me double check that ssh and http work before itting enter | 16:22 |
| corvus | raise BadHostKeyException(hostname, server_key, our_key) | 16:22 |
| corvus | paramiko.ssh_exception.BadHostKeyException: ('review.openstack.org', <paramiko.rsakey.RSAKey object at 0x7f76a19076d8>, <paramiko.rsakey.RSAKey object at 0x7f76bb63dfd0>) | 16:22 |
| corvus | zuul is unhappy | 16:22 |
| clarkb | ya local ls-projects over ssh doesn't seem to work either | 16:23 |
| fungi | i wonder if mina-sshd was returning garbage for a bit at start | 16:23 |
| clarkb | fungi: anything in the error log? | 16:23 |
| clarkb | oh there it goes | 16:23 |
| fungi | probably the sshd_log, checking | 16:23 |
| clarkb | though still haven't receied complete output yet | 16:24 |
| clarkb | and now it is done /me tries again | 16:24 |
| clarkb | much quicker now. I think ssh must take a while to startup beyond gerrit saying "ready" | 16:24 |
| clarkb | corvus: is zuul looking happier? | 16:24 |
| fungi | third-party ci systems spam that ssh api really, really hard too, looking at the log | 16:25 |
| fungi | still trying to find the error | 16:25 |
| corvus | clarkb: no the host key is wrong | 16:25 |
| clarkb | corvus: hrm I didn't have to reconfirm it here | 16:25 |
| mordred | I didn't have to reconfirm it here either | 16:25 |
| clarkb | did we configure the wrong key in zuul via puppet? | 16:25 |
| mordred | ssh -p 29418 review.openstack.org gerrit ls-projects | wc -l ... worked for me | 16:26 |
| corvus | gerrit-code-review@review-dev.openstack.org | 16:26 |
| corvus | clarkb: i believe we did | 16:26 |
| fungi | whups! | 16:26 |
| clarkb | ugh | 16:26 |
| corvus | i will manually fix the known_hosts file on zuul | 16:26 |
| clarkb | corvus: thank you | 16:26 |
| fungi | and i guess this is the first time the scheduler had to reconnect after that file got updated | 16:26 |
| clarkb | I'll wait for zuul to be confirmed happy before I hit enter in ansible | 16:27 |
| clarkb | unless you all think we should just continue? | 16:27 |
| fungi | is it only the scheduler which uses ssh access to gerrit, or do we need to worry about mergers? | 16:27 |
| fungi | [2018-08-03 16:24:26,165] [SSH gerrit ls-projects (mordred)] ERROR com.google.gerrit.sshd.BaseCommand : Internal server error (user mordred account 2) during gerrit ls-projects | 16:28 |
| fungi | stream is already closed | 16:28 |
| fungi | i guess he disconnected before it completed | 16:28 |
| corvus | fungi: only the scheduler | 16:28 |
| corvus | i think zuul is gtg now | 16:28 |
| clarkb | ok I'm hitting enter in the screen now to continue with group renames and reindexing | 16:28 |
| mordred | fungi: I just tried again - but without the | wc -l | 16:28 |
| mordred | clarkb: ++ | 16:29 |
| fungi | yeah, i just saw zuul report a build succeeded according to the gerrit logs | 16:29 |
| clarkb | and that is the playbook done | 16:29 |
| *** psachin has joined #openstack-infra | 16:29 | |
| clarkb | fungi: want to handle storyboard? I'm making openstack/openstack-chef active now | 16:30 |
| fungi | yep | 16:30 |
| corvus | i manually re-enqueued the tripleo changes which failed to report | 16:30 |
| clarkb | and I am off to do the github renaming as soon as I get my 2fa token | 16:30 |
| fungi | Rows matched: 1 Changed: 1 Warnings: 0 | 16:31 |
| fungi | sb project-group renamed | 16:31 |
| *** jpich has quit IRC | 16:32 | |
| fungi | the online reindexing seems to have gone extremely quickly | 16:33 |
| clarkb | for the ownership transfer the new organization name is 'openstack' right? | 16:33 |
| clarkb | I guess it doesn't take a full url just the name /me goe sfor it | 16:34 |
| fungi | yeah, just the short org name | 16:34 |
| fungi | as long as you're an admin in both it should work | 16:34 |
| clarkb | yup seems to be there and even asked me if I wanted to give gerrit perms on it | 16:35 |
| clarkb | I think we are to the point where we can reenable puppet, please check me on that | 16:35 |
| fungi | yeah, seems we're safe to go ahead | 16:38 |
| clarkb | looks like cgit isn't serving the new repos yet. I believe because we need manage projects there to update the index lists | 16:38 |
| clarkb | enabling puppet should take care of that | 16:38 |
| clarkb | if I bypass cgit and clone directly the repos seem to be there | 16:38 |
| clarkb | alright reenabling puppet cron now | 16:39 |
| clarkb | done | 16:39 |
| clarkb | should start running in ~6 minutes | 16:39 |
| fungi | gerrit started at around 15k queued tasks and is now under 10k | 16:41 |
| clarkb | java 8 so fast | 16:41 |
| mordred | yay java 8 | 16:41 |
| fungi | so so fast, the java 8 | 16:41 |
| clarkb | the two todo items are fix zuul ssh known hosts and give gerrit a copy of its .my.cnf? | 16:42 |
| corvus | that's my recollection | 16:42 |
| fungi | and maybe add functionality to support project-group renames | 16:42 |
| mordred | we could also just add --defaults-file to the mysql invocation in the playbook | 16:43 |
| clarkb | mordred: actually that might be better | 16:43 |
| fungi | the playbook could in theory add a parameter for the mysql config file name and then | 16:43 |
| fungi | yeah | 16:43 |
| clarkb | since its explicit | 16:43 |
| fungi | exactly what i was thinking too | 16:43 |
| fungi | should do the same for the storyboard mysqlclient config | 16:43 |
| clarkb | corvus: I'm looking at system-config/manifests/site.pp and it isn't immediately clear to me what was wrong with the ssh host key | 16:43 |
| fungi | for consistency | 16:43 |
| * mordred on mysql change | 16:43 | |
| *** jpena is now known as jpena|off | 16:44 | |
| fungi | puppet waking up in 15 seconds | 16:44 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Add .gerrit_db.cfg to project rename playbook https://review.openstack.org/588620 | 16:44 |
| fungi | do we want to tail that log? | 16:44 |
| clarkb | fungi: I'm tailing it outside the screen | 16:45 |
| clarkb | also tailing manage projects log on review.o.o | 16:45 |
| fungi | cool | 16:45 |
| clarkb | fungi: I dind't want puppet log to push the ansible stuff off the screen buffer | 16:46 |
| fungi | yep, good call | 16:46 |
| fungi | (though could just ctrl-a,c and add a new window in it) | 16:46 |
| clarkb | as an aside github didn't actually end up asking for 2fa even though it asked for a password when transfering repo owernship. I think because i was already auth'd with 2fa odd that they wouldn't enforce it anytime they ask for a password though | 16:48 |
| clarkb | git* done now | 16:48 |
| clarkb | on to review | 16:48 |
| clarkb | cgit looks good | 16:49 |
| *** openstackgerrit has quit IRC | 16:49 | |
| clarkb | manage projects is running now | 16:49 |
| fungi | yeah, makes sense that gh would just consider you already authed | 16:50 |
| *** gfidente has quit IRC | 16:50 | |
| *** agopi_ has joined #openstack-infra | 16:51 | |
| clarkb | manage projects ran against both api-sig and openstack-chef and seems to have been happy with them | 16:53 |
| *** agopi has quit IRC | 16:54 | |
| *** agopi_ is now known as agopi | 16:54 | |
| tosky | mordred: now that the gerrit crysis is over, if you have some time for my question... basically https://review.openstack.org/#/c/588515/ does not apply the variant as it should | 16:55 |
| tosky | same question for corvus | 16:56 |
| corvus | clarkb, mordred: i think the fix for the ssh key thing is just in hiera. what do i need to do? edit the same files on both puppetmaster and bridge? | 16:56 |
| * fungi questions the use of "crisis" to refer to a scheduled maintenance | 16:56 | |
| corvus | tosky: gimme 5 mins and i can take a look :) | 16:56 |
| clarkb | corvus: yes I think so. I don't see it in the in repo hiera | 16:56 |
| tosky | fungi: ups, sorry, you are right; it was an hyperbolic way to express the busy time on the channel | 16:56 |
| clarkb | corvus: fyi Aug 3 16:54:02 zuul01 puppet-user[32202]: (/Stage[main]/Zuul::Scheduler/Exec[zuul-reload]) Triggered 'refresh' from 1 events | 16:57 |
| clarkb | I believe that we have cross the point of zuul concern | 16:57 |
| fungi | excellent | 16:57 |
| fungi | i need to spend a few minutes cooking lunch since maintenance has concluded. gerrit queue is down to ~3k tasks now so should clear out in a few more minutes | 16:57 |
| corvus | also "gerrit_zuul_user_ssh_key_contents" is a... weird... name for the public gerrit server host key. | 16:58 |
| mordred | corvus: I agree - and yes to also editing on bridge | 16:58 |
| corvus | i guess i should make the commit on puppetmaster and push the commit to bridge? | 16:59 |
| corvus | that way the git repos stay in sync | 16:59 |
| clarkb | corvus: ya | 16:59 |
| mordred | corvus: there is an extra commit on bridge | 16:59 |
| corvus | mordred: can that commit be pushed to puppetmaster? | 16:59 |
| mordred | no - it removes the production subdirectory | 17:00 |
| corvus | okay, so we're forking the repos, and puppetmaster is a dead-end | 17:00 |
| mordred | yah. I'd just make it both places and we'll work to finish killing puppetmaster as quickly as we can | 17:00 |
| *** derekh is now known as derekh_afk | 17:00 | |
| mordred | corvus: also - we should maybe put on the todo list pulling that out of secrets and into in-tree variables | 17:02 |
| fungi | i had taken a stab at putting the gerrit ssh public host key in public hiera already | 17:02 |
| corvus | mordred: there's a bunch of renamed files on puppetmaster... should i just avoid committing there? | 17:02 |
| mordred | corvus: hrm. lemme look? | 17:03 |
| fungi | but i guess not all uses of the host key got switched over to use the centralized copy | 17:03 |
| mordred | corvus: oh that's the rename - that's totally safe to commit, sorry, I should have committed it yesterday | 17:04 |
| mordred | corvus: we're live with the change to consume that rename | 17:04 |
| corvus | mordred: ok... making them two separate commits at this point will be tricky, but i'll give it a shot | 17:05 |
| corvus | mordred, clarkb, fungi: okay, everything committed on both hosts | 17:07 |
| mordred | \o/ | 17:07 |
| corvus | we might want to send a note to openstack-infra describing the interim procedure for other roots | 17:07 |
| mordred | I shall do that | 17:07 |
| corvus | cool, i will look at tosky's thing | 17:08 |
| mordred | cool | 17:08 |
| clarkb | I'm waiting for puppet run all to finish before giving the all clear on the rename but I think we are all clear on the rename at this point | 17:08 |
| clarkb | ok puppet is done | 17:12 |
| *** calebb has quit IRC | 17:13 | |
| mordred | corvus, clarkb, fungi: https://etherpad.openstack.org/p/FTD8VBMWfw read ok? | 17:14 |
| mordred | (to send to openstack-infra@ | 17:14 |
| fungi | gerrit task queue has stabilized around 450 tasks | 17:15 |
| fungi | looks like it's still indexing nova, neutron and openstack-manuals | 17:15 |
| fungi | the rest are done | 17:16 |
| clarkb | mordred: I'm not quite sure I understand what you mean about removing the production subdir | 17:16 |
| fungi | oh, actually the task queue count is still falling, that's good | 17:16 |
| clarkb | but I think that is necessary for puppet 4? | 17:16 |
| mordred | clarkb: it's necessary on the remote hosts - it is not useful on bridge itself, as "environments" are a puppet concept | 17:17 |
| mordred | clarkb: the syncing code has to put data into two different paths depending on 3 or 4 already anyway | 17:18 |
| clarkb | ya I guess we don't need it source side | 17:18 |
| mordred | so the directory on the puppetmaster/bridge host is not used for anything | 17:18 |
| mordred | yah | 17:18 |
| *** dtantsur is now known as dtantsur|afk | 17:20 | |
| clarkb | I'm not sure the current code copies it where we need it htough? | 17:21 |
| clarkb | er for puppet 4 specifically | 17:21 |
| corvus | tosky: left comment on https://review.openstack.org/581781 | 17:22 |
| logan- | looks like limestone is stuck deleting in nodepool again | 17:22 |
| *** dmellado has quit IRC | 17:22 | |
| *** stevebaker has quit IRC | 17:23 | |
| *** gouthamr has quit IRC | 17:23 | |
| tosky | corvus: oh, thanks a lot; a PEBKAC | 17:23 |
| mordred | clarkb: it's in ansible-role-puppet in tasks/main.yaml - look for Set management server hieradata var | 17:23 |
| tosky | I will go for the branches syntax, it's more clear | 17:23 |
| clarkb | logan-: we'll have t osee if we have logs explaining what happened this time | 17:24 |
| corvus | tosky: if you set this attribute on a project-pipeline in a change where you can't figure out why zuul didn't apply a job (ie, include it in 588515 in this case), zuul will report back with a complete list of all the jobs and variants it considered applying: https://zuul-ci.org/docs/zuul/user/config.html#attr-project.%3Cpipeline%3E.debug | 17:24 |
| mordred | clarkb: ah - however, there is definitely a deficiency there that we'll need to fix before we start running puppet from bridge | 17:24 |
| corvus | tosky: that can be useful to find out why zuul didn't run a job | 17:24 |
| mordred | fixing now | 17:25 |
| tosky | corvus: I see, thanks | 17:25 |
| corvus | tosky: you'd get a line like: Variant <Job sahara-tests-tempest branches: {MatchAny:{BranchMatcher:^(stable/(ocata|pike|queens)).$}} source: openstack/sahara-tests/.zuul.yaml@master#58> did not match <Change 0x7f76ba962f98 588515,1> | 17:25 |
| clarkb | I guess we should send a followup its done notice | 17:26 |
| tosky | corvus: did you manually build that line, or did you check the server logs? | 17:26 |
| *** gouthamr has joined #openstack-infra | 17:26 | |
| clarkb | how about #statuc notice Project renames and review.openstack.org downtime are complete without any major issue. | 17:26 |
| clarkb | corvus: mordred fungi ^ | 17:26 |
| mordred | ++ | 17:26 |
| clarkb | #status notice Project renames and review.openstack.org downtime are complete without any major issue. | 17:27 |
| openstackstatus | clarkb: sending notice | 17:27 |
| corvus | tosky: that's in the debug logs. we log a lot of stuff so that if something weird happens, we can reconstruct it and fix the bug. but the debug:True bit makes it self-service for any user | 17:27 |
| -openstackstatus- NOTICE: Project renames and review.openstack.org downtime are complete without any major issue. | 17:28 | |
| tosky | yeah, makes sense | 17:28 |
| mnaser | clarkb: when you have a second, can you comment/see https://review.openstack.org/#/c/588598/ with openstack infra ptl hat? | 17:28 |
| clarkb | mnaser: ya | 17:29 |
| clarkb | infra-root please review https://review.openstack.org/#/c/588620/1 so that we don't forget it for next time | 17:29 |
| *** psachin has quit IRC | 17:29 | |
| openstackstatus | clarkb: finished sending notice | 17:30 |
| *** openstackgerrit has joined #openstack-infra | 17:30 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/ansible-role-puppet master: Allow explicit override for mgmt_hieradata https://review.openstack.org/588626 | 17:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/system-config master: Set mgmt_hieradata variable for bridge.openstack.org https://review.openstack.org/588627 | 17:31 |
| clarkb | mnaser: done | 17:31 |
| mordred | clarkb: ^^ those two should fix that - good catch | 17:31 |
| clarkb | logan-: ok, you are next on my list then mordreds stack of bridge changes | 17:31 |
| mordred | (we haven't run puppet remotely from bridge yet) | 17:32 |
| mnaser | clarkb: cool, thanks! AJaeger hopefully concerns here are addressed :) https://review.openstack.org/#/c/588598/ | 17:32 |
| *** gouthamr has quit IRC | 17:33 | |
| clarkb | logan-: pabelanger hrm not seeing much in the logs. mordred can you look at nl02.openstack.org:/var/log/nodepool/launcher-debug.log and see if the sahde logs are in there as expected? | 17:35 |
| clarkb | mordred: if I grep for shade and sdk I see nothing | 17:36 |
| mordred | looking | 17:36 |
| clarkb | I thinkwe may not have set up the logging properly like we thought we did | 17:36 |
| mordred | clarkb: what sort of thing are yu looking to see? | 17:36 |
| mordred | clarkb: I see things like 2018-08-03 17:36:28,450 DEBUG nodepool.TaskManager: Manager packethost-us-west-1 running task ComputeGetFlavorsDetail (queue 0) | 17:37 |
| clarkb | mordred: limestone hsa completely stopped processing requests, see grep limestone /var/log/nodepool/launcher-debug.log.2018-08-03_01 | 17:37 |
| clarkb | mordred: no data to the cloud and no new node requests managed | 17:37 |
| clarkb | I think the hope was to see if we sent data to the cloud that caused us to go out to lunch | 17:38 |
| clarkb | 2018-08-03 05:19:51,810 INFO nodepool.PoolWorker.limestone-regionone-main: Assigning node request <NodeRequest {'nodes': [], 'id': '200-0005370277', 'state_time': 1533273591.6037893, 'requestor': 'zuul01.openstack.org', 'reuse': True, 'declined_by': [], 'stat': ZnodeStat(czxid=3971832287, mzxid=3971832287, ctime=1533273591606, mtime=1533273591606, version=0, cversion=0, aversion=0, | 17:38 |
| clarkb | ephemeralOwner=99428362472261746, dataLength=161, numChildren=0, pzxid=3971832287), 'state': 'requested', 'node_types': ['ubuntu-xenial']}> is the last real thing it seems to try to do | 17:38 |
| clarkb | after that its just "deleting nodes" for nodes that are used | 17:38 |
| clarkb | I'm going to try a thread dump if nodepool suppots that | 17:38 |
| mordred | 2018-08-03 05:12:09,447 DEBUG nodepool.TaskManager: Manager limestone-regionone ran task ComputeGetServersDetail in 68.9221122264862s | 17:38 |
| mordred | is the last thing I see where nodepool tried to do somethign via openstacksdk | 17:39 |
| clarkb | I don't see a sigusr2 in nodepool, we should add that (I would do it but parents are here and I'm likely to need to pop out soon and do some family stuff) | 17:39 |
| *** hemna_ has quit IRC | 17:41 | |
| clarkb | 2018-08-03 05:19:41,785 DEBUG nodepool.PoolWorker.limestone-regionone-main: Active requests: ['100-0005370268'] seems to be the active thing it was trying to do | 17:41 |
| corvus | clarkb: it should support sigusr2 | 17:41 |
| mnaser | does infra have/use a role that maintains clouds.yaml file via ansible? | 17:42 |
| clarkb | corvus: oh its in the base class | 17:42 |
| clarkb | but sigusr1 is in the child class | 17:42 |
| clarkb | ok running sigusr2 against nl02 launcher | 17:43 |
| mordred | mnaser: not yet | 17:44 |
| mnaser | mordred: when the time comes, i'd love to work together on making https://github.com/openstack/openstack-ansible-openstack_openrc more generic | 17:44 |
| mnaser | it does 1 cloud only right now but yeah | 17:44 |
| * mnaser is in the process of ansible-izing everything here and sharing stuff with infra would be good | 17:44 | |
| mordred | mnaser: we will soon - although I wasn't really planning on making a generic role though - since you'd basically need to store the clouds.yaml contents in the ansible variables to get it to work | 17:45 |
| mnaser | yeah that's what bothers me a tad | 17:45 |
| mordred | mnaser: that said - if we can come up with a way to make one that's sensible, I'd be all for it | 17:45 |
| mnaser | ++ | 17:45 |
| clarkb | mordred: corvus http://paste.openstack.org/show/727270/ its blocking on that compute detail call finishing I think | 17:47 |
| openstackgerrit | Merged openstack-infra/project-config master: Add cloud launcher role to infra. channel https://review.openstack.org/588598 | 17:48 |
| corvus | mordred, clarkb: does openstacksdk/requests timeout http requests eventually? | 17:49 |
| corvus | i think requests sessions have a timeout option | 17:50 |
| clarkb | corvus: based on 2018-08-03 05:12:09,447 DEBUG nodepool.TaskManager: Manager limestone-regionone ran task ComputeGetServersDetail in 68.9221122264862s it seems like maybe it completed? possibly it timed out and then the cleanup code doesn't release the lock? | 17:50 |
| openstackgerrit | Merged openstack-infra/system-config master: Add .gerrit_db.cfg to project rename playbook https://review.openstack.org/588620 | 17:50 |
| corvus | clarkb: oh, you're saying that the get limits call is waiting to start? | 17:51 |
| corvus | we need to find out what the task manager thread is doing | 17:52 |
| mordred | yeah. there should be a running and a ran line at the top and bottom of each Task | 17:52 |
| clarkb | corvus: the pool worker thread I pasted is waiting for the openstacksdk/shade task to complete. That is signaled by bumping the waiter aiui. I think the log message there shows the task did complete but took a long time (68 seconds), possibly it never bumped the waiter in handling the a timeout if there was one? | 17:53 |
| clarkb | let me see if I can grep for another get server detail | 17:53 |
| corvus | clarkb: that's a get limits call, not get servers detail | 17:53 |
| corvus | clarkb: the stacktrace you posted is the pool worker waiting on a get_compute_limits() call, so the complete ComputeGetServersDetail task is something else | 17:54 |
| clarkb | ah | 17:54 |
| clarkb | ok I don't see a running for get limits | 17:55 |
| clarkb | so maybe it is blocking on another task | 17:55 |
| mnaser | can i get very quick eyes on https://review.openstack.org/#/c/588599/1 ? | 17:56 |
| clarkb | corvus: mordred everything else with limestone in the thread names is a node deleter | 17:56 |
| clarkb | these requests all go through the same task manager to serialize them right? possibly one of the deletes is hanging | 17:56 |
| clarkb | corvus: mordred: if that is the case then adding a timeout would probably be a good idea | 17:56 |
| mordred | they do - althugh I'd still expect to see a "running" line | 17:57 |
| mordred | which the task manager should log before it actually attempts to execute the sdk operation | 17:57 |
| fungi | looks like the gerrit task queue is caught up except for still indexing nova (no surprise) | 17:58 |
| clarkb | mordred: corvus looking at the logs for tasks that ran they were all very fast (subsecond) then we have that slow 68second one last | 17:58 |
| mordred | (adding a timeout is probably a good idea ... I'm just bothered by not seeing breadcrumbs leading me to believe there's a hung http task) | 17:58 |
| corvus | mordred: what are the openstacksdk taskmanager threads named? | 17:58 |
| corvus | or does nodepool create those? | 17:59 |
| mordred | corvus: I believe all of that is still in nodepool | 17:59 |
| corvus | yeah, so they should just be named with the name of the provider | 17:59 |
| corvus | like just "limestone" i think | 17:59 |
| corvus | there is no limestone task manager thread | 18:00 |
| corvus | launcher-debug.log.2018-08-03_09:Thread: citycloud-la1 (140679866984192) | 18:00 |
| corvus | that's what a task_manager thread looks like | 18:00 |
| corvus | launcher-debug.log.2018-08-03_01:2018-08-03 05:12:09,450 ERROR nodepool.TaskManager: Task manager died. | 18:01 |
| corvus | sigh. that's not the way we do error handling. | 18:01 |
| *** rlandy|mtg is now known as rlandy|brb | 18:02 | |
| fungi | on par with using assert ;) | 18:02 |
| corvus | yeah | 18:02 |
| corvus | http://paste.openstack.org/show/727274/ | 18:02 |
| corvus | so something *did* timeout, and apparently we thought the right way to handle that was to die | 18:02 |
| corvus | patch incoming | 18:03 |
| *** amoralej is now known as amoralej|off | 18:03 | |
| mordred | corvus: well, at least some fraction of things 'worked' | 18:03 |
| corvus | actually, this fix is going to take me a bit. i have to do a bunch of research. | 18:04 |
| *** ccamacho has quit IRC | 18:04 | |
| *** ccamacho has joined #openstack-infra | 18:05 | |
| *** jmorgan1 has quit IRC | 18:05 | |
| *** electrofelix has quit IRC | 18:06 | |
| *** ccamacho has quit IRC | 18:09 | |
| corvus | actually i need to afk for a bit; expect the nodepool fix sometime after (my) lunch | 18:10 |
| mordred | corvus: let me know if I can be helpful | 18:10 |
| mordred | corvus: kk | 18:10 |
| corvus | mordred: will do, thx | 18:11 |
| *** jmorgan1 has joined #openstack-infra | 18:11 | |
| *** jmorgan1 has joined #openstack-infra | 18:12 | |
| corvus | if folks want to restart that launcher it's ok with me, i don't need any more debug info | 18:13 |
| clarkb | my quick read of it is that openstack task manager rereaises any exceptions that tasks caught. So if we have an exception that makes it allt hte way to our task manager. We likely just need to catch, log, and then continue rather than die? I think part of the behavior expectation is that shade only raises exceptions in fatal situations though | 18:13 |
| clarkb | shade returns real data for success, falsy data for failures and exceptions for fatal errors. A timeout is probably the exception to this if we only want to catch timeout exceptions and handle those | 18:13 |
| clarkb | I will restart nl02 now | 18:14 |
| clarkb | done, the sha1 reported by pbr freeze has not changed since the lsat restart | 18:14 |
| *** e0ne has joined #openstack-infra | 18:15 | |
| *** openstackgerrit has quit IRC | 18:19 | |
| *** derekh_afk has quit IRC | 18:21 | |
| clarkb | ok things look good I'm going to afk, now. mordred I'll try to get to your changes post lunch | 18:23 |
| *** rlandy|brb is now known as rlandy | 18:34 | |
| *** openstackgerrit has joined #openstack-infra | 18:35 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Swift logs: don't allow links outside of the supplied path https://review.openstack.org/587580 | 18:35 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add functional tests with DevStack https://review.openstack.org/588594 | 18:38 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add extra_specs to flavor https://review.openstack.org/588332 | 18:38 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add os_project_access https://review.openstack.org/588335 | 18:38 |
| mnaser | if anyone has a quick second -- https://review.openstack.org/#/c/588599/ | 18:39 |
| mnaser | that'll help make my stack pass for review | 18:40 |
| mordred | clarkb: ossum | 18:40 |
| fungi | nova changes still being reindexed | 18:43 |
| *** hemna_ has joined #openstack-infra | 18:48 | |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add functional tests with DevStack https://review.openstack.org/588594 | 18:49 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add extra_specs to flavor https://review.openstack.org/588332 | 18:49 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add os_project_access https://review.openstack.org/588335 | 18:49 |
| *** jtomasek has quit IRC | 18:51 | |
| *** bobh has joined #openstack-infra | 18:57 | |
| *** bobh has quit IRC | 18:58 | |
| openstackgerrit | K Jonathan Harker proposed openstack-infra/puppet-elasticsearch master: Add support for setting the bind address https://review.openstack.org/588643 | 18:59 |
| *** gouthamr has joined #openstack-infra | 19:04 | |
| *** derekh has joined #openstack-infra | 19:13 | |
| *** kgiusti has left #openstack-infra | 19:15 | |
| *** derekh has quit IRC | 19:15 | |
| *** fried_rolls is now known as fried_rice | 19:18 | |
| *** jcoufal has quit IRC | 19:26 | |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add functional tests with DevStack https://review.openstack.org/588594 | 19:28 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add extra_specs to flavor https://review.openstack.org/588332 | 19:28 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add os_project_access https://review.openstack.org/588335 | 19:28 |
| *** jcoufal has joined #openstack-infra | 19:29 | |
| mnaser | ahem does anyone know how to locally clone gerrit changes, i'm trying to use this in an ansible requirements.yml | 19:39 |
| clarkb | mnaser: git review -d change number? | 19:39 |
| mnaser | http://paste.openstack.org/show/727279/ (change: https://review.openstack.org/#/c/588335/10 ) seems to fail | 19:39 |
| mnaser | ansible seems to complain about `command git checkout 760da4a77bcd4ffd60eed15556c96b9b99728f01 failed in directory /Users/mnaser/.ansible/tmp/ansible-local-25557CD47wd/tmpTrWvNz/openstack.cloud-launcher` | 19:39 |
| jlvillal | mnaser, What clarkb said. Or if you want to get complicated: git pull origin refs/changes/20/1220/2 | 19:40 |
| mnaser | but i guess i can look at what git review does and put the right values | 19:40 |
| clarkb | Ya I would exec git or git review | 19:40 |
| mnaser | i tried putting version: refs/changes/35/588335/10 to no avail either | 19:41 |
| jlvillal | mnaser, If you look at: https://review.openstack.org/#/c/588335/ And on the right are the download links. git fetch https://git.openstack.org/openstack/ansible-role-cloud-launcher refs/changes/35/588335/10 | 19:42 |
| jlvillal | mnaser, It might need to be a two-step process. Checkout the repo and then do the fetch. | 19:44 |
| mnaser | jlvillal: yeah it doesnt look like i'd be able to do it through ansible requirements.yml | 19:53 |
| *** mdrabe has quit IRC | 19:54 | |
| cgoncalves | very often I get "Cannot retrieve metalink for repository: epel/x86_64. Please verify its path and try again" when building a CentOS 7 DIB image. anyone has an idea? | 19:55 |
| cgoncalves | http://logs.openstack.org/14/587414/5/check/octavia-v2-dsvm-scenario-centos.7/b7ce6d4/job-output.txt.gz#_2018-08-03_19_04_34_400376 | 19:55 |
| *** dmellado has joined #openstack-infra | 19:56 | |
| logan- | mnaser: you need to use a refspec.. let me find you an example | 19:59 |
| logan- | mnaser: https://github.com/logan2211/openstack-ansible-overlay/blob/cb9d8397f1419986178aace282dd2fd25c134410/overlay/env/ansible-role-requirements.yml#L11-L15 | 20:00 |
| *** gouthamr has quit IRC | 20:00 | |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add functional tests with DevStack https://review.openstack.org/588594 | 20:09 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add extra_specs to flavor https://review.openstack.org/588332 | 20:09 |
| openstackgerrit | Mohammed Naser proposed openstack/ansible-role-cloud-launcher master: Add os_project_access https://review.openstack.org/588335 | 20:09 |
| mnaser | logan-: awesome! thank you | 20:09 |
| AJaeger | any infra-root has time for infra-manual reviews, please? https://review.openstack.org/587620 , https://review.openstack.org/586549 and https://review.openstack.org/#/c/570498/ are open reviews | 20:10 |
| AJaeger | infra-root, ianw has done a couple of changes for accessbot, please review the stack at https://review.openstack.org/#/c/588106 | 20:11 |
| *** stevebaker has joined #openstack-infra | 20:14 | |
| *** harlowja has joined #openstack-infra | 20:21 | |
| fungi | [2018-08-03 19:33:55,596] [Reindex v32-v32] INFO com.google.gerrit.server.index.OnlineReindexer : Reindex to version 32 complete | 20:21 |
| fungi | ~3 hours 5 minutes to reindex | 20:22 |
| *** jcoufal has quit IRC | 20:29 | |
| *** e0ne has quit IRC | 20:40 | |
| *** e0ne has joined #openstack-infra | 20:41 | |
| *** agopi has quit IRC | 20:43 | |
| *** e0ne has quit IRC | 20:45 | |
| *** beekneemech is now known as bnemec-pto | 20:55 | |
| pabelanger | dmsimard: mind a +3 on https://review.openstack.org/588368/ for ara, moves to fedora-latest | 20:56 |
| pabelanger | in this case fedora-28 | 20:56 |
| corvus | clarkb, mordred: remote: https://review.openstack.org/588656 Don't wait for task in submit_task | 20:56 |
| openstackgerrit | Merged openstack-infra/infra-manual master: fix URL markup for PEP 440 https://review.openstack.org/587620 | 20:57 |
| corvus | clarkb, mordred: i'm pushing that up at the moment mostly as a question. that appears to me to be the flaw, but if that's the case, i don't understand how things have worked at all -- this should mean that *any* exception from the cloud via openstacksdk should kill nodepool task managers... | 20:57 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Update Zuul Status Page to correct URL https://review.openstack.org/570498 | 20:58 |
| corvus | clarkb, mordred: i'm expecting someone to tell my why that's wrong, and then we can proceed from there :) | 20:58 |
| clarkb | corvus: yes I think any exception to kill the managers, I think it works becaus shade doesn't raise exceptions unless there is a fatal flaw, it returns falsy values for failure | 20:58 |
| corvus | clarkb: wow, so we just haven't been getting exceptions? it looks like this is hitting the "retry network errors once" code path, so i guess like you said earlier, that's the exception to the rule. so we get exceptions if the network is bad.... and somehow, for the past few months, the Internet has been *working*? | 21:00 |
| clarkb | corvus: ya I think so | 21:00 |
| clarkb | mordred should confirm but that is my guess at the moment | 21:00 |
| corvus | i feel like i missed my chance to buy a lottery ticket | 21:00 |
| *** rfolco|ruck is now known as rfolco|off | 21:01 | |
| pabelanger | corvus: clarkb: Oh, thanks for pointing this out | 21:02 |
| pabelanger | 2018-08-02 01:25:27,956 ERROR nodepool.TaskManager: Task manager died. | 21:02 |
| pabelanger | we also get it in rdoproject | 21:02 |
| pabelanger | been trying to figure out for the last few days why our nodepool stopped working there too | 21:03 |
| corvus | pabelanger: strangely, that's reassuring. i figured lack of errors suggested my analysis was wrong, but this improve my confidence. thanks :) | 21:03 |
| pabelanger | yah, in this case, rdocloud goes doen for reasons not related to nodepool, but nodepool never recovers from it., | 21:04 |
| pabelanger | taskmanager dying now explains why that it | 21:04 |
| pabelanger | s/doen/down | 21:04 |
| clarkb | AJaeger: thanks and done | 21:05 |
| clarkb | or at least half done | 21:05 |
| openstackgerrit | Merged openstack-infra/infra-manual master: Remove Zuul v2 content https://review.openstack.org/586549 | 21:07 |
| clarkb | corvus: did you want to review the accessbot changes? https://review.openstack.org/#/c/588134/1 and its parent | 21:12 |
| clarkb | corvus: ianw In any case I +2'd but didn't approve to give corvus a chance to review since I think corvus had indicated interest in this the other day | 21:17 |
| corvus | clarkb: ya, i'll look now | 21:18 |
| clarkb | mordred: https://review.openstack.org/#/c/587540/9 responses to that change make sense to me should we approve it? | 21:20 |
| mordred | clarkb: yes! | 21:20 |
| mordred | also - reading scrollback about shade things above | 21:21 |
| clarkb | mordred: and what about child changes jus approve as things look good? | 21:21 |
| openstackgerrit | Merged openstack-infra/puppet-accessbot master: accessbot logs : add timestamp and rotate https://review.openstack.org/588106 | 21:22 |
| mordred | clarkb: yup. there should be notning in that stack until the one marked WIP that should impact current production at all | 21:22 |
| mordred | clarkb, corvus: I agree with clarkb, we've been lucky because shade/sdk mostly doesn't throw exceptions | 21:23 |
| corvus | mordred: ok, you think my change is the thing to do? | 21:24 |
| mordred | for network issues, there is actually a retry baked in somewhere that will cause a certainly class of exception to get auto-retried | 21:24 |
| mordred | corvus: I haven't digest the change yet - mostly responding to the writeup and discussion | 21:24 |
| corvus | mordred: yeah, there's a retry-once in the task manager | 21:24 |
| corvus | mordred: ok, while you continue to digest, i'll go look into what might be involved in adding a test | 21:25 |
| mordred | corvus: but yes - I believe that is correct - pending that it passes the sdk tests and the nodepool tests | 21:25 |
| corvus | clarkb, ianw: accessbot lgtm, thanks. i'm not in the right frame of mind to approve those right now. | 21:25 |
| mordred | corvus: we should also potentially (not this afternoon necessarily) finish https://review.openstack.org/#/c/574285/ and then get some good explicit testing of both task managers | 21:26 |
| clarkb | mordred: I got as far as https://review.openstack.org/#/c/587985/7 before running out of necessary +2's to approve fwiw | 21:27 |
| mordred | clarkb: awesome. the one 2 after that: https://review.openstack.org/#/c/588394 | 21:27 |
| mordred | is the one I need landed most to be able to continue on bridge itself (need the install_modules change to actually land :) ) | 21:28 |
| corvus | mordred: in 587985 why is git-server set to python2? | 21:28 |
| corvus | like, isn't that the default? | 21:28 |
| corvus | oh you were talking about inverting that | 21:29 |
| corvus | in which case, why is bridge set to python3? :) | 21:29 |
| mordred | not if ansible is installed with python3 - that changes the default remote python it looks for | 21:29 |
| pabelanger | re: 587985 shouldn't we consider a virtualenv for ansible, given recent pip10 issues we had with sudo and os packages? | 21:29 |
| clarkb | pabelanger: possibly | 21:29 |
| clarkb | I mostly wasn't going to require that to start since we don't venv today | 21:30 |
| mordred | corvus: so, I figured go ahead and be explicit with the centos boxen, since they don't have 3 anyway | 21:30 |
| corvus | another option is using ansible from packages. bionic has 2.5.1. | 21:30 |
| mordred | we can invert the other pretty easily- either by putting an ansible_python_interpreter in group_vars/all.yaml or by reinstsalling ansible with python2 | 21:30 |
| corvus | mordred: it's confusing to me to have both of those | 21:31 |
| corvus | mordred: maybe we can drop the one for bridge? | 21:31 |
| clarkb | corvus: re openstacksdk patch nodepool task manager overrides submit_task and does wait on the task there | 21:31 |
| mordred | clarkb: both pythons? | 21:31 |
| mordred | gah | 21:31 |
| mordred | corvus: both pythons? | 21:31 |
| corvus | mordred: yeah. like, if one of those is the default, then one of those shouldn't be needed, right? | 21:31 |
| mordred | corvus: yes. there is currently no python2 installed on brige | 21:32 |
| mordred | bridge | 21:32 |
| * mordred cannot type | 21:32 | |
| corvus | it's spelled "bilge" | 21:32 |
| clarkb | corvus: oh but then it calls into _run_task | 21:32 |
| mordred | zomg. we should have a bilge | 21:32 |
| clarkb | so ya I think that may make it happier | 21:32 |
| * fungi is not volunteering to man the bilge pump | 21:32 | |
| corvus | "captain in the bilge!" | 21:32 |
| clarkb | so ya I think nodepool's code did/does the right thing but then that behavior change in sdk caught nodepool by surprise | 21:33 |
| clarkb | mordred: fungi bilge.pump.openstack.org can be the bug tracker | 21:34 |
| *** apetrich has quit IRC | 21:34 | |
| pabelanger | lolz | 21:37 |
| *** pcaruana has quit IRC | 21:38 | |
| *** boden has quit IRC | 21:40 | |
| openstackgerrit | Merged openstack-infra/system-config master: Add base playbooks and roles to bootstrap a new server https://review.openstack.org/587540 | 21:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Add sudoers file and groups https://review.openstack.org/587854 | 21:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Add bridge.openstack.org to trusted ssh list https://review.openstack.org/587855 | 21:43 |
| openstackgerrit | Merged openstack-infra/system-config master: Add emacs and vim to base-server packages https://review.openstack.org/587983 | 21:44 |
| openstackgerrit | Merged openstack-infra/system-config master: Add pip and virtualenv to bridge.openstack.org https://review.openstack.org/587984 | 21:46 |
| dhellmann | is there a zuul API to tell me how many active jobs are running? I would like to watch for a low point before submitting a bunch of the zuul setting migration patches for Oslo. | 21:47 |
| dhellmann | I've just been looking at the web page and eyeballing the check queue as a proxy for business | 21:47 |
| *** njohnston has quit IRC | 21:47 | |
| *** gouthamr has joined #openstack-infra | 21:47 | |
| clarkb | dhellmann: the status.json file has that info (which is what the dashboard uses to render things). you can also use grafana | 21:48 |
| clarkb | dhellmann: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 and something like https://zuul.openstack.org/api/status | 21:48 |
| dhellmann | ok, I was thinking I might write a little script to fetch it and tell me when it dips below a threshold, but now that I'm thinking out loud I don't know what that value might be | 21:48 |
| dhellmann | it's probably just as easy for me to check it periodically during off-peak times | 21:49 |
| clarkb | dhellmann: I personally like to use the test nodes graph on graphana for this sort of thing because it shows you how much capacity we have free | 21:49 |
| clarkb | right now almost 50% of our capacity is unused | 21:49 |
| dhellmann | aha | 21:50 |
| dhellmann | that's better than what I was doing | 21:50 |
| dhellmann | so we have ~1000 nodes? | 21:50 |
| clarkb | dhellmann: yes | 21:50 |
| clarkb | and actually that reminds me I was going to increase packethost back to 100 if it looked happy today | 21:50 |
| dhellmann | ok. I have somewhere around 150 patches to submit here, so I think I'll see if that number dips lower as the day moves on | 21:50 |
| dhellmann | while my script generates those, I'm going to go run an errand | 21:51 |
| openstackgerrit | Clark Boylan proposed openstack-infra/project-config master: Bump packethost back to 100 max-servers https://review.openstack.org/588669 | 21:53 |
| clarkb | infra-root ^ I think we may be good to do that based on current observations | 21:53 |
| *** rh-jelabarre has quit IRC | 21:56 | |
| clarkb | fungi: pabelanger ^ if still around second revie won that would be much appreciated | 22:12 |
| fungi | yup, looking now | 22:14 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Update fontawesome to version 5 https://review.openstack.org/545676 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Make board and worklist icons unique https://review.openstack.org/545677 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Convert less to scss https://review.openstack.org/379595 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Redesign the sidebar to neaten code and improve contrast https://review.openstack.org/549010 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Reduce the number of items in the sidebar https://review.openstack.org/549059 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Make the page background less bright https://review.openstack.org/549210 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add a drop shadow beneath the header navbar https://review.openstack.org/549211 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Add shadow to board lanes https://review.openstack.org/549212 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Redesign dashboard to reduce clutter https://review.openstack.org/549333 | 22:19 |
| openstackgerrit | Adam Coldrick proposed openstack-infra/storyboard-webclient master: Improve appearance of search bars https://review.openstack.org/588675 | 22:19 |
| *** hongbin has quit IRC | 22:19 | |
| openstackgerrit | Merged openstack-infra/project-config master: Bump packethost back to 100 max-servers https://review.openstack.org/588669 | 22:20 |
| clarkb | fungi: thanks! | 22:20 |
| corvus | clarkb, mordred: i updated remote: https://review.openstack.org/588656 Don't wait for task in submit_task | 22:24 |
| corvus | clarkb, mordred: i added a test and comments | 22:24 |
| corvus | mordred: i agree that adding https://review.openstack.org/574285 will help. we'll still need a test like that with that change. we can incorporate the test i wrote into that later. | 22:26 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Install ca-certificate with redhat-common https://review.openstack.org/588676 | 22:32 |
| *** rlandy has quit IRC | 22:42 | |
| *** derekh has joined #openstack-infra | 22:43 | |
| *** nicolasbock has quit IRC | 22:43 | |
| corvus | clarkb: when you have a sec, can you take a look at https://review.openstack.org/588383 ? | 22:46 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-jobs master: Un-wip upload-logs-swift https://review.openstack.org/588677 | 22:48 |
| clarkb | corvus: yup | 22:48 |
| *** agopi has joined #openstack-infra | 22:51 | |
| *** tpsilva has quit IRC | 22:54 | |
| ianw | corvus: thanks for reviews. i can watch it on my monday when it's quiet. it actually didn't fully run last time, as it took longer than the puppet timeout and got killed. i'll run it manually and see what's up with timing etc | 22:55 |
| ianw | accessbot i mean | 22:55 |
| clarkb | corvus: the change looks functional. I'm a little concerned about some of the performance impact ( we would still filter with crm114 ) left more details inline | 22:58 |
| clarkb | corvus: I think given the existing code it would be a bit more cmplicated to implement my suggestion :/ | 22:58 |
| clarkb | corvus: maybe we still go through the filters but don't run f.process if a flag is set | 22:59 |
| corvus | clarkb: oh, good point. i'll ponder that. | 23:02 |
| corvus | i was also thinking of having a filter, erm, actually *filter* things, like, be able to reject a line. that'd be a change in how it's currently set up, but maybe worth doing based on that concern. | 23:03 |
| clarkb | ya | 23:03 |
| *** mschuppert has quit IRC | 23:03 | |
| *** tosky has quit IRC | 23:04 | |
| *** larainema has quit IRC | 23:07 | |
| *** derekh has quit IRC | 23:08 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add HTMLify logs role https://review.openstack.org/588105 | 23:09 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: import zuul job settings from project-config https://review.openstack.org/588700 | 23:14 |
| openstackgerrit | Doug Hellmann proposed openstack-dev/pbr master: switch documentation job to new PTI https://review.openstack.org/588701 | 23:14 |
| *** rosmaita has quit IRC | 23:17 | |
| dhellmann | does git-review have a mode where it won't prompt when there are multiple commits to propose? like a -y option? | 23:18 |
| dhellmann | ah, yes, it does | 23:18 |
| *** xarses has joined #openstack-infra | 23:24 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for Oslo repositories https://review.openstack.org/588842 | 23:26 |
| *** neiloy has quit IRC | 23:27 | |
| *** rpioso is now known as rpioso|afk | 23:45 | |
| *** sthussey has quit IRC | 23:47 | |
| *** harlowja has quit IRC | 23:48 | |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for Oslo repositories https://review.openstack.org/588842 | 23:51 |
| mordred | corvus: patch looks great. I approved it, and also cherry-picked it back to stable/rocky and added a release note on that branch remote: https://review.openstack.org/588845 Don't wait for task in submit_task | 23:51 |
| mordred | corvus: once that lands I'll get a bugfix release cut | 23:52 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: remove job settings for Oslo repositories https://review.openstack.org/588842 | 23:54 |
| openstackgerrit | Doug Hellmann proposed openstack-infra/project-config master: fix api-wg/sig project settings https://review.openstack.org/588846 | 23:54 |
| dhellmann | corvus , clarkb : I have submitted a bunch of patches to move the zuul job settings from project-config to the oslo repos. See https://review.openstack.org/#/q/topic:python3-first | 23:56 |
| dhellmann | and https://review.openstack.org/588842 is the related project-config change to delete them | 23:56 |
| dhellmann | I would appreciate it if you would look over that project-config change when you have some time, and maybe spot-check some of the other ones | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!