| *** lseki has quit IRC | 00:00 | |
| *** roman_g has quit IRC | 00:01 | |
| *** weifan has quit IRC | 00:03 | |
| *** markvoelker has quit IRC | 00:07 | |
| *** markvoelker has joined #openstack-infra | 00:07 | |
| *** michael-beaver has quit IRC | 00:08 | |
| *** diablo_rojo has quit IRC | 00:11 | |
| *** aaronsheffield has quit IRC | 00:11 | |
| *** rfarr has joined #openstack-infra | 00:11 | |
| *** markvoelker has quit IRC | 00:12 | |
| *** mattw4 has quit IRC | 00:15 | |
| *** mattw4 has joined #openstack-infra | 00:15 | |
| *** mattw4 has quit IRC | 00:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 00:31 |
|---|---|---|
| *** auristor has quit IRC | 00:33 | |
| *** rkukura has quit IRC | 00:38 | |
| *** auristor has joined #openstack-infra | 00:40 | |
| *** rfarr has quit IRC | 00:44 | |
| *** rcernin has joined #openstack-infra | 00:45 | |
| *** rcernin has quit IRC | 00:45 | |
| *** rcernin has joined #openstack-infra | 00:45 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] backups https://review.opendev.org/662657 | 00:50 |
| openstackgerrit | Merged opendev/system-config master: centos-mirror-update: handle non-x86 architectures https://review.opendev.org/662727 | 00:57 |
| *** bobh has quit IRC | 00:59 | |
| *** spsurya has joined #openstack-infra | 01:01 | |
| *** zhurong has quit IRC | 01:04 | |
| *** slaweq has quit IRC | 01:07 | |
| tkajinam | Hi. Can anybody from infra can check my patch for opendev/sandbox? https://review.opendev.org/#/c/663463 | 01:16 |
| tkajinam | I know that I can merge this by myself, but as it is related to infra configuration, I want to get some confirmation | 01:16 |
| *** rcernin has quit IRC | 01:21 | |
| *** rcernin has joined #openstack-infra | 01:21 | |
| *** rkukura has joined #openstack-infra | 01:24 | |
| *** bobh has joined #openstack-infra | 01:26 | |
| ianw | tkajinam: it looks fine :) fungi i think was responsible for https://opendev.org/opendev/sandbox/commit/ab26fe63cb025b2de30b49f46fe8e8cf3c0b49e4 ; not exactly sure why that one didn't rename | 01:27 |
| tkajinam | ianw, thanks for your check. Can I land it or is it better to wait for fungi? | 01:29 |
| ianw | i'd say merge away! | 01:30 |
| *** dave-mccowan has joined #openstack-infra | 01:31 | |
| tkajinam | ianw, thanks for the quick response :-) | 01:36 |
| *** mriedem has quit IRC | 01:38 | |
| *** rkukura has quit IRC | 01:41 | |
| fungi | ianw: to answer your question, it wasn't renamed during the migration, we renamed it last friday with https://review.opendev.org/657275 | 01:43 |
| fungi | but nobody pushed a .gitreview update after that maintenance | 01:44 |
| ianw | ahh, well that explains that :) | 01:44 |
| fungi | so it's been broken less than a week, at least ;) | 01:44 |
| *** bobh has quit IRC | 01:48 | |
| *** hongbin has joined #openstack-infra | 01:55 | |
| *** ykarel|away has joined #openstack-infra | 01:57 | |
| ianw | do we know who runs mirror.lstn.net/centos/ ... it seems it has a AAAA record, but does not actually like ipv6 | 01:59 |
| *** weifan has joined #openstack-infra | 01:59 | |
| *** weifan has quit IRC | 02:03 | |
| *** whoami-rajat has joined #openstack-infra | 02:07 | |
| *** markvoelker has joined #openstack-infra | 02:08 | |
| *** ykarel|away has quit IRC | 02:09 | |
| clarkb | ianw: logan- iirc | 02:17 |
| logan- | hmm | 02:24 |
| *** Goneri has joined #openstack-infra | 02:24 | |
| *** gfidente is now known as gfidente|afk | 02:26 | |
| logan- | ianw clarkb: should be fixed now | 02:29 |
| logan- | thanks for pointing it out. i'm not sure why keepalived didn't have the vip bound, but ill keep an eye on it for a while and make sure it doesn't disappear again. | 02:33 |
| ianw | logan-: cool; i'm just finishing the manual sync https://review.opendev.org/#/c/662727/ ... once it's done with the alt-arch stuff i'll double back to be sure, but i just connected ok, thanks | 02:37 |
| *** jamesmcarthur has joined #openstack-infra | 02:41 | |
| *** markvoelker has quit IRC | 02:42 | |
| *** rlandy|ruck|bbl has quit IRC | 02:45 | |
| *** yamamoto has joined #openstack-infra | 02:50 | |
| *** jamesmcarthur has quit IRC | 03:08 | |
| *** eernst has quit IRC | 03:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add centos/ to centos mirror path https://review.opendev.org/663491 | 03:24 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add centos/ to centos mirror path https://review.opendev.org/663491 | 03:37 |
| ianw | hrw / fungi: ^ still syncing (screen on mirror-update) | 03:40 |
| *** dikonoor has joined #openstack-infra | 03:41 | |
| *** dave-mccowan has quit IRC | 03:50 | |
| fungi | that'll take a while, and so will the vos release | 03:57 |
| *** udesale has joined #openstack-infra | 04:00 | |
| *** yamamoto has quit IRC | 04:00 | |
| ianw | yep it's into the "j" of ppc64 atm | 04:03 |
| *** yamamoto has joined #openstack-infra | 04:06 | |
| *** ramishra has joined #openstack-infra | 04:11 | |
| *** hongbin has quit IRC | 04:11 | |
| *** ykarel|away has joined #openstack-infra | 04:14 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Ansible roles for backup https://review.opendev.org/662657 | 04:27 |
| *** ykarel|away has quit IRC | 04:39 | |
| *** pcaruana has joined #openstack-infra | 04:50 | |
| *** ykarel|away has joined #openstack-infra | 04:57 | |
| *** dikonoor has quit IRC | 04:59 | |
| *** tkajinam has quit IRC | 05:00 | |
| *** ykarel|away is now known as ykarel | 05:03 | |
| *** gregoryo has joined #openstack-infra | 05:05 | |
| *** dikonoor has joined #openstack-infra | 05:12 | |
| *** janki has joined #openstack-infra | 05:30 | |
| ianw | hrw: look ok? http://mirror.dfw.rax.opendev.org/centos/altarch/7/ | 05:34 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable https://review.opendev.org/662662 | 05:38 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Ansible roles for backup https://review.opendev.org/662657 | 05:38 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to ansible-lint 4.1.0 https://review.opendev.org/663501 | 05:38 |
| ianw | yak shaved ... | 05:39 |
| *** jtomasek has joined #openstack-infra | 05:45 | |
| *** janki has quit IRC | 05:49 | |
| *** dtantsur|afk is now known as dtantsur | 05:49 | |
| *** janki has joined #openstack-infra | 05:49 | |
| *** janki has quit IRC | 05:50 | |
| *** janki has joined #openstack-infra | 05:51 | |
| openstackgerrit | Merged opendev/system-config master: Add centos/ to centos mirror path https://review.opendev.org/663491 | 05:55 |
| *** tkajinam has joined #openstack-infra | 06:00 | |
| *** lpetrut has joined #openstack-infra | 06:03 | |
| *** slaweq has joined #openstack-infra | 06:14 | |
| *** gregoryo has quit IRC | 06:16 | |
| ianw | dmsimard: http://logs.openstack.org/57/662657/24/check/system-config-run-base-ansible-devel/5dfe1c5/job-output.txt.gz#_2019-06-06_05_44_55_614770 | 06:16 |
| ianw | ERROR! Unexpected Exception, this is probably a bug: Unable to import Django: the server dependencies can be installed with 'pip install ara[server]' | 06:16 |
| ianw | dunno if that is expected on master branch or not (devel job) | 06:16 |
| *** xek has joined #openstack-infra | 06:16 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 06:17 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 06:19 |
| *** e0ne has joined #openstack-infra | 06:19 | |
| *** udesale has quit IRC | 06:21 | |
| *** udesale has joined #openstack-infra | 06:21 | |
| *** dpawlik has joined #openstack-infra | 06:22 | |
| *** raukadah is now known as chandankumar | 06:24 | |
| *** kopecmartin|off is now known as kopecmartin | 06:25 | |
| *** pgaxatte has joined #openstack-infra | 06:29 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Allow download-artifact to download multiple files https://review.opendev.org/662876 | 06:33 |
| *** dklyle has quit IRC | 06:38 | |
| *** dklyle has joined #openstack-infra | 06:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Return javascript content artifact records to Zuul https://review.opendev.org/663056 | 06:39 |
| *** jtomasek has quit IRC | 06:42 | |
| *** udesale has quit IRC | 06:49 | |
| *** udesale has joined #openstack-infra | 06:50 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Return python artifact records to Zuul https://review.opendev.org/663053 | 06:57 |
| *** gyee has quit IRC | 07:04 | |
| *** e0ne has quit IRC | 07:05 | |
| *** e0ne has joined #openstack-infra | 07:06 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** rcernin has quit IRC | 07:14 | |
| lpetrut | mordred: Hi, we were talking yesterday about transfering the cloudbase-init repo to github.com/cloudbase. We've just deleted the existing repo and should be good to go with the transfer. | 07:17 |
| *** rpittau|afk is now known as rpittau | 07:19 | |
| *** haleyb has quit IRC | 07:22 | |
| *** otherwiseguy has quit IRC | 07:22 | |
| *** weshay has quit IRC | 07:22 | |
| *** mhu has quit IRC | 07:22 | |
| *** mhu has joined #openstack-infra | 07:22 | |
| *** weshay has joined #openstack-infra | 07:23 | |
| *** otherwiseguy has joined #openstack-infra | 07:26 | |
| *** e0ne has quit IRC | 07:26 | |
| *** janki has quit IRC | 07:27 | |
| *** janki has joined #openstack-infra | 07:27 | |
| *** janki has quit IRC | 07:28 | |
| *** jpena|off is now known as jpena | 07:38 | |
| *** roman_g has joined #openstack-infra | 07:52 | |
| *** dchen has quit IRC | 07:56 | |
| *** ralonsoh has joined #openstack-infra | 07:58 | |
| *** markvoelker has joined #openstack-infra | 07:58 | |
| *** pkopec has joined #openstack-infra | 07:59 | |
| *** udesale has quit IRC | 08:01 | |
| *** udesale has joined #openstack-infra | 08:03 | |
| ianw | sigh, looks like there's something up with the handlers for letsencrypt and ansible 2.8 ... so ... yeah | 08:03 |
| ianw | http://logs.openstack.org/57/662657/24/check/system-config-run-mirror/8db2d14/ if anyone feels like ansible deep dive before i get to it | 08:03 |
| *** ociuhandu has joined #openstack-infra | 08:07 | |
| *** ociuhandu has quit IRC | 08:07 | |
| *** ociuhandu has joined #openstack-infra | 08:08 | |
| *** ykarel_ has joined #openstack-infra | 08:10 | |
| *** ykarel has quit IRC | 08:12 | |
| *** ykarel_ is now known as ykarel | 08:14 | |
| *** happyhemant has joined #openstack-infra | 08:19 | |
| *** udesale has quit IRC | 08:24 | |
| *** markvoelker has quit IRC | 08:32 | |
| *** derekh has joined #openstack-infra | 08:41 | |
| *** ginopc has joined #openstack-infra | 08:43 | |
| *** ociuhandu has quit IRC | 08:44 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: WIP: Move FS_TYPE set to after elements are sourced https://review.opendev.org/663543 | 08:46 |
| *** e0ne has joined #openstack-infra | 08:49 | |
| *** udesale has joined #openstack-infra | 08:51 | |
| *** lucasagomes has joined #openstack-infra | 08:51 | |
| *** lucasagomes has quit IRC | 08:52 | |
| *** lucasagomes has joined #openstack-infra | 08:53 | |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Move FS_TYPE set post-element source https://review.opendev.org/663543 | 08:56 |
| *** gfidente|afk has quit IRC | 08:56 | |
| *** ykarel is now known as ykarel|lunch | 09:00 | |
| *** tkajinam has quit IRC | 09:01 | |
| *** jtomasek has joined #openstack-infra | 09:03 | |
| *** sshnaidm|afk is now known as sshnaidm | 09:05 | |
| *** priteau has joined #openstack-infra | 09:12 | |
| *** iokiwi has quit IRC | 09:17 | |
| *** jistr is now known as jistr|lnl | 09:28 | |
| *** markvoelker has joined #openstack-infra | 09:29 | |
| *** janki has joined #openstack-infra | 09:29 | |
| *** ramishra has quit IRC | 09:30 | |
| *** pkopec has quit IRC | 09:32 | |
| *** ykarel|lunch is now known as ykarel | 09:46 | |
| *** Lucas_Gray has joined #openstack-infra | 09:48 | |
| *** gfidente has joined #openstack-infra | 09:57 | |
| *** aedc has joined #openstack-infra | 10:00 | |
| *** ociuhandu has joined #openstack-infra | 10:01 | |
| *** markvoelker has quit IRC | 10:03 | |
| *** ramishra has joined #openstack-infra | 10:11 | |
| *** pkopec has joined #openstack-infra | 10:17 | |
| *** dikonoor has quit IRC | 10:17 | |
| *** priteau has quit IRC | 10:35 | |
| *** jpena is now known as jpena|away | 10:36 | |
| *** yamamoto has quit IRC | 10:37 | |
| *** happyhemant has quit IRC | 10:38 | |
| *** ginopc has quit IRC | 10:47 | |
| *** ginopc has joined #openstack-infra | 10:48 | |
| *** markvoelker has joined #openstack-infra | 10:59 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| *** takamatsu has quit IRC | 11:01 | |
| *** takamatsu has joined #openstack-infra | 11:02 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/openstack-zuul-jobs master: Adds openstack-tox-mol job https://review.opendev.org/663599 | 11:02 |
| openstackgerrit | Sorin Sbarnea proposed openstack/openstack-zuul-jobs master: Adds openstack-tox-mol job https://review.opendev.org/663599 | 11:03 |
| *** yamamoto has quit IRC | 11:06 | |
| slaweq | infra-root: hi, can You do me a favor and put on hold nodes with job networking-ovn-tempest-slow run on my patch https://review.opendev.org/#/c/658326/ - I'm trying to understand what is missing there that there is no connectivty to floating IPs there as on single node job all worked fine | 11:07 |
| slaweq | infra-root: just let me know when I should recheck this job :) | 11:08 |
| slaweq | thx in advance | 11:08 |
| *** pkopec has quit IRC | 11:08 | |
| *** pkopec has joined #openstack-infra | 11:11 | |
| *** pkopec has quit IRC | 11:11 | |
| *** Lucas_Gray has quit IRC | 11:11 | |
| *** Lucas_Gray has joined #openstack-infra | 11:12 | |
| *** yamamoto has joined #openstack-infra | 11:13 | |
| *** pkopec has joined #openstack-infra | 11:14 | |
| frickler | slaweq: give me a couple of minutes | 11:17 |
| *** jistr|lnl is now known as jistr | 11:18 | |
| *** udesale has quit IRC | 11:23 | |
| *** pkopec has quit IRC | 11:26 | |
| zbr | ianw: clarkb fungi : please let me know if you have anything against adding this new tox job as we need it for multiple projects: https://review.opendev.org/#/c/663599/ | 11:30 |
| *** pkopec has joined #openstack-infra | 11:31 | |
| slaweq | frickler: thx a lot | 11:31 |
| *** markvoelker has quit IRC | 11:32 | |
| openstackgerrit | Merged opendev/base-jobs master: Add jobs to publish artifacts to tarballs.opendev.org https://review.opendev.org/662877 | 11:32 |
| openstackgerrit | Carlos Goncalves proposed openstack/diskimage-builder master: Move FS_TYPE set post-element source https://review.opendev.org/663543 | 11:34 |
| frickler | slaweq: always welcome, can you paste me your public key again, please? | 11:46 |
| frickler | nodes will be 149.202.191.161 and 149.202.161.239 | 11:47 |
| *** aedc has quit IRC | 11:48 | |
| *** dikonoor has joined #openstack-infra | 11:59 | |
| *** rh-jelabarre has joined #openstack-infra | 11:59 | |
| *** rlandy has joined #openstack-infra | 11:59 | |
| *** lpetrut has quit IRC | 12:00 | |
| *** lpetrut has joined #openstack-infra | 12:00 | |
| *** rlandy is now known as rlandy|ruck | 12:00 | |
| *** ramishra_ has joined #openstack-infra | 12:03 | |
| slaweq | frickler: my key http://paste.openstack.org/show/752589/ | 12:04 |
| *** ramishra has quit IRC | 12:05 | |
| frickler | slaweq: o.k., added to both nodes. the nodes will only be held if the job actually fails, but you may want to take a look at it while the job is still running | 12:06 |
| slaweq | frickler: yes, I will look in few minutes, thx a lot | 12:06 |
| *** eharney has quit IRC | 12:14 | |
| *** aedc has joined #openstack-infra | 12:20 | |
| *** derekh has quit IRC | 12:21 | |
| *** trident has quit IRC | 12:21 | |
| *** trident has joined #openstack-infra | 12:26 | |
| *** markvoelker has joined #openstack-infra | 12:29 | |
| *** dave-mccowan has joined #openstack-infra | 12:29 | |
| *** Emine has joined #openstack-infra | 12:32 | |
| *** jchhatbar has joined #openstack-infra | 12:36 | |
| *** janki has quit IRC | 12:38 | |
| *** yamamoto has quit IRC | 12:44 | |
| *** jchhatbar has quit IRC | 12:49 | |
| *** yamamoto has joined #openstack-infra | 12:51 | |
| *** tdasilva has joined #openstack-infra | 12:51 | |
| *** aaronsheffield has joined #openstack-infra | 12:51 | |
| *** pcaruana has quit IRC | 12:52 | |
| *** yamamoto has quit IRC | 13:00 | |
| *** dikonoor has quit IRC | 13:01 | |
| AJaeger | zbr: you can add that tox job to any repo - and use it from there everywhere. We share one namespace, everything is global. | 13:02 |
| *** markvoelker has quit IRC | 13:03 | |
| *** haleyb has joined #openstack-infra | 13:03 | |
| *** rfarr has joined #openstack-infra | 13:04 | |
| *** Goneri has quit IRC | 13:04 | |
| *** ekultails has joined #openstack-infra | 13:05 | |
| *** rfarr_ has joined #openstack-infra | 13:05 | |
| zbr | AJaeger: i am not sure it would make sense that his would work. i am concerned that if I add it to, let's say, tripleo-common, others from outside tripleo would try to use it. 2nd is that once is added to a specific repo, if someone decides to change it, it may break unrelated projects. With centralized location I am sure more attention would be made if someone want to modify it. What do you think? | 13:05 |
| *** derekh has joined #openstack-infra | 13:06 | |
| zbr | adding openstack-tox-mol does not block me from adding this job in multiple repos, but wanted to avoid duplication. | 13:06 |
| zbr | the trick is that this requires docker installation as you probably observed with the pre-run.yaml | 13:06 |
| AJaeger | zbr: I read your request - and commit message as: It will not work if it does not life in that repo. And I was replying on that technical argument. I agree that there are process issues... | 13:07 |
| *** rfarr has quit IRC | 13:08 | |
| *** aakarsh has quit IRC | 13:10 | |
| zbr | AJaeger: i didn't create a depends-on change as I was not sure if it people would appreciate it or not, but I do have at least 4 different repositories that could use it, all of them happening to be tripleo*, but there is nothing specific to tripleo about it, could apply to any repo with some ansible roles insie. | 13:10 |
| AJaeger | zbr: on the other hand having it in a central location, means that people review the change that have no clue about molecule while having it in tripleo-common you have tight coupling ;) | 13:10 |
| AJaeger | zbr: I asked for a test in just *one* repo to show that the job is working as expected. | 13:11 |
| zbr | AJaeger: sure, doing it right away. | 13:11 |
| *** bnemec has joined #openstack-infra | 13:11 | |
| AJaeger | zbr: note I'm not objecting to add the job - I just need more information ;) | 13:13 |
| *** ykarel is now known as ykarel|afk | 13:18 | |
| *** Goneri has joined #openstack-infra | 13:22 | |
| *** mriedem has joined #openstack-infra | 13:23 | |
| *** hwoarang has quit IRC | 13:24 | |
| *** hwoarang has joined #openstack-infra | 13:25 | |
| *** tdasilva has quit IRC | 13:28 | |
| *** mloza has joined #openstack-infra | 13:29 | |
| *** udesale has joined #openstack-infra | 13:29 | |
| openstackgerrit | Sorin Sbarnea proposed openstack/openstack-zuul-jobs master: Adds openstack-tox-mol job and template https://review.opendev.org/663599 | 13:30 |
| slaweq | frickler: I hope I found what was missing there so I pushed new PS and probably those nodes which You hold are already destroyed but if not, You can destroy them | 13:35 |
| slaweq | thx a lot for help | 13:35 |
| *** tdasilva has joined #openstack-infra | 13:38 | |
| frickler | slaweq: oh, you pushed the new ps before the job failed. yeah, that deleted those nodes, but also causes the hold request still to be in place, so you will get the next set of nodes held, too ;) | 13:39 |
| *** aakarsh has joined #openstack-infra | 13:39 | |
| *** yamamoto has joined #openstack-infra | 13:39 | |
| slaweq | frickler: ok, good to know | 13:42 |
| slaweq | if those job will still be failing, I will be able to check again | 13:43 |
| slaweq | if not, I will tell here to unhold those nodes than | 13:43 |
| frickler | slaweq: well, if the job passes, the nodes won't be held, either. so we'll have a stale hold request until the next zuul reboot, which has happened before, so it isn't a real issue. also Shrews has some patches up that will hopefully allow us to also delete no-longer-needed hold requests again ;) | 13:45 |
| *** whoami-rajat has quit IRC | 13:47 | |
| *** jaosorior has joined #openstack-infra | 13:47 | |
| *** ramishra_ is now known as ramishra | 13:47 | |
| *** jaosorior has quit IRC | 13:51 | |
| *** jpena|away is now known as jpena | 13:53 | |
| *** whoami-rajat has joined #openstack-infra | 13:57 | |
| *** ricolin has joined #openstack-infra | 13:57 | |
| *** markvoelker has joined #openstack-infra | 13:59 | |
| *** Lucas_Gray has quit IRC | 14:01 | |
| *** kjackal_v2 has joined #openstack-infra | 14:04 | |
| *** kjackal has quit IRC | 14:05 | |
| *** jamesmcarthur has joined #openstack-infra | 14:05 | |
| *** yamamoto has quit IRC | 14:10 | |
| *** dikonoor has joined #openstack-infra | 14:14 | |
| *** jcoufal has joined #openstack-infra | 14:15 | |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Load subproject data from governance https://review.opendev.org/653024 | 14:15 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Switch to stestr https://review.opendev.org/655506 | 14:15 |
| *** lpetrut has quit IRC | 14:16 | |
| *** jcoufal_ has joined #openstack-infra | 14:20 | |
| *** jcoufal has quit IRC | 14:22 | |
| *** Lucas_Gray has joined #openstack-infra | 14:22 | |
| mordred | fungi, corvus: +A'd the opendev-tarballs patches outstanding. they made me think - should we tee up making an opendev.org AFS cell at some point? | 14:24 |
| corvus | mordred: theoretically, but it's both difficult and hidden enough that i think we can defer that for a while | 14:25 |
| mordred | corvus: yeah - I think "at some point" can be a decent ways out into the future | 14:25 |
| *** ykarel|afk is now known as ykarel | 14:25 | |
| clarkb | ya I pondered that when upgrading servers and agree on hidden and difficult | 14:26 |
| corvus | i'm going to wip the final system-config change so we don't add the vhost until we can confirm the LE certs are in place | 14:27 |
| mordred | corvus: ah - good point | 14:28 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Load subproject data from governance https://review.opendev.org/653024 | 14:30 |
| openstackgerrit | Ben Nemec proposed openstack/reviewstats master: Switch to stestr https://review.opendev.org/655506 | 14:30 |
| *** jcoufal_ has quit IRC | 14:31 | |
| *** jcoufal has joined #openstack-infra | 14:31 | |
| *** markvoelker has quit IRC | 14:33 | |
| *** priteau has joined #openstack-infra | 14:36 | |
| *** jaosorior has joined #openstack-infra | 14:37 | |
| *** iurygregory has quit IRC | 14:38 | |
| *** jaosorior has quit IRC | 14:39 | |
| clarkb | re LE ianw found an issue with our le stuff and ansible 2.8 in scrollback | 14:41 |
| *** jaosorior has joined #openstack-infra | 14:41 | |
| clarkb | the upgrade hasnt merged yet so doesnt affect this but testing of the upgrade has a sad | 14:42 |
| *** xek_ has joined #openstack-infra | 14:42 | |
| *** xek has quit IRC | 14:43 | |
| *** rpittau is now known as rpittau|brb | 14:45 | |
| *** yamamoto has joined #openstack-infra | 14:47 | |
| openstackgerrit | Lon Hohberger proposed openstack/diskimage-builder master: Use architecture-specific grub2 RPMs on RHEL8 https://review.opendev.org/663693 | 14:50 |
| *** lseki has joined #openstack-infra | 14:53 | |
| openstackgerrit | Merged opendev/system-config master: Get an LE cert for tarballs.opendev.org https://review.opendev.org/663428 | 14:53 |
| Shrews | frickler: you can delete no longer needed hold requests now. just set count to 0 | 14:54 |
| *** jaosorior has quit IRC | 14:55 | |
| *** yamamoto has quit IRC | 14:55 | |
| *** jaosorior has joined #openstack-infra | 14:57 | |
| frickler | Shrews: oh, cool, seems I missed that update | 14:59 |
| clarkb | it seems that http://logs.openstack.org/57/662657/24/check/system-config-run-mirror/8db2d14/job-output.txt.gz#_2019-06-06_06_42_03_692428 shows ansible 2.8 crashing after running the restart apache handler? | 15:00 |
| clarkb | pabelanger: dmsimard do you know of any known issues in 2.8 with handlers? | 15:01 |
| dmsimard | have not personally come across a particular issue yet | 15:01 |
| asettle | oh clarkb I lost my logs - did you see my ping about the rackspace server? | 15:03 |
| dmsimard | clarkb: "ERROR! The requested handler 'letsencrypt updated mirror01-region-provider-opendev-org-main' was not found in either the main handlers list nor in the listening handlers list" | 15:04 |
| clarkb | dmsimard: why does it report no failures at the play recap then? | 15:04 |
| dmsimard | clarkb: nested playbooks ? I found it in http://logs.openstack.org/57/662657/24/check/system-config-run-mirror/8db2d14/ara-report/result/4776b797-5bfb-4166-acdf-0daf01c23624/ which runs a few of them | 15:05 |
| clarkb | asettle: much of our control plane still runs on rax, though we are becoming more and more multicloud for the control plane | 15:06 |
| asettle | Gotcha. Is that of concern since RAX isn't that integrated with stack anymore? | 15:06 |
| dmsimard | playbooks/service-mirror.yaml looks fine, the error is for playbooks/service-letsencrypt.yaml | 15:06 |
| clarkb | asettle: moving the web server is straightforward as all the data lievs on afs (which can be accessed globally). Moving the filesystems if more difficult | 15:06 |
| asettle | Ah okay | 15:06 |
| frickler | clarkb: the play recap you pointed to was for bridge. the complete recap is here http://logs.openstack.org/57/662657/24/check/system-config-run-mirror/8db2d14/job-output.txt.gz#_2019-06-06_06_42_03_767929 | 15:07 |
| clarkb | asettle: It is of some concern, but they haven't indicated we need to evacuate and we are trying to distribute the load across other servers. However we don't have volunteers to host everything nor the manpower to make such a switch | 15:07 |
| asettle | Oh | 15:08 |
| asettle | That... that's the real problem | 15:08 |
| clarkb | basically what we've been doing is try to make it a rax + others control plane and not a not rax control plane. Distribute the load and have redundancy etc | 15:08 |
| *** pkopec_ has joined #openstack-infra | 15:10 | |
| jamesmcarthur | fungi, rpioso: We are now reading directly from driverlog: https://www.openstack.org/marketplace/drivers/ | 15:10 |
| *** pkopec has quit IRC | 15:10 | |
| jamesmcarthur | I think you'll see many previously missing drivers there. This list will update nightly. | 15:10 |
| *** chandankumar is now known as raukadah | 15:10 | |
| jamesmcarthur | I've confirmed the missing Ironic drivers are there, plus the ones from this review: https://review.opendev.org/#/c/661407/ | 15:11 |
| evrardjp | pabelanger: I am whipping votes on https://review.opendev.org/#/c/662817/ could you give me your opinion? | 15:13 |
| pabelanger | evrardjp: seems fine to me, +2, but cannot watch it merge sadly | 15:13 |
| pabelanger | maybe corvus or clarkb can +A | 15:13 |
| evrardjp | pabelanger: thanks :) | 15:14 |
| evrardjp | I guess I will see with them! I am glad the approach doesn't sound too bad | 15:14 |
| pabelanger | clarkb: speaking of 2.8, we should consder doing a round of executor restarts to pick up new 2.8 support in zuul. I don't believe we have done that yet | 15:14 |
| *** gyee has joined #openstack-infra | 15:16 | |
| clarkb | dmsimard: ok I've confirmed that handlers is in that role's handlers main.yaml https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml#L40-L41 | 15:17 |
| clarkb | https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/tasks/acme.yaml#L5 is that task in the same role | 15:18 |
| mnaser | infra-root: there is a lot of old glance images that have not been deleted by nodepool in sjc1. can i clean those up? | 15:18 |
| mnaser | tc | 15:19 |
| clarkb | mnaser: yes or we can. Shrews ^ I guess we are still leaking images. I think in the past we thought that may be due to sdk saying the deletion was a success witout it actually being? | 15:19 |
| *** jeremyfreudberg has joined #openstack-infra | 15:19 | |
| Shrews | i dunno. i don't have time to look at that now | 15:19 |
| jeremyfreudberg | hi infra, I did a patch a while ago to update the description of some Sahara repos. https://review.opendev.org/#/c/655527/ . i just noticed that the change took effect only github, and not on opendev.org | 15:20 |
| mnaser | clarkb: if ya can do them now, that'd be great, if not, i can do some basic introspection on whats being used right now (or the 2 newest ones) | 15:20 |
| jeremyfreudberg | compare https://opendev.org/openstack/sahara and https://github.com/openstack/sahara | 15:20 |
| clarkb | its been a chronic issue over time that we thought may be due to gearman based rpc then it persisted when we switched to zk and then we thought it could be related to shade and I thought we thought we fixed it but I Guess it still persists | 15:20 |
| clarkb | mnaser: ya I can take a look in a few minutes | 15:20 |
| clarkb | jeremyfreudberg: we don't update opendev.org unless it is repo creation or a manual trigger as it takes many hours to do a full reconciliation | 15:21 |
| clarkb | this is due to ansible task spinup cost so we are going to rewrite those ansible tasks into python script (which is how github updates) | 15:22 |
| jeremyfreudberg | clarkb: got it, thanks! | 15:23 |
| clarkb | mordred: we have control plane image in vexxhost sjc1 now btw | 15:23 |
| clarkb | our test node images don't seem to be updating as expected though | 15:24 |
| clarkb | this means I/we can spin up a new gitea06 | 15:24 |
| clarkb | oh should double check the ext4 journal size first | 15:25 |
| *** rpittau|brb is now known as rpittau | 15:26 | |
| *** jeremyfreudberg has quit IRC | 15:26 | |
| frickler | ianw: from https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.8.html#imports-as-handlers I'd say you need to move from import_tasks to include_tasks here https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml | 15:27 |
| frickler | dmsimard: clarkb: ^^ | 15:27 |
| clarkb | frickler: thanks! | 15:29 |
| openstackgerrit | Merged zuul/zuul-jobs master: Explicitly store date facts for promote https://review.opendev.org/662817 | 15:29 |
| clarkb | mnaser: they are failing to delete | 15:29 |
| mnaser | clarkb: all of them or some of them? | 15:29 |
| mnaser | usually if it fails to delete, that's because there is a volume booted using it | 15:29 |
| clarkb | all of the ones I have attempted so far (just our leaked centos-7 images) | 15:29 |
| clarkb | I'll get a paste for you | 15:29 |
| *** markvoelker has joined #openstack-infra | 15:29 | |
| clarkb | oh wait | 15:30 |
| clarkb | I didn't specify a region | 15:30 |
| clarkb | why is it working at all then | 15:30 |
| clarkb | trying agin with explicit region set | 15:30 |
| *** iurygregory has joined #openstack-infra | 15:31 | |
| frickler | clarkb: mnaser: there were centos-7 nodes stuck in ERROR yesterday or so, maybe that's blocking the deletion | 15:31 |
| clarkb | ok that makes it happier. mordred ^ any idea why some clouds yell at you when you don't set a region and others don't? seems like that should be consistent | 15:31 |
| mnaser | i'd bet some of those are because of the quota issue :\ | 15:31 |
| frickler | and with that, /me goes home | 15:32 |
| clarkb | frickler: good night and thanks for the find on that ansible item | 15:32 |
| *** ykarel is now known as ykarel|away | 15:36 | |
| *** AJaeger has quit IRC | 15:38 | |
| *** pgaxatte has quit IRC | 15:41 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch https://review.opendev.org/644927 | 15:44 |
| *** e0ne has quit IRC | 15:45 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A reporter for Elasticsearch https://review.opendev.org/644927 | 15:46 |
| *** pkopec__ has joined #openstack-infra | 15:46 | |
| *** rpittau is now known as rpittau|afk | 15:46 | |
| *** pkopec__ has quit IRC | 15:47 | |
| *** pkopec_ has quit IRC | 15:49 | |
| *** jaosorior has quit IRC | 15:50 | |
| clarkb | the reason our image builds have stopped functioning is we ran out of disk | 15:51 |
| clarkb | as soon as my oneliner is done removing sjc1 images I'll see if I can't delete some iamges on nb01 to get some disk back | 15:53 |
| *** diablo_rojo has joined #openstack-infra | 15:54 | |
| clarkb | mnaser: can you check your side of things? I think I got all images but those nodepool has records for and those glance says are used | 15:58 |
| *** kjackal_v2 has quit IRC | 16:00 | |
| mnaser | clarkb: thanks, it looks much better.. we can look at the other extras later | 16:00 |
| *** rkukura has joined #openstack-infra | 16:01 | |
| *** piotrowskim has quit IRC | 16:01 | |
| *** anteaya has joined #openstack-infra | 16:02 | |
| *** markvoelker has quit IRC | 16:02 | |
| *** ykarel|away has quit IRC | 16:02 | |
| clarkb | mnaser: I'm asking nodepool to delete the extras it knows about (these aren't leaked but appear to have been overlapped uploads) | 16:03 |
| clarkb | once that is done we should be left with only the images glance says are in use | 16:03 |
| *** tesseract has quit IRC | 16:03 | |
| clarkb | ok those are queued up, just a matter of deletions happening now | 16:05 |
| clarkb | and now to see if there is disk that can be freed on nb01 and nb02 | 16:05 |
| *** whoami-rajat has quit IRC | 16:07 | |
| *** lucasagomes has quit IRC | 16:07 | |
| *** Lucas_Gray has quit IRC | 16:12 | |
| *** whoami-rajat has joined #openstack-infra | 16:12 | |
| fungi | sorry i've been on silent running today, it's been meetings most of my morning and i need to go run errands now but will hopefully finish catching up and pitch in as soon as i get back | 16:13 |
| clarkb | ok I only got about 20GB cleared on nb01 by cleaning up /opt/nodepool_dib | 16:18 |
| clarkb | I'm running a du -hs * in /opt to see whereelse I can clear things out. If the cache is quite large I think I'll stop the builder, delete the cache, reboot to clear mounts, then let it rebuild its cache | 16:19 |
| *** rkukura has quit IRC | 16:19 | |
| clarkb | its possible our cache is leaky as we change what we need tocache over time so that may clear out a bit of space | 16:19 |
| clarkb | oh and clear out dib_tmp | 16:21 |
| clarkb | mordred: you have a mordred dir on nb01:/opt/ can we delete that? looks like you used it to build and stash your control plane images from before | 16:22 |
| clarkb | mordred: or maybe I can move it into your homedir which is on a different device? | 16:22 |
| clarkb | ya I'm gonna move your /opt/mordred dir into /home/mordred/opt/mordred | 16:25 |
| *** tjgresha has joined #openstack-infra | 16:25 | |
| *** dikonoor has quit IRC | 16:27 | |
| *** gfidente has quit IRC | 16:28 | |
| clarkb | similarly I'm going to move /opt/ianw to /home/ianw/opt/ianw | 16:28 |
| *** rkukura has joined #openstack-infra | 16:30 | |
| clarkb | oh wow dib_tmp is half our disk? we must not actually be deleting things because they are mounted? | 16:30 |
| clarkb | so ya I think the cleanup here is disable builder in systemctl, reboot (clears out mounts), delete dib_tmp, enable builder, reboot | 16:30 |
| *** hwoarang has quit IRC | 16:31 | |
| *** hwoarang has joined #openstack-infra | 16:32 | |
| clarkb | I am in the delete dib_tmp contents step | 16:35 |
| clarkb | ianw: any thoughts on how we can make dib more resilient to leaking this stuff? | 16:35 |
| clarkb | we must have many many really small files as this rm is not fast | 16:40 |
| clarkb | one idea is we could have nodepool-builder clear whatever files it owns in the specified tempdir on startup | 16:43 |
| clarkb | might make for the occasional slow startup but should help to keep things in check | 16:43 |
| *** xek_ has quit IRC | 16:45 | |
| *** mgoddard has quit IRC | 16:48 | |
| *** mgoddard has joined #openstack-infra | 16:50 | |
| clarkb | nb01 is done and has a lot of disk now. Doing the same on nb02 | 16:50 |
| *** Emine has quit IRC | 16:50 | |
| mordred | clarkb: sorry - was on the phone - I do not need that /opt/mordred anymore | 16:54 |
| *** ricolin has quit IRC | 16:54 | |
| clarkb | mordred: it is in your homedir so feel free to clean up as you wish :) | 16:55 |
| *** hwoarang has quit IRC | 16:56 | |
| clarkb | nb02 is now done too | 16:56 |
| clarkb | both builders have a ton of disk and are each building an image now. Note I did not end up deleting the cache as it was only 35GB or so | 16:56 |
| clarkb | clearing dib_tmp was the ticket | 16:56 |
| *** weifan has joined #openstack-infra | 16:57 | |
| *** hwoarang has joined #openstack-infra | 16:57 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Add caching of autohold requests https://review.opendev.org/663412 | 16:58 |
| *** markvoelker has joined #openstack-infra | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable https://review.opendev.org/662662 | 17:01 |
| clarkb | ianw: ^ went ahead and updated that for you to reduce rtt on test results | 17:01 |
| *** mrhillsman is now known as openlab | 17:02 | |
| clarkb | infra-root https://review.opendev.org/#/c/663146/ whcih adds db backups to gitea servers is +1 from zuul now | 17:03 |
| clarkb | I'm gonna pop out for a bike ride but will be back to check that the cleaned up builders are successfully building now and I want to do scan/cleanup of leaked images in clouds that are not vexxhost | 17:03 |
| *** dpawlik has quit IRC | 17:04 | |
| *** dpawlik has joined #openstack-infra | 17:04 | |
| *** openlab is now known as codebauss | 17:05 | |
| *** dtantsur is now known as dtantsur|afk | 17:06 | |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Increase the size of ext4 journal on control plane images https://review.opendev.org/663726 | 17:06 |
| clarkb | infra-root I think we want ^ before we rebuild gitea06 | 17:06 |
| clarkb | and now I'm really popping out. Back in a bit | 17:07 |
| mordred | clarkb: yay control plane images! | 17:07 |
| *** dpawlik has quit IRC | 17:10 | |
| *** ginopc has quit IRC | 17:10 | |
| *** codebauss is now known as openlab | 17:13 | |
| *** openlab is now known as codebauss | 17:14 | |
| *** codebauss is now known as openlab | 17:15 | |
| *** bobh has joined #openstack-infra | 17:15 | |
| *** openlab is now known as codebauss | 17:16 | |
| *** kopecmartin is now known as kopecmartin|off | 17:17 | |
| *** jtomasek has quit IRC | 17:19 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Store autohold requests in zookeeper https://review.opendev.org/661114 | 17:19 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Add caching of autohold requests https://review.opendev.org/663412 | 17:19 |
| *** yolanda__ has joined #openstack-infra | 17:20 | |
| *** mattw4 has joined #openstack-infra | 17:22 | |
| *** bobh has quit IRC | 17:22 | |
| *** yolanda has quit IRC | 17:23 | |
| *** jpena is now known as jpena|off | 17:23 | |
| *** codebauss is now known as mrhillsman | 17:24 | |
| *** dpawlik has joined #openstack-infra | 17:26 | |
| *** dpawlik has quit IRC | 17:31 | |
| *** markvoelker has quit IRC | 17:32 | |
| *** aedc has quit IRC | 17:32 | |
| *** aedc has joined #openstack-infra | 17:33 | |
| *** jamesmcarthur has quit IRC | 17:33 | |
| *** aedc has quit IRC | 17:37 | |
| openstackgerrit | Merged openstack/project-config master: Increase the size of ext4 journal on control plane images https://review.opendev.org/663726 | 17:39 |
| *** ociuhandu_ has joined #openstack-infra | 17:39 | |
| *** spsurya has quit IRC | 17:40 | |
| *** udesale has quit IRC | 17:42 | |
| *** dpawlik has joined #openstack-infra | 17:42 | |
| *** ociuhandu has quit IRC | 17:42 | |
| *** ociuhandu_ has quit IRC | 17:43 | |
| *** dpawlik has quit IRC | 17:46 | |
| *** tdasilva has quit IRC | 17:48 | |
| corvus | clarkb, mordred, ianw, fungi: there's a LE cert on files02 for tarballs.opendev.org :) | 17:49 |
| corvus | i will re-enqueue my deployment change once my schedule clears | 17:50 |
| *** Vadmacs has joined #openstack-infra | 17:53 | |
| mnaser | clarkb: thanks, it looks much better.. we can look at the other extras later | 17:56 |
| mnaser | oops, sorry, accidental arrow up and enter. | 17:56 |
| *** priteau has quit IRC | 17:56 | |
| fungi | corvus: excellent! | 17:57 |
| fungi | do we know yet why the apache restart handler crashes ansible 2.8? | 17:58 |
| *** panda has quit IRC | 17:59 | |
| rpioso | jamesmcarthur: Well done! The change you made is a nice improvement. And it's good to see all of those previously missing in action drivers. Thank you very much :-) | 17:59 |
| *** lennyb has quit IRC | 18:00 | |
| *** panda has joined #openstack-infra | 18:01 | |
| *** michael-beaver has joined #openstack-infra | 18:02 | |
| *** mriedem is now known as mriedem_lunch | 18:02 | |
| *** mgoddard has quit IRC | 18:03 | |
| *** mgoddard has joined #openstack-infra | 18:03 | |
| *** ralonsoh has quit IRC | 18:04 | |
| *** roman_g has quit IRC | 18:04 | |
| *** Goneri has quit IRC | 18:05 | |
| *** sshnaidm is now known as sshnaidm|off | 18:06 | |
| clarkb | fungi: yes https://review.opendev.org/662662 updated to fix it | 18:09 |
| clarkb | has a link to the docs too | 18:09 |
| fungi | awesome | 18:11 |
| fungi | seems i skimmed scrollback too quickly | 18:11 |
| clarkb | nb02 successfully built a bionic image ~6 minutes ago | 18:11 |
| clarkb | nb01 is in the middle of a centos-7 build | 18:12 |
| fungi | and looks like 662662 is failing all the run jobs besides run-base | 18:13 |
| clarkb | possible that include_tasks there isn't the drop in replacement I expected | 18:14 |
| fungi | "Could not find or access '/home/zuul/src/opendev.org/opendev/system-config/playbooks/touch_file.yaml' on the Ansible Controller." | 18:15 |
| openstackgerrit | Paul Belanger proposed zuul/nodepool master: Toggle host-key-checking for openstack provider.labels https://review.opendev.org/663378 | 18:16 |
| fungi | i *think* that's the error, but hard to pick out of the nested ara report | 18:16 |
| fungi | oh, right, we have a second layer ara in there | 18:16 |
| clarkb | ya thats one of the handlers that got renamed from import_tasks to include_tasks | 18:16 |
| clarkb | the path should be local to the roles/foo/handlers dir | 18:16 |
| fungi | but yep http://logs.openstack.org/62/662662/8/check/system-config-run-letsencrypt/ac4c99a/bridge.openstack.org/ara-report/result/dd3ee162-9245-469b-adbc-3b69ffb4adcc/ | 18:17 |
| *** Goneri has joined #openstack-infra | 18:17 | |
| clarkb | the docs say nothing about pathing for that :/ | 18:17 |
| clarkb | https://github.com/ansible/ansible/issues/33784 says this should've been fixed in 2.5 | 18:20 |
| clarkb | new regression possibly? | 18:20 |
| clarkb | dmsimard: pabelanger ^ is there a canonical way to do this? I guess provide the entire relative path relative to the playbook dir? | 18:24 |
| * clarkb updates to see if that works | 18:24 | |
| *** aedc has joined #openstack-infra | 18:24 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: bridge.opendev.org: use Ansible 2.8.0 stable https://review.opendev.org/662662 | 18:25 |
| *** e0ne has joined #openstack-infra | 18:26 | |
| *** mattw4 has quit IRC | 18:26 | |
| *** mattw4 has joined #openstack-infra | 18:27 | |
| *** markvoelker has joined #openstack-infra | 18:29 | |
| clarkb | nb01 did successfully build that centos-7 image | 18:30 |
| clarkb | I've queued up a control plane image rebuild as the ext4 journal update seems to be in place | 18:31 |
| *** e0ne has quit IRC | 18:31 | |
| *** e0ne has joined #openstack-infra | 18:32 | |
| pabelanger | clarkb: I've never really used include_tasks in a handler, would have to debug more | 18:36 |
| pabelanger | possible you need to add something like: include_tasks: ../tasks/foo.yaml | 18:36 |
| pabelanger | as it might be trying to load in the handlers directory | 18:37 |
| *** dpawlik has joined #openstack-infra | 18:37 | |
| *** e0ne has quit IRC | 18:38 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Add caching of autohold requests https://review.opendev.org/663412 | 18:39 |
| *** dpawlik has quit IRC | 18:42 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: DNM: Debug wsgi ci failures https://review.opendev.org/663745 | 18:42 |
| clarkb | can I tempt anyone with reviews for gitea db backups https://review.opendev.org/663146 | 18:46 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 18:48 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 18:48 |
| fungi | clarkb: yup! you certainly can | 18:49 |
| fungi | and then i have some changes i need to update | 18:49 |
| fungi | i should get some dnm changes up to use base-test so we can exercise the fetch-zuul-cloner removal today too | 18:50 |
| clarkb | ++ | 18:51 |
| *** mattw4 has quit IRC | 18:53 | |
| *** kjackal has joined #openstack-infra | 18:53 | |
| *** mattw4 has joined #openstack-infra | 18:53 | |
| *** dpawlik has joined #openstack-infra | 18:53 | |
| *** yamamoto has joined #openstack-infra | 18:53 | |
| *** jamesmcarthur has joined #openstack-infra | 18:56 | |
| *** mattw4 has quit IRC | 18:57 | |
| *** yamamoto has quit IRC | 18:58 | |
| *** dpawlik has quit IRC | 18:58 | |
| *** markvoelker has quit IRC | 19:02 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: DNM: Debug wsgi ci failures https://review.opendev.org/663745 | 19:03 |
| *** Vadmacs has quit IRC | 19:03 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Update axios version and yarn.lock https://review.opendev.org/662316 | 19:06 |
| *** rkukura has quit IRC | 19:07 | |
| fungi | clarkb: small question on 663146 but otherwise lgtm | 19:09 |
| clarkb | fungi: responded | 19:12 |
| *** jamesmcarthur has quit IRC | 19:12 | |
| corvus | i also responded with very similar words | 19:13 |
| *** mriedem_lunch is now known as mriedem | 19:13 | |
| *** jamesmcarthur has joined #openstack-infra | 19:14 | |
| fungi | thanks! | 19:14 |
| clarkb | hopefully the way I folded the string makes it easier to read that too | 19:15 |
| clarkb | https://review.opendev.org/#/c/662662/ passes testing after I updated the paths to be relative to the playbooks dir | 19:16 |
| clarkb | that is the ansible 2.8 change | 19:16 |
| *** e0ne has joined #openstack-infra | 19:17 | |
| *** owalsh has quit IRC | 19:18 | |
| *** bnemec has quit IRC | 19:23 | |
| *** d34dh0r53 has quit IRC | 19:25 | |
| *** zbr has quit IRC | 19:25 | |
| *** cloudnull has quit IRC | 19:25 | |
| *** Goneri has quit IRC | 19:25 | |
| *** bnemec has joined #openstack-infra | 19:25 | |
| *** cloudnull has joined #openstack-infra | 19:30 | |
| *** AJaeger has joined #openstack-infra | 19:31 | |
| *** bnemec has quit IRC | 19:31 | |
| AJaeger | mordred, fungi, do we need a variant of https://review.opendev.org/#/c/535836 for bindep removal? Or can it be abandoned? | 19:31 |
| *** d34dh0r53 has joined #openstack-infra | 19:31 | |
| fungi | AJaeger: i think clarkb has sufficiently convinced me that it already works that way | 19:34 |
| *** owalsh has joined #openstack-infra | 19:34 | |
| *** bnemec has joined #openstack-infra | 19:34 | |
| mordred | want me to abandon? | 19:34 |
| fungi | AJaeger: oh, my mistake, it already avoids *running* bindep, but 535836 avoids installing it when it's not going to be used | 19:34 |
| fungi | i don't think that one is urgent | 19:35 |
| fungi | it just improves performance in jobs slightly | 19:35 |
| mordred | ah - yeah | 19:35 |
| fungi | it does still seem useful | 19:36 |
| AJaeger | mordred: it's ancient - want to update? | 19:36 |
| openstackgerrit | Merged zuul/nodepool master: Toggle host-key-checking for openstack provider.labels https://review.opendev.org/663378 | 19:36 |
| AJaeger | mordred: you have quite a few open changes that are stale - want to cleanup that list one time? ;) | 19:37 |
| openstackgerrit | Merged opendev/system-config master: Add db backups to gitea https://review.opendev.org/663146 | 19:38 |
| AJaeger | fungi: want to abandon https://review.opendev.org/461878 - flatten namespace infra-specs change... | 19:38 |
| fungi | oh, yep, that has recently become redundant. thanks! | 19:39 |
| *** Goneri has joined #openstack-infra | 19:39 | |
| *** ekultails has quit IRC | 19:40 | |
| AJaeger | ;) | 19:41 |
| *** bnemec has quit IRC | 19:41 | |
| *** AJaeger has quit IRC | 19:42 | |
| *** bnemec has joined #openstack-infra | 19:42 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add caching of autohold requests https://review.opendev.org/663412 | 19:43 |
| *** ekultails has joined #openstack-infra | 19:44 | |
| *** Goneri has quit IRC | 19:47 | |
| *** rkukura has joined #openstack-infra | 19:50 | |
| openstackgerrit | George Dimitrov proposed opendev/git-review master: Fix parsing of review over http https://review.opendev.org/641484 | 19:52 |
| *** imacdonn has quit IRC | 19:53 | |
| *** imacdonn has joined #openstack-infra | 19:54 | |
| *** jamesmcarthur has quit IRC | 19:55 | |
| *** jamesmcarthur has joined #openstack-infra | 19:56 | |
| *** markvoelker has joined #openstack-infra | 19:59 | |
| *** Goneri has joined #openstack-infra | 20:00 | |
| *** rlandy|ruck is now known as rlandy|ruck|brb | 20:02 | |
| openstackgerrit | Merged opendev/system-config master: Add tarballs.opendev.org vhost https://review.opendev.org/663431 | 20:03 |
| *** eernst has joined #openstack-infra | 20:04 | |
| *** panda has quit IRC | 20:04 | |
| *** e0ne has quit IRC | 20:05 | |
| *** eernst has quit IRC | 20:05 | |
| clarkb | oh also if anyone is wondering the gitea backups are fairly small < 10MB once compressed | 20:05 |
| *** panda has joined #openstack-infra | 20:05 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: DNM: Debug wsgi ci failures https://review.opendev.org/663745 | 20:06 |
| mordred | clarkb: yay | 20:06 |
| *** mattw4 has joined #openstack-infra | 20:07 | |
| *** ccamacho has quit IRC | 20:08 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: Make WSGI tests listen on localhost https://review.opendev.org/663758 | 20:11 |
| corvus | looks like the merge just missed the ansipuppet cycle | 20:13 |
| *** bnemec has quit IRC | 20:16 | |
| *** rlandy|ruck|brb is now known as rlandy|ruck | 20:17 | |
| fungi | clarkb: makes sense, it's almost all metadata i'm sure | 20:17 |
| *** bnemec has joined #openstack-infra | 20:17 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Auto-delete expired autohold requests https://review.opendev.org/663762 | 20:20 |
| *** tjgresha has quit IRC | 20:21 | |
| openstackgerrit | Ben Nemec proposed openstack/pbr master: DNM: Debug wsgi ci failures https://review.opendev.org/663745 | 20:23 |
| *** jamesmcarthur has quit IRC | 20:24 | |
| fungi | infra-root: from fanatical support, "This message is to inform you that our monitoring systems have detected a problem with the server which hosts your Cloud Block Storage device 'afs02.dfw.openstack.org/main02' at 19:50 UTC. We are currently investigating the issue and will update you as soon as we have additional information regarding the alert. Please do not access or modify | 20:26 |
| fungi | 'afs02.dfw.openstack.org/main02' during this process." | 20:26 |
| *** jamesmcarthur has joined #openstack-infra | 20:27 | |
| *** jamesmcarthur_ has joined #openstack-infra | 20:29 | |
| fungi | looks like afs started panicking on the server as of 17:01:35z | 20:29 |
| fungi | currently complaining about i/o errors on xvdc | 20:29 |
| *** jamesmcarthur has quit IRC | 20:31 | |
| *** mattw4 has quit IRC | 20:31 | |
| fungi | per that message, i guess we should avoid rebooting it just yet | 20:31 |
| *** mattw4 has joined #openstack-infra | 20:31 | |
| *** markvoelker has quit IRC | 20:32 | |
| clarkb | iirc RW is on afs01.dfw across the baord anyway | 20:33 |
| corvus | yeah, i think 02 is all RO replicas | 20:47 |
| fungi | okay, so after we get the all-clear from fanatical support we can just reboot it | 20:50 |
| fungi | and presumably make sure it resyncs correctly (however long that takes) | 20:50 |
| *** jcoufal has quit IRC | 20:50 | |
| fungi | "This message is to inform you that your Cloud Block Storage device afs02.dfw.openstack.org/main02 has been returned to service." | 20:53 |
| fungi | guess i'll reboot it now and see what happens | 20:53 |
| corvus | i'm working on fixing files.o.o | 20:54 |
| fungi | what happened to files.o.o? | 20:54 |
| corvus | change merged | 20:54 |
| fungi | oh, that | 20:54 |
| *** dpawlik has joined #openstack-infra | 20:54 | |
| corvus | ugh, the vhost files are completely wrong | 20:54 |
| fungi | #status log rebooted afs02.dfw.openstack.org following a cinder volume outage for xvdc | 20:56 |
| openstackstatus | fungi: finished logging | 20:56 |
| fungi | it's not come up yet, likely fsck underway | 20:57 |
| *** dpawlik has quit IRC | 20:59 | |
| fungi | if you need help with files.o.o, lmk | 20:59 |
| corvus | back up now | 20:59 |
| corvus | so what it did is it wrote blank certificate path names for tarballs.opendev.org, docs.opendev.org, docs.starlingx.io, and zuul-ci.org | 21:00 |
| corvus | https://review.opendev.org/663431 | 21:00 |
| fungi | hrm, undefined variables in the erb? | 21:01 |
| corvus | i don't understand what's missing there, but i admit, i don't understand variables in puppet | 21:01 |
| *** ekultails has quit IRC | 21:02 | |
| fungi | i suspect it's a puppet scoping thing, and @ is referring to variables in the vhost resource scope not the containing manifest? checking for another place we've done this | 21:02 |
| *** jamesmcarthur_ has quit IRC | 21:03 | |
| clarkb | the problem is scoping | 21:03 |
| clarkb | @ssl_cert_file and friends are only in the website.pp scope not the vhost scope | 21:03 |
| fungi | yeah, other places where we're doing this we use: | 21:03 |
| fungi | SSLCertificateFile <%= scope['openstack_project::static::cert_file'] %> | 21:03 |
| clarkb | so you have to look up the other location's value | 21:03 |
| fungi | or similar | 21:03 |
| fungi | because those are classvars for the files class, not attributes of the vhost resource | 21:04 |
| fungi | er, website class | 21:04 |
| corvus | do either of you know which keyboard buttons to press to fix this? | 21:05 |
| fungi | yeah, i can copy what we're using on static.o.o | 21:05 |
| TheJulia | Question, is there any way to configure node selection such that a job will always land on an ipv6 enabled cloud? | 21:05 |
| fungi | just a moment | 21:05 |
| *** jamesmcarthur has joined #openstack-infra | 21:05 | |
| clarkb | TheJulia: no | 21:05 |
| TheJulia | that is going to be ouchy for ipv6 testing :( | 21:06 |
| clarkb | TheJulia: what do you need an ipv6 enabled cloud for? | 21:06 |
| clarkb | right so I've expalined this to the TC a couple times now | 21:06 |
| fungi | TheJulia: nope, but all our nodes are ipv6-enabled, they may just lack global v6 routes. it's best not to have job behaviors depend on the routing available to the nodes | 21:06 |
| clarkb | what probably should be done to test the ipv6 only is supported assertion is do packet captures | 21:06 |
| clarkb | then allow external connectivity to external resources (zuul, dns, mirrors, etc) via a whitelist | 21:06 |
| fungi | i did mention that as a suggestion in my goal rewrite too | 21:06 |
| clarkb | then if you have any other ipv4 traffic fail the test | 21:06 |
| TheJulia | clarkb: well, the devstack provided host_ipv6 var can't be trusted unless devstack changes :\ | 21:07 |
| clarkb | TheJulia: not sure I understand. The trust comes from the external packet capture check | 21:07 |
| TheJulia | clarkb: so not about validation, more about job mechanics of actually having things work reliably because I've seen it set and not set on job runs. | 21:07 |
| clarkb | if you see traffic that isn't whitelisted you fail | 21:07 |
| clarkb | TheJulia: any traffic that ends up on v4 would show up in your capture though | 21:08 |
| TheJulia | clarkb: I'm talking about something different than goal validation, I'm talking about actually being able to trust the variables in the jobs | 21:08 |
| clarkb | that should cover your concern? | 21:08 |
| clarkb | TheJulia: and I'm saying if the variable isn't working properly the validation should catch that | 21:08 |
| fungi | the job ought not depend on what routing is available to the node | 21:08 |
| TheJulia | clarkb: that would be nice :) | 21:09 |
| fungi | as an analog, swift wants to test that their software works with xfs, but our nodes are provisioned with ext4 filesystems. the solution is to create a virtual filesystem and use that, not expect special nodes which are booted on xfs instead | 21:09 |
| TheJulia | fungi: If that is what causes HOST_IPV6 to be set or not :\ | 21:09 |
| TheJulia | fungi: doing that already but yeah, it creates a fun code path | 21:10 |
| *** dpawlik has joined #openstack-infra | 21:10 | |
| clarkb | ok I think I understand better you are saying devstack auto detects that for you and won't on ipv4 only clouds. For that we should just set the value | 21:10 |
| clarkb | don't rely on devstack to autodetect | 21:10 |
| clarkb | that could be via virtual network that is the same on all test runs | 21:11 |
| fungi | yep, we should be asking devstack for consistent behavior in our jobs, not context-dependent behavior guessed from the system | 21:11 |
| TheJulia | ++++++ | 21:11 |
| clarkb | (or ansible in zuul that checks host facts | 21:11 |
| TheJulia | fungi: that | 21:11 |
| fungi | the context-dependent guessing is likely useful for local use, but doesn't lead to consistent results | 21:12 |
| clarkb | the gitea backup cron is in place now | 21:13 |
| *** dpawlik has quit IRC | 21:14 | |
| clarkb | we did similar with nova net once upon a time and always set up networking for it as if it were multinode (whether or not it was actually multinode) | 21:15 |
| clarkb | then the testing consistently used the virtual network across the baord | 21:15 |
| clarkb | could do similar here | 21:16 |
| *** jamesmcarthur_ has joined #openstack-infra | 21:16 | |
| clarkb | or use the scope local addr (though I bet routing gets weird with that) | 21:16 |
| TheJulia | We do the same actually for multinode. I'll ask the QA team because I'd like a simpler code path. :) | 21:16 |
| TheJulia | The downside is for baremetal, we need to fake out the end virtual machines to think they have global for $reasons | 21:17 |
| clarkb | fungi: in your fix for corvus remember the scoping is slightly weird because the scope you need to lookup is for a define which is instantiated multiple times | 21:17 |
| clarkb | fungi: I think you have to stick the name of the define in the lookup path somewhere | 21:18 |
| *** jamesmcarthur has quit IRC | 21:18 | |
| fungi | i'm about to push it as soon as i finish the commit message, and then you can tell me it's wrong ;) | 21:18 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Correctly scope SSL path lookup for website vhosts https://review.opendev.org/663774 | 21:20 |
| fungi | corvus: clarkb: ^ that's what i *think* we want | 21:20 |
| clarkb | fungi: ya, my only concern is that there are many website instantiations each with different values for those vars | 21:21 |
| clarkb | so I think we may have to lookup the specific instances of the defined type? | 21:22 |
| clarkb | cmurphy: ^ do you know | 21:22 |
| corvus | (restarted apache again) | 21:23 |
| fungi | oh, i see, as in need some way to ask the vhost resource what its parent instance of the webserver class is? | 21:23 |
| * cmurphy looks | 21:23 | |
| clarkb | fungi: ya | 21:23 |
| clarkb | or maybe that is automagic within the context of each website definition | 21:23 |
| clarkb | (and nothing extra is needed) | 21:23 |
| fungi | i guess this works for our "static" class because it only refers to one server | 21:23 |
| clarkb | https://puppet.com/docs/puppet/4.8/lang_defined_types.html#containment may be the answer | 21:25 |
| clarkb | (in that it is automagic) | 21:25 |
| *** mriedem is now known as mriedem_afk | 21:26 | |
| *** whoami-rajat has quit IRC | 21:27 | |
| *** weifan has quit IRC | 21:27 | |
| fungi | there are many things about puppet which seem magic to me, so i'll just not be surprised that it's not a problem | 21:28 |
| cmurphy | i'm not sure if puppet 4 can handle this better than puppet 3, what i think we should do is use this https://review.opendev.org/570824 | 21:28 |
| cmurphy | is it really not a problem? | 21:29 |
| *** markvoelker has joined #openstack-infra | 21:29 | |
| clarkb | cmurphy: we don't know yet. But also that change you linked won't fix this because those vars will still be outside of scope? or do you mean evaluate template first then pass to vhost content? | 21:29 |
| clarkb | ya that must be it, so basically pass a really large string in that has already been interpolated | 21:30 |
| *** weifan has joined #openstack-infra | 21:30 | |
| clarkb | I guess we can try fungi's change and see if it is automagic | 21:30 |
| clarkb | (our testing may even do this for us already?) | 21:30 |
| cmurphy | clarkb: yeah evaluate the template first which would have the variables set properly at that scope and then pass the result to httpd | 21:31 |
| corvus | i trust our testing on this as far as i can throw the cloud it runs on | 21:31 |
| cmurphy | i don't think we trust the ssl paths very well | 21:31 |
| cmurphy | i sort of doubt that there is automagic that works for this | 21:31 |
| fungi | we do it this way for the vhosts in openstack_project::static but we only instantiate that class once | 21:33 |
| fungi | and it provides multiple vhost resources | 21:33 |
| cmurphy | fungi: that is because it's a class, you can only instantiate it once | 21:33 |
| cmurphy | the openstack_project::website is a defined type | 21:33 |
| cmurphy | is the problem | 21:33 |
| cmurphy | it doesn't have the same place in the hierarchy as a class does | 21:33 |
| fungi | oh, wow, i completely missed that it's a define not a class | 21:34 |
| fungi | can we define these at the node level? | 21:35 |
| clarkb | fwiw I have an alternate change in the works | 21:35 |
| *** iokiwi has joined #openstack-infra | 21:36 | |
| *** jamesmcarthur_ has quit IRC | 21:36 | |
| *** jamesmcarthur has joined #openstack-infra | 21:37 | |
| fungi | though i guess it would be a lot of repeated conditional logic in the node definition if we did it thatway | 21:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Evaluate files website vhosts in context of website not vhost https://review.opendev.org/663777 | 21:37 |
| clarkb | something like that (I think I got the mapping of the var names correct too but double check me on that) | 21:37 |
| clarkb | cmurphy: ^ | 21:38 |
| cmurphy | lgtm | 21:40 |
| corvus | whoah. | 21:42 |
| corvus | would have taken me the rest of the day to get to that point. :) | 21:42 |
| *** kjackal has quit IRC | 21:42 | |
| clarkb | I guess I can double check the @name value == @srvname value via the logs path | 21:43 |
| fungi | that's... deceptively simple | 21:43 |
| * clarkb does this | 21:43 | |
| clarkb | ya they seem to line up | 21:43 |
| *** jamesmcarthur has quit IRC | 21:46 | |
| corvus | i'll direct-enqueue that | 21:47 |
| *** aakarsh has quit IRC | 21:47 | |
| corvus | cmurphy, clarkb, fungi: thank you! | 21:48 |
| fungi | i'll abandon mine | 21:49 |
| clarkb | this is one of those really weird corner cases in puppet | 21:51 |
| clarkb | I assume the role path thing we ran into with ansible 2.8 is similar | 21:51 |
| clarkb | in theory the tool knows the data you want is right there | 21:51 |
| corvus | (restarted apache) | 21:51 |
| clarkb | you just can't get it | 21:51 |
| *** cloudnull has quit IRC | 22:00 | |
| *** d34dh0r53 has quit IRC | 22:01 | |
| ianw | frickler: thanks for pointing out https://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.8.html#imports-as-handlers ... i'm certain that has something to do with it ... | 22:01 |
| ianw | i tried "listen:" which seemed like the logical thing to do https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/letsencrypt-create-certs/handlers/main.yaml#L6 | 22:02 |
| ianw | i'll go back to that, maybe it's fixed | 22:02 |
| *** markvoelker has quit IRC | 22:03 | |
| clarkb | ianw: yes should be fixed (though the child chagnes may need a rebase) | 22:03 |
| fungi | afs02.dfw did come back up and mount /vicepa successfully, btw | 22:04 |
| fungi | i just checked in on it | 22:04 |
| corvus | fungi: yay! | 22:06 |
| fungi | vos listvldb is reporting "Old release" on a bunch of its volume copies | 22:06 |
| fungi | i assume as each of those gets a fresh vos release it'll do a full release for afs02.dfw? | 22:07 |
| corvus | yeah it should; we can probably leave it to its own devices and just check back tomorrow | 22:07 |
| corvus | (unless we're really antsy, in which case we should take the cron job out of service and run releases manually) | 22:08 |
| fungi | i am not in the least bit antsy | 22:08 |
| fungi | though my yard is increasingly so | 22:08 |
| fungi | ants are on a rampage this year | 22:09 |
| * corvus suddenly feels itchy | 22:09 | |
| clarkb | http://logs.openstack.org/77/663777/1/check/infra-puppet-apply-4-ubuntu-xenial/aa31350/applytest/puppetapplytest24.final.out.FAILED is alias a special word in ruby? | 22:10 |
| ianw | clarkb: oh, awesome, sorry missed your updates :) | 22:11 |
| cmurphy | clarkb: til yes https://blog.bigbinary.com/2012/01/08/alias-vs-alias-method.html | 22:11 |
| corvus | the GOOD news is the test caught it! :) | 22:12 |
| fungi | alias_ ftw | 22:12 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to ansible-lint 4.1.0 https://review.opendev.org/663501 | 22:12 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Evaluate files website vhosts in context of website not vhost https://review.opendev.org/663777 | 22:12 |
| clarkb | cmurphy: cool I just changed it to alias_name | 22:12 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Update to ansible-lint 4.1.0 https://review.opendev.org/663501 | 22:13 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Ansible roles for backup https://review.opendev.org/662657 | 22:13 |
| corvus | re-enqueued | 22:14 |
| ianw | assuming CI is happy with the new stuff, ^ should be ready for review too | 22:14 |
| clarkb | ianw: no worries the path thing on the include_tasks is weird | 22:14 |
| clarkb | but seems happy with using the longer path so good enoguh rpobably | 22:14 |
| ianw | i can retry the listen: thing too ... it's the more logical thing to register a handler but i have no desire to dive deep into how listen & include_tasks etc interact in the ansible code base :) | 22:16 |
| *** aakarsh has joined #openstack-infra | 22:19 | |
| *** weifan has quit IRC | 22:22 | |
| corvus | (restarted apache) | 22:22 |
| *** rcernin has joined #openstack-infra | 22:23 | |
| *** weifan has joined #openstack-infra | 22:23 | |
| *** rcernin has quit IRC | 22:23 | |
| *** rcernin has joined #openstack-infra | 22:24 | |
| clarkb | zuul says 13 minutes to merging the fix | 22:24 |
| *** jcoufal has joined #openstack-infra | 22:25 | |
| *** rcernin has quit IRC | 22:26 | |
| *** rfarr_ has quit IRC | 22:27 | |
| fungi | old release copies on afs02 have fallen from 14 to 10 since i last checked | 22:27 |
| fungi | so it does appear to be catching up | 22:27 |
| *** slaweq has quit IRC | 22:27 | |
| *** cloudnull has joined #openstack-infra | 22:27 | |
| *** rcernin has joined #openstack-infra | 22:28 | |
| *** d34dh0r53 has joined #openstack-infra | 22:28 | |
| clarkb | nodepool builders are continuing to successfully build images too | 22:29 |
| openstackgerrit | Merged opendev/system-config master: Evaluate files website vhosts in context of website not vhost https://review.opendev.org/663777 | 22:36 |
| corvus | \o/ | 22:38 |
| *** yamamoto has joined #openstack-infra | 22:47 | |
| ianw | fungi: do the fanatical support messages line up at all with any of the afs errors we saw on the opendev.org mirror? | 22:51 |
| fungi | nope | 22:52 |
| fungi | they were about a volume for afs02 which the mirrors aren't using | 22:52 |
| corvus | i restarted apache again. this is why i think there's something weird about the ansipuppet loop that sometimes requires 2 passes for a change to go into effect. | 22:53 |
| *** tkajinam has joined #openstack-infra | 22:53 | |
| *** slaweq has joined #openstack-infra | 22:55 | |
| *** jcoufal has quit IRC | 22:55 | |
| clarkb | corvus: we only update it at the very start of the cron | 22:56 |
| clarkb | so if you merge after that then you have that pass then the next before it applies | 22:56 |
| corvus | clarkb: ah, looks like we're on half-hour pulses (2230-2253 was most recent) | 22:58 |
| corvus | so that jives | 22:59 |
| *** markvoelker has joined #openstack-infra | 22:59 | |
| corvus | so this should finally go onto the fs around 23:22 | 23:00 |
| fungi | so we have a pulse starting now which should hopefully apply it | 23:00 |
| *** slaweq has quit IRC | 23:08 | |
| *** slaweq has joined #openstack-infra | 23:11 | |
| *** dpawlik has joined #openstack-infra | 23:11 | |
| *** jamesmcarthur has joined #openstack-infra | 23:13 | |
| *** dpawlik has quit IRC | 23:15 | |
| *** jamesmcarthur has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-infra | 23:17 | |
| corvus | broken; fixing | 23:22 |
| corvus | okay, so i guess that didn't work? | 23:23 |
| corvus | result was the same -- same sites had nothing in the ssl cert path sections | 23:23 |
| clarkb | hrm | 23:23 |
| clarkb | the three variable names seem correct | 23:24 |
| *** slaweq has quit IRC | 23:24 | |
| corvus | #status log added files02.openstack.org to emergency file due to recent system-config changes breaking apache config | 23:24 |
| clarkb | cmurphy: ^ any other ideas? | 23:25 |
| openstackstatus | corvus: finished logging | 23:25 |
| corvus | i can undertake to set up a more complete test and fix tomorrow | 23:26 |
| *** dpawlik has joined #openstack-infra | 23:27 | |
| clarkb | oh I see it ugh | 23:27 |
| clarkb | maybe | 23:27 |
| clarkb | its an order of operations thing I think | 23:27 |
| clarkb | change in a sec | 23:27 |
| *** dchen has joined #openstack-infra | 23:29 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Evaluate files vhosts after we determine ssl file paths https://review.opendev.org/663796 | 23:29 |
| clarkb | that | 23:29 |
| corvus | does that really change the order? | 23:30 |
| corvus | i have no idea when order does or does not matter in puppet | 23:31 |
| clarkb | in modern puppet it evaluates files top to bottom because people found the other behavior too confusing | 23:31 |
| corvus | but i guess that's about a sequence of steps to create a graph, and the graph itself may be unordered? | 23:31 |
| clarkb | so I'm fairly certain now that we are in puppet 4 the order matters there | 23:31 |
| corvus | oh, ok, so it's just ordered :) | 23:31 |
| *** markvoelker has quit IRC | 23:31 | |
| *** dpawlik has quit IRC | 23:31 | |
| corvus | if that lands, i can mind it tomorrow | 23:38 |
| cmurphy | clarkb: ah yeah the ordering of variables matters in puppet 3 and puppet 4 | 23:39 |
| *** yamamoto has quit IRC | 23:41 | |
| *** yamamoto has joined #openstack-infra | 23:41 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** lseki has quit IRC | 23:42 | |
| *** jamesmcarthur has joined #openstack-infra | 23:45 | |
| clarkb | ianw: https://review.opendev.org/663796 care to get that one in so it is ready for corvus tomorrow? | 23:46 |
| ianw | lgtm ... yeah ansible got it right just being (mostly) a glorified shell script | 23:47 |
| clarkb | ianw: I rechecked the ansible lint change as it failed on beaker | 23:48 |
| ianw | btw on that ansible-lint change, there was a response to https://github.com/ansible/ansible-lint/issues/534 | 23:48 |
| clarkb | and well those two things shouldn't be related | 23:48 |
| *** weifan has quit IRC | 23:48 | |
| *** hwoarang has quit IRC | 23:48 | |
| ianw | having tried doing similar things with bashate, i can somewhat feel the pain | 23:48 |
| *** weifan has joined #openstack-infra | 23:49 | |
| *** hwoarang has joined #openstack-infra | 23:50 | |
| clarkb | still don't have new control plane images with the ext4 journal fix. Hopefully we'll have one tomorrow and I can start poking at a replacement gitea06 | 23:50 |
| *** weifan has quit IRC | 23:50 | |
| *** weifan has joined #openstack-infra | 23:51 | |
| *** weifan has quit IRC | 23:51 | |
| *** aaronsheffield has quit IRC | 23:51 | |
| clarkb | basically every image has to be rebuilt now that builders are happier | 23:51 |
| clarkb | and the control plane image is stuck in the queue | 23:51 |
| *** weifan has joined #openstack-infra | 23:53 | |
| *** weifan has quit IRC | 23:53 | |
| *** weifan has joined #openstack-infra | 23:54 | |
| *** mriedem_afk has quit IRC | 23:54 | |
| *** rcernin has quit IRC | 23:55 | |
| *** rcernin has joined #openstack-infra | 23:56 | |
| *** jamesmcarthur has quit IRC | 23:57 | |
| *** weifan has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!