| openstackgerrit | Kendall Nelson proposed opendev/storyboard master: Add StoryBoard Story Tags to Docs https://review.opendev.org/667482 | 00:00 |
|---|---|---|
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** ianychoi has quit IRC | 00:12 | |
| *** jklare has quit IRC | 00:14 | |
| *** ianychoi has joined #openstack-infra | 00:14 | |
| *** jklare has joined #openstack-infra | 00:15 | |
| *** slaweq has quit IRC | 00:15 | |
| *** ianychoi has quit IRC | 00:20 | |
| *** ianychoi has joined #openstack-infra | 00:21 | |
| *** hamzy has joined #openstack-infra | 00:27 | |
| *** ianychoi has quit IRC | 00:30 | |
| *** ianychoi has joined #openstack-infra | 00:31 | |
| ianw | fungi: i get the feeling that iad.rax ipv6 ticket is now going to start going in loops ... | 00:33 |
| fungi | maybe best we move the server to a new port and take the hit on a new ip address? | 00:34 |
| ianw | yeah, just with a pattern of behaviour emerging i thought rax might want to really root cause it ... then again turn it off and on again is tried and true | 00:37 |
| *** gregoryo has joined #openstack-infra | 00:38 | |
| *** sthussey has joined #openstack-infra | 00:41 | |
| openstackgerrit | Merged openstack/diskimage-builder master: Move existing Zuul project config to projects.yaml https://review.opendev.org/667220 | 00:45 |
| *** smarcet has joined #openstack-infra | 00:48 | |
| *** uberjay has quit IRC | 01:03 | |
| *** ianychoi has quit IRC | 01:03 | |
| *** ianychoi has joined #openstack-infra | 01:05 | |
| *** uberjay has joined #openstack-infra | 01:20 | |
| *** mgoddard has quit IRC | 01:40 | |
| *** smarcet has quit IRC | 01:46 | |
| *** smarcet has joined #openstack-infra | 01:47 | |
| *** mgoddard has joined #openstack-infra | 01:48 | |
| *** noama has quit IRC | 01:49 | |
| *** apetrich has quit IRC | 01:57 | |
| *** diablo_rojo has quit IRC | 01:57 | |
| *** igordc has quit IRC | 01:58 | |
| *** smarcet has left #openstack-infra | 02:03 | |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** ianychoi has quit IRC | 02:14 | |
| *** ianychoi has joined #openstack-infra | 02:16 | |
| *** slaweq has quit IRC | 02:16 | |
| *** smarcet has joined #openstack-infra | 02:25 | |
| *** smarcet has quit IRC | 02:29 | |
| *** smarcet has joined #openstack-infra | 02:32 | |
| *** smarcet has left #openstack-infra | 02:32 | |
| *** ianychoi has quit IRC | 02:38 | |
| *** ianychoi has joined #openstack-infra | 02:40 | |
| *** hongbin has joined #openstack-infra | 02:44 | |
| ianw | corvus: there's a volume in vexxhost/ymq -> kubernetes-dynamic-pvc-1adf74dd-71bd-11e9-8638-fa163ed67b15 not attached to anything. can we remove it? | 02:45 |
| ianw | corvus: there is also one attached to jeblair test, not sure if this host is still needed | 02:49 |
| ianw | (i mention this because we're at quota limits :) | 02:49 |
| *** sthussey has quit IRC | 02:51 | |
| *** goldyfruit has quit IRC | 02:52 | |
| *** ricolin has joined #openstack-infra | 03:01 | |
| *** xinranwang has joined #openstack-infra | 03:02 | |
| *** whoami-rajat has joined #openstack-infra | 03:04 | |
| clarkb | ianw I just deleted jeblairtest did it not delete the volume too? | 03:05 |
| clarkb | oh wait that was sjc1 | 03:05 |
| clarkb | sorry | 03:05 |
| ianw | so much testing :) | 03:07 |
| *** hongbin has quit IRC | 03:18 | |
| *** ykarel|afk has joined #openstack-infra | 03:19 | |
| corvus | ianw: yeah, if it's not attached that should be fine | 03:20 |
| ianw | corvus: thanks; and do you still want your ymq test host? | 03:24 |
| corvus | nope, i'm all done with test hosts | 03:24 |
| ianw | thanks, will clean up both in a bit | 03:25 |
| corvus | thanks, and sorry about the mess | 03:25 |
| ianw | when i figure out why launching a node in ovh/gra thinks it has a public ipv6 but doesn't | 03:25 |
| *** bhavikdbavishi has joined #openstack-infra | 03:34 | |
| *** psachin has joined #openstack-infra | 03:35 | |
| clarkb | ianw thats because neutron reports ipv6 but its not in cofnig drive and no TAs | 03:36 |
| clarkb | er RAs | 03:36 |
| *** ramishra has joined #openstack-infra | 03:36 | |
| ianw | yes, i just realised the api is giving an address but it's not in config drive :) | 03:36 |
| clarkb | so you ha e to set the cloud config option to force ipv4 | 03:37 |
| clarkb | nodepool does this | 03:37 |
| ianw | this is for launching a node ... where it tries to ping v6 when it thinks it has an address ... perhaps the last time we did this, we just hacked around that and moved on | 03:38 |
| ianw | ... which is what i'll probably do now :) | 03:38 |
| *** bhavikdbavishi has quit IRC | 03:38 | |
| clarkb | oh hrm | 03:39 |
| *** bhavikdbavishi has joined #openstack-infra | 03:46 | |
| *** ykarel|afk is now known as ykarel | 03:48 | |
| *** udesale has joined #openstack-infra | 03:51 | |
| *** raukadah is now known as chandankumar | 03:52 | |
| *** hongbin has joined #openstack-infra | 03:54 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** hongbin has quit IRC | 04:12 | |
| *** slaweq has quit IRC | 04:16 | |
| *** viks___ has joined #openstack-infra | 04:17 | |
| *** jhesketh has quit IRC | 04:19 | |
| *** jhesketh has joined #openstack-infra | 04:19 | |
| *** ykarel is now known as ykarel|afk | 04:51 | |
| *** ykarel|afk has quit IRC | 04:57 | |
| *** ykarel|afk has joined #openstack-infra | 05:14 | |
| *** e0ne has joined #openstack-infra | 05:19 | |
| *** e0ne has quit IRC | 05:23 | |
| *** ykarel|afk is now known as ykarel | 05:25 | |
| AJaeger | cmurphy, dirk , could you look at glean openSUSE jobs, please? See discussion from last night that lead to https://review.opendev.org/667459 | 05:50 |
| *** xinranwang has quit IRC | 05:52 | |
| *** mgoddard has quit IRC | 05:54 | |
| *** lpetrut has joined #openstack-infra | 05:56 | |
| *** lpetrut has quit IRC | 05:57 | |
| *** lpetrut has joined #openstack-infra | 05:57 | |
| *** dpawlik has joined #openstack-infra | 06:05 | |
| *** slaweq has joined #openstack-infra | 06:11 | |
| *** jtomasek has joined #openstack-infra | 06:14 | |
| *** yboaron_ has joined #openstack-infra | 06:16 | |
| *** slaweq has quit IRC | 06:16 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml https://review.opendev.org/667516 | 06:17 |
| *** Adri2000 has quit IRC | 06:17 | |
| *** kjackal has joined #openstack-infra | 06:25 | |
| *** pcaruana has joined #openstack-infra | 06:26 | |
| *** pgaxatte has joined #openstack-infra | 06:30 | |
| *** apetrich has joined #openstack-infra | 06:31 | |
| *** ianychoi has quit IRC | 06:34 | |
| AJaeger | ianw: do we need an updated version of https://review.opendev.org/#/c/550582/ for ARM64 wheels? | 06:37 |
| AJaeger | pabelanger: I saw you working on those as well ^ | 06:40 |
| *** ianychoi has joined #openstack-infra | 06:41 | |
| *** kopecmartin has joined #openstack-infra | 06:43 | |
| *** slaweq has joined #openstack-infra | 06:44 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove release-openstack-python-without-pypi https://review.opendev.org/531829 | 06:54 |
| ianw | AJaeger: oh geez, i'd have to switch that all back in with our current arm state :/ | 06:58 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove nodepool functional job attachments https://review.opendev.org/667213 | 06:58 |
| AJaeger | ianw: ;( | 06:59 |
| AJaeger | ianw: what do you think about https://review.opendev.org/#/c/631049/ ? | 07:00 |
| dirk | AJaeger, so if I understand correctly this is about 42.3 failing to build related to systems logger | 07:05 |
| AJaeger | dirk: that's what I got from the scrollback as well... | 07:05 |
| dirk | I guess I would question why were voting on 42.3, it is dead | 07:05 |
| AJaeger | so, update to 15? | 07:06 |
| dirk | The jobs should check for 15.0 and 15.1 only | 07:06 |
| *** rcernin has quit IRC | 07:06 | |
| dirk | Yeah, need to figure out where it pulls that information from | 07:06 |
| *** virendra-sharma has joined #openstack-infra | 07:11 | |
| dirk | Ok, found it | 07:11 |
| dirk | I have a patch with a revert. | 07:12 |
| openstackgerrit | Dirk Mueller proposed openstack/project-config master: Revert "Make glean opensuse job non-voting" https://review.opendev.org/667527 | 07:13 |
| *** ginopc has joined #openstack-infra | 07:15 | |
| *** tesseract has joined #openstack-infra | 07:16 | |
| *** tosky has joined #openstack-infra | 07:16 | |
| AJaeger | dirk, please propose for glean repo instead - corvus is moving everything there... | 07:18 |
| AJaeger | dirk, see also https://review.opendev.org/#/c/667213/ | 07:18 |
| AJaeger | dirk, also https://review.opendev.org/#/c/610477/ should be proposed for nodepool repo now | 07:19 |
| dirk | AJaeger, so how do I get it back to voting? The moved job is no voting as well | 07:20 |
| dirk | AJaeger, no review in over half a year.. | 07:21 |
| openstackgerrit | Merged openstack/project-config master: Normalize projects.yaml https://review.opendev.org/667516 | 07:25 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add OVH GRA1 opendev.org mirror https://review.opendev.org/667529 | 07:28 |
| openstackgerrit | Dirk Mueller proposed zuul/nodepool master: Switch to openSUSE 15.1 https://review.opendev.org/667532 | 07:33 |
| *** ccamacho has joined #openstack-infra | 07:37 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Switch to opensuse-15 nodeset for bindep testing https://review.opendev.org/667533 | 07:40 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 07:43 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 07:43 |
| *** rajinir has quit IRC | 07:45 | |
| *** e0ne has joined #openstack-infra | 07:47 | |
| yoctozepto | hey infra, thanks for sorting out those issues with CI :-) | 07:48 |
| *** ralonsoh has joined #openstack-infra | 07:49 | |
| yoctozepto | I have a question regarding test node specs; is there a place I could check the specs of centos-7 vs ubuntu-bionic? | 07:50 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:50 |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Enable nodepool testing for opensuse 15.x https://review.opendev.org/667537 | 07:51 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 07:53 |
| *** yboaron_ has quit IRC | 07:53 | |
| ianw | yoctozepto: the specs are all the same ... they don't run on different flavors or anything. 8 core 8gb basically | 07:55 |
| *** psachin has quit IRC | 07:55 | |
| ianw | cgoncalves has identified that centos-7 is a lot slower at times, we haven't figured out why though | 07:55 |
| cgoncalves | if only at times... :) | 07:56 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-jobs master: install-nodejs: add support for RPM-based OSes https://review.opendev.org/631049 | 07:57 |
| cgoncalves | ianw, I think we reached the point where we ran out of ideas. johnsom also threw a couple of patches to try other things. none helped | 07:58 |
| *** ianychoi has quit IRC | 07:59 | |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 08:00 |
| *** ianychoi has joined #openstack-infra | 08:01 | |
| *** yboaron_ has joined #openstack-infra | 08:02 | |
| *** jpich has joined #openstack-infra | 08:03 | |
| *** mgoddard has joined #openstack-infra | 08:05 | |
| *** lucasagomes has joined #openstack-infra | 08:11 | |
| *** dchen has quit IRC | 08:14 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add OVH GRA1 mirror https://review.opendev.org/667543 | 08:15 |
| *** tkajinam has quit IRC | 08:16 | |
| *** pkopec has joined #openstack-infra | 08:19 | |
| *** gregoryo has quit IRC | 08:21 | |
| *** yboaron_ has quit IRC | 08:22 | |
| ianw | corvus / mnaser : 4aa57cc6-8710-4952-87e6-ddad1db2bb12 (jeblairtest) in ymq-1 seems like it will not die ... despite my delete attempt it's still showing as running, but no ping or console or anything else useful. | 08:24 |
| ianw | i did manage to clear out the unused kube volume, though (08bd4bb9-b45f-48a4-b5f8-a901625f218f) | 08:25 |
| *** ykarel is now known as ykarel|lunch | 08:28 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: launch-node.py : add option to skip ipv6 address checks https://review.opendev.org/667548 | 08:29 |
| *** dtantsur|afk is now known as dtantsur|mtg | 08:37 | |
| *** tesseract has quit IRC | 08:38 | |
| *** tesseract has joined #openstack-infra | 08:40 | |
| *** rpittau|afk is now known as rpittau|mtg | 08:41 | |
| rpittau|mtg | good morning folks! Quick question if anyone is available, we're seeing failures in tempest jobs, are you aware of any recent change on your side that could have impact on that ? | 08:42 |
| *** imacdonn has quit IRC | 08:42 | |
| *** imacdonn has joined #openstack-infra | 08:43 | |
| *** rcernin has joined #openstack-infra | 08:46 | |
| tosky | rpittau|mtg: legacy tempest jobs (non zuulv3)? This happened two days ago: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007313.html | 08:46 |
| *** ociuhandu has joined #openstack-infra | 08:47 | |
| dtantsur|mtg | tosky: no, not legacy | 08:48 |
| tosky | oook | 08:48 |
| dtantsur|mtg | tosky: rings any bells? http://logs.openstack.org/31/667531/1/check/python-ironic-inspector-client-tempest-python2/604d2d4/job-output.txt.gz#_2019-06-26_08_17_37_653753 | 08:48 |
| dtantsur|mtg | it seems that zuul is cloning that thing | 08:48 |
| dtantsur|mtg | maybe ansible_user_dir stopped being the correct path? | 08:49 |
| dtantsur|mtg | yeah, it's cloned here http://logs.openstack.org/31/667531/1/check/python-ironic-inspector-client-tempest-python2/604d2d4/job-output.txt.gz#_2019-06-26_07_40_08_955720 | 08:53 |
| dtantsur|mtg | mmm, maybe quotation marks? | 08:53 |
| tosky | I don't really know, and I don't see any relevant change in tempest and devstack either | 08:58 |
| rpittau|mtg | thanks tosky , we probably found the issue | 09:02 |
| tosky | rpittau|mtg: uh, where? Now I'm curious :) | 09:03 |
| rpittau|mtg | quotation marks :) this should fix the issue: https://review.opendev.org/667555 | 09:04 |
| *** ociuhandu has quit IRC | 09:04 | |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Fix parsing on egg names with dashes from git URLs https://review.opendev.org/648727 | 09:04 |
| tosky | rpittau|mtg: I'd suggest to also move away from TEMPEST_PLUGINS - there is a native zuul variable | 09:05 |
| tosky | rpittau|mtg: https://review.opendev.org/#/c/639374/ | 09:05 |
| rpittau|mtg | tosky: thanks, can you point to any docs on that or example ? | 09:05 |
| rpittau|mtg | thanks :) | 09:05 |
| *** ianychoi has quit IRC | 09:07 | |
| *** rcernin has quit IRC | 09:07 | |
| *** ianychoi has joined #openstack-infra | 09:10 | |
| *** ykarel|lunch is now known as ykarel | 09:19 | |
| *** jaosorior has quit IRC | 09:22 | |
| *** jaosorior has joined #openstack-infra | 09:24 | |
| *** gfidente has joined #openstack-infra | 09:30 | |
| *** Lucas_Gray has joined #openstack-infra | 09:35 | |
| yoctozepto | <ianw> cgoncalves has identified that centos-7 is a lot slower at times, we haven't figured out why though | 09:38 |
| yoctozepto | my issue exactly ;-) | 09:38 |
| *** psachin has joined #openstack-infra | 09:39 | |
| cgoncalves | yoctozepto, where is it slow for you? | 09:39 |
| yoctozepto | kolla jobs timeout on centos-7 | 09:39 |
| yoctozepto | especially after we got update from horizon making us compile many more messages (django) | 09:40 |
| yoctozepto | ubuntu almost no hit | 09:40 |
| yoctozepto | centos-7 killed | 09:40 |
| cgoncalves | yoctozepto, I don't know those jobs. Is it that centos is slower to boot? | 09:41 |
| yoctozepto | cgoncalves: to operate | 09:41 |
| yoctozepto | the same actions can take literally 3x time | 09:41 |
| yoctozepto | sometimes less, sometimes more | 09:41 |
| virendra-sharma | can anyone update on ci-watch ? unable to see latest result on ci-watch (http://ciwatch.mmedvede.net/project?project=cinder&time=7+days) | 09:43 |
| cgoncalves | yoctozepto, on that kolla job, is centos the host operating system? or something different like ubuntu on host and centos on containers? | 09:44 |
| yoctozepto | cgoncalves: we had centos+centos and ubuntu+ubuntu, I am trying out ubuntu(host)+centos(containers) but it seems they are slow the same :D | 09:46 |
| yoctozepto | or maybe not | 09:47 |
| yoctozepto | https://review.opendev.org/667518 | 09:47 |
| yoctozepto | https://review.opendev.org/667519 | 09:47 |
| AJaeger | virendra-sharma: we do not maintain ciwatch, no idea who owns it | 09:48 |
| *** Lucas_Gray has quit IRC | 09:48 | |
| *** Lucas_Gray has joined #openstack-infra | 09:49 | |
| AJaeger | virendra-sharma: I just saw it points to us here. infra-root, any idea about ciwatch at http://ciwatch.mmedvede.net ? | 09:50 |
| AJaeger | mmedvede: do you run that ? ^ | 09:50 |
| cgoncalves | yoctozepto, ok. if centos is slower than ubuntu on the host, the symptoms are different from what we see in Octavia but the root cause could be the same. | 09:51 |
| *** bhavikdbavishi has quit IRC | 09:52 | |
| cgoncalves | we observed that, on a local test, centos cloud image performs fine but the one diskimage-builder produces is slower | 09:52 |
| yoctozepto | cgoncalves: no, it did not help much | 09:55 |
| *** virendra-sharma9 has joined #openstack-infra | 09:55 | |
| *** xek has joined #openstack-infra | 09:55 | |
| yoctozepto | this job used ubuntu as host and centos in containers to build | 09:55 |
| yoctozepto | and it was still noticeably slower | 09:56 |
| yoctozepto | kolla-build-centos-sourceSUCCESS in 1h 39m 02s | 09:56 |
| yoctozepto | kolla-build-ubuntu-sourceSUCCESS in 47m 23s | 09:56 |
| yoctozepto | kolla-build-centos-source fails us recently | 09:56 |
| yoctozepto | ;-( | 09:56 |
| yoctozepto | the odd thing seems the most impactful is horizon with gettext invocations | 09:58 |
| cgoncalves | yoctozepto, uh. I'm afraid I don't know what's going on there either :/ | 09:58 |
| yoctozepto | as if centos was suboptimal on this hw | 09:58 |
| virendra-sharma9 | AJaeger, I am not sure owner of ci-watch, but from last two week unable to see consolidated result of all latest patch with voting of vendors. if could get any update here then it will be healpful | 09:58 |
| yoctozepto | cgoncalves: it's ok, let's keep this in mind though if someone wise enough steps upon this holy channel | 09:58 |
| yoctozepto | and blesses us with the wisdom | 09:58 |
| yoctozepto | ;D | 09:59 |
| *** virendra-sharma has quit IRC | 09:59 | |
| cgoncalves | amen! | 09:59 |
| yoctozepto | amen! | 09:59 |
| *** yboaron_ has joined #openstack-infra | 10:00 | |
| yoctozepto | alas, it looks as if our favorite mirror still has its personal issues: 2019-06-26 09:46:03.784177 | primary | INFO:kolla.common.utils.designate-worker:http://mirror.iad.rax.opendev.org/centos/7/os/x86_64/repodata/repomd.xml: [Errno 14] curl#7 - "Failed connect to mirror.iad.rax.opendev.org:80; No route to host" | 10:00 |
| yoctozepto | if woken up at midnight, asked who failed you the most, I will recite mirror.iad.rax.opendev.org | 10:01 |
| *** tdasilva_ has joined #openstack-infra | 10:02 | |
| jrosser | yoctozepto: i believe that we see the same thing in openstack-ansible too, that the centos tests are just somehow systematically slower than the same thing on other OS | 10:02 |
| *** tdasilva has quit IRC | 10:03 | |
| yoctozepto | jrosser: thanks, at least we are not alone | 10:04 |
| jrosser | yoctozepto: we did find this https://github.com/openstack/openstack-ansible/blob/d178cbccc8339f23367631725a0862e583cf3ab4/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L219-L223 | 10:04 |
| yoctozepto | some ubuntu marketing guys should come up with "with no effort on your side we give you 30% more performance on the same hardware" | 10:04 |
| jrosser | but i don't think that is the whole story somehow | 10:04 |
| yoctozepto | jrosser: probably not because we switched the kernel to ubuntu there | 10:05 |
| yoctozepto | and it did not help | 10:05 |
| yoctozepto | as if the centos libs and utils were unoptimized | 10:05 |
| yoctozepto | strange | 10:05 |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: Enable nodepool testing for opensuse 15.x https://review.opendev.org/667537 | 10:07 |
| *** tdasilva_ has quit IRC | 10:22 | |
| *** virendra-sharma9 has quit IRC | 10:25 | |
| *** virendra-sharma has joined #openstack-infra | 10:26 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Switch to opensuse-15 nodeset for bindep testing https://review.opendev.org/667533 | 10:28 |
| *** dpawlik has quit IRC | 10:37 | |
| *** dpawlik has joined #openstack-infra | 10:38 | |
| *** dpawlik has quit IRC | 10:42 | |
| *** dpawlik has joined #openstack-infra | 10:45 | |
| rpittau|mtg | tosky: sorry to bug you again, the change is unfortunately not working for us, the path is still wrapped in multiple quotation marks http://logs.openstack.org/55/667555/2/check/ironic-tempest-functional-python2/6ba2f9f/job-output.txt.gz#_2019-06-26_09_36_21_817124 | 10:46 |
| *** dpawlik has quit IRC | 10:50 | |
| *** dpawlik has joined #openstack-infra | 10:53 | |
| *** bhavikdbavishi has joined #openstack-infra | 10:59 | |
| *** dpawlik has quit IRC | 11:00 | |
| *** iurygregory has joined #openstack-infra | 11:00 | |
| *** dpawlik has joined #openstack-infra | 11:01 | |
| tosky | rpittau|mtg: than there is something specific in your job | 11:06 |
| tosky | rpittau|mtg, dtantsur|mtg you should change the other job | 11:08 |
| tosky | uh, there is a problem with rax mirrors | 11:10 |
| rpittau|mtg | tosky: thanks, yeah, just noticed :/ | 11:11 |
| *** tosky has quit IRC | 11:15 | |
| *** tosky has joined #openstack-infra | 11:16 | |
| *** shachar has quit IRC | 11:18 | |
| tosky | so, regarding the rax mirrors, it's also about the normal mirrors: | 11:20 |
| tosky | http://logs.openstack.org/90/638390/35/check/sahara-grenade/275261c/job-output.txt.gz#_2019-06-26_09_34_04_040694 | 11:20 |
| tosky | http://logs.openstack.org/90/638390/35/check/sahara-grenade/171cd45/job-output.txt.gz#_2019-06-26_08_49_55_436255 | 11:21 |
| *** ociuhandu has quit IRC | 11:23 | |
| *** ociuhandu has joined #openstack-infra | 11:23 | |
| *** happyhemant has joined #openstack-infra | 11:28 | |
| ianw | infra-root: rax.iad.opendev.org mirror went into "shutoff" state | 11:36 |
| ianw | i was capturing netconsole ... nothing appeared :/ i have rebooted it now, and reset the console | 11:38 |
| ianw | have to think about it in the morning. i don't really trust this host ... given the ipv6 issues on the interface too. rebuilding it might not be a bad idea | 11:39 |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Switch to opensuse-15 nodeset for bindep testing https://review.opendev.org/667533 | 11:40 |
| *** ianychoi has quit IRC | 11:40 | |
| *** roman_g has joined #openstack-infra | 11:43 | |
| zbr|ruck | apparently doc8 needs some cores, i see plan reviews ignores for too long: example: https://review.opendev.org/#/c/649590/ -- i still see enough openstack projects using it, so we have an interest in maintaining it, right? Last merge was >2 years ago. | 11:44 |
| *** ianychoi has joined #openstack-infra | 11:45 | |
| *** ykarel is now known as ykarel|afk | 11:49 | |
| zbr|ruck | yeah, I think there are some mirror issues as I got failure to find "No package matching 'libssl-dev' is available" | 11:50 |
| zbr|ruck | this is really annoying because I would have preffered an error telling user that repository could not be accessed, not that is empty. | 11:51 |
| *** udesale has quit IRC | 11:51 | |
| *** udesale has joined #openstack-infra | 11:52 | |
| *** ykarel_ has joined #openstack-infra | 11:53 | |
| *** eharney has quit IRC | 11:55 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/627534 | 11:55 |
| *** jcoufal has joined #openstack-infra | 11:55 | |
| AJaeger | tobias-urdin: I've picked up the change, could you review whether that is fine - I'll split now in two changes so that we can merge without breakage | 11:55 |
| *** jcoufal has quit IRC | 11:55 | |
| *** ykarel|afk has quit IRC | 11:56 | |
| *** jcoufal has joined #openstack-infra | 11:56 | |
| yoctozepto | <ianw> have to think about it in the morning. i don't really trust this host ... given the ipv6 issues on the interface too. rebuilding it might not be a bad idea | 11:57 |
| yoctozepto | you have the whole kolla team on your side | 11:57 |
| *** rh-jelabarre has joined #openstack-infra | 11:58 | |
| *** _erlon_ has joined #openstack-infra | 11:59 | |
| *** goldyfruit has joined #openstack-infra | 11:59 | |
| *** ekultails has joined #openstack-infra | 12:01 | |
| *** ykarel_ has quit IRC | 12:04 | |
| *** goldyfruit has quit IRC | 12:05 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove bindep role for Puppet build job https://review.opendev.org/627554 | 12:06 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Install PDK dependencies for puppet building https://review.opendev.org/667589 | 12:06 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Prepare: Use PDK to build puppet module https://review.opendev.org/667590 | 12:07 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/667591 | 12:07 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Prepare: Use PDK to build puppet module https://review.opendev.org/667590 | 12:09 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/627534 | 12:09 |
| *** virendra-sharma has quit IRC | 12:10 | |
| *** tdasilva has joined #openstack-infra | 12:11 | |
| *** ianychoi has quit IRC | 12:14 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove bindep role for Puppet build job https://review.opendev.org/627566 | 12:15 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Install PDK dependencies for puppet building https://review.opendev.org/667595 | 12:15 |
| mnaser | o/ ianw | 12:16 |
| *** ianychoi has joined #openstack-infra | 12:22 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Prepare: Use PDK to build puppet module https://review.opendev.org/667590 | 12:22 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/627534 | 12:22 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Install PDK dependencies for puppet building https://review.opendev.org/667589 | 12:24 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove bindep role for Puppet build job https://review.opendev.org/627554 | 12:24 |
| *** rlandy has joined #openstack-infra | 12:25 | |
| *** mriedem has joined #openstack-infra | 12:25 | |
| *** aaronsheffield has joined #openstack-infra | 12:27 | |
| *** ykarel_ has joined #openstack-infra | 12:28 | |
| AJaeger | tobias-urdin: now updated everything, let's see whether tests are all fine. Is this what you had in mind? | 12:28 |
| tobias-urdin | AJaeger: thanks, i'll have a look | 12:29 |
| * AJaeger had to split them up so that we can merge step by step. | 12:30 | |
| AJaeger | tobias-urdin: do you have a way to test the job? | 12:31 |
| *** ykarel_ is now known as ykarel | 12:37 | |
| fungi | ianw: should we switch that region back to using the openstack.org mirror until we can rebuild the opendev one? | 12:37 |
| *** pkopec has quit IRC | 12:37 | |
| tobias-urdin | AJaeger: the code that was in my latest PS is tested | 12:38 |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Fix parsing on egg names with dashes from git URLs https://review.opendev.org/648727 | 12:38 |
| tobias-urdin | your patches seems fine, since it's just restructure it should work straight up on ubuntu 18.04 that was tested | 12:38 |
| AJaeger | tobias-urdin: glad to hear! | 12:38 |
| * AJaeger will push another update for zuul-jobs in 5 mins (after other tests finished) to add a Co-Authored-By for tobias-urdin ... | 12:39 | |
| AJaeger | Sorry for omitting this initially | 12:39 |
| *** dpawlik has quit IRC | 12:40 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Prepare: Use PDK to build puppet module https://review.opendev.org/667590 | 12:40 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Use PDK to build puppet module https://review.opendev.org/627534 | 12:40 |
| tobias-urdin | AJaeger: no worries, let me do another test to make sure | 12:40 |
| AJaeger | tobias-urdin: thanks - I'm done now ;) | 12:41 |
| *** pkopec has joined #openstack-infra | 12:51 | |
| *** dave-mccowan has joined #openstack-infra | 12:52 | |
| *** ykarel has quit IRC | 12:52 | |
| *** dpawlik has joined #openstack-infra | 12:55 | |
| tobias-urdin | AJaeger: yeah that code is fine when testing on ubuntu 18.04 | 12:57 |
| *** rfarr has joined #openstack-infra | 12:58 | |
| *** ykarel has joined #openstack-infra | 13:01 | |
| AJaeger | tobias-urdin: can you comment this in https://review.opendev.org/627534 , please? Thanks! | 13:03 |
| AJaeger | config-core, could you review the stack by tobias-urdin in https://review.opendev.org/#/q/topic:forge-publish . A small change but to be able to merge this without breakage, we need many small changes... | 13:03 |
| *** xek_ has joined #openstack-infra | 13:05 | |
| *** xek has quit IRC | 13:07 | |
| *** yamamoto has joined #openstack-infra | 13:11 | |
| tobias-urdin | AJaeger: done, posted what was used and output as well | 13:11 |
| AJaeger | thanks | 13:13 |
| roman_g | Hello team. IRC question. Our manuals say that someone first needs to register channel https://docs.openstack.org/infra/system-config/irc.html#access, but Freenode support says that openstack infra needs extend their registration to cover new channel. How is that done? | 13:13 |
| AJaeger | roman_g: what are you trying to do? | 13:13 |
| roman_g | Freenode support: we'll need them to send us an email to extend the namespace | 13:13 |
| roman_g | Trying to register a channel and move it to under openstack control. | 13:14 |
| AJaeger | roman_g: which channel? | 13:14 |
| roman_g | AJaeger: #airship. It used to be occupied, and now it is free to be registered and used. | 13:15 |
| *** eharney has joined #openstack-infra | 13:15 | |
| AJaeger | roman_g: yeah, we only maintain "openstack" namespace. maybe fungi, clarkb, or corvus can help further | 13:16 |
| fungi | roman_g: all you need is to a channel operator to enter the registration commands... what challenge are you running into? | 13:17 |
| roman_g | fungi: can't get OP, because there are people on the channel. Channel is not registered. | 13:17 |
| fungi | yeah, you need everyone to leave the channel first | 13:17 |
| fungi | is that the problem? | 13:18 |
| *** rajinir has joined #openstack-infra | 13:18 | |
| fungi | if everyone evacuates the channel, then the first person to re-enter it will automatically be a channel operator | 13:18 |
| *** rf0lc0 has joined #openstack-infra | 13:18 | |
| roman_g | Freenode staff says they can OP someone, but need e-mail from someone responsible for IRC from openstack that channel I mention above is openstack (-related) channel | 13:19 |
| *** sthussey has joined #openstack-infra | 13:19 | |
| fungi | if you can't get everyone to leave the channel first, then yeah we'll need to figure out how to engage freenode support to perform a takeover | 13:20 |
| fungi | i'll see if i have some notes on how we've authorized them to do that in the past | 13:21 |
| *** rfolco has quit IRC | 13:21 | |
| fungi | but probably won't be able to pick that up for about an hour | 13:21 |
| roman_g | no problem | 13:21 |
| roman_g | making people leave channel will anyhow take longer, as they are connected via irccloud (like forever) | 13:22 |
| *** bhavikdbavishi has quit IRC | 13:28 | |
| *** eharney has quit IRC | 13:29 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Use Python 3.x by default for testing https://review.opendev.org/667614 | 13:30 |
| *** yamamoto has quit IRC | 13:32 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Switch to opensuse-15 nodeset for bindep testing https://review.opendev.org/667533 | 13:32 |
| *** lseki has joined #openstack-infra | 13:32 | |
| *** Goneri has joined #openstack-infra | 13:37 | |
| *** rf0lc0 has quit IRC | 13:42 | |
| *** rf0lc0 has joined #openstack-infra | 13:42 | |
| corvus | clarkb: still seeing inability to connect to port 22 on the trusty/bionic (but not xenial) jobs -- the libvirt log doesn't seem to have any additional information: http://logs.openstack.org/21/667221/6/check/dib-nodepool-functional-openstack-ubuntu-trusty-src/34c6811/libvirt/qemu/instance-00000002.log | 13:44 |
| *** mloza has quit IRC | 13:47 | |
| *** eharney has joined #openstack-infra | 13:48 | |
| *** eharney has quit IRC | 13:48 | |
| *** eharney has joined #openstack-infra | 13:49 | |
| *** jistr is now known as jistr|call | 14:05 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP test ubuntu builds https://review.opendev.org/667624 | 14:06 |
| cloudnull | is something like https://pasted.tech/pastes/4b9cf2c88db2635fc9f3f58909265e9d4c0eb686.raw possible - basically I'm looking to reuse that template but I only want it to run when someone changes the docs? | 14:06 |
| corvus | clarkb: ^ i think that will get us the instance console logs | 14:06 |
| openstackgerrit | Sean McGinnis proposed openstack/reviewstats master: Raise hacking version and fix pep8 errors https://review.opendev.org/655911 | 14:06 |
| cloudnull | if that does work then I'd want to do the same for release notes too | 14:06 |
| *** bhavikdbavishi has joined #openstack-infra | 14:07 | |
| cloudnull | derived from here - https://github.com/openstack/tripleo-ansible/blob/master/zuul.d/layout.yaml#L1-L6 | 14:07 |
| corvus | cloudnull: project.templates is strictly a list: https://zuul-ci.org/docs/zuul/user/config.html#attr-project.templates | 14:08 |
| corvus | you can put files matchers on the job in the template definition | 14:09 |
| *** rfarr_ has joined #openstack-infra | 14:09 | |
| fungi | you can also have a local variant of one or more jobs from the template and specify a files matcher on those, right? | 14:09 |
| corvus | yeah | 14:09 |
| fungi | (unless the job is marked final) | 14:09 |
| corvus | but that particular template is system-wide, so i don't think we'd want to change it | 14:10 |
| *** rfarr has quit IRC | 14:10 | |
| corvus | cloudnull: so, really, all the answers involve dropping the use of that template and replacing it with something more explicit | 14:10 |
| *** jpena|off is now known as jpena | 14:10 | |
| cloudnull | bummer, but totally understandable. yea , reading https://zuul-ci.org/docs/zuul/user/config.html#project-template it looks like it merges ? | 14:10 |
| *** jpena is now known as jpena|mtg | 14:10 | |
| cloudnull | corvus ++ | 14:11 |
| corvus | yeah, even if you added the job some other way, the job in the template would still say to run, so it would run | 14:11 |
| *** rfarr__ has joined #openstack-infra | 14:11 | |
| fungi | i meant in the individual project, include the template but also call out the job itself in your project's pipeline definition and make a variant there... i guess that wouldn't end up excluding things that the version from the template doesn't exclude? | 14:11 |
| fungi | ahh, yep | 14:11 |
| corvus | fungi: right, that's the case i just described right before you hit enter | 14:11 |
| fungi | indeed ;) | 14:11 |
| cloudnull | thanks fungi corvus | 14:12 |
| *** rfarr_ has quit IRC | 14:14 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Use Python 3.x by default for testing https://review.opendev.org/667614 | 14:17 |
| *** bhavikdbavishi has quit IRC | 14:18 | |
| *** iurygregory has quit IRC | 14:19 | |
| *** jistr|call is now known as jistr | 14:22 | |
| *** _erlon_ has quit IRC | 14:23 | |
| *** mattw4 has joined #openstack-infra | 14:35 | |
| *** bhavikdbavishi has joined #openstack-infra | 14:40 | |
| *** yamamoto has joined #openstack-infra | 14:42 | |
| *** yamamoto has quit IRC | 14:42 | |
| *** yamamoto has joined #openstack-infra | 14:43 | |
| stephenfin | Could a kind devstack core send these two trivial patches on their way for me, pretty please? :) https://review.opendev.org/#/q/topic:bp/remove-cells-v1+status:open+project:openstack/devstack | 14:44 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP test ubuntu builds https://review.opendev.org/667624 | 14:45 |
| *** rfarr__ has quit IRC | 14:45 | |
| *** rfarr__ has joined #openstack-infra | 14:45 | |
| *** yamamoto has quit IRC | 14:49 | |
| *** yamamoto has joined #openstack-infra | 14:50 | |
| *** yamamoto has quit IRC | 14:51 | |
| *** yamamoto has joined #openstack-infra | 14:51 | |
| *** lpetrut has quit IRC | 14:52 | |
| zbr|ruck | fungi: what I do not underatand is why not running ansible-lint as part of the standard tox-linters standard job. | 14:53 |
| openstackgerrit | Hervé Beraud proposed openstack/pbr master: Fix parsing on egg names with dashes from git URLs https://review.opendev.org/648727 | 14:55 |
| *** yboaron_ has quit IRC | 14:57 | |
| *** yamamoto has quit IRC | 14:58 | |
| *** yamamoto has joined #openstack-infra | 14:59 | |
| *** michael-beaver has quit IRC | 15:00 | |
| *** Lucas_Gray has quit IRC | 15:02 | |
| *** asettle is now known as asettle-PTO | 15:03 | |
| *** xek__ has joined #openstack-infra | 15:03 | |
| *** ccamacho has quit IRC | 15:04 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 15:05 |
| *** mattw4 has quit IRC | 15:05 | |
| *** xek_ has quit IRC | 15:05 | |
| *** witek has joined #openstack-infra | 15:08 | |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Switch to opensuse-15 nodeset for bindep testing https://review.opendev.org/667533 | 15:09 |
| *** Lucas_Gray has joined #openstack-infra | 15:12 | |
| *** psachin has quit IRC | 15:19 | |
| *** whoami-rajat has quit IRC | 15:22 | |
| clarkb | I've approved the gitea06 inventory enrollment change. Will keep an eye on that this morning | 15:23 |
| *** dpawlik has quit IRC | 15:28 | |
| *** icarusfactor has joined #openstack-infra | 15:28 | |
| *** jamesdenton has joined #openstack-infra | 15:29 | |
| *** factor has quit IRC | 15:29 | |
| fungi | awesome. i'm around to help if things go sideways of course | 15:30 |
| clarkb | my biggest concern is that the exclusion for git repo creation will not work and the ansipuppet cron will take ~4 hours | 15:31 |
| clarkb | shouldn't be catastrophic but worth keeping an eye on | 15:31 |
| *** diablo_rojo has joined #openstack-infra | 15:31 | |
| *** tosky has quit IRC | 15:37 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: URLTrigger driver time based https://review.opendev.org/635567 | 15:37 |
| *** whoami-rajat has joined #openstack-infra | 15:38 | |
| fungi | infra-root: i'm not having much luck identifying where we might have documented any formal process for contacting freenode staff on behalf of the openstack project (and the last vague mention i could find in channel logs was years ago). any idea how they identify someone as a representative? access list for the #openstack channel maybe? | 15:40 |
| clarkb | fungi: I think corvus is our official contact with them | 15:40 |
| clarkb | (so requests from corvus are golden) | 15:40 |
| corvus | fungi: i think i am the only contact for openstack | 15:40 |
| corvus | when we added zuul, i think i added clarkb for that namespace | 15:40 |
| *** sshnaidm is now known as sshnaidm|afk | 15:40 | |
| corvus | i'm in a meeting, back in a few | 15:41 |
| *** armstrong has joined #openstack-infra | 15:41 | |
| fungi | corvus: not urgent, but when you get time can you reach out and ask for control of the #airship channel on behalf of roman_g? (see request above at 13:13z) | 15:42 |
| *** ykarel is now known as ykarel|away | 15:42 | |
| fungi | it's not registered and the airship project wants to register it, and for whatever reason freenode staff think it's related to the openstack project | 15:42 |
| fungi | more likely they need something set up similar to the openstack, zuul and opendev prefix associations i guess | 15:43 |
| *** witek has quit IRC | 15:47 | |
| *** ramishra has quit IRC | 15:49 | |
| *** pgaxatte has quit IRC | 15:49 | |
| *** rfarr__ has quit IRC | 15:54 | |
| *** rfarr__ has joined #openstack-infra | 15:55 | |
| openstackgerrit | Merged zuul/zuul master: Add missing docker daemon installation for Ubuntu/Debian https://review.opendev.org/665520 | 15:55 |
| *** mattw4 has joined #openstack-infra | 15:57 | |
| *** igordc has joined #openstack-infra | 15:59 | |
| *** tdasilva has quit IRC | 15:59 | |
| zbr|ruck | clarkb: fungi: re our talks yesterday about optimizing use of resources. Do you see any problems with an approach like https://review.opendev.org/#/c/666755/1/zuul.d/layout.yaml ? | 16:01 |
| *** _erlon_ has joined #openstack-infra | 16:02 | |
| clarkb | zbr|ruck: functionally it looks fine. It would be good to identify a metric or two that we can measure (via graphite, logstash, zuul logs, something) to determine if that improves zuul's behavior for tripleos needs | 16:03 |
| *** jpena|mtg is now known as jpena|off | 16:04 | |
| *** rfarr__ has quit IRC | 16:04 | |
| *** rfarr__ has joined #openstack-infra | 16:05 | |
| zbr|ruck | clarkb: yeah, i would love to be able to track this but I have no idea what to measure, especially as these changes are applied to each repo and their impact is like grain of sand. | 16:05 |
| zbr|ruck | their effective outcome should become visible only when you scale it. | 16:05 |
| clarkb | I think the measurement of node time that we have in graphite now is probably the one we want since you had mentioned wanting to reduce tripleos resource impact | 16:06 |
| zbr|ruck | but we may be able to estimate the impact: calculate lint fail rate, avg number of jobs. and this should allow us to estimate how many job executions we could save. if that would be >5% it would worth scaling, I think. | 16:07 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Enable testing against Python 3.6/3.7 (switch to stestr) https://review.opendev.org/667666 | 16:08 |
| clarkb | zbr|ruck: https://review.opendev.org/#/c/616306/16/doc/source/admin/monitoring.rst is the chnage that merged to report usage numbers to graphite | 16:08 |
| clarkb | zbr|ruck: that has been in place on our opendev zuul for a couple weeks now I think | 16:08 |
| clarkb | maybe 3 weeks | 16:08 |
| clarkb | so we should be able to pull those numbers over time | 16:09 |
| zbr|ruck | cool, thanks, I will look next week as I am overloaded now, trying to build a presentation for the summit. | 16:10 |
| openstackgerrit | Merged zuul/zuul master: Return store object instead of a function https://review.opendev.org/665642 | 16:12 |
| *** lucasagomes has quit IRC | 16:15 | |
| *** gfidente has quit IRC | 16:18 | |
| *** bhavikdbavishi has quit IRC | 16:27 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Enable nodepool testing for opensuse 15.x https://review.opendev.org/667537 | 16:27 |
| *** tdasilva has joined #openstack-infra | 16:29 | |
| *** eernst has joined #openstack-infra | 16:29 | |
| openstackgerrit | Merged zuul/zuul master: Add missing start-message in pipeline config schema https://review.opendev.org/665936 | 16:30 |
| corvus | fungi, clarkb: oh, sorry, it was #opendev that got registered as a group with clarkb and i as contacts, and it's separate from openstack | 16:30 |
| corvus | fungi, clarkb: is that what we want to have happen with #airship too? | 16:30 |
| clarkb | I'm ok with that | 16:31 |
| clarkb | hogepodge is probably asleep now but he may have ideas for who would be appropriate on that? | 16:31 |
| corvus | i'm imagining that the freenode admins asked about openstack since roman_g's request was originally to get openstackinfra added. i do think getting a new group registration for airship is probably the best route, with at least one opendev admin as the group contact. | 16:34 |
| openstackgerrit | Merged opendev/system-config master: Enroll new gitea06 into ansible inventory https://review.opendev.org/667474 | 16:34 |
| fungi | yeah, that's where i'm leaning as well | 16:35 |
| corvus | and i'm happy to help make the request (as an openstack contact) to clear up any lingering ambiguity | 16:35 |
| fungi | ultimately the openstack group should probably be for openstack channels | 16:35 |
| *** tesseract has quit IRC | 16:35 | |
| fungi | and airship having its own group doesn't preclude opendev/openstack-infra bot operators from having access in their channel acls | 16:35 |
| fungi | just means that the folks who call the shots on any future channels starting #airship-... are an independent set of people | 16:36 |
| *** ykarel|away has quit IRC | 16:37 | |
| *** jpich has quit IRC | 16:39 | |
| *** bhavikdbavishi has joined #openstack-infra | 16:43 | |
| corvus | assuming this isn't super-urgent, let's wait for hogepodge to weigh in, and then i'll send in a group request, and hopefully things will be resolved in something like days/1-week timeframe. if it's very urgent, i could go beg that they provisionally assign that to openstack while we go through the group registration process. | 16:43 |
| clarkb | +1 to waiting for input as I don't think it is urgent | 16:44 |
| corvus | clarkb: is this the first time we've captured an instance log from devstack? http://logs.openstack.org/24/667624/2/check/dib-nodepool-functional-openstack-ubuntu-trusty-src/39feb07/instances/a914df0d-5bd8-47d4-ad72-cc7a36012eb5/console.log | 16:45 |
| corvus | "Serious errors were found while checking the disk drive for /." and "keys:Press I to ignore, S to skip mounting, or M for manual recovery" certainly are suggestive of the sort of problem that would prevent ssh from working | 16:46 |
| clarkb | corvus: tempest captures them when tests fail against a node, but outside of that I think this may be the first time | 16:47 |
| AJaeger | clarkb, could you put review of the stack by tobias-urdin in https://review.opendev.org/#/q/topic:forge-publish on your list for some time this week, please?. A small change but to be able to merge this without breakage, we need many small changes... | 16:48 |
| clarkb | AJaeger: yes | 16:48 |
| clarkb | corvus: and ya that seems like the type of error that would prevent ssh from working | 16:48 |
| AJaeger | thanks, clarkb | 16:48 |
| openstackgerrit | Dirk Mueller proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 16:49 |
| corvus | clarkb: weird... the bionic job doesn't have that error -- http://logs.openstack.org/24/667624/2/check/dib-nodepool-functional-openstack-ubuntu-bionic-src/05fce53/ | 16:51 |
| corvus | it says ssh has started | 16:52 |
| clarkb | corvus: maybe we want to hold the test node then manually inspect the disk that was built? | 16:53 |
| clarkb | corvus: what is extra odd is we produce working production images with those elements too | 16:53 |
| openstackgerrit | Merged zuul/zuul master: Add missing doc for pipeline start-message https://review.opendev.org/665930 | 16:53 |
| corvus | clarkb: okay, i'll recheck that and hold all 3 jobs | 16:53 |
| clarkb | could it be that they are gpt formatted and not mbr and xenial and trusty libvirt/qemu/kvm only support mbr | 16:54 |
| clarkb | (I thought you hda to opt into gpt with dib though) | 16:54 |
| corvus | clarkb: my missing a package or option like that sounds very plausible for this kind of work -- but it looked like the trusty job got farther than the partition table | 16:54 |
| clarkb | and I guess looking at the image build log is the other place to investigate /me pulls that up | 16:56 |
| *** ginopc has quit IRC | 16:56 | |
| clarkb | http://logs.openstack.org/24/667624/2/check/dib-nodepool-functional-openstack-ubuntu-trusty-src/39feb07/nodepool/builds/test-image-0000000001.log for the trusty job above | 16:56 |
| clarkb | oh could this be the deboostrap bug on bionic? | 16:57 |
| clarkb | in production (and with the older jobs) we ran the builds on xenial with a different version of debootstrap | 16:57 |
| clarkb | and I recall there was a bug in bionic's debootstrap? fungi do you remember what the details around that were? | 16:57 |
| *** e0ne has quit IRC | 16:58 | |
| corvus | clarkb: wow, there is such a hack in the devstack plugin source | 16:59 |
| corvus | add-apt-repository ppa:openstack-ci-core/debootstrap | 16:59 |
| clarkb | ya I think we had to do that in production too? | 16:59 |
| clarkb | ubuntu debootstrap was broken somehow | 16:59 |
| corvus | yeah, puppet-diskimage_builder does that | 17:01 |
| fungi | i'll have to dig a bit to refresh my memory | 17:02 |
| corvus | https://review.opendev.org/659031 -- memory refresher | 17:02 |
| fungi | i take it there are no breadcrumbs in the devstack plugin with a bug number | 17:02 |
| corvus | fungi: ^ bug number breadcrumbs are there | 17:02 |
| *** ricolin has quit IRC | 17:02 | |
| fungi | ahh, yup | 17:02 |
| fungi | so ubuntu hasn't been backporting debootstrap updates with new release/suite names | 17:03 |
| corvus | clarkb: i'll cancel the autoholds and find a way to work this into the jobs | 17:04 |
| clarkb | corvus: ok | 17:04 |
| corvus | clarkb: thanks :) | 17:04 |
| *** yamamoto_ has joined #openstack-infra | 17:04 | |
| *** Lucas_Gray has quit IRC | 17:04 | |
| *** yamamoto_ has quit IRC | 17:05 | |
| corvus | (i was like, pfft, i *know* how to install nodepool, so i just skipped over the "install_nodepool" function) | 17:05 |
| *** yamamoto_ has joined #openstack-infra | 17:06 | |
| *** yamamoto has quit IRC | 17:06 | |
| clarkb | base.yaml is running now. the next remote puppet git run should half configure gitea06 | 17:09 |
| corvus | i haven't run the debian versions of the new jobs yet (they're all in experimental, so not really on my radar), so i haven't observed this behavior wrt debian | 17:10 |
| openstackgerrit | Merged zuul/zuul master: Add support for item.change for pipeline start-message formater https://review.opendev.org/665968 | 17:12 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP test ubuntu builds https://review.opendev.org/667624 | 17:15 |
| *** eernst has quit IRC | 17:16 | |
| corvus | dirk: mind if i take over my glean change (667225) again? :) | 17:16 |
| clarkb | ok new gitea06 is being ignored due to ssh host key not being accepted | 17:17 |
| clarkb | I'm going to get a bike ride in before I do that | 17:18 |
| dirk | corvus: maybe you could help me getting the depending reviews reviewed + merged? | 17:18 |
| *** zbr|ruck is now known as zbr | 17:18 | |
| dirk | corvus: e.g. https://review.opendev.org/#/c/666066/ not being merged is slightly annoying | 17:18 |
| *** weifan has joined #openstack-infra | 17:19 | |
| corvus | dirk: that seems fine, but i don't have any background there; let's ask clarkb or fungi for that one | 17:20 |
| dirk | corvus: yesterday the glean gate made suse completely nonvoting because leap 42.3 was failing | 17:21 |
| dirk | 42.3 is unmaintained and end of life, so we should just remove that part instead of making everything suse related nonvoting | 17:21 |
| openstackgerrit | Merged openstack/project-config master: Remove nodepool functional job attachments https://review.opendev.org/667213 | 17:22 |
| dirk | corvus: potentially this conflicts with your stuff: https://review.opendev.org/#/c/667532/ | 17:22 |
| dirk | when this is merged I believe the revert of the change from yesterday could be merged | 17:22 |
| corvus | dirk: yes, i did that. the current/old jobs combine all of the suse builds into one job. my new jobs are one build-per image (at least for now, we can change that later) | 17:23 |
| corvus | dirk: i'm happy to make whatever suse versions are current voting as part of my work | 17:23 |
| corvus | dirk: so i'll just drop 42.3 entirely from the new jobs? | 17:23 |
| dirk | corvus: yes | 17:24 |
| *** kopecmartin is now known as kopecmartin|off | 17:24 | |
| corvus | dirk: ok, i'll have a new revision of 667221 in a few mins | 17:25 |
| dirk | corvus: there is a depöending change (667537 ) | 17:25 |
| dirk | ah, ok, but I left out the 42.3 stuff. ok. I'll just rebase it then | 17:26 |
| corvus | dirk: 15.0 should still work right? | 17:26 |
| dirk | yes | 17:26 |
| dirk | but the idea was to no longer test each opensuse 15.x release separately but only have a opensuse-15 job (which is pointing to the latest enabled opensuse-15.x release) | 17:27 |
| dirk | similar to *-fedora-latest that is used elsewhere | 17:27 |
| corvus | dirk: ok, let's leave 667537 as a followup (and rebase) -- i think it deserves its own change+discussion because of that ^ | 17:27 |
| corvus | dirk: (and, fwiw, there's a "fedora-29" job, so i'm not sure that's strictly true) | 17:28 |
| *** udesale has quit IRC | 17:29 | |
| fungi | i think it's more like how we test ubuntu 16.04 instead of 16.04.3 | 17:29 |
| dirk | corvus: yeah, the good thing about naming standards is that there are so many of them | 17:29 |
| AJaeger | cloudnull: do you have a governance change up as followup for https://review.opendev.org/667418 ? | 17:29 |
| fungi | or centos 7 instead of 7.5 | 17:29 |
| *** rf0lc0 is now known as rfolco | 17:30 | |
| fungi | presumably when opensuse 16.0 comes along we'd still transition with a different image name | 17:30 |
| corvus | ah, heh :) | 17:30 |
| dirk | luckily we only have to worry about that 2024+ | 17:31 |
| *** panda has quit IRC | 17:31 | |
| dirk | there won't be a 16.0 for the forseeable future | 17:31 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Replace nodepool func jobs https://review.opendev.org/667221 | 17:32 |
| fungi | dirk: so more like the cadence fro centos/rhel 5 to 6 to 7 to 8 | 17:32 |
| corvus | okay, that drops 42.3 and fixes our best guess of what the ubuntu problem is | 17:32 |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Add and switch to the newly created opensuse-15 nodeset https://review.opendev.org/666066 | 17:33 |
| openstackgerrit | Merged zuul/zuul master: Allow config projects to override allowed-projects https://review.opendev.org/666733 | 17:33 |
| cloudnull | AJaeger I do, though it looks like I need to give it another spin due to some docs issue - https://review.opendev.org/#/c/667416 | 17:33 |
| *** panda has joined #openstack-infra | 17:35 | |
| *** eernst has joined #openstack-infra | 17:35 | |
| AJaeger | cloudnull: that doc issue looks unrelated - thanks | 17:35 |
| openstackgerrit | James E. Blair proposed opendev/glean master: Replace nodepool func jobs https://review.opendev.org/667225 | 17:36 |
| AJaeger | any other config-core around to remove some retired repos? PLease review https://review.opendev.org/663355 - I'll +2 in a minute... | 17:36 |
| *** rfarr_ has joined #openstack-infra | 17:37 | |
| cloudnull | thanks AJaeger | 17:37 |
| AJaeger | cloudnull: recheck won't help on it - see https://review.opendev.org/#/q/project:openstack/governance - all those -1s are from tox-docs ;( | 17:38 |
| corvus | AJaeger: that's probably not the right change | 17:38 |
| AJaeger | corvus: indeed - https://review.opendev.org/667418 is the one I wanted... | 17:39 |
| AJaeger | sorry | 17:39 |
| dirk | clarkb: thanks! | 17:39 |
| *** eernst has quit IRC | 17:40 | |
| *** rfarr__ has quit IRC | 17:40 | |
| cloudnull | AJaeger ah, bummer | 17:41 |
| corvus | clarkb, fungi: it looks like the openstack - | 17:44 |
| corvus | gr | 17:44 |
| corvus | clarkb, fungi: it looks like the openstack-ci-core ppa only has xenial packages | 17:44 |
| fungi | that makes sense | 17:45 |
| fungi | we presumably need to build the same newer version for bionic as well | 17:45 |
| corvus | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/debootstrap | 17:45 |
| corvus | so.... how does that happen? | 17:46 |
| *** eernst has joined #openstack-infra | 17:46 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Enable nodepool testing for opensuse 15.1 https://review.opendev.org/667537 | 17:47 |
| pabelanger | corvus: you should be able to use UI to copy package into bionic | 17:48 |
| pabelanger | which does a rebuild | 17:48 |
| *** jamesdenton has quit IRC | 17:49 | |
| *** eernst has quit IRC | 17:49 | |
| pabelanger | https://launchpad.net/~openstack-ci-core/+archive/ubuntu/debootstrap/+packages in top right, I can see copy packages | 17:49 |
| *** pleia2_ has joined #openstack-infra | 17:49 | |
| pabelanger | that should be the entry point | 17:49 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP test ubuntu builds https://review.opendev.org/667624 | 17:49 |
| *** eernst has joined #openstack-infra | 17:50 | |
| *** zerick_ has joined #openstack-infra | 17:50 | |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Replace nodepool func jobs https://review.opendev.org/667221 | 17:50 |
| corvus | pabelanger: thanks -- i think i'll pin the debian and ubuntu builds to xenial for now, and later we can do that and test the upgrade | 17:51 |
| *** e0ne has joined #openstack-infra | 17:51 | |
| *** jistr_ has joined #openstack-infra | 17:52 | |
| *** dtantsur has joined #openstack-infra | 17:52 | |
| johnsom | Did the path to "required-projects" clones change recently? I'm getting "It looks like a path. File ''/opt/stack/octavia-tempest-plugin'' does not exist." this morning. http://logs.openstack.org/61/665861/6/check/octavia-v2-dsvm-noop-api/dac4dba/job-output.txt.gz#_2019-06-26_14_13_23_428457 | 17:52 |
| johnsom | We are explicitly defining that path for the tempest plugin here: https://github.com/openstack/octavia-tempest-plugin/blob/master/zuul.d/jobs.yaml#L52 which probably isn't the right thing. | 17:53 |
| johnsom | But it worked until yesterday/today | 17:53 |
| fungi | unless that will successfully work on bionic too | 17:53 |
| *** pleia2 has quit IRC | 17:53 | |
| *** zaro has quit IRC | 17:53 | |
| *** zerick has quit IRC | 17:53 | |
| *** dtantsur|mtg has quit IRC | 17:53 | |
| *** jistr has quit IRC | 17:53 | |
| *** yamamoto_ has quit IRC | 17:53 | |
| *** rfarr_ has quit IRC | 17:54 | |
| corvus | johnsom: http://lists.openstack.org/pipermail/openstack-discuss/2019-June/007313.html could be related | 17:54 |
| *** yamamoto has joined #openstack-infra | 17:54 | |
| corvus | johnsom: but not if it has worked after monday | 17:54 |
| *** rfarr_ has joined #openstack-infra | 17:54 | |
| johnsom | Yeah, I fixed the one docs issue we had as fallout from that on Monday. | 17:54 |
| johnsom | That fix has already merged, so we were good after that. This seems new. | 17:55 |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Replace Trusty with Bionic in the testing https://review.opendev.org/667694 | 17:55 |
| *** altlogbot_2 has quit IRC | 17:55 | |
| openstackgerrit | Merged openstack/project-config master: Remove retired repos https://review.opendev.org/667418 | 17:55 |
| *** zaro has joined #openstack-infra | 17:56 | |
| *** jamesdenton has joined #openstack-infra | 17:56 | |
| johnsom | I thought that if we defined "openstack/octavia-tempest-plugin" in the zuul job definition for "required-projects" it should always be cloned to /opt/stack/octavia-tempest-plugin. | 17:56 |
| corvus | johnsom: is double quoting a problem? | 17:56 |
| *** altlogbot_2 has joined #openstack-infra | 17:57 | |
| *** altlogbot_2 has quit IRC | 17:57 | |
| *** bhavikdbavishi has quit IRC | 17:58 | |
| corvus | johnsom: it will be cloned to ~zuul/src/opendev.org/openstack/octavia/tempest-plugin there's some stuff in the devstack job which moves repos to /opt/stack/... i think it should move octavia-tempest-plugin too but i'm not 100% sure | 17:58 |
| openstackgerrit | Merged zuul/zuul master: Strengthen the caution about allowed-projects https://review.opendev.org/667241 | 17:58 |
| *** altlogbot_1 has joined #openstack-infra | 17:58 | |
| johnsom | corvus Yeah, maybe it's the quoting. though that has been there for 11 months.... Something changed. Maybe it was something in devstack, I didn't know it was involved in that file path. I will look at recently merged stuff there. | 17:59 |
| corvus | this is amazing: pip install -c u-c-m.txt ''"'"'/opt/stack/octavia-tempest-plugin'"'"'' | 17:59 |
| *** eernst has quit IRC | 18:00 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: [Configuration] Switch to stestr. https://review.opendev.org/629414 | 18:01 |
| *** weifan has quit IRC | 18:01 | |
| johnsom | Yeah, that is.... interesting | 18:01 |
| *** eernst has joined #openstack-infra | 18:01 | |
| *** weifan has joined #openstack-infra | 18:01 | |
| *** altlogbot_1 has quit IRC | 18:01 | |
| fungi | nutty is more like it | 18:01 |
| *** weifan has quit IRC | 18:01 | |
| *** weifan has joined #openstack-infra | 18:02 | |
| *** altlogbot_2 has joined #openstack-infra | 18:02 | |
| johnsom | Nothing obvious in devstack, pip hasn't changed since may. hmmm | 18:02 |
| corvus | johnsom: yeah -- the last successful run has less quoting around that: http://logs.openstack.org/49/667249/1/check/octavia-v2-dsvm-noop-api/a118c14/job-output.txt.gz#_2019-06-25_00_23_20_238939 | 18:03 |
| corvus | so i think that's the immediate cause; what caused that i don't know | 18:04 |
| dirk | likely a quoting fix in bash? | 18:07 |
| *** rfarr__ has joined #openstack-infra | 18:07 | |
| *** rfarr_ has quit IRC | 18:10 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Advance ansible-lint cap to test with 4 https://review.opendev.org/667695 | 18:11 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Make all lines less than 160 characters long https://review.opendev.org/667696 | 18:11 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Don't compare to literal True/False https://review.opendev.org/667697 | 18:11 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 18:11 |
| johnsom | No two projects quote it the same, lol | 18:12 |
| fungi | the beauty of diversity! | 18:13 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: Run linters via pre-commit https://review.opendev.org/667699 | 18:14 |
| *** weifan has quit IRC | 18:15 | |
| *** weifan has joined #openstack-infra | 18:16 | |
| johnsom | Giving it a go here: https://review.opendev.org/667700 | 18:16 |
| openstackgerrit | Merged zuul/zuul master: Stop bind mounting zuul dir into bwrap https://review.opendev.org/666972 | 18:20 |
| *** weifan has quit IRC | 18:20 | |
| zbr | fungi: cool, after your patches go in my POC will be much smaller. Btw, I did not know that ansible would have an `unless` command, that is new to me, apparently not even documented on https://docs.ansible.com/ansible/latest/user_guide/playbooks_conditionals.html#conditionals | 18:21 |
| zbr | are you sure that is not a puppet stuff? | 18:21 |
| fungi | zbr: it's possible i made it up or confused it with puppet. i thought it was a thing, but can switch those to some sort of inversion operator if needed? | 18:21 |
| fungi | (also, what, ansible-lint doesn't actually evaluate ansible syntax?) | 18:21 |
| zbr | i used: when: not foo | 18:21 |
| fungi | ahh, i'll give that a shot instead, thanks | 18:21 |
| zbr | it does not, ansible-lint does its own parsing, so that's why it had bugs. it does some tricks and loads some code from ansible but cannot do it at the same level. | 18:22 |
| *** eernst has quit IRC | 18:22 | |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Don't compare to literal True/False https://review.opendev.org/667697 | 18:23 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 18:23 |
| *** eernst has joined #openstack-infra | 18:23 | |
| zbr | fungi: example of stuff you may encounter: https://github.com/ansible/ansible-lint/issues/505#issuecomment-505920271 | 18:24 |
| fungi | yeah, possible we just exclude 206 for much longer if we do want to continue using ansible-lint but consider that rule intractably broken | 18:26 |
| smcginnis | Are there any changes to our images where the DejaVuSans font would not be installed? | 18:27 |
| smcginnis | I have a patch that looks like https://opendev.org/openstack/governance/src/branch/master/doc/source/_exts/badges.py#L86 is hitting that. | 18:27 |
| smcginnis | http://logs.openstack.org/64/667664/2/check/openstack-tox-docs/b555073/job-output.txt.gz#_2019-06-26_17_33_47_148005 | 18:27 |
| fungi | smcginnis: maybe it was in the bindep fallback list before? seems like an odd one to have been including but... | 18:27 |
| AJaeger | fungi: "fonts-nanum" is in the list | 18:28 |
| smcginnis | fungi: Is that fallback file still accessible somewhere that I can check? | 18:28 |
| AJaeger | fungi: and "fonts-takao" | 18:28 |
| AJaeger | smcginnis: yeah, let me grab it | 18:28 |
| smcginnis | AJaeger: Thanks. I can track down which package it was in and add it in to the governance bindep.txt. | 18:28 |
| AJaeger | https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/bindep-fallback.txt | 18:29 |
| fungi | yeah, i guess there may have been some fonts in that case. if we need one for governance doc builds then i guess we should add a bindep.txt to governance | 18:29 |
| *** e0ne has quit IRC | 18:29 | |
| smcginnis | AJaeger: Thanks! | 18:29 |
| *** efried has joined #openstack-infra | 18:30 | |
| efried | donnyd: like this | 18:31 |
| donnyd | i see | 18:31 |
| smcginnis | Looks like it's in fonts-dejavu-core, which was not in bindep-fallback.txt. But guessing it was maybe a dependency for one of the other font packages. | 18:31 |
| clarkb | I have accepted new gitea06's ssh host key now | 18:32 |
| smcginnis | Hmm, no bindep.txt at all in governance. That seems like it could be an issue for other things. | 18:32 |
| clarkb | so next pass through should do the things to it | 18:32 |
| donnyd | clarkb what are the ipv6 requirements? anything special or just a routable subnet? | 18:32 |
| AJaeger | smcginnis: just add a file with the single line in it ;) | 18:32 |
| dirk | corvus: https://review.opendev.org/#/c/667221/8 still doesn't seem to be working.. | 18:32 |
| smcginnis | AJaeger: I guess so. :) | 18:32 |
| openstackgerrit | Alex Schultz proposed zuul/zuul master: Additional note about branches for implied-branches https://review.opendev.org/667415 | 18:32 |
| clarkb | donnyd: we need ssh access into the nodes from the internet (because we run services in various clouds) so one ipv6 address per test node (I think even a /128 is fine) | 18:32 |
| clarkb | donnyd: then because there are things like dockerhub and github that only ipv4 we need ipv4 NAT outbound for the instances, this can be shared behind a single ipv4 address | 18:33 |
| donnyd | Ok, thats not an issue at all. So public ipv6 and private ipv4 will work fine for the network is what I am hearing | 18:34 |
| corvus | dirk: ah, i see the error, thx | 18:34 |
| clarkb | donnyd: yes it should. Then we need a single ipv4 address (in addition to the nat addr) for our mirror node | 18:34 |
| clarkb | donnyd: that acts as the primary ipv4 gateway for things as we mirror and cache stuff on that node (keeping test nodes talking on network as much as possible) | 18:34 |
| donnyd | Ok, no problem with that either.. I am hoping to work on it later tonight when I am done at my day job :) | 18:35 |
| donnyd | Had to shift a bunch of stuff around, so its taking a little longer than I wanted it to | 18:36 |
| corvus | dirk: i'm going to wait until i have results on 667624 before i fix that so i don't loose my place in line for node allocations. | 18:36 |
| *** ociuhandu has quit IRC | 18:36 | |
| clarkb | donnyd: no worries, and thank you for the help | 18:36 |
| donnyd | :) | 18:36 |
| donnyd | its fun, so I don't mind at all | 18:37 |
| *** eernst has quit IRC | 18:37 | |
| dirk | corvus: wfm | 18:38 |
| openstackgerrit | Merged zuul/zuul master: Fix build leak with paused jobs https://review.opendev.org/667072 | 18:39 |
| *** e0ne has joined #openstack-infra | 18:44 | |
| *** michael-beaver has joined #openstack-infra | 18:44 | |
| *** kjackal has quit IRC | 18:51 | |
| *** kjackal has joined #openstack-infra | 18:52 | |
| *** jcoufal has quit IRC | 18:56 | |
| openstackgerrit | Merged zuul/zuul master: Split ansible output into its own sub logger https://review.opendev.org/667131 | 18:59 |
| *** hwoarang has quit IRC | 19:03 | |
| *** hwoarang has joined #openstack-infra | 19:04 | |
| *** e0ne has quit IRC | 19:08 | |
| *** dklyle has quit IRC | 19:09 | |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Cleanup deprecated ansible syntax https://review.opendev.org/667710 | 19:14 |
| clarkb | fungi: corvus ^ cleans up deprecation notices I saw when watching gitea06 | 19:14 |
| openstackgerrit | Merged zuul/zuul master: Remove non working tests/base.py ZuulTestCase.getPipeline method https://review.opendev.org/667351 | 19:14 |
| *** ralonsoh has quit IRC | 19:17 | |
| *** phughk has quit IRC | 19:18 | |
| clarkb | ok we have a mostly empty gitea install on gitea06 | 19:19 |
| clarkb | I'm going to backup the db in its current state in case I need the user table (or other table for some reason) | 19:19 |
| *** dims has quit IRC | 19:20 | |
| clarkb | Then restore gitea01's db | 19:20 |
| *** whoami-rajat has quit IRC | 19:22 | |
| *** rfarr__ has quit IRC | 19:24 | |
| *** rfarr__ has joined #openstack-infra | 19:24 | |
| *** dims has joined #openstack-infra | 19:28 | |
| *** kjackal has quit IRC | 19:30 | |
| *** kjackal has joined #openstack-infra | 19:31 | |
| fungi | clarkb: exim paniclog on gitea06.opendev.org has non-zero size | 19:31 |
| fungi | rm it if you get a chance | 19:31 |
| fungi | there are more which complain nightly i haven't gotten around to cleaning up, but seems we still end up with errors in paniclog which happen during server provisioning for pretty much every new one we launch | 19:32 |
| *** dims has quit IRC | 19:32 | |
| fungi | "User 0 set for local_delivery transport is on the never_users list" | 19:32 |
| clarkb | fungi: will do | 19:33 |
| clarkb | /usr/bin/docker-compose -f /etc/gitea-docker/docker-compose.yaml exec -T mariadb bash -c '/usr/bin/mysql -uroot -p"$MYSQL_ROOT_PASSWORD"' < /home/clarkb/gitea-mariadb.sql | 19:33 |
| clarkb | does that look obviously wrong for a restore action? because that didn't seem to update the database | 19:33 |
| *** dims has joined #openstack-infra | 19:33 | |
| clarkb | maybe -T doesn't work wiht < | 19:33 |
| fungi | maybe needs to be inside the bash | 19:34 |
| fungi | in which case you need the file local | 19:34 |
| clarkb | paniclog rm'd | 19:34 |
| fungi | thanks! | 19:34 |
| fungi | er, need the file inside the container's filesystem tree | 19:34 |
| clarkb | ya maybe | 19:34 |
| fungi | docker-compose may not pass along its stdin | 19:35 |
| *** weifan has joined #openstack-infra | 19:35 | |
| *** weifan has quit IRC | 19:35 | |
| *** weifan has joined #openstack-infra | 19:35 | |
| clarkb | ah yup mariadb's docker image example shows docker exec -i being used | 19:35 |
| clarkb | so I think if I drop the -T it will work | 19:35 |
| *** dklyle has joined #openstack-infra | 19:36 | |
| * clarkb tries | 19:36 | |
| roman_g | Hello team. Trying to use opensuse-tumbleweed image in Zuul job, but failing to get packages installed with the following error: | 19:37 |
| roman_g | [repo-oss|http://mirror.ord.rax.openstack.org/opensuse/tumbleweed/repo/oss/] Valid metadata not found at specified URL | 19:37 |
| fungi | what's -T do" assign a tty? | 19:37 |
| clarkb | fungi: does not assign a tty | 19:37 |
| fungi | ahh | 19:37 |
| roman_g | Is this just abad luck and I just retry, or there is some problem? | 19:37 |
| AJaeger | roman_g: that URL does not exist at all | 19:37 |
| roman_g | well.. it's set in image | 19:38 |
| AJaeger | cmurphy, dirk: any ideas on roman_g 's problem? ^ | 19:38 |
| fungi | it's possible we're back to having trouble mirroring tumbleweed again | 19:38 |
| fungi | i can take a quick look in the mirror-update log | 19:38 |
| *** panda has quit IRC | 19:39 | |
| roman_g | thank you, fungi | 19:39 |
| openstackgerrit | Merged zuul/zuul master: Parallelize github event processing https://review.opendev.org/666982 | 19:39 |
| *** panda has joined #openstack-infra | 19:40 | |
| clarkb | db seems to actually be working this time with -T removed | 19:41 |
| clarkb | I also stopped the two gitea containers before doing this to avoid conflicts | 19:41 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Include overcloud deploy in logstash config https://review.opendev.org/666883 | 19:41 |
| clarkb | I'll get dns update pushed, we can confirm things look correct then remove gitea06 project creation exclusion and add it back to haproxy if things look happy | 19:42 |
| openstackgerrit | Dirk Mueller proposed opendev/bindep master: Replace Trusty with Bionic in the testing https://review.opendev.org/667694 | 19:42 |
| *** sreejithp has joined #openstack-infra | 19:43 | |
| clarkb | one observation: docker-compose seems to become the bottleneck when doing the restore | 19:44 |
| fungi | looks like we're only mirroring 15.0, 15.1 and 42.3 | 19:44 |
| clarkb | fungi: AJaeger I want to say we stopped mirroring tumbleweed because it changes often and the upstream mirror source wsn't able to keep up or something like that? | 19:45 |
| openstackgerrit | Sorin Sbarnea proposed openstack/project-config master: Include overcloud deploy in logstash config https://review.opendev.org/666883 | 19:45 |
| clarkb | so it might be a bug in our job setup to use our mirror on tumbleweed? | 19:45 |
| *** altlogbot_2 has quit IRC | 19:45 | |
| fungi | aha! https://review.opendev.org/660125 | 19:45 |
| fungi | yes, it's provided as a caching mirror instead but the mirror data maybe hasn't been adjusted to accommodate that? | 19:46 |
| *** dims has quit IRC | 19:46 | |
| clarkb | fungi: ya I think its a bug in our base job setup stuff for mirrors | 19:46 |
| *** altlogbot_1 has joined #openstack-infra | 19:47 | |
| fungi | i can't dig deeper right now, have to disappear for a bit, but should return in an hour or so | 19:48 |
| openstackgerrit | Clark Boylan proposed opendev/zone-opendev.org master: Update gitea06 IP address https://review.opendev.org/667717 | 19:48 |
| clarkb | we can probably go ahead and make that dns update now | 19:48 |
| corvus | clarkb: how's d-c the bottleneck? | 19:49 |
| *** jtomasek has quit IRC | 19:49 | |
| clarkb | corvus: top shows it using an entire cpu during this db restore | 19:49 |
| clarkb | corvus: maybe shuffling data between namespaces? | 19:49 |
| clarkb | (and the db restore isn't particularly fast) if I have to do it again I'll try with docker exec instead | 19:50 |
| corvus | clarkb: why not use docker exec | 19:50 |
| corvus | yeah that | 19:50 |
| corvus | i think d-c only adds value when creating, updating, or stopping containers | 19:50 |
| clarkb | we use d-c for the dump creation (which is quick) because it makes naming things easier | 19:51 |
| corvus | oh interesting; i thought we used docker directly for that | 19:51 |
| clarkb | so I was just updating that command to do the restore | 19:51 |
| clarkb | corvus: using d-c makes it a one liner, simplified your example you shared | 19:51 |
| clarkb | (also its not a problem on the mysqldump side of things, that is quick) | 19:52 |
| *** dims has joined #openstack-infra | 19:52 | |
| *** dims has quit IRC | 19:57 | |
| clarkb | thinking I'll ^C it then start a screen and use docker exec instead at this point | 19:57 |
| openstackgerrit | Merged zuul/zuul master: Differentiate between queued and waiting jobs in zuul web UI https://review.opendev.org/660878 | 19:59 |
| clarkb | (that left the mysql client process running and its still doing work, maybe faster now?) | 20:00 |
| roman_g | fungi: thank you for your support. I will re-run jobs tomorrow. | 20:01 |
| corvus | clarkb: when you have a second, i'm still seeing an error with the bionic job (though trusty seems to be better now with the debootstrap ppa) | 20:04 |
| corvus | clarkb: i set an autohold, so should be no rush | 20:04 |
| *** mriedem has quit IRC | 20:04 | |
| clarkb | corvus: k I'm about to eat lunch while I let mysql decide what it is doing but can look after | 20:04 |
| *** mriedem has joined #openstack-infra | 20:07 | |
| corvus | the console log looks clean, glean claims to have configured an interface, but ssh gives 'no route to host' | 20:08 |
| openstackgerrit | Dirk Mueller proposed zuul/nodepool master: Switch to openSUSE 15.1 https://review.opendev.org/667532 | 20:09 |
| openstackgerrit | Dirk Mueller proposed zuul/nodepool master: Add Python 3.7 testing https://review.opendev.org/667720 | 20:09 |
| *** diablo_rojo has quit IRC | 20:09 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Enable nodepool testing for opensuse 15.1 https://review.opendev.org/667537 | 20:12 |
| corvus | i don't suppose our test images have any actual user accounts with passwords so we can log in on the console? | 20:13 |
| *** altlogbot_1 has quit IRC | 20:15 | |
| *** eharney has quit IRC | 20:16 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Enable nodepool testing for opensuse 15.1 https://review.opendev.org/667537 | 20:17 |
| openstackgerrit | Merged zuul/zuul master: [doc][monitoring] Fix the wait_time parent attribute https://review.opendev.org/667342 | 20:18 |
| *** altlogbot_2 has joined #openstack-infra | 20:18 | |
| clarkb | corvus: they have the devuser user and the root user | 20:20 |
| clarkb | er but those are both with keys I think | 20:21 |
| corvus | yeah, i just mounted the glance image and there's nothing in shadow | 20:22 |
| corvus | i'll try performing some surgery to see if i can get a console login | 20:22 |
| clarkb | ERROR 1728 (HY000): Cannot load from mysql.proc. The table is probably corrupted | 20:22 |
| clarkb | I get that on gitea06 now | 20:22 |
| clarkb | (was trying to drop the gitea db to do another restore) | 20:23 |
| corvus | clarkb: was that one of the tables that was restored? | 20:23 |
| corvus | (we should not be backing that up) | 20:23 |
| clarkb | let me check the backup file | 20:23 |
| *** jsmdk has joined #openstack-infra | 20:24 | |
| openstackgerrit | Dirk Mueller proposed openstack/diskimage-builder master: DNM: Enable nodepool testing for opensuse 15.1 https://review.opendev.org/667537 | 20:25 |
| clarkb | corvus: yes it is in there. So our exclude isn't sufficient? | 20:25 |
| clarkb | --opt --ignore-table mysql.event --all-databases | 20:25 |
| corvus | clarkb: seems like it... i'm not sure the best way to recover here.... | 20:26 |
| corvus | clarkb: do our playbooks do anything to the mysql container after it's created? | 20:27 |
| clarkb | not directly. But they do a couple things to gitea that updates the db | 20:27 |
| corvus | because if not, we may just be able to delete the container and volume, then docker-compose it up again | 20:27 |
| corvus | but that'll be restored | 20:27 |
| clarkb | ya should be | 20:27 |
| corvus | then i think that process is worth a try, should be quick | 20:28 |
| clarkb | and then edit the db restore file to only restore the gitea db? | 20:28 |
| corvus | clarkb: yep | 20:28 |
| corvus | (the mysql user stuff should be created when the mariadb container initializes) | 20:29 |
| corvus | (and that's the only other "user" data we have, besides the gitea database) | 20:29 |
| corvus | yay i have a root shell on the inner-vm | 20:36 |
| corvus | okay, i can see that it has a private interface with an ip that matches what nova thinks it should have | 20:37 |
| corvus | so i'm running out of ideas for why i can't connect | 20:38 |
| clarkb | is sshd running? | 20:39 |
| corvus | yes | 20:39 |
| clarkb | and maybe we need haveged for entropy? | 20:39 |
| corvus | (and 'ssh localhost' inside the vm works) | 20:39 |
| clarkb | ok I think I've restored gitea06 at this point | 20:39 |
| corvus | and... sshing from outside the vm now works | 20:39 |
| corvus | sigh | 20:39 |
| clarkb | https://38.108.68.250:3000 the gitea data seems there, just lacking the repo data | 20:40 |
| clarkb | if we get https://review.opendev.org/#/c/667717/ in I'll push up a change to replicate to gitea06 now | 20:40 |
| clarkb | er actually maybe we've been replicating all along | 20:41 |
| clarkb | so dns update is all that is required | 20:41 |
| clarkb | ya should just be dns | 20:42 |
| clarkb | corvus: was the only change you made adding a password? | 20:42 |
| corvus | clarkb: i stopped the vm, added a password to its disk image, started it, logged in on the console, and ran 'ssh localhost'. and all that took a while (so count "waited a while" as a difference too) | 20:43 |
| *** altlogbot_2 has quit IRC | 20:43 | |
| corvus | so the biggest differences seem to be: 1) it booted a second time; 2) 'ssh localhost'; 3) more time | 20:44 |
| *** altlogbot_3 has joined #openstack-infra | 20:44 | |
| clarkb | can you ssh from the host node too? | 20:45 |
| clarkb | that should rule out any problem with network connectivity | 20:45 |
| clarkb | I would not be surprised if 3) is the answer | 20:46 |
| *** jsmdk has quit IRC | 20:46 | |
| clarkb | maybe we can check dmesg for rough idea of how long it takes to boot up fully | 20:46 |
| *** kjackal has quit IRC | 20:47 | |
| corvus | clarkb: yes i can ssh from the host node -- but am i missing something subtle about your question? (sshing from the host node into the inner vm is the objective -- the thing that wasn't working earlier and now works) | 20:47 |
| corvus | or were you just confirming you understood my haphazard updates? :) | 20:48 |
| clarkb | corvus: yes, the 'ssh localhost' comment made me wonder if you hda only tested ssh to localhost in the nested vm from its console. So clarifying you also tested ssh from outer vm to inner vm worked | 20:49 |
| corvus | gotcha. yeah, i confirmed that ssh-from-host failed, poked around, ran ssh-from-inner-vm, that worked, then checked ssh-from-host again and it worked | 20:51 |
| corvus | i'm skeptical that it's time alone though, since nodepool waits 10 minutes and this has been so reliably failing | 20:51 |
| corvus | clarkb: journalctl claims that sshd was listening before the system even finished booting | 20:52 |
| clarkb | hrm maybe we can take the old original image, edit it with a passwd user and do a new fresh boot on it? | 20:52 |
| corvus | and before i logged in on the console | 20:52 |
| clarkb | if that reproduces the error then we've got a first boot issue maybe | 20:53 |
| corvus | ok, i'll nova delete this instance, edit the glance image, then start up nodepool to let it recreate | 20:53 |
| corvus | guestmount is slow | 20:55 |
| *** altlogbot_3 has quit IRC | 21:00 | |
| *** Goneri has quit IRC | 21:00 | |
| corvus | hrm. i guess i didn't edit the correct image... i probably edited the version in glance, not the cached nova copy | 21:02 |
| corvus | oh well... i'll try the second-boot experiment still | 21:02 |
| *** tosky has joined #openstack-infra | 21:02 | |
| fungi | okay, back and taking a look at the mirror config on executors as it pertains to tumbleweed | 21:02 |
| clarkb | fungi: https://review.opendev.org/#/c/667717/ would help with gitea06 work | 21:03 |
| corvus | clarkb: on the second boot, i saw openssh key generation, and ssh started accepting connections right around the time the boot finished | 21:03 |
| clarkb | hrm it should generate the key on the first boot | 21:04 |
| *** altlogbot_0 has joined #openstack-infra | 21:04 | |
| *** pcaruana has quit IRC | 21:05 | |
| corvus | in the old devstack plugin config, i don't see any differences between the xenial and bionic configurations | 21:07 |
| corvus | wait | 21:09 |
| corvus | http://logs.openstack.org/14/629414/9/check/nodepool-functional-py35-ubuntu-src/df3d204/controller/nodepool/nodepool-list.txt | 21:09 |
| corvus | that says "building" for bionic | 21:09 |
| openstackgerrit | Merged opendev/zone-opendev.org master: Update gitea06 IP address https://review.opendev.org/667717 | 21:09 |
| corvus | that's a job that passed | 21:09 |
| corvus | well, it looks like the check script did ssh to the right ip address... so maybe that's just the last listing before it switched to ready | 21:11 |
| *** rfarr has joined #openstack-infra | 21:14 | |
| *** rfarr__ has quit IRC | 21:15 | |
| corvus | clarkb, fungi, ianw: do you have any ideas about how to proceed? | 21:16 |
| corvus | also, are there any other dib maintainers who might be interested in this? | 21:16 |
| clarkb | corvus: do we have the boot log for the first boot? Curious if that indicates why it might not be generating the ssh host keys then | 21:16 |
| clarkb | but I think that is likely the thread to pull on | 21:17 |
| clarkb | is it because of those early errors you found before? | 21:17 |
| corvus | clarkb: not any more, i should be able to repeat the process | 21:17 |
| corvus | clarkb: i saw no errors | 21:17 |
| fungi | i do find it strange that you saw ssh host key generation logged during the *second* boot. that suggests the host keys were removed on or never created during the first boot | 21:19 |
| clarkb | oh to follow up on db restore was slow with docker compose exec, using docker exec was really fast | 21:20 |
| corvus | i saw key generation during the first boot this time | 21:20 |
| fungi | i believe for debian's openssh packages (which ubuntu uses) host keys are generated if missing at service start time (which usually happens at package installation, but since we presumably install it during image creation we have to wipe them) | 21:20 |
| corvus | (which is not to say that it wasn't there the last time, just that i didn't see it) | 21:20 |
| fungi | i wonder if it happens at every boot, indicating the keys aren't being written or are being written to the wrong location or are being deleted after write or... | 21:21 |
| corvus | first boot: http://paste.openstack.org/show/753471/ | 21:21 |
| clarkb | and as expected ssh continues to fail ? | 21:21 |
| corvus | clarkb: correct, it's been 30+ seconds since it finished booting and ssh says 'no route to host' | 21:21 |
| corvus | i'll reboot now and get the log from the second boot | 21:22 |
| fungi | firewall rules allowing 22/tcp? | 21:22 |
| corvus | the iptables is not installed on the system | 21:22 |
| corvus | the iptables command | 21:22 |
| corvus | (on the inner vm, to be clear) | 21:23 |
| *** ianychoi has quit IRC | 21:23 | |
| fungi | what ip address is it expected to listen on from the perspective of where you're trying to initiate the client? | 21:23 |
| clarkb | ok I have a hunch. the glean stuff that configures the interfaces runs very late according to that paste | 21:23 |
| clarkb | it seems that network manager starts before we write out the network configuration files | 21:23 |
| corvus | fungi: i don't understand the question | 21:24 |
| clarkb | second boot could be working bceause those files are already written then the order of glean vs network manager doesn't matter | 21:24 |
| clarkb | now, I thought we were only network managering on fedora and centos | 21:24 |
| fungi | simpler version of the question: what ip address are you trying to ssh to? | 21:24 |
| clarkb | so that may be the difference here? | 21:24 |
| *** Lucas_Gray has joined #openstack-infra | 21:24 | |
| *** pkopec has quit IRC | 21:24 | |
| fungi | the address for which you're getting "no route to host" errors | 21:25 |
| corvus | on the second boot, i get 'no route to host' during the first part, 'connection refused' during the last 20 seconds of the sequence, and successful connection right around the time the login prompt appears, which is just before the explosion of glean.sh messages. 1 sec and i'll paste the log | 21:26 |
| corvus | second boot: http://paste.openstack.org/show/753472/ | 21:26 |
| corvus | fungi: 172.24.5.65 which i got from: | 590e9aed-4151-4d8e-919f-0ec43bd855b7 | test-image-devstack-0000000011 | ACTIVE | private=10.1.0.44, fd79:6010:8c34:0:f816:3eff:fea3:ddb, 172.24.5.65 | test-image-1561575944 | nodepool-512 | | 21:26 |
| corvus | fungi: i believe that's a fip | 21:26 |
| clarkb | corvus: I think the way to test my hunch is to do first boot, log in via console, restart networking via systemctl then test ssh (do this after the glean stuff is logged) | 21:27 |
| clarkb | if that works then I think it is a dependency ordering issue in the bootup | 21:27 |
| fungi | okay. so the "no route to host" is expected if the fip isn't created or maybe if the address to which it's translating isn't bound to anything yet. "connection refused" would probably indicate no sshd listening at that address even though it's reachable | 21:27 |
| *** ianychoi has joined #openstack-infra | 21:28 | |
| corvus | clarkb: i have a hunch as to how to explain your hunch in relation to the fact that this is working on the current jobs -- the current jobs build and launch 3 images + vms -- is it possible that contention is so high during that that glean runs in a different order? | 21:28 |
| clarkb | corvus: yes | 21:29 |
| fungi | if openssh is configured to listen on :: then it should be fine to start even when only linklocal addresses exist and before the routable interfaces get addresses assigned | 21:29 |
| corvus | clarkb: okay, i'll destroy this vm, try to find the right glance image to edit, add the password, then boot another | 21:29 |
| clarkb | fungi: ya my hunch is that routable interfaces don't get assigned because the config is written after things check the config | 21:29 |
| corvus | clarkb, fungi: with that, i should also be able to examine the sshd config and see what it's sposed to be listening on | 21:29 |
| fungi | yep | 21:30 |
| fungi | to both of you | 21:30 |
| clarkb | corvus: also check ifconfig/ip addr prior to restarting networking | 21:30 |
| corvus | heh, anyone know where devstack-nova keeps its cached disk images? :) | 21:30 |
| fungi | ianw: ^ if you're around yet? | 21:31 |
| * fungi has no idea | 21:32 | |
| corvus | oh, maybe it's the *glance* cache | 21:33 |
| corvus | i have found another copy of the image which i haven't edited yet there, i'll try that :) | 21:34 |
| corvus | booting now | 21:34 |
| corvus | okay, that clearly wasn't the right image either | 21:37 |
| fungi | on an unrelated note, anybody happen to know where we configure opensuse to talk to our mirrors for yast or zypper or whatever? | 21:37 |
| fungi | zuul site vars on the executors? | 21:38 |
| corvus | fungi: that's where we set the info, yes | 21:39 |
| fungi | or dib elements? | 21:39 |
| fungi | nodepool ready scripts are no longer a thing, right? | 21:39 |
| corvus | fungi: then there's a configure-mirrors role in zuul-jobs which uses the values set by the zuul site vars | 21:39 |
| *** diablo_rojo has joined #openstack-infra | 21:40 | |
| corvus | fungi: the only dib involvement is there is an element for the purpose of using a mirror when building images, but assuming you're working on issues related to "regular jobs", that shouldn't come into play (only if you're looking into issues building our images, or issues with nodepool/dib/glean test jobs) | 21:40 |
| *** panda has quit IRC | 21:42 | |
| clarkb | re gitea06 the problem I'm facing now is gitea06 has a different hostkey (because I didn't copy those from the old host). I accepted the new hostkeys on review01.opendev.org but that doesn't seem ot have updated what the replication plugin sees as the hostkey | 21:42 |
| fungi | yeah, looking for where to differentiate the url to opensuse tumbleweed mirrors (which are now a caching proxy) vs opensuse leap mirrors | 21:42 |
| clarkb | anyone know if that is something cached in memory or if there is a file I can update for the replication plugin? Otherwise I guess I can try to copy the old host keys | 21:43 |
| corvus | clarkb: i don't know | 21:43 |
| fungi | clarkb: the gerrit2 user's known_hosts list | 21:43 |
| clarkb | fungi: ya that is what I updated | 21:43 |
| fungi | is what you updated, right? | 21:43 |
| clarkb | and I used -p222 | 21:44 |
| fungi | may also need a gerrit restart to notice? | 21:44 |
| clarkb | ya before we do that I'm thinking I should restore the old hostkeys and see if that fixes it | 21:44 |
| fungi | or a hot reconfig of the replication plugin, which i think we haven't enabled? | 21:45 |
| *** panda has joined #openstack-infra | 21:45 | |
| fungi | yeah, using old hostkeys is likely simpler | 21:45 |
| clarkb | ya working on that now | 21:46 |
| *** sreejithp has quit IRC | 21:48 | |
| corvus | clarkb, fungi: i'm not sure how to get that console login on first boot. i'm inclined to say that we've tracked this down to the point where we think that the problem is pre-existing but doesn't manifest due to the busier environment of the current jobs. i'd like to mark this job as non-voting and ask that someone else find a solution. any objections? | 21:49 |
| fungi | no objection here | 21:49 |
| fungi | unless there's something we actually need to test specific to those platforms it seems okay to punt on them and focus on the ones we can test | 21:50 |
| corvus | (i think this fairly demonstrates the entire point of this exercise -- that nodepool maintainers are not the right people to be maintaining these jobs) | 21:50 |
| corvus | fungi: well, i mean, opendev uses bionic quite a bit, so i feel certain that some of us are going to want that to be tested | 21:51 |
| *** tdasilva_ has joined #openstack-infra | 21:52 | |
| fungi | yes, but i don't think it demonstrates a problem with the jobs themselves | 21:52 |
| corvus | right. the opposite, i hope :) | 21:52 |
| *** tdasilva has quit IRC | 21:54 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Switch functional testing to a devstack consumer job https://review.opendev.org/665023 | 21:55 |
| clarkb | ok ssh host keys should've been updated (I stopped the openssh container, moved new old keys aside, moved old old keys inplace, started the ssh container) but gerrit is still complaining | 21:57 |
| openstackgerrit | James E. Blair proposed openstack/diskimage-builder master: Replace nodepool func jobs https://review.opendev.org/667221 | 21:57 |
| clarkb | I'm going to do a docker-compose restart just to ensure there isn't anything weird with stop start using just docker | 21:58 |
| corvus | okay, my current expectation is that all 3 of those patches should work now. we'll see. | 21:59 |
| clarkb | that doesn't seem to have helped. The other idea I've got is to remove gitea06.opendev.org from our replication config then add it back in again | 21:59 |
| clarkb | since we do reload the config when it changes on disk | 21:59 |
| corvus | clarkb: you mean manually remove it because gerrit watches the file? | 21:59 |
| clarkb | corvus: yes | 21:59 |
| corvus | seems reasonable | 22:00 |
| fungi | yeah, i agree that's worth a try | 22:00 |
| clarkb | basically comment that block out. Let gerrit notice. Then uncomment the block and see if it checks the host key again | 22:00 |
| clarkb | ok doing that now | 22:00 |
| *** rcernin has joined #openstack-infra | 22:00 | |
| corvus | i'm going to afk for some errands | 22:01 |
| clarkb | looks like I'll have to kill the existing tasks that are in a retry loop so doing that before I add gitea06 back to the config | 22:02 |
| clarkb | nevermind all the tasks went away on their own | 22:03 |
| clarkb | adding gitea06 back now | 22:03 |
| *** sgw has joined #openstack-infra | 22:04 | |
| *** michael-beaver has quit IRC | 22:04 | |
| *** Lucas_Gray has quit IRC | 22:05 | |
| clarkb | in the replication log I see it cancelling the tasks | 22:05 |
| clarkb | but it is back to failing again so it must have that value cached | 22:05 |
| clarkb | though it should've cached the values that are there now? | 22:06 |
| sgw | mordred: clarkb: Hi guys another dumb'ish PBR question, can pbr be used with non-pythonic packages, say a C library or a C-based systemd service ?? adding setup.py and setup.cfg seems like it would cause confusions, I guess I am looking for a way to extract the same info for non-pythonic packages | 22:07 |
| fungi | clarkb: you undid/redid the known_hosts acceptance after you restored the old host key? | 22:07 |
| clarkb | fungi: yes | 22:08 |
| clarkb | fungi: and ssh -p 222 git@gitea06.opendev.org as gerrit2 works fine | 22:08 |
| fungi | sgw: pbr is very much just focused on being a python solution (partly because if its adherence to pep 440 version standards) | 22:08 |
| fungi | sgw: other language ecosystems would need their own analogues of pbr | 22:08 |
| *** xek__ has quit IRC | 22:10 | |
| clarkb | any objection to run gerrit flush-caches --cache sshkeys | 22:11 |
| clarkb | I wonder if the plugin uses the same cache tooling as gerrit itself | 22:11 |
| fungi | sgw: basically the typical way to version a python package kept under revision control was to commit a change to the metadata configuration indicating the version you were about to tag, and then tag that change with the same version... racy double-entry often leading to mistakes. pbr was written as a solution to that problem for python packages (by hooking into internal implementation details of the | 22:12 |
| fungi | python package building libraries), but i can imagine a similar solution might be warranted for other language ecosystems | 22:12 |
| sgw | fungi: thanks, I will do some googling, I was hoping that we could reuse the tagging/git walking as a general solution, since package versioning is close enough | 22:12 |
| *** eharney has joined #openstack-infra | 22:12 | |
| fungi | clarkb: go for it. you could try to flush --all for that matter | 22:12 |
| clarkb | fungi: that didn't work | 22:15 |
| clarkb | rereading the replication plugin docs I think we may have to restart gerrit :/ | 22:15 |
| *** weifan has quit IRC | 22:15 | |
| clarkb | though what I don't understand is why it isn't working after setting the host key to the old hostkey | 22:15 |
| clarkb | maybe it is paying attention to the ip address? | 22:16 |
| fungi | sgw: where i think you're likely to find a disconnect is that pbr is focused on package versioning. python doesn't have a concept of module versions any more than c does (unless you're talking about libc-style shared object versions but that mostly comes down to file naming?). it's how you package those c builds which would be sensitive to versioning i would expect | 22:16 |
| fungi | clarkb: tcpdmp ought to tell you | 22:16 |
| *** rfarr has quit IRC | 22:17 | |
| sgw | fungi: right it's more about the package versions than the libc numbering | 22:17 |
| clarkb | tcpdump shows it hitting the new gitea06 | 22:18 |
| *** rfarr has joined #openstack-infra | 22:18 | |
| clarkb | hrm I accepted ecdsa keys I think | 22:18 |
| clarkb | do I need to accept rsa? | 22:18 |
| fungi | sgw: in that case it's probably a question of whether someone's written something which can auto-version rpm and deb packages and the like based on git tags | 22:18 |
| sgw | generating that tag-<dev>git-offset value for a package for python and non-python packages in a consistent fashion | 22:18 |
| sgw | fungi: yeah, well I guess I am kind of working on that ;-) | 22:19 |
| *** rcernin has quit IRC | 22:19 | |
| fungi | for deb packages i'd expect a debhelper plugin maybe (though that probably gets into editing contents of the debian/changelog file on the fly). there's presumably something similar in rpm specfiles | 22:20 |
| *** rcernin has joined #openstack-infra | 22:20 | |
| sgw | I am playing with Zuul and OBS based on the existing work that the rpm-packaging project does, but without jenkins | 22:20 |
| *** rlandy is now known as rlandy|bbl | 22:20 | |
| sgw | fungi: not that I have found yet, version info is mostly hardwired, I would need to template the specfile and then fill it automagically | 22:21 |
| fungi | it's... not a trivial problem to solve ;) | 22:21 |
| *** goldyfruit has joined #openstack-infra | 22:21 | |
| fungi | clarkb: it likely is only trying rsa | 22:22 |
| fungi | clarkb: given what we know about the odd elliptic curve problems in the mina-ssh implementation in that gerrit version | 22:22 |
| sgw | fungi: right, having dealt with this in the Yocto Project / Embedded world, versioning is hard! | 22:23 |
| *** rfarr has quit IRC | 22:23 | |
| clarkb | fungi: ya that seem to have been it | 22:24 |
| clarkb | I removed the ecdsa hosts keys for gitea06 and added rsa and now it is happy | 22:24 |
| fungi | clarkb: oh, good, (bizarre) problem solved | 22:25 |
| clarkb | its still not replicating successfully but ya that problem is sovled | 22:26 |
| clarkb | I think I may need to manually create some git repos | 22:26 |
| clarkb | we didn't ask gitea to create the projects we only copied them over via the db | 22:27 |
| clarkb | so there are no on disk repos | 22:27 |
| fungi | ahh, yep, gerrit won't be able to do that like it did via ssh | 22:28 |
| fungi | (presumably) | 22:28 |
| fungi | and i guess there's still an on-disk component? | 22:28 |
| fungi | should we rsync the repos over from another gitea server? | 22:29 |
| *** weifan has joined #openstack-infra | 22:29 | |
| clarkb | possibly. I'm going to create a system-config bare repo and see if that fixes it | 22:29 |
| clarkb | https://gitea06.opendev.org:3000/opendev/system-config that seems to have done it | 22:32 |
| clarkb | ok so I can either git init a bunch of bare repos or we can copy them over from anothre host. I think my one concern with git initing a bunch of repos is that gitea may use different options than a simple git init --bare | 22:32 |
| clarkb | corvus: fungi ^ thoughts on that? | 22:32 |
| fungi | seems like rsync would be less work and possibly less load on gerrit | 22:33 |
| fungi | clarkb: we seem to use a consistent var for the repo base for all mirrors, any idea how to make it tack on :8080 for tumbleweed but not leap? https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/tasks/mirror/Suse.yaml#L8 | 22:33 |
| clarkb | fungi: ya the one gotcha with rsync is that you can't safely rsync an in use git repo | 22:33 |
| fungi | mm... as in can't copy *from* an in-use repo because it may contain tempfiles? | 22:35 |
| clarkb | fungi: or things could be in the middle of a repack | 22:35 |
| clarkb | its safe to do git actions in parallel but not rsync aiui | 22:35 |
| clarkb | fungi: I think you update that line to be {{ package_mirror }}:8080 | 22:36 |
| *** ekultails has quit IRC | 22:36 | |
| clarkb | (with the appropriate path) | 22:36 |
| fungi | huh, except that's in zuul-jobs | 22:36 |
| fungi | it's not specific to opendev's deployment | 22:37 |
| clarkb | oh | 22:37 |
| clarkb | so where is package_mirror defined | 22:37 |
| *** mattw4 has quit IRC | 22:37 | |
| clarkb | before I forget: config-core can we hold off on merging new project requests until we have gitea06 happy? its sort of in an inbetween state right now and would be good to make it happy before we add new projets that have to be reconciled there | 22:38 |
| * mnaser ack | 22:38 | |
| clarkb | fungi: https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/configure-mirrors/vars/Suse.yaml is where we define it | 22:39 |
| clarkb | fungi: so I think what we can do is override that package_mirror value where we call the config mirror role if on tumbleweed | 22:39 |
| clarkb | then we'll not violate zuul-jobs generalism and can have an override ourselves | 22:40 |
| clarkb | fungi: https://opendev.org/opendev/base-jobs/src/branch/master/playbooks/base/pre.yaml#L36-L37 is the bit to update | 22:40 |
| fungi | oh, yep. thanks! do a conditional value in there with jinja based on... the buildset name? | 22:42 |
| clarkb | probably based on whatever the os release ansible fact is | 22:43 |
| fungi | for mirror_fqdn | 22:43 |
| clarkb | or is that distribution name? something like that | 22:43 |
| *** diablo_rojo has quit IRC | 22:44 | |
| fungi | package_mirror: "{%- if ansible_distribution == 'openSUSE Tumbleweed' -%}http://{{ mirror_fqdn }}:8080/opensuse{%- endif -%}" | 22:45 |
| clarkb | I am rtfs'ing gitea and there is an admin dashboard function to reinit missing repos | 22:45 |
| fungi | something like that? | 22:45 |
| clarkb | fungi: ya | 22:45 |
| clarkb | but also rtfsing it seems to just do a git init --bare | 22:46 |
| clarkb | so I think I can either figure out how to login as an admin and click that button or just make a bash script to mkdir all the things | 22:46 |
| clarkb | corvus: ^ curious about what you think of those options when you get back. And I'm gonna step out for a few myself | 22:46 |
| fungi | seems like it ought to be fine. a worthy experiment anyway | 22:47 |
| clarkb | ya Ifigure its easy to loop over projects.yaml mkdir -p name.git && cd name.git && git init --bare | 22:47 |
| *** dchen has joined #openstack-infra | 22:48 | |
| *** armstrong has quit IRC | 22:51 | |
| corvus | clarkb, fungi: i also think either is fine. you should be able to log in as root using the password in bridge secrets if you wanted to go that route. | 22:51 |
| corvus | clarkb: https://gitea06.opendev.org:3000/user/login is the url | 22:52 |
| corvus | dirk: do you know what happened with these builds? http://logs.openstack.org/25/667225/11/check/dib-nodepool-functional-openstack-opensuse-150-src/da5452a/nodepool/builds/test-image-0000000001.log and http://logs.openstack.org/25/667225/11/check/dib-nodepool-functional-openstack-opensuse-tumbleweed-src/49446e4/nodepool/builds/test-image-0000000001.log ? | 22:59 |
| *** tkajinam has joined #openstack-infra | 23:05 | |
| *** diablo_rojo has joined #openstack-infra | 23:07 | |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Test setting OpenSUSE Tumbleweed mirror proxy port https://review.opendev.org/667750 | 23:09 |
| openstackgerrit | Jeremy Stanley proposed opendev/base-jobs master: Set OpenSUSE Tumbleweed mirror proxy port https://review.opendev.org/667751 | 23:09 |
| fungi | config-core: 667750 should be safe to merge asap if it looks okay, and then i'll set up some dnm depends-on changes to make sure it's not utterly broken before i un-wip 667751 | 23:11 |
| *** tosky has quit IRC | 23:12 | |
| fungi | dirk: ^ that's probably relevant to you too since you got the first part of that transition committed | 23:13 |
| fungi | roman_g: ^ we're working to get it fixed | 23:14 |
| *** sgw has quit IRC | 23:16 | |
| corvus | fungi: my guess is that won't work because package_mirror is set in a vars file, not a defaults file | 23:18 |
| fungi | oh :/ | 23:19 |
| fungi | should we extend the zuul-jobs variable with a port default? | 23:19 |
| corvus | fungi: i'm happy to +2 it to verify, but i think it's unlikely enough that i think local testing will save time -- maybe make a copy of the configure-mirrors role and change it to just print out values instead of writing files, and see what happes | 23:19 |
| fungi | yeah, i have a feeling you're right that it won't work | 23:20 |
| fungi | what's the recommended approach? | 23:20 |
| fungi | we could just revert 660125 or drop the tumbleweed nodes | 23:20 |
| corvus | fungi: honestly, what we need to do is throw out configure-mirrors and build a new system from scratch which is basically structured data for each kind of mirror a site supports (mordred would agree with this; i think he just hasn't had time to write down what a new approach would look like) | 23:21 |
| fungi | seems like 660125 only addressed half of the problem, but dropped the working mirror at the same time it added the proxy | 23:22 |
| fungi | however, if it's taken 5 weeks to find out tumbleweed nodes can't install packages now... | 23:22 |
| corvus | fungi: given that, i'm not sure further calcification of configure-mirrors is worth it... maybe reverting 660125 until we can replace configure-mirrors is the best use of everyone's time? is continuing to use the afs mirror okay? | 23:22 |
| fungi | yeah, i'm leaning toward at least a partial revert of 660125 (don't have to remove the proxy, but still put the afs mirror back) | 23:23 |
| corvus | (the commit message for 660125 reads like it's an efficiency change, not a fix for an existing problem) | 23:23 |
| fungi | possible this was done to conserve space when the 15.1 mirror was added | 23:24 |
| fungi | i'll check filesystem utilization first | 23:25 |
| *** igordc has quit IRC | 23:25 | |
| fungi | looks like it's only 53% used with 3 versions in place | 23:25 |
| corvus | and i actually don't think the configure-mirrors thing needs to be too hard -- i think we just need to define a data structure, create a "configure-mirrors2" role that uses it, then drop our use of configure-mirrors, and tell the world to do the same before we remove it. | 23:25 |
| fungi | so ought to be enough space | 23:26 |
| fungi | configure-mirrors-harder | 23:26 |
| corvus | the big advantage of the new approach will be: 1) we can address problems like this :) 2) other roles can use the mirror information (eg, dib testing roles or nested docker containers can pick out just the mirror info they need) | 23:26 |
| *** auristor has quit IRC | 23:28 | |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Revert "Move openSUSE Tumbleweed into a caching mirror instead" https://review.opendev.org/667754 | 23:29 |
| *** sthussey has quit IRC | 23:29 | |
| *** goldyfruit has quit IRC | 23:30 | |
| clarkb | clicking the gitea site admin button was easy so I did that | 23:31 |
| clarkb | that is actually a useful little button for this redeployment process | 23:31 |
| ianw | i think it was more that rsync kept breaking in magic ways ... and we don't do a good job of exposing that to non-root | 23:31 |
| clarkb | I hvae triggered a full rereplication of gitea06 | 23:31 |
| corvus | fungi: i'm imagining that our site vars would end up looking like this: https://etherpad.openstack.org/p/P8MVuDozNG | 23:32 |
| fungi | corvus: thanks, i tried to encapsulate those recommendations within a comment in the revert | 23:32 |
| *** goldyfruit has joined #openstack-infra | 23:32 | |
| corvus | then roles can do things like "{{ if 'opensuse' in zuul_site_mirror }}" to say "if the zuul site has an opensuse mirror... else just use upstream" | 23:32 |
| ianw | fungi: ^ we probably could expose the mirroring logs on mirror-update? | 23:32 |
| ianw | or maybe a better idea would be to put the mirroring logs into a volume and expose that | 23:33 |
| fungi | ianw: i can't think of anything sensitive which could possibly appear in those (it's not like kerberos keys are going to get splatted out by cron) so that sounds fie | 23:33 |
| fungi | fine | 23:33 |
| ianw | yeah, just that host *does* have a lot of key material in various ways, so keeping apache off it would be good | 23:33 |
| fungi | also a log volume sounds like a very afs solution ;) | 23:34 |
| fungi | wfm. easy enough to serve from files.o.o or similar if we want | 23:34 |
| clarkb | redirects seem to work too | 23:34 |
| fungi | clarkb: awesome | 23:35 |
| *** aaronsheffield has quit IRC | 23:36 | |
| corvus | fungi: in fact, it will automatically be served from files.o.o | 23:37 |
| openstackgerrit | Clark Boylan proposed opendev/system-config master: Put gitea06 back in the rotation https://review.opendev.org/667759 | 23:37 |
| corvus | http://files.openstack.org/ is /afs/openstack.org | 23:38 |
| clarkb | infra-root ^ I'm happy to approve and watch that tomorrow once replication is done. Reviews now would be great | 23:38 |
| fungi | corvus: fair point, no need for any new vhost at all | 23:39 |
| corvus | clarkb: did you do a targeted replication to that host? | 23:39 |
| clarkb | corvus: yup that is in progress (change is WIP until that is compltee) | 23:39 |
| clarkb | I ran replication start --url gitea06.opendev.org | 23:39 |
| corvus | cool -- and yeah i see 1291 tasks most with gitea06 in the name | 23:39 |
| clarkb | seemed to queue up all the things | 23:39 |
| fungi | yeah, i figured we don't want to go approving that until the repos are in sync again | 23:39 |
| *** rh-jelabarre has quit IRC | 23:39 | |
| clarkb | then tomorrow I might fiddle with having ansible triggerthe make all repos that are missing button | 23:42 |
| clarkb | we can probably run that in our normal ansible run which will remove a step from this deployment process | 23:42 |
| *** mriedem has quit IRC | 23:42 | |
| clarkb | (basically process becomes, restore db, wait for ansible to trigger repo creation, trigger replication, done) | 23:42 |
| *** slaweq has quit IRC | 23:50 | |
| *** icarusfactor has quit IRC | 23:51 | |
| clarkb | I think it is actually done replicating | 23:52 |
| clarkb | queue doesn't show any outstanding gitea06 entries and I even git fetched a change ref for neutron against that node (cloned neutron from that node too) | 23:53 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add OVH GRA1 opendev.org mirror https://review.opendev.org/667529 | 23:53 |
| clarkb | I have removed my WIP on the change above | 23:53 |
| *** auristor has joined #openstack-infra | 23:54 | |
| *** bobh has joined #openstack-infra | 23:54 | |
| clarkb | Do we want to add AAAA records for all of the gitea backends now that they ipv6? | 23:57 |
| *** weifan has quit IRC | 23:57 | |
| *** hwoarang has quit IRC | 23:57 | |
| ianw | infra-root: could i get a couple of eyes on https://review.opendev.org/667543 & https://review.opendev.org/667529 to get the ovh gra1 mirror into the system; i'll hand configure it with the kafs stuff (which just got a pull request upstream) and we can switch it at some point after | 23:59 |
| *** hwoarang has joined #openstack-infra | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!