| *** diablo_rojo has quit IRC | 00:01 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add ansible nodepool-builder https://review.opendev.org/710888 | 00:02 |
|---|---|---|
| *** threestrands has joined #openstack-infra | 00:03 | |
| *** mattw4 has quit IRC | 00:05 | |
| *** weifan has joined #openstack-infra | 00:05 | |
| *** weifan has quit IRC | 00:05 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: nodepool: Add nb01-test.opendev.org config file https://review.opendev.org/710700 | 00:06 |
| ianw | clarkb: ^ sorry just updated the needed-by link | 00:07 |
| *** slaweq has joined #openstack-infra | 00:11 | |
| *** pkopec has quit IRC | 00:12 | |
| *** zxiiro has joined #openstack-infra | 00:14 | |
| clarkb | k. I'll look closer in a few. Parenting dutiee calling currently | 00:15 |
| ianw | np | 00:16 |
| *** slaweq has quit IRC | 00:17 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: executor: do not blacklist host-vars for trusted context https://review.opendev.org/710890 | 00:24 |
| *** tetsuro has joined #openstack-infra | 00:42 | |
| *** tetsuro has quit IRC | 00:47 | |
| *** tetsuro has joined #openstack-infra | 00:47 | |
| *** diablo_rojo has joined #openstack-infra | 00:47 | |
| *** aedc has quit IRC | 00:50 | |
| *** owalsh has quit IRC | 00:54 | |
| *** aedc has joined #openstack-infra | 00:58 | |
| *** owalsh has joined #openstack-infra | 01:00 | |
| *** slaweq has joined #openstack-infra | 01:11 | |
| *** yolanda has quit IRC | 01:15 | |
| *** slaweq has quit IRC | 01:16 | |
| *** ijw has quit IRC | 01:25 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add ansible for nodepool hosts https://review.opendev.org/710888 | 01:31 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 01:31 |
| openstackgerrit | Merged zuul/zuul master: executor: do not blacklist host-vars for trusted context https://review.opendev.org/710890 | 01:39 |
| *** mtreinish has joined #openstack-infra | 01:42 | |
| *** jamesmcarthur has joined #openstack-infra | 01:45 | |
| *** cdearborn has quit IRC | 01:52 | |
| clarkb | ianw 710700 lgtm and maybe need to lamd that before 710888 will pass testing? | 02:02 |
| ianw | clarkb: yep, will be required, but still sorting out some other issues too | 02:02 |
| *** gyee has quit IRC | 02:08 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add ansible for nodepool hosts https://review.opendev.org/710888 | 02:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 02:11 |
| *** slaweq has joined #openstack-infra | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:13 | |
| openstackgerrit | Merged openstack/project-config master: nodepool: Add nb01-test.opendev.org config file https://review.opendev.org/710700 | 02:13 |
| *** yamamoto has joined #openstack-infra | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 02:17 |
| *** jamesmcarthur has joined #openstack-infra | 02:21 | |
| *** kaisers_ has quit IRC | 02:28 | |
| *** diablo_rojo has quit IRC | 02:33 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add ansible for nodepool hosts https://review.opendev.org/710888 | 02:36 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 02:36 |
| *** Goneri has quit IRC | 02:41 | |
| *** tbachman has joined #openstack-infra | 02:42 | |
| *** jamesmcarthur has quit IRC | 02:50 | |
| *** rcernin_ has joined #openstack-infra | 02:52 | |
| *** jamesmcarthur has joined #openstack-infra | 02:53 | |
| *** rcernin has quit IRC | 02:55 | |
| *** jamesmcarthur has quit IRC | 02:58 | |
| *** jamesmcarthur has joined #openstack-infra | 03:04 | |
| *** slaweq has joined #openstack-infra | 03:11 | |
| *** rlandy has quit IRC | 03:15 | |
| *** slaweq has quit IRC | 03:16 | |
| *** apetrich has quit IRC | 03:17 | |
| *** weifan has joined #openstack-infra | 03:22 | |
| *** weifan has quit IRC | 03:26 | |
| *** weifan has joined #openstack-infra | 03:28 | |
| *** weifan has quit IRC | 03:33 | |
| ianw | sigh, it looks like the scripts directory is embedded in too many places | 03:48 |
| *** jamesmcarthur has quit IRC | 03:50 | |
| *** lbragstad has quit IRC | 03:58 | |
| openstackgerrit | Ian Wienand proposed openstack/project-config master: Restore blank nodepool/scripts https://review.opendev.org/710900 | 04:01 |
| *** admcleod has quit IRC | 04:02 | |
| *** dave-mccowan has quit IRC | 04:04 | |
| *** raukadah is now known as chandankumar | 04:04 | |
| *** dSrinivas has joined #openstack-infra | 04:05 | |
| *** slaweq has joined #openstack-infra | 04:11 | |
| *** ricolin has quit IRC | 04:15 | |
| *** slaweq has quit IRC | 04:17 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] add ansible for nodepool hosts https://review.opendev.org/710888 | 04:45 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 04:45 |
| *** ociuhandu has joined #openstack-infra | 04:55 | |
| *** ykarel|away is now known as ykarel | 04:58 | |
| *** admcleod has joined #openstack-infra | 04:59 | |
| *** ociuhandu has quit IRC | 04:59 | |
| *** jamesdenton has quit IRC | 05:03 | |
| *** jamesdenton has joined #openstack-infra | 05:03 | |
| *** slaweq has joined #openstack-infra | 05:11 | |
| *** slaweq has quit IRC | 05:16 | |
| *** rcernin_ has quit IRC | 05:24 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 05:28 |
| *** vesper has quit IRC | 05:34 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** vesper11 has joined #openstack-infra | 05:37 | |
| *** lmiccini has joined #openstack-infra | 06:06 | |
| *** igordc has joined #openstack-infra | 06:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add initial Ansible for nodepool hosts https://review.opendev.org/710888 | 06:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] deploy nodepool-builder container https://review.opendev.org/710891 | 06:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add install zookeeper role; use for nodepool-builder testing https://review.opendev.org/710908 | 06:11 |
| *** slaweq has joined #openstack-infra | 06:11 | |
| AJaeger | ianw: want me to +A the scripts directory change? | 06:12 |
| *** igordc has quit IRC | 06:13 | |
| ianw | AJaeger: probably thanks; maybe if you could recheck a system-config change and make sure it fixes the puppet job too? (i'll have to run out soon) | 06:13 |
| ianw | i just don't think anyone wants to go through all the puppet and remove the various places it gets references; might as well push on the with the container/ansible efforts for the builders then we can just rm all the puppet stuff | 06:14 |
| *** slaweq has quit IRC | 06:16 | |
| AJaeger | yeah, might be easier to kill puppet... | 06:17 |
| AJaeger | ianw: approved | 06:17 |
| ianw | thanks! | 06:17 |
| ianw | i'm not too sure on podman-compose ... but for one day i think i've made some decent progress on the nodepool-builder container deployment anyway | 06:18 |
| *** toabctl has quit IRC | 06:18 | |
| AJaeger | cool! | 06:19 |
| *** toabctl has joined #openstack-infra | 06:22 | |
| openstackgerrit | Merged openstack/project-config master: Restore blank nodepool/scripts https://review.opendev.org/710900 | 06:32 |
| *** diablo_rojo has joined #openstack-infra | 06:33 | |
| *** matt_kosut has quit IRC | 06:44 | |
| *** kaisers has joined #openstack-infra | 06:56 | |
| *** matt_kosut has joined #openstack-infra | 07:06 | |
| *** matt_kosut has quit IRC | 07:08 | |
| *** matt_kos_ has joined #openstack-infra | 07:08 | |
| *** diablo_rojo has quit IRC | 07:09 | |
| *** slaweq has joined #openstack-infra | 07:11 | |
| *** rcernin has joined #openstack-infra | 07:13 | |
| *** slaweq has quit IRC | 07:16 | |
| *** hwoarang has quit IRC | 07:18 | |
| *** yolanda has joined #openstack-infra | 07:21 | |
| *** hwoarang has joined #openstack-infra | 07:21 | |
| *** ijw has joined #openstack-infra | 07:30 | |
| *** gshippey has quit IRC | 07:33 | |
| *** ricolin has joined #openstack-infra | 07:33 | |
| *** ykarel is now known as ykarel|lunch | 07:39 | |
| *** matt_kos_ has quit IRC | 07:41 | |
| *** matt_kosut has joined #openstack-infra | 07:41 | |
| *** happyhemant has joined #openstack-infra | 07:50 | |
| *** slaweq has joined #openstack-infra | 07:52 | |
| AJaeger | ianw: puppet run succeeds now on system-config | 07:54 |
| *** jcapitao has joined #openstack-infra | 08:10 | |
| *** jcapitao has quit IRC | 08:11 | |
| *** jcapitao has joined #openstack-infra | 08:11 | |
| *** dchen has quit IRC | 08:13 | |
| *** njohnston has quit IRC | 08:14 | |
| *** lmiccini has quit IRC | 08:19 | |
| *** tosky has joined #openstack-infra | 08:20 | |
| *** yamamoto has quit IRC | 08:22 | |
| *** roman_g has joined #openstack-infra | 08:25 | |
| *** lmiccini has joined #openstack-infra | 08:25 | |
| *** iurygregory has joined #openstack-infra | 08:26 | |
| *** yamamoto has joined #openstack-infra | 08:27 | |
| *** hwoarang has quit IRC | 08:27 | |
| *** hwoarang has joined #openstack-infra | 08:28 | |
| *** tesseract has joined #openstack-infra | 08:30 | |
| *** roman_g has quit IRC | 08:35 | |
| *** iurygregory has quit IRC | 08:36 | |
| *** jpena|off is now known as jpena | 08:41 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** iurygregory has joined #openstack-infra | 08:41 | |
| *** rcernin has quit IRC | 08:41 | |
| *** ralonsoh has joined #openstack-infra | 08:42 | |
| *** sshnaidm|pto has quit IRC | 08:45 | |
| *** ysastri has joined #openstack-infra | 08:48 | |
| *** sshnaidm|pto has joined #openstack-infra | 08:59 | |
| *** apetrich has joined #openstack-infra | 09:00 | |
| *** ccamacho has quit IRC | 09:02 | |
| *** FlorianFa has quit IRC | 09:05 | |
| *** ociuhandu has joined #openstack-infra | 09:06 | |
| *** threestrands has quit IRC | 09:09 | |
| *** matt_kos_ has joined #openstack-infra | 09:10 | |
| *** matt_kosut has quit IRC | 09:10 | |
| *** ociuhandu has quit IRC | 09:12 | |
| *** hashar has joined #openstack-infra | 09:14 | |
| *** udesale has joined #openstack-infra | 09:14 | |
| *** ociuhandu has joined #openstack-infra | 09:15 | |
| *** sshnaidm|pto is now known as sshnaidm | 09:17 | |
| *** tesseract has quit IRC | 09:25 | |
| *** tesseract has joined #openstack-infra | 09:26 | |
| *** iurygregory has quit IRC | 09:27 | |
| frickler | so we fixed the importlib-resources bug, but not the virtualenv bug it seems, but I fail to find the reasoning for that in the backlog | 09:29 |
| *** Lucas_Gray has joined #openstack-infra | 09:29 | |
| *** ociuhandu has quit IRC | 09:37 | |
| *** ysastri has quit IRC | 09:40 | |
| kevinz | Hi Infra, we meet a network problem here in linaro us, can anybody help to check why: https://zuul.opendev.org/t/openstack/build/b4eca2de0b0f465f90e483567ccb8896/log/kolla/build/000_FAILED_base.log | 09:40 |
| *** gfidente has joined #openstack-infra | 09:41 | |
| *** yamamoto has quit IRC | 09:42 | |
| *** zxiiro has quit IRC | 09:43 | |
| *** ijw has quit IRC | 09:43 | |
| *** ijw_ has joined #openstack-infra | 09:43 | |
| *** iurygregory has joined #openstack-infra | 09:43 | |
| *** derekh has joined #openstack-infra | 09:43 | |
| *** ociuhandu has joined #openstack-infra | 09:45 | |
| frickler | kevinz: that looks more like an kolla issue to me than an infra issue. maybe something related to docker and IPv6? or do you see similar failures for non-kolla jobs? otherwise best ask in the kolla channel | 09:53 |
| frickler | fwiw I tried connecting to those sites from the linaro-us mirror node and that doesn't see any issues | 09:53 |
| kevinz | hrw: ^^ | 09:54 |
| kevinz | frickler: thx | 09:54 |
| *** gshippey has joined #openstack-infra | 09:54 | |
| kevinz | I will trigger a devstack test to check | 09:55 |
| *** ociuhandu has quit IRC | 09:58 | |
| *** yamamoto has joined #openstack-infra | 09:58 | |
| *** dtantsur|afk is now known as dtantsur | 10:03 | |
| *** rcernin has joined #openstack-infra | 10:04 | |
| *** pkopec has joined #openstack-infra | 10:04 | |
| *** roman_g has joined #openstack-infra | 10:06 | |
| iurygregory | we still have the problem with py27 in CI? | 10:10 |
| iurygregory | https://18eaa349c9388c8b78c5-fa065b3d3badff7cb87ccafc6f749b5a.ssl.cf5.rackcdn.com/709146/2/check/openstack-tox-py27/8126af3/job-output.txt | 10:11 |
| *** ociuhandu has joined #openstack-infra | 10:25 | |
| *** ociuhandu has quit IRC | 10:27 | |
| *** ociuhandu has joined #openstack-infra | 10:27 | |
| frickler | iurygregory: yes, it seems so. https://review.opendev.org/710729 is my sample fix in reqs, I'll propose to do the same in the base job, but if you need fast progress, you could add that workaround within our project | 10:28 |
| sshnaidm | just fyi, seems like no space on device in https://nb02.openstack.org/ | 10:37 |
| *** rcernin has quit IRC | 10:37 | |
| *** rcernin has joined #openstack-infra | 10:38 | |
| *** ociuhandu has quit IRC | 10:39 | |
| *** ykarel|lunch is now known as ykarel | 10:40 | |
| openstackgerrit | Jens Harbott (frickler) proposed opendev/base-jobs master: Pin virtualenv in order to fix tox on py27 https://review.opendev.org/710967 | 10:41 |
| *** dSrinivas has quit IRC | 10:42 | |
| *** ccamacho has joined #openstack-infra | 10:52 | |
| *** ccamacho has quit IRC | 10:53 | |
| *** ccamacho has joined #openstack-infra | 10:54 | |
| *** openstackstatus has quit IRC | 10:59 | |
| *** tomaw_ has joined #openstack-infra | 11:00 | |
| *** arif-ali has quit IRC | 11:00 | |
| *** Lucas_Gray has quit IRC | 11:01 | |
| *** arif-ali has joined #openstack-infra | 11:04 | |
| *** roman_g has quit IRC | 11:05 | |
| *** sshnaidm has quit IRC | 11:07 | |
| *** tomaw has quit IRC | 11:08 | |
| *** sshnaidm has joined #openstack-infra | 11:12 | |
| *** ociuhandu has joined #openstack-infra | 11:12 | |
| *** sshnaidm has quit IRC | 11:13 | |
| *** sshnaidm has joined #openstack-infra | 11:13 | |
| iurygregory | frickler, no worries I can wait for the fix to avoid doing a workaround =) | 11:16 |
| *** rpittau is now known as rpittau|bbl | 11:17 | |
| *** hashar has quit IRC | 11:21 | |
| *** hashar has joined #openstack-infra | 11:22 | |
| *** trident has quit IRC | 11:46 | |
| *** david-lyle has joined #openstack-infra | 11:47 | |
| *** dklyle has quit IRC | 11:47 | |
| kevinz | frickler: hey, is it an easy way for me to login in the linaro-us vms to check the network reason? | 11:47 |
| *** trident has joined #openstack-infra | 11:49 | |
| frickler | kevinz: not directly I think, easiest way would be to hold a node for a job and then I or some other infra-root can add your ssh key to give you access | 11:49 |
| frickler | kevinz: do you want the devstack job or a kolla job? or both? | 11:50 |
| kevinz | both would be better | 11:50 |
| *** Lucas_Gray has joined #openstack-infra | 11:50 | |
| *** tbachman has quit IRC | 11:50 | |
| *** rcernin has quit IRC | 11:52 | |
| kevinz | frickler: ssh-key pub has been sent to you via irc | 11:53 |
| frickler | kevinz: o.k., should be able to catch the nodes for the currently running jobs on https://review.opendev.org/710970 and https://review.opendev.org/708317 | 11:53 |
| kevinz | thx a lot | 11:54 |
| *** Lucas_Gray has quit IRC | 11:55 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Cache getUser in Github connection https://review.opendev.org/710985 | 11:56 |
| kevinz | frickler: can not login at 2604:1380:4111:3e56:f816:3eff:fed6:f2d1, which user for devstack? | 11:56 |
| openstackgerrit | Sagi Shnaidman proposed openstack/project-config master: Add option to get ethX interfaces on centos-8 https://review.opendev.org/710986 | 11:56 |
| sshnaidm | cores, please review ^^ | 11:56 |
| *** hashar has quit IRC | 11:57 | |
| *** tetsuro has quit IRC | 11:57 | |
| frickler | kevinz: root, I just added your key. note that the job is still running, though | 11:58 |
| kevinz | frickler: thx, I login :D | 11:59 |
| frickler | kevinz: https://review.opendev.org/710970 passed, the no held node for that one :( otoh it probably means that yoctozepto's fix helped | 11:59 |
| frickler | s/the/thus | 12:00 |
| kevinz | frickler: OK, yes I login in another node, runing devstack, it looks that curl -L https://packages.grafana.com/gpg.key works very well | 12:04 |
| * yoctozepto reporting for duty | 12:05 | |
| yoctozepto | hrw: I think I will repropose as non-WIP then | 12:07 |
| frickler | yoctozepto: looks like your patch may indeed be the fix for the linaro issue that kevinz was looking at | 12:07 |
| kevinz | nice job | 12:08 |
| yoctozepto | then same to k-a and we are done | 12:11 |
| * yoctozepto was tired of red colour on those builds | 12:11 | |
| yoctozepto | ;D | 12:11 |
| frickler | yoctozepto: kevinz: do you still want to take a deeper look at the devstack node? seems the timeout isn't happening just because things are slow, there's also some nova/neutron related-looking errors in the logs | 12:13 |
| *** jcapitao is now known as jcapitao_lunch | 12:14 | |
| yoctozepto | frickler: /me letting kevinz have some fun | 12:14 |
| yoctozepto | I'm fixing k-a and moving on to $job | 12:14 |
| *** nicolasbock has joined #openstack-infra | 12:15 | |
| yoctozepto | kevinz, frickler: k-a fix https://review.opendev.org/710987 | 12:17 |
| yoctozepto | this one would normally fail -> https://zuul.opendev.org/t/openstack/stream/3a09d0a13ded42a2a3907014631eebfa?logfile=console.log | 12:18 |
| yoctozepto | (in case you want to look at it ;-) ) | 12:18 |
| *** Lucas_Gray has joined #openstack-infra | 12:20 | |
| *** rlandy has joined #openstack-infra | 12:24 | |
| *** Lucas_Gray has quit IRC | 12:31 | |
| *** jpena is now known as jpena|lunch | 12:32 | |
| *** aedc has quit IRC | 12:34 | |
| *** aedc has joined #openstack-infra | 12:35 | |
| *** jcoufal has joined #openstack-infra | 12:35 | |
| *** jcoufal has quit IRC | 12:48 | |
| *** aedc has quit IRC | 12:50 | |
| *** aedc has joined #openstack-infra | 12:51 | |
| *** yamamoto has quit IRC | 12:52 | |
| *** matt_kos_ has quit IRC | 12:56 | |
| *** matt_kosut has joined #openstack-infra | 12:57 | |
| *** matt_kosut has quit IRC | 13:02 | |
| *** rpittau|bbl is now known as rpittau | 13:04 | |
| *** yamamoto has joined #openstack-infra | 13:06 | |
| *** ociuhandu has quit IRC | 13:06 | |
| *** matt_kosut has joined #openstack-infra | 13:06 | |
| *** lbragstad has joined #openstack-infra | 13:07 | |
| *** ociuhandu has joined #openstack-infra | 13:07 | |
| *** tbachman has joined #openstack-infra | 13:08 | |
| *** matt_kosut has quit IRC | 13:08 | |
| *** matt_kosut has joined #openstack-infra | 13:08 | |
| *** Lucas_Gray has joined #openstack-infra | 13:11 | |
| *** jaosorior has joined #openstack-infra | 13:11 | |
| *** ociuhandu has quit IRC | 13:12 | |
| *** matt_kosut has quit IRC | 13:12 | |
| openstackgerrit | Benedikt Löffler proposed zuul/zuul master: Fix override variables in zuul_return https://review.opendev.org/711002 | 13:13 |
| *** matt_kosut has joined #openstack-infra | 13:14 | |
| *** matt_kosut has quit IRC | 13:15 | |
| *** matt_kosut has joined #openstack-infra | 13:16 | |
| *** udesale_ has joined #openstack-infra | 13:17 | |
| *** udesale has quit IRC | 13:18 | |
| *** jcoufal has joined #openstack-infra | 13:19 | |
| *** udesale_ has quit IRC | 13:24 | |
| *** udesale has joined #openstack-infra | 13:24 | |
| *** matt_kosut has quit IRC | 13:26 | |
| *** yamamoto has quit IRC | 13:29 | |
| *** udesale has quit IRC | 13:31 | |
| *** udesale has joined #openstack-infra | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** ykarel is now known as ykarel|afk | 13:35 | |
| *** matt_kosut has joined #openstack-infra | 13:35 | |
| *** matt_kosut has quit IRC | 13:35 | |
| *** matt_kosut has joined #openstack-infra | 13:36 | |
| *** tbachman has quit IRC | 13:53 | |
| *** udesale has quit IRC | 13:58 | |
| *** udesale has joined #openstack-infra | 13:58 | |
| *** zigo has quit IRC | 13:58 | |
| *** lbragstad has quit IRC | 13:59 | |
| *** ykarel|afk is now known as ykarel | 13:59 | |
| *** eharney has joined #openstack-infra | 14:02 | |
| *** aaronsheffield has joined #openstack-infra | 14:06 | |
| sshnaidm | AJaeger, can you take a look please, to unblock us https://review.opendev.org/#/c/710986/ | 14:07 |
| AJaeger | sshnaidm: I'd like ianw to review that, he's familiar with both areas. Also, I need to think more about these implications - and can't right now. | 14:09 |
| sshnaidm | AJaeger, ack | 14:10 |
| sshnaidm | ianw, ^^ | 14:10 |
| frickler | infra-root: LE has some issues, I'm currently scanning the provided lists of affected certs https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864 | 14:15 |
| frickler | up to now only found review-dev.o.o | 14:15 |
| openstackgerrit | Ghanshyam Mann proposed openstack/os-testr master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705522 | 14:16 |
| *** yamamoto has joined #openstack-infra | 14:18 | |
| *** lpetrut has joined #openstack-infra | 14:18 | |
| frickler | o.k., checked all other occurrences of opendev&openstack, we seem to be fine otherwise. the foundation may want to look into it's use of cert.openstack.org though http://paste.openstack.org/show/790250/ | 14:20 |
| fungi | i gave them a heads up a few minutes ago, thanks frickler! | 14:21 |
| *** Goneri has joined #openstack-infra | 14:21 | |
| fungi | though for that specific cert, i think lf may be managing it | 14:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Report aborted changes via Github checks API https://review.opendev.org/711023 | 14:21 |
| *** matt_kosut has quit IRC | 14:22 | |
| *** yamamoto has quit IRC | 14:23 | |
| frickler | infra-root: we also still need to fix or rather work around the virtualenv/tox/py27 issue, please review https://review.opendev.org/710967 | 14:26 |
| *** iurygregory has quit IRC | 14:26 | |
| *** iurygregory has joined #openstack-infra | 14:28 | |
| *** tkajinam has quit IRC | 14:30 | |
| *** tkajinam has joined #openstack-infra | 14:31 | |
| corvus | frickler: did that go through the base-test process, or are we confident enough in it to merge it untested? | 14:31 |
| *** zigo has joined #openstack-infra | 14:32 | |
| mordred | corvus: on clarkb's patch from yesterday, https://review.opendev.org/#/c/710851/ - he notes that we skipped base-test because the task itself is set to not fail on errors | 14:33 |
| corvus | okiedokie | 14:34 |
| mordred | based on that, and frickler having tested that particular pin in the requirements repo - I'm ok with it | 14:34 |
| *** xek has joined #openstack-infra | 14:37 | |
| *** ijw_ has quit IRC | 14:41 | |
| openstackgerrit | Merged opendev/base-jobs master: Pin virtualenv in order to fix tox on py27 https://review.opendev.org/710967 | 14:46 |
| *** tbachman has joined #openstack-infra | 14:51 | |
| *** artom has joined #openstack-infra | 14:51 | |
| *** michael-beaver has joined #openstack-infra | 14:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: DNM: test depends-on cherrypy https://review.opendev.org/711034 | 14:54 |
| *** ykarel is now known as ykarel|away | 14:59 | |
| *** jcapitao_lunch is now known as jcapitao | 15:00 | |
| *** lbragstad has joined #openstack-infra | 15:01 | |
| *** roman_g has joined #openstack-infra | 15:03 | |
| *** ociuhandu has joined #openstack-infra | 15:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Move stackviz publish in-tree https://review.opendev.org/710756 | 15:04 |
| *** tkajinam has quit IRC | 15:09 | |
| *** KeithMnemonic has quit IRC | 15:16 | |
| *** chandankumar is now known as raukadah | 15:18 | |
| corvus | i'm going to restart all of zuul in a few minutes; i've cleared it with the release team | 15:21 |
| fungi | infra-root: inap reports 198.72.124.203 was exposing an open dns resolver at 2020-02-28T00:38:48 (i believe they've said in the past their event times are in utc) | 15:22 |
| fungi | see e-mail to the shared root inbox | 15:23 |
| fungi | i'm not really around to investigate what job that might have been | 15:23 |
| *** jamesmcarthur has joined #openstack-infra | 15:23 | |
| *** lpetrut has quit IRC | 15:23 | |
| Shrews | 2020-02-27 22:13:13,136 DEBUG nodepool.NodeLauncher: [node_request: 300-0007671594] [node: 0014869987] Node is running [region: mtl01, az: nova, ip: 198.72.124.203 ipv4: 198.72.124.203, ipv6: , hostid: 5442f8bb7d56a01546982e7683455ecf9b2e0487a20777fc54868dea] | 15:28 |
| Shrews | only reference to that IP before 00:38:48 | 15:28 |
| corvus | 2020-02-28 01:19:30,808 INFO nodepool.NodeDeleter: Deleting ZK node id=0014869987, state=deleting, external_id=498f1740-7cb5-4a25-9bca-e6527402dcba | 15:30 |
| corvus | looks like that node was still in use during the notification time | 15:30 |
| corvus | long job | 15:31 |
| Shrews | yeah | 15:31 |
| corvus | (or maybe it was during our troubles?) | 15:31 |
| clarkb | frickler: what is the LE issue? | 15:32 |
| corvus | clarkb: https://community.letsencrypt.org/t/revoking-certain-certificates-on-march-4/114864 | 15:33 |
| clarkb | and ya I wouldve done base-test first if this is a long term change to the base job but upstream importlib-resources already landed a fix so we just need a release now | 15:33 |
| clarkb | I expect we'll delete that code by the end of the week | 15:33 |
| corvus | clarkb: well, we run base-test changes to make sure we don't break the entire system with a typo | 15:33 |
| corvus | i don't at all doubt that the solution works | 15:33 |
| corvus | but if everyone else is confident in their ability as an ansible syntax parser, sure, i'll go along with it. until the next time we break it and we reaffirm our commitment to the process. | 15:34 |
| clarkb | oh I was relying on linters for that | 15:35 |
| clarkb | maybe I shpuldnt | 15:35 |
| corvus | they are not sufficient | 15:35 |
| corvus | we have broken the entire system nearly every time we merge a change to base without going through base-test | 15:35 |
| clarkb | it did catch that ansible doesnt like unmatched ' in comments at least | 15:35 |
| *** udesale_ has joined #openstack-infra | 15:35 | |
| corvus | but most of those times, it's me writing the change, so there you go :) | 15:36 |
| *** Goneri has quit IRC | 15:36 | |
| *** udesale has quit IRC | 15:37 | |
| corvus | Shrews, fungi: Execute job kuryr-kubernetes-tempest-containerized-ovn (uuid: 166301f57b21402d8d8443bb1e17f970) on nodes <NodeSet openstack-single-node-bionic [<Node 0014869987 ('controller',):ubuntu-bionic>]> | 15:39 |
| *** ociuhandu has quit IRC | 15:39 | |
| fungi | so there's a possibility that job exposes a recursive dns resolver to the internet at large | 15:40 |
| corvus | Shrews, fungi: https://zuul.opendev.org/t/openstack/build/166301f57b21402d8d8443bb1e17f970 | 15:40 |
| clarkb | fungi typically k8s jobs ime | 15:40 |
| corvus | #status log restarted all of zuul on commit 3acc00a30eb967556e7e6484f17e1b9019d51c85 | 15:40 |
| clarkb | the last one was airship? or osh I wrote email to the list and they said they would address it iirc | 15:40 |
| fungi | corvus: thanks for looking into it! | 15:41 |
| corvus | fungi, clarkb, Shrews: what should we do now with that information? | 15:41 |
| *** dave-mccowan has joined #openstack-infra | 15:41 | |
| clarkb | corvus: last time I sent email to opemstack-discuss and tagged the involved team and asked them to address it | 15:41 |
| clarkb | looks like this is kuryr+k8s not osh+k8s | 15:42 |
| Shrews | looks like their devstack enables coredns. not sure if that's the culprit | 15:43 |
| *** tesseract has quit IRC | 15:43 | |
| fungi | yeah, appears to be a deliverable of the kuryr team in openstack: https://governance.openstack.org/tc/reference/projects/kuryr.html#kuryr-kubernetes | 15:44 |
| fungi | so maybe a note on openstack-discuss ml with [kuryr] in the subject | 15:44 |
| fungi | asking them to work out whether it's an endemic problem in that job, and if so to see if they're opening the iptables rules too liberally and whether they can find a way not to | 15:45 |
| *** dave-mccowan has quit IRC | 15:46 | |
| corvus | hrm, i can't find a previous message to -discuss | 15:49 |
| *** jcapitao is now known as jcapitao_afk | 15:49 | |
| corvus | i'll come up with something and send it | 15:51 |
| *** tesseract has joined #openstack-infra | 15:53 | |
| *** tomaw_ is now known as tomaw | 15:56 | |
| corvus | clarkb, fungi, Shrews: this look good? http://paste.openstack.org/show/790259/ | 15:57 |
| Shrews | lgtm | 15:58 |
| clarkb | for some reason I'm having a hard time finding the email too. Did I dream that interaction? Pretty sure it happened | 15:58 |
| clarkb | looking | 15:58 |
| corvus | clarkb: if your todo list is anything like mine, it's probably there :) | 15:58 |
| clarkb | corvus: email looks good to me | 15:59 |
| fungi | corvus: lgtm, thanks!!! | 16:03 |
| clarkb | corvus: found it. "[OSH][Infra] Open UDP port 111 on test nodes" issue was the service wasn't dns | 16:05 |
| corvus | aha :) | 16:05 |
| clarkb | I had dreamed that aspect then :) | 16:06 |
| clarkb | frickler: thank you for checking out certs | 16:08 |
| clarkb | mordred: frickler to reissue review-dev's cert we delete those files in root's homedir for the acme config? | 16:08 |
| mordred | clarkb: no - it was something else | 16:10 |
| mordred | clarkb: we should get clarity from ianw when he's up, then add it to our letsencrypt docs | 16:11 |
| *** mattw4 has joined #openstack-infra | 16:11 | |
| clarkb | mordred: ++ have we done the necessary rotation yet or is that to be done after clarification? | 16:11 |
| clarkb | (its not a huge rush given its the -dev server) | 16:11 |
| mordred | clarkb: review-dev was reissued to include review-dev.openstack.org - do we need to do an additional one? | 16:13 |
| mordred | clarkb: oh - I now see the thing to which you are referring | 16:13 |
| clarkb | mordred: ya we need to reissue before tomorrow or LE's revocation list will break clients that check revocation lists | 16:14 |
| clarkb | but since this is -dev and many clients don't actually check those lists we are likely fine to wait for input | 16:14 |
| mordred | clarkb: nod. yeah - let's get clarity from ianw on how to handle it - since maybe next time we need to do it it might not be review-dev :_) | 16:15 |
| clarkb | ++ and ++ to documentation addition | 16:15 |
| *** jcapitao_afk is now known as jcapitao | 16:21 | |
| clarkb | frickler: re the two issues around virtualenv yesterday, I just wasn't aware that virtuaelnv needed pinning on py27 | 16:23 |
| *** jamesmcarthur has quit IRC | 16:23 | |
| clarkb | frickler: thank you for catching up on that | 16:23 |
| *** jamesmcarthur has joined #openstack-infra | 16:23 | |
| *** gyee has joined #openstack-infra | 16:27 | |
| *** udesale_ has quit IRC | 16:29 | |
| *** jamesmcarthur has quit IRC | 16:30 | |
| *** lmiccini has quit IRC | 16:33 | |
| corvus | mordred: do you understand the error here? https://zuul.opendev.org/t/openstack/build/745af467bc0442d49e808223d8af47f8 | 16:41 |
| *** jpena is now known as jpena|brb | 16:46 | |
| *** tbachman has quit IRC | 16:53 | |
| *** jamesmcarthur has joined #openstack-infra | 16:54 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 16:56 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Build an openstackclient container for use on servers https://review.opendev.org/711057 | 16:57 |
| *** tesseract has quit IRC | 16:58 | |
| *** jamesmcarthur has quit IRC | 17:03 | |
| *** matt_kosut has joined #openstack-infra | 17:04 | |
| *** matt_kosut has quit IRC | 17:08 | |
| *** zxiiro has joined #openstack-infra | 17:10 | |
| *** rpittau is now known as rpittau|afk | 17:10 | |
| *** bdodd has quit IRC | 17:12 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP tox https://review.opendev.org/711058 | 17:12 |
| *** dtantsur is now known as dtantsur|afk | 17:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: test-ensure-tox: Ignore errors uninstalling tox the second time https://review.opendev.org/711068 | 17:20 |
| *** ociuhandu has joined #openstack-infra | 17:21 | |
| *** tbachman has joined #openstack-infra | 17:22 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: use provided tox_executable for tests https://review.opendev.org/711058 | 17:22 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: DO NOT MERGE test base-test with no virtualenv perms modifications https://review.opendev.org/680178 | 17:22 |
| clarkb | frickler: fyi https://review.opendev.org/#/c/680178/ is testing your question at https://review.opendev.org/#/c/710858/ around checking with base-test | 17:23 |
| clarkb | frickler: basically we already don't do the workaround on base-test so reparenting those jobs to base-test then checking that they run bindep successfully should confirm we can safely remove the workaround in base | 17:24 |
| clarkb | infra-root ^ fyi. I also think that will reduce the spam in logs about console log streamer not being started yet (as it reduces work done prior to starting the streamer) | 17:24 |
| *** jpena|brb is now known as jpena | 17:25 | |
| mordred | clarkb: woot | 17:25 |
| mordred | clarkb: I *sortof* understand that error- I feel like I went down a rabbithole with it a few days ago | 17:26 |
| clarkb | corvus: ^ that was for you (I was confused by what error mordred meant :) ) | 17:26 |
| *** keith28 has joined #openstack-infra | 17:28 | |
| *** derekh has quit IRC | 17:28 | |
| mordred | clarkb: well - I think I'm actually the right person | 17:29 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 17:29 |
| clarkb | mordred: ya but corvus asked you about the error | 17:29 |
| clarkb | mordred: so making sure he sees your response | 17:29 |
| mordred | clarkb: OH | 17:30 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 17:30 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 17:31 |
| corvus | mordred: it sort of looks like something external has broken all of our gerrit image builds? | 17:33 |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** ociuhandu_ has joined #openstack-infra | 17:40 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add python3-distutils to gerrit-base https://review.opendev.org/711072 | 17:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable recommends in python-base and python-builder https://review.opendev.org/711073 | 17:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add python3-distutils to gerrit-base https://review.opendev.org/711072 | 17:41 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Disable recommends in python-base and python-builder https://review.opendev.org/711073 | 17:41 |
| mordred | corvus: ^^ I think the python3-distutils should fix it | 17:42 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Switch to distro pip for gerrit images https://review.opendev.org/711074 | 17:43 |
| *** ociuhandu has quit IRC | 17:43 | |
| *** ociuhandu_ has quit IRC | 17:44 | |
| mordred | corvus: my hunch is that something changed in the openjdk:8 image - potentially a base OS update - that pulled us into the world where we need to explicitly install python3-distutils | 17:45 |
| corvus | mordred: ok, thanks! | 17:45 |
| mordred | clarkb: https://review.opendev.org/711057 is the result of me getting annoyed and nerdsniping myself | 17:48 |
| clarkb | infra-root anyone else want to review https://review.opendev.org/#/c/710858/1 to cleanup some of the previous virtualenv workarounds (this helps reduce the number of total workarounds we need to manage) | 17:51 |
| clarkb | mordred: I think https://review.opendev.org/#/c/711057/1/docker/openstackclient/Dockerfile copies the wrong context. We are in system-config there so asssemble won't produce us a osc, it will likely fail to produce us a thing system-config like | 17:54 |
| mordred | clarkb: look at line 62 of https://review.opendev.org/#/c/711057/1/.zuul.yaml | 17:55 |
| clarkb | mordred: I see. I guess we should expect that to be more normal for us, having dockerfiles in system-config to execute against different contexts? Maybe we can document those cases from within the dockerfile? | 17:56 |
| corvus | clarkb: looking at 710858 where did all the virtualenv pinning go? | 17:56 |
| mordred | clarkb: that tells the roles to run in that dir, so that'll be the context | 17:56 |
| mordred | corvus, clarkb: we should rebase that change | 17:57 |
| *** keith28 has left #openstack-infra | 17:57 | |
| clarkb | ya 710858 was written before virtualenv pinning landed | 17:57 |
| clarkb | I can rebase to make the diff cleaner | 17:57 |
| clarkb | *clearer | 17:57 |
| *** diablo_rojo has joined #openstack-infra | 17:57 | |
| corvus | ++ i think we need to see the correct result reviewing this | 17:57 |
| mordred | clarkb: yes - I think we can document that in the dockerfiles - it's definitely a more normal pattern for the ones in system-config | 17:57 |
| clarkb | corvus: ok let me finish this review for mordred then I can rebase | 17:58 |
| *** jcapitao has quit IRC | 17:58 | |
| corvus | mordred: why not have openstackclient build its image in its own repo? | 17:58 |
| *** igordc has joined #openstack-infra | 17:59 | |
| openstackgerrit | Clark Boylan proposed opendev/base-jobs master: Remove temporary workaround for virtualenv install behavior https://review.opendev.org/710858 | 18:02 |
| clarkb | corvus: ^ done | 18:02 |
| corvus | clarkb: +3 | 18:10 |
| *** Lucas_Gray has quit IRC | 18:12 | |
| *** jcoufal has quit IRC | 18:16 | |
| *** jcoufal has joined #openstack-infra | 18:17 | |
| *** yamamoto has joined #openstack-infra | 18:21 | |
| *** jamesmcarthur has joined #openstack-infra | 18:22 | |
| openstackgerrit | Merged opendev/base-jobs master: Remove temporary workaround for virtualenv install behavior https://review.opendev.org/710858 | 18:22 |
| *** yamamoto has quit IRC | 18:26 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 18:28 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 18:28 |
| openstackgerrit | Merged zuul/zuul master: Add destructor for SshAgent https://review.opendev.org/709609 | 18:29 |
| *** jcoufal_ has joined #openstack-infra | 18:31 | |
| *** jcoufal has quit IRC | 18:34 | |
| *** jpena is now known as jpena|off | 18:41 | |
| *** rosmaita has quit IRC | 19:01 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 19:03 |
| *** andrewbonney has quit IRC | 19:09 | |
| *** michael-beaver has quit IRC | 19:13 | |
| *** rosmaita has joined #openstack-infra | 19:14 | |
| ianw | frickler: thanks for pointing that out. i didn't write up docs to replace certs because i thought i fixed the problem issuing new certs when we added domains ... but i for sure didn't think of having to reissue certs because of LE bugs! | 19:17 |
| ianw | i can do that today and get review-dev reissued | 19:18 |
| ianw | and i can check the logs to see if we renewed anything else in the period | 19:18 |
| *** eharney has quit IRC | 19:19 | |
| *** ralonsoh has quit IRC | 19:21 | |
| AJaeger | config-core, please put https://review.opendev.org/710756 and https://review.opendev.org/710654 on your review queue | 19:24 |
| *** gfidente is now known as gfidente|AFK | 19:24 | |
| *** mattw4 has quit IRC | 19:25 | |
| *** tbachman has quit IRC | 19:34 | |
| *** tbachman has joined #openstack-infra | 19:38 | |
| *** tbachman_ has joined #openstack-infra | 19:40 | |
| *** tbachman_ has left #openstack-infra | 19:40 | |
| *** hashar has joined #openstack-infra | 19:42 | |
| *** tbachman has quit IRC | 19:43 | |
| *** eharney has joined #openstack-infra | 19:50 | |
| ianw | we didn't get emails about our bad LE certs because we're not using a common account with email registered to issue certs ... looking if we can fix that | 19:50 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 20:01 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Rewrite logs.o.o post reference https://review.opendev.org/711098 | 20:02 |
| AJaeger | ianw, fungi, one place where we use logs.o.o ^ | 20:02 |
| frickler | one other idea regarding #opendev, maybe we can merge what remains here with #openstack-qa, which is difficult to find anyway for people with devstack related questions | 20:03 |
| clarkb | frickler: I was wondering about that too after corvus' thoughts on devstack questions | 20:03 |
| openstackgerrit | Andreas Jaeger proposed opendev/base-jobs master: Remove site_logs secret https://review.opendev.org/711099 | 20:04 |
| AJaeger | ianw: one more ^ | 20:04 |
| clarkb | a lot of the questions I imagine that will be openstack specific will have overlap with the qa team | 20:04 |
| clarkb | having direct involvement from both sides would be helpful | 20:04 |
| frickler | in fact that might even work team-wise. like devstack and the remainder of openstack-infra could be openstack-ci | 20:05 |
| AJaeger | ianw: and logs.openstack.org is in use in project-config, these needs a bit more investigation | 20:06 |
| ianw | AJaeger: thanks ... i will investigate and probably add something to the static.o.o story ... there's a few other cleanup tasks there too | 20:07 |
| frickler | anyways, that's it for me for today, have a nice super tuesday still ;) | 20:07 |
| AJaeger | thanks, ianw | 20:07 |
| *** ociuhandu has joined #openstack-infra | 20:10 | |
| *** aedc has quit IRC | 20:11 | |
| *** aedc has joined #openstack-infra | 20:11 | |
| *** ociuhandu has quit IRC | 20:15 | |
| openstackgerrit | Merged opendev/system-config master: Add python3-distutils to gerrit-base https://review.opendev.org/711072 | 20:18 |
| clarkb | I'm going to find lunch then ride the bike. Back in a bit to review the changes AJaeger called out and then do zuul sruvey stuff | 20:19 |
| openstackgerrit | Merged zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 20:21 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Cleanup #opendev gerritbot https://review.opendev.org/711105 | 20:21 |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Cleanup #openstack-infra gerritbot https://review.opendev.org/711106 | 20:21 |
| AJaeger | clarkb: have fun. | 20:21 |
| AJaeger | infra-root, here are the two changes for gerritbot, please review ^ | 20:22 |
| *** tosky has quit IRC | 20:22 | |
| *** trident has quit IRC | 20:23 | |
| *** trident has joined #openstack-infra | 20:26 | |
| *** aedc_ has joined #openstack-infra | 20:27 | |
| *** aedc has quit IRC | 20:27 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove site_specs and playbooks/publish/service-types.yaml https://review.opendev.org/711107 | 20:30 |
| AJaeger | ianw: that's one part of logs removal, I leave the rest for now - happy if you can do that. | 20:31 |
| ianw | AJaeger: thanks! will look in on it today | 20:31 |
| AJaeger | cool | 20:32 |
| * AJaeger waves good bye for today | 20:32 | |
| artom | donnyd, IIRC FN was supposed to change labels (or something similar?) I guess it happened? | 20:33 |
| *** jamesmcarthur has quit IRC | 20:36 | |
| *** jamesmcarthur has joined #openstack-infra | 20:37 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| fungi | artom: i believe fn is out of service a week or so for some upgrades/redeployment | 20:39 |
| donnyd | yea I am still out doing network tuning | 20:40 |
| donnyd | its pretty close though | 20:40 |
| donnyd | ovs seems to be dropping a lot of packets - and I am not entirely sure why - when this is sorted it will be back online | 20:44 |
| *** slaweq has quit IRC | 20:46 | |
| *** Goneri has joined #openstack-infra | 20:47 | |
| *** aaronsheffield has quit IRC | 20:55 | |
| *** nicolasbock has quit IRC | 20:58 | |
| mordred | corvus: have a sec: https://review.opendev.org/#/c/711073 ? | 21:00 |
| *** jamesmcarthur has joined #openstack-infra | 21:01 | |
| *** jcoufal_ has quit IRC | 21:02 | |
| *** lbragstad_ has joined #openstack-infra | 21:04 | |
| *** lbragstad has quit IRC | 21:07 | |
| *** igordc has quit IRC | 21:08 | |
| *** igordc has joined #openstack-infra | 21:08 | |
| *** igordc has quit IRC | 21:10 | |
| *** slaweq has joined #openstack-infra | 21:11 | |
| *** igordc has joined #openstack-infra | 21:12 | |
| *** igordc has quit IRC | 21:12 | |
| *** slaweq has quit IRC | 21:16 | |
| *** ijw has joined #openstack-infra | 21:20 | |
| *** xek has quit IRC | 21:20 | |
| *** Lucas_Gray has joined #openstack-infra | 21:22 | |
| corvus | mordred: +2 with comment | 21:29 |
| *** happyhemant has quit IRC | 21:29 | |
| mordred | corvus: totally agree | 21:30 |
| mordred | infra-root: https://review.opendev.org/#/c/699416 is green, which is a patch to replace python-glanceclient with openstacksdk in python-openstackclient ... I know we have no workflows relying on that - but I thought I'd give a heads up in case any humans wanted to test it before we land it | 21:37 |
| *** mattw4 has joined #openstack-infra | 21:42 | |
| weshay|ruck | anyone see requirements-check fail today? couple times in our gates.. | 21:53 |
| weshay|ruck | c | /tmp/tmp5wcvj6e9/venv/bin/pip install /home/zuul/src/opendev.org/openstack/requirements | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919102 | ubuntu-bionic | Traceback (most recent call last): | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919139 | ubuntu-bionic | File "src/opendev.org/openstack/requirements/playbooks/files/project-requirements-change.py", line 171, in <module> | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919171 | ubuntu-bionic | main() | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919202 | ubuntu-bionic | File "src/opendev.org/openstack/requirements/playbooks/files/project-requirements-change.py", line 130, in main | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919237 | ubuntu-bionic | install_and_load_requirements(reqroot, reqdir) | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919272 | ubuntu-bionic | File "src/opendev.org/openstack/requirements/playbooks/files/project-requirements-change.py", line 96, in install_and_load_requirements | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919304 | ubuntu-bionic | from openstack_requirements import check # noqa | 21:53 |
| weshay|ruck | 2020-03-03 20:17:31.919337 | ubuntu-bionic | ModuleNotFoundError: No module named 'openstack_requirements' | 21:53 |
| weshay|ruck | works locally fine for me | 21:53 |
| *** Lucas_Gray has quit IRC | 21:54 | |
| *** rcernin has joined #openstack-infra | 21:56 | |
| *** hashar has quit IRC | 21:58 | |
| fungi | please link to the failure. filling the channel with a traceback not only gets annoying but provides no context to look into what led up to that exception case | 21:58 |
| openstackgerrit | Lance Bragstad proposed openstack/project-config master: Add a new project and repository for tripleo-ipa https://review.opendev.org/711114 | 21:58 |
| openstackgerrit | Lance Bragstad proposed openstack/project-config master: Add jobs for tripleo-ipa project https://review.opendev.org/711115 | 21:58 |
| fungi | (also there is a channel for #openstack-requirements) | 21:58 |
| *** dpawlik has quit IRC | 22:00 | |
| *** owalsh has quit IRC | 22:03 | |
| *** owalsh has joined #openstack-infra | 22:03 | |
| *** imacdonn has quit IRC | 22:04 | |
| *** imacdonn has joined #openstack-infra | 22:04 | |
| *** slaweq has joined #openstack-infra | 22:11 | |
| *** dkehn has joined #openstack-infra | 22:15 | |
| *** slaweq has quit IRC | 22:16 | |
| lbragstad_ | hey folks, i'm trying to reserve a pypi namespace for an ansible role (^ links above), and I'm referencing the project creator guide | 22:20 |
| *** lbragstad_ is now known as lbragstad | 22:20 | |
| lbragstad | do i need to include the python specifics described in https://packaging.python.org/tutorials/packaging-projects/ if this is going to be for an ansible role and not a python project? | 22:21 |
| sshnaidm | ianw, hi, can you please take a look in your time? https://review.opendev.org/#/c/710986/ it's for centos8 and it's blocking tripleo CI currently, thanks! | 22:24 |
| *** jamesmcarthur has quit IRC | 22:24 | |
| ianw | sshnaidm: hrm ... i don't see that as a straightforward thing we'd put in project-config. my initial feeling is that it feels odd | 22:26 |
| sshnaidm | ianw, what is making it odd? | 22:27 |
| *** jamesmcarthur has joined #openstack-infra | 22:27 | |
| ianw | well, it sort of makes me think that tripleo can't work on any system not booted with the right kernel flags | 22:27 |
| ianw | which doesn't seem right | 22:27 |
| ianw | and we switch this here and it applies for everyone who ever uses centos8 ... | 22:28 |
| sshnaidm | ianw, well, as I mentioned in the patch for centos7 and rhel8 it were always interfaces ethX and all our CI code is relying on them being the same | 22:28 |
| sshnaidm | ianw, tripleo can configure any interface for a customer | 22:28 |
| ianw | it's just that we make settings like this and it always comes back to bite us later when someone says "hey, upstream doesn't do that, and you're overriding the defaults" | 22:29 |
| sshnaidm | ianw, mmm.. we don't override defaults | 22:30 |
| sshnaidm | ianw, and when automatic CI job runs it's different from customer putting his configuration in tripleo, choosing slowly what is convenient for him | 22:30 |
| sshnaidm | ianw, and having in mind rhel8 has the same settings, we actually bring by that CI closer to a customer | 22:31 |
| ianw | i'm not like -2 on it ... but i just don't want us to make centos8 in infra something special that doesn't reflect anything like what people get in the wild | 22:31 |
| ianw | perhaps in that case it would be better to commit this as a default to dib, so that anyone building a dib centos image gets that behaviour? | 22:32 |
| sshnaidm | ianw, you mean to add this setting to DIB_BOOTLOADER_DEFAULT_CMDLINE in defaults? | 22:33 |
| sshnaidm | ianw, like here? https://opendev.org/openstack/diskimage-builder/src/branch/master/diskimage_builder/elements/bootloader/environment.d/10-bootloader-default-cmdline | 22:33 |
| corvus | mordred, clarkb: i'm happy enough with the results of the non-gerrit image build jobs in https://review.opendev.org/710532 to proceed with the provides/requires change | 22:34 |
| corvus | i'll (re-)propose those changes now | 22:35 |
| ianw | sshnaidm: well, it would have to be additive for only centos | 22:36 |
| sshnaidm | ianw, so, where can I put it then? In my patch I added it to centos8 only, actually | 22:37 |
| clarkb | lbragstad: you need enough that python setup.py sdist and python setup.py bdist_wheel work | 22:37 |
| clarkb | lbragstad: I'm not quite sure where that line is, but I'm guessing its a pretty minimal setup.py/setup.cfg | 22:37 |
| lbragstad | clarkb ok - that sounds good | 22:37 |
| lbragstad | after that, we can manage everything via gerrit (assuming the project is created), yeah? | 22:38 |
| clarkb | lbragstad: `twine` is the upload command and I think it has a linter/checker subcommand too | 22:38 |
| clarkb | lbragstad: yu | 22:38 |
| lbragstad | sweet | 22:38 |
| lbragstad | thanks clarkb | 22:39 |
| clarkb | ianw: sshnaidm I agree that we should probably stick to defaults as much as possible. I think we've learned hard lessons straying from those | 22:39 |
| clarkb | sshnaidm: what is wrong with devbiosname interfaces? | 22:39 |
| ianw | sshnaidm: does rhel8 really by default set net.ifnames=0? i need to look at my image of it | 22:39 |
| sshnaidm | clarkb, well, it depends what you call defaults, I think stock centos8 qcow2 images does have interfaces ethX | 22:40 |
| clarkb | sshnaidm: I mean what you get installing the kernel/bootloader | 22:40 |
| sshnaidm | ianw, we run jobs on it, it has this setting | 22:40 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Pul from intermediate registry in image build job https://review.opendev.org/711119 | 22:41 |
| mordred | if the centos8 qcow2 images have the setting - that sound like they're setting something in the bootloader special for the qcow - which does make it seem more like a thing that woudl be appropriate for dib | 22:41 |
| *** mattw4 has quit IRC | 22:41 | |
| clarkb | mordred: agreed. And possibly even the actual distro bootloader/kernel packages | 22:41 |
| sshnaidm | clarkb, all our CI that worked on centos7 has ethX interfaces, on rhel8 has ethX interfaces. It's not a "customer" case, it's matter of CI code that relies on interface name | 22:42 |
| mordred | that said - if the default behavior changed from centos7 -> centos8, it seems like it would be better to update tripleo to deal with that being a thing in the real world | 22:42 |
| clarkb | sshnaidm: I undersatnd. What I am telling you is we just install the packages. Its up to the distro to decide if they want devbiosname or not, it isn't really us | 22:42 |
| sshnaidm | mordred, it's not matter of tripleo, it's matter of tripleo *CI* | 22:42 |
| clarkb | we can modify that, but ideally we'd stick to what you get if you install rhel/centos 8 | 22:43 |
| clarkb | and not stray from that | 22:43 |
| clarkb | then if necessary you should update your CI jobs to use the appropriate interface | 22:43 |
| ianw | sshnaidm: in my rhel-guest-image-8.0-1854.x86_64.qcow2 i don't have net.ifnames=0 in the kernel config ... are you possibly using a more custom image? | 22:43 |
| sshnaidm | clarkb, but nobody installs centos8 from openstack dib images used in CI | 22:43 |
| mordred | sshnaidm: right. but tripleo CI's job is to test how tripleo would work on the OS in question - if the CI is depending on a configuation that's different than what is actually out there, you could be missing real-world bugs | 22:43 |
| openstackgerrit | James E. Blair proposed opendev/base-jobs master: Pull from intermediate registry in image build job https://review.opendev.org/711119 | 22:43 |
| sshnaidm | ianw, it runs on 3 party job, built in RDO nodepool | 22:43 |
| ianw | sshnaidm: from my POV, if you're saying that this is set in default rhel8 images, and now default centos8 images, and dib is the outlier here, i would consider that a very strong argument to turn it on in dib to be consistent | 22:44 |
| sshnaidm | mordred, not really, CI code configures various things to set environment for tripleo, in tripleo itself you can configure whatever interface | 22:44 |
| ianw | sshnaidm: but if dib is going to be the odd one out, then i'm not really convinced we should change. because someone always comes back and says "hey, why is infra ci different to the upstream images" | 22:45 |
| clarkb | ianw: ++ | 22:45 |
| openstackgerrit | James E. Blair proposed opendev/system-config master: Use explicit provides/requires for container jobs https://review.opendev.org/710106 | 22:46 |
| sshnaidm | I'm not sure I understand what you mean by defaults, but testing code - which CI code is, needs to rely on interfaces to be the same because it's automatic, it's not a customer case | 22:47 |
| *** mattw4 has joined #openstack-infra | 22:47 | |
| openstackgerrit | James E. Blair proposed opendev/storyboard master: Use explicit provides/requires for container jobs https://review.opendev.org/710113 | 22:47 |
| sshnaidm | I don't know any customer that installs servers from CI openstack images | 22:48 |
| clarkb | sshnaidm: one of the intents of our CI platform is to represent software you would find in the wild. We've not always done that and its almost always come back to hurt us | 22:48 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Use explicit provides/requires for container jobs https://review.opendev.org/710115 | 22:48 |
| clarkb | sshnaidm: we don't have customers, that is the wrong way of looking at it | 22:48 |
| clarkb | sshnaidm: instead we have users that want to test their software, and they should be testing that their software works on these platforms when end users deploy them | 22:48 |
| *** tkajinam has joined #openstack-infra | 22:48 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Use explicit provides/requires for container jobs https://review.opendev.org/710116 | 22:48 |
| clarkb | if we stray from the defaults we make that more difficult | 22:48 |
| ianw | sshnaidm: i mean if i download https://cloud.centos.org/centos/8/x86_64/images/CentOS-8-GenericCloud-8.1.1911-20200113.3.x86_64.qcow2 or a rhel-guest image and boot it, i don't see they are setting this | 22:49 |
| mordred | which makes me not want to change it on our side - otherwise we're inventing a whole new OS deployment just for CI - which is the opposite of correct | 22:49 |
| sshnaidm | dynamic interfaces names is one of worst ideas I saw, really.. | 22:49 |
| mordred | sshnaidm: oh - I don't disagree with you there ... so is systemd - but both of them are bad decisions made by other people that are the way the world works now | 22:50 |
| mordred | and we need to provide environments that are just as broken as what people get in real life | 22:50 |
| sshnaidm | mordred, but it's exactly what I mean - it's not a real life here | 22:51 |
| sshnaidm | sorry :) | 22:51 |
| *** kozhukalov has joined #openstack-infra | 22:51 | |
| clarkb | sshnaidm: it is an approximation | 22:51 |
| mordred | it's supposed to be a real life here - it's supposed to be as close to a real life as possible - ti's not supposed to be different except when it is completely unavoidable | 22:51 |
| clarkb | and the closer we get the better | 22:51 |
| mordred | otherwise tests are invalid | 22:51 |
| sshnaidm | when for real person it's fine to set "ens3" instead of "eth1", CI code that tries to guess which interface to use doesn't really make sense | 22:52 |
| corvus | ci used to be an 8-bit platformer, but we're running the unreal engine here. :) | 22:52 |
| mordred | corvus: nice | 22:52 |
| *** mattw4 has quit IRC | 22:53 | |
| sshnaidm | well, it kinds blocks tripleo jobs to use c8 for now.. | 22:54 |
| corvus | sshnaidm: how would a person know how to set that? | 22:55 |
| openstackgerrit | Merged openstack/cookiecutter master: Minimise info in CONTRIBUTING.rst https://review.opendev.org/708672 | 22:55 |
| corvus | sshnaidm: just wondering if there's a way to mimic or automate that. since presumably that's what a person would need to do in real life. | 22:56 |
| sshnaidm | corvus, in real life person knows what interfaces are in his machine usually, and set their names in configs | 22:57 |
| jrosser | for test environments ansible_default_ipv4.interface pretty much does what you want | 22:57 |
| mordred | sshnaidm: not if they don't have consistent names they don't | 22:57 |
| mordred | jrosser: ++ | 22:57 |
| sshnaidm | mordred, well, it's only once - ip a | 22:57 |
| mordred | sshnaidm: and what does that do for them? | 22:57 |
| sshnaidm | person doesn't do it thousands times in a day like we | 22:57 |
| corvus | so maybe using that ansible variable is an approximation of "the user somehow figured out what the interface name is" | 22:57 |
| sshnaidm | mordred, shows the interfaces | 22:58 |
| mordred | sshnaidm: right. and then the human looks at that? | 22:58 |
| mordred | sshnaidm: so write a script which does ip a and then looks at the output | 22:58 |
| mordred | that's, in fact, a GREAT thing to have the CI do - because presumably there is a quality of a given interface you are looking for in that output - and that is what your user should be looking for | 22:59 |
| corvus | OMG i just ran that and literally have 14 interfaces | 22:59 |
| sshnaidm | mordred, yeah, and push this script everywhere and change all hardcoded names of interfaces, and and and | 22:59 |
| mordred | sshnaidm: yes | 22:59 |
| mordred | sshnaidm: exactly | 22:59 |
| sshnaidm | mordred, I don't say it's impossible, but it's like a huge overhead | 22:59 |
| mordred | of course it is | 22:59 |
| clarkb | corvus: I only have 7 | 22:59 |
| mordred | that's why distros doing crap like that is horrible | 22:59 |
| corvus | clarkb: probably kind | 22:59 |
| *** mattw4 has joined #openstack-infra | 23:00 | |
| mordred | but it's the new state of the world | 23:00 |
| corvus | clarkb: i mean "KIND" not, you know, the word kind | 23:00 |
| sshnaidm | mordred, well, they still leave a nice way to get rid off this crap by configuration settings in grub | 23:00 |
| mordred | and if we change it in CI - then there might be a place that isn't just CI setup script but is actually in the code under test somewhere that you'd miss | 23:00 |
| mordred | and then there someone in the world who gets hit for real | 23:00 |
| sshnaidm | I'd add just a note in tripleo howto: "CONFIGURE net.ifnames=0 !" :) | 23:01 |
| *** pkopec has quit IRC | 23:02 | |
| mordred | and now you just made a user go through another kernel setup and reboot step :) | 23:02 |
| fungi | lbragstad: clarkb: but you only need to reserve the project on pypi if you're worried about someone else sniping it out from under you. the release job will create it on pypi automatically if it doesn't already exist (because it calls twine upload, which autocreates new pypi projects) | 23:03 |
| *** mattw4 has quit IRC | 23:03 | |
| sshnaidm | I'm not sure how it was always for centos7, didn't you have something for that? Because these names should have came up already there | 23:04 |
| ianw | fungi / lbragstad : yes, i may have led people down the wrong path suggesting it wouldn't do that in the recent past | 23:04 |
| corvus | sshnaidm: that's the second-best outcome from this discussion -- because it means that users and CI are performing the same actions. the best outcome would be if that were the case and we were not suggesting they reboot with new kernel options. | 23:05 |
| sshnaidm | mordred, actually tripleo does build images for itself, so it's fine to set there anything, but it's for overcloud hosts | 23:06 |
| sshnaidm | the only problem is with undercloud host, when all this runs on | 23:06 |
| clarkb | infra-root on https://review.opendev.org/#/c/711105/ and https://review.opendev.org/#/c/711106/ I think I'll look for quorum before approving. And then also set a transition day whcih we can announce broadly before merging those changes. | 23:06 |
| *** jamesmcarthur has quit IRC | 23:06 | |
| sshnaidm | mordred, actually tripleo allows you to insert in images whatever you want and any settings you like, it's pretty flexible | 23:07 |
| mordred | sshnaidm: I know - I was there when we designed it originally ;) | 23:08 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update k8s log streaming release note https://review.opendev.org/711124 | 23:09 |
| sshnaidm | mordred, so, you know that in upstream CI we have already deployed server :) and can't actually provision it from image | 23:09 |
| sshnaidm | and this server has these awful names.. | 23:09 |
| clarkb | mordred: https://review.opendev.org/#/c/711107/1 is a cleanup for service-types old publishing jobs if you have a minute to review it | 23:09 |
| clarkb | what is wrong with the names? | 23:10 |
| sshnaidm | what is not really a usual customer case that I'd think about | 23:10 |
| clarkb | oh interface names not hostnames | 23:10 |
| clarkb | I think your customers will end up with devbiosnames too | 23:10 |
| mordred | sshnaidm: I do! I just think it's not that awful to figure out which one is the right one based on criteria not hardcoded names | 23:10 |
| clarkb | I mean I have eno1 on my desktop and wlp1s0 | 23:10 |
| mordred | hardcoding names makes code brittle | 23:10 |
| *** slaweq has joined #openstack-infra | 23:11 | |
| fungi | well, they'll end up with devbiosnames in the undercloud, and tripleo will build images for them with nonstandard kernel configurations so that the overcloud is inconsistent with red hat's standard operating system deployments | 23:11 |
| corvus | not to distract too much with procticalities, but were we able to either confirm or exclude jrosser's idea of using ansible_default_ipv4.interface ? | 23:11 |
| mordred | that too | 23:11 |
| mordred | corvus: I have not checked that - but I think the main issue is actually that eth0 is hardcoded in a bunch of places | 23:12 |
| clarkb | corvus: https://zuul.opendev.org/t/openstack/build/3adb1e70100e4cef8e662ab7d5b7f7be/log/zuul-info/host-info.ubuntu-bionic.yaml#44 seems like it would work fine | 23:12 |
| ianw | infra-root: apropos nothing recent -- i've looked more closely at account keys for letsencrypt and i don't think we want one global account key we share across all hosts. the only thing you can do with it is revoke a certificate if you loose the private key portion | 23:13 |
| ianw | so it seems like if we ever leak the shared key, someone could revoke all our certificates | 23:13 |
| fungi | ianw: i concur, after having researched it before | 23:13 |
| ianw | as opposed to each host having its own account, and never having the key kept as a global secret | 23:13 |
| cloudnull | sorry super late into this thread, excuse me if this was already mentioned, but couldn't we query the system for the interface with the default route? Something like: `awk 'NR==2{print $1}' /proc/net/route` or `ip -o r g 1 | awk '{print $5}'`? | 23:14 |
| openstackgerrit | Merged openstack/project-config master: Finish retiring faafo https://review.opendev.org/710654 | 23:14 |
| corvus | ianw: based on my 95% understanding of what's going on, that seems reasonable | 23:14 |
| cloudnull | instead of hard coding eth0, p1p1, eno1, etc | 23:14 |
| ianw | fungi: cool. what i *am* preparing is a change to add our email contact to the accounts, which i didn't put in at first | 23:14 |
| fungi | cloudnull: probably yes, but it sounds like the concern is updating a bajillion places tripleo-ci has embedded the string "eth0" | 23:14 |
| sshnaidm | mordred, yeah, if it was only eth0, the problem we use also eth1 and eth2 :) | 23:14 |
| ianw | and is why we havne't got an email about the revocation of review-dev | 23:14 |
| fungi | ianw: yeah, caught that earlier too, thanks, i agree that's a good addition | 23:14 |
| clarkb | ianw: makes sense | 23:15 |
| cloudnull | I think in nmcli we can change the name without impacting the actual nic ? | 23:15 |
| fungi | cloudnull: yes, you could use renaming or aliasing, most likely | 23:15 |
| mordred | cloudnull: good point! | 23:15 |
| fungi | in the job at runtime | 23:16 |
| mordred | also - if y'all are using the multinode base job there's also br-infra you could use | 23:16 |
| *** slaweq has quit IRC | 23:16 | |
| *** jamesmcarthur has joined #openstack-infra | 23:16 | |
| fungi | yes, more generally, attaching configuration in your job to the instance's default network interfaces is a questionable choice, things with network focus should probably be creating virtual interfaces to use | 23:16 |
| cloudnull | something like `sudo nmcli con mod eth2 connection.id NewNic1` would modify the ID | 23:17 |
| fungi | (and if necessary bridging them to actual interfaces, or, more likely tunnels so that they can control addressing and traffic) | 23:17 |
| cloudnull | https://pasted.tech/pastes/049001c48762d0dab16a07221c12896a873129e1 | 23:18 |
| cloudnull | which looks like it works | 23:18 |
| fungi | debian used to do basically that via udev ruls, to work around remapping from devbiosnames to compatible names at boot/hotplug for upgraded systems | 23:19 |
| fungi | so as to avoid modifying configuration at upgrade time | 23:20 |
| sshnaidm | cloudnull, that's great idea | 23:20 |
| sshnaidm | cloudnull, forgot we even have nmcli.. | 23:20 |
| *** jamesmcarthur has quit IRC | 23:20 | |
| sshnaidm | cloudnull, just need to know what exactly to rename to eth0 | 23:21 |
| mordred | sshnaidm: yah - but that's great- you can run one script once to find the thing, do the nmcli - and then you're good! | 23:21 |
| sshnaidm | yeah, maybe we can do it in on of pre-playbooks | 23:21 |
| cloudnull | sshnaidm should get it done `sudo nmcli con mod $(awk 'NR==2{print $1}' /proc/net/route) connection.id NewNic1` | 23:22 |
| sshnaidm | cloudnull, we need to rename eth0, eth1, eth2 | 23:22 |
| sshnaidm | cloudnull, but we can figure out them with nmcli too | 23:22 |
| fungi | which is i guess why just ansible_default_ipv4.interface won't cut it | 23:22 |
| cloudnull | we could also just use a macvlan off of whatever the gateway is ? | 23:23 |
| clarkb | note our instances start with a min of 1 interface and a max of 2 | 23:23 |
| cloudnull | then we could name the interafaces whatever we want | 23:23 |
| clarkb | any additional interfaces need to be created, in which case you can set the nam to whatever you want | 23:23 |
| cloudnull | ^ | 23:23 |
| fungi | yeah, maybe this is an opportunity to not rename the test vm's interface, and instead create veths and rename them | 23:24 |
| cloudnull | a macvlan is simple to make go | 23:24 |
| cloudnull | https://pasted.tech/pastes/fe68c2d3498cb91b47f524228a4ff78e9bc9ffc2 | 23:24 |
| *** owalsh_ has joined #openstack-infra | 23:25 | |
| sshnaidm | fungi, does ansible_default_ipv4.interface report interfaces in down as well? | 23:25 |
| fungi | sshnaidm: i have no idea, that was jrosser's suggestion | 23:26 |
| fungi | cloudnull: ahh, i see, right creating vlan subinterfaces basically | 23:26 |
| cloudnull | ++ | 23:26 |
| fungi | just not with 802.1q tagging associated with them | 23:26 |
| cloudnull | exactly | 23:26 |
| *** dchen has joined #openstack-infra | 23:26 | |
| cloudnull | and we can create all the interfaces off the gw device we need | 23:27 |
| sshnaidm | cloudnull, can you connect from localhost to macvlan interface? | 23:27 |
| cloudnull | then vlan tag on top of that if needed | 23:27 |
| cloudnull | sshnaidm yes, you just need to ip them | 23:27 |
| fungi | every interface within the same network namespace is connected to the loopback | 23:27 |
| *** owalsh has quit IRC | 23:28 | |
| fungi | unless the kernel's tcp/ip stack has changed massively since the last time i looked | 23:28 |
| sshnaidm | cloudnull, ok, I just remember playing with macvlan, dhcp and containers, having some troubles there | 23:28 |
| cloudnull | https://pasted.tech/pastes/bdb22e04d4ed5fe5d5203b536a9849e8f56863a4 | 23:29 |
| *** dychen has joined #openstack-infra | 23:29 | |
| cloudnull | so long as the IP address on the macvlan is on the same subnet as the vms or containers within the environment it should be able to connect. | 23:29 |
| *** dychen has quit IRC | 23:29 | |
| * cloudnull did a lot of this same stuff when i was working on OSA | 23:30 | |
| sshnaidm | cloudnull, cool | 23:30 |
| clarkb | fungi: its not so much that they are connected to the loopback as you have direct access to everything from the host aiui | 23:30 |
| clarkb | fungi: so if you need to talk to $otherinterface you know how to route directly to it | 23:30 |
| fungi | "connected" by the routing table (as long as you don't disable forwarding) | 23:31 |
| sshnaidm | cloudnull, thanks, maybe we can try these ways in patches.. I'm fine to prepare a patch with nmcli renaming | 23:31 |
| clarkb | fungi: in this case I don't even think you need forwarding because you are directly attached, but ya if trying to route through then you'd need forwarding | 23:32 |
| *** jamesmcarthur has joined #openstack-infra | 23:32 | |
| fungi | i'm assuming "connected" was in the sense of something like a guest vm on a bridge which had that subinterface in it | 23:33 |
| cloudnull | sshnaidm happy to help in any way I can :D | 23:33 |
| sshnaidm | cloudnull, I recalled what I tried, https://blog.oddbit.com/post/2018-03-12-using-docker-macvlan-networks/ - "Host access" section, but this seems like a different story | 23:34 |
| sshnaidm | cloudnull, just an offtop :) | 23:34 |
| *** dkehn has quit IRC | 23:36 | |
| cloudnull | sshnaidm ++ "without special support from a network switch, your host is unable to send packets to its own macvlan interfaces." is vepa mode | 23:37 |
| ianw | fungi: i'm sure we've discussed this before, is there some sort of auto-filtering on infra-root email for like "infra-root+letsencrypt" or similar? i feel like i've been confused when mails got auto-filtered in that account somehow | 23:38 |
| sshnaidm | cloudnull, ack, I'll google it :D | 23:38 |
| clarkb | ianw: I think fungi sets up those rules in the horde instance (or whatever system the email provider uses) | 23:40 |
| cloudnull | sshnaidm i love this blog post - https://sreeninet.wordpress.com/2016/05/29/macvlan-and-ipvlan/ - i've read and reread it more times than I'd like to admit. lots of good tidbits in there. | 23:40 |
| fungi | clarkb: ianw: actually i believe the last person to configure those may have been lyz | 23:40 |
| sshnaidm | cloudnull, cool, thanks, like this networking stuff | 23:40 |
| * mordred hands cloudnull a fluffy bunny | 23:41 | |
| cloudnull | :) | 23:41 |
| ianw | fungi / clarkb: yeah, testing shows that infra-root+<thing> goes to folder <thing> | 23:42 |
| lbragstad | fungi ack - thanks for the confirmation | 23:43 |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-jobs master: ensure-tox: add prefer_python2 https://review.opendev.org/711056 | 23:44 |
| clarkb | infra-root I'm triggering replication to gitea backends in order to ensure we are caught up after bnemec found we were behind on a server in a repo | 23:45 |
| clarkb | details on the openstack-discuss list | 23:45 |
| mordred | clarkb: makes me think we should code that sequential upgrade playbook | 23:50 |
| clarkb | mordred: ya, however I think it may have been a db update that did it this time | 23:50 |
| mordred | take backend out, upgrade, trigger repl, put back | 23:50 |
| clarkb | because it merged about 4 hours before the upgrade on that day | 23:50 |
| mordred | mmm | 23:50 |
| clarkb | maybe we should pin the mariadb image in addition to ^ | 23:50 |
| mordred | maybe so? | 23:50 |
| clarkb | but ya I like the idea of doing that in a more controlled fashion | 23:51 |
| clarkb | note replication takes about 15-20 minutes per backend if done sequentially | 23:51 |
| clarkb | I wonder if we could make replication fail during the upgrade intsead | 23:52 |
| clarkb | then gerrit would retry | 23:52 |
| clarkb | mordred: maybe if we stopped the ssh container whenever we stop oen of the other two | 23:52 |
| clarkb | then gerrit wouldn't be able to connect and it would start queueing the replication events | 23:52 |
| clarkb | (and that way we only catch up on what was missed and don't have to do a full replication) | 23:52 |
| mordred | clarkb: oh - that's actually a good idea | 23:53 |
| mordred | stop ssh container first | 23:53 |
| clarkb | ya | 23:53 |
| clarkb | we might even be able to express that in the compose file? | 23:53 |
| mordred | now - figuring out how to do that with compose ... | 23:53 |
| mordred | you think? | 23:53 |
| clarkb | maybe? not sure | 23:53 |
| mordred | that would be ideal; | 23:53 |
| *** jamesmcarthur has quit IRC | 23:53 | |
| mordred | clarkb, ianw: you're the two most likely to potentailly try to do something with openstackclient and glance - don't know if you saw the note earlier, but https://review.opendev.org/#/c/699416/ is close to landing for switching to sdk | 23:55 |
| clarkb | mordred: https://docs.docker.com/compose/compose-file/#depends_on | 23:55 |
| mordred | clarkb: so maybe what we want to do is just do a stop a pull and a start? | 23:56 |
| clarkb | I think that may be halfway to what we want | 23:56 |
| clarkb | mordred: ya that may be the simplest thing | 23:56 |
| clarkb | but then we'd need to update to only trigger the pull when we have new things, but that is doable | 23:57 |
| mordred | clarkb: oh - and maybe make gitea_web depends_on gitea_ssh? | 23:57 |
| clarkb | mordred: other way around ssh depends on web | 23:57 |
| clarkb | so that it stops first | 23:57 |
| mordred | yeah | 23:57 |
| clarkb | and starts after web is closer to being up | 23:57 |
| ianw | mordred: cool ... i think that the devstack test is probably pretty convincing there as it builds and uploads an image ... if it can do that infra should be happy :) | 23:58 |
| clarkb | and ya really the only time I have image upload weirdness is uploading to rax | 23:58 |
| clarkb | having osc use sdk gets us closer to that working in osc I think | 23:58 |
| clarkb | so I'm all for it | 23:58 |
| clarkb | mordred: so we don't lose this train of thought on restarting things are you going to write that change or should I? | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!