| *** mattw4 has quit IRC | 00:00 | |
| fungi | but yeah, having a separate ca allows you to not have to update the set of allowed clients basically | 00:05 |
|---|---|---|
| fungi | of course, you could simply rely on static source addresses in your firewall rules to serve as your authentication, and then allow any ol' cert so you still get encryption/privacy for the sessions between them | 00:06 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add storyboard story for test node cleanup https://review.opendev.org/711567 | 00:39 |
| clarkb | fungi: ^ to be clear it appears you are good if I land ^ and its parent together? (I'll do similar with the website activity stats spec if so) | 00:40 |
| clarkb | fungi: and re volunteer for ^ I was half hoping ianw might do that since ianw has proposed changes for the spec already. But if not I can probably drive it | 00:41 |
| clarkb | (I don't want to pile everything on ianw who is also involved in the wbsite stats stuff) | 00:41 |
| clarkb | and I figured I'd put myself down for website stats since I guess I already did a bunch of the investigating | 00:42 |
| clarkb | ianw: ^ let me know what you think | 00:42 |
| ianw | i think if that's the way we're going, at this point the chagnes to dib just need to land | 00:42 |
| fungi | yeah, wfm, just didn't want those placeholders slipping through the cracks | 00:42 |
| fungi | reviewing 711567 now | 00:43 |
| fungi | thanks! | 00:43 |
| *** diablo_rojo has quit IRC | 00:43 | |
| clarkb | ianw: ya thats what I think, in that case should I amend 711567 to put you down? or maybe you'd prefer to do that? | 00:43 |
| ianw | clarkb: you can put me down, i can get it going | 00:44 |
| clarkb | ok I'll amend once I've got webstats updated | 00:44 |
| *** larainema has joined #openstack-infra | 00:46 | |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add story for website stats spec https://review.opendev.org/711569 | 00:48 |
| clarkb | fungi: ^ ok I think that should be good to go? | 00:48 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add storyboard story for test node cleanup https://review.opendev.org/711567 | 00:49 |
| clarkb | and now for the other spec | 00:49 |
| clarkb | fungi: if you confirm that addresses your pre merge concerns I'll proceed with landing those | 00:50 |
| fungi | yeah, both lgtm | 00:52 |
| clarkb | great I'll hit the go button. | 00:53 |
| clarkb | I think the next step on the python images changes are to land the testing with a non production image, but also to start communicating it | 00:58 |
| clarkb | ianw: ^ I can send emails tomorrow morning to start communicating if you like | 00:58 |
| openstackgerrit | Merged opendev/infra-specs master: Add spec for website activity stats effort https://review.opendev.org/709236 | 00:59 |
| openstackgerrit | Merged opendev/infra-specs master: Add story for website stats spec https://review.opendev.org/711569 | 00:59 |
| openstackgerrit | Merged opendev/infra-specs master: Spec to cleanup python dev tools on our test images https://review.opendev.org/709579 | 01:00 |
| *** tetsuro has joined #openstack-infra | 01:02 | |
| ianw | clarkb: yeah, we need a dib release first i think. maybe it's best to get the images sort of there and working before we announce it? | 01:03 |
| openstackgerrit | Merged opendev/infra-specs master: Add storyboard story for test node cleanup https://review.opendev.org/711567 | 01:03 |
| clarkb | ianw: maybe? I guess the risk is testing showing that we have to change details | 01:04 |
| clarkb | ianw: perhaps we should get as far as glean and friends working on the test image first | 01:04 |
| clarkb | as that should prove out the major aspects of the change | 01:05 |
| ianw | yeah that's what i mean, like the images booted -- not so much the whole replacing pip installation in ansible | 01:07 |
| *** yamamoto has joined #openstack-infra | 01:07 | |
| clarkb | gotcha, ya I think we can wait for that | 01:08 |
| clarkb | I need to figure out dinner, but I suppose the next from your perspective is getting reviews? | 01:08 |
| ianw | clarkb: if you want to look over https://review.opendev.org/#/c/707513/ that's the start | 01:09 |
| *** yamamoto has quit IRC | 01:09 | |
| *** yamamoto has joined #openstack-infra | 01:09 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: deploy from container https://review.opendev.org/710891 | 01:11 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: Add webserver https://review.opendev.org/711574 | 01:11 |
| *** diablo_rojo has joined #openstack-infra | 01:11 | |
| clarkb | ianw: I guess we can fill out https://review.opendev.org/#/c/707513/5/diskimage_builder/elements/ensure-venv/pkg-map for suse and gentoo later? I'm worried if we release with that and the glean change we'll break those images | 01:13 |
| clarkb | ianw: I believe suse is just python3 as well | 01:13 |
| ianw | hrm now you mention it i think they're in experimental for boot tests | 01:15 |
| clarkb | I'm leaving a proper comment too so we don't lose that thought | 01:15 |
| clarkb | (now I'm double checking that pip install glean doesn't do magic to install the systemd stuff) | 01:16 |
| *** zxiiro has quit IRC | 01:18 | |
| *** jamesmcarthur has joined #openstack-infra | 01:18 | |
| clarkb | ya the glean-install command is run after to do that and it shouldn't care about venv setup so that all looks good | 01:18 |
| *** igordc has quit IRC | 01:18 | |
| mnaser | clarkb: https://review.opendev.org/#/c/418748/5 is a simple change if you have a second :) | 01:20 |
| clarkb | ianw: one other concern noted inline, but other than those two things I think we are good | 01:21 |
| *** ijw has quit IRC | 01:22 | |
| ianw | clarkb: ooohhh, yeah right ... the opensuse + bionic failure. i'd put that to the back of my mind | 01:23 |
| *** tosky has quit IRC | 01:39 | |
| *** jamesmcarthur has quit IRC | 01:41 | |
| *** jamesmcarthur has joined #openstack-infra | 01:42 | |
| *** jamesmcarthur has quit IRC | 01:47 | |
| openstackgerrit | Merged opendev/lodgeit master: Removes unnecessary utf-8 encoding https://review.opendev.org/418748 | 01:49 |
| *** KeithMnemonic1 has quit IRC | 01:52 | |
| openstackgerrit | Merged opendev/system-config master: ansible-lint : disable 503 https://review.opendev.org/711149 | 01:53 |
| *** KeithMnemonic1 has joined #openstack-infra | 01:53 | |
| *** diablo_rojo has quit IRC | 01:53 | |
| *** adriant has quit IRC | 01:53 | |
| *** diablo_rojo has joined #openstack-infra | 01:54 | |
| *** lbragstad_ has joined #openstack-infra | 01:54 | |
| *** adriant has joined #openstack-infra | 01:54 | |
| *** lbragstad has quit IRC | 01:57 | |
| openstackgerrit | Ian Wienand proposed openstack/diskimage-builder master: Add ensure-venv element, install glean with it https://review.opendev.org/707513 | 01:57 |
| *** owalsh has quit IRC | 01:58 | |
| *** gyee has quit IRC | 02:00 | |
| *** diablo_rojo has quit IRC | 02:04 | |
| *** owalsh has joined #openstack-infra | 02:07 | |
| *** jamesmcarthur has joined #openstack-infra | 02:12 | |
| *** Goneri has quit IRC | 02:18 | |
| *** jamesmcarthur has quit IRC | 02:21 | |
| *** jamesmcarthur has joined #openstack-infra | 02:21 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: Add webserver https://review.opendev.org/711574 | 02:24 |
| *** jamesmcarthur has quit IRC | 02:24 | |
| *** igordc has joined #openstack-infra | 02:25 | |
| *** jamesmcarthur has joined #openstack-infra | 02:26 | |
| *** matt_kosut has joined #openstack-infra | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:32 | |
| *** happyhemant has quit IRC | 02:34 | |
| *** matt_kosut has quit IRC | 02:36 | |
| *** tetsuro_ has joined #openstack-infra | 02:44 | |
| *** tetsuro has quit IRC | 02:48 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add nb01.opendev.org https://review.opendev.org/693119 | 02:53 |
| *** jamesmcarthur has joined #openstack-infra | 02:54 | |
| *** tetsuro has joined #openstack-infra | 02:54 | |
| *** tetsuro_ has quit IRC | 02:56 | |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add nb01.opendev.org https://review.opendev.org/693118 | 02:59 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: Add nb01.opendev.org https://review.opendev.org/693118 | 03:01 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add initial Ansible for nodepool hosts https://review.opendev.org/710888 | 03:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add install zookeeper role; use for nodepool-builder testing https://review.opendev.org/710908 | 03:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: deploy from container https://review.opendev.org/710891 | 03:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: Add webserver https://review.opendev.org/711574 | 03:04 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add nb01.opendev.org https://review.opendev.org/693119 | 03:04 |
| *** zxiiro has joined #openstack-infra | 03:19 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: nodepool-builder: Add webserver https://review.opendev.org/711574 | 03:27 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Add nb01.opendev.org https://review.opendev.org/693119 | 03:27 |
| *** ijw has joined #openstack-infra | 03:48 | |
| *** jamesmcarthur has quit IRC | 03:50 | |
| *** jamesmcarthur has joined #openstack-infra | 03:51 | |
| *** rlandy has quit IRC | 03:56 | |
| *** jamesmcarthur has quit IRC | 03:56 | |
| *** apetrich has quit IRC | 04:00 | |
| *** michael-beaver has quit IRC | 04:11 | |
| *** jamesmcarthur has joined #openstack-infra | 04:21 | |
| *** hashar has joined #openstack-infra | 04:23 | |
| *** jamesmcarthur has quit IRC | 04:26 | |
| *** roman_g has quit IRC | 04:27 | |
| *** ykarel|away is now known as ykarel | 04:31 | |
| *** udesale has joined #openstack-infra | 04:55 | |
| *** matt_kosut has joined #openstack-infra | 05:00 | |
| *** weshay has quit IRC | 05:01 | |
| *** matt_kosut has quit IRC | 05:04 | |
| *** weshay has joined #openstack-infra | 05:05 | |
| *** dave-mccowan has quit IRC | 05:17 | |
| *** jamesmcarthur has joined #openstack-infra | 05:22 | |
| *** raukadah is now known as chandankumar | 05:24 | |
| *** jamesmcarthur has quit IRC | 05:26 | |
| *** psachin has joined #openstack-infra | 05:29 | |
| *** ijw_ has joined #openstack-infra | 05:30 | |
| *** ijw has quit IRC | 05:33 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** ijw has joined #openstack-infra | 05:38 | |
| *** ijw_ has quit IRC | 05:42 | |
| *** lbragstad_ has quit IRC | 05:50 | |
| *** jamesmcarthur has joined #openstack-infra | 05:53 | |
| *** jamesmcarthur has quit IRC | 05:58 | |
| *** ondrejburian has joined #openstack-infra | 06:03 | |
| *** ondrejbu- has quit IRC | 06:03 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix translation-source location https://review.opendev.org/711582 | 06:12 |
| AJaeger | config-core, please get this fix for the translations in ^ | 06:12 |
| *** udesale_ has joined #openstack-infra | 06:21 | |
| *** udesale has quit IRC | 06:23 | |
| *** udesale has joined #openstack-infra | 06:28 | |
| *** udesale_ has quit IRC | 06:29 | |
| *** udesale_ has joined #openstack-infra | 06:33 | |
| *** udesale has quit IRC | 06:35 | |
| *** hashar has quit IRC | 06:40 | |
| *** igordc has quit IRC | 06:41 | |
| *** jamesmcarthur has joined #openstack-infra | 06:54 | |
| *** jamesmcarthur has quit IRC | 06:59 | |
| *** lpetrut has joined #openstack-infra | 07:10 | |
| *** lpetrut has quit IRC | 07:11 | |
| *** lpetrut has joined #openstack-infra | 07:11 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Allow check runs to be configured as required status in pipeline config https://review.opendev.org/711241 | 07:17 |
| *** iurygregory has joined #openstack-infra | 07:18 | |
| *** slaweq has joined #openstack-infra | 07:23 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Make github file annotation levels configurable via zuul return https://review.opendev.org/711179 | 07:28 |
| *** jamesmcarthur has joined #openstack-infra | 07:28 | |
| frickler | AJaeger: +3 | 07:30 |
| *** jamesmcarthur has quit IRC | 07:33 | |
| openstackgerrit | Merged openstack/project-config master: Fix translation-source location https://review.opendev.org/711582 | 07:38 |
| *** apetrich has joined #openstack-infra | 07:40 | |
| *** johnmp has joined #openstack-infra | 07:40 | |
| *** ykarel is now known as ykarel|lunch | 07:43 | |
| *** jcapitao_off has joined #openstack-infra | 07:46 | |
| AJaeger | thanks, frickler | 07:52 |
| *** jamesmcarthur has joined #openstack-infra | 07:53 | |
| *** hashar has joined #openstack-infra | 07:55 | |
| *** jcapitao_off is now known as jcapitao | 07:55 | |
| *** tesseract has joined #openstack-infra | 07:56 | |
| *** sshnaidm_ has joined #openstack-infra | 07:57 | |
| *** matt_kosut has joined #openstack-infra | 07:57 | |
| *** sshnaidm|afk has quit IRC | 07:58 | |
| *** jamesmcarthur has quit IRC | 07:58 | |
| *** matt_kosut has quit IRC | 08:01 | |
| *** matt_kosut has joined #openstack-infra | 08:03 | |
| *** pgaxatte has joined #openstack-infra | 08:03 | |
| *** ccamacho has quit IRC | 08:09 | |
| *** ccamacho has joined #openstack-infra | 08:09 | |
| *** ccamacho has quit IRC | 08:10 | |
| *** ccamacho has joined #openstack-infra | 08:10 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update elastic-recheck comment https://review.opendev.org/711586 | 08:12 |
| *** ociuhandu has joined #openstack-infra | 08:17 | |
| *** ociuhandu has quit IRC | 08:27 | |
| *** ociuhandu has joined #openstack-infra | 08:29 | |
| *** tkajinam has quit IRC | 08:29 | |
| *** jamesmcarthur has joined #openstack-infra | 08:29 | |
| *** ociuhandu has quit IRC | 08:34 | |
| *** jamesmcarthur has quit IRC | 08:35 | |
| *** matt_kosut has quit IRC | 08:38 | |
| *** matt_kosut has joined #openstack-infra | 08:38 | |
| *** matt_kosut has quit IRC | 08:39 | |
| *** amoralej|off is now known as amoralej | 08:39 | |
| *** tosky has joined #openstack-infra | 08:40 | |
| *** Lucas_Gray has joined #openstack-infra | 08:42 | |
| *** Lucas_Gray has quit IRC | 08:45 | |
| *** Lucas_Gray has joined #openstack-infra | 08:46 | |
| *** jpena|off is now known as jpena | 08:47 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Report canceled changes via Github checks API https://review.opendev.org/711023 | 08:49 |
| *** dtantsur|afk is now known as dtantsur | 08:55 | |
| *** matt_kosut has joined #openstack-infra | 08:56 | |
| *** matt_kosut has quit IRC | 08:57 | |
| *** matt_kosut has joined #openstack-infra | 08:57 | |
| *** ykarel|lunch is now known as ykarel | 08:58 | |
| *** rpittau|afk is now known as rpittau | 08:58 | |
| *** tetsuro has quit IRC | 09:00 | |
| *** tetsuro_ has joined #openstack-infra | 09:01 | |
| *** tetsuro_ has quit IRC | 09:03 | |
| *** lxkong has quit IRC | 09:04 | |
| *** lxkong has joined #openstack-infra | 09:04 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Provide some documentation for the checks API implementation https://review.opendev.org/711493 | 09:05 |
| *** happyhemant has joined #openstack-infra | 09:06 | |
| openstackgerrit | Antoine Musso proposed zuul/zuul master: Enhance some logging messages https://review.opendev.org/704331 | 09:07 |
| *** gfidente has joined #openstack-infra | 09:07 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Provide some documentation for the checks API implementation https://review.opendev.org/711493 | 09:12 |
| *** ricolin has quit IRC | 09:12 | |
| *** zxiiro has quit IRC | 09:25 | |
| *** Lucas_Gray has quit IRC | 09:29 | |
| *** yolanda has quit IRC | 09:29 | |
| *** jamesmcarthur has joined #openstack-infra | 09:31 | |
| *** Lucas_Gray has joined #openstack-infra | 09:32 | |
| *** yolanda has joined #openstack-infra | 09:33 | |
| *** trident has quit IRC | 09:34 | |
| *** jamesmcarthur has quit IRC | 09:35 | |
| *** trident has joined #openstack-infra | 09:36 | |
| *** yamamoto has quit IRC | 09:58 | |
| *** xek_ has joined #openstack-infra | 10:09 | |
| *** derekh has joined #openstack-infra | 10:13 | |
| *** yamamoto has joined #openstack-infra | 10:15 | |
| *** ralonsoh has joined #openstack-infra | 10:17 | |
| *** aedc has joined #openstack-infra | 10:39 | |
| *** pkopec has joined #openstack-infra | 10:46 | |
| *** yamamoto has quit IRC | 11:01 | |
| *** yamamoto has joined #openstack-infra | 11:01 | |
| *** yamamoto has quit IRC | 11:11 | |
| *** jamesmcarthur has joined #openstack-infra | 11:11 | |
| *** yamamoto has joined #openstack-infra | 11:12 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 11:13 | |
| *** jamesmcarthur has quit IRC | 11:16 | |
| *** gshippey has joined #openstack-infra | 11:32 | |
| *** jcapitao is now known as jcapitao_lunch | 11:35 | |
| *** rpittau is now known as rpittau|bbl | 11:36 | |
| *** hashar has quit IRC | 11:38 | |
| *** jamesmcarthur has joined #openstack-infra | 11:43 | |
| openstackgerrit | Mark Goddard proposed openstack/project-config master: Add publish-to-pypi template for tenks https://review.opendev.org/711612 | 11:46 |
| *** Lucas_Gray has quit IRC | 11:46 | |
| *** ociuhandu has joined #openstack-infra | 11:46 | |
| *** Lucas_Gray has joined #openstack-infra | 11:47 | |
| *** jamesmcarthur has quit IRC | 11:48 | |
| *** ijw has quit IRC | 12:02 | |
| *** rkukura_ has joined #openstack-infra | 12:02 | |
| *** zigo has quit IRC | 12:02 | |
| *** ijw has joined #openstack-infra | 12:02 | |
| *** irclogbot_0 has quit IRC | 12:05 | |
| *** amotoki has quit IRC | 12:05 | |
| *** ddurst has quit IRC | 12:06 | |
| *** zzehring has quit IRC | 12:06 | |
| *** irclogbot_3 has joined #openstack-infra | 12:06 | |
| *** rkukura has quit IRC | 12:06 | |
| *** aarents has quit IRC | 12:06 | |
| *** rkukura_ is now known as rkukura | 12:06 | |
| *** irclogbot_3 has quit IRC | 12:07 | |
| *** zzehring has joined #openstack-infra | 12:07 | |
| *** aarents has joined #openstack-infra | 12:07 | |
| *** irclogbot_0 has joined #openstack-infra | 12:08 | |
| *** amotoki has joined #openstack-infra | 12:09 | |
| *** ddurst has joined #openstack-infra | 12:09 | |
| *** zigo has joined #openstack-infra | 12:11 | |
| *** yamamoto has quit IRC | 12:11 | |
| *** psachin has quit IRC | 12:13 | |
| *** nicolasbock has joined #openstack-infra | 12:18 | |
| *** zigo has quit IRC | 12:31 | |
| *** dtantsur is now known as dtantsur|afk | 12:33 | |
| *** jpena is now known as jpena|lunch | 12:37 | |
| *** jamesmcarthur has joined #openstack-infra | 12:45 | |
| *** hashar has joined #openstack-infra | 12:46 | |
| *** zigo has joined #openstack-infra | 12:48 | |
| *** jamesmcarthur has quit IRC | 12:49 | |
| *** yamamoto has joined #openstack-infra | 12:53 | |
| *** rlandy has joined #openstack-infra | 12:55 | |
| *** udesale has joined #openstack-infra | 12:55 | |
| *** yamamoto has quit IRC | 12:57 | |
| *** udesale_ has quit IRC | 12:57 | |
| *** psachin has joined #openstack-infra | 13:03 | |
| *** jcapitao_lunch is now known as jcapitao | 13:03 | |
| *** Lucas_Gray has quit IRC | 13:07 | |
| *** ociuhandu has quit IRC | 13:08 | |
| *** ociuhandu has joined #openstack-infra | 13:08 | |
| *** Lucas_Gray has joined #openstack-infra | 13:12 | |
| *** ociuhandu has quit IRC | 13:15 | |
| *** jamesmcarthur has joined #openstack-infra | 13:16 | |
| *** rpittau|bbl is now known as rpittau | 13:18 | |
| *** jamesmcarthur has quit IRC | 13:23 | |
| *** jamesmcarthur_ has joined #openstack-infra | 13:23 | |
| *** derekh has quit IRC | 13:25 | |
| *** yamamoto has joined #openstack-infra | 13:27 | |
| *** ociuhandu has joined #openstack-infra | 13:27 | |
| *** trident has quit IRC | 13:30 | |
| *** trident has joined #openstack-infra | 13:32 | |
| *** jpena|lunch is now known as jpena | 13:33 | |
| *** rh-jelabarre has joined #openstack-infra | 13:34 | |
| *** LiangFang has quit IRC | 13:45 | |
| *** jamesmcarthur_ has quit IRC | 13:49 | |
| *** yamamoto has quit IRC | 14:04 | |
| openstackgerrit | Merged openstack/cookiecutter master: Fix links formatting for the contributing documentation https://review.opendev.org/711407 | 14:06 |
| *** derekh has joined #openstack-infra | 14:08 | |
| *** owalsh has quit IRC | 14:10 | |
| *** psachin has quit IRC | 14:11 | |
| *** weshay is now known as weshay|ruck | 14:12 | |
| *** roman_g has joined #openstack-infra | 14:13 | |
| *** owalsh has joined #openstack-infra | 14:18 | |
| *** lbragstad has joined #openstack-infra | 14:18 | |
| *** trident has quit IRC | 14:22 | |
| *** trident has joined #openstack-infra | 14:24 | |
| *** dalvarez has joined #openstack-infra | 14:28 | |
| dalvarez | hi folks, we're exploring how to log IRC meetings in #openvswitch using Meetbot. Would it be possible to use the same meetbot that we have in OpenStack? | 14:29 |
| *** cgoncalves has quit IRC | 14:32 | |
| *** cgoncalves has joined #openstack-infra | 14:34 | |
| *** iurygregory has quit IRC | 14:34 | |
| *** Lucas_Gray has quit IRC | 14:38 | |
| *** Liang__ has joined #openstack-infra | 14:39 | |
| *** cgoncalves has quit IRC | 14:39 | |
| *** iurygregory has joined #openstack-infra | 14:40 | |
| *** cgoncalves has joined #openstack-infra | 14:41 | |
| *** amoralej is now known as amoralej|lunch | 14:41 | |
| *** ociuhandu has quit IRC | 14:42 | |
| *** Lucas_Gray has joined #openstack-infra | 14:42 | |
| fungi | dalvarez: first blp (who is the only listed admin for that channel according to its registration with chanserv) will need to grant permissions to our openstackinfra account as outlined at https://docs.openstack.org/infra/system-config/irc.html#access | 14:42 |
| dalvarez | fungi: great, i was wondering if we're fine to do that or it has to be strictly an openstack project channel | 14:43 |
| dalvarez | if we can do it, i'll talk to blp. It'd be awesome | 14:43 |
| fungi | nope. we eventually want to rename the account, the bot and the logs site to be opendevish rather than openstackish | 14:43 |
| fungi | so it's not intended to be openstack-specific | 14:44 |
| *** yamamoto has joined #openstack-infra | 14:44 | |
| fungi | though it is intended primarily for projects who are hosting their development in opendev | 14:44 |
| fungi | i don't think we've drawn a hard line in the past saying we wouldn't perform channel logging for other projects | 14:45 |
| *** udesale has quit IRC | 14:45 | |
| fungi | dalvarez: out of curiosity, seeing as how ovs is listed as an lf-supported project, does lf not provide similar options? | 14:46 |
| dalvarez | fungi: great question, let me explore | 14:47 |
| *** eharney has joined #openstack-infra | 14:47 | |
| dalvarez | fungi: https://docs.releng.linuxfoundation.org/en/latest/meetbot.html looks like it | 14:47 |
| *** jlibosva has joined #openstack-infra | 14:51 | |
| *** soumith has joined #openstack-infra | 14:52 | |
| AJaeger | fungi: we're at capacity AFAIK, so better run a second bot... | 14:52 |
| *** soumith has quit IRC | 14:52 | |
| *** soumith has joined #openstack-infra | 14:53 | |
| *** ykarel is now known as ykarel|away | 14:53 | |
| *** dave-mccowan has joined #openstack-infra | 14:54 | |
| fungi | AJaeger: yeah, we could probably also stand to do another audit looking for defunct channels to /part the bot from | 14:54 |
| fungi | i'll see if i have my old one-liner command for finding those somewhere handy | 14:55 |
| fungi | currently digging in the logs to see if i can identify any channels which have been silent (other than bots and parts/joins) since the end of september | 15:00 |
| *** owalsh has quit IRC | 15:00 | |
| fungi | these are channels which have had no human conversation in ~6 months: #scientific-wg #openstack-women #openstack-sprint #openstack-outreachy #openstack-net-bgpvpn #openstack-heat-translator #openstack-forum #openstack-dragonflow #congress | 15:01 |
| *** amoralej|lunch is now known as amoralej | 15:02 | |
| fungi | and these channels have averaged fewer than one human message a month in that timeframe: #openstack-browbeat #openstack-ec2api #openstack-performance #openstack-tricircle #murano | 15:02 |
| *** Lucas_Gray has quit IRC | 15:06 | |
| *** matt_kosut has quit IRC | 15:09 | |
| *** sreejithp has joined #openstack-infra | 15:11 | |
| *** yamamoto has quit IRC | 15:13 | |
| *** ociuhandu has joined #openstack-infra | 15:16 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update sandbox intro for OpenDev https://review.opendev.org/711639 | 15:16 |
| AJaeger | frickler: thanks for reviews, here's the tiny update ^ | 15:16 |
| *** matt_kosut has joined #openstack-infra | 15:18 | |
| openstackgerrit | Merged openstack/infra-manual master: Update Developer's Guide for OpenDev https://review.opendev.org/711536 | 15:20 |
| AJaeger | gmann: is https://review.opendev.org/711531 now fine? | 15:20 |
| AJaeger | gmann: thanks for your reviews on that one | 15:20 |
| *** ociuhandu has quit IRC | 15:21 | |
| *** ociuhandu has joined #openstack-infra | 15:21 | |
| *** matt_kosut has quit IRC | 15:23 | |
| *** jlibosva is now known as Guest92703 | 15:24 | |
| *** jlibosva has joined #openstack-infra | 15:25 | |
| *** lpetrut has quit IRC | 15:25 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update "Outbound Third-Party Testing" for OpenDev https://review.opendev.org/711508 | 15:26 |
| *** Guest92703 has quit IRC | 15:26 | |
| frickler | AJaeger: you're amending patches faster than I can review them, that's not fair ;) | 15:29 |
| openstackgerrit | Merged openstack/infra-manual master: Update Driver's Guide for OpenDev https://review.opendev.org/711534 | 15:29 |
| gmann | AJaeger: your update lgtm from /join page but not with legal page. not sure its just me reading it as conflict between both. | 15:29 |
| openstackgerrit | Merged ttygroup/gertty master: Fix error in message refresh https://review.opendev.org/711523 | 15:31 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update "Code Review" for OpenDev https://review.opendev.org/711514 | 15:32 |
| *** Goneri has joined #openstack-infra | 15:33 | |
| AJaeger | frickler: I'll try to keep you busy ;) Thanks! | 15:33 |
| openstackgerrit | Zane Bitter proposed openstack/cookiecutter master: Autofill bug tracker link in contributor docs https://review.opendev.org/708703 | 15:34 |
| *** owalsh has joined #openstack-infra | 15:34 | |
| AJaeger | gmann: I suggest we merge what we have - and let diablo_rojo_phon figure out the legalese on that one. | 15:34 |
| *** Goneri has quit IRC | 15:38 | |
| *** owalsh has quit IRC | 15:40 | |
| *** soumith has quit IRC | 15:41 | |
| *** owalsh has joined #openstack-infra | 15:50 | |
| *** jcapitao is now known as jcapitao_afk | 15:50 | |
| *** bnemec is now known as beekneemech | 15:53 | |
| AJaeger | infra-root, translation files where published to wrong place, it was fixed earlier today. Could you remove content from https://tarballs.opendev.org/translation-source/ , please? Correct location is https://tarballs.opendev.org/openstack/translation-source/ | 15:53 |
| *** owalsh_ has joined #openstack-infra | 15:53 | |
| *** owalsh- has joined #openstack-infra | 15:54 | |
| *** hashar has quit IRC | 15:55 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Remove "Eligibility to Vote in Elections" section https://review.opendev.org/711532 | 15:55 |
| *** owalsh has quit IRC | 15:57 | |
| *** owalsh_ has quit IRC | 15:58 | |
| clarkb | AJaeger: do we need to mv any of that content or just let it be repopulated as jobs run? | 16:03 |
| clarkb | I'm having another slow start but will try to resolve that soon | 16:05 |
| AJaeger | clarkb: repopulate is good enough | 16:05 |
| AJaeger | thanks, clarkb | 16:05 |
| mordred | clarkb: could you add https://review.opendev.org/#/c/578557/ to your review list? | 16:05 |
| clarkb | mordred: sure | 16:05 |
| mordred | clarkb: it's got 2x+2 but it's an important change | 16:06 |
| AJaeger | clarkb, fungi, mordred, I pushed a few more changes for infra-manual, reviews welcome ( frickler did a round already), please | 16:06 |
| *** ykarel|away is now known as ykarel | 16:11 | |
| *** mattw4 has joined #openstack-infra | 16:12 | |
| *** pgaxatte has quit IRC | 16:12 | |
| fungi | need to go run some lunch errands but should return in an hour or so | 16:12 |
| *** jaosorior has quit IRC | 16:14 | |
| *** jcapitao_afk is now known as jcapitao | 16:16 | |
| *** jamesmcarthur has joined #openstack-infra | 16:19 | |
| *** kopecmartin is now known as kopecmartin|afk | 16:29 | |
| *** gyee has joined #openstack-infra | 16:29 | |
| *** matt_kosut has joined #openstack-infra | 16:31 | |
| *** owalsh has joined #openstack-infra | 16:32 | |
| *** owalsh_ has joined #openstack-infra | 16:33 | |
| *** owalsh| has joined #openstack-infra | 16:34 | |
| *** ijw has quit IRC | 16:35 | |
| *** owalsh^ has joined #openstack-infra | 16:35 | |
| *** matt_kosut has quit IRC | 16:35 | |
| *** owalsh- has quit IRC | 16:35 | |
| *** owalsh has quit IRC | 16:37 | |
| *** chandankumar is now known as raukadah | 16:38 | |
| *** owalsh_ has quit IRC | 16:38 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Update 'Feature Branches' for OpenDev https://review.opendev.org/711650 | 16:38 |
| *** owalsh| has quit IRC | 16:39 | |
| *** pkopec has quit IRC | 16:39 | |
| *** armax has joined #openstack-infra | 16:41 | |
| clarkb | ok I'm finally actually at my desk | 16:43 |
| clarkb | gonna start with the zuul review then figure out afs (probably on a server since i think tumbleweed isn't working again) | 16:43 |
| openstackgerrit | Zane Bitter proposed openstack/cookiecutter master: Autofill bug tracker link in contributor docs https://review.opendev.org/708703 | 16:47 |
| openstackgerrit | Merged zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 16:48 |
| *** ociuhandu_ has joined #openstack-infra | 16:54 | |
| openstackgerrit | Merged zuul/zuul master: Test that retries don't trigger fail-fast https://review.opendev.org/704983 | 16:54 |
| *** ociuhandu has quit IRC | 16:57 | |
| *** ociuhandu_ has quit IRC | 16:58 | |
| *** rpittau is now known as rpittau|afk | 17:04 | |
| clarkb | AJaeger: infra-root I've rm'ed that translation-source/ dir and I think vos release just happened automatically and it isn't listed at https://tarballs.opendev.org/ anymore | 17:05 |
| *** jamesmcarthur has quit IRC | 17:05 | |
| *** iurygregory has quit IRC | 17:05 | |
| *** hashar has joined #openstack-infra | 17:06 | |
| clarkb | AJaeger: if you get a chance to confirm it looks good now that would be great | 17:06 |
| *** gfidente has quit IRC | 17:08 | |
| *** ijw has joined #openstack-infra | 17:10 | |
| *** jlibosva has quit IRC | 17:10 | |
| *** yamamoto has joined #openstack-infra | 17:11 | |
| clarkb | fungi: maybe send notice to openstack-discuss since those are all openstack channels then pull them next week if there isn't any objection? | 17:18 |
| openstackgerrit | Merged zuul/zuul master: Don't reset file list to None in updateChange https://review.opendev.org/704328 | 17:20 |
| openstackgerrit | Merged zuul/zuul master: Protect prime_installation_map with mutex https://review.opendev.org/704932 | 17:20 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/cookiecutter master: [WIP] Add CI job to generate and build docs for cookiecutter https://review.opendev.org/711674 | 17:20 |
| AJaeger | thanks, clarkb - all fine | 17:21 |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/cookiecutter master: [WIP] Add CI job to generate and build docs for cookiecutter https://review.opendev.org/711674 | 17:23 |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Redraw code_review image https://review.opendev.org/711675 | 17:25 |
| AJaeger | team, I redraw the code-review graphic on the Developers's Guide to get rid of jenkins. Please review and tell me what to change. And if any of you is a better artist than me, feel free to take over ;) I can provide a libreoffice source as well if wanted. ^ | 17:26 |
| *** diablo_rojo has joined #openstack-infra | 17:28 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/cookiecutter master: [WIP] Add CI job to generate and build docs for cookiecutter https://review.opendev.org/711674 | 17:29 |
| *** dtroyer has quit IRC | 17:33 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/cookiecutter master: [WIP] Add CI job to generate and build docs for cookiecutter https://review.opendev.org/711674 | 17:33 |
| *** evrardjp has quit IRC | 17:35 | |
| openstackgerrit | Jean-Philippe Evrard proposed openstack/cookiecutter master: [WIP] Add CI job to generate and build docs for cookiecutter https://review.opendev.org/711674 | 17:35 |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** artom has quit IRC | 17:38 | |
| *** artom has joined #openstack-infra | 17:38 | |
| *** ykarel is now known as ykarel|away | 17:54 | |
| *** amoralej is now known as amoralej|off | 17:54 | |
| openstackgerrit | Merged zuul/zuul master: Centralize merge handling https://review.opendev.org/707692 | 17:56 |
| *** jpena is now known as jpena|off | 17:57 | |
| *** derekh has quit IRC | 18:00 | |
| openstackgerrit | Merged zuul/zuul master: Uncap GitPython https://review.opendev.org/708593 | 18:03 |
| openstackgerrit | Merged zuul/zuul master: Log duration of tenant reconfigurations https://review.opendev.org/708403 | 18:04 |
| openstackgerrit | Merged zuul/zuul master: Reduce gearman logging in tests https://review.opendev.org/707587 | 18:04 |
| openstackgerrit | Merged zuul/zuul master: Increase timeout in timeout test https://review.opendev.org/710146 | 18:04 |
| openstackgerrit | Merged zuul/zuul master: web ui: fix buildset display when no builds https://review.opendev.org/710508 | 18:04 |
| openstackgerrit | Merged zuul/zuul master: Optimize canMerge using graphql https://review.opendev.org/709836 | 18:08 |
| clarkb | AJaeger: the diagram looks good, but is missing the commit amend portion of the flow | 18:09 |
| clarkb | AJaeger: do you want to add that in or would you prefer someone else give it a go? | 18:09 |
| AJaeger | clarkb: oh, indeed, let me add... | 18:09 |
| AJaeger | I can add it | 18:09 |
| clarkb | AJaeger: also if you can round off the arrow lines they render a bit nicer (with anti aliasing or whatever) | 18:10 |
| clarkb | (not critical, the information is still carried just fine) | 18:10 |
| *** jcapitao has quit IRC | 18:10 | |
| *** eharney has quit IRC | 18:12 | |
| openstackgerrit | Merged zuul/zuul master: Increase zookeeper timeout during tests https://review.opendev.org/707945 | 18:12 |
| openstackgerrit | Merged zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 18:12 |
| *** rlandy is now known as rlandy|brb | 18:13 | |
| *** dtroyer has joined #openstack-infra | 18:14 | |
| *** tesseract has quit IRC | 18:15 | |
| *** larainema has quit IRC | 18:16 | |
| *** igordc has joined #openstack-infra | 18:20 | |
| *** igordc has quit IRC | 18:20 | |
| *** gmann is now known as gmann_lunch | 18:27 | |
| openstackgerrit | Merged openstack/infra-manual master: Update elastic-recheck comment https://review.opendev.org/711586 | 18:28 |
| openstackgerrit | Merged openstack/infra-manual master: Document why drafts are disabled https://review.opendev.org/711550 | 18:32 |
| *** gmann_lunch is now known as gmann | 18:32 | |
| openstackgerrit | Merged openstack/infra-manual master: Update sandbox intro for OpenDev https://review.opendev.org/711639 | 18:32 |
| *** hashar has quit IRC | 18:33 | |
| *** rlandy|brb is now known as rlandy | 18:33 | |
| openstackgerrit | Andreas Jaeger proposed openstack/infra-manual master: Redraw code_review image https://review.opendev.org/711675 | 18:35 |
| AJaeger | clarkb: updated - if that needs fine tuning, I need help... | 18:35 |
| openstackgerrit | Merged zuul/zuul master: Add reference pipelines file for Github driver https://review.opendev.org/672712 | 18:40 |
| *** ralonsoh has quit IRC | 18:40 | |
| *** happyhemant has quit IRC | 18:40 | |
| *** jamesmcarthur has joined #openstack-infra | 18:41 | |
| fungi | clarkb: yeah, i'll send something out letting folks know we'll be removing those channels unless there are objections | 18:42 |
| *** ociuhandu has joined #openstack-infra | 18:47 | |
| *** diablo_rojo has quit IRC | 18:50 | |
| *** gshippey has quit IRC | 18:52 | |
| *** ociuhandu has quit IRC | 18:52 | |
| *** pkopec has joined #openstack-infra | 18:56 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add a test for retroactive branch tag application https://review.opendev.org/711741 | 18:58 |
| donnyd | fungi - do you think you would have time today to start working at bringing FN back online | 18:59 |
| clarkb | I can probably help too (though I'm juggling some reviews at the moment) | 19:01 |
| donnyd | oh yes - i will take any help I can get | 19:01 |
| clarkb | donnyd: I'm not quite up to date on what needs doing next, can you fill me (the channel too) in | 19:02 |
| donnyd | I am just getting your networking finished up now | 19:02 |
| donnyd | So FN was actually acquired by an Italian company called Open Edge | 19:03 |
| clarkb | AJaeger: the diagram looks good to me now | 19:03 |
| donnyd | So I have been working at moving all of the api endpoints, refactoring the infra for HA, and building the final component of what they needed it to do | 19:03 |
| donnyd | There is an agreement that the Openstack CI work will continue | 19:04 |
| clarkb | gotcha so on our side we'll need to update our clouds.yaml connection settings for the APIs. Will we need to rebuild the mirror (assuming that existing resources may have been purged)? | 19:04 |
| donnyd | Correct clarkb | 19:04 |
| *** yamamoto has quit IRC | 19:04 | |
| *** pkopec has quit IRC | 19:04 | |
| AJaeger | infra-root, would be great to get some reviews on the next round of changes for infra-manual, please: https://review.opendev.org/#/q/project:openstack/infra-manual+is:open | 19:05 |
| donnyd | Just putting the finishing touches on the network setup and I will be ready to build and test the mirror | 19:05 |
| clarkb | donnyd: is that something we should work wtih Open Edge to do? (thinking out loud that that might help us build the relationship there) | 19:06 |
| donnyd | Well the relationship with the Openstack community is still managed by me | 19:06 |
| donnyd | :) | 19:06 |
| donnyd | So yes - and you are | 19:06 |
| clarkb | got it, thanks for clarifying that | 19:07 |
| clarkb | donnyd: so the first step for us then before building a mirror is updating our clouds.yaml stuff and ensuring that our cloud resources provisioner (security groups, ssh keys, etc) runs | 19:07 |
| donnyd | all things considered - not much is changing for Openstack | 19:07 |
| donnyd | yes - should I paste the creds here? | 19:08 |
| donnyd | you rotate them anyways if I am not mistaken | 19:08 |
| clarkb | we do, but maybe PM them to someone who can rotate them relatively quickly (I'm not quite yet in that spot, but probably after lunch I can do that) | 19:08 |
| donnyd | okie dokey | 19:09 |
| clarkb | and ya typically step 0 is get working clouds.yaml with details, then step 1 is rotate the passwd | 19:09 |
| *** eharney has joined #openstack-infra | 19:09 | |
| openstackgerrit | Merged openstack/infra-manual master: Update "Outbound Third-Party Testing" for OpenDev https://review.opendev.org/711508 | 19:09 |
| clarkb | fungi: ^ not sure if you are in a better spot for that | 19:10 |
| fungi | i'm catching back up, but also still trying to do debrief in the wake of the open cloud workshop. so i might be able to in the next couple of hours sure | 19:10 |
| donnyd | no rush | 19:11 |
| openstackgerrit | Merged zuul/zuul master: Improve error reporting when pr merge fails https://review.opendev.org/707637 | 19:12 |
| donnyd | fungi: clarkb I think its also just fine to not have to do the work to rename everything | 19:14 |
| donnyd | we can keep what we have to minimize the amount of work to get this back into operation | 19:14 |
| clarkb | I think its ok to update. Especially if that is cleaner long term since it salready off right now | 19:16 |
| *** ahosam has joined #openstack-infra | 19:18 | |
| donnyd | does it matter what private network I create for the mirror? | 19:21 |
| *** jamesmcarthur has quit IRC | 19:24 | |
| *** diablo_rojo has joined #openstack-infra | 19:25 | |
| clarkb | donnyd: it shouldnt, ideally its different than the test nodes | 19:25 |
| fungi | for some reason i thought we had some details published as to what rfc 1918 addressing gets used in devstack jobs... but i'm not finding it | 19:25 |
| clarkb | and will need ipv4 fip to be attached | 19:25 |
| donnyd | yes it will be different | 19:25 |
| donnyd | yes | 19:25 |
| donnyd | I used 192.168.120.0/24 for the v4 network | 19:26 |
| fungi | though if the test nodes are going to talk to the mirror's global address then it shouldn't matter | 19:26 |
| donnyd | can you ping6 this ```2001:470:e126:0:f816:3eff:fe5b:8501``` | 19:27 |
| fungi | yep | 19:27 |
| fungi | i can reach that from home | 19:27 |
| donnyd | and how about ```108.44.198.35``` | 19:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Implement zookeeper-auth https://review.opendev.org/619155 | 19:28 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Add mypy to linter test https://review.opendev.org/711750 | 19:28 |
| fungi | donnyd: yep, reachable from here | 19:29 |
| fungi | coincidentally, i discovered a few weeks ago that my broadband provider has finally rolled out v6 prefix delegation via dhcp6, so i've dropped my gre tunnel to hurricane electric once and for all | 19:29 |
| donnyd | fios? | 19:29 |
| donnyd | I was trying to get mine working - but no luck | 19:30 |
| fungi | charter/specrtrum cable | 19:30 |
| donnyd | and the HE thing seems to work pretty good | 19:30 |
| fungi | i'm only getting a /64 from them, though i gather that with the right option set in my dhcp6 request i may be able to get a /56 or /48 | 19:30 |
| donnyd | I have no complaints - I am really really close to the tunnel endpoint | 19:30 |
| donnyd | its 15 miles from where my edge sits | 19:30 |
| donnyd | but it does effect the MTU | 19:31 |
| fungi | right, for me iad was the closest endpoint in he but the backhaul to get there was not so close | 19:31 |
| donnyd | yea that makes sense - iad is a stones thrown away from here | 19:31 |
| donnyd | just making sure migration works - all the other functions have been tested | 19:32 |
| fungi | i figured. i'm (barely) over the border into nc and as far east as you can get, and then they backhaul everything from the island to somewhere way inland before they connect to a common backbone | 19:32 |
| donnyd | yea that was probably a little latent | 19:32 |
| fungi | so even though geographical distance for me is only a few hundred miles to iad, as the packet flies it's probably a lot more | 19:33 |
| fungi | also happy to no longer need the workarounds for things like netflix blocking he's v6 addresses | 19:34 |
| fungi | traceroute from here to 2001:470:e126:0:f816:3eff:fe5b:8501 takes me through a peer in atl :/ | 19:36 |
| donnyd | what is latency like | 19:37 |
| fungi | and traceroute to 108.44.198.35 goes through chicago | 19:37 |
| donnyd | totally random | 19:38 |
| fungi | average rtt for both the v4 and v6 addresses is ~53ms at the moment | 19:38 |
| fungi | i'm sure charter is just buying space on whichever links they're getting cheapest for the month, so in a few days it could be totally different | 19:39 |
| openstackgerrit | Merged zuul/zuul master: Add load-branch to tenant configuration https://review.opendev.org/705664 | 19:41 |
| donnyd | that isn't too bad though | 19:42 |
| donnyd | considering where you are | 19:42 |
| fungi | yup | 19:42 |
| fungi | when it manages to stay up anyway ;) | 19:43 |
| *** matt_kosut has joined #openstack-infra | 19:53 | |
| donnyd | ok - migrations finally work like they should | 19:54 |
| donnyd | I am ready whenever you are | 19:54 |
| fungi | just a sec and i can get a revert going and edit up the necessary bits | 19:55 |
| fungi | i guess i should update the creds in our ansible vars first | 19:56 |
| *** matt_kosut has quit IRC | 19:58 | |
| *** diablo_rojo has quit IRC | 19:58 | |
| donnyd | both projects are setup with networking (on different v4 subnets) | 20:01 |
| *** diablo_rojo has joined #openstack-infra | 20:02 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Implement zookeeper-auth https://review.opendev.org/619155 | 20:02 |
| donnyd | pm sent fungi | 20:03 |
| openstackgerrit | Daniel Alvarez proposed opendev/system-config master: Add #openvswitch to meetbot channels https://review.opendev.org/711756 | 20:03 |
| fungi | thanks donnyd, received! | 20:08 |
| donnyd | I will work at swapping out all the names | 20:08 |
| donnyd | and updating the region data | 20:08 |
| openstackgerrit | Donny Davis proposed openstack/project-config master: Moving FortNebula to OpenEdge https://review.opendev.org/711760 | 20:16 |
| * donnyd waves at AJaeger | 20:18 | |
| * AJaeger waves back | 20:19 | |
| fungi | okay, i have the openedge credentials for openstackci and openstackzuul reset, tested and recorded in our private ansible vars | 20:24 |
| donnyd | awesome fungi - thank you | 20:25 |
| donnyd | so now we wait | 20:25 |
| donnyd | so now we wait for the next run | 20:25 |
| fungi | variable names are basically s/fortnebula/openedge/ | 20:26 |
| clarkb | thanks! I've finished lunch and am applying updates and revooting | 20:26 |
| fungi | upi | 20:26 |
| fungi | er, that was a strange finger misalignment | 20:26 |
| fungi | clarkb: you're revolting? | 20:26 |
| donnyd | oh no - clarkb I hope this didn't make you that upset to revolt | 20:26 |
| fungi | viva la revootućion! | 20:26 |
| *** ahosam has quit IRC | 20:29 | |
| clarkb | the reboot revolution | 20:29 |
| *** ahosam has joined #openstack-infra | 20:30 | |
| donnyd | :) | 20:30 |
| * fungi forgot to start that with a ¡ | 20:31 | |
| fungi | okay, our backup encrypted credential record is updated to add these as well | 20:31 |
| donnyd | I will be standing by to watch for errors | 20:32 |
| fungi | is there already a change up to update the references under https://opendev.org/opendev/system-config/src/branch/master/playbooks ? if not i'll start on that next | 20:34 |
| donnyd | no | 20:34 |
| fungi | on it | 20:34 |
| donnyd | I didn't touch system-config | 20:34 |
| donnyd | mainly because I didn't know I was supposed to | 20:34 |
| donnyd | thank you fungi | 20:35 |
| fungi | no worries, that's how the clouds.yaml files get generated | 20:35 |
| donnyd | is the mirror nodes flavors supposed to be 250G disk? | 20:36 |
| fungi | that sounds right, or we need to attach ~200gb of block storage for the caches | 20:36 |
| donnyd | for some reason I thought it was 200 | 20:36 |
| clarkb | ya can ptobably get away with 220 ish | 20:36 |
| donnyd | I can update the flavor no prob | 20:37 |
| fungi | right, that would work | 20:37 |
| fungi | 220 ought to be fine | 20:37 |
| donnyd | looking is system-config it looks like its supposed to be 250 | 20:37 |
| donnyd | I will update it to the proper amount | 20:37 |
| donnyd | https://opendev.org/opendev/system-config/src/branch/master/playbooks/clouds_layouts.yml | 20:38 |
| *** weshay|ruck is now known as weshay | 20:41 | |
| openstackgerrit | Merged openstack/project-config master: Moving FortNebula to OpenEdge https://review.opendev.org/711760 | 20:42 |
| clarkb | and now rebooting is done | 20:42 |
| donnyd | I very un-creatively named the mirrors flavor 8cpu-8GBram-250GBdisk | 20:43 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Moving FortNebula to OpenEdge https://review.opendev.org/711768 | 20:44 |
| fungi | donnyd: clarkb: ^ see if that looks correct to you | 20:44 |
| clarkb | looking | 20:45 |
| fungi | it's times like this when i realize how much repetition we have in that repo | 20:46 |
| clarkb | donnyd: fungi are the network details still correct at https://review.opendev.org/#/c/711768/1/playbooks/templates/clouds/nodepool_builder_clouds.yaml.j2 ? | 20:46 |
| clarkb | basically ipv6 only, no ipv4 except through NAT as well as the name | 20:47 |
| fungi | good q, i assumed that was the case but did not ask | 20:47 |
| donnyd | https://www.irccloud.com/pastebin/joQv1alE/ | 20:49 |
| donnyd | they look correct to me | 20:49 |
| *** diablo_rojo has quit IRC | 20:50 | |
| clarkb | fungi: +2'd based on donnyd's paste above. I think we can land that as soon as hiera/hostvars are updated | 20:51 |
| donnyd | nothing has changed in the network architecture here | 20:51 |
| fungi | clarkb: yeah, per earlier i already updated our private host/group vars on bridge.o.o to add those | 20:52 |
| fungi | (i haven't deleted the old fortnebula entries yet though, will do that once the system-config change lands) | 20:53 |
| clarkb | sounds good | 20:53 |
| fungi | i went ahead and self-approved it noting that | 20:54 |
| *** diablo_rojo has joined #openstack-infra | 20:55 | |
| *** ociuhandu has joined #openstack-infra | 20:55 | |
| *** dave-mccowan has quit IRC | 21:11 | |
| *** lxkong has quit IRC | 21:11 | |
| *** jaicaa has quit IRC | 21:11 | |
| *** mwhahaha has quit IRC | 21:11 | |
| *** gagehugo has quit IRC | 21:11 | |
| *** Shrews has quit IRC | 21:11 | |
| *** dayou has quit IRC | 21:11 | |
| *** rosmaita has quit IRC | 21:11 | |
| *** stevebaker has quit IRC | 21:11 | |
| *** iokiwi has quit IRC | 21:11 | |
| *** tinwood has quit IRC | 21:13 | |
| *** portdirect has quit IRC | 21:13 | |
| *** StevenK has quit IRC | 21:13 | |
| *** ChanServ has quit IRC | 21:13 | |
| *** adam_g has quit IRC | 21:13 | |
| *** diablo_rojo_phon has quit IRC | 21:13 | |
| *** clayg has quit IRC | 21:13 | |
| *** melwitt has quit IRC | 21:13 | |
| *** johnsom has quit IRC | 21:13 | |
| *** tonyb[m] has quit IRC | 21:13 | |
| *** jdelaros1 has quit IRC | 21:13 | |
| *** kevinz has quit IRC | 21:13 | |
| *** jungleboyj has quit IRC | 21:13 | |
| *** noonedeadpunk has quit IRC | 21:13 | |
| *** aspiers has quit IRC | 21:13 | |
| *** jkt has quit IRC | 21:13 | |
| *** corvus has quit IRC | 21:13 | |
| *** jonher has quit IRC | 21:13 | |
| *** mugsie has quit IRC | 21:13 | |
| *** Jeffrey4l has quit IRC | 21:13 | |
| *** jrosser has quit IRC | 21:13 | |
| *** donnyd has quit IRC | 21:13 | |
| *** knikolla has quit IRC | 21:13 | |
| *** logan- has quit IRC | 21:13 | |
| *** davecore has quit IRC | 21:13 | |
| *** rajinir has quit IRC | 21:13 | |
| *** MarkMaglana has quit IRC | 21:13 | |
| *** srwilkers has quit IRC | 21:13 | |
| *** evgenyl has quit IRC | 21:13 | |
| *** rpioso has quit IRC | 21:13 | |
| *** dustinc has quit IRC | 21:13 | |
| *** mudpuppy has quit IRC | 21:13 | |
| *** jroll has quit IRC | 21:13 | |
| *** pots has quit IRC | 21:13 | |
| *** darvon has quit IRC | 21:13 | |
| *** DinaBelova has quit IRC | 21:13 | |
| *** ociuhandu has quit IRC | 21:13 | |
| *** diablo_rojo has quit IRC | 21:13 | |
| *** roman_g has quit IRC | 21:13 | |
| *** zigo has quit IRC | 21:13 | |
| *** ddurst has quit IRC | 21:13 | |
| *** zzehring has quit IRC | 21:13 | |
| *** mgoddard has quit IRC | 21:13 | |
| *** zzzeek has quit IRC | 21:13 | |
| *** SpamapS has quit IRC | 21:13 | |
| *** Ramereth has quit IRC | 21:13 | |
| *** kmarc has quit IRC | 21:13 | |
| *** benj_ has quit IRC | 21:13 | |
| *** edausq has quit IRC | 21:13 | |
| *** eandersson has quit IRC | 21:13 | |
| *** amoralej|off has quit IRC | 21:13 | |
| *** asettle has quit IRC | 21:13 | |
| *** ildikov has quit IRC | 21:13 | |
| *** arxcruz|rover has quit IRC | 21:13 | |
| *** jpena|off has quit IRC | 21:13 | |
| *** dalvarez has quit IRC | 21:13 | |
| *** spotz has quit IRC | 21:13 | |
| *** rh-jelabarre has quit IRC | 21:13 | |
| *** nicolasbock has quit IRC | 21:13 | |
| *** philroche has quit IRC | 21:13 | |
| *** howell has quit IRC | 21:13 | |
| *** beisner has quit IRC | 21:13 | |
| *** dougwig has quit IRC | 21:13 | |
| *** seongsoocho has quit IRC | 21:13 | |
| *** zbr|pto has quit IRC | 21:13 | |
| *** gmann has quit IRC | 21:13 | |
| *** bstinson has quit IRC | 21:13 | |
| *** tristanC has quit IRC | 21:13 | |
| *** smcginnis has quit IRC | 21:13 | |
| *** csatari has quit IRC | 21:13 | |
| *** cjohnston has quit IRC | 21:13 | |
| *** persia has quit IRC | 21:13 | |
| *** jmccrory has quit IRC | 21:13 | |
| *** Ng has quit IRC | 21:13 | |
| *** lseki has quit IRC | 21:13 | |
| *** odyssey4me has quit IRC | 21:13 | |
| *** davidlenwell has quit IRC | 21:13 | |
| *** kukacz has quit IRC | 21:13 | |
| *** cgoncalves has quit IRC | 21:13 | |
| *** lbragstad has quit IRC | 21:13 | |
| *** yolanda has quit IRC | 21:13 | |
| *** slaweq has quit IRC | 21:13 | |
| *** weshay has quit IRC | 21:13 | |
| *** exsdev has quit IRC | 21:13 | |
| *** kaisers has quit IRC | 21:13 | |
| *** nhicher has quit IRC | 21:13 | |
| *** calbers has quit IRC | 21:13 | |
| *** prometheanfire has quit IRC | 21:13 | |
| *** fdegir has quit IRC | 21:13 | |
| *** jhesketh has quit IRC | 21:13 | |
| *** yoctozepto has quit IRC | 21:13 | |
| *** freerunner has quit IRC | 21:13 | |
| *** Qiming has quit IRC | 21:13 | |
| *** tobias-urdin has quit IRC | 21:13 | |
| *** dmsimard|off has quit IRC | 21:13 | |
| *** lifeless has quit IRC | 21:13 | |
| *** JayF has quit IRC | 21:13 | |
| *** tbarron has quit IRC | 21:13 | |
| *** ianw has quit IRC | 21:13 | |
| *** tobberydberg has quit IRC | 21:13 | |
| *** EmilienM has quit IRC | 21:13 | |
| *** antonym has quit IRC | 21:13 | |
| *** amorin has quit IRC | 21:13 | |
| *** dtroyer has quit IRC | 21:13 | |
| *** owalsh^ has quit IRC | 21:13 | |
| *** irclogbot_0 has quit IRC | 21:13 | |
| *** ccamacho has quit IRC | 21:13 | |
| *** apetrich has quit IRC | 21:13 | |
| *** KeithMnemonic1 has quit IRC | 21:13 | |
| *** ramishra has quit IRC | 21:13 | |
| *** dmellado has quit IRC | 21:13 | |
| *** johnthetubaguy has quit IRC | 21:13 | |
| *** tomaw has quit IRC | 21:13 | |
| *** vesper11 has quit IRC | 21:13 | |
| *** evrardjp has quit IRC | 21:13 | |
| *** aedc has quit IRC | 21:13 | |
| *** hwoarang has quit IRC | 21:13 | |
| *** dciabrin__ has quit IRC | 21:13 | |
| *** mrmartin has quit IRC | 21:13 | |
| *** masayukig has quit IRC | 21:13 | |
| *** onovy has quit IRC | 21:13 | |
| *** dancek has quit IRC | 21:13 | |
| *** arne_wiebalck has quit IRC | 21:13 | |
| *** jberg-dev has quit IRC | 21:13 | |
| *** jbryce has quit IRC | 21:13 | |
| *** mnasiadka has quit IRC | 21:13 | |
| *** hrybacki has quit IRC | 21:13 | |
| *** cyberpear has quit IRC | 21:13 | |
| *** Adri2000 has quit IRC | 21:13 | |
| *** fungi has quit IRC | 21:13 | |
| *** d34dh0r53 has quit IRC | 21:13 | |
| *** cloudnull has quit IRC | 21:13 | |
| *** tonyb has quit IRC | 21:13 | |
| *** cjloader has quit IRC | 21:13 | |
| *** rm_work has quit IRC | 21:13 | |
| *** ab-a has quit IRC | 21:13 | |
| *** ahosam has quit IRC | 21:13 | |
| *** artom has quit IRC | 21:13 | |
| *** rkukura has quit IRC | 21:13 | |
| *** otherwiseguy has quit IRC | 21:13 | |
| *** factor has quit IRC | 21:13 | |
| *** toabctl has quit IRC | 21:13 | |
| *** Tengu has quit IRC | 21:13 | |
| *** admcleod has quit IRC | 21:13 | |
| *** bdodd has quit IRC | 21:13 | |
| *** tobiash has quit IRC | 21:13 | |
| *** jistr has quit IRC | 21:13 | |
| *** elod has quit IRC | 21:13 | |
| *** timburke has quit IRC | 21:13 | |
| *** clarkb has quit IRC | 21:13 | |
| *** redrobot has quit IRC | 21:13 | |
| *** ianychoi has quit IRC | 21:13 | |
| *** dansmith has quit IRC | 21:13 | |
| *** brwyatt has quit IRC | 21:13 | |
| *** aluria has quit IRC | 21:13 | |
| *** calebb has quit IRC | 21:13 | |
| *** hemna has quit IRC | 21:13 | |
| *** dabukalam has quit IRC | 21:13 | |
| *** guilhermesp has quit IRC | 21:13 | |
| *** dirk has quit IRC | 21:13 | |
| *** ericyoung has quit IRC | 21:13 | |
| *** tdasilva has quit IRC | 21:13 | |
| *** fnordahl has quit IRC | 21:13 | |
| *** mnencia has quit IRC | 21:13 | |
| *** gouthamr has quit IRC | 21:13 | |
| *** frickler has quit IRC | 21:13 | |
| *** Diabelko has quit IRC | 21:13 | |
| *** sdoran has quit IRC | 21:13 | |
| *** mordred has quit IRC | 21:13 | |
| *** andreaf has quit IRC | 21:13 | |
| *** dtantsur|afk has quit IRC | 21:13 | |
| *** armax has quit IRC | 21:13 | |
| *** mattw4 has quit IRC | 21:13 | |
| *** sreejithp has quit IRC | 21:13 | |
| *** tosky has quit IRC | 21:13 | |
| *** sshnaidm|afk has quit IRC | 21:13 | |
| *** ondrejburian has quit IRC | 21:13 | |
| *** johnmp has quit IRC | 21:13 | |
| *** harlowja has quit IRC | 21:13 | |
| *** arif-ali has quit IRC | 21:13 | |
| *** mtreinish has quit IRC | 21:13 | |
| *** lyarwood has quit IRC | 21:13 | |
| *** TViernion has quit IRC | 21:13 | |
| *** icey has quit IRC | 21:13 | |
| *** bauzas has quit IRC | 21:13 | |
| *** jrist has quit IRC | 21:13 | |
| *** raukadah has quit IRC | 21:13 | |
| *** thedac has quit IRC | 21:13 | |
| *** npochet has quit IRC | 21:13 | |
| *** TheJulia has quit IRC | 21:13 | |
| *** andrewbonney has quit IRC | 21:13 | |
| *** rledisez has quit IRC | 21:13 | |
| *** eharney has quit IRC | 21:13 | |
| *** ijw has quit IRC | 21:13 | |
| *** gyee has quit IRC | 21:13 | |
| *** trident has quit IRC | 21:13 | |
| *** amotoki has quit IRC | 21:13 | |
| *** aarents has quit IRC | 21:13 | |
| *** xek_ has quit IRC | 21:13 | |
| *** auristor has quit IRC | 21:13 | |
| *** takamatsu has quit IRC | 21:13 | |
| *** verdurin has quit IRC | 21:13 | |
| *** andreykurilin has quit IRC | 21:13 | |
| *** dklyle has quit IRC | 21:13 | |
| *** openstackgerrit has quit IRC | 21:13 | |
| *** zaro has quit IRC | 21:13 | |
| *** lennyb has quit IRC | 21:13 | |
| *** stephenfin has quit IRC | 21:13 | |
| *** mpjetta has quit IRC | 21:13 | |
| *** beekneemech has quit IRC | 21:13 | |
| *** strigazi has quit IRC | 21:13 | |
| *** pcrews has quit IRC | 21:13 | |
| *** gary_perkins has quit IRC | 21:13 | |
| *** rakhmerov has quit IRC | 21:13 | |
| *** lsell has quit IRC | 21:13 | |
| *** osmanlicilegi has quit IRC | 21:13 | |
| *** dannins has quit IRC | 21:13 | |
| *** vdrok has quit IRC | 21:13 | |
| *** petevg has quit IRC | 21:13 | |
| *** jamespage has quit IRC | 21:13 | |
| *** mnaser has quit IRC | 21:13 | |
| *** crodriguez has quit IRC | 21:13 | |
| *** nickv1985 has quit IRC | 21:13 | |
| *** rpittau|afk has quit IRC | 21:13 | |
| *** coreycb has quit IRC | 21:13 | |
| *** lathiat has quit IRC | 21:13 | |
| *** kota_ has quit IRC | 21:13 | |
| *** cmurphy has quit IRC | 21:13 | |
| *** ttx has quit IRC | 21:13 | |
| *** Anticimex has quit IRC | 21:13 | |
| *** ociuhandu has joined #openstack-infra | 21:13 | |
| *** diablo_rojo has joined #openstack-infra | 21:13 | |
| *** ahosam has joined #openstack-infra | 21:13 | |
| *** eharney has joined #openstack-infra | 21:13 | |
| *** dtroyer has joined #openstack-infra | 21:13 | |
| *** artom has joined #openstack-infra | 21:13 | |
| *** evrardjp has joined #openstack-infra | 21:13 | |
| *** ijw has joined #openstack-infra | 21:13 | |
| *** armax has joined #openstack-infra | 21:13 | |
| *** owalsh^ has joined #openstack-infra | 21:13 | |
| *** gyee has joined #openstack-infra | 21:13 | |
| *** mattw4 has joined #openstack-infra | 21:13 | |
| *** sreejithp has joined #openstack-infra | 21:13 | |
| *** dave-mccowan has joined #openstack-infra | 21:13 | |
| *** cgoncalves has joined #openstack-infra | 21:13 | |
| *** dalvarez has joined #openstack-infra | 21:13 | |
| *** trident has joined #openstack-infra | 21:13 | |
| *** lbragstad has joined #openstack-infra | 21:13 | |
| *** roman_g has joined #openstack-infra | 21:13 | |
| *** rh-jelabarre has joined #openstack-infra | 21:13 | |
| *** zigo has joined #openstack-infra | 21:13 | |
| *** nicolasbock has joined #openstack-infra | 21:13 | |
| *** ddurst has joined #openstack-infra | 21:13 | |
| *** amotoki has joined #openstack-infra | 21:13 | |
| *** irclogbot_0 has joined #openstack-infra | 21:13 | |
| *** aarents has joined #openstack-infra | 21:13 | |
| *** zzehring has joined #openstack-infra | 21:13 | |
| *** rkukura has joined #openstack-infra | 21:13 | |
| *** aedc has joined #openstack-infra | 21:13 | |
| *** xek_ has joined #openstack-infra | 21:13 | |
| *** yolanda has joined #openstack-infra | 21:13 | |
| *** lxkong has joined #openstack-infra | 21:13 | |
| *** tosky has joined #openstack-infra | 21:13 | |
| *** ccamacho has joined #openstack-infra | 21:13 | |
| *** sshnaidm|afk has joined #openstack-infra | 21:13 | |
| *** johnmp has joined #openstack-infra | 21:13 | |
| *** apetrich has joined #openstack-infra | 21:13 | |
| *** slaweq has joined #openstack-infra | 21:13 | |
| *** ondrejburian has joined #openstack-infra | 21:13 | |
| *** weshay has joined #openstack-infra | 21:13 | |
| *** KeithMnemonic1 has joined #openstack-infra | 21:13 | |
| *** exsdev has joined #openstack-infra | 21:13 | |
| *** harlowja has joined #openstack-infra | 21:13 | |
| *** kaisers has joined #openstack-infra | 21:13 | |
| *** ramishra has joined #openstack-infra | 21:13 | |
| *** auristor has joined #openstack-infra | 21:13 | |
| *** takamatsu has joined #openstack-infra | 21:13 | |
| *** jaicaa has joined #openstack-infra | 21:13 | |
| *** StevenK has joined #openstack-infra | 21:13 | |
| *** portdirect has joined #openstack-infra | 21:13 | |
| *** tinwood has joined #openstack-infra | 21:13 | |
| *** iokiwi has joined #openstack-infra | 21:13 | |
| *** stevebaker has joined #openstack-infra | 21:13 | |
| *** rosmaita has joined #openstack-infra | 21:13 | |
| *** dayou has joined #openstack-infra | 21:13 | |
| *** Shrews has joined #openstack-infra | 21:13 | |
| *** gagehugo has joined #openstack-infra | 21:13 | |
| *** mwhahaha has joined #openstack-infra | 21:13 | |
| *** verdurin has joined #openstack-infra | 21:13 | |
| *** andreykurilin has joined #openstack-infra | 21:13 | |
| *** otherwiseguy has joined #openstack-infra | 21:13 | |
| *** dannins has joined #openstack-infra | 21:13 | |
| *** factor has joined #openstack-infra | 21:13 | |
| *** nhicher has joined #openstack-infra | 21:13 | |
| *** dklyle has joined #openstack-infra | 21:13 | |
| *** toabctl has joined #openstack-infra | 21:13 | |
| *** Tengu has joined #openstack-infra | 21:13 | |
| *** hwoarang has joined #openstack-infra | 21:13 | |
| *** admcleod has joined #openstack-infra | 21:13 | |
| *** mgoddard has joined #openstack-infra | 21:13 | |
| *** dmellado has joined #openstack-infra | 21:13 | |
| *** dciabrin__ has joined #openstack-infra | 21:13 | |
| *** openstackgerrit has joined #openstack-infra | 21:13 | |
| *** bdodd has joined #openstack-infra | 21:13 | |
| *** vdrok has joined #openstack-infra | 21:13 | |
| *** mrmartin has joined #openstack-infra | 21:13 | |
| *** thedac has joined #openstack-infra | 21:13 | |
| *** npochet has joined #openstack-infra | 21:13 | |
| *** masayukig has joined #openstack-infra | 21:13 | |
| *** TheJulia has joined #openstack-infra | 21:13 | |
| *** johnthetubaguy has joined #openstack-infra | 21:13 | |
| *** andrewbonney has joined #openstack-infra | 21:13 | |
| *** prometheanfire has joined #openstack-infra | 21:13 | |
| *** calbers has joined #openstack-infra | 21:13 | |
| *** arif-ali has joined #openstack-infra | 21:13 | |
| *** tomaw has joined #openstack-infra | 21:13 | |
| *** vesper11 has joined #openstack-infra | 21:13 | |
| *** mtreinish has joined #openstack-infra | 21:13 | |
| *** tobiash has joined #openstack-infra | 21:13 | |
| *** rledisez has joined #openstack-infra | 21:13 | |
| *** lyarwood has joined #openstack-infra | 21:13 | |
| *** ChanServ has joined #openstack-infra | 21:13 | |
| *** jistr has joined #openstack-infra | 21:13 | |
| *** elod has joined #openstack-infra | 21:13 | |
| *** zaro has joined #openstack-infra | 21:13 | |
| *** lennyb has joined #openstack-infra | 21:13 | |
| *** TViernion has joined #openstack-infra | 21:13 | |
| *** stephenfin has joined #openstack-infra | 21:13 | |
| *** mpjetta has joined #openstack-infra | 21:13 | |
| *** osmanlicilegi has joined #openstack-infra | 21:13 | |
| *** lsell has joined #openstack-infra | 21:13 | |
| *** Anticimex has joined #openstack-infra | 21:13 | |
| *** bauzas has joined #openstack-infra | 21:13 | |
| *** jrist has joined #openstack-infra | 21:13 | |
| *** andreaf has joined #openstack-infra | 21:13 | |
| *** dtantsur|afk has joined #openstack-infra | 21:13 | |
| *** tdasilva has joined #openstack-infra | 21:13 | |
| *** fnordahl has joined #openstack-infra | 21:13 | |
| *** mnencia has joined #openstack-infra | 21:13 | |
| *** gouthamr has joined #openstack-infra | 21:13 | |
| *** frickler has joined #openstack-infra | 21:13 | |
| *** Diabelko has joined #openstack-infra | 21:13 | |
| *** dansmith has joined #openstack-infra | 21:13 | |
| *** calebb has joined #openstack-infra | 21:13 | |
| *** brwyatt has joined #openstack-infra | 21:13 | |
| *** aluria has joined #openstack-infra | 21:13 | |
| *** dabukalam has joined #openstack-infra | 21:13 | |
| *** hemna has joined #openstack-infra | 21:13 | |
| *** tonyb has joined #openstack-infra | 21:13 | |
| *** cjloader has joined #openstack-infra | 21:13 | |
| *** ab-a has joined #openstack-infra | 21:13 | |
| *** rm_work has joined #openstack-infra | 21:13 | |
| *** ttx has joined #openstack-infra | 21:13 | |
| *** kota_ has joined #openstack-infra | 21:13 | |
| *** bstinson has joined #openstack-infra | 21:13 | |
| *** tristanC has joined #openstack-infra | 21:13 | |
| *** smcginnis has joined #openstack-infra | 21:13 | |
| *** csatari has joined #openstack-infra | 21:13 | |
| *** cjohnston has joined #openstack-infra | 21:13 | |
| *** orwell.freenode.net sets mode: +o ChanServ | 21:13 | |
| *** persia has joined #openstack-infra | 21:13 | |
| *** jmccrory has joined #openstack-infra | 21:13 | |
| *** Ng has joined #openstack-infra | 21:13 | |
| *** lseki has joined #openstack-infra | 21:13 | |
| *** odyssey4me has joined #openstack-infra | 21:13 | |
| *** davidlenwell has joined #openstack-infra | 21:13 | |
| *** kukacz has joined #openstack-infra | 21:13 | |
| *** tobias-urdin has joined #openstack-infra | 21:13 | |
| *** freerunner has joined #openstack-infra | 21:13 | |
| *** Qiming has joined #openstack-infra | 21:13 | |
| *** dmsimard|off has joined #openstack-infra | 21:13 | |
| *** lifeless has joined #openstack-infra | 21:13 | |
| *** JayF has joined #openstack-infra | 21:13 | |
| *** tbarron has joined #openstack-infra | 21:13 | |
| *** ianw has joined #openstack-infra | 21:13 | |
| *** tobberydberg has joined #openstack-infra | 21:13 | |
| *** EmilienM has joined #openstack-infra | 21:13 | |
| *** antonym has joined #openstack-infra | 21:13 | |
| *** amorin has joined #openstack-infra | 21:13 | |
| *** ildikov has joined #openstack-infra | 21:13 | |
| *** arxcruz|rover has joined #openstack-infra | 21:13 | |
| *** jpena|off has joined #openstack-infra | 21:13 | |
| *** spotz has joined #openstack-infra | 21:13 | |
| *** ericyoung has joined #openstack-infra | 21:13 | |
| *** lathiat has joined #openstack-infra | 21:13 | |
| *** kmarc has joined #openstack-infra | 21:13 | |
| *** benj_ has joined #openstack-infra | 21:13 | |
| *** edausq has joined #openstack-infra | 21:13 | |
| *** eandersson has joined #openstack-infra | 21:13 | |
| *** asettle has joined #openstack-infra | 21:13 | |
| *** diablo_rojo_phon has joined #openstack-infra | 21:13 | |
| *** clayg has joined #openstack-infra | 21:13 | |
| *** melwitt has joined #openstack-infra | 21:13 | |
| *** johnsom has joined #openstack-infra | 21:13 | |
| *** tonyb[m] has joined #openstack-infra | 21:13 | |
| *** jdelaros1 has joined #openstack-infra | 21:13 | |
| *** kevinz has joined #openstack-infra | 21:13 | |
| *** jungleboyj has joined #openstack-infra | 21:13 | |
| *** jkt has joined #openstack-infra | 21:13 | |
| *** noonedeadpunk has joined #openstack-infra | 21:13 | |
| *** aspiers has joined #openstack-infra | 21:13 | |
| *** corvus has joined #openstack-infra | 21:13 | |
| *** jonher has joined #openstack-infra | 21:13 | |
| *** logan- has joined #openstack-infra | 21:13 | |
| *** rajinir has joined #openstack-infra | 21:13 | |
| *** davecore has joined #openstack-infra | 21:13 | |
| *** MarkMaglana has joined #openstack-infra | 21:13 | |
| *** evgenyl has joined #openstack-infra | 21:13 | |
| *** srwilkers has joined #openstack-infra | 21:13 | |
| *** rpioso has joined #openstack-infra | 21:13 | |
| *** dustinc has joined #openstack-infra | 21:13 | |
| *** mudpuppy has joined #openstack-infra | 21:13 | |
| *** jroll has joined #openstack-infra | 21:13 | |
| *** pots has joined #openstack-infra | 21:13 | |
| *** darvon has joined #openstack-infra | 21:13 | |
| *** DinaBelova has joined #openstack-infra | 21:13 | |
| *** zbr|pto has joined #openstack-infra | 21:13 | |
| *** gmann has joined #openstack-infra | 21:13 | |
| *** seongsoocho has joined #openstack-infra | 21:13 | |
| *** knikolla has joined #openstack-infra | 21:13 | |
| *** dougwig has joined #openstack-infra | 21:13 | |
| *** rpittau|afk has joined #openstack-infra | 21:13 | |
| *** coreycb has joined #openstack-infra | 21:13 | |
| *** nickv1985 has joined #openstack-infra | 21:13 | |
| *** crodriguez has joined #openstack-infra | 21:13 | |
| *** beisner has joined #openstack-infra | 21:13 | |
| *** donnyd has joined #openstack-infra | 21:13 | |
| *** philroche has joined #openstack-infra | 21:13 | |
| *** jrosser has joined #openstack-infra | 21:13 | |
| *** mnaser has joined #openstack-infra | 21:13 | |
| *** cmurphy has joined #openstack-infra | 21:13 | |
| *** cloudnull has joined #openstack-infra | 21:13 | |
| *** d34dh0r53 has joined #openstack-infra | 21:13 | |
| *** rakhmerov has joined #openstack-infra | 21:13 | |
| *** dirk has joined #openstack-infra | 21:13 | |
| *** gary_perkins has joined #openstack-infra | 21:13 | |
| *** fungi has joined #openstack-infra | 21:13 | |
| *** Ramereth has joined #openstack-infra | 21:13 | |
| *** pcrews has joined #openstack-infra | 21:13 | |
| *** yoctozepto has joined #openstack-infra | 21:13 | |
| *** amoralej|off has joined #openstack-infra | 21:13 | |
| *** jamespage has joined #openstack-infra | 21:13 | |
| *** Adri2000 has joined #openstack-infra | 21:13 | |
| *** cyberpear has joined #openstack-infra | 21:13 | |
| *** hrybacki has joined #openstack-infra | 21:13 | |
| *** guilhermesp has joined #openstack-infra | 21:13 | |
| *** mnasiadka has joined #openstack-infra | 21:13 | |
| *** jhesketh has joined #openstack-infra | 21:13 | |
| *** SpamapS has joined #openstack-infra | 21:13 | |
| *** fdegir has joined #openstack-infra | 21:13 | |
| *** petevg has joined #openstack-infra | 21:13 | |
| *** zzzeek has joined #openstack-infra | 21:13 | |
| *** jberg-dev has joined #openstack-infra | 21:13 | |
| *** arne_wiebalck has joined #openstack-infra | 21:13 | |
| *** Jeffrey4l has joined #openstack-infra | 21:13 | |
| *** mordred has joined #openstack-infra | 21:13 | |
| *** sdoran has joined #openstack-infra | 21:13 | |
| *** dancek has joined #openstack-infra | 21:13 | |
| *** ianychoi has joined #openstack-infra | 21:13 | |
| *** icey has joined #openstack-infra | 21:13 | |
| *** raukadah has joined #openstack-infra | 21:13 | |
| *** redrobot has joined #openstack-infra | 21:13 | |
| *** clarkb has joined #openstack-infra | 21:13 | |
| *** onovy has joined #openstack-infra | 21:13 | |
| *** timburke has joined #openstack-infra | 21:13 | |
| *** strigazi has joined #openstack-infra | 21:13 | |
| *** mugsie has joined #openstack-infra | 21:13 | |
| *** jbryce has joined #openstack-infra | 21:13 | |
| *** adam_g has joined #openstack-infra | 21:13 | |
| *** beekneemech has joined #openstack-infra | 21:13 | |
| *** howell has joined #openstack-infra | 21:13 | |
| *** factor has quit IRC | 21:13 | |
| fungi | i suppose there's a reason it can't override that url with a local file path | 21:13 |
| clarkb | jrosser: just so you know https://pypi.org/project/nova/#files has wheels too for releases | 21:13 |
| *** factor has joined #openstack-infra | 21:13 | |
| clarkb | I feel like this has come up before and ew've basically said that we have a system intended to avoid doing that as we can act like our own ddos | 21:13 |
| *** lastmikoi has quit IRC | 21:13 | |
| jrosser | i know | 21:13 |
| jrosser | i've not managed to get the OSA gates green for weeks | 21:14 |
| fungi | oh, yeah that too. if what you want is to pip install a release version of a nova wheel, we also cache those in the test regions | 21:14 |
| jrosser | which has prevented me getting that fixed | 21:14 |
| fungi | gitea01 seems to let me clone nova. one down, seven to go | 21:15 |
| fungi | cloning nova is not quick, in case anyone wondered (i doubt anyone did) | 21:15 |
| clarkb | is there a timeout on that command? | 21:16 |
| *** factor has quit IRC | 21:16 | |
| *** factor has joined #openstack-infra | 21:16 | |
| fungi | i doubt a timeout would result in "The requested URL returned error: 500" | 21:16 |
| jrosser | https://opendev.org/openstack/ansible-role-python_venv_build/src/branch/master/tasks/python_venv_wheel_build.yml#L135 | 21:16 |
| *** lastmikoi has joined #openstack-infra | 21:16 | |
| fungi | but maybe. timeouts sometimes abort things in ways which raise pathological conditions | 21:17 |
| clarkb | fungi: that only happened one time | 21:17 |
| clarkb | the others errors seem to be relatd to tls stopping uncleanly | 21:17 |
| fungi | ahh | 21:18 |
| fungi | i was more concerned about the 500 anyway | 21:18 |
| fungi | but i hadn't looked at the others yet and assumed they were all the same | 21:18 |
| jrosser | theres a few jobs done this and not all with a 500 https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_8ea/711706/1/check/openstack-ansible-deploy-aio_lxc-debian-buster/8eaef4f/logs/ara-report/result/4c62dd39-ab96-49c9-a74d-8d4b1a3a3105/ | 21:19 |
| fungi | 02 cloned clean | 21:19 |
| clarkb | it is possible that whatever backend this node was talking to simply OOM'd under the pressure of "lets clone nova a bunch" | 21:20 |
| fungi | yeah, oom on the backend would explain it | 21:21 |
| clarkb | gitea03 [Fri Mar 6 20:08:53 2020] Out of memory: Kill process 19722 (git) score 57 or sacrifice child | 21:22 |
| clarkb | the others look fine | 21:22 |
| fungi | that was around 2020-03-06 19:43:56 | 21:22 |
| fungi | that the error was logged in the job | 21:23 |
| clarkb | ya so the OOM is a bit late, but possible memory pressure caused problems early | 21:23 |
| clarkb | then finally resulted in kernel doing something about it | 21:23 |
| fungi | possible | 21:23 |
| fungi | 03 clones nova clean | 21:24 |
| clarkb | I guess we can check syslog on opendev.org to see where those requests were balanced to | 21:24 |
| * clarkb tries to figure that out | 21:24 | |
| clarkb | confirmed Mar 6 19:43:56 gitea-lb01 haproxy[6]: ::ffff:198.72.124.197:56228 [06/Mar/2020:19:40:52.805] balance_git_https balance_git_https/gitea03.opendev.org | 21:25 |
| fungi | oof, yeah, gitea03 went into a world of sad around then http://cacti.openstack.org/cacti/graph.php?action=view&local_graph_id=66679&rra_id=all | 21:26 |
| fungi | looks like it got slammed hard | 21:26 |
| clarkb | fungi: possibly because osa jobs all happened to hash there and also want nova | 21:26 |
| fungi | some indications of a mess of swap thrash happening too, when snmp was able to get a datagram in edgewise | 21:27 |
| fungi | but i was able to clone nova from gitea03 so it seems to have recovered from that | 21:27 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Match tag items against containing branches https://review.opendev.org/578557 | 21:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add a test for retroactive branch tag application https://review.opendev.org/711741 | 21:29 |
| clarkb | this is basically the same underlying issue that I debugged a while back. We either need smaller repos, less requests, better distribution (could add more server), or bigger servers | 21:30 |
| clarkb | zuul jobs should definitely rely on our extensive caching | 21:31 |
| clarkb | (that helps with "less requests") | 21:31 |
| clarkb | if we were to build bigger nodes we really just need more memory | 21:34 |
| clarkb | I suspect that adding more nodes won't completely make this go away as we'll still get hot nodes | 21:34 |
| fungi | until we can properly cluster the backends and start distributing by request hash instead of source and destination hash | 21:35 |
| clarkb | I wonder if haproxy could rehash based on load. Basically if we figure you'll fail if you hit $backend anyway, trying another (and hoping the repo state is consistent enough) will work? | 21:39 |
| clarkb | really the problem we have with load balacning now is clients failing as requests switch nodes | 21:40 |
| openstackgerrit | Merged opendev/system-config master: Moving FortNebula to OpenEdge https://review.opendev.org/711768 | 21:42 |
| *** nicolasbock has quit IRC | 21:42 | |
| *** hashar has joined #openstack-infra | 21:46 | |
| fungi | clarkb: that's essentially what a health reporter agent would get you anyway. agent on gitea03 sees memory pressure increase, tells haproxy which then puts it in graceful down to stop sending new requests there until it recovers | 21:47 |
| fungi | though it does also mean a serious flood of requests could strafe the entire cluster and take every backend offline | 21:47 |
| *** dannins has quit IRC | 21:48 | |
| *** mattw4 has quit IRC | 21:52 | |
| clarkb | https://github.com/reincubate/haproxy-dynamic-weight | 21:52 |
| clarkb | in that solution you have each backend tell a central service what weight they want then that updates haproxy | 21:53 |
| clarkb | we'll have to change the hash-type we use to support dynamic weights but that could be an option. And ya it would be at risk of that behavior | 21:53 |
| *** mattw4 has joined #openstack-infra | 21:56 | |
| fungi | i'm almost done testing that i can clone nova from all the backends... 07 underway now | 21:56 |
| clarkb | what that will do though is potentially break clients that switch from one host to another due to a change in weights | 21:57 |
| clarkb | if we don't tune that well enough that that only happens when the original server would have failed then we're just making things worse | 21:57 |
| clarkb | (and that seems tricky to get right) | 21:57 |
| clarkb | I think the health status checker version of that has a similar problem | 21:58 |
| fungi | right, we basically wouldn't want to use it to even out the load distribution, just to ease off backends which are on the verge of falling over | 21:59 |
| fungi | which is why something more like a detailed health check would make more sense | 22:00 |
| fungi | (haproxy pings the health check script which returns a bad result if memory is almost full) | 22:00 |
| clarkb | fungi: ya the problem there becomes memory needs per request vary by size of repo | 22:01 |
| clarkb | and we won't know what the next request to the server will be so can under or over estimate pretty easily | 22:01 |
| jrosser | tbh i think i need to put everything else down for a bit and fix up OSA to use the locally available repos | 22:01 |
| clarkb | if all our repos were nova sized or all of them git restack sized it would be easier | 22:01 |
| jrosser | part of the issue is that we are multi-repo, at about ~50 currently | 22:02 |
| fungi | right, it would be something like an 80% utilization threshold. whatever puts it in range of toppling if it gets another `git clone nova` request | 22:02 |
| *** mattw4 has quit IRC | 22:02 | |
| *** mattw4 has joined #openstack-infra | 22:02 | |
| jrosser | and we use proposal bot to maintain some things across those repos, which is what has most probably caused the large number of requests today | 22:02 |
| jrosser | i wish i understood why the local repos havent been used before, it's been like this for a *long* time | 22:05 |
| *** imacdonn has quit IRC | 22:05 | |
| *** imacdonn has joined #openstack-infra | 22:05 | |
| fungi | also whatever health checker we use should incorporate a mechanism to diffuse/attenuate possible hysteresis around the set point | 22:08 |
| fungi | something as simple as stop sending requests if we reach 80% memory used, but don't start sending requests again until use drops below 60% | 22:09 |
| fungi | if we end up wanting to go that route | 22:10 |
| clarkb | fungi: thats where the weight based system can be good I think. Its not a complete on oroff, but will balance based on what other servers weight is | 22:10 |
| fungi | otherwise we could end up in some nasty flapping cycles | 22:10 |
| clarkb | this way you don't turn all the servers off too quickly :) | 22:10 |
| fungi | oh, i see, drop the weight only if things get dire rather than evenly weight by utilization at all times | 22:11 |
| clarkb | ya | 22:11 |
| clarkb | then you'll still get connections if necessary | 22:11 |
| clarkb | but otherwise other servers will pick up the load | 22:11 |
| *** factor has quit IRC | 22:15 | |
| *** diablo_rojo has quit IRC | 22:17 | |
| *** mattw4 has quit IRC | 22:17 | |
| *** sreejithp has quit IRC | 22:26 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Implement zookeeper-auth https://review.opendev.org/619155 | 22:27 |
| *** bdodd has quit IRC | 22:29 | |
| *** hashar has quit IRC | 22:31 | |
| *** diablo_rojo has joined #openstack-infra | 22:32 | |
| *** rlandy has quit IRC | 22:33 | |
| donnyd | fungi: clarkb AJaeger any ideas when the next run will be | 22:35 |
| * donnyd patiently waits for nodepool to upload images | 22:35 | |
| *** xek_ has quit IRC | 22:37 | |
| clarkb | donnyd: uploads started just under 2 minutes ago | 22:39 |
| *** Tengu has quit IRC | 22:39 | |
| donnyd | oh yep - i see them now | 22:39 |
| *** Tengu has joined #openstack-infra | 22:39 | |
| *** ociuhandu has quit IRC | 22:40 | |
| fungi | we still need to build a mirror in there too | 22:40 |
| fungi | i'll see if i can work on launching one | 22:41 |
| *** ociuhandu has joined #openstack-infra | 22:41 | |
| donnyd | yey - load balancing is working correctly out of the gate this time - all three glance servers actually have load on them | 22:41 |
| fungi | also i did manage to confirm that i can currently clone the nova repo cleanly from all 8 of our gitea backends, so i don't think any are presently broken | 22:41 |
| *** Tengu has quit IRC | 22:43 | |
| donnyd | its only going at about 300M/s so its probably going to be a while before they are loaded up - do you use a different image for the mirror fungi | 22:44 |
| *** Tengu has joined #openstack-infra | 22:44 | |
| donnyd | is there an image I can upload for you ? | 22:44 |
| clarkb | donnyd: we typically ues an upstream ubuntu bionic image | 22:45 |
| donnyd | rgr - I will get it loaded directly | 22:45 |
| *** ociuhandu has quit IRC | 22:45 | |
| openstackgerrit | Merged openstack/infra-manual master: Redraw code_review image https://review.opendev.org/711675 | 22:46 |
| fungi | i guess https://cloud-images.ubuntu.com/bionic/current/bionic-server-cloudimg-amd64.img is the current raw cloud image? | 22:46 |
| donnyd | I just uploaded that image for you | 22:46 |
| clarkb | that looks riht (though I think thats a qcow2?) | 22:46 |
| fungi | i don't see anything in there claiming to be qcow2 | 22:47 |
| fungi | but yeah, maybe it is | 22:47 |
| openstackgerrit | Merged openstack/infra-manual master: Update 'Feature Branches' for OpenDev https://review.opendev.org/711650 | 22:47 |
| fungi | thanks donnyd! | 22:47 |
| fungi | that'll make this a little faster | 22:47 |
| donnyd | its a donny@office:~/Downloads$ qemu-img info bionic-server-cloudimg-amd64.img | 22:47 |
| donnyd | image: bionic-server-cloudimg-amd64.img | 22:47 |
| donnyd | file format: qcow2 | 22:47 |
| donnyd | virtual size: 2.2G (2361393152 bytes) | 22:47 |
| donnyd | disk size: 329M | 22:47 |
| donnyd | cluster_size: 65536 | 22:48 |
| donnyd | Format specific information: | 22:48 |
| donnyd | compat: 0.10 | 22:48 |
| donnyd | refcount bits: 16 | 22:48 |
| fungi | perfect | 22:48 |
| donnyd | the image is called ```ubuntu-bionic``` | 22:48 |
| donnyd | My dinner just walked in the house - I will be back in a bit | 22:49 |
| donnyd | thanks everyone for helping me (by help I mean doing all of the work) to get this back online | 22:49 |
| donnyd | :) | 22:49 |
| *** jrosser has quit IRC | 22:51 | |
| fungi | thanks for all the help! | 22:51 |
| openstackgerrit | Merged openstack/infra-manual master: Remove "Eligibility to Vote in Elections" section https://review.opendev.org/711532 | 22:52 |
| openstackgerrit | Merged openstack/infra-manual master: Update "Code Review" for OpenDev https://review.opendev.org/711514 | 22:52 |
| fungi | also you did all the hard work getting it back online, we're just syncing up our side | 22:52 |
| fungi | network list shows three in that tenant: openstackci public-v4 public-v6 | 22:52 |
| *** jrosser has joined #openstack-infra | 22:53 | |
| fungi | looks like previously the fortnebula mirror had interfaces on networks named public_v4 and public_v6 | 22:53 |
| fungi | though the last time i launched one for it i only specified --network=public_v4 | 22:54 |
| clarkb | fungi: those exist in the new cloud too | 22:54 |
| fungi | so i'll try with --network=public-v4 here initially | 22:54 |
| fungi | clarkb: note the underscore vs hyphen | 22:55 |
| clarkb | ah yup | 22:55 |
| fungi | sudo OS_CLIENT_CONFIG_FILE=/etc/openstack/all-clouds.yaml /opt/system-config/launch/launch-node.py mirror01.us-east.openedge.opendev.org --flavor=8cpu-8GBram-250GBdisk --cloud=openstackci-openedge --region=us-east --image=ubuntu-bionic --config-drive --network=public-v4 | 22:55 |
| fungi | that's how i'm booting this one | 22:56 |
| fungi | we'll see how it fares | 22:56 |
| donnyd | We only have 1 interface now that has both v4 and v6 | 22:56 |
| donnyd | I think before we ended up with two | 22:56 |
| clarkb | donnyd: ya that sounds right | 22:56 |
| fungi | ahh, okay, so doesn't matter which network i specify in the server boot call? | 22:57 |
| fungi | or should i use the network called openstackci? | 22:57 |
| clarkb | fungi: one clue for that is to subnet show the subnets on those networks | 22:57 |
| clarkb | (I haven't done that here but have had to in othe rclouds in the psat) | 22:58 |
| fungi | ahh | 22:58 |
| clarkb | you can usually infer what routable ranges and ip versions are available that way | 22:58 |
| fungi | yeah, looks like the two subnets are bound to the openstackci network | 22:59 |
| *** matt_kosut has joined #openstack-infra | 23:00 | |
| clarkb | fungi: note the other subnets as shown at https://www.irccloud.com/pastebin/joQv1alE/ | 23:01 |
| fungi | deleted the old server (which wound up in an error state) and doing again with --network=openstackci this time | 23:01 |
| clarkb | I think permission stuff gets weird if doing a listing as a normal user but you can show the subnets by id directly because you actually have access to them | 23:01 |
| fungi | ahh | 23:02 |
| fungi | yeah i used subnet list and then subnet show and then network list to match the network uuid | 23:03 |
| *** yamamoto has joined #openstack-infra | 23:03 | |
| clarkb | In this case I expect openstackci maybe correct. You'll get an rfc1918 ip addr on ipv4 then you fip from public v4 on that? and the other subnet I'm assuming is a working ipv6? | 23:04 |
| *** matt_kosut has quit IRC | 23:04 | |
| fungi | the only v4 subnet on the openstackci network is an rfc 1918 one though, so i guess we need to create and bind a fip? | 23:04 |
| donnyd | Correct | 23:04 |
| fungi | ahh, yeah, what you said | 23:04 |
| fungi | looks like it tries to fip for us... | 23:05 |
| fungi | --- Running initial configuration on host 108.44.198.37 --- | 23:05 |
| fungi | unfortunately... | 23:05 |
| fungi | While testing ssh access: [Errno 110] Connection timed out | 23:05 |
| *** mattw4 has joined #openstack-infra | 23:05 | |
| donnyd | Hrm | 23:05 |
| fungi | though maybe it's just taking a bit | 23:05 |
| fungi | i think we retry | 23:06 |
| fungi | may just be waiting for sshd to start | 23:06 |
| donnyd | checking now | 23:07 |
| clarkb | yes our laucnh stuff should detect when an fip is needed then repeatedly try to ssh | 23:07 |
| donnyd | dinner complete | 23:07 |
| clarkb | which can take some time as its waiting for boot to happen | 23:07 |
| *** yamamoto has quit IRC | 23:08 | |
| donnyd | lol - security group | 23:08 |
| fungi | possible our cloud configurator hasn't run to adjust the security groups yet | 23:08 |
| clarkb | oh we should automatically adjust that | 23:08 |
| clarkb | ya it runs hourly iirc | 23:08 |
| donnyd | don't we have to allow ipv4 taffic inbound | 23:08 |
| clarkb | but also if a cloud ahead fails it doesn't get through to the others | 23:09 |
| clarkb | donnyd: ya we have a tool that should set it up for us | 23:09 |
| donnyd | https://usercontent.irccloud-cdn.com/file/fF56AmPD/image.png | 23:09 |
| donnyd | here is what it is RN | 23:09 |
| fungi | oh, right, so maybe we're currently failing to reach another cloud and it's short-circuiting. i'll check | 23:09 |
| donnyd | try ssh now | 23:09 |
| clarkb | it started at 2300 | 23:10 |
| clarkb | I think it wasn't configured with the new cloud at 2200 | 23:10 |
| donnyd | do you have any any rules or something specific | 23:10 |
| clarkb | and now we just have to wait for it to get there | 23:10 |
| fungi | do i have to manually delete the fip when i delete a server out from behind it? | 23:10 |
| clarkb | donnyd: we do any any rules and then manage things on the host with iptables | 23:10 |
| clarkb | fungi: I don't remember | 23:10 |
| donnyd | i will just update them and then the automation will take over later if needed | 23:11 |
| clarkb | donnyd: I think we can just wait a few minutes | 23:11 |
| fungi | (for having been on the openstack tc for years, i know surprisingly little about how to use openstack!) | 23:11 |
| clarkb | donnyd: ya that should work too | 23:11 |
| donnyd | you should be good to go now | 23:12 |
| donnyd | any any for v4 and v6 | 23:12 |
| fungi | okay, i manually deleted the prior floating ip for 108.44.198.37 | 23:13 |
| fungi | booting again | 23:13 |
| donnyd | it got 36 this time | 23:14 |
| fungi | paramiko.ssh_exception.SSHException: Error reading SSH protocol banner | 23:15 |
| fungi | ugh | 23:15 |
| donnyd | what does that mean | 23:15 |
| clarkb | fungi: I think it will retry? | 23:15 |
| fungi | yeah, seriously | 23:15 |
| fungi | clarkb: you might think that, but after a lengthy traceback it returned control to the shell and deleted the instance | 23:16 |
| clarkb | huh | 23:16 |
| clarkb | TASK [cloud-launcher : Processing security_group_rule for openstackci-openedge us-east] *** | 23:16 |
| clarkb | that ran successfully at least | 23:16 |
| *** igordc has joined #openstack-infra | 23:17 | |
| fungi | looks like paramiko maybe has started raising new exceptions while we're handling the socket timeout exception | 23:17 |
| donnyd | we also have a number of images that have gone active | 23:17 |
| fungi | since this was a nested traceback | 23:17 |
| fungi | trying to boot yet again | 23:17 |
| fungi | maybe it was a fluke | 23:17 |
| donnyd | https://www.irccloud.com/pastebin/kBDyLPun/ | 23:18 |
| fungi | yeah, makes me wonder if something has changed with paramiko's error handling recently | 23:20 |
| donnyd | oh it got kilt again | 23:20 |
| *** diablo_rojo has quit IRC | 23:20 | |
| clarkb | fungi: maybe tell it to keep the host then try openssh and see what that says? | 23:20 |
| fungi | i'll do that one the next try. i started running again to tee the output to a log i can paste | 23:21 |
| fungi | here's what the tracebacks look like: http://paste.openstack.org/show/790427/ | 23:24 |
| fungi | keeping the next build and will test with openssh | 23:24 |
| clarkb | hrm ya almost looks like while processing the timeout the connection finally goes through and then fails | 23:25 |
| donnyd | I moved to a DVR setup for v4 if that matters | 23:25 |
| donnyd | shouldn't - but I have noticed it takes like maybe 1 extra second for the connection to be available | 23:26 |
| clarkb | donnyd: I don't think so, this is probably a bug in the launch script that this tickles | 23:26 |
| fungi | okay, this time it got 108.44.198.34 for the fip | 23:26 |
| clarkb | fungi: you might try change the timeout value in the launch script? | 23:26 |
| fungi | the instance and fip are still up | 23:26 |
| fungi | but i can't ssh, looks like it's timing out | 23:26 |
| fungi | i can telnet to 22/tcp | 23:27 |
| donnyd | ``` ssh 108.44.198.34 | 23:27 |
| donnyd | Warning: Permanently added '108.44.198.34' (ECDSA) to the list of known hosts. | 23:27 |
| donnyd | donny@108.44.198.34: Permission denied (publickey). | 23:27 |
| donnyd | ``` | 23:27 |
| fungi | but i get no banner | 23:27 |
| donnyd | I can get all the way through to auth denied - which is not a connection error | 23:28 |
| fungi | yeah, for me i seem to be able to establish a tcp socket | 23:28 |
| fungi | just don't get any banner back | 23:28 |
| clarkb | fungi: like the tcp handshake succeeds but then doesn't get handed over to the serice? | 23:29 |
| fungi | that's what it seems like to me | 23:29 |
| fungi | tried it from bridge.o.o and from my workstation | 23:29 |
| donnyd | interesting - let me try something | 23:29 |
| donnyd | you can kilt this machine | 23:30 |
| fungi | will do | 23:30 |
| fungi | deleted server instance and fip now | 23:30 |
| *** rh-jelabarre has quit IRC | 23:31 | |
| donnyd | ok try again | 23:32 |
| fungi | on it | 23:32 |
| donnyd | I moved the v4 router from Distributed to Central | 23:32 |
| donnyd | we will see what happens | 23:32 |
| clarkb | oh I suppose that could happen if the floating ip nat was working but then not forwarding properly | 23:32 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Implement zookeeper-auth https://review.opendev.org/619155 | 23:32 |
| clarkb | though its not proxying it should just rewrite | 23:32 |
| donnyd | right - its a one to one | 23:32 |
| fungi | seems like the nat would have to be forwarding to something to wind up establishing a socket, yeah | 23:32 |
| *** rcernin has joined #openstack-infra | 23:33 | |
| donnyd | only have the fedora image left to be uploaded | 23:33 |
| fungi | same behavior. got 108.44.198.37 this time. it gives me an established socket on 22/tcp but no ssh banner | 23:34 |
| fungi | pmtud blackhole? | 23:34 |
| fungi | maybe the tcp handshake makes it through but the packet with the banner is too fat to make it through? | 23:35 |
| donnyd | hrm - well I have the edge setup to fragment those things anyways | 23:35 |
| *** ahosam has quit IRC | 23:35 | |
| donnyd | but everything is 1500 mtu - no funny business this time | 23:36 |
| fungi | ahh, yeah and ipv4 so it's not like this is a v6 tunnel | 23:36 |
| clarkb | note that systemd will open sockets for things then you have to wait for it to finish booting | 23:36 |
| clarkb | I don't know what sort of time frame we are talking but maybe we do just need to wait longer? | 23:36 |
| fungi | right, but donnyd is getting all the way to key exchange | 23:37 |
| donnyd | which makes me think something is up with my edge FW | 23:37 |
| fungi | and even after that i continued to get a dead socket | 23:37 |
| donnyd | because I am behind it | 23:37 |
| clarkb | ah | 23:37 |
| fungi | so i don't think it's systemd's proxy waiting for sshd to start in this case | 23:37 |
| *** mattw4 has quit IRC | 23:38 | |
| donnyd | try ssh again | 23:40 |
| fungi | still the same | 23:40 |
| donnyd | interesting | 23:40 |
| fungi | no change (ssh spins forever, telnet gives me an established socket and no banner from the sshd) | 23:41 |
| *** mattw4 has joined #openstack-infra | 23:41 | |
| donnyd | ok - I have one more idea about what is broke - but i need you to kilt that instance so I can redo something | 23:41 |
| fungi | note that if i try the same with one of the previous ip addresses which is no longer associated with a fip, i just get a socket timeout as expected | 23:41 |
| fungi | baleeting now | 23:41 |
| fungi | server instance and fip deleted | 23:42 |
| donnyd | its possible I may have to re-do some of my edge networking | 23:42 |
| fungi | it happens | 23:44 |
| *** lbragstad_ has joined #openstack-infra | 23:46 | |
| donnyd | ok - one mo gin | 23:46 |
| fungi | should i pour a double? | 23:47 |
| donnyd | LOL | 23:47 |
| donnyd | may as well - it is almost 7 on a friday and you are working on building a mirror | 23:48 |
| * fungi had trouble parsing, assumed it was your drink order | 23:48 | |
| fungi | i'm drinking virginian beer, should be good enough | 23:48 |
| fungi | did you want me to try to boot another instance? or were you saying it would be one more minute? | 23:49 |
| *** lbragstad has quit IRC | 23:49 | |
| donnyd | oh yes - boot another instacne | 23:49 |
| fungi | on it | 23:49 |
| donnyd | https://www.urbandictionary.com/define.php?term=mo%27gin | 23:50 |
| fungi | woohoo! | 23:50 |
| donnyd | ok - so it was dvr | 23:51 |
| donnyd | LOL | 23:51 |
| donnyd | darn - i was really looking forward to saving some IP's | 23:51 |
| *** tosky has quit IRC | 23:51 | |
| fungi | any idea what about dvr was broken in that case? | 23:52 |
| donnyd | yes - in my DVR setup which worked in all testing I did - i used a different set of IP's for the outbound route | 23:52 |
| donnyd | but that is only north - south traffic | 23:52 |
| fungi | ahh | 23:52 |
| donnyd | in DVR it's supposed to NAT outbound on the FIP - which in testing seemed to work fine | 23:53 |
| donnyd | but it did not - or I am misunderstanding how DVR actually works | 23:54 |
| donnyd | anyways i am happy that the mirror is building | 23:55 |
| fungi | there might be people around here who know how dvr is supposed to work, but i'm definitely not one of them | 23:55 |
| fungi | once this finishes configuring i'll add its dns entries and push up a change to put it into the server inventory | 23:55 |
| donnyd | images are loaded | 23:55 |
| fungi | nice | 23:55 |
| donnyd | that seemed faster than last time | 23:55 |
| fungi | yeah, a bit | 23:55 |
| ianw | fungi: don't forget LE setup too | 23:55 |
| fungi | ianw: yep, i'll try to readd the things i deleted for the old fn mirror basically | 23:56 |
| donnyd | it loaded about 180G worth of images in about 1.5 hours | 23:57 |
| fungi | that's blazin' fast | 23:57 |
| donnyd | 14 images which are mostly close to 10G each - so that is not bad at all | 23:58 |
| donnyd | not bad for this shabby little cloud | 23:58 |
| clarkb | zoom zoom | 23:58 |
| donnyd | I used to use filestore for glance - but now they are all in swift | 23:59 |
| donnyd | oh yes - I can be a log provider again too | 23:59 |
| donnyd | plenty of space to house logs | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
