| openstackgerrit | Merged opendev/zone-opendev.org master: Add _acme-challenge.docs.opendev.org https://review.opendev.org/709377 | 00:01 |
|---|---|---|
| *** ociuhandu has quit IRC | 00:02 | |
| *** jamesmcarthur has joined #openstack-infra | 00:02 | |
| *** jamesmcarthur has quit IRC | 00:21 | |
| *** dklyle has joined #openstack-infra | 00:24 | |
| *** jamesmcarthur has joined #openstack-infra | 00:26 | |
| openstackgerrit | Merged opendev/system-config master: static: move afs sites from files.openstack.org to static.opendev.org https://review.opendev.org/709024 | 00:39 |
| *** jamesmcarthur has quit IRC | 01:05 | |
| *** tosky has quit IRC | 01:06 | |
| *** jamesmcarthur has joined #openstack-infra | 01:06 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/hacking master: [ussuri][goal] Drop python 2.7 support and testing https://review.opendev.org/705514 | 01:08 |
| *** jamesmcarthur has quit IRC | 01:09 | |
| *** jamesmcarthur_ has joined #openstack-infra | 01:09 | |
| *** jamesmcarthur_ has quit IRC | 01:26 | |
| *** jamesmcarthur has joined #openstack-infra | 01:27 | |
| *** jamesmcarthur has quit IRC | 01:29 | |
| *** jamesmcarthur_ has joined #openstack-infra | 01:29 | |
| *** ricolin has quit IRC | 01:32 | |
| *** jamesmcarthur_ has quit IRC | 01:39 | |
| *** jamesmcarthur has joined #openstack-infra | 01:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: Framework for periodic 404 scraping job https://review.opendev.org/709382 | 01:46 |
| *** jamesmcarthur has quit IRC | 01:54 | |
| *** jamesmcarthur has joined #openstack-infra | 01:54 | |
| ianw | ERROR: No matching distribution found for gitdb2===4.0.1 (from -c /opt/stack/requirements/upper-constraints.txt (line 229)) | 01:55 |
| ianw | do we know about this? | 01:55 |
| ianw | sigh, it looks like a deleted release | 01:56 |
| ianw | https://github.com/gitpython-developers/gitdb/issues/59 ... and this makes my brain hurt | 01:57 |
| *** verdurin has quit IRC | 02:04 | |
| ianw | https://review.opendev.org/#/c/709389/ ... fixes it ... maybe | 02:07 |
| *** jamesmcarthur has quit IRC | 02:09 | |
| *** jamesmcarthur has joined #openstack-infra | 02:11 | |
| *** verdurin has joined #openstack-infra | 02:19 | |
| *** jamesmcarthur has quit IRC | 02:29 | |
| *** jamesmcarthur has joined #openstack-infra | 02:29 | |
| ianw | ... there's two guys outside chest deep in the pit with my cable connection with a jackhammer ... wish my connection luck | 02:36 |
| adriant | ianw: gitdb vs gitdb2 hurts my brain too | 02:49 |
| ianw | adriant: yeah, afaict gidb 4.0.1 == gitdb2 4.0.2 | 02:50 |
| * adriant sighs | 02:50 | |
| ianw | i'm not sure when people will learn that deleting a release never helps anything | 02:50 |
| iokiwi | you beat me to it ianw https://review.opendev.org/#/c/709392/ | 02:51 |
| fungi | ianw: did you catch the oslo.limit release error thread on openstack-discuss? | 02:52 |
| *** yamamoto has joined #openstack-infra | 02:52 | |
| ianw | btw not throwing stones because who has not ended up with project renaming issues, etc. just that deleting the release can only make whatever is wrong worse | 02:52 |
| ianw | fungi: i'll open that inbox now :) | 02:53 |
| ianw | fungi: ohh, right, sorry that. yeah | 02:54 |
| fungi | on the topic of whether or not to delete a release from pypi, yeah | 02:54 |
| ianw | yeah, it's never a nice linear process that's led up to whatever is going on. sorry i thought you meant maybe they were flagging a failing release job | 02:57 |
| ianw | due to something i'd done to releases/dns/etc lately :) | 02:58 |
| fungi | nah | 02:58 |
| openstackgerrit | Ian Wienand proposed opendev/zone-opendev.org master: docs.opendev.org : switch to static.opendev.org https://review.opendev.org/709402 | 03:12 |
| ianw | AJaeger: if you'd like to confirm with /etc/hosts that "23.253.245.150 developer.openstack.org docs.openstack.org docs.starlingx.io" i think i can switch the DNS entries so they're served by static.opendev.org | 03:15 |
| *** hongbin has joined #openstack-infra | 03:19 | |
| *** jamesmcarthur has quit IRC | 03:21 | |
| *** ricolin has joined #openstack-infra | 03:36 | |
| *** yamamoto has quit IRC | 03:40 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: Revert "Generate list of 404s for docs.o.o" https://review.opendev.org/709038 | 03:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: add static.openstack.org/files.openstack.org https://review.opendev.org/709042 | 03:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: provide git.openstack.org https://review.opendev.org/709403 | 03:49 |
| *** ykarel|away is now known as ykarel | 03:52 | |
| *** yamamoto has joined #openstack-infra | 04:12 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** jamesmcarthur has joined #openstack-infra | 04:30 | |
| *** ricolin has quit IRC | 04:31 | |
| *** ociuhandu has joined #openstack-infra | 04:33 | |
| *** ociuhandu has quit IRC | 04:37 | |
| *** sshnaidm|afk has quit IRC | 04:38 | |
| *** udesale has joined #openstack-infra | 04:45 | |
| *** imacdonn has quit IRC | 04:46 | |
| *** imacdonn has joined #openstack-infra | 04:47 | |
| *** sshnaidm|afk has joined #openstack-infra | 04:51 | |
| *** bdodd has joined #openstack-infra | 05:04 | |
| *** hongbin has quit IRC | 05:04 | |
| *** ramishra has joined #openstack-infra | 05:05 | |
| *** jamesmcarthur has quit IRC | 05:29 | |
| *** ociuhandu has joined #openstack-infra | 05:30 | |
| *** udesale has quit IRC | 05:31 | |
| *** udesale has joined #openstack-infra | 05:32 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #openstack-infra | 05:35 | |
| *** ociuhandu has quit IRC | 05:35 | |
| AJaeger | ianw, those three sites look fine - thanks | 05:41 |
| *** slaweq has joined #openstack-infra | 05:47 | |
| ianw | AJaeger: ok, i'll switch dns, seems as good a time as any and i can check back in a few hours | 05:50 |
| AJaeger | ianw: left a +1 on the 404 change... | 05:56 |
| ianw | #status log switched developer.openstack.org docs.openstack.org and docs.starlingx.io CNAME to static.opendev.org (see https://review.opendev.org/709024) | 05:57 |
| openstackstatus | ianw: finished logging | 05:57 |
| ianw | AJaeger: thanks, i think clarkb's spec would be the best way, and more reliable too probably | 05:57 |
| AJaeger | ianw: yes. Hope somebody implements it... | 05:58 |
| AJaeger | ianw: could you review https://review.opendev.org/709368 - URL change for publications - , please? | 06:00 |
| openstackgerrit | Merged opendev/publications master: Fix links to doc-contrib-guide https://review.opendev.org/709368 | 06:01 |
| *** yamamoto has joined #openstack-infra | 06:12 | |
| *** happyhemant has joined #openstack-infra | 06:13 | |
| *** yamamoto has quit IRC | 06:16 | |
| *** udesale has quit IRC | 06:23 | |
| *** udesale has joined #openstack-infra | 06:25 | |
| frickler | ianw: I just checked the logs again, yes ze04 matched that error, starting about 22:00 on Saturday | 06:27 |
| *** rui-zang has joined #openstack-infra | 06:32 | |
| rui-zang | https://opendev.org/openstack/requirements/commit/ffcbb0278fd2aa0b5559a86a3bba6aa3a6fbc94d | 06:34 |
| rui-zang | https://www.irccloud.com/pastebin/sQW1yzi9/ | 06:35 |
| rui-zang | gitdb2==4.0.1 | 06:35 |
| rui-zang | however, I can not find gitdb2 version 4.0.1 -- https://pypi.org/project/gitdb2/#history | 06:36 |
| rui-zang | This fails neutron installation, thus devstack | 06:37 |
| *** lmiccini has joined #openstack-infra | 06:42 | |
| *** ociuhandu has joined #openstack-infra | 06:46 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: install-javascript-packages: add tox_constraints_file https://review.opendev.org/709414 | 06:48 |
| *** psachin has joined #openstack-infra | 06:50 | |
| *** ociuhandu has quit IRC | 06:57 | |
| *** ociuhandu has joined #openstack-infra | 06:58 | |
| ianw | rui-zang: yes, this is rather broken and unfortunately the problem has become worse with the horizon cross job failing, which blocks the gate | 06:58 |
| ianw | there's some changes out but i'm not sure what way to go at this point | 06:59 |
| *** AJaeger has quit IRC | 06:59 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: install-javascript-packages: add tox_constraints_file https://review.opendev.org/709414 | 07:00 |
| *** AJaeger has joined #openstack-infra | 07:03 | |
| *** ociuhandu has quit IRC | 07:07 | |
| *** lpetrut has joined #openstack-infra | 07:09 | |
| *** lpetrut has quit IRC | 07:10 | |
| *** lpetrut has joined #openstack-infra | 07:11 | |
| *** pgaxatte has joined #openstack-infra | 07:14 | |
| rui-zang | @ianw: Do we have a bug or email communication to openstack-discussion? Otherwise I am afraid lots of people will come up tommorrow | 07:15 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: install-javascript-packages: add tox_constraints_file https://review.opendev.org/709414 | 07:25 |
| *** dpawlik has joined #openstack-infra | 07:29 | |
| *** ccamacho has joined #openstack-infra | 07:33 | |
| *** ijw_ has quit IRC | 07:37 | |
| *** ijw has joined #openstack-infra | 07:38 | |
| *** matt_kosut has joined #openstack-infra | 07:40 | |
| *** sshnaidm|afk has quit IRC | 07:41 | |
| *** ykarel is now known as ykarel|lunch | 07:41 | |
| *** ijw has quit IRC | 07:44 | |
| *** sshnaidm|afk has joined #openstack-infra | 07:55 | |
| *** ociuhandu has joined #openstack-infra | 08:04 | |
| *** sshnaidm|afk is now known as sshnaidm | 08:04 | |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: provide git.openstack.org https://review.opendev.org/709403 | 08:09 |
| *** ociuhandu has quit IRC | 08:09 | |
| *** tesseract has joined #openstack-infra | 08:12 | |
| openstackgerrit | Mohammed Naser proposed zuul/zuul-helm master: Drop args from container spec https://review.opendev.org/709424 | 08:17 |
| *** tosky has joined #openstack-infra | 08:18 | |
| *** amoralej|off is now known as amoralej | 08:22 | |
| *** jcapitao has joined #openstack-infra | 08:23 | |
| *** Lucas_Gray has joined #openstack-infra | 08:24 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: install-javascript-packages: add tox_constraints_file https://review.opendev.org/709414 | 08:24 |
| *** dchen has quit IRC | 08:25 | |
| *** ccamacho has quit IRC | 08:27 | |
| *** jcapitao has quit IRC | 08:29 | |
| *** jcapitao has joined #openstack-infra | 08:29 | |
| *** ccamacho has joined #openstack-infra | 08:33 | |
| *** ykarel|lunch is now known as ykarel | 08:46 | |
| *** ralonsoh has joined #openstack-infra | 08:48 | |
| openstackgerrit | Chandan Kumar (raukadah) proposed openstack/diskimage-builder master: Change tgt pkg-map to target-restore CentOS/RHEL-8 https://review.opendev.org/709159 | 08:48 |
| *** udesale_ has joined #openstack-infra | 08:49 | |
| *** udesale has quit IRC | 08:52 | |
| *** ricolin has joined #openstack-infra | 08:56 | |
| *** lucasagomes has joined #openstack-infra | 08:57 | |
| *** slaweq_ has joined #openstack-infra | 09:00 | |
| *** ccamacho has quit IRC | 09:01 | |
| *** slaweq has quit IRC | 09:02 | |
| evrardjp | hey | 09:15 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 09:20 |
| evrardjp | on governance.o.o the date is very ancient, and doesn't correspond to latest content: last updated: 2019-03-01 17:32:45 | 09:24 |
| evrardjp | anyone knows if there is something going on? | 09:24 |
| AJaeger | evrardjp: let me check... | 09:30 |
| AJaeger | that date is the last date the RST page was changed | 09:30 |
| AJaeger | evrardjp: which content on that page is outdated? | 09:31 |
| AJaeger | evrardjp: Oh, I see - you're missing ideas? | 09:33 |
| *** gfidente has joined #openstack-infra | 09:33 | |
| AJaeger | ianw: something is wrong with governance.o.o: http://files.openstack.org/project/governance.openstack.org/ shows "Ideas" at bottom whily https://governance.openstack.org/ does not | 09:34 |
| AJaeger | ianw: is the DNS entry wrong? | 09:34 |
| AJaeger | evrardjp: FYI ^ | 09:35 |
| *** yolanda__ has joined #openstack-infra | 09:37 | |
| *** apetrich has joined #openstack-infra | 09:37 | |
| evrardjp | oh yeah I see | 09:38 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app factory https://review.opendev.org/708812 | 09:39 |
| *** yolanda has quit IRC | 09:40 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Make test setup_config more pure https://review.opendev.org/709084 | 09:42 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler test app factory https://review.opendev.org/708812 | 09:42 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Add spec for scale out scheduler https://review.opendev.org/621479 | 09:50 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Enforce sql connections for scheduler and web https://review.opendev.org/630472 | 09:51 |
| AJaeger | ianw: or is the webserver serving old data? Looking at DNS, it seems fine | 09:52 |
| openstackgerrit | Jens Harbott (frickler) proposed opendev/system-config master: static: provide git.openstack.org https://review.opendev.org/709403 | 09:52 |
| AJaeger | evrardjp: hope ianw can shed light on this but if he's already asleep down under, we need another infra-root to investigate. | 09:53 |
| *** ociuhandu has joined #openstack-infra | 10:06 | |
| evrardjp | AJaeger: I am not in a hurry | 10:17 |
| frickler | apache serves that site from /afs/openstack.org/project/governance.openstack.org/governance , while the updated content seems to be in /afs/openstack.org/project/governance.openstack.org | 10:26 |
| AJaeger | frickler: so, job and serving is using wrong location? | 10:27 |
| frickler | the governance/ dir seems to be a copy created by ianw last year. not sure whether the apache config is wrong or the publishing | 10:27 |
| AJaeger | frickler: thanks. | 10:27 |
| AJaeger | since evrardjp is not in a hurry: Let's wait for ianw to untangle this. | 10:28 |
| frickler | ack | 10:28 |
| *** ociuhandu has quit IRC | 10:34 | |
| evrardjp | it used to be okay, like last week | 10:44 |
| *** slaweq_ is now known as slaweq | 10:51 | |
| frickler | infra-root: wanting to promote the reqs fix https://review.opendev.org/#/c/709389/ , zuul command is now broken by the thing this patch is fixing, too http://paste.openstack.org/show/789920/ | 10:52 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 10:52 |
| frickler | o.k., using the zuul cmd from a freshly created venv worked, but someone might want to look into how the existing installation got broken | 10:59 |
| *** chandankumar has joined #openstack-infra | 11:02 | |
| *** Lucas_Gray has quit IRC | 11:04 | |
| *** ociuhandu has joined #openstack-infra | 11:04 | |
| jrosser | could i get a hold on the openstack-ansible-functional-debian-buster job in https://review.opendev.org/#/c/707943/ | 11:07 |
| *** ociuhandu has quit IRC | 11:12 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Fix governance-website publishing https://review.opendev.org/709449 | 11:12 |
| AJaeger | evrardjp, frickler, ianw, this is my proposal to fix it ^ | 11:12 |
| *** Lucas_Gray has joined #openstack-infra | 11:21 | |
| *** ociuhandu has joined #openstack-infra | 11:22 | |
| *** udesale_ has quit IRC | 11:28 | |
| evrardjp | AJaeger: was it me dreaming that it has ever worked then? | 11:28 |
| AJaeger | evrardjp: ianw synced the data to the place I change it to, so it works. Just job and apache have different views | 11:29 |
| AJaeger | so, depends on what you mean with working. We moved from one server to another last month and that needed changes in both places - and we broke it ;) | 11:29 |
| evrardjp | no I meant how is that possible that I saw it working last week, was it me that dreamt or not? | 11:31 |
| *** ccamacho has joined #openstack-infra | 11:31 | |
| evrardjp | it is totally possible | 11:31 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 11:37 |
| AJaeger | evrardjp: different repo? only governance-website is broken | 11:41 |
| brtknr | Whats up with Zuul today? | 11:42 |
| *** priteau has joined #openstack-infra | 11:43 | |
| *** pgaxatte has quit IRC | 11:45 | |
| evrardjp | AJaeger: yes it's probably an issue on my side | 11:50 |
| brtknr | is it just Magnum or is Zuul broken atm? | 11:50 |
| *** ociuhandu has quit IRC | 11:56 | |
| *** ociuhandu has joined #openstack-infra | 11:56 | |
| AJaeger | brtknr: see the discussion about gitdb2 above. Is that the background of your question? Then 709389 should fix it. IF not, please give details | 11:57 |
| frickler | meh, that patch is failing again, ianw forgot to remove the failing job from gate :( | 12:00 |
| *** jcapitao is now known as jcapitao_lunch | 12:01 | |
| *** nicolasbock has joined #openstack-infra | 12:01 | |
| *** ociuhandu has quit IRC | 12:01 | |
| *** ociuhandu has joined #openstack-infra | 12:08 | |
| *** ociuhandu has quit IRC | 12:08 | |
| *** ociuhandu has joined #openstack-infra | 12:08 | |
| *** ociuhandu has quit IRC | 12:14 | |
| *** ociuhandu has joined #openstack-infra | 12:15 | |
| *** ociuhandu has quit IRC | 12:19 | |
| *** ociuhandu has joined #openstack-infra | 12:19 | |
| noonedeadpunk | also it would be great to get an autohold for https://zuul.opendev.org/t/openstack/build/e35a42f4e739453ea293fc0bc717c231 | 12:24 |
| *** ociuhandu has quit IRC | 12:27 | |
| *** ociuhandu has joined #openstack-infra | 12:27 | |
| *** artom has joined #openstack-infra | 12:28 | |
| frickler | noonedeadpunk: autohold created, feel free to recheck | 12:28 |
| noonedeadpunk | frickler: oh, nice, thanks. Also jrosser asked another hold previously, could you kindly check it? | 12:30 |
| frickler | noonedeadpunk: oh, I missed that in the zuul mess, setting it up now | 12:31 |
| noonedeadpunk | thanks:) | 12:31 |
| *** roman_g has joined #openstack-infra | 12:33 | |
| *** ociuhandu has quit IRC | 12:33 | |
| frickler | jrosser: hold set up, feel free to recheck | 12:33 |
| *** ociuhandu has joined #openstack-infra | 12:34 | |
| jrosser | frickler: thanks! | 12:34 |
| frickler | #status log force-merged requirements fix https://review.opendev.org/709389 to unblock gate | 12:35 |
| openstackstatus | frickler: finished logging | 12:35 |
| *** happyhemant has quit IRC | 12:43 | |
| openstackgerrit | Merged opendev/irc-meetings master: Rename meeting name for Automation SIG https://review.opendev.org/708384 | 12:45 |
| *** Lucas_Gray has quit IRC | 12:56 | |
| *** rlandy has joined #openstack-infra | 12:56 | |
| *** Lucas_Gray has joined #openstack-infra | 12:58 | |
| *** factor has quit IRC | 12:58 | |
| *** factor has joined #openstack-infra | 12:58 | |
| *** ociuhandu_ has joined #openstack-infra | 12:59 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: DNM: testing spaces in topic https://review.opendev.org/709464 | 13:01 |
| *** nicolasbock has quit IRC | 13:01 | |
| *** redrobot has joined #openstack-infra | 13:02 | |
| *** ociuhandu has quit IRC | 13:02 | |
| *** ociuhandu has joined #openstack-infra | 13:02 | |
| *** ociuhandu_ has quit IRC | 13:03 | |
| *** rfolco has joined #openstack-infra | 13:04 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: DNM: testing commas in topic https://review.opendev.org/709464 | 13:06 |
| *** nicolasbock has joined #openstack-infra | 13:06 | |
| *** ociuhandu has quit IRC | 13:07 | |
| *** smarcet has joined #openstack-infra | 13:07 | |
| *** jamesmcarthur has joined #openstack-infra | 13:13 | |
| *** pgaxatte has joined #openstack-infra | 13:14 | |
| jrosser | frickler: i think my job with a hold might be ready? | 13:15 |
| *** jcapitao_lunch is now known as jcapitao | 13:19 | |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Removed stale config variables https://review.opendev.org/709525 | 13:19 |
| *** amoralej is now known as amoralej|lunch | 13:28 | |
| *** dave-mccowan has joined #openstack-infra | 13:29 | |
| frickler | jrosser: yes. which ssh key should I use for you? | 13:31 |
| jrosser | this one please http://jrosser.woaf.net/openstack.pub | 13:31 |
| *** jamesmcarthur has quit IRC | 13:32 | |
| frickler | jrosser: o.k., should be ready to go, 198.72.124.26 | 13:32 |
| jrosser | great, thanks | 13:33 |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: Removed OpenStackID stale config variables https://review.opendev.org/709529 | 13:35 |
| *** dave-mccowan has quit IRC | 13:40 | |
| *** dave-mccowan has joined #openstack-infra | 13:41 | |
| *** jamesmcarthur has joined #openstack-infra | 13:45 | |
| openstackgerrit | Jeremy Stanley proposed opendev/git-review master: Support spaces and other characters in topic https://review.opendev.org/681906 | 13:47 |
| *** adriancz has joined #openstack-infra | 14:06 | |
| *** rfarr has joined #openstack-infra | 14:06 | |
| *** lbragstad has joined #openstack-infra | 14:12 | |
| *** psachin has quit IRC | 14:14 | |
| *** yamamoto has joined #openstack-infra | 14:16 | |
| *** Goneri has joined #openstack-infra | 14:19 | |
| *** yamamoto has quit IRC | 14:20 | |
| *** jamesmcarthur has quit IRC | 14:22 | |
| *** amoralej|lunch is now known as amoralej | 14:24 | |
| *** eharney has joined #openstack-infra | 14:32 | |
| *** smarcet has quit IRC | 14:34 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Use scheduler factory consistently in tests https://review.opendev.org/709542 | 14:40 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 14:42 |
| *** smarcet has joined #openstack-infra | 14:47 | |
| *** matt_kosut has quit IRC | 14:50 | |
| *** smarcet has quit IRC | 14:51 | |
| *** ociuhandu has joined #openstack-infra | 14:54 | |
| *** ociuhandu_ has joined #openstack-infra | 14:54 | |
| *** jackedin has joined #openstack-infra | 14:58 | |
| *** jackedin has quit IRC | 14:58 | |
| *** ociuhandu has quit IRC | 14:58 | |
| *** rfarr_ has joined #openstack-infra | 14:58 | |
| *** jackedin has joined #openstack-infra | 14:59 | |
| *** rfarr has quit IRC | 15:00 | |
| *** rfarr_ has quit IRC | 15:04 | |
| fungi | infra-root: we just got an e-mail from ovh which my terrible grasp of french suggests is about them switching from keystone v2 to v3 api, which i think has been scheduled for months (i vaguely recall them warning everyone) and i don't expect it to pose any problems for us, but keep an eye out | 15:05 |
| *** haleyb has joined #openstack-infra | 15:06 | |
| fungi | ahh, it's a month from today they plan to switch (2020-03-24) | 15:07 |
| fungi | seems i got an english version to my personal account since i'm also a customer | 15:08 |
| Shrews | fungi: we might need to add identity_api_version: '3' to clouds.yml files | 15:09 |
| Shrews | e.g., https://opendev.org/opendev/system-config/src/branch/master/playbooks/templates/clouds/nodepool_clouds.yaml.j2#L95 | 15:10 |
| fungi | do we use v2 by default? | 15:11 |
| fungi | i'm surprised if so | 15:11 |
| Shrews | fungi: i forget offhand how the default works (discovered or otherwise) | 15:11 |
| Shrews | i think if the catalog is updated properly it should *just work* | 15:12 |
| *** ykarel is now known as ykarel|afk | 15:12 | |
| mordred | if SHOULD just work | 15:12 |
| Shrews | BUT if it doesn't, that's a thing we can do | 15:12 |
| *** matt_kosut has joined #openstack-infra | 15:12 | |
| mordred | we shoudln't need to do anythign with identity_api_version ... but yeah, also what Shrews says | 15:12 |
| Shrews | i mean, clouds always work as we expect them too, right? | 15:13 |
| mordred | actually - ovh vendor profile in openstacksdk already sets identity_api_version to 3 - AND we point at the unversioned endpoint | 15:13 |
| mordred | so I am 100% positive this will not impact us | 15:13 |
| *** dave-mccowan has quit IRC | 15:14 | |
| mordred | https://opendev.org/openstack/openstacksdk/src/branch/master/openstack/config/vendors/ovh.json | 15:14 |
| *** ykarel|afk is now known as ykarel|away | 15:15 | |
| mordred | in fact, last vendor profile change was by OVH: https://opendev.org/openstack/openstacksdk/commit/5cac3f7240637eea934a172aee2db1a7e25dd7aa :) | 15:15 |
| Shrews | well now i'm confused why we have that as the default if they've been on v2? | 15:15 |
| rajinir | we are seeing this error in out third party builds.. anyone else facing it? ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1 | 15:15 |
| rledisez | fungi: mordred: a targetted communication will be send soon for people that we know (by logs) are using the keystone v2 API. If you don't get this specific email, you're fine | 15:15 |
| *** ociuhandu has joined #openstack-infra | 15:15 | |
| Shrews | oh, so both have been available i suppose | 15:16 |
| fungi | Shrews: yeah, the announcement is that they're upgrading to a keystone which lacks v2 api support, i believe | 15:16 |
| *** matt_kosut has quit IRC | 15:16 | |
| *** ociuhandu_ has quit IRC | 15:17 | |
| mordred | fungi, rledisez: if we got an email though, that means somewhere we *are* using v2? | 15:17 |
| mordred | oh | 15:17 |
| mordred | will be sent soon | 15:17 |
| mordred | PHEW | 15:17 |
| mordred | I was worried there was a massive bug in sdk that I didn't want to have to track down | 15:18 |
| fungi | rajinir: yes, gitdb2===4.0.1 was deleted from pypi, there's a transition underway to call it gitdb (without the 2), and i think requirements changes are up and perhaps already merged | 15:18 |
| fungi | rledisez: thanks for clarifying! | 15:18 |
| fungi | i was fairly sure we looked into it back when the earlier notice was sent (around october) and concluded then it wouldn't need any further changes on our end | 15:18 |
| rajinir | fungi: thank you I see it now | 15:20 |
| *** lpetrut has quit IRC | 15:20 | |
| prometheanfire | ya, deleting releases, that's the third time in a week | 15:23 |
| *** dave-mccowan has joined #openstack-infra | 15:28 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add jobs to build gerrit 3.1 https://review.opendev.org/709295 | 15:30 |
| mordred | prometheanfire: I really wish we could convince people to stop doing that | 15:30 |
| mordred | prometheanfire: or convince pypi to remove the ability :) | 15:30 |
| *** ociuhandu has quit IRC | 15:34 | |
| *** ociuhandu has joined #openstack-infra | 15:34 | |
| *** dave-mccowan has quit IRC | 15:35 | |
| brtknr | Hmm CI still appears broken even after the 709389 merge: https://review.opendev.org/#/c/705984/ | 15:36 |
| *** Lucas_Gray has quit IRC | 15:40 | |
| *** ociuhandu has quit IRC | 15:40 | |
| AJaeger | brtknr: only magnum-functional-api is failing - are you sure it's gitdb? If it is: Check that the job uses constraints correctly. As the other jobs pass, it's likely to be job specific | 15:45 |
| *** psachin has joined #openstack-infra | 15:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Ensure job.queued is correct after reconfig https://review.opendev.org/709559 | 15:49 |
| *** matt_kosut has joined #openstack-infra | 15:52 | |
| brtknr | mnaser: You might now? I see 404 for an image that its trying to download from Vexxhost mirror: https://zuul.opendev.org/t/openstack/build/b14a29db60324cfaae695ea3b57ecb3b/log/logs/devstack-early.txt#41801 | 15:52 |
| *** ociuhandu has joined #openstack-infra | 15:54 | |
| fungi | brtknr: i think we're not mirroring that | 15:56 |
| fungi | shouldn't be vexxhost-specific | 15:56 |
| *** cmurphy|afk is now known as cmurphy | 15:57 | |
| fungi | see http://mirror.sjc1.vexxhost.openstack.org/fedora/releases/29/README | 15:57 |
| fungi | fedora archived f29 | 15:57 |
| mordred | fungi: oh - well that explains another zuul job issue | 15:58 |
| *** lmiccini has quit IRC | 15:59 | |
| brtknr | :( so what now? | 16:00 |
| clarkb | mordred: Shrews fungi when ovh notified us of that change last year we went ahead and forcefully transitioned early | 16:00 |
| *** pgaxatte has quit IRC | 16:01 | |
| *** diablo_rojo has joined #openstack-infra | 16:02 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Fix unittests for python2 and fedora https://review.opendev.org/709302 | 16:02 |
| mordred | brtknr: switch to fedora-30, like that ^^ | 16:02 |
| clarkb | I'm not sure fedora-30 is fully operational fwiw | 16:03 |
| clarkb | because you can't actually bootstrap it or something so we paused image builds then upstream never fixed dnf or whatever it was | 16:03 |
| mordred | clarkb: so should I just get rid of fedora from zuul jobs tests yet? | 16:04 |
| mordred | clarkb: *awesome* | 16:04 |
| brtknr | mordred: my hero! | 16:04 |
| clarkb | I mean it might half work depending on what you are doing | 16:04 |
| clarkb | definitely let zuul run the jobs and see what happens first | 16:04 |
| *** ociuhandu has quit IRC | 16:05 | |
| mordred | kk | 16:06 |
| *** ociuhandu has joined #openstack-infra | 16:06 | |
| mordred | brtknr: see the followup from clarkb before you get happy :) | 16:06 |
| *** jcapitao has quit IRC | 16:07 | |
| brtknr | yes i saw, didnt wanna be overly negative | 16:07 |
| brtknr | i think fedora 31 is stable now | 16:07 |
| brtknr | might it be better to use that for longer term stablility if it works? | 16:08 |
| clarkb | brtknr: I think it too suffered the bootstrapping issues | 16:08 |
| *** jcapitao has joined #openstack-infra | 16:08 | |
| clarkb | ianw would know what the current status is, this is from memory when it first cropped up. But basically dnf can't isntall a fedora | 16:08 |
| clarkb | because there is some unexpressed dep or something | 16:09 |
| clarkb | oh no it was segfaults | 16:09 |
| clarkb | or similar | 16:09 |
| mordred | clarkb: maybe this is where we need to explore that "just export a docker container contents to make the initial root fs" patch series | 16:09 |
| *** Lucas_Gray has joined #openstack-infra | 16:09 | |
| clarkb | mordred: yes, though upstream was working to address the problem. I think the issue was being such a central utility people are cautious | 16:10 |
| mordred | clarkb: fair | 16:10 |
| *** Lucas_Gray has quit IRC | 16:13 | |
| *** xek_ has joined #openstack-infra | 16:14 | |
| *** Lucas_Gray has joined #openstack-infra | 16:16 | |
| *** xek has quit IRC | 16:17 | |
| *** jcapitao is now known as jcapitao_afk | 16:18 | |
| noonedeadpunk | gerrit irc bot is dead? | 16:19 |
| clarkb | noonedeadpunk: it just posted data here ~17 minutes ago and is in channel. We have too many channels for it to be in at once so it will connect on demand using a rotation | 16:20 |
| noonedeadpunk | oh, no, just some delay | 16:20 |
| clarkb | ok | 16:20 |
| noonedeadpunk | yeah, ok, I see | 16:20 |
| noonedeadpunk | thanks for clarificaton) | 16:20 |
| *** mattw4 has joined #openstack-infra | 16:23 | |
| *** jcapitao_afk has quit IRC | 16:23 | |
| *** gyee has joined #openstack-infra | 16:24 | |
| *** jcapitao_afk has joined #openstack-infra | 16:25 | |
| *** jcapitao_afk is now known as jcapitao | 16:32 | |
| *** ccamacho has quit IRC | 16:36 | |
| *** jamesmcarthur has joined #openstack-infra | 16:38 | |
| *** tesseract has quit IRC | 16:47 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Fix unittests for python2 and fedora https://review.opendev.org/709302 | 16:49 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Fix unittests for python2 and fedora https://review.opendev.org/709302 | 16:51 |
| openstackgerrit | Merged zuul/zuul master: Make most test cases work on MacOS https://review.opendev.org/707585 | 16:52 |
| *** bdodd has quit IRC | 16:56 | |
| *** bdodd has joined #openstack-infra | 16:57 | |
| clarkb | I'm going to find breakfast then go into the spec writing cave | 17:04 |
| *** Lucas_Gray has quit IRC | 17:04 | |
| *** lucasagomes has quit IRC | 17:08 | |
| *** Lucas_Gray has joined #openstack-infra | 17:10 | |
| *** Lucas_Gray has quit IRC | 17:12 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add pause-buildset-registry role https://review.opendev.org/709256 | 17:17 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Fix cleanup of symlink fixtures https://review.opendev.org/709306 | 17:17 |
| *** jamesmcarthur has quit IRC | 17:17 | |
| *** jamesmcarthur has joined #openstack-infra | 17:18 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: k8s: allow port forwarding https://review.opendev.org/709259 | 17:19 |
| *** Lucas_Gray has joined #openstack-infra | 17:20 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:20 | |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** smarcet has joined #openstack-infra | 17:21 | |
| *** sreejithp has joined #openstack-infra | 17:22 | |
| smarcet | fungi: sorry to bother you, but i cant figure what is wrong with https://review.opendev.org/#/c/709525/ | 17:23 |
| *** psachin has quit IRC | 17:23 | |
| fungi | smarcet: i saw that and suspect your depends-on is in the wrong direction but haven't had a free moment to pull up logs yet | 17:24 |
| smarcet | oh ok | 17:24 |
| smarcet | will check thx u :0 | 17:25 |
| smarcet | :) | 17:25 |
| *** jcapitao has quit IRC | 17:25 | |
| fungi | you'll need to stop setting classvars from the instantiations in system-config before you remove support for those classvars from the module | 17:25 |
| fungi | otherwise puppet will choke (which explains why the module change is failing but the system-config change depending on it passes the same job) | 17:26 |
| smarcet | fungi: ack thx u for the info :) | 17:26 |
| fungi | yw | 17:26 |
| fungi | sorry i haven't had a chance to look at the job failure yet, that was just my first take skimming the changes earlier | 17:26 |
| openstackgerrit | sebastian marcet proposed opendev/system-config master: Removed OpenStackID stale config variables https://review.opendev.org/709529 | 17:26 |
| smarcet | fungi: its ok thx u again :) | 17:26 |
| fungi | but if you see errors logged by puppet about nonexistent class variables or something like that then it's almost certainly why | 17:27 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Remove enum34 from test-requirements https://review.opendev.org/709575 | 17:33 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-infra | 17:35 | |
| *** jamesmcarthur_ has quit IRC | 17:35 | |
| *** jamesmcarthur_ has joined #openstack-infra | 17:38 | |
| *** smarcet has quit IRC | 17:41 | |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add spec for website activity stats effort https://review.opendev.org/709236 | 17:44 |
| clarkb | fungi: ianw ^ thank you for the revews. I think I've managed to capture your feedback there | 17:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stream output from kubectl pods https://review.opendev.org/709261 | 17:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: Fix unittests for python2 and fedora https://review.opendev.org/709302 | 17:49 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add pause-buildset-registry role https://review.opendev.org/709256 | 17:49 |
| *** ijw has joined #openstack-infra | 17:51 | |
| *** smarcet has joined #openstack-infra | 17:55 | |
| *** ijw has quit IRC | 17:56 | |
| *** smarcet has quit IRC | 17:59 | |
| *** ijw has joined #openstack-infra | 17:59 | |
| sshnaidm | clarkb, hi, can we mirror fedora 31 repositories on afs mirrors? | 18:00 |
| clarkb | sshnaidm: yes, though we don't have fedora 31 images as far as I know so that would be the first step | 18:02 |
| sshnaidm | clarkb, ack | 18:02 |
| sshnaidm | chandankumar, ^^ | 18:02 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add jobs to build gerrit 3.1 https://review.opendev.org/709295 | 18:06 |
| AJaeger | config-core, a few cleanups for review, please https://review.opendev.org/705373 and https://review.opendev.org/709051 | 18:06 |
| AJaeger | config-core and also, please: https://review.opendev.org/708920 and https://review.opendev.org/708925 | 18:07 |
| weshay|ruck | fyi... need reviews on https://review.opendev.org/#/c/708402/ getting the right version of openvswitch installed for centos-8 please | 18:07 |
| *** priteau has quit IRC | 18:07 | |
| AJaeger | weshay|ruck: is that passing jobs now? | 18:08 |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch https://review.opendev.org/708402 | 18:08 |
| weshay|ruck | AJaeger, gentoo timed out | 18:08 |
| AJaeger | yatin, no need to rebase ;( | 18:08 |
| AJaeger | weshay|ruck: we cannot merge without Zuul passing, so let's first solve that. | 18:09 |
| weshay|ruck | ack | 18:09 |
| ykarel|away | AJaeger, ohhk sorry | 18:10 |
| AJaeger | ykarel|away: with every recheck and approval we automatically rebase. So, if a simple "recheck" would have been enough | 18:10 |
| *** Lucas_Gray has quit IRC | 18:10 | |
| ykarel|away | AJaeger, hmm got that, but doing rebase have some side effect ? | 18:11 |
| AJaeger | it can be of you base changes on top of it and don't rebase them as well... | 18:12 |
| AJaeger | ykarel|away: yes, there's a side affect: It makes diff between versions more complicated, so we prefer to only rebase in case of conflicts | 18:13 |
| AJaeger | thanks, mordred for reviewing | 18:13 |
| *** kozhukalov has joined #openstack-infra | 18:14 | |
| AJaeger | config-core, any takes for a second +2 on https://review.opendev.org/708920, please? | 18:14 |
| *** jamesmcarthur_ has quit IRC | 18:14 | |
| *** jamesmcarthur has joined #openstack-infra | 18:15 | |
| ykarel|away | AJaeger, ack | 18:17 |
| ykarel|away | hmm for dependency chain it makes sense | 18:17 |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove enum34 from test-requirements https://review.opendev.org/709575 | 18:19 |
| *** ociuhandu_ has joined #openstack-infra | 18:20 | |
| *** jamesmcarthur has quit IRC | 18:20 | |
| openstackgerrit | Merged opendev/system-config master: Build gerrit images with bazelisk https://review.opendev.org/708018 | 18:20 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Spec to cleanup python dev tools on our test images https://review.opendev.org/709579 | 18:21 |
| clarkb | infra-root ianw mordred ^ that is the python cleanup spec. I think I've largely captured where we are at. Feedback very welcome | 18:21 |
| clarkb | AJaeger: looking | 18:21 |
| AJaeger | thanks, clarkb | 18:23 |
| *** ociuhandu has quit IRC | 18:24 | |
| *** ociuhandu_ has quit IRC | 18:25 | |
| openstackgerrit | Merged openstack/project-config master: grafana afs: add project.releases volume https://review.opendev.org/709051 | 18:25 |
| openstackgerrit | Merged openstack/project-config master: Remove unused files in playbooks/publish https://review.opendev.org/708925 | 18:25 |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| *** jamesmcarthur has quit IRC | 18:26 | |
| *** jamesmcarthur has joined #openstack-infra | 18:26 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove deprecated upload-afs role https://review.opendev.org/705373 | 18:27 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Use Fedora 30 for openshift functional job https://review.opendev.org/709581 | 18:29 |
| *** smarcet has joined #openstack-infra | 18:29 | |
| Shrews | corvus: i suspect you'll need ^^ for your nodepool change to merge | 18:29 |
| openstackgerrit | sebastian marcet proposed opendev/puppet-openstackid master: Removed stale config variables https://review.opendev.org/709525 | 18:31 |
| *** ralonsoh has quit IRC | 18:33 | |
| *** igordc has joined #openstack-infra | 18:36 | |
| corvus | Shrews: aha, thanks! | 18:37 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: k8s: allow port forwarding https://review.opendev.org/709259 | 18:37 |
| corvus | rebased | 18:37 |
| openstackgerrit | Merged openstack/project-config master: Remove obsolete releases publishing https://review.opendev.org/708920 | 18:37 |
| *** chandankumar is now known as raukadah | 18:38 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Use CentOS8 dependencies repo to install openvswitch https://review.opendev.org/708402 | 18:40 |
| *** smarcet has quit IRC | 18:45 | |
| *** sean-k-mooney has quit IRC | 18:45 | |
| noonedeadpunk | folks, you can release autohold 198.72.124.26 for https://review.opendev.org/#/c/707943 | 18:45 |
| mordred | clarkb: https://review.opendev.org/#/c/709579/1/specs/cleanup-test-node-python.rst is a spec that we would ultimately want to get an ack from our opendev customers in the new world order, right? | 18:46 |
| clarkb | noonedeadpunk: I'll get it, thanks | 18:46 |
| clarkb | mordred: yup I think this is our first example of that | 18:46 |
| *** ociuhandu has joined #openstack-infra | 18:48 | |
| clarkb | noonedeadpunk: and done | 18:48 |
| mordred | clarkb: spec looks great | 18:48 |
| *** dpawlik has quit IRC | 18:49 | |
| *** ociuhandu has quit IRC | 18:54 | |
| noonedeadpunk | clarkb: thanks! | 18:57 |
| *** amoralej is now known as amoralej|off | 19:00 | |
| *** smarcet has joined #openstack-infra | 19:03 | |
| mordred | clarkb, corvus: https://review.opendev.org/#/c/709295/ gerrit 3.1 build now | 19:07 |
| *** eharney has quit IRC | 19:07 | |
| mordred | corvus: the approach to dealing with per-repo submodules is clearly bong and needs to be replaced with the stuff from that dude | 19:08 |
| smarcet | fungi:sorry to bother you again, but this job its failing http://zuul.openstack.org/build/229f62d9591e486a9a7c65b433c7c964 | 19:11 |
| smarcet | fungi: seems that i can not create a dir | 19:11 |
| smarcet | fungi: at opendev.org/openstack/project-config/playbooks/publish/openstack-artifacts.yaml | 19:11 |
| corvus | mordred: you copied in the role i wrote, right? why is the polymer-bridges playbook needed? | 19:13 |
| fungi | smarcet: this looks like possible fallout from the tarball site move... ianw awake yet? i'll start looking into it | 19:14 |
| corvus | mordred: oh, wait, did you not copy in my repo setup script? | 19:14 |
| corvus | mordred: mine was based on yours from opendev, but i made some changes -- maybe we can fold those back into opendev | 19:14 |
| fungi | smarcet: ianw: my guess is i just need to manually create the /afs/.openstack.org/project/tarballs.opendev.org/ parent directory | 19:15 |
| clarkb | fungi: https://review.opendev.org/708920 just merged | 19:15 |
| fungi | clarkb: related, but probably not the direct cause | 19:15 |
| clarkb | fungi: oh huh. We may want to streamline that so that new projects don't need manual intervention | 19:15 |
| fungi | clarkb: i agree, maybe we should be creating parent dirs in the job if they don't exist | 19:16 |
| corvus | mordred: yeah, why don't we just pull in prepare-gerrit-repos for now, and then set up that jobvar datastructure on the jobs to map in the gitmodules? | 19:16 |
| fungi | smarcet: ianw: also i meant create the /afs/.openstack.org/project/tarballs.opendev.org/osf parent directory | 19:16 |
| mordred | corvus: cool - I'll do that | 19:17 |
| corvus | mordred: that's basically a generalized form of those one-off playbooks you have for jgit and polymer | 19:17 |
| corvus | mordred: this role https://gerrit.googlesource.com/zuul/jobs/+/refs/heads/master/roles/prepare-gerrit-repos/ | 19:17 |
| corvus | mordred: though i don't see where gerrit_project_mapping ended up | 19:18 |
| mordred | corvus: yah. totes. looking at it right now | 19:19 |
| corvus | mordred: oh we actually don't need it for jgit or polymer-bridges | 19:19 |
| corvus | mordred: so yeah, the role should just work as-is | 19:20 |
| corvus | (and the gerrit_project_mapping var is just a get-out-of-jail card if we run into trouble, but so far, the project names == their directory locations inside the gerrit repo) | 19:20 |
| mordred | corvus: jgit is different | 19:21 |
| corvus | where does it go? | 19:21 |
| mordred | jgit is the project name, it's in modules/jgit in the repo | 19:21 |
| openstackgerrit | Merged zuul/nodepool master: Use Fedora 30 for openshift functional job https://review.opendev.org/709581 | 19:21 |
| smarcet | fungi: you mean to update the opendev.org/openstack/project-config/playbooks/publish/openstack-artifacts.yaml job to create the parent dir ? | 19:22 |
| corvus | mordred: oh, i guess we haven't done a full gerrit build upstream yet so i didn't notice that | 19:22 |
| corvus | mordred: so yeah, i think wherever you add jgit to required-projects, also add vars: gerrit_project_mapping: {'jgit', 'modules/jgit'} | 19:23 |
| corvus | mordred: lol, that's the default from the role :) | 19:24 |
| mordred | corvus: also I need to modify the role a bit too ... | 19:24 |
| corvus | mordred: so that's why it's not needed | 19:24 |
| mordred | corvus: becuase it checks things about project.branch | 19:24 |
| mordred | but in our case, project == system-config | 19:24 |
| fungi | smarcet: i think we should do it in the upload-afs-synchronize role, i'm just trying to find the role definition now | 19:25 |
| AJaeger | fungi: zuul/zuul-jobs | 19:25 |
| openstackgerrit | Merged zuul/nodepool master: k8s: allow port forwarding https://review.opendev.org/709259 | 19:25 |
| mordred | corvus: I'm just going to modify it a bit for system-config for now ... then I think we should think about how we could modify things so that system-config could consume these directly from gerrit | 19:25 |
| corvus | mordred: yeah, i agree they are targeting 2 different audiences | 19:26 |
| mordred | corvus: it might be splitting it into 2 roles - one that does the guard conditions and one that does the submodules | 19:26 |
| fungi | AJaeger: thanks, just found it myself. i forgot to checkout origin/master and it was added more recently than my workdir | 19:26 |
| corvus | mordred: in your change, why do you do a submodule init instead of a mv? i thought jgit was a subscribed submodule? | 19:28 |
| corvus | (same for polymer) | 19:29 |
| mordred | corvus: because you only need to do a mv if it's not a subscribed submodule - if it's subscirbed, with no origin remote, it'll find things correctly relative to the gerrit repo on disjk | 19:31 |
| corvus | mordred: oh right | 19:31 |
| clarkb | subscribed submodule? | 19:31 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Precreate AFS target directory in upload https://review.opendev.org/709594 | 19:31 |
| corvus | mordred: ok, then if you go with the prepare-gerrit-repos role, i think we can just drop lines 36-40 and keep only lines 42-44 -- the "move" case | 19:32 |
| corvus | clarkb: gerrit submodule subscriptions | 19:32 |
| fungi | ianw: AJaeger: smarcet: i think https://review.opendev.org/709594 should do it | 19:32 |
| corvus | clarkb: where gerrit automatically updates the .gitmodules pointer for submodules on the same gerrit (eg, what we do with openstack/openstack) | 19:32 |
| mordred | clarkb: it's really just whether it's a submodule - the other things that we're moving are plugins that are not submodules | 19:32 |
| smarcet | fungi: thx u | 19:32 |
| clarkb | gotcha we move repos into place if they are not proepr submodules, otherwise it just works | 19:33 |
| corvus | mordred: prepare-gerrit-roles moves submodules (it does not do the remote editing trick) | 19:33 |
| mordred | corvus: it does remove the origin remote | 19:34 |
| mordred | corvus: so it could just do submodule init | 19:34 |
| AJaeger | fungi: thanks, reviewing... | 19:34 |
| corvus | oh, hrm | 19:34 |
| mordred | corvus: it's possible this is a complex thing :) | 19:34 |
| *** matt_kosut has quit IRC | 19:35 | |
| corvus | mordred: but we generally want to use 'mv' so that we get speculative repo state | 19:36 |
| AJaeger | fungi: should we use the file module of ansible instead of calling mkdir directly? https://stackoverflow.com/questions/22844905/how-to-create-a-directory-using-ansible ? | 19:36 |
| *** matt_kosut has joined #openstack-infra | 19:37 | |
| corvus | mordred: oh i have another idea | 19:37 |
| fungi | AJaeger: as long as it's idempotent and creates all parents | 19:37 |
| openstackgerrit | Clark Boylan proposed opendev/infra-specs master: Add spec for website activity stats effort https://review.opendev.org/709236 | 19:37 |
| fungi | AJaeger: i guess recurse:yes solves the second need | 19:37 |
| clarkb | fungi: it is idempotent | 19:38 |
| AJaeger | fungi: yes. Not sure what's better here on style | 19:38 |
| corvus | mordred: you can change the 'project_branch_exists' check to test 'zuul.projects['gerrit'].checkout' rather than 'zuul.branch'. | 19:38 |
| mordred | corvus: nod. to check that the matching branch exists | 19:38 |
| corvus | yeah, that should be an equivalent test for us | 19:38 |
| corvus | (as long as we aren't using this to build bare plugins) | 19:39 |
| fungi | clarkb: AJaeger: oh, actually state:directory does the equivalent of -p, not recurse:yes | 19:39 |
| corvus | (and we aren't, we'd always build them with gerrit) | 19:39 |
| corvus | i'm going to grab lunch | 19:39 |
| openstackgerrit | Jeremy Stanley proposed zuul/zuul-jobs master: Precreate AFS target directory in upload https://review.opendev.org/709594 | 19:40 |
| fungi | AJaeger: clarkb: ^ | 19:40 |
| AJaeger | fungi: +2A | 19:41 |
| AJaeger | thanks, fungi ! | 19:41 |
| fungi | smarcet: as soon as that merges i can reenqueue that ref in the post pipeline | 19:42 |
| fungi | should hopefully only be a few minutes | 19:42 |
| clarkb | I'm going to find lunch now. Then will address any new comments on those specs and send out the infra meeting agenda | 19:44 |
| fungi | enjoy | 19:44 |
| *** jamesmcarthur has quit IRC | 19:44 | |
| *** dpawlik has joined #openstack-infra | 19:45 | |
| *** eharney has joined #openstack-infra | 19:49 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 19:51 |
| mordred | corvus: ^^ that may do the right thing | 19:51 |
| zbr | what is the status of gitea? any chance to be able to enable the issue tracker on it in the near future? | 19:56 |
| fungi | zbr: i don't think we ever planned to enable the issue tracker in gitea? | 19:57 |
| fungi | we discussed enabling the wiki | 19:58 |
| *** igordc has quit IRC | 19:58 | |
| fungi | but that all hinges on the elasticsearch indexing backend for gitea getting completed first so we can cluster the servers | 19:58 |
| zbr | i remember that I asked that at some point and I was told that there was an issue with our muti-server deployment | 19:58 |
| fungi | yeah, it would also prevent using the issue tracker in gitea, but whether opendev chooses to support that feature is independent from whether we technically can use it | 19:59 |
| openstackgerrit | Merged zuul/zuul-jobs master: Precreate AFS target directory in upload https://review.opendev.org/709594 | 20:00 |
| *** ociuhandu has joined #openstack-infra | 20:01 | |
| *** TristanSullivan has joined #openstack-infra | 20:05 | |
| fungi | smarcet: ^ i'm reenqueuing your branch tip now | 20:09 |
| smarcet | fungi: thx u :) | 20:09 |
| fungi | or i would be if not for "ImportError: 'gitdb' could not be found in your PYTHONPATH" | 20:09 |
| fungi | geez. just a sec ;) | 20:09 |
| clarkb | python3 | 20:10 |
| fungi | clarkb: yeah, that's what the shebang says | 20:11 |
| fungi | `sudo zuul enqueue-ref ...` on zuul.o.o is coming back with that exception | 20:11 |
| clarkb | huh | 20:11 |
| fungi | `python3 -m pip list|grep gitdb` says we have both gitdb (0.6.4) and gitdb2 (2.0.6) installed | 20:12 |
| fungi | i have a feeling if we were to restart services on our zuul servers right now they'd all fall over | 20:12 |
| fungi | if i interactively `import gitdb` under python3 there i get "ImportError: 'smmap' could not be imported, assure it is located in your PYTHONPATH" | 20:13 |
| AJaeger | fungi: frickler run into that earlier, I think he left a question for us on that | 20:13 |
| *** dpawlik has quit IRC | 20:14 | |
| fungi | and pip3 says we have smmap (3.0.1) and smmap2 (3.0.1) installed | 20:14 |
| *** jackedin has quit IRC | 20:14 | |
| fungi | but interactive python3 can't `import smmap` | 20:14 |
| fungi | ImportError: No module named 'smmap' | 20:15 |
| fungi | i have a feeling we need to pip install --upgrade some python libs | 20:15 |
| AJaeger | fungi: http://eavesdrop.openstack.org/irclogs/%23openstack-infra/latest.log.html#t2020-02-24T10:59:01 | 20:17 |
| clarkb | is smmap in /usr/local/lib? I wonder if pip tracks that separate to what is on disk | 20:17 |
| AJaeger | ianw: once you're awake, please review https://review.opendev.org/709449, see backscroll for discussion with evrardjp | 20:18 |
| fungi | i did confirm that a local zuul virtualenv using python3 is able to enqueue-ref just fine (so the job in question is rerunning now) | 20:18 |
| fungi | clarkb: the /usr/local/lib/python3.5/dist-packages/smmap* directories only have a handful of text files in them for metadata | 20:19 |
| fungi | only the dist-info trees | 20:20 |
| fungi | no actual python module trees | 20:20 |
| clarkb | I wonder if that waspart of the broken releases that got deleted | 20:20 |
| clarkb | then pip thinks the version we have is new enpugh so isnt upgrading it | 20:21 |
| fungi | strangely, my local virtualenv claims to have the same smmap{,2} releases | 20:22 |
| ianw | AJaeger: looking :) | 20:23 |
| fungi | doesn't look like there are any .post releases for those on pypi either | 20:23 |
| ianw | oh, one line of scrollback and i see smmap2 and i think i might go back to bed :) | 20:23 |
| fungi | ianw: you, sir, are very wise | 20:24 |
| fungi | (also thanks for tackling all that while most of us were sleeping) | 20:24 |
| *** ahosam has joined #openstack-infra | 20:26 | |
| AJaeger | ianw: thanks. We first need to merge and then you move - or you copy files over, merge, remove. | 20:26 |
| AJaeger | ianw: but get your coffee etc done first - just wanted to be sure to remind you before I call it a day... | 20:27 |
| *** matt_kosut has quit IRC | 20:27 | |
| fungi | "TASK [upload-afs-synchronize : Precreate AFS target directory]" took 53 seconds to complete :/ | 20:28 |
| fungi | but worked! ;) | 20:28 |
| ianw | brtknr: from scrollback, the issue with fedora31 is that they changed the compression format of rpms for this release, and we can't build f31 on xenial any more | 20:28 |
| ianw | so we need to upgrade our builders to bionic (or, do more drastic things like build from docker containers as a base etc) that haven't quite made it into production yet | 20:29 |
| AJaeger | fungi: 53seconds? Let's go back to mkdir ;/ | 20:29 |
| clarkb | ianw: was the segfault issue fixed or was that the same issue? iirc we have paused f30 builds for months? | 20:30 |
| *** ociuhandu has quit IRC | 20:32 | |
| ianw | ummm, i can't remember to be honest. i spent a lot of time getting nodepool-builder jobs using containers which are live. i just need to get back to putting one in production | 20:32 |
| *** ociuhandu_ has joined #openstack-infra | 20:32 | |
| ianw | mordred: can you look at https://review.opendev.org/#/c/709414/ which adds tox_constraints_file to install-javascript-packages; see https://review.opendev.org/#/c/709412/ for the change driving that | 20:33 |
| fungi | okay, next challenge i've hit... i think we need to copy old content from https://tarballs.opendev.org/openstack/openstackid/ to https://tarballs.opendev.org/osf/openstackid/ and then set up a redirect or rewrite | 20:35 |
| *** ijw has quit IRC | 20:35 | |
| fungi | the new jobs have basically made a non-backward-compatible change for any repos outside the openstack namespace which were publishing tarballs | 20:35 |
| clarkb | fungi: your change fixes the compat issue though right? | 20:37 |
| clarkb | now just need to sync content? | 20:37 |
| fungi | clarkb: no, the old url is hard-coded in puppet-openstackid | 20:38 |
| fungi | the new publication location is osf/openstackid but we're redirecting to openstack/openstackid | 20:38 |
| clarkb | oh I see | 20:39 |
| fungi | just need to find where we're doing the redirect and add a more fine-grained one to the new publication path | 20:39 |
| mordred | ianw: o_O | 20:39 |
| ianw | fungi: that's in system-config:playbooks/roles/static/files/50-tarballs.opendev.org.conf | 20:39 |
| fungi | thanks ianw! you just saved me 5 minutes of searching ;) | 20:40 |
| mordred | ianw: I mean ... | 20:41 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: static: provide git.openstack.org https://review.opendev.org/709403 | 20:41 |
| mordred | ianw: you have a comment that says you proposed not running tox in the npm for horizon but the ref you linked is the change in question | 20:41 |
| mordred | ianw: nm. I found it | 20:42 |
| ianw | mordred: oh, i might have copy pasted. i think however that breaks xstatic which is why it's doing it | 20:42 |
| mordred | nod | 20:42 |
| ianw | yeah, https://review.opendev.org/#/c/709415/ | 20:42 |
| *** matt_kosut has joined #openstack-infra | 20:42 | |
| ianw | i think this must have been lurking, but just happened to be the first time packages that horizon cared about installing actually broke | 20:43 |
| clarkb | so it doesn't get lost do we have a fix for zuul installs? | 20:44 |
| mordred | so - the *real* answer here would be to stop doing that xstatic thing - which is a much bigger effort | 20:44 |
| mordred | clarkb: aroo? | 20:44 |
| clarkb | because I suspect fungi is right, restarting those services will cause them all to fail | 20:44 |
| clarkb | mordred: basically smmap isn't properly installed and gitpython needs that for gitdb | 20:44 |
| mordred | clarkb: sob | 20:45 |
| clarkb | mordred: its related to the pypi package problems but separate issue from the data-require-python problem I think | 20:45 |
| clarkb | there are only .dist-info dirs for smmap and smmap2 and no actual package content | 20:45 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Redirect openstackid tarball URLs to osf namespace https://review.opendev.org/709608 | 20:46 |
| fungi | clarkb: ianw: smarcet: ^ | 20:46 |
| clarkb | smmap did release yesterday | 20:46 |
| clarkb | the package doesn't seem to be broken | 20:47 |
| clarkb | fungi: is the $1 matcher correct in that? It should include openstackid in the group right? | 20:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stream output from kubectl pods https://review.opendev.org/709261 | 20:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add destructor for SshAgent https://review.opendev.org/709609 | 20:48 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Redirect openstackid tarball URLs to osf namespace https://review.opendev.org/709608 | 20:48 |
| fungi | clarkb: good eye! | 20:48 |
| corvus | clarkb: wow i just got back from lunch and started reading scrollback backwards and wondered why fungi was getting matchers from the dollar store | 20:49 |
| fungi | corvus: matches, and they sell them by the case for a dollar, just don't count on every one to light | 20:49 |
| ianw | fungi: sorry, i clearly hadn't even considered that namespacing mingling. any thoughts on how we could find others? | 20:50 |
| *** nicolasbock has quit IRC | 20:50 | |
| fungi | ianw: probably by guessing that most of our projects have unique short names and then matching existing directories from tarballs.o.o against them | 20:50 |
| *** eharney has quit IRC | 20:50 | |
| fungi | i can probably take a stab at that once we're sure this first one is working | 20:51 |
| clarkb | "Feb 23 16:17:04 zuul01 puppet-user[13997]: (/Stage[main]/Zuul/Exec[install_zuul]) Triggered 'refresh' from 1 events" seems to be all we get in the syslog logs on zuul01 re pip weirdness | 20:51 |
| fungi | ianw: and no need to apologize, none of us considered that | 20:51 |
| *** nicolasbock has joined #openstack-infra | 20:51 | |
| ianw | clarkb: is the dependency for smmap or smmap2? | 20:52 |
| clarkb | ianw: they appear to be the same thing | 20:53 |
| ianw | yeah, as of like yesterday | 20:53 |
| clarkb | smmap2 is a mirror package for smmap | 20:53 |
| clarkb | oh | 20:53 |
| clarkb | that is what changed gotcha | 20:53 |
| clarkb | ianw: it comes from gitdb2/gitdb | 20:54 |
| corvus | fungi: i think you don't need a '/?' in your matcher, but i don't think it's wrong, it's just superfluous and is canceled out... | 20:54 |
| clarkb | whcih I'm guessing is a similar situation | 20:54 |
| ianw | clarkb: yeah ... interesting that zuul depends on gitdb but it's not in requirements explicitly, something else drags it in? | 20:54 |
| fungi | corvus: will it still redirect openstackid without anything after it (including a /)? | 20:54 |
| clarkb | ianw: GitPython | 20:55 |
| fungi | corvus: oh, yep, you're right | 20:55 |
| fungi | as /?.* is equivalent to .* | 20:55 |
| corvus | yep that | 20:55 |
| corvus | i approved but will re +2 if you decide to simplify | 20:56 |
| fungi | corvus: it's probably first in a list, so we can just clean it up when we add more | 20:56 |
| corvus | kk | 20:56 |
| fungi | what i wanted to convey was that we should only redirect openstackid or openstackid/something but not openstackidsomething | 20:56 |
| openstackgerrit | Zane Bitter proposed openstack/cookiecutter master: Minimise info in CONTRIBUTING.rst https://review.opendev.org/708672 | 20:56 |
| openstackgerrit | Zane Bitter proposed openstack/cookiecutter master: Autofill bug tracker link in contributor docs https://review.opendev.org/708703 | 20:56 |
| fungi | that would require deeper regex hackery though | 20:56 |
| fungi | and probably a corner case we hit with 0 actual projects which predate the new site | 20:57 |
| corvus | fungi: yeah, that getting into "|$" territory i think | 20:57 |
| fungi | but i'll mull it over | 20:57 |
| mordred | corvus: zuul.projects['gerrit'].checkout is unhappy - that should probably fully qualify? | 20:57 |
| corvus | mordred: yep | 20:57 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Use prepare-gerrit-repo role for gerrit submodules https://review.opendev.org/709602 | 20:58 |
| clarkb | I'm beginning to wonder if there was some sort of magic done to transition smmap2 to smmap at a package level and that is what broke things for us | 21:01 |
| clarkb | I'm going to try and mock that up locally and see if that reproduces | 21:01 |
| corvus | clarkb: ++ | 21:01 |
| *** smarcet has quit IRC | 21:02 | |
| corvus | because we should be running "pip3 install -U -r/opt/zuul/requirements.txt" on zuul01 | 21:02 |
| *** matt_kosut has quit IRC | 21:04 | |
| clarkb | "smmap2 v3.0.0 was released as the entire smmap library while still requiring smmap" "smmap2 v3.0.1 has been released on PyPI from the new smmap2 branch (2422b98)" "smmap2 v3.0.0 has been deleted on PyPI" from https://github.com/gitpython-developers/smmap/issues/44 helps track some of the history for smmap | 21:05 |
| corvus | zuul01 has smmap==3.0.1 and smmap2==3.0.1 | 21:06 |
| fungi | well, it sort of does | 21:07 |
| clarkb | corvus: it thinks it has those packages but the packages aren't actually there, just the dist-info dir and metadata is there | 21:07 |
| fungi | it has the dist-info directories only | 21:07 |
| fungi | not the library trees | 21:07 |
| corvus | "cool" | 21:07 |
| clarkb | fwiw I can't reproduce via new virtualenv installing smmap2==2.0.5 then installing zuul over the top of it | 21:07 |
| fungi | yeah, wtg pip | 21:07 |
| clarkb | note smmap2 is imported as smmap not smmap2 | 21:07 |
| fungi | but smmap2 is now a transitional package to smmap anyway, right? | 21:08 |
| clarkb | fungi: correct | 21:08 |
| fungi | smarcet: i've temporarily copied openstackid-3.0.0.tar.gz and openstackid-latest.tar.gz into the directory which tarballs.openstack.org/openstackid redirects to. the correct new url for those files is going to be https://tarballs.opendev.org/osf/openstackid/ but we've got a redirect fix in flight and i need to copy the old content into that new directory. once that's done it should continue to work | 21:08 |
| fungi | transparently with the code in puppet-openstackid but i'll push up a patch for it to switch to the new url so we're not hitting the redirect | 21:08 |
| clarkb | also running pip install -U zuul doesn't upgrade smmap/smmap2 to 3.0.1 | 21:10 |
| clarkb | I still have 2.0.5 | 21:10 |
| corvus | clarkb: yeah, but that's not what we do, see above | 21:10 |
| corvus | clarkb: "pip3 install -U -r/opt/zuul/requirements.txt" is what we run (then we install zuul with nodeps) | 21:10 |
| clarkb | gotcha | 21:10 |
| clarkb | ok let me make a new venv and try again | 21:10 |
| corvus | just thinking ahead to possible fixes: maybe we pip uninstall those, then pip install them again? | 21:12 |
| clarkb | corvus: ya I think we pip uninstall both then only pip install smmap not smmap2 | 21:12 |
| clarkb | similar with gitdb | 21:12 |
| clarkb | fwiw doing what puppet does still doesn't upgrade smmap2==2.0.5 to smmap==3.0.1 | 21:13 |
| clarkb | which makes me think that perhaps the intermediate packages that have been deleted (as noted in that github issue) may be to blame | 21:13 |
| corvus | i wonder if we got here through some very specific series of conditions and timing | 21:13 |
| corvus | yeah that | 21:13 |
| mordred | have I mentioned how much people shouldn't delete released packages? | 21:14 |
| clarkb | corvus: I think what we should do is pip uninstall GitPython, gitdb, gitdb2, smmap, smmap2. then do what puppet does to install zuul requirements | 21:14 |
| *** slaweq has quit IRC | 21:14 | |
| mordred | clarkb: ++ | 21:14 |
| corvus | mordred: i don't recall you saying so in the last 5 minutes or so | 21:14 |
| *** slaweq_ has joined #openstack-infra | 21:14 | |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-openstackid master: Update tarball publication URLs https://review.opendev.org/709613 | 21:14 |
| mordred | corvus: that's surprising | 21:14 |
| mordred | clarkb: I think that's a good plan | 21:14 |
| corvus | clarkb: ++ want to try that on zuul01 | 21:14 |
| clarkb | corvus: ya I'll give that a go on zuul01 | 21:15 |
| clarkb | or actually would one of the executors be better? | 21:15 |
| clarkb | since we have many of those but only one scheduler | 21:15 |
| fungi | that's basically what i was considering, but didn't know which packages to do it with and how deep this rabbit hole goes | 21:15 |
| corvus | the executors actually use those code paths, so i'm certain they will break; weirdly the scheduler may not. but i can't promise that. :) | 21:15 |
| corvus | clarkb: tell you what | 21:15 |
| corvus | clarkb: the safest+easiest thing to do is to shut down a merger and try it there | 21:16 |
| clarkb | ok I'll use zm01 | 21:16 |
| corvus | clarkb: and you can run "zuul" (with no args) on the merger to to verify | 21:16 |
| clarkb | I've confirmed that zm01 exhibits the same setup in /usr/local/lib | 21:17 |
| fungi | ianw: what's your recommendation for a "fast" (in afs terms) server to do the rsync from openstack/openstackid to osf/openstackid on? | 21:17 |
| clarkb | smmap2-3.0.1.dist-info and smmap-3.0.1.dist-info exist but no smmap | 21:17 |
| clarkb | stopping the merger service now | 21:17 |
| ianw | fungi: i always log into mirror-update.opendev.org and kinit/aklog from there | 21:18 |
| ianw | fungi: ... document has moved <a href="https://opendev.org//openstack/nova">here</a> ... do you remember anything to do with "//" when doing the git redirects? for https://review.opendev.org/#/c/709403/ where i'm moving to static.opendev.org | 21:19 |
| ianw | it might be the curl and the name resolve override it does | 21:20 |
| clarkb | hrm do mergers take time to stop? sudo systemctl stop zuul-merger doesn't seem to have done much | 21:21 |
| ianw | ...oohhhh i think i know. in htaccess you're relative to the directory, in virtualhost relative to the root | 21:22 |
| clarkb | systemctl list-units doesn't show any zuul units this gets weirder | 21:23 |
| *** smarcet has joined #openstack-infra | 21:23 | |
| *** jamesmcarthur has joined #openstack-infra | 21:23 | |
| clarkb | hahahahaok | 21:24 |
| clarkb | Feb 24 21:17:42 zm01 zuul-merger[26951]: ImportError: 'gitdb' could not be found in your PYTHONPATH | 21:24 |
| clarkb | it couldn't stop the service because of this issue | 21:24 |
| mordred | clarkb: wow | 21:25 |
| clarkb | should I try to surgery this with the service running or manually stop it (with kill and rm of pid file) | 21:25 |
| corvus | clarkb: 1 sec | 21:27 |
| *** sreejithp has quit IRC | 21:27 | |
| clarkb | `sudo systemctl status zuul-merger` is how I got to see where the error above came from | 21:27 |
| clarkb | `sudo pip3 uninstall smmap smmap2 gitdb gitdb2 GitPython && sudo pip3 install -U -r/opt/zuul/requirements.txt` is the planned "fix" command | 21:27 |
| fungi | ianw: yep, minor differences in how redirects work for .htaccess can get you that extra / | 21:30 |
| *** igordc has joined #openstack-infra | 21:30 | |
| corvus | clarkb: just "kill" it, it should shut down nicely | 21:30 |
| clarkb | corvus: ok will do | 21:30 |
| corvus | or, as nice as it would the other way | 21:31 |
| clarkb | looks like that even cleaned up the merger.pid file | 21:31 |
| clarkb | running the above pip3 commands now | 21:31 |
| clarkb | that seems to have worked though I had to manually confirm each of the uninstalls. We'll want to run with -y if scripting it | 21:33 |
| clarkb | going to start the merger on zm01 now | 21:33 |
| *** matt_kosut has joined #openstack-infra | 21:33 | |
| clarkb | seems to have started ok | 21:33 |
| corvus | clarkb: cool, maybe next step is to do that with ansible on zm* ? | 21:33 |
| corvus | or do we want to shut all of zuul down, do that, then restart | 21:34 |
| corvus | cause we probably want to run the new executor code to protect from oom | 21:34 |
| clarkb | corvus: do we want to see zm01 process some merge requests first? I think that is the last thing we are waiting on. Otherwise ya I think we can run `sudo pip3 uninstall -y smmap smmap2 gitdb gitdb2 GitPython && sudo pip3 install -U -r/opt/zuul/requirements.txt` in batches across zuul hosts | 21:34 |
| clarkb | corvus: oh good point re the OOM | 21:34 |
| corvus | clarkb: yeah, let's observe zm01 | 21:34 |
| corvus | clarkb: maybe we can take a middle path: shut down zm* and ze*, leave the scheduler running but idle, then do the pip install then start everyone? | 21:35 |
| clarkb | corvus: and the issue with doing the pip uninstall && install while services run is we fork now for git operations? | 21:36 |
| clarkb | otherwise we could do the updates first, then stop and start? | 21:36 |
| corvus | clarkb: well, they're polled, so they might survive okay? | 21:36 |
| corvus | pooled | 21:36 |
| corvus | but there's a non-zero chance they could try an import while it's happening | 21:37 |
| *** sreejithp has joined #openstack-infra | 21:37 | |
| *** matt_kosut has quit IRC | 21:37 | |
| clarkb | got it, in that case I think your middle road option sounds good | 21:37 |
| clarkb | (still waiting for zm01 to run a merge job) | 21:38 |
| *** jamesmcarthur has quit IRC | 21:39 | |
| clarkb | corvus: do we want to use the playbook for that? I'm guessing it might be simpler to just do a few ad hoc ansible commands | 21:40 |
| *** matt_kosut has joined #openstack-infra | 21:40 | |
| corvus | clarkb: yeah, i think ad-hoc | 21:40 |
| *** Goneri has quit IRC | 21:40 | |
| clarkb | ansible 'zm*' -m shell -a 'kill $(cat /var/run/zuul/merger.pid)' Then once stopped run the pip3 command then start the service? | 21:41 |
| clarkb | then repeat for ze*? My concern is that with broken init management we may need to be a bit more hands on | 21:41 |
| corvus | clarkb: yep | 21:41 |
| clarkb | alright I'll start that for zm* now | 21:42 |
| *** slaweq_ has quit IRC | 21:42 | |
| clarkb | or should we wait for zm01 to do something? | 21:43 |
| fungi | #status log copied content from /afs/.openstack.org/project/tarballs.opendev.org/openstack/openstackid to ../../osf/openstackid to reflect new publication path | 21:43 |
| openstackstatus | fungi: finished logging | 21:43 |
| clarkb | corvus: ^ what do you think? I think its probably fine to move ahead since the problems are import time | 21:44 |
| clarkb | but there could be additional problems we haven't seen yet | 21:45 |
| *** matt_kosut has quit IRC | 21:45 | |
| fungi | ianw: can i get your blessing on 709613 now that i've rsync'd the content? | 21:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stream output from kubectl pods https://review.opendev.org/709261 | 21:46 |
| clarkb | oh cool zm01 just did something | 21:46 |
| clarkb | it updated /var/lib/zuul/git/opendev.org/x/ranger | 21:46 |
| clarkb | and no tracebacks so I'm going to proceed with the other stuff | 21:46 |
| corvus | clarkb: sounds like we're gtg | 21:46 |
| fungi | ianw: thanks! i've left myself a to do entry to clean up the old directory once the redirect is confirmed working too | 21:47 |
| ianw | ++ | 21:47 |
| *** smarcet has quit IRC | 21:50 | |
| clarkb | ok one small hiccup in that running that uninstall command against zm01 was no longer valid because we don't reinstall all of the same packages that we uninstalled | 21:50 |
| clarkb | this meant I had to manually rerun the pip3 install -U -r requirements.txt since we used and && | 21:51 |
| *** ijw has joined #openstack-infra | 21:51 | |
| clarkb | all 8 mergers should be up and running with the fixes though | 21:51 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Stream output from kubectl pods https://review.opendev.org/709261 | 21:52 |
| clarkb | corvus: is it safe to kill $(cat /var/run/zuul/executor.pid) ? | 21:52 |
| clarkb | I think it should be as gearman should sort it out for us then we clean up on startup | 21:52 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add destructor for SshAgent https://review.opendev.org/709609 | 21:52 |
| openstackgerrit | Merged opendev/system-config master: Redirect openstackid tarball URLs to osf namespace https://review.opendev.org/709608 | 21:53 |
| corvus | clarkb: yeah, and i think that should be a controlled shutdown too. do it for all executors at once though (with '-f 20') | 21:53 |
| corvus | clarkb: (to make sure we don't keep rolling over retried jobs to executors that subsequently shut down) | 21:53 |
| clarkb | corvus: ok. Also zm05 appears to have gotten work and hasn't failed yet | 21:53 |
| clarkb | I didn't check what version of zuul is installed first | 21:54 |
| clarkb | Let me go and do that really quickly | 21:54 |
| clarkb | (to ensure we get the forked worker pool fix) | 21:54 |
| openstackgerrit | Merged openstack/project-config master: Fix governance-website publishing https://review.opendev.org/709449 | 21:55 |
| clarkb | zuul==3.16.2.dev63 # git sha 32bfbae is what is one zm01 so that should be fine | 21:55 |
| corvus | agreed | 21:55 |
| *** jamesmcarthur has joined #openstack-infra | 21:55 | |
| clarkb | proceeding with executors now | 21:56 |
| clarkb | er /me checks for release jobs | 21:57 |
| clarkb | there is one thing queued in release-approval which I think is safe to have rerun | 21:57 |
| clarkb | notifying the release team now | 21:57 |
| *** artom has quit IRC | 21:58 | |
| clarkb | notified. Now I am proceeding | 21:59 |
| *** ociuhandu_ has quit IRC | 22:00 | |
| *** ociuhandu has joined #openstack-infra | 22:01 | |
| clarkb | and done | 22:03 |
| clarkb | I think zuul01 is the last remaining host that is sad | 22:03 |
| clarkb | and that one doesn't fork so we should be able to run the pip3 uninstall && pip install under the running services? | 22:04 |
| clarkb | corvus: ^ let me know if that sounds right and I can go ahead and do that there without stopping services | 22:04 |
| corvus | clarkb: yep, i think it's worth a shot :) | 22:04 |
| clarkb | ok proceeding with that on zuul01 now | 22:04 |
| *** ijw has quit IRC | 22:05 | |
| clarkb | and done | 22:05 |
| clarkb | I can import smmap on zuul01 now | 22:06 |
| mordred | \o/ | 22:06 |
| *** ociuhandu_ has joined #openstack-infra | 22:07 | |
| corvus | clarkb: thanks! | 22:08 |
| clarkb | infra-root please keep an eye on zuul and point out any git related weirdness | 22:08 |
| corvus | clarkb: have the executors finished stopping yet? | 22:08 |
| fungi | thanks for the heads up (and for working through the damage) | 22:08 |
| clarkb | corvus: ya they stopped really quickly I think kill may have been slightly more forceful than normal | 22:08 |
| corvus | oh, then they'll probably be slow to start | 22:08 |
| clarkb | corvus: ze01 at least is doing things that look nromal to me | 22:09 |
| *** ociuhandu has quit IRC | 22:09 | |
| clarkb | #status log On all Zuul hosts: uninstalled smmap, smmap2, gitdb, gitdb2, GitPython then ran pip3 install -U -r /opt/zuul/requirements.txt to reinstall Zuul's deps against current happy pypi state as package movement over the weekend left us in a sad state withe smmap, gitdb, and GitPython. | 22:10 |
| openstackstatus | clarkb: finished logging | 22:10 |
| corvus | ok, it looks like ze01 did delete stale jobdirs, but not very many of them | 22:10 |
| corvus | we may want to check on that in a couple days :) | 22:10 |
| *** rkukura_ has joined #openstack-infra | 22:16 | |
| *** ociuhandu_ has quit IRC | 22:17 | |
| *** ociuhandu_ has joined #openstack-infra | 22:17 | |
| *** rkukura has quit IRC | 22:18 | |
| *** rkukura_ is now known as rkukura | 22:18 | |
| clarkb | meeting agenda sent | 22:19 |
| *** dave-mccowan has joined #openstack-infra | 22:22 | |
| *** auristor has quit IRC | 22:25 | |
| ianw | fungi: do you happen to have a bunch of urls you might have tested with when writing the redirect stuff? i'd like to add it to testinfra just because we can and ensure we never break anything | 22:26 |
| ianw | git redirect stuff i mean | 22:26 |
| *** smarcet has joined #openstack-infra | 22:26 | |
| *** dave-mccowan has quit IRC | 22:27 | |
| fungi | ianw: i based it on corvus's earlier work, so it was mostly copy/paste. he might have some notes for the more novel bits | 22:28 |
| *** ociuhandu_ has quit IRC | 22:29 | |
| openstackgerrit | Merged opendev/system-config master: Removed OpenStackID stale config variables https://review.opendev.org/709529 | 22:29 |
| openstackgerrit | Merged opendev/puppet-openstackid master: Removed stale config variables https://review.opendev.org/709525 | 22:29 |
| openstackgerrit | Merged opendev/puppet-openstackid master: Update tarball publication URLs https://review.opendev.org/709613 | 22:29 |
| *** ociuhandu has joined #openstack-infra | 22:30 | |
| fungi | smarcet: ^ | 22:30 |
| fungi | those should take effect on the next config management pulse | 22:30 |
| *** smarcet has quit IRC | 22:30 | |
| *** Goneri has joined #openstack-infra | 22:32 | |
| openstackgerrit | James E. Blair proposed ttygroup/gertty master: Fix "Save and Submit" button https://review.opendev.org/709623 | 22:32 |
| clarkb | ianw: when you get a chance can you review https://review.opendev.org/#/c/709579/ the pip/python/mess spec for our dib images. Want to make sure I haven't missed anything important there | 22:34 |
| openstackgerrit | Merged ttygroup/gertty master: Handle binary data in diffs https://review.opendev.org/708932 | 22:34 |
| openstackgerrit | Merged ttygroup/gertty master: Hide errors when syncing conflicts https://review.opendev.org/708933 | 22:34 |
| openstackgerrit | Merged ttygroup/gertty master: Display line comments from all patchsets in change view https://review.opendev.org/708934 | 22:35 |
| clarkb | (and that it accurate reflects the plan we've congealed around) | 22:35 |
| openstackgerrit | Merged ttygroup/gertty master: Support robot comments https://review.opendev.org/708944 | 22:35 |
| openstackgerrit | Merged ttygroup/gertty master: Add support for checks plugin https://review.opendev.org/708992 | 22:35 |
| openstackgerrit | Merged ttygroup/gertty master: Fix "Save and Submit" button https://review.opendev.org/709623 | 22:35 |
| *** tkajinam has joined #openstack-infra | 22:35 | |
| *** ociuhandu has quit IRC | 22:35 | |
| *** mattw4 has quit IRC | 22:36 | |
| *** jamesmcarthur has quit IRC | 22:38 | |
| *** ijw has joined #openstack-infra | 22:39 | |
| openstackgerrit | Amy Marrich (spotz) proposed opendev/irc-meetings master: Update chairs list https://review.opendev.org/709626 | 22:40 |
| *** sreejithp has quit IRC | 22:40 | |
| *** Goneri has quit IRC | 22:40 | |
| *** ijw has quit IRC | 22:44 | |
| *** auristor has joined #openstack-infra | 22:44 | |
| *** dchen has joined #openstack-infra | 22:48 | |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Upgrade gitea to 1.11.1 https://review.opendev.org/705807 | 22:48 |
| *** jamesmcarthur has joined #openstack-infra | 22:48 | |
| mordred | clarkb: ^^ gitea has released 1.11.0 (and 1.11.1) - I believe that syncs template changes appropriately | 22:49 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] static: provide git services https://review.opendev.org/709403 | 22:49 |
| *** auristor has quit IRC | 22:51 | |
| *** ijw has joined #openstack-infra | 22:53 | |
| *** ijw has quit IRC | 22:58 | |
| *** jamesmcarthur has quit IRC | 23:02 | |
| clarkb | mordred: I'm reading the changelog, https://github.com/go-gitea/gitea/blob/master/CHANGELOG.md#1110---2020-02-10, and have two concerns. The first is the change of the cert and key file paths. Second is the enforce gitea env on pushes (I'm not sure yet if we are pushing into gitea in the expected manner) | 23:05 |
| clarkb | mordred: for the first thing I'm pretty confident we need to update our configuration mgmt stuff in ansible to write the correct config and/or change the paths of the ssl files | 23:05 |
| openstackgerrit | Merged zuul/gcp-authdaemon master: Initial commit https://review.opendev.org/707438 | 23:06 |
| *** ijw has joined #openstack-infra | 23:06 | |
| clarkb | ya for the certs I think the docker compose file wants to chnge the mount point from /var/gitea/certs:/certs to /var/gitea/certs:/custom/certs | 23:07 |
| *** auristor has joined #openstack-infra | 23:09 | |
| clarkb | and I've just confirmed the gitea role manages the gerrit ssh key through the gitea api so we should be fine on the push question | 23:09 |
| clarkb | I'll leave a -1 for the ssl key/cert pathing issue | 23:09 |
| *** ahosam has quit IRC | 23:10 | |
| clarkb | oh wait reading the actual issue for the key/cert issue I think it is only applicable to relative paths | 23:10 |
| clarkb | confirming against the code now | 23:10 |
| clarkb | https://github.com/go-gitea/gitea/pull/9868/files#diff-60f5fca6d29eafbb30eb645a0cd4041fR557-R562 confirmed, it will only affect non rooted paths | 23:12 |
| *** auristor has quit IRC | 23:13 | |
| mordred | clarkb: yeah - I think we're fine | 23:14 |
| mordred | on the key/cert | 23:14 |
| mordred | playbooks/roles/gitea/templates/app.ini.j2:CERT_FILE = /certs/cert.pem | 23:14 |
| mordred | also | 23:14 |
| mordred | relative paths to certs? oy | 23:14 |
| clarkb | +2'd with a note on what I learned in case others have questions | 23:16 |
| *** mattw4 has joined #openstack-infra | 23:18 | |
| mordred | woot | 23:19 |
| ianw | do we know about openstackci-beaker-puppet-4 failing? | 23:24 |
| *** auristor has joined #openstack-infra | 23:29 | |
| ianw | looks like nodepool launcher and builder failures to start | 23:29 |
| *** mattw4 has quit IRC | 23:29 | |
| ianw | rror: /Stage[main]/Diskimage_builder/Apt::Ppa[ppa:openstack-ci-core/vhd-util]/Exec[add-apt-repository-ppa:openstack-ci-core/vhd-util]/returns: change from notrun to 0 failed: Command exceeded timeout | 23:30 |
| ianw | ok, so probably just a random timeout on the ppa ... | 23:30 |
| openstackgerrit | Ian Wienand proposed opendev/system-config master: [wip] static: provide git services https://review.opendev.org/709403 | 23:30 |
| *** mattw4 has joined #openstack-infra | 23:34 | |
| *** auristor has quit IRC | 23:35 | |
| ianw | infra-root: i've switched all the other docs sites to static.opendev.org so just docs.opendev.org remains if you could have a look -> https://review.opendev.org/#/c/709402/ | 23:37 |
| *** auristor has joined #openstack-infra | 23:38 | |
| *** auristor has quit IRC | 23:42 | |
| *** tosky has quit IRC | 23:43 | |
| clarkb | ianw: looks like we only host a mod autoindex index at https://docs.opendev.org? Appears both the old and new site are that way (telling the difference by inspecting the ssl cert) | 23:44 |
| *** jamesmcarthur has joined #openstack-infra | 23:45 | |
| ianw | clarkb: yeah, i believe that's the idea, you really want to go deeper with https://docs.opendev.org/opendev/infra-specs/latest/ say | 23:45 |
| ianw | if you feel like https://review.opendev.org/#/c/709038/ https://review.opendev.org/#/c/709042/5 too that would be good :) | 23:46 |
| ianw | just the git.* change i have in flight now after that, and files.openstack.org can disappear | 23:47 |
| ianw | and all that's left of static.openstack.org is the random small redirects | 23:47 |
| *** jamesmcarthur has quit IRC | 23:49 | |
| *** jamesmcarthur has joined #openstack-infra | 23:50 | |
| ianw | failed to fetch PPA information, error was: Request failed: <urlopen error _ssl.c:835: The handshake operation timed out>" | 23:50 |
| ianw | boo | 23:50 |
| *** auristor has joined #openstack-infra | 23:52 | |
| *** jamesmcarthur has quit IRC | 23:55 | |
| clarkb | looks like AJaeger acked the removal of the 404s script | 23:55 |
| clarkb | worst case we can manually generate that list if necessary too | 23:55 |
| clarkb | (until we have an alternative in place) | 23:56 |
| *** auristor has quit IRC | 23:56 | |
| clarkb | and the second change removed the extra test case | 23:56 |
| clarkb | they lgtm | 23:57 |
| *** igordc has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!