| *** e0ne has joined #openstack-infra | 00:05 | |
| *** dychen has quit IRC | 00:08 | |
| * fungi stands down on force-merge alert in thaht case | 00:11 | |
| *** jamesmcarthur has quit IRC | 00:16 | |
| *** e0ne has quit IRC | 00:18 | |
| *** weifan has quit IRC | 00:21 | |
| *** jamesmcarthur has joined #openstack-infra | 00:21 | |
| *** weifan has joined #openstack-infra | 00:24 | |
| *** armax has quit IRC | 00:26 | |
| *** ociuhandu has joined #openstack-infra | 00:31 | |
| *** e0ne has joined #openstack-infra | 00:31 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** e0ne has quit IRC | 00:40 | |
| *** e0ne has joined #openstack-infra | 00:43 | |
| *** e0ne has quit IRC | 00:47 | |
| *** tkajinam_ has quit IRC | 00:49 | |
| *** tkajinam has joined #openstack-infra | 00:49 | |
| openstackgerrit | Merged openstack/devstack-gate master: Add gate job for stable/train https://review.opendev.org/686776 | 00:49 |
|---|---|---|
| *** e0ne has joined #openstack-infra | 00:54 | |
| *** weifan has quit IRC | 00:57 | |
| *** e0ne has quit IRC | 00:58 | |
| *** jamesmcarthur has quit IRC | 00:58 | |
| *** hamzy has joined #openstack-infra | 01:01 | |
| *** jamesmcarthur has joined #openstack-infra | 01:03 | |
| *** adriant has quit IRC | 01:04 | |
| *** iokiwi has quit IRC | 01:04 | |
| *** e0ne has joined #openstack-infra | 01:05 | |
| ianw | pabelanger/clarkb: https://bugzilla.redhat.com/show_bug.cgi?id=1761282 & https://review.opendev.org/688580 ... looks like the dnf issues are probably falling out as being fixed | 01:09 |
| openstack | bugzilla.redhat.com bug 1761282 in dnf "dnf segfault after "Generating completion cache"; likely libdnf 0.35.5" [High,Assigned] - Assigned to lhrazky | 01:09 |
| ianw | if i had to choose one term to describe it, "hot mess" comes to mind :) | 01:10 |
| *** bnemec has joined #openstack-infra | 01:12 | |
| *** e0ne has quit IRC | 01:12 | |
| fungi | (that's two words) | 01:14 |
| fungi | oh, you said "term" ;) | 01:14 |
| fungi | yeah, that does look un-fun | 01:17 |
| *** e0ne has joined #openstack-infra | 01:18 | |
| *** e0ne has quit IRC | 01:22 | |
| *** bnemec has quit IRC | 01:25 | |
| *** e0ne has joined #openstack-infra | 01:28 | |
| *** weifan has joined #openstack-infra | 01:29 | |
| *** weifan has quit IRC | 01:33 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** ykarel|afk has joined #openstack-infra | 01:34 | |
| *** e0ne has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #openstack-infra | 01:41 | |
| *** e0ne has joined #openstack-infra | 01:42 | |
| *** bnemec has joined #openstack-infra | 01:42 | |
| *** e0ne has quit IRC | 01:46 | |
| *** adriant has joined #openstack-infra | 01:47 | |
| *** ykarel|afk has quit IRC | 01:49 | |
| *** e0ne has joined #openstack-infra | 01:51 | |
| *** Garyx has quit IRC | 01:55 | |
| *** Garyx has joined #openstack-infra | 01:56 | |
| *** e0ne has quit IRC | 02:00 | |
| *** jamesmcarthur has quit IRC | 02:02 | |
| *** Garyx has quit IRC | 02:05 | |
| *** jamesmcarthur has joined #openstack-infra | 02:05 | |
| *** e0ne has joined #openstack-infra | 02:06 | |
| *** dave-mccowan has joined #openstack-infra | 02:06 | |
| *** Garyx has joined #openstack-infra | 02:07 | |
| *** apetrich has quit IRC | 02:09 | |
| *** e0ne has quit IRC | 02:14 | |
| *** e0ne has joined #openstack-infra | 02:17 | |
| *** ianychoi has quit IRC | 02:19 | |
| *** ianychoi has joined #openstack-infra | 02:21 | |
| *** e0ne has quit IRC | 02:22 | |
| *** ykarel has joined #openstack-infra | 02:22 | |
| *** e0ne has joined #openstack-infra | 02:23 | |
| *** weifan has joined #openstack-infra | 02:27 | |
| *** e0ne has quit IRC | 02:27 | |
| *** e0ne has joined #openstack-infra | 02:30 | |
| *** weifan has quit IRC | 02:30 | |
| *** markvoelker has joined #openstack-infra | 02:32 | |
| *** ramishra has joined #openstack-infra | 02:38 | |
| *** ricolin has joined #openstack-infra | 02:40 | |
| *** e0ne has quit IRC | 02:41 | |
| *** e0ne has joined #openstack-infra | 02:44 | |
| *** e0ne has quit IRC | 02:49 | |
| *** markvoelker has quit IRC | 02:52 | |
| *** e0ne has joined #openstack-infra | 02:55 | |
| *** e0ne has quit IRC | 03:00 | |
| *** jamesmcarthur has quit IRC | 03:26 | |
| *** slaweq has joined #openstack-infra | 03:31 | |
| *** hongbin has joined #openstack-infra | 03:35 | |
| *** slaweq has quit IRC | 03:35 | |
| *** psachin has joined #openstack-infra | 03:37 | |
| *** ykarel has quit IRC | 03:39 | |
| *** exsdev has quit IRC | 03:44 | |
| *** ykarel has joined #openstack-infra | 03:57 | |
| *** exsdev has joined #openstack-infra | 04:03 | |
| *** dave-mccowan has quit IRC | 04:09 | |
| *** FlorianFa has quit IRC | 04:18 | |
| *** hongbin has quit IRC | 04:22 | |
| *** FlorianFa has joined #openstack-infra | 04:26 | |
| *** rh-jelabarre has quit IRC | 04:27 | |
| *** larainema has joined #openstack-infra | 04:30 | |
| *** ykarel has quit IRC | 04:31 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 04:32 |
| *** fresta has quit IRC | 04:42 | |
| *** raukadah is now known as chandankumar | 04:47 | |
| *** diablo_rojo has quit IRC | 04:48 | |
| *** surpatil has joined #openstack-infra | 04:49 | |
| *** hwoarang has quit IRC | 04:57 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:06 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:06 |
| *** ykarel has joined #openstack-infra | 05:06 | |
| *** hwoarang has joined #openstack-infra | 05:09 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:11 |
| *** ramishra has quit IRC | 05:12 | |
| *** surpatil has quit IRC | 05:14 | |
| *** surpatil has joined #openstack-infra | 05:14 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:14 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:22 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:22 |
| *** ramishra has joined #openstack-infra | 05:27 | |
| *** e0ne has joined #openstack-infra | 05:30 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:31 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:31 |
| *** udesale has joined #openstack-infra | 05:33 | |
| *** ykarel_ has joined #openstack-infra | 05:34 | |
| *** ykarel has quit IRC | 05:36 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 05:38 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 05:38 |
| *** udesale has quit IRC | 05:38 | |
| *** udesale has joined #openstack-infra | 05:39 | |
| *** markvoelker has joined #openstack-infra | 05:47 | |
| *** xarses_ has quit IRC | 05:53 | |
| *** takamatsu has quit IRC | 05:54 | |
| *** signed8bit has quit IRC | 05:55 | |
| *** e0ne has quit IRC | 05:58 | |
| *** ccamacho has joined #openstack-infra | 05:59 | |
| *** sshnaidm_ has joined #openstack-infra | 05:59 | |
| *** sshnaidm|pto has quit IRC | 06:00 | |
| AJaeger | clarkb: I went now through all jobs on http://zuul.opendev.org/t/openstack/jobs and hope I found the last remaining ones, https://review.opendev.org/#/q/topic:goodbye_42.3+is:open shows the few left | 06:02 |
| AJaeger | jrosser: thanks for approving two of them! | 06:02 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:02 |
| jrosser | AJaeger: no prob :) | 06:03 |
| *** sshnaidm has joined #openstack-infra | 06:03 | |
| *** sshnaidm_ has quit IRC | 06:04 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 06:05 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:05 |
| AJaeger | jrosser: care to review https://review.opendev.org/688621 as well, please? | 06:07 |
| *** sshnaidm_ has joined #openstack-infra | 06:07 | |
| *** sshnaidm has quit IRC | 06:10 | |
| *** slaweq has joined #openstack-infra | 06:10 | |
| *** igordc has quit IRC | 06:10 | |
| *** ykarel__ has joined #openstack-infra | 06:14 | |
| *** ykarel_ has quit IRC | 06:17 | |
| *** exsdev has quit IRC | 06:19 | |
| *** yolanda__ has quit IRC | 06:20 | |
| *** yolanda has joined #openstack-infra | 06:20 | |
| *** ykarel__ is now known as ykarel | 06:28 | |
| *** pcaruana has joined #openstack-infra | 06:30 | |
| *** xenos76 has joined #openstack-infra | 06:31 | |
| *** vesper11- has quit IRC | 06:33 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 06:34 |
| *** vesper11 has joined #openstack-infra | 06:36 | |
| *** exsdev has joined #openstack-infra | 06:37 | |
| *** exsdev has quit IRC | 06:38 | |
| *** exsdev has joined #openstack-infra | 06:43 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 06:46 |
| *** iurygregory has joined #openstack-infra | 06:51 | |
| *** xarses has joined #openstack-infra | 06:52 | |
| *** dpawlik has joined #openstack-infra | 06:52 | |
| *** trident has quit IRC | 06:52 | |
| *** exsdev has quit IRC | 06:54 | |
| *** exsdev has joined #openstack-infra | 06:56 | |
| *** trident has joined #openstack-infra | 06:56 | |
| *** jpena|off is now known as jpena | 07:00 | |
| *** pgaxatte has joined #openstack-infra | 07:05 | |
| *** tesseract has joined #openstack-infra | 07:12 | |
| *** udesale has quit IRC | 07:13 | |
| *** udesale has joined #openstack-infra | 07:13 | |
| *** pkopec has joined #openstack-infra | 07:14 | |
| *** rcernin has quit IRC | 07:16 | |
| *** tosky has joined #openstack-infra | 07:19 | |
| *** e0ne has joined #openstack-infra | 07:25 | |
| *** tkajinam has quit IRC | 07:25 | |
| *** apetrich has joined #openstack-infra | 07:26 | |
| *** tkajinam has joined #openstack-infra | 07:26 | |
| *** ralonsoh has joined #openstack-infra | 07:28 | |
| *** ociuhandu has joined #openstack-infra | 07:30 | |
| jrosser | AJaeger: this will need to be force merged I think https://review.opendev.org/#/c/688620/ | 07:30 |
| *** pgaxatte has quit IRC | 07:32 | |
| *** ociuhandu has quit IRC | 07:35 | |
| *** pgaxatte has joined #openstack-infra | 07:35 | |
| *** ykarel is now known as ykarel|lunch | 07:35 | |
| *** ociuhandu has joined #openstack-infra | 07:35 | |
| *** kopecmartin|off is now known as kopecmartin | 07:38 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: config-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:38 |
| *** pgaxatte has quit IRC | 07:40 | |
| AJaeger | jrosser: yeah ;( Will work with clarkb later on it. thanks | 07:40 |
| *** ociuhandu has quit IRC | 07:42 | |
| *** dpawlik has quit IRC | 07:43 | |
| *** ramishra has quit IRC | 07:47 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: fix mirror_fqdn variable https://review.opendev.org/688468 | 07:49 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:50 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 07:51 |
| *** priteau has joined #openstack-infra | 07:52 | |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 07:53 |
| *** dtantsur|afk is now known as dtantsur | 07:53 | |
| *** e0ne has quit IRC | 07:55 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch https://review.opendev.org/644927 | 07:58 |
| *** tkajinam has quit IRC | 08:02 | |
| *** kjackal has quit IRC | 08:03 | |
| *** kjackal has joined #openstack-infra | 08:04 | |
| *** rpittau|afk is now known as rpittau | 08:07 | |
| *** ramishra has joined #openstack-infra | 08:08 | |
| *** ociuhandu has joined #openstack-infra | 08:13 | |
| *** dchen has quit IRC | 08:15 | |
| *** dpawlik has joined #openstack-infra | 08:15 | |
| *** e0ne has joined #openstack-infra | 08:17 | |
| *** dpawlik has quit IRC | 08:20 | |
| *** ociuhandu has quit IRC | 08:20 | |
| *** lucasagomes has joined #openstack-infra | 08:24 | |
| *** ociuhandu has joined #openstack-infra | 08:25 | |
| *** e0ne_ has joined #openstack-infra | 08:25 | |
| *** takamatsu has joined #openstack-infra | 08:26 | |
| *** e0ne has quit IRC | 08:26 | |
| *** gfidente has joined #openstack-infra | 08:29 | |
| *** takamatsu has quit IRC | 08:29 | |
| *** ykarel|lunch is now known as ykarel | 08:29 | |
| *** jamesmcarthur has joined #openstack-infra | 08:30 | |
| *** ociuhandu has quit IRC | 08:31 | |
| *** ociuhandu has joined #openstack-infra | 08:31 | |
| *** jamesmcarthur has quit IRC | 08:34 | |
| *** ociuhandu has quit IRC | 08:35 | |
| *** ociuhandu has joined #openstack-infra | 08:35 | |
| *** takamatsu has joined #openstack-infra | 08:36 | |
| *** kjackal_v2 has joined #openstack-infra | 08:39 | |
| *** kjackal has quit IRC | 08:41 | |
| *** takamatsu has quit IRC | 08:41 | |
| *** ociuhandu has quit IRC | 08:42 | |
| openstackgerrit | Jan Kubovy proposed opendev/gear master: Add BSD/Darwin support. https://review.opendev.org/671674 | 08:45 |
| *** prometheanfire has quit IRC | 08:45 | |
| *** prometheanfire has joined #openstack-infra | 08:46 | |
| *** dpawlik has joined #openstack-infra | 08:47 | |
| *** takamatsu has joined #openstack-infra | 08:48 | |
| *** lxkong has quit IRC | 08:49 | |
| *** dpawlik has quit IRC | 08:52 | |
| *** bnemec has quit IRC | 08:53 | |
| *** xarses has quit IRC | 09:06 | |
| *** xarses has joined #openstack-infra | 09:06 | |
| *** e0ne_ has quit IRC | 09:09 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add nodesets API route https://review.opendev.org/605877 | 09:11 |
| *** fresta has joined #openstack-infra | 09:22 | |
| *** sshnaidm_ is now known as sshnaidm|pto | 09:27 | |
| AJaeger | mordred: you also need a change for governance repo for puppet-yum retirement, rest looks great | 09:39 |
| *** derekh has joined #openstack-infra | 09:43 | |
| openstackgerrit | Andreas Jaeger proposed openstack/project-config master: Remove legacy-puppet4-centos-7 https://review.opendev.org/688664 | 09:45 |
| openstackgerrit | Merged openstack/project-config master: Stop running centos-7 puppet jobs https://review.opendev.org/688560 | 09:47 |
| *** ociuhandu has joined #openstack-infra | 09:47 | |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove legacy-puppet-syntax-centos job https://review.opendev.org/688666 | 09:48 |
| openstackgerrit | Merged openstack/project-config master: Retire puppet-yum - remove gate jobs https://review.opendev.org/688563 | 09:49 |
| openstackgerrit | Merged opendev/puppet-yum master: Retire puppet-yum https://review.opendev.org/688565 | 09:52 |
| *** ociuhandu has quit IRC | 09:53 | |
| ykarel | Is there some issue related to rax centos mirror? 3 jobs in one of my patch failed with [Errno 256] No more mirrors to try. | 10:00 |
| ykarel | https://review.opendev.org/#/c/688035/ don't want to recheck if the issue is still going on | 10:00 |
| ykarel | passed job ran on ovh | 10:00 |
| *** dpawlik has joined #openstack-infra | 10:02 | |
| *** dpawlik has quit IRC | 10:06 | |
| *** e0ne has joined #openstack-infra | 10:20 | |
| *** ociuhandu has joined #openstack-infra | 10:31 | |
| *** markvoelker has quit IRC | 10:33 | |
| *** ociuhandu has quit IRC | 10:36 | |
| *** dpawlik has joined #openstack-infra | 10:42 | |
| *** dpawlik has quit IRC | 10:46 | |
| *** takamatsu has quit IRC | 10:56 | |
| *** pkopec has quit IRC | 10:58 | |
| *** pkopec has joined #openstack-infra | 11:10 | |
| *** factor has joined #openstack-infra | 11:15 | |
| *** udesale has quit IRC | 11:16 | |
| *** markvoelker has joined #openstack-infra | 11:19 | |
| *** raschid has joined #openstack-infra | 11:29 | |
| *** takamatsu has joined #openstack-infra | 11:34 | |
| *** dpawlik has joined #openstack-infra | 11:35 | |
| *** markvoelker has quit IRC | 11:39 | |
| *** rosmaita has quit IRC | 11:39 | |
| *** rosmaita has joined #openstack-infra | 11:40 | |
| *** jpena is now known as jpena|lunch | 11:42 | |
| *** ccamacho has quit IRC | 11:46 | |
| *** takamatsu has quit IRC | 11:51 | |
| *** dpawlik has quit IRC | 11:52 | |
| mnasiadka | yoctozepto: do we have some problem with bindep, or I'm just special? https://review.opendev.org/#/c/688636/2 | 11:58 |
| *** goldyfruit has quit IRC | 11:58 | |
| yoctozepto | mnasiadka: how would I know? everyone is special | 12:00 |
| mnasiadka | yoctozepto: for me it looks like on k-a jobs libffi-devel doesn't get installed | 12:00 |
| yoctozepto | Failed to build cffi | 12:01 |
| yoctozepto | fatal error: ffi.h: No such file or directory | 12:01 |
| yoctozepto | indeed | 12:01 |
| yoctozepto | wonder if it's related to that ceph funkiness | 12:02 |
| yoctozepto | now that I look at it | 12:02 |
| yoctozepto | maybe they changed packaging | 12:02 |
| yoctozepto | related to python | 12:02 |
| yoctozepto | having suddenly python2 and libffi-devel gone | 12:02 |
| yoctozepto | mgoddard | 12:03 |
| mnasiadka | yoctozepto: I guess everything will start failing now :) | 12:03 |
| yoctozepto | mnasiadka: world will crumble | 12:03 |
| mgoddard | yoctozepto: o/ | 12:08 |
| mgoddard | do we actually execute the bindep zuul ansible role? | 12:09 |
| yoctozepto | mgoddard: no sign | 12:09 |
| yoctozepto | in kolla-ansible | 12:10 |
| yoctozepto | but yes in kolla | 12:10 |
| mgoddard | e.g. https://opendev.org/openstack/tripleo-ci/src/branch/master/playbooks/tripleo-buildcontainers/pre.yaml | 12:10 |
| mnasiadka | so how did it work previously? :) | 12:10 |
| *** gridinv has joined #openstack-infra | 12:10 | |
| mgoddard | blind luck | 12:11 |
| yoctozepto | ^ | 12:11 |
| mgoddard | change in the nodepool image? | 12:11 |
| yoctozepto | just like with change in ubuntu container image | 12:11 |
| yoctozepto | anyone proposing it? | 12:11 |
| mnasiadka | let me add it | 12:11 |
| openstackgerrit | Mohammed Naser proposed opendev/gear master: Added Docker image builds https://review.opendev.org/688446 | 12:11 |
| mgoddard | mnasiadka: need to fix yamllint at the same time | 12:12 |
| mgoddard | I'll chuck it in with my fix | 12:12 |
| yoctozepto | mgoddard: rightyo | 12:13 |
| mnasiadka | ok, whatever is wrong with yamllint :) | 12:13 |
| yoctozepto | apocalypse :D | 12:13 |
| yoctozepto | mgoddard: though you noticed yamllint is still broken | 12:13 |
| yoctozepto | (maybe back to #..kolla?) | 12:13 |
| mgoddard | yoctozepto: I did. Hadn't realised warn=err | 12:13 |
| *** takamatsu has joined #openstack-infra | 12:14 | |
| *** takamatsu has quit IRC | 12:16 | |
| *** rh-jelabarre has joined #openstack-infra | 12:18 | |
| *** larainema has quit IRC | 12:18 | |
| *** dpawlik has joined #openstack-infra | 12:21 | |
| *** takamatsu has joined #openstack-infra | 12:21 | |
| *** paladox has quit IRC | 12:23 | |
| *** paladox has joined #openstack-infra | 12:24 | |
| *** dpawlik has quit IRC | 12:26 | |
| openstackgerrit | Eyal proposed openstack/devstack-gate master: Work around cffi install error https://review.opendev.org/688696 | 12:28 |
| *** takamatsu has quit IRC | 12:29 | |
| *** takamatsu has joined #openstack-infra | 12:32 | |
| *** ykarel is now known as ykarel|afk | 12:32 | |
| *** hamzy has quit IRC | 12:34 | |
| *** xek has joined #openstack-infra | 12:36 | |
| *** hamzy has joined #openstack-infra | 12:37 | |
| *** markvoelker has joined #openstack-infra | 12:38 | |
| *** jpena|lunch is now known as jpena | 12:41 | |
| *** xek has quit IRC | 12:51 | |
| *** xek has joined #openstack-infra | 12:52 | |
| *** xek has quit IRC | 12:53 | |
| *** xek has joined #openstack-infra | 12:54 | |
| *** markvoelker has quit IRC | 12:57 | |
| *** dpawlik has joined #openstack-infra | 12:59 | |
| *** panda|off is now known as panda | 13:01 | |
| *** xek has quit IRC | 13:01 | |
| *** mriedem has joined #openstack-infra | 13:01 | |
| *** xek has joined #openstack-infra | 13:02 | |
| *** dpawlik has quit IRC | 13:04 | |
| *** ykarel|afk is now known as ykarel|away | 13:04 | |
| *** takamatsu has quit IRC | 13:06 | |
| *** takamatsu has joined #openstack-infra | 13:10 | |
| *** ykarel|away has quit IRC | 13:10 | |
| *** liuyulong has joined #openstack-infra | 13:16 | |
| openstackgerrit | Merged zuul/nodepool master: Validate openstack provider pool labels have top-level labels https://review.opendev.org/686514 | 13:16 |
| *** lpetrut has joined #openstack-infra | 13:21 | |
| *** goldyfruit has joined #openstack-infra | 13:22 | |
| *** goldyfruit has quit IRC | 13:24 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add the process environment to zuul.conf parser https://review.opendev.org/612824 | 13:28 |
| *** goldyfruit has joined #openstack-infra | 13:32 | |
| *** pcaruana has quit IRC | 13:32 | |
| *** sreejithp has joined #openstack-infra | 13:33 | |
| *** dpawlik has joined #openstack-infra | 13:34 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add the process environment to zuul.conf parser https://review.opendev.org/612824 | 13:35 |
| *** takamatsu has quit IRC | 13:36 | |
| *** sreejithp has quit IRC | 13:38 | |
| *** sreejithp_ has joined #openstack-infra | 13:38 | |
| *** dpawlik has quit IRC | 13:39 | |
| *** spsurya has joined #openstack-infra | 13:42 | |
| *** bnemec has joined #openstack-infra | 13:43 | |
| *** KeithMnemonic has joined #openstack-infra | 13:43 | |
| *** Goneri has joined #openstack-infra | 13:44 | |
| *** ykarel has joined #openstack-infra | 13:46 | |
| *** jamesdenton has quit IRC | 13:46 | |
| openstackgerrit | Douglas Mendizábal proposed openstack/project-config master: Add openstack/barbican-ui https://review.opendev.org/688714 | 13:53 |
| *** sreejithp_ has quit IRC | 13:54 | |
| *** surpatil has quit IRC | 13:55 | |
| *** jamesmcarthur has joined #openstack-infra | 13:55 | |
| *** xek_ has joined #openstack-infra | 13:55 | |
| *** iurygregory has quit IRC | 13:56 | |
| *** xek has quit IRC | 13:58 | |
| *** ociuhandu has joined #openstack-infra | 13:58 | |
| *** sreejithp has joined #openstack-infra | 14:01 | |
| *** pcaruana has joined #openstack-infra | 14:04 | |
| *** michael-beaver has joined #openstack-infra | 14:04 | |
| *** gridinv has quit IRC | 14:05 | |
| *** jamesmcarthur has quit IRC | 14:09 | |
| openstackgerrit | Slawek Kaplonski proposed ttygroup/gertty master: Update information about config file location https://review.opendev.org/655338 | 14:13 |
| *** dklyle has quit IRC | 14:18 | |
| *** iurygregory has joined #openstack-infra | 14:19 | |
| *** ykarel is now known as ykarel|away | 14:20 | |
| *** markvoelker has joined #openstack-infra | 14:23 | |
| *** electrofelix has joined #openstack-infra | 14:25 | |
| *** jamesmcarthur has joined #openstack-infra | 14:27 | |
| *** ociuhandu has quit IRC | 14:33 | |
| *** dtantsur is now known as dtantsur|brb | 14:40 | |
| *** dklyle has joined #openstack-infra | 14:43 | |
| clarkb | ykarel|away: can you link to specific jobs instead of changes? | 14:44 |
| clarkb | ykarel|away: I've pulled up one of the failing centos7 jobs but it looks like selinux problems not mirror issues | 14:44 |
| *** pkopec has quit IRC | 14:45 | |
| ykarel|away | clarkb, sure, /me get's link | 14:45 |
| AJaeger | clarkb: good morning, https://review.opendev.org/#/q/status:open+topic:goodbye_42.3 are the final 3 changes that I'm aware off - and the devstack failed building wheels for cffi, something looks broken with libffi (see the kolla discusssion above), so a recheck should heal it once cffi/libffi is fixed, so only two need force merge. Could you do so sometime today, please? | 14:47 |
| clarkb | AJaeger: yup let me see | 14:48 |
| ykarel|away | clarkb, https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/zuul_opendev_logs_ec7/688035/1/gate/puppet-openstack-integration-5-scenario001-tempest-centos-7-luminous/ec7fef5/logs/puppet.txt.gz | 14:48 |
| ykarel|away | No more mirrors to try. | 14:48 |
| openstackgerrit | Merged zuul/zuul-jobs master: configure-mirrors: make separate template directories for each platform https://review.opendev.org/688616 | 14:48 |
| ykarel|away | also i rechecked the patch and this time it passed, i didn't checked in which infra job landed | 14:48 |
| clarkb | AJaeger: the osa and ara changes will be force merged and devstack should be rechecked if I am reading that correctly? | 14:49 |
| clarkb | dmsimard: fyi we'll force merge https://review.opendev.org/688618 into ara shortly to remove testing for opensuse 42.3 | 14:49 |
| *** eharney has joined #openstack-infra | 14:49 | |
| AJaeger | clarkb: yes, that was the idea. Looking at devstack, I wouldn't mind force merging instead of rechecking - but can wait... | 14:49 |
| clarkb | ykarel|away: do we know what urls it is trying to fetch there? It looks like those could be rdo proxy packages (eg packages we don't mirror but instead cache and if the backend for those is down then it will fail) | 14:51 |
| *** armax has joined #openstack-infra | 14:51 | |
| clarkb | ykarel|away: but without the actual url it is hard to say | 14:51 |
| *** gagehugo has joined #openstack-infra | 14:55 | |
| clarkb | yoctozepto: mgoddard what is the libffi/cffi story? Looks like it is affecting more than just kolla | 14:57 |
| mgoddard | clarkb: not certain, but looks like a dependency on libffi-dev[el] was removed from the images, and anyone not installing them gets hurt | 14:58 |
| mgoddard | clarkb: if projects have libffi-dev[el] in bindep, and use the bindep role they should be fine | 14:59 |
| mgoddard | I noticed ironic inspector hit the issue | 14:59 |
| AJaeger | also showing up in https://review.opendev.org/#/c/688619/ - see https://zuul.opendev.org/t/openstack/build/29e5ad77aaa148e78b58d6671cad038e/log/job-output.txt | 14:59 |
| AJaeger | or cffi has no wheels build... | 14:59 |
| AJaeger | when do we build wheels? Rebuilding our own wheels might work around it, won't it? | 15:00 |
| clarkb | our wheels happen every 4 hours iirc | 15:00 |
| clarkb | cffi 1.13.0 has wheels as of 3 hours ago looks like | 15:01 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch https://review.opendev.org/644927 | 15:03 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for test_minimum_basic_scenario bug 1838202 https://review.opendev.org/688730 | 15:07 |
| *** ociuhandu has joined #openstack-infra | 15:07 | |
| openstack | bug 1838202 in tempest "TestMinimumBasicScenario.test_minimum_basic_scenario race fail comparing volume to expected values with updated_at diff" [Undecided,Confirmed] https://launchpad.net/bugs/1838202 | 15:07 |
| *** ociuhandu has quit IRC | 15:10 | |
| *** ociuhandu has joined #openstack-infra | 15:11 | |
| *** gyee has joined #openstack-infra | 15:13 | |
| clarkb | AJaeger: I have rechecked the devstack change after confirming it tried to use cffi 1.13.0 prior to the wheels being present according to pypi timestmaps | 15:17 |
| clarkb | mgoddard: yoctozepto fyi ^ the wheels seem to exist on pypi now | 15:17 |
| clarkb | and crisis might be averted by that | 15:17 |
| *** mriedem has quit IRC | 15:18 | |
| fungi | mgoddard: this tends to happen every time there is a new cffi release. we haven't (in modern history) shipped the dependencies necessary to build cffi sdists by default in our node images. when there are wheels of the latest cffi available jobs tend to install those and so don't need to build them. when there are only sdists of a new release suddenly everything breaks because jobs want to build the | 15:18 |
| fungi | new version from sdist | 15:18 |
| *** mriedem has joined #openstack-infra | 15:18 | |
| mgoddard | clarkb: thanks for the update. presumably we will not see this in future after adding libffi-dev* to bindep? | 15:19 |
| clarkb | ya if you have libffi-dev then cffi can be built from source | 15:19 |
| fungi | correct, that should solve it (assuming that's the only package it needs) | 15:19 |
| clarkb | your jobs might run a little longer as they will be compiling that library but otherwise that should make it happy always | 15:19 |
| fungi | one way to be sure is to have a job which sets pip options to never install wheels only sdists | 15:20 |
| clarkb | all four nodepool launchers have nodepool==3.8.1.dev36 # git sha 5a69b6a installed | 15:24 |
| *** sreejithp has quit IRC | 15:24 | |
| openstackgerrit | Merged zuul/nodepool master: Fix builder shutdown race in tests https://review.opendev.org/687965 | 15:24 |
| *** sreejithp has joined #openstack-infra | 15:24 | |
| clarkb | I'm going to restart nl02 first since it hosts fortnebula and the DELETED state handling is thought to help that cloud in particular | 15:24 |
| clarkb | I'll restart that one now and let it run for a bit then do the other three if things look happy | 15:24 |
| yoctozepto | clarkb: thanks! | 15:25 |
| *** markvoelker has quit IRC | 15:28 | |
| *** diablo_rojo has joined #openstack-infra | 15:29 | |
| clarkb | we've successfully launched at least 2 nodes in fn since the nl02 launcher restart | 15:30 |
| clarkb | I'll give it a few minutes more but so far things look happy | 15:30 |
| clarkb | donnyd: ^ fyi this is the thing we think might address the long term deleting state for nodes | 15:31 |
| donnyd | I haven't looked at the logs for the node state in a while, but it looks like yesterday it happened again for most of the day | 15:32 |
| donnyd | hopefully that resolves it | 15:32 |
| donnyd | its weird though that it doesn't happen all the time | 15:32 |
| donnyd | http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=now-6h&to=now | 15:32 |
| donnyd | looking here its been fine most of the day | 15:32 |
| donnyd | but here http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=1571102554425&to=1571131321530 | 15:33 |
| donnyd | Need to compare actual FN state with what nodepool was reporting | 15:33 |
| clarkb | we should know if a few days if it doesn't happen anymore based on that rate of occurence | 15:33 |
| clarkb | as why why it happens if it is the DELETED status issue we are fixing then it will depend on why nova isn't cleaning up those instance records quickly (perhaps it is busy with other background tasks at that time) | 15:33 |
| *** pgaxatte has joined #openstack-infra | 15:36 | |
| clarkb | alright not really seeing any problems. I'll go ahead with the others now | 15:37 |
| *** igordc has joined #openstack-infra | 15:37 | |
| fungi | though i suppose it could lead to "no host available" or similar errors if nova doesn't deprovision them quickly enough and gets overrun | 15:37 |
| fungi | so we may want to keep a closer eye on the launch errors graph | 15:37 |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for test_minimum_basic_scenario bug 1838202 https://review.opendev.org/688730 | 15:37 |
| openstack | bug 1838202 in tempest "TestMinimumBasicScenario.test_minimum_basic_scenario race fail comparing volume to expected values with updated_at diff" [Undecided,Confirmed] https://launchpad.net/bugs/1838202 | 15:37 |
| *** pmatulis1 has joined #openstack-infra | 15:38 | |
| pmatulis1 | hi. i'm trying to figure out why i'm not getting gerrit mail for microstack since this one: https://review.opendev.org/#/c/675390/ | 15:38 |
| donnyd | fungi: I do that that job that runs every 2 minutes and check the state of that tenant | 15:38 |
| *** mattw4 has joined #openstack-infra | 15:38 | |
| fungi | good news and bad news on the debian mirror rebuild front... the bad news is that it's still running. the good news is that it logged noticing i had removed the buster packages for base-files and replaced them, so it doesn't seem to be replacing any packages which were already present in the mirror | 15:39 |
| clarkb | hrm vexxhost sjc1 might not appreciate that restart | 15:39 |
| clarkb | Block Device Mapping is Invalid: failed to get image 74c266ae-e8bd-442e-8b48-2931a6edb9d2 | 15:40 |
| clarkb | not sure yet if it was doing that pre restart | 15:40 |
| clarkb | Error finding address for https://image-sjc1.vexxhost.us/v2/schemas/image: ('Connection aborted.', RemoteDisconnected('Remote end closed connection without response',)) | 15:41 |
| clarkb | that is from osc on bridge | 15:41 |
| clarkb | mnaser: ^ just in case this isn't user error you may want to know about that | 15:41 |
| donnyd | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/fortnebula-infra-logs/20191014-082001-openstack-server-stat.json | 15:42 |
| fungi | clarkb: looks like it was doing it before the restart too | 15:42 |
| clarkb | fungi: cool thanks for checking | 15:42 |
| donnyd | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/fortnebula-infra-logs/20191014-081801-openstack-server-stat.json | 15:42 |
| fungi | hunting for the earliest occurrence now | 15:42 |
| donnyd | https://openstack.fortnebula.com:13808/v1/AUTH_e8fd161dc34c421a979a9e6421f823e9/fortnebula-infra-logs/20191014-081601-openstack-server-stat.json | 15:42 |
| clarkb | pmatulis1: we can take a look when we complete this nodepool maintenance | 15:43 |
| donnyd | So those three logs center around this occurrence http://grafana.openstack.org/d/3Bwpi5SZk/nodepool-fortnebula?orgId=1&from=1571040723883&to=1571043437761 | 15:43 |
| clarkb | fungi: should I proceed to nl04 and nl01 then? | 15:43 |
| clarkb | donnyd: no DELETED states there | 15:43 |
| mnaser | clarkb: thats a weird error | 15:43 |
| pmatulis1 | clarkb, thank you | 15:44 |
| *** jamesdenton has joined #openstack-infra | 15:44 | |
| donnyd | right.. it usually only takes 30(+-10) seconds according to my logs here | 15:44 |
| donnyd | But i need to do some diffs on those files to see if the instances that were called to be deleted were actually being deleted | 15:45 |
| *** jpena is now known as jpena|brb | 15:45 | |
| fungi | seems it started erroring at 14:37:47z | 15:45 |
| clarkb | I'm going to proceed to nl04 since the issue we've identified was happening prior to the restarts | 15:45 |
| donnyd | and we need to look at the nodepool logs to compare the instance id's for instances which delete was called for | 15:45 |
| fungi | i don't see earlier errors for other uuids, so going to see if that roughly corresponds to when it was uploaded | 15:46 |
| openstackgerrit | Merged zuul/zuul-jobs master: configure-mirrors: add CentOS 8 https://review.opendev.org/688617 | 15:46 |
| clarkb | fungi: well I can't talk to the api at all right now | 15:46 |
| *** xek__ has joined #openstack-infra | 15:46 | |
| fungi | oh, that's "neat" | 15:46 |
| clarkb | fungi: I'm thinking it is related to a specific instance that needs to be deleted and that is why we hit only the one image uuid | 15:46 |
| clarkb | or something along those lines | 15:46 |
| *** xek_ has quit IRC | 15:48 | |
| clarkb | #status log Restarted nl01 and nl04 on nodepool==3.8.1.dev38 # git sha 0a010d9 and nl02 and nl03 on nodepool==3.8.1.dev36 # git sha 5a69b6a. The only difference between these commits is a test only change. | 15:49 |
| openstackstatus | clarkb: finished logging | 15:49 |
| clarkb | rax ord has some quota sync issues but those are "normal" | 15:49 |
| clarkb | otherwise I think things are happy from what I see (and the vexxhost api weirdness) | 15:50 |
| *** lucasagomes has quit IRC | 15:50 | |
| *** liuyulong has quit IRC | 15:51 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:51 |
| clarkb | pmatulis1: "250 2.0.0 Ok: queued as $ID" is what your MX seems to report back to us | 15:54 |
| clarkb | pmatulis1: my read of gerrit's exim logs is that gerrit has successfully managed to deliver emails to your MX via the exim running on our gerrit | 15:55 |
| clarkb | I can share delivery ids I don't think those need to be secret (but fungi and corvus can confirm) | 15:56 |
| fungi | they are not secret | 15:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:57 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 15:57 |
| fungi | well, that is to say, they're not generally known, but i can't imagine any circumstance where publishing a queue id string for an mta would be a security risk | 15:57 |
| corvus | agreed | 15:57 |
| AJaeger | clarkb: thanks for looking at cffi | 15:57 |
| fungi | i mean, it's reported from their mta to ours, an untrusted relationship | 15:57 |
| AJaeger | and for the merges | 15:57 |
| clarkb | pmatulis1: 2019-10-15 15:51:29 $STUFF C="250 2.0.0 Ok: queued as 44C5E1A3D22" | 15:58 |
| clarkb | pmatulis1: that timestamp is in UTC | 15:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 15:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 15:58 |
| *** xenos76 has quit IRC | 15:59 | |
| fungi | so ~8.5 minutes ago | 16:00 |
| clarkb | looking at grafana graphs the vexxhost issues seem to ahve started around 14:36UTC | 16:02 |
| clarkb | about an hour before my restart | 16:02 |
| fungi | yep, that roughly coincides with the first incidence of that error i found | 16:02 |
| openstackgerrit | Matt Riedemann proposed opendev/elastic-recheck master: Add query for bug 1836754 https://review.opendev.org/671051 | 16:03 |
| openstack | bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Medium,Confirmed] https://launchpad.net/bugs/1836754 | 16:03 |
| fungi | er, first incidence i found of that error | 16:03 |
| * fungi shakes fist at misplaced modifier | 16:03 | |
| *** rpittau is now known as rpittau|afk | 16:03 | |
| *** xek__ has quit IRC | 16:05 | |
| *** xek__ has joined #openstack-infra | 16:05 | |
| *** xek_ has joined #openstack-infra | 16:09 | |
| *** hemna_afk is now known as hemna_ | 16:11 | |
| *** kopecmartin is now known as kopecmartin|off | 16:11 | |
| *** xek__ has quit IRC | 16:12 | |
| *** dtantsur|brb is now known as dtantsur | 16:13 | |
| mnaser | clarkb, fungi: yeah, systemd is fun. | 16:14 |
| mnaser | it's nice when it works and it's not nice when it locks you out of machines because systemd-logind is having a field day | 16:15 |
| mnaser | and most other services are still functional and running | 16:15 |
| *** e0ne has quit IRC | 16:16 | |
| *** jpena|brb is now known as jpena | 16:18 | |
| *** sgw has joined #openstack-infra | 16:21 | |
| *** pmatulis1 has left #openstack-infra | 16:23 | |
| openstackgerrit | Merged opendev/elastic-recheck master: Add query for bug 1836754 https://review.opendev.org/671051 | 16:26 |
| openstack | bug 1836754 in OpenStack Compute (nova) "Conflict when deleting allocations for an instance that hasn't finished building" [Low,Confirmed] https://launchpad.net/bugs/1836754 | 16:26 |
| fungi | mnaser: ouch, fun indeed | 16:30 |
| *** igordc has quit IRC | 16:31 | |
| clarkb | https://etherpad.openstack.org/p/rCF58JvzbF opendev governance email draft (it feels more like an outline at this point, I tried to bring up important items without pushing too much in any specific direction. Please jump in and add thoughts) | 16:33 |
| clarkb | now Iget an http 401 talking to vexxhost | 16:34 |
| clarkb | something has changed at least | 16:34 |
| *** dtantsur is now known as dtantsur|afk | 16:37 | |
| *** ociuhandu_ has joined #openstack-infra | 16:40 | |
| *** ociuhandu has quit IRC | 16:44 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 16:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add support for selecting the kubernetes version https://review.opendev.org/688743 | 16:46 |
| openstackgerrit | Gabor Lekeny proposed zuul/zuul master: Fix github auth token authentication https://review.opendev.org/686355 | 16:47 |
| *** derekh has quit IRC | 16:47 | |
| *** ociuhandu_ has quit IRC | 16:47 | |
| *** pgaxatte has quit IRC | 16:48 | |
| *** takamatsu has joined #openstack-infra | 16:49 | |
| *** xenos76 has joined #openstack-infra | 16:52 | |
| *** diablo_rojo has quit IRC | 16:55 | |
| *** diablo_rojo has joined #openstack-infra | 16:56 | |
| *** ociuhandu has joined #openstack-infra | 16:56 | |
| clarkb | I can list images in vexxhost now | 16:56 |
| mnaser | clarkb: yes its recovering | 16:58 |
| mnaser | and also in other newsd | 16:58 |
| mnaser | dockerhub is broken :( | 16:58 |
| mnaser | `docker pull busybox` => 503 and now 502 | 16:58 |
| clarkb | exciting | 16:58 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Mandatory SQL connection https://review.opendev.org/688755 | 16:59 |
| *** jpena is now known as jpena|off | 17:01 | |
| *** ociuhandu has quit IRC | 17:01 | |
| *** lpetrut has quit IRC | 17:01 | |
| *** gfidente is now known as gfidente|adfk | 17:01 | |
| *** gfidente|adfk is now known as gfidente|afk | 17:02 | |
| *** e0ne has joined #openstack-infra | 17:02 | |
| *** priteau has quit IRC | 17:03 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Mandatory SQL connection https://review.opendev.org/688755 | 17:04 |
| *** e0ne has quit IRC | 17:08 | |
| *** armax has quit IRC | 17:08 | |
| *** takamatsu has quit IRC | 17:12 | |
| *** weifan has joined #openstack-infra | 17:15 | |
| *** goldyfruit_ has joined #openstack-infra | 17:15 | |
| *** ykarel|away is now known as ykarel | 17:15 | |
| ykarel | clarkb, http://mirror.iad.rax.openstack.org:8080/rdo/centos7/7e/78/7e78ebdc0f3678afddea3bdab2007c7b6ac92776_2932899d/ repo is configured in that job | 17:15 |
| ykarel | openstack-neutron-ml2-15.1.0-0.20191014113745.7150014.el7.noarch: [Errno 256] No more mirrors to try | 17:16 |
| clarkb | ykarel: ya that is a proxy not a mirror | 17:17 |
| clarkb | ykarel: chances are the backend was having trouble at that time | 17:17 |
| AJaeger | clarkb: regarding the OpenDev etherpad: line 13 has "tevels" - do you mean "levels"? | 17:17 |
| clarkb | we do our best and will cache responses for up to 24 hours (if ttl isn't shorter) but when we have to hit the backend we are susceptible to failures | 17:18 |
| clarkb | AJaeger: yup! thanks | 17:18 |
| AJaeger | good | 17:18 |
| *** goldyfruit has quit IRC | 17:18 | |
| ykarel | clarkb, okk, Thanks for checking and confirming | 17:18 |
| ykarel | but /me noticed most of the time it's rax proxy | 17:19 |
| clarkb | ykarel: we run more jobs on rax than elsewhere | 17:19 |
| ykarel | clarkb, ohhk | 17:19 |
| clarkb | but also could be that the rdo distribution is less reliable for rax | 17:19 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: fix mirror_fqdn variable https://review.opendev.org/688468 | 17:19 |
| *** ricolin has quit IRC | 17:20 | |
| AJaeger | clarkb: why is membership in the council an even number (line 22), you list in line 12 only 3 members. | 17:21 |
| AJaeger | That makes it odd | 17:21 |
| clarkb | AJaeger: opendev ptl + openstack + kata + zuul | 17:22 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: configure-mirrors: Consolidate dnf/yum handler https://review.opendev.org/688479 | 17:22 |
| AJaeger | clarkb: then please update the text to reflect that, it's not obvious to me. | 17:22 |
| AJaeger | clarkb: ah, now I parse the sentence properly... | 17:22 |
| clarkb | I can try to reword it to be more clear | 17:23 |
| AJaeger | clarkb: Let me think myself | 17:23 |
| AJaeger | it's clear now, I stumbled earlier | 17:24 |
| clarkb | ok | 17:24 |
| * AJaeger did some minor edits | 17:24 | |
| clarkb | thank you for reviewing | 17:25 |
| * mordred added an s - otherwise looked good | 17:25 | |
| *** ralonsoh has quit IRC | 17:25 | |
| *** xek__ has joined #openstack-infra | 17:27 | |
| AJaeger | clarkb: for the council, I suggest that each governing body, like the OpenStack TC, nominates a member. | 17:27 |
| AJaeger | mordred: could you send an update for puppet-yum to openstack/goverance - as long as we're part of OpenStack, please? | 17:28 |
| mordred | AJaeger: oh - yeah - thanks! | 17:28 |
| AJaeger | clarkb: text is fine IMHO, if you want to add my proposal, go for it - or consider it something for later | 17:29 |
| clarkb | AJaeger: I added a line for council nominations | 17:29 |
| clarkb | lines 23-24 | 17:29 |
| AJaeger | clarkb: LGTM | 17:30 |
| mordred | AJaeger: does it want a depends-on the removal? | 17:30 |
| *** xek_ has quit IRC | 17:30 | |
| AJaeger | mordred: no need, I'll +2A your change once I see the governance change ;) | 17:30 |
| *** ociuhandu has joined #openstack-infra | 17:31 | |
| mordred | AJaeger: https://review.opendev.org/688761 Remove puppet-yum from Infra | 17:31 |
| AJaeger | mordred: you need to move the entry to the legacy file | 17:31 |
| mnaser | https://status.docker.com/pages/533c6539221ae15e3f000031 | 17:31 |
| mnaser | its an actual ~incident~ now | 17:32 |
| mnaser | just fyi in case people mention breakages | 17:32 |
| mordred | AJaeger: done | 17:32 |
| *** electrofelix has quit IRC | 17:32 | |
| mordred | AJaeger: thanks | 17:32 |
| AJaeger | mordred: LGTM, +2A on the project-config change | 17:33 |
| mordred | \o/ | 17:33 |
| *** spsurya has quit IRC | 17:33 | |
| AJaeger | any config-core to review https://review.opendev.org/688664 so that we can remove those jobs in https://review.opendev.org/688666, please? | 17:34 |
| *** ociuhandu has quit IRC | 17:35 | |
| *** goldyfruit___ has joined #openstack-infra | 17:36 | |
| *** psachin has quit IRC | 17:38 | |
| *** goldyfruit_ has quit IRC | 17:39 | |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-mediawiki master: Move cache directory out of mediawiki Git worktree https://review.opendev.org/688762 | 17:41 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-mediawiki master: Make sure the images tree is owned by www-data https://review.opendev.org/688763 | 17:41 |
| openstackgerrit | Jeremy Stanley proposed opendev/puppet-mediawiki master: Fetch external dependencies https://review.opendev.org/688764 | 17:41 |
| clarkb | I'm cleaning up leaked images in vexxhost now (noticed them when poking at why things weren't happy there) | 17:42 |
| *** e0ne has joined #openstack-infra | 17:42 | |
| clarkb | AJaeger: is fuel actually inactive? | 17:43 |
| *** eharney has quit IRC | 17:43 | |
| openstackgerrit | Merged openstack/project-config master: Finish retiring puppet-yum https://review.opendev.org/688566 | 17:43 |
| AJaeger | clarkb: nothing merged in the last two years ;( | 17:44 |
| fungi | until... now! | 17:44 |
| *** armax has joined #openstack-infra | 17:44 | |
| fungi | ;) | 17:44 |
| AJaeger | https://review.opendev.org/#/q/project:openstack/fuel-library+is:merged | 17:44 |
| clarkb | fair neough | 17:44 |
| AJaeger | clarkb: and we still leave jobs for them to merge anything if they want... | 17:45 |
| fungi | oh, i guess they're not defined in-repo | 17:45 |
| AJaeger | fungi: yes, they are in project-config, they never imported | 17:45 |
| AJaeger | that's why we can easily remove them... | 17:45 |
| AJaeger | I would remove them from infrastructure, since they just confuse people | 17:46 |
| AJaeger | but that's too much of a cleanup for now ;/ | 17:46 |
| *** e0ne has quit IRC | 17:47 | |
| AJaeger | fungi: see 688664 - that triggered clarkb's question | 17:48 |
| fungi | yep, it's the one i just approved | 17:48 |
| *** ociuhandu has joined #openstack-infra | 17:48 | |
| AJaeger | thanks! | 17:49 |
| mordred | AJaeger: I agree - removing them seems nice - but also non-urgent | 17:49 |
| *** ramishra has quit IRC | 17:50 | |
| fungi | ideally those repositories should be retired, yes. i even wrote this with fuel in mind: https://governance.openstack.org/tc/resolutions/20190711-mandatory-repository-retirement.html | 17:52 |
| fungi | since nobody asked to have them renamed out of the openstack namespace, the opendev hosting for their former deliverables falls under the jurisdiction of the openstack tc, who can now choose to take that action | 17:52 |
| fungi | that was basically openstack "serving notice" to teams maintaining former openstack deliverables | 17:53 |
| AJaeger | fungi: but isn't fuel not part of it - meaning, it got moved out of openstack TC before train? | 17:54 |
| fungi | that last paragraph could have been more clear | 17:55 |
| AJaeger | so, what was the intention: Allow TC to remove fuel? | 17:56 |
| *** ociuhandu has quit IRC | 17:56 | |
| AJaeger | (as side effect, not main driver) | 17:56 |
| fungi | the main thrust of the resolution was to deal with projects which want to rename out of the openstack namespace to instead fork | 17:56 |
| AJaeger | yes, got taht | 17:56 |
| fungi | since being in the openstack namespace now connotes openstack project officialness again | 17:57 |
| AJaeger | ++ | 17:57 |
| AJaeger | clarkb: btw want to move openstack/infra-manual to opendev? | 17:57 |
| fungi | the grace period for this cycle was to allow projects who want to switch away from the openstack namespace (for example, infra projects going into the opendev namespace) to do so | 17:58 |
| clarkb | AJaeger: I think that depends on whether or not we want to keep openstack specific documentation there (and can spin out more generic opendev docs into a different document) | 17:58 |
| clarkb | AJaeger: a lot of the user setup stuff is very openstack specific like signing the cla | 17:58 |
| AJaeger | clarkb: That is indeed an option | 17:58 |
| fungi | my take is that openstack-specific workflow stuff should move into openstack's contributor guide | 17:59 |
| AJaeger | still, I rather maintain one document with a couple of "This section applies to OpenStack" than two with duplication | 17:59 |
| AJaeger | fungi: so, back to fuel: Should we ask TC to fully retire fuel or is that not covered by the resolution (my interpretation)? | 18:00 |
| fungi | maintaining one guide with project-specific bits risks growing without bound as other projects hosted in opendev end up with their own project-specific requirements | 18:00 |
| fungi | AJaeger: i think it's worth reminding the tc that they have the ability to follow retirement process for any former deliverables which remain in the openstack namespace | 18:01 |
| fungi | fuel happens to represent a large set of those | 18:01 |
| openstackgerrit | Merged openstack/project-config master: Remove legacy-puppet4-centos-7 https://review.opendev.org/688664 | 18:01 |
| clarkb | fuel also represents a large chunk of our git cache iirc | 18:02 |
| *** raschid has quit IRC | 18:03 | |
| AJaeger | let me send an email to fuel and fuel-ccp cores and see whether I find anybody to drive the process... | 18:03 |
| openstackgerrit | Monty Taylor proposed opendev/puppet-openstack_infra_spec_helper master: Pin dry-inflector to <0.2 https://review.opendev.org/688555 | 18:04 |
| fungi | well, really anyone in the openstack community (with the blessing of the tc) can carry that out | 18:05 |
| *** goldyfruit_ has joined #openstack-infra | 18:05 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:05 |
| fungi | i have no problem updating the core review groups for those projects to add the openstack tc and then they can delegate change approval in them to whoever they like | 18:06 |
| *** diablo_rojo has quit IRC | 18:06 | |
| *** diablo_rojo has joined #openstack-infra | 18:06 | |
| *** markvoelker has joined #openstack-infra | 18:07 | |
| mnaser | are things healthy again on our side? grafana nodepool makes it look like it is ok | 18:07 |
| AJaeger | fungi: let me try those - and then try tc next. If those waste git cache, it's worth cleaning up... | 18:07 |
| fungi | yep, shrinking our images would be nice | 18:07 |
| *** openstackgerrit has quit IRC | 18:07 | |
| *** goldyfruit___ has quit IRC | 18:07 | |
| fungi | thanks AJaeger! | 18:08 |
| fungi | mnaser: yeah, looks like things are okay, but skimming our nodepool launcher logs is like sipping from a barrel | 18:09 |
| fungi | mnaser: thanks for getting it all back in shape! sorry you had to deal with that on release-eve | 18:09 |
| *** priteau has joined #openstack-infra | 18:10 | |
| * fungi half wonders if this is a hint that vexxhost is about to once again announce on release day that they're already running the new release | 18:10 | |
| mnaser | hah | 18:10 |
| mnaser | we've done a lot of infra improvements which seem to be helping in the inconsistent states we would get | 18:10 |
| mnaser | i.e. the cinder volume quota black magic | 18:11 |
| fungi | that'sexcellent | 18:11 |
| fungi | hopefully that means we'll be making more efficient use of our quota there | 18:11 |
| *** jamesmcarthur has quit IRC | 18:12 | |
| *** markvoelker has quit IRC | 18:12 | |
| clarkb | ya I've not seen any new issues | 18:13 |
| fungi | okay, more good news on the debian mirror front. the refresh finished just a bit ago: http://mirror.dfw.rax.openstack.org/debian/dists/ | 18:16 |
| fungi | it did indeed recreate indices | 18:16 |
| fungi | Package: base-files | 18:17 |
| fungi | Version: 10.3+deb10u1 | 18:17 |
| fungi | yasss | 18:17 |
| fungi | okay, so apparently the bludgeon of blowing away the reprepro database and letting it be recreates has at least gotten us a correct index now | 18:18 |
| fungi | s/recreates/recreated/ | 18:19 |
| fungi | next to see whether that fixes the debian-buster image builds | 18:19 |
| *** markvoelker has joined #openstack-infra | 18:21 | |
| fungi | release completed at 18:09, so the most recent build failure logged at https://nb01.openstack.org/debian-buster-0000038616.log was just a few minutes too early | 18:22 |
| fungi | fingers crossed the next one gets well past there, at least | 18:22 |
| *** kjackal has joined #openstack-infra | 18:23 | |
| *** chandankumar is now known as raukadah | 18:24 | |
| *** kjackal_v2 has quit IRC | 18:25 | |
| *** openstackgerrit has joined #openstack-infra | 18:25 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:25 |
| *** ociuhandu has joined #openstack-infra | 18:26 | |
| *** weifan has quit IRC | 18:28 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 18:30 |
| *** markvoelker has quit IRC | 18:31 | |
| *** eharney has joined #openstack-infra | 18:32 | |
| corvus | #status log really disabled zuul-registry prune cronjob on intermediate registry | 18:34 |
| openstackstatus | corvus: finished logging | 18:34 |
| corvus | mordred: ^ i missed that before, so our testing before now might be invalid | 18:34 |
| AJaeger | config-core, https://review.opendev.org/688666 is ready to remove the now unused puppet4 syntax jobs | 18:35 |
| AJaeger | please review | 18:35 |
| *** igordc has joined #openstack-infra | 18:36 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 18:36 |
| *** michael-beaver has quit IRC | 18:44 | |
| *** ykarel is now known as ykarel|away | 18:47 | |
| fungi | infra-root: we've received a notification from rackspace monitoring that our 10gb filesystem on the "zuul-v3" trove instance is down to 10% free. any objection to me... say... doubling it to 20gb? | 18:54 |
| mordred | fungi: can that be done with the db being up? | 18:55 |
| *** ociuhandu has quit IRC | 18:55 | |
| fungi | i believe so, yes | 18:55 |
| clarkb | and if not that might be a thursday or friday activity (I think reporters failing will fail jobs in zuul?) | 18:56 |
| *** iurygregory has quit IRC | 18:56 | |
| fungi | resizing ram or, amusingly, changing mysql configuration, requires a reboot | 18:56 |
| fungi | but increasing volume size does not | 18:56 |
| mordred | how weird | 18:56 |
| fungi | the power of lvm, i suppose | 18:57 |
| openstackgerrit | Andreas Jaeger proposed openstack/openstack-zuul-jobs master: Remove puppet-yum https://review.opendev.org/688785 | 18:57 |
| fungi | or more likely network filesystems | 18:57 |
| AJaeger | mordred, clarkb , any other fallout from puppet-yum retirement besides 688785? Please review | 18:57 |
| clarkb | codesearch isn't finding other zuul config with it | 18:59 |
| *** e0ne has joined #openstack-infra | 18:59 | |
| AJaeger | good | 18:59 |
| *** jamesmcarthur has joined #openstack-infra | 19:00 | |
| *** markvoelker has joined #openstack-infra | 19:01 | |
| *** weifan has joined #openstack-infra | 19:01 | |
| *** ykarel|away has quit IRC | 19:05 | |
| *** weifan has quit IRC | 19:06 | |
| *** markvoelker has quit IRC | 19:06 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:07 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Further split Debian and Ubuntu reprepro configs https://review.opendev.org/688788 | 19:11 |
| *** tbarron_ is now known as tbarron | 19:13 | |
| *** efried is now known as efried_afk | 19:15 | |
| openstackgerrit | Merged openstack/openstack-zuul-jobs master: Remove puppet-yum https://review.opendev.org/688785 | 19:16 |
| *** prometheanfire has quit IRC | 19:22 | |
| *** prometheanfire has joined #openstack-infra | 19:23 | |
| *** markvoelker has joined #openstack-infra | 19:25 | |
| *** trident has quit IRC | 19:29 | |
| *** efried_afk is now known as efried | 19:33 | |
| *** trident has joined #openstack-infra | 19:33 | |
| mordred | corvus, clarkb, fungi: if you get a sec, https://review.opendev.org/#/c/688555/ is ready | 19:36 |
| mordred | as ishttps://review.opendev.org/#/c/688523/2 | 19:37 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:37 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:39 |
| clarkb | mordred: I'm not sure I grok https://review.opendev.org/#/c/686868/1 we still use the bazel image to build gerrit right? so we'd want to run that one first? | 19:39 |
| *** weifan has joined #openstack-infra | 19:39 | |
| mordred | clarkb: it's promote - so it shouldn't be re-building - only fetching/pushing | 19:40 |
| *** factor has quit IRC | 19:43 | |
| openstackgerrit | Merged opendev/puppet-openstack_infra_spec_helper master: Pin dry-inflector to <0.2 https://review.opendev.org/688555 | 19:45 |
| *** weifan has quit IRC | 19:46 | |
| *** tesseract has quit IRC | 19:49 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 19:50 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 19:51 |
| *** e0ne has quit IRC | 19:55 | |
| openstackgerrit | Mohammed Naser proposed opendev/project-config master: Added {success,failure}-message for pipelines https://review.opendev.org/688801 | 19:55 |
| mnaser | clarkb, mordred ^ | 19:56 |
| openstackgerrit | Merged opendev/project-config master: Added {success,failure}-message for pipelines https://review.opendev.org/688801 | 19:57 |
| *** markvoelker has quit IRC | 20:00 | |
| *** eernst has joined #openstack-infra | 20:00 | |
| *** weifan has joined #openstack-infra | 20:04 | |
| *** eernst has quit IRC | 20:04 | |
| *** jamesmcarthur has quit IRC | 20:09 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 20:10 |
| mordred | corvus: can haz +A on https://review.opendev.org/#/c/688523 - clarkb: can haz +A on https://review.opendev.org/#/c/678875 ? (you've each +2'd one of them) | 20:11 |
| openstackgerrit | Ian Wienand proposed zuul/zuul master: Remove python-path auto release note https://review.opendev.org/688809 | 20:12 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add jobs to build gerrit master branch https://review.opendev.org/688482 | 20:13 |
| mordred | oh - duh - I wanted to do the opposite there | 20:13 |
| openstackgerrit | Merged opendev/system-config master: Remove dependencies from promote https://review.opendev.org/686868 | 20:15 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add jobs to build gerrit master branch https://review.opendev.org/688482 | 20:16 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Add checks plugin to gerrit master build https://review.opendev.org/688483 | 20:16 |
| *** jamesmcarthur has joined #openstack-infra | 20:19 | |
| fungi | okay, the debian-security mirror wasn't updating at all, reprepro was complaining that the keys listed in the updates file were insufficient to verify the release signatures. turns out we needed subkeys allowed there, according to some manual testing. i'll get a fix pushed up shortly for that | 20:19 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 20:21 |
| openstackgerrit | Jeremy Stanley proposed opendev/system-config master: Further split Debian and Ubuntu reprepro configs https://review.opendev.org/688788 | 20:22 |
| fungi | it was only a couple of extra characters, so i just incorporated it into those fixes | 20:22 |
| *** pcaruana has quit IRC | 20:25 | |
| fungi | though that's not actually why the image build is failing | 20:27 |
| fungi | it's looking in the wrong path for the security repository's dists | 20:27 |
| fungi | and it seems our debian-stretch image builds are failing exactly the same way | 20:27 |
| fungi | i wonder for how long | 20:27 |
| fungi | last successful build was 6 days 15.5 hours ago | 20:29 |
| clarkb | I'm going to try and get a bike ride in back after to review all the things | 20:29 |
| *** dpawlik has joined #openstack-infra | 20:29 | |
| fungi | oh, i bet that's when the dib fix merged for being able to reference our security mirror | 20:30 |
| fungi | rather than having images build directly against security.debian.org | 20:30 |
| *** kjackal has quit IRC | 20:30 | |
| fungi | so this one is probably my fault ;) | 20:30 |
| *** pcaruana has joined #openstack-infra | 20:34 | |
| *** slaweq has quit IRC | 20:38 | |
| *** jamesmcarthur has quit IRC | 20:38 | |
| *** ociuhandu has joined #openstack-infra | 20:46 | |
| *** jamesmcarthur has joined #openstack-infra | 20:48 | |
| *** weifan has quit IRC | 20:50 | |
| fungi | well, not directly my fault, but the debian-minimal element hard-codes a /updates in the path for the security mirror | 20:51 |
| fungi | our mirror doesn't have /updates in its path | 20:51 |
| *** pcaruana has quit IRC | 20:51 | |
| fungi | changing that string likely makes the element non-backwards-compatible, changing our mirror is likely non-backward-compatible for our existing debian-stretch jobs | 20:53 |
| fungi | so i think the only sane solution there is to add yet another variable in the element, and then wait for another dib release | 20:53 |
| *** weifan has joined #openstack-infra | 20:53 | |
| *** dpawlik has quit IRC | 20:55 | |
| openstackgerrit | Merged opendev/system-config master: Use bazel 1.0.0 https://review.opendev.org/688523 | 20:56 |
| *** xenos76 has quit IRC | 21:02 | |
| fungi | dib change is almost done... anybody know if i can just randomly generate the 16-character hex component of the release note filename? | 21:02 |
| fungi | i guess that's just in case the prefix isn't unique? | 21:03 |
| *** dpawlik has joined #openstack-infra | 21:06 | |
| *** sreejithp has quit IRC | 21:10 | |
| *** Goneri has quit IRC | 21:10 | |
| *** dpawlik has quit IRC | 21:11 | |
| clarkb | fungi: reno command does it for you | 21:11 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Pull minikube log data https://review.opendev.org/688775 | 21:12 |
| clarkb | infra-root I think https://review.opendev.org/#/c/688442/ can merge now, the devstack change has merged | 21:14 |
| fungi | clarkb: i didn't see a tox env for running reno to do that, but it's just yaml so easy enough to do by hand | 21:18 |
| clarkb | fungi: I have a reno install in my "global" venv for it | 21:19 |
| clarkb | not sure if it is typically run via a tox env | 21:19 |
| clarkb | mordred: can you see my comment on https://review.opendev.org/#/c/678875/13 | 21:21 |
| mordred | clarkb: nope. invisible ink | 21:22 |
| clarkb | I hear you spray the computer with citric acid then light it on fire to read it | 21:22 |
| openstackgerrit | Jeremy Stanley proposed openstack/diskimage-builder master: Add security suite name override in debian-minimal https://review.opendev.org/688816 | 21:22 |
| clarkb | then depending on how hungry you are it might make lunch | 21:22 |
| mordred | clarkb: oh - I didn't know about order | 21:22 |
| clarkb | mordred: at least I took that away from reading some docs really quickly | 21:23 |
| mordred | clarkb: cool. I'll update it - no reason not to | 21:23 |
| clarkb | though rereading it it seems the test isn't optional when specifying the -v | 21:24 |
| *** diablo_rojo has quit IRC | 21:24 | |
| clarkb | so maybe I've misparsed the earlier bit | 21:24 |
| clarkb | the mount options are optional though | 21:24 |
| openstackgerrit | Monty Taylor proposed opendev/system-config master: Expose volumes for the gerrit hooks, static and tmp dirs https://review.opendev.org/678875 | 21:25 |
| *** gagehugo has quit IRC | 21:25 | |
| mordred | clarkb: I needed to update it for the missing 2.13 tmp anyway - so might as well update the order | 21:26 |
| mordred | corvus: if you have a sec to re-+2 that ^^ | 21:26 |
| corvus | mordred: i never did see an answer about my question of why volumes are needed there | 21:28 |
| openstackgerrit | Jeremy Stanley proposed openstack/project-config master: Override security suite for Debian image builds https://review.opendev.org/688817 | 21:29 |
| corvus | mordred: we can bind-mount stuff into the container without it being a volume -- so if that's why we're adding hooks, why make a volume? | 21:29 |
| corvus | (same for static) | 21:29 |
| corvus | i guess i understand tmp, if that's actually used for stateful info (plugin installation?) | 21:30 |
| mordred | corvus: oh. hrm. that's a great question (I don't see that question, but no matter) | 21:30 |
| mordred | yeah - tmp is where plugins get extracted to - so it's oddly more stateful | 21:30 |
| corvus | mordred: well, we were having a conversation about it in channel the other day... i guess you hit turbulence or something | 21:30 |
| mordred | corvus: ah - sorry - I though you meant on the review | 21:31 |
| mordred | corvus: I guess my brain has been thinking of declared volumes partially as a declaration of "these are things someone might want to bindmount in that the container content is going to use" - but that's probably me *way* over thinking it and your version of only using it for things that would need to be actually stateful if docker were to create volumes for you is more accurate | 21:32 |
| corvus | mordred: yeah -- i sorta think minimal volume count is desirable | 21:33 |
| openstackgerrit | Clark Boylan proposed openstack/project-config master: Remove opensuse-423 from nodepool's config https://review.opendev.org/688820 | 21:33 |
| *** gagehugo has joined #openstack-infra | 21:33 | |
| clarkb | I didn't see a change for ^ already. The depends on should ensure we don't merge it early | 21:33 |
| clarkb | fungi: why not update our mirror to match debian's repos? | 21:34 |
| corvus | clarkb: 688443 688444 | 21:35 |
| clarkb | aha | 21:35 |
| mordred | corvus: ok. I'll update the patch | 21:36 |
| clarkb | I knew about the first one but somehow missed the second | 21:36 |
| clarkb | I'll abandon mine | 21:36 |
| clarkb | corvus: can you check the comment on https://review.opendev.org/#/c/688444/1 | 21:36 |
| fungi | clarkb: if we do that we also need to update anywhere we're consuming those mirrors (so presumably in the variables we write out in images) | 21:37 |
| fungi | and it's too late in the day for me to reason through how to safely make that transition | 21:38 |
| clarkb | fungi: we should double check we don't already make that assumption and then have to undo it later | 21:38 |
| clarkb | deb {{ security_mirror }} {{ ansible_distribution_release }} main | 21:39 |
| clarkb | I think that confirms we don't do buster/updates and instead do buster | 21:40 |
| corvus | clarkb: sure, maybe do that in another step? | 21:40 |
| clarkb | corvus: wfm | 21:40 |
| *** ociuhandu has quit IRC | 21:40 | |
| clarkb | fungi: thinking out loud a bit more here, wouldn't it be better to match upstream so that our zuul-jobs roles line up with how other people will be mirroring debian? | 21:40 |
| fungi | it does make sense, yes | 21:40 |
| clarkb | fungi: for example if they did a straight up rsync type mirror rather than reprepro they would end up with buster/updates iirc | 21:40 |
| clarkb | (and then zuul-jobs would be wrong) | 21:41 |
| openstackgerrit | Nate Johnston proposed openstack/project-config master: Add tracking for Train jobs https://review.opendev.org/688822 | 21:41 |
| fungi | mostly trying to figure out what the fastest path is to getting this working before the impending job update breaks debian-stretch jobs, as i did not realize until an hour or so ago we were no longer building new images for that for the past week | 21:42 |
| fungi | also need to work out how/why our reprepro configuration was set up to omit /updates in the first place | 21:42 |
| clarkb | fungi: I think the older stretch images may have the necessary updates? (I don't recall exactly when those merged) | 21:42 |
| fungi | ahh, maybe, if they merged a week ago | 21:43 |
| fungi | in which case we could do a more orderly transition and rearrange the mirror layout | 21:43 |
| fungi | the debian-security mirror update has finally completed with the patch for the debian-security-updates file from 688788 manually applied, the vos release is underway | 21:45 |
| *** jamesmcarthur has quit IRC | 21:45 | |
| clarkb | fungi: https://review.opendev.org/#/c/682934/ the change merged almost a moneth ago | 21:47 |
| clarkb | *month even | 21:47 |
| clarkb | I think your 6 day old stretch image should be fine | 21:47 |
| fungi | phew | 21:48 |
| fungi | in that case i'm dropping this for the moment so i can start writing newsletter content i promised people i'd have done by the end of today | 21:48 |
| mnaser | fungi: i remember you showing me some neat trick on how to view big console files in my terminal.. | 21:50 |
| mnaser | view $url is just not doing the trick (thinking its an actual real file path) | 21:50 |
| clarkb | mnaser: `vim` should do it | 21:51 |
| clarkb | I don't know if `view` has different behavior there | 21:51 |
| clarkb | you should be able to give vim any url that curl can understand and it will download it and open it for you | 21:51 |
| mnaser | ah there ya go, i didnt have vim, so it worked fine with vim | 21:52 |
| mnaser | thanks clarkb | 21:53 |
| clarkb | oh ya if your view is from vi and not vim that probably explains it | 21:53 |
| clarkb | oh huh my view doesn't do it either, could be behavior compatibility then | 21:53 |
| clarkb | vim is the new command so can safely do new things | 21:54 |
| fungi | yeah, `view` work for me because i have vim-tiny set as my vi and view alternative | 21:55 |
| fungi | i think technically you want `vim -r` for an equivalent read-only mode to what `view` is expected to provide | 21:56 |
| fungi | i'm just habituated to many years of running `view` to prevent myself from inadvertently editing a file when i only wanted to look at it | 21:57 |
| clarkb | ++ | 21:57 |
| *** armax has quit IRC | 21:57 | |
| clarkb | and then you decide you actually do want to edit it and :w fails | 21:57 |
| fungi | that old sysadmin addage, "trust no one, but especially not yourself" | 21:57 |
| clarkb | so you do :w! and hope it works | 21:58 |
| clarkb | (half the time it fails because I also didn't raise my privs and need root to :w anyway) | 21:58 |
| *** markvoelker has joined #openstack-infra | 22:00 | |
| *** markvoelker has quit IRC | 22:04 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 22:05 |
| * mordred goes to find breakfast | 22:07 | |
| *** goldyfruit_ has quit IRC | 22:08 | |
| *** jamesmcarthur has joined #openstack-infra | 22:13 | |
| *** rcernin has joined #openstack-infra | 22:30 | |
| *** jamesmcarthur has quit IRC | 22:31 | |
| *** bnemec has quit IRC | 22:38 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 22:43 |
| *** mattw4 has quit IRC | 22:45 | |
| *** harlowja has quit IRC | 22:51 | |
| *** armax has joined #openstack-infra | 23:02 | |
| *** dchen has joined #openstack-infra | 23:09 | |
| *** verdurin has quit IRC | 23:11 | |
| *** tkajinam has joined #openstack-infra | 23:16 | |
| *** verdurin has joined #openstack-infra | 23:19 | |
| *** mriedem has quit IRC | 23:27 | |
| *** gfidente|afk has quit IRC | 23:29 | |
| *** calbers has quit IRC | 23:33 | |
| *** jamesmcarthur has joined #openstack-infra | 23:35 | |
| *** tosky has quit IRC | 23:38 | |
| *** eharney has quit IRC | 23:41 | |
| *** jklare has joined #openstack-infra | 23:44 | |
| *** markvoelker has joined #openstack-infra | 23:45 | |
| *** markvoelker has quit IRC | 23:55 | |
| *** jamesmcarthur has quit IRC | 23:59 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime https://review.opendev.org/688578 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!