Thursday, 2019-10-17

*** aaronsheffield has quit IRC00:05
*** Lucas_Gray has joined #openstack-infra00:09
*** markvoelker has joined #openstack-infra00:15
*** markvoelker has quit IRC00:20
*** markvoelker has joined #openstack-infra00:22
*** mriedem_afk has quit IRC00:30
*** bnemec has quit IRC00:45
*** rfolco has joined #openstack-infra00:49
*** rcernin has joined #openstack-infra00:51
*** dchen has joined #openstack-infra00:52
*** bnemec has joined #openstack-infra00:56
*** dychen has joined #openstack-infra00:58
*** dingyichen has joined #openstack-infra01:06
*** bnemec has quit IRC01:07
*** dychen has quit IRC01:09
openstackgerritMerged openstack/diskimage-builder master: Blacklist sphinx 2.1.0 (autodoc bug)  https://review.opendev.org/68747501:16
*** weifan has joined #openstack-infra01:24
*** mtreinish has quit IRC01:27
*** weifan has quit IRC01:29
*** rfolco has quit IRC01:42
*** jamesmcarthur has joined #openstack-infra01:53
*** jamesmcarthur has quit IRC01:53
*** jamesmcarthur has joined #openstack-infra01:53
*** Lucas_Gray has quit IRC01:56
*** rh-jelabarre has joined #openstack-infra01:57
*** dchen has quit IRC01:59
*** weifan has joined #openstack-infra02:09
*** apetrich has quit IRC02:10
*** weifan has quit IRC02:10
*** weifan has joined #openstack-infra02:12
*** weifan has quit IRC02:14
*** jamesmcarthur has quit IRC02:26
*** jamesmcarthur has joined #openstack-infra02:27
*** jamesmcarthur has quit IRC02:28
*** jamesmcarthur has joined #openstack-infra02:28
*** jamesmcarthur has quit IRC02:29
*** jamesmcarthur has joined #openstack-infra02:37
*** jamesmcarthur has quit IRC02:42
*** ricolin has joined #openstack-infra02:55
*** jamesmcarthur has joined #openstack-infra03:00
*** pleia2 has quit IRC03:04
*** xinranwang has joined #openstack-infra03:07
*** jamesmcarthur has quit IRC03:08
*** rh-jelabarre has quit IRC03:11
*** pleia2 has joined #openstack-infra03:13
*** rosmaita has left #openstack-infra03:31
*** dingyichen has quit IRC03:36
*** psachin has joined #openstack-infra03:40
*** ykarel has joined #openstack-infra03:41
*** jamesmcarthur has joined #openstack-infra03:42
*** ramishra has joined #openstack-infra03:44
*** ykarel has quit IRC03:51
*** larainema has joined #openstack-infra03:55
*** ramishra has quit IRC03:56
*** igordc has quit IRC04:15
*** dchen has joined #openstack-infra04:16
*** diablo_rojo has quit IRC04:17
*** ramishra has joined #openstack-infra04:17
*** pcaruana has joined #openstack-infra04:18
*** jamesmcarthur has quit IRC04:35
*** jamesmcarthur has joined #openstack-infra04:37
*** jamesmcarthur has quit IRC04:43
*** markvoelker has quit IRC04:45
*** lbragstad has quit IRC04:53
*** lbragstad has joined #openstack-infra04:53
*** dmsimard has quit IRC04:54
*** dansmith has quit IRC04:54
*** dmsimard has joined #openstack-infra04:54
*** ianw has quit IRC04:54
*** ianw_ has joined #openstack-infra04:55
*** dansmith has joined #openstack-infra04:55
*** xenos76 has joined #openstack-infra04:56
*** ianw_ is now known as ianw04:56
*** hwoarang has quit IRC05:01
*** jamesmcarthur has joined #openstack-infra05:05
*** hwoarang has joined #openstack-infra05:11
*** jamesmcarthur has quit IRC05:12
*** jamesmcarthur has joined #openstack-infra05:19
*** surpatil has joined #openstack-infra05:22
*** jamesmcarthur has quit IRC05:24
openstackgerritMerged zuul/nodepool master: Set default python-path to "auto"  https://review.opendev.org/68279705:26
*** ociuhandu has joined #openstack-infra05:26
openstackgerritMerged zuul/nodepool master: Remove unused playbook  https://review.opendev.org/68901405:28
*** ociuhandu has quit IRC05:31
*** xenos76 has quit IRC05:45
*** takamatsu has quit IRC05:47
*** raukadah is now known as chandankumar05:49
*** lpetrut has joined #openstack-infra06:04
*** lpetrut has quit IRC06:05
*** lpetrut has joined #openstack-infra06:05
openstackgerritOpenStack Proposal Bot proposed openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/68907806:10
*** whoami-rajat has joined #openstack-infra06:12
*** janki has joined #openstack-infra06:18
*** rcernin has quit IRC06:18
*** udesale has joined #openstack-infra06:22
*** pgaxatte has joined #openstack-infra06:25
openstackgerritMerged openstack/project-config master: Normalize projects.yaml  https://review.opendev.org/68907806:42
*** dpawlik has joined #openstack-infra06:42
*** markvoelker has joined #openstack-infra06:46
*** markvoelker has quit IRC06:51
*** FlorianFa has quit IRC06:53
*** trident has quit IRC06:55
*** slaweq has joined #openstack-infra06:55
*** trident has joined #openstack-infra06:58
*** FlorianFa has joined #openstack-infra07:06
*** tesseract has joined #openstack-infra07:09
*** tosky has joined #openstack-infra07:16
*** takamatsu has joined #openstack-infra07:18
*** jpena|off is now known as jpena07:24
*** dychen has joined #openstack-infra07:30
*** dchen has quit IRC07:33
*** dingyichen has joined #openstack-infra07:46
*** dychen has quit IRC07:48
*** jaicaa has quit IRC07:51
*** rpittau|afk is now known as rpittau07:52
*** xek has joined #openstack-infra07:52
openstackgerritSimon Westphahl proposed zuul/nodepool master: Fix register race condition in static provider  https://review.opendev.org/68909307:52
*** jaicaa has joined #openstack-infra07:54
*** xek_ has joined #openstack-infra07:54
*** kjackal has joined #openstack-infra07:56
*** xek has quit IRC07:57
*** xinranwang has quit IRC07:57
*** ralonsoh has joined #openstack-infra07:57
*** dtantsur|afk is now known as dtantsur07:59
*** e0ne has joined #openstack-infra08:01
*** tkajinam has quit IRC08:05
*** noama has joined #openstack-infra08:05
*** ociuhandu has joined #openstack-infra08:06
*** dingyichen has quit IRC08:07
*** dchen has joined #openstack-infra08:09
*** xek__ has joined #openstack-infra08:11
*** xek_ has quit IRC08:15
*** lucasagomes has joined #openstack-infra08:15
*** ociuhandu has quit IRC08:15
*** e0ne_ has joined #openstack-infra08:17
*** e0ne has quit IRC08:17
*** dchen has quit IRC08:20
*** whoami-rajat has quit IRC08:22
*** kopecmartin|off is now known as kopecmartin08:23
*** xenos76 has joined #openstack-infra08:25
*** dpawlik has quit IRC08:31
*** takamatsu has quit IRC08:32
*** pkopec has joined #openstack-infra08:34
*** xenos76 has quit IRC08:38
*** derekh has joined #openstack-infra08:40
*** ociuhandu has joined #openstack-infra08:40
*** ociuhandu has quit IRC08:42
*** xenos76 has joined #openstack-infra08:43
openstackgerritIvan Kolodyazhny proposed openstack/openstack-zuul-jobs master: Add openstack-tox-pep8 to openstack-python3-ussuri-jobs template.  https://review.opendev.org/68899708:54
*** dpawlik has joined #openstack-infra08:57
*** kjackal has quit IRC09:02
*** kjackal_v2 has joined #openstack-infra09:02
*** zbr has quit IRC09:07
*** janki has quit IRC09:09
*** xenos76 has quit IRC09:09
*** zbr has joined #openstack-infra09:10
*** e0ne has joined #openstack-infra09:10
*** e0ne_ has quit IRC09:11
*** ociuhandu has joined #openstack-infra09:21
*** xenos76 has joined #openstack-infra09:21
*** ociuhandu has quit IRC09:25
*** ociuhandu has joined #openstack-infra09:31
*** ykarel has joined #openstack-infra09:31
*** ociuhandu has quit IRC09:32
*** ociuhandu has joined #openstack-infra09:33
*** pgaxatte has quit IRC09:45
*** xenos76 has quit IRC09:46
*** ricolin has quit IRC09:50
*** xenos76 has joined #openstack-infra09:52
*** gfidente has joined #openstack-infra09:58
*** xenos76 has quit IRC09:59
*** whoami-rajat has joined #openstack-infra10:00
*** pgaxatte has joined #openstack-infra10:00
openstackgerritMonty Taylor proposed opendev/system-config master: Add jobs to build gerrit master branch  https://review.opendev.org/68848210:12
openstackgerritMonty Taylor proposed opendev/system-config master: Add checks plugin to gerrit master build  https://review.opendev.org/68848310:12
*** ykarel is now known as ykarel|afk10:27
*** ociuhandu has quit IRC10:27
*** ociuhandu has joined #openstack-infra10:27
*** dpawlik has quit IRC10:43
*** dmellado has quit IRC10:55
*** dmellado has joined #openstack-infra10:57
*** udesale has quit IRC10:57
*** jamesmcarthur has joined #openstack-infra11:00
*** jamesmcarthur has quit IRC11:04
*** ykarel|afk is now known as ykarel11:06
*** jpena is now known as jpena|lunch11:09
*** piotrowskim has joined #openstack-infra11:11
*** mtreinish has joined #openstack-infra11:12
*** ykarel is now known as ykarel|afk11:15
*** dpawlik has joined #openstack-infra11:16
*** hwoarang has quit IRC11:31
*** hwoarang has joined #openstack-infra11:42
*** ociuhandu has quit IRC11:47
*** ociuhandu has joined #openstack-infra11:48
*** markvoelker has joined #openstack-infra11:48
*** ociuhandu has quit IRC11:54
*** ykarel|afk is now known as ykarel11:58
*** rh-jelabarre has joined #openstack-infra12:02
*** ociuhandu has joined #openstack-infra12:03
*** jamesden_ is now known as jamesdenton12:09
*** ociuhandu has quit IRC12:11
*** jamesmcarthur has joined #openstack-infra12:11
*** ociuhandu has joined #openstack-infra12:11
*** rlandy has joined #openstack-infra12:15
*** ociuhandu has quit IRC12:16
*** jpena|lunch is now known as jpena12:19
*** rfolco has joined #openstack-infra12:23
*** rosmaita has joined #openstack-infra12:24
*** larainema has quit IRC12:26
*** takamatsu has joined #openstack-infra12:35
*** jamesmcarthur has quit IRC12:36
*** dtantsur is now known as dtantsur|brb12:38
*** mriedem has joined #openstack-infra12:42
*** dave-mccowan has joined #openstack-infra12:43
*** jamesmcarthur has joined #openstack-infra12:45
*** eharney has quit IRC12:45
*** derekh has quit IRC12:49
*** ramishra has quit IRC12:49
*** whoami-rajat has quit IRC12:50
*** whoami-rajat has joined #openstack-infra12:52
*** dmsimard has quit IRC12:54
*** dmsimard has joined #openstack-infra12:55
fricklerAJaeger: I'm assuming you are the one to know most about doc publishing, I found that updates to governance.o.o/tc/ leave the old content in place, see https://review.opendev.org/689143 for an example. I also vaguely remember that we had some similar issue a long time ago, but I'm not sure whether that got resolved or is a known todo item still12:59
*** rfolco is now known as rfolco|ruck13:03
*** ociuhandu has joined #openstack-infra13:09
*** iurygregory has joined #openstack-infra13:12
*** derekh has joined #openstack-infra13:12
* iurygregory grabs coffee to wake up13:12
*** takamatsu has quit IRC13:12
*** udesale has joined #openstack-infra13:16
*** jtomasek_ has joined #openstack-infra13:18
AJaegerfrickler: corvus did some great work but that is only implemented on docs.o.o. The publishing we use for static.o.o - where governance is hosted today - is not using that framework.13:19
AJaegerfrickler: once we switch to AFS publishing, it should work...13:19
*** jtomasek has quit IRC13:21
fricklerAJaeger: ah, so is that due to be part of the static.o.o upgrade? or more likely a further step after that is done?13:22
AJaegerfrickler: once we use the publishing for AFS, we get it for free ;)13:22
AJaegerfrickler: I pushed up some changes that should just do it, see https://review.opendev.org/#/c/681583/13:23
*** Goneri has joined #openstack-infra13:28
*** eharney has joined #openstack-infra13:30
*** factor has quit IRC13:36
*** lbragsta_ has joined #openstack-infra13:42
*** efried is now known as efried_afk13:46
openstackgerritMatt Riedemann proposed opendev/elastic-recheck master: Add query for networkx 2.4 bug 1848499  https://review.opendev.org/68916013:49
openstackbug 1848499 in OpenStack Compute (nova) "tests fail with networkx 2.4: "AttributeError: 'DiGraph' object has no attribute 'node'"" [Critical,In progress] https://launchpad.net/bugs/1848499 - Assigned to Balazs Gibizer (balazs-gibizer)13:49
*** ociuhandu has quit IRC13:57
fricklerAJaeger: ah, great, thx13:57
*** ykarel is now known as ykarel|away14:00
*** ociuhandu has joined #openstack-infra14:02
*** ramishra has joined #openstack-infra14:03
*** ociuhandu has quit IRC14:07
*** slittle1 has joined #openstack-infra14:09
slittle1I would like to correct a tag I pushed to opendev.org/starlingx/manifest.14:09
slittle1Pushed tag 2.0.1 before the reviews were completed, and there where revisions.14:10
*** bnemec has joined #openstack-infra14:11
slittle1tag 2.0.1 should point to c89aeff3401fa37ca679bf170cd2f239a51b639f  in opendev.org/starlingx/manifest. .... thanks14:11
openstackgerritMerged opendev/elastic-recheck master: Add query for networkx 2.4 bug 1848499  https://review.opendev.org/68916014:12
openstackbug 1848499 in OpenStack Compute (nova) "tests fail with networkx 2.4: "AttributeError: 'DiGraph' object has no attribute 'node'"" [Critical,In progress] https://launchpad.net/bugs/1848499 - Assigned to Balazs Gibizer (balazs-gibizer)14:12
clarkbslittle1: we generally avoid moving tags as downstream pullers wont get that update automatically14:12
*** efried_afk is now known as efried14:12
clarkbslittle1: instead it is generally best to push a new tag/release to supplant the old one.14:13
*** dtantsur|brb is now known as dtantsur14:14
*** ociuhandu has joined #openstack-infra14:15
openstackgerritTobias Henkel proposed zuul/zuul master: Extract an abstract base Parser class  https://review.opendev.org/54561014:15
*** surpatil has quit IRC14:16
*** lpetrut has quit IRC14:18
*** harlowja has quit IRC14:19
*** jamesmcarthur has quit IRC14:20
*** dpawlik has quit IRC14:27
*** takamatsu has joined #openstack-infra14:32
openstackgerritOliver Walsh proposed openstack/diskimage-builder master: WIP: ensure nouveau is blacklisted in initramfs too  https://review.opendev.org/67999214:34
*** pgaxatte has quit IRC14:37
openstackgerritTobias Henkel proposed zuul/nodepool master: Add libssl-dev to bindep  https://review.opendev.org/68918314:38
*** pmatulis has left #openstack-infra14:38
*** noama has quit IRC14:38
*** gyee has joined #openstack-infra14:40
*** ykarel|away has quit IRC14:44
*** jpena is now known as jpena|off14:47
*** ricolin has joined #openstack-infra14:47
sean-k-mooneyclarkb: so the main blocker to testing xen with devstack would be the fact you have to reboot into dom0 after install xen before running devstack14:50
clarkbyup thats doable14:51
sean-k-mooneyclarkb: but i assume we could skip that by having a ubuntu xen image14:51
sean-k-mooneythen just run devstack as normal14:51
clarkbI dont want a xen specific image to become the infra teams proble.14:51
clarkbbut you can reboot in your job no problem14:51
sean-k-mooneyreally i though zuul killed hte instance14:51
clarkbjenkins does, zuul does not14:52
fungiand, yay, we're no longer relying on jenkins14:52
clarkbthis was avery explicit thing in using ansible as a jenkins replacement14:52
sean-k-mooneydo we have any jobs that do that today?14:52
clarkbsean-k-mooney: I think there are some but I dont know of any specific ones14:52
clarkb(ylu have o restart the zuul log daemon thing and we've gotten questions about that so people must be doing it14:53
fungiin particular i don't know of devstack jobs doing it, since you have to go to extra lengths to restart services on reboot14:53
sean-k-mooneyi was under the impression we still could not do that but i guess we coudl try14:53
sean-k-mooneyfungi: well the idea woudl be a pre playbook to install xen14:53
*** efried is now known as efried_afk14:53
clarkbin your ansible you reboot then have a wait for ssh task then pick up once ssh is back14:53
sean-k-mooneythen a normal devstack job runs form dom0 with a few tweks in the local.conf for xen14:54
clarkbits a common ansible thing outside of zuul too14:54
jrosserare the cirros images kept somewhere inside the infra CI images?14:54
sean-k-mooneyclarkb: i have done it in ansible before but i thought nodepool got unhappy when the ssh connection dropped14:54
sean-k-mooneyi guess not14:54
clarkbsean-k-mooney: nodepool.has no clue14:54
sean-k-mooneyclarkb: cool good to know14:55
clarkbjrosser: yes in /home/zuul/cache or somethibg like that14:55
* clarkb looks at dib elements14:55
sean-k-mooneyi guess the node is reserved untill the zuul job finsihes so as long as it has more stuff to do and we wait in the job it should be fine14:55
clarkbexactly14:56
clarkbjrosser: /opt/cache/files https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements/cache-devstack/extra-data.d/55-cache-devstack-repos#L14214:59
jrosserclarkb: excellent thanks15:00
jrossercurros download site is/was down and bust a bunch of osa jobs15:01
jrosser*cirros15:01
*** dpawlik has joined #openstack-infra15:02
*** diablo_rojo has joined #openstack-infra15:03
*** dpawlik has quit IRC15:07
*** dmsimard has quit IRC15:11
*** dmsimard has joined #openstack-infra15:12
*** jamesmcarthur has joined #openstack-infra15:12
*** jamesmcarthur_ has joined #openstack-infra15:17
*** mattw4 has joined #openstack-infra15:18
*** jamesmcarthur has quit IRC15:21
*** igordc has joined #openstack-infra15:24
*** surpatil has joined #openstack-infra15:25
*** surpatil is now known as surpatil|dinner15:26
*** zbr has quit IRC15:29
*** zbr__ has joined #openstack-infra15:29
openstackgerritDavid Shrewsbury proposed zuul/zuul-jobs master: Increase minikube logging output to maximum  https://review.opendev.org/68902115:29
*** e0ne has quit IRC15:29
openstackgerritMerged zuul/zuul-registry master: Add docker buildset test  https://review.opendev.org/68795315:29
openstackgerritMerged zuul/nodepool master: Add libssl-dev to bindep  https://review.opendev.org/68918315:32
*** jamesmcarthur_ has quit IRC15:33
*** prometheanfire has quit IRC15:33
*** prometheanfire has joined #openstack-infra15:33
fnordahlWould there be a config-core around with a spare moment for https://review.opendev.org/#/c/687925/ ?15:34
*** jtomasek_ has quit IRC15:34
mnaserfnordahl: +w15:35
fnordahlexcelent, ta mnaser15:35
openstackgerritOliver Walsh proposed openstack/diskimage-builder master: WIP: ensure nouveau is blacklisted in initramfs too  https://review.opendev.org/67999215:39
*** ramishra has quit IRC15:41
openstackgerritMerged zuul/zuul-registry master: Add podman buildset test  https://review.opendev.org/68798615:43
openstackgerritMerged openstack/project-config master: Add OVN charms  https://review.opendev.org/68792515:44
noonedeadpunkclarkb: returning to the images cache question - so path to cirros should be just /opt/cache/files/cirros-0.4.0-x86_64-disk.img ?15:47
*** dpawlik has joined #openstack-infra15:47
*** surpatil|dinner is now known as surpatil15:47
clarkbI think so but can't confirm right now. Running errands15:47
slittle1clarkb:  did you been I can overwrite the 2.0.1 tag... or that I issue a new 2.0.1.1 tag?15:48
slittle1I don't have the power to alter a tag once created.15:48
clarkbslittle1: you should make a 2.0.2 tag15:48
clarkb2.0.1 should not be updatef as you cant garuntee others will get that update (due to how git works)15:48
slittle1We talked that over in our meeting, and that is not the preferred option.15:48
*** jamesmcarthur has joined #openstack-infra15:49
slittle1We would prefer to alter 2.0.1 and announce the change via IRC and mailing list15:49
clarkbslittle1: was thr fact that git wont update that tag on a pull considered?15:49
*** whoami-rajat has quit IRC15:50
clarkbthis means anyone that has 2.0.1 already will continue to have that old tag unless they manually update it15:50
slittle12.0.1 has not been announced to the public as of yet.15:50
slittle1We don't expect it to be in use15:50
clarkbbut git pull will fetch it15:50
clarkbthe concern is with git15:50
slittle1We know15:50
clarkband you know that it will not be updated anywhere it has already been fetched?15:51
*** dpawlik has quit IRC15:52
slittle1yes.  We'll make sure folks know that they must manually pull the tag, or start with a fresh clone.15:52
slittle1Saul: you want to +2 the change ?15:53
slittle1tag 2.0.1 should point to c89aeff3401fa37ca679bf170cd2f239a51b639f  in opendev.org/starlingx/manifest.15:54
clarkbin that case the best thing to do is send the request to the infra list eith specifcs, project, tagname, sha so that we have a record of it and remove the correct item. Then once removed you can create a new one15:55
clarkbfungi: ^ I'm not missing anything in that process right?15:55
fungii'm honestly not sure zuul will handle that well. executors/mergers will have the original tag and won't know that it's been replaced because git does not communicate changes in tags15:57
fungiwe can try it, but there's a good chance that any jobs triggered from the tag push will either generate artifacts based on the original tag state or simply break15:58
fungialso if there are jobs which generate artifacts from that tag, those may not get overwritten depending on the platform hosting them15:59
*** lucasagomes has quit IRC15:59
fungiand yeah, from a logistical standpoint in gerrit, it's just a matter of deleting the tag and letting people push a new one the same way they usually do16:00
slittle1I'm not aware of anything dependent on that tag, but I can take your concerns back our primes for debate.16:00
*** armstrong has joined #openstack-infra16:02
*** efried_afk is now known as efried16:02
openstackgerritMerged zuul/nodepool master: Fix register race condition in static provider  https://review.opendev.org/68909316:04
clarkbre zuul that is a good point16:05
fungiif there are no release jobs for that repo, then it may be a moot point from zuul's perspective16:05
sgwright, we have not setup any release jobs yet since our actual builds happen on the CENGN server where we release artifacts from16:06
fungibut for example, if this were a python project publishing sdists and wheels to pypi, just deleting and replacing the tag would fail because pypi doesn't allow replacement of files with the same name (even if they've been deleted first)16:06
fungisgw: are the cengn builds triggered manually, or will they possibly have already cached the original tag?16:07
sgwWe can fix all that as they are dealt with manually.16:07
fungijust be aware that if they rely on a git repository cache then they may need the tag deleted locally before they'll fetch the replacement16:08
slittle1automatic build nightly, but it doesn't use the manifest tags16:08
slittle1yes, we are aware16:08
fungiokay, cool16:09
fungias they say, "if it breaks you get to keep both pieces" ;)16:09
slittle1:)16:09
Shrewsinfra-root: Are we good for a nodepool launcher restart to pick up the python-path changes in https://review.opendev.org/682797 to go along with the similar recent zuul change?16:12
fungiShrews: seems fine to me, sure16:14
ShrewsActually, I want to wait a few minutes for a recent merge to propagate over.16:14
*** udesale has quit IRC16:22
*** ociuhandu_ has joined #openstack-infra16:23
*** ociuhandu has quit IRC16:26
*** markvoelker has quit IRC16:26
*** rpittau is now known as rpittau|afk16:27
*** ociuhandu_ has quit IRC16:30
*** SurajPatil has joined #openstack-infra16:33
Shrewsok, it's there now.16:35
*** surpatil has quit IRC16:36
fungirestarting those still seems fine to me16:36
Shrewsfungi: the zuul change *is* in place, right?16:37
Shrewscorvus: ^^ ?16:37
*** dtantsur is now known as dtantsur|afk16:37
clarkbYesterdays restart shouldve picked it up16:38
*** smarcet has joined #openstack-infra16:38
Shrewsok, i'll start with nl03 first and watch it a bit16:38
*** lpetrut has joined #openstack-infra16:39
fungiyeah16:39
fungithat was the reason for the restart16:39
fungihttps://wiki.openstack.org/wiki/Infrastructure_Status says "2019-10-16 20:03:10 UTC restarted all of zuul zuul on tag 3.11.1"16:40
*** lpetrut has quit IRC16:40
*** lpetrut has joined #openstack-infra16:40
fungiso i guess not the reason for the restart technically? the restart was to clear out the hung changes and bring in the fix for what caused them to get stuck there16:41
fungibut regardless we're running 3.11.1 and need at least 3.11.0 for the feature in nodepool right?16:41
ShrewsI'm not sure how this happened:  http://paste.openstack.org/raw/784544/16:44
Shrews'pool' should not be None16:44
*** jpena|off is now known as jpena16:44
Shrewsbut everything else seems to be humming along smoothly16:44
Shrewsfungi: that's my understanding, yes16:45
*** kjackal_v2 has quit IRC16:47
Shrewsclarkb: fungi: is packethost-us-west-1 still a thing? lots of errors on nl02 about it. maybe we can delete that if it isn't needed. it's already disabled with max-servers:0 and unable to cleanup resources16:49
*** kjackal has joined #openstack-infra16:49
fungiit is offline16:52
fungiso removing it seems reasonable to me16:53
*** weifan has joined #openstack-infra16:53
clarkbit was going to be revived as an osa deployed cloud but havent heard anynews on that recently16:54
clarkbcleanup should be fine16:54
*** derekh has quit IRC16:58
Shrews#status log Restarted nodepool launchers on commit cec796eaf2419b8741d2ca10720601485a5d5750 to pick up new python-path support.16:58
openstackstatusShrews: finished logging16:58
openstackgerritJames E. Blair proposed zuul/zuul-registry master: Disable namespacing  https://review.opendev.org/68921917:00
*** lpetrut has quit IRC17:01
openstackgerritDavid Shrewsbury proposed openstack/project-config master: Remove packethost provider  https://review.opendev.org/68922017:04
Shrewsclarkb: fungi: ^^17:05
*** markvoelker has joined #openstack-infra17:05
*** smarcet has quit IRC17:17
ShrewsAJaeger: do i just remove those 2 lines from the grafana files?17:29
AJaegerShrews: remove the one line and delete the other file completely17:30
Shrewsah17:30
AJaegerIt's named packethost ;)17:30
AJaegersorry for not beeing clear in my review17:30
openstackgerritDavid Shrewsbury proposed openstack/project-config master: Remove packethost provider  https://review.opendev.org/68922017:30
AJaegercorvus: want to merge https://review.opendev.org/#/c/682935/ now?17:31
*** lpetrut has joined #openstack-infra17:37
*** ociuhandu has joined #openstack-infra17:38
*** psachin has quit IRC17:46
*** dpawlik has joined #openstack-infra17:48
*** armax has quit IRC17:48
corvusAJaeger: done17:53
*** dpawlik has quit IRC17:53
openstackgerritBrian Haley proposed openstack/project-config master: Update neutron stable grafana dashboards  https://review.opendev.org/68923317:54
*** jamesmcarthur has quit IRC17:55
*** mriedem has quit IRC17:55
openstackgerritMerged openstack/project-config master: Remove packethost provider  https://review.opendev.org/68922017:56
*** mriedem has joined #openstack-infra17:56
*** harlowja has joined #openstack-infra18:00
*** jerryz has joined #openstack-infra18:08
*** ociuhandu has quit IRC18:13
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Use zuul-registry as buildset registry  https://review.opendev.org/68923818:14
*** weifan has quit IRC18:23
*** jpena is now known as jpena|off18:24
*** ociuhandu has joined #openstack-infra18:26
*** smarcet has joined #openstack-infra18:28
*** lbragsta_ has quit IRC18:29
*** armax has joined #openstack-infra18:30
*** zbr__ has quit IRC18:32
*** zbr has joined #openstack-infra18:32
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime  https://review.opendev.org/68857818:35
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Use zuul-registry as buildset registry  https://review.opendev.org/68923818:35
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable  https://review.opendev.org/68924018:35
AJaegercorvus: we need to merge https://review.opendev.org/#/c/680696/ first - I'll review now...18:36
*** weifan has joined #openstack-infra18:37
*** SurajPatil has quit IRC18:40
*** jtomasek has joined #openstack-infra18:41
*** armstrong has quit IRC18:41
*** weifan has quit IRC18:42
*** weifan has joined #openstack-infra18:43
*** weifan has quit IRC18:45
*** weifan has joined #openstack-infra18:46
openstackgerritMerged opendev/base-jobs master: Switch to prepare-workspace-git role  https://review.opendev.org/68069618:46
*** lpetrut has quit IRC18:46
*** weifan has quit IRC18:48
*** dpawlik has joined #openstack-infra18:51
*** ociuhandu has quit IRC18:52
*** dpawlik has quit IRC18:55
*** gfidente has quit IRC18:59
openstackgerritMerged openstack/project-config master: Cache repos in /opt/git/opendev.org  https://review.opendev.org/68293519:00
AJaegerianw: could you review https://review.opendev.org/688479 , please? It's a change for configure-mirrors and since you worked on it recently...19:01
mordredclarkb, corvus: https://review.opendev.org/#/c/688482 and https://review.opendev.org/#/c/688483 are green and ready to go19:02
openstackgerritMonty Taylor proposed opendev/system-config master: Run a gerrit container (test only)  https://review.opendev.org/63040619:03
*** tesseract has quit IRC19:07
corvusmordred: lgtm!19:08
openstackgerritMerged openstack/project-config master: Remove qcow2 version 0.10 compat from our images  https://review.opendev.org/68905519:13
*** iurygregory has quit IRC19:18
mordredcorvus: woot!19:23
mordredcorvus: also - yay for cross-source depends-on!19:23
openstackgerritClark Boylan proposed opendev/puppet-openstackci master: Add a warning to the third party CI doc  https://review.opendev.org/68924819:23
clarkbrecent email to the infra list and discussion in #openstack-cinder has led to ^19:23
clarkbI think getting that in would help set expectations19:23
openstackgerritClark Boylan proposed opendev/puppet-openstackci master: Add a warning to the third party CI doc  https://review.opendev.org/68924819:25
*** e0ne has joined #openstack-infra19:25
*** weifan has joined #openstack-infra19:26
openstackgerritClark Boylan proposed opendev/puppet-openstackci master: Add a warning to the third party CI doc  https://review.opendev.org/68924819:26
clarkbtyping is hard19:26
openstackgerritMonty Taylor proposed opendev/puppet-openstackci master: Add a warning to the third party CI doc  https://review.opendev.org/68924819:27
mordredclarkb: jinx19:28
clarkbbah19:28
*** weifan has quit IRC19:31
*** e0ne has quit IRC19:34
*** ralonsoh has quit IRC19:47
*** pcaruana has quit IRC19:49
*** weifan has joined #openstack-infra19:49
*** e0ne has joined #openstack-infra19:52
*** e0ne has quit IRC19:53
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: install-kubernetes: add sanity check for crio  https://review.opendev.org/68900119:59
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Increase minikube logging output to maximum  https://review.opendev.org/68902120:00
*** bnemec has quit IRC20:03
*** bnemec has joined #openstack-infra20:05
*** ricolin_ has joined #openstack-infra20:06
openstackgerritMerged zuul/zuul-jobs master: install-kubernetes: add support for cri-o runtime  https://review.opendev.org/68857820:08
*** ricolin has quit IRC20:09
*** kjackal has quit IRC20:10
*** kjackal has joined #openstack-infra20:10
openstackgerritMerged zuul/zuul-jobs master: Increase minikube logging output to maximum  https://review.opendev.org/68902120:14
*** efried is now known as efried_afk20:19
openstackgerritMerged zuul/zuul-jobs master: install-kubernetes: add sanity check for crio  https://review.opendev.org/68900120:19
* fungi curses at 15 minutes of his life lost to not realizing `time` is a bash builtin with different behavior from gnu /usr/bin/time20:19
corvusclarkb: i'm seriously considering writing a skopeo replacement that handles ipv6 addresses20:21
corvusclarkb: i honestly think it would be less work than writing playbooks that edit /etc/hosts everytime we want to copy images from one registry to another20:21
corvus(for our purposes, we pretty much only need the registry <-> registry functionality)20:22
clarkbcorvus: maybe it is worth bringing that back up with the conatiners folks on github again?20:24
clarkb"hey listen we know you said you want bug compat here but the end result is we are going to rewrite skopeo..."20:24
clarkbcorvus: re registry <-> registry ya though currently we stage it locally in a working location I think. But nothing would be lost if we staged in an arbitrary location and just forwarded on20:26
*** igordc has quit IRC20:26
corvusclarkb: i think the push/pull roles do http to http20:27
corvusclarkb: also "copyo"?  "scopyo"?  :)20:28
clarkb"docker-image-client-that-understands-tcp20:28
clarkbthat is too long to type20:28
mordredcorvus: seems reasonable to me20:30
fungidocker-deserves-ipv6-too-client20:31
*** jamesmcarthur has joined #openstack-infra20:32
fungiskopeo comes out of the same group at red hat doing podman?20:33
clarkbfungi: yes20:34
slittle12.0.1 tag still pointing to wrong commit.  Are we waiting on the mailing list request?  It's awaiting moderator approval.20:36
clarkbI was waiting on that yes20:37
slittle1who is the moderator ?20:37
fungioh, i'll go check the moderation queue again, though i looked a few hours ago20:37
clarkbfungi: typically pushes things through20:37
fungiapproved20:38
fungii usually only check the moderation queue on that list (and half a dozen more) once or maybe twice a day20:39
slittle1no prob.   Just hoping to be able to announce the new release before vacation starts  :)20:42
clarkbfungi: and we decided that the zuul risk was low given there are no jobs running currently on tags right?20:42
corvuszuul risk is high20:42
slittle1oh ?20:42
corvusif you delete the tag, please perform a full zuul shutdown and remove affected git repos from all systems and restart.  also beware that behavior may be erratic if the wrong tag is on nodepool images20:43
clarkbcorvus: reading the discussion from this morning it was asserted that there are no tag jobs that run currently and no artifacts produced from tags (so while zuul's local tags would never update that shouldn't matter)20:43
clarkbin general though I agree. Does the fact that tags aren't used by zuul currently change ^20:44
clarkb(in thatrepo)20:44
corvusclarkb: yes, but there will be repos with that tag, and they will be wrong, indefinitely, until they are deleted20:44
clarkbcorrect whcih is what I've warned them against several times now and they seem fine with that?20:44
corvusso any job at any point in the future could end up incorrect20:44
corvusthis could affect things years down the road and no one will remember this20:44
clarkbslittle1: what is the reason for not pushing a 2.0.2?20:44
clarkbcorvus: thats a good point20:45
clarkbif it happens tomorrow we'll likely associate the two, if next year gl20:45
corvus(i know that we have done this in extraordinary circumstances before, and i'm pretty sure we did remove the zuul repos)20:45
openstackgerritMonty Taylor proposed opendev/system-config master: Run a gerrit container (test only)  https://review.opendev.org/63040620:46
openstackgerritMonty Taylor proposed opendev/system-config master: Squash two ansible warnings  https://review.opendev.org/68926220:46
corvus(i guess we should document this, since it's come up more than once)20:46
clarkbslittle1: I'm unlikely to be able to do the full cleanup corvus describes this week (as I have more errands to run today and am traveling tomorrow)20:46
clarkbit might be helpful to understand why a 2.0.2 is not feasible or undesirable20:47
corvusbased on what i've read, we could probably postpone the cleanup.  but we should make sure it happens.20:48
slittle1I can take the 2.0.2 counter-proposal back to the StarlingX executive.  So the concern is that zuul will retain a copy of the old tag, and if we need to reference it at some future date, zuul may behave unexpectedly?20:48
*** mattw4 has quit IRC20:48
clarkbslittle1: yes, and while we can clean zuul up to avoid that it would require a fair bit of cleaning that I am not able to do in the near future. Also any other git repos that have fetched the ref are in the same position20:48
*** mattw4 has joined #openstack-infra20:48
slittle1I think they had the press releases ready to fly.   Not good20:48
fungiit will likely continue to act on the old content the tag points to, but that's uncertain20:48
clarkbslittle1: s/2.0.1/2.0.2/ in your press release file :)20:49
corvusslittle1: yes, including things not directly related to releases (for instance, pbr version calculations, reno, etc).20:49
*** mriedem has quit IRC20:49
*** eharney has quit IRC20:49
slittle1THis tag does not feed into any pbr calculations20:50
fungii think the point was if you run jobs on that repo which take any tags connected with the branch history into account, either now or in the future, that was merely an example20:50
*** dpawlik has joined #openstack-infra20:51
fungiit's really unfortunate for technical choices like this to be driven by marketing forces20:51
slittle1The only purpose of this git repo is to hold manifests for the 'repo' tool20:52
slittle1Packages are independently versioned within their own gits.20:53
corvusi believe you when you say you are certain you won't see any immediate impacts.  but we're asked to debug issues when zuul behaves strangely, and we know that this can cause that.  we need zuul's repos to *exactly* match the authoritative source, so if we violate protocol and change that source, we need to make zuul match.  otherwise it's a potential future bug and there is just no way to predict20:54
corvushow it will manifest.20:54
corvusso, if the tag is deleted, we must delete the zuul repos, whether that's today or next week.  but it must happen.20:55
*** dpawlik has quit IRC20:56
clarkbfor a list of repos to delete (so that they will get updated on the next pull) we need to clear out the nodepool builder dib cache for that repo (or just clear it out entirely and let it rebuild), we need to remove the repo on all zuul mergers and executors (and do a restart?). We may need to update codesearch too. Am I forgetting anything? gerrit would be the updated as would everything it pushes to20:56
slittle1k, I'll propose 2.0.2 to our community20:56
*** e0ne has joined #openstack-infra20:57
openstackgerritMerged zuul/zuul master: Fix github auth token authentication  https://review.opendev.org/68635520:57
corvusclarkb: oh, i forgot about the dib builder cache -- i was thinking that the images get rebuilt, but of course they get rebuilt from the cache.   so yeah, i think delete the cache and let it rebuild.20:57
clarkbcorvus: ya20:57
*** igordc has joined #openstack-infra20:58
*** trident has quit IRC20:58
fungiso 12 executors, 8 mergers and 3 image builders?20:59
clarkbfungi: ya20:59
*** jamesmcarthur has quit IRC20:59
clarkbplus a zuul restart20:59
clarkb(and maybe codesearch)20:59
fungioh yep20:59
fungiit fetches, so probably codesearch21:00
fungithough it may not fetch tags at all, i have no idea21:00
fungii've never seen tag annotations or filtering options in hound, but then again i haven't looked for them21:00
clarkbok I need to pop out now. back in a bit21:01
*** igordc has quit IRC21:03
*** trident has joined #openstack-infra21:04
fungii'll follow up to the ml with a recap of what we discussed21:04
mordredclarkb, corvus: I think you'll "enjoy" https://review.opendev.org/68926221:06
mordredthat patch is mostly because the log from the gerrit container is hard to read because of warning messages21:06
clarkbfungi: tyty21:07
*** jamesmcarthur has joined #openstack-infra21:08
*** panda has quit IRC21:08
*** igordc has joined #openstack-infra21:09
*** pkopec has quit IRC21:10
*** lpetrut has joined #openstack-infra21:11
*** panda has joined #openstack-infra21:11
mordredfungi: I'm sure you already would - but in that writeup maybe mention that while we're talking about remdiations in the CI system that we have to take, those are just manifestations of the reality that git doesn't propogate tag deletes or retags - so it's a good reminder that once a git tag is pushed it should basically be considered immutable because git itself just simply doesn't handle the case21:11
*** e0ne has quit IRC21:11
openstackgerritDrew Walters proposed opendev/system-config master: lists: Add Airship VMP mailing lists  https://review.opendev.org/68927121:11
fungimordred: yep21:12
*** hwoarang has quit IRC21:14
*** lbragstad has quit IRC21:14
*** hwoarang has joined #openstack-infra21:15
*** rfolco|ruck has quit IRC21:17
*** lpetrut has quit IRC21:17
openstackgerritKendall Nelson proposed opendev/storyboard-webclient master: Show Email Addresses when Searching  https://review.opendev.org/58971321:19
*** jamesmcarthur has quit IRC21:20
*** gyee has quit IRC21:20
*** smarcet has left #openstack-infra21:21
*** lbragstad has joined #openstack-infra21:24
openstackgerritDrew Walters proposed opendev/system-config master: lists: Add Airship VMP mailing lists  https://review.opendev.org/68927121:25
*** jamesmcarthur has joined #openstack-infra21:28
openstackgerritDrew Walters proposed opendev/system-config master: lists: Add Airship VMP mailing lists  https://review.opendev.org/68927121:33
openstackgerritKendall Nelson proposed opendev/storyboard-webclient master: Show Email Addresses when Searching  https://review.opendev.org/58971321:35
*** HenryG has quit IRC21:35
*** jamesmcarthur has quit IRC21:36
*** HenryG has joined #openstack-infra21:37
*** rcernin has joined #openstack-infra21:39
*** markvoelker has quit IRC21:39
*** kjackal has quit IRC21:41
*** kjackal has joined #openstack-infra21:42
*** Goneri has quit IRC21:45
fungiinteresting to nobody, probably, but if you want to postprocess the output of bash's `time` builtin, it's not as straightforward as you might think... it seems to create a separate stderr stream which can't be piped or redirected normally and needs to use a subshell21:51
fungiso `time foo 2>&1 | grep ^real` does not work, but `( time foo ) 2>&1 | grep ^real` does21:52
fungithe bash manpage and builtin help are fairly useless for discovering this quirk of implementation21:53
*** rcernin has quit IRC21:53
openstackgerritKendall Nelson proposed opendev/storyboard-webclient master: Show Email Addresses when Searching  https://review.opendev.org/58971321:53
*** slaweq has quit IRC21:56
*** weifan has quit IRC21:59
*** weifan has joined #openstack-infra22:01
*** tosky has quit IRC22:02
*** weifan has quit IRC22:05
*** smarcet has joined #openstack-infra22:08
*** rcernin has joined #openstack-infra22:09
*** smarcet has quit IRC22:10
*** ricolin_ has quit IRC22:10
mordredfungi: wow22:11
*** armax has quit IRC22:17
*** weifan has joined #openstack-infra22:18
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928022:22
openstackgerritJames E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker  https://review.opendev.org/68928022:23
*** weifan has quit IRC22:24
*** rcernin has quit IRC22:27
*** rcernin has joined #openstack-infra22:27
clarkbI've double checked that the opensuse-423 images are cleaned up on the builders and they were. We do have two failed build records in zk though22:30
clarkbdo those normally stick around? I thought they didn't but may be misremembering22:30
clarkbtheir impact is small but might be something to clean up when we remove the image22:35
clarkbShrews: ^ fyi22:36
*** diablo_rojo has quit IRC22:44
*** adriant has quit IRC22:46
*** diablo_rojo has joined #openstack-infra22:49
*** jbadiapa has quit IRC22:50
*** dpawlik has joined #openstack-infra22:52
clarkbmordred: for your ansible change around groups should we rename our groups to be foo_bar?22:52
*** kopecmartin is now known as kopecmartin|off22:56
*** dpawlik has quit IRC22:56
*** mattw4 has quit IRC22:58
*** mattw4 has joined #openstack-infra22:59
mordredclarkb: no - it's not necessary23:00
mordredI mean - we CAN if we want to dot-notation access them23:00
mordredbut I think group_vars['foo-bar'] is fine23:01
clarkbwfm23:01
*** xek__ has quit IRC23:01
mordredclarkb: I had to read a very long github issue thread to eventually learn that23:01
clarkbmordred: there was a weird puppet issue around git which I rechecked for. Want to see if that is persistent23:03
clarkb(not sure if you saw)23:03
*** jbadiapa has joined #openstack-infra23:04
*** weifan has joined #openstack-infra23:05
*** weifan has quit IRC23:05
*** weifan has joined #openstack-infra23:05
*** jamesmcarthur has joined #openstack-infra23:06
mordredyeah - I saw the issue but hadn't figured out what was wrong with it yet23:09
*** jbadiapa has quit IRC23:09
mordredthanks for the recheck23:09
*** weifan has quit IRC23:10
*** tkajinam has joined #openstack-infra23:11
*** rlandy has quit IRC23:11
*** rlandy has joined #openstack-infra23:15
*** jbadiapa has joined #openstack-infra23:16
*** weifan has joined #openstack-infra23:21
mordredclarkb: oh - if you're in the reviewing frame of mine - https://review.opendev.org/#/c/688482 and https://review.opendev.org/#/c/688483 are ready23:23
mordredOMG. ALL morning I've been trying to figure out why ansible can't find a file that should totally be there23:25
mordredmisspelled filename23:25
clarkbmordred: you saw my typing earlier. It is no better23:25
openstackgerritMonty Taylor proposed opendev/system-config master: Run a gerrit container (test only)  https://review.opendev.org/63040623:26
mordredclarkb: apparently "nofity_impact.yaml" is different than "notify_impact.yaml"23:26
clarkbcan confirm23:27
*** dchen has joined #openstack-infra23:27
*** jamesmcarthur has quit IRC23:29
fungif and t look similar when mirrored across the horizontal axis23:29
fungigives my dyslexia a heck ot a fime23:29
*** Lucas_Gray has joined #openstack-infra23:31
*** Lucas_Gray has quit IRC23:35
*** markvoelker has joined #openstack-infra23:40
*** mattw4 has quit IRC23:41
*** jamesmcarthur has joined #openstack-infra23:42
*** Lucas_Gray has joined #openstack-infra23:44
*** markvoelker has quit IRC23:45
openstackgerritOliver Walsh proposed openstack/diskimage-builder master: WIP: ensure nouveau is blacklisted in initramfs too  https://review.opendev.org/67999223:49
mordredcorvus: ZOMG. the run-a-gerrit-container job is almost green!23:55
mordredit's failing puppet23:55
mordredShrews, clarkb: does https://zuul.opendev.org/t/openstack/build/e1327d32752844c2bc47c6496e3d4ff4/console make any sense to you?23:57

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!