Tuesday, 2021-02-02

clarkb2021-02-02 00:04:09,065 INFO cherrypy.access.139663382723600: ::ffff:172.17.0.1 - - [02/Feb/2021:00:04:09] "GET /v2/upstream/image/blobs/sha256:358792b44b713f905789d54d9733e75c702792522c07f812bf873b5ffc5efe77?ns=docker.io HTTP/1.1" 404 735 "" "containerd/1.4.0+unknown" from https://zuul.opendev.org/t/zuul/build/02a0af1e9c23470ab93eac0160354449/log/builder/docker/buildset_registry.txt#18300:26
clarkbI'll get a change up shortly to handle the ns parameter there00:26
openstackgerritMerged zuul/zuul master: gitlab: handle protected branches  https://review.opendev.org/c/zuul/zuul/+/75690000:30
*** rlandy has quit IRC00:30
openstackgerritClark Boylan proposed zuul/zuul-registry master: Handle ns parameter in get_blob  https://review.opendev.org/c/zuul/zuul-registry/+/77354900:30
clarkbcorvus: ianw tristanC ^ fyi00:30
ianwclarkb: are we sure it's the buildx client and not some other part of the job using podman somehow?  but otherwise lgtm00:31
clarkbianw: yes it fails in the multiarch image build when buildx pulls the upstream image to build on top of00:32
clarkbI don't think that uses podman00:32
ianwi can't find the reference in the buildx code, but that doesn't mean much00:33
clarkbI bet its calling out to containerd00:35
clarkbI think all that container code gets passed around a lot. Like docker's failure to deal with ipv6 literals is everywhere too including podman00:35
clarkband that is because they all source the same file in some docker repo00:35
ianwyeah, there is also a vendor/ directory with ... a lot of stuff00:39
ianwhere's where it possibly came into buildx via vendored code updates https://github.com/docker/buildx/commit/2d720a1e0b99988f510b54f0192e3a612e564d1700:47
*** tjgresha has joined #zuul00:48
corvusmhu: this look right for tagging?  commit 7bab2f829165602af49ce42dfb05fd2a391f65ca (HEAD -> master, tag: 0.0.4, origin/master, origin/HEAD)00:52
-openstackstatus- NOTICE: The Gerrit service on review.opendev.org is being quickly restarted to apply a new security patch00:58
*** tjgresha has quit IRC01:30
openstackgerritMerged zuul/nodepool master: Connect root logger to null handler by default  https://review.opendev.org/c/zuul/nodepool/+/77324802:07
*** tjgresha has joined #zuul02:13
*** tjgresha has quit IRC02:30
*** hamalq has quit IRC02:36
*** irclogbot_0 has quit IRC03:27
*** ykarel has joined #zuul04:11
*** ykarel_ has joined #zuul04:15
*** ykarel has quit IRC04:17
*** irclogbot_3 has joined #zuul04:43
*** irclogbot_3 has quit IRC04:54
*** irclogbot_0 has joined #zuul04:59
*** vishalmanchanda has joined #zuul05:07
*** ykarel_ is now known as ykarel05:16
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** jfoufas1 has joined #zuul05:49
*** ykarel_ has joined #zuul05:51
*** ykarel has quit IRC05:53
*** ykarel_ is now known as ykarel06:22
*** jcapitao has joined #zuul07:00
*** hashar has joined #zuul08:13
*** rpittau|afk is now known as rpittau08:25
*** jfoufas1 has quit IRC08:40
*** nils has joined #zuul08:53
openstackgerritGuillaume Chauvel proposed zuul/zuul master: gitlab: Add access token name, Update docs, Fix webhook  https://review.opendev.org/c/zuul/zuul/+/77118408:53
*** jpena|off is now known as jpena08:56
*** tosky has joined #zuul09:02
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Update tests/base.py to use proper git data  https://review.opendev.org/c/zuul/zuul/+/74274609:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix gerrit merge commit change with zuul configuration  https://review.opendev.org/c/zuul/zuul/+/76288609:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix zuul-client enqueue-ref when oldrev/newrev aren't provided  https://review.opendev.org/c/zuul/zuul/+/76576709:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Improve merger getFilesChanges, Fix edge cases  https://review.opendev.org/c/zuul/zuul/+/76288709:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Add tests  https://review.opendev.org/c/zuul/zuul/+/74274709:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Reorg scheduler event process loop  https://review.opendev.org/c/zuul/zuul/+/74274809:09
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Scheduler ref-updated create/delete  https://review.opendev.org/c/zuul/zuul/+/73919809:10
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Allow ref-updated newrev+oldrev reconfiguration  https://review.opendev.org/c/zuul/zuul/+/74274909:10
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Get ref-updated modified files  https://review.opendev.org/c/zuul/zuul/+/73907809:10
mhucorvus: yes I think so09:13
openstackgerritClément Mondion proposed zuul/zuul master: [UI][Nodes] use patternfly 4  https://review.opendev.org/c/zuul/zuul/+/76819209:54
*** jfoufas1 has joined #zuul10:18
*** tosky has quit IRC10:33
*** tosky has joined #zuul10:34
openstackgerritMatthieu Huin proposed zuul/nodepool master: Move zuul integration testing to nodepool project  https://review.opendev.org/c/zuul/nodepool/+/76056410:37
openstackgerritMatthieu Huin proposed zuul/zuul master: Move the testing framework under zuul  https://review.opendev.org/c/zuul/zuul/+/76036110:41
*** zbr1 has joined #zuul11:14
*** zbr has quit IRC11:16
*** zbr1 is now known as zbr11:16
*** sshnaidm|ruck is now known as sshnaidm|afk11:38
*** jcapitao is now known as jcapitao_lunch12:08
*** piotrowskim has joined #zuul12:12
*** rlandy has joined #zuul12:26
*** hashar is now known as hasharLunch12:29
*** jpena is now known as jpena|lunch12:36
*** sshnaidm|afk is now known as sshnaidm|ruck12:36
openstackgerritMatthieu Huin proposed zuul/zuul master: Remove obsolete testenv "zuul_client"  https://review.opendev.org/c/zuul/zuul/+/77368113:08
*** jcapitao_lunch is now known as jcapitao13:14
*** hasharLunch is now known as hashar13:18
*** jpena|lunch is now known as jpena13:25
openstackgerritClément Mondion proposed zuul/zuul master: [UI][Nodes] use patternfly 4  https://review.opendev.org/c/zuul/zuul/+/76819213:51
openstackgerritClément Mondion proposed zuul/zuul master: [UI][Nodes] use patternfly 4  https://review.opendev.org/c/zuul/zuul/+/76819214:47
openstackgerritSlawek Kaplonski proposed zuul/zuul-jobs master: [multi-node-bridge] Add script to configure connectivity  https://review.opendev.org/c/zuul/zuul-jobs/+/76265014:50
openstackgerritClément Mondion proposed zuul/zuul master: [UI][Nodes] use patternfly 4  https://review.opendev.org/c/zuul/zuul/+/76819214:50
openstackgerritClément Mondion proposed zuul/zuul master: [UI][Nodes] use patternfly 4  https://review.opendev.org/c/zuul/zuul/+/76819214:51
tobiashmhu: I've commented on 73408214:54
openstackgerritClément Mondion proposed zuul/zuul master: [web][Labels] update table to new lib  https://review.opendev.org/c/zuul/zuul/+/77370315:07
mhutobiash, thanks, I'll have a look15:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Update tests/base.py to use proper git data  https://review.opendev.org/c/zuul/zuul/+/74274615:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix gerrit merge commit change with zuul configuration  https://review.opendev.org/c/zuul/zuul/+/76288615:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix zuul-client enqueue-ref when oldrev/newrev aren't provided  https://review.opendev.org/c/zuul/zuul/+/76576715:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Improve merger getFilesChanges, Fix edge cases  https://review.opendev.org/c/zuul/zuul/+/76288715:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Add tests  https://review.opendev.org/c/zuul/zuul/+/74274715:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Reorg scheduler event process loop  https://review.opendev.org/c/zuul/zuul/+/74274815:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Scheduler ref-updated create/delete  https://review.opendev.org/c/zuul/zuul/+/73919815:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Allow ref-updated newrev+oldrev reconfiguration  https://review.opendev.org/c/zuul/zuul/+/74274915:13
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Get ref-updated modified files  https://review.opendev.org/c/zuul/zuul/+/73907815:13
mhuzuul-maint, I've revisited moving zuul's test framework into the zuul package so it can be imported in functional tests for other projects: https://review.opendev.org/q/topic:%22zuul.tests%22+(status:open%20OR%20status:merged)15:15
mhuit works well for zuul-client but I have trouble doing it with the zuul-nodepool tests; does it actually need to be done for nodepool too?15:15
zbrmhu: good idea15:18
openstackgerritClément Mondion proposed zuul/zuul master: [web][Labels] update table to new lib  https://review.opendev.org/c/zuul/zuul/+/77370315:18
*** hashar is now known as hasharAway15:27
fungimhu: one problem i expect that to bring is that testing something like zuul-client will result in you having to install all of zuul's dependencies just to use the test framework from it... though maybe extras is a way to work around that15:36
tristanCfungi: mhu: and the other way around, the test requirements will be zuul requirements too15:46
tristanCif the goal is to re-use the tests function, could they be published as a separate deliverable?15:47
fungii think we can still artificially split those test requirements, it's how we've done unit tests with the tests inside the package15:48
fungiin other projects i mean15:49
fungie.g., nodepool is already that way15:49
fungithe tests are accessible as nodepool.tests.something, but the requirements for running those tests are tracked in test-requirements.txt instead of requirements.txt so they don't pollute the setup-requires in the package15:50
*** jamesmcarthur has joined #zuul16:02
*** jamesmcarthur has quit IRC16:03
*** jamesmcarthur has joined #zuul16:03
*** hasharAway is now known as hashar16:04
corvusmhu: we explicityl decided not to do that16:20
*** ykarel has quit IRC16:20
corvusmhu: i understand the issue with zuul-client, but i don't think we should pursue that right now.  the zuul tests are huge, and they're not actually something we consider to be a deliverable of zuul.16:21
corvusmhu: it might be better to try to brainstorm a different way of testing zuul-client with zuul.16:22
corvusmhu: maybe you could use fakes in z-c for unit tests, then set up a real zuul (in containers, like quickstart) for integration tests.16:30
corvusmhu: or maybe the status quo is okay?  i know it's not ideal, but zuul-client shouldn't be changing too much; it may not be worth a huge investment to improve its tests.16:31
tristanCcorvus: adding the tests to zuul deliverable would not be too heavy, compressed the tests directory adds 1.8MB to the 10.9MB zuul-3.19.1.tar.gz, and it would be barely noticable for the container image16:36
tristanCcorvus: though i also think it's better to keep them separated16:37
mhucorvus, I can work with the status quo, provided we're okay with functional testing of new features in follow-up patches. Like https://review.opendev.org/c/zuul/zuul-client/+/752909 then https://review.opendev.org/c/zuul/zuul/+/75898516:38
avasstristanC: I think software factory is being sad: https://review.opendev.org/c/zuul/zuul-jobs/+/77158516:38
avassnot sure if that's because of my change or not but it looks irrelevant16:38
corvusmhu: yep, i'm okay with the followup testing approach; i think that's a good balance of effort/reward for now.  :)16:39
tristanCavass: oops, it's the reno test that doesn't work on python2, let me disable it quickly16:39
clarkbtristanC: corvus if you get a chance https://review.opendev.org/c/zuul/zuul-registry/+/773549 should fix the next thing with the buildx multiarch jobs16:52
*** jcapitao has quit IRC16:52
clarkbI'll continue to shepherd that along if the registry update lands16:52
corvusclarkb: lgtm thx16:52
*** jamesmcarthur has quit IRC16:52
*** jamesmcarthur has joined #zuul16:54
*** jamesmcarthur has quit IRC16:57
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Update tests/base.py to use proper git data  https://review.opendev.org/c/zuul/zuul/+/74274616:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix gerrit merge commit change with zuul configuration  https://review.opendev.org/c/zuul/zuul/+/76288616:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Fix zuul-client enqueue-ref when oldrev/newrev aren't provided  https://review.opendev.org/c/zuul/zuul/+/76576716:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Improve merger getFilesChanges, Fix edge cases  https://review.opendev.org/c/zuul/zuul/+/76288716:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Add tests  https://review.opendev.org/c/zuul/zuul/+/74274716:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Reorg scheduler event process loop  https://review.opendev.org/c/zuul/zuul/+/74274816:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Scheduler ref-updated create/delete  https://review.opendev.org/c/zuul/zuul/+/73919816:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Allow ref-updated newrev+oldrev reconfiguration  https://review.opendev.org/c/zuul/zuul/+/74274916:59
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Get ref-updated modified files  https://review.opendev.org/c/zuul/zuul/+/73907817:00
*** jamesmcarthur has joined #zuul17:00
openstackgerritMerged zuul/zuul-registry master: Handle ns parameter in get_blob  https://review.opendev.org/c/zuul/zuul-registry/+/77354917:06
*** zbr1 has joined #zuul17:16
*** zbr has quit IRC17:18
*** zbr1 is now known as zbr17:18
openstackgerritHemant Jain proposed zuul/nodepool master: Changed ExecStart address location in etc/nodepool-launcher.service.d/centos.conf as original configuration was giving an error. Explained in story https://storyboard.openstack.org/#!/story/2008569  https://review.opendev.org/c/zuul/nodepool/+/77366717:20
*** rlandy is now known as rlandy|biab17:40
clarkbzbr: corvus ianw fungi that latest patch to the zuul-registry seems to have fixed the multiarch test jobs17:41
corvusclarkb: thank you for fiking that!17:41
fungithanks clarkb!17:41
corvusfixing even17:41
corvusit was already fiked, i'd say17:42
clarkbmy do not merge change is now only failing on the k8s crio job17:42
corvus(dvorak typos are the best)17:42
clarkblooks like a post run playbook does a health check and etcd returns a failure (not sure if that is a consistent or one off failure though)17:42
clarkbhttps://zuul.opendev.org/t/zuul/build/a6fbef4038824d8fbdc87fba870ceb73/log/job-output.txt#623-659 is the crio error17:43
zbrclarkb: a recheck to see if is random?17:44
*** jamesmcarthur has quit IRC17:45
clarkbzbr: https://zuul.opendev.org/t/zuul/builds?job_name=zuul-jobs-test-ensure-kubernetes-crio implies it isn't a consistent error17:46
clarkbunless it is a very new issue17:46
*** jpena is now known as jpena|off17:55
*** jamesmcarthur has joined #zuul17:59
*** nils has quit IRC18:11
*** jfoufas1 has quit IRC18:14
*** fbo|off is now known as fbo18:22
*** jamesmcarthur has quit IRC18:24
*** sshnaidm|ruck is now known as sshnaidm|afk18:34
*** rpittau is now known as rpittau|afk18:39
avasstristanC: thanks!18:39
*** hashar is now known as hasharDinner18:41
*** jamesmcarthur has joined #zuul18:52
*** jamesmcarthur has quit IRC19:03
*** jamesmcarthur has joined #zuul19:15
*** rlandy|biab is now known as rlandy19:35
openstackgerritTristan Cacqueray proposed zuul/nodepool master: quota: ignore node without a type attribute  https://review.opendev.org/c/zuul/nodepool/+/77378819:37
*** hasharDinner is now known as hashar19:39
openstackgerritTristan Cacqueray proposed zuul/nodepool master: quota: ignore node without a type attribute  https://review.opendev.org/c/zuul/nodepool/+/77378819:43
*** evrardjp has quit IRC19:51
*** jamesmcarthur has quit IRC20:05
*** jamesmcarthur has joined #zuul20:06
*** jamesmcarthur has quit IRC20:07
*** jamesmcarthur has joined #zuul20:12
*** evrardjp has joined #zuul20:18
*** evrardjp has quit IRC20:26
*** evrardjp has joined #zuul20:27
*** jamesmcarthur has quit IRC20:40
*** vishalmanchanda has quit IRC21:15
masterpeI'm sorry if I'm ask stupid questions but I'm trying to convert my static nodepool instance to a dynamic one with using https://zuul-ci.org/docs/nodepool/openstack.html21:24
masterpebut when I try to start nodepool-launcher  I get a error: https://gist.github.com/mpiscaer/ef07d155f533ce089fda77a9f7bcc28121:26
corvusmasterpe: no stupid questions around here, only bad documentation (i'm allowed to say that since i wrote a lot of it) :)21:28
ianw    diskimage = config.diskimages[i.name]21:28
ianwKeyError: 'ubuntu-bionic'21:28
corvusmasterpe: the nodepool config is probably missing a standalone diskimages section21:28
ianwmasterpe: ^ that's telling you it can't find the 'ubuntu-bionic' diskimage ... maybe it's not defined?21:28
masterpeYes, in the /etc/nodepool/nodepool.yaml I have change that.21:29
corvusmasterpe: however, if you aren't building images, then you probably don't need that.  if that's the case, then you probably want to switch from using diskimages to 'cloud-images' in the launcher config21:29
corvusmasterpe: so first question is: are you trying to build your own images, or do you just want to use an existing image in the cloud?21:29
masterpeah21:29
masterpeNo I want to use a existing image.21:30
corvusmasterpe: i'm literally doing the exact same thing right now, let me paste you the config i just wrote :)21:30
corvusmasterpe: http://paste.openstack.org/show/802261/21:32
corvusmasterpe: you may or may not need the "networks" section21:32
corvusmasterpe: but that's a minimal launcher config file for an openstack cloud using an existing image21:33
corvusmasterpe: the all caps stuff (like IMAGE_NAME_HERE) are things you'll need to fill in based on the values in the cloud provider.21:35
*** hashar has quit IRC21:40
*** jamesmcarthur has joined #zuul21:41
masterpeI also had to solve other issues, in my config. corvus Thanks for helping out. Is the openstack driver generating dynamic ssh-keys or what is the best solution to that?21:53
clarkbmasterpe: no, you are expected to pregenerate a key and then bake that into the imge or supply it via nova's ssh key management21:55
clarkbmasterpe: then on the zuul side it bootstraps the node and switches the keys over to a dynamic key21:55
clarkbthe nodepool side is relatively simple in comparison21:55
masterpeso the private key needs to be in the home directory of the nodepool user?21:56
clarkbI want to say no, only zuul needs the private key side to bootstrap the dynamic keys21:57
clarkbbut I could be wrong about that21:57
corvuszuul only, not nodepool22:04
corvusmasterpe: in the config i pasted, it references an openstack 'keypair'.  zuul needs the public key for that.22:04
clarkbcorvus: unrelated, but do we need to consider a zuul-regsitry release?22:05
corvusclarkb: couldn't hurt, though i don't think it's urgent.22:10
clarkbok22:10
openstackgerritGuillaume Chauvel proposed zuul/zuul master: Tenant reconfiguration: Get ref-updated modified files  https://review.opendev.org/c/zuul/zuul/+/73907822:10
clarkbit seems that the intermediate registry for opendev has updated in any case22:11
corvusmhu: i've pushed the 0.0.4 tag; can you please check pypi and dockerhub to make sure both updated (after the jobs finish) and let me know if it's okay or there are any anomalies22:15
*** jamesmcarthur has quit IRC22:49
*** asettle has joined #zuul22:56
*** tjgresha has joined #zuul23:13
corvusOpen10K8S: fyi 772364 needs an update (see the log from the test failure)23:21
tjgreshamy builds are passing, but on the buildset tab on the web ui, all are are result of error - any way to see what that error is?23:28
corvustjgresha: a zuul admin (dunno if that's you or not :) will need to look at the zuul executor logs to find out23:30
tjgreshayeah that is me..23:30
corvustjgresha: either the executor logs or possible the scheduler.  relevant log lines should have the build uuid in them, so you can grep for that.23:31
corvustjgresha: but a result of error does mean an error related to zuul (as opposed to some kind of failure of the job)23:33
tjgreshai been called worse23:33
*** tjgresha has quit IRC23:34
*** tjgresha has joined #zuul23:35
corvustjgresha: oh, also the error cause may be available in the web ui under the build page23:35
tjgreshaok that is good to know23:36
corvusi'm still looking into that23:36
tjgreshaappreciate it.. will dig some more..23:36
corvushuh; i think we may expose that in the api but not on the web page (doh)23:37
corvusyep that's the case23:37
corvustjgresha: okay, so, in addition to checking the log files, you may have success with this:23:38
corvushere's an example error build: https://zuul.opendev.org/t/openstack/build/10a13bb2a40f490b9d6c3437bc3b41e623:38
corvushere's the api query for the same build: https://zuul.opendev.org/api/tenant/openstack/build/10a13bb2a40f490b9d6c3437bc3b41e623:38
corvusnote the "error_detail" field there23:38
corvuswe need to plumb that through to the web page, obviously23:39
corvusbut in the interim, that might be an easy thing for you to check23:39
tjgreshagood stuff - i will give it go..23:42
corvusi'm working on a patch for the web ui23:46
*** rlandy has quit IRC23:55

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!