Wednesday, 2019-05-15

*** jamesmcarthur has joined #zuul00:22
*** jamesmcarthur has quit IRC00:46
*** jamesmcarthur has joined #zuul00:47
*** jamesmcarthur has quit IRC00:52
*** mattw4 has quit IRC00:56
*** jamesmcarthur has joined #zuul01:11
*** SpamapS has quit IRC01:16
*** SpamapS has joined #zuul01:16
*** sshnaidm|afk has quit IRC01:42
*** SpamapS has quit IRC01:52
*** SpamapS has joined #zuul01:53
*** rlandy|bbl is now known as rlandy02:12
*** jamesmcarthur has quit IRC02:14
*** jamesmcarthur has joined #zuul02:32
*** rlandy has quit IRC02:40
*** bhavikdbavishi has joined #zuul03:00
*** SpamapS has quit IRC03:08
*** SpamapS has joined #zuul03:09
openstackgerritTristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port  https://review.opendev.org/65920903:21
*** SpamapS has quit IRC03:30
*** SpamapS has joined #zuul03:32
*** SpamapS has quit IRC03:40
*** jamesmcarthur has quit IRC03:40
*** SpamapS has joined #zuul03:41
*** jamesmcarthur has joined #zuul03:58
*** jamesmcarthur has quit IRC04:03
*** hashar has joined #zuul04:19
*** mattw4 has joined #zuul04:24
SpamapStristanC: worth noting that you can add the same host twice, as long as it has different inventory_hostname's04:37
SpamapSIt's a problem for 'hosts: all', but I actually avoid that in my playbooks in general, because I often have ancillary temporary hosts in the inventory for delegations.04:39
*** mattw4 has quit IRC04:39
tristanCSpamapS: good idea, thanks, one less patch then :-)04:56
*** bhavikdbavishi1 has joined #zuul05:12
*** bhavikdbavishi has quit IRC05:14
*** bhavikdbavishi1 is now known as bhavikdbavishi05:14
*** hashar has quit IRC05:24
openstackgerritTristan Cacqueray proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid  https://review.opendev.org/65921805:35
*** bjackman_ has joined #zuul05:44
*** spsurya has joined #zuul05:49
tristanCclarkb: corvus: it seems like a new version of skopeo is installed on the executor and the jobs using push-to-intermediate-registry are now failing because of a missing subuid file, for example: http://zuul.openstack.org/build/f385f07558b44495815fa7e65cf3726705:52
tristanCi think https://review.opendev.org/659218 should fix that, but i don't know if your ubuntu ship with that file on the host05:53
tristanCiiuc, the ppa is building skopeo continuously and it pulled a change that is not released yet in: https://launchpadlibrarian.net/420875087/skopeo_0.1.36-1~dev~ubuntu18.04~ppa16_0.1.36-1~dev~ubuntu18.04~ppa17.diff.gz05:54
*** bjackman_ has quit IRC05:57
*** bjackman_ has joined #zuul06:09
*** threestrands has joined #zuul06:12
*** bjackman_ has quit IRC06:23
*** abraverm has quit IRC06:41
*** bhavikdbavishi1 has joined #zuul06:49
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Fix scheduler breakage when using periodic jobs with artifacts  https://review.opendev.org/65922206:50
*** bhavikdbavishi has quit IRC06:51
*** bhavikdbavishi1 is now known as bhavikdbavishi06:51
*** bhavikdbavishi1 has joined #zuul06:55
*** bhavikdbavishi has quit IRC06:56
*** bhavikdbavishi1 is now known as bhavikdbavishi06:56
*** bjackman_ has joined #zuul06:56
*** abraverm has joined #zuul07:15
*** themroc has joined #zuul07:26
*** markwork has joined #zuul07:31
*** hashar has joined #zuul07:41
*** corvus has quit IRC07:47
*** bhavikdbavishi1 has joined #zuul07:51
*** corvus has joined #zuul07:52
*** bhavikdbavishi has quit IRC07:52
*** bhavikdbavishi1 is now known as bhavikdbavishi07:52
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: remove SafeLoader left-over from ZuulJSONEncoder  https://review.opendev.org/65902607:59
*** markwork has quit IRC08:01
*** themroc has quit IRC08:06
*** themroc has joined #zuul08:06
*** bhavikdbavishi has quit IRC08:13
*** lennyb has joined #zuul08:16
*** threestrands has quit IRC08:18
*** abraverm has quit IRC08:22
*** abraverm has joined #zuul08:23
corvustristanC: the pre-run idea sounds promising :)08:37
corvustobiash: i'm in munich airport (transferring to barcelona)  :)08:38
corvusi reviewed a lot of changes on the plane08:39
corvustristanC: https://review.opendev.org/659218 looks good -- the file that references exists on our executors08:44
corvusso to fix our jobs we're going to need to restart executors with that08:45
corvuswe can either manually apply the patch, or make the jobs non-voting to get that merged, or update the executor config to add that path, or force-merge it08:47
*** pcaruana has joined #zuul08:51
*** bhavikdbavishi has joined #zuul08:51
corvusi'm sleep deprived, so i think i will take the safest route and make the jobs non-voting08:52
* AJaeger waves to corvus (munich is just one hour south from here ;)08:57
openstackgerritJames E. Blair proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid  https://review.opendev.org/65921808:57
openstackgerritJames E. Blair proposed zuul/zuul master: Make image build jobs voting again  https://review.opendev.org/65924208:57
corvuszuul-maint: ^ those are high priority :)  thanks tristanC08:58
tobiashcorvus: looks like you had a long flight ;)08:59
corvustobiash: 10.5 hours... then i have a 2 hour flight... then a 1 hour flight...08:59
corvusAJaeger: o/09:00
corvuscause in the world of star alliance, munich is on the way to barcelona...09:00
* tobiash waves at corvus09:00
corvusthe food situation at MUC is *much* better than FRA09:00
tobiash:)09:01
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket build status reporter  https://review.opendev.org/65833509:06
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883509:06
*** jangutter has quit IRC09:06
fbo_Hi, we discovered that issue with dequeue-ref usage yesterday: https://review.opendev.org/659110/ project name is not take in account there.09:13
corvusfbo_: thanks, lgtm09:17
openstackgerritFabien Boucher proposed zuul/zuul master: Prevent Zuul scheduler to crash at startup if gerrit down  https://review.opendev.org/57619209:18
fbo_thanks :)09:20
*** sshnaidm has joined #zuul09:21
corvusclarkb: i think https://review.opendev.org/649900 will be of interest to you09:32
*** markwork has joined #zuul09:38
markworkHey ho, James E Blair? Can I'd like to tack on the functional testing in part to the third patchset and then maybe add a fourth with any leftovers09:40
openstackgerritMonty Taylor proposed zuul/zuul master: Remove unneeded extras definition  https://review.opendev.org/65925509:40
corvusmarkwork: that sounds great09:40
*** bhavikdbavishi1 has joined #zuul09:40
markworkcorvus: Cool, so for me I'll mark the first patchset as done and am waiting for the final review on the 2nd, and will continue with the event source today.09:41
*** bhavikdbavishi has quit IRC09:41
corvusmarkwork: given that, you can consider my -1 on that first patch temporary, otherwise it looks fine.  we'll probably just land the whole series once everything is in place09:42
corvusmarkwork: and i suspect you can copy a lot of the functional test stuff from the pagure patch (and later, once both bitbucket and pagure merge, we'll probably swing back around and refactor that to reduce the duplication)09:44
*** bhavikdbavishi1 has quit IRC09:45
*** sshnaidm is now known as sshnaidm|afk09:49
*** bhavikdbavishi has joined #zuul09:52
markworkcorvus: that sounds like a good deal :)09:54
markworkI got a RETRY_LIMIT from zuul-build-image... all the other checks look good. Something broken in the build infra?09:56
*** hashar has quit IRC09:57
corvusmarkwork: yeah, we have a patch working its way through to fix it09:57
corvus https://review.opendev.org/65921809:57
*** bhavikdbavishi has quit IRC10:17
openstackgerritSimon Westphahl proposed zuul/zuul master: Fix missing check if logger is None  https://review.opendev.org/65926210:26
*** yolanda_ has joined #zuul10:39
*** bhavikdbavishi has joined #zuul10:41
*** jangutter has joined #zuul11:14
*** abraverm has quit IRC11:17
*** hashar has joined #zuul11:30
tristanCcorvus: binding the subuid map may not be enough, it seems like you can't use that ppa version of skopeo (git master) in bubblewrap11:50
*** bhavikdbavishi has quit IRC11:50
tristanCcorvus: a quick test show that it's going to fail with: "ERRO[0000] error opening "allow" to /proc/31/setgroups: write /proc/31/setgroups: operation not permitted"11:50
*** panda is now known as panda|launch12:01
*** spsurya has quit IRC12:08
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888912:17
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889512:18
*** bhavikdbavishi has joined #zuul12:19
*** jamesmcarthur has joined #zuul12:19
*** dkehn has quit IRC12:25
pabelangerwow, that is a large queue of patches in gate for zuul :D12:25
pabelangermerge all the things12:26
*** jamesmcarthur has quit IRC12:29
*** rlandy has joined #zuul12:36
tristanCcorvus: the proper fix for skopeo failing would be to add all those capabilities to the bwrap command: https://github.com/containers/skopeo/blob/master/cmd/skopeo/unshare_linux.go#L912:36
tristanCcorvus: or rollback to skopeo-0.1.3512:37
tristanChere is a quick reproducer: http://paste.openstack.org/show/751417/12:42
tristanCi think we should at least ensure the user running bwrap is not root if we set those caps12:45
*** jamesmcarthur has joined #zuul12:50
*** hashar has quit IRC12:59
*** panda|launch is now known as panda13:05
*** bjackman_ has quit IRC13:06
*** bhavikdbavishi has quit IRC13:10
*** themr0c has joined #zuul13:20
*** themroc has quit IRC13:22
dmsimardFYI ansible 2.8 is still on track for release tomorrow13:27
tobiashcool13:31
mordredtristanC: I think I'd vote for adding the capabilities and ensuring the user isn't root13:53
tristanCmordred: but if it was working without, maybe skopeo needs an option/fix to not create namespace for opendev use case (iiuc, remote docker api shouldn't need client cap)13:55
mordredtristanC: yeah - that too13:55
mordredtristanC: maybe we file a bug with skopeo (and that's a nice reproducer there) - but add the caps to zuul while we wait for it - and we can drop them again later once skopeo is updated. or we can keep them if they reject the bug :)13:57
tristanCmordred: couldn't we rollback the skopeo version installed on the executor nodes?13:58
mordredmaybe that's an easier temporary fix14:00
tristanCmordred: iiuc, SETFCAP may be dangerous, e.g. breakout vector, or creating file that can't be removed...14:00
tristanCthe version before skopeo_0.1.36-1~dev~ubuntu18.04~ppa16_0.1.36-1~dev~ubuntu18.04~ppa17 shouldn't be affected14:01
mordredthat's such a nice version number14:01
mordredtristanC: so 0.1.36-1~dev~ubuntu16.04.2~ppa17 is the bad one, and you're saying kopeo - 0.1.36-1~dev~ubuntu16.04.2~ppa16 should be good?14:05
mordredI shouldn't ask questions about the actual package version not getting updated should I?14:05
tristanCmordred: iiuc yes, ppa16 shouldn't be affected14:05
tristanCmordred: there are no version because it seems like that ppa is continuously packaged, the offending commit isn't tagged (or packaged in fedora/centos) yet14:06
mordredah - makes sense14:08
mordredtristanC: https://review.opendev.org/659296 Pin skopeo to unbreak skopeo+bubblewrap14:08
mordredtristanC: how does that look then?14:08
tristanCmordred: i can't tell how long previous version are available, isn't there a rollback/downgrade command to restore the previous installed version14:10
mordredtristanC: in this case, since it's a PPA, the previous versions are there indefinitely14:10
mordredtristanC: you can look at the 'superseeded' packages:14:11
mordredhttps://launchpad.net/~projectatomic/+archive/ubuntu/ppa/+packages?field.name_filter=skopeo&field.status_filter=&field.series_filter=xenial14:11
tristanCmordred: hum, ppa16 was for bionic, in xenial that number seems affected14:14
tristanCyou need ppa15, at least according to: https://launchpadlibrarian.net/420904271/skopeo_0.1.36-1~dev~ubuntu16.04.2~ppa14_0.1.36-1~dev~ubuntu16.04.2~ppa15.diff.gz14:15
mordredoh good14:15
tristanCi meant, ppa14, bug was introduced in ppa1514:16
mordreddoh14:16
*** jamesmcarthur has quit IRC14:25
clarkbmordred: corvus it is my understanding that 649900 will return opendev's zuul to the state before zaro's gerrit change which we specificially had zaro change in gerrit because we didn't like that old behavior14:47
clarkbin particular users have to remove their approval and reapply it for things to trigger when they recomment14:47
clarkbrather than doing what they mean and treating it as another approval if they haven't removed their vote14:48
clarkb(I've left a -1 comment on the change basically saying we wanted this behavior, got khai to change gerrit, and the old behavior is confusing)14:50
*** gchenuet has joined #zuul15:12
gchenuetHi guys! I noticed that our `/data` partition in executor are growing fast as Zuul keep all Ansible run (code + logs). Is it the correct or a bug ? Should I need to create a crontask ?15:14
clarkbgchenuet: by default zuul should clean that up, there is a debugging flag you can set to keep that data though. Is it possible you have that flag set?15:14
gchenuetclarkb: yes! I have enable the debug flag for a job. Thanks for the reminder.15:19
clarkbgchenuet: I think you may have to cleanup the data that was kept by the debug state manually, but once unset any new jobs should clean up after themselves15:19
*** rfolco|ruck is now known as rfolco15:22
*** panda is now known as panda|ruck15:26
gchenuetclarkb: ok. thanks!15:32
*** themr0c has quit IRC15:52
*** hashar has joined #zuul15:53
openstackgerritClark Boylan proposed zuul/zuul master: Handle artifacts on non change types.  https://review.opendev.org/65932915:54
clarkbtobiash: ^ if you are still around your review on that would be helpful15:54
clarkb(you wrote the earlier change that caused the issue so I'm guessing you understand the problem space a bit better than I do)15:54
tobiashclarkb: I wrote a similar fix today, I'll look in a bit16:08
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/  https://review.opendev.org/60440416:11
tobiashclarkb: we also had another run_handler exception problem today: https://review.opendev.org/65926216:15
tobiashmordred: ^16:15
tobiashclarkb: lgtm, but we definitely should test this16:18
clarkbya I wanted to make sure I wasn't missing something important first16:18
clarkbI can work on tests in a bit16:18
tobiashclarkb: do you need assistance with the test?16:18
clarkbmaybe? I was going to look at the test you wrote for the earlier change and udpate it to process tag events16:19
clarkbtobiash: I do need to step away from the computer for a bit and eat braekfasty16:20
clarkbso if you would like to add a test more quickly than I can feel free16:20
tobiashI have to grab dinner first as well ;)16:21
clarkbactually it might be easy to make this test so I'll try a quick stab at it16:25
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/  https://review.opendev.org/60440416:25
*** hashar has quit IRC16:31
*** hashar has joined #zuul16:32
*** hashar has quit IRC16:52
*** flaper87 has joined #zuul17:10
*** gchenuet has quit IRC17:13
flaper87clarkb: o/ if you're around and have like 5mins to help me out, I'd super appreciate it. If you wonder why I'm pinging you directly is because someone told me that you're very good at debugging zuul deployments :D17:20
flaper87I'm running the docker-compose example, slightly modified, and I'm seeing this in the scheduler: https://paste.fedoraproject.org/paste/BQlF9VexjUvrkWWnAbKtgw17:20
flaper87I'm trying to point zuul to github instead of using the local gerrit (That's the slightly modified part)17:21
flaper87I honestly have no idea where to look :D17:21
nickx-intelcan you please paste your ../zuul/.gitreview  flaper87 ?17:22
nickx-intelI don't know if/how zuul works with github but whatever config you have in there is likely relevant17:23
pabelangerflaper87: if you are using github, it looks like your merger might not be able to communicate with it. I'd check merger-debug.log for exception17:23
pabelangeryou might have wrong credentials17:23
flaper87pabelanger: that is entirely possible. I created the webhook token but I'm not sure if it's correct. :/17:23
flaper87pabelanger: lemme look17:24
flaper87pabelanger: where can I find those logs in the docker-compose example?17:24
clarkbflaper87: docker logs $containername17:25
clarkbflaper87: and docker ps -a will give you the containers it is running17:25
flaper87yeah, I meant in what container17:25
flaper87:)17:25
flaper87I don't see a merger container17:25
pabelangerdo we run a dedicated merger for docker-compose17:25
pabelangeryou might want to look in zuul-exeuctor too17:25
flaper87I see a bunch of failures saying the executor couldn't talk to the schedule.17:27
flaper87lemme check if I messed something up17:27
flaper87oh, yeha, I see the GH error now17:28
pabelangercool17:29
flaper87pabelanger: atal: could not read Username for 'https://github.com': No such device or address'17:29
* flaper87 digs into this17:30
pabelangerflaper87: did you setup your zuul.conf like: https://zuul-ci.org/docs/zuul/admin/github_setup.html17:30
pabelangerthat's for using a github app17:30
flaper87pabelanger: I didn't use apps17:30
flaper87do I have to use GH apps?17:30
flaper87I just created a token17:31
clarkbtobiash: there is a bug in my chagne (caught by tests so that is good) I have a working test for the tag case but the existing test fails17:31
tobiashcool17:31
flaper87pabelanger: token is the issue17:34
flaper87thanks17:34
pabelangercool17:34
pabelangerflaper87: I think there is more coverage on github apps, but non github apps should work17:34
clarkbtobiash: ok I think I see the issue, it is the addition of the branch info17:35
tobiashah ok17:36
openstackgerritClark Boylan proposed zuul/zuul master: Handle artifacts on non change types.  https://review.opendev.org/65932917:38
clarkbtobiash: ^ there that should pass tests now and actually test the change. I've put a TODO in there about how things get weird in a particular spot (basically we assume changes for artifacts in the test suite)17:39
*** mattw4 has joined #zuul17:40
clarkbwhere did that .keep file come from?17:42
clarkbwhy does that file keep causing problems for us?17:44
clarkbI have ahd to untangle it ~3 times now17:44
openstackgerritClark Boylan proposed zuul/zuul master: Handle artifacts on non change types.  https://review.opendev.org/65932917:45
tobiashclarkb: the yarn thing deletes it17:45
clarkblatest patchset adds it back in again17:45
*** mattw4 has quit IRC17:45
clarkbshould be ready for review17:46
tobiashthe keep file is there so the target dir exists in git, but yarn deletes the content of that dir during build17:46
tobiashlgtm17:46
clarkbcan we maybe add the favico.ico instead as that should be fairly static and yarn will put it back again?17:46
pabelangeris it same to recheck zuul failures or is docker job still failing?17:46
tobiashclarkb: maybe we can on demand the dir instead before running yarn17:47
clarkbpabelanger: the tls'd mirrors is still a wip17:47
clarkbpabelanger: if you'd like to help review changes I can dig them up for you17:47
clarkbhttps://review.opendev.org/#/c/658281/ and its children17:48
pabelangerclarkb: sorry, I mean the current zuul-upload-image failures, is that related to tls mirrors?17:48
clarkbpabelanger: I believe the issues we have been having all stem back to trying to use shared NAT for ipv4 to talk to dockerhub via the buildset registry17:49
tobiashpabelanger: I at least saw no successful build yet17:49
clarkbbut i haven't looked again today to see if there are new problems17:49
pabelangerhttp://logs.openstack.org/49/649949/1/gate/zuul-upload-image/a8b9625/job-output.txt.gz#_2019-05-15_13_06_50_88041917:49
pabelangeropen /etc/subuid: no such file or directory17:49
pabelangerchecking backscroll now17:49
tobiashclarkb: there was/is a skopeo update that now requires privileges17:49
clarkbtobiash: oh fun17:49
pabelangerokay17:50
clarkbI jumped right into fixing the pipelines issue I didn't realize there was another issue17:50
pabelangerI see https://review.opendev.org/65929617:50
pabelangerthat is to pin17:50
pabelangerhttps://review.opendev.org/659218/ looks to also be related17:50
pabelangerbut would need a release of zuul17:50
clarkbso fix process is rollback skopeo, merge zuul side fix, release that, then we can update skopeo again?17:51
tobiashpabelanger: tristanC pointed out that https://review.opendev.org/659218 won't fix it unfortunately17:51
*** hashar has joined #zuul17:52
*** hashar is now known as hasharAway17:52
tobiashrelated backscroll: http://eavesdrop.openstack.org/irclogs/%23zuul/%23zuul.2019-05-15.log.html#t2019-05-15T12:36:3017:53
tobiashwhy do we run skopeo on the executor?17:54
pabelangertobiash: ah, I see that now17:54
pabelangertobiash: not sure, maybe to reduce the cost of using nodepool?17:54
pabelangerI have a recheck, +3 on 659296 now17:57
tobiashclarkb: did i understand correctly that the ipv6 problems required to run skopeo on the executor combined with socat magic?18:03
pabelangerokay, looks like some fallout to puppet breaking something in system-config, so zuul will be blocked for a bit, unless we flip job non-voting18:03
clarkbtobiash: yes though that was a different ipv6 problem than the current one18:03
*** rlandy is now known as rlandy|brb18:12
*** ianychoi has quit IRC18:42
*** hasharAway is now known as hashar18:45
*** rlandy|brb is now known as rlandy18:45
*** mattw4 has joined #zuul19:09
*** mattw4 has quit IRC19:13
*** dkehn has joined #zuul19:19
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883519:56
*** markwork has quit IRC19:59
*** pcaruana has quit IRC20:14
*** rlandy has quit IRC20:15
*** rlandy has joined #zuul20:16
*** panda|ruck has quit IRC21:10
*** panda has joined #zuul21:14
*** hashar has quit IRC21:43
*** jlk has quit IRC21:51
*** jlk has joined #zuul21:53
*** mattw4 has joined #zuul21:56
*** mattw4 has quit IRC22:03
*** rlandy is now known as rlandy|bbl22:18
nickx-intelI have a question ..... as I'm looking at zuul-jobs, this syntax seems pretty arcane22:28
nickx-intel.. zuul:rolevar:: zuul_output_dir22:28
nickx-intel   :default: {{ ansible_user_dir }}/zuul-output22:28
nickx-intelI'm gathering that {{ zuul_output_dir }} is actually the syntax22:29
clarkbnickx-intel: those are python sphinx rst annotations. They make the rendered output prettier22:31
clarkbhttps://zuul-ci.org/docs/zuul/developer/docs.html#zuul-sphinx-directives is where they are documented22:32
clarkbnow to find a link to the rendered docs for zuul_output_dir22:32
clarkbhttps://zuul-ci.org/docs/zuul-jobs/log-roles.html#rolevar-ensure-output-dirs.zuul_output_dir there22:33
nickx-intelhmmm .... thanks clarkb .... I'm just swimming in this right now22:38
*** mattw4 has joined #zuul23:01
*** mattw4 has quit IRC23:21
*** panda has quit IRC23:30
*** panda has joined #zuul23:31

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!