Tuesday, 2019-12-17

*** mattw4 has quit IRC00:07
pabelangercorvus: thanks! Confirmed it worked as expected00:14
pabelanger(zuul.artifacts)00:14
*** armstrongs has quit IRC00:38
*** smyers has quit IRC00:50
*** rlandy|bbl is now known as rlandy00:58
openstackgerritClark Boylan proposed zuul/zuul-jobs master: DNM test all the things on ansible 2.9 with default python  https://review.opendev.org/69834401:01
clarkbpabelanger: corvus I think something like ^ is a first step to evaluating if ansible 2.9 can be a default01:02
*** jamesmcarthur has joined #zuul01:53
*** jamesmcarthur has quit IRC02:02
*** rlandy has quit IRC02:08
*** bhavikdbavishi has joined #zuul02:14
*** Goneri has quit IRC02:21
*** bhavikdbavishi has quit IRC02:33
*** jamesmcarthur has joined #zuul02:38
*** rfolco|bbl has quit IRC02:46
*** jamesmcarthur has quit IRC02:54
*** bhavikdbavishi has joined #zuul03:21
*** jamesmcarthur has joined #zuul03:54
*** jamesmcarthur has quit IRC04:26
*** raukadah is now known as chkumar|rover05:33
openstackgerritMerged zuul/nodepool master: Dockerfile: install sudo for nodepool-builder  https://review.opendev.org/69470906:26
openstackgerritMerged zuul/nodepool master: Dockerfile: add DEBUG environment flag  https://review.opendev.org/69484506:27
*** jamesmcarthur has joined #zuul06:28
*** jamesmcarthur has quit IRC06:33
*** jcapitao has joined #zuul07:25
*** avass has joined #zuul07:32
*** pcaruana has joined #zuul07:52
*** tosky has joined #zuul08:20
*** saneax has joined #zuul08:22
*** hashar has joined #zuul08:36
*** jpena|off is now known as jpena08:49
*** themroc has joined #zuul08:49
*** clayg_ has joined #zuul09:03
*** ChrisShort_ has joined #zuul09:03
*** dcastellani_ has joined #zuul09:04
*** tosky_ has joined #zuul09:04
*** zxiiro_ has joined #zuul09:04
*** klindgren_ has joined #zuul09:04
*** shanemcd- has joined #zuul09:09
*** amotoki_ has joined #zuul09:09
*** tosky has quit IRC09:10
*** klindgren has quit IRC09:10
*** shanemcd has quit IRC09:10
*** AJaeger has quit IRC09:10
*** SotK has quit IRC09:10
*** ChrisShort has quit IRC09:10
*** zxiiro has quit IRC09:10
*** dcastellani has quit IRC09:10
*** amotoki has quit IRC09:10
*** clayg has quit IRC09:10
*** clayg_ is now known as clayg09:12
*** ChrisShort_ is now known as ChrisShort09:12
*** zxiiro_ is now known as zxiiro09:12
*** dcastellani_ is now known as dcastellani09:12
*** SotK has joined #zuul09:12
*** AJaeger has joined #zuul09:16
*** tosky_ is now known as tosky09:19
*** sshnaidm|afk is now known as sshnaidm09:34
*** bhavikdbavishi has quit IRC09:53
*** hashar has quit IRC10:24
*** bhavikdbavishi has joined #zuul10:31
*** jcapitao is now known as jcapitao|afk11:21
Shrewsianw: him, https://review.opendev.org/693464 pretty much doubles the nodepool test run time because of the opendev-buildset-registry job11:35
Shrewss/him/hrm/11:35
*** ianychoi has quit IRC11:38
Shrewsianw: why do we need that job and the dependency on it for the new jobs? can we not use the current intermediate registry?11:40
*** ianychoi has joined #zuul11:40
ShrewsI left a question inline on the review.11:55
*** rfolco|bbl has joined #zuul11:57
*** amotoki_ is now known as amotoki12:12
openstackgerritMerged zuul/nodepool master: Also build sibling container images  https://review.opendev.org/69739312:19
*** rfolco|bbl is now known as rfolco12:27
*** jpena is now known as jpena|lunch12:38
*** armstrongs has joined #zuul12:45
*** jcapitao|afk is now known as jcapitao12:49
*** armstrongs has quit IRC12:54
*** jamesmcarthur has joined #zuul12:56
*** sshnaidm is now known as sshnaidm|afk12:58
*** rlandy has joined #zuul12:59
*** jamesmcarthur has quit IRC13:02
*** jamesmcarthur has joined #zuul13:08
mordredShrews: the buildset-registry and the intermediate registry work in concert with each other - you need the buildset registry when you build or use images.13:17
Shrewsmordred: right, I got that part, but I thought the buildset registry was a system-level thing, not project level?13:17
mordredit's project - but also - that time is going to be misleading13:18
mordredsince the buildset registry job pauses13:18
Shrewsotherwise, how are the current nodepool jobs working?13:18
mordredso it's going to run, then the other jobs that need it are going to run, then it's going to finish13:18
ShrewsI think I fully misunderstand how the pieces fit then13:19
mordredShrews: I'm only on sip one of coffee one - so I'm probably not smart enough to explain it yet13:20
*** bhavikdbavishi has quit IRC13:20
Shrewsaye. I started way too early, too, so I'm blaming lack of caffeine as well13:20
*** bhavikdbavishi has joined #zuul13:20
Shrewsalthough I've had two cups at this point... there may not be enough caffeine in my house13:21
mordredShrews: the tl;dr is that the jobs themselves that do things with speculative container images always want to talk to the buildset registry13:21
mordredthe buildset registry jobs will grab image content as appropriate from the intermediate registry and put it in the buildset registry so that a request for, say, docker.io/opendevorg/nodepool actually comes from the buildset registry and returns the correct image content13:22
fungia job starts which sets up the buildset registry server running from a job node, then pauses. another job starts and configures docker or whatever to connect to the registry running on the node for the paused job. when that second job finishes, the buildset registry job resumes and cleans up, then exits as well13:22
mordredthe intermediate registry is where built but not published image content goes - but it goes there tagged with its change info so that a buildset registry can pull it and rename it apporiately13:22
mordredif you do image jobs without a buildset registry, the content you're going to work with is whatever that job itself built and any other images will be whatever is in dockerhub13:24
*** jamesmcarthur has quit IRC13:31
ShrewsI think the bit about the buildset registry being a project level thing helps to reconfigure my brain13:34
*** bhavikdbavishi has quit IRC13:35
*** jpena|lunch is now known as jpena13:41
Shrewsand after re-re-rereading https://zuul-ci.org/docs/zuul-jobs/docker-image.html#system-architecture, I can see that info is there, just sort of implied.  *sigh*13:42
*** jamesmcarthur has joined #zuul13:42
mordredShrews: this is all a super new concept  that only exists in zuul ... we should probably improve that doc if it was confusing?13:58
*** sshnaidm|afk is now known as sshnaidm14:02
Shrewsmordred: not sure if it’s the doc so much as it is me being thick. I’m a visual learner for sure.14:02
mordrednod14:03
*** panda has quit IRC14:44
*** panda has joined #zuul14:44
tristanCafter upgrading to zuul-3.13.0, some of our jobs got broken because it seems like ansible changed something with regards to delegated and become:true synchronize task which now fails with permission error: https://softwarefactory-project.io/r/#/c/17241/15:10
AJaegertristanC: corvus pointed out https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/use-buildset-registry/tasks/main.yaml#L96-L103 - might be related to your problem...15:13
openstackgerritAlbin Vass proposed zuul/nodepool master: 'keys' must be defined for host-key-checking: false  https://review.opendev.org/69802915:19
tristanCAJaeger: maybe... well pinning ansible-version fixed the issue for now.15:27
corvusoh we haven't done the 2.5 removal / 3.14 release yet...15:29
corvuswe should do that today15:29
corvusi think the issue with delegate+synchronize sounds new to me15:32
corvusi'm not sure how else to write that :/15:32
corvustristanC: it might be interesting to try it with 2.9.  i have no reason to think that would fix it, but it would provide more data15:33
tristanCcorvus: alright, i'm trying 2.9 with https://softwarefactory-project.io/r/1724615:37
*** michael-beaver has joined #zuul15:45
*** jamesmcarthur has quit IRC15:47
*** jamesmcarthur has joined #zuul15:48
*** chkumar|rover is now known as raukadah15:50
*** saneax has quit IRC15:52
*** jamesmcarthur has quit IRC15:53
*** jcapitao is now known as jcapitao|afk15:56
*** jamesmcarthur has joined #zuul15:59
pabelangertristanC: which version of ansible are you upgrading from?16:03
pabelangermaybe try copy, see how that works16:03
openstackgerritClark Boylan proposed zuul/zuul-jobs master: DNM test all the things on ansible 2.9 with default python  https://review.opendev.org/69834416:04
tristanCpabelanger: we went from zuul-3.11 to zuul-3.1316:06
pabelangertristanC: sorry, ansible side16:06
tristanCpabelanger: corvus: setting ansible-version to 2.9 didn't work either. pinning to 2.6 does work16:07
pabelangeransible 2.9 really isn't tested at this point, so I'd expect some playbook issues16:08
clarkbtristanC: we did see other changes with become and become_user between 2.7 and 2.816:08
clarkb2.8 doesn't let you become on include roles now16:08
tristanCpabelanger: well the job didn't had an ansible-version set, we had to pin it to 2.6 to make it work again16:08
clarkbwould not surprise me if there were other updates around that16:08
clarkbpabelanger: see 698344 :)16:08
clarkbtrying to get some testing done16:08
tristanCclarkb: here is the affected task: https://softwarefactory-project.io/logs/46/17246/1/check/sf-tenants/bff3e43/ara-report/file/00a68911-aa1b-48b8-9e2d-fc113d79a7c3/#line-116:08
pabelangertristanC: k, so default version of ansible in 3.11 is 2.7 IIRC16:09
pabelangerclarkb: yah, I was going to suggest uplifting all of zuul tenant to 2.9, if basic smoke test worked. But haven't had time to do that yet16:10
clarkbtristanC: https://opendev.org/openstack/openstack-helm-infra/src/branch/master/roles/deploy-docker/tasks/deploy-ansible-docker-support.yaml#L26-L49 is the thing we found as no longer working16:10
pabelangerclarkb: no, you can do become on include_role, you just need to use apply parameter16:11
tristanCclarkb: that seems like another issue, in our case there is no block or import propagation, become and delegate are set to the task directly16:11
pabelangerclarkb: https://docs.ansible.com/ansible/latest/modules/include_role_module.html16:11
pabelangerwas added in ansible 2.716:11
clarkbpabelanger: ok. The error message totally doesn't say that fwiw16:11
clarkbit says become is not valid on includerole16:12
pabelangeryah, if you move into apply param, it should work16:12
pabelangerlet me check porting guide16:12
clarkbright, maybe ansible should say "become is not valid on icnludrole. Use apply parameter16:12
clarkbinstead of "this isn't possible"16:12
pabelanger+116:12
pabelangerwow, isn't in porting guide16:13
pabelanger;(16:13
pabelangerhttps://docs.ansible.com/ansible/latest/porting_guides/porting_guide_2.7.html#include-role-and-import-role-variable-exposure is only real notes16:14
tristanCor we could have use deprecation warning too16:14
clarkbre 2.9 looks like we have some failures from zuul-jobs tests. We'll probably want to make that green as step 0 in 2.9 by default16:14
pabelangerthat reminds me, it would be nice if we bubbles up deprecated warnings into console for zuul16:14
pabelangertoday, you can only see them in debug logs on executor16:14
clarkb"msg": "value of state must be one of: absent, build-dep, fixed, latest, present, got: installed" small things like that. I'll try to get some changes up after breakfast16:15
pabelangeroh16:15
pabelangeryah, that is deprecated now16:15
*** themroc has quit IRC16:15
pabelangers/installed/present16:15
pabelangerthat is actually a good example of zuul user doesn't see that warning in playbook runs16:16
pabelangerbut if you look in executor debug logs, you will see it16:16
clarkbits also a good example of a change that is user facing but doesn't really help users :/16:16
pabelangerIMO, we should add that warning to zuul console logs too16:16
pabelangerwell, to be fair, the warning has been around since 2.7 I think16:17
clarkbsure, but how does it help users?16:17
clarkbprseent and installed can be equivalent16:17
clarkbjust accept both since you have accepted one or the other already16:17
pabelangeryah, you got me on that. It was core who removed it16:17
pabelangerclarkb: expect a lot of fall out in ansible 2.10.0, that is what I am dealing with now16:18
clarkbpabelanger: note windmill uses a lot of state:installed16:22
clarkbhttp://codesearch.openstack.org/?q=state%3A%20installed&i=nope&files=&repos= the list isn't as large as I feared16:22
clarkb(for things we host)16:23
*** jcapitao|afk is now known as jcapitao16:23
pabelangerclarkb: yah, I've slowly been migrating them16:23
pabelangerhaven't fixed them all yet16:23
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Use present for package state instead of installed  https://review.opendev.org/69945016:24
pabelangereg https://review.opendev.org/675339/16:24
corvusis present supported in 2.5?16:31
corvusyes16:31
corvushttps://docs.ansible.com/ansible/2.5/modules/package_module.html16:32
corvuswow, installed isn't even documented as far back as 2.516:32
clarkbya I checked the docs but didn't test it16:33
clarkbis that something a linter should be catching?16:34
clarkb(these are the things I actually want linting to warn me about)16:34
pabelangeransible-review was to add the ability for custom hooks16:35
pabelangerbut not sure what happened to that tool16:35
openstackgerritMerged zuul/nodepool master: Add container-with-siblings functional test  https://review.opendev.org/69346416:37
*** jpena is now known as jpena|brb16:45
*** jamesmcarthur has quit IRC16:48
*** jamesmcarthur has joined #zuul16:48
*** jamesmcarthur has quit IRC16:53
*** jamesmcarthur has joined #zuul16:53
*** jcapitao is now known as jcapitao|afk16:54
*** jamesmcarthur has quit IRC16:56
SpamapSYesterday whilst walking the dog I had an epiphany. The #1 reason my devs bounce off Zuul is Ansible complexity. I'm really good with Ansible, but ultimately, it's not a pleasing language.  So.. what I wish we had, was a really well thought out set of roles that do really basic stuff, so that most users could construct most tests entirely from playbooks....16:59
SpamapSI know that seems obvious, but.. right now I feel that zuul-jobs is really focused on enabling tools, not necessarily on enabling devs to grok.17:00
SpamapSAnyway, just a thought.17:00
SpamapSOh also, more simple examples in the docs. :)17:01
clarkbSpamapS: re enabling tools you mean "here is the tox role" and "here is the go test" role?17:02
corvusSpamapS: i agree, and i think it's slowly going in the right direction (see recent addition of 'go' jobs).  but i would also hope that before someone bounces off of zuul because of complexity, they might at least try https://zuul-ci.org/docs/zuul-jobs/general-jobs.html#job-run-test-command  whenever i talk about zuul and ansible, i try to always mention "if you don't like ansible, just tell ansible to run17:03
corvuswhatever you do like"17:03
clarkbmordred: is SELECT COUNT(*) from zuul_build WHERE end_time BETWEEN '2019-01-01 00:00:00' AND '2019-12-31 11:59:59'; how you would count the number of builds done by a zuul install over the last year?17:03
clarkbmordred: this would be for zuuls that don't have a statsd or other counting mechanism17:04
SpamapScorvus: yeah, more roles and generic jobs like that. :)17:05
openstackgerritMerged zuul/zuul master: Remove support for ansible 2.5  https://review.opendev.org/65043117:06
mordredclarkb: yah, that looks right (assuming between is an operator17:08
Shrewsit is17:09
clarkbtyty17:12
clarkbcorvus: SpamapS: maybe we should make the "have ansible run whatever you prefer" job as a prominent example in the zuul documentation (not just the zuul-jobs docs)17:13
corvus++17:13
fungiyes, that definitely cries out for an example17:19
*** Goneri has joined #zuul17:20
fungii agree saying "zuul can have very simple jobs, here's a pile of ansible documentation to read" is probably a tad unhelpful ;)17:21
openstackgerritMerged zuul/zuul-jobs master: Use present for package state instead of installed  https://review.opendev.org/69945017:22
*** jpena|brb is now known as jpena17:23
dmsimardclarkb: I had a similar query for build time (instead of build count): SELECT job_name, result, start_time, end_time, TIMEDIFF(end_time, start_time) as duration FROM zuul_build WHERE start_time BETWEEN '2019-01-01 00:00:00' AND '2019-01-31 23:59:59'"17:31
*** jamesmcarthur has joined #zuul17:31
dmsimardalso noticed your query has 11:59:59 instead of 23:59:5917:33
clarkbdmsimard: ya I noticed that too but 12-31 is 2 weeks away so good enough for now :)17:33
*** jamesmcarthur has quit IRC17:33
dmsimardsure :p17:33
*** jamesmcarthur has joined #zuul17:34
*** mattw4 has joined #zuul17:38
*** michael-beaver has quit IRC17:55
corvusSpamapS: any thoughts about where in the docs to convey this wisdom?18:02
fungihttps://zuul-ci.org/docs/zuul/user/jobs.html already has some examples for slightly more esoteric tasks18:09
fungii wonder if a basic example would fit in there fairly early18:10
tristanCfwiw i like the documentation layout introduced here: https://www.divio.com/blog/documentation/18:13
tristanCe.g. move most of the existing content to a reference category, and pull the few tutorial/how-to to a top level category easier to find18:15
*** rlandy has quit IRC18:15
*** rlandy has joined #zuul18:16
*** jpena is now known as jpena|off18:33
*** pcaruana has quit IRC18:38
corvustristanC: sounds great18:47
*** panda has quit IRC18:49
*** panda has joined #zuul18:49
openstackgerritClark Boylan proposed zuul/zuul-jobs master: Fix ansible use of filters and tests  https://review.opendev.org/69947818:50
clarkbmore 2.9 fixes ^ should be backward compat to 2.518:50
*** jamesmcarthur has quit IRC18:51
SpamapScorvus: I've actually been thinking we need a "Getting Started as a user" section.19:12
SpamapStristanC: yeah, that sounds good19:12
SpamapSour docs right now are definitely in the "power user" category19:12
corvusif someone wants to propose an outline, that would be great.  that's probably the best way to get started.19:14
*** pcaruana has joined #zuul19:18
tristanCcorvus: i won't have time for that until next year, but i'm happy to help implement this structure.19:21
tristanCthough it seems like we are a bit short of tutorials, how-tos and discussions content...19:22
tristanCthus the initial outline might looks quite unbalanced19:22
corvuswell, i think the outline should look like we want it to look; if it names a section we don't have any content for, we'll know what to work on19:23
SpamapS+119:26
*** hashar has joined #zuul19:31
SpamapSI've been meaning to carve out a few hours to write just such a tutorial up. Never seem to quite get there though. :P19:44
*** ianw is now known as ianw_pto19:50
*** jamesmcarthur has joined #zuul19:55
*** pcaruana has quit IRC19:59
*** jamesmcarthur has quit IRC20:11
*** jamesmcarthur has joined #zuul20:12
*** jamesmcarthur has quit IRC20:19
*** jamesmcarthur has joined #zuul20:34
*** pcaruana has joined #zuul20:42
openstackgerritMerged zuul/nodepool master: Dockerfile: install nodepool-builder dependencies  https://review.opendev.org/69330620:50
clarkbanyone else able to review https://review.opendev.org/#/c/699478/ fixes some ansible 2.9 support things in zuul-jobs20:51
openstackgerritMerged zuul/nodepool master: Add a container-with-releases functional test  https://review.opendev.org/69881820:58
*** jcapitao|afk has quit IRC21:18
*** jamesmcarthur has quit IRC21:28
*** pcaruana has quit IRC21:41
openstackgerritMerged zuul/zuul-jobs master: Fix ansible use of filters and tests  https://review.opendev.org/69947821:44
*** panda has quit IRC22:07
*** panda has joined #zuul22:10
clarkbansible 2.9 support in zuul-jobs is looking good after ^22:20
corvushow does this message to repo-discuss about setting up zuul for gerrit look?  https://etherpad.openstack.org/p/j5EifZIKD322:36
corvuspaladox, mordred: ^22:36
* paladox looks22:37
clarkbhttps://review.opendev.org/#/c/698344/3 passees testing now. pabelanger for next steps on 2.9 I think we want to search for occurences of the installed vs present issue as well as filters vs tests being used improperly. Then we can probably switch zuul tenant to ansible 2.922:37
corvusclarkb: i love that change22:38
corvusclarkb, fungi, Shrews: ^ also, as opendev folks with much zuul experience, how does that look as a greenfield deployment of zuul in gitops?22:39
clarkbcorvus: the major question I have after reading that is how does the zuul/ops config get into the google k8s22:40
clarkbmaybe that is getting too far ahead of baby steps but I doubt you want to be running that by hand often22:40
paladoxcorvus +1 (looks good to me!)22:41
corvusclarkb: yeah, i expect by hand at first, and by zuul asap :)22:41
corvus(probably some sort of throwaway GCE vm where we check it out and run kubectl apply or something)22:42
corvusbut a zuul job which checks it out and uses a zuul secret to run kubectl apply shouldn't be too hard to do22:42
corvus(that zuul secret could be for a google cloud service account with creds to do that)22:43
corvusthat it should be that simple is one of the main draws of running the control plane in k8s22:43
corvusclarkb: i added a little bit on line 62 about that22:44
clarkblgtm22:44
clarkbcorvus: and I guess you won't bother with image builds to start?22:56
corvusclarkb: yeah22:56
clarkb(that would depend on ianw's nodepool-builder container work to run in k8s and probably need privileged containers?)22:56
corvuswell, executors need privileged containers anyway22:57
clarkboh right22:57
clarkbor user namespacing22:57
corvus(which is the default in gke)22:57
clarkbI think on opensuse you don't need privileged containers to run the executor22:57
clarkbbut on centos you do22:57
clarkbyou'll also want to be careful to toggle th gke security settings such that zuul and nodepool don't have admin access to the google cloud account22:58
clarkbI think they may do that by default now22:58
clarkbbut was not default for a long time22:58
fungicorvus: just read the draft e-mail and it makes sense to me, even if i pretend to have a lot less context about how zuul works23:03
*** saneax has joined #zuul23:03
*** sgw has quit IRC23:19
*** hashar has quit IRC23:20
*** avass has quit IRC23:31
mnaserFWIW: I am currently in the process of writing helm charts for zuul...23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!