Monday, 2020-01-20

*** armstrongs has joined #zuul00:02
*** armstrongs has quit IRC00:12
mordredtobiash: I think that's nicer to read00:19
*** smcginnis|PTO has quit IRC01:51
*** smcginnis|PTO has joined #zuul01:52
*** bhavikdbavishi has joined #zuul03:31
*** bhavikdbavishi1 has joined #zuul03:36
*** bhavikdbavishi has quit IRC03:38
*** bhavikdbavishi1 is now known as bhavikdbavishi03:38
*** saneax has joined #zuul04:26
*** raukadah is now known as chandankumar04:51
*** evrardjp has quit IRC05:34
*** evrardjp has joined #zuul05:34
openstackgerritSimon Westphahl proposed zuul/nodepool master: Cleanup exception logging in static provider  https://review.opendev.org/70282806:14
*** themroc has joined #zuul06:46
*** tosky has joined #zuul08:17
*** jpena|off is now known as jpena08:50
*** arxcruz|off is now known as arxcruz09:22
*** jhesketh has quit IRC09:32
*** jhesketh has joined #zuul09:34
*** sgw has quit IRC09:40
*** sgw has joined #zuul09:53
*** AJaeger has quit IRC10:05
openstackgerritAntoine Musso proposed zuul/zuul master: doc: add links to components documentation  https://review.opendev.org/70310510:40
arxcruzhello guys, I need to run an ansible-role in the post10:54
arxcruzwhat's the best way to add the role in zuul ansible path?10:55
arxcruzif i add roles: zuul: openstack/myrole in job definition will it work ?10:55
zbr|droverarxcruz: has a valid question, i wondered this myself because "roles" in job config make zuul run the roles, so you cannot use it make a role available for later using in post-run10:58
zbr|droveri also got a weird error trying to build zuul docs locally, http://paste.openstack.org/show/788593/11:09
arxcruzcorvus: ^11:10
arxcruzfungi: ^11:10
reiterativeI'm trying to get a basic zuul setup working on AWS, using the quick start tutorial example as a reference, but my jobs are failing to load the general roles (e.g. prepare-workspace, upload-logs) from zuul/zuul-jobs. I can see via the web interface that the zuul-jobs repo from opendev  has been loaded, and confirmed this by running zuul-scheduler with debug on. I've also tried running zuul-executor in debug mode, but could not see any clues as to why11:38
reiterativeit isn't working. Can anyone suggest what else I can do to debug this?11:38
openstackgerritSorin Sbarnea proposed zuul/zuul master: WIP: docs: improve job.role documentation  https://review.opendev.org/70337211:39
*** AJaeger has joined #zuul11:56
openstackgerritSorin Sbarnea proposed zuul/zuul master: docs: improve job.role documentation  https://review.opendev.org/70337212:00
*** mnaser has quit IRC12:27
*** adam_g has quit IRC12:27
*** mnaser has joined #zuul12:27
*** adam_g has joined #zuul12:28
*** rfolco has joined #zuul12:29
zbr|droverarxcruz: added your question at https://stackoverflow.com/q/59822978/99834 as it seems to be of general interest12:29
*** jpena is now known as jpena|lunch12:35
arxcruzzbr|drover: thanks12:39
arxcruzperhaps jpena|lunch  knows12:39
*** fbo|off is now known as fbo12:56
*** rfolco is now known as rfolco|bbl12:56
*** avass has joined #zuul12:58
*** rlandy has joined #zuul13:00
*** electrofelix has joined #zuul13:01
*** electrofelix has quit IRC13:01
*** smcginnis|PTO is now known as smcginnis13:09
*** jpena|lunch is now known as jpena13:30
jpenaarxcruz: I see https://zuul-ci.org/docs/zuul/reference/job_def.html#attr-job.roles which should match your need, but I'm not sure if I understood the question13:59
zbr|droverjpena: ins't https://stackoverflow.com/q/59822978/99834 clear enough?14:06
jpenazbr|drover: I didn't see the example when I first checked14:06
zbr|droveri added it few seconds ago14:07
jpenathat's weird, the zuul docs do not imply that the role will be auto-run14:07
zbr|droveri think we should start using examples in the docs, lots of them.14:07
jpenaagreed14:07
zbr|droverfirst sentence from https://zuul-ci.org/docs/zuul/reference/job_def.html?highlight=job#attr-job.roles makes me believe zuul will run the role14:08
zbr|droverif adding roles there only makes them available, is ok. arx will know what to do.14:09
zbr|droverand I have to update my doc patch to make this clear.14:09
zbr|droveri wonder what "prepared and installed" really means14:10
zbr|droveruse of prepare there is really confusing, especially because there is a pre-run.. which translated to "prepare-run"14:10
zbr|droverplease review https://review.opendev.org/#/c/703372/14:15
*** rfolco|bbl is now known as rfolco14:16
zbr|droveri was also not able to run "tox -e docs" on zuul, tried macos and centos-8, choked on both with http://paste.openstack.org/show/788593/14:20
openstackgerritSorin Sbarnea proposed zuul/zuul master: bindep: fixed wrong dep names on rpm platform  https://review.opendev.org/70340314:30
tobiashadding the roles there exactly makes them available to ansible, no role is automatically run14:31
tobiashzuul executes exactly the pre, run, post and cleanup playbooks14:31
tobiashif those playbooks don't use a role, it's not executed14:31
zbr|droverso cool, https://pypi.org/project/gear/#files does not even have a link to the source code, lucky that I know where it should be.14:34
zbr|droverproject homepage links to the pypa itself, not really useful :D14:35
*** zxiiro has joined #zuul14:36
zbr|drovertobiash: thanks for the confirmation, I will update the doc.14:39
fungizbr|drover: are you able to run `zuul-manage-ansible -h` yourself from the venv? does it give an error when you do that (in addition to a nonzero exit code)?14:42
fungi(sphinx is configured to capture the output of that command to incorporate into the generated documentation)14:43
fungilike `/Users/ssbarnea/c/rdo/zuul/.tox/docs/bin/zuul-manage-ansible -h` maybe14:43
zbr|droverfungi: i managed to narrow down to https://opendev.org/opendev/gear/src/branch/master/gear/__init__.py#L2732-L273514:44
zbr|drovernone of these are defined on macos, and at least first one neither on centos.14:45
zbr|droverand this code runs even when trying to build the docs....14:45
zbr|droversadly i have no knowledge of gear, so not sure with what it would be acceptable to replace them wiht14:45
openstackgerritSimon Westphahl proposed zuul/nodepool master: Handle event id in node requests  https://review.opendev.org/70340614:46
openstackgerritSimon Westphahl proposed zuul/nodepool master: Centralize logging adapters  https://review.opendev.org/70340714:46
fungiyeah, sounds like sockets on the macos mach kernel lack epoll?14:46
Shrewsos x has poll and kqueue14:46
fungiwe might be able to improve zuul-ansible-manage so that it doesn't result in gear being force-imported in that code path14:46
fungii can't imagine gear itself being needed for that utility14:47
fungialternatively, maybe someone wants to make gear work on macos14:47
zbr|droveryeah, i wonder what happens if put a fallback to set all of them to zero14:47
zbr|droveri have no plans to write services for macos, i only want to build zuul docs, and hopfully running some unittests.14:48
zbr|droverfunny part that i get the same error on centos, too.14:48
Shrewsfungi: https://review.opendev.org/67167414:48
zbr|droverShrews: super! testing locally now.14:51
zbr|drovercan we merge ^ and release it?14:56
fungizbr|drover: as in `zuul-ansible-manage -h` exits 1 on centos too?14:56
fungithat i find more surprising14:56
openstackgerritSimon Westphahl proposed zuul/nodepool master: Make flake8 config compatible with latest version  https://review.opendev.org/70341014:58
tristanCcorvus: mnaser: mordred: pabelanger: clarkb: tobiash: Shrews: here are my notes about how I implemented the zuul-operator and how it can be extended by our sf-operator: https://etherpad.openstack.org/p/zuul-operator-dhall14:59
zbr|droverfungi: i think i seen it failing too there but probably for different reason.14:59
zbr|droverbut on macos that patch is fixing the problem14:59
zbr|droverwe need to make the stderr visible because when it fails now, you do not see any error.15:00
zbr|droveri had to enter the venv and run the command manually to see what it was producing15:00
tristanCfungi: ^ (sorry i forgot you were in the call too)15:01
fungizbr|drover: well, the error says what command failed, so presumably it can be executed directly to get the stdout/stderr if needed, but i don't see any problem with echoing it if there's some way to get that sphinx extension to do so15:02
zbr|drovercentos error is unrelated, pip._vendor.pkg_resources.ContextualVersionConflict: (urllib3 1.25.7 (/home/ssbarnea/rmux/zuul/.tox/docs/lib/python3.6/site-packages), Requirement.parse('urllib3<=1.25.3'), {'zuul'}15:02
*** johanssone has quit IRC15:03
zbr|droveri will change the reqs15:04
fungiahh, pip is embedding a newer urllib3 than zuul is pinned to15:04
funginote in requirements.txt refers to that pin being a workaround for https://github.com/urllib3/urllib3/pull/168415:04
zbr|drovercorrect it would be >=1.25.515:04
fungior at least !=1.25.415:06
fungiare you sure 1.25.5 included the fix?15:07
openstackgerritSorin Sbarnea proposed zuul/zuul master: Unlock urllib pinning  https://review.opendev.org/70341415:07
fungizbr|drover: it doesn't15:09
fungigit tag --contains 9167b58128dbfe3ddcbab253166697348d8d364c15:09
fungiwe need !=1.25.4,!=1.25.515:09
fungii'll comment in the review15:09
*** johanssone has joined #zuul15:09
zbr|droverfungi: you preffer not instead of >= ?15:09
fungiyes, because zuul still works on platforms with older urllib315:11
openstackgerritSorin Sbarnea proposed zuul/zuul master: Unpin urllib  https://review.opendev.org/70341415:11
*** armstrongs has joined #zuul15:14
*** chandankumar is now known as raukadah15:15
zbr|drovermy impression is that gear does not work with newer pythons, i got timeout errors running tests. i am going to enable nv jobs.15:21
*** armstrongs has quit IRC15:26
*** bhavikdbavishi has quit IRC15:37
arxcruzzbr|drover: jpena just tested, zuul doesn't run the role, it just install on /var/lib/zuul/build and so be available in pre/post/run15:37
jpenaaha, that makes sense then15:38
*** avass has quit IRC15:53
clarkbyes it is a dependency specification15:54
clarkbit makes the role available to the job15:54
clarkbreiterative: ^ that may actually be what your job needs too15:54
clarkbreiterative: update the base job to include roles from zuul-jobs to make them available in the job context15:54
*** openstackgerrit has quit IRC16:13
Shrewsugh. i did not notice, but we lost our links to the yaml definition files in the doc reference section16:16
Shrewscorvus: i don't think that was intentional, was it? should be easy to add back16:17
corvusShrews: which links?16:17
Shrewsto the new *_def.rst files16:18
Shrewsmissing from the reference/user.rst toc16:18
corvusShrews: i see them at https://zuul-ci.org/docs/zuul/reference/user.html16:18
corvusalso https://zuul-ci.org/docs/zuul/reference/config.html#configuration-items16:18
Shrewsright, but i mean from the root doc page16:18
corvusShrews: i don't think we ever had that.  we agreed it would be nice, but there are complications16:19
corvusoh nope i'm wrong16:19
Shrewscorvus: we had that when I initially added them under the common Reference section16:19
corvusthey are here https://zuul-ci.org/docs/zuul/3.15.0/16:19
Shrewsyeah. got a fix coming16:20
corvusthe tricky thing is that they should be under "user reference > project configuration"16:21
*** rishabhhpe has joined #zuul16:22
Shrewscorvus: yeah, that we can't do as far as i've been able to find16:22
*** tosky has quit IRC16:23
Shrewsbut without them on the root level page, they are hidden yet again  :(16:24
corvusShrews: we could manually create the user reference TOC16:24
Shrewscorvus: how?16:25
corvusShrews: create a hidden tocree, then just make some bullet lists16:26
rishabhhpeHi All , Can anyone help in resolving this issue -: http://paste.openstack.org/show/788608/ during building the image16:26
Shrewscorvus: is there an example of how to do that?16:27
corvusShrews: not that i'm aware of16:28
corvusShrews: the :hidden: flag is doc'd here https://www.sphinx-doc.org/en/master/usage/restructuredtext/directives.html16:30
corvusShrews: we should think really hard about whether that should be on this index though.  i agree it's really useful, but otoh, if this is supposed to be an effective index of many kinds of documentation, having too much detail may not be best.  it might be better to figure out a section title other than "project configuration" that tells people that's what they're looking for.16:35
corvussort of like how i know to go to the library reference to see all the python modules: https://docs.python.org/3/16:35
rishabhhpeclarkb: fungi: can you please help me with this -:  http://paste.openstack.org/show/788608/16:45
tristanCcorvus: Shrews: actually, the django doc doesn't even list the tutorials or howtos from the main toc, perhaps the main index.rst should only contains the 4 categories links (tutorial/howto/discussion/reference) ?16:53
tristanCand within each category, we could increase the toc depth to include nested elements like the project configuration16:54
corvustristanC: zuul is a different kind of application than dango.  it has distinct audiences of end-users and administrators, so our top-level has to accomodate that.16:54
corvusso we currently have 6 sections16:54
corvus(we might have 8, if we end up just doubling the 4 for both audiences)16:55
corvusbut yeah16:55
tristanCcorvus: so just 6 links from the main index.rst (user|admin)-tutorials and (user|admin)-howtos16:55
corvusyeah, that's probably worth a look16:55
corvusbecause if we can get someone to click on "user reference" then they get this, which is great: https://zuul-ci.org/docs/zuul/reference/user.html16:56
corvusShrews: ^ the "less is more" approach16:56
tristanCShrews: i can propose a draft for above approach if you want16:57
corvus(meanwhile, i'm working on updating the redirects)16:58
fungirishabhhpe: sorry, i was in a meeting (so was clarkb). looking at your paste, diskimage-builder is complaining about being unable to deploy the root element, but i don't see any details there as to why17:01
fungicorvus: i wonder if the shell script i embedded in my .htaccess change to parse git's file list could also work on a git diff between then and master tip?17:02
corvusfungi: i dunno... i just did this vvv  semi-manually17:02
fungier, shell script i used in my .htaccess change, noted in a review comment17:02
*** openstackgerrit has joined #zuul17:03
openstackgerritJames E. Blair proposed zuul/zuul-website master: Update redirects  https://review.opendev.org/70345417:03
fungicool, reviewing now17:03
corvusthe shell script did end up adding more redirects than we really need (but the extras are easy to remove now that i've noticed them)17:04
ShrewstristanC: sure, if you want. I’m at lunch now17:04
corvusthe changes are mostly paired -- i wonder if it would be more readable if i organized that differently17:04
rishabhhpefungi: only this much information is published in the log17:06
openstackgerritJames E. Blair proposed zuul/zuul-website master: Update redirects  https://review.opendev.org/70345617:09
openstackgerritJames E. Blair proposed zuul/zuul-website master: Update redirects  https://review.opendev.org/70345617:13
openstackgerritJames E. Blair proposed zuul/zuul-website master: Remove some redirects  https://review.opendev.org/70345717:13
*** tobiash has quit IRC17:14
corvusfungi: there we go. i like this better  ignore the first change in favor of 457 and 45617:14
corvusi think that's in a much more reviewable form17:14
pabelangerDid we land the logging of retry_failures into builds UI?  IIRC, we only report the last failure, not the previous 2 (if 3 is max times)17:21
clarkbpabelanger: not in the UI, but it is in the job info and can be indexed now17:23
*** tobiash has joined #zuul17:23
clarkbpabelanger: because ya the db only gets the last one17:23
pabelangerclarkb: indexed from where?17:24
clarkbpabelanger: in opendev's case elasticsearch17:25
rishabhhpeclarkb: fungi: did u get any updates over that config-drive issue in which my disk-image-builder was not able to attach config-drive to the spawned VM17:26
clarkb(basically its in your job inventory data so however you get at that historical info)17:26
pabelangerclarkb: ah, yah. That is our issue today, we 100% rely on zuul UI for that info17:27
fungirishabhhpe: it's working for us, so no i honestly have no idea why it's not working for you17:27
clarkbpabelanger: I believe tobiash and crew have a change up to record intermediate builds in the buildsets17:27
clarkbpabelanger: I don't remember where it was in review though. I recall corvus explaining that it is complicated or something17:27
rishabhhpeok17:28
fungirishabhhpe: we suggested configuring openstacksdk to perform debug logging so that the log will include the exact api call parameters it's making to nova so you can see whether nodepool is failing to pass the config-drive option vs whether nova is ignoring it for some reason17:28
rishabhhpecan you share your nodepool.yaml and the changes which you did in your disk-builder so that i can also try with same setup17:29
rishabhhpefungi: i am not sure how i can enable the debug logging if you can guide surely i will move forward17:29
pabelangerclarkb: k, will see what others say. Thanks!17:29
tobiashclarkb, pabelanger: https://review.opendev.org/#/c/633501 and related change17:31
pabelanger++17:34
*** evrardjp has quit IRC17:34
fungirishabhhpe: our nodepool configuration is here: https://opendev.org/openstack/project-config/src/branch/master/nodepool17:34
*** evrardjp has joined #zuul17:34
fungirishabhhpe: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nl01.openstack.org.yaml is a good example of the nodepool.yaml on one of our launchers17:35
fungirishabhhpe: https://opendev.org/openstack/project-config/src/branch/master/nodepool/nodepool.yaml is what we use on our builders17:35
fungirishabhhpe: and https://opendev.org/openstack/project-config/src/branch/master/nodepool/elements is where we put additional dib elements we use17:36
tristanCi've proposed to add zuul to https://github.com/ligurio/awesome-ci/pull/9117:36
fungithanks tristanC!17:38
openstackgerritTristan Cacqueray proposed zuul/zuul master: docs: remove generated toc from the main index  https://review.opendev.org/70346817:40
tristanCcorvus: Shrews: here is the lighter main toc we discussed earlier ^17:41
*** Goneri has quit IRC17:47
*** gouthamr_ has joined #zuul17:47
*** jpena is now known as jpena|off17:48
openstackgerritJames E. Blair proposed zuul/zuul master: Docs: change "config" title  https://review.opendev.org/70347117:51
openstackgerritTristan Cacqueray proposed zuul/zuul master: docs: remove generated toc from the main index  https://review.opendev.org/70346817:58
*** jamesmcarthur has joined #zuul18:03
corvustristanC: would your sf operator basically include a copy of the zuul operator in order to extend it?  would your plan be to copy changes from zuul-operator into sf-operator?18:08
corvustristanC: (that's what https://softwarefactory-project.io/r/#/c/17346/2/conf/zuul/applications/Zuul.dhall looks like to me)18:09
*** gouthamr_ has quit IRC18:10
*** Shrews has quit IRC18:11
*** gundalow has quit IRC18:11
*** tributarian has quit IRC18:11
*** Shrews has joined #zuul18:11
*** gundalow has joined #zuul18:12
*** tributarian has joined #zuul18:12
tristanCcorvus: i only included a copy because it is not merged yet upstream, otherwise we would import it like a regular package18:22
tristanCcorvus: the conf/zuul is a copy of https://review.opendev.org/#/c/702105/18:23
corvustristanC: okay, so there is an import mechanism18:23
corvustristanC: would you be able to add mqtt using that?18:24
tristanCcorvus: here is how import works: https://review.opendev.org/#/c/702104/6/conf/operator/Prelude.dhall18:24
corvus(i guess it was just easier to modify the local copy rather than making a new mqtt file that imports zuul?)18:24
corvustristanC: is the sha256 required?18:25
tristanCcorvus: right, it was just easier like that, this https://softwarefactory-project.io/r/#/c/17346/2/conf/zuul/applications/Zuul.dhall needs to be proposed in zuul/zuul-operator18:25
corvusyeah, you're right, mqtt belongs there anyway18:26
tristanCcorvus: the sha256 is a semantic hash, see https://github.com/dhall-lang/dhall-lang/wiki/Safety-guarantees#code-injection18:26
tristanCand if the hash is already in the cache, then it's used without network lookup18:27
tristanCcorvus: more generally, any dhall expression can be a path or an url, and in that case, the content is loaded as if it was defined locally18:29
tristanCcorvus: that can also be used to load text file, here i'm doing that to fetch the uid_entrypoint script: https://softwarefactory-project.io/r/#/c/17345/3/conf/sf/applications/helpers.dhall@23018:29
corvusthat is very powerful indeed; i understand the sha256 reasoning now18:30
rishabhhpefungi: i am following the nodepool.yaml file recommend by you .. and also while cloning the elements from opendev.org for nodepool image building i am getting duplicate entries for control-plane-minimal and in infra-package-needs packages .. last time also i faced same thing i deleted the duplicate items from infra-package-needs elements can that cause the issue for glean not getting the ip info from nodepool ?18:31
corvustristanC: do you think kopf would support the extension mechanisms you'd need?  i imagine it could be published to pypi and imported...18:31
tristanCcorvus: i can't see why it wouldn't let you import and use functions defined in another project18:31
*** jamesmcarthur has quit IRC18:36
tristanCi'm not using any special mechanisms, the dhall code could be simply rewritten using python lambda or pure functions18:37
*** tosky has joined #zuul18:50
*** rishabhhpe has quit IRC19:04
mnasercould I get reviews for https://review.opendev.org/#/c/702963/ and https://review.opendev.org/#/c/702965/ ? :>19:08
tristanCcorvus: mnaser: my worries with helm is that the template configuration might end up like this one: https://github.com/helm/charts/blob/master/stable/prometheus/values.yaml19:16
mnasertristanC: In my experience that’s just a part of doing helm and one of the more poor experiences about it19:18
mnaserFor all the good stuff there is a few annoying issues like these19:18
*** themroc has quit IRC19:19
corvusi wonder if we can automatically pass some things through19:21
corvuslike this: executor: {replicas: 3, args: {disk_limit_per_job: 250}}19:22
corvusand just add args to the template19:22
*** igordc has joined #zuul19:23
pabelangeryes, I would like to see that too19:24
pabelangerthe link tristanC shared is very much like how we did it in puppet, which was painful to me19:24
tristanCif zuul-helm just implements the spec defined in https://zuul-ci.org/docs/zuul/reference/developer/specs/kubernetes-operator.html#custom-resource-definitions , then the values.yaml shouldn't grow out of control19:28
*** adam_g has quit IRC19:28
tristanCbut i found that as you add more services, such as elasticsearch or mqtt, then it becomes difficult to manage the resources using values and template only.19:29
pabelangercan the zuul.conf template file be read from file or does it need to exist in the crd yaml (sorry, not sure if the correct term).19:32
corvuscrd is an operator term19:32
corvusprobably best not to confuse helm and operator here19:32
tristanCpabelanger: because the zuul-operator may manage the database connection, then we decided that the zuul.conf would be managed by the operator19:33
tristanCcorvus: actually, iiuc the operator-sdk support helm, thus it might be possible to implement the crd using the zuul-helm charts19:34
corvusa fifth option19:34
pabelangerso, as an example, with ansible-role-zuul, there is a default template use to populate zuul.conf: https://opendev.org/windmill/ansible-role-zuul/src/branch/master/defaults/main.yaml#L102 but, if it needs to be more complex, that is exposed as ansible variable for user to provide own. From what I understand, you might not be able to do that in helm?19:34
tristanChttps://github.com/operator-framework/operator-sdk/blob/master/doc/helm/user-guide.md19:35
tristanCpabelanger: you could do that with helm or any operator language, but then if the system creates the database connection for you, then you would have to concat ini files. that works, but that is also error prone19:36
pabelangertristanC: right, you'd do that or leave database bits to human. Or, maybe look to support zuul.d folder for zuul.conf?19:37
pabelangerI want to say, maybe there was a patch up to support that19:37
pabelangerI think it was flaper87 that added a patch19:37
tristanCpabelanger: agreed, perhaps the operator (or the chart) shouldn't bother with db or zk. I find that a bit arbritary that the operator would manage the database but not the monitoring for example19:38
corvusthe database is required19:38
corvusthis is what we wrote about that: https://zuul-ci.org/docs/zuul/reference/developer/specs/kubernetes-operator.html#external-dependencies19:39
pabelangerhttps://review.opendev.org/660977/ was PR I was thinking of19:42
corvuseven without that, combining ini files should not be difficult19:42
pabelangeryup, puppet had a module that would do that, seemed to work okay19:43
pabelangerlooking at spec, I see spec.database and spec.connections but not a way to handle disk_limit_per_job or maybe read/write bindmounts19:45
tristanCpabelanger: zuul can template zuul.conf from environment since https://review.opendev.org/#/c/612824/ , which i think render https://review.opendev.org/660977/ less useful . I'm already using env templating to implement the zuul-operator external zookeeper or db-uri19:46
pabelangermerger might need git_user_email and git_user_name also19:46
pabelangerjust looking at our zuul.conf file19:46
corvusyeah, i think we didn't consider that case well.  adding explicit options, an args dict, or just passing through unrecognized options all seem like valid approaches.19:46
tristanCpabelanger: https://review.opendev.org/#/c/702105/6/conf/zuul/applications/Zuul.dhall@584 implements the spec.database and spec.connections option19:46
pabelangerI like args dict, that would be less churn on operator for exposing config settings over explicit options19:47
tristanCpabelanger: and merger git_user_email / name is already implemented here too: https://review.opendev.org/#/c/702105/6/conf/zuul/applications/Zuul.dhall@21519:47
pabelangertristanC: ack, I could see that approach being more work, to keep in sync with zuul.conf changes, but is something we've done before19:48
openstackgerritMerged zuul/zuul-helm master: Added support for configuring disk_limit_per_job  https://review.opendev.org/70296319:48
tristanCcorvus: should we replace the spec.connections by a scheduler.raw_config option? And when no db or zookeeper are provided, then the operator "inject" the setting in zuul.conf19:48
openstackgerritMerged zuul/zuul-helm master: Add extra files for secret  https://review.opendev.org/70296519:48
corvustristanC: it looks like it's already meant to be a pass-through.19:50
tristanCcorvus: the spec does mention "Zuul files like /etc/nodepool/secure.conf and /etc/zuul/zuul.conf should be managed by the Operator and their options should be represented in the CRD."19:50
corvusalso "Connections should each have a stanza that is mostly a passthrough representation of what would go in the corresponding section of zuul.conf."19:51
tristanCpabelanger: i don't mind either way, i would prefer we hide the zuul.conf details from the user, but it is more work indeed19:51
pabelangertristanC: as example, reading change relative_priority is missing for scheduler, which we enable.19:51
pabelangerI think you have SSL change up for gearman?19:52
tristanCpabelanger: yes, it is: https://review.opendev.org/#/c/702716/419:52
corvusit's certainly possible for an operator to inspect and manipulate the options it needs (like the secrets) and just pass-through anything else unknown.  we could also add each option explicitly.  they don't change *that* often.19:53
corvuslet's see if we can have another meeting on wednesday, come to some kind of agreement on the underlying technology, then dig into these problems :)19:54
tristanCcorvus: right, so if we do that, then the spec needs a 'zuul_conf' secret config to enable the user to provide the zuul.conf content'19:55
corvusoh no no no19:55
corvusi outlined 3 options and none of them needed that19:55
corvusoption 1: add each option explicitly to the zuul crd and copy it over (like mnaser just did in helm).  option 2: add an args dict and copy its contents as the values (what i suggested for the helm charts earlier; i don't like this for the operator).  option 3: just have the operator pass-through any options it doesn't recognize.19:57
*** hashar has joined #zuul19:58
tristanCfor what it worth, i find that using a strict language like dhall or go, it's not too difficult to express the zuul.conf content structure, then it is easy to expose any settings (such as relative_priority) and let the user tune them safely.19:59
corvusyeah, that has advantages in validation too.20:00
mnaserbtw, we can get away with templating all of the configuration variables in helm. You can see how I did it for “connections”20:00
mnaserThe reason I did it for connections only was that it was predictable. I worried that reimplementing the entire config in ini and doing go trmplating for the whole file might introduce some weird things20:01
corvusis wednesday at 1600utc an okay time to have another call?20:01
pabelangermnaser: where can one see that?20:01
mnaserIt might totally be ok too.  But again this is a fast moving target so none of the stuff we’re doing is something that I’d consider stable in terms of choices in helm implementation until we decide we have a “1.0.0”20:01
mnaserwhich means now is a good time to experiment and see if we wanna have the entire file in yaml converted to ini.. I think that’s cleaner than having a ini string blob20:02
pabelangermnaser: I'm guessing you'd inspect driver value, then template based on that?20:02
mnaserpabelanger: mot really even more trivial20:03
mnaserhttps://opendev.org/zuul/zuul-helm/src/branch/master/charts/zuul/templates/secret.yaml#L2820:03
tristanCcorvus: i'd rather do that tuesday or later on wednesday please20:03
corvusmnaser, pabelanger: tuesday (tomorrow) 1600 utc okay?20:04
pabelangerI think I have a conflict, but should be able to join20:04
mnaserI don’t mind day but personally it would have to be after 1700 utc20:04
mnaser(I’m in Europe right now and I have stuff that pretty much goes all day)20:05
tristanCmnaser: how one is supposed to set the ssh key for a gerrit connection with that secret.yaml#L28 template?20:05
corvusmnaser: heh, i was going earlier to try to accomodate europe :)20:05
corvustristanC: what's your earliest time on wednesday?20:05
mnasertristanC: define extra files which would then mean they would appear in /etc/nodepool20:06
mnaserYeah outside Europe business hours works better for me20:06
pabelangermnaser: yah, that is kinda what corvus said for config options, just pass straigh through, no validation20:06
pabelangerI don't mind that approach20:07
pabelangerfor us, rather then loop, we just did it directly in tempalte file: https://github.com/ansible-network/windmill-config/blob/master/zuul/zuul.conf.j2#L80 because older I get, harder to loop all the template stuff in brain :)20:07
tristanCcorvus: i can do utc noon up to 1400 utc, then i have sprint review the whole morning20:07
mnaserya I’m ok with doing that got the charts, but I’d want the config to be yaml rather than an ini you plop down20:08
corvustristanC: oh i thought you said later on wed was ok20:08
corvustristanC: so wed is completely out?20:09
tristanCcorvus: up-to 1400 utc on wednesday, isn't that earlier than 1600 ? :)20:09
corvustristanC: " corvus: i'd rather do that tuesday or later on wednesday please" is why i am confused20:09
tristanCmnaser: the issue with that secrets.yaml template is you have sync the secret mount path (or the file name) along with the zuul.conf entry20:10
pabelangerhowever, https://github.com/ansible-network/windmill-config/blob/master/ansible/group_vars/zuul-connections.yaml is how we added content to disk20:10
pabelangerwhich was loop'd20:10
pabelangerfor SSH keys20:10
tristanCcorvus: oops, later, or earlier on wednesay, i opt for earlier because of mnaser tz20:10
corvusmnaser requests > 1700 utc, so wed is incompatible20:11
mnaseryeah except it’s better for me to meet outside business hours, so the later for me, the better :p20:11
corvusso, tues 1700 or thurs 1700?20:11
tristanCpabelanger: that strategy has the same issue where you have to keep the zuul.conf content in sync with the secret definition20:11
tristanCcorvus: tues and thurs both works for me20:11
mnasertristanC: yeah but I’m mounting the secret as a directory, not a file.20:11
mnaserSo any secrets within it appear inside the directory20:12
corvusmnaser: tues 1700 or thurs 1700?20:12
mnaserSo it will always be in /etc/zuul20:12
mnasercorvus: let’s do Thursday 170020:12
pabelangertristanC: yup, but that is limited to my local install. So, I am happy to break that if needed.  I also find if easier now to look at final produced config file in git, over templated out onto running system20:13
tristanCmnaser: because we support multiple gerrits or githubs, then it's tricky to avoid file collision. For the operator spec, I used a directory per connection like so: https://review.opendev.org/#/c/702105/6/conf/zuul/applications/Zuul.dhall@34320:13
tristanCwhere the key name defaults to `id_rsa`, but that can be overriden by the user20:13
mnasertristanC: you could just have multiple files too?  I tried not to make a lot of assumptions and let the user wire up things to their will20:13
tristanCmnaser: well yes, any secret can contains multiple files20:14
mnaseralso the chart doesn’t actually have an understanding of a connection natively20:14
mnaserIt’s just taking things pass through right now20:14
mnaserSo In my case I have extra files defined with the private key, and then point the connection config to it20:15
tristanCwhat i meant is that the operator implementation doesn't expose to the user where the secrets are mounted on the container, thus the user can't mess with file name colission or bad file path20:15
tristanCwhich seems like a good thing to have, at the cost of a structured zuul.conf20:16
pabelangerwe do it more like mnaser today, define where we want to ssh stored, and ansible creates it. Then we update zuul.conf data, to point to it20:23
*** adam_g has joined #zuul20:23
mnaseryeah, I agree on that. But the connection stuff are different from github to gerrit and I didn’t want to build the charts with certain baked in assumptions of the connection they use20:24
mnaserSo yeah, took the more piped path20:24
tristanCthat is simpler to manage for the confmgmt implementation, but that is not has been specified in the zuul-operator crd.20:24
pabelangerthat's what I did with roles, I didn't want to manage that. Expose enough knobs for the user to do that, and if they wanted something finer, they could build atop of the role20:25
tristanCpabelanger: that way it's easier to implement but it also seems harder to use. iirc the goal of that operator is to make it easier to use zuul.20:30
tristanCi don't mind making the operator simpler, but we should update the spec to remove the 'secretName' attributes from the connections20:36
pabelangertristanC: it pushes more work to operator (or next level up operator), and less to community to manage, for sure. Historically, I've found it easier myself to point to know file I create, then update config for it. I'm not sure what the right way moving forward is.20:37
pabelangerWe are also at a point with zuul.a.c, were we don't really have to make many operations changes, so also kinda a step back on recent operations discussions. Mostly focused on onboarding new projects / users to zuul workflow, which is an effort in itself. :)20:38
tristanCpabelanger: i like how easy you can use the zuul cr as it is defined, but i get your need for fine-tuning, thus i don't mind either way.20:40
tristanCthough i think the fear of having the same issues we had in the past with puppet or ansible templating does not applies where the config generator is using strict type. It can be safe to derive complex schemas and implement as many knobs as needed.20:43
pabelangertristanC: for me, if we end up with something like https://opendev.org/opendev/puppet-zuul/src/branch/master/manifests/init.pp#L20 we've repeated history. https://opendev.org/opendev/puppet-openstackci was help help hide the interface to 3pci users, for most part worked, but was always a balance act between everybody20:46
pabelangerthat's mostly why I like the args dict approach20:46
pabelangernot much business logic by default20:46
tristanCpabelanger: then i think we should drop the `secretName` attribute from the zuul cr spec, as it seems like it is what prevents the args dict approach.20:48
corvusi don't think that's necessary20:49
pabelangerI'm not familiar enough with CRDs honestly, so would defer to others20:49
corvusif we don't want to explicitly name every option, then i think the partial pass-through approach is fine20:50
tristanCcorvus: how would you know user secrets needs to be mounted inside the zuul container?20:50
tristanCwhat* user secrets20:50
corvustristanC: last paragraph of https://zuul-ci.org/docs/zuul/reference/developer/specs/kubernetes-operator.html#zuul-config20:52
corvusbasically, the operator handles secrets.  users never see a filename.20:53
corvus(that's an important part of how the operator takes care of administering the system)20:54
tristanCcorvus: right, so if the operator handles secrets and manage their mount path, then I don't understand how the user can provide a raw zuul.conf, or args dict?20:55
tristanCunless the operator knows the structure of each connection, and be able to set the right sshkey path value20:56
corvustristanC: right, which is why the spec does not have a facility for a user-supplied zuul.conf20:56
corvusand like i said earlier, i don't think we should have an args dict for the operator20:57
tristanCcorvus: yes, and that's how i implemented the zuul-operator in topic:zuul-crd :)20:57
corvusyes, i'm pretty sure we agree here20:57
corvusi mean, that's the point of a spec, to get agreement on stuff like this20:58
tristanCcorvus: oh, well i suggested to drop that secret management part of the spec based on pabelanger and mnaser feedback20:58
corvusi think pabelanger raises a good point that adding each option is annoying, but if we find that's the case, i think we can implement that via pass-through of unknown options20:59
mnaserfwiw I am not following the zuul operator in the charts20:59
corvusno change to the spec is required either way20:59
mnaserI think the operator and the charts are two different things imho20:59
mnaserOne is just kubernetes application “packaging”, the other is fully fledged operator21:00
mnaserOne strictly gets the application installed, the other one helps with the actual lifecycle and operations of it21:01
tristanCmnaser: the operator can also helps with the installation too, how would you create the gearman tls certs with helm?21:03
mnaserYou don’t, you generate them elsewhere and point to them21:03
mnaserHelm isn’t supposed to make life “easier” but more of help package the common bits to get an app running. It’s far more “low level”21:04
tristanCmnaser: another big difference is that you can use operator by just applying kubernetes resources, for helm you need a cli and a place to generate runtime secrets21:04
pabelangerI figure, take my feedback with grain of salt, I don't really have too much time to help drive operator stuff. So deferring to others to drive it. :)  Most of the stuff so far, is based on how I did it on zuul.a.c with playbooks and roles, which was heavily inspired based on how puppet-zuul and openstack-infra grew over the years. So far, the roles I created have had minimal (to none) changes to them related to21:04
pabelangercfg settings, but they are also not the most user friendly. You do need to understand how ansible works. And like I mentioned, I'm really not a fan of having something do log of magic related to configuration, as each time I learn how it works and step away, I end up forgetting stuff each time. That's mostly why I like static cfg files these days21:04
mnaserFor my use case, I prefer helm. I use Argo which templates and applies it21:05
mnaserI am not against an operator, but in my case, and probably some other users, that’s enough21:06
mnaserI think we should have both an operator and helm charts which are two distinct ways of deployment21:06
mnaserJust like how you can deploy Prometheus using an operator or simply use helm charts21:06
corvusi've been setting up these meetings in the hope of getting us all to agree on how to move the operator forward -- is everyone here actually willing to participate in that?21:08
corvusbecause i think i just say pabelanger and mnaser both say they aren't particularly interested in the operator21:08
corvuss/just say/just saw/21:08
pabelangerI am intersted, I have zero time to help right now21:09
corvuspabelanger: you just spent 2 hours talking about it, are you sure "zero" is the right value?  perhaps it's just "less than you would like"?21:10
mnasercorvus: I don’t presently have time to drive an operator, unfortunately given my huge time constraint, the charts are a reasonable stop gap to actually get zuul on kubernetes21:10
mnaserAnd even if I end up investing time into the operator eventually, I think I would still be happy to maintain the charts too for users who want them21:11
mnaserI think it is good for zuul to have two deployment options21:11
corvusi'm just trying to figure out if we're doing something productive here, or are we all wasting our time?21:11
corvusmnaser: i don't think you'd need to drive it21:11
mnaserI would love to migrate to an operator eventually personally21:12
pabelangercorvus: well, I am happy to talk while things compile, but I really haven't reviews any patches or tried the dhall or helm patches. So, happy to share POV, but taking a task item and completing, I really am tight on time. it would have to be after hours / weekends for me21:12
mnaserI just don’t think it would be operator vs helm21:12
tristanCmnaser: fwiw the current operator already get zuul on kubernetes21:12
corvusmnaser: okay, cool, that level of commitment is great.  :)21:12
pabelangercorvus: however, with zuul.a.c working, migrating to new operator is lower priority, if I am being honest.21:12
tristanCmnaser: once the image is published, you could get a zuul by applying https://review.opendev.org/#/c/702106/15/deploy/crds/zuul-ci_v1alpha1_zuul_cr.yaml21:13
mnasertristanC: that’s awesome, honestly, I haven’t had too much time to look into it. I’m sure a migration would be relatively trivial21:13
mnaserIt’s just the time investment of moving to it right now is a lot for not much value right now. Once we have some of the really useful things we discussed, say like dumping and restoring queues on upgrade for example21:14
mnaserIt starts to be far more valuable21:14
corvuspabelanger: your input is very welcome, including feedback on how much you would be able to contribute to each of the various options.21:15
mnaserBut the time investment to pay right now based on how little I can find doesn’t drive huge benefit :(21:15
tristanCmnaser: alright, but that's a minor change, a task to dump before apply, then a a task to reload if the scheduler got restarted...21:15
corvusthe main thing i just wanted to avoid is spending a bunch of time talking about operators, and ending up, once again, with a repo collecting dust.  :)21:15
tristanCmnaser: there is also got to make it scale executor and merger on demand too, just need a bit of wiring up21:15
mnaserYes, I don’t want that either. Something is better than nothing21:15
tristanCcode*21:15
mnaserI spent a sizeable investment of time to get those helm charts up and running, so In terms of time21:17
mnaserTime planning, it would be a bit unproductive to spend a bunch more refactoring into the operator21:17
mnaserGiven my current time constraints anyways21:18
corvusi also think the helm charts are valuable -- i think one of the ways to keep them so is to make them as "thin" as possible.  i think pabelanger raised a good point earlier that adding options like the executor storage size is hard to manage/scale, so as we work on it, we may want to try to think about the best ways to minimize that.21:18
mnaserIdeally I would have spent that time on the operator, but writing one is far more complex and I believe at the time tristanC wasn’t working on it yet21:19
corvusand that plays to their strong point: just getting the app up and running like mnaser said, with the user supplying most everything it needs.21:19
mnaserI agree. As I started adding more values I realized just how this can become a pain over time21:19
mnaserSo I’m actually in favour of refactoring it in order to actually have the config all yaml and templated into ini21:20
mnaserAnd I can work on that.. probably the next time I have to change a zuul config, I’ll probably just change it to take a config only21:20
mnaserRather than implement the one thing I want to use21:21
tristanCcorvus: we are very much interested in getting a zuul operator available, and i think the spec is almost completed now (beside things like the mqtt or pagure connections).21:25
corvus++21:25
tristanCi would be happy to investigate another framework such as kopf or golang, but if i'll be in the charge of the implementation, then i'd rather stick to what is known to work for the problem i explained previously21:27
tristanC(e.g. use dhall to generate the resources, and ansible to change the state)21:29
corvustristanC: i'm worried that if we use dhall, you will be the only one able to either write or review changes.  but if we go with kopf, then everyone working on zuul should be able to understand it.  that seems like a big win to me, so i hope you'd be willing to consider that.21:30
tristanCcorvus: kopf looks interesting, but i couldn't find any success stories, like not even one complete operator.21:35
corvustristanC: didn't you invent the dhall operator? :)21:38
tristanCcorvus: that's just a minor abstraction, the core come from the ansible operator-sdk and the https://github.com/dhall-lang/dhall-kubernetes21:39
tristanCwhich are both used successfully in other projects, to solve the same problem we are facing with zuul21:40
tristanCwhat i named `dhall operator`, is mostly that function: https://review.opendev.org/#/c/702104/6/conf/operator/deploy/Kubernetes.dhall , which i also implemented for pure ansible or podman compose script. but that's just using the kubernetes binding21:43
zbr|droveri have a set of simple fixes for zuul https://review.opendev.org//#/q/topic:zuuly+(status:open+OR+status:merged) -- lets merge them and move on. thanks.21:48
hasharzbr|drover: I was just about to wonder how to add support for py37 and others ;)21:50
zbr|drovernot sure where my extra slash came from but underlined an interesting bug in zuul21:51
zbr|droverpage loads but you cannot click any link.21:52
zbr|droverwith correct number of slashes, it works https://review.opendev.org/#/q/topic:zuuly+(status:open+OR+status:merged)21:52
fungibug in zuul?21:52
zbr|droverfungi: more or less21:53
fungipresumably you meant bug in (opendev's ancient) gerrit21:53
mnaserI agree regarding the sustainability factor of kopf fwiw21:53
zbr|droveraahh, right.21:53
mnaserBut it seems pretty active and it seems to21:53
zbr|droverfungi: i would be very happy to see gerrit bumped21:53
mnaserBe maintained by a popular Swedish e-commerce which has been known to its open source work21:54
mnaserhttps://github.com/zalando is their main repo21:55
tristanCmnaser: and it seems like they are also using golang for https://github.com/zalando/postgres-operator21:58
mnaserInteresting21:59
fungizbr|drover: yep, step 1 is our upcoming maintenance to swap out to container-deployed gerrit, then we can more easily migrate to newer builds (so... rsn)22:00
*** jamesmcarthur has joined #zuul22:16
*** adam_g has quit IRC22:40
*** adam_g has joined #zuul22:43
*** hashar has quit IRC23:14
*** tosky has quit IRC23:51

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!