| *** _KaszpiR_ has left #zuul | 00:01 | |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 02:57 |
|---|---|---|
| *** remi_ness has quit IRC | 03:01 | |
| *** remi_ness has joined #zuul | 03:15 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: add playbooks to job.toDict() https://review.openstack.org/621343 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Get executor job params https://review.openstack.org/607078 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Separate out executor server from runner https://review.openstack.org/607079 | 03:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 03:22 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 03:49 |
| tristanC | jhesketh: is there a reason why https://review.openstack.org/607081 isn't part of https://review.openstack.org/607080 ? It seems like it would better if those 2 were squashed | 04:45 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: tests: improve test_web to only provision events when needed https://review.openstack.org/630575 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: config: add playbooks to job.toDict() https://review.openstack.org/621343 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Add API endpoint to get frozen jobs https://review.openstack.org/607077 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Get executor job params https://review.openstack.org/607078 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Separate out executor server from runner https://review.openstack.org/607079 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Move common AnsibleJob prep tasks into a base class https://review.openstack.org/607080 | 05:19 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Implement a local zuul-runner https://review.openstack.org/607082 | 05:19 |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 06:33 |
| *** remi_ness has quit IRC | 06:51 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 07:08 |
| *** pcaruana has joined #zuul | 07:11 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Move common AnsibleJob prep tasks into a base class https://review.openstack.org/607080 | 07:31 |
| openstackgerrit | Rui Chen proposed openstack-infra/zuul master: Avoid using list branches with protected=1 in github driver https://review.openstack.org/630038 | 07:58 |
| *** themroc has joined #zuul | 08:11 | |
| *** avass has joined #zuul | 08:13 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add action to task result in zuul_json callback https://review.openstack.org/630622 | 08:28 |
| *** gtema has joined #zuul | 08:47 | |
| *** avass has quit IRC | 08:49 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: [wip] Use bindep for devstack jobs https://review.openstack.org/626068 | 08:51 |
| quiquell | hello | 08:53 |
| quiquell | is possible to hold a node on "success" with a openstack nodepool provider ? | 08:53 |
| *** ssbarnea|rover has joined #zuul | 08:54 | |
| openstackgerrit | Felix Schmidt proposed openstack-infra/zuul master: Add action to task result in zuul_json callback https://review.openstack.org/630622 | 08:54 |
| quiquell | Ok I see it's not possible https://github.com/openstack-infra/zuul/blob/2fd688352f5e220fda0dfc72b164144910670d95/zuul/scheduler.py#L1249 | 08:55 |
| quiquell | corvus: ^ It make sense if I put in place a review to be able to hold without filtering result if it's propertly configured ? | 08:58 |
| quiquell | corvus: in the scheduler config | 08:58 |
| *** jpena|off is now known as jpena | 08:58 | |
| quiquell | clarkb: ^ | 08:58 |
| *** panda|off is now known as panda | 09:12 | |
| *** avass has joined #zuul | 09:14 | |
| *** hashar has joined #zuul | 09:20 | |
| *** chandan_kumar is now known as chandankumar | 09:21 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 09:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: WIP: Implement a local zuul-runner https://review.openstack.org/607082 | 09:40 |
| tristanC | jhesketh: just got "zuul-runner prep-workspace" command working :-) | 09:40 |
| tristanC | jhesketh: i'll work on the "execute" sub-command tomorrow | 09:46 |
| *** sshnaidm|off is now known as sshnaidm | 10:27 | |
| jhesketh | tristanC: that's awesome, thanks so much! | 10:39 |
| jhesketh | I'm traveling for the next couple of weeks so I won't be able to do much upstream stuff, but I'm looking forward to taking a closer look as soon as I can :-) | 10:40 |
| tristanC | jhesketh: you're welcome, thanks for putting the fondation | 10:40 |
| jhesketh | re the order of the patches, they were kept like that because of the way I was working through the problem in my head. I knew some of them should be more logically grouped and it was my intention to reorder and squash etc down the line, so that's fine :-) | 10:40 |
| quiquell | tristanC: hello, I was trying to figure out how to do autohold on success jobs | 10:44 |
| quiquell | tristanC: Does it make sense to put in place a review to configure that per tenant at tenant_configuration ? | 10:44 |
| tristanC | quiquell: perhaps an auto-hold option to force hold even on success? | 10:45 |
| *** electrofelix has joined #zuul | 10:45 | |
| quiquell | tristanC: Looks like job results are harcoded at scheduler https://github.com/openstack-infra/zuul/blob/2fd688352f5e220fda0dfc72b164144910670d95/zuul/scheduler.py#L1249 | 10:46 |
| quiquell | tristanC: Humm wait I see it now kwno to do it | 10:47 |
| quiquell | tristanC: We get the autohold tuple key, we can also put there the job results we are interested in :-) | 10:48 |
| quiquell | tristanC: Changing the zuul client acording to that, is that it ? | 10:48 |
| tristanC | quiquell: maybe? not sure why the hold command was removed from nodepool in the first place | 10:49 |
| quiquell | tristanC: what do you mean ? do we have an unfiltered autohold at nodepool before ? | 10:49 |
| quiquell | tristanC: I am very new to all this | 10:49 |
| tristanC | quiquell: yes, it was named nodepool hold | 10:51 |
| quiquell | tristanC: do we have a nodepool command too at launchers ? | 10:51 |
| quiquell | yep just checked it | 10:52 |
| jkt | in the openstack's zuul config, am I correct in assuming that https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml is the place which lists all projects that Zuul cares about? | 10:53 |
| jkt | how is that file distributed to the scheduler? | 10:53 |
| jkt | I'm asking because the quick tutorial (with a prebuild Docker image) apparently has a hard-coded list of projects in a file /etc/zuul/main.yaml which looks like being provisioned out-of-band | 10:55 |
| quiquell | jkt: you have the scheduler config pointing to this file directly, with tenant_config | 10:55 |
| quiquell | jkt: in the zuul.cfg file | 10:55 |
| jkt | quiquell: so in case of openstack's CI, is the openstack-infra/project-config managed out-of-band via, e.g., Puppet/cron/soomething? | 10:58 |
| quiquell | jkt: Could be that zuul.cfg is generated | 10:59 |
| quiquell | jkt: maybe at #openstack-infra they know better | 10:59 |
| * quiquell remember reading about it but has forgot | 11:00 | |
| jkt | quiquell: let me rephrase, I'm being confused because in case of that CI, there's a git repo which contains both the top-level tenant definition with a list of projects etc, and that same git repo is also being "included" into zuul from that same tenant config | 11:05 |
| jkt | so I wonder how this got bootstrapped | 11:05 |
| quiquell | jkt: For example you can clone it start zuul and with this config and commit changes to this it will be hot reconfigured | 11:07 |
| quiquell | jkt: and if you restart you have to be sure that you are refreshing the cloned repo | 11:07 |
| quiquell | jkt: out of my mind, don't know how the real deal is done | 11:07 |
| jkt | okay, thanks, I'll wait for the admins to respond, I know they are here as well | 11:08 |
| *** gtema has quit IRC | 11:09 | |
| jkt | quiquell: it seems like it's provisioned out-of-band for the tenant config, https://git.openstack.org/cgit/openstack-infra/system-config/tree/manifests/site.pp#n862 | 11:13 |
| quiquell | jkt: so they just clone and push changes later on | 11:14 |
| jkt | quiquell: perhaps it's the cron+puppet for content of the tenant file (which is https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml ), and then zuul's builtin stuff for https://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.d | 11:16 |
| openstackgerrit | Sagi Shnaidman proposed openstack-infra/nodepool master: Support userdata for instances in openstack https://review.openstack.org/630649 | 11:19 |
| quiquell | sshnaidm: ^ needed for the cloud-init + zuul ? | 11:20 |
| sshnaidm | quiquell, yep | 11:21 |
| quiquell | sshnaidm: cool cool | 11:22 |
| quiquell | sshnaidm: so at the end is not an image issue ? | 11:22 |
| sshnaidm | quiquell, there is no other way to inject key for non-default user except using userdata :( | 11:23 |
| sshnaidm | quiquell, and I don't know if we want to go this way tbh.. | 11:23 |
| quiquell | sshnaidm: let's talk about this this afternoon meeting | 11:23 |
| sshnaidm | quiquell, yeah | 11:24 |
| sshnaidm | but still nice to have this functionality in nodepool, seems like very trivial change | 11:24 |
| openstackgerrit | Tobias Urdin proposed openstack-infra/zuul-jobs master: Add upload-puppetforge role https://review.openstack.org/627553 | 11:30 |
| quiquell | sshnaidm: Can you put the possible options at the taiga thingy for the meeting ? | 11:34 |
| *** gtema has joined #zuul | 11:35 | |
| *** rfolco has joined #zuul | 11:38 | |
| sshnaidm | quiquell, yeah, doing.. | 11:39 |
| avass | zuul encryption keys seem to be changing every time the containers are restarted. any way to make them static? | 11:42 |
| avass | keys for zuul secrets i mean | 11:42 |
| tobiash | avass: you need to mount a volume (/var/lib/zuul) by default into the container | 11:46 |
| jkt | I wonder how sane it would be to use https://softwarefactory-project.io/docs/zuul/admin/components.html#attr-scheduler.tenant_config_script to fetch the main config file from Gerrit :) | 11:46 |
| avass | tobiash: into the scheduler? | 11:53 |
| avass | tobiash: looks like it's already doing that in the docker-compose file | 11:54 |
| tristanC | jkt: you still need to reload the scheduler procedure when the file change, so you might as well drop the new version before doing that | 11:54 |
| tristanC | jkt: but it's sounds sane to me to fetch the config dynamically through config_script, but put some retry logic in case gerrit is slow, not available | 11:55 |
| *** dkehn has quit IRC | 12:25 | |
| *** jpena is now known as jpena|lunch | 12:27 | |
| tobiash | avass: yes, into the scheduler. It by default stores its keys in /var/lib/zuul/keys | 12:31 |
| *** quiquell is now known as quiquell|lunch | 12:48 | |
| *** rlandy has joined #zuul | 12:58 | |
| *** needssleep is now known as TheJulia | 13:02 | |
| avass | tobiash: strange, it's mounted and the keys exist but they change when the containers are restarted. | 13:02 |
| *** quiquell|lunch is now known as quiquell | 13:05 | |
| avass | tobiash: looks like it's creating a new volume every time it's restarted | 13:14 |
| jkt | tristanC: thanks (...or one could always rely on systemd to restart this if the initial attempt fails, I guess) | 13:23 |
| jkt | it seems that I will have to always restart the scheduler when I add/register/... more projects, right? | 13:23 |
| *** dkehn has joined #zuul | 13:31 | |
| *** jpena|lunch is now known as jpena | 13:31 | |
| tobiash | jkt: no, you can update the main.yaml and then trigger a reconfiguration | 13:33 |
| tobiash | https://zuul-ci.org/docs/zuul/admin/components.html?highlight=reconfigure#operation | 13:33 |
| tristanC | jkt: no need to restart the process, you can use "zuul-scheduler -d full-reconfigure" as ExecReload= | 13:34 |
| jkt | thanks, that's nice, I must have been a bit overwhelmed by the combination of docs *and* the openstack-specific operation cookbooks | 13:57 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Document default values of runtime arguments https://review.openstack.org/630679 | 13:59 |
| Shrews | ianw: re: your suggestion to my nodepool build timeout change, do you know how dib handles stderr if you use the --logfile option? | 14:16 |
| *** gtema has quit IRC | 14:18 | |
| tobiash | Shrews: shouldn't dib isolate concurrent image builds? | 14:30 |
| tobiash | I just switched nodepool to do 2 image builds in parallel and now the builds fail with this: E: Could not get lock /var/cache/apt/archives/lock - open (11: Resource temporarily unavailable) | 14:31 |
| Shrews | tobiash: i don't know | 14:31 |
| openstackgerrit | Quique Llorente proposed openstack-infra/zuul-jobs master: WIP: Default private_ipv4 to use public_ipv4 address when null https://review.openstack.org/623294 | 14:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 14:42 |
| *** pabelanger has joined #zuul | 14:43 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 14:44 |
| quiquell | tobiash: Do you know if it's easy to have the seam console stream we have at zuul web for jobs but at command line ? | 14:51 |
| quiquell | tobiash: like piping it with a command | 14:52 |
| quiquell | I suppose it involes using websockets and all | 14:52 |
| Shrews | quiquell: finger UUID@zuul.openstack.org where UUID is the running job uuid (can pull this from the web interface path) | 14:55 |
| quiquell | Shrews: thanks | 15:03 |
| corvus | jkt: your assumption about how openstack-infra sets up main.yaml is correct: out-of-band ansible/puppet | 15:07 |
| corvus | tristanC: 'nodepool hold' was removed because it doesn't work with the nodepool locking system. | 15:08 |
| avass | zuul_return doesn't seem to work on windows nodes | 15:08 |
| corvus | avass: that's unexpected; do you know why? | 15:09 |
| corvus | avass: oh wait | 15:10 |
| corvus | avass: zuul_return has to run on the executor | 15:10 |
| avass | one second I'll try to reproduce it. It works with delegate_to: localhost but throws an exception otherwise | 15:10 |
| corvus | avass: so yes, delegate_to: localhost is the right way to use it | 15:10 |
| corvus | we should, erm, probably document that. | 15:11 |
| avass | alright, but then the variables seem to be undefined in the child job | 15:11 |
| avass | haha | 15:11 |
| corvus | okay, we did "document" it, in that we mentioned it in a sentence, and then proceeded to omit the localhost part from every example. i'll make a patch | 15:12 |
| avass | ah, I see it now | 15:13 |
| corvus | avass: oh, what was the issue? | 15:13 |
| avass | I mean the documentation :) | 15:13 |
| corvus | avass: oh, so the vars in child job is still an issue? can you pastebin the task that uses zuul_return so i can look at it? | 15:14 |
| jkt | corvus: thanks; I saw some initial (?) work for auto-restarting scheduler upon changes to the tenant config, but it appears to be disabled | 15:15 |
| jkt | corvus: is that a manual step now? | 15:16 |
| avass | http://paste.openstack.org/show/742309/ | 15:17 |
| corvus | jkt: i don't recall that -- we've needed a reconfiguration (which is much less disruptive than a restart) to reload main.yaml since 3.0. the main.yaml basically bootstraps the system to tell zuul where all the git repos are so it can find the rest of the configuration. | 15:18 |
| tobiash | corvus, avass: there is a change in review that converts zuul_return into a plugin: https://review.openstack.org/591168 | 15:19 |
| tobiash | this way we won't have to delegate it to localhost | 15:19 |
| corvus | that'll be nice :) | 15:19 |
| jkt | corvus: https://git.openstack.org/cgit/openstack-infra/system-config/tree/playbooks/zuul_reconfigure.yaml looks like a stub | 15:19 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Include "delegate_to: localhost" in zuul_return examples https://review.openstack.org/630700 | 15:20 |
| corvus | jkt: ah, yes, that -- sorry, i thought you meant in zuul itself. yes, that should, eventually, be part of a post-merge job in openstack's zuul to reload zuul's configuration when changes land to that file. it's waiting on us to move more of our config management to ansible. | 15:21 |
| tobiash | corvus: what do you think about adding run: once to this too? ^ | 15:21 |
| corvus | tobiash: probably a good idea | 15:21 |
| pabelanger | avass: corvus: https://review.openstack.org/591168/ makes zuul_return more user friendly for playbooks, but still need to address tobiash comments | 15:21 |
| pabelanger | ah, just seen tobiash pointing to it | 15:22 |
| jkt | corvus: and all projects to "handle" via zuul must be defined in a per-tenant manner via its main config file, right? (and only activated by that reconfig op) | 15:23 |
| corvus | jkt: exactly | 15:23 |
| corvus | avass: there were some yaml errors in that (colon on line 2, indentation on lines 4-6); it should look like http://paste.openstack.org/show/742310/ | 15:24 |
| jkt | thanks, I hope this is all from me for now :) | 15:24 |
| corvus | avass: wait | 15:24 |
| corvus | avass: sorry, like this: http://paste.openstack.org/show/742311/ | 15:24 |
| corvus | jkt: you're welcome :) | 15:25 |
| avass | corvus: yeah, that makes sense | 15:26 |
| avass | corvus: but it doens't seem like it fixed it | 15:29 |
| corvus | avass: do you have the inventory file from the child job? | 15:29 |
| avass | corvus: http://paste.openstack.org/show/742313/ | 15:32 |
| corvus | avass: i mean like this: http://logs.openstack.org/68/630468/8/check/system-config-build-image-gerrit/ca32dee/zuul-info/inventory.yaml (we have our base job set up to save the inventory file on the log server) | 15:33 |
| avass | corvus: oh, I'm not using the default base job | 15:37 |
| corvus | avass: if you add https://zuul-ci.org/docs/zuul-jobs/roles.html#role-log-inventory to your base job, you'll get the inventory saved too; it can be very useful in debugging. | 15:38 |
| corvus | avass: if you want to continue debugging without that change, you can run 'zuul-executor keep' on an executor node, and it will not delete the build directory when it's finished a job. you can then look at the logs to see what the full path is (should be like: /var/lib/zuul/builds/<uuid>/...) and look in there to see the inventory file | 15:39 |
| corvus | avass: that might give us a clue if we see the variable showing up someplace unexepected in that file, or not at all. | 15:39 |
| corvus | avass: ('zuul-executor unkeep' turns deletion back on again when you're done) | 15:40 |
| corvus | avass: also, can you pastebin the project-pipeline configuration where these jobs are running? | 15:41 |
| avass | corvus: not sure if I'm allowed to do that to be honest | 15:43 |
| avass | corvus: post the pipeline that is | 15:44 |
| corvus | avass: no problem; i mostly wanted to see what the "dependencies" configuration between the jobs was | 15:45 |
| avass | corvus: oh, it's independent | 15:46 |
| corvus | avass: i assume it looks like this? http://paste.openstack.org/show/742314/ | 15:46 |
| corvus | (since i haven't fixed that bug where dependencies are ignored if they aren't in the project-pipeline config; that's on my list this week) | 15:47 |
| avass | orvus: oh, no. the job is inheriting from the base job that fetches information and tries to return to jobs inheriting from it | 15:47 |
| corvus | avass: ah there's our problem; i think it's one of terminology. zuul_return is used to pass information from one job to a second (dependent) job, which runs after the first. | 15:49 |
| avass | corvus: ah | 15:49 |
| *** hashar has quit IRC | 15:50 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 15:50 |
| corvus | avass: i'm guessing you have two playbooks, one in a parent job, and one in a child job which inherits from the parent. and when the child job runs, both playbooks are run, and you want to pass data from the parent's playbook to the child's. | 15:50 |
| avass | corvus: exactly | 15:51 |
| corvus | avass: i'd probably recommend using plain ansible to share that data... maybe via json files... | 15:51 |
| corvus | avass: maybe write a json file with the "copy" module, and then use "include_vars" to load it | 15:52 |
| avass | corvus: yeah, that will probably do it. | 15:53 |
| corvus | avass: the nice thing about that is it's testable outside of zuul | 15:53 |
| *** avass has quit IRC | 16:05 | |
| *** hashar has joined #zuul | 16:11 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add governance document https://review.openstack.org/622439 | 16:12 |
| * fungi spies https://zuul-ci.org/docs/zuul/governance.html | 16:19 | |
| *** pcaruana has quit IRC | 16:20 | |
| corvus | tobiash: i'm worried about 630472 landing too early (mordred has +2d it already). i'd like to wait until we've actually landed the spec before we starting doing big changes like that. i'm going to put an administrative -2 on it for now, ok? | 16:24 |
| mordred | corvus: reading scrollback - we could make zuul_return into an action plugin instead of a normal module so that it always runs on the executor regardless of the playbook ... which I now see tobiash has linked to a patch doing | 16:27 |
| fungi | could it depends-on the spec change? | 16:27 |
| tobiash | corvus: fine for me | 16:27 |
| pabelanger | mordred: Yup, do you have thoughts on path there? | 16:30 |
| pabelanger | I can push up a patch to get it landed | 16:30 |
| *** chandankumar is now known as codemonster | 16:31 | |
| mordred | pabelanger: well - if we keep it, we should definitely do the path validation on the value ... I don't see anywhere that we're passing path - but it has been part of the argspec for the module version, so removing it would probably need a deprecation period | 16:34 |
| tobiash | mordred: the path argument is completely undocumented | 16:35 |
| tobiash | I even wasn't aware of it until I reviewed pabelanger's change | 16:35 |
| mordred | tobiash: hrm. good point | 16:36 |
| mordred | corvus: do you have an opinion? | 16:36 |
| corvus | mordred: i think it's probably okay to drop; i added it in case it was useful, but i don't think a use has come up in the past year | 16:37 |
| mordred | cool. yeah - it also doesn't seem useful to me | 16:37 |
| *** rfolco is now known as rfolco|brb | 16:37 | |
| *** corvus is now known as thecount | 16:38 | |
| *** thecount is now known as corvus | 16:38 | |
| *** zxiiro is now known as zxiiro-away | 16:40 | |
| *** themroc has quit IRC | 16:46 | |
| *** codemonster is now known as chkumar|out | 16:49 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 16:54 |
| pabelanger | corvus: mordred: tobiash: i believe ^ should address safe_path comment | 16:54 |
| pabelanger | I can also send on ML post about removing it, just to be safe | 16:54 |
| tobiash | pabelanger: commented | 16:59 |
| pabelanger | tobiash: ah, thanks | 17:02 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:03 |
| pabelanger | tobiash: I didn't run remote tests, lets see what zuul.o.o says now | 17:04 |
| corvus | if it's an action plugin, is it okay to still have delegate_to: localhost? | 17:04 |
| corvus | (will that just be ignored?) | 17:05 |
| pabelanger | believe it is ignored | 17:06 |
| pabelanger | we can add test for that too | 17:06 |
| tobiash | corvus: yes, that should be ok | 17:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:09 |
| pabelanger | and test to confirm | 17:09 |
| tobiash | corvus: further docs to action plugins: https://docs.ansible.com/ansible/latest/plugins/action.html | 17:11 |
| tobiash | So in our case we actually just skip the actual module | 17:11 |
| *** gtema has joined #zuul | 17:14 | |
| *** pcaruana has joined #zuul | 17:15 | |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add upload-puppetforge role https://review.openstack.org/627553 | 17:17 |
| pabelanger | has anybody thought about submitting a talk for open infra summit in Denver? I know CFP is still open, but haven't found the motivation to write up anything yet | 17:22 |
| clarkb | pabelanger: I haven't, but was thinking a good talk could be a bit more in depth presentation about how to run zuul (and in general CI) at scale | 17:28 |
| clarkb | things like the fair scheduling, distributed mergers and executors, multiple clouds (or other nodepool providers etc | 17:28 |
| pabelanger | I'd attend :) | 17:29 |
| pabelanger | would be good to also give numbers to people | 17:29 |
| pabelanger | was thinking of submitting one about migrating from jenkins to zuul | 17:29 |
| clarkb | I don't know that I'll be able to do that myself or at least by myself considering I'll be prepping for PTG and other infra tasks | 17:29 |
| pabelanger | or how to CD an openstack cloud from zuul, for the purpose of using it for testing resources | 17:29 |
| *** gtema has quit IRC | 17:38 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 17:39 |
| *** hashar has quit IRC | 17:42 | |
| *** hashar has joined #zuul | 17:42 | |
| *** hashar_ has joined #zuul | 17:45 | |
| *** hashar has quit IRC | 17:46 | |
| *** hashar_ has quit IRC | 17:47 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 17:52 |
| *** electrofelix has quit IRC | 17:57 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Add a timeout for the image build https://review.openstack.org/629923 | 18:06 |
| pabelanger | tobiash: corvus: mordred: https://review.openstack.org/591168/ is ready for review again | 18:08 |
| pabelanger | I guess, maybe we should add a reno note about the change | 18:08 |
| *** panda is now known as panda|off | 18:12 | |
| *** jpena is now known as jpena|off | 18:12 | |
| tobiash | I think reno makes sense | 18:14 |
| *** rfolco|brb is now known as rfolco | 18:14 | |
| corvus | yeah, maybe just to let folks know they don't need 'delegate_to' anymore | 18:35 |
| *** remi_ness has joined #zuul | 18:47 | |
| *** remi_ness has quit IRC | 18:54 | |
| *** pcaruana has quit IRC | 19:08 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Add release note for zuul_return action plugin update https://review.openstack.org/630760 | 19:09 |
| corvus | pabelanger: can you squash that? | 19:09 |
| pabelanger | sure, was trying to keep tobiash review | 19:09 |
| corvus | if the only change is a reno, i'm sure the re-review will be easy :) | 19:10 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul master: Convert zuul_return into action plugin https://review.openstack.org/591168 | 19:11 |
| pabelanger | doh | 19:11 |
| pabelanger | Oh, that is right. | 19:11 |
| openstackgerrit | Merged openstack-infra/zuul master: sql: add buildset uuid column https://review.openstack.org/630034 | 19:13 |
| *** remi_ness has joined #zuul | 19:25 | |
| pabelanger | corvus: Shrews: do you think we'll have a nodepool release this week too? The 1/2 of code for zuul-executor zones is still unreleased. | 19:32 |
| corvus | Shrews: ^? :) | 19:54 |
| clarkb | openstack is running up to date launchers and builders. I dont think there are problems other than the dib build timeout fix | 19:55 |
| clarkb | may want to get that in or do a releasethen quickly get timeout released after | 19:56 |
| pabelanger | wfm | 19:59 |
| *** remi_ness has quit IRC | 20:05 | |
| corvus | what's the urgency for the dib timeout fix? | 20:08 |
| pabelanger | I just realized, we don't actually install encrypt_secret if you pip install zuul. Think we should maybe update that to zuul-encrypt(-secrets) and add entrypoint | 20:08 |
| clarkb | corvus: openstack found that dib builds were hanging blocking the image build queues twice in two months. Probably low overall | 20:08 |
| corvus | pabelanger: well, it's not really intended to be run on zuul servers; it's intended to be run by anyone, via wget. we *could* do that, but then i feel like we're undercutting the message that "you don't need to install zuul to run this" | 20:09 |
| corvus | clarkb: okay, so there isn't some new change in the landscape that we urgently need to respond to (obviously getting the fix in asap is good, but it doesn't need to supercede normal process) | 20:10 |
| clarkb | correct | 20:10 |
| corvus | Shrews, pabelanger, clarkb: so it sounds like we could tag master now, then do a new release once the timeout stuff has settled | 20:10 |
| clarkb | yes I think so | 20:11 |
| pabelanger | corvus: sure, I don't have a strong view on it. I needed it on a remote server, and defaulted to pip install zuul, and noticed it wasn't there. I then proceeded to git clone the repo for it. | 20:11 |
| pabelanger | wget would have worked, just didn't think to do it. | 20:11 |
| corvus | clarkb: we've restarted with the config parsing changes? | 20:11 |
| clarkb | corvus: let me find where I noted the sha1 | 20:12 |
| clarkb | f8bf6afd08bcdcee06b377fda71ad543a63f2ffc was what i restarted on | 20:13 |
| clarkb | that does not include config parsing changes | 20:13 |
| corvus | pabelanger: in this case, it sounds like a win, since you've now realized that you don't need to install zuul just to get the tool; the main thing that could be improved is finding a way for you to have realized that before installing it. :) | 20:14 |
| pabelanger | agree | 20:14 |
| corvus | clarkb: ok. nothing user-visible depends on the config changes, so i think we can release that sha (but i wouldn't want to release later without testing in case we missed an edge case in config loading) | 20:15 |
| pabelanger | okay, I need to #dadops here for a bit, will be back online in a few hours. any version of nodepool works for me, if openstack isn't running master | 20:15 |
| corvus | clarkb, pabelanger, Shrews: i think we can tag nodepool f8bf6afd08bcdcee06b377fda71ad543a63f2ffc as 3.4.0. sound good? | 20:15 |
| clarkb | ++ there was at least one feature added (the metadata support in openstack driver) | 20:16 |
| clarkb | so commit and tag lgtm | 20:16 |
| tobiash | corvus, clarkb: that will miss this fix for static winrm nodes: bcaa264 - Default host_keys to empty list in static driver | 20:30 |
| tobiash | but I think the one who had this problem (I forgot the name) doesn't depend on a release | 20:31 |
| corvus | drat, sorry i missed that | 20:32 |
| corvus | but it sounds like we can still go ahead and release f8bf, and that will end up in the timeout fix which will probably follow shortly | 20:32 |
| Shrews | corvus: yeah. changing the dib timeout stuff for ian is proving a bit problematic atm | 20:33 |
| tobiash | ok, ++ for f8bf | 20:33 |
| tobiash | btw, think we should land 613261 and do another zuul release soon (maybe next week) | 20:34 |
| tobiash | without this the skip child jobs feature is unusable and can even wedge zuul | 20:35 |
| corvus | Shrews: any reason that's a CLI argument and not a config file entry? | 20:35 |
| Shrews | corvus: no, only following the pattern for the num uploaders/builders. we don't have any "count" of things in the config for builder, atm | 20:36 |
| corvus | Shrews: ok, i'm going to suggest the other way with more words on the review | 20:37 |
| Shrews | corvus: that's fine, thx | 20:37 |
| Shrews | i don't know how my most recent change to the fake dib command "fixed" the nodepool-zuul-functional job. that's a puzzler | 20:40 |
| Shrews | *sigh* | 20:40 |
| Shrews | corvus: clarkb: tobiash: can we please get a consensus on whether the timeout should be per-image or global? folks keep suggesting either way and i'd like to get that settled | 20:43 |
| clarkb | Shrews: the ideal would be to be per image | 20:43 |
| tobiash | Shrews: I'm ok with either | 20:44 |
| clarkb | Shrews: that way you don't have to set all iamges to a ~12 hour timeout if you are tobiash. That said I think it is ok to do a global timeout value if it is siginificantly simpler to implement | 20:44 |
| Shrews | so, i hear "either" again. corvus, final vote? | 20:45 |
| tobiash | Shrews: you can rrad my 'either' as take the choice of less effort ;) | 20:46 |
| Shrews | I'll make the executive decision then... i'll just do the per-image route, even though it's more code. | 20:49 |
| Shrews | Disagreeing voices may show their disapproval with a patch set of their own. :) | 20:50 |
| corvus | Shrews: that wfm. we can always upgrade it to "both" later. :) | 20:51 |
| *** hashar has joined #zuul | 21:03 | |
| dmsimard | This isn't the first time I see something like this when trying to see the console recently: https://i.imgur.com/zZ3j3rl.png .. I noticed that when it happens, the progress bar on zuul web doesn't seem to actually progress. | 21:09 |
| clarkb | dmsimard: it means the jobs hasn't actually started aiui | 21:09 |
| clarkb | but there is a node allocated for the job | 21:10 |
| dmsimard | clarkb: right, but the "end of stream" loop ? | 21:10 |
| clarkb | probably due to the governor in the executors | 21:10 |
| clarkb | dmsimard: the web client is trying to connect to the stream ina loop | 21:10 |
| dmsimard | the end of stream loop is new to me | 21:10 |
| clarkb | eventually it should connect once the job starts running | 21:10 |
| dmsimard | oh, is it, huh | 21:10 |
| dmsimard | http://zuul.openstack.org/stream/4324b9b7d7ed45278a9ec8e75595d115?logfile=console.log has been looping for a while now ... | 21:11 |
| *** maxamillion has quit IRC | 21:11 | |
| clarkb | dmsimard: http://grafana.openstack.org/d/T6vSHcSik/zuul-status?orgId=1 zero accepting executors | 21:12 |
| clarkb | for about half an hour, that seems long | 21:12 |
| dmsimard | ok, we can move to -infra | 21:12 |
| dmsimard | eh, of course now it starts | 21:13 |
| dmsimard | so should we wait until an executor has picked up the nodeset before making the console available ? | 21:17 |
| dmsimard | otherwise the UX is not awesome | 21:17 |
| corvus | dmsimard: theoretically that would mean waiting for the build to have reported 'start'; i'm not sure what the trigger is for that atm. | 21:19 |
| dmsimard | clarkb: I was looking at the ram sensor http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/sensors/ram.py | 21:37 |
| dmsimard | It doesn't take swap into account... should it ? | 21:37 |
| dmsimard | They're two different things from the perspective of psutils | 21:37 |
| dmsimard | ex: http://paste.openstack.org/show/742354/ | 21:37 |
| clarkb | dmsimard: possibly? or maybe it should be a different sensor | 21:38 |
| clarkb | dmsimard: however in this case it would just result in us not running any more jobs | 21:38 |
| clarkb | really what we need to figure out is how to reduce the memory overhead of the zuul executor (I think) | 21:38 |
| dmsimard | Browsing through upstream Ansible issues to see if there's anything obvious standing out... Unless mistaken, Zuul is currently running Ansible 2.5.14 which contains some significant memory usage improvements that landed either in 2.4 or early on in 2.5 | 21:45 |
| dmsimard | They're up to 2.6.11 and 2.7.5 now but the issues I found were fixed before then | 21:50 |
| fungi | the osf newsletter editors want to run a spotlight piece on zuul for this week's edition, so i've been trying to put together some notes at https://etherpad.openstack.org/p/a5OnSIreGH and would appreciate links anyone has handy for proposed changes or design discussions about adding gitlab or bitbucket connection drivers to zuul, and a gce provider driver to nodepool (did i perhaps imagine one or more of | 21:51 |
| fungi | those?) | 21:51 |
| clarkb | dmsimard: manageLoad which determines if the governor should kick is is run after grabbing a job handle but before executing it | 21:55 |
| clarkb | dmsimard: this explains your time gap in the log | 21:55 |
| clarkb | (I think) | 21:55 |
| clarkb | comment says this is done to make unittesting easier | 21:57 |
| clarkb | we might be able to turn that into a test only behavior | 21:57 |
| pabelanger | corvus: f8bf6a wfm | 22:02 |
| *** hashar has quit IRC | 22:13 | |
| fungi | grepping the mailing list archives turns up no mention of gce, only a note that someone at ansiblefest had requested bitbucket, and no mention of gitlab since november 28 (though that thread is probably at least notable enough to link for it) | 22:38 |
| fungi | hrm, or not, it's more about aggregating changes under test | 22:41 |
| fungi | aha, found some irc conversation for bitbucket | 22:42 |
| fungi | found people in the channel log talking about the possibility they might write drivers for bitbucket and gitlab, but no firm commitments | 22:46 |
| fungi | and none in the past 6 months at any rate | 22:47 |
| fungi | amd last real gce driver mention in channel was last april | 22:47 |
| fungi | so i'll just take those out | 22:47 |
| mordred | fungi: yah - I don't think those have made much traction | 22:50 |
| fungi | i don't suppose tristanC's vmware driver has garnered much interest either? https://review.openstack.org/554463 | 22:51 |
| fungi | maybe mentioning and linking to it will at least encourage someone to help finish it | 22:57 |
| ianw | Shrews: stderr & stdout shoudl be caught by dib's --logfile output | 22:59 |
| ianw | tobiash: dib in parallel probably isn't as isolated as you'd hope. certainly we have a few lockfiles and points where we deal with concurrency, but it's not very well tested | 23:01 |
| mordred | fungi: fbo has made good progress on the pagure driver: https://review.openstack.org/#/c/604404/ | 23:03 |
| fungi | i saw! included a mention/link for it in the writeup already | 23:04 |
| mordred | fungi: which, incidentally, has led to improvements to pague in a good virtuous circle | 23:04 |
| mordred | yay! | 23:04 |
| fungi | excellent | 23:04 |
| ianw | Shrews: the capturing is based on what devstack does redirects via a backgrounded python process. it's been working fairly well in unit tests where builds are now split out for easier tracing of issues, e.g. http://logs.openstack.org/20/619120/18/gate/dib-functests-xenial-python3/a9d9b45/logs/ | 23:05 |
| *** rlandy has quit IRC | 23:25 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Add dogpile.cache master to the -src tests https://review.openstack.org/625457 | 23:33 |
| *** dkehn has left #zuul | 23:33 | |
| *** dkehn has joined #zuul | 23:37 | |
| pabelanger | fungi: one of the human interested in gitlab has sadly moved on from the original company, they are no longer doing zuul | 23:41 |
| fungi | :( | 23:43 |
| fungi | sad for us anyway | 23:43 |
| fungi | perhaps it's happy for them though | 23:43 |
| clarkb | https://gitlab.com/gitlab-org/gitlab-ce/blob/master/app/models/project_services/ci_service.rb looks like the starting point of adding such support to gitlab for zuul | 23:43 |
| clarkb | (you can also do more restricted webhooks( | 23:43 |
| tristanC | fungi: thanks for putting the list together, i'm not aware of any wip for the vmware driver | 23:44 |
| fungi | well, at least there's work someone can build on for it | 23:44 |
| *** rfolco has quit IRC | 23:45 | |
| clarkb | fungi: oh well thats just their generic ci service base class in gitlab | 23:45 |
| clarkb | unsure of any specific zuul work | 23:45 |
| fungi | clarkb: sorry, i was responding to tristanC with that ;) | 23:45 |
| *** rfolco has joined #zuul | 23:45 | |
| corvus | pabelanger: oh, then maybe we should put out a call for interest on the mailing list, since i think some folks were waiting on that work to start; maybe there's enough interest to put a group together? | 23:47 |
| pabelanger | +1 | 23:48 |
| pabelanger | we in ansible-network are likely still interested in it, just haven't discussed it much on our side recently | 23:49 |
| tristanC | perhaps i should abandon those untested driver that are starting to rot, and replace them with a "how to write a driver" documentation | 23:49 |
| tristanC | Shrews: do you think we can mark the current drivers api as stable (e.g. version=1) ? | 23:51 |
| fungi | mmm, if you're planning to abandon the vmware and azure drivers, maybe i shouldn't mention them in the article | 23:52 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!
