| *** jamesmcarthur has joined #zuul | 00:01 | |
| mordred | clarkb: nope. I lost the session and wasn't running it in screen because FAIL | 00:03 |
|---|---|---|
| mordred | clarkb: oh - that was after the time I ran in to duplicate key issues because the first mttest image didn't get deleted :) | 00:03 |
| mordred | clarkb: trying once more | 00:03 |
| clarkb | ya they leak :) | 00:04 |
| clarkb | note I did not delete mttest as part of my leak cleanups | 00:04 |
| *** michael-beaver has quit IRC | 00:09 | |
| fungi | i've got tests.unit.test_scheduler.TestScheduler.test_head_is_dequeued_once failing with MismatchError: 14 != 15 | 00:24 |
| fungi | is that a known instability in the unit tests? | 00:25 |
| fungi | failed on py35 but not py36 | 00:25 |
| fungi | http://logs.openstack.org/70/662870/8/check/tox-py35/9aabd57/ | 00:25 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add sanity checks https://review.opendev.org/664471 | 00:48 |
| *** jamesmcarthur has quit IRC | 00:52 | |
| *** igordc has quit IRC | 00:56 | |
| *** mattw4 has quit IRC | 00:57 | |
| *** threestrands_ has joined #zuul | 01:11 | |
| *** swest has quit IRC | 01:12 | |
| *** threestrands has quit IRC | 01:14 | |
| *** threestrands_ has quit IRC | 01:21 | |
| *** swest has joined #zuul | 01:27 | |
| *** sshnaidm has quit IRC | 01:33 | |
| *** sshnaidm has joined #zuul | 01:35 | |
| *** jamesmcarthur has joined #zuul | 01:35 | |
| *** jamesmcarthur has quit IRC | 01:37 | |
| *** bhavikdbavishi has joined #zuul | 02:05 | |
| *** bhavikdbavishi has quit IRC | 02:09 | |
| *** michael-beaver has joined #zuul | 02:12 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Update the readme for new role options https://review.opendev.org/664484 | 02:14 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 02:22 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Update the readme for new role options https://review.opendev.org/664484 | 02:49 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 02:49 |
| *** bhavikdbavishi has joined #zuul | 02:56 | |
| *** bhavikdbavishi1 has joined #zuul | 02:59 | |
| *** bhavikdbavishi has quit IRC | 03:01 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:01 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Update the readme for new role options https://review.opendev.org/664484 | 03:12 |
| *** bhavikdbavishi1 has joined #zuul | 03:18 | |
| *** bhavikdbavishi has quit IRC | 03:20 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:20 | |
| *** swest has quit IRC | 04:33 | |
| *** bjackman_ has joined #zuul | 04:36 | |
| *** michael-beaver has quit IRC | 05:02 | |
| *** saneax has joined #zuul | 05:03 | |
| *** saneax has quit IRC | 05:07 | |
| *** swest has joined #zuul | 05:16 | |
| *** swest has quit IRC | 05:18 | |
| *** bhavikdbavishi1 has joined #zuul | 05:19 | |
| *** bhavikdbavishi has quit IRC | 05:19 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:19 | |
| *** swest has joined #zuul | 05:19 | |
| *** bhavikdbavishi has quit IRC | 05:23 | |
| *** bhavikdbavishi has joined #zuul | 05:41 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 05:47 |
| *** saneax has joined #zuul | 05:48 | |
| openstackgerrit | Merged zuul/zuul master: Annotate logs around build states https://review.opendev.org/661489 | 05:51 |
| *** threestrands has joined #zuul | 05:52 | |
| *** threestrands has quit IRC | 05:53 | |
| *** pcg has joined #zuul | 05:54 | |
| *** pcg has left #zuul | 05:54 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: document multiple labels shortcomming https://review.opendev.org/662954 | 05:55 |
| openstackgerrit | Merged zuul/zuul master: Annotate logs around reporting https://review.opendev.org/661490 | 06:12 |
| openstackgerrit | Merged zuul/zuul master: Annotate logs around finished builds https://review.opendev.org/661491 | 06:33 |
| *** pcaruana has joined #zuul | 06:55 | |
| *** badboy has joined #zuul | 07:13 | |
| *** saneax has quit IRC | 07:17 | |
| *** flepied has quit IRC | 07:20 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:22 |
| *** gtema has joined #zuul | 07:24 | |
| *** bjackman_ has quit IRC | 07:42 | |
| *** hashar has joined #zuul | 07:45 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** flepied has joined #zuul | 07:54 | |
| *** saneax has joined #zuul | 07:58 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 08:07 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 08:35 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Parallelize github event processing https://review.opendev.org/662818 | 08:39 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 08:52 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 08:52 |
| *** tobberydberg has quit IRC | 09:22 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:25 |
| *** tobberydberg has joined #zuul | 09:31 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:45 |
| *** gtema has quit IRC | 09:47 | |
| *** panda has quit IRC | 09:47 | |
| *** panda has joined #zuul | 09:49 | |
| *** gtema has joined #zuul | 09:58 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 09:59 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Allow to select the merge method in Github https://review.opendev.org/617949 | 10:08 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support squash merge in Github https://review.opendev.org/661096 | 10:08 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 10:13 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Disable gc in test_scheduler.TestExecutor https://review.opendev.org/661316 | 10:14 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 10:14 |
| fbo | corvus: hi, thanks for the review on the Pagure driver, I've fixed gc patch https://review.opendev.org/661316 according to your request. | 10:17 |
| *** pcaruana has quit IRC | 10:19 | |
| *** gtema has quit IRC | 10:28 | |
| *** gtema has joined #zuul | 10:28 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 10:31 |
| *** bhavikdbavishi has quit IRC | 10:39 | |
| *** pcaruana has joined #zuul | 11:09 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 11:31 |
| *** jpena is now known as jpena|lunch | 11:38 | |
| *** bhavikdbavishi has joined #zuul | 11:45 | |
| *** rlandy has joined #zuul | 11:58 | |
| *** rlandy is now known as rlandy|ruck | 11:59 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 12:02 |
| *** badboy has quit IRC | 12:13 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 12:24 |
| *** jpena|lunch is now known as jpena | 12:43 | |
| *** zbr|ruck is now known as zbr|rover | 12:44 | |
| zbr|rover | does anyone knows if is legal to add extra text after URL on Depens-On or Needed-By:? | 12:51 |
| *** jamesmcarthur has joined #zuul | 12:51 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Adds tox-molecule job https://review.opendev.org/664230 | 12:54 |
| fungi | for needed-by it doesn't matter because that's not a zuul feature. i do not know if it will confuse zuul on depends-on, but it's probably easy to try | 12:55 |
| fungi | just remember that git considers footers to be one thing per line, so at least don't go adding comments so long that you need to wrap them onto additional lines | 12:56 |
| fungi | (granted, git commit footers are merely a convention, but some applications do expect to interpret them one per line) | 12:57 |
| zbr|rover | fungi: i asked because on more complex changes where you have 2-3 depends-on, it makes code easier to read if you mention repo-name after the URL. | 12:57 |
| zbr|rover | i think i already did this in the past and apparently it does not break anything, but I wanted to ask, just in case. | 12:58 |
| fungi | zbr|rover: makes sense. if it helps, when we upgrade to current gerrit you'll get that by default (change numbers are scoped to projects in notedb, so the project names appear in the change urls) | 12:58 |
| zbr|rover | fungi: super cool! | 12:59 |
| zbr|rover | i used https://review.gerrithub.io/ in the past and loved the UI, i am sure others may not share the same love of it :) | 13:00 |
| *** jamesmcarthur has quit IRC | 13:02 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 13:10 |
| tobiash | corvus: btw, this is getting battle tested since three weeks here in production ^ | 13:25 |
| tobiash | so we're shaking out most bugs on this atm ;) | 13:26 |
| *** jamesmcarthur has joined #zuul | 13:26 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:28 |
| *** armstrongs has joined #zuul | 13:33 | |
| *** bhavikdbavishi has quit IRC | 13:33 | |
| armstrongs | hi guys, i am trying to execute a playbook from a required projects workspace using the run: with a full relative path of the dependent project but it isn't working. Is this possible to do? | 13:34 |
| pabelanger | armstrongs: no, playbooks need to live next to zuul.yaml file | 13:35 |
| pabelanger | you cannot get them from cross projects | 13:35 |
| pabelanger | you're best to move the logic into a role, they create a smaller wrapper playbook | 13:36 |
| armstrongs | cool so to call cross project i could have a playbook that calls the dependant one | 13:36 |
| armstrongs | as long as original playbook lives in the same repo as zuul.yaml | 13:36 |
| pabelanger | yup | 13:38 |
| pabelanger | or, move the playbook into a share repo | 13:38 |
| pabelanger | and update both jobs to then use it | 13:38 |
| armstrongs | so theres no way shadow jobs could allow this at all | 13:38 |
| armstrongs | i have the scenario where we have a standard set of playbooks to deploy apps, teams have config repos to provide inputs to self-serve against them. So really the use case is each team needs to use the playbooks in the central repo and provide the supplimentary self-service repo to give inputs | 13:40 |
| armstrongs | if that makes sense the self-service repos are just var files for each app | 13:40 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 13:42 |
| mordred | clarkb, fungi, corvus, Shrews: patch up in sdk https://review.opendev.org/#/c/664585/ to fix the nodepool regression. I predict you're going to "love" the patch | 13:44 |
| *** electrofelix has joined #zuul | 13:44 | |
| mordred | I particularly enjoy that we got a raw 400 error back with no error message on a "this field wants a string and you passed a boolean" issue | 13:46 |
| pabelanger | armstrongs: yes, you can create new child jobs, which setup vars to the parent and that will work too | 13:47 |
| fungi | right, jobs and roles can be reused between projects (with some limitations around secrets in untrusted repositories), you just can't refer to playbooks in other projects as far as i know | 13:49 |
| fungi | and you can certainly derive new jobs by parenting them to jobs in other repos, or create "variants" of jobs in other repos if you just need to override some parameters in them | 13:50 |
| *** hashar has quit IRC | 13:58 | |
| *** pcaruana has quit IRC | 14:04 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:10 |
| *** michael-beaver has joined #zuul | 14:10 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:11 |
| *** jpena is now known as jpena|afk | 14:14 | |
| *** chandankumar is now known as raukadah | 14:23 | |
| zbr|rover | fungi: clarkb mordred pabelanger : i think I implemented all comments on tox-molecule job at https://review.opendev.org/#/c/664230/ | 14:29 |
| *** pcaruana has joined #zuul | 14:30 | |
| pabelanger | +2 | 14:30 |
| *** pcaruana has quit IRC | 14:31 | |
| *** pcaruana has joined #zuul | 14:31 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:32 |
| corvus | evrardjp, clarkb, pabelanger: i'm trying to think of alternate approaches to https://review.opendev.org/662828 -- i'm frustrated that the date/time is not available to an untrusted zero-node job. that seems like it would be generally very useful. perhaps we should consider modifying zuul to stick that into the executor's fact cache? or, add a new zuul. variable? | 14:32 |
| evrardjp | so... Yeah I tried to have this facts gathered selectively, but as you could guess, the whole setup, even filtered, was not allowed. | 14:33 |
| evrardjp | so to think of alternative approaches, I think what I would like was to freeze by sha | 14:34 |
| evrardjp | that could work, as the lookup('url' would be allowed. | 14:34 |
| evrardjp | but then the build and publish would become quite ugly | 14:34 |
| evrardjp | corvus: not sure what you mean by adding a new zuul variable | 14:35 |
| evrardjp | I mean I am not sure what this technically involves | 14:35 |
| corvus | evrardjp: like "{{ zuul.executor.datetime }}" | 14:35 |
| evrardjp | yeah got that part :) | 14:35 |
| clarkb | corvus: may even be worth adding to ansible as a var that isnt a fact | 14:35 |
| clarkb | since it can lookup time too | 14:35 |
| corvus | evrardjp: both of my suggestions are small code changes to the executor source code | 14:36 |
| corvus | clarkb: yeah, i think if we're automatically adding variables which aren't facts, we should stick it in the zuul. hierarchy to avoid collisions | 14:36 |
| evrardjp | does the executor now the target state of the repo for which the change is made? | 14:36 |
| evrardjp | (sorry I am thinking outside the box here) | 14:37 |
| corvus | evrardjp: yes, but i don't think we set that as a variable for changes (only for ref-updates) | 14:38 |
| evrardjp | corvus: could you remind me why the change_<changeid> was added in the past, and not make the final tag part of the upload? | 14:38 |
| evrardjp | corvus: ok on that | 14:38 |
| evrardjp | I suppose the change_<change number> question is because we only want the tag applied post-merge | 14:39 |
| pabelanger | I wonder what facts are gathered with gather_subset=!all' | 14:39 |
| pabelanger | will have to test that out locally | 14:39 |
| corvus | evrardjp: because when we upload an image, we don't know yet if the change for it will actually land, so promote is responsible for finding the changes which actually did land and retagging them with the 'real' tags. | 14:39 |
| evrardjp | pabelanger: I think in this case, it's the setup module that wouldn't be allowed to run in promote pipeline, so it doesn't really matter | 14:40 |
| corvus | clarkb: but i figured it would be okay to "shadow" a known ansible fact with an equivalent value. it would make more things work out of the box with less zuul-specific knowledge. | 14:41 |
| pabelanger | evrardjp: I was thinking more of https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L2062 | 14:41 |
| evrardjp | corvus: well that could as well be a post job | 14:41 |
| pabelanger | all jobs run that today | 14:41 |
| pabelanger | execpt localhost | 14:41 |
| fungi | it's unclear to me how a static timestamp is especially useful... or is there some way for it to be the actual time and not just the time at which the variable was resolved and stored for that build? | 14:41 |
| pabelanger | but cannot remember what facts it gather | 14:41 |
| corvus | evrardjp: it's not actually the pipeline that restricts it, but rather that the playbook is in an untrusted repo (zuul-jobs) | 14:42 |
| evrardjp | corvus: thanks for improving my knowledge there :) | 14:42 |
| fungi | it the timestamp was stored in a fact, then it's frozen at the point at which fact gathering happened, right? | 14:42 |
| corvus | fungi: evrardjp is going for an approximate date, so the slight error/race isn't too important | 14:42 |
| fungi | ahh | 14:43 |
| corvus | basically a "one-per-day" archive | 14:43 |
| fungi | and having the timestamp be static throughout the build is also acceptable i guess | 14:43 |
| corvus | yeah, if we were to define it as a zuul var, i'd describe it as the "start time" for the build | 14:43 |
| evrardjp | fungi: yeah I don't even mind if the day is different when it starts and when it ends | 14:44 |
| fungi | seems like it would be especially flexible as an ansible feature where the value was expanded at the moment of execution rather than compilation | 14:44 |
| evrardjp | it's just better than not freezing it _at all_ | 14:44 |
| corvus | fungi: that could be done as a lookup plugin or module | 14:44 |
| corvus | (ideally lookup plugin) | 14:44 |
| fungi | yeah, i suppose if one existed we could whitelist it fairly easily | 14:44 |
| fungi | especially if that's all it did | 14:44 |
| evrardjp | it looks better to have a lookup plugin for date to whitelist, than whitelist setup | 14:45 |
| fungi | but sure, build start time as a zuul var also makes fine sense | 14:45 |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 14:45 |
| evrardjp | I think build start time would be cleaner though | 14:45 |
| fungi | i can see both approaches being useful for different things, and not necessarily exclusive of one another | 14:45 |
| evrardjp | I have the impression it's more work long term to have a lookup and whitelist it | 14:46 |
| evrardjp | than expose the current build start time | 14:46 |
| evrardjp | but I might be wrong | 14:46 |
| corvus | yeah, i think it would be weird to have a zuul-specific lookup plugin which does that (though we certainly could) -- that would make more sense to me as something to add to upstream ansible. so that's not short-term :) | 14:47 |
| *** kklimonda has quit IRC | 14:47 | |
| evrardjp | pabelanger: not really sure where this runAnsibleSetup is involved and would be able to expose more data | 14:47 |
| fungi | if a plugin is tightly scoped and poses no breakout risk, whitelisting it in zuul for use on an executor is fairly trivial | 14:47 |
| *** kklimonda has joined #zuul | 14:47 | |
| *** kklimonda has left #zuul | 14:48 | |
| evrardjp | I guess the real question is... does this really need to be in zuul, or can this be in the role | 14:48 |
| evrardjp | (for my case) | 14:48 |
| fungi | and certainly a plugin of this nature could be as simple as a few lines to call and return a single python stdlib function | 14:48 |
| evrardjp | yup | 14:48 |
| evrardjp | fungi: https://review.opendev.org/#/c/662828/1/roles/promote-docker-image/tasks/promote-retag-inner.yaml L12 -- something like that I suppose | 14:49 |
| fungi | but i agree that ideally would go into the ansible source tree as a community plugin | 14:49 |
| pabelanger | evrardjp: we use it before every job starts to confirm the network is up and working as expect, so we prime all nodes (minus executor) with some facts | 14:49 |
| evrardjp | it's unlikely to happen there, as setup covers the same thing | 14:49 |
| pabelanger | eg: http://paste.openstack.org/show/752749/ | 14:49 |
| evrardjp | pabelanger: so you propose the executor to also have this but not expose it? | 14:50 |
| evrardjp | or expose it partially? | 14:50 |
| pabelanger | we could update that to filter that output for facts we consider safe, and add them to localhost fact cache | 14:50 |
| pabelanger | but that is a change in out we gather facts for localhost | 14:50 |
| fungi | evrardjp: is the ansible community opposed to a proliferation of simple and tightly-scoped lookup plugins? seems like there's a definite use case for deployments deciding which plugins are safe and blacklisting the rest (we have at least one such use case) | 14:51 |
| evrardjp | I don't have enough view on how best it would be maintained, this is why I am looking for help :) | 14:51 |
| *** jpena|afk is now known as jpena | 14:51 | |
| fungi | kitchen sink plugins like setup are likely not to meet that use case | 14:52 |
| evrardjp | fungi: I guess you could run lookup shell for the same results, so I see a reason to not carry code | 14:52 |
| evrardjp | which would be an opposite reason to what we want to do (i.e.: NOT run shell) | 14:52 |
| fungi | right, hence my question about use cases involving only allowing certain plugins for safety reasons | 14:53 |
| corvus | the thing i would suggest would be useful upstream is a plugin which gets the *current* time like fungi was suggesting (not the playbook start time as the current setup/fact does). it would help people with issues like this: https://stackoverflow.com/questions/31323604/ansible-date-variable | 14:53 |
| corvus | and https://gist.github.com/halberom/b452df40828839fecabf | 14:53 |
| fungi | right, could be as simple as just returning the result of time.time() | 14:54 |
| corvus | or as complicated as accepting all the arguments to datetime + timedelta :) | 14:54 |
| fungi | that can always be postprocessed | 14:54 |
| fungi | but sure | 14:54 |
| evrardjp | corvus: I should have started by the fact I have strict deadlines | 14:55 |
| evrardjp | hahaha | 14:55 |
| fungi | anyway, i do still think a build start timestamp as a zuul variable is also a good idea | 14:55 |
| evrardjp | so the summary, is ... do everything? :p | 14:55 |
| fungi | i think the summary is tackle whichever one you're keen to work on | 14:56 |
| corvus | evrardjp: yeah, upstream isn't going to be an immediate answer, if we do it, we should do it out of generosity with a view to improving things in the future, but still do something else now :) | 14:56 |
| evrardjp | I am worried about converting timestamp to dates for my use case, as it mean I would also need to "run something" :D | 14:56 |
| corvus | let's see what pabelanger comes up with in exploring what facts get collected with !all | 14:56 |
| dmsimard | fwiw I've been experimenting with a lookup plugin to the new ARA API so you can query the API during the the execution of the playbook itself https://review.opendev.org/#/c/663968/1/tests/integration/lookups.yaml | 14:57 |
| corvus | if we can extend the fact gathering of the executor itself, then that would be the simplest, nicest way to go | 14:57 |
| *** saneax has quit IRC | 14:57 | |
| evrardjp | I would prefer that tbh | 14:57 |
| fungi | facts incorporate a timestamp for when they were gathered, i guess? | 14:58 |
| corvus | if we can't then i think our best options are to shadow ansible_date_time in the executor, or create zuul.build_start_time | 14:58 |
| evrardjp | but I am not really sure where to start debugging this | 14:58 |
| evrardjp | what do you mean by "shadow ansible_date_time" ? | 14:59 |
| corvus | fungi: one of the facts is "ansible_date_time" which you can see here: https://ttl255.com/ansible-getting-date-and-timestamp/ | 14:59 |
| evrardjp | sorry for asking so many questions :) | 14:59 |
| *** hashar has joined #zuul | 14:59 | |
| fungi | got it | 14:59 |
| fungi | so basically if we *can't* safely gather facts, then have zuul somehow inject a fact with the appropriate data type? | 15:00 |
| corvus | evrardjp: we can put our own "facts" in without getting ansible involved: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L411 | 15:00 |
| corvus | fungi: right | 15:00 |
| evrardjp | corvus: haha exactly what I thought doing :p | 15:01 |
| fungi | i suppose that keeps it somewhat ansibleish | 15:01 |
| evrardjp | seed the facts there | 15:01 |
| fungi | so you don't need zuul-specific roles when you're consumnig it | 15:01 |
| evrardjp | I can do a patch to keep that content, have the same structure as ansible_date_time | 15:01 |
| corvus | options: A) pabelanger says it's safe to gather more facts on the executor; B) have the executor write an ansible_date_time fact; C) have the executor add a zuul.build_start_time variable | 15:01 |
| evrardjp | we can still call it ansible_date_time too | 15:01 |
| pabelanger | I think it would be helpful, at least for me, what is considered a safe fact from executor POV. ansible_date_time seems good, but is there others we need / want? | 15:01 |
| evrardjp | I will ansible setup locally real quick | 15:02 |
| corvus | i will eat breakfast | 15:03 |
| evrardjp | corvus: sounds great! | 15:03 |
| evrardjp | pabelanger: hostname maybe? | 15:04 |
| evrardjp | the rest sound very sensitive | 15:04 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add multi-distro support to install-docker https://review.opendev.org/664455 | 15:05 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 15:05 |
| pabelanger | evrardjp: okay, I'll work up a test to see if my hunch is correct | 15:07 |
| evrardjp | pabelanger: that would be awesome. So basically approach A said by corvus. I will propose approach B in parallel. | 15:07 |
| pabelanger | wfm | 15:07 |
| evrardjp | if you could ping me, that would be amazing, as I would 1) learn 2) review | 15:07 |
| evrardjp | pabelanger: gather_subset="date_time" would also be interesting btw | 15:08 |
| evrardjp | oh wait | 15:09 |
| evrardjp | nevermind | 15:09 |
| *** pcaruana has quit IRC | 15:17 | |
| *** bhavikdbavishi has joined #zuul | 15:21 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Remove fact cache restriction for localhost (zuul-executor) https://review.opendev.org/664620 | 15:24 |
| pabelanger | evrardjp: ^ is what I am thinking we could do, however left some questions in commit message | 15:24 |
| pabelanger | also, we'll need to add various tests to validate we are not leaking info | 15:24 |
| evrardjp | oh I see, simple as that. I thought we had some extra validation. I feel dumb now to not have proposed that | 15:29 |
| pabelanger | evrardjp: I think the safest approve, would be 2nd setup task just for localhost (running it first), then we can only place filters on that host | 15:33 |
| pabelanger | vs across all hosts | 15:34 |
| evrardjp | pabelanger: I thought of that :) | 15:37 |
| evrardjp | well my implementation for B could be close to that tbh | 15:38 |
| evrardjp | finishing my series of meetings then will take a crack at it | 15:38 |
| *** pcaruana has joined #zuul | 15:43 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: download-artifacts: only consider the most recent build https://review.opendev.org/664624 | 15:44 |
| *** igordc has joined #zuul | 15:44 | |
| corvus | fungi, clarkb, tobiash: ^ that's the next thing needed to fix the tarball promote jobs | 15:45 |
| *** pcaruana has quit IRC | 15:55 | |
| *** flepied has quit IRC | 15:59 | |
| zbr|rover | clarkb: can we wf tox-molecule now? https://review.opendev.org/#/c/664230/ | 16:13 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Add variable to set the docker download url https://review.opendev.org/664485 | 16:13 |
| openstackgerrit | Merged zuul/zuul-jobs master: download-artifacts: only consider the most recent build https://review.opendev.org/664624 | 16:14 |
| *** gtema has quit IRC | 16:22 | |
| pabelanger | I've raised an item on zuul-discuss about an issue we are having with the github driver, to humans that are intersted | 16:23 |
| clarkb | zbr|rover: one small thing then I think it is ready | 16:24 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Adds tox-molecule job https://review.opendev.org/664230 | 16:25 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Adds tox-molecule job https://review.opendev.org/664230 | 16:25 |
| clarkb | zbr|rover: looks great now. pabelanger if you want to rereview ^ | 16:26 |
| pabelanger | +3 | 16:29 |
| *** kklimonda has joined #zuul | 16:32 | |
| *** jamesmcarthur has quit IRC | 16:32 | |
| clarkb | pabelanger: your email thread honestly seems like one better suited for github and vexxhost support :/ | 16:35 |
| pabelanger | clarkb: yah, expect that to be true. | 16:35 |
| pabelanger | Wanted to get it out there, to spark some discussion. Because I imagine us being the only one who has the isssue | 16:36 |
| pabelanger | clarkb: I looked at zuul.o.o last week, and also seem the same tracebacks | 16:36 |
| corvus | well, if two systems in different cloud providers have the same issue, it's not a network problem in one cloud provider | 16:37 |
| pabelanger | let me look again at logs on zuul.o.o for signs it is happening today | 16:37 |
| clarkb | I did quickly double check there are no AAAA records for github in dns still (there aren't any) as ipv6 has been flaky in $clouds | 16:38 |
| clarkb | also they report no errors on their status site | 16:38 |
| clarkb | they being github | 16:38 |
| *** hashar has quit IRC | 16:39 | |
| pabelanger | http://paste.openstack.org/show/752758/ | 16:39 |
| pabelanger | yah, I can see 55 errors just from today on zuul.o.o | 16:39 |
| corvus | pabelanger: all read timeouts? | 16:40 |
| pabelanger | corvus: yah | 16:41 |
| pabelanger | let me figure out if they are in the same time period | 16:41 |
| corvus | good idea | 16:42 |
| pabelanger | I can also ask sf.io to check their logs | 16:42 |
| corvus | pabelanger: does the system get events for ansible/ansible ? | 16:43 |
| *** [GNU] has quit IRC | 16:43 | |
| pabelanger | corvus: now yes | 16:43 |
| pabelanger | we enabled it maybe 3 weeks ago | 16:43 |
| corvus | pabelanger: if they are for the same time period, it may be worth checking if they are for the same events | 16:44 |
| pabelanger | kk | 16:44 |
| corvus | ie, is it a really big query against the ansible repo or something | 16:44 |
| corvus | pabelanger: i replied to the ml, but also if you wanted to expirement with a longer read timeout, i think this is how you would do it: https://paste.ubuntu.com/p/FYxyzXHxfQ/ | 16:44 |
| pabelanger | ack | 16:44 |
| pabelanger | thanks | 16:45 |
| fungi | well, also github probably has all manner of global load distribution, so just because we saw timeouts from one place on the internet doesn't mean someone else did as they could be hitting entirely different github systems | 16:48 |
| fungi | so a failure correlating these sorts of things doesn't mean it wasn't still broken | 16:48 |
| *** pcaruana has joined #zuul | 16:49 | |
| pabelanger | zuul.a.c: http://paste.openstack.org/show/752762/ | 16:50 |
| pabelanger | zuul.o.o: http://paste.openstack.org/show/752763/ | 16:51 |
| pabelanger | both start having failures around the same time | 16:51 |
| pabelanger | in fact the same event a45aa186-8c16-11e9-9e52-4a6f1ad91061 | 16:51 |
| corvus | pabelanger: you could have github resend that event and see if it happens again | 16:51 |
| pabelanger | corvus: okay, let me figure out how to find it | 16:52 |
| pabelanger | let me do a more recently failure | 16:55 |
| pabelanger | there doesn't seem to be any pagenation here | 16:55 |
| pabelanger | okay, f63ca21a-8c61-11e9-9dc1-c61cd291984c | 16:56 |
| corvus | mhu: i did a quick scan of the jwt stack and it looks good; i think the only thing missing is docs | 16:58 |
| mhu | corvus: great! Yeah I was waiting for validation before adding the docs. I'll add some based on the spec | 17:00 |
| mhu | corvus, I'm also revisiting my docker-compose PoC so that folks can experiment easily, I'll push something this evening | 17:01 |
| *** gtema has joined #zuul | 17:01 | |
| mhu | the hardest part so far is configuring gerrit with oidc, it makes it harder to setup admin and zuul accounts, but this could be skipped | 17:02 |
| openstackgerrit | Merged zuul/zuul-jobs master: Adds tox-molecule job https://review.opendev.org/664230 | 17:03 |
| *** jamesmcarthur has joined #zuul | 17:10 | |
| zbr|rover | pabelanger: clarkb: thanks. I guess next one is the openstack one https://review.opendev.org/#/c/663599/ | 17:15 |
| *** jpena is now known as jpena|off | 17:17 | |
| *** bhavikdbavishi has quit IRC | 17:22 | |
| *** jamesmcarthur has quit IRC | 17:26 | |
| *** jamesmcarthur has joined #zuul | 17:26 | |
| *** electrofelix has quit IRC | 17:27 | |
| tobiash | corvus: pagure driver lgtm, do you want mor eyes on it? | 17:57 |
| corvus | tobiash: mordred reviewed an earlier version; my guess is he'd be happy with the current PS but he can speak up now if he wants to review again... let's ask SpamapS if he's interested since he's been diving in on bitbucket.... otherwise, unless someone jumps up and says "me!" i think we're good. | 18:00 |
| pabelanger | corvus: retrigger worked this time: http://paste.openstack.org/show/752771/ | 18:01 |
| pabelanger | corvus: but did notice a 20min delay on processing it | 18:01 |
| pabelanger | sorry, 10min | 18:01 |
| corvus | pabelanger: so it was more likely a transient issue on the github side before | 18:02 |
| pabelanger | agree | 18:02 |
| clarkb | pabelanger: re the delay check your github event queue processing lengths | 18:03 |
| clarkb | pabelanger: ansible/ansible in particular is quite susceptible to those delays due to long queues due to its post merge testing | 18:03 |
| pabelanger | clarkb: okay, let me check logs | 18:03 |
| corvus | that's about to get better -- https://review.opendev.org/662818 | 18:04 |
| clarkb | ya I need to rereview that apparently | 18:04 |
| pabelanger | oh, neat | 18:04 |
| tobiash | yeah, needed a rebase because of log stack | 18:05 |
| pabelanger | jlk: do you happen to know when the next release of github3.py will be happening? | 18:05 |
| *** armstrongs has quit IRC | 18:10 | |
| pabelanger | corvus: re: pastebin for default_read_timeout, docs seem to say the following: https://github3.readthedocs.io/en/master/api-reference/github.html#githubsession-object | 18:13 |
| clarkb | I've approved the github multiprocessing change | 18:13 |
| clarkb | pabelanger: ^ so you should be able to update to that and see if it is happier soon | 18:13 |
| pabelanger | clarkb: great, thanks | 18:14 |
| fungi | at least it looks like the session timeout parameters are configurable | 18:14 |
| corvus | pabelanger: er yes, do that. :) | 18:16 |
| corvus | pabelanger: you can skip the connect timeout if it's not a problem. | 18:16 |
| pabelanger | +1 | 18:16 |
| *** [GNU] has joined #zuul | 18:17 | |
| pabelanger | corvus: should we expose that value as a config setting? Or 300 for all? | 18:18 |
| corvus | pabelanger: i think you should pick a value and see if it makes things better, and if it does, we'll hardcode it | 18:19 |
| pabelanger | understood | 18:20 |
| corvus | no one should have to set that on their own | 18:20 |
| tobiash | fyi, I'll roll out the mitogen change to production tomorrow morning and try it with some jobs | 18:21 |
| * tobiash is looking forward to see if it improves things | 18:22 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add caching of autohold requests https://review.opendev.org/663412 | 18:23 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Add autohold-info CLI command https://review.opendev.org/662487 | 18:23 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Record held node IDs with autohold request https://review.opendev.org/662498 | 18:23 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 18:23 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 18:23 |
| jlk | it's not a scheduled event, one of us will decide that enough has changed to need a release, or somebody asks really nicely. | 18:27 |
| jlk | pabelanger: ^^ | 18:28 |
| tobiash | jlk: I guess pabelanger chooses to ask really nicely ;) | 18:28 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 18:30 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 18:30 |
| fungi | recent problem with the zuul-stream-functional-2.5 job? | 18:30 |
| fungi | E: Failed to fetch http://mirror.dfw.rax.opendev.org/ubuntu/dists/xenial-updates/main/binary-amd64/Packages.gz Hash Sum mismatch | 18:31 |
| fungi | ahh, must have been a problem with opendev's mirrors | 18:31 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 18:36 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 18:36 |
| *** gtema has quit IRC | 18:38 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add spec for enhanced regional executor distribution https://review.opendev.org/663413 | 18:38 |
| pabelanger | jlk: tobiash: a release would be nice, if able :) | 18:38 |
| *** pcaruana has quit IRC | 18:39 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Auto-delete expired autohold requests https://review.opendev.org/663762 | 18:39 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: Mark nodes as USED when deleting autohold https://review.opendev.org/664060 | 18:39 |
| Shrews | Frankie says "Relax. This time for sure" | 18:40 |
| *** jamesmcarthur has quit IRC | 18:40 | |
| jlk | pabelanger: it's been a while since I've done a release. I'll try to remember how to do it. | 18:40 |
| pabelanger | tyty | 18:44 |
| jlk | pabelanger: this is to fix the issue w/ pull reviews? | 18:44 |
| pabelanger | jlk: yah | 18:45 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul master: Fix minor typos in Nodepool OpenStack instructions https://review.opendev.org/664665 | 18:49 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 18:52 |
| openstackgerrit | Paul Belanger proposed zuul/zuul master: WIP: Bump default_read_timeout for github driver to 300 https://review.opendev.org/664667 | 18:52 |
| *** jamesmcarthur has joined #zuul | 18:59 | |
| *** jamesmcarthur has quit IRC | 19:01 | |
| *** jamesmcarthur has joined #zuul | 19:03 | |
| *** jamesmcarthur has quit IRC | 19:05 | |
| *** jamesmcarthur has joined #zuul | 19:07 | |
| openstackgerrit | Jean-Philippe Evrard proposed zuul/zuul master: Expose date time as facts https://review.opendev.org/664674 | 19:09 |
| openstackgerrit | Merged zuul/zuul master: Parallelize github event processing https://review.opendev.org/662818 | 19:14 |
| *** jamesmcarthur has quit IRC | 19:15 | |
| *** hashar has joined #zuul | 19:23 | |
| *** igordc has quit IRC | 19:26 | |
| *** jamesmcarthur has joined #zuul | 19:34 | |
| *** armstrongs has joined #zuul | 19:44 | |
| *** armstrongs has quit IRC | 19:58 | |
| *** jamesmcarthur has quit IRC | 20:03 | |
| fungi | the most recent recheck of 662870 hit a TimeoutException in TestAnsible28.test_plugins at 260s in the py35 job... the change sets wait_timeout=270 though so not sure why it bailed 10s early. regardless, should i be upping it even further? | 20:06 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul master: Limit jobs runs with standard set of irrelevant files https://review.opendev.org/664680 | 20:07 |
| openstackgerrit | Evgeniy L proposed zuul/nodepool master: Add `floating-ip-pool` option to specify custom pool name https://review.opendev.org/664681 | 20:11 |
| openstackgerrit | Sean McGinnis proposed zuul/zuul master: Limit jobs runs with standard set of irrelevant files https://review.opendev.org/664680 | 20:15 |
| *** jamesmcarthur has joined #zuul | 20:25 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: download-artifacts: correct build selection https://review.opendev.org/664685 | 20:28 |
| corvus | clarkb, fungi: ^ sorry about that | 20:28 |
| *** jamesmcarthur has quit IRC | 20:28 | |
| fungi | oops, i didn't even consider stack vs queue | 20:30 |
| *** jamesmcarthur has joined #zuul | 20:53 | |
| *** jamesmcarthur has quit IRC | 21:04 | |
| openstackgerrit | Merged zuul/zuul-jobs master: download-artifacts: correct build selection https://review.opendev.org/664685 | 21:12 |
| clarkb | corvus: sorry just got back from lunch | 21:14 |
| fungi | i ended up taking a ~30-minute break to solder longer leads onto a replacement cpu fan and then tear down my workstation and replace its fan (for some reason the replacement fans i found come with leads just a few mm too short and rerouting them causes the case to press them into the fan housing distorting it until the fins rub against the inside, slowing it down measurably and making my cpu not cool off so | 21:21 |
| fungi | well) | 21:21 |
| fungi | here's to less thermal throttling and not having to press f1 to ignore fan errors at boot | 21:21 |
| *** igordc has joined #zuul | 21:31 | |
| *** hashar has quit IRC | 21:37 | |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Enhance upstream installation mode https://review.opendev.org/664711 | 22:06 |
| openstackgerrit | Kevin Carter (cloudnull) proposed zuul/zuul-jobs master: Isolate installation mode https://review.opendev.org/664711 | 22:29 |
| SpamapS | $ git describe | 22:55 |
| SpamapS | 3.8.1-138-g1fab39cc | 22:55 |
| SpamapS | Anything I can do to help get a 3.9 out? (I assume we'd go 3.9 since there are at least 2 new minor features) | 22:55 |
| SpamapS | I'm about to deploy 3.8.1-138-xxxx but was thinking it would be nice if this was 3.9.0 instead. ;) | 22:56 |
| clarkb | we did just redeploy opendev's zuul and haven't had problems with it though we know there is a memory leak hiding somewhere | 22:57 |
| clarkb | that restart did not include the github multi processing though | 22:57 |
| pabelanger | multi processing would be nice | 22:58 |
| SpamapS | If there's a leak, mine doesn't seem to be showing it. Mine is also super tiny. | 22:58 |
| clarkb | SpamapS: it shows up "randomly" after a few weeks of running. So opendev's last restart included the repl change to aid in debugging | 22:58 |
| SpamapS | I'm at 40 days with my 3.8.0 install | 22:59 |
| pabelanger | I've done some basic testing of ansible 2.8, and it seems to work | 22:59 |
| SpamapS | http://paste.openstack.org/show/752782/ | 23:00 |
| pabelanger | we just did a reset to apply https://review.opendev.org/664667/ | 23:01 |
| pabelanger | hoping that 'fixes' github | 23:01 |
| clarkb | SpamapS: http://cacti.openstack.org/cacti/graph.php?action=zoom&local_graph_id=64792&rra_id=3&view_type=&graph_start=1557615695&graph_end=1560294095 you can see it hit us there | 23:02 |
| *** rlandy|ruck has quit IRC | 23:14 | |
| SpamapS | yeah I suspect my activity is too low to notice it | 23:26 |
| SpamapS | BTW, I just experienced something unexpected. Two PR's, one to a branch (prod), one to master. They both got approved in a short time, and the second one is showing as dependent on the first in the Zuul UI. | 23:27 |
| SpamapS | I'd have thought different branches would mean different queues. | 23:27 |
| fungi | branches all share the queue because you might do upgrade testing or similar branch-dependent things | 23:28 |
| fungi | so a change to one branch could influence the result of jobs run for a change on another branch | 23:28 |
| clarkb | and it isn't a strict dependency. If the parent fails jobs reschedule for the child without the parent | 23:29 |
| fungi | i think we had at one point talked about some way to make that configurable | 23:29 |
| clarkb | which is different than proper git or depends-on deps | 23:29 |
| fungi | (configurable branch-independent queuing i mean) | 23:29 |
| fungi | but yeah, they're not "dependent" they're merely "sequenced" | 23:30 |
| fungi | in the same way that multiple projects sharing the same change queue would be | 23:30 |
| SpamapS | clarkb:ah, I wonder if we could make that show up as a dotted line in the UI or something. | 23:31 |
| fungi | well, you might wind up with changes a,b,c in a queue where c depends on a but b got approved between them | 23:32 |
| fungi | so using different line styles wouldn't cut it | 23:32 |
| fungi | the graph of dependencies and the graph of sequencing would be hard to effectively overlay | 23:34 |
| fungi | though that might be a really useful alternative status view... ability to switch between queue and deptree views | 23:35 |
| *** ianychoi has quit IRC | 23:48 | |
| *** ianychoi has joined #zuul | 23:48 | |
| SpamapS | to quote the great Avril Levine: Why'd you have to go and make things so complicated? | 23:48 |
| *** jamesmcarthur has joined #zuul | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!