| *** jesusaur has quit IRC | 00:04 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove implied playbook extensions https://review.openstack.org/525239 | 01:02 |
|---|---|---|
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove file extension when building SimpleLayout https://review.openstack.org/525356 | 01:02 |
| *** jesusaur has joined #zuul | 01:23 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove implied playbook extensions https://review.openstack.org/525239 | 01:24 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Update playbook paths with extension https://review.openstack.org/525359 | 01:24 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove file extension when building SimpleLayout https://review.openstack.org/525356 | 02:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Update playbook paths with extension https://review.openstack.org/525359 | 02:03 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove implied playbook extensions https://review.openstack.org/525239 | 02:03 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Move send_command() into ZuulApp https://review.openstack.org/523211 | 02:05 |
| *** jesusaur has quit IRC | 02:30 | |
| *** jesusaur has joined #zuul | 02:32 | |
| dmsimard | pabelanger: you used a set of playbooks for standing up zuul/nodepool in Sydney right ? was that windmill ? | 03:03 |
| pabelanger | dmsimard: yes, it will use ansible-role-nodepool in git.o.o | 03:08 |
| pabelanger | still working on coverting all the jobs to native zuulv3 | 03:08 |
| dmsimard | ok, I might re-use that to test a nested zuul+nodepool for ara integration jobs | 03:08 |
| dmsimard | unless it's a horrible idea but I thought might as well test like.. everything | 03:09 |
| pabelanger | sure, should work. I still don't have devstack integration just yet, but plan on spending some time on it in the next week for so | 03:10 |
| dmsimard | I don't want to refactor ARA's unit/functional/integration tests ever again, this is mind numbing | 03:17 |
| dmsimard | definitely one of the worst part of making maintainable software, the tests :( | 03:19 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove file extension when building SimpleLayout https://review.openstack.org/525356 | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Update playbook paths with extension https://review.openstack.org/525359 | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove implied playbook extensions https://review.openstack.org/525239 | 03:20 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove file extension when building SimpleLayout https://review.openstack.org/525356 | 03:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Update playbook paths with extension https://review.openstack.org/525359 | 03:26 |
| openstackgerrit | Ian Wienand proposed openstack-infra/zuul feature/zuulv3: Remove implied playbook extensions https://review.openstack.org/525239 | 03:26 |
| *** threestrands has joined #zuul | 03:27 | |
| *** bhavik1 has joined #zuul | 04:25 | |
| *** threestrands has quit IRC | 05:25 | |
| *** jaianshu has joined #zuul | 07:06 | |
| *** hashar has joined #zuul | 08:21 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: revoke-sudo: only revoke when zuul is sudoer https://review.openstack.org/522261 | 08:30 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: revoke-sudo: only revoke when zuul is sudoer https://review.openstack.org/522261 | 08:52 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: bindep: fix install bindep_command fact name https://review.openstack.org/525503 | 08:52 |
| *** jpena|off is now known as jpena | 08:55 | |
| *** sshnaidm|off is now known as sshnaidm|rover | 08:58 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: bindep: skip package install when not needed https://review.openstack.org/525506 | 09:09 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: tox: make install_sibling_packages pip import optional https://review.openstack.org/525507 | 09:13 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: fetch-tox-output: remove --ignore-missing-args https://review.openstack.org/525510 | 09:20 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: DNM: negative test https://review.openstack.org/522438 | 09:22 |
| *** sshnaidm|rover is now known as sshnaidm|rover|a | 10:18 | |
| *** sshnaidm|rover|a is now known as sshnaidm|afk | 10:18 | |
| *** bhavik1 has quit IRC | 10:26 | |
| *** sshnaidm|afk is now known as sshnaidm|rover | 10:43 | |
| *** jkilpatr has quit IRC | 11:36 | |
| jaianshu | Hi...can anyone look at my config file of nodepool once, i'm getting " return self.config.provider_managers[provider_name] | 11:41 |
| jaianshu | TypeError: unhashable type: 'Provider' | 11:41 |
| jaianshu | " error for `nodepool alien-list`. | 11:41 |
| jaianshu | files - nodepool.yaml-> http://paste.openstack.org/show/628150/ , clouds.yaml -> http://paste.openstack.org/show/628151/ | 11:41 |
| *** jkilpatr has joined #zuul | 11:55 | |
| tobiash | jaianshu: what are you expecting from alien-list? | 12:16 |
| tobiash | jaianshu: the targets section looks bogus to me | 12:18 |
| tobiash | jaianshu: alien-list is about to be removed: https://review.openstack.org/#/c/525177/ | 12:19 |
| jaianshu | tobiash: i have a node provisioned in my provider..i can get the info from OS_CLOUD=example openstack server list but not from nodepool command. What would be the alternative to get node info then if alien command is going to be removed? | 12:23 |
| tobiash | jaianshu: for listing nodepool managed nodes you should use 'nodepool list' | 12:24 |
| tobiash | alien list is something different | 12:24 |
| jaianshu | tobiash: ok..i have removed the targets section..still its not able to get nodes info. | 12:34 |
| jaianshu | anything else i'm missing out here | 12:34 |
| Shrews | jaianshu: there's a bug in alien-list keeping it from working, but we are removing it entirely soon | 12:35 |
| jaianshu | ahh..ok | 12:38 |
| *** jpena is now known as jpena|lunch | 12:48 | |
| rcarrillocruz | Shrews: you run your nodepool dev in Docker don't you? do you run in xenial or centos? | 13:04 |
| Shrews | rcarrillocruz: i do not run a nodepool. i only use docker for running zookeeper for unit tests | 13:06 |
| rcarrillocruz | ah ok | 13:06 |
| *** jaianshu has quit IRC | 13:30 | |
| *** jkilpatr has quit IRC | 13:47 | |
| *** jkilpatr has joined #zuul | 13:48 | |
| *** dkranz has joined #zuul | 13:50 | |
| *** jpena|lunch is now known as jpena | 13:57 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: WIP: Git driver https://review.openstack.org/525614 | 14:00 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix variable name in npm-publish job https://review.openstack.org/525326 | 14:05 |
| fbo_ | jeblair: pabelanger hi, I proposed a wip for the git driver. When you have time, could you let me know if I'm on the right track :) | 14:20 |
| _ari__ | msg NickServ identify r3dh@t | 14:26 |
| pabelanger | fbo_: great! | 14:29 |
| Shrews | _ari__: oopsie :) | 14:44 |
| Shrews | tristanC: about https://review.openstack.org/523640 ... maybe that would be a good thing to make configurable via the [zookeeper] config section? i'm not entirely sure one behavior has a clear advantage over the other, so making it configurable would allow both? | 15:26 |
| tristanC | Shrews: sounds good to me, i'll add the option tomorrow | 15:27 |
| Shrews | tristanC: i'd like to hear what jeblair thinks about it, too, since we'd probably do the same with zuul | 15:27 |
| jeblair | Shrews, tristanC: hrm -- if we can't tell which one is better, how will our users know which option to set? | 15:28 |
| Shrews | jeblair: that's why i'm having trouble just accepting it, i guess | 15:28 |
| tristanC | jeblair: ideally, nodepool would wait/retry zk for at least a long time | 15:29 |
| jeblair | we've seen a lot of zk errors, and i don't remember it just getting stuck.... | 15:29 |
| tristanC | jeblair: i put a short timeout because it was enough for the connectionlost issue and because it doesn't change how nodepool currently works (it raises exception if zk isn't available) | 15:30 |
| tristanC | jeblair: the issue with having no retry limit, is that kazoo doesn't seems to log what's going on, and if zk is not started, then nodepool may looks stall while it is in fact waiting for zk | 15:31 |
| Shrews | well, it does change how it works, doesn't it? if a command would have waitied until ZK returns (may be slower, but would eventually work), now it fails quickly | 15:32 |
| Shrews | that's where i'm having my doubts | 15:32 |
| jeblair | i don't know which thing we should do yet, i'll probably need to go read some more code before i have an opinion on that. but i'm pretty sure i don't want to add an option -- i think it's too much to ask users to know which thing should work better, and it sounds like it would fundamentally change how nodepool works in poor conditions. that will make it really hard for us to think about problems when debugging if we have to think about two ... | 15:32 |
| jeblair | ... completely different failure paths. | 15:32 |
| Shrews | more heads thinking about this is very good, IMO :) | 15:34 |
| Shrews | i suggested the config option because, as jeblair says, it's been working very well for us as-is and I didn't want to change that out from underneath us. But yeah, I can see having that as an option to be very confusing | 15:37 |
| jeblair | one thought i have: why is retrying forever bad? certainly in some programs you don't want to do that, but nodepool can't do anything at all if it can't contact zookeeper, so is it bad for it to essentially pause until the connection comes back? | 15:39 |
| tristanC | jeblair: Shrews: i don't mind eitherway, as long as we can silence those connectionlost exception that seems to happen randomly | 15:39 |
| Shrews | tristanC: oh, this is just to clean up logging? i thought there was a concrete example you had on your system | 15:40 |
| jeblair | tristanC: if connection lost exceptions are happening, that's pretty bad -- we should fix those | 15:40 |
| tristanC | jeblair: perhaps there is a way to make kazoo log that it's stuck retrying | 15:40 |
| jeblair | whenever a connection is lost, we lose ephemeral nodes, and locks, etc.... | 15:40 |
| jeblair | (if zuul loses its connection to zk, *all* of the jobs running abort) | 15:40 |
| tristanC | Shrews: well clean-up the logging, and I guess improve service quality, I guess node request fail when storenode fails | 15:40 |
| jeblair | we talked about making a mini-howto for tuning zookeeper, maybe we should now... | 15:41 |
| Shrews | we can't fix connection lost exceptions, just make sure we are handling them correctly. :) but if we're overlogging, would be nice to fix that | 15:41 |
| jeblair | Shrews: you and I may not be able to, but tristanC may be able to :) | 15:42 |
| tristanC | heh, for what it worth, we cherry-pick that 1.7 second retry fix, and nodepool isn't raising connectionlost anymore | 15:43 |
| tristanC | i got to go now, i can work on the review tomorrow | 15:44 |
| jeblair | tristanC: do you mean with that change it isn't losing the connection in the first place? | 15:44 |
| tristanC | jeblair: it's probably still loosing the connection, but at least it retry and proceed as if it didn't happen | 15:44 |
| Shrews | tristanC: kazoo.exceptions.ConnectionLoss can be silenced by changing the kazoo logging level. I don't think we can silence that from nodepool | 15:46 |
| tristanC | this is the kind of exception we were getting every few hours or so: http://paste.openstack.org/show/628181/ | 15:46 |
| *** openstackgerrit has quit IRC | 15:48 | |
| jeblair | tristanC: when you have time, let's focus on why you were getting those -- your users are going to be really unhappy if all their jobs reset every few hours | 15:48 |
| tristanC | jeblair: interestingly enough, these errors only happen with the nodepool services | 15:53 |
| tristanC | jeblair: i'll investigate if this is related to how instances are deployed on this cloud, perhaps the qrouter path is different | 15:53 |
| jeblair | tristanC: yeah, that's one thing to look at. also, whether the host(s) is/are cpu bound (that can cause starvation and timeouts) | 15:54 |
| pabelanger | mordred: do you have some time this week to comment on https://review.openstack.org/521324/ ? Would like to see if we could continue forward on it, to help optimize CI resources. I have some jobs I'd like to update for native v3 and this is a good optimaztion to have if possible | 16:11 |
| mordred | pabelanger: sorry - I keep opening it and then not actually reviewing it - so yes | 16:11 |
| pabelanger | mordred: np, and thanks | 16:13 |
| *** openstackgerrit has joined #zuul | 16:53 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add use_python3 variable to sphinx role and job https://review.openstack.org/525688 | 16:53 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: DNM testing new build-sphinx job works for python3 projects https://review.openstack.org/525690 | 16:55 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add use_python3 variable to sphinx role and job https://review.openstack.org/525688 | 16:57 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Add use_python3 variable to sphinx role and job https://review.openstack.org/525688 | 17:00 |
| mordred | pabelanger: +2 from me on https://review.openstack.org/#/c/521324 - leaving un +A'd so jeblair can re-review | 17:04 |
| pabelanger | mordred: tyty | 17:06 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Add use_python3 variable to sphinx role and job https://review.openstack.org/525688 | 17:35 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update sphinx jobs to use python3 https://review.openstack.org/525690 | 17:37 |
| *** sshnaidm|rover is now known as sshnaidm|off | 17:47 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Update sphinx jobs to use python3 https://review.openstack.org/525690 | 17:54 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use python3 for docs publication https://review.openstack.org/525718 | 17:55 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul feature/zuulv3: WIP: Git driver https://review.openstack.org/525614 | 18:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Add finger gateway https://review.openstack.org/525276 | 18:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Update sphinx jobs to use python3 https://review.openstack.org/525690 | 18:19 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul feature/zuulv3: Use python3 for docs publication https://review.openstack.org/525718 | 18:21 |
| *** jpena is now known as jpena|off | 18:35 | |
| *** JasonCL has joined #zuul | 19:16 | |
| *** electrofelix has quit IRC | 19:27 | |
| Shrews | anyone want to +2A https://review.openstack.org/525177 to finally get rid of alien-list? | 20:00 |
| *** JasonCL has quit IRC | 20:29 | |
| *** JasonCL has joined #zuul | 20:39 | |
| mordred | Shrews: done | 20:45 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove alien_list command https://review.openstack.org/525177 | 20:54 |
| Shrews | w00t | 20:55 |
| *** JasonCL has quit IRC | 21:03 | |
| tobiash | is the gearman client certificate needed on zuul side? | 21:07 |
| tobiash | ah, that's for client authentication | 21:09 |
| *** JasonCL has joined #zuul | 21:11 | |
| *** jkilpatr has quit IRC | 21:35 | |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add sphinx_python variable to sphinx role and job https://review.openstack.org/525763 | 21:39 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Add sphinx_python variable to sphinx role and job https://review.openstack.org/525763 | 21:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Correct exception for missing job dependencies https://review.openstack.org/525773 | 22:01 |
| *** threestrands has joined #zuul | 22:05 | |
| *** threestrands has quit IRC | 22:05 | |
| *** threestrands has joined #zuul | 22:05 | |
| dmsimard | jeblair: btw still seeing an END OF STREAM (although I guess for a shorter duration now?) with that job starting message | 22:07 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul feature/zuulv3: Correct exception for missing job dependencies https://review.openstack.org/525773 | 22:09 |
| jeblair | dmsimard: i haven't checked to see if it's running on ze* yet; did it land before friday? | 22:09 |
| dmsimard | Yeah it does | 22:11 |
| dmsimard | Well, er, yes it landed | 22:12 |
| dmsimard | And it's printed in the consoles | 22:12 |
| *** JasonCL has quit IRC | 22:23 | |
| jesusaur | wanna see something interesting? here's a change that got a full +2 +2 +1 but didn't merge: https://review.openstack.org/#/c/517658/ | 22:25 |
| jesusaur | are there any known issues around changes not merging after passing the gate jobs? | 22:27 |
| dmsimard | jesusaur: when that happens it's usually because either the parent patch has not merged or a Depends-On patch hasn't merged. I don't think that's the case here, looks like a legit issue. | 22:28 |
| dmsimard | jesusaur: I think an obvious attempt at working around the issue would be to rebase the patch and +W again | 22:31 |
| dmsimard | until someone gets a chance to look at logs or something like that. | 22:31 |
| dmsimard | the patch mentions the parent is out of date so it can be rebased, maybe it failed a merge silently ? | 22:32 |
| dmsimard | ¯\_(ツ)_/¯ | 22:32 |
| jesusaur | well, zuul was able to rebase otherwise it wouldn't start the gate jobs, right? | 22:34 |
| jesusaur | but maybe it's that jgit vs cgit merge issue? I'll try rebasing | 22:34 |
| dmsimard | jesusaur: It's probably not that, hard to tell without looking at logs :( | 22:35 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Correct exception for missing job dependencies https://review.openstack.org/525773 | 22:40 |
| *** JasonCL has joined #zuul | 22:45 | |
| *** JasonCL has quit IRC | 22:51 | |
| SpamapS | do we have examples of the best "zuulv3 way" for jobs to have one job build an artifact, and many jobs consume that artifact? | 22:56 |
| SpamapS | Like I want to build a docker image, and have it used by 3 integration test jobs. | 22:56 |
| SpamapS | oh! zuul_return will dump them in, neat. | 22:57 |
| dmsimard | SpamapS: there's been different discussions about that, there is nothing yet built in afaik | 23:10 |
| dmsimard | In v2 for RDO we have a parent job that uploads it's log at a predictable location and then the child jobs are able to reconstruct that location to download the artifacts | 23:11 |
| dmsimard | Would need to look at chat logs but jeblair and mordred definitely have something in the backlog | 23:12 |
| dmsimard | SpamapS: those are other ideas we had discussed back in Denver in the particular use case of container images https://etherpad.openstack.org/p/rdo-registry-ideas | 23:13 |
| *** hashar has quit IRC | 23:15 | |
| *** JasonCL has joined #zuul | 23:33 | |
| mordred | dmsimard: what did I do? | 23:35 |
| mordred | SpamapS, dmsimard: ah - yah, that old chestnut ... | 23:36 |
| *** JasonCL has quit IRC | 23:39 | |
| SpamapS | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/jobs.html#return-values | 23:41 |
| SpamapS | Suggests that I should get anything that isn't under zuul: will get set as a variable in child jobs | 23:42 |
| * SpamapS has broken brain | 23:42 | |
| *** JasonCL has joined #zuul | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!