| jeblair | i just found ze02 stuck on a git fetch | 00:39 |
|---|---|---|
| jeblair | i think that meant that all of the jobs it claimed were stuck waiting on that to complete | 00:39 |
| jeblair | (since they use the internal merger to set up their repos, and that is centralized and locked) | 00:40 |
| jeblair | so i think we may need to have the merger timeout on git fetch operations to protect us from that | 00:40 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add netaddr requirements for running ipv4|ipv6 filters https://review.openstack.org/504522 | 00:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /tenants route https://review.openstack.org/503268 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/status route https://review.openstack.org/503269 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs route https://review.openstack.org/503270 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/builds route https://review.openstack.org/466561 | 01:01 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: make console-stream tenant scoped https://review.openstack.org/505452 | 01:02 |
| jamielennox | SpamapS, jlk: the problem with kube is that you are expecting to get something you can kubectl apply to in ci | 01:15 |
| jamielennox | i think it's ok to say that each job should get a namespace that then gets torn down at the end | 01:16 |
| jamielennox | and then you should be able to RBAC that the user only has priv within the namespace | 01:16 |
| jamielennox | and anyone testing cross-namespace stuff has to figure out there own things | 01:16 |
| jamielennox | in theory i think that's reasonable, in practice it changes nodepool's role which may or may not be a problem | 01:17 |
| jamielennox | there are questions then on what happens on executor vs on the node, and i have no idea because how exactly do you run functional tests against a deployed kube app? | 01:21 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up hosts file https://review.openstack.org/504552 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 01:49 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Pin ansible<2.4 in test-requirements for ansible-lint https://review.openstack.org/505467 | 02:12 |
| mordred | jamielennox, SpamapS, jlk: I think a job that wants to do kubectl apply things is a great thing we should have - but is also a whole new can of worms with tons of things to figure out. there's also the possibilty of just using k8s behind the scenes like we do with openstack today to get a container $somewhere that is ssh-able086146 | 02:28 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Pin ansible<2.4 in test-requirements for ansible-lint https://review.openstack.org/505467 | 02:31 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Support zuul.d directories https://review.openstack.org/504797 | 03:14 |
| dmsimard | jlk: made some minimal 2.4 progress on ara, two hurdles down. Giving up on the third one for now, I'll revisit later. | 03:31 |
| dmsimard | updated the pad. | 03:31 |
| *** yolanda has quit IRC | 03:34 | |
| *** yolanda has joined #zuul | 03:34 | |
| jamielennox | the question is going to be then what from ansible makes sense. if i've done kubectl i don't expect to be able to ansible into those container | 03:42 |
| jamielennox | however either way i think you probably need a runner there somewhere that is not a kube container | 03:43 |
| jamielennox | like build containers, kubectl apply, and then somewhere that a functional test suite can run from | 03:44 |
| SpamapS | mordred: you don't really SSH to containers. | 04:49 |
| SpamapS | that implies systemd.. other things | 04:49 |
| SpamapS | jamielennox: a namespace per build is not a bad idea at all. But my point about k8s is more that users don't necessarily want to "test things on OpenStack", they want to "test things on reasonable facsimilies of servers" so they use OpenStack to do that. | 04:51 |
| SpamapS | Likewise, I think people don't really want to test how their software interacts with k8s. They just want to run something. | 04:52 |
| jamielennox | SpamapS: i can fairly easily come up with a functional testing scenario for testing k8s like that and afaik there is nothings else that can do that | 05:00 |
| jamielennox | like tempest tests are in their own repo of code that you run after devstack-gate | 05:00 |
| jamielennox | so the test path is tempest goes onto the node | 05:00 |
| jamielennox | kubectl is invoked either from executor or node | 05:01 |
| jamielennox | then the test process is to run tempest against the thing that was deployed | 05:01 |
| SpamapS | I'm not sure what you're saying. | 05:02 |
| SpamapS | What I'm suggesting is that it's most important that people be able to run jobs inside containers. It's not very important that they get API access to k8s. If they want to test against versions of k8s, I suggest they use VMs, and deploy k8s to the specification they need it to work with. | 05:03 |
| jamielennox | i'm saying that i don't think k8s is like openstack in that sense, for that target i really want to test that my kube scripts are correct | 05:04 |
| SpamapS | You won't have any control over what version of k8s is deployed. | 05:04 |
| SpamapS | Or how it is configured. | 05:04 |
| SpamapS | It's all int he Zuul operator's hands. | 05:04 |
| jamielennox | there is definitely an "i want to run tox and i don't really need a vm for that" but i don't think that's a k8s use case | 05:04 |
| SpamapS | Right, I was saying earlier, that's a docker swarm use case. | 05:05 |
| jamielennox | agreed | 05:05 |
| SpamapS | But, I think it's more likely you'll find k8s deployed than swarm. | 05:05 |
| SpamapS | and k8s is perfectly capable of doing that. | 05:05 |
| SpamapS | and you can do things like multi-container functional testing | 05:05 |
| jamielennox | versions of k8s are the same as versions of ubuntu, the deployer is going to have to set that up for you | 05:05 |
| SpamapS | so you can have a job writer that drops a mysql service, a rabbitmq service, and then runs neutron's functional tests as a 3rd service utilizing those. | 05:06 |
| SpamapS | (well maybe not neutron... grr ovs) | 05:07 |
| SpamapS | but there's really two separate things | 05:07 |
| SpamapS | 1) Lighter weight "dumb" jobs that don't need VMs. | 05:07 |
| SpamapS | 2) k8s/coe/etc. heavy jobs that don't want to deploy their own k8s/coe/etc. | 05:07 |
| SpamapS | I'm only interested in (1) | 05:08 |
| jamielennox | ah, ok | 05:08 |
| SpamapS | And I think using k8s for it means more adoption. | 05:08 |
| jamielennox | that's why we're missing each other, 1 i think is not that hard, 2 is interesting | 05:08 |
| SpamapS | Yeah, (1) is easy | 05:08 |
| SpamapS | want it yesterday ;) | 05:08 |
| SpamapS | writing a spec begging for it actually :) | 05:09 |
| SpamapS | (an internal spec) | 05:09 |
| jamielennox | right, ok. 2 is what i think brings more uniqueness to zuul | 05:09 |
| jamielennox | but yea, we're not there yet | 05:09 |
| SpamapS | zuul kinda doesn't need more uniqueness at this point :) | 05:10 |
| SpamapS | It will | 05:10 |
| SpamapS | some day soon | 05:10 |
| tobiash | I think in the long run both use cases should be supported | 05:10 |
| SpamapS | but today.. it needs to stabilize and release a 3.0 and then start seeing if we can get more flowers to bloom. | 05:11 |
| jamielennox | for 1, kube is useful (and we want), but it seems like swarm can just be replaced by something exposing a docker socket and nodepool | 05:11 |
| jamielennox | here's 10 static machines with a docker socket, nodepool loads up a min-ready number of containers via that | 05:12 |
| SpamapS | Yeah I kind of was suggesting that in the past. Just layer a docker driver over top of regular VM nodepool. | 05:13 |
| tobiash | that sounds cool | 05:14 |
| SpamapS | That would be a great way to get lightweight containery tests without having to fit it into k8s. | 05:14 |
| SpamapS | However, MANY potential users of Zuul will not have an OpenStack cloud. | 05:15 |
| qwc | True. | 05:20 |
| *** yolanda has quit IRC | 06:27 | |
| *** yolanda has joined #zuul | 06:29 | |
| *** openstackgerrit has quit IRC | 07:02 | |
| *** hashar has joined #zuul | 07:40 | |
| *** electrofelix has joined #zuul | 09:34 | |
| *** hashar is now known as hasharAway | 09:52 | |
| *** toabctl has quit IRC | 10:01 | |
| *** jkilpatr has quit IRC | 10:44 | |
| *** jkilpatr has joined #zuul | 11:16 | |
| *** toabctl has joined #zuul | 11:17 | |
| *** yolanda has quit IRC | 11:18 | |
| *** yolanda has joined #zuul | 11:24 | |
| *** hasharAway is now known as hashar | 11:39 | |
| *** dkranz has joined #zuul | 12:44 | |
| *** hashar is now known as hasharAway | 13:22 | |
| *** openstackgerrit has joined #zuul | 13:31 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Handle first play in playbook matching no nodes https://review.openstack.org/505626 | 13:31 |
| Shrews | mordred: seems 2.4 makes many things unhappy: http://logs.openstack.org/54/505354/1/check/tox-py35/8e3b3e1/testr_results.html.gz | 13:37 |
| Shrews | mordred: will begin digging into some of those when i return from my morning run | 13:37 |
| mordred | Shrews: yah - that patch above is from the first traceback I saw | 13:41 |
| mordred | Shrews: which I think has actually always been there - but 2.4 will actually show us callback plugin tracebacks now! :) | 13:41 |
| mordred | Shrews: anywho - if you didn't see I pushed up two patches on top of your 2.4 DNM patch | 13:42 |
| dmsimard | I think I managed to get ara passing integration and unit tests on 2.4 | 13:42 |
| mordred | dmsimard: woot! | 14:38 |
| Shrews | mordred: rebased your 2.4 things to see what effect they'll have on the tests | 15:07 |
| Shrews | we'll likely have to combine any fixes into one review at some point | 15:07 |
| mordred | Shrews: totally agree | 15:08 |
| mordred | Shrews: also - feel free to squash/modify/take-over anything I've got there | 15:09 |
| Shrews | sure | 15:09 |
| Shrews | hrm, well, they didn't eliminate any failures, but added on instead. progress! :) | 15:25 |
| Shrews | added one* | 15:25 |
| mordred | Shrews: woot! | 15:31 |
| mordred | dmsimard: don't know if you saw from earlier - but 2.4 added one very important thing ... | 15:31 |
| mordred | dmsimard: callback tracebacks are now actually emitted!!!! | 15:32 |
| dmsimard | mordred: wow | 15:32 |
| dmsimard | Best feature evet | 15:32 |
| dmsimard | ever* | 15:32 |
| mordred | right? | 15:33 |
| dmsimard | mordred: however the deprecation notices are real https://twitter.com/dmsimard/status/910337157495181312 | 15:34 |
| dmsimard | Everything, everywhere. Deprecated. | 15:35 |
| dmsimard | I actually had to redefine a function from Ansible to avoid printing 20 lines of deprecation when ara bootstrapped | 15:36 |
| Shrews | jeblair: where is the job.timeout value used in zuul? | 15:42 |
| Shrews | grepping for timeout is... not helpful | 15:42 |
| *** leifmadsen has quit IRC | 15:46 | |
| *** leifmadsen has joined #zuul | 15:47 | |
| jeblair | Shrews: executor/server.py | 15:49 |
| Shrews | jeblair: yeah, looked there first. didn't see any timeout references for a job. | 15:55 |
| * Shrews looks again | 15:56 | |
| Shrews | oh, it's hidden in args[] | 15:57 |
| jeblair | Shrews: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?id=feature/zuulv3#n1108 | 15:57 |
| jeblair | ya | 15:57 |
| Shrews | args = json.loads(self.job.arguments) | 15:57 |
| Shrews | yup | 15:57 |
| jeblair | Shrews: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/client.py?id=feature/zuulv3#n170 | 15:58 |
| jeblair | that's the transition from job.timeout to args (when we serialize to go over gearman) | 15:58 |
| Shrews | seems 2.4 is reporting SUCCESS for timeouts instead of TIMED_OUT. trying to track that down | 15:58 |
| Shrews | or perhaps the executor is now misinterpreting it | 15:59 |
| jeblair | Shrews: the mechanism of timeouts is the watchdog killing the ansible process | 15:59 |
| Shrews | hrm | 15:59 |
| jeblair | Shrews: if that happens, we shouldn't actually be getting anything from ansible -- we ask the watchdog if it timed out: https://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/executor/server.py?id=feature/zuulv3#n1599 | 16:00 |
| jeblair | Shrews: so it's pretty surprising that behavior would change -- unless ansible is actually finishing before the timeout for some reason? maybe what we're doing to slow things down doesn't work anymore? | 16:01 |
| Shrews | jeblair: shell: sleep(60) | 16:01 |
| Shrews | that should still work | 16:02 |
| jeblair | i agree | 16:02 |
| Shrews | oh, i might be chasing the wrong thing. I see this in the output: http://paste.openstack.org/show/621551/ | 16:07 |
| Shrews | i think i recall changes around hostvars in 2.4 | 16:08 |
| *** hasharAway has quit IRC | 16:57 | |
| *** hashar has joined #zuul | 16:57 | |
| mordred | dmsimard: AWESOME | 17:35 |
| mordred | Shrews: that output is what https://review.openstack.org/#/c/505626/2 is supposed to deal with | 17:35 |
| dmsimard | mordred: ? | 17:36 |
| Shrews | mordred: yeah, eventually put 2&2 together on that. still tracking this timeout thing | 17:39 |
| mordred | dmsimard: Everything, everywhere. Deprecated. | 17:39 |
| dmsimard | Oh, yes. Everything. | 17:40 |
| mordred | dmsimard: do you have any handy examples of stuff you had todo to avoid the deprecation? | 17:40 |
| *** electrofelix has quit IRC | 17:42 | |
| dmsimard | mordred: just one thing that I don't think zuul would use, the entire patch for 2.4 support in ara is https://review.openstack.org/#/c/505480/ | 17:43 |
| dmsimard | I basically redefined https://github.com/ansible/ansible/blob/devel/lib/ansible/constants.py#L18 to do nothing | 17:43 |
| dmsimard | It's temporary anyway, it's just to get something out and working with 2.4, I'll do things properly in 1.0. | 17:44 |
| dmsimard | there are deprecation notices all over the integration tests too but I haven't (and won't) address those just yet | 17:45 |
| dmsimard | A big reason of the deprecation spam is tied to includes which have been replaced by imports | 17:45 |
| dmsimard | like to include tasks from another file is now "import_tasks", there's "import_role" and "import_playbook" too. | 17:46 |
| jeblair | tristanC: i still haven't had a chance to review the web stack, but we will want tests and docs to go along with them; maybe that's something you can start working on? | 17:47 |
| mordred | dmsimard: ah. fun | 17:53 |
| Shrews | ooh, i think the timeout thing is our hosts not matching, thus it skips the playbook, and SUCCESS is the result | 18:08 |
| Shrews | neat | 18:08 |
| Shrews | DEBUG [build: 3bac474e4df340038de6564dbb3697f2] Ansible output: b'skipping: no hosts matched' | 18:10 |
| Shrews | there is also: | 18:10 |
| Shrews | DEBUG [build: a2cc1056fe9d4449a4022e209fc4cd43] Ansible output: b'[DEPRECATION WARNING]: [defaults]hostfile option, The key is misleading as it can also be a list of hosts, a directory or a list of paths . This feature will be removed in version 2.8. Deprecation warnings can be disabled by' | 18:10 |
| pabelanger | well, neat... | 18:11 |
| Shrews | but, that *should* still work, according to the warning text | 18:12 |
| Shrews | oh, this might be it: b' [WARNING]: Unable to parse /etc/ansible/hosts as an inventory source' | 18:13 |
| pabelanger | Shrews: is this local testing of 2.4? | 18:13 |
| Shrews | yeah | 18:13 |
| pabelanger | k | 18:13 |
| pabelanger | ya, we shouldn't have anything in /etc/ansible/hosts | 18:14 |
| pabelanger | we do setup hostfile IIRC | 18:14 |
| Shrews | right | 18:14 |
| Shrews | so, maybe their definition of "deprecation" is "we've already removed it! jokes on you!" | 18:15 |
| pabelanger | ha | 18:15 |
| pabelanger | Is ansible-playbook -i inventory still valid? | 18:15 |
| mordred | what's the new option supposed to be? | 18:16 |
| Shrews | yes. you can, in fact, use -i multiple times now | 18:16 |
| Shrews | there were LOTS of changes around inventory | 18:16 |
| Shrews | https://github.com/ansible/ansible/blob/devel/CHANGELOG.md | 18:16 |
| Shrews | beginning with "Inventory has been revamped:" | 18:16 |
| pabelanger | It is now possible to specify mulitple inventory sources in the command line (-i /etc/hosts1 -i /opt/hosts2) | 18:18 |
| pabelanger | that is fun | 18:18 |
| mordred | Shrews: does that warning go away if we change hostfile= to inventory= in the ansible.cfg we write? | 18:31 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up hosts file https://review.openstack.org/504552 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up firewalls https://review.openstack.org/504553 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 18:33 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 18:33 |
| Shrews | mordred: changing it to inventory causes all sorts of fail | 18:34 |
| Shrews | still poking | 18:34 |
| tristanC | jeblair: sure, I was waiting for some feedback regarding the zuul-web interfaces implementation before writting tests... | 18:40 |
| tristanC | Is there a reason why ZUUL_COMMIT is omited from http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/ansible/filter/zuul_filters.py?h=feature/zuulv3#n20 ? | 18:43 |
| jeblair | tristanC: we no longer have the information that was in ZUUL_COMMIT | 18:52 |
| tristanC | jeblair: hum, the use-case was for CD job in post pipeline, where we rely on the ZUUL_COMMIT to deploy merged changes in order | 18:56 |
| jeblair | tristanC: zuul.newrev should work for that purpose | 18:56 |
| tristanC | oh indeed, thanks! | 18:57 |
| jeblair | (aka ZUUL_NEWREV) | 18:57 |
| *** rbergeron has quit IRC | 19:20 | |
| *** rbergeron has joined #zuul | 19:20 | |
| Shrews | fyi, seems there may be a bug in ansible regarding 'all' in hosts file. working up a bug report now | 19:31 |
| Shrews | jeblair: pabelanger: ^^^ | 19:31 |
| Shrews | for 2.4, that is | 19:32 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Set up connectivity with default OVS bridge https://review.openstack.org/504554 | 19:34 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Multi-node: Streamline multi-node-known-hosts https://review.openstack.org/504629 | 19:34 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul-jobs master: Append /sbin and /usr/sbin to $PATH for 'ip' commands https://review.openstack.org/504743 | 19:34 |
| dmsimard | jeblair, mordred: now that the zuul-sphinx zuul.d fix is in, it's really itching me to separate things but I'll hold off on it in the current multinode stacks.. but I'll submit something to do it after everything lands | 19:39 |
| mordred | dmsimard: yah - I think as soon as we get o-z-j stack cleared we should do the split patch | 19:42 |
| mordred | dmsimard: and similarly for project-config once it's clear of v3 patches | 19:43 |
| dmsimard | right | 19:43 |
| dmsimard | should we agree on a convention for splitting ? are we splitting by components ? (nodes, nodesets, jobs, templates, projects, etc.) or by logical groups ? (base-integration, multinode-integration, etc.) | 19:43 |
| dmsimard | since order matters, it's probably better to split by logical chunks | 19:44 |
| mordred | pabelanger: -1 on https://review.openstack.org/#/c/504785 - but I thnk it's ready to land - I can also just update the patch if you'd prefer | 19:44 |
| mordred | dmsimard: I was doing components before - since order does not matter across them | 19:45 |
| mordred | dmsimard: logical groups seem like understanding ordering would be hard | 19:45 |
| dmsimard | mordred: ah, order matters only within the same set of components ? it doesn't matter when "merging" things ? | 19:46 |
| dmsimard | mordred: bah, I guess there's pros and cons, going for logical chunks also makes it semi-obvious where you need to go to check something | 19:46 |
| mordred | dmsimard: right - the most important thing is that order matters for jobs | 19:46 |
| dmsimard | OH, that reminds me, I have a special request | 19:46 |
| mordred | uhoh | 19:46 |
| dmsimard | you know how in JJB we had the print-template macro, that gave you the template the job was from | 19:47 |
| dmsimard | I really want something like that that identifies where things are coming from | 19:47 |
| pabelanger | mordred: sure, if you want too. Head deep into image failures | 19:47 |
| mordred | pabelanger: kk | 19:47 |
| pabelanger | mordred: but, job did build properly | 19:47 |
| dmsimard | mordred: like... node from openstack-zuul-jobs, job from zuul-jobs, pipeline from project-config | 19:47 |
| Shrews | for anyone interested in the 2.4 inventory bug: https://github.com/ansible/ansible/issues/30654 | 19:48 |
| dmsimard | mordred: otherwise it can get sorta confusing where you need to go look | 19:48 |
| pabelanger | mordred: last step is to have zuul return docs-draft.openstack.org/85/504785/1/check/openstack-doc-build/97151a2/ for openstack-doc-build, current it does logs.openstack.org/85/504785/1/check/openstack-doc-build/97151a2/ | 19:48 |
| pabelanger | mordred: couldn't figure out how to make that happen | 19:48 |
| mordred | dmsimard: ah, interesting. we have that somewhat for playbooks - but I can see where you're going with that - it might be a bit harder to produce ... | 19:49 |
| mordred | pabelanger: ah - nod - SO ... | 19:49 |
| mordred | pabelanger: I think what we want there is success-url, no? | 19:50 |
| mordred | pabelanger: anywho- I'll follow up on those real quick | 19:51 |
| mordred | pabelanger: you pay attention to image failures :) | 19:51 |
| mordred | Shrews: we should be able to work around that in the meantime by just generating an all group when we write out the inventory | 19:51 |
| pabelanger | mordred: right, I think so. But looking at the current code, we'd need to use log_url, which is setup in base playbook. I _think_ we need to maybe we work the setting of that value? | 19:51 |
| pabelanger | kk | 19:51 |
| dmsimard | mordred: surely zuul knows which file things are being loaded from ? | 19:52 |
| mordred | dmsimard: oh - it surely does - I just don't think it's available in a consumable form in a convenient place right now | 19:56 |
| * dmsimard nods | 19:56 | |
| jeblair | jobs have an attribute that elucidates their inheritance hierarchy; step 1 is probably changing that from strings to structured data, then exposing it as a zuul var | 19:57 |
| jeblair | .inheritance_path | 19:57 |
| Shrews | mordred: maybe? or we could flog bcoca to get a fix in. the latter seems more fun :) | 19:58 |
| mordred | jeblair: ++ | 19:58 |
| mordred | jeblair: I had a hunch you'd have a more complete answer than I | 19:58 |
| jeblair | dmsimard: i'm ready to be your root buddy; are you ready? | 19:58 |
| jeblair | mordred: what needs to happen before we can start throwing some auto-migrated jobs at v3? | 20:00 |
| jeblair | i reviewed a bunch of migartion changes earlier | 20:00 |
| dmsimard | jeblair: putting finishing touches on the multi node integration tests, soon :D | 20:01 |
| jeblair | hrm, i think zuul is stuck | 20:03 |
| dmsimard | did I kill it ? | 20:03 |
| mordred | jeblair: I'm making another pass through the open ones - and also following up on this jeepyb/git thing real quick | 20:03 |
| jeblair | somehow the item in post has an invalid configuration | 20:04 |
| jeblair | which is a neat trick. | 20:04 |
| dmsimard | jeblair: I'll go ahead and confirm that my new patchsets have not been enqueued indeed | 20:04 |
| mordred | jeblair: nice! did we manage to land something to project-config with a v3 config issue perhaps? | 20:05 |
| jeblair | mordred: nope, it's in zuul-jobs | 20:05 |
| jeblair | there are likely 2 issues | 20:05 |
| jeblair | why it (thinks?) it has an invalid config | 20:06 |
| jeblair | and why it isn't removed from the pipeline | 20:06 |
| jeblair | i'm going to be digging into this for a bit | 20:06 |
| mordred | jeblair: kk. by the time you're done with that I'll likely be ready to start throwing some auto-migrated jobs | 20:07 |
| Shrews | mordred: so, looks like the workaround is to use 'inventory =' instead of 'hostfile =', but that caused other issues that we'll have to begin tracking down | 20:15 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support Gerrit 2.13 ref-updated events https://review.openstack.org/505795 | 20:19 |
| dmsimard | afk for a bit. | 20:20 |
| jeblair | that's one issue ^ | 20:20 |
| *** pabelanger has quit IRC | 20:21 | |
| *** pabelanger has joined #zuul | 20:21 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: DNM: Test Ansible 2.4 https://review.openstack.org/505354 | 20:24 |
| mordred | jeblair: oh - right - we upgraded to 2.13 :) | 20:24 |
| Shrews | mordred: that ^^^ includes your https://review.openstack.org/505626 change now. feel free to abandon | 20:25 |
| mordred | Shrews: cool | 20:25 |
| Shrews | also changes hostfile to inventory so we can see the entirety of the fail | 20:26 |
| mordred | cool. I wonder if we'll need to do the whole plugin config thing we were poking at the other day with the new openstack inventory plugin | 20:26 |
| Shrews | mordred: refresh my memory on that? | 20:27 |
| mordred | Shrews: well, we did "inventory_enabled = openstack" in the ansible.cfg to enable the new openstack inventory plugin | 20:28 |
| mordred | Shrews: BUT - I would expect to not have to do that for inventories that are handled by the pre-plugin inventory code | 20:29 |
| Shrews | mordred: oh right. yeah, we shouldn't have to do that | 20:29 |
| mordred | Shrews: I see this in the yaml inventory plugin docs though: | 20:30 |
| mordred | - It takes the place of the previously hardcoded YAML inventory. | 20:30 |
| Shrews | well... mabye. the errors i saw were "cannot connect" or some such | 20:30 |
| Shrews | mordred: my test case in that PR works w/o using inventory_enabled | 20:30 |
| mordred | cool. | 20:31 |
| mordred | Shrews: oh - SO ... | 20:31 |
| mordred | Shrews: together with the "all" issue ... | 20:31 |
| mordred | Shrews: ifyou look at the description of the format for the new yaml plugin in lib/ansible/plugins/inventory/yaml.py | 20:32 |
| Shrews | this fixes the "all" issue when using "hostfile": https://github.com/ansible/ansible/pull/30651 | 20:32 |
| mordred | oh - nevermind ... it's the same format :) | 20:32 |
| mordred | I was about to say I was wondering if we were sending old format to new parser or something | 20:33 |
| mordred | Shrews: cool! | 20:33 |
| *** dkranz has quit IRC | 20:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix infinite loop on reconfiguration exception https://review.openstack.org/505807 | 20:51 |
| jeblair | okay, there's the other one ^ | 20:51 |
| jeblair | we should land that and https://review.openstack.org/505795 asap | 20:52 |
| * jlk reads, votes | 20:53 | |
| mordred | jeblair: jlk and I have +3d both (although in alternating order) | 20:55 |
| jlk | tag team! | 20:56 |
| jeblair | i'll restart when those land, then we'll probably need a bunch of rechecks | 20:57 |
| mordred | yah | 21:04 |
| *** jkilpatr has quit IRC | 21:05 | |
| mordred | dmsimard: in https://review.openstack.org/#/c/504787/5/tests/multi-node-known-hosts.yaml you say "Figure out why doing a plain lookup isn't working properly" | 21:17 |
| dmsimard | Yeah. | 21:17 |
| mordred | dmsimard: I believe lookup executes on the executor, not the remote node | 21:17 |
| dmsimard | Oh damn you're right | 21:17 |
| mordred | TODO -> done | 21:17 |
| dmsimard | And I just did a lookup in another place too | 21:18 |
| * dmsimard needs more sleep | 21:18 | |
| mordred | also, TIL that you can put crazy jinja into a set_fact :) | 21:18 |
| dmsimard | You can put crazy jinja in anything | 21:18 |
| dmsimard | Learned that from openstack-ansible, gotta give them credit | 21:18 |
| dmsimard | It allows for some crazy flexibility | 21:19 |
| mordred | jeblair: do you have a second to ponder an issue with me? | 21:25 |
| *** jkilpatr has joined #zuul | 21:25 | |
| jeblair | mordred: yes! what's that? :) | 21:49 |
| *** hashar has quit IRC | 21:50 | |
| mordred | WELL | 21:50 |
| jeblair | dmsimard: before i jump into nodeset stuff, do you want to do that thing on the executor? | 21:50 |
| mordred | jeblair: so (and I'm mostly summarizing a thing pabelanger discovered) | 21:50 |
| mordred | jeblair: if you look at http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul.yaml#n380 | 21:51 |
| mordred | jeblair: we need to add success-url: "http://docs-draft.openstack.org/{{ log_path }}" ... except chicken-and-egg - we don't have log_path so we can't set it there | 21:52 |
| mordred | jeblair: so I think we might need to be able to set success-url via zuul_return? (and by I think, I mean that's what pabelanger was saying and I wasn't properly hearing) | 21:53 |
| jeblair | we can't use zuul_return to set log_path in that job's post because base post will override it | 21:53 |
| mordred | right. we can't use it to set log_path | 21:53 |
| mordred | but we could use it (with code added) to set success-url | 21:54 |
| mordred | OR - we can do something completely different | 21:54 |
| jeblair | mordred: i have 2 alternate suggestions: | 21:54 |
| mordred | \o/ | 21:54 |
| jeblair | 1) instead of setting success-url with zuul_return, set an arbitrary var with zuul_return and make all zuul_return vars available in the format url method (this should be really easy) | 21:55 |
| jeblair | 2) have the base job only set log_url in zuul_return if it isn't already set | 21:55 |
| jeblair | the downside to 1 is that if something goes wrong, and $docs_draft_url (or whatever we call it) isn't actually set by zuul_return, the success-url will reference an undefined variable. and we, for good reason, don't do crazy jinja in zuul, so that will just have to fall back as if it weren't defined. that's probably fine. | 21:56 |
| jeblair | really, that's not much of a downside. :) | 21:56 |
| mordred | yah | 21:57 |
| mordred | 2 I don't think works - because we don't actually want to overide log_url ... oh- this is extra awkward | 21:57 |
| mordred | jeblair: there won't be anything reporting the location of the logs if we do success-url to https://docs-draft... | 21:58 |
| mordred | butI guess that's consistent with today, right? | 21:58 |
| jeblair | well, the nice thing about 2 is that we can have the docs-draft post playbook use whatever ansible/jinja logic we need to say "if this looks like it worked, set log_url here, otherwise, don't and let the base job do it) | 21:58 |
| jeblair | mordred: correct, that's today's behavior | 21:58 |
| mordred | gotcha. ok | 21:59 |
| jeblair | mordred: (success-url actually works a bit better if we use it the way we accidentally used it in v3 so far -- only reference files *under* the log publish location; i think we had a TODO to double check on sizes and retention to see if that's viable) | 21:59 |
| jeblair | (cause then it's easy to backspace up the url hierarchy) | 21:59 |
| mordred | yah. I kind of prefer that myself - if it's workable | 22:00 |
| mordred | I mean, I'm not sure what it is that's going to docs-draft that might be larger than devstack logs already are ... | 22:00 |
| jeblair | things have changed significantly with docs builds since we did that :) | 22:01 |
| mordred | yah | 22:01 |
| jeblair | let's move this to #-infra | 22:01 |
| *** tobiash has quit IRC | 22:11 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Support Gerrit 2.13 ref-updated events https://review.openstack.org/505795 | 22:25 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix infinite loop on reconfiguration exception https://review.openstack.org/505807 | 22:26 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/505845 | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:01 |
| jeblair | mordred, dmsimard: ^ | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add job.nodeset parameter to supercede job.nodes https://review.openstack.org/505845 | 23:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove Job.nodes https://review.openstack.org/505846 | 23:27 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!