| *** rfolco|rover has joined #zuul | 00:32 | |
| *** panda has quit IRC | 01:15 | |
| *** panda has joined #zuul | 01:19 | |
| *** rfolco|rover has quit IRC | 01:30 | |
| *** jamesmcarthur has joined #zuul | 02:06 | |
| *** swest has quit IRC | 02:37 | |
| *** ianychoi__ has joined #zuul | 02:38 | |
| *** ianychoi_ has quit IRC | 02:41 | |
| *** swest has joined #zuul | 02:51 | |
| *** jamesmcarthur has quit IRC | 03:02 | |
| *** jamesmcarthur has joined #zuul | 03:03 | |
| *** jamesmcarthur has joined #zuul | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:05 | |
| *** jamesmcarthur has joined #zuul | 03:06 | |
| *** jamesmcarthur has quit IRC | 03:11 | |
| *** jamesmcarthur has joined #zuul | 03:36 | |
| *** jamesmcarthur has quit IRC | 03:42 | |
| *** bhavikdbavishi has joined #zuul | 03:48 | |
| *** jamesmcarthur has joined #zuul | 04:16 | |
| *** jamesmcarthur has quit IRC | 04:20 | |
| *** panda has quit IRC | 04:34 | |
| *** panda has joined #zuul | 04:37 | |
| *** jamesmcarthur has joined #zuul | 04:48 | |
| *** jamesmcarthur has quit IRC | 04:53 | |
| *** sshnaidm_ has joined #zuul | 05:05 | |
| *** sshnaidm|off has quit IRC | 05:06 | |
| *** bolg has joined #zuul | 05:15 | |
| *** bhavikdbavishi has quit IRC | 05:29 | |
| *** jamesmcarthur has joined #zuul | 05:49 | |
| *** jamesmcarthur has quit IRC | 05:54 | |
| *** jamesmcarthur has joined #zuul | 06:10 | |
| *** panda has quit IRC | 06:41 | |
| *** panda has joined #zuul | 06:43 | |
| *** jamesmcarthur has quit IRC | 06:44 | |
| *** gtema has joined #zuul | 07:10 | |
| *** jamesmcarthur has joined #zuul | 07:47 | |
| *** jamesmcarthur has quit IRC | 07:51 | |
| *** gtema has quit IRC | 07:53 | |
| *** tosky has joined #zuul | 08:03 | |
| *** jpena|off is now known as jpena | 08:05 | |
| *** themroc has joined #zuul | 08:07 | |
| *** sshnaidm__ has joined #zuul | 08:27 | |
| *** sshnaidm_ has quit IRC | 08:27 | |
| *** hashar has joined #zuul | 08:35 | |
| *** panda has quit IRC | 08:45 | |
| *** panda has joined #zuul | 08:47 | |
| *** jamesmcarthur has joined #zuul | 08:48 | |
| *** jamesmcarthur has quit IRC | 08:53 | |
| *** jangutter has joined #zuul | 09:04 | |
| *** saneax has joined #zuul | 09:23 | |
| *** jamesmcarthur has joined #zuul | 09:24 | |
| *** jamesmcarthur has quit IRC | 09:28 | |
| *** sshnaidm__ is now known as sshnaidm | 09:43 | |
| *** dustinc is now known as dustinc_pto | 10:07 | |
| *** jamesmcarthur has joined #zuul | 10:24 | |
| *** saneax has quit IRC | 10:25 | |
| *** saneax has joined #zuul | 10:28 | |
| *** jamesmcarthur has quit IRC | 10:29 | |
| *** panda is now known as panda|pto | 10:33 | |
| *** tosky_ has joined #zuul | 11:08 | |
| *** tosky has quit IRC | 11:12 | |
| *** tosky_ is now known as tosky | 11:18 | |
| *** jangutter has quit IRC | 11:24 | |
| *** jamesmcarthur has joined #zuul | 11:25 | |
| *** jamesmcarthur has quit IRC | 11:30 | |
| *** jangutter has joined #zuul | 11:43 | |
| *** rfolco|rover has joined #zuul | 11:56 | |
| *** rfolco|rover is now known as rfolco|rucker | 11:58 | |
| *** rfolco|rucker is now known as rfolco|ruck | 11:58 | |
| *** jamesmcarthur has joined #zuul | 12:11 | |
| *** rlandy has joined #zuul | 12:13 | |
| *** jamesmcarthur has quit IRC | 12:29 | |
| *** jangutter has quit IRC | 12:43 | |
| *** jamesmcarthur has joined #zuul | 12:46 | |
| *** jpena is now known as jpena|lunch | 12:49 | |
| *** sanjayu_ has joined #zuul | 13:04 | |
| *** saneax has quit IRC | 13:07 | |
| *** sanjayu_ has quit IRC | 13:28 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Make reporting asynchronous https://review.opendev.org/691253 | 13:34 |
|---|---|---|
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Take advantage of fetch-output https://review.opendev.org/689473 | 13:55 |
| Shrews | I'm confused why that ^^ passes check but not gate | 13:55 |
| *** themroc has quit IRC | 13:55 | |
| *** themroc has joined #zuul | 13:58 | |
| *** bolg has quit IRC | 14:03 | |
| *** themroc has quit IRC | 14:05 | |
| *** themr0c has joined #zuul | 14:05 | |
| *** jamesmcarthur has quit IRC | 14:06 | |
| *** jpena|lunch is now known as jpena | 14:08 | |
| *** ffloreth__ has joined #zuul | 14:23 | |
| *** themr0c has quit IRC | 14:25 | |
| *** raukadah is now known as chandankumar | 14:26 | |
| *** jamesmcarthur has joined #zuul | 14:28 | |
| *** jamesmcarthur has quit IRC | 14:34 | |
| openstackgerrit | Merged zuul/zuul-website master: Correct mistyped name in Tungsten Fabric case https://review.opendev.org/691522 | 14:36 |
| *** jamesmcarthur has joined #zuul | 14:42 | |
| *** ffloreth__ has quit IRC | 15:13 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 15:22 |
| *** todun has joined #zuul | 15:43 | |
| tristanC | Shrews: it seems like the gate failure is unrelated to the fetch-output change, e.g. https://zuul.opendev.org/t/zuul/build/71466f70171a4e4ea88557d84b61ccfd/log/job-output.txt#32000-32001 | 15:43 |
| *** mattw4 has joined #zuul | 15:48 | |
| *** igordc has joined #zuul | 16:05 | |
| dmellado | mnaser: o/ ping, around? | 16:05 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 16:06 |
| Shrews | tristanC: no, it's related. see a few lines above that: ls: cannot access '/home/zuul/zuul-output/logs/nodepool/builds/': No such file or directory | 16:06 |
| Shrews | i just don't understand why that would sometimes succeed | 16:07 |
| *** Goneri has joined #zuul | 16:07 | |
| dmellado | Goneri: o/ | 16:08 |
| dmellado | I was just pinging mnaser on the flavor sizes@vexxhost | 16:08 |
| mnaser | hi dmellado | 16:15 |
| *** todun has quit IRC | 16:34 | |
| *** todun has joined #zuul | 16:35 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: Don't add implicit role if plugin found https://review.opendev.org/691715 | 16:38 |
| *** bhavikdbavishi has joined #zuul | 16:40 | |
| *** phildawson has quit IRC | 16:42 | |
| *** jamesmcarthur has quit IRC | 16:49 | |
| *** jpena is now known as jpena|off | 16:50 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Don't add implicit role if plugin found https://review.opendev.org/691715 | 16:59 |
| tristanC | Shrews: oh indeed... so in check it does create the directory: https://zuul.opendev.org/t/zuul/build/d1413f987c4c41c2ad14721d125296d1/log/job-output.txt#29238-29241 | 17:02 |
| tristanC | but not in gate: https://zuul.opendev.org/t/zuul/build/71466f70171a4e4ea88557d84b61ccfd/log/job-output.txt#29243-29245 | 17:02 |
| tristanC | it's as if the gate pipeline didn't picked the change | 17:04 |
| *** hashar has quit IRC | 17:05 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 17:08 |
| *** Goneri has quit IRC | 17:14 | |
| *** Goneri has joined #zuul | 17:15 | |
| corvus | tristanC, clarkb: take a look at https://review.opendev.org/689280 - it passes! :) there's still at least one spot where we can get hit by a skopeo ipv6 error. but it's really close. | 17:27 |
| Shrews | tristanC: the check log you pasted is with the newest patchset which I just pushed up this morning that makes sure that directory is created. | 17:29 |
| Shrews | it's weird that sometimes that isn't needed, apparently *shrugs* | 17:29 |
| *** pcaruana has joined #zuul | 17:31 | |
| *** Goneri has quit IRC | 17:35 | |
| *** igordc has quit IRC | 17:43 | |
| *** Goneri has joined #zuul | 17:48 | |
| *** bhavikdbavishi has quit IRC | 18:13 | |
| *** pcaruana has quit IRC | 18:30 | |
| *** pcaruana has joined #zuul | 18:31 | |
| clarkb | corvus: left a couple of comments. What are the remaining ipv6 issues? | 18:44 |
| corvus | clarkb: there's a todo comment in the change for one of them, and i think a couple of the recent failures were ipv6 as well. | 18:46 |
| clarkb | I guess this isn't using socat? | 18:47 |
| corvus | i'm unclear about whether we can solve them with hostname entries, or if we need a second socat or what.... | 18:47 |
| clarkb | fwiw I believe that things work if names resolve to ipv6 addrs | 18:47 |
| clarkb | we didn't end up using that method in production because it required updating /etc/hosts within executor bwrap contexts and that got a little hacky | 18:48 |
| corvus | yeah, but i think the exector (or the fake executor) may be involved, and i think we used socat because we didn't want to make /etc/hosts changes on the executor? | 18:48 |
| clarkb | yup | 18:48 |
| corvus | though -- did it work? | 18:48 |
| clarkb | yes I believe it worked | 18:48 |
| corvus | but we just found it distasteful? | 18:48 |
| clarkb | well I should rephrase I don't know if we could update /etc/hosts with the existing bwrap mount rules | 18:48 |
| clarkb | but when tested without bwrap interferring docker push and pull seemd to work with /etc/host hancks | 18:49 |
| corvus | anyway, the action item is to dig into that at least enough to articulate the question properly :) | 18:49 |
| *** openstackgerrit has quit IRC | 18:50 | |
| *** Goneri has quit IRC | 18:51 | |
| *** mordred has quit IRC | 18:52 | |
| *** todun_ has joined #zuul | 19:09 | |
| *** todun has quit IRC | 19:10 | |
| *** todun_ is now known as todun | 19:10 | |
| *** hashar has joined #zuul | 19:20 | |
| *** igordc has joined #zuul | 19:27 | |
| *** Goneri has joined #zuul | 19:40 | |
| *** hashar is now known as hasharAway | 19:47 | |
| *** todun has quit IRC | 19:50 | |
| *** ryanpetrello has joined #zuul | 19:51 | |
| *** todun has joined #zuul | 19:53 | |
| *** pcaruana has quit IRC | 20:02 | |
| *** hasharAway is now known as hashar | 20:07 | |
| *** todun has quit IRC | 20:14 | |
| *** mordred has joined #zuul | 20:43 | |
| *** jamesmcarthur has joined #zuul | 20:47 | |
| corvus | clarkb: i *think* the only remaining ipv6 issue is that todo -- and the problem is basically this: on the executor, we pull from the intermediate registry and push into the buildset registry with "skopeo copy docker://intermediate-registry/image docker://127.0.0.1:12345" where 127.0.0.1:12345 is a socat to the buildset registry. but the buildset registry is now using the new jwt auth thing, and that | 20:50 |
| corvus | requires putting in a URL (which points back to the registry) in the www-authenticate header. afaict, it has to be a full url, it can't be relative. and we can't use the host header to construct it dynamically because that doesn't tell us the socat port. the role that starts the buildset registry shouldn't (or can't) know about the socat in order to pre-construct that as part of the config. i do believe | 20:50 |
| corvus | that using hostnames on the executor instead of socat would address this problem, and i'm not immediately coming up with other ideas. | 20:50 |
| clarkb | how does it work in production? that is doesnt use token autg? | 20:52 |
| clarkb | a layer 7 proxy could rewrite the header but that is a lot heavier than just socat | 20:53 |
| *** tosky has quit IRC | 20:53 | |
| corvus | clarkb: we're not using zuul-registry as a buildset registry in production yet (this is the change series that does that) | 20:53 |
| clarkb | right but we still do auth to the registry in production through skopeo amd socat | 20:55 |
| corvus | (we are using it in prod as an intermediate registry, but that has hostnames. and likewise, we use hostnames for the "fake intermediate registry" in our testing (it's running on a fake executor, so it's less controversial to add an entry to /etc/hosts there, *and* it makes it behave more like prod). | 20:55 |
| corvus | clarkb: no, we only use skopeo for the buildset registry | 20:55 |
| corvus | er socat | 20:55 |
| clarkb | but it authenticates to the buildset registry? | 20:55 |
| clarkb | is that authentication different? | 20:56 |
| corvus | yes, that's using the docker registry with basic auth | 20:56 |
| clarkb | if we put socat on a fixed port it would be maybe less problematic then we always 'localhost' to it but that is hacky and maybe more hacky than editing /etc/hosts | 20:58 |
| corvus | that also imposes some requirements on the executors | 20:59 |
| corvus | i guess they both do -- either "you must have network port isolation between jobs" or "you must be able to write to /etc/hosts" | 21:00 |
| corvus | clarkb: *or* we write a tool to do what skopeo is doing here. | 21:01 |
| corvus | this is one of those cases where if the tool just worked over v6, we would not have a problem. and this is constrained to a couple of executor-only roles. the choice of tool doesn't really impact anything about jobs that use it. | 21:01 |
| clarkb | ya | 21:04 |
| corvus | mordred, tristanC: thoughts ^ ? | 21:05 |
| mordred | corvus: reading | 21:07 |
| clarkb | to be clear the issue is that docker doesn't allow for [] or : in the host portion of their "urls" (you can use : to separate host and port but that is it) so you cannot use ipv6 literals in the url | 21:08 |
| clarkb | skopeo includes docker's lib for parsing these urls | 21:08 |
| clarkb | and refuses to change according to a response on a bug we filed | 21:08 |
| corvus | oh hrm | 21:09 |
| mordred | my first inclination is to just use hostnames | 21:09 |
| corvus | actually, even that might not solve the problem | 21:09 |
| mordred | it's a little bit of test node pollution- but it's unlikely to be pollution that would substantively change things for test workloads | 21:10 |
| corvus | er, let me revise that to: i am not certain that would solve the issue -- i'm not sure that we've tested that skopeo and docker can handle ipv6 literal auth urls (nevermind image names). it seems plausible that they would (they probably use url handling libs for that). but we should verify that first if we're going to go down that path. | 21:11 |
| clarkb | mordred: its not the test nodes it is the executor | 21:11 |
| clarkb | mordred: we do use hostnames on the test nodes already | 21:11 |
| clarkb | corvus: oh good point | 21:11 |
| mordred | ohhh. duh. yeah | 21:12 |
| mordred | corvus: well, if we wrote a tool, it could just be part of zuul and would reduce the operator installation burden on the executors | 21:14 |
| mordred | otoh - that's more writing of things | 21:15 |
| corvus | mordred: it could, or we could put it in zuul-jobs, or we could make it standalone and pip install --user it in the jobs | 21:15 |
| mordred | yeah | 21:15 |
| corvus | i agree we shouldn't require the operator to install something | 21:15 |
| mordred | but basically - we have many options that would be easier on the operator than them needing to install skopeo on the executors | 21:16 |
| mordred | how hard of a tool do you think it would be to write? it gets to be pretty single-purpose I think, right? | 21:17 |
| tristanC | providing a correct /etc/hosts per execution context would fix that issue? if it does, that seems like a reasonable thing to do | 21:18 |
| corvus | huh. rfc2617 suggests that an relative path is okay.... i should verify that i've adequately tested that (vs docker just claiming to implement 2617 without having read it) | 21:18 |
| corvus | mordred: i think it would be simple. i'd estimate the main methods to be < 100 lines of python. | 21:19 |
| corvus | okay, i did try a relative url, and this is what skopeo/moby returned: https://zuul.opendev.org/t/zuul/build/9dda450a63b54b05aebdbb0028169832 | 21:26 |
| corvus | shock bombshell: moby does not follow rfc2617 though it claims to | 21:27 |
| corvus | maybe we can split brain this though | 21:28 |
| mordred | corvus, tristanC: if we wanted to go /etc/hosts route, what if zuul just *always* provided /etc/hosts entries for all of the nodes in a job's nodeset? paused nodes would start to get weird ... | 21:28 |
| clarkb | mordred: I'm not sure how that helps with the exectuor? | 21:28 |
| corvus | maybe if moby supports fully qualified ipv6 literal urls as auth urls, maybe we can tell docker to pull from "localhost:12345" which then tells moby to get an auth token from [literal]:9000 | 21:28 |
| corvus | i'm assuming mordred means to have the zuul executor add all the nodes to an overlaid /etc/hosts | 21:29 |
| mordred | clarkb: why not? if the executor bind-mounted in an /etc/hosts to the bubblewrap context | 21:29 |
| mordred | yeah | 21:29 |
| clarkb | I see | 21:29 |
| corvus | basically doing something similar to what we'd do in the role, but systemically and less hacky | 21:29 |
| corvus | that might open some attack vectors though | 21:29 |
| corvus | (request a nodeset with some name to trick a trusted playbook into sending creds to a test node or something) | 21:30 |
| corvus | yeah, i think that'd be a can of worms. broadly speaking, i think that would mean "trusted playbooks can't trust dns" | 21:31 |
| mordred | yeah | 21:31 |
| corvus | shame. it's really elegant. :) | 21:31 |
| mordred | well - what if part of the hostname was generated? | 21:32 |
| corvus | clarkb: there's no single way to construct an ipv4/ipv6 literal, right? we have to switch (to know to add []) ? | 21:32 |
| tristanC | mordred: why would it get weird with paused nodes? | 21:32 |
| mordred | like - if the hostname put in /etc/hosts was "{node-name}.{job-name}.zuul" or something | 21:32 |
| corvus | mordred: oh yeah, good point. we do something similar with our in-job hostnames to make sure they never collide. | 21:33 |
| clarkb | corvus: if you aren't specifying a port too you can avoid caring. It is the port specification that cauises problems because the : becomes ambiguous | 21:33 |
| mordred | yah | 21:33 |
| clarkb | but in general ya if using a port you have to know. There is an ansible filter for this | 21:33 |
| corvus | clarkb: oh, do we have an example? | 21:34 |
| mordred | tristanC: other builds in the buildset would need the hostname of paused nodes in their /etc/hosts - but the executor usually doesn't have that information since it's just focused on the one build, so we'd have to pass some stuff along somewhere - probably not too terrible to do - just something we'd have to sort | 21:34 |
| corvus | clarkb: ipwrap? https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters_ipaddr.html#wrapping-ipv6-addresses-in-brackets | 21:36 |
| clarkb | corvus: https://docs.ansible.com/ansible/latest/user_guide/playbooks_filters_ipaddr.html yup | 21:36 |
| *** openstackgerrit has joined #zuul | 21:37 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: test ipwrap https://review.opendev.org/691758 | 21:37 |
| corvus | clarkb: ^ does that look right? | 21:37 |
| clarkb | ya | 21:38 |
| corvus | clarkb, mordred, tristanC: okay, let's see if that works, and if so, then great. if not, we'll mull over the "hostnames or new tool" question some more | 21:38 |
| mordred | ++ | 21:39 |
| *** EmilienM has quit IRC | 21:46 | |
| *** EmilienM has joined #zuul | 21:47 | |
| *** todun has joined #zuul | 21:52 | |
| *** jamesmcarthur has quit IRC | 22:28 | |
| *** mattw4 has quit IRC | 22:28 | |
| *** mattw4 has joined #zuul | 22:28 | |
| corvus | clarkb, mordred, tristanC: i think that change worked! https://zuul.opendev.org/t/zuul/build/db01a689f6624ad3a09bfff96eb7e709/console | 22:29 |
| corvus | in just a few, i'll rework that into the appropriate point in the stack, and then i think it'll be ready to merge | 22:30 |
| mordred | corvus: oh cool! | 22:30 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Use zuul-registry as buildset registry https://review.opendev.org/689238 | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Make the buildset registry port configurable https://review.opendev.org/689240 | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Adjust buildset registry container name https://review.opendev.org/690992 | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Don't sudo when saving container logs in registry test https://review.opendev.org/690993 | 22:45 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 22:46 |
| *** hashar has quit IRC | 22:47 | |
| mordred | corvus: in https://review.opendev.org/#/c/690993 - do we put the zuul user into the docker group somewhere? | 23:14 |
| *** mattw4 has quit IRC | 23:19 | |
| mordred | I mean - it's working, so I guess we do :) | 23:20 |
| openstackgerrit | Merged zuul/zuul-jobs master: Open iptables ports in registry test job https://review.opendev.org/690425 | 23:30 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!