| *** irclogbot_1 has quit IRC | 00:16 | |
| *** rlandy has joined #zuul | 00:18 | |
| *** irclogbot_3 has joined #zuul | 00:21 | |
| *** bolg has quit IRC | 00:27 | |
| *** rfolco has quit IRC | 00:33 | |
| *** saneax has joined #zuul | 02:21 | |
| *** bhavikdbavishi has joined #zuul | 02:35 | |
| *** bhavikdbavishi1 has joined #zuul | 03:10 | |
| *** bhavikdbavishi has quit IRC | 03:12 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:12 | |
| *** saneax has quit IRC | 03:58 | |
| *** bhavikdbavishi has quit IRC | 04:20 | |
| *** bhavikdbavishi has joined #zuul | 04:21 | |
| *** vishalmanchanda has joined #zuul | 04:51 | |
| *** saneax has joined #zuul | 05:49 | |
| *** bhavikdbavishi1 has joined #zuul | 06:01 | |
| *** bhavikdbavishi has quit IRC | 06:02 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:02 | |
| *** marios has joined #zuul | 06:28 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Remove copy paste from upload-logs-swift https://review.opendev.org/741840 | 06:41 |
|---|---|---|
| *** bhavikdbavishi has quit IRC | 06:41 | |
| *** holser has joined #zuul | 06:53 | |
| zbr|ruck | reviews needed on https://review.opendev.org/#/c/740733/ and https://review.opendev.org/#/c/739482/ | 07:00 |
| *** bhavikdbavishi has joined #zuul | 07:07 | |
| *** bhavikdbavishi has quit IRC | 07:13 | |
| *** jcapitao has joined #zuul | 07:16 | |
| *** bhavikdbavishi has joined #zuul | 07:22 | |
| *** yolanda has joined #zuul | 07:31 | |
| *** tosky has joined #zuul | 07:34 | |
| *** jpena|off is now known as jpena | 07:46 | |
| *** bhavikdbavishi has quit IRC | 07:57 | |
| *** wuchunyang has joined #zuul | 08:03 | |
| *** bhavikdbavishi has joined #zuul | 08:07 | |
| *** nils has joined #zuul | 08:09 | |
| *** wuchunyang has quit IRC | 08:31 | |
| *** bolg has joined #zuul | 08:42 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler's pause/resume functionality https://review.opendev.org/709735 | 08:50 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Separate connection registries in tests https://review.opendev.org/712958 | 08:50 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the labels requirement https://review.opendev.org/741893 | 08:58 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Prepare Zookeeper for scale-out scheduler https://review.opendev.org/717269 | 09:13 |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: Bump ansible-lint to speed it up https://review.opendev.org/741897 | 09:27 |
| *** sshnaidm|off is now known as sshnaidm | 09:27 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Mandatory Zookeeper connection for ZuulWeb in tests https://review.opendev.org/721254 | 09:30 |
| *** bhavikdbavishi has quit IRC | 09:45 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect merger to Zookeeper https://review.opendev.org/716221 | 09:46 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect fingergw to Zookeeper https://review.opendev.org/716875 | 09:46 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Connect executor to Zookeeper https://review.opendev.org/716262 | 09:46 |
| *** bhavikdbavishi has joined #zuul | 09:48 | |
| webknjaz | Folks, Cheroot needs more eyes on the new regression fix PR | 10:04 |
| webknjaz | https://github.com/cherrypy/cheroot/pull/301 | 10:04 |
| webknjaz | If somebody can run it in their envs and it explodes, plz tell me. | 10:05 |
| *** tosky has quit IRC | 10:08 | |
| *** holser has quit IRC | 10:08 | |
| *** irclogbot_3 has quit IRC | 10:08 | |
| *** reiterative has quit IRC | 10:08 | |
| *** fdegir has quit IRC | 10:09 | |
| *** decimuscorvinus has quit IRC | 10:09 | |
| *** tosky has joined #zuul | 10:10 | |
| *** holser has joined #zuul | 10:10 | |
| *** irclogbot_3 has joined #zuul | 10:10 | |
| *** reiterative has joined #zuul | 10:10 | |
| *** fdegir has joined #zuul | 10:10 | |
| *** decimuscorvinus has joined #zuul | 10:10 | |
| *** smyers has quit IRC | 10:14 | |
| *** smyers has joined #zuul | 10:15 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/717299 | 10:32 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 10:45 |
| *** bhavikdbavishi has quit IRC | 10:47 | |
| zbr|ruck | is anyone working on publishing zuul-roles as a collection? I would really want to make use of some of them outside zuul | 10:54 |
| zbr|ruck | stuff like ensure-docker are very useful | 10:54 |
| *** wuchunyang has joined #zuul | 10:57 | |
| *** jcapitao is now known as jcapitao_lunch | 11:13 | |
| *** bhavikdbavishi has joined #zuul | 11:19 | |
| *** wuchunyang has quit IRC | 11:23 | |
| *** bhavikdbavishi has quit IRC | 11:30 | |
| *** bhavikdbavishi has joined #zuul | 11:34 | |
| *** weshay_pto is now known as weshay_ | 11:34 | |
| *** jpena is now known as jpena|lunch | 11:45 | |
| *** rfolco has joined #zuul | 11:59 | |
| *** yolanda has quit IRC | 12:14 | |
| *** yolanda has joined #zuul | 12:15 | |
| *** Goneri has joined #zuul | 12:17 | |
| *** jcapitao_lunch is now known as jcapitao | 12:31 | |
| *** jpena|lunch is now known as jpena | 12:32 | |
| tristanC | zbr|ruck: what would need to happen to use zuul-jobs roles outside of zuul? | 12:34 |
| tristanC | zbr|ruck: (not sure what you meant by `zuul-roles` | 12:35 |
| tristanC | iirc we talked about publishing the zuul_console and zuul_return action modules, those are currently in zuul source tree | 12:36 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the merge reported https://review.opendev.org/741931 | 12:50 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the merge reporter https://review.opendev.org/741931 | 12:51 |
| *** rlandy has joined #zuul | 12:52 | |
| *** rlandy is now known as rlandy|ruck | 12:54 | |
| fungi | zbr|ruck: if you mean zuul-jobs, we've previously made choices about that repo on the basis that we would not be publishing it as a collection (disabling certain ansible-lint checks, continuing to use hyphens in role names...) | 13:01 |
| clarkb | https://zuul.opendev.org/t/zuul/build/55dcbafa54f04804afd0b073e87fdc27/log/job-output.txt#5305 seems to be breaking nodepool iamge builds. I'm not quite sure yet what layer is missing make but will look into it as I can during the opendev event today (feel free to dig in if you can without interruption) | 13:21 |
| clarkb | I think that is happening in the arm64 builder's assemble call. Likely fallout from building an arm64 python-builder image | 13:23 |
| clarkb | I'm guessing the two images, amd64 and arm64 actually have different sets of preinstalled packages /me working to confirm that now | 13:24 |
| clarkb | but not sure how to run arm64 container on local machine | 13:24 |
| fungi | you should be able to do it under qemu, in theory | 13:26 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: remove default mutables https://review.opendev.org/741938 | 13:30 |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Install make to build wheels https://review.opendev.org/741942 | 13:33 |
| clarkb | I think ^ will do it (eventually decided just having zuul run the job is probably easiest) | 13:33 |
| *** saneax has quit IRC | 13:34 | |
| *** saneax has joined #zuul | 13:35 | |
| *** bhavikdbavishi has quit IRC | 13:43 | |
| fungi | it's not necessarily the preinstalled packages differing, but that there's a lot more stuff on pypi prebuilt for x86 and not arm, therefore you wind up having to install and build a lot of additional architecture-dependent wheels on the fly from sdist tarballs instead | 13:44 |
| clarkb | ah | 13:45 |
| clarkb | ya that could explain it, we're actually building wheels on arm but not amd64 | 13:45 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: update wording patchset to patch_number in getChange https://review.opendev.org/740504 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: implement the open pipeline requirement https://review.opendev.org/740707 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: implement the merged pipeline requirement https://review.opendev.org/740708 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: implement the approval reporter https://review.opendev.org/741438 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: update naming in reference pipeline https://review.opendev.org/741450 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the merge request approval event https://review.opendev.org/741483 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the approval requirement https://review.opendev.org/741637 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the labeled event https://review.opendev.org/741667 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the labels requirement https://review.opendev.org/741893 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: support the merge reporter https://review.opendev.org/741931 | 14:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab: remove default mutables https://review.opendev.org/741938 | 14:05 |
| clarkb | I think we've also discovered where arm64 emulation on amd64 VMs is slow: building wheels | 14:12 |
| corvus | clarkb: bummer; before it was like just a few minutes slower, is it much slower now? | 14:16 |
| corvus | i wonder if we could speed that up either with layer caching, or adding a new image that has all our pre-reqs... | 14:17 |
| corvus | looks like the build took 30m | 14:17 |
| clarkb | corvus: well it timed out | 14:18 |
| corvus | oh :( | 14:18 |
| corvus | i guess it has a 30m limit? | 14:18 |
| corvus | previous builds seem to have taken 15-20m | 14:19 |
| clarkb | ya, I'm going to bump it to an hour to gather more data | 14:19 |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Install make to build wheels https://review.opendev.org/741942 | 14:20 |
| clarkb | corvus: we could potentially use some of the wheel cache stuff that ianw and kevinz are working on for testing in the linaro cloud | 14:21 |
| corvus | clarkb: how's that work? | 14:22 |
| clarkb | corvus: so far just our normal wheel mirrors | 14:22 |
| fungi | wheel build jobs run on arm nodes to natively build arm binary wheels and then we cache those alongside our x86 ones | 14:22 |
| clarkb | I'm not sure what platforms they are planning to build them for yet, but I know debian is one of the images in the linaro cloud so adding it for debian isn't a crazy idea | 14:23 |
| fungi | so i guess the suggestion is to use our prebuilt binary wheelhouse during image builds? | 14:23 |
| clarkb | then we could have a switch in python-builder's assemble that checks arch and if arm64 adds a pointer to that wheel cache? | 14:23 |
| fungi | we already use our distro package mirrors during image builds, so this wouldn't be a stretch in opendev i suppose | 14:24 |
| clarkb | fungi: we don't for the containers | 14:24 |
| fungi | oh! right, containers | 14:24 |
| fungi | my head was in nodepool image builds, sorry | 14:24 |
| fungi | nodepool node image builds i mean | 14:24 |
| corvus | what decides what wheels get built? | 14:25 |
| clarkb | corvus: I think it may still be openstack's global requirements list | 14:25 |
| clarkb | but those pip indexes are additive so we can safely add it and if it has the packages we need we win if it doesn't we build | 14:25 |
| clarkb | unless you have pip8 | 14:25 |
| corvus | so we'd need to generalize this for opendev by adding new sources | 14:26 |
| fungi | that's been a lingering to do item anyway (make opendev's wheel cache not openstack-specific) | 14:26 |
| clarkb | if that was going to be a long term solution to the problem then ya we'd probably want to have a "this is a list of python things that are generally useful to have wheels for" list | 14:26 |
| clarkb | cryptography and its deps would be high on that list | 14:26 |
| corvus | clarkb: or rather a list of lists? | 14:27 |
| fungi | numpy, et cetera | 14:27 |
| clarkb | corvus: ya I do worry about scaling that particularly for these slow platforms | 14:27 |
| clarkb | it would be easy to have a 48 hour build job | 14:27 |
| clarkb | another approach could be to work with upstream pacakges to add arm64 wheels to pypi | 14:27 |
| fungi | it can also be parallelized, no reason those all have to be built in a single job | 14:27 |
| fungi | but yeah, getting arm wheels into pypi would help lots of people beyond opendev | 14:28 |
| zbr|ruck | while I find a wheel cache very useful, i need to warn others that the common list is more like >200 packages, and likely a couple of GB in size. | 14:28 |
| zbr|ruck | packages, not files, multiply by flavours. | 14:29 |
| clarkb | zbr|ruck: yes, we've been running a wheel mirror cache for years | 14:29 |
| clarkb | its just been x86 specific so far | 14:29 |
| fungi | zbr|ruck: well, the job in question optimizes by only caching wheels that need to be built, and skips those which are already available on pypi (as pure python or manylinux1 for example) | 14:29 |
| clarkb | I'm not super concerned about the size of the data as much as the cost to produce it, but we can also worry about that as we hit those problems | 14:30 |
| zbr|ruck | arm and ppc is clearly making the issue more complex | 14:30 |
| clarkb | well we don't need to worry about ppc, at least not yet | 14:30 |
| clarkb | "Building wheels for collected packages: PyYAML, PrettyTable, voluptuous, openshift, bcrypt, cryptography, pynacl, netifaces, dogpile.cache, cffi, MarkupSafe, ruamel.yaml.clib" thats our wheel build list for nodepool on arm64 fwiw | 14:31 |
| fungi | if anybody knows people who work on openshift, maybe they'd be an easy one to get releasing arm wheels on pypi soon | 14:32 |
| corvus | given entries like crypto, maybe we would immediately benefit from adding the openstack cache? | 14:33 |
| clarkb | it was processing those in that order and everything before bcrypt was a few seconds. bcrypt took about a minute then cryptography was running for almost 10 minutes when the job timed out | 14:33 |
| clarkb | corvus: yes, that was my point | 14:33 |
| clarkb | but it doesn't look like we have the arm64 stuff up and running yet | 14:33 |
| clarkb | oh wait I know I have to look at the linaro mirror | 14:34 |
| corvus | i'm guessing bcrypt is transitive from cryptography too | 14:34 |
| fungi | though the wheel build jobs build specific versions pinned in openstack's central constraints list, so might lag behind latest releases of stuff regularly | 14:34 |
| fungi | https://static.opendev.org/mirror/wheel/debian-10-aarch64/ | 14:35 |
| clarkb | and I'm guessing the NAT issues there persist because I can't hit the mirror directly | 14:35 |
| clarkb | fungi: thanks | 14:35 |
| clarkb | maybe we should add the serving of all the wheel caches to all the mirrors even if they dno't directly host that arch in that provider | 14:36 |
| clarkb | since cross platform builds are a thing (as nodepool is trying to do) | 14:36 |
| fungi | seems reasonable to me. multi-arch clouds are a thing. but probably better to discuss that in #opendev | 14:37 |
| clarkb | corvus: one potential issue here is we've gnerally build these wheels for distro python. Taking the example we've got here our debian 10 wheel mirror builds for debian python2.7 and debian python3.7. Our python-base images use python on debian (not debian python) | 14:45 |
| corvus | is that a different kind of wheel? | 14:45 |
| clarkb | I don't know if we should expect python3.7 wheels build for debian python to work with python3.7 on debian | 14:45 |
| clarkb | corvus: not a different kind of wheel, but wheels built for slgihtly different targets | 14:46 |
| clarkb | it may be the case that the targets are similar enough to each other thing things just work | 14:46 |
| corvus | clarkb: ok. it sounds like we have some steps we can take to experiment, but quick sanity check: is this easier/better than inserting a new image into the stack that just builds nodepool dependencies? | 14:47 |
| corvus | eg nodepool -> nodepool-dependencies -> python-base | 14:48 |
| clarkb | I think the advantage to using a mirror cache like that is we'd keep the build deps reasonably fresh on every build. THe advantage to a new image layer is it is simple with the existing tooling and avoids the target platform mismatch, but we'd not be rebuilding them every time | 14:48 |
| corvus | clarkb: right, but we can rebuild them daily and on-demand | 14:48 |
| clarkb | for now that is probably fine and have nodepool-dependencies run a daily job to update | 14:49 |
| corvus | we do have a system for speculative dependency building | 14:49 |
| clarkb | interestingly my latest image build job has decided to build pynacl before bycrypt. It too is slow | 14:49 |
| corvus | (also, we could still probably build new things in our nodepool image itself if a dependency changed) | 14:50 |
| clarkb | corvus: that would likely automatically happen, basically if the dep isn't there it would build itb eacuse that is how pip would handle the situation | 14:50 |
| corvus | then the dependencies layer just becomes a cache | 14:52 |
| clarkb | pynacl is a manylinux wheel. manylinux publishes an aar64 image: https://quay.io/repository/pypa/manylinux2014_aarch64?tab=info I wonder if we could work with pypa to switch their default build tooling/docs to buildx instead of regular docker build then people would start to build arm and ppc wheels by default if using manylinux | 15:01 |
| clarkb | that is likely way out of scope here, but does point to an advantage of the manylinux system: if it can be convinced to do more we get more out of it | 15:01 |
| fungi | i think you're going to find quite a few in that situation | 15:03 |
| clarkb | https://github.com/pypa/manylinux/issues/84 raspi to the rescue | 15:06 |
| clarkb | we could try https://www.piwheels.org/ | 15:06 |
| clarkb | the latest raspi is arm64 and maybe things will work? | 15:06 |
| fungi | hah, i was just responding on a piwheels issue over the weekend | 15:08 |
| clarkb | no their pacakges are all 32bit | 15:08 |
| clarkb | and no python3.8 | 15:08 |
| clarkb | https://www.piwheels.org/simple/cryptography/ for example | 15:08 |
| clarkb | also my idea to builx has received a thumbs down: https://github.com/pypa/manylinux/issues/84#issuecomment-514585688 I wish the system made you express why you do't like an idea :/ | 15:09 |
| *** weshay_ is now known as weshay|ruck | 15:12 | |
| fungi | someone mentioned buildx in that issue a year ago | 15:12 |
| clarkb | fungi: ya thats my link above, it got a thumbsdown | 15:12 |
| clarkb | also pynacl build is approaching 30 minutes | 15:12 |
| *** zbr|ruck is now known as zbr|rover | 15:13 | |
| fungi | ahh, yeah i thought you meant you had suggested in that issue it too | 15:13 |
| clarkb | no someone else had the same idea and was given negative feedback with no actual reason for it | 15:13 |
| clarkb | maybe the issue is it takes 30 minutes or more to build pynacl | 15:13 |
| *** bhavikdbavishi has joined #zuul | 15:14 | |
| *** hamalq has joined #zuul | 15:15 | |
| zbr|rover | clarkb: fungi: need review on cmd: https://review.opendev.org/#/c/740733/ and preferences https://review.opendev.org/#/c/739482/ | 15:16 |
| zbr|rover | i updated them over the weekend, hopefully they are ok now. | 15:16 |
| zbr|rover | i wonder if reviewers would appreciate if I include screenshot in review message, to highlight UI area affected. | 15:26 |
| clarkb | zbr|rover: for the cmd change I wonder why make it a tooltip and not just a table filed? | 15:28 |
| clarkb | I wouldn't think to hover for that info but if it said 'cmd: this is the cmd' that would be helpful? | 15:28 |
| clarkb | the job with the hour timeout timed out building pynacl | 15:32 |
| clarkb | so ya I think probably should start looking at another layer. I'm not sure I'll be able to sort that out in the near future. Distracted by opendev event and have a few things I need to do today | 15:33 |
| corvus | clarkb: should we try using the openstack arm mirrors as well? | 15:35 |
| fungi | we'd need to switch to python3.7 or get some 3.8 wheel builds going there | 15:35 |
| clarkb | ya we could try switching to 3.7 (its easy) and then use the mirror to see how that does | 15:36 |
| corvus | we can "switch" experimentally right? | 15:36 |
| clarkb | corvus: yes | 15:36 |
| fungi | right now they're building for debian buster, which doesn't have 3.8 so they're 3.7 wheels | 15:36 |
| clarkb | its just a change of the parent image name | 15:36 |
| clarkb | we should update our non arm mirrors to serve that content too, but I'll write a change to get it from the arm mirror | 15:37 |
| *** bhavikdbavishi1 has joined #zuul | 15:37 | |
| corvus | k, i can start on a new image in a bit | 15:37 |
| *** bhavikdbavishi has quit IRC | 15:39 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:39 | |
| *** marios is now known as marios|out | 15:48 | |
| *** marios|out has quit IRC | 15:49 | |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 15:51 |
| clarkb | I think I got the conditional correct there | 15:51 |
| corvus | clarkb: out of curiosity, do you actually need a conditional? | 15:53 |
| clarkb | corvus: I don't think so because pip will check the arch types on those wheels and see they are all wrong when running on x86, but I thought it was nice to be explicit. At least until we decide we are comfortable with not being explicit | 15:54 |
| clarkb | I guess for testing it may reduce a rtt | 15:54 |
| clarkb | ya I think we can simplify that to just copy the pip.conf always and let pip sort out what wheels are appropraite for the given platform | 15:55 |
| fungi | i agree we should probably be relying on the cache/mirror for both arches for consistency | 15:56 |
| openstackgerrit | Merged zuul/zuul-jobs master: add-build-sshkey: Ensure .ssh exists, enable admin authorized_keys https://review.opendev.org/740350 | 15:57 |
| *** sshnaidm is now known as sshnaidm|afk | 15:57 | |
| clarkb | I think the risk here is ipv4 has been flaky in that linaro cloud recently | 15:57 |
| *** armstrongs has joined #zuul | 15:58 | |
| clarkb | but if we get the other mirrors hosting the same content we can point at the mirror local to the job and get better reliability | 15:58 |
| clarkb | and then it really shouldn't matter if we just set the values | 15:58 |
| fungi | yeah, i don't see a good reason to make the mirrors architecture-specific there | 15:58 |
| *** hamalq has quit IRC | 16:01 | |
| *** hamalq has joined #zuul | 16:01 | |
| *** hamalq_ has joined #zuul | 16:07 | |
| *** holser has quit IRC | 16:07 | |
| *** holser has joined #zuul | 16:08 | |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 16:11 |
| *** hamalq has quit IRC | 16:11 | |
| *** holser_ has joined #zuul | 16:11 | |
| *** yoctozepto has quit IRC | 16:12 | |
| *** yoctozepto has joined #zuul | 16:13 | |
| *** holser has quit IRC | 16:14 | |
| *** nils has quit IRC | 16:21 | |
| openstackgerrit | Clark Boylan proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 16:30 |
| zbr|rover | clarkb: cmd change is consitent with current implementation of other fields, that is why it does not have a visible label. | 16:31 |
| zbr|rover | it is not really a problem and tbh I do not care much about the tooltip, I care more to about using <pre> to display it. | 16:31 |
| *** jcapitao has quit IRC | 16:31 | |
| zbr|rover | because in some cases cmd can be 4-10 multiline bash script. | 16:31 |
| zbr|rover | we should also try to avoid cluttering the UI with repetitive info that does not add much value, like label. | 16:33 |
| zbr|rover | tooltips are quite an useful way to provide more info | 16:34 |
| zbr|rover | who was complaining about ansible-lint being slow? https://review.opendev.org/#/c/741897/ makes it 10x faster. | 16:36 |
| zbr|rover | well, real value is around 9x, but rounded up for marketing purposes :D | 16:37 |
| *** jlk has quit IRC | 16:43 | |
| *** jpena is now known as jpena|off | 16:45 | |
| *** bhavikdbavishi has quit IRC | 17:24 | |
| *** bhavikdbavishi has joined #zuul | 17:25 | |
| clarkb | corvus: it seems that my change helped, we only needed to build openshift, dogpile.cache and ruamel.yaml.clib wheels. That took just under 7 minutes. However on the python-base side it has decided that it needs to build those packages anyway so something with the wheel caching there isn't working | 17:31 |
| *** armstrongs4 has joined #zuul | 17:31 | |
| clarkb | it fails beacuse there is no gcc on python-base side | 17:31 |
| corvus | clarkb: hrm, why does python-base build stuff? | 17:32 |
| corvus | i thought the idea was to only install previously built things (from python-builder) into python-base | 17:32 |
| clarkb | that I haven't figured out yet. I'm guessing either we don't copy the arm wheels properly between the builder and base side or we've got the wrong wheel type somehow even though the builder side grabbed them | 17:32 |
| clarkb | hrm I wonder if it is related to the pip.conf? on the x86 side it says Stored in directory: /output/wheels/wheels/5f/09/cf/2b1aa8371c071fa89518ac0bbda1b8cca4e65b6e2538af4192 for the wheels it is building but I don't see that for arm side | 17:34 |
| clarkb | oh wait maybe it does hrm that makes it even more confusing | 17:35 |
| clarkb | Created wheel for ruamel.yaml.clib: filename=ruamel.yaml.clib-0.2.0-cp37-cp37m-linux_aarch64.whl size=137954 sha256=42e6e1cb1e9ebd5d809244c7b20134212c95f22f096b540e5a18818d48a1202d and Stored in directory: /output/wheels/wheels/fb/6d/cd/41c880c9ec9a01bee4c222b8bda87ea703b35ae591b9c91975 | 17:36 |
| corvus | clarkb: can you link to that? i can't find that line | 17:36 |
| clarkb | corvus: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c80/741973/3/check/nodepool-build-image/c80bc21/job-output.txt that file (buildest hasn't reported yet) | 17:36 |
| corvus | clarkb: thx, i was looking at wrong build :/ | 17:37 |
| clarkb | corvus: reading the log around where it failed it is finding the openshift wheel we built and cached I think | 17:39 |
| clarkb | as well as many wheels that were arch independent from pypi that we cached | 17:39 |
| clarkb | I wonder if it is something about that wheel in particular that it doesn't like for the current platform | 17:39 |
| clarkb | I've got to pop out for a bike ride now otherwise it will be surface of the sun temps when I do it, but this is a curious behavior | 17:39 |
| corvus | clarkb: i agree that it looks like the builder was happy with the downloaded version of cryptography, that wheels were copied over to the base, that the base was happy with the openshift wheel it built, but thought it needed to build cryptography | 18:11 |
| corvus | clarkb: oh -- i have an idea | 18:11 |
| corvus | clarkb: maybe it doesn't copy the wheel into the wheel cache if it downloads it; it only ends up there if it builds? so maybe we need to add the pip.conf to the base image too? | 18:12 |
| corvus | clarkb: i'll push up a new ps with that | 18:12 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 18:13 |
| fungi | an alternative might be if you could bindmount a shared wheel cache into each layer | 18:14 |
| fungi | er, pip cache i mean | 18:14 |
| fungi | (pip caches things it downloads) | 18:14 |
| corvus | fungi: we could copy the cache from the builder | 18:35 |
| *** harrymichal has joined #zuul | 18:41 | |
| *** armstrongs has quit IRC | 18:49 | |
| *** armstrongs4 has quit IRC | 18:49 | |
| *** bhavikdbavishi has quit IRC | 18:51 | |
| *** tosky has quit IRC | 18:57 | |
| corvus | i'm going to create a 13.x branch in zuul so we can make the 3.x point release | 19:33 |
| corvus | i don't think we need to do the same for nodepool? i think we can probably just tag master as the next 3.x in nodepool | 19:33 |
| corvus | i'll name it stable/3.x | 19:34 |
| corvus | and use 3.19.0 as the branch point | 19:35 |
| fungi | that seems reasonable, nodepool doesn't have breaking changes on master | 19:35 |
| fungi | also i assume (from subsequent comments) you meand a 3.x branch not 13.x | 19:37 |
| fungi | er, meant | 19:37 |
| corvus | yep | 19:37 |
| fungi | cool, just making sure | 19:37 |
| corvus | remote: https://review.opendev.org/742013 Update .gitreview for 3.x branch | 19:37 |
| corvus | (the bot doesn't announce changes on non-master branches for zuul) | 19:38 |
| corvus | remote: https://review.opendev.org/742015 Require kazoo 2.8.0 | 19:39 |
| *** vishalmanchanda has quit IRC | 19:41 | |
| clarkb | corvus: oh interesting idea | 19:41 |
| corvus | clarkb: oh i failed at that change, sorry | 19:42 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 19:45 |
| *** kmalloc has quit IRC | 20:03 | |
| clarkb | zuulians https://review.opendev.org/741820 is a small change to our ensure-devstack role to help run it in a more production like manner. Nodepool seems to show it working https://review.opendev.org/741821 | 20:22 |
| clarkb | corvus: your change to also set the mirror in base seems to have failed with the same error | 20:24 |
| clarkb | maybe we can run it with pip -vvv | 20:24 |
| clarkb | just thinking that may give us a clue as to why pip is ignoring the cached wheels we copied in | 20:24 |
| corvus | clarkb: i'm confused -- current error still looks like i don't know how to bash | 20:25 |
| corvus | clarkb: https://zuul.opendev.org/t/zuul/build/b1f5c34ac0404239a4d1858434ea079a/log/job-output.txt#5261 ? | 20:26 |
| clarkb | corvus: https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3ce/741973/5/check/nodepool-build-image/3ce6cb2/ | 20:26 |
| clarkb | I'm looking at the jobs before the buildset reports | 20:27 |
| corvus | clarkb: are we sure that pip.conf was installed? | 20:27 |
| corvus | i guess we should have put an echo in there | 20:28 |
| clarkb | oh it may not be | 20:28 |
| corvus | i'm not sure we'd see one way or the other | 20:28 |
| clarkb | its pip.conf.arm64 not pip.conf in /output | 20:29 |
| clarkb | the -f is failing as a result I think | 20:29 |
| corvus | ok, i'll fix once more | 20:29 |
| clarkb | need to update the cp too | 20:29 |
| *** y2kenny has joined #zuul | 20:30 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Test nodepool build with opendev wheel cache https://review.opendev.org/741973 | 20:30 |
| corvus | clarkb: how's that ? | 20:30 |
| clarkb | ya that looks right | 20:31 |
| openstackgerrit | Merged zuul/zuul-jobs master: Enable tls-proxy in ensure-devstack https://review.opendev.org/741820 | 20:49 |
| avass | the s3 log upload role should be ready: https://review.opendev.org/#/c/741809/ | 20:58 |
| corvus | avass: is the python module copied from gcs? | 20:59 |
| corvus | (or swift?) | 21:00 |
| avass | corvus: should be gcs, but I noticed I might have grabbed pieces from both | 21:01 |
| avass | do they differ a lot? | 21:02 |
| corvus | nope | 21:02 |
| corvus | they should be pretty close, and identical if you strip out the cloud-specific parts | 21:02 |
| corvus | just gcs is a simpler starting point | 21:02 |
| avass | I think the python module could be swift and added the path where gcs adds 'index.html' to the index file | 21:02 |
| *** y2kenny has quit IRC | 21:02 | |
| corvus | do you think we should try symlinking? | 21:03 |
| avass | that or tobiash idea of trying to combine them into one role | 21:03 |
| avass | I don't know how zuul handles symlinking if a project imports only one of the roles | 21:03 |
| corvus | oh, that's an interesting idea | 21:03 |
| corvus | maybe we should try to start with that | 21:04 |
| avass | corvus: yeah, I can take a look at that later. Maybe sticking to zuul_log_container instead of zuul_log_bucket and more generic names is a good idea then :) | 21:15 |
| avass | I'm planning on adding a role for azure as well. it should be possible to test that in ci since microsoft has supplied an azure storage emulator image: https://hub.docker.com/r/microsoft/azure-storage-emulator/ | 21:16 |
| tobiash | corvus: are you aware of https://storyboard.openstack.org/#!/story/2007897 ? I think that might indicate a misunderstanding of gerritci users. | 21:17 |
| avass | oh, and I just found https://github.com/fsouza/fake-gcs-server for testing gcs :) | 21:18 |
| corvus | tobiash: no i haven't seen that, thanks | 21:23 |
| clarkb | corvus: latest failure in the nodepool image build was due to a ppa error before it got to pypi things. I think we need to recheck it once we can | 21:40 |
| clarkb | I need to pop out for a few now but will be back in a bit to continue chekcing on that | 21:40 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Test nodepool build with opendev wheel cache. https://review.opendev.org/741973 | 21:42 |
| corvus | clarkb: i just edited commit message to speed that up | 21:42 |
| *** holser has joined #zuul | 21:48 | |
| *** holser_ has quit IRC | 21:50 | |
| clarkb | corvus: I think it may be working now | 22:13 |
| clarkb | so that was it, the cached wheels incorporate info about their origin | 22:14 |
| clarkb | https://zuul.opendev.org/t/zuul/stream/a67e0f3763ee452eb83207bfce7eeda0?logfile=console.log for the live stream | 22:14 |
| fungi | ahh, yep, pip's download cache is keyed by url | 22:19 |
| fungi | i forgot about that | 22:20 |
| clarkb | our assemble script ends up installing things twice | 22:30 |
| clarkb | I think because we install all the wheels first then the software itself, and when we install the software it uninstalls things then installs for some reason? That may be something we can optimize by streamlining, but I have another very early morning tomorrow so I'll look at that closer tomorrow | 22:30 |
| clarkb | ok the ppa thing happened again and it is actually after the installs. I got confused because its nodepool-builder not python-builder but I read it as python-builder and got the relative phase step wrong | 22:33 |
| clarkb | I'm guessing this is related to trying to install the arm package on debian via our ppa, it works installing the x86_64 package via our ppa on debian but maybe something about debian? | 22:33 |
| clarkb | ianw: ^ maybe that is familiar to you? | 22:34 |
| clarkb | ianw: https://63964e6d65ae1ec0ac34-67674e607a46c69293c37e4affbebdc2.ssl.cf2.rackcdn.com/741973/7/check/nodepool-build-image/a67e0f3/ the ppa vhd-util failure there | 22:34 |
| ianw | umm, interesting, no that's not familiar but possible, i will have a look | 22:35 |
| ianw | 2020-07-20 22:32:17.102743 | ubuntu-bionic | #19 37.57 E: Unable to locate package vhd-util | 22:36 |
| clarkb | ya I checked earlier and it seemed we had arm64 packages | 22:36 |
| clarkb | but maybe we need to select the repo more specifically? | 22:36 |
| clarkb | like maybe the ppa add is adding x86? | 22:37 |
| ianw | hrm, maybe i did add arm64 builds for it, not sure i remember doing that though. however we don't need that for arm64, because we're not building vhd images anyway | 22:37 |
| ianw | we should probably put it behind a if | 22:37 |
| ianw | yeah, the arm64 has a "X" against it, it didn't build. maybe i did try (dates are 2019-10 ... seems like ancient history!!!!) | 22:38 |
| clarkb | generally I think it would be good for nodepool's arm images to support vhd if that is possible | 22:39 |
| clarkb | since other users may have it running against a vhd cloud. Though if that isn't practical I can see it being a known limitation | 22:39 |
| clarkb | eg document that the arm image can't build vhd images and live with it | 22:39 |
| ianw | so my notes on 2019-10-16 say i was updating vhdutil for bionic | 22:41 |
| ianw | so that would have been when we were initially looking at cutting over from the older hosts | 22:41 |
| ianw | i must have ticked the arm64 box and it tried when i uploaded the bionic builds, but i have no record of me actually trying to fix it | 22:42 |
| ianw | Missing build dependencies: gcc-multilib | 22:42 |
| ianw | i take that back actually, it's even older | 22:43 |
| ianw | Finished on 2018-02-20 | 22:43 |
| ianw | i'll take a look today, see what's up | 22:44 |
| clarkb | thanks! feel free to recheck that change ir update it if necessary | 22:45 |
| *** holser has quit IRC | 23:12 | |
| *** sgw1 has quit IRC | 23:16 | |
| *** harrymichal has quit IRC | 23:24 | |
| *** sgw1 has joined #zuul | 23:25 | |
| *** holser has joined #zuul | 23:27 | |
| *** rlandy|ruck has quit IRC | 23:33 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!