| @iwienand:matrix.org | 2022-01-26 22:56:25.404274 | TASK [Produce minikube log] | 00:16 |
|---|---|---|
| 2022-01-26 22:56:40.783614 | ubuntu-focal | E0126 22:56:40.783464 10451 logs.go:190] command /bin/bash -c "docker logs --tail 60 b21d05bbe185" failed with error: /bin/bash -c "docker logs --tail 60 b21d05bbe185": exit status 1 | ||
| @iwienand:matrix.org | is this a known issue? (https://5b53b22152836ac2a1ba-584a141890d4c711c7adef07d7bdf32d.ssl.cf1.rackcdn.com/826537/1/gate/nodepool-functional-k8s/329dce6/job-output.txt_ | 00:17 |
| @clarkb:matrix.org | corvus: left some questions/suggestions/comments on https://review.opendev.org/c/zuul/zuul/+/824482 if you have time to take a look at that | 00:19 |
| @jim:acmegating.com | ianw: can you link to the build instead? | 00:19 |
| @iwienand:matrix.org | corvus: that was from https://zuul.opendev.org/t/zuul/build/329dce6f55014044a93898182ee18aac | 00:20 |
| @clarkb:matrix.org | ianw: not known to me but appears the coredns container didn't start or get networking | 00:21 |
| @jim:acmegating.com | ianw: the coredns log yeah what clarkb said: https://zuul.opendev.org/t/zuul/build/329dce6f55014044a93898182ee18aac/log/minikube.txt | 00:21 |
| @jim:acmegating.com | also unknown to me | 00:22 |
| @iwienand:matrix.org | [FATAL] plugin/loop: Loop (127.0.0.1:59383 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 1386818609659094942.7024570961074799855." | 00:28 |
| @iwienand:matrix.org | that seems DNS-y | 00:29 |
| @iwienand:matrix.org | 5791 exec_runner.go:52] Run: /bin/bash -c "sudo /var/lib/minikube/binaries/v1.21.2/kubectl --kubeconfig=/var/lib/minikube/kubeconfig -n kube-system get configmap coredns -o yaml | sed '/^ forward . \/etc\/resolv.conf.*/i \ hosts {\n 127.0.0.1 host.minikube.internal\n fallthrough\n }' | sudo /var/lib/minikube/binaries/v1.21.2/kubectl --kubeconfig=/var/lib/minikube/kubeconfig replace -f -" | 00:34 |
| @iwienand:matrix.org | seems to be where forwarding would be setup | 00:34 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 824482: Add "zuul delete-pipeline-state" command https://review.opendev.org/c/zuul/zuul/+/824482 | 00:35 | |
| @jim:acmegating.com | Clark: ^ done thx :) | 00:35 |
| @jim:acmegating.com | oh forgot the docs fix | 00:35 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 824482: Add "zuul delete-pipeline-state" command https://review.opendev.org/c/zuul/zuul/+/824482 | 00:36 | |
| @josefwells:matrix.org | I think I found my issue with SQL connections. I was using the old "connections" specifications. When I switched to [database] things got better. Following this: https://zuul-ci.org/docs/zuul/latest/configuration.html#database I looked at SqlAlchemy, which indicated I needed to encode my password's special characters, however executor seems to barf on URL encoded special chars, per: https://docs.sqlalchemy.org/en/14/core/engines.html#database-urls | 00:47 |
| @jim:acmegating.com | josefwells: that's fixed in 4.9.0 | 00:59 |
| @josefwells:matrix.org | Oh, ok, I'll give it a go | 01:00 |
| @josefwells:matrix.org | I was worried about the not reading keys from the filesystem, I think that was around that release too, but I'll just tackle it | 01:00 |
| @josefwells:matrix.org | also, last curious thing, in the docker-quickstart, The dburi is: dburi=mysql+pymysql://zuul:%(ZUUL_MYSQL_PASSWORD)s@mysql/zuul What is the deal with the "s" after %(ZUUL_MYSQL_PASSWORD) ? | 01:02 |
| @jim:acmegating.com | josefwells: 2nd paragraph here: https://zuul-ci.org/docs/zuul/latest/configuration.html | 01:04 |
| @jim:acmegating.com | josefwells: https://docs.python.org/3/library/configparser.html#interpolation-of-values is the underlying implementation | 01:05 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 826537: Stop running siblings jobs and pin SDK https://review.opendev.org/c/zuul/nodepool/+/826537 | 01:17 | |
| @josefwells:matrix.org | corvus: that helps a bunch, I did not previously understand that at all | 01:17 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 826549: Annotate variable interpolation in the tutorial zuul.conf https://review.opendev.org/c/zuul/zuul/+/826549 | 01:20 | |
| @jim:acmegating.com | josefwells: that's good feedback, thanks. ^ :) | 01:20 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 817640: Add some pipeline processing stats https://review.opendev.org/c/zuul/zuul/+/817640 | 01:45 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 826572: nodepool-zuul-functional: install into a venv https://review.opendev.org/c/zuul/nodepool/+/826572 | 02:01 | |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 826572: nodepool-zuul-functional: install into a venv https://review.opendev.org/c/zuul/nodepool/+/826572 | 02:41 | |
| @stevebak:matrix.org | https://review.opendev.org/c/zuul/nodepool/+/826538 is green btw | 03:03 |
| @clarkb:matrix.org | I can reapprove it in a bit. Need to get to a real computer | 03:26 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 826572: [wip] nodepool-zuul-functional: install into a venv https://review.opendev.org/c/zuul/nodepool/+/826572 | 03:35 | |
| @clarkb:matrix.org | I was looking at why our zuul images are so big and it seems a decent portion of the reason is .pyc files for all the python things | 04:12 |
| @clarkb:matrix.org | I wonder if it makes sense to trim those out and let python generate them when it runs and needs them. | 04:12 |
| @clarkb:matrix.org | Anyway its late. THought I'd mention it in case anyone had noticed this before or worked around it in other contexts | 04:13 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 826572: [wip] nodepool-zuul-functional: install into a venv https://review.opendev.org/c/zuul/nodepool/+/826572 | 04:26 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 826579: debug config check https://review.opendev.org/c/zuul/zuul/+/826579 | 04:31 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 826579: debug config check https://review.opendev.org/c/zuul/zuul/+/826579 | 04:55 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 826585: debug https://review.opendev.org/c/zuul/zuul/+/826585 | 05:39 | |
| @iwienand:matrix.org | https://zuul.opendev.org/t/zuul/build/ff001ba71a104f5c8da5180966cc38d8/log/minikube.txt hit what appears to be the minikube dns "loop" thing again | 05:59 |
| @gobi_g:matrix.org | Hi, | 07:06 |
| What happens when new commit added to the MR while zuul job running for the that MR? | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 826538: Remove openstacksdk from siblings job https://review.opendev.org/c/zuul/nodepool/+/826538 | 10:14 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 10:17 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 10:29 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 10:46 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 826609: Refresh cached branches in timer driver https://review.opendev.org/c/zuul/zuul/+/826609 | 10:48 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 11:26 | |
| - [zuul/zuul] 823587: Add some ZK debug scripts https://review.opendev.org/c/zuul/zuul/+/823587 | ||
| - [zuul/zuul] 824482: Add "zuul delete-pipeline-state" command https://review.opendev.org/c/zuul/zuul/+/824482 | ||
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 11:29 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 825014: Remove "sql connection" backwards compatability for database https://review.opendev.org/c/zuul/zuul/+/825014 | 11:44 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826648: DNM test only https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 12:22 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 12:55 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 12:57 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826648: pin oauthlib version for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 13:10 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826648: pin oauthlib version for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 13:32 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed on behalf of Alfredo Moralejo: [zuul/zuul-jobs] 826648: pin oauthlib version for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 13:32 | |
| -@gerrit:opendev.org- Dr. Jens Harbott proposed on behalf of Alfredo Moralejo: [zuul/zuul-jobs] 826648: pin oauthlib version for python2.7 https://review.opendev.org/c/zuul/zuul-jobs/+/826648 | 13:35 | |
| -@gerrit:opendev.org- Alfredo Moralejo proposed: [zuul/zuul-jobs] 826603: Add CentOS Stream 9 to configure-mirrors role https://review.opendev.org/c/zuul/zuul-jobs/+/826603 | 14:25 | |
| @josefwells:matrix.org | Clark: re: pyc files. I generally use the ENV var: PYTHONDONTWRITEBYTECODE, seems reasonable when building a container image. For ci we use "dive" which has configurable targets for wasted space in containers as well. | 14:58 |
| @fungicide:matrix.org | yeah, the tradeoff is between transferring larger images with bytecode precompiled, vs everyone's python compiling it at start. slower downloads vs slower startup, there's probably no one right answer | 16:22 |
| @fungicide:matrix.org | in debian, for example, python bytecode is precompiled at package installation so that startup doesn't incur the overhead, and in order to make it available to all users on the system. in a container context, that may still be relevant if you're running as non-root inside the container | 16:23 |
| @fungicide:matrix.org | the user invoking python may not have anywhere to persist the bytecode, so instead of incuring the startup overhead for it the first time you run a script, you get to enjoy it on every single invocation | 16:25 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 824077: Add a zk-shell debug script https://review.opendev.org/c/zuul/zuul/+/824077 | 16:25 | |
| @jim:acmegating.com | Clark: tobiash ^ updated zk-shell to better handle missing nodes | 16:26 |
| @clarkb:matrix.org | fungi: that is a good point about not being able to write the pyc files as a different user. I think zuul owns the ansible installations which are the bulk of it. But definitely somethign to check before making any changes | 16:46 |
| @jim:acmegating.com | Clark: we install ansible during the build; it's owned as root. | 17:04 |
| @jim:acmegating.com | i think keeping .pyc files around for speeding up invocations of ansible is desirable | 17:05 |
| @clarkb:matrix.org | fast ansible is definitely desirebale and if we can't write pyc on the fly then having them prebuilt seems appropriate | 17:05 |
| @jim:acmegating.com | maybe there's some kind of way to do a lookaside cache for pyc files that could be added as a volume. but i have no idea. might be a fun rabbit hole. :) | 17:06 |
| @fungicide:matrix.org | shortcut to the bottom of that rabbit hole: https://docs.python.org/3/using/cmdline.html#envvar-PYTHONPYCACHEPREFIX | 18:02 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 826357: Add QuotaSupport to drivers that don't have it https://review.opendev.org/c/zuul/nodepool/+/826357 | 18:11 | |
| @jim:acmegating.com | Clark: ^ so setting that as an ENV in the dockerfile and pointing it at a VOLUME might get what you're after | 18:13 |
| @jim:acmegating.com | re fungi 's link... i'm still bad at hitting the reply button | 18:13 |
| @clarkb:matrix.org | corvus: ya but I'm not sure we should expect people to have to do that to deploy zuul | 18:13 |
| @jim:acmegating.com | Clark: you can do that in the Dockerfile | 18:13 |
| @jim:acmegating.com | no deployment changes needed | 18:14 |
| @clarkb:matrix.org | Oh I see | 18:14 |
| @clarkb:matrix.org | I think we may not want to do it with a VOLUME though because if the code chagnes you need new pyc files | 18:14 |
| @clarkb:matrix.org | and a VOLUME will persist right? | 18:14 |
| @tobias.henkel:matrix.org | do we have any idea about how the startup times for ansible differ with and without the pyc files? | 18:14 |
| @jim:acmegating.com | yeah, but it's writable, so should end up getting overwritten? | 18:14 |
| @clarkb:matrix.org | tobiash: it should only affect the first job that runs | 18:15 |
| @tobias.henkel:matrix.org | Clark: but only if the pyc files can be written right? | 18:15 |
| @tobias.henkel:matrix.org | if the pyc files cannot be written it would affect every job? | 18:15 |
| @jim:acmegating.com | thus the suggestion to have a docker volume where they can be written | 18:16 |
| @clarkb:matrix.org | tobiash: yes we would need to make the pyc writeable via PYTHONPYCACHEPREFIX | 18:16 |
| @jim:acmegating.com | i'm not convinced this is a win; just an interesting thing worth exploring. but if we assume that we're going to have the pyc files on disk eventually anyway, then it's really just a network bandwidth savings, and that isn't a huge deal these days | 18:18 |
| @clarkb:matrix.org | ya this was mostly me noticing the image is 1.3GB last night and wondering why. Turns out ansible is like half that and largely due to pyc files | 18:18 |
| @jim:acmegating.com | many ansibles, even :) | 18:19 |
| @clarkb:matrix.org | But ansible is slow enough with pycs and we shouldn't impact that :) | 18:19 |
| @jim:acmegating.com | we can make the image look smaller and fetch faster, and then surprise it's 1.3G of disk space used anyway :) | 18:20 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Dong Zhang: [zuul/zuul] 823242: Display overall duration in buidset page in zuul web https://review.opendev.org/c/zuul/zuul/+/823242 | 18:38 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 824077: Add a zk-shell debug script https://review.opendev.org/c/zuul/zuul/+/824077 | 19:02 | |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 826357: Add QuotaSupport to drivers that don't have it https://review.opendev.org/c/zuul/nodepool/+/826357 | 20:01 | |
| @jim:acmegating.com | nice: gating in action: https://review.opendev.org/825022 just legit failed tests after a change merged ahead of it | 20:15 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 825022: Add a model API version https://review.opendev.org/c/zuul/zuul/+/825022 | 20:19 | |
| @jim:acmegating.com | that should fix it | 20:19 |
| @clarkb:matrix.org | corvus: I'm having a hard time parsing what the diff is shwoing me. I see the delete pipeline state and gearmanectomy removals in client.py. But nothing related to the model change? Gerrit says client.py is the only file that chagned between patchsets | 20:25 |
| @clarkb:matrix.org | did it just need a rebase to help out git merge? | 20:26 |
| @jim:acmegating.com | Clark: no there was a small change, but unfortunately the rebase hides it; here's the diff: https://paste.opendev.org/show/812411/ | 20:32 |
| @jim:acmegating.com | oh, since client.py wasn't in the old patchset, you can actually just look at BASE..4 in gerrit and see it | 20:34 |
| @clarkb:matrix.org | ah thanks for the hint | 20:37 |
| @clarkb:matrix.org | and that totally makes sense now | 20:37 |
| @clarkb:matrix.org | +2 from me | 20:38 |
| @jim:acmegating.com | i think we can go ahead and re+w that | 20:39 |
| @clarkb:matrix.org | WFM. It is a smal change directly related to one of the parent chagnes that just landed | 20:40 |
| @iwienand:matrix.org | is the nodepool-zuul-functional job worth saving? i spent a bit of time getting things installing and a tls zookeeper working, but it still fails in the actual tests now | 20:56 |
| @iwienand:matrix.org | https://fb73578e32dc0eaefa59-f67dc2afe768675df5d36f2071b73ef5.ssl.cf2.rackcdn.com/826572/4/check/nodepool-zuul-functional/37fa4e1/testr_results.html | 20:56 |
| @iwienand:matrix.org | is the unit test failures now | 20:56 |
| @iwienand:matrix.org | it's been broken for a long time, pulling it out might be the other option if it's past it's use-by date | 20:57 |
| @fungicide:matrix.org | ianw: should i abandon https://review.opendev.org/790826 in favor of yours? and yeah, i suspect the tests themselves are bitrotten because the job hasn't run successfully for so long | 20:58 |
| @iwienand:matrix.org | oh, heh, i didn't scroll back that far in changes :) i got the zuul side working (i think) by adding the ZK flags like https://review.opendev.org/c/zuul/nodepool/+/826572/4/.zuul.yaml | 21:01 |
| @iwienand:matrix.org | it is a bit of a weird one in that nodepool sets up the job and the ZK environment, then runs tox out of a zuul checkout | 21:02 |
| @fungicide:matrix.org | yeah, as best i was able to tell it broke a year ago when the zk tls work landed but the job was already non-voting because it had some inherent instability, so nobody really noticed it went from randomly broken to entirely broken | 21:05 |
| @fungicide:matrix.org | i started trying to fix it around the middle of the year, but got a bit out of my depth and ended up bancburnering the fix | 21:06 |
| @jim:acmegating.com | i feel it's probably mostly covered by quick-start... | 21:06 |
| @fungicide:matrix.org | * i started trying to fix it around the middle of the year, but got a bit out of my depth and ended up back-burnering the fix | 21:06 |
| @iwienand:matrix.org | fungi: haha i was like "oh bancumburnering", that must be a new word the kids are using :) | 21:07 |
| @fungicide:matrix.org | nope, just my terribad typing | 21:08 |
| @iwienand:matrix.org | in that case i might propose to just take it out, rather than spend more time trying to fix it when it's sort of oddly bespoke | 21:08 |
| @fungicide:matrix.org | i'm good with that | 21:08 |
| @jim:acmegating.com | i assume fungi just mistyped beancurding | 21:08 |
| -@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 826609: Refresh cached branches in timer driver https://review.opendev.org/c/zuul/zuul/+/826609 | 21:09 | |
| @jim:acmegating.com | there's a lot that can be ripped out if we do that; playbooks and unit tests, etc | 21:09 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/zuul] 826772: nodepool-zuul-functional: remove https://review.opendev.org/c/zuul/zuul/+/826772 | 21:18 | |
| @clarkb:matrix.org | ya nodepool tests the booting of things with its functional cloud provider jobs and then quick start checks the node request interface | 21:21 |
| -@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] 826773: nodepool-zuul-functional: remove https://review.opendev.org/c/zuul/nodepool/+/826773 | 21:21 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 826400: Add zuul-scheduler tenant-reconfigure https://review.opendev.org/c/zuul/zuul/+/826400 | 21:25 | |
| @jim:acmegating.com | fixed a merge conflict on that | 21:25 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 825022: Add a model API version https://review.opendev.org/c/zuul/zuul/+/825022 | 22:46 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Benjamin Schanzel: [zuul/nodepool] 825426: Add tests for exporting tenant limit stats https://review.opendev.org/c/zuul/nodepool/+/825426 | 23:06 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!