Tuesday, 2021-08-31

@fungicide:matrix.orgcorvus: right, that's where the default value is, and a big part of the reason why i added 806621 to begin with00:39
@fungicide:matrix.orgbut the problem with the current failures on 806612 seems to be that it's being passed either a different tox.ini or a different env name than is intended00:40
@jim:acmegating.comfungi: no it's running tox showconfig with -vv which causes a [tox] section to be emitted which changes the code path in get_envlist00:41
@jim:acmegating.comfungi: when you added tox_extra_args to the showconfig line, it added the "-vv" from the default value00:41
@fungicide:matrix.orgoh, does --showconfig not normally include a [tox] section? i didn't notice that, testing again00:42
@fungicide:matrix.orgi'm aware the default is getting added, i mean i included an entire additional change to deal with the non-ini output which -vv seems to want to emit ahead of the config, i just didn't notice it also alters the outputted config sections00:43
@fungicide:matrix.orgcorvus: so, if i `tox --showconfig` without adding -vv, i still get a [tox] section00:44
@jim:acmegating.comfungi: try `tox --showconfig -epy39` vs `tox --showconfig -epy39 -vv`00:44
@jim:acmegating.comthose are the commands that ansible runs00:44
@fungicide:matrix.orgoh, yep okay, i didn't notice that -vv causes it not to filter when specifying an envlist00:45
@jim:acmegating.comit does filter, it just also include [tox]00:46
@jim:acmegating.com(note that [linters] is still not in the output)00:46
@fungicide:matrix.orgit also seems to cause it to add a [tox:versions] section00:47
@jim:acmegating.combut since there's a [tox] section, the get_envlist method in the siblings helper uses tox.envlist_default instead of the implied list of section headers00:47
@fungicide:matrix.orgoh, thanks! i'll see if i can make that bit smarter00:48
@fungicide:matrix.orgmmm, yeah the ``if 'tox' in tox_config.sections()`` is being used to determine whether the output was filtered by envlist00:50
@fungicide:matrix.orglooks like i can check if tox.option has verbose_level != 0 in its Namespace object keyargs and env != None maybe00:57
@fungicide:matrix.orgeither that or parsing tox.args which might be more fiddly00:57
@fungicide:matrix.orgi'll sleep on it. thanks for the help corvus !00:58
-@gerrit:opendev.org- Ian Wienand proposed: [zuul/nodepool] [dnm] testing ubuntu builds on bullseye https://review.opendev.org/c/zuul/nodepool/+/80665901:50
@iwienand:matrix.orgi think that on bullseye, something part of debootstrap is unmounting /proc.  this seems really familiar but i haven't pinpointed it yet ...04:43
@iwienand:matrix.orgaahhh, i knew it : https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=91965904:48
@iwienand:matrix.org1.0.123~salsa2627+1focal ... so that sorts before 1.0.123 (which bullseye has)05:05
@iwienand:matrix.org2021-08-31 05:58:17.788 | Build completed successfully05:58
@iwienand:matrix.orgi've rebuilt the ppa as debootstrap - 1.0.123+ppa1.  so in short, i think that bullseye based nodepool-builder is probably looking good 05:59
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] Reporter: add info on job retries https://review.opendev.org/c/zuul/zuul/+/80672210:18
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] Reporter: add info on job retries https://review.opendev.org/c/zuul/zuul/+/80672213:41
-@gerrit:opendev.org- Simon Westphahl proposed:14:09
- [zuul/zuul] Add source interface for setting change attributes https://review.opendev.org/c/zuul/zuul/+/805836
- [zuul/zuul] Reference change dependencies by key https://review.opendev.org/c/zuul/zuul/+/805844
- [zuul/zuul] Implement ABC for caching changes in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805835
- [zuul/zuul] Cache Gerrit refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805837
- [zuul/zuul] Cache Github refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805838
- [zuul/zuul] Cache Pagure refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806556
- [zuul/zuul] Cache Gitlab refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806557
- [zuul/zuul] Cache Git refs (driver) in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806755
- [zuul/zuul] wip: Periodically prune connection caches https://review.opendev.org/c/zuul/zuul/+/806756
-@gerrit:opendev.org- Simon Westphahl proposed:14:19
- [zuul/zuul] Implement ABC for caching changes in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805835
- [zuul/zuul] Cache Gerrit refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805837
- [zuul/zuul] Cache Github refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/805838
- [zuul/zuul] Cache Pagure refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806556
- [zuul/zuul] Cache Gitlab refs in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806557
- [zuul/zuul] Cache Git refs (driver) in Zookeeper https://review.opendev.org/c/zuul/zuul/+/806755
@jpew:matrix.orgI'm trying to use the kubernetes nodepool driver and it appears to spin up the pods correctly to run jobs, but they fail with the error: 14:50
Failed to connect to the host via ssh: ssh: Could not resolve hostname
pod-ubuntu-focal: No address associated with hostname
@clarkb:matrix.orgjpew: I think that nodepool is supposed to tell zuul to talk to those "instances" via kubectl?15:01
@clarkb:matrix.orgtristanC: does that require any special config? I would've expected that would be done for you when using the kubernetes driver in nodepool15:02
@jim:acmegating.comno special config required15:02
@jim:acmegating.comit might be useful to know what emitted that error15:02
@clarkb:matrix.orghttps://opendev.org/zuul/nodepool/src/branch/master/nodepool/driver/kubernetes/handler.py#L47 yup that seems to confirm it is automatic15:03
@clarkb:matrix.orgoh ya I guess a playbook may have assumed ssh15:03
@avass:vassast.orgI'm guessing it's the synchronize-repo role failing 15:05
@jpew:matrix.orgI think its failing gathering facts15:06
-@gerrit:opendev.org- Jeremy Stanley proposed:16:15
- [zuul/zuul-jobs] Add tox_config_file rolevar to tox https://review.opendev.org/c/zuul/zuul-jobs/+/806613
- [zuul/zuul-jobs] Support verbose showconfig in tox siblings https://review.opendev.org/c/zuul/zuul-jobs/+/806621
- [zuul/zuul-jobs] Include tox_extra_args in tox siblings tasks https://review.opendev.org/c/zuul/zuul-jobs/+/806612
@clarkb:matrix.orgfungi: zuul is happy with https://review.opendev.org/c/zuul/zuul-jobs/+/806621 but software factory ci isnt'16:39
@clarkb:matrix.orgfungi: left a note on where I think the issue is but I'm not entirely certain of what the fix is16:48
@clarkb:matrix.orgMaybe we ignore any cases where the len(sections.split(':')) <= 1?16:49
@jim:acmegating.comdo you know why sf.io reports different results?16:51
@jim:acmegating.commaybe because it's using an empty tox.ini?16:52
@clarkb:matrix.orgI don't which is part of why I'm not sure what the correct fix is here16:53
@clarkb:matrix.organd ya maybe their tox.ini example is different in a corner case way like that16:54
@clarkb:matrix.orgI think tox allows you to have arbitrary sections as well for configuring things like linters.16:55
@clarkb:matrix.orgIt may be they have one of those in their example16:55
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Don't treat failed requirement jobs as ready https://review.opendev.org/c/zuul/zuul/+/80678116:59
@jim:acmegating.comClark: well, i assume the tox.ini is literally empty because there is a task named "Setup empty tox.ini"16:59
@jim:acmegating.comthough i don't know if something augments it later.17:00
@clarkb:matrix.orggot it17:02
@clarkb:matrix.orgThough it is interesting we would get a section returned by the for loop in that case. Maybe that is a configparser default behavior to give you a default section for an empty file?17:07
@jim:acmegating.comi also don't know empty it really is17:08
@jim:acmegating.comi'm totally just going on the task name17:08
@clarkb:matrix.orgoh ya I guess emtpy here could be a section with no content or similar17:08
@jim:acmegating.comincidentally, all the zuul unit tests are failing with AttributeError: module 'alembic' has no attribute 'migration'17:09
@clarkb:matrix.orgYou need a fairly up to date sqlalchemy for that iirc17:10
@clarkb:matrix.orgif that is local maybe you need to rebuild your tox env (or just update sqlalchemy?)17:10
@jim:acmegating.comwas there an alembic release to go along with the sqlalchemy change?17:10
@jim:acmegating.comClark: not local17:11
@jim:acmegating.comhere's what the jobs are currently installing: SQLAlchemy==1.4.23 alembic==1.7.117:12
@clarkb:matrix.orgAlembic 1.7.1 (and 1.7.0) are both from yesterday17:12
@clarkb:matrix.orgI wonder if they inadverdently broke alembic with new sqlalchemy?17:12
@jim:acmegating.com1.6.5 is the last success17:12
@clarkb:matrix.orgheh there is a different alembic also on github with similar but different version numbers of maximum confusion17:15
-@gerrit:opendev.org- Jeremy Stanley proposed:17:15
- [zuul/zuul-jobs] Support verbose showconfig in tox siblings https://review.opendev.org/c/zuul/zuul-jobs/+/806621
- [zuul/zuul-jobs] Include tox_extra_args in tox siblings tasks https://review.opendev.org/c/zuul/zuul-jobs/+/806612
@jim:acmegating.comi don't see anything in https://alembic.sqlalchemy.org/en/latest/changelog.html that makes this look like an intentional change where we should be expected to change code17:16
@jim:acmegating.com(but there are some import related changes that could (inadvertently?) be affecting us...maybe?)17:16
@clarkb:matrix.orgI'll ping zzzeek in #openstack-oslo on oftc17:17
@clarkb:matrix.orgI agree this doesn't appear intentional17:17
@clarkb:matrix.orgcorvus: can you share a link to a failing job?17:17
@jim:acmegating.comClark: https://zuul.opendev.org/t/zuul/build/177414f0c97f4a75804501ea55db3dc1/logs17:18
@jim:acmegating.comnote that it's the "everything bombs and produces gigabytes of useless output" failure case17:18
@clarkb:matrix.orgok I'll see if I can trim it to relevant info in a paste for zzzeek17:19
@jim:acmegating.com```17:19
2021-08-31 14:29:53.668319 | ubuntu-bionic | Captured traceback:
2021-08-31 14:29:53.668332 | ubuntu-bionic | ~~~~~~~~~~~~~~~~~~~
2021-08-31 14:29:53.668344 | ubuntu-bionic | Traceback (most recent call last):
2021-08-31 14:29:53.668357 | ubuntu-bionic |
2021-08-31 14:29:53.668370 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/tests/base.py", line 4641, in setUp
2021-08-31 14:29:53.668382 | ubuntu-bionic | self.createScheduler()
2021-08-31 14:29:53.668394 | ubuntu-bionic |
2021-08-31 14:29:53.668406 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/tests/base.py", line 4654, in createScheduler
2021-08-31 14:29:53.668417 | ubuntu-bionic | self.scheds.create(
2021-08-31 14:29:53.668429 | ubuntu-bionic |
2021-08-31 14:29:53.668440 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/tests/base.py", line 4350, in create
2021-08-31 14:29:53.668477 | ubuntu-bionic | app = SchedulerTestApp(log, scheduler_config, changes,
2021-08-31 14:29:53.668492 | ubuntu-bionic |
2021-08-31 14:29:53.668505 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/tests/base.py", line 4289, in __init__
2021-08-31 14:29:53.668516 | ubuntu-bionic | self.connections.registerScheduler(self.sched)
2021-08-31 14:29:53.668538 | ubuntu-bionic |
2021-08-31 14:29:53.668557 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/zuul/lib/connections.py", line 78, in registerScheduler
2021-08-31 14:29:53.668570 | ubuntu-bionic | connection.onLoad()
2021-08-31 14:29:53.668583 | ubuntu-bionic |
2021-08-31 14:29:53.668594 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/zuul/driver/sql/sqlconnection.py", line 288, in onLoad
2021-08-31 14:29:53.668606 | ubuntu-bionic | self._migrate()
2021-08-31 14:29:53.668618 | ubuntu-bionic |
2021-08-31 14:29:53.668629 | ubuntu-bionic | File "/home/zuul/src/opendev.org/zuul/zuul/zuul/driver/sql/sqlconnection.py", line 264, in _migrate
2021-08-31 14:29:53.668648 | ubuntu-bionic | context = alembic.migration.MigrationContext.configure(conn)
2021-08-31 14:29:53.668674 | ubuntu-bionic |
2021-08-31 14:29:53.668700 | ubuntu-bionic | AttributeError: module 'alembic' has no attribute 'migration'```
@jim:acmegating.commaybe just need an "import alembic.migration" ?17:20
@clarkb:matrix.orgya looking at it more closely I think it is just circumstance that this has happened recently after landing the sqlalchemy compat update17:20
@clarkb:matrix.orgI suspect this is an unintentional break independent of that work17:21
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Import alembic.migration https://review.opendev.org/c/zuul/zuul/+/80678317:21
@jim:acmegating.comClark: yeah, i haven't dug into it too deeply, but based on release notes they may have tightened up their import structure and we were depending on their internal import paths.  we'll see if that ^ fixes.17:22
@pabelanger:matrix.orghi, we seem to be having some sort of deadlock in our zuul-registry on zuul.a.c.  https://paste.opendev.org/show/808475/ is the traceback17:34
@pabelanger:matrix.orgwhen we stop / start the registry works again17:35
@jpew:matrix.orgOk, I figured out part of my k8s pod problem; I was missing `python` on the build nodes and also a `zuul` user. *now* it's failing to synchronize the source to the workspace directory17:36
@jim:acmegating.comjpew: you may need special base job roles for that17:37
@jpew:matrix.orgDoes someone know how to fix that? Also, is there perhaps an example of a build node k8s Dockerfile I could use as a reference to make sure I have everything I need?17:37
@pabelanger:matrix.orgI am wondering if our SSL certs changed due to certbot running17:39
@pabelanger:matrix.organd somehow confused the registry17:39
@jim:acmegating.comjpew: check out https://zuul-ci.org/docs/zuul-jobs/general-roles.html#role-prepare-workspace-openshift17:39
@jpew:matrix.orgcorvus: Will do. Thanks17:39
@pabelanger:matrix.orgit doesn't appear to be our SSL cert it was issued on Monday, 9 August 2021 at 21:06:41 17:41
@hwangbro:matrix.orgIf I wanted to test source code changes in Zuul and deploy it to another machine, what are the commands to build a new (wheel?) package from scratch17:49
@jpew:matrix.orgcorvus: Success! Thanks!17:49
@jpew:matrix.orgNow I just need to figure out how to do logging17:50
@fungicide:matrix.orgcorvus: Clark: it looks like the error returned by sf.io on the previous revision of 806621 was spotted by opendev's zuul on the child change 806612 (both seem happy now with the latest patchset), so presumably due to testing with a different tox.ini yes17:51
@clarkb:matrix.orgpabelanger: that error is happening when clients communicate to the zuul-registry. Due to a bad diffie hellmman exchange? The first thing I would try is check if openssl s_client also complains. If it does that may give you better info on why it fails17:51
@clarkb:matrix.orghwangbro: I think pip wheel /path/to/zuul will do it17:52
@hwangbro:matrix.orgThanks, I'17:52
@hwangbro:matrix.org * Thanks, I'll try that!17:52
@fungicide:matrix.orghwangbro: for precisely how we build the wheels and sdist tarballs we publish to pypi, here's the command zuul ran for the most recent release: https://zuul.opendev.org/t/zuul/build/aab1f733fa9344e2ae67099aa1a7b351/console17:54
@fungicide:matrix.orghwangbro: more precisely, https://zuul.opendev.org/t/zuul/build/aab1f733fa9344e2ae67099aa1a7b351/console#3/0/3/ubuntu-focal17:54
@fungicide:matrix.orgso it was `python3 setup.py sdist bdist_wheel`17:55
@fungicide:matrix.orgthough the https://pypi.org/project/build/ tool should also work, i've used it to create wheels of similarly organized projects17:57
@fungicide:matrix.orgbuild is supposedly what the python packaging ecosystem is moving to (away from invoking setup.py or calling the wheel module directly)17:58
@fungicide:matrix.orgbased on my preliminary tests, it works fine with pbr-based projects lacking a pyproject.toml, since it falls back to the setuptools backend by default17:59
@hwangbro:matrix.orgI'm noticing that when I build it locally, there's no step involving running yarn, but I see it in the build logs you linked. Is that another dependency?18:05
@hwangbro:matrix.orgI'm building in a WSL Ubuntu environment18:06
@hwangbro:matrix.org * I'm building in a WSL Ubuntu environment for reference18:06
@clarkb:matrix.orgif you need the zuul-web component then yes there is another additional step18:07
@clarkb:matrix.orgHowever, I thought that was automated18:07
@clarkb:matrix.orgaha it relies on yarn being installed18:07
@clarkb:matrix.orghwangbro: https://opendev.org/zuul/zuul/src/branch/master/zuul/_setup_hook.py#L25-L2618:08
@hwangbro:matrix.orgAh there it is. This looks promising, thanks18:09
@clarkb:matrix.orgthat hook will run automatically during the wheel build, but it exits early if yarn isn't present18:09
@hwangbro:matrix.orgThat did the trick, thanks for the help!20:21
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:20:40
- [zuul/zuul] Remove returnNodeSet calls from scheduler https://review.opendev.org/c/zuul/zuul/+/806062
- [zuul/zuul] Remove nodeset from NodeRequest https://review.opendev.org/c/zuul/zuul/+/806063
- [zuul/zuul] Make node requests persistent https://review.opendev.org/c/zuul/zuul/+/806280
- [zuul/zuul] Add node request cache to zk nodepool interface https://review.opendev.org/c/zuul/zuul/+/806639
- [zuul/zuul] Wrap nodepool request completed events with election https://review.opendev.org/c/zuul/zuul/+/806653
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Felix Edel: [zuul/zuul] WIP NodeRequest watches https://review.opendev.org/c/zuul/zuul/+/80496120:40
-@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] Import alembic.migration https://review.opendev.org/c/zuul/zuul/+/80678321:06
@clarkb:matrix.orgianw: looking at https://review.opendev.org/c/zuul/nodepool/+/806312/11/Dockerfile was the ubuntu-keyring package just completely unused?21:21
@iwienand:matrix.org> <@clarkb:matrix.org> ianw: looking at https://review.opendev.org/c/zuul/nodepool/+/806312/11/Dockerfile was the ubuntu-keyring package just completely unused?21:21
sorry yeah i still need to document that change a bit better. i think it may not be verifying installs now
@clarkb:matrix.orghrm, the installs should all be for debian though? and the ubuntu-keyring package would just be for the ubuntu repositories? Maybe our PPA key is chained off of the ubuntu keys somehow?21:22
@iwienand:matrix.orghttps://3173935d1683b1ac754b-b45f7fdceccc06baa60c38e38ca87fcf.ssl.cf2.rackcdn.com/806660/3/check/dib-nodepool-functional-openstack-ubuntu-focal-src/7d445cd/nodepool/builds/test-image-0000000001.log21:22
@clarkb:matrix.orgOh is this actually used by dib on the image build side? not the docker image build itself?21:23
@iwienand:matrix.orgno sorry i didn't mean to paste that :)21:23
@clarkb:matrix.orgah ok and looking at that it seems the VM image builds bootstrap the keyring directly anyway21:24
@iwienand:matrix.orghttps://5132fc4a29e1cfa921ad-21b0bd21f45f83c83b77a87ffa43cecc.ssl.cf5.rackcdn.com/806660/3/check/nodepool-build-image-siblings/be5c505/job-output.txt is the docker image build21:25
@iwienand:matrix.orgi think they must come from the ppa?21:27
@clarkb:matrix.orgWe install the ppa key separately though. We also install the debian-keyring21:27
@clarkb:matrix.orgI think it may be correct to drop the ubuntu-keyring unless there is a specific log entry I'm missing21:27
@iwienand:matrix.orgbut we already install the ppa key directly COPY tools/openstack-ci-core-ppa.asc /etc/apt/trusted.gpg.d/21:27
@iwienand:matrix.orgyeah, that was my thinking.  i can re-propose with change comment explicitly calling it out21:28
@iwienand:matrix.orgrather than "various things" :)21:28
@clarkb:matrix.orgI don't think that is necessary, just wanted to be sure I understood what was going on there. Basically it isn't directly related to bullseye update but should be done21:29
@clarkb:matrix.orgthough maybe a commit message note or similar would be good21:29
@iwienand:matrix.orgthe other one was that option for the podman install.  fedora built using hte containerfile element, so that shows that removal is safe21:30
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:22:01
- [zuul/zuul] Don't treat failed requirement jobs as ready https://review.opendev.org/c/zuul/zuul/+/806781
- [zuul/zuul] Remove returnNodeSet calls from scheduler https://review.opendev.org/c/zuul/zuul/+/806062
- [zuul/zuul] Remove nodeset from NodeRequest https://review.opendev.org/c/zuul/zuul/+/806063
- [zuul/zuul] Make node requests persistent https://review.opendev.org/c/zuul/zuul/+/806280
- [zuul/zuul] Add node request cache to zk nodepool interface https://review.opendev.org/c/zuul/zuul/+/806639
- [zuul/zuul] Wrap nodepool request completed events with election https://review.opendev.org/c/zuul/zuul/+/806653
- [zuul/zuul] Remove hold request cache from nodepool interface https://review.opendev.org/c/zuul/zuul/+/806813
- [zuul/zuul] Remove unecessary node request cancelation code https://review.opendev.org/c/zuul/zuul/+/806814
@iwienand:matrix.orgcorvus: are we ok to restart zuul-web to fix the severity buttons?22:19
@jim:acmegating.comianw: i think it should be fine; i don't remember anything that requires the web and scheduler be in sync, but it still might be a good idea to do a full system restart.  would take care of the zk key question anyway.22:36
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed:22:40
- [zuul/zuul] Wrap nodepool request completed events with election https://review.opendev.org/c/zuul/zuul/+/806653
- [zuul/zuul] Remove unecessary node request cancelation code https://review.opendev.org/c/zuul/zuul/+/806814
- [zuul/zuul] Refactor the checkNodeRequest method https://review.opendev.org/c/zuul/zuul/+/806816
-@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] Don't store node requests/nodesets on queue items https://review.opendev.org/c/zuul/zuul/+/80682123:38

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!