| @spamaps:spamaps.ems.host | Hrm. nodepool marks nodes ready about 30s before they can be SSH'd to properly... | 00:24 |
|---|---|---|
| @spamaps:spamaps.ems.host | (So the ansible setup task times out) | 00:26 |
| @jim:acmegating.com | should be an ssh check in nodepool unless it's disabled | 00:26 |
| @spamaps:spamaps.ems.host | It just checks the port | 00:26 |
| @spamaps:spamaps.ems.host | the user can't SSH yet | 00:26 |
| @jim:acmegating.com | it should get the keys | 00:26 |
| @spamaps:spamaps.ems.host | cloud-init problems | 00:27 |
| @spamaps:spamaps.ems.host | It seems much faster on 20.04.. | 00:27 |
| @jim:acmegating.com | ah, it starts with keys then changes them | 00:27 |
| @spamaps:spamaps.ems.host | Well it starts with a functional sshd, but no authorized_keys .. and then installs them some time later. | 00:27 |
| @spamaps:spamaps.ems.host | Just late enough on 18.04 that if the node hasn't been sitting about a minute before the job starts.. it fails. | 00:28 |
| @spamaps:spamaps.ems.host | 20.04 seems to get it done much faster. | 00:28 |
| @jim:acmegating.com | spamaps: did you see the nodepool.yaml i have for gerrit? | 00:28 |
| @spamaps:spamaps.ems.host | No? | 00:28 |
| @jim:acmegating.com | spamaps: https://gerrit.googlesource.com/zuul/ops/+/refs/heads/master/nodepool/nodepool.yaml there's some comments in there about it | 00:29 |
| @jim:acmegating.com | spamaps: it doesn't look like that's necessarily a solution to the problem you're seeing. but it's more data. | 00:30 |
| @spamaps:spamaps.ems.host | Oh this isn't an ssh thing.. | 00:31 |
| @spamaps:spamaps.ems.host | the network actually seems to go down for a minute. | 00:31 |
| @spamaps:spamaps.ems.host | 🤷 | 00:31 |
| @jim:acmegating.com | i could see how that would affect connectivity :) | 00:31 |
| @spamaps:spamaps.ems.host | I'm going to side-step this .. I don't actually want to use this particular image.. In fact I really want to run 90% of things in gke pods anyway. | 00:31 |
| @spamaps:spamaps.ems.host | I can get my job to try and run on 20.04 but because of the way they've configured networking and DNS our internal ubuntu mirrors don't have 20.04 .. so.. yeah.. n/m ... onward to GKE. | 00:32 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 817479: Fix buildset config_errors https://review.opendev.org/c/zuul/zuul/+/817479 | 00:34 | |
| @spamaps:spamaps.ems.host | Hrm, I think I may have to invent a new kubernetes type | 00:36 |
| @spamaps:spamaps.ems.host | There's no way I'm getting namespace-creation permissions | 00:36 |
| @spamaps:spamaps.ems.host | and I don't even want that | 00:36 |
| @spamaps:spamaps.ems.host | I just want pods... lots of pods.. in one namespace. | 00:37 |
| @jim:acmegating.com | spamaps: is https://zuul-ci.org/docs/nodepool/kubernetes.html#value-providers.[kubernetes].pools.labels.type.pod not sufficient? | 00:39 |
| @spamaps:spamaps.ems.host | No, that always creates a namespace | 00:39 |
| @jim:acmegating.com | oh i see, that makes a namespace for the pod. nm. | 00:39 |
| @spamaps:spamaps.ems.host | I'm going to see if I can add a namespace: setting to it, and then skip the createNamespace part. | 00:40 |
| @spamaps:spamaps.ems.host | That may actually be sufficient. | 00:40 |
| @jim:acmegating.com | pretty sure there was a good reason for the way that was made. so expect some discussion on that. :) | 00:40 |
| @spamaps:spamaps.ems.host | Sure. For me, I just want to run stuff in a container with a particular image and thus contents. I can't expect admin level controls and don't need separation at the namespace level anyway. | 00:41 |
| @spamaps:spamaps.ems.host | I can imagine that a reason not to do it this way is that one pod can probably spy on other running pods if the k8s perms aren't set up just so. | 00:42 |
| @spamaps:spamaps.ems.host | But I think that can be handled by locking down the particular namespace to deny that. | 00:43 |
| @jim:acmegating.com | yeah. i'm not sure if we documented the reasons; so we might have to poll some people to figure it out. i agree what you want is sensible, and that implementing it and pushing up a change for review is a great way to start the discussion. just wanted to flag that as potentially something that might not sail right through :) | 00:44 |
| @spamaps:spamaps.ems.host | Yeah, if I could get namespace creation access I wouldn't care, I'd just let this happen. :) | 00:45 |
| @spamaps:spamaps.ems.host | But.. trying to take advantage of the well-managed k8s clusters we already have if I can. :) | 00:45 |
| @spamaps:spamaps.ems.host | For what I'm trying to prove right now.. I can probably run on VMs for a while anyway. We have a big monorepo build that is killing the jenkins nodes we have access to (32GB 16 core vms). So I want to just break that up into 7 or so zuul jobs and have them all run on their own right-sized vms. But.. if I can't use k8s.. I have to figure out how to run the action containers on the vms. | 00:49 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: | 00:53 | |
| - [zuul/zuul] 817484: Use a stable hash for ConfigurationErrorKeys https://review.opendev.org/c/zuul/zuul/+/817484 | ||
| - [zuul/zuul] 817486: Identify the object in ZKObject (de)serialization errors https://review.opendev.org/c/zuul/zuul/+/817486 | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 817490: Log null change key deference https://review.opendev.org/c/zuul/zuul/+/817490 | 00:53 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 817343: Fix a bug in getting changed files https://review.opendev.org/c/zuul/zuul/+/817343 | 04:20 | |
| -@gerrit:opendev.org- Dong Zhang proposed: [zuul/zuul] 817343: Fix a bug in getting changed files https://review.opendev.org/c/zuul/zuul/+/817343 | 05:46 | |
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 817518: Add an icon for each type of component to the components page https://review.opendev.org/c/zuul/zuul/+/817518 | 08:19 | |
| -@gerrit:opendev.org- Felix Edel proposed: [zuul/zuul] 817518: Add an icon for each type of component to the components page https://review.opendev.org/c/zuul/zuul/+/817518 | 08:26 | |
| -@gerrit:opendev.org- Zuul merged on behalf of Felix Edel: [zuul/zuul] 814711: UI: Fix build time calculation for empty buildsets https://review.opendev.org/c/zuul/zuul/+/814711 | 09:04 | |
| @mordred:inaugust.com | > <@spamaps:spamaps.ems.host> woo I got a test written and it even does some decorating | 10:20 |
| did you eventually discover that you could run the unit tests on mac directly? | ||
| -@gerrit:opendev.org- Zuul merged on behalf of Felix Edel: [zuul/zuul] 814717: UI: Ignore empty timestamps in build time calculation on buildset page https://review.opendev.org/c/zuul/zuul/+/814717 | 10:32 | |
| @avass:vassast.org | corvus: spamaps regarding https://review.opendev.org/c/zuul/zuul-jobs/+/817291, can we try to make that more generic? I know we talked about the revoke-sudo role previously where we didn't want to revoke sudo on static nodes. The idea there was to configure something like `attribute.revoke_sudo: false` on the label or node in nodepool which then gets passed to the role as `nodepool.revoke_sudo` or `nodepool.attributes.revoke_sudo`. Maybe the sudoers file should be configured in nodepool too? | 11:22 |
| @emacchi:matrix.org | Hey folks, I'm using Zuul from openlab recently (for Gophercloud CI) and we added new CI jobs in .zuul.yaml of the project, but somehow Zuul still runs the old job. If I look at the footer of https://status.openlabtesting.org/status - I can see that the last reconfiguration was more than a month ago. Does Zuul needs to be reconfigured when a new job is added? | 13:34 |
| @fungicide:matrix.org | emacchi: typically, you should expect that timestamp to update any time a configuration change merges to a tracked repository/branch, or when a reconfiguration is requested (e.g. after updating the tenant config to add/remove repos), or when the scheduler was last restarted, whichever happened more recently | 13:49 |
| @fungicide:matrix.org | if zuul is tracking the repository/branch and configured to load job configs from it, then it should notice when those configs change and reconfigure itself automatically | 13:50 |
| @fungicide:matrix.org | it's possible this configuration error is preventing the scheduler from being able to complete tenant reconfiguration: https://status.openlabtesting.org/config-errors | 13:53 |
| @emacchi:matrix.org | > <@fungicide:matrix.org> it's possible this configuration error is preventing the scheduler from being able to complete tenant reconfiguration: https://status.openlabtesting.org/config-errors | 14:01 |
| oh so if we fix the error, there is a good chance that zuul scheduler will successfully restart and load the new config. | ||
| @fungicide:matrix.org | emacchi: yes, well it should reload its configuration, normally a restart shouldn't be necessary for that | 14:04 |
| @tobias.henkel:matrix.org | mordred, spamaps nodepool unit tests on mac? That should be possible but requires some hacks | 14:08 |
| @tobias.henkel:matrix.org | but nodepool is easier than zuul I think | 14:09 |
| @tobias.henkel:matrix.org | but it's already a while I ran the nodepool tests locally last time | 14:14 |
| @apevec:matrix.org | emacchi: what is openlab and what Zuul do they run? | 14:43 |
| @jim:acmegating.com | apevec: https://openlabtesting.org/ | 14:44 |
| @apevec:matrix.org | hmm doesn't say much "managed by open source community members" | 14:45 |
| @emacchi:matrix.org | > <@apevec:matrix.org> emacchi: what is openlab and what Zuul do they run? | 14:46 |
| they run their own infra | ||
| @jim:acmegating.com | apevec: was started by huawei. mostly for openstack<->k8s integration testing | 14:46 |
| @emacchi:matrix.org | we'll probably leave it btw | 14:47 |
| @emacchi:matrix.org | we're discussing about it (see openstack-discuss) | 14:47 |
| @apevec:matrix.org | interesting they've choosen Zuul | 14:47 |
| @spamaps:spamaps.ems.host | > <@mordred:inaugust.com> did you eventually discover that you could run the unit tests on mac directly? | 15:07 |
| Some, not all. | ||
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 817491: Add repr to FrozenJob https://review.opendev.org/c/zuul/zuul/+/817491 | 15:23 | |
| @fungicide:matrix.org | > <@apevec:matrix.org> interesting they've choosen Zuul | 16:33 |
| you'll find them listed at https://zuul-ci.org/users.html with a link to an article/interview from several years ago | ||
| @fungicide:matrix.org | it explains some of their reasons for that choice | 16:33 |
| -@gerrit:opendev.org- Tobias Henkel proposed: [zuul/zuul] 817604: DNM: Upload build requests async https://review.opendev.org/c/zuul/zuul/+/817604 | 17:15 | |
| @tobias.henkel:matrix.org | this is an analysis that lead to this change | 17:21 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 817626: WIP: Remove RPC client from autohold tests https://review.opendev.org/c/zuul/zuul/+/817626 | 17:41 | |
| @tobias.henkel:matrix.org | corvus: +2 with comment on 817196 | 18:58 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 817640: Add some pipeline processing stats https://review.opendev.org/c/zuul/zuul/+/817640 | 19:03 | |
| @jim:acmegating.com | tobiash, swest: ^ that may give us some interesting performance stats | 19:04 |
| @jim:acmegating.com | tobiash: good point; i think i'm willing to run that in prod on opendev for just a bit to see how bad it is :) | 19:08 |
| @tobias.henkel:matrix.org | I think the badness of that scales with the number of pipelines and tenants :) | 19:09 |
| @jim:acmegating.com | yep. but we'll get a baseline... some value that we don't know how to scale ;) | 19:10 |
| @tobias.henkel:matrix.org | first item in gate will fail | 20:15 |
| @tobias.henkel:matrix.org | looks like again the test_zookeeper_disconnect test case that's the first that failed | 20:16 |
| @jim:acmegating.com | sigh. i'll try to dig into that more | 20:17 |
| @jim:acmegating.com | or maybe we should delete the test | 20:19 |
| @jim:acmegating.com | okay, the issue with that is the stats thread. it start the election after the reconnect, but it doesn't win it, and the cancel doesn't appear to have canceled the election. | 20:32 |
| @iwienand:matrix.org | just checking we're not in any state that merging https://review.opendev.org/c/zuul/nodepool/+/817482 to bump dib in nodepool would be a concern? | 21:23 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 817646: Re-order scheduler shutdown https://review.opendev.org/c/zuul/zuul/+/817646 | 21:23 | |
| @jim:acmegating.com | ianw: +2 i think it's clear | 21:24 |
| -@gerrit:opendev.org- Zuul merged on behalf of Felix Edel: [zuul/zuul] 816807: Split up registerScheduler() and onLoad() methods https://review.opendev.org/c/zuul/zuul/+/816807 | 21:53 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-sphinx] 817650: Add :zuul:path support https://review.opendev.org/c/zuul/zuul-sphinx/+/817650 | 22:23 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-sphinx] 817650: Add :zuul:path support https://review.opendev.org/c/zuul/zuul-sphinx/+/817650 | 22:24 | |
| @harrymichal:matrix.org | Hi folks! I'm wondering about the behaviour of artifacts in jobs. Do the artifacts live across pipeline invocations? E.g., a build job caches some data that speeds up future builds -> possible speed up of future pipeline runs. | 22:27 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 817652: Prevent duplicate config file entries https://review.opendev.org/c/zuul/zuul/+/817652 | 22:32 | |
| @jim:acmegating.com | martymichal: yep, the info is stored in the sql database, so if a required artifact is satisfied by a build that's currently running, it gets the data from memory; if its satisfied by a previous build, it gets it from the db. | 22:33 |
| @clarkb:matrix.org | corvus: ^ I think that change largely addresses the problem I discovered the other day with extra-config-paths potentailly doubling up configs. However, the tests don't pass yet because I think they expose another bug where we early load configs in the scheduler before validating them. This means you get a much more cryptic error message than you would from the validator (my test should hopefully illustrate this). You might want to take a look at that to double check it isn't a larger issue | 22:33 |
| @jim:acmegating.com | martymichal: (you may know this, but just to be clear: zuul manages metadata about the artifact, like the storage location; it's up to jobs and roles to implement the actual storage and fetching of it; the zuul-jobs repo has some roles to help with that) | 22:34 |
| @jim:acmegating.com | Clark: ack, thx | 22:34 |
| @harrymichal:matrix.org | > martymichal: (you may know this, but just to be clear: zuul manages metadata about the artifact, like the storage location; it's up to jobs and roles to implement the actual storage and fetching of it; the zuul-jobs repo has some roles to help with that) | 22:36 |
| Didn't know that. So, I'll have to check those out before I start writing the config for the artifacts. Before I do so, does this require the project to acquire some actual storage space? | ||
| @harrymichal:matrix.org | Anyway, thanks for the clarification! | 22:42 |
| -@gerrit:opendev.org- Zuul merged on behalf of Ian Wienand: [zuul/nodepool] 817482: Bump DIB to 3.15.1 https://review.opendev.org/c/zuul/nodepool/+/817482 | 22:59 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-sphinx] 817650: Add :zuul:path support https://review.opendev.org/c/zuul/zuul-sphinx/+/817650 | 23:07 | |
| @jim:acmegating.com | ianw: would you mind taking a quick look at https://review.opendev.org/817650 ? | 23:37 |
| @jim:acmegating.com | https://review.opendev.org/809300 is the use case (cc: tristanC) | 23:38 |
| @iwienand:matrix.org | sure, just let me get into sphinx mode :) | 23:38 |
| @jim:acmegating.com | yeah... it's been a few years :) | 23:40 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!