| @iwienand:matrix.org | > <@jim:acmegating.com> ianw: Clark anyone else involved in that convo that can throw another +2 on that now? | 01:42 |
|---|---|---|
| corvus: this is done so if you can approve that message at your convenience that would be great, thanks | ||
| @jim:acmegating.com | ianw: i would... but i can't connect to lists.zuul-ci.org... | 01:59 |
| @jim:acmegating.com | ianw: done! | 03:11 |
| @iwienand:matrix.org | thanks, a little more dramatic than i'd hoped :) | 03:11 |
| @jim:acmegating.com | ianw: the universe is asking if you're *sure* you want to deprecate that ;) | 03:14 |
| @iwienand:matrix.org | it all relates to installing pip globally from upstream. that has been the gift that has just kept giving for the past ... 10 years or so | 03:15 |
| @blaisep-sureify:matrix.org | Fair enough | 03:15 |
| ... and it's quite easy | ||
| https://zero2datadog.readthedocs.io/en/latest/faas.html# | ||
| @jkt_:matrix.org | hi, I'm on some ancient nodepool on top of a pretty oversubscribed private cloud, which means that my openstack-level quotas are often higher than the number of actually available hypervisors | 13:18 |
| @jkt_:matrix.org | that often leads to `Detailed node error: No valid host was found. There are not enough hosts available.`, which is a hard error in (at least my, rather old one) nodepool | 13:18 |
| @jkt_:matrix.org | are there any recent changes or plans which would make this a transient error? | 13:19 |
| @jkt_:matrix.org | our private openstack operators are hesitant to reflect the amount of nodes which are "actually available" in a quote because their terraform (?) thingy which they use for tenant and quota management won't play nicely with that, apparently | 13:21 |
| @jkt_:matrix.org | so my backup plan is to have them publish a simple HTTP endpoint with a number of VMs I can launch, and manipulating my `nodepool.yaml` :) | 13:22 |
| @q:fricklercloud.de | Jan Kundrát: iiuc this should be handled by https://review.opendev.org/c/zuul/nodepool/+/839226 | 13:36 |
| @jkt_:matrix.org | frickler: my openstack-level quota is telling me "you can run 100 VMs", while I can only run 15 because they are almost fully utilized; that patch looks like something else, doesn't it? | 13:37 |
| @clarkb:matrix.org | I think 839226 only helps if you've got an openstack that expands it's size based on requests. I doubt your openstack does that. Instead you can set max-servers on your end to try and limit the blast zone. You can also set the number of retries iirc. You can also add additional providers that can act as a secondary option | 13:40 |
| @clarkb:matrix.org | But ultimately if you've told nodepool that it can expect X resources and it finds X-N then that is an error and it reports it as such | 13:41 |
| @jim:acmegating.com | Jan Kundrát: i think you're looking for commit 31c276c234 | 13:41 |
| @jkt_:matrix.org | yeah, I was suspecting that my use case is generally hard to hit on a public cloud because it's really a case of "we don't want your money" :) | 13:41 |
| @jim:acmegating.com | that first appeared in 3.2.0 | 13:43 |
| @jkt_:matrix.org | corvus: maybe I'm on a too old nodepool, but from the logs I cannot tell whether I get `ABORTED` or `FAILED` | 13:43 |
| @jkt_:matrix.org | 3.12.1.dev3 | 13:43 |
| @jkt_:matrix.org | I was thinking that *if* they manipulated my quota so that it reflected what I can "quite probably" get right now, that would get me almost there | 13:44 |
| @fungicide:matrix.org | yeah, modern nodepool will query the quotas/limits in your providers and dynamically adjust its expectations | 13:46 |
| @jkt_:matrix.org | so yeah, the real clause is that this particular private cloud is "broken" because the quota is "too high", I guess. OK, that makes sense. | 13:46 |
| @fungicide:matrix.org | for opendev, we have providers who lower our quotas temporarily or raise them when they're okay with us bursting additional load there | 13:47 |
| @fungicide:matrix.org | and that can be done on the fly by them without needing us to adjust nodepool configs | 13:48 |
| @fungicide:matrix.org | something that i end up doing as a zuul administrator semi-regularly, is helping users identify the causes of post_failure results with no log uploads. my usual workflow is to take the build uuid and then grep the debug logs of all 12 of our executors in order to find which one handled that build (i know i could find that out by grepping the scheduler's log as well, but can never remember the exact pattern for the log message which says which executor took it, so just grepping all the executor logs is a little faster for me). would it make sense to include the executor information in the build result dashboard in order to make this case easier? does that need an extension of the db scehma or are we already tracking it there and would just be a simple ui tweak? | 15:57 |
| @fungicide:matrix.org | or is there a more straightforward way to do what i've been doing? | 15:58 |
| @jim:acmegating.com | fungi: i like the idea of storing the executor; would need a schema update. (in the mean time, ansible can do the grep for you across all executors) | 16:00 |
| @fungicide:matrix.org | yeah, the ansible solution is pretty good, i just need to remember the syntax | 16:01 |
| @fungicide:matrix.org | ssh in a for loop is often easier to wrap my head around | 16:01 |
| @fungicide:matrix.org | `sudo ansible zuul-executor -m shell -a 'grep ffb35b4d680d48a8bd21b1964964019d /var/log/zuul/executor-debug.log'` | 16:06 |
| @fungicide:matrix.org | that seems to do the trick, thanks! | 16:06 |
| @clarkb:matrix.org | https://eng.uber.com/how-we-halved-go-monorepo-ci-build-time/ via the gerrit random channel on slack. Reading that is interesting beacuse they seem ot have employed a number of strategies we've already been using for many years | 17:03 |
| @clarkb:matrix.org | There is a bit about a remote cache near the end that includes the caching of test results. That is an interesting idea. I think this only works if you have no external dependencies though which many of OpenDev's users do actually care about so not a general approach. | 17:06 |
| -@gerrit:opendev.org- Kenny Ho proposed: [zuul/zuul] 837063: Add build set URL to reporter https://review.opendev.org/c/zuul/zuul/+/837063 | 21:53 | |
| -@gerrit:opendev.org- Kenny Ho proposed wip: [zuul/zuul] 837063: Add build set URL to reporter https://review.opendev.org/c/zuul/zuul/+/837063 | 23:53 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!