| corvus | and so i think what's happening with download plugin is that as changes merge to d-p master, since that doesn't match stable-3.0, the pointer from gerrit/stable3.0 to download-plugin does not advance (but the pointer in gerrit/master to download-plugin does advance) | 00:00 |
|---|---|---|
| corvus | so yeah, things are effectively "frozen" until either download-plugin gets a stable-3.0 branch and merges changes to it, or someone moves the submodule pointer manually | 00:00 |
| mordred | yah | 00:01 |
| fungi | maybe this helps them improve their coordination between repos | 00:02 |
| mordred | whcih makes zuul dependencies hard to do anything with other than bail | 00:02 |
| corvus | mordred, fungi: yes to both | 00:03 |
| *** Defolos has quit IRC | 00:09 | |
| *** sgw has joined #zuul | 00:17 | |
| *** wxy-xiyuan has joined #zuul | 00:46 | |
| *** sgw has quit IRC | 00:52 | |
| *** sshnaidm is now known as sshnaidm|afk | 01:20 | |
| *** jamesmcarthur has joined #zuul | 01:22 | |
| *** jamesmcarthur has quit IRC | 01:34 | |
| *** jamesmcarthur has joined #zuul | 01:41 | |
| *** bhavikdbavishi has joined #zuul | 01:52 | |
| *** bhavikdbavishi1 has joined #zuul | 01:55 | |
| *** bhavikdbavishi has quit IRC | 01:57 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:57 | |
| *** jamesmcarthur has quit IRC | 02:05 | |
| *** jamesmcarthur has joined #zuul | 02:11 | |
| *** jamesmcarthur has quit IRC | 02:40 | |
| *** jamesmcarthur has joined #zuul | 02:43 | |
| *** jamesmcarthur has quit IRC | 02:48 | |
| *** zxiiro has quit IRC | 02:54 | |
| *** jamesmcarthur has joined #zuul | 02:57 | |
| *** jamesmcarthur has quit IRC | 03:25 | |
| *** jamesmcarthur has joined #zuul | 03:27 | |
| *** rlandy|bbl is now known as rlandy | 03:29 | |
| *** jamesmcarthur has quit IRC | 03:55 | |
| *** jamesmcarthur has joined #zuul | 03:56 | |
| *** jamesmcarthur has quit IRC | 04:01 | |
| *** rlandy has quit IRC | 04:03 | |
| *** jamesmcarthur has joined #zuul | 04:26 | |
| *** igordc has joined #zuul | 04:37 | |
| *** igordc has quit IRC | 04:48 | |
| *** igordc has joined #zuul | 04:48 | |
| *** igordc has quit IRC | 04:48 | |
| *** raukadah is now known as chandankumar | 04:59 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #zuul | 05:34 | |
| *** swest has joined #zuul | 05:36 | |
| *** reiterative has quit IRC | 05:39 | |
| *** reiterative has joined #zuul | 05:39 | |
| *** sgw has joined #zuul | 05:53 | |
| openstackgerrit | yatin proposed zuul/zuul-jobs master: Ensure indirect deps are updated too https://review.opendev.org/707569 | 06:32 |
| openstackgerrit | Merged zuul/zuul master: Don't report enqueue of non-live item https://review.opendev.org/707479 | 06:50 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: Ensure correct cleanup on repo update and reset https://review.opendev.org/701531 | 07:02 |
| *** saneax has joined #zuul | 07:20 | |
| *** AJaeger has quit IRC | 07:35 | |
| *** AJaeger has joined #zuul | 07:38 | |
| *** Defolos has joined #zuul | 07:39 | |
| *** jpena|off is now known as jpena | 07:51 | |
| *** armstrongs has joined #zuul | 08:04 | |
| *** armstrongs has quit IRC | 08:13 | |
| *** hashar has joined #zuul | 08:14 | |
| *** tosky has joined #zuul | 08:19 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS https://review.opendev.org/707585 | 08:41 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Reduce gearman logging in tests https://review.opendev.org/707587 | 08:52 |
| *** mhu has joined #zuul | 09:11 | |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds in a build set via mqtt. https://review.opendev.org/632727 | 09:11 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 09:11 |
| *** NBorg has joined #zuul | 09:25 | |
| *** avass has joined #zuul | 09:44 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Make most test cases work on MacOS https://review.opendev.org/707585 | 10:02 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 10:07 |
| *** AJaeger has quit IRC | 10:10 | |
| *** AJaeger has joined #zuul | 10:11 | |
| fbo | Hi, maybe someone could confirm, regarding provide/require for artifacts. Here https://src.fedoraproject.org/rpms/python-setuptools/pull-request/31 we see "Requirements ['repo'] not met by build defd1e2b60f6420faa59f4d2e51e575e" I guess it means a depends PR should have generated an artifact but in reality it has not. So I guess that the reason ? | 10:12 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Add support for tracing using OpenTelemetry https://review.opendev.org/705170 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Enable tracing of trigger events https://review.opendev.org/706289 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Github webhook trigger events https://review.opendev.org/706290 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Gerrit trigger events https://review.opendev.org/706291 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace timer trigger events https://review.opendev.org/707606 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace git trigger nevents https://review.opendev.org/707607 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace zuul trigger events https://review.opendev.org/707608 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace Pagure webhook trigger events https://review.opendev.org/707609 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace queue items https://review.opendev.org/707610 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace merge jobs (client) https://review.opendev.org/707611 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: Trace merge jobs (server) https://review.opendev.org/707612 | 10:26 |
| openstackgerrit | Simon Westphahl proposed zuul/zuul master: wip: DNM test https://review.opendev.org/707613 | 10:26 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add load-branch to tenant configuration https://review.opendev.org/705664 | 10:31 |
| tobiash | fbo: not necessarily a depends, a gate job that provides the artifact that resulted in a merge is sufficient | 10:49 |
| tobiash | for introducing a provides/requires it might be needed to get the job that 'provides' merged first to get it working | 10:50 |
| tobiash | but corvus knows that better | 10:50 |
| fbo | tobiash: my testing regarding theprovide/require shown even not merged change can expose artifacts. See this example | 10:58 |
| fbo | This PR https://github.com/rpm-software-management/dnf/pull/1584 (last rpm-mock-build-source) in the inventory the artifact produced by the 'Depends-on: rpm-software-management/libdnf#896' is exposed | 11:00 |
| fbo | tobiash: ^ | 11:00 |
| *** AJaeger has quit IRC | 11:04 | |
| *** hashar has quit IRC | 11:05 | |
| *** AJaeger has joined #zuul | 11:09 | |
| tobiash | yes, you need just a provides job that ran before the requires job regardless if that's merged or not | 11:15 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add project, change in RequirementError message https://review.opendev.org/707620 | 11:17 |
| fbo | the behavior of zuul when an artifact is missing is to make the require job fail. but in my use case I would like prefer like a soft require that would have just added in the inventory that the artifact is missing. That way I could handle that case in the job. | 11:20 |
| *** sshnaidm|afk is now known as sshnaidm|brb | 11:25 | |
| tobiash | fbo: do you need the artifact within a buildset or from different changes? | 11:27 |
| mhu | tobiash, from depends-on | 11:29 |
| tobiash | hrm, then a soft-requirement would be needed | 11:29 |
| *** jpena is now known as jpena|lunch | 11:56 | |
| *** sshnaidm|brb is now known as sshnaidm | 11:56 | |
| fbo | tobiash: in fact from both, depends-on provide dependents 'rpms repo' artifacts to a buildset that run a parent job (that provide also a 'rpms repo' artifact) and child jobs require a 'rpms repo' artifact. But here https://src.fedoraproject.org/rpms/python-setuptools/pull-request/31#comment-37696 we see that childs jobs failed (even before the parent job run) | 12:05 |
| *** rfolco has joined #zuul | 12:25 | |
| *** rlandy has joined #zuul | 13:00 | |
| *** jamesmcarthur has quit IRC | 13:12 | |
| *** jamesmcarthur has joined #zuul | 13:12 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Improve error reporting when pr merge fails https://review.opendev.org/707637 | 13:12 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 13:19 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make ensure-tox pass cross-platform https://review.opendev.org/707439 | 13:19 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add foreground option https://review.opendev.org/635649 | 13:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Deprecate -d switch for running in foreground https://review.opendev.org/705185 | 13:29 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Don't enforce foreground with -d switch https://review.opendev.org/705189 | 13:29 |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** jamesmcarthur has quit IRC | 13:38 | |
| *** Goneri has quit IRC | 13:40 | |
| *** Goneri has joined #zuul | 14:31 | |
| mordred | corvus: https://gerrit-review.googlesource.com/c/zuul/jobs/+/254752 is missing a change-id - which has shown me a gerrit behavior I'd never seen before! | 14:38 |
| mordred | corvus: also - the second change has a typo, so if you decide to rectify the change-id in the first (not strictly necessary) - I'd go ahead and fix the second thing too | 14:39 |
| mordred | but otherwise, both look great | 14:39 |
| *** NBorg has quit IRC | 14:58 | |
| *** chandankumar is now known as raukadah | 14:59 | |
| *** rlandy is now known as rlandy|brb | 15:10 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Move oc download to before src copy https://review.opendev.org/707250 | 15:20 |
| *** rlandy|brb is now known as rlandy | 15:26 | |
| *** saneax has quit IRC | 15:34 | |
| *** michael-beaver has joined #zuul | 15:53 | |
| *** zxiiro has joined #zuul | 15:58 | |
| *** Defolos has quit IRC | 16:02 | |
| *** Defolos has joined #zuul | 16:06 | |
| *** jpena is now known as jpena|off | 16:06 | |
| *** tosky has quit IRC | 16:21 | |
| corvus | fbo, tobiash: a requires job should not fail if a provides job does not produce an artifact. | 16:36 |
| *** Defolos has quit IRC | 16:36 | |
| fungi | that's simply used for execution ordering the builds, right? | 16:38 |
| fungi | so that the requiring job doesn't run until the providing one finishes? | 16:38 |
| fungi | though obviously if the requiring job makes use of that provided artifact in some way then it could end up failing because the artifact wasn't available, but that's under control of the job itself | 16:39 |
| corvus | if the providing job failed though... | 16:39 |
| corvus | i'm trying to figure out exactly what's going on with the examples fbo linked | 16:40 |
| *** jamesmcarthur has joined #zuul | 16:41 | |
| *** tosky has joined #zuul | 16:43 | |
| corvus | fbo: it looks like this is the build that failed: https://fedora.softwarefactory-project.io/zuul/build/defd1e2b60f6420faa59f4d2e51e575e | 16:43 |
| fbo | corvus: yes, some of the depends-on change did not produced the expected artifact | 16:44 |
| corvus | fbo: the situation looks to me like this: a job on a change says it requires "repo" artifacts, the change depends on another change which runs jobs which provide "repo" artifacts, but that job failed. | 16:45 |
| corvus | fbo: well, it more than just didn't produce it -- the job that was supposed to produce it failed | 16:45 |
| corvus | that really seems to me like the sort of case that zuul should not assume that it's okay not to have an artifact. | 16:45 |
| corvus | (it would be something else if the job that was supposed to produce it was skipped, or ran successfully but produced zero artifacts as a result) | 16:46 |
| fungi | though presumably the job could be made to still succeed if there are expected conditions under which the artifact can't be produced, and the requiring job has a reasonable fallback | 16:46 |
| fungi | or basically what corvus just said | 16:47 |
| corvus | yeah. it just seems like the one case where we couldn't say "it's okay not to have an artifact" is when the job producing it really does fail. | 16:47 |
| fbo | corvus ok but the failure let the user with a not human friendly info thus https://review.opendev.org/707620 | 16:47 |
| mhu | corvus, yeah that's a contingency we have, if the artifact doesn't exist we can rebuild it in the job that required it | 16:48 |
| fbo | corvus: ok but could we have a solution to still run the job but have this information (artifact missing) in thi sinventory | 16:48 |
| mhu | corvus, fbo: I was thinking of adding a "soft" option to requires, similar to job dependencies | 16:48 |
| fungi | if it's expected that the providing job sometimes doesn't produce an artifact, then why not just make it succeed under those circumstances? | 16:50 |
| fungi | (especially if you have a good way of detecting when non-production of an artifact is expected vs when it's not expected) | 16:51 |
| corvus | mhu: what happens with a soft dependency if the parent job fails? | 16:52 |
| corvus | mhu: i'm pretty sure the child job does not run | 16:52 |
| corvus | so i agree that it makes sense to look into situations where a requires job should still run even if a provides job doesn't provide an artifact, but if the provides job errors, that sounds really dangerous | 16:53 |
| corvus | is the use-case here opportunistic artifact caching? | 16:54 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Centralize merge handling https://review.opendev.org/707692 | 16:55 |
| mhu | yeah: if the artifact exists, use it, otherwise build it | 16:55 |
| fbo | yes eventually that could be handled by the job itself. But yes, you are true corvus that's not safe. I need to think more about it. | 16:55 |
| fbo | the fact is today zuul returns a not obvious warning to the user | 16:56 |
| mhu | but the only case an artifact wouldn't exist is if the provides job didn't succeed | 16:56 |
| fbo | Maybe better to have such a message "Unable to run jobs as depends artifacts are misssing for X Y Z projects" | 16:57 |
| mhu | fbo: although it's true that you can search by build id in the web ui | 16:57 |
| fbo | s/projects/changes/ | 16:57 |
| fungi | except it didn't refuse to run because the artifact was missing, it refused to run because the job which would have provided that artifact did not succeed | 16:57 |
| mhu | fungi, right | 16:57 |
| corvus | mhu: it could not exist because the job that produces it was intentionally not run. yeah what fungi said | 16:57 |
| fbo | yes | 16:58 |
| corvus | basically the system was predicated on the idea that A depends on B, and if B doesn't get built, then A is invalid. you seem to be describing a system where even if that's true, A can build both A and B. | 16:59 |
| corvus | i guess i don't understand why if B wasn't able to build itself, why would A have any better luck building B? | 17:00 |
| corvus | mhu, fbo: yes we should absolutely improve the messages. even i find them confusing. :) | 17:00 |
| *** jamesmcarthur has quit IRC | 17:02 | |
| fbo | yes you're right. thanks to have clarified the issue :) | 17:03 |
| mhu | ditto! | 17:03 |
| *** jamesmcarthur has joined #zuul | 17:06 | |
| *** jamesmcarthur has quit IRC | 17:09 | |
| *** jamesmcarthur has joined #zuul | 17:15 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add project, change in RequirementError message https://review.opendev.org/707620 | 17:21 |
| *** jpena|off is now known as jpena | 17:27 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #zuul | 17:34 | |
| *** jamesmcarthur has quit IRC | 17:47 | |
| *** jamesmcarthur has joined #zuul | 17:50 | |
| *** Defolos has joined #zuul | 17:59 | |
| *** bhavikdbavishi has quit IRC | 18:13 | |
| *** igordc has joined #zuul | 18:28 | |
| *** igordc has quit IRC | 18:28 | |
| *** adamw has quit IRC | 18:29 | |
| *** jpena is now known as jpena|off | 18:31 | |
| *** adamw has joined #zuul | 18:32 | |
| *** adamw has quit IRC | 18:32 | |
| *** adamw has joined #zuul | 18:32 | |
| corvus | mordred: https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994 is the repo setup from yesterday. it has a configuration error in that it refers to a bunch of unknown projects. but it got reported as "not relevant" because the config error caused no jobs to match. it also didn't post the line comments about the error. | 18:36 |
| corvus | so i think there's some bugs there we should fix | 18:36 |
| corvus | i think line comments should always be posted, and config errors should probably be failure, not "not relevant" | 18:37 |
| *** jamesmcarthur has quit IRC | 18:37 | |
| *** jamesmcarthur has joined #zuul | 18:38 | |
| mordred | corvus: yes - I agree with that | 18:39 |
| *** jamesmcarthur has quit IRC | 18:44 | |
| corvus | mordred: oh, hrm, it looks like it hit the not-relevant path because we haven't actually landed any changes which add the project to the check pipeline. | 18:48 |
| corvus | maybe we should go ahead and land https://gerrit-review.googlesource.com/c/zuul/jobs/+/254752/1 | 18:49 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Report robot comments to gerrit https://review.opendev.org/707708 | 19:02 |
| corvus | mordred: that looks like that's why that didn't show up | 19:03 |
| mordred | corvus: I merged the change, which felt very wrong | 19:29 |
| corvus | mordred: yeah, it's an icky feeling :) i figure we should be able to gate those repos eventually. just wanted to get gerrit-focused stuff going first | 19:40 |
| mordred | ++ | 19:40 |
| mordred | corvus: https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994/1/roles/prepare-gerrit-repos/README.rst is nicely written | 19:43 |
| corvus | mordred: thanks, i figured we probably needed to brain dump somewhere :) | 19:45 |
| mordred | corvus: I look forward to pulling that role back in to zuul jobs and then using it for our own gerrit builds | 19:46 |
| corvus | mordred: you think that one should migrate? i was thinking that should maybe live in gerrit's zuul. i guess it probably only needs a little bit of generalization then it could be in zuul-jobs. or we could pull gerrit/zuul/jobs into opendev's zuul | 19:47 |
| corvus | mordred: w00t! https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994 has comments now | 19:48 |
| corvus | (running with a locally built image to get the robot comments fix) | 19:48 |
| mordred | corvus: hrm. yeah - maybe it doesn't need to be in zuul-jobs ... it's _similar_ to our pre-playbook, but in honestly the chances that we're going to need the error condition check in ours are *way* low | 19:50 |
| mordred | corvus: I left a +2 with a nit comment in case you need to respin for other reasons | 19:51 |
| fungi | the "run details" link to the buildset info gives a blank page in the zuul dashboard... is that expected at this stage? | 19:52 |
| fungi | https://gerrit-zuul.inaugust.com/t/gerrit/buildset/c47cf6a1c0324d509ffffd65d2d1e8b3 | 19:52 |
| corvus | fbo: yeah, i would describe that as an "expected bug". no builds reported since the config was invalid, so there's no buildset record in the db. we should do something different, but i'm unsure what. | 19:53 |
| corvus | fungi: sorry fbo | 19:53 |
| corvus | mordred: https://gerrit-review.googlesource.com/c/zuul/ops/+/255012 is the next step | 19:57 |
| mordred | corvus: do we have any more cores? or should I just straight approve things? | 19:58 |
| corvus | mordred: i think the gerrit maintainers are included too, but i think we should probably just approve that one | 20:00 |
| corvus | i'll make sure to mention it in my report so that people can see the whole process (including the changes we haad to merge) | 20:00 |
| mordred | kk | 20:01 |
| fungi | oh, right-o | 20:01 |
| fungi | it didn't run anything | 20:01 |
| fungi | well, didn't run any builds anyway | 20:01 |
| corvus | yeah. i think the first obvious step would be to report that as a null url. but maybe long-term after the db rework we can have buildsets with no builds | 20:01 |
| corvus | cause that's a thing internally in zuul. that buildset has the result "CONFIG_ERROR". it's not crazy to think that visualizing that in the zuul web ui would be useful. | 20:02 |
| corvus | mordred: an interesting thing happened on the gerrit zuul. the scheduler lost its zk connection in a way that seemed permanent | 21:13 |
| corvus | also, the debug log is completely spammed with refs from the additional repos. so my first step in addressing this problem is rejiggering the io debug logging a bit so "-d" is more sensible and maybe we can actually tell what's going on if it happens again. | 21:14 |
| mordred | corvus: oh weird - and yeah - that is an interesting thing | 21:21 |
| mordred | corvus: it's running in a GKE - I wonder if the containers are co-located such that zuul noisy-neighbored itself enough to make the zk unhappy | 21:22 |
| mordred | (just thinking out loud) | 21:22 |
| corvus | mordred: yeah, could be, or maybe some migration thingy happened or something | 21:23 |
| mordred | yeah | 21:24 |
| mordred | as you said - mo bettah logging | 21:24 |
| mordred | and by mo, I mean less | 21:24 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Adjust io-level logging in gerrit/git drivers https://review.opendev.org/707728 | 21:28 |
| corvus | mordred: ^ more lessness | 21:28 |
| *** dpawlik is now known as dpawlik_off | 21:30 | |
| corvus | mordred: potentially related, i see "end of stream" sometimes when watching the console streaming. it's not ended, and it resumes (by starting over again). i wonder if some intra-k8s connection for the log streamer is getting interrupted? | 21:36 |
| corvus | tristanC, tobiash, SpamapS: ^ ever seen anything like that? | 21:37 |
| *** avass has quit IRC | 21:53 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Gerrit checks: trigger new patchset behavior https://review.opendev.org/707729 | 22:01 |
| *** sshnaidm is now known as sshnaidm|afk | 22:21 | |
| openstackgerrit | Merged zuul/zuul master: Report robot comments to gerrit https://review.opendev.org/707708 | 22:28 |
| tristanC | corvus: i've not seen that. could it be the inbound that drop long websocket connection? | 22:50 |
| *** Goneri has quit IRC | 22:51 | |
| SpamapS | corvus: yes, I've seen it a few times when my connection was wonky, like on a train tethered. | 22:51 |
| SpamapS | I believe that it's an exception bubbling up from the streaming code. | 22:51 |
| SpamapS | Should not blank an existing stream, and it should also not say "END OF STREAM" but probably something like a spinner. | 22:52 |
| corvus | tristanC: yeah, and that seems to jive with SpamapS's observation | 22:53 |
| corvus | i don't see that watching opendev streams, so maybe something about my terminal <-> GKE is the issue | 22:53 |
| SpamapS | websocket timeouts would explain it too | 22:53 |
| corvus | or the particular configuration of the load balancer | 22:53 |
| corvus | in fact, that's probably the place to start looking | 22:54 |
| SpamapS | Like, if the LB has a lower timeout than the server, you'll get dropped websocket conns when there's no output. | 22:54 |
| SpamapS | I raised my websocket timeout to 30 minutes. | 22:54 |
| corvus | SpamapS: yep. and i haven't paid attention to that at all :) | 22:54 |
| SpamapS | (on the origin).. but I don't know what it is on my LB. | 22:55 |
| corvus | i was just really happy i convinced GKE to make the right kind of load balancer at all. :) | 22:55 |
| SpamapS | yeah, maybe GKE's LB's have short websocket timeouts. | 22:55 |
| SpamapS | It might make sense to change the streaming code to send empty messages if nothing sends for more than 60s or something. | 22:55 |
| SpamapS | oh websocket has ping/pong | 22:56 |
| corvus | SpamapS: gke lb websocket timeout is 30s by default! | 22:56 |
| corvus | so that's almost certainly the problem. i'll look into fixing that, and hopefully we can also do a ping as well | 22:57 |
| SpamapS | yep | 22:57 |
| corvus | oh wow, it's actually not even an idle timeout. it's just 30s max connection length. | 23:04 |
| corvus | looks like *all* i need to do is create a BackendConfig then annotate the Service with the BackendConfig name then bob's my uncle | 23:08 |
| corvus | mordred: looks like https://gerrit-review.googlesource.com/c/zuul/jobs/+/254994 is working now | 23:08 |
| fungi | make sure you double-check with bob on that though | 23:09 |
| corvus | fungi: you are not going to let us leave today without a msft bob joke are you? | 23:10 |
| fungi | he was such a cute doggo though | 23:11 |
| fungi | i guess rover was the dog | 23:12 |
| *** paladox|UKInEU has quit IRC | 23:14 | |
| fungi | huh, i never knew (or conveniently forgot) this, per wikipedia anyway: "The typeface Comic Sans was created for (but not used in) Microsoft Bob." | 23:14 |
| fungi | i'm sure there's a good comic sans joke somewhere in that, it's just too late in the day for me to tease it out | 23:15 |
| corvus | fungi: really? i thought it was created, for, well, comics | 23:15 |
| *** paladox has joined #zuul | 23:17 | |
| fungi | wp's citation goes to an interview with the creator of the microsoft font file for it, anyway | 23:17 |
| fungi | who claims that's what he created it for | 23:17 |
| fungi | granted, this is on the internet, so who knows really | 23:18 |
| *** rlandy is now known as rlandy|bbl | 23:30 | |
| *** wxy-xiyuan has quit IRC | 23:35 | |
| corvus | mordred, paladox: https://ci.gerritcodereview.com/ is live | 23:45 |
| paladox | ohhhh | 23:45 |
| paladox | wooooo! | 23:45 |
| paladox | corvus <3 | 23:46 |
| fungi | it's running jobs!!! | 23:53 |
| * fungi stares intently at the test-gerrit-base output stream | 23:54 | |
| *** Defolos has quit IRC | 23:54 | |
| fungi | this is *so* cool | 23:54 |
| paladox | indeed, it's awesome! | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!