| -@gerrit:opendev.org- Simon Westphahl proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/nodepool] 885099: Use asynchronous fetch operations for tree cache https://review.opendev.org/c/zuul/nodepool/+/885099 | 06:31 | |
| -@gerrit:opendev.org- Guillaume Chauvel proposed: | 09:31 | |
| - [zuul/zuul] 875057: quick-start: run additional tutorials using var run_playbooks https://review.opendev.org/c/zuul/zuul/+/875057 | ||
| - [zuul/zuul] 875639: quick-start: Change Gerrit wait method & increase Scheduler gerrit wait time https://review.opendev.org/c/zuul/zuul/+/875639 | ||
| - [zuul/zuul] 875640: quick-start: recheck as PATCHSET_LEVEL comment https://review.opendev.org/c/zuul/zuul/+/875640 | ||
| - [zuul/zuul] 732067: tutorial: Add "gate your first patch" https://review.opendev.org/c/zuul/zuul/+/732067 | ||
| @rancher:matrix.org | > <@fungicide:matrix.org> now it's starting to sound like two different zuul deployments, and you're seeing different behavior depending on which one you connect to | 11:04 |
|---|---|---|
| Exactly that. The given `curl` command works and gives the same output on both computers. :( | ||
| ``` | ||
| [{"name": "tenant", "projects": 2, "queue": 0}] | ||
| ``` | ||
| @rancher:matrix.org | > <@fungicide:matrix.org> now it's starting to sound like two different zuul deployments, and you're seeing different behavior depending on which one you connect to | 11:05 |
| * Exactly that. The given `curl` command works and gives the same output on both Zuul installations. :( | ||
| ``` | ||
| [{"name": "tenant", "projects": 2, "queue": 0}] | ||
| ``` | ||
| @fungicide:matrix.org | Rancher: and that's using curl to test from the remote machine that was unable to see dashboard content rendered for one of them? next guess, maybe the javascript files used to render the dashboard aren't being returned? are you serving those and similar static files directly from the filesystem with your webserver, or through the zuul-web port? for an example, here's the apache vhost config template we deploy for our zuul-web servers in opendev: https://opendev.org/opendev/system-config/src/branch/master/playbooks/roles/zuul-web/templates/zuul.vhost.j2 | 12:11 |
| @rancher:matrix.org | Yeah, I tried the curl command both on the local machine where "non-working" Zuul is installed and a remote one. | 12:14 |
| @fungicide:matrix.org | it looks like we're proxying ours to 9000/tcp on the loopback interface, and then we have caching set up for /static/* to improve performance | 12:14 |
| @rancher:matrix.org | I'm not sure about the other question. I used the default config from your repository located at: https://opendev.org/zuul/zuul | 12:18 |
| Nothing's changed except the `zuul.conf` and `main.yaml` files. | ||
| @fungicide:matrix.org | Rancher: looking back at your initial question, you say you get no content other than a page heading when opening a project view in the dashboard. are the other views (the tenant list on the main page for example, or the status page) populated with more content? | 12:20 |
| @fungicide:matrix.org | i think we're seeing it in opendev too | 12:25 |
| @fungicide:matrix.org | https://zuul.opendev.org/t/zuul/project/opendev.org/zuul/zuul | 12:25 |
| @rancher:matrix.org | > Can someone take a look at this: https://privatebin.net/?53abf0f3d1708270#DX11H95THXJoZf2Z65BGs7wVmWr3RMXv2J6d8xKxnZRn | 12:25 |
| > | ||
| > Logs are included. Is the syntax okay? | ||
| > | ||
| > TLDR: The config project files in my repositories (local and remote) are copied from the Acme Gating repository (which is the only method that works), but all of them end up with the `GetProjectMetadata` error. | ||
| Yes, tenants are visible (I've got only one) as well as pipelines in the Status page. But pipelines only work when I load them via the remote Acme Gating repository on GitHub; the identical config files copied from that repository don't work on my GitLab and GitHub instance (local and web versions), i.e. nothing is shown in the Status page. I quoted my message from the other day with more info and logs. | ||
| @fungicide:matrix.org | that just displays the project name (zuul/zuul) | 12:25 |
| @rancher:matrix.org | > <@fungicide:matrix.org> i think we're seeing it in opendev too | 12:26 |
| Yes, that's it, for each project I have.. | ||
| @rancher:matrix.org | > <@fungicide:matrix.org> i think we're seeing it in opendev too | 12:26 |
| * Yes, that's it, for each project I have. | ||
| @fungicide:matrix.org | i agree it seems like something might be incomplete or mis-plumbed for the project detail view | 12:28 |
| @fungicide:matrix.org | is it possible your two deployments showing different behaviors are running slightly different versions of zuul? | 12:28 |
| @rancher:matrix.org | No, I cloned them several times in the same day, so source files are identical. I also tried the same Ubuntu version as the one that runs the working Zuul, but it's the same behaviour. | 12:29 |
| @fungicide:matrix.org | ours, in theory, is running from the latest official zuul container images as of roughly one week ago | 12:29 |
| @rancher:matrix.org | I cloned them yesterday. | 12:29 |
| @fungicide:matrix.org | we perform automated weekly rolling updates of opendev's zuul deployment early utc on saturdays, so we're nearly due to update again | 12:30 |
| @fungicide:matrix.org | unfortunately, i'm not very good at javascript debugging. my eyes glaze over when i try to reverse-engineer what i'm seeing in the browser, and i'm not finding anything that looks like outbound api calls (it doesn't help that the script names are pure gibberish) | 12:38 |
| @fungicide:matrix.org | i would expect it to pull https://zuul.opendev.org/api/tenant/zuul/project/zuul/zuul and use that to populate the page content | 12:39 |
| @fungicide:matrix.org | and the api is definitely returning solid data about that project | 12:40 |
| @rancher:matrix.org | Okay, thanks. Do you know what could be the problem with loading a config project: https://privatebin.net/?53abf0f3d1708270#DX11H95THXJoZf2Z65BGs7wVmWr3RMXv2J6d8xKxnZRn | 12:41 |
| That one bothers me more. | ||
| @fungicide:matrix.org | oh, right this was the one where you were running into problems trying to use the git connection driver to load your config project from the local filesystem on the scheduler? | 12:50 |
| @fungicide:matrix.org | and you're saying it works when you use an `https://` url to the config project's git repository but not when you use a `file://` url? | 12:51 |
| @fungicide:matrix.org | looking back at the project page issue, the oldest dashboard preview build we have archived is from may 22, and it's displaying the same blank project details, so i'm guessing whatever regressed happened longer ago than that | 12:59 |
| @rancher:matrix.org | That was the other user, but yeah, inspired by him, i also tried `file://` URL, but nothing works except using the `git` driver with the Acme Gating repository located at: https://github.com/acmegating | 12:59 |
| As I said, using the `git` driver and/or `gitlab`/`github` with the same config files from the Acme Gating repository on my GitLab/Github servers doesn't work. Nothing's shown in the Status page and there's the `GetProjectMetadata` error when opening projects in the Projects section. | ||
| @fungicide:matrix.org | okay, so remote `https://` to a gitlab.com repository url with the git driver is also not working in this case? | 13:01 |
| @rancher:matrix.org | > <@fungicide:matrix.org> okay, so remote `https://` to a gitlab.com repository url with the git driver is also not working in this case? | 13:02 |
| It doesn't. | ||
| @fungicide:matrix.org | as well as remote `http://` to a gitlab deployment on your network | 13:02 |
| @rancher:matrix.org | Yep. | 13:02 |
| @fungicide:matrix.org | but from the zuul server you're able to clone the repositories remotely via those same urls | 13:03 |
| @rancher:matrix.org | Is it relevant what I enter as a `server` parameter? | 13:03 |
| @rancher:matrix.org | I used `baseurl` for full URLs. | 13:03 |
| @rancher:matrix.org | `http` | 13:03 |
| @rancher:matrix.org | > <@fungicide:matrix.org> but from the zuul server you're able to clone the repositories remotely via those same urls | 13:05 |
| From the machine that runs the Zuul server? Yes. | ||
| @rancher:matrix.org | * That was another user, but yeah, inspired by him, i also tried `file://` URL, but nothing works except using the `git` driver with the Acme Gating repository located at: https://github.com/acmegating | 13:05 |
| As I said, using the `git` driver and/or `gitlab`/`github` with the same config files from the Acme Gating repository on my GitLab/Github servers doesn't work. Nothing's shown in the Status page and there's the `GetProjectMetadata` error when opening projects in the Projects section. | ||
| @fungicide:matrix.org | also the service logs in your paste say zuul-scheduler and zuul-web successfully loaded configuration from the main branch of your zuul-config repository | 13:08 |
| @rancher:matrix.org | That's what's strange. | 13:08 |
| @rancher:matrix.org | Maybe it's another interface issue. | 13:08 |
| @rancher:matrix.org | * Maybe it's another web interface issue. | 13:09 |
| @fungicide:matrix.org | Rancher: that's a good theory. try hitting the api method for the project details instead and see if the json contains branch info et cetera | 13:17 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 885140: DNM: bisecting empty project view https://review.opendev.org/c/zuul/zuul/+/885140 | 13:38 | |
| @rancher:matrix.org | > <@fungicide:matrix.org> Rancher: that's a good theory. try hitting the api method for the project details instead and see if the json contains branch info et cetera | 14:05 |
| Can you provide an example? Sorry, I'm new to this. | ||
| @amberdev:matrix.org | After setting up demo via `podman kube play deployment.yaml` and executing last instruction (type `recheck` for test1) from the Quick-Start, it leads to `Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.`. Currently not sure whats happening there. Everything else seems to be working fine. | 14:05 |
| @fungicide:matrix.org | Rancher: sorry, it's the api path mentioned in your error log, so http://localhost:9000/api/tenant/tenant/project/gitlab.com/saneladinic/zuul-config though your log says that returned an http 500 (internal server) error | 14:22 |
| @clarkb:matrix.org | > <@amberdev:matrix.org> After setting up demo via `podman kube play deployment.yaml` and executing last instruction (type `recheck` for test1) from the Quick-Start, it leads to `Merge Failed. This change or one of its cross-repo dependencies was unable to be automatically merged with the current state of its repository. Please rebase the change and upload a new patchset.`. Currently not sure whats happening there. Everything else seems to be working fine. | 14:23 |
| Check your merger and executor logs this is where the git merges occur and the error there will hopefully identify the problem | ||
| @amberdev:matrix.org | Yeah... already looking into that. Tnx. | 14:23 |
| @amberdev:matrix.org | `fatal: unable to access 'https://opendev.org/zuul/zuul-jobs/': Could not resolve proxy: https_proxy'` | 14:24 |
| @amberdev:matrix.org | Will fix thix | 14:25 |
| @amberdev:matrix.org | * Will fix this first | 14:25 |
| @fungicide:matrix.org | as far as the empty project detail view in the dashboard we're also seeing in opendev, my attempt to roll back to the 8.2.0 state from february is still showing the same with the built preview, so it may have be due to a change in the api rather than in the javascript itself (or maybe this is just not an effective way to bisect the start of the problem, or maybe it regressed before 8.2.0?) | 14:27 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 885140: DNM: bisecting empty project view https://review.opendev.org/c/zuul/zuul/+/885140 | 14:31 | |
| @fungicide:matrix.org | seeing if 8.1.0 from january shows anything different | 14:31 |
| @fungicide:matrix.org | yeah, still the same, i'm not convinced this is an effective means of identifying the cause | 14:59 |
| @jim:acmegating.com | fungi: https://review.opendev.org/879440 is the direct cause | 15:06 |
| @fungicide:matrix.org | corvus: huh, i wondered about that one since it does touch the project data, but it wasn't immediately obvious why it would stop the dashboard from loading it | 15:07 |
| @fungicide:matrix.org | problem with the null value for default-branch? | 15:07 |
| @jim:acmegating.com | the change is on the api end, not the js side. unfortunately the original implementation of that page took a shortcut and inferred information about projects from the default_branch attribute. the fix is probably going to be to actually implement the necessary api data to render that page as desired. | 15:08 |
| @fungicide:matrix.org | yep, that explains why rolling back to an earlier dashboard didn't seem to make any difference | 15:09 |
| @jim:acmegating.com | https://opendev.org/zuul/zuul/src/branch/master/web/src/actions/project.js#L25 | 15:09 |
| @jim:acmegating.com | fungi: note the comments there on lines 26, 27, and 32 | 15:09 |
| @fungicide:matrix.org | i'm mostly baffled as to why i couldn't find any errors coming from the js in my browser though | 15:09 |
| @jim:acmegating.com | i think because the error happens in an async background thread, so that function just bombs and doesn't report anything | 15:10 |
| @fungicide:matrix.org | aha | 15:10 |
| @jim:acmegating.com | error handling for that should be possible, but i think it's extra work to add a catch that propogates it up somewhere useful | 15:11 |
| @amberdev:matrix.org | I would suggest not to use commonly used service ports in Quick-Start examples. Like 8080, 8000 and such. Because, depending on the local network setup those ports most likely are already used by other services. I mean... developer who went so far that he is looking into Zuul most likely have some things running. :) | 15:21 |
| @amberdev:matrix.org | Otherwise, in order to follow quick start, you need to wipe out entire stack, or modify quick-start source. | 15:22 |
| @amberdev:matrix.org | Just an suggestion nobody asked for. :) | 15:22 |
| @fungicide:matrix.org | you may have to do that anyway no matter what ports are used by default, if you're not testing it on a dedicated server/vm | 15:23 |
| @amberdev:matrix.org | Yeah... i have "dirty, throw-away workstation" where I am working on. I run and test gazillion of things there. Zuul is not the simplest system to dive into. And if you add conflicting common port, it's makes thing bit complicated to follow. | 15:25 |
| I'm not complaining. | ||
| @amberdev:matrix.org | In case of zuul i prefixed all port with 92. 9280, 9281, 9205 and so on | 15:26 |
| @amberdev:matrix.org | Just a random number | 15:27 |
| @clarkb:matrix.org | Personally I think fixed ports and adjusting to accommodate local conflicts is far less confusing than using 0 and trying to synchronize that across everything and ending up with inconsistent results each time you deploy | 15:30 |
| @clarkb:matrix.org | its also an opportunity for people to look and see what the network topology looks like | 15:31 |
| @jim:acmegating.com | perhaps an inventory of used ports in the documentation would help | 15:32 |
| @jim:acmegating.com | * perhaps an inventory of used ports in the tutorial would help | 15:32 |
| @amberdev:matrix.org | ^ | 15:39 |
| @amberdev:matrix.org | At least. I's definitely a huge issue. I'm talking more from a perspective to bring more community. And simplifying/clarifying the Demo IMHO is one tool to use. | 15:42 |
| @amberdev:matrix.org | * At least. It's definitely not a huge issue. I'm talking more from a perspective to bring more community. And simplifying/clarifying the Demo IMHO is one tool to use. | 15:42 |
| @amberdev:matrix.org | To eliminate friction as much as possible. | 15:43 |
| @clarkb:matrix.org | yup documentation can only help | 15:43 |
| @clarkb:matrix.org | But when it comes to choosing ports your two options are typically 1) use fixed ports and risk conflicts or 2) use 0 and coordinate who owns what port. And I think the tradeoffs with 1) are better than with 2) | 15:44 |
| @amberdev:matrix.org | Fixed ports is fine. But random (not common) fixed ports is even better. It's way easier to follow what is talking to what, and possibility of collision is low. | 15:50 |
| @amberdev:matrix.org | * Fixed ports are fine. But random (not common) fixed ports is even better. It's way easier to follow what is talking to what, and possibility of collision is low. | 15:50 |
| @amberdev:matrix.org | * Fixed ports are fine. But random (not common) fixed ports are even better. It's way easier to follow what is talking to what, and possibility of collision is low. | 15:50 |
| @fungicide:matrix.org | swest: corvus: so should we temporarily revert https://review.opendev.org/879440 since it only merged after the last release, or try to roll forward with a fix for the project detail view in the dashboard? i'm willing to attempt the latter, but my grasp of the javascript parts is limited so i'd be relying heavily on other reviewers for that | 16:07 |
| @fungicide:matrix.org | i guess i would start with trying to make a new test that fails with that change but works without it | 16:08 |
| @fungicide:matrix.org | assuming i can find where the other dashboard tests are | 16:11 |
| @jim:acmegating.com | fungi: i think 440 is something we should avoid rolling back (it's a pretty important fix and the reverse behavior change could be problematic for folks). meanwhile, the annoyance of the broken projects page is small. i think we should just roll forward and expect it may take us a few days to get a fix. | 16:12 |
| @fungicide:matrix.org | thanks, that's where i was leaning as well | 16:12 |
| @fungicide:matrix.org | as this is mainly a cosmetic issue | 16:12 |
| @jim:acmegating.com | this js code is testable, but i believe has no tests. but i think a good fix for it may eliminate most of that function anyway. | 16:13 |
| @jim:acmegating.com | (basically: update zuul-web to provide the info that the js actually needs in a usable format) | 16:14 |
| @fungicide:matrix.org | specifically whether or not the project is a template? | 16:14 |
| @jim:acmegating.com | yep | 16:15 |
| @jim:acmegating.com | (but there's also some transforms it does; i think we need to understand that) | 16:16 |
| @jim:acmegating.com | i think the whole point is to merge templates into the project stanza that includes them, but i'm not 100% on that | 16:16 |
| @jim:acmegating.com | so we may want to keep that in js, but just add a "template" bit to the rest api. | 16:17 |
| @jim:acmegating.com | and if so, we could probably add a test, but i wouldn't require it -- because it would use mock data that we'd have to update manually anyway. since it wouldn't be an effective regression test, i probably wouldn't bother. | 16:18 |
| @jim:acmegating.com | (a better regression test would be a python unit test that did approximately the same thing as the js, or at least, asserted that what it needed was present) | 16:19 |
| @fungicide:matrix.org | yeah, i can take a stab at that since it's mostly python-side work and a simple adjustment to a condition in the js (hopefully). not sure i could engineer dashboard testing from scratch with the time i have available | 16:19 |
| @jim:acmegating.com | so maybe that's the way to go; add the template bit; add a py unit test to assert it and the structure, then update the js to use the template bit. and otherwise things stay more or less the same. | 16:19 |
| @jim:acmegating.com | fungi: ++ | 16:19 |
| @fungicide:matrix.org | trying to trace back to where the templates get merged in, it seems that zuul.model.Layout.getAllProjectConfigs() returns the direct configs and also iterates over the templates list attribute for each config and appends the results of getProjectTemplates(), so i guess it would make more sense to extend the project config schema to include something like an is_template boolean attribute to the zuul.model.ProjectConfig class and then toggle that at project-template loading so it carries all the way through? | 16:54 |
| @fungicide:matrix.org | sadly, this is the deepest i've delved into zuul internals on my own in quite some time | 16:55 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 885155: Add config metadata to identify project-templates https://review.opendev.org/c/zuul/zuul/+/885155 | 17:15 | |
| @fungicide:matrix.org | that's probably all sorts of wrong, but hopefully folks can nudge me in a better direction | 17:16 |
| @fungicide:matrix.org | well, i suppose it's not entirely terrible. it does seem to fix the problem judging from the dashboard build, though if you look at a project with templates applied it lists the name of the project the templates came from. i guess that works (and is probably what we had before the regression?) | 17:45 |
| @fungicide:matrix.org | oh, nevermind, the dashboard build is pointing at the production opendev api, so everything is going to be treated as a non-template | 17:48 |
| @amberdev:matrix.org | In any case will leave it there - https://pastebin.com/sjhvNJ6i | 21:25 |
| There are some missconfiguration on my side, but ... this error is rapidly flooded dozens of times per second. Is that expected? | ||
| @clarkb:matrix.org | I don't think that error is expected. Seems like you've set an https proxy that it cannot resolve | 21:28 |
| @clarkb:matrix.org | That is happening outside of zuul and in git if I read things ocrrectly | 21:29 |
| @amberdev:matrix.org | Yeah... that is fine. I'm more like about the fact of that rapid flooding. Scheduler is running in the container as per Quick-Start. IDK... mby there should be some controlled retry logic. Have no idea. Could it lead in some unordinary expenses if some would run such accidental thing in the cloud? | 21:34 |
| @clarkb:matrix.org | The git driver necessarily polls so its happening over and over again because each poll is failing | 21:35 |
| @clarkb:matrix.org | I don't think it would result in any extra chargers because nothing is happening. | 21:35 |
| @amberdev:matrix.org | Ok. I'm just learning this thing. Leaving a trail "from a newbie perspective". | 21:36 |
| @amberdev:matrix.org | * Ok. I'm just learning this thing. Leaving a trail "as from a newbie perspective". | 21:37 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!