| @flaper87:matrix.org | > <@flaper87:matrix.org> Looking at the code, I see that the web client does remove the tenant scope: https://opendev.org/zuul/zuul/src/commit/a9146705148afba092e31a7013676f9c5661a2c6/web/src/api.js#L70-L73 | 06:51 |
|---|---|---|
| > | ||
| > This makes sense, and it seems to work when the correct (`/auth_callback`) url is added in the Google's configs. However, that callback URL does not seem to be working. That is, The browser is redirected to `https://<fqdn>/auth_callback` but it then gets stuck there in a infinite loop. If I add `/t/auth_callback` manually, then I can see the auth flow complete | ||
| - Any guidance on where the routing for `auth_callback` happens? | ||
| - Also, where is the `auth_callback` URL built? | ||
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 884807: dnm: Debug pragma w/ project-template https://review.opendev.org/c/zuul/zuul/+/884807 | 07:18 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 884807: dnm: Debug pragma w/ project-template https://review.opendev.org/c/zuul/zuul/+/884807 | 07:21 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 884807: dnm: Debug pragma w/ project-template https://review.opendev.org/c/zuul/zuul/+/884807 | 08:03 | |
| @omning:matrix.org | do I understand this correctly: if I lose Zookeeper data I lose stuff like SSH and secret keys, which means that all my existing secrets in the repositories are rendered unusable if scheduler regenerates new keys? | 08:54 |
| @omning:matrix.org | because this is something that happened to me and I wonder how bad my situation is going to be as ZooKeeper ended up with corrupted data and after backing it up I recreated a fresh scheduler and I'm now seeing all keys being generated | 08:55 |
| @omning:matrix.org | oh well, doesn't look like it did anything good anyway as I'm still getting `ERROR zuul.Scheduler Exception loading ZKObject <zuul.model.PipelineSummary object at 0x7fd0183847f0> at /zuul/tenant/xx/pipeline/gate/status` errors for all pipelines | 08:58 |
| @jjbeckman:matrix.org | Yes, I'm new to Ansible and thank you for the explanation. | 09:09 |
| > playbooks would need to be copied somewhere and wrapped by job definitions | ||
| Got it! | ||
| @omning:matrix.org | > <@omning:matrix.org> do I understand this correctly: if I lose Zookeeper data I lose stuff like SSH and secret keys, which means that all my existing secrets in the repositories are rendered unusable if scheduler regenerates new keys? | 09:12 |
| Just noticed that yes indeed:) I'm losing both private keys and nodepool data this way. Back to backup | ||
| @fungicide:matrix.org | we could probably do a better job of encouraging running https://zuul-ci.org/docs/zuul/latest/client.html#export-keys for backups, and then you can import those in a disaster | 11:55 |
| @omning:matrix.org | you think about it when it's far too late usually:) | 11:59 |
| @omning:matrix.org | in my case I finally managed to run the delete-state command, had an operator sticking its head out and this was the reason why I was getting errors after running it. You really need to take everything down before running it | 12:00 |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 884883: Don't set buildset files for non-live items ahead https://review.opendev.org/c/zuul/zuul/+/884883 | 12:31 | |
| @fungicide:matrix.org | but yeah, we deploy our schedulers with a cronjob that runs it on all of them daily: https://opendev.org/opendev/system-config/src/commit/11c604a/playbooks/roles/zuul-scheduler/tasks/main.yaml#L52-L59 | 13:00 |
| @fungicide:matrix.org | that way even if something happens to the entire zk cluster, we can still reimport the keys from any one of the scheduler servers | 13:01 |
| @omning:matrix.org | I love the idea, thank you | 13:39 |
| @rancher:matrix.org | > <@fungicide:matrix.org> that data is populated through browser-side javascript based on api calls, so could be your browser is failing to call the api (maybe firewalling or cors settings?) | 14:07 |
| I don't see any errors on the debugging console. The output in Firefox on both computers (with and without issues) is the same: | ||
| ``` | ||
| This page uses the non standard property “zoom”. Consider using calc() in the relevant property values, or using “transform” along with “transform-origin: 0 0”. neuronix-devops | ||
| ZuulAuthProvider rendering with params | ||
| Object { authority: "", client_id: "", scope: "", loadUserInfo: true } | ||
| ZuulAuthProvider.jsx:56:12 | ||
| Disabled service worker registerServiceWorker.js:24:12 | ||
| Asking other tabs for auth tokens ZuulAuthProvider.jsx:142:16 | ||
| No OpenIDConnect provider found auth.js:54:12 | ||
| ``` | ||
| Zuul is freshly cloned and only the configuration files (`main.yaml` and `zuul.conf`) are modified (they're the same on both computers). | ||
| @fungicide:matrix.org | maybe i misunderstood your observation. you said you're able to get dashboard content when browsing from one computer but not when browsing it from another? | 14:16 |
| @fungicide:matrix.org | i.e. you have one zuul deployment and are seeing different behavior depending on which client machine you connect to it from? | 14:17 |
| @fungicide:matrix.org | now it's starting to sound like two different zuul deployments, and you're seeing different behavior depending on which one you connect to | 14:21 |
| @fungicide:matrix.org | if that's the case, make sure the api is accessible, maybe by trying to fetch something like the tenants list data (example pulling it from our zuul): `curl -X GET "https://zuul.opendev.org/api/tenants" -H "accept: application/json"` | 14:24 |
| @flaper87:matrix.org | > <@flaper87:matrix.org> - Any guidance on where the routing for `auth_callback` happens? | 15:57 |
| > - Also, where is the `auth_callback` URL built? | ||
| Ok, looks like it's all handled in the jsx. The redirect URL is built in [ZuulAuthProvider.jsx](https://opendev.org/zuul/zuul/src/commit/a9146705148afba092e31a7013676f9c5661a2c6/web/src/ZuulAuthProvider.jsx#L62) | ||
| @flaper87:matrix.org | Can someone confirm that the right URL for Google Authenticator is `/auth_callback` instead of `t/<tenant>/auth_callback` as described in the docs? | 15:58 |
| @flaper87:matrix.org | The former seems to work well with Google but then it seems like it's not completing the Zuul login workflow (it works if I manually change the URL in the browser to point to `/t/<tenant>/auth_callback` | 15:59 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul] 884904: Add more info to merge mode selection errors https://review.opendev.org/c/zuul/zuul/+/884904 | 16:16 | |
| @clarkb:matrix.org | flaper87: I don't use google auth so can't say for sure but grepping around in the source for auth_callback the urls are generated using getHomepageUrl() and that trims off the tenant portion of the url. I don't see any special casing of this for google auth soI think /auth_callback and not /tenant/auth_callback is correct | 16:21 |
| @clarkb:matrix.org | it is possible that is buggy? Again I don't use that setup so can't speak directly to it just looks like the code and routers don't expect a tenant prefix for auth urls | 16:21 |
| @flaper87:matrix.org | > <@clarkb:matrix.org> flaper87: I don't use google auth so can't say for sure but grepping around in the source for auth_callback the urls are generated using getHomepageUrl() and that trims off the tenant portion of the url. I don't see any special casing of this for google auth soI think /auth_callback and not /tenant/auth_callback is correct | 16:23 |
| Indeed! Found the same and the authentication flow seems to work until the redirect. That is, Google matches the URL, allows to authenticate, and does the redirect. Once the flow is back on Zuul, then it gets stuck reloading the `auth_callback` | ||
| @flaper87:matrix.org | > <@clarkb:matrix.org> it is possible that is buggy? Again I don't use that setup so can't speak directly to it just looks like the code and routers don't expect a tenant prefix for auth urls | 16:25 |
| Maybe corvus knows better. I can tell for sure two things: | ||
| 1. The MSFT and Google docs differ. The former uses `/auth_callback` the latter says to use `/t/<tenant>/auth_callback`. | ||
| 2. When the flow redirects back to Zuul, if I change the URL manually to `/t/<tenant>/auth_callback` then the flow completes | ||
| @flaper87:matrix.org | Clark: thanks for checking 🙏 | 16:25 |
| @fungicide:matrix.org | if i were to insert a sentence or three into the docs recommending using `zuul-admin export-keys` to backup project keys from zk, where would be the best/most visible place to add it? the zk section of the components list? the places we talk about secrets and encryption are more user-facing than admin-facing | 16:29 |
| @flaper87:matrix.org | > <@fungicide:matrix.org> if i were to insert a sentence or three into the docs recommending using `zuul-admin export-keys` to backup project keys from zk, where would be the best/most visible place to add it? the zk section of the components list? the places we talk about secrets and encryption are more user-facing than admin-facing | 16:30 |
| Maybe https://zuul-ci.org/docs/zuul/latest/operation.html ? | ||
| @clarkb:matrix.org | I was just going to link to that | 16:31 |
| @clarkb:matrix.org | A section for ongoing operational tasks would probably work well there | 16:31 |
| @jim:acmegating.com | that or installation | 16:31 |
| @fungicide:matrix.org | yeah, that's probably why i'm having a dilemma. we lack an appropriate existing section. i'll add one. thanks! | 16:31 |
| @fungicide:matrix.org | right, installation also didn't really have an existing section where it would make sense | 16:32 |
| @flaper87:matrix.org | > <@jim:acmegating.com> that or installation | 16:32 |
| Installation would also work but I would probably have a link in installation that says "Now go and read how to operate this thing" | ||
| @flaper87:matrix.org | considering the backup should be do as a periodic task during the entire lifecycle of zuul | 16:33 |
| @flaper87:matrix.org | * considering the backup should be done as a periodic task during the entire lifecycle of zuul | 16:33 |
| @fungicide:matrix.org | yeah, maybe there could be components of both. recommend setting up periodic exports in the installation doc, recommend backing them up in the operation doc | 16:33 |
| @fungicide:matrix.org | or talk about importing them as part of a disaster recovery plan | 16:34 |
| @fungicide:matrix.org | if i write something about disaster recovery, is it true that the only non-ephemeral state that's critical to be able to restore is the project (secret and ssh) keys? or is there other data that should be backed up as well? | 16:36 |
| @jim:acmegating.com | that's the only critical thing. nodepool images might be a time saver, but not required, and not appropriate for the zuul docs. | 16:38 |
| @jim:acmegating.com | (though a pointer from zuul to nodepool might be appropriate) | 16:38 |
| @fungicide:matrix.org | thanks, that's what i thought. for non-recreatable data (at least if you don't want to have to replace all your existing secrets and ssh authorizations), it's just the keystore content then | 16:39 |
| @flaper87:matrix.org | > <@flaper87:matrix.org> Maybe corvus knows better. I can tell for sure two things: | 16:43 |
| > | ||
| > 1. The MSFT and Google docs differ. The former uses `/auth_callback` the latter says to use `/t/<tenant>/auth_callback`. | ||
| > 2. When the flow redirects back to Zuul, if I change the URL manually to `/t/<tenant>/auth_callback` then the flow completes | ||
| Ok, it looks like zuul stores the `t/<tenant>/auth_callback` url in the browser session to then use it during the oidc login. So, either I have something not configured properly, or there's a bug somewhere | ||
| @flaper87:matrix.org | > <@flaper87:matrix.org> Ok, it looks like zuul stores the `t/<tenant>/auth_callback` url in the browser session to then use it during the oidc login. So, either I have something not configured properly, or there's a bug somewhere | 16:51 |
| nvm, this is just the redirect_target that is set in the browser session | ||
| @flaper87:matrix.org | Ok, I'm starting to consider just keeping the oauth-proxy I have in front of Zuul. Is there a way to give admin to anonymous users? I basically would like for everyone to be able to queue jobs, dequeue jobs, etc. | 16:57 |
| @flaper87:matrix.org | * Ok, I'm starting to consider just keeping the oauth-proxy I have in front of Zuul. Is there a way to give admin to anonymous users? I basically would like for everyone to be able to queue jobs, dequeue jobs, etc. At this point, it'd be better if they don't have to login through Zuul | 16:57 |
| @fungicide:matrix.org | flaper87: depending on your budget for this project, it might also be something acmegating.com would be able to dig into and get working or documented more clearly | 17:00 |
| @fungicide:matrix.org | i don't think the auth layer has really widespread adoption yet, so there may be corner cases lurking in different identity providers | 17:00 |
| @flaper87:matrix.org | > <@fungicide:matrix.org> flaper87: depending on your budget for this project, it might also be something acmegating.com would be able to dig into and get working or documented more clearly | 17:03 |
| For now, it's a bit "contained" given the project is still smallEngaging with acmegating.com has been considered for some other things but still need to get there. Thanks for the suggestion, tho. | ||
| @jim:acmegating.com | i can confirm that microsoft and keycloak work well; i've used them in all configurations recently. it may have been a while since anyone has set up google. | 17:03 |
| @flaper87:matrix.org | > <@jim:acmegating.com> i can confirm that microsoft and keycloak work well; i've used them in all configurations recently. it may have been a while since anyone has set up google. | 17:04 |
| Thanks, this helps. I wonder if some more recent changes broke the Google workflow. It looks like only the last bit of the redirect is not really working | ||
| @jim:acmegating.com | there's no option to allow admin access for anonymous users afaik | 17:04 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 884910: Document backing up and restoring project keys https://review.opendev.org/c/zuul/zuul/+/884910 | 17:21 | |
| @fungicide:matrix.org | omning: please let me know if that ^ would have caught your attention in the docs and averted the problem in your case | 17:22 |
| @jim:acmegating.com | fungi: might be worth mentioning that the export is still encrypted with the keystore password, so that folks can accurately evaluate the safeguards needed for backup. | 17:26 |
| @fungicide:matrix.org | corvus: oh, thanks i'll throw that in for good measure | 17:27 |
| @jim:acmegating.com | (i should say, the individual keys within the export are still so encrypted) | 17:27 |
| @fungicide:matrix.org | yep, will clarify | 17:27 |
| @jim:acmegating.com | (also, so, erm, don't lose the keystore password :) | 17:28 |
| @fungicide:matrix.org | though i'm also lazily waiting for zuul to let me know what i messed up with my rst/sphinx bits so will push a new revision once it gets back to me | 17:28 |
| @clarkb:matrix.org | ya losing the keystore passwd is effectively the same as deleting the data as is changing the value | 17:28 |
| @clarkb:matrix.org | I think someone ran into this a while back | 17:28 |
| @fungicide:matrix.org | right, definitely merits a mention | 17:30 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 884910: Document backing up and restoring project keys https://review.opendev.org/c/zuul/zuul/+/884910 | 19:23 | |
| @fungicide:matrix.org | corvus: Clark: ^ new revision mentions the role of the keystore.password in all of this | 19:24 |
| -@gerrit:opendev.org- Tony Breeds proposed: [zuul/zuul-jobs] 884935: [configure-mirrors] Allow per distribution disabling of mirrors https://review.opendev.org/c/zuul/zuul-jobs/+/884935 | 19:54 | |
| @clarkb:matrix.org | fungi: in that change you are using external link format but those links are all in the same site? or is internal per file only? | 20:07 |
| @clarkb:matrix.org | fungi: looks like it is failing on the keystore.password ref too | 20:09 |
| @fungicide:matrix.org | Clark: it's using `:ref:` syntax which is for references between different files in the same set of documents, but yes i have a feeling `.` isn't allowed in a refname | 20:16 |
| -@gerrit:opendev.org- Jeremy Stanley https://matrix.to/#/@fungicide:matrix.org proposed: [zuul/zuul] 884910: Document backing up and restoring project keys https://review.opendev.org/c/zuul/zuul/+/884910 | 20:18 | |
| @fungicide:matrix.org | that switched is to `-` | 20:18 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 20:26 | |
| - [zuul/zuul] 884939: Web: convert config errors to table https://review.opendev.org/c/zuul/zuul/+/884939 | ||
| - [zuul/zuul] 884940: Web: remove config error drawer https://review.opendev.org/c/zuul/zuul/+/884940 | ||
| @jim:acmegating.com | Clarkfungi since you mentioned that openstack is interested in cleaning up config errors, those may be useful ^ | 20:28 |
| @jim:acmegating.com | the preview will be available soon, but here's a screenshot | 20:29 |
| @jim:acmegating.com | also, wow that's really exciting when i paste a screenshot from a high dpi display | 20:30 |
| @jim:acmegating.com | i have more work planned in this area, but that's a start | 20:31 |
| @clarkb:matrix.org | cool I'll take a look soon | 20:31 |
| @fungicide:matrix.org | looks nice! | 20:32 |
| @jim:acmegating.com | currently everything there is severity=error and most errors are name=Unknown (but you can see a pipeline error in the screenshot) -- thus the more work planned :) | 20:33 |
| @clarkb:matrix.org | fungi: the reason I asked about the links is that https://docutils.sourceforge.io/docs/user/rst/quickref.html#hyperlink-targets seems to only document that method with a uri but you are using the tags instead. Maybe it works nad that doc is old or somethng | 20:34 |
| @clarkb:matrix.org | corvus: jrosser indicated yesterday that it wasn't clear how to find the errors listing and we may want ot make it more prominent. One suggestion was a link on the tenant list page to the tenant errors page | 20:35 |
| @fungicide:matrix.org | Clark: it's a sphinx link role not a rst hyperlink target specifically: https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-ref | 20:36 |
| @clarkb:matrix.org | fungi: oh I see | 20:37 |
| @jim:acmegating.com | Clark: we don't have infinite space on the tenants page, so we should be selective about what we put there; but having said that, i think we should actually replace the "queue" column with "errors". i think that operators or multi-tenant users are much more likely to be interested in that. | 20:39 |
| @clarkb:matrix.org | ++ | 20:40 |
| @fungicide:matrix.org | makes sense to me. could also consider switching some of that to pictograms in order to reduce space, though i kinda dislike the modern trend of webpages which just use icons to link to things, after all it's called hyperTEXT for a reason | 20:42 |
| @jim:acmegating.com | yeah, a bitwise errors vs no errors icon would also be an option i think. but i agree, it's less optimal | 20:43 |
| @fungicide:matrix.org | error count would make sense there if that's what you were suggesting | 20:43 |
| @jim:acmegating.com | yep | 20:43 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: | 21:07 | |
| - [zuul/zuul] 884939: Web: convert config errors to table https://review.opendev.org/c/zuul/zuul/+/884939 | ||
| - [zuul/zuul] 884940: Web: remove config error drawer https://review.opendev.org/c/zuul/zuul/+/884940 | ||
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!