| *** Defolos has quit IRC | 00:10 | |
| *** Defolos has joined #zuul | 00:12 | |
| *** Defolos has quit IRC | 00:19 | |
| *** smyers has quit IRC | 00:49 | |
| *** smyers has joined #zuul | 00:56 | |
| *** rlandy has joined #zuul | 01:32 | |
| *** rlandy has quit IRC | 01:33 | |
| *** saneax has quit IRC | 02:58 | |
| *** bhavikdbavishi has joined #zuul | 03:02 | |
| *** bhavikdbavishi has quit IRC | 03:45 | |
| *** jamesmcarthur has joined #zuul | 04:44 | |
| *** jamesmcarthur has quit IRC | 04:46 | |
| *** jamesmcarthur has joined #zuul | 04:46 | |
| *** dpawlik has joined #zuul | 05:21 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #zuul | 05:34 | |
| *** saneax has joined #zuul | 05:57 | |
| *** jamesmcarthur has quit IRC | 06:22 | |
| *** felixedel has joined #zuul | 06:26 | |
| *** dpawlik has quit IRC | 06:31 | |
| *** dpawlik has joined #zuul | 06:31 | |
| *** dpawlik has quit IRC | 06:42 | |
| *** dpawlik has joined #zuul | 06:48 | |
| *** AJaeger has quit IRC | 06:57 | |
| *** gothicmindfood has quit IRC | 06:59 | |
| *** AJaeger has joined #zuul | 07:02 | |
| *** Defolos has joined #zuul | 07:41 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 08:15 |
|---|---|---|
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 08:18 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 08:21 |
| mnaser | appreciate reviews on very trivial https://review.opendev.org/#/q/topic:install-docker-compose :) | 08:50 |
| *** felixedel has quit IRC | 08:50 | |
| *** tosky has joined #zuul | 08:51 | |
| *** jpena|off is now known as jpena | 08:52 | |
| *** avass has joined #zuul | 08:59 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure: remove connectors burden and simplify code https://review.opendev.org/696134 | 09:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure - Refresh token and recheck sign when token regenerated https://review.opendev.org/698208 | 09:05 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: A Zuul reporter for Elasticsearch https://review.opendev.org/644927 | 09:15 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: Add reference pipelines file for Github driver https://review.opendev.org/672712 | 09:17 |
| mnaser | discussion topic: we use swift for our logging right now and obviously things are uploaded in a public bucket. i'd like to know if there's a "generic" way we can come up with to be able to expose signed tempurls inside the zuul webui (and really, this applies for s3/gcs/swift logs): https://boto3.amazonaws.com/v1/documentation/api/latest/guide/s3-presigned-urls.html , | 09:19 |
| mnaser | https://cloud.google.com/storage/docs/access-control/signed-urls , https://docs.openstack.org/swift/latest/api/temporary_url_middleware.html | 09:19 |
| *** felixedel has joined #zuul | 09:23 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: URLTrigger driver time based https://review.opendev.org/635567 | 09:38 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Fix evaluation of range file_comments https://review.opendev.org/708076 | 09:44 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Clarify RequirementError message https://review.opendev.org/707620 | 09:49 |
| *** felixedel has quit IRC | 11:33 | |
| *** felixedel has joined #zuul | 11:39 | |
| zbr | any change to get the build history link added https://review.opendev.org/#/c/705049/ ? | 11:44 |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Fix github app authentication to work with checks API endpoints (2) https://review.opendev.org/708109 | 11:59 |
| *** rfolco has joined #zuul | 12:04 | |
| *** ianychoi has joined #zuul | 12:22 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: Report retried builds via sql reporter. https://review.opendev.org/633501 | 12:24 |
| *** jpena is now known as jpena|lunch | 12:35 | |
| *** rlandy has joined #zuul | 12:55 | |
| *** jpena|lunch is now known as jpena | 13:34 | |
| *** saneax has quit IRC | 13:44 | |
| *** Goneri has joined #zuul | 13:53 | |
| fungi | mnaser: i assume the use case is for tenants/deployments which want to keep their log data private but served from a public object store? | 14:04 |
| mnaser | fungi: exactly | 14:04 |
| mnaser | fungi: or really we can figure out to make this something that can be replicated to even storing log files on a static server too, just an idea that encapsulates logs within zuul | 14:04 |
| mnaser | i mean, the more crazy wild idea would be to entirely pipe all logs through zuul and eliminate all object store access but that might be too much traffic through zuul | 14:05 |
| fungi | a lower-tech alternative might be to assign "unguessable" tokens (e.g., type 4 uuid) in the urls themselves | 14:08 |
| fungi | that doesn't prevent an unauthorized user from getting access to a log if someone leaks url, but then again the same could be said of authentication credentials. however, individual credentials are easier to revoke than a single shared key | 14:20 |
| fungi | so i agree per-request tempurls would be the more secure option, though i expect that would require something to plumb tempurl creation creds through the web dashboard/rest api | 14:21 |
| mnaser | fungi: yeah i'm thinking it might be hard to fully decouple this | 14:22 |
| mnaser | fungi: thinking out loud i guess you could have the role create tempurl and return that to zuul, and then make sure zuul is behind $something_authenticated | 14:23 |
| mnaser | or we might need to write a zuul logserver :X | 14:24 |
| fungi | what role? the tempurl would have to be created when a user requests access to the log, right? | 14:24 |
| fungi | not at job runtime | 14:24 |
| mnaser | fungi: the idea is that for example the swift logs role would return a url to zuul that contains a tempurl | 14:25 |
| mnaser | with no expiry time | 14:25 |
| fungi | how does that help? | 14:25 |
| mnaser | in a way that's kinda like your idea of "very unguessable url" | 14:25 |
| fungi | yeah, it's really no better than embedding a shared secret in the log path | 14:25 |
| fungi | either way, if the url gets leaked, anyone can access that log | 14:25 |
| mnaser | i feel like a small static server microservice that integrates with zuul might really make sense here | 14:26 |
| mnaser | and then you'd just make sure everything inside zuul is authenticated (in my case using oauth2 to protect the tenant endpoint) | 14:26 |
| fungi | the main thing that loses is direct log serving from the object store. your authenticated proxy will end up as a funnel with all requests going through it | 14:26 |
| mnaser | fungi: unless said service could do something similar to how docker does, where it can serve stuff directly, but when using a block storage system, it will redirect | 14:27 |
| fungi | but if it redirects to a consistent url, you're back to the same problem again | 14:27 |
| mnaser | fungi: well i'm assuming this system has credentials and generates a tempurl for 2 minutes for example | 14:28 |
| fungi | the sensitive url becomes whatever you've redirected the client to | 14:28 |
| fungi | ahh, yeah as long as it's a new tempurl per request that could work | 14:28 |
| mordred | yeah - the main issue is that by the time the url gets to zuul-web there's no longer any concept of swift or not - it's just a url | 14:28 |
| mnaser | whats interesting is you can actually use tempurls for uploads too (also on s3 and gcs) | 14:28 |
| mordred | mnaser: in fact, we do! | 14:29 |
| mnaser | so if we come up with some sort of 'log server broker' that's zuul aware, we dont actually have to do all of the uploading/downloading through it, but it simplifies things, and zuul still gets "a url" to display | 14:29 |
| mnaser | cause i think logging is one of those harder things to get right now as it's pretty decoupled (nice) but a little too much at times | 14:30 |
| mnaser | mordred: TIL | 14:30 |
| mordred | mnaser: or - maybe we used to - or something | 14:30 |
| mnaser | my use case is "i have a user who wants to use a multi-tenant zuul that needs their log data not visible to the world" | 14:31 |
| mnaser | you can hide everything but the logs are still out there | 14:31 |
| mordred | yeah. so - tobiash handles this use case with apache I believe | 14:31 |
| mordred | he has _many_ tenants that have to be isolated from each other | 14:31 |
| fungi | right, if you control a web service fronting all your log access you can do it fairly trivially by just tacking on an auth layer | 14:31 |
| mordred | mnaser: (totally understand the use case - it's totally valid) | 14:32 |
| mnaser | hmm, yeah, i guess it can be worked | 14:32 |
| fungi | the challenge with object storage is that the zuul admin (usually) doesn't control the object store's webserver | 14:32 |
| mnaser | yeah, i mean, i could probably work around it by using a bucket per tenant, switch to private buckets, and then have a simple service that proxy's things | 14:33 |
| mnaser | but i usually try to think on how we can reasonably solve this to make a lot more people's lives easier | 14:33 |
| mnaser | i wonder if there's a nice clean OSS project that provides something along these lines | 14:36 |
| fungi | i like your lightweight tempurl indirection service idea. allows you to decouple the durable object store creds from the web dashboard code | 14:36 |
| fungi | and yeah, maybe someone's already written it | 14:37 |
| mnaser | fungi: and it can help simplify poc-ing for something where it can be configured with local storage too | 14:37 |
| fungi | true | 14:38 |
| mnaser | trivial change: https://review.opendev.org/#/q/topic:install-docker-compose (it's blocking a merge locally here =p) | 14:44 |
| corvus | mnaser, mordred, fungi: a proxy (whether that's zuul-web or something else) makes deployment simpler in that you don't have to worry about cors. otoh, we put a lot of work into not needing a proxy because the maintenance was annoying (i mean, the point of the object store is that someone else runs the highly scalable storage infrastructure) | 14:45 |
| mnaser | corvus: i agree, i dont want to see 30MB/s of traffic being sent pushed through zuul-web as folks are streaming logs/downloading artifacts/etc | 14:46 |
| pabelanger | we have a proxy for zuul.a.c logs, currently disabled, but we might enabled it again to hid some logs | 14:53 |
| pabelanger | but mostly ran it for more user friendly logs URLs | 14:53 |
| *** jhesketh has quit IRC | 14:55 | |
| pabelanger | we just upgraded to zuul 3.16.1, looks great | 14:56 |
| pabelanger | really like estimated time remaining tool tip feature | 14:56 |
| *** jhesketh has joined #zuul | 14:56 | |
| fungi | corvus: though the proxy mnaser is suggesting might be a reasonable middle ground in that it only serves redirects to tempurls (after generating them) and the client still retrieves the files from the object store directly | 14:57 |
| fungi | and of course it wouldn't be needed in deployments where all logs are expected to be public (like opendev) | 14:58 |
| *** mhu has quit IRC | 15:02 | |
| corvus | fungi: yes, though while a pass-through proxy may have performance considerations, a tempurl-based one may be more complex (and difficult to apply in all situations like static fileservers). and perhaps mordred suggests, apache could be put in service here as the core of a pass-through proxy. i don't know the right answer -- there are a bunch of trade-offs here. | 15:04 |
| fungi | absolutely | 15:05 |
| mordred | corvus: incidentally, one of the issues with the gerrit build job is that they added a new submodule which we didn't have (also something with directory permissions) | 15:10 |
| corvus | mordred: ah that'll do it | 15:11 |
| corvus | i'm curious about the dir perms part... | 15:11 |
| mordred | corvus: me too. let me find a link | 15:12 |
| felixedel | Hi, could I get another review on https://review.opendev.org/#/c/708109/ and https://review.opendev.org/#/c/633501/21 ? | 15:14 |
| mordred | corvus: you know what - I think the permissions thing was a red herring | 15:21 |
| mordred | corvus: it was all originally the repo - it just looked like that | 15:21 |
| corvus | aha | 15:22 |
| mordred | corvus: I have updated the patch to remove the directory perms sledgehammer | 15:23 |
| mordred | corvus: zuul double-commented on https://review.opendev.org/#/c/708018/ - is that a thing we expect? | 15:26 |
| corvus | mordred: yeah i think it's in 2 tenants. i agree it's a little messy. | 15:27 |
| *** chandankumar is now known as raukadah | 15:33 | |
| mordred | ah - nod | 15:35 |
| mordred | yeah - I remember that now | 15:35 |
| avass | this is strange, for some reason the install-docker role fails because the user can't talk to the docker socket, it looks like the user isn't part of the docker group so I'm guessing meta: reset_connection isn't enough for some reason. Any ideas? | 15:42 |
| avass | https://review.opendev.org/gitweb?p=zuul/zuul-jobs.git;a=blob;f=roles/install-docker/tasks/docker-setup.yaml;h=e398c70eedd0652dbb62ea312f2ac92355fcb6e7;hb=refs/heads/master#l47 | 15:42 |
| clarkb | avass: I would double check that your ansible_user is the user attempting to talk to the docker damon | 15:43 |
| clarkb | (and that the subsequent tasks aren't running as a different user, just to rule that out) | 15:43 |
| avass | Yeah all tasks looks like they are runing as the same user | 15:44 |
| clarkb | I would also run the `groups` command to see group membership for the current user and probably ls -l /var/run/docker.sock to cross check | 15:46 |
| clarkb | if groups doesn't show the membership then likely the connection reset isn't sufficient for some reason | 15:47 |
| avass | done that and it looks like it should (except for groups since the user isn't part of the docker group) | 15:47 |
| avass | don't know if it matters but I'm using ansible 2.8.1 and not through zuul | 15:48 |
| clarkb | avass: you might try a newer 2.8.x in case this was a bug they fixed | 15:49 |
| clarkb | but it should be ok to run this outside of zuul (as the connection reset construct is an ansible thing) | 15:49 |
| *** felixedel has quit IRC | 15:49 | |
| avass | I'll try that | 15:49 |
| avass | didn't work with 2.8.0, 2.8.8 or 2.9.5 so it's probably something else | 16:05 |
| tobiash | mnaser, mordred, fungi, corvus: we use a proxy that authenticates the user and authenticates itself to the object store. Basically we're using a quite simple wsgi python script that serves the objects from the object store. | 16:09 |
| tobiash | that's running with multiple replicas along zuul-web in openshift | 16:10 |
| avass | clarkb: and the user is part of the docker group according to /etc/group *shrug* | 16:12 |
| tristanC | fwiw, in sf we use a logserver (and logstash/classify models service) per tenant | 16:12 |
| clarkb | avass: I wonder if this requires control persistence to work properly | 16:12 |
| clarkb | avass: are you usign the control persist options with ansible outside of zuul? if not maybe try enabling that? | 16:12 |
| mnaser | tobiash: yeah that is the simplest alternative | 16:13 |
| clarkb | hrm looks like control persist might be the default now | 16:14 |
| clarkb | so should be there automatically | 16:14 |
| avass | clarkb: not sure what that means | 16:14 |
| mnaser | tobiash: do you serve any big files through that? | 16:14 |
| clarkb | avass: https://docs.ansible.com/ansible/latest/reference_appendices/config.html#ansible-ssh-args it is an ssh communication method | 16:15 |
| tobiash | mnaser: mostly not (most files are <100MB), but I tested streaming big files and it worked quite well | 16:15 |
| pabelanger | avass: when do you add user to docker group? | 16:15 |
| pabelanger | when jobs is running? | 16:15 |
| avass | pabelanger: using the install-docker role, outside of zuul | 16:15 |
| avass | trying to build and AMI with Packer | 16:16 |
| pabelanger | did you reset connection, and force ansible to log back in? | 16:16 |
| pabelanger | meta: reset_connection | 16:16 |
| avass | yep | 16:16 |
| pabelanger | k, out of ideas | 16:16 |
| AJaeger | config-core, zuul-jobs experts, couple of open reviews for your consideratoin, please: https://review.opendev.org/707902 https://review.opendev.org/681603 https://review.opendev.org/704414 https://review.opendev.org/681882 | 16:16 |
| mordred | WEIRD | 16:16 |
| avass | :( | 16:16 |
| AJaeger | config-core and also https://review.opendev.org/#/c/704045/ | 16:18 |
| mnaser | tobiash: oh interesting, i assume this uses s3 | 16:18 |
| tobiash | mnaser: I'm using swift using openstacksdk | 16:18 |
| tobiash | the only thing you need to make sure is that you do chunked streaming of data | 16:19 |
| corvus | tobiash: pushed gear 0.15.0 | 16:22 |
| tobiash | corvus: awesome, thanks! | 16:22 |
| tobiash | then I'd only need https://review.opendev.org/707585 :) | 16:23 |
| tobiash | (for unit tests on macos) | 16:23 |
| corvus | tobiash: i'm hesitant to alter zuul server code to make it "work" on macos (since it can not work on macos) -- is there some other way to implement 585 that only touches test code? | 16:26 |
| corvus | tobiash: maybe add a little bit of indirection to the cpu time gathering that you then override in tests on macos? | 16:26 |
| tobiash | corvus: we can also just catch the exception when askung for cpu usage, it's for logging only | 16:27 |
| tobiash | Or I can try to monkey patch during tests | 16:27 |
| avass | 'transport = paramiko' crashes when trying to add the user to docker group instead *shrug* | 16:36 |
| avass | I think I'm going to quit for today, hopefully it's one of those bugs that only appear on mondays | 16:37 |
| openstackgerrit | Merged zuul/zuul-jobs master: install-docker: option to install docker-compose https://review.opendev.org/707902 | 16:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-javascript-tarball: introduce zuul_use_fetch_output https://review.opendev.org/681603 | 16:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: Skip test-setup.sh in pep8 jobs https://review.opendev.org/704414 | 16:42 |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-subunit-output: introduce zuul_use_fetch_output https://review.opendev.org/681882 | 16:42 |
| *** avass has quit IRC | 16:43 | |
| sugaar | Hello again! So as a reminder I am trying to setup zuul running in k8s for the celduin project. For that I am having as a reference your docker-compose example and the zuul helm project. So far I have the executor, the scheduller, the database and zookepeer working. Here the MR if somebody wants to have a look: | 16:47 |
| sugaar | https://gitlab.com/celduin/infrastructure/celduin-infra/-/merge_requests/12 | 16:47 |
| sugaar | Now I am trying to incorporate gerrit to the system, I managed to make gerrit work, but I am fighting to achieve a solution to create the zuul user on it. Here the MR: https://gitlab.com/celduin/infrastructure/celduin-infra/-/merge_requests/15 | 16:48 |
| sugaar | So my question is: My first idea was to create the zuul user using the "gerrit create-user" command, but turns out that the gerrit command is not included in the gerrit container. So I decided to create a script to run in a executor container wraped in the same pod in which is running the gerrit container. This scrip will search for the "GERRIT IS | 16:51 |
| sugaar | READY" string and when that happens would execute the ansible config that is included in your docker-compose | 16:51 |
| sugaar | however, I am aware that this might not be the best solution, so I wonder if you came across this problem and if you have any hints about how to solve it | 16:52 |
| *** Defolos has quit IRC | 16:52 | |
| clarkb | sugaar: zuul-deployment/playbooks.yaml seems to be your updated version of how docker-compose does it. It uses the rest api | 16:52 |
| clarkb | re the gerrit command, those are executed via ssh | 16:53 |
| clarkb | so you do `ssh -p 29418 gerrit-service gerrit create-user` rather than executing them directly on the host | 16:53 |
| mnaser | tobiash: and how do you upload, using the role? | 16:53 |
| mnaser | tobiash: and is that something that we can maybe work together on? if there $politics in play, i can write something again from scratch that you can use at your side to maintain?\ | 16:56 |
| *** jamesmcarthur has joined #zuul | 16:57 | |
| sugaar | clarkb that sounds much easier and nicer than what I was going to do, thanks! But I still need a executor constantly checking if the gerrit container has been rebooted, so if that happens it ca create the user right? | 17:00 |
| clarkb | sugaar: the user should persist through the database | 17:02 |
| clarkb | sugaar: you should only need to do it when creating a new gerrit install | 17:02 |
| sugaar | mm nice, I was just reading about that in https://gerrit-review.googlesource.com/Documentation/dev-design.html#_infrastructure but thanks for confirming it. | 17:03 |
| AJaeger | corvus, thanks for reviews. | 17:12 |
| AJaeger | anybody to review https://review.opendev.org/704045 as well, please? | 17:12 |
| clarkb | AJaeger: do you know where that is tested in https://review.opendev.org/#/c/697683/ ? | 17:13 |
| clarkb | system-config run base maybe? /me looks | 17:13 |
| clarkb | yes https://zuul.opendev.org/t/openstack/build/b83b212f378449ad919f7f54d86a4548 has an ara report link | 17:14 |
| mordred | clarkb: yeah - if you click on run-base you'll see the artifact | 17:14 |
| AJaeger | fbo: see clarkb's question above | 17:18 |
| AJaeger | clarkb, I agree with mordred | 17:19 |
| clarkb | yup it seems to work | 17:19 |
| clarkb | it took me a minute to figure out where we ere running nested ara is all | 17:19 |
| AJaeger | clarkb: want to abandon https://review.opendev.org/#/c/697683/ now? not sure whether dmsimard|off is around to do it... | 17:20 |
| AJaeger | thanks, clarkb and mordred | 17:20 |
| *** jamesmcarthur has quit IRC | 17:21 | |
| *** dpawlik has quit IRC | 17:22 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Make ara-report role to zuul_return an artifact https://review.opendev.org/704045 | 17:27 |
| clarkb | AJaeger: change has been abandoned | 17:31 |
| tobiash | mnaser: I can push up something later | 17:32 |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #zuul | 17:34 | |
| tobiash | mnaser: and we're using the upstream role for upload (with multiple tries to different regions if something fails) | 17:35 |
| *** SpamapS has quit IRC | 18:05 | |
| *** paulalbertella has quit IRC | 18:05 | |
| *** mordred has quit IRC | 18:06 | |
| *** reiterative has joined #zuul | 18:06 | |
| *** masterpe has quit IRC | 18:08 | |
| *** mordred has joined #zuul | 18:09 | |
| *** jpena is now known as jpena|off | 18:09 | |
| *** SpamapS has joined #zuul | 18:13 | |
| *** masterpe has joined #zuul | 18:19 | |
| *** at_work has quit IRC | 18:20 | |
| AJaeger | thanks, clarkb | 18:21 |
| *** jamesmcarthur has joined #zuul | 18:37 | |
| mnaser | tobiash: thank you, i think it would be super useful to get that somewhat integrated to zuul (or some sort of 'contrib' thing so we can maintain it there) | 18:43 |
| mnaser | dependency for https://review.opendev.org/#/c/707903/ has merged if anyone wants to push that through :) | 18:43 |
| *** sshnaidm is now known as sshnaidm|afk | 18:57 | |
| corvus | mnaser, tobiash: i'd be okay hosting that as part of the zuul project | 19:00 |
| tobiash | corvus, mnaser: I'm preparing a change to add it to contrib. If there is more interest, it might even make sense to host it in its own repo | 19:00 |
| corvus | yeah, it sounds like building its own container image might be the best way to deploy it? | 19:01 |
| mnaser | yeah personally i was thinking of deploying it on it's own deployment (probably would push up a patch for zuul-helm for this) | 19:02 |
| mnaser | that way that can scale independently, but yeah, having an image of that would be very neat, esp if we can very easily/simply configure it using environment variables. | 19:03 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add swift proxy to contrib https://review.opendev.org/708233 | 19:03 |
| corvus | it might need/want to read the zuul-web jwt config eventually | 19:03 |
| tobiash | mnaser, corvus ^ | 19:03 |
| tobiash | this is like it's in our prod system atm | 19:03 |
| tobiash | it has some tweaks that are taylored to our system that might need some work to make that more generic | 19:04 |
| mnaser | corvus: yeah, i was thinking of that too, but i was also thinking that zuul-web jwt stuff might need to be split out to a per-tenant config but thats a whole another bag of worms to open :) | 19:04 |
| tobiash | e.g. it handles put for a specific subdir that is used to upload log analysis data from a third party project (that doesn't have access to swift) | 19:05 |
| tobiash | user authentication is done in our system by an apache in front of that | 19:05 |
| mnaser | yes i agree for the first 'iteration' it would make sense to let the deployer handle authentication infront of that | 19:06 |
| tobiash | it also can take multiple clouds which it tries as fallback (which is used together with multi-cloud log upload) | 19:06 |
| tobiash | and it assumes that you choose one container per tenant | 19:09 |
| tobiash | (during log upload) | 19:09 |
| mnaser | tobiash: are your object storages publicly accessible? | 19:10 |
| tobiash | mnaser: only for admins | 19:11 |
| mnaser | tobiash: ah ok, so a tempurl based solution wouldn't really make sense to you | 19:11 |
| tobiash | they have its own auth system so we needed a proxy that does swift auth towards the object store while authing the user itself | 19:11 |
| mnaser | cause im thinking how to extend this to generate tempurls and redirect rather than proxy (which at least, in my case, is useful) | 19:12 |
| tobiash | if the tempurls are predictable by the proxy this should be fairly easy | 19:13 |
| mnaser | yes tempurls are generated via the swift api | 19:13 |
| tobiash | ah ok | 19:14 |
| tobiash | on the fly? | 19:14 |
| mnaser | the only thing is you need to have a connection to the swift endpoint | 19:14 |
| clarkb | you can generate them completely on the client side iirc | 19:14 |
| mnaser | oh really | 19:14 |
| mnaser | i havent done as much | 19:14 |
| mnaser | https://docs.openstack.org/swift/latest/api/temporary_url_middleware.html | 19:14 |
| clarkb | it uses a shared secret and a known algorithm I think | 19:14 |
| mnaser | yes clarkb is right | 19:14 |
| mnaser | so in that case, instead of proxying it, you can generate the tempurl and send a 301 | 19:14 |
| tobiash | ah ok, so the connection is also a problem for us because some users only can route to some specific ip addresses (the object store is not among them) | 19:14 |
| tobiash | but I don't think it's a problem to make this configurable | 19:15 |
| mnaser | gotcha so that wouldn't make your life easier but that does cover one of my use cases so im sure its something i could add as a toggle in there | 19:15 |
| tobiash | ++ | 19:16 |
| tobiash | and it also doesn't have test cases yet apart from flake8 ;) | 19:16 |
| mordred | flake8 catches all important bugs | 19:16 |
| tobiash | mordred: yay, so we can drop py35 and py37 jobs in zuul? | 19:18 |
| tobiash | :) | 19:18 |
| tobiash | make testing much faster | 19:18 |
| *** Defolos has joined #zuul | 19:22 | |
| *** masterpe has quit IRC | 19:46 | |
| *** openstackstatus has quit IRC | 19:49 | |
| *** masterpe has joined #zuul | 20:17 | |
| *** felixedel has joined #zuul | 20:34 | |
| tobiash | corvus: looks like all py jobe timeout now with a >1gb log file | 20:43 |
| tobiash | I'm not at the computer right now so I cannot check the logs | 20:45 |
| *** Goneri has quit IRC | 20:59 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Cleanup handlers defined in OS_LOG_DEFAULTS https://review.opendev.org/707942 | 21:05 |
| tobiash | corvus: the log seems to be flooded by https://opendev.org/opendev/gear/src/branch/master/gear/__init__.py#L2994 | 21:20 |
| tobiash | oh no, from here: https://opendev.org/opendev/gear/src/branch/master/gear/__init__.py#L878 | 21:22 |
| corvus | tobiash: look like 942 fixes it? | 21:30 |
| tobiash | corvus: 942 probably just hides it by suppressing that log | 21:30 |
| tobiash | corvus: I think there is some behavior change in the poll change in gear | 21:31 |
| corvus | tobiash: should we pin gear in zuul for now? | 21:31 |
| clarkb | do we also need to update the tox.ini? | 21:31 |
| tobiash | corvus: yes, that makes sense | 21:31 |
| tobiash | corvus: when looking closely here: https://review.opendev.org/#/c/671674/9/gear/__init__.py I saw that it changes all poll(in/out) to epoll(in/out) regardless if epoll was requested or not | 21:32 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Avoid gear version 0.15.0 https://review.opendev.org/708259 | 21:33 |
| clarkb | oh the tox change is in a followup | 21:34 |
| clarkb | corvus: https://review.opendev.org/#/c/707783/ is related to 492 and should probably be landed too | 21:36 |
| tobiash | remote: https://review.opendev.org/708261 Revert "Add BSD/Darwin support." | 21:37 |
| clarkb | does logging that large imply the zuul per job disk limits are not working? | 21:38 |
| tobiash | clarkb: disk limits are enforced by regularly running 'du'. If the job downloads faster than that run it will allow more than that | 21:40 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Don't set OS_LOG_DEFAULTS if unset https://review.opendev.org/707783 | 21:41 |
| clarkb | tobiash: corvus: that revert doesn't change the code that is causing the logging does it? | 21:50 |
| tobiash | clarkb: that revert changes the polling in gear. It looks like somehow the poll change changed one of those poll loops into a busy loop wich logs that log line | 21:52 |
| clarkb | I see so the log happens more frequently | 21:52 |
| tobiash | yes | 21:52 |
| tobiash | the log should happen on every received packet I guess but is printed now constantly | 21:53 |
| *** jamesmcarthur has quit IRC | 22:01 | |
| *** jamesmcarthur has joined #zuul | 22:02 | |
| clarkb | I see the issue | 22:02 |
| clarkb | getting a link | 22:03 |
| *** Goneri has joined #zuul | 22:04 | |
| clarkb | https://opendev.org/opendev/gear/src/commit/4fbb6d1a57de3ad1e6cb5207c2f4e09be9580a7a/gear/__init__.py#L2740 the poll(0) there means no timeout | 22:05 |
| clarkb | so its always returning | 22:05 |
| clarkb | regardless of events being present or not | 22:05 |
| clarkb | should I push up an unrevert that removes the timeout entirely (the old code had no timeout) | 22:05 |
| tobiash | clarkb: I think that will break it because the poll change unintendetly changes all polls to epoll (it was suppost to change nothing on linux) | 22:07 |
| *** jamesmcarthur has quit IRC | 22:07 | |
| clarkb | tobiash: epoll is what it was using on linux before. Do you mean poll? | 22:07 |
| tobiash | clarkb: we're not using epollet everywhere as far as I saw | 22:08 |
| clarkb | tobiash: based on the early returns? that is due to the timeout | 22:08 |
| clarkb | I think the other parts of it are fine | 22:08 |
| clarkb | it uses epoll by default and checks if select.epoll is valid otherwise uses select.poll | 22:08 |
| tobiash | clarkb: https://review.opendev.org/#/c/671674/9/gear/__init__.py line 2936 | 22:09 |
| clarkb | ah it isn't masking edge in there | 22:11 |
| tobiash | the original uses select.POLLIN while the poller uses EPOLLIN | 22:11 |
| clarkb | so a couple bugs then | 22:11 |
| tobiash | yes | 22:11 |
| clarkb | let me get a patchset up so we don't foreget these things | 22:11 |
| tobiash | I guess that would block there if we remove the timeout (which probably was the reason for removing the timeout without my fully understanding of edge and level triggering at that time) | 22:12 |
| clarkb | right you want to block until there is work to be done | 22:14 |
| clarkb | that should be true of edge or level triggering | 22:14 |
| clarkb | setting the timeout to 0 means it always returned then we checked events and nothing was happening | 22:14 |
| clarkb | well most of the time it would be a noop | 22:15 |
| clarkb | but we would still trigger the log message | 22:15 |
| tobiash | correct | 22:15 |
| tobiash | and it's busy waiting which is not ideal either | 22:16 |
| *** lennyb has quit IRC | 22:17 | |
| clarkb | oh and we shouldn't need to check for edge polling at 2936 | 22:18 |
| clarkb | because its checking the event mask not setting the listen mask | 22:18 |
| *** lennyb has joined #zuul | 22:20 | |
| clarkb | tobiash: remote: https://review.opendev.org/708267 Revert "Revert "Add BSD/Darwin support."" | 22:20 |
| clarkb | I think that may do it | 22:20 |
| tobiash | clarkb: commented | 22:24 |
| tobiash | I think that's exactly the bug that made poll(0) neccessary so the tests work | 22:24 |
| clarkb | oh I get it now | 22:26 |
| clarkb | ya I think that is an easy fix | 22:26 |
| clarkb | tobiash: new patchset take care of that I think | 22:32 |
| tobiash | lgtm, I'm wondering if we can test against that faulty behavior somehow | 22:34 |
| tobiash | maybe counting the polls during a simple test case by overriding poller._poll in the test case | 22:35 |
| tobiash | maybe I'll look tomorrow how easy this is | 22:36 |
| *** openstackstatus has joined #zuul | 23:02 | |
| *** ChanServ sets mode: +v openstackstatus | 23:02 | |
| *** igordc has joined #zuul | 23:03 | |
| openstackgerrit | Merged zuul/zuul master: Avoid gear version 0.15.0 https://review.opendev.org/708259 | 23:25 |
| *** Defolos has quit IRC | 23:40 | |
| *** tosky has quit IRC | 23:49 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!