| fungi | tristanC: if the question's what's wrong with reusing that branch for future 3.19.x point releases, it's that reno isn't designed to fold those back into a single-branch project and would need some significant logical rework to support that release model. on a side note, for a single-branch project i'm not entirely convinced reno and all its black magic are really a benefit to a single-branch project, when | 00:01 |
|---|---|---|
| fungi | compared with just committing updates to a news file or the like | 00:01 |
| fungi | the problems reno solves (branched release notes, collating multiple notes while avoiding merge conflicts) aren't that much of a challenge in a project like zuul | 00:03 |
| fungi | probably the main benefit is reno's pbr-like behavior of deriving version numbers from git tags, so we neither need to predict the next version in advance nor need to merge a commit containing version information immediately before tagging | 00:05 |
| *** wuchunyang has joined #zuul | 00:38 | |
| *** weshay|ruck is now known as weshay|pto | 00:41 | |
| *** wuchunyang has quit IRC | 00:42 | |
| *** wuchunyang has joined #zuul | 00:58 | |
| *** Goneri has quit IRC | 01:27 | |
| tristanC | fungi: i see, thanks for the details. I guess my question was about if we should keep a stable branch around until the work on master is completed | 01:31 |
| tristanC | for example we had to create a special 3.19.1-3 to include https://review.opendev.org/c/zuul/zuul/+/764069 for gerrit-3.x performance issue | 01:34 |
| *** zenkuro has quit IRC | 02:14 | |
| *** rlandy has quit IRC | 02:15 | |
| *** ajitha has joined #zuul | 02:32 | |
| *** bhavikdbavishi has joined #zuul | 02:48 | |
| *** rfolcOUT has joined #zuul | 02:50 | |
| *** bhavikdbavishi1 has joined #zuul | 02:51 | |
| *** bhavikdbavishi has quit IRC | 02:53 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 02:53 | |
| *** iurygregory|pto has quit IRC | 03:01 | |
| *** rfolcOUT has quit IRC | 03:05 | |
| *** wuchunyang has quit IRC | 04:05 | |
| *** bhavikdbavishi has quit IRC | 04:22 | |
| *** bhavikdbavishi has joined #zuul | 04:22 | |
| *** ikhan has quit IRC | 04:38 | |
| *** vishalmanchanda has joined #zuul | 04:38 | |
| *** evrardjp has quit IRC | 05:33 | |
| *** evrardjp has joined #zuul | 05:33 | |
| *** wuchunyang has joined #zuul | 05:35 | |
| *** zbr has quit IRC | 06:13 | |
| *** zbr has joined #zuul | 06:14 | |
| *** y2kenny has quit IRC | 06:14 | |
| *** hamalq has quit IRC | 06:17 | |
| *** hamalq has joined #zuul | 06:18 | |
| *** jfoufas1 has joined #zuul | 06:25 | |
| *** bhavikdbavishi1 has joined #zuul | 06:27 | |
| *** bhavikdbavishi has quit IRC | 06:29 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 06:29 | |
| *** saneax has joined #zuul | 06:52 | |
| *** bhavikdbavishi has quit IRC | 06:53 | |
| *** mach1na has joined #zuul | 07:05 | |
| *** mach1na has quit IRC | 07:31 | |
| *** hashar has joined #zuul | 08:00 | |
| *** mach1na has joined #zuul | 08:01 | |
| *** mach1na has quit IRC | 08:04 | |
| *** hashar has quit IRC | 08:04 | |
| *** mach1na has joined #zuul | 08:04 | |
| *** hashar has joined #zuul | 08:04 | |
| *** jcapitao has joined #zuul | 08:12 | |
| *** rpittau|afk is now known as rpittau | 08:28 | |
| *** rishabhhpe has joined #zuul | 08:35 | |
| *** zbr has quit IRC | 08:44 | |
| *** zbr has joined #zuul | 08:45 | |
| *** zbr has quit IRC | 08:47 | |
| *** zbr has joined #zuul | 08:47 | |
| rishabhhpe | Hi .. did anyone facing issue with posting results of CI to upstream ? my gerrit is taking the events starting the job .. but after completion of job it is not able to post the result to community | 08:49 |
| *** zbr has quit IRC | 08:49 | |
| *** zbr has joined #zuul | 08:49 | |
| openstackgerrit | Felix Edel proposed zuul/zuul master: WIP Switch to ZooKeeper backed result event queues https://review.opendev.org/c/zuul/zuul/+/764344 | 08:50 |
| *** zbr has quit IRC | 08:52 | |
| *** zbr has joined #zuul | 08:52 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** rishabhhpe has quit IRC | 09:03 | |
| *** rishabhhpe has joined #zuul | 09:04 | |
| *** tosky has joined #zuul | 09:05 | |
| *** nils has joined #zuul | 09:16 | |
| *** sshnaidm is now known as sshnaidm|ruck | 09:34 | |
| *** bhavikdbavishi has joined #zuul | 09:54 | |
| *** sshnaidm|ruck has quit IRC | 10:01 | |
| *** sshnaidm has joined #zuul | 10:01 | |
| *** bhavikdbavishi has quit IRC | 10:07 | |
| *** wuchunyang has quit IRC | 10:07 | |
| *** nils has quit IRC | 10:30 | |
| *** hashar is now known as hasharLunch | 10:32 | |
| *** nils has joined #zuul | 10:35 | |
| *** wuchunyang has joined #zuul | 10:41 | |
| *** wuchunyang has quit IRC | 10:42 | |
| *** msuszko4 has joined #zuul | 10:53 | |
| *** msuszko4 is now known as msuszko | 10:54 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:55 | |
| msuszko | Do you have recommendations on sampling profiler to watch zuul-scheduler on production? | 10:56 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-client master: encrypt: fix bad indentation https://review.opendev.org/c/zuul/zuul-client/+/765112 | 11:01 |
| msuszko | I'm observing Zuul pausing for some time (3-15 minutes) when there is > 80 jobs running. During this pause changes with success on all jobs are not finished, queued jobs are not started. | 11:01 |
| msuszko | And I'm observing scheduler trigger queue steadily rising from few to sometimes over thousand events during this period | 11:03 |
| avass | rishabhhpe: maybe the scheduler doesn't have access or labels aren't configured correctly, does the logs say anything? | 11:04 |
| msuszko | then trigger event queue suddenly drops, jobs are started, changes finished, and another cycle follows | 11:05 |
| avass | msuszko: 3-15minutes sounds like a lot but I know bmw has been working on making reporting asynchronous https://review.opendev.org/c/zuul/zuul/+/691253 | 11:06 |
| avass | msuszko: looking at the commit message it sounds like it could stall the system if you have long merge times | 11:07 |
| msuszko | avass: thanks, I'll take a look | 11:09 |
| avass | tobiash: probably has more information when he comes online :) | 11:11 |
| msuszko | I'm already running with his change-queues changes integrated :-) | 11:13 |
| *** rishabhhpe has quit IRC | 11:14 | |
| *** rishabhhpe has joined #zuul | 11:14 | |
| *** bhavikdbavishi has joined #zuul | 11:14 | |
| *** bhavikdbavishi1 has joined #zuul | 11:19 | |
| *** bhavikdbavishi has quit IRC | 11:21 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:21 | |
| *** rishabhhpe has quit IRC | 11:26 | |
| *** rishabhhpe has joined #zuul | 11:27 | |
| *** sshnaidm|afk is now known as sshnaidm|ruck | 11:39 | |
| tobiash | msuszko: we're running with async reporting since ~3 months in production | 11:41 |
| tobiash | msuszko: which reporters are you using? | 11:41 |
| tobiash | we observed different bottlenecks there in the past | 11:42 |
| tobiash | e.g. the sql db can be overloaded without https://review.opendev.org/c/zuul/zuul/+/758579 and many jobs with artifacts which can slow down sql reporting | 11:43 |
| *** zenkuro has joined #zuul | 11:44 | |
| tobiash | or if you report to gerrit it is crucual to have proper git gc settings there, otherwise this can cause very long pauses in reporting to gerrit on heavily used repos | 11:44 |
| tobiash | and not to forget, tenant reconfigurations (done after every merge of a zuul.yaml change) can also stall zuul (~3min for our largest tenant) | 11:46 |
| tobiash | rishabhhpe: you mean the opendev gerrit? | 11:47 |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection https://review.opendev.org/c/zuul/zuul-jobs/+/765125 | 11:50 |
| msuszko | tobiash: I have gerrit and postgres. PostgreSQL is an instance just for Zuul, and in looks unused on monitoring. | 11:52 |
| tobiash | msuszko: you could add this to see in the logs if there are problems with reporting delays: https://review.opendev.org/c/zuul/zuul/+/752443 | 11:54 |
| msuszko | change thoughput is fairly low, less than 3 patchsets per minute | 11:55 |
| msuszko | tobiash: oh, I'd definitely add this change with reporting | 11:57 |
| tobiash | msuszko: do you see larger jumps in the log timestamps? | 11:57 |
| tobiash | that could possibly also hint towards a problem already without that change | 11:58 |
| *** holser has joined #zuul | 11:59 | |
| rishabhhpe | tobiash: yes for openstack thirdparty CI gerrit i am talking | 11:59 |
| msuszko | Juest need to wait till the evening, I have one particular repo with over 2000 branches and few million objects which makes zuul-scheduler more than 15 minutes to start (using 14 merger instances) | 11:59 |
| tobiash | rishabhhpe: since the gerrit update you need to switch to basic auch if you've not done that yet: https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.auth_type | 11:59 |
| tobiash | 2000 branches on a gerrit repo? impressive :D | 12:00 |
| msuszko | tobiash: yes, there are certain types of log entries missing during pause, but events from gerrit are appearing for instance | 12:01 |
| *** holser is now known as holser2 | 12:01 | |
| msuszko | 2000 after automated cleanup, I started with over 6000 | 12:01 |
| tobiash | you want to filter out the events from gerrit since they're in a different thread | 12:01 |
| *** holser2 is now known as holser | 12:01 | |
| msuszko | this is basically 400 man doing some kind of gitflow inspired freestyle on gerrit | 12:03 |
| rishabhhpe | well i tried with auth_type as basic and my zuul version is 3.18 | 12:03 |
| rishabhhpe | but still the error persists | 12:03 |
| *** iurygregory has joined #zuul | 12:03 | |
| tobiash | rishabhhpe: can you post the error? | 12:04 |
| *** mach1na has quit IRC | 12:05 | |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection https://review.opendev.org/c/zuul/zuul-jobs/+/765125 | 12:06 |
| *** jcapitao is now known as jcapitao_lunch | 12:06 | |
| rishabhhpe | tobiash: log is pasted here - https://paste.ubuntu.com/p/b6kfSYdVFY/ | 12:06 |
| *** rfolcOUT has joined #zuul | 12:06 | |
| *** rfolcOUT has quit IRC | 12:07 | |
| *** rfolco has joined #zuul | 12:07 | |
| tobiash | rishabhhpe: so authentication failed. I guess this is a topic for #opendev then | 12:07 |
| tobiash | maybe there is a problem with the account, but I'm no opendev code so can't help in that regard | 12:08 |
| tobiash | s/code/core/ | 12:08 |
| openstackgerrit | Sorin Sbârnea proposed zuul/zuul-jobs master: Enable ensure ansible to install local collection https://review.opendev.org/c/zuul/zuul-jobs/+/765125 | 12:09 |
| msuszko | tenent reconfiguration is slow, but it is not common, and far quicker than pauses i'm investigating: Tenant reconfiguration complete (duration: 162.127 seconds) | 12:13 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add logging and monitoring of reporting timings https://review.opendev.org/c/zuul/zuul/+/752443 | 12:16 |
| rishabhhpe | tobiash: do u know how to pass proxy if we are behind firewall while changing the authtype to basic in zuul.conf ? | 12:21 |
| *** jfoufas1 has quit IRC | 12:21 | |
| *** zenkuro has quit IRC | 12:21 | |
| tobiash | rishabhhpe: I don't use a proxy towards gerrit, but zuul uses the requests library which should take care of this automatically if you set the http proxy env vars | 12:22 |
| *** zenkuro has joined #zuul | 12:22 | |
| msuszko | tobiash: Do you run on production Zuul with relation chain up to https://review.opendev.org/c/zuul/zuul/+/691253/33 applied? | 12:23 |
| tobiash | msuszko: yes | 12:24 |
| rishabhhpe | tobiash: or anybody out there can guide me on this a little because normal environmental proxy varibale are already imported but i am seeing the issue related to that | 12:25 |
| msuszko | Perfect. I already have branche queues applied, which is zuul-enabler for this crazy branch repo I mentioned | 12:25 |
| msuszko | s/branche/branch/ | 12:25 |
| tobiash | rishabhhpe: zuul has no specific way to enforce proxy usage other than http_proxy and https_proxy vars (maybe the https_proxy var is missing?) | 12:28 |
| avass | msuszko: ooh git inspired freestyle is our speciality | 12:31 |
| *** jpena is now known as jpena|lunch | 12:36 | |
| *** ikhan has joined #zuul | 12:50 | |
| openstackgerrit | Jonas Sticha proposed zuul/nodepool master: WIP: aws: add support for uploading diskimages https://review.opendev.org/c/zuul/nodepool/+/735217 | 12:50 |
| *** Goneri has joined #zuul | 12:55 | |
| *** rishabhhpe has quit IRC | 12:55 | |
| *** mach1na has joined #zuul | 12:56 | |
| *** jfoufas1 has joined #zuul | 13:00 | |
| *** hamalq has quit IRC | 13:01 | |
| *** jcapitao_lunch is now known as jcapitao | 13:03 | |
| *** hamalq has joined #zuul | 13:05 | |
| tobiash | avass: did you see tristanC's comment on https://review.opendev.org/c/zuul/zuul/+/764584? | 13:06 |
| *** rlandy has joined #zuul | 13:18 | |
| *** ikhan has quit IRC | 13:32 | |
| *** ikhan has joined #zuul | 13:35 | |
| avass | tobiash: nope but I'll take a look at doing that instead | 13:36 |
| *** bhavikdbavishi has quit IRC | 13:38 | |
| *** hasharLunch is now known as hashar | 13:43 | |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Throw a warning if the executors ssh-key can't be loaded https://review.opendev.org/c/zuul/zuul/+/764584 | 13:43 |
| avass | tobiash: how about logging a warning if it doesn't exist. That should make it a bit easier to debug | 13:43 |
| *** manoj_kumar_kata has joined #zuul | 13:48 | |
| manoj_kumar_kata | Hello | 13:48 |
| manoj_kumar_kata | with the gerrit upgrade, we fixed zuul issue internally to launch the jobs. | 13:49 |
| manoj_kumar_kata | Now after execution, zuul is failing to report the jobs to reviews. | 13:49 |
| manoj_kumar_kata | zuul.reporter.gerrit.Reporter is failing with exception : | 13:49 |
| manoj_kumar_kata | 2020-12-02 12:13:44,731 ERROR zuul.source.Gerrit: Exception looking for ref refs/heads/master | 13:49 |
| manoj_kumar_kata | Traceback (most recent call last): | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/source/gerrit.py", line 49, in getRefSha | 13:49 |
| manoj_kumar_kata | refs = self.connection.getInfoRefs(project) | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py", line 399, in getInfoRefs | 13:49 |
| manoj_kumar_kata | raise Exception("Gerrit repository does not support " | 13:49 |
| manoj_kumar_kata | Exception: Gerrit repository does not support git-upload-pack | 13:49 |
| manoj_kumar_kata | 2020-12-02 12:13:44,873 ERROR zuul.IndependentPipelineManager: Exception while reporting: | 13:49 |
| manoj_kumar_kata | Traceback (most recent call last): | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1764, in _reportItem | 13:49 |
| manoj_kumar_kata | ret = self.sendReport(actions, self.pipeline.source, item) | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/scheduler.py", line 1291, in sendReport | 13:49 |
| manoj_kumar_kata | ret = reporter.report(source, self.pipeline, item) | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/reporter/gerrit.py", line 39, in report | 13:49 |
| manoj_kumar_kata | message, self.reporter_config) | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py", line 279, in review | 13:49 |
| manoj_kumar_kata | out, err = self._ssh(cmd) | 13:49 |
| manoj_kumar_kata | File "/usr/local/lib/python2.7/dist-packages/zuul/connection/gerrit.py", line 384, in _ssh | 13:49 |
| manoj_kumar_kata | raise Exception("Gerrit error executing %s" % command) | 13:49 |
| manoj_kumar_kata | Exception: Gerrit error executing gerrit review --project openstack/cinder --message "Build | 13:49 |
| manoj_kumar_kata | would appreciate some help here. | 13:49 |
| *** jpena|lunch is now known as jpena | 13:51 | |
| avass | manoj_kumar_kata: hi! please use a paste service like http://paste.openstack.org/ to post logs in the future | 13:53 |
| *** sugaar has quit IRC | 14:02 | |
| manoj_kumar_kata | avass: sure , will do it next time :) | 14:08 |
| zbr | out of curiosity, does zuul really need these huge UUIDS? can't we just use abridged versions like git at least for urls? | 14:20 |
| fungi | zbr: that might be doable but would need a different kind of indexing in the database to efficiently perform substring lookups, i think? someone with a better grasp of mysql would need to confirm that though | 14:22 |
| zbr | i suspect that there are concerns about clashing, but lets be realistic...even 6 chars should provide more than enough entropy for practical reasons. | 14:22 |
| zbr | shorter urls could improve display in several places, like the "build succeeded message". | 14:23 |
| zbr | https://sbarnea.com/ss/Screen-Shot-2020-12-02-14-23-59.75.png | 14:24 |
| zbr | not sure why it is wrapped like this bug i suspect is a feature of wrapping long text to ~80 columns to make it easier to read. | 14:24 |
| zbr | in our particular case it works against its original intention | 14:24 |
| corvus | zbr: i don't think we should change zuul to accomodate that display issue in gerrit; instead we should update gerrit not to display urls as hyperlink titles | 14:26 |
| zbr | corvus: that was one example, sharing links to builds, or even the links to logs are very long. i think fixing one does not rule out the other. | 14:27 |
| zbr | i am sure that the short-uuid in zuul is not very easy to fix | 14:27 |
| zbr | another think that i find bit annoying is searching for a specific build by UI, many times i endup searching for it on the wrong tenant. I wonder if we could make it look on other tenants. | 14:29 |
| corvus | zbr: tenants are intentionally completely separate | 14:30 |
| corvus | not only for scoping but for security/privacy reasons | 14:30 |
| zbr | still, the web interface knows about all of them, so the react side could do some magic to help the user. | 14:31 |
| fungi | it's even set up so that deployments can require different authentication for accessing information about different tenants | 14:31 |
| corvus | zbr: my point is that's counter to the design so we aren't likely to do that | 14:32 |
| corvus | zbr: (opendev's deployment of the web ui is not the only way the system can be deployed; as fungi says, the web uis can be completely separate for tenants) | 14:32 |
| fungi | and the authentication may be done as an external mechanism based on url path, which react isn't going to even know exists, so could return results to which the user has no access thereby leaking some information | 14:32 |
| avass | build/buildset could be b/bs to make it shorter. like: /t/opendev/b/<uuid> | 14:34 |
| zbr | afaik, react is client only, which means it cannot expose stuff that user does not have access to, only a backend can do this kind of mistake. | 14:34 |
| avass | but I can't say I've thought the url was too long | 14:34 |
| corvus | zbr: the web ui should also not mislead users into thinking that information is shared between tenants. it's good that the ui reflects the actual system. | 14:37 |
| avass | also it don't think I wanted to make the url build uuid shorten unless it's shorter everywhere else, I use that to take me to the correct build page pretty often | 14:38 |
| zbr | ok, lets forget about combining results from multiple tenants in the UI, at least for now. | 14:39 |
| avass | though I have to agree a multi-tenant dashboard would be nice to have | 14:41 |
| *** bhavikdbavishi has joined #zuul | 14:45 | |
| *** bhavikdbavishi1 has joined #zuul | 14:48 | |
| *** bhavikdbavishi has quit IRC | 14:50 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 14:50 | |
| *** sanjayu_ has joined #zuul | 14:50 | |
| fungi | i also use it to get to other things, like build artifacts | 14:51 |
| fungi | yesterday i was testing a build of a container image for gerrit, and it was convenient to be able to just construct the url to it knowing the build id, without needing to go look up the artifact's url | 14:52 |
| *** saneax has quit IRC | 14:52 | |
| fungi | for reference, our test builds end up looking like insecure-ci-registry.opendev.org:5000/opendevorg/gerrit:5d927e040a1d4ffeb8baf17cad2acc16_3.2 | 14:54 |
| fungi | so just have to update the bit between the : and the _ with the build uuiid to get the new one i want to try for gerrit 3.2 | 14:55 |
| fungi | which means seeing the build result url posted on the change, i was able to set the correct image without ever needing to actually visit the zuul webui | 14:56 |
| *** mach1na has quit IRC | 14:57 | |
| *** bhavikdbavishi has quit IRC | 14:59 | |
| *** mach1na has joined #zuul | 15:00 | |
| *** sugaar has joined #zuul | 15:03 | |
| *** hashar is now known as hasharAway | 15:05 | |
| *** bhavikdbavishi has joined #zuul | 15:17 | |
| *** bhavikdbavishi1 has joined #zuul | 15:20 | |
| *** bhavikdbavishi has quit IRC | 15:22 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 15:22 | |
| *** jfoufas1 has quit IRC | 15:22 | |
| corvus | tobiash: is https://review.opendev.org/752664 really required for mandatory sql? the existing code was very deliberately written and well tested; i'd like to have a thorough conversation about changing it. | 15:27 |
| corvus | bolg: ^ | 15:27 |
| corvus | (but i'd like to have that thorough conversation after v4 if possible) | 15:28 |
| *** jfoufas1 has joined #zuul | 15:32 | |
| msuszko | There is mqtt reporter. Does anyone have kafka reporter by any change? | 15:53 |
| msuszko | s/change/chance/ | 15:53 |
| pabelanger | morning! IIRC, there was talk of zuul 4.0 soon, is that still the case? I _think_ there was talk of this or next week? | 15:56 |
| clarkb | pabelanger: corvus was working on release notes stuff yesterday I think in preparation for that | 15:58 |
| clarkb | I'm not sure what else needs to be done though | 15:58 |
| pabelanger | okay, cool! | 15:59 |
| pabelanger | we still need to enable SSL on zookeeper, so I guess now is the time to prepare | 15:59 |
| corvus | clarkb: that and we need the require-sql change and require-zk | 15:59 |
| corvus | pabelanger: yes! | 15:59 |
| pabelanger | thanks | 16:00 |
| pabelanger | could some one point me to Felix Edel? I had a question about https://review.opendev.org/c/zuul/zuul/+/709135 | 16:00 |
| pabelanger | specifically, how aborts are setup in github | 16:00 |
| corvus | pabelanger: if you have zk+ssl enabled and reachable from all zuul components, along with a sql connection (which you can do now because it's optional), then upgrade should be painless | 16:00 |
| corvus | pabelanger: meet felixedel | 16:00 |
| pabelanger | corvus: will confirm | 16:01 |
| pabelanger | and thanks | 16:01 |
| pabelanger | felixedel: can you point me to more info about custom aborts | 16:01 |
| corvus | pabelanger: i got the idea that it was automatic | 16:01 |
| *** savihou has joined #zuul | 16:02 | |
| corvus | pabelanger: ie, line 2053 in that change causes the abort action to be present on github's ui | 16:02 |
| corvus | pabelanger: so you'll get that automatically if you run master or the upcoming v4 release | 16:03 |
| corvus | at least, that's my understanding; haven't used it myself | 16:03 |
| pabelanger | ah, I see. Zuul creates the abort button | 16:04 |
| pabelanger | okay, that explains it | 16:04 |
| pabelanger | does that mean anyone can click it, or only the PR owner? | 16:05 |
| tobiash | corvus: no, that is not required for mandatory sql, no idea why it's in that order (this change was an attempt to improve reporting performance of the sql reporter) | 16:05 |
| corvus | tobiash: cool, so maybe when you/bolg have a chance to refresh that, you could swap the order? | 16:05 |
| tobiash | on it | 16:05 |
| fungi | msuszko: i haven't seen any reporter implementation for apache kafka proposed, though it probably wouldn't be hard to add one based on the mqtt reporter if you wanted to try | 16:11 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Required SQL reporters https://review.opendev.org/c/zuul/zuul/+/630472 | 16:11 |
| fungi | msuszko: just be aware that runs as a reporter, so only emits events based on completion of all builds for a buildset, it's not a granular event stream | 16:12 |
| tobiash | pabelanger: I guess anyone with write access to the repo can click it | 16:13 |
| pabelanger | tobiash: okay cool that is what I was hoping, write access was the acl | 16:13 |
| pabelanger | tobiash: with that in mind, I also wonder if a new action for say 'auto-hold' could be setup via github actions | 16:13 |
| tobiash | pabelanger: I guess we could add any arbitrary type of action there | 16:15 |
| tobiash | clicking on the button would result in an according check_run event with the info we supply to the action | 16:15 |
| pabelanger | k, I'd have to look more into it. I think it would be neat, to expose some sort of 'auto-hold', which the check_run event exposed the person who clicked it, then injected their ssh keys, from github.com into the node. | 16:16 |
| pabelanger | For that, I guess we'd need to some how map the check_run event, into running playbooks | 16:17 |
| tobiash | that should be configurable though, I don't want an auto hold button where everyone of my users can click on ;) | 16:17 |
| tobiash | hm, that would be difficult, each event is change related | 16:17 |
| msuszko | fungi: Writing kafka reporter is indeed easy, far easier than tests for it. | 16:18 |
| tobiash | so you could only add autoholds for changes | 16:18 |
| pabelanger | tobiash: yah, that is right | 16:18 |
| tobiash | or you add an autohold button for each job :D | 16:18 |
| pabelanger | otherwise, it would be using SpamapS idea of adding your ssh key into the PR | 16:18 |
| *** jfoufas1 has quit IRC | 16:19 | |
| pabelanger | with some sort of 2 hour busy-loop | 16:19 |
| tobiash | that's our default way for on node debugging | 16:19 |
| pabelanger | feels so dirty | 16:19 |
| fungi | msuszko: an alternative might also be an mqtt broker acting as a kafka proxy, so publish zuul reports into it and then convert them on the fly to kafka events, as a stand-alone daemon/service | 16:19 |
| SpamapS | pabelanger: I always hoped we might build in something to make it more web-based. ;) | 16:19 |
| msuszko | fungi: I settled with https://github.com/nodefluent/mqtt-to-kafka-bridge | 16:20 |
| fungi | msuszko: aha, so it already exists. i thought it might | 16:20 |
| corvus | SpamapS, pabelanger, tobiash: i think mhu has WIP on that (using zuul-web authn) | 16:20 |
| corvus | so the button would be in zuul's ui, not github's | 16:20 |
| tobiash | which would be the better more generic solution | 16:21 |
| msuszko | i was asking, because last time I've made upload-logs-s3 role just to find it in zuul-jobs few weeks later | 16:22 |
| pabelanger | corvus: k, I haven't looked into zuul webui actions yet. Need to better understand how auth for it works | 16:22 |
| corvus | pabelanger: it's incomplete; i think the backend api is there but not the frontend yet | 16:23 |
| pabelanger | do you know if it is expect to work with SSO, say from github? | 16:24 |
| pabelanger | perhaps I should sync with mhu more | 16:24 |
| mhu | OHAI | 16:24 |
| avass | now might be a good time to tell everyone that upload-logs-s3 is broken: https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 16:25 |
| avass | msuszko: ^ | 16:25 |
| *** mach1na has quit IRC | 16:26 | |
| avass | I think I made that fix in only upload-logs-s3 before the roles were merged so it got left out | 16:26 |
| corvus | avass: any idea why that [apparently] works as written for the others? | 16:26 |
| avass | I guess they specify a read length | 16:26 |
| mhu | pabelanger, I wanted to revisit the scoped admin ui after the migration to patternfly was done, then I got into zuul-client | 16:26 |
| pabelanger | mhu: so, zuul web auth, can you use github as source for user credentials? | 16:27 |
| corvus | avass: that gets passed to a boto class which uses it? | 16:27 |
| avass | corvus: yeah, it crashes when boto calls read() | 16:27 |
| corvus | avass: makes sense; clarkb, tobiash: can one of you look at 764478 ? | 16:27 |
| clarkb | I can take a look | 16:28 |
| corvus | clarkb: (and parent) | 16:28 |
| avass | also I'm using that fix at https://zuul.vassast.org so I know it works :) | 16:28 |
| clarkb | avass: do you know why the gzip thing broke? that is vendored code only minimally modified to handle python2 and python3 and its worked for us as far as I can tell? | 16:29 |
| corvus | this is the second personal/private zuul instance i've seen; i feel like we may be starting to round the curve on "zuul is hard to install/run" :) | 16:29 |
| avass | well it's easy when you know how to do it :) | 16:29 |
| tobiash | avass: is that your personal home-zuul? | 16:30 |
| avass | clarkb: I guess the other role that uses it calls read() with and argument and doesn't read until EOF | 16:30 |
| avass | tobiash: yeah | 16:30 |
| tobiash | awesome :) | 16:30 |
| avass | running in k8s in digitalocean | 16:30 |
| clarkb | avass: well the current api there requires you always pass an argument? | 16:30 |
| avass | yep | 16:30 |
| clarkb | (I mean I don't think the chagne is necessarily wrong, I just don't see how it could've worked before without passing in -1 already) | 16:31 |
| msuszko | corvus: hardly anything to curve, docker-compose is a breeze | 16:31 |
| mhu | pabelanger, not sure if github uses OIDC or old oauth | 16:31 |
| avass | clarkb: I think I fixed it when I copy pasted it to upload-logs-s3 but not in swift :) | 16:31 |
| mhu | pabelanger, there was a doc I wrote for google auth: https://review.opendev.org/c/zuul/zuul/+/734082/14/doc/source/howtos/openid-with-google.rst | 16:31 |
| tobiash | oh and I broke it during consolidation? | 16:31 |
| avass | but then it broke when it got merged into upload-logs-base and the tests didn't run since the file: attribute wasn't updated | 16:31 |
| avass | yeah | 16:32 |
| tobiash | sorry | 16:32 |
| mhu | pabelanger, worst case scenario, you can configure zuul with keycloak, and have keycloak authenticate users on github | 16:32 |
| avass | np we didn't start using it anyway | 16:32 |
| avass | but are gonna do that soonish probably | 16:32 |
| mhu | pabelanger, https://review.opendev.org/c/zuul/zuul/+/734082/14/doc/source/howtos/openid-with-keycloak.rst | 16:32 |
| avass | corvus: also, I'm gonna push the helm charts we use at volvo to zuul-helm whenever I find time. It automates zookeeper tls, whitelabeling, separates config per components so only relevant components gets restarted | 16:34 |
| avass | uuh and uses a prestop hook to gracefully shut down executors for roling updates | 16:34 |
| avass | probably something else :) | 16:34 |
| pabelanger | mhu: okay, thanks. Are you running the web-ui bits in rdo zuul? or some other sf install | 16:35 |
| pabelanger | I'd love to look at it | 16:35 |
| mhu | corvus, tobiash do you think you'd have time in the coming weeks to review the scoped admin web UI changes if I refresh them? | 16:35 |
| clarkb | avass: looking at the code in that file all the read()s are specifying a size from what I see | 16:35 |
| msuszko | avass: do you have gracefull shutdown of executors? | 16:35 |
| clarkb | avass: do you know where the short read call was being made? | 16:35 |
| mhu | pabelanger, not yet, besides these UI patches probably need a refresh - I can set up a demo some time soon if you'd like | 16:36 |
| tobiash | mhu: I think so | 16:36 |
| mhu | I use keycloak as the SSO | 16:36 |
| pabelanger | mhu: yah, demo would be cool | 16:37 |
| clarkb | I'm wary of approving that change since I don't understand why it should be necessary. At the same time I'm reasonably convinced it would be ok since we pass a length everywhere else? corvus tobiash I'll defer to you on that I think | 16:37 |
| avass | clarkb: botocore or boto3 somewhere | 16:38 |
| mhu | pabelanger, what's your TZ these days? | 16:38 |
| msuszko | speaking of helm, current charts are restarting scheduler on tenant config change | 16:38 |
| avass | msuszko: there's `zuul-executor graceful` but I think it's just a stub on 3.19 | 16:38 |
| avass | msuszko: oh we set the updatestrategy to OnDelete for the scheduler | 16:38 |
| tobiash | clarkb: that's the base class: https://docs.python.org/3/library/io.html#io.RawIOBase.read | 16:39 |
| pabelanger | mhu: UTC-5 | 16:39 |
| tobiash | docs say that size defaults to -1 so the change looks correct to me | 16:39 |
| msuszko | I'm using curl -sSn https://gerrit/a/plugins/gitiles/zuul-setup/+archive/refs/heads/master/tenants.tar.gz | tar xzf - -O as tenant_config_script | 16:39 |
| tobiash | although we may even rename length to size to accomodate for the interface | 16:40 |
| avass | clarkb, tobiash: yeah the current GZip implementation is incompatible to the superclass | 16:40 |
| clarkb | tobiash: right, the problem is more that when we call GZIPCompressedStream.read() previously length was always required. So we should never get a short read as you were forced to apss that argument | 16:40 |
| avass | tobiash: true | 16:40 |
| fungi | could that vary by python version? | 16:40 |
| clarkb | oh I see what the issue is now, compat with parent interface | 16:40 |
| tobiash | fungi: 3.5 has the same interface | 16:41 |
| msuszko | avass: great! | 16:41 |
| tobiash | yes | 16:41 |
| clarkb | https://docs.python.org/2.7/library/io.html#io.RawIOBase.read 2.7 calls it n | 16:42 |
| clarkb | I think we go for 3 compat here and if 2.7 is a problem due to named args we firgure that out when it becomse a problem? | 16:42 |
| tobiash | at least the default is the same since 2.7 | 16:43 |
| clarkb | yup so only a problem if using a named arg | 16:43 |
| tobiash | so do we want to rename that arg as well or just set the default? | 16:43 |
| avass | renaming it probably avoids a future headache :) | 16:44 |
| clarkb | If the goal is parent interface compat we should rename the arg imo | 16:44 |
| tobiash | probably :D | 16:44 |
| clarkb | and maybe add a note about it to the vendored code comment | 16:44 |
| clarkb | so that we can track our delta from the upstream more easily | 16:44 |
| openstackgerrit | Merged zuul/zuul-jobs master: upload-logs-s3: fix test-jobs files: attribute https://review.opendev.org/c/zuul/zuul-jobs/+/764476 | 16:45 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read length to -1 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 16:52 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: GZIPCompressedStream default read size to -1 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 16:53 |
| avass | how about that ^ | 16:53 |
| avass | tobiash: too quick :) | 16:54 |
| tobiash | yeah... | 16:54 |
| clarkb | thats great thank you. I +2'd but didn't approve in case corvus wants to rereview | 16:54 |
| clarkb | but if it isn't approved in a half an hour I think we can just approve it? | 16:55 |
| tobiash | ++ | 16:55 |
| avass | also s3 doesn't redirect to index.html so only the cors logs work. but if you report the log_url to point at the index.html logs the cors logs breaks.. | 16:59 |
| avass | not sure if there's a good way to fix that | 16:59 |
| avass | you can see it here: https://zuul.vassast.org/build/2ec604b42feb4f769cb8aebe8b61e007/logs | 17:00 |
| *** bhavikdbavishi has quit IRC | 17:01 | |
| *** rpittau is now known as rpittau|afk | 17:04 | |
| *** bhavikdbavishi has joined #zuul | 17:07 | |
| corvus | avass: yeah, gcs has the same problem; i think we need some flag to link to '/index.html' in the web page | 17:09 |
| *** manoj_kumar_kata has quit IRC | 17:14 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 17:18 |
| *** manoj_kumar_kata has joined #zuul | 17:27 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 17:38 |
| *** sshnaidm|ruck is now known as sshnaidm|afk | 17:47 | |
| avass | corvus: yep | 17:47 |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 17:48 |
| *** polls45 has joined #zuul | 17:49 | |
| *** jcapitao has quit IRC | 17:54 | |
| *** jpena is now known as jpena|off | 18:02 | |
| *** wuchunyang has joined #zuul | 18:02 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 18:04 |
| *** wuchunyang has quit IRC | 18:07 | |
| *** bhavikdbavishi has quit IRC | 18:08 | |
| *** hasharAway is now known as hashar | 18:15 | |
| *** nils has quit IRC | 18:18 | |
| avass | corvus: if you have a second to approve https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 18:21 |
| msuszko | while profiling zuul-scheduler found out it spent 30% time in chardet | 18:23 |
| msuszko | i guess it is requests calling gerrit | 18:23 |
| *** manoj_kumar_kata has quit IRC | 18:39 | |
| openstackgerrit | Paul Belanger proposed zuul/zuul-jobs master: ensure-podman: Use official podman repos for ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/765177 | 18:43 |
| mnaser | talking out loud -- i'm trying to look at replacing terraform cloud by zuul, however, a use case that i'm finding a bit harder is the 'retry' story | 18:48 |
| mnaser | inside terraform clouds ui, you can go back and 'queue' a run if it failed | 18:48 |
| mnaser | i cant think of any simple way of doing that within zuul other than making an empty change to retrigger it | 18:49 |
| openstackgerrit | Merged zuul/zuul-jobs master: GZIPCompressedStream default read size to -1 https://review.opendev.org/c/zuul/zuul-jobs/+/764478 | 18:49 |
| corvus | mnaser: we should be able to handle that with a direct-enqueue via api; combined with mhu's authenticated ui work, could be a button on a the build result page. | 18:49 |
| mnaser | corvus: has the auth'd ui work itself landed (i.e. would the work needed essentially be enqueue on the build result page or more?) | 18:51 |
| corvus | mnaser: no; only the backend api part | 18:51 |
| pabelanger | mnaser: you are using github? | 18:52 |
| mnaser | i think something that might be tricky in my case is uh, auth for api per tenant | 18:52 |
| mnaser | pabelanger: yes | 18:52 |
| pabelanger | won't the re-run button in check runs work for this? | 18:52 |
| corvus | mnaser: api auth is per-tenant | 18:52 |
| mnaser | pabelanger: hmm, maybe? i wonder if you can ask to rerun the post jobs in this case | 18:53 |
| mnaser | corvus: ok, so that's really interesting then.. i should play with that code a bit more | 18:53 |
| corvus | mnaser: the cli should support it; so you should be able to get an auth token and use the cli to demo | 18:53 |
| corvus | mnaser: zuul-client | 18:54 |
| mnaser | hrm, got it | 18:54 |
| mnaser | i might write up something if i end up coming up with a full terraform pipeline for this | 18:55 |
| pabelanger | what we came up with was 1hr periodic pipeline, if we needed something to re-run | 18:56 |
| pabelanger | then, re-enqueue with CLI if we need faster | 18:56 |
| mnaser | got it | 18:57 |
| corvus | that's pretty much the opendev strategy too | 18:58 |
| *** vishalmanchanda has quit IRC | 18:58 | |
| fungi | almost exactly, in fact | 18:58 |
| mnaser | to be honest, the only concern i have a little bit right now is running this on our existing shared zuul or dedicating one for this use case | 18:58 |
| mnaser | one side says "dogfood" the other says "risk-mitigating" :) | 18:59 |
| fungi | the biggest logistical challenge we face is triggering deployments by changes landing in related repositories | 18:59 |
| fungi | so we still rely on hourly deploys for that sort of stuff too | 18:59 |
| *** polls45 has quit IRC | 19:00 | |
| mnaser | hmm | 19:01 |
| mnaser | i wonder if i can have multiple zuuls talking to the same nodepool instance | 19:01 |
| mnaser | :p | 19:01 |
| corvus | mnaser: theoretically yes; i don't think it's well tested | 19:02 |
| corvus | (it definitely was a design intent) | 19:03 |
| mnaser | i guess i could technically run a seperate zuul that uses the same nodepool | 19:04 |
| *** reiterative has quit IRC | 19:06 | |
| *** reiterative has joined #zuul | 19:07 | |
| corvus | fungi, clarkb, pabelanger: stack at https://review.opendev.org/765023 is ready for review; that's the first blocker for v4 | 19:16 |
| *** sugaar has quit IRC | 19:17 | |
| *** manoj_kumar_kata has joined #zuul | 19:22 | |
| *** iurygregory has quit IRC | 19:29 | |
| *** msuszko has quit IRC | 19:53 | |
| fungi | corvus: thanks for the reminder! looking at it now | 19:54 |
| *** reiterative has quit IRC | 19:57 | |
| *** reiterative has joined #zuul | 19:57 | |
| fungi | unrelated, but has anyone started looking into the nodepool-zuul-functional build timeouts? i haven't been following closely if so | 19:58 |
| *** hashar has quit IRC | 20:23 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul-client master: API, CLI: Allow use of tenant-scoped API URLs https://review.opendev.org/c/zuul/zuul-client/+/765203 | 20:33 |
| openstackgerrit | Merged zuul/zuul master: Temporarily remove pending release notes in order to merge 3.19.1 https://review.opendev.org/c/zuul/zuul/+/765023 | 21:00 |
| openstackgerrit | Merged zuul/zuul master: Merge tag '3.19.1' https://review.opendev.org/c/zuul/zuul/+/765024 | 21:02 |
| *** rfolco has quit IRC | 21:20 | |
| *** iurygregory has joined #zuul | 21:28 | |
| openstackgerrit | Merged zuul/zuul master: Re-add pending release notes https://review.opendev.org/c/zuul/zuul/+/765025 | 22:09 |
| *** ajitha has quit IRC | 22:35 | |
| fungi | corvus: ^ okay those blockers are in, thanks for working through the last of that! | 23:03 |
| * fungi missed that merging by nearly an hour | 23:03 | |
| *** rlandy has quit IRC | 23:23 | |
| *** holser_ has joined #zuul | 23:48 | |
| *** Goneri has quit IRC | 23:50 | |
| *** holser has quit IRC | 23:51 | |
| *** sanjayu_ has quit IRC | 23:52 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
