Wednesday, 2021-02-24

openstackgerritJames E. Blair proposed zuul/zuul-storage-proxy master: DNM: Break the proxy  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77726500:06
openstackgerritJames E. Blair proposed zuul/zuul-storage-proxy master: Add an integration test to the image build  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77726200:07
openstackgerritJames E. Blair proposed zuul/zuul-storage-proxy master: DNM: Break the proxy  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77726500:07
*** tosky has quit IRC00:10
corvusmordred: w00t!  got it in 2 :)  https://zuul.opendev.org/t/zuul/build/2112787d3afd4f6791286de1657bc312 fails as expected; https://zuul.opendev.org/t/zuul/build/a55c05d58531448da074c9f5fb3f4d99/console worked00:29
corvuszuul-maint: https://review.opendev.org/777262 is worth a look from everyone as we may be able to establish that as a pattern for functional testing of code and/or roles that interact with swift (ie, some stuff in zuul-jobs).  i haven't looked into it, but we might be able to use that as a stand-in for s3 as well.00:31
corvusi'm going to push up one more patchset without the extra whitespace :)00:31
openstackgerritJames E. Blair proposed zuul/zuul-storage-proxy master: Add an integration test to the image build  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77726200:32
openstackgerritJames E. Blair proposed zuul/zuul-storage-proxy master: DNM: Break the proxy  https://review.opendev.org/c/zuul/zuul-storage-proxy/+/77726500:32
mordredcorvus: lovely!!!00:42
*** rlandy has quit IRC00:46
*** dmsimard0 has joined #zuul00:50
*** dmsimard has quit IRC00:51
*** dmsimard0 is now known as dmsimard00:51
*** ikhan has quit IRC01:09
*** hamalq has quit IRC01:53
*** hamalq has joined #zuul01:54
*** hamalq has quit IRC02:08
openstackgerritOleksandr Kozachenko proposed zuul/zuul-jobs master: Revert "Revert "Update upload-logs roles to support endpoint override""  https://review.opendev.org/c/zuul/zuul-jobs/+/77667702:25
*** decimuscorvinus has quit IRC03:34
*** decimuscorvinus has joined #zuul03:36
*** decimuscorvinus has quit IRC03:50
*** decimuscorvinus has joined #zuul03:54
*** decimuscorvinus has quit IRC04:00
*** decimuscorvinus has joined #zuul04:05
*** ykarel has joined #zuul04:11
*** evrardjp has quit IRC05:33
*** evrardjp has joined #zuul05:33
*** jfoufas1 has joined #zuul05:40
*** ricolin has quit IRC06:00
*** saneax has joined #zuul06:07
*** ricolin has joined #zuul06:28
*** jcapitao has joined #zuul07:38
*** sshnaidm|afk is now known as sshnaidm07:38
dryIs there a way to configure Zuul, to make it rebase Gerrit changes based on version older than HEAD before gate begins?07:54
*** dry is now known as msuszko07:55
*** okamis has joined #zuul08:02
msuszkoI have no Workflow label, and Code-Review with ignoreSelfApproval. This effectively prevents reviewer from rebasing change while doint review08:04
msuszkos/doint/doing/08:05
*** rpittau|afk is now known as rpittau08:11
*** fdegir has joined #zuul08:18
openstackgerritMerged zuul/zuul master: Simplify ZooKeeper client initialization  https://review.opendev.org/c/zuul/zuul/+/75436008:23
openstackgerritMerged zuul/zuul master: Use ZooKeeperClient.fromConfig in tests  https://review.opendev.org/c/zuul/zuul/+/77683808:31
*** ykarel_ has joined #zuul08:31
*** ykarel has quit IRC08:34
*** ykarel_ is now known as ykarel08:38
*** jpena|off is now known as jpena08:57
*** holser_ has joined #zuul09:10
*** nils has joined #zuul09:17
*** tosky has joined #zuul09:18
*** okamis has quit IRC10:05
avasstobiash: are you running parallel jobs on windows? if so do you ever run into problems with locked authorized_keys when adding/removing build keys?10:47
avasstrying to figure out how to use build-keys without needing multiple users10:48
openstackgerritMerged zuul/zuul master: Add tenant to reconfiguration end log message  https://review.opendev.org/c/zuul/zuul/+/77710911:06
*** jcapitao is now known as jcapitao_lunch11:20
*** rlandy has joined #zuul12:32
*** jpena is now known as jpena|lunch12:38
*** vishalmanchanda has joined #zuul12:46
*** ajitha has joined #zuul12:47
avassI think I stumbled into a weird edge case where renaming a file and editing it isn't picked up by git as a rename (it's a deleted file and an added file). So what zuul does is load the same job twice and runs the pre/post playbooks from that job twice12:48
avassfor some reason?12:49
*** jcapitao_lunch is now known as jcapitao13:00
fungiavass: file "renames" aren't really special in git, they're encoded as normal deletions and additions. some git clients will use heuristics to spot addition of a file similar to one deleted in the same commit and call that a rename, but it's just guessing13:02
fungiavass: were you renaming a file containing zuul configuration?13:02
fungiit's not obvious to me why zuul would treat file deletion and addition as a reason to double playbooks for a job13:04
avassfungi: yeah exactly13:04
avassI'm not sure if it's specific to delete+add or if a rename would produce the same result13:05
fungimaybe you could demonstrate it with a unit test and that reproducer would make it easier to work on a behavior correction13:05
avass++, just gotta find time to do that :)13:06
avassI don't understand why zuul would load configuration from the deleted file however.13:07
fungineither do i13:07
fungiunless maybe it was shadowing a similar configuration and creating a variant?13:08
avassthe new file had the exact same job+project config only with other additions13:08
fungimsuszko: as far as i know zuul doesn't have options to avoid merging to latest state of target branches. it wants to test what will get merged, and if you're not able to rebase a change to avoid merge conflicts with the actual branch state then those changes won't be able to merge anyway. the only times a reviewer should need to rebase a change are when that change has an ancestor which is also in review and13:09
fungihas received a new revision, or when the change merge-conflicts with the state of the branch to which it will ultimately merge13:09
fungiin both of those cases, a rebase is mandatory anyway because otherwise there's no way to merge the change following approval13:11
msuszkoI want to merge to the last branch state. Nothing unusual there.13:12
msuszkoWhen there are changes based on latest master, and label conditions are met Zuul will start parallel gate for these changes13:16
msuszkoMy scenario is: there is change 1 that is not yet approved, and change 2 that is approved, gated, and merged. Then change 1 gets aproval. With my pipeline config it will be gated, but not merged, because it is based on HEAD~1.13:25
*** flaper87 has quit IRC13:26
*** flaper87 has joined #zuul13:26
fungii still don't understand, zuul should still merge that change to HEAD to test it, and then ask gerrit to merge the change to the target branch (which will also use HEAD)13:27
msuszkohmm, i'll check it again13:31
fungiour reviewers in opendev only ever need to manually rebase a change when there's a merge conflict identified13:34
*** jpena|lunch is now known as jpena13:39
fungi(or if they're working on a strictly ordered change series and revise one of the changes, of course)13:55
*** rpittau is now known as rpittau|afk14:04
*** imtiazc has joined #zuul15:03
*** saneax has quit IRC15:19
*** bhagyashris is now known as bhagyashri|ruck15:20
tobiashavass: no, we only run one job per windows node15:46
avasstobiash: got it, we still have some things limiting us from doing that still15:48
openstackgerritJames E. Blair proposed zuul/zuul master: doc: update the config to mention the `load-branch` attribute  https://review.opendev.org/c/zuul/zuul/+/77290415:53
*** rlandy is now known as rlandy|mtg16:17
openstackgerritMerged zuul/zuul master: mqtt: document the trigger and enqueue time attribute  https://review.opendev.org/c/zuul/zuul/+/77621216:26
openstackgerritTristan Cacqueray proposed zuul/zuul master: web: disable logfile line rendering when the size exceed a threshold  https://review.opendev.org/c/zuul/zuul/+/77572616:28
openstackgerritMerged zuul/zuul master: merger fileschanges: remove self._update duplicates  https://review.opendev.org/c/zuul/zuul/+/77684216:31
*** vishalmanchanda has quit IRC16:36
openstackgerritMerged zuul/zuul master: Update react-json-view  https://review.opendev.org/c/zuul/zuul/+/77722316:43
openstackgerritGuillaume Chauvel proposed zuul/zuul master: merger cat: remove self._update duplicates  https://review.opendev.org/c/zuul/zuul/+/77743216:48
fungiran into a weird situation in opendev today. zuul left a verified -2 on https://review.opendev.org/777076 after it was approved, because one of the jobs reported "ERROR Failed to update project None"16:56
fungiin the executor's log it was raising BrokenProcessPool inside zuul.executor.server.ExecutorServer._innerUpdateLoop()16:57
fungithis is one of two places we catch BrokenProcessPool, the other being zuul.executor.server.AnsibleJob.execute()16:58
fungiwhen we catch it in AnsibleJob.execute() we additionally call AnsibleJob._send_aborted()16:59
fungiwhich results in the build being retried16:59
fungibut in ExecutorServer._innerUpdateLoop() a BrokenProcessPool just falls through to task.setComplete() in a finally clause, which i suppose is why we report error17:00
fungidoes that sound right? should we abort in the second case too?17:00
*** ykarel has quit IRC17:05
tobiashfungi: I think you're right17:06
corvusagree17:06
tobiashbut secondly brokenprocesspool hints towards oomkiller17:07
fungiooh, thanks, i'll check dmesg on that executor17:07
corvustristanC: do you agree with the proposal in comments on patchset 21 of https://review.opendev.org/720182 ?  (emit warnings then remove later?)17:07
fungitobiash: yep! coincides with the time we hit that exception... [Tue Feb 23 23:29:02 2021] Out of memory: Kill process 6396 (zuul-executor) score 93 or sacrifice child17:08
fungigood call17:08
fungianyway, i'll work on a patch17:08
tristanCcorvus: i agree, but still, this is asking operators to look for warning in their logs to check if they are affected17:13
corvustristanC: i was thinking we could make the warnings config errors; then they would show up in the web ui17:14
corvustristanC: they are non-fatal for existing errors, but if you make a patch that touches a file with an error, you have to fix it17:15
corvusif we don't like that last part, we could make a new "config warnings" feature, but i think the errors as written might be okay.17:16
tristanCi think the issue is that the zuul configuration model can be spread on multiple repositories / branch, and that is difficult to check and update to accomodate such change17:21
corvustristanC: right -- but with the config errors, you can click the bell icon in the web ui and see all of them17:21
corvusso we'd be able to open that up and then just run through the list fixing them17:21
tobiashmaybe more phases? First warnings, second config error, third removal?17:21
tristanCcorvus: the bell is only available per tenants, which is not very practical when you have many of them17:22
corvustobiash: well, that's the nice thing about config errors -- they could be phase 1+2 together, since they're basically non-fatal warnings for existing "errors" but you can't merge changes that make new errors17:23
corvustristanC: well, it's the owner of the tenant responsible for fixing this, not the operator17:23
corvus(i realise they could be the same person, but presumably an operator who owns many tenants is autogenerating pipelines or something and has tools for mass updates)17:24
corvusie, in opendev we would send out an announcement to service discuss and say "look at these warnings, go fix them please :)"17:24
tristanCcorvus: could zuul provides such mass updates tool?17:25
fungithe mechanism for pushing updates would differ depending on your code platform17:25
corvustristanC: well, i was speculating about the case of a zuul as a service where an operator has tools to auto-generate config projects.  you do that right?17:25
tristanCcorvus: to some extend, but we don't touch configuration outside of the tenant configuration17:26
corvusgot it17:27
tristanCa warning/error for pipeline queue has the potencial to affect many users, which do not necessarly care or understand the implication of the change, which seems quite specific to the circular dependency feature17:28
corvusit's not really specific to circular deps, it's more that since circular deps is the *second* thing that's on a project-pipeline that probably should have been on the project, let's move it.17:29
corvustristanC: honestly, if a user doesn't understand the change, then they probably shouldn't have named a queue in the first place17:30
tristanCif there is a solution to avoid that, or minimize the change, then it seems worthy to consider17:30
corvusa user will have named a queue only if they understand gating and how to construct a shared change queue with multiple projects.  users of those projects those are the only users who will see the warning "this setting should be moved from the project-pipeline definition to the project".17:31
corvusi don't see a good way to automate this.  either the tool will be exessively complex as it tries to understand all the different potential project stanzas (possibly in different repos) that would be appropriate to move the setting to, or excessively simple and just moves it 2 lines up in the same project definition.17:34
corvustristanC: we could write a tool to do the latter, and tell operators, "hey if you want to run this on all your projects, then figure out how to commit and merge the resulting changes, go for it".  would you find that tool helpful?17:36
tristanCi think that would helpful for such situations, e.g. get all the configuration model for a given tenant/deployment17:37
tristanCbut perhaps a good documentation on how to fix the config-error would work too17:38
corvusi think that is a generalizable rule -- that if you have it in a project-pipeline definition, you can just move it up a few lines to the project definition.  it should be valid as long as there isn't a conflicting value on a different pipeline.17:38
corvustristanC: yeah, i've written tools to mass-update zuul configs before.  my feeling is that it would be of limited use in this case, and just a good doc would be easier than a tool, but i'm open to it.17:39
corvustristanC: how about this: since it seems like we're in agreement on the overall approach, we merge these changes; i'll prepare a WIP change to make config errors, and we can look at what that looks like along with documentation on how to make the change, and see if that's sufficient or if we want to write a tool.  how's that sound?17:40
tristanCcorvus: that works for me17:44
openstackgerritJeremy Stanley proposed zuul/zuul master: Abort build if update raises BrokenProcessPool  https://review.opendev.org/c/zuul/zuul/+/77744117:45
openstackgerritMerged zuul/zuul master: doc: update the config to mention the `load-branch` attribute  https://review.opendev.org/c/zuul/zuul/+/77290417:51
*** jcapitao has quit IRC17:55
openstackgerritMerged zuul/zuul master: Support emitting warnings via zuul_return  https://review.opendev.org/c/zuul/zuul/+/65152617:59
*** jpena is now known as jpena|off17:59
*** iurygregory_ has joined #zuul18:04
*** iurygregory has quit IRC18:05
*** iurygregory_ is now known as iurygregory18:05
*** nils has quit IRC18:38
*** rlandy|mtg is now known as rlandy18:56
*** nils has joined #zuul18:56
*** harrymichal has joined #zuul19:08
*** jfoufas1 has quit IRC19:20
avassoh when was warnings added?19:30
fungiavass: looks like https://review.openstack.org/590442 added it, merged in 2018, included in zuul 3.3.019:36
tobiashjepp, long ago19:37
fungi"A new facility for reporting warning messages is added, and if the executor is unable to perform the mapping, or the file comment syntax is incorrect, a warning is reported."19:37
tobiashand now we can use that for emitting e.g. deprecation warnings in zuul-jobs :)19:38
avass++, I think we discussed that some time ago. thought it would need more work than that :)19:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: testing parallel bindep on multiple hosts  https://review.opendev.org/c/zuul/zuul-jobs/+/77746220:14
avassI cannot reproduce what happens with bindep not looping in parallel locally :(20:14
clarkbavass: do yo uhave a link to a failure I can look at?20:15
clarkbthat sounds like something that would be good for us to improve as well20:16
avassclarkb: not yet but that change should hopefully reproduce it20:17
fungioh, this is the "if we run bindep across multiple hosts, the execution happens serially" issye?20:19
fungiissue20:20
fungithat is, the bindep role not bindep the utility20:20
clarkbya I don't think the utility is a problem20:20
avassfungi: yeah exactly. I was trying to reproduce it with loops, include_task and roles but everything runs in parallel20:22
*** sshnaidm is now known as sshnaidm|afk20:28
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: testing parallel bindep on multiple hosts  https://review.opendev.org/c/zuul/zuul-jobs/+/77746220:29
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: testing parallel bindep on multiple hosts  https://review.opendev.org/c/zuul/zuul-jobs/+/77746220:41
avassoh interesting, and that make a bit more sense20:46
avassso it happens specifically because I'm dumping bindep content to a tempfile as a quickfix. ansible only does loops in parallel if it's looping over the same list for all hosts20:48
openstackgerritAlbin Vass proposed zuul/zuul-jobs master: DNM: testing parallel bindep on multiple hosts  https://review.opendev.org/c/zuul/zuul-jobs/+/77746220:58
ianwany clues on why https://review.opendev.org/c/opendev/system-config/+/777298 fails with "ERROR Project gerrit.googlesource.com/plugins/zuul-results-summary does not have the default branch master in 5s" ?21:02
ianwi think override is set correctly to "main" everywhere; e.g. https://opendev.org/opendev/system-config/src/branch/master/zuul.d/docker-images/gerrit.yaml#L8621:02
ianwopendev-tox-docs ... ERROR Project gerrit.googlesource.com/plugins/zuul-results-summary does not have the default branch master in 5s21:05
ianwit doesn't seem like these jobs should have requirements on this, i wonder where it comes from21:05
clarkbianw: the master branch doesn't exist though right? (so this isn't a random failure)21:06
ianwyeah, it's "main"21:06
avasswould bindep error if two packages were included twice in the same bindep.txt file? I'm working around the loop by just combining all bindep files21:09
clarkbavass: I want to say no21:09
clarkbI am not aware of any parser checks for uniqueness21:09
avasslet me create a test for that quickly. maybe there could be a problem doing that if the same package is included twice with different rules?21:10
fungiavass: it's even expected that you might have multiple entries for the same package with different profiles associated (though probably less efficient than combining the profiles into a single entry)21:12
fungiavass: though if you have conflicting "rules" (profiles) it's not particularly smart, the entries should be evaluated independently21:13
fungithe logic is basically, for each line: check the list of profiles, and if one of the conditions is met, check whether that package is present21:14
avassfungi: so combining two files would produce the same result as installing packages from them in series21:14
fungii dunno about that. bindep doesn't every install anything. its sole purpose is to ingest a list of names of expected packages and report which ones are not installed21:15
fungier, doesn't ever install anything21:15
mordredI feel like it does de-dupe the final list is outputs though doesn't it?21:16
avassfungi: yeah but the bindep role installs packages _from_ bindep output :)21:16
fungione of teh popular ways to use bindep, and the way we do in the zuul-jobs role, is to then take that returned list of missing package names and feed it to a package manager asking for them to be installed21:16
fungimordred: i don't recall if bindep dedupes the output, but it likely does. one of the use cases we had was to be able to take bindep files from multiple projects and shove them all into bindep in one go21:17
avassjust checked and I only get one line of 'neovim' from a bindep.txt with two lines of 'neovim' :)21:18
fungiavass: what platform(s) are you running into problems on? one thing we discovered for red hat based platforms was that unless you call yum/dnf on one package at a time, you can't catch failures of some package installs because they report success as long as at least one requested package was installed in a given run21:19
avassfungi: technically not a problem since I'm going to remove the tempfile later anyway. but looping over the bindep files runs sequentially for each host since the lists aren't equal21:20
fungioh, though i think we ended up mitigating that by rerunning bindep after calling the package manager to check that it reports no missing packages21:20
fungii was thinking we may have done the yum/dnf interface in the role to loop over individual packages instead of just passing the list in a single command21:21
fungibut that shouldn't be necessary as long as we check that no packages are missing afterward21:21
avasswone host has bindep_file: ["/tmp/tmp.fXYXN5k51M"] while the other had bindep_file: ["/tmp/tmp.j7EK91T9Cd/"] for example.21:22
openstackgerritMerged zuul/zuul master: merger cat: remove self._update duplicates  https://review.opendev.org/c/zuul/zuul/+/77743221:22
avassso ansible doesn't check len(list1) == len(list2) but list1 == list2 which can cause a performance hit if there are different bindep files for each host for whatever reason21:23
*** harrymichal has quit IRC21:42
*** harrymichal has joined #zuul21:42
*** harrymichal has quit IRC21:44
*** harrymichal has joined #zuul21:44
*** harrymichal has quit IRC21:46
*** harrymichal has joined #zuul21:46
*** harrymichal has joined #zuul21:46
*** hamalq has joined #zuul22:11
corvusi just ran the command "find broken" in my shell and chuckled22:27
corvus(as in: cd zuul/tests/fixtures/config/ && find broken) totally legit22:28
ianwcorvus: i'm heading down a rabbit hole and not sure i'm on the right path here with errors in https://review.opendev.org/c/opendev/system-config/+/777298.  if i depends-on a project with no "master" branch, but the jobs do not have that project as a required-project (and hence no branch override) ... what is supposed to happen?22:40
corvusianw: did you set the default branch?  https://zuul-ci.org/docs/zuul/reference/project_def.html#attr-project.default-branch22:50
corvusianw: also, wasn't this working?  what changed?22:51
*** harrymichal has quit IRC22:51
ianwi thought it was too ... hence why i've got myself so confused :)22:51
*** harrymichal has joined #zuul22:51
corvusianw: oh, well if nothing changed then we should consider recent changes to zuul may have introduced a bug22:52
openstackgerritJames E. Blair proposed zuul/zuul master: Emit config warnings if shared queues on pipelines  https://review.opendev.org/c/zuul/zuul/+/77747922:52
*** harrymichal has quit IRC22:52
*** harrymichal has joined #zuul22:53
*** harrymichal has joined #zuul22:53
ianwcorvus: i can't set the default branch @ https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L64 right?22:54
corvusno it's in the project stanza22:54
corvusianw: can do it here https://opendev.org/opendev/project-config/src/branch/master/zuul.d/projects.yaml22:55
corvusianw: i can't think of any recent changes that could affect that (and i've looked at a couple that are barely related).  so if it is a recent change, it's something really unexpected.22:58
ianwcorvus: hrm, so i can add a project "name: gerrit.googlesource.com/plugins/zuul-results-summary" with just a default-branch?  i don't think i'd considered that22:58
corvusianw: yep22:59
corvustobiash, tristanC: i did a little more work and made a softer config loading warnings feature.  so you can still even propose a change that adds the deprecated syntax and it will only emit a warning.  and existing deprecations will show up in the web ui config error list, but nothing will actually be broken.  i think that should give us a nice gentle ramp for deprecation: https://review.opendev.org/77747923:01
tristanCcorvus: thank you, that sounds like very user friendly23:03
*** rlandy is now known as rlandy|bbl23:13
*** harrymichal has quit IRC23:18
*** harrymichal has joined #zuul23:18
*** ajitha has quit IRC23:26
*** harrymichal has quit IRC23:38
*** harrymichal has joined #zuul23:38
*** harrymichal has quit IRC23:40

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!