| @dancywmf:matrix.org | Scenario: | 17:42 |
|---|---|---|
| Using Zuul + Gitlab driver. | ||
| I have a config-project named "zuul-config" with zuul.d/*.yaml files | ||
| defined in branch "main". zuul.d/jobs.yaml defines a job named | ||
| "gitlab-pipeline" which runs a particular playbook. | ||
| I have an untrusted project named "mediawiki/core" with a .zuul.yaml file in branch "master" which has: | ||
| ``` | ||
| - project: | ||
| check: | ||
| jobs: | ||
| - gitlab-pipeline: | ||
| required-projects: &required | ||
| - list-of-requred-projects... | ||
| ``` | ||
| I create a commit, push it to a topic branch (something like | ||
| "master-I1"), and create a | ||
| merge request w/ the topic branch as the source branch and "master" as | ||
| the target branch. Everything works as expected when I create a merge | ||
| request... the gitlab-pipeline job runs for the MR. | ||
| Problem: | ||
| If I create a second commit on top of the first one, push it to a | ||
| different topic branch (e.g., "master-I2"), then I create a second MR | ||
| with "master-I2" as the source branch and "master-I1" as the target | ||
| branch, the gitlab-pipeline job is NOT created for this MR. | ||
| Note: This pattern of merge requests is arranged by the Gerritlab tool | ||
| (https://github.com/yaoyuannnn/gerritlab). | ||
| I haven't been able to figure out why the job is not created. My | ||
| intuition says that the .zuul.yaml exists in all of the branches | ||
| (master, master-I1, and master-I2) so it should be useable in all | ||
| circumstances, but clearly my mental model doesn't align with how | ||
| things really work. I tried adding "branches: .*" in various places | ||
| (in the job configuration in "zuul-config" or in the job configuration | ||
| in ".zuul.yaml" on "master") but that didn't get things working. | ||
| I do see this difference in zuul-scheduler log messages between the | ||
| working and non-working case: | ||
| Working (MR#6 has target branch of "master"): | ||
| Project <ProjectConfig gitlab.devtools.wmcloud.org/repos/mediawiki/core source: repos/mediawiki/core/.zuul.yaml@master {ImpliedBranchMatcher:master}> matched item <QueueItem 3dd02675e0094bc39ce6a68bf1164ffc live for <Change 0x7f5c30177650 project: repos/mediawiki/core number: 6 patchset: 8a83609f64ca8d23b860 | ||
| f623fb9e5e39c93f1dd6 updated: 1704384483 state: open approval: approved> in check> | ||
| Not-working (MR#7 has target branch of "master-I1"): | ||
| Project <ProjectConfig gitlab.devtools.wmcloud.org/repos/mediawiki/core source: repos/mediawiki/core/.zuul.yaml@master {ImpliedBranchMatcher:master}> did not match item <QueueItem d51c4cb72a8e43f1a1c3f9a7997611f2 live for <Change 0x7f5c2056a510 project: repos/mediawiki/core number: 7 patchset: 296540103868be | ||
| b1ae9d3b55d71ddad685474291 updated: 1704384430 state: open> in check> | ||
| So, "matched item" vs "did not match item". | ||
| I'm hoping someone can help me understand Zuul's behavior better and | ||
| ideally propose a solution. Thanks in advance! | ||
| @clarkb:matrix.org | dancywmf: there should be an event id in the log messages. If you grep that event id value in the logs you should get a fairly complete accounting of what is happening according ot zuul. Note you may need to turn on debug logging ? | 17:51 |
| That said I think the issue is likely due to the second merge request having a target branch that is itself a topic branch? | ||
| @clarkb:matrix.org | Is it possible that zuul is aware of some sort of merge policy that makes such a situation unmergeable? | 17:51 |
| @clarkb:matrix.org | an alternative you may want to explore is using Zuul's depends-on rather than stacking merge requests in that weird way. I know github doesn't support such a workflow and maybe it isn't quite right in gitlab either? | 17:52 |
| @thcipriani:matrix.org | Both merge requests are in a mergable state. We thought about "Depends-on" vs. stacking. The problem is the dependent patchset may not be mergable without its dependency. The only way we could figure to express a dependency among merge requests (akin to Gerrit's stacked patchsets) is by stacking merge requests in this way. You may be right about the second merge request targeting a topic branch—is there some way to work around that? | 18:27 |
| @clarkb:matrix.org | How is the second MR mergable of its target isn't a valid target? | 18:29 |
| @clarkb:matrix.org | I don't use gitlab so I'm not sure if that is a thing or not. I suppose it may be | 18:29 |
| @clarkb:matrix.org | Depends-on enforces a strict order and should address the dependency issue. That is what it is for. I mentioned depends-on because I know GitHub users have to rely on it due to a lack of built in dependencies between PRs | 18:31 |
| @thcipriani:matrix.org | there is a subtle way to make explicit dependencies between PRs. If you have a PR that targets `master` (topic branch `master-I1`) and you want to make a new PR that depends on that PR: create a topic branch `master-I2` and then instead of targeting `master` you target `master-I1`. So the chain looks like: `master-I2` → `master-I1` → `master`. | 18:37 |
| This is a pain to do manually, but there are a lot of tools that do this for github/gitlab users (the pattern is documented in a few places, notably: https://www.michaelagreiler.com/stacked-pull-requests/ ) | ||
| Basically, this fakes Gerrit's model in a pull-request based system. | ||
| @clarkb:matrix.org | Looks like gitlab does understand dependencies. If that is how it represents them (the target is a MR) then the gitlab driver in zuul may need to be updated to transitively discover the ultimate target? However that also implies you can only merge to the same branch as your dependencies which seems wrong | 18:38 |
| @clarkb:matrix.org | I would look at the debug logs for the event I'd as mentioned earlier to get more info on the state in gitlab as zuul sees it and work backward from there | 18:39 |
| @thcipriani:matrix.org | > that also implies you can only merge to the same branch as your dependencies which seems wrong | 18:44 |
| It's possible to merge down (i.e., to merge `master-I2` into `master-I1`) depending on your config. And that's confusing, yes. | ||
| The log pasted earlier is some of the more relevant bits from the debug log. I think the part that I couldn't figure out was ` {ImpliedBranchMatcher:master}> did not match item`—what is zuul looking for there? the target branch? `.zuul.yaml` is present on `master-I2`, what does it need from master? | ||
| @clarkb:matrix.org | The implied branch matcher comes from the branch the config is defined on. And then for dynamic configs should be related to the target of the change | 18:45 |
| @clarkb:matrix.org | I suspect that since the target is not actually master things are getting confused somewhere. The actual gitlab event data indicating these attributes is what I think may be helpful grepping the event id in the logs | 18:46 |
| @clarkb:matrix.org | And not just not master but the target is some temporary MR topic branch which zuul may ignore as a non mergable invalid target? | 18:47 |
| @thcipriani:matrix.org | hrm. dancywmf has been deeper in this than I, lemme see if I can dig up relevant event logs. IIRC, `master-I2` sees a target of `master-I1` (which, I'd guess, didn't exist at zuul startup in case that matters). | 18:52 |
| @dancywmf:matrix.org | Hi Clark! Thanks for responding. I just now tried adding a `Depends-In:` to the second MR, pointing to the first. I do see in the logs that Zuul is recognizing this relationship (`zuul.Pipeline.wikimedia.check: ...Needed change is already ahead in the queue`). However, the end result is the same (`Project <projectconfig> repos/mediawiki/core/.zuul.yaml@master {ImpliedBranchMatcher:master}> did not match item `) and no job created. | 18:59 |
| @dancywmf:matrix.org | (I'll answer your other questions shortly) | 19:08 |
| @clarkb:matrix.org | > <@dancywmf:matrix.org> Hi Clark! Thanks for responding. I just now tried adding a `Depends-In:` to the second MR, pointing to the first. I do see in the logs that Zuul is recognizing this relationship (`zuul.Pipeline.wikimedia.check: ...Needed change is already ahead in the queue`). However, the end result is the same (`Project <projectconfig> repos/mediawiki/core/.zuul.yaml@master {ImpliedBranchMatcher:master}> did not match item `) and no job created. | 19:09 |
| Did you change the target of the second MR yo master instead of the first MR? | ||
| @dancywmf:matrix.org | I did not, since the second MR (for the purposes of this discussion) has changes that depends on the changes made in the first MR. | 19:10 |
| @clarkb:matrix.org | right depends-on should deal with those | 19:11 |
| @dancywmf:matrix.org | in other words, the second MR is not rebaseable on top of master | 19:12 |
| @clarkb:matrix.org | and I think the problem is due to the target being an MR if we can try both targeting master and use depends on we should at least be able to determine if the jobs trigger | 19:12 |
| @clarkb:matrix.org | another thing you could try is modifying the job in the second MR so that it is dynamically configured for that MR | 19:13 |
| @clarkb:matrix.org | I think what is happening here is the second MR is trying to execute a job that is defined in the master branch statically. Since there is no dynamic update to the job zuul is looking to see if the current branch matches the branch for the defined job. It does not because neither the current branch of the MR nor the target of the MR are the branch where the job is defined (master) | 19:15 |
| @clarkb:matrix.org | dancywmf: thcipriani in the case of these MR/PR dependencies master <- A <-B when A merges to master we get master' which includes A. Do github/gitlab know to automatically merge B to master' at this point or do you need to update B to target master'? | 19:17 |
| @thcipriani:matrix.org | gitlab is smart enough to retarget B so B then thinks master is the target (there are some fiddly details here and it's all an async queue inside gitlab so sometimes things go wrong, but in the general case that's true) | 19:19 |
| @clarkb:matrix.org | ok in that case I suspect that the zuul gitlab driver may need to transitively inspect branch targets. But I don't know enough about the gitlab driver or gitlab to say for sure | 19:22 |
| @clarkb:matrix.org | basically in the situation I describe above Zuul could discover that the second MR is ultimately looking to merge to master and treat master as the logical target | 19:22 |
| @clarkb:matrix.org | then it would match the implied branch matcher | 19:23 |
| @clarkb:matrix.org | ideally gitlab would provide that info in the event it sends and we just need to check for that attribute | 19:24 |
| @clarkb:matrix.org | https://gitlab.com/gitlab-org/gitlab/-/issues/11393 is an itneresting read (and I recognize some of the names) | 19:38 |
| @clarkb:matrix.org | basically you can only do dependencies one deep? | 19:39 |
| @clarkb:matrix.org | https://gitlab.com/gitlab-org/gitlab/-/issues/12551 indicates that there isn't a programmatic way to inspect dependencies | 19:39 |
| @clarkb:matrix.org | the practice of touting the numebr of seats you pay for in order to get attention on issues is interesting to me as well | 19:40 |
| @thcipriani:matrix.org | heh, tell me about it, and it doesn't seem to work that well :\ | 20:14 |
| @thcipriani:matrix.org | yeah, transitive dependences seem like we'd have to crawl the tree as far as I can tell. I know there's a branch cache in zuul (vaguely) but I do wonder how expensive that would be. | 20:16 |
| @clarkb:matrix.org | well until the next release we would only have to go one hop beacuse they don't do deeper dependencies | 20:16 |
| @clarkb:matrix.org | but looks like 11393 is fixed with the latest release (not sure if that is released yet or not) | 20:16 |
| @fungicide:matrix.org | but is that the next release of gitlab enterprise or ce? | 20:39 |
| @clarkb:matrix.org | fungi: I think this feature is enterprise only | 20:39 |
| @fungicide:matrix.org | yeah, from what i understand they often hold back those sorts of features so the open source version never gets them (or gets them years later) | 20:40 |
| @fungicide:matrix.org | i know folks from debian (they have a ce instance) who have submitted code for new features in it only to have them rejected because it would compete with exclusive enterprise functionality | 20:41 |
| @thcipriani:matrix.org | we are in the same boat as debian, using the open source version. Similar experience. | 21:39 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!