| @jim:acmegating.com | zuul-maint: how does this look for a zuul release? commit c1b0a00c607c13c52991e14056e6bcac4a49d5d5 (HEAD -> master, tag: 8.3.1, origin/master, gerrit/master, refs/changes/43/883443/3) | 01:03 |
|---|---|---|
| @jim:acmegating.com | (tomorrow at this point) | 01:03 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/nodepool] 883463: Use cached ids in node iterator more often https://review.opendev.org/c/zuul/nodepool/+/883463 | 03:04 | |
| @rancher:matrix.org | > <@clarkb:matrix.org> the error with getProjectMetadata() implies the project has no config. I'm not sure this is a gitlab issue. It may just be an improper/incomplete configuration. I think you need to look in your logs for where that project is getting loaded to see the initial error | 07:25 |
| But it's strange that the same projects work fine on the older version of GitLab. Even the default "zuul/zuul-jobs" (opendev.org) throws the same error on the newer versions: | ||
| opendev.org: | ||
| untrusted-projects: | ||
| - zuul/zuul-jobs: | ||
| include: | ||
| - job | ||
| @rancher:matrix.org | [connection "opendev.org"] | 07:26 |
| name=opendev | ||
| driver=git | ||
| baseurl=https://opendev.org | ||
| @rancher:matrix.org | I think these are the only references to that project. | 07:27 |
| @rancher:matrix.org | * I think these are the only references to that project, it should work, right? | 07:27 |
| @newbie23:matrix.org | hi guys, | 08:01 |
| is it possible to configure Nodepool + Azure Compute Driver to define some sort of retention policy | ||
| so that nodes are kept around for a while and, if still up and running, they are reused for jobs? | ||
| Context: reusing nodes will enable us several use cases involving using cached data, but just using pre-defined pre-existing nodes might be expensive as there might be no workload | ||
| @rancher:matrix.org | I now tried with github.com, but the following errors show: https://privatebin.net/?74639cd6d74df993#79prpzrqVynsUcQ9Cqtgfy5TyeyZPuugMFwffU2fo9BG | 13:44 |
| There's no mention of "gitlab" anywhere in config files, and default GitHub reference pipelines are used. | ||
| @clarkb:matrix.org | your yaml tenant config is where I would expect the gitlab to be defined that is failing to map to the connection config | 13:47 |
| @clarkb:matrix.org | What I was trying to get at before is that the failure is happening before the traceback you previously posted. You need to look in early startup to see why there is no metadata for that setup. A keyerror like this could be causing it | 13:48 |
| @clarkb:matrix.org | I think | 13:48 |
| @clarkb:matrix.org | > <@newbie23:matrix.org> hi guys, | 13:49 |
| > is it possible to configure Nodepool + Azure Compute Driver to define some sort of retention policy | ||
| > so that nodes are kept around for a while and, if still up and running, they are reused for jobs? | ||
| > | ||
| > Context: reusing nodes will enable us several use cases involving using cached data, but just using pre-defined pre-existing nodes might be expensive as there might be no workload | ||
| The metastatic and static drivers are going to be the ones you want to look at. Metastatic won't schedule them in an intelligent reuse manner. But depending on your needs this may be fine. With static you would set up static nodes and specific labels that could be used. | ||
| @newbie23:matrix.org | > <@clarkb:matrix.org> The metastatic and static drivers are going to be the ones you want to look at. Metastatic won't schedule them in an intelligent reuse manner. But depending on your needs this may be fine. With static you would set up static nodes and specific labels that could be used. | 13:56 |
| Thanks for the pointer! I will look into it. | ||
| @rancher:matrix.org | > <@clarkb:matrix.org> your yaml tenant config is where I would expect the gitlab to be defined that is failing to map to the connection config | 14:05 |
| Yeah, but I didn't define one, only Github. I suppose the syntax is correct, correct me if I'm wrong. I'm new to this. | ||
| @rancher:matrix.org | > <@clarkb:matrix.org> What I was trying to get at before is that the failure is happening before the traceback you previously posted. You need to look in early startup to see why there is no metadata for that setup. A keyerror like this could be causing it | 14:06 |
| Is every log displayed by default when cloning the repository (Quick-Start Guide)? What should I look for? I didn't see any error apart from gerrit ones that I don't use. | ||
| @rancher:matrix.org | > <@clarkb:matrix.org> What I was trying to get at before is that the failure is happening before the traceback you previously posted. You need to look in early startup to see why there is no metadata for that setup. A keyerror like this could be causing it | 14:06 |
| * Is every log displayed by default when cloning the repository (Quick-Start Guide)? What should I look for? I didn't see any error apart from Gerrit ones, which I don't use. | ||
| @rancher:matrix.org | * Is every log displayed by default when cloning the repository (Quick-Start Guide)? What should I look for? I didn't see any error apart from Gerrit ones, which I don't use. I removed the gerrit connection string. | 14:07 |
| @rancher:matrix.org | * Is every log displayed by default when cloning the repository (Quick-Start Guide)? What should I look for? I didn't see any error apart from Gerrit ones, which I don't use. I removed the Gerrit connection string. | 14:07 |
| @jim:acmegating.com | Rancher: you might find the configurator at https://acmegating.com/acme-enterprise-zuul/#start helpful. it will help you make the configuration files with gitlab. you can use it on its own (it produces a docker-compose file like the quick-start), or just take the gitlab parts and splice them into the zuul quick-start. | 14:09 |
| @jim:acmegating.com | 8.3.1 promote succeeded, i'll send out the email shortly | 14:55 |
| @fungicide:matrix.org | thanks! | 15:01 |
| @jim:acmegating.com | Clark: okay i held a quickstart node and i get different behavior on it than i do locally | 16:49 |
| @jim:acmegating.com | locally when i run 'host gerrit' in a pod i get a response, but on the test node, i get NXDOMAIN | 16:50 |
| @jim:acmegating.com | local dns: | 16:50 |
| @jim:acmegating.com | search dns.podman | 16:51 |
| nameserver 10.89.1.1 | ||
| @jim:acmegating.com | test node: | 16:51 |
| @jim:acmegating.com | nameserver 8.8.8.8 | 16:51 |
| nameserver 8.8.4.4 | ||
| @jim:acmegating.com | same podman/podman-compose versions | 16:52 |
| @jim:acmegating.com | maybe something about podman-dnsname | 16:54 |
| @clarkb:matrix.org | I wonder if that is a side effect of some plugin installation? Like containernetworking-plugin? | 16:54 |
| @jim:acmegating.com | exactly | 16:54 |
| @jim:acmegating.com | i have golang-github-containernetworking-plugin-dnsname locally and it's not on the remote node | 16:54 |
| @jim:acmegating.com | how does adding that to ensure-podman (without an option to enable, so always installed) sound? | 16:55 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul-jobs] 883552: Add podman dns plugin to ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/883552 | 17:00 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed on behalf of Tristan Cacqueray: [zuul/zuul] 687135: Replace docker by podman for quick-start https://review.opendev.org/c/zuul/zuul/+/687135 | 17:01 | |
| @jim:acmegating.com | Clark: ^ | 17:01 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul-jobs] 883552: Add podman dns plugin to ensure-podman https://review.opendev.org/c/zuul/zuul-jobs/+/883552 | 18:11 | |
| @michael_kelly_anet:matrix.org | Is there an (easy) way to exclude a specific repo/project from a pipeline? eg: I'm PoCing some check jobs for someone in a large repo and I don't want Zuul to just start auto-merging everyone's changes (yet) because multiple different teams live in this repo and it'll be surprising. | 19:05 |
| @michael_kelly_anet:matrix.org | Or is the sane thing to do here just to create a separate tenant to handle this casE? | 19:06 |
| @michael_kelly_anet:matrix.org | * Or is the sane thing to do here just to create a separate tenant to handle this case? | 19:06 |
| @jim:acmegating.com | just don't add it to the pipeline | 19:09 |
| @fungicide:matrix.org | projects have pipelines explicitly added to them, what's the situation where you're ending up with a project added that you didn't intend? | 19:09 |
| @fungicide:matrix.org | maybe it's coming in via a project-template? in that case, the simple solution is to probably have a separate project-template which omits the problem pipeline and apply that to the project instead, or just don't use a project-template for that project, or split the existing project-template to move the pipeline in question into a second template that most projects get both of but that project only gets the one that doesn't have that pipeline... | 19:11 |
| @jim:acmegating.com | or maybe via a catch-all project regex, in which case adjust the regex | 19:11 |
| @michael_kelly_anet:matrix.org | This is happening before I've even added a zuul.yaml to the repo. :) | 19:12 |
| @fungicide:matrix.org | do you add pipelines to that project in your config project maybe? | 19:12 |
| @michael_kelly_anet:matrix.org | That project only exists in Zuul's universe in as much as I've added it to my tenant config. The config project has no awareness of it as such. | 19:13 |
| @fungicide:matrix.org | and there's no project pattern matching it in your config project that might add pipelines? | 19:14 |
| @michael_kelly_anet:matrix.org | Ah, there you have it. | 19:14 |
| @fungicide:matrix.org | but to restate, zuul won't automatically add pipelines for a project just by adding it to the tenant config | 19:14 |
| @michael_kelly_anet:matrix.org | I get what you're saying now. | 19:14 |
| @michael_kelly_anet:matrix.org | Yea, I think content in my config project is derived from the quickstart/walkthrough so yes, the regex in there is matching the entire world. | 19:15 |
| @fungicide:matrix.org | that'd do it. so like corvus said, adjust that pattern | 19:15 |
| @michael_kelly_anet:matrix.org | Sounds good. | 19:16 |
| @michael_kelly_anet:matrix.org | Thanks for the pointer. :) | 19:16 |
| @michael_kelly_anet:matrix.org | I was a bit excited that folks were going to hunt me down like the monster that I am :) | 19:16 |
| @fungicide:matrix.org | my monster-hunting days are long over | 19:16 |
| @michael_kelly_anet:matrix.org | Hopefully more folks will join the religion and see the light. | 19:17 |
| @clarkb:matrix.org | corvus: https://zuul.opendev.org/t/zuul/build/7f3ac7f8dd4f4d55a0ad41b4bcc0e30c/log/job-output.txt#734 is https://github.com/containers/podman/issues/6368#issuecomment-669836088 I think | 20:04 |
| @clarkb:matrix.org | I'm not sure how to plumb that into podman compose though | 20:04 |
| @clarkb:matrix.org | I think I haven't hit this in opendev bceaues I'm not trying to run as a regular user there | 20:06 |
| @clarkb:matrix.org | so that might be a workaround here if all else fails | 20:07 |
| @jim:acmegating.com | Clark: oh interesting; look a few lines above that | 20:10 |
| @jim:acmegating.com | 720-724 | 20:10 |
| @jim:acmegating.com | (this is during the image build of the "node" container) | 20:10 |
| @jim:acmegating.com | that suggests that it is using cgroupfs but that's failing? | 20:11 |
| @clarkb:matrix.org | oh ya it is | 20:11 |
| @jim:acmegating.com | but at any rate, for our "test what the user runs" approach, we may want to see about using systemd anyway? | 20:11 |
| @clarkb:matrix.org | yes and systemd is there and I think when you ssh in you get a session of some sort. But maybe it is too minimal forwhat podman wants | 20:12 |
| @jim:acmegating.com | so maybe we should try the `sudo loginctl enable-linger 1000` or... at least learn what that is :) | 20:12 |
| @jim:acmegating.com | Clark: i still have the held node at 213.32.76.175 | 20:13 |
| @clarkb:matrix.org | looks like sshd pam config is what sets up systemd sessions via ssh | 20:13 |
| @jim:acmegating.com | yeah, the loginctl linger thing doesn't look like what we want. | 20:15 |
| @jim:acmegating.com | Clark: wow i added the cgroup cli and it's working | 20:16 |
| @jim:acmegating.com | so i guess when it said "falling back on cgroupfs" it really meant "just kidding" | 20:16 |
| @clarkb:matrix.org | hahaha | 20:18 |
| @clarkb:matrix.org | ok | 20:18 |
| @jim:acmegating.com | https://github.com/containers/podman-compose/issues/209 | 20:19 |
| @clarkb:matrix.org | looking at the session stuff our sshd_config does say use pam and pam sshd config includes common-session which includes pam-systemd.so which should do it | 20:19 |
| @clarkb:matrix.org | but i don't have the XDG_SESSION_BUS env var logging in over ssh | 20:19 |
| @jim:acmegating.com | that last comment appears to work | 20:19 |
| @jim:acmegating.com | `podman-compose --podman-build-args="--cgroup-manager cgroupfs" ...` | 20:20 |
| @jim:acmegating.com | but we might want to consider https://github.com/containers/podman-compose/issues/209#issuecomment-655276317 | 20:20 |
| @jim:acmegating.com | and maybe doing that in the ensure-podman role? | 20:21 |
| @clarkb:matrix.org | I think the config option would be good if we think othe rpeople won't have this issue. Theoretically they won't if logged in via a desktop | 20:21 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Tristan Cacqueray: [zuul/zuul] 687135: Replace docker by podman for quick-start https://review.opendev.org/c/zuul/zuul/+/687135 | 20:32 | |
| @clarkb:matrix.org | hopefully I did that right. We should know soon enough | 20:32 |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 883319: Refresh builds even less https://review.opendev.org/c/zuul/zuul/+/883319 | 20:35 | |
| @jim:acmegating.com | Clark: looks good technically (but it should actually be in a different playbook; i can move it once it works) -- but should we put that in the ensure-podman role too? | 20:36 |
| @jim:acmegating.com | er s/too/instead | 20:36 |
| @clarkb:matrix.org | I thought about that, but I think this is only necessary if you aren't in a systemd session with dbus info | 20:36 |
| @clarkb:matrix.org | and so it is a question of whether or not we expect that problem to be universal enough on ubuntu at least to just hard code it | 20:37 |
| @jim:acmegating.com | Clark: given that zuul-jobs roles are basically for running on systems like this, i think it would be pretty universal? | 20:37 |
| @clarkb:matrix.org | thats a good point and I did confirm that we appear to have the pam stuff that would make this work. | 20:38 |
| @clarkb:matrix.org | I also hcecked that libpam-systemd is installed and it is | 20:38 |
| @clarkb:matrix.org | if this works I can remove it from the zuul change and add it to the jammy tasks for ensure-podman | 20:39 |
| @jim:acmegating.com | the libpam-systemd stuff is required for the cgroupfs manager? | 20:39 |
| @clarkb:matrix.org | I think libpam-systemd is what sshd would trigger via pam to create systemd + dbus session stuff in your login environment | 20:40 |
| @clarkb:matrix.org | and that isn't happenign whihc makes me think we aren't missing anything with our sshd and pam setup and instead its something they just aren't doing | 20:40 |
| @clarkb:matrix.org | wouldn't surprise me if on rhel you get this but not on ubuntu | 20:40 |
| @jim:acmegating.com | okay, i think i grok now what you were saying, thx :) | 20:40 |
| @jim:acmegating.com | > <@clarkb:matrix.org> if this works I can remove it from the zuul change and add it to the jammy tasks for ensure-podman | 20:40 |
| sounds like a plan | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Tristan Cacqueray: [zuul/zuul] 687135: Replace docker by podman for quick-start https://review.opendev.org/c/zuul/zuul/+/687135 | 20:54 | |
| @jim:acmegating.com | Clark: i'm seeing errors about dependencies... i wonder if we should just remove them from the compose file since they don't do anything useful anyway | 21:22 |
| @clarkb:matrix.org | corvus: hrm I think on the opendev gitea side of things we use them (dependenceis) too and they work. | 21:23 |
| @clarkb:matrix.org | but ya if they aren't helpful the ncleaning them up seems fine | 21:23 |
| @jim:acmegating.com | i mean... in docker they "work" inasmuch as they'll cause things to start up in a certain order, but they don't actually cause docker to wait until the dependency is up and running | 21:24 |
| @jim:acmegating.com | so as long as things are fast enough, they can make startup more smooth, but if anything is slow, then they might as well not be there | 21:24 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/bb0abe9346ed45e898a4752756722dd6/log/job-output.txt#1287 is this what you think is a dependency error? | 21:25 |
| @jim:acmegating.com | if i'm reading the logs right, in podman it looks like we might be waiting on builds | 21:25 |
| @jim:acmegating.com | Clark: no; `2023-05-18 21:16:49.074241 | ubuntu-jammy | Error: error generating dependency graph for container 5e613a47ab513f153865b6d6fd87adcac6952c9c1932cc2b174d9d1f53d88b3f: container 5bb4da0c942019132b5be057f2e86d0ca074bc1b8cf2be2e5dbf52091e0629ba depends on container 6e20e3f34ef8bfcfea4b9546004c11292fa7ae812dc6c1699990a1aae07a5483 not found in input list: no such container` from the streaming log | 21:26 |
| @jim:acmegating.com | i think that says "executor depends on scheduler" ; honestly i'm not sure what the deal with the scheduler is | 21:27 |
| -@gerrit:opendev.org- Clark Boylan proposed: [zuul/zuul-jobs] 883593: Force cgroupfs cgroup manager with podman on ubuntu https://review.opendev.org/c/zuul/zuul-jobs/+/883593 | 21:27 | |
| @clarkb:matrix.org | ok ya its trying to ru nthe executor and failing to find its dependencies maybe due to transitivitiy too? | 21:31 |
| @clarkb:matrix.org | but thats the first non zero exit code so theoretically every container is running? | 21:32 |
| @clarkb:matrix.org | I wonder if it is gerritconfig in particular since it is fire and foreget so may not be running anymore at that point | 21:33 |
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Tristan Cacqueray: [zuul/zuul] 687135: Replace docker by podman for quick-start https://review.opendev.org/c/zuul/zuul/+/687135 | 21:33 | |
| @clarkb:matrix.org | lets try removing just gerritconfig to start | 21:33 |
| -@gerrit:opendev.org- Michael Kelly proposed: | 21:50 | |
| - [zuul/zuul-operator] 853695: Prefix zuul-specific resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/853695 | ||
| - [zuul/zuul-operator] 853696: Prefix nodepool specific resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/853696 | ||
| - [zuul/zuul-operator] 867938: Prefix managed resources with instance name https://review.opendev.org/c/zuul/zuul-operator/+/867938 | ||
| - [zuul/zuul-operator] 861488: helm: Add a basic helm chart for zuul-operator https://review.opendev.org/c/zuul/zuul-operator/+/861488 | ||
| - [zuul/zuul-operator] 862390: helm: Add cert-manager as optional dependency https://review.opendev.org/c/zuul/zuul-operator/+/862390 | ||
| - [zuul/zuul-operator] 863191: helm: Add pxc-operator as optional dependency https://review.opendev.org/c/zuul/zuul-operator/+/863191 | ||
| - [zuul/zuul-operator] 865945: test: Restructure zuul-operator-functional-k8s layout https://review.opendev.org/c/zuul/zuul-operator/+/865945 | ||
| - [zuul/zuul-operator] 863579: test: Introduce Helm-based functional test https://review.opendev.org/c/zuul/zuul-operator/+/863579 | ||
| - [zuul/zuul-operator] 863474: k8s: Provide an option to disable PXC operator installer https://review.opendev.org/c/zuul/zuul-operator/+/863474 | ||
| - [zuul/zuul-operator] 866231: k8s: Clean up cert-manager installer https://review.opendev.org/c/zuul/zuul-operator/+/866231 | ||
| - [zuul/zuul-operator] 863475: k8s: Provide an option to disable cert-manager installation https://review.opendev.org/c/zuul/zuul-operator/+/863475 | ||
| - [zuul/zuul-operator] 863586: helm: Remove unnecessary CRD access from clusterrole https://review.opendev.org/c/zuul/zuul-operator/+/863586 | ||
| - [zuul/zuul-operator] 863476: k8s: Enable administrator to limit the watched namespace scope https://review.opendev.org/c/zuul/zuul-operator/+/863476 | ||
| - [zuul/zuul-operator] 863477: k8s: Allow use of a default image version besides latest https://review.opendev.org/c/zuul/zuul-operator/+/863477 | ||
| - [zuul/zuul-operator] 863571: web: Enable custom metadata for Service resources https://review.opendev.org/c/zuul/zuul-operator/+/863571 | ||
| - [zuul/zuul-operator] 861279: bug: Select scheduler pod based on instance name on update https://review.opendev.org/c/zuul/zuul-operator/+/861279 | ||
| - [zuul/zuul-operator] 863572: bug: Properly parameterize zookeeper-client-tls everywhere https://review.opendev.org/c/zuul/zuul-operator/+/863572 | ||
| - [zuul/zuul-operator] 866295: k8s: Remove unused ClusterRole from rbac-admin https://review.opendev.org/c/zuul/zuul-operator/+/866295 | ||
| - [zuul/zuul-operator] 866296: helm: Support clusteradmin role binding https://review.opendev.org/c/zuul/zuul-operator/+/866296 | ||
| - [zuul/zuul-operator] 866297: k8s: Added deploy cluster admin template https://review.opendev.org/c/zuul/zuul-operator/+/866297 | ||
| - [zuul/zuul-operator] 866406: k8s: Inject rbac.yaml into operator.yaml https://review.opendev.org/c/zuul/zuul-operator/+/866406 | ||
| - [zuul/zuul-operator] 866407: k8s: Provide tools and checker for deploy templates https://review.opendev.org/c/zuul/zuul-operator/+/866407 | ||
| - [zuul/zuul-operator] 863439: doc: Rework install doc to cover both template and helm install https://review.opendev.org/c/zuul/zuul-operator/+/863439 | ||
| - [zuul/zuul-operator] 883595: deploy: Surface disk_limit_per_job in operator CRD https://review.opendev.org/c/zuul/zuul-operator/+/883595 | ||
| - [zuul/zuul-operator] 883596: Broaden scope of potential config updates https://review.opendev.org/c/zuul/zuul-operator/+/883596 | ||
| -@gerrit:opendev.org- Clark Boylan proposed on behalf of Tristan Cacqueray: [zuul/zuul] 687135: Replace docker by podman for quick-start https://review.opendev.org/c/zuul/zuul/+/687135 | 22:10 | |
| @clarkb:matrix.org | corvus: ^ I think that is really close now. It is failing to get builds listed back after some down and up stuff. I think maybe zookeeper may be losing its data? | 22:10 |
| @clarkb:matrix.org | I don't see a bind mount for zk's data storage. But it wasn't there under docker either... I'm a bit confused about that | 22:10 |
| @jim:acmegating.com | Clark: just goes into a volume | 22:10 |
| @clarkb:matrix.org | were we relying on docker volumes maybe and podman doesn't have that? | 22:11 |
| @clarkb:matrix.org | ya I wonder if podman isn't persisting volumes for us | 22:11 |
| @clarkb:matrix.org | https://zuul.opendev.org/t/zuul/build/a4802526d4564247a0e3d8dfe98ae086/logs was the most recent build. The patchset I just pushed cleans up the container.conf config in zuul | 22:11 |
| @clarkb:matrix.org | looks like podman does do volume management, but maybe it isn't doing it based on the image data and we need to list them? | 22:12 |
| @jim:acmegating.com | fascinating | 22:13 |
| @clarkb:matrix.org | this is my hunch at least due to the failure and the scheduler generating new keys on what Ithink is the last time it started: https://zuul.opendev.org/t/zuul/build/a4802526d4564247a0e3d8dfe98ae086/log/container_logs/scheduler.log | 22:15 |
| @jim:acmegating.com | Clark: verified locally that the volume doesn't survive a "up/down/up" cycle | 22:16 |
| @jim:acmegating.com | looks like "down" removes the volume | 22:17 |
| @clarkb:matrix.org | oof | 22:17 |
| @clarkb:matrix.org | https://github.com/containers/podman-compose/issues/105 that implies it didn't do this at one time | 22:19 |
| @jim:acmegating.com | volumes are still there, but it makes new ones | 22:20 |
| @clarkb:matrix.org | https://github.com/containers/podman-compose/blob/devel/podman_compose.py#L2322-L2332 oh ok that would explain it I guess | 22:21 |
| @clarkb:matrix.org | I was about to say it doesn't delete them unless you pass the --volumes flag | 22:21 |
| @clarkb:matrix.org | this is fun behavior | 22:21 |
| @jim:acmegating.com | https://github.com/containers/podman-compose/issues/378#issuecomment-1019310463 is interesting info (not directly our problem) | 22:22 |
| @clarkb:matrix.org | If we manually define them I wonder if that makes podman-compose smart and it stops creating new ones | 22:22 |
| @clarkb:matrix.org | but when they are dynamic based on the image it just naively recreates them every time | 22:22 |
| @jim:acmegating.com | yeah this is i think the first troubling behavior... guaranteed to result in data loss | 22:22 |
| @jim:acmegating.com | (unless we find a way to get docker-compose behavior) | 22:23 |
| @jim:acmegating.com | (because if we rely on "manually inspect the container image to see if any anonymous volumes have changed and update the compose file accordingly" we are hosed) | 22:23 |
| @clarkb:matrix.org | agreed this is seems problematic | 22:25 |
| @jim:acmegating.com | Clark: er, are we sure this isn't docker-compose behavior? | 22:27 |
| @clarkb:matrix.org | corvus: well if it were the tests would fail I think? | 22:27 |
| @jim:acmegating.com | i think i just repeated that test with d-c and got the same behavior | 22:27 |
| @clarkb:matrix.org | how is the test passing with docker then? | 22:27 |
| @clarkb:matrix.org | I may be misreading the job but I think it does multiple up and downs and then later validates it can still fetch data that should be present and that build data should be stored in mysql and/or zk and persisted via volumes? | 22:29 |
| @clarkb:matrix.org | both mysql and zookeeper appear to lack explicit volumes for data dirs | 22:29 |
| @jim:acmegating.com | oh we use "stop" not "down" | 22:30 |
| @jim:acmegating.com | the task names are lying | 22:30 |
| @jim:acmegating.com | let me repeat local tests with those commands | 22:30 |
| @jim:acmegating.com | Clark: with up/stop/up they both keep the volume, and i can see data persisting | 22:32 |
| @clarkb:matrix.org | I don't understand the failure then. Maybe we need to hold a newer node and check it directly. Maybe I'm missing something with what the test job is doing | 22:33 |
| @jim:acmegating.com | log says it's "recreating" the containers | 22:33 |
| @jim:acmegating.com | so it must have decided they have a different configuration? | 22:34 |
| @clarkb:matrix.org | oh so we lose the volumes for a different reason maybe? | 22:34 |
| @jim:acmegating.com | looks like that part of the process is changing an env variable, and i guess that doesn't cause docker-compose to recreate containers, but it does podman-compose | 22:35 |
| @jim:acmegating.com | and actually that env var is used for a mount, so what we're really doing is changing a volume mount | 22:36 |
| @jim:acmegating.com | Clark: yes, that appears to be a behavior difference. i can change a volume mount with docker-compose and it will keep the old anonymous volumes, but with podman-compose i get a new set | 22:40 |
| @clarkb:matrix.org | corvus: what bit is changing? | 22:41 |
| @clarkb:matrix.org | is it `ZUUL_TUTORIAL_CONFIG`? | 22:41 |
| @jim:acmegating.com | yeah | 22:41 |
| @jim:acmegating.com | https://opendev.org/zuul/zuul/src/branch/master/doc/source/examples/docker-compose.yaml#L71 | 22:42 |
| @clarkb:matrix.org | so bind mounts and maybe specific volumes would address this | 22:42 |
| @jim:acmegating.com | i don't want to use bind mounts for data, that's an anti-goal for the tutorial, but if specific mounts for container data would solve it, that might be okay, but it's pretty sub-optimal (since again we'd be in a position of needing to track any changes to dependent containers) but at least it's not a data loss scenario | 22:44 |
| @clarkb:matrix.org | or can we accomplish what the test is doing by not changing mounts? | 22:46 |
| @clarkb:matrix.org | perhaps by stopping and the modifying the data under the mount instead? | 22:47 |
| @jim:acmegating.com | Clark: local testing says defined volumes should work | 22:47 |
| @jim:acmegating.com | Clark: not what i want to be doing in the repo | 22:47 |
| @clarkb:matrix.org | and probably worth filing a bug too since it is potentially lossy | 22:47 |
| @jim:acmegating.com | (i really like how easy it is for users and for testing right now -- we can run the same system with any number of configs just by tweaking an env var) | 22:48 |
| @jim:acmegating.com | i wonder what docker-compose does to change the mount without recreating | 22:49 |
| @jim:acmegating.com | apparently it *does* recreate the container, it apparently just also reattaches the existing volumes | 22:53 |
| @jim:acmegating.com | Clark: fyi changing the image name will do this too | 23:00 |
| @clarkb:matrix.org | corvus: that I half expected. I wonder if doing foo/bar -> docker.io/foo/bar does too | 23:02 |
| @clarkb:matrix.org | that would be docker-compose specific since podman doesn't support unqualified names | 23:03 |
| @jim:acmegating.com | Clark: well, it seems like the best thing to do might be to add explicit volumes :/ | 23:03 |
| @jim:acmegating.com | (because, ideally, we want this to work with both things, and explicit volumes aren't bad, per se) | 23:04 |
| @clarkb:matrix.org | ya in theory we'll catch this if it breaks due to a change in implicit volumes not matching the explicit ones | 23:04 |
| @jim:acmegating.com | and we may want to make opendev admins aware of the behavior difference. at least we won't lose data in case of an error -- since the old volumes just get orphaned. they can be recovered. | 23:05 |
| @clarkb:matrix.org | ++ | 23:05 |
| @jim:acmegating.com | i need to task switch away from this for a bit | 23:05 |
| @clarkb:matrix.org | ya I actually need to pop out for dinner momentarily | 23:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!