| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add build target job variable https://review.opendev.org/726266 | 00:22 |
|---|---|---|
| *** y2kenny has joined #zuul | 00:22 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add build target job variable https://review.opendev.org/726267 | 00:22 |
| *** jamesmcarthur has quit IRC | 00:24 | |
| *** jamesmcarthur has joined #zuul | 00:42 | |
| *** jamesmcarthur has quit IRC | 00:49 | |
| *** jamesmcarthur has joined #zuul | 00:50 | |
| *** jamesmcarthur has quit IRC | 00:55 | |
| *** Goneri has quit IRC | 01:05 | |
| *** jamesmcarthur has joined #zuul | 01:16 | |
| *** jamesmcarthur has quit IRC | 01:16 | |
| *** jamesmcarthur has joined #zuul | 01:16 | |
| *** jamesmcarthur has quit IRC | 01:21 | |
| *** jamesmcarthur has joined #zuul | 01:24 | |
| *** jamesmcarthur has quit IRC | 01:36 | |
| *** jamesmcarthur has joined #zuul | 01:36 | |
| *** jamesmcarthur has quit IRC | 01:38 | |
| *** jamesmcarthur has joined #zuul | 01:39 | |
| *** rlandy has quit IRC | 01:45 | |
| *** swest has quit IRC | 01:52 | |
| *** swest has joined #zuul | 02:07 | |
| *** bhavikdbavishi has joined #zuul | 03:04 | |
| *** y2kenny has quit IRC | 03:05 | |
| *** bhavikdbavishi1 has joined #zuul | 03:07 | |
| *** bhavikdbavishi has quit IRC | 03:08 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:08 | |
| *** jamesmcarthur has quit IRC | 03:09 | |
| *** jamesmcarthur has joined #zuul | 03:10 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-registry master: [dnm] buildx https://review.opendev.org/726276 | 03:13 |
| *** jamesmcarthur has quit IRC | 03:15 | |
| *** jamesmcarthur has joined #zuul | 03:20 | |
| *** jamesmcarthur has quit IRC | 03:23 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-registry master: [dnm] buildx https://review.opendev.org/726276 | 03:31 |
| *** bhavikdbavishi has quit IRC | 03:54 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-registry master: [dnm] buildx https://review.opendev.org/726276 | 03:56 |
| *** zxiiro has quit IRC | 04:07 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-registry master: [dnm] buildx https://review.opendev.org/726276 | 04:12 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Scheduler's pause/resume functionality https://review.opendev.org/709735 | 04:32 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Separate connection registries in tests https://review.opendev.org/712958 | 04:32 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Prepare Zookeeper for scale-out scheduler https://review.opendev.org/717269 | 04:32 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Mandatory Zookeeper connection for ZuulWeb in tests https://review.opendev.org/721254 | 04:32 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/717299 | 04:33 |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #zuul | 04:36 | |
| *** mhu has quit IRC | 05:07 | |
| *** sgw has quit IRC | 05:39 | |
| *** dpawlik has joined #zuul | 05:55 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/717299 | 06:16 |
| *** saneax has joined #zuul | 06:17 | |
| openstackgerrit | masterpe proposed zuul/zuul master: Use the --user to install bindep https://review.opendev.org/726288 | 06:42 |
| masterpe | I think I have done something wrong, | 06:43 |
| masterpe | patch 726288 was a, correction of https://review.opendev.org/#/c/726179/ | 06:44 |
| avass | masterpe: I'm going to guess you're use to githubs workflow :) | 06:54 |
| avass | masterpe: In gerrit you usually amend the commit instead of pushing commit after commit on a branch | 06:55 |
| masterpe | I did used the command "git review to push the change. | 06:55 |
| avass | masterpe: you can squash those commits together by using 'git rebase -i HEAD~2' to do an interactive rebase, and choose 'squash' latest commit | 06:56 |
| avass | masterpe: and then when you want to update the change you do whatever changes you want, 'git add <file>' and instead of comitting like usual use 'git commit --amend' and then push | 06:58 |
| masterpe | I have used atom to commit | 06:59 |
| masterpe | but I need to use git commit --amend and then git push, instead of git review? | 07:00 |
| avass | corvus: oh, I wasn't expecting a response that fast :) | 07:00 |
| avass | you can still use git review | 07:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Validate ansible extra packages https://review.opendev.org/724110 | 07:03 |
| avass | corvus: Looks like I missed linting, should be fixed now, want to +2 again? | 07:03 |
| avass | masterpe: I think there's a gerrit workflow guide for opendev, let me check | 07:03 |
| avass | masterpe: I guess this part is relevant: https://docs.opendev.org/opendev/infra-manual/latest/developers.html#submitting-a-change-for-review | 07:04 |
| *** toabctl has joined #zuul | 07:05 | |
| openstackgerrit | masterpe proposed zuul/zuul master: add installation manual for Ubuntu https://review.opendev.org/726288 | 07:08 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Driver event ingestion https://review.opendev.org/717299 | 07:08 |
| *** yolanda has joined #zuul | 07:12 | |
| *** saneax has quit IRC | 07:15 | |
| *** saneax has joined #zuul | 07:16 | |
| *** tosky has joined #zuul | 07:17 | |
| noonedeadpunk | hey there:) have another portion of questions:( so my job is stuck in queued state forever. Like seems that nodepool don't give a node for the check. but nodepool eventually has node http://paste.openstack.org/show/793301/ | 07:21 |
| noonedeadpunk | for project-config I have the following nodesets.yaml http://paste.openstack.org/show/793302/ | 07:22 |
| avass | noonedeadpunk: that looks correct | 07:23 |
| noonedeadpunk | and the following zuul.yaml for the repo itself http://paste.openstack.org/show/793303/ | 07:24 |
| avass | noonedeadpunk: just to be sure, the label shows up in the dashboard right? | 07:24 |
| noonedeadpunk | hm. it's not | 07:25 |
| avass | noonedeadpunk: so either nodepool and the scheduler can't communicate, or you've limited the allowed labels in the tenant config | 07:26 |
| avass | I'm guessing | 07:26 |
| noonedeadpunk | but I kinda didn't configure websocket proxy yet so didn't pay much attention (like builds and console are not supposing to display for sure) | 07:26 |
| noonedeadpunk | and have the following in log | 07:26 |
| avass | noonedeadpunk: But I thought that would result in an error, not in a queued state | 07:26 |
| noonedeadpunk | sec | 07:27 |
| noonedeadpunk | 2020-05-08 07:29:26,059 INFO zuul.nodepool: [e: 370e220200994ff18ed5a3b619c9abd0] Submitted node request <NodeRequest 300-0000000003 <NodeSet ubuntu-bionic [<Node None ('node',):bionic-default>]>> | 07:30 |
| noonedeadpunk | like "Node None" does not look correct.... | 07:30 |
| noonedeadpunk | and see nothing in nodepool log | 07:34 |
| noonedeadpunk | Like feel that maybe zuul/nodepool linking is broken | 07:34 |
| noonedeadpunk | Btw I didn't specify nodepool port in zuul.conf, but it's listening on 127.0.0.1:8005 | 07:35 |
| avass | noonedeadpunk: that log is the scheduler just saying that it has requested a node | 07:37 |
| avass | you should be able to follow the request id, 300-0000000003, in nodepool and see what happens to it | 07:38 |
| noonedeadpunk | nodepool request-list is just empty | 07:40 |
| avass | maybe nodepool isn't connected to zookeeper | 07:40 |
| noonedeadpunk | I think it is? http://paste.openstack.org/show/793304/ | 07:41 |
| noonedeadpunk | maybe zuul isn't..... | 07:42 |
| avass | one of them at least :) | 07:43 |
| noonedeadpunk | so zuul and nodepool are connected not via api but via zookeeper? | 07:48 |
| noonedeadpunk | not sure how can I check zuul - like logs says nothing and don;t really see clli command to verify | 07:49 |
| avass | yeah | 07:50 |
| avass | I think the web grabs the node info from zookeeper as well | 07:50 |
| avass | you'll have to check the logs, are you using docker? | 07:51 |
| avass | then you can do: docker logs <container> or docker-compose logs <container> | 07:51 |
| avass | and it helps if you're running everything in debug mode :) | 07:52 |
| noonedeadpunk | Nope, it;s not in docker | 07:55 |
| avass | I guess you can also check journalctl | 07:56 |
| *** jpena|off is now known as jpena | 07:56 | |
| avass | of you're running it as a service | 07:56 |
| noonedeadpunk | hm, btw, I see no gearman process running | 07:58 |
| avass | noonedeadpunk: I think it's built-in to the scheduler so it should be fine | 07:59 |
| avass | noonedeadpunk: since you're already at the stage of requesting nodes the scheduler has already requested a merge job and gotten the job config | 08:00 |
| noonedeadpunk | Yeah, from architecture https://zuul-ci.org/docs/zuul/discussion/components.html it seems that gearman should be already present | 08:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:02 |
| noonedeadpunk | avass: oh, cant it be because of chroot I provided for nodepool.yaml? http://paste.openstack.org/show/793305/ | 08:03 |
| noonedeadpunk | So they're just using their own prfixes | 08:03 |
| avass | noonedeadpunk: could e | 08:05 |
| avass | be | 08:05 |
| noonedeadpunk | eventually I jsut copied from https://zuul-ci.org/docs/nodepool/configuration.html#attr-zookeeper-servers | 08:05 |
| noonedeadpunk | But see no relevant setting for zuul itself | 08:06 |
| avass | me neither :) | 08:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:08 |
| *** bhagyashris|ruck has joined #zuul | 08:10 | |
| noonedeadpunk | ok, will try that out - images are rebuilding now | 08:10 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:13 |
| noonedeadpunk | avass: thanks! it worked | 08:16 |
| avass | woo! | 08:16 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:18 |
| noonedeadpunk | so now I should build image that zuul can access))) | 08:26 |
| noonedeadpunk | as "Data could not be sent to remote host \"172.16.10.10\". Make sure this host can be reached over ssh: zuul@172.16.10.10: Permission denied (publickey)" | 08:28 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:28 |
| avass | yeah, I guess you're just missing the publickey in authorized_keys | 08:28 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:29 |
| noonedeadpunk | yeah, indeed, just didn't specify keypair while launching node | 08:29 |
| noonedeadpunk | avass: thanks so much for your time and having to answer my stupid questions | 08:31 |
| avass | no problem :) | 08:31 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 08:36 |
| zbr | avass: let me know what you think about https://github.com/ansible/ansible-lint/issues/778 | 08:38 |
| avass | zbr: couldn't that also be configurable in .ansible-lint? | 08:41 |
| zbr | yes it could, any configuration should override the auto mode. | 08:42 |
| avass | I think it sounds reasonable at least | 08:43 |
| zbr | btw, i was looking at your linenumber change, but i fail to identify a change in output | 08:44 |
| avass | you'd need to create a role that uses matchplay and return a linenumber :) | 08:44 |
| avass | since currently if matchplay detects a block it will report the result for the beginning of the block. You can't return the linenumber of the task you return the result of | 08:45 |
| avass | for* | 08:46 |
| avass | which is a bit annoying | 08:46 |
| avass | zbr: as an example, this would report line 1 instead of line 13: http://paste.openstack.org/show/793308/ | 08:49 |
| zbr | something is wrong with the example as it does not fail with a bare ansible-lint on it | 08:56 |
| avass | zbr: ah no, I meant if you write a rule that would fail on line 13 for a matchplay, that matchpal would reportl line 1 | 09:00 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 09:00 |
| noonedeadpunk | avass: is it the way I set username for zuul to connect to the node? https://zuul-ci.org/docs/nodepool/configuration.html#attr-diskimages.username | 09:01 |
| avass | noonedeadpunk: never used the openstack driver, but I would guess so yes | 09:01 |
| avass | zbr: this rule currently reports the line where the block is instead of the task not namespacing it's loop variable: https://opendev.org/zuul/zuul-jobs/src/branch/master/.rules/ZuulJobsNamespaceLoopVar.py | 09:03 |
| avass | zbr: and only matchplay was checking includes and content inside blocks it seems | 09:03 |
| zbr | maybe i was not clear: i want to see one example yml file where running the linter on it does produce a change in line number w/ and w/o your change. | 09:04 |
| zbr | i am still trying to find one... | 09:04 |
| avass | zbr: I'll push a change | 09:05 |
| noonedeadpunk | but it seems to be not provider-specific setting | 09:06 |
| zbr | there are about ~10 calls to matchplay, me just wanted to be sure we fix something and avoid regression | 09:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: return linenumber in matchplay https://review.opendev.org/726312 | 09:09 |
| avass | ^ that changes the role to return the linenumber, test it against test-playbooks/ansible-lint-rules/ZUULJOBS0001/faulty/roles/task-nested-block-missing-loopvar and you'll see a difference | 09:10 |
| avass | rule* | 09:10 |
| avass | without my patch it returns line: 1, with my patch and the updated rule it return line: 3 | 09:11 |
| avass | zbr: but I'm not sure why we return what we do, instead of just returning a Match object from matchplay | 09:15 |
| noonedeadpunk | also wondering why do I have formats: ['raw'] but got qcow image uploaded... | 09:17 |
| avass | noonedeadpunk: I think you'll get more information about the openstack driver later when the rest wakes up :) | 09:18 |
| noonedeadpunk | is there a doc page that describes all available options for zuul.conf? | 09:23 |
| noonedeadpunk | ah, it's in components | 09:23 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add upload-artifactory role https://review.opendev.org/725678 | 09:24 |
| avass | corvus: now that ^ is flipping the delegate_to when testing instead, that way we get to test zuul_return :) | 09:33 |
| *** threestrands has quit IRC | 09:41 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add ansible-lint rule to check owner and group is not preserved https://review.opendev.org/724855 | 10:09 |
| *** sshnaidm|afk is now known as sshnaidm|off | 10:31 | |
| *** tumble has joined #zuul | 10:51 | |
| veecue | hey tumble: Unfortunately, the gitea driver isn't in a working state yet. Mostly missing are webhooks (and of course basic testing). If you want to work on it today, I can hand the code over to you so you can continue, until we have a basic working version and can parallelize | 11:01 |
| tumble | I'd at least like to have a look at it to get a feeling how it's going. :) Where are you hosting the code? | 11:09 |
| tumble | and cool that you started, veecue ! | 11:10 |
| veecue | On /home/veecue/repos/opendev.org/zuul/zuul ;). Where should I upload it? | 11:12 |
| tumble | how about github? as far as I can see opendev doesn't allow forks, does it? | 11:16 |
| avass | tumble: not in an easy way that github does it as far as I know | 11:29 |
| veecue | tumble: https://github.com/veecue/zuul/pull/5 | 11:31 |
| tumble | good stuff, thanks veecue. I will go refill the sugar + caffeine tank now and sit down in the evening to see if I can get a local zuul scheduler environment up and running with this code. :) | 11:34 |
| veecue | You won't, pretty sure ;). But maybe I also have some time to work on it later today (no public holiday in bavaria :( ) | 11:35 |
| tumble | hehe, well once in a lifetime us Berliners need to be lucky too ;P | 11:37 |
| *** jpena is now known as jpena|lunch | 11:37 | |
| *** tumble has quit IRC | 11:46 | |
| tobiash | veecue, tumble: why not using directly zuul's gerrit? | 11:50 |
| *** rfolco|rover|off is now known as rfolco|rover | 11:54 | |
| veecue | tobiash: I would need an account. Also I wanted to wait with an upstream commit until the minimum viable driver is done. | 12:10 |
| *** rlandy has joined #zuul | 12:27 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add build target job variable https://review.opendev.org/726266 | 12:38 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add build target job variable https://review.opendev.org/726267 | 12:38 |
| *** jpena|lunch is now known as jpena | 12:40 | |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Push arm64 specific tags for images https://review.opendev.org/726372 | 12:43 |
| *** yolanda has quit IRC | 12:44 | |
| *** yolanda has joined #zuul | 12:44 | |
| *** bolg has quit IRC | 12:49 | |
| tobiash | corvus, AJaeger: I can confirm now with a test case and production usage that 723800 fixes the gc issue | 12:51 |
| AJaeger | great, tobiash | 12:52 |
| fungi | veecue: it's not really an upstream commit, just a change proposal which you can mark as "not ready for review" by setting a workflow -1 vote once you've pushed the change | 13:28 |
| fungi | i often use gerrit as a place to stash my in-progress work, because it's safer than trusting i won't accidentally prune a local branch with work in it or something | 13:29 |
| fungi | and also makes it available for others to collaborate with me on | 13:29 |
| noonedeadpunk | folks, any idea why do I have 404 for builds and buildsets only? http://paste.openstack.org/show/793317/ | 13:35 |
| tobiash | noonedeadpunk: did you configure an sql connection? | 13:36 |
| tobiash | builds and buildsets are only available when adding a database to zuul | 13:36 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: cabal-test: add build target job variable https://review.opendev.org/726266 | 13:38 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add build target job variable https://review.opendev.org/726267 | 13:38 |
| noonedeadpunk | tobiash: ah, I guess I didn't | 13:38 |
| noonedeadpunk | just error message dissapoints a bit | 13:39 |
| tobiash | noonedeadpunk: the sql connection will become mandatory in one of the next releases anyway. Then zuul will refuse to start without one (with a message). | 13:41 |
| *** guilhermesp has joined #zuul | 13:41 | |
| guilhermesp | hi there! After this commit https://opendev.org/zuul/zuul-jobs/commit/6d78fc4f900415b4e33db1653a8be64ee62a6ee6 we have seen this | 13:42 |
| guilhermesp | https://www.irccloud.com/pastebin/GDDny13Z/ | 13:42 |
| guilhermesp | i guess we'd need another role that ensures-venv? | 13:42 |
| noonedeadpunk | tobiash: so I set https://zuul-ci.org/docs/zuul/reference/drivers/sql.html for zuul.conf and update pipeline to use it, correct? | 13:44 |
| tobiash | noonedeadpunk: correct | 13:44 |
| noonedeadpunk | cool, thanks for the help | 13:45 |
| openstackgerrit | Justus proposed zuul/zuul master: copy gitea driver to pagure driver https://review.opendev.org/726389 | 13:48 |
| openstackgerrit | Justus proposed zuul/zuul master: first work on gitea driver https://review.opendev.org/726390 | 13:48 |
| openstackgerrit | Justus proposed zuul/zuul master: more work towards a working gitea driver https://review.opendev.org/726391 | 13:48 |
| noonedeadpunk | guilhermesp: I think this might be related to the opnestack infra cleanup as they've removed venv pre-installation by dib while following https://review.opendev.org/#/c/709579/ | 13:50 |
| veecue | tobiash: as you can see, you convinced me ;) | 13:50 |
| corvus | veecue: yay! now others can pull down those changes, and push up new changes based on them; that makes collaborating on something like this pretty easy. individual changes (commits) in the stack can also be updated as we go. we'll probably wait to merge the whole stack until it's ready. | 13:56 |
| corvus | veecue: this bit about amending changes may be helpful later: https://docs.opendev.org/opendev/infra-manual/latest/developers.html#submitting-a-change-for-review | 13:58 |
| corvus | clarkb, fungi, ianw: can you see guilhermesp's message at 13:42 -- did we miss sending an announcement about the pip work? | 14:01 |
| * guilhermesp wonders what can we do in our side | 14:02 | |
| guilhermesp | thanks for the attention noonedeadpunk corvus :) | 14:02 |
| veecue | corvus: I guess, I'll just squash the minimum viable thing down to one commit when its ready and add more on top | 14:04 |
| noonedeadpunk | LIke I know this as asked AJaeger this morning :P But I believe I could miss notice | 14:04 |
| corvus | veecue: either way -- we're happy with a patch series if it makes sense; we usually just squash when a later commit undoes something in an earlier commit. whatever is easier for authors and reviewers :) | 14:05 |
| noonedeadpunk | tobiash: should I somehow populate db? | 14:06 |
| corvus | guilhermesp: do you use the ensure-pip role? | 14:06 |
| corvus | guilhermesp: also, are you using one of the 'tox' jobs defined in zuul-jobs? | 14:06 |
| corvus | guilhermesp: i'm guessing you need either ensure-pip or ensure-virtualenv. but it doesn't look like the 'tox' job in zuul-jobs uses either of those.... | 14:10 |
| corvus | guilhermesp: are you just using python3? | 14:10 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add build target job variable https://review.opendev.org/726267 | 14:11 |
| mordred | corvus: hrm. I feel like the tox job should use those if it decides it needs to install tox from pip | 14:11 |
| mordred | or - rather ensure-tox should if it decides it needs to install something | 14:12 |
| *** hashar has joined #zuul | 14:13 | |
| guilhermesp | hey corvus well accordingly to traces, the ensure-pip role is being executed right before ensure-tox role | 14:13 |
| *** hashar is now known as hasharAway | 14:14 | |
| guilhermesp | and the tox.ini we use http://paste.openstack.org/show/793326/ | 14:14 |
| corvus | ah yeah, ensure-tox does call ensure-pip | 14:14 |
| tristanC | zuul-maint : i keep on receiving questions about the status of zuul-runner . Could the spec be reviewed please? it's https://review.opendev.org/681277 | 14:14 |
| guilhermesp | what i feel here is since we started using virtualevn, debian images should have python3-venv package | 14:15 |
| corvus | guilhermesp, mordred: yeah, and it seems like ensure-pip probably should take care of that | 14:16 |
| corvus | guilhermesp: it looks like ensure-pip is supposed to install python3-venv on debian | 14:16 |
| guilhermesp | yeah i do see https://opendev.org/zuul/zuul-jobs/src/branch/master/roles/ensure-pip/tasks/Debian.yaml | 14:17 |
| corvus | guilhermesp: any idea why that isn't running? is it because "Check if pip is installed" (the previous task) did find a "pip" or "pip3" command? | 14:18 |
| guilhermesp | https://www.irccloud.com/pastebin/1WAzwFsH/ | 14:18 |
| mordred | that's only if ensure_pip_from_packages is called | 14:18 |
| corvus | mordred: you mean if ensure_pip_from_packages is true -- and it is by default | 14:19 |
| mordred | oh wow. it is? | 14:19 |
| mordred | that's just ... | 14:19 |
| guilhermesp | https://www.irccloud.com/pastebin/WMbaoZqU/ | 14:19 |
| guilhermesp | and that's just the wholoe trace of the ensure-pip role ^ | 14:19 |
| *** sgw has joined #zuul | 14:20 | |
| guilhermesp | and then, when starts ensure-tox | 14:20 |
| guilhermesp | https://www.irccloud.com/pastebin/2vtqPXfq/ | 14:20 |
| corvus | guilhermesp: okay, so it looks like it found a pre-installed pip3. do you know where it came from? | 14:20 |
| guilhermesp | not sure, let me dig and try to see where I find it | 14:20 |
| corvus | mordred: so it looks like guilhermesp has /usr/local/bin/pip3 -- probably not installed by packages. but ensure-pip figures that should be enough for a working venv module. but the tox module in ansible wants to use the "ensurepip" module which isn't present. where is that supposed to come from? | 14:25 |
| *** Goneri has joined #zuul | 14:28 | |
| mordred | corvus: I do not know. I have completely lost track of what is supposed to be happening here - and I'm having a bit of a grump about some of what is going on but am trying not just just run around ranting about it | 14:28 |
| mordred | corvus: at the mininum I'd say our logic here is flawed in that we're just looking for a pip3 in the from-packages workflow rather than lookign at package manager status | 14:29 |
| mordred | or - maybe we should just install python3-venv regardless of whether we find pip or not - as I think we're making assumptions that that should always exist | 14:29 |
| mordred | also, fwiw: l# python3 -m ensurepip | 14:30 |
| mordred | ensurepip is disabled in Debian/Ubuntu for the system python. | 14:30 |
| mordred | corvus: so - I think that tox in this case is expecting ensurepip to be in the virtualenv that it creates | 14:31 |
| corvus | mordred: hrm, but ensure-pip is designed to install pip from upstream to.... | 14:31 |
| corvus | guilhermesp: is this an emergency for you? do you have a local copy of zuul-jobs where you can revert back to a working state? | 14:32 |
| mordred | corvus: let me try to reproduce real quick | 14:32 |
| guilhermesp | hum actually we have a customer waiting for a fix, currently we cant run any jobs in there | 14:33 |
| corvus | maybe we should just revert that commit then | 14:34 |
| guilhermesp | I'm afraid I dont have much permissions or/and not sure how to proceed on our evn, Maybe mnaser could try out something | 14:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Revert "ensure-tox: use venv to install" https://review.opendev.org/726404 | 14:36 |
| avass | corvus: I fixed the linting on this, wanna +3 it? https://review.opendev.org/#/c/724110/ | 14:37 |
| mordred | corvus: I can't reproduce that error message | 14:38 |
| mnaser | corvus: these are dib built images, i can grab the config.. | 14:38 |
| avass | corvus: thanks! | 14:39 |
| mnaser | corvus: http://paste.openstack.org/show/793333/ is pretty much how those images appear | 14:39 |
| corvus | mnaser: that seems similar, but not quite the same as the '-plain' images that ianw has been using for testing. so maybe we need to figure out the difference there, or maybe we need to add a debian-buster-plain to the set. | 14:44 |
| corvus | mnaser, mordred, guilhermesp: i'm going to spend way too much time trying to catch up on all the particulars here, so i think i'd like to put out the immediate fire by merging https://review.opendev.org/726404 and then seeing if ianw or clarkb who know much more of the nuances of this can suggest a way forward. what do you think? | 14:46 |
| mordred | corvus: ++ | 14:47 |
| guilhermesp | we appreciate that for now corvus and it is going to unblock us | 14:47 |
| mnaser | happy to provide whatever is necessary to replicate this here | 14:47 |
| corvus | mnaser: are you in a place you could third-party ci zuul-jobs? | 14:56 |
| mnaser | corvus: its just a matter of setting up a pipeline and figuring out how i can load the jobs | 14:57 |
| mnaser | as i think i need to get them loaded from zuul-tests.d | 14:57 |
| mnaser | but the opendev-related infra is all setup, we can report | 14:57 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "ensure-tox: use venv to install" https://review.opendev.org/726404 | 14:58 |
| corvus | mnaser: https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1608-L1611 | 14:58 |
| corvus | mnaser: add "include: [job]" so you don't get the pipelines, etc | 14:58 |
| corvus | mnaser: then at least for the moment, maybe just add the related tox jobs | 14:59 |
| mnaser | right yes, i think we'll have to figure out getting matching nodesets but that should be (relatively) trivial | 15:00 |
| corvus | ya -- and of course we want to end up using your debian nodes... | 15:01 |
| corvus | hopefully we can get something working :) | 15:01 |
| mordred | yeah - breaking mnaser makes me sad | 15:03 |
| fungi | mordred: corvus: guilhermesp: sorry, was out on an errand and catching up now, whether or not a pip3 exists is not necessarily an indicator of whether the venv module is available to the default interpreter (and in fact venv doesn't use the installed pip, it relies on a wheel bundle which includes pip and some other bits which are used to seed the environments it creates) | 15:04 |
| clarkb | I think the bug here is everywhere but debian python3 existing implies venv exists | 15:05 |
| clarkb | this is the silly corner case where deboan insists on splitting out python stdlib | 15:05 |
| clarkb | fungi: only because debian is specoal. its part of python stdlib and available everywhere else and we just have to force ourselves to rber debian is different | 15:06 |
| clarkb | *to remember | 15:06 |
| mordred | clarkb: yeah - so it might be as simple as adding an "if platform == debian: unconditinoally apt-get instal python3-venv" | 15:06 |
| fungi | yes, debian slpits python stdlib up into multiple packages (and separate from the package for the interpreter and builtins) | 15:06 |
| mordred | (in the ensure-pip role) | 15:06 |
| guilhermesp | corvus: fungi clarkb mnaser FYI that fixes for us now https://review.opendev.org/#/c/726404/ jobs are happy again | 15:06 |
| mordred | well - I say that | 15:07 |
| mordred | I think the logic might want to be "if platform is debian and python3 -m venv does not work apt-get install python3-venv" - because if we unconditionally install it doesn't let people make base images that have everything that don't need sudo | 15:08 |
| mordred | yeah? | 15:08 |
| clarkb | mordred: ya we should do a separate chevk for python3 -m venv and install if missing | 15:08 |
| fungi | anyway, having a working venv module and having a working pip module are orthogonal. if you're using "pip exists" as a proxy for "python exists" and then assume that the presence of a python interpreter further means all of python's stdlib is installed, then i guess it makes sense | 15:08 |
| clarkb | fungi: its specifically if pip is installed for python3 iirc but ya | 15:09 |
| fungi | but as i said, venv doesn't use the installed copy of pip anyway, it uses a bundled wheel of pip which gets bootstrapped into the new environment | 15:09 |
| fungi | so you can have a working venv module with no pip executable and no importable pip module | 15:09 |
| fungi | (and vice versa) | 15:10 |
| clarkb | yes, but the no pip case doesnt matter as it is ensure-pip | 15:10 |
| clarkb | I think mordred's suggestion is the proper fix | 15:11 |
| clarkb | separately check for venv particularly on debian. Install extra packages if necessary | 15:11 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726413 | 15:12 |
| mordred | clarkb: ^^ like that | 15:12 |
| mordred | I should put in more of a comment | 15:13 |
| *** zxiiro has joined #zuul | 15:13 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726413 | 15:14 |
| mordred | clarkb, corvus, fungi, mnaser: the new code that's not just the revert is in workarounds.yaml | 15:15 |
| mordred | I'd love to see that working on vexxhost though | 15:15 |
| avass | mordred: should that not use the python ansible is using instead of /usr/bin/python3 ? | 15:16 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 15:16 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 15:16 |
| fungi | clarkb: mordred: yep, something simple like `python3 -c 'import venv'` would do it | 15:17 |
| clarkb | mordred: ++ if we can get vexxhost to report back that that fixes them that would be great info to have | 15:17 |
| fungi | or s/python3/whatever the relevant ansible variable is/ | 15:17 |
| avass | ansible_python_interpreter it seems | 15:18 |
| clarkb | no you shuoldn't use ansible_python_interpreter here | 15:18 |
| clarkb | we really need to get away from that | 15:18 |
| mordred | avass: not necessarily, no - the pyton on the node for ansible and the python for running tox are treated differently here | 15:18 |
| clarkb | this is specifically a python3 module and ansible can run under python2 | 15:18 |
| clarkb | mordred: ++ | 15:18 |
| mordred | yeah | 15:19 |
| avass | ah | 15:19 |
| clarkb | ansible_python_interpreter should be used when interacting with the things ansible itself is doing or if you don't particularly care what python is used and so using the one ansible is using is a convenient option | 15:20 |
| fungi | granted, tox also doesn't itself utilize the venv module unless you install the tox-venv plugin | 15:20 |
| *** hasharAway has quit IRC | 15:20 | |
| clarkb | fungi: correct this is to install tox into a virtualenv | 15:21 |
| clarkb | fungi: not to have tox use a particular virtualenv | 15:21 |
| fungi | (or install tox into a venv i guess) | 15:21 |
| *** hasharAway has joined #zuul | 15:21 | |
| clarkb | mordred: I've +1'd that change to idnicate I've reviewed it and it lgtm but vexxhost confirming their side before we land it would be good hence not +2 | 15:22 |
| fungi | and coming back to the question of announcements, i'm guessing this change wasn't announced because it was assumed it would be transparent? but we ended up with an unanticipated regression in behavior | 15:23 |
| tobiash | avass: I commented on the callback chance | 15:23 |
| tobiash | change | 15:23 |
| avass | tobiash: yeah that seems reasonable, I'll update it | 15:25 |
| clarkb | fungi: correct it was intended to be unnoticeable to users | 15:26 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 15:26 |
| clarkb | fungi: basically switch from pip install --user tox to python3 -m venv somepath && somepath/bin/pip install tox | 15:26 |
| clarkb | fungi: the gap there was in assuming a valid python3 install implied a venv module existed | 15:26 |
| clarkb | whcih it does basically everywhere but debian and we have to get into remembering debian is special | 15:26 |
| clarkb | (and we can probably construct testing for that on our side too, but thats part of the remembering. In particular we can uninstall python3-venv before running that in our tests I guess) | 15:27 |
| avass | mordred, clarkb: I guess the problem is if someone for some reason moves their /usr/bin/python3, then python3-venv will always be installed | 15:27 |
| avass | but I'm not sure why someone would do that :) | 15:27 |
| clarkb | avass: ya I think we have to make the distionction between supporting normal valid python installations and people doing stuff on their own outside of the distro and zuul-jobs frameworks | 15:28 |
| avass | yeah | 15:28 |
| clarkb | avass: we should definitely support people working within the frameworks the distro and zuul-jobs give them | 15:28 |
| avass | mordred: why do we no_log looking for venv? | 15:31 |
| *** dpawlik has quit IRC | 15:32 | |
| clarkb | avass: because it spits out a bunch of output probably. There might be a less verbose command we can use to check if it works though | 15:32 |
| *** hasharAway has quit IRC | 15:32 | |
| avass | clarkb: yeah, I'd rather have the output in case it fails when it shouldn't for some reason | 15:33 |
| clarkb | avass: mordred: `python3 -c 'import venv'` was fungi's suggestion | 15:33 |
| fungi | yep, and that's silent unless it doesn't exist. is there more that it needs to do than that? | 15:33 |
| clarkb | that seems to exit 0 if import works and exits 1 with a traceback when it doesn't | 15:33 |
| fungi | if there's no venv module, you get a nonzero exit and a traceback/exceptiom | 15:33 |
| avass | or python3 -m venv > /dev/null, so we get stderr at least | 15:33 |
| clarkb | fungi: no I think that is it. avass fungi would be a good suggestion for the change | 15:33 |
| avass | that seems better | 15:34 |
| fungi | i wouldn't even bother with the redirect. that should never return anything on stdout | 15:34 |
| fungi | it will either print nothing because it worked (and was a no-op) or it'll spew a very brief traceback on stderr | 15:35 |
| avass | python3 -m venv does, python3 -c 'import venv' doesn't :) | 15:35 |
| fungi | but you can just rely on the exit code | 15:35 |
| fungi | right, yes, i was talking about using -c 'import venv' | 15:35 |
| fungi | sorry | 15:35 |
| *** armstrongs has joined #zuul | 15:45 | |
| armstrongs | Is there any documentation anywhere on how to set-up the gitlab driver I couldn't find any docs...? | 15:50 |
| avass | armstrongs: it's not completely ready yet, there's a discussion about it in zuul-discuss, let me link it | 15:50 |
| armstrongs | thanks | 15:51 |
| avass | armstrongs: http://lists.zuul-ci.org/pipermail/zuul-discuss/2020-May/001232.html | 15:51 |
| openstackgerrit | David Moreau Simard proposed zuul/zuul master: Remove dmsimard from zuul-jobs maintainers https://review.opendev.org/726432 | 16:09 |
| fungi | armstrongs: but to reiterate the bits on the ml, any help finishing up that driver for better feature parity with the other code review source connection drivers (at least sufficient to get a project gating workflow supported) would be awesome | 16:16 |
| fungi | in theory what's there can already do advisory testing, fbo|off posted examples months ago of having it report on gitlab.com merge requests for a test project | 16:17 |
| armstrongs | My company is likely switching from Github to Gitlab shortly so I should have access to a Gitlab enterprise instance shortly where I can do some testing with it. | 16:19 |
| *** rlandy is now known as rlandy|biab | 16:19 | |
| fungi | awesome, that's a huge help | 16:20 |
| *** armstrongs has quit IRC | 16:28 | |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #zuul | 16:36 | |
| avass | AJaeger: just went through all the changes for renaming install-* to ensure-* and I don't think there are any projects other than openstacck/magnum that has stable branches, so we should be good | 16:41 |
| avass | unless I missed a project | 16:41 |
| AJaeger | avass: great! should be good then. | 16:42 |
| AJaeger | When did you send out the announcement? | 16:42 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support per branch change queues https://review.opendev.org/718531 | 16:43 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Move queue from pipeline to project https://review.opendev.org/720182 | 16:43 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add optional support for circular dependencies https://review.opendev.org/685354 | 16:43 |
| avass | AJaeger: April 28, but that got a bit delayed and following the dates in the announcement we should start erroring with a message in four days | 16:44 |
| avass | AJaeger: I'll prepare a change for that | 16:44 |
| AJaeger | avass: best write the date in the commit message ;) Thanks | 16:45 |
| avass | AJaeger: sure :) | 16:45 |
| avass | AJaeger: it's just x/pbrx and airship/airshipctl that hasn't merged their changes yet | 16:46 |
| clarkb | roman_g is probably the person to bug re airshipctl | 16:46 |
| avass | already have a +2 from him :) | 16:47 |
| AJaeger | clarkb, mordred, want to retire x/pbrx - or merge that change? | 16:48 |
| AJaeger | pbrx change: https://review.opendev.org/#/c/719402/ | 16:49 |
| avass | AJaeger: airshipctl might be accidentally using the role from zuul-jobs since they have their own docker-install | 16:57 |
| avass | but is using install-docker | 16:57 |
| *** jpena is now known as jpena|off | 17:00 | |
| AJaeger | want to hop over into #airshipit and ask? | 17:00 |
| AJaeger | Just point it out and let them fix it either way | 17:00 |
| avass | AJaeger: already did :) | 17:00 |
| avass | and I pushed a change to fix it for them | 17:00 |
| AJaeger | great | 17:03 |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726413 | 17:04 |
| *** tumble has joined #zuul | 17:05 | |
| openstackgerrit | Merged x/pbrx master: Use ensure-* roles https://review.opendev.org/719402 | 17:09 |
| mordred | AJaeger: ^^ but let's also retire that | 17:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove install-* roles https://review.opendev.org/719322 | 17:29 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fail and direct user to use ensure-* version of roles https://review.opendev.org/726448 | 17:29 |
| avass | AJaeger: I think that should be correct | 17:30 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Revert "Revert "ensure-tox: use venv to install"" https://review.opendev.org/726413 | 17:31 |
| openstackgerrit | Merged zuul/zuul master: Validate ansible extra packages https://review.opendev.org/724110 | 17:32 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Remove install-* roles https://review.opendev.org/719322 | 17:33 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: return linenumber in matchplay https://review.opendev.org/726312 | 17:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Fix bad path in ansible-lint test job files https://review.opendev.org/726449 | 17:35 |
| avass | AJaeger: ^ :) | 17:36 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: DNM: return linenumber in matchplay https://review.opendev.org/726312 | 17:37 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ansible-lint-rules: Fix bad path and filename https://review.opendev.org/726449 | 17:39 |
| *** rlandy|biab is now known as rlandy | 18:17 | |
| tumble | I might have missed it in the usage examples, but… is it possible to run a subset of tests with tox? | 18:20 |
| avass | tumble: yeah, tox passes any arguments give to stestr | 18:23 |
| avass | tumble: so the same way you would with stestr | 18:23 |
| tumble | ok ty, will look at that | 18:23 |
| clarkb | which is basically tox -epy38 -- 'my.test.regex.here' | 18:23 |
| tumble | excellent, thanks for the shortcut ;D | 18:24 |
| clarkb | usually I get away with just the last bit of the testname since our tests tend to be unique enough | 18:24 |
| tumble | damn the entire test suite runs for long on my old machine :P | 18:24 |
| avass | it takes over an hour on my laptop :) | 18:24 |
| tumble | haha so I better quit it xD | 18:25 |
| veecue | Isn't that why people invented CIs? | 18:25 |
| tumble | yeah but the feedback cycle is a little slow while actually writing the tests | 18:26 |
| avass | technically you don't need tests for CI ;) | 18:26 |
| *** yolanda has quit IRC | 18:35 | |
| *** avass has quit IRC | 18:37 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Fail and direct user to use ensure-* version of roles https://review.opendev.org/726448 | 19:03 |
| *** dpawlik has joined #zuul | 19:03 | |
| AJaeger | mordred: regarding pbrx, where should we announce retirement? | 19:05 |
| AJaeger | zuul-discuss? | 19:06 |
| mordred | AJaeger: sure? it's a good question | 19:07 |
| AJaeger | gerritbot announces on #zuul. But we could also use service-discuss or openstack-discuss. I consider it a formality... | 19:08 |
| mordred | AJaeger: I think zuul is the best bet | 19:09 |
| AJaeger | mail sent. | 19:14 |
| openstackgerrit | Andreas Jaeger proposed x/pbrx master: Retire repository https://review.opendev.org/726462 | 19:18 |
| openstackgerrit | Andreas Jaeger proposed x/pbrx master: Retire repository https://review.opendev.org/726462 | 19:21 |
| AJaeger | mordred: everything pushed | 19:32 |
| *** bstinson has quit IRC | 19:56 | |
| mordred | AJaeger: \o/ thank you! | 19:58 |
| *** avass has joined #zuul | 19:59 | |
| avass | AJaeger: all ensure-* role updates has been merged and we're ready to retire install-* on tuesday :) | 20:01 |
| *** avass has quit IRC | 20:07 | |
| *** bstinson has joined #zuul | 20:07 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Increase cherrypy thread pool https://review.opendev.org/708377 | 20:08 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw https://review.opendev.org/664950 | 20:10 |
| *** avass has joined #zuul | 20:10 | |
| *** dpawlik has quit IRC | 20:12 | |
| *** hashar has joined #zuul | 20:13 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add --all to skopeo copy from insecure registry https://review.opendev.org/726469 | 20:23 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Add --all to skopeo copy from insecure registry https://review.opendev.org/726469 | 20:25 |
| AJaeger | avass: great, thanks! | 20:28 |
| *** avass has quit IRC | 20:34 | |
| *** bstinson has quit IRC | 20:44 | |
| *** bstinson has joined #zuul | 21:04 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Add --all to skopeo copy from insecure registry https://review.opendev.org/726469 | 21:23 |
| *** avass has joined #zuul | 21:23 | |
| *** hashar has quit IRC | 21:24 | |
| *** dustinc has joined #zuul | 21:34 | |
| *** hashar has joined #zuul | 21:46 | |
| *** avass has quit IRC | 21:47 | |
| *** hashar has quit IRC | 21:48 | |
| openstackgerrit | Monty Taylor proposed zuul/nodepool master: Add podman and config to the nodepool-builder image https://review.opendev.org/726477 | 21:52 |
| mordred | corvus: ^^ that should capture enough of my exploration from today to make it repeatable | 21:52 |
| mordred | clarkb: ^^ | 21:54 |
| mordred | clarkb: idea being we can actually just put that command in the image defiitions for the arm64 images in nodepool.yaml - and we should be able to build them on any of our builders instead of needing a special host os | 21:55 |
| mordred | clarkb: I haven't written a lot of prose yet, becuase it's late in the day - but I think hopefully that should be enough to get the general picture | 21:55 |
| clarkb | mordred: but will thta actually run everything else as if it were arm64? | 21:56 |
| clarkb | mordred: I think if binfmt did a "proper" vm it would but worry the emulated process alone won't | 21:56 |
| clarkb | (I mean we can test all of that I think so we'll find out) | 21:56 |
| mordred | clarkb: yeah - it'll run the container containing dib as if it were | 21:56 |
| clarkb | oh I see it will be the whole container not just the process | 21:57 |
| mordred | clarkb: my initial tests show it working pretty well | 21:57 |
| mordred | yeah | 21:57 |
| mordred | that's why the nested podman invocatino | 21:57 |
| mnaser | opendev is running zuul-scheduler in containers, right? | 22:51 |
| mnaser | https://www.irccloud.com/pastebin/lNIZsM4G/ | 22:52 |
| mordred | mnaser: yes | 22:53 |
| mordred | mnaser: uhm | 22:53 |
| mordred | corvus: ^^ | 22:53 |
| clarkb | that looks like a type mismatch | 22:54 |
| clarkb | I think it wants the dict lookup there | 22:54 |
| clarkb | not a string index | 22:54 |
| mnaser | i mean, this happened after we made a tenant config change (but that also coincided with a rollout) | 22:54 |
| mordred | mnaser: we _just_ restarted our scheduler | 22:54 |
| mnaser | so it could also be our fault though it shouldn't be fatal | 22:54 |
| mnaser | ok, so that's good | 22:54 |
| mnaser | means it's likely here | 22:54 |
| mordred | yeah - but I agree, that traceback is ugly | 22:55 |
| mordred | mnaser: I'm guessing you saw we updated the images to remove jemalloc and it seems to be helping with memory | 22:55 |
| clarkb | though we've been running on 3.8 for a while now | 22:55 |
| mordred | yeah | 22:55 |
| mnaser | this is exactly what was added (work to get 3pci stuff) | 22:55 |
| mnaser | https://www.irccloud.com/pastebin/JeCyZI5J/ | 22:55 |
| clarkb | we just removed jemalloc this time around I think | 22:55 |
| clarkb | though maybe only web was on 3.8? | 22:55 |
| mnaser | and that was inspired by https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L1608-L1611 | 22:56 |
| openstackgerrit | Justus proposed zuul/zuul master: more work towards a working gitea driver https://review.opendev.org/726391 | 22:56 |
| mordred | mnaser: I don't see an issue with that | 22:57 |
| mnaser | and vexxhost is also a valid connection too | 22:57 |
| mnaser | so is opendev | 22:57 |
| mordred | mnaser: maybe you had a latent error somewhere else and a config error before this? | 22:57 |
| mordred | but didn;t notice cause you weren't watching during a restart? | 22:57 |
| mnaser | i can try and look but unlikely because the config is managed via git+argocd so we'd have been alerted if there was drift | 22:57 |
| mordred | nod | 22:57 |
| *** tumble has quit IRC | 22:59 | |
| mnaser | https://opendev.org/zuul/zuul/src/branch/master/zuul/model.py#L3629-L3631 | 23:00 |
| mnaser | ok so old_obj must be a string here for some reason. | 23:00 |
| corvus | yeah, this is a pretty nonsensical error | 23:00 |
| mnaser | https://opendev.org/zuul/zuul/commit/0971847a50c317233733ab6dd921ae3b36f7793e | 23:00 |
| mnaser | "Ignore source_context and description in job changes" | 23:00 |
| mnaser | i mean, i dont understand the change yet, but.. it's.. close? | 23:01 |
| ianw | mordred: so the venv thing ... that is annoying. the reason we have this "check if pip is installed and do nothing" is because of the requirement to not have become: yes -- what i really want is to just use package: and see if pip is installed as a package and move on | 23:01 |
| mordred | ianw: yah. srrsly | 23:02 |
| mordred | ianw: well - I put up a revert revert with a stab at adding in a new piece of logic for it - it's TOTALLY failing all of the tests right now | 23:02 |
| corvus | mnaser: that shouldn't be related | 23:02 |
| mordred | ianw: https://review.opendev.org/#/c/726413/ | 23:02 |
| corvus | mnaser: what's your status? have you restarted, or is it still in this state? and is it completely broken, or running okay on the prior config? | 23:03 |
| mnaser | corvus: crash looping right now, i have not tried reverting config | 23:03 |
| mnaser | but before i do that, i think i might want to print(old_obj) | 23:03 |
| mnaser | just to see what it throws out | 23:04 |
| mordred | corvus, mnaser the other thing that changed recently is we installed libyaml in the image | 23:04 |
| mnaser | could it be the k8s-y yaml format of the config | 23:04 |
| corvus | mnaser: oh, it's refusing to start due to this error? | 23:04 |
| mnaser | corvus: yep | 23:04 |
| mordred | mnaser: the yaml format shouldn't matter - that's valid yaml | 23:04 |
| mnaser | see top of traceback with the "2020-05-08 23:02:41,397 ERROR zuul.Scheduler: Error starting Zuul:" | 23:04 |
| mnaser | (other services went up fine) | 23:05 |
| corvus | mnaser: if you print(old_obj) also print(attr) | 23:07 |
| *** tosky has quit IRC | 23:11 | |
| *** rfolco|rover has quit IRC | 23:12 | |
| mnaser | proving to be less than trivial because k8s wipes the whole thing clean | 23:13 |
| mnaser | oh i can try and use an initcontainer | 23:13 |
| clarkb | I've checked opendev's zuul for that error and ew don't get it | 23:15 |
| clarkb | (just as a sanity check) | 23:15 |
| mordred | mnaser: jeez | 23:16 |
| corvus | mnaser: is the ci/config/opendev repo publicly accessible by any chance? | 23:17 |
| mnaser | corvus: there's no secret sauce in it, it's 2 or 3 files i think. it's just behind an oauth2 proxy, noonedeadpunk can probably paste the contents | 23:18 |
| noonedeadpunk | corvus: it's eventually pipelines.yaml http://paste.openstack.org/show/793346/ | 23:19 |
| mnaser | oh | 23:20 |
| noonedeadpunk | and projects.yaml http://paste.openstack.org/show/793347/ | 23:20 |
| mnaser | i just noticed it | 23:20 |
| corvus | line 2 | 23:20 |
| mnaser | yeah | 23:20 |
| noonedeadpunk | ? | 23:20 |
| mnaser | noonedeadpunk: pipeline is not a list, | 23:20 |
| mnaser | its a dict | 23:20 |
| noonedeadpunk | oh | 23:20 |
| mordred | bad dash evil evil bad | 23:21 |
| corvus | we have a lot of code to catch syntax errors like that; that's a really creative new one | 23:21 |
| mordred | yeah - I'm impressed with that one | 23:21 |
| noonedeadpunk | pushed patch | 23:22 |
| corvus | we should be able to catch that earlier | 23:22 |
| mnaser | ok lets see if that fixed it | 23:22 |
| mnaser | maybe if noonedeadpunk is feeling adventerous he can help out with a patch to catch that :P | 23:23 |
| noonedeadpunk | I feel but not now :p | 23:23 |
| mnaser | haha fair enough | 23:23 |
| mordred | noonedeadpunk: :) | 23:24 |
| mnaser | yeah tha was it | 23:24 |
| mordred | \o/ | 23:24 |
| mnaser | 2020-05-08 23:24:00,971 INFO zuul.Scheduler: Full reconfiguration complete (duration: 50.321 seconds) | 23:24 |
| mnaser | https://zuul.vexxhost.dev/t/opendev/status | 23:24 |
| mnaser | we've got some cleanup to do, but yeah. | 23:24 |
| mordred | corvus: oh bother - I think I just realized _another_ issue with the multi-arch | 23:25 |
| mordred | corvus: bear with me here | 23:25 |
| mordred | corvus: when we push to intermediate registry, we do so from the buildset registry ... oh, nevermind, that should be fine | 23:27 |
| mordred | corvus: I was thinking we were doing it from the local docker cache which would be bad, but from the buildset regsistry should be fine | 23:27 |
| mordred | corvus: but still not working right | 23:28 |
| corvus | mordred: is it like everything broken for all image builds and we need to revert? or just doesn't work for the multi-arch thing yet? | 23:29 |
| mordred | corvus: no - just multi-arch | 23:29 |
| mordred | corvus: there is luckily a really clear chunk of code showing it's clearly wrong in the logs of this last run though | 23:30 |
| mordred | so that's nice | 23:30 |
| mordred | corvus: if you look at https://f27413429b5f9d129ea6-391ca9f25982feff002f89e6b05bcdaa.ssl.cf1.rackcdn.com/726458/1/check/system-config-build-image-uwsgi-base-3.8/7bcc7b9/job-output.txt and scroll to 2020-05-08 23:15:03.027081 | 23:30 |
| mordred | corvus: you'll see both arm64 and amd64 are pulling the same sha of python-base and python-builder | 23:31 |
| mordred | corvus: I think I'm at the end of todays' debugging on that | 23:32 |
| corvus | mordred: but theoretically the previous build should have pushed both; so at this point we can inspect the intermediate registry and see if things made it that far, yeah? | 23:32 |
| corvus | mordred: yeah me too. see you later :) | 23:33 |
| mordred | corvus: hrm. | 23:34 |
| mordred | corvus: so - (sorry, I lied, still looking | 23:34 |
| mordred | corvus: if you look at 2020-05-08 23:13:20.942417 | 23:34 |
| mordred | we sure do seem to be pulling 4 sets of shas | 23:34 |
| mordred | which would be consistent with {base,builder}:{amd64,arm64} | 23:34 |
| mordred | but there's only actually two sets of distinct shas | 23:35 |
| mordred | so -- yeah | 23:35 |
| mordred | same with push: https://zuul.opendev.org/t/openstack/build/aebd30b779ca498a966184fbf3211815/log/job-output.txt#1145 | 23:36 |
| mordred | ok. for real -that's it for me | 23:36 |
| *** rlandy has quit IRC | 23:45 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!