Thursday, 2021-11-18

@iwienand:matrix.orghttps://graphite.dev/blog/post/DThX8ffP1gmxWJChEv0y01:03
@iwienand:matrix.orgwhen stuff like that makes the front page of HN it does make me realise that I live in a bit of a start-of-the-art development bubble :)01:04
@jim:acmegating.commaybe i ought to post a comment about zuul...01:09
@jim:acmegating.comianw: https://news.ycombinator.com/item?id=29260997 if you want to upvote and/or reply :)01:13
@iwienand:matrix.orgi do try as well sometimes - https://news.ycombinator.com/item?id=26920815 :)01:16
@jim:acmegating.comi can't believe they called it graphite.  like.  um.  i get names are hard and they're all taken, but that one is really taken.01:27
@jim:acmegating.comi think i'll call my next project "kubernetes"01:28
@gtema:matrix.org> <@jim:acmegating.com> gtema: i don't think your squash change is complete: https://opendev.org/zuul/zuul/src/branch/master/zuul/driver/gitlab/gitlabconnection.py#L433-L434  params appears unused.  if you fix that, it would be good to add a test.07:24
yes corvus: https://review.opendev.org/c/zuul/zuul/+/817393 is already open for that. Adding tests is clearly good but that might be not that trivial due to how tests look like now. Will try to add something
-@gerrit:opendev.org- Zuul merged on behalf of Simon Westphahl: [zuul/zuul] 818206: Log and ignore exceptions in pipeline processing https://review.opendev.org/c/zuul/zuul/+/81820607:56
-@gerrit:opendev.org- Artem Goncharov proposed: [zuul/zuul] 817393: Fix gitlab squash merge https://review.opendev.org/c/zuul/zuul/+/81739308:23
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 818360: Load job from pipeline state on executors https://review.opendev.org/c/zuul/zuul/+/81836010:13
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 818295: web UI: Add a credentials renew modal https://review.opendev.org/c/zuul/zuul/+/81829510:15
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 818360: Load job from pipeline state on executors https://review.opendev.org/c/zuul/zuul/+/81836010:25
-@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 818360: Load job from pipeline state on executors https://review.opendev.org/c/zuul/zuul/+/81836010:29
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 810699: Web UI: Show pipeline types as icons https://review.opendev.org/c/zuul/zuul/+/81069910:56
-@gerrit:opendev.org- Simon Westphahl proposed:13:48
- [zuul/zuul] 818360: Load job from pipeline state on executors https://review.opendev.org/c/zuul/zuul/+/818360
- [zuul/zuul] 818385: Fix job data attribute serialization https://review.opendev.org/c/zuul/zuul/+/818385
@spamaps:spamaps.ems.host> <@tristanc_:matrix.org> spamaps: that's good to hear, thanks for the feedback. I guess we could replicate this setting https://zuul-ci.org/docs/nodepool/aws.html#attr-providers.[aws].pools.labels.tags to set custom metadata14:49
Yeah I think that may be how I do it. For now I've got pods to spin up, they just come along with a big java sidecar that everything at Spotify gets unless explicitly disabled. :P
@spamaps:spamaps.ems.hostNow, a new and possibly more controvertial idea that I will probably take to the mailing list when it is better informed by the struggle I've just set out on. I may ask for dynamic label configuration. What i mean is.. right now, labels are static links to resources+base images. But, what I think I want, is to be able to define jobs that say "I want a pod with 2cpu and 4GB RAM, and image: xxxxxx" .. and then nodepool starts spinning those up. The reason I want this is that we currently have 8 action containers in our CI/CD flow for the app I'm PoC'ing with Zuul.. and those action containers are maintained by the devs in many cases. They cache things in them for speed and stability.. and I'd really like to enable them without having to require PR's to nodepool.yaml.14:52
@avass:vassast.orgspamaps: ++14:55
@clarkb:matrix.orgNote that is roughly how Jenkins resource request system works and I'm trying to remember why we rejected it. Its been so long now14:58
@spamaps:spamaps.ems.hostAnother use-case for this, btw, is that it'd be great if nodepool-builder was building images that have the git repos cached. :)14:58
@clarkb:matrix.orgthough jenkins doesn't quite do it that way I guess. It does many arbitrary labels on a single system then you can match any number of them14:58
@clarkb:matrix.orgmaybe that was the issue, it was impreceise14:59
@spamaps:spamaps.ems.host> <@clarkb:matrix.org> Note that is roughly how Jenkins resource request system works and I'm trying to remember why we rejected it. Its been so long now14:59
I'd suspect that the constraints put on OpenDev's zuul are far different than those we'll encouter behind Spotify's firewall.
@spamaps:spamaps.ems.hostI'll be quite happy to write some code to implement such a system. My thinking is that we just need some plugin points for Nodepool.14:59
@spamaps:spamaps.ems.hostRight now the only source for nodepool config is a file on disk. We can add a REST API endpoint as an option, for instance, and that would allow us to write a service in Zuul which gathers dynamic nodes and applies any centralized control before exposing it to Nodepool.15:00
@jim:acmegating.comgtema: /buffer 5115:01
@jim:acmegating.comgerr15:01
@spamaps:spamaps.ems.hostAnyway, my need for this is speculative at the moment. I'm just raising it now to see if anyone else might be interested. :)15:01
@spamaps:spamaps.ems.hostWe may be fine with PRs to the nodepool.yaml. :)15:01
@clarkb:matrix.org> <@spamaps:spamaps.ems.host> Another use-case for this, btw, is that it'd be great if nodepool-builder was building images that have the git repos cached. :)15:02
The source-repositories element makes this straightforward, though our use of it includes a script to filter out repos we don't want as well as find the current set dynamically so that we don't need to update a file each time a new repo is created
@spamaps:spamaps.ems.host> <@clarkb:matrix.org> The source-repositories element makes this straightforward, though our use of it includes a script to filter out repos we don't want as well as find the current set dynamically so that we don't need to update a file each time a new repo is created15:04
diskimage-builder is .. not a popular image builder. :-/
@clarkb:matrix.orgspamaps: sure, but that is what nodepool-builder supports out of the box? I guess I don't undersatnd the suggestion then if yuo want nodepool-builder to do this15:04
@clarkb:matrix.orgit also has the benefit of being able to run without virtualization. Which maybe isn't super important on google cloud because they support nested virt15:06
@gtema:matrix.org> <@jim:acmegating.com> gtema: /buffer 5115:06
corvus: you wanted to say something particular?
@jim:acmegating.comgtema: no sorry15:06
@jim:acmegating.comtypo15:07
@gtema:matrix.orgno problem15:07
@jim:acmegating.comspamaps: https://review.opendev.org/775042 is some light reading you may find interesting15:07
@spamaps:spamaps.ems.host> <@clarkb:matrix.org> it also has the benefit of being able to run without virtualization. Which maybe isn't super important on google cloud because they support nested virt15:15
I have very little interest in VMs at all.
@clarkb:matrix.orgAh, Zuul alreasy has a fair bit of tooling to build container images15:16
@clarkb:matrix.orgI guess the question is whether we can improve that for use in direct consumption by zuul for jobs15:16
@spamaps:spamaps.ems.hostIt's possible that the whole use case can be handled through careful management of container image tags.15:19
@clarkb:matrix.orgspamaps: just thinking out loud you could have periodic jobs that build and publish your job container images. Then the label would refer to that image name? It is possible that tighter integration between the zuul system and the job based building could be useful 775042 covers some of that iirc (but it has been a while since I read that spec)15:23
@spamaps:spamaps.ems.host> <@clarkb:matrix.org> spamaps: just thinking out loud you could have periodic jobs that build and publish your job container images. Then the label would refer to that image name? It is possible that tighter integration between the zuul system and the job based building could be useful 775042 covers some of that iirc (but it has been a while since I read that spec)15:35
Yeah, I'm just thinking out loud too. As long as it's clear to people what the nodeset is doing and where it's getting the contents, I think that should work.
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed:15:37
- [zuul/zuul] 781858: web UI: allow a privileged user to promote a change https://review.opendev.org/c/zuul/zuul/+/781858
- [zuul/zuul] 802559: Web UI: Add "Create Autohold Request" form, improve API error messages https://review.opendev.org/c/zuul/zuul/+/802559
- [zuul/zuul] 818295: web UI: Add a credentials renew modal https://review.opendev.org/c/zuul/zuul/+/818295
- [zuul/zuul] 769943: Example Docker compose: keycloak integration https://review.opendev.org/c/zuul/zuul/+/769943
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com:15:39
- [zuul/zuul] 816918: Upgrade react-router-dom https://review.opendev.org/c/zuul/zuul/+/816918
- [zuul/zuul] 817861: Support auth in multiple tabs https://review.opendev.org/c/zuul/zuul/+/817861
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed:15:39
- [zuul/zuul] 734082: web UI: user login with OpenID Connect https://review.opendev.org/c/zuul/zuul/+/734082
- [zuul/zuul] 734850: web UI: allow a privileged user to dequeue a change https://review.opendev.org/c/zuul/zuul/+/734850
- [zuul/zuul] 736772: web UI: allow a privileged user to re-enqueue a change https://review.opendev.org/c/zuul/zuul/+/736772
- [zuul/zuul] 768115: Web UI: allow a privileged user to request autohold https://review.opendev.org/c/zuul/zuul/+/768115
- [zuul/zuul] 768199: Web UI: add Autoholds, Autohold page https://review.opendev.org/c/zuul/zuul/+/768199
- [zuul/zuul] 810699: Web UI: Show pipeline types as icons https://review.opendev.org/c/zuul/zuul/+/810699
- [zuul/zuul] 781858: web UI: allow a privileged user to promote a change https://review.opendev.org/c/zuul/zuul/+/781858
- [zuul/zuul] 802559: Web UI: Add "Create Autohold Request" form, improve API error messages https://review.opendev.org/c/zuul/zuul/+/802559
- [zuul/zuul] 818295: web UI: Add a credentials renew modal https://review.opendev.org/c/zuul/zuul/+/818295
- [zuul/zuul] 769943: Example Docker compose: keycloak integration https://review.opendev.org/c/zuul/zuul/+/769943
@jim:acmegating.comi'm going to be afk until monday; see ya'll next week :)16:14
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 817861: Support auth in multiple tabs https://review.opendev.org/c/zuul/zuul/+/81786116:40
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 734850: web UI: allow a privileged user to dequeue a change https://review.opendev.org/c/zuul/zuul/+/73485016:40
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 736772: web UI: allow a privileged user to re-enqueue a change https://review.opendev.org/c/zuul/zuul/+/73677216:40
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 768115: Web UI: allow a privileged user to request autohold https://review.opendev.org/c/zuul/zuul/+/76811516:41
-@gerrit:opendev.org- Matthieu Huin https://matrix.to/#/@mhuin:matrix.org proposed: [zuul/zuul] 768199: Web UI: add Autoholds, Autohold page https://review.opendev.org/c/zuul/zuul/+/76819916:41
@spamaps:spamaps.ems.hostTricky thing I've never liked: getting all the right things into zuul executor's ansible venvs.17:23
@clarkb:matrix.orgya, there is an option somewhere to give those hints to the naisble install process, but you still need to sort them all out based on what ansible modules you'll be using. I think by default we've added a few cloud related thigns we think are useful, but certainly not a complete list depending on what your jobs do17:26
@spamaps:spamaps.ems.host> <@clarkb:matrix.org> ya, there is an option somewhere to give those hints to the naisble install process, but you still need to sort them all out based on what ansible modules you'll be using. I think by default we've added a few cloud related thigns we think are useful, but certainly not a complete list depending on what your jobs do17:31
Yeah there are too many things to hit them all. But it's just a base job problem so hopefully it's a rare thing.
@spamaps:spamaps.ems.hostStill it's one of those insider things that I am not sure how to genericize.17:31
@clarkb:matrix.orgThe way ansible users typically boot strap this is to have raw module installation steps which get what they need onto the system. But that doesn't work for zuul due to security concerns17:32
@spamaps:spamaps.ems.hostexactly17:33
@clarkb:matrix.orgor at least I think that is our assumption, there might be a way to get it to work with trusted jobs somehow?17:33
@clarkb:matrix.orgI think we RO mount the ansible install in bwrap though17:33
@spamaps:spamaps.ems.hostWeird.. I monkeyed with my zuul-executor image and now my scheduler is refusing to process GHE hooks17:34
@spamaps:spamaps.ems.host```scheduler_1  | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: [e: a690d7c0-4895-11ec-9d9b-25ad476b54d3] Exception when processing event:17:34
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: Traceback (most recent call last):
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: File "/usr/local/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 339, in run
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: self._process_event()
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: File "/usr/local/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 397, in _process_event
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: change = self.connection._getChange(
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: File "/usr/local/lib/python3.8/site-packages/zuul/driver/github/githubconnection.py", line 1319, in _getChange
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: pull = self.getPull(change.project.name, change.number,
scheduler_1 | 2021-11-18 17:33:32,823 ERROR zuul.GithubEventProcessor: AttributeError: 'NoneType' object has no attribute 'name'
scheduler_1 | 2021-11-18 17:33:32,824 DEBUG zuul.GithubEventProcessor: [e: a690d7c0-4895-11ec-9d9b-25ad476b54d3] Finished event processing```
@spamaps:spamaps.ems.host(I know it can talk to GHE because it's showing the proper configs from GHE.. just.. hooks are bouncing off with this error.)17:35
@clarkb:matrix.org`change.project` is `None` maybe check the event that was emitted?17:36
@clarkb:matrix.orggrepping by the event id there `a690d7c0-4895-11ec-9d9b-25ad476b54d3` might also give clues17:37
@spamaps:spamaps.ems.hostOk so this is a weird thing.. I noticed it before. The close/reopen PR flow seems to break if zuul scheduler is restarted.17:40
@spamaps:spamaps.ems.hostI had to open a new PR17:40
@spamaps:spamaps.ems.hostI think it may be a bug in the GH driver.17:42
@spamaps:spamaps.ems.hostOk, on to new problem...17:42
@spamaps:spamaps.ems.host```2021-11-18 17:41:09.637411 | TASK [Gathering Facts]17:42
2021-11-18 17:41:18.230033 | ubuntu-pod | ERROR
2021-11-18 17:41:18.230474 | ubuntu-pod | {
2021-11-18 17:41:18.230530 | ubuntu-pod | "msg": "Data could not be sent to remote host \"ubuntu-pod\". Make sure this host can be reached over ssh: ssh: Could not resolve hostname ubuntu-pod: Name or service not known\r\n",
2021-11-18 17:41:18.230565 | ubuntu-pod | "unreachable": true
2021-11-18 17:41:18.230596 | ubuntu-pod | }```
@spamaps:spamaps.ems.hostARGH17:42
@spamaps:spamaps.ems.hostWhy is ansible trying to use SSH on a kubectl node? :-P17:42
@clarkb:matrix.orgThe connection method should be an attribute of the node set by nodepool17:55
@clarkb:matrix.orgThen zuul+Ansible run with what nodepool said17:55
@spamaps:spamaps.ems.hostYeah I don't know how to inspect the nodes for connection type.. --detail doesnt' show it19:15
@spamaps:spamaps.ems.host```executor_1   | 2021-11-18 17:42:21,763 ERROR zuul.ExecutorServer: [e: a93882b0-4896-11ec-80b4-40764ccd821b] [build: c246d676787048cfa06f84dbc2d388e0] Could not find the forwarded port: error: the server doesn't have a resource type "pod"```19:19
@spamaps:spamaps.ems.hostOh I think I see.. kubeconfig was wonk19:20
@spamaps:spamaps.ems.hostNow I'm stuck on `No Auth Provider found for name "gcp"`19:38
@fungicide:matrix.orglooks like that's coming from the kubernetes cli not having the gcp plugin?19:54
@spamaps:spamaps.ems.hostYeah, it's a little confusing since the config was written by the gcloud CLI installed from the same google cloud APT repo as this kubectl20:17
@spamaps:spamaps.ems.hostI think there's something missing that's preventing the gcp plugin from initializing.20:17
@spamaps:spamaps.ems.hostIt works fine in the nodepool image20:17
@spamaps:spamaps.ems.hostbut not the zuul executor20:17
@michiel:piscaer.comI try to upgrade zuul to version 4.10.4 on a Ubuntu 18.04. Do do this I run sudo pip3 install . I get "RuntimeError: Yarn build failed" with the error "FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory"20:46
@michiel:piscaer.comThe system has 8GB of RAM.20:47
@clarkb:matrix.orgmasterpe: our build nodes also have 8GB of RAM. I wonder if this is a nodejs version specific problem. How are you installing nodejs? We install it from nodesource and use 0.14.x I think. But let me check on that20:48
@clarkb:matrix.orgYup the Dockerfile uses docker.io/library/node:14-bullseye20:49
@clarkb:matrix.orgwhich is a nodesource based docker image for 0.14.latest20:49
@michiel:piscaer.comClark: I don't use docker20:49
@clarkb:matrix.orgI understand, I'm saying that is how we're doing it on 8GB of RAM machines. It might offer a clue that nodejs 0.14.latest will work better than what you have if different20:50
@michiel:piscaer.comroot@zuul-ci:~/zuul# yarn --version20:51
1.22.15
I'm sorry I'm little bit in the dark.
@clarkb:matrix.orgmasterpe: the javascript heap is likely to be node.js. I think the command is often `node` or `nodejs`20:52
@michiel:piscaer.comClark: Thanks for helping out, https://stackoverflow.com/a/59572966 helpt me that nodejs version 10 has a memory leak. And after upgrading to 14 it got solved somewhere. 21:05
@michiel:piscaer.comAfter the upgrade I start the zuul web and I get the error: "AttributeError: module 'sqlalchemy.orm' has no attribute 'declarative_base'"21:25
@clarkb:matrix.orgmasterpe: I think you need to upgrade sqlalchemy as well21:28
@spamaps:spamaps.ems.host> <@spamaps:spamaps.ems.host> Now I'm stuck on `No Auth Provider found for name "gcp"`22:24
This was because for some reason there was a /usr/local/bin/kubectl from who knows where
@spamaps:spamaps.ems.hostHrm.. so.. does the kubectl connection type just like.. not work at all with gather_facts?22:53
@spamaps:spamaps.ems.host```all:22:54
hosts:
ubuntu-pod:
ansible_connection: kubectl
ansible_host: ubuntu-pod-0000000853```
@clarkb:matrix.orgtristanC: ^ would probably know22:58
@tristanc_:matrix.orgspamaps: is there an error in the logs? Perhaps you can try using ansible by hand on a min-ready pod to check the image has the requirements, e.g. gather_facts might need iproute or something. 23:06
@spamaps:spamaps.ems.host```2021-11-18 22:53:03.033228 | TASK [Gathering Facts]23:07
2021-11-18 22:53:11.874577 | ubuntu-pod | ERROR
2021-11-18 22:53:11.875304 | ubuntu-pod | {
2021-11-18 22:53:11.875418 | ubuntu-pod | "msg": "Data could not be sent to remote host \"ubuntu-pod\". Make sure this host can be reached over ssh: ssh: Could not resolve hostname ubuntu-pod: Name or service not known\r\n",
2021-11-18 22:53:11.875488 | ubuntu-pod | "unreachable": true
2021-11-18 22:53:11.875543 | ubuntu-pod | }```
@spamaps:spamaps.ems.hostit's trying to SSH23:07
@tristanc_:matrix.orgthat's odd, `ansible_connection: kubectl` should make it use kubectl exec instead23:09
@spamaps:spamaps.ems.hostKinda stuck. :-/23:13
@spamaps:spamaps.ems.host```root@fe7d6781e92f:/var/lib/zuul/builds/7773c8e9f1424c34914686211a13549e# ANSIBLE_CONFIG=$PWD/ansible/playbook_0/ansible.cfg /usr/local/lib/zuul/ansible/2.9/bin/ansible -i ansible/inventory.yaml -m setup all23:13
[WARNING]: Skipping plugin (/usr/local/lib/zuul/ansible/2.9/lib/python3.8/site-packages/ara/plugins/actions/ara_read.py) as it seems to be invalid:
module 'ansible.plugins.callback.ara_read' has no attribute 'CallbackModule'
[WARNING]: Skipping plugin (/usr/local/lib/zuul/ansible/2.9/lib/python3.8/site-packages/ara/plugins/actions/ara_record.py) as it seems to be invalid:
module 'ansible.plugins.callback.ara_record' has no attribute 'CallbackModule'
[WARNING]: Skipping plugin (/var/lib/zuul/ansible/2.9/zuul/ansible/callback/zuul_json.py) as it seems to be invalid: No module named 'zuul'
[WARNING]: Skipping plugin (/var/lib/zuul/ansible/2.9/zuul/ansible/callback/zuul_stream.py) as it seems to be invalid: No module named 'zuul'
An exception occurred during task execution. To see the full traceback, use -vvv. The error was: ModuleNotFoundError: No module named 'zuul'
ubuntu-pod | FAILED! => {
"msg": "Unexpected failure during module execution.",
"stdout": ""
}```
@spamaps:spamaps.ems.hostTried to run ansible with that inventory but I get that23:14
@spamaps:spamaps.ems.hostI don't see where we're ever passing kubectl_kubeconfig23:23
@spamaps:spamaps.ems.hostI guess we're setting HOME do workdir so it should work?23:23
@spamaps:spamaps.ems.hostOk enough staring at errors23:23

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!