| openstackgerrit | Merged zuul/zuul-jobs master: download-artifact: cleanup long when statement https://review.opendev.org/733446 | 00:00 |
|---|---|---|
| *** smyers has quit IRC | 00:11 | |
| *** smyers has joined #zuul | 00:12 | |
| ianw | ^ just a heads up on that; as AJaeger mentioned the artifact in that test will eventually expire. perhaps by that time we'll have got it converted to a useful mock of zuul api for testing, or perhaps we'll need to to choose a new artifact if the job is updated | 00:38 |
| *** kklimonda has quit IRC | 00:55 | |
| *** kklimonda has joined #zuul | 00:57 | |
| *** wuchunyang has quit IRC | 01:11 | |
| *** swest has quit IRC | 01:22 | |
| *** ysandeep|away is now known as ysandeep | 01:29 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] download-artifact: fix compression https://review.opendev.org/733728 | 01:33 |
| *** swest has joined #zuul | 01:37 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] download-artifact: fix compression https://review.opendev.org/733728 | 02:28 |
| *** wuchunyang has joined #zuul | 02:50 | |
| *** hashar has joined #zuul | 02:52 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] download-artifact: fix compression https://review.opendev.org/733728 | 02:58 |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] download-artifact: fix compression https://review.opendev.org/733728 | 03:12 |
| *** rfolco|rover has quit IRC | 03:15 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: [wip] download-artifact: fix compression https://review.opendev.org/733728 | 03:21 |
| *** bhavikdbavishi has joined #zuul | 03:26 | |
| *** bhavikdbavishi1 has joined #zuul | 03:33 | |
| *** bhavikdbavishi has quit IRC | 03:35 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:35 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Revert "download-artifact : support recursive download" https://review.opendev.org/733739 | 03:43 |
| ianw | zuul-maint: ^ that has turned out to be a real PITA; not only does it not work everywhere due to gzip issues with some providers, but it can affect the non-recursive path. i'm going to take the liberty of merging the revert until we can rewrite it better | 03:59 |
| *** wuchunyang has quit IRC | 04:03 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "download-artifact : support recursive download" https://review.opendev.org/733739 | 04:08 |
| bhavikdbavishi | Hi, I'm looking for Github PR webhook proxy to internal network, like Amazon SQS or some thing similar. just for background, we are migrating hosted server to close internal network, with this zuul endpoint will not have public IP address and so it won't be accessible to github for webhook push. | 04:13 |
| bhavikdbavishi | in case ngrok we can directly use it and forward PR to internal host (IT might not approve use of ngrok), or probably host on EC2 with https://github.com/BonnyCI/github-webhook-proxy to forward/push PR to internal host. but we are looking for using of Amazon SQS to avoid maintaining any external host. is there any reference of implementation of zuul using Amazon SQS or similar that I can take a look? | 04:13 |
| *** ysandeep is now known as ysandeep|off | 04:27 | |
| *** evrardjp has quit IRC | 04:33 | |
| *** evrardjp has joined #zuul | 04:33 | |
| *** wuchunyang has joined #zuul | 04:43 | |
| *** hashar has quit IRC | 05:15 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: download-artifact: fix missed part of revert https://review.opendev.org/733744 | 05:25 |
| *** wuchunyang has quit IRC | 05:27 | |
| openstackgerrit | Merged zuul/zuul-jobs master: download-artifact: fix missed part of revert https://review.opendev.org/733744 | 05:34 |
| AJaeger | ianw: thanks for the revert, I saw promote jobs failing like in https://review.opendev.org/728569 | 05:36 |
| ianw | AJaeger: yeah, 733744 sorry :( | 05:41 |
| *** bhavikdbavishi has quit IRC | 05:42 | |
| AJaeger | yeah, reviewed now and figured it out ;/ | 05:43 |
| AJaeger | at least the promote from that one worked ;) | 05:43 |
| ianw | AJaeger: i should have taken it as a sign when I had about 8 options to wget that it was not heading in the right direction | 05:47 |
| ianw | anyway, i think there *is* a pretty nice way to achieve it via the manifest, when someone feels like writing a minor reimplementation of wget :) | 05:48 |
| *** bhavikdbavishi has joined #zuul | 06:07 | |
| *** evgenyl has quit IRC | 06:10 | |
| *** evgenyl has joined #zuul | 06:10 | |
| *** hashar has joined #zuul | 07:04 | |
| *** rpittau|afk is now known as rpittau | 07:04 | |
| *** felixedel has joined #zuul | 07:09 | |
| *** bhavikdbavishi has quit IRC | 07:11 | |
| *** jpena|off is now known as jpena | 07:15 | |
| *** saneax has joined #zuul | 07:15 | |
| frickler | so is there a zuul-native way to make dns lookups in zuul? seems using the "dig" lookup plugin on the executors is forbidden, see https://review.opendev.org/733614 for the intended use | 07:25 |
| *** yolanda has joined #zuul | 07:25 | |
| frickler | we cannot use DNS resolution within instances easily because of the way unbound is set up | 07:26 |
| *** tosky has joined #zuul | 07:31 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul-jobs master: tox: allow tox to be upgraded https://review.opendev.org/690057 | 07:34 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 07:40 |
| *** jcapitao has joined #zuul | 07:45 | |
| *** bhavikdbavishi has joined #zuul | 07:57 | |
| openstackgerrit | Felix Edel proposed zuul/zuul-jobs master: Return upload_results in upload-logs-swift role https://review.opendev.org/733564 | 08:06 |
| *** nils has joined #zuul | 08:14 | |
| openstackgerrit | Felix Edel proposed zuul/zuul-jobs master: Return upload_results in upload-logs-swift role https://review.opendev.org/733564 | 08:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 08:21 |
| openstackgerrit | Felix Edel proposed zuul/zuul-jobs master: Return upload_results in upload-logs-swift role https://review.opendev.org/733564 | 08:39 |
| *** wuchunyang has joined #zuul | 08:56 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs", test without depends-on https://review.opendev.org/733520 | 09:18 |
| *** felixedel has quit IRC | 09:43 | |
| openstackgerrit | Sorin Sbarnea (zbr) proposed zuul/zuul master: Make task errors expandable https://review.opendev.org/723534 | 09:53 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs", test without depends-on https://review.opendev.org/733520 | 10:01 |
| *** wuchunyang has quit IRC | 10:01 | |
| *** rpittau is now known as rpittau|bbl | 10:08 | |
| *** wuchunyang has joined #zuul | 10:15 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 10:16 |
| *** bhavikdbavishi has quit IRC | 10:27 | |
| tobiash | frickler: the dig lookup plugin seems safe to me for local usage so we might be able to whitelist it in zuul | 10:28 |
| tobiash | corvus: what do you think? ^ | 10:28 |
| zbr | ianw thanks for the comment on https://review.opendev.org/#/c/690057 -- i fixed it, apparently tristanC and avass missed it. | 10:28 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "gate pipeline" https://review.opendev.org/732069 | 10:29 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "job secrets" https://review.opendev.org/732070 | 10:29 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "job dependencies" https://review.opendev.org/732071 | 10:29 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: build tutorial image one time and use registry https://review.opendev.org/733005 | 10:29 |
| *** wuchunyang has quit IRC | 10:41 | |
| *** hashar has quit IRC | 10:50 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 11:03 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add support for overriding parameters https://review.opendev.org/733809 | 11:03 |
| *** bhavikdbavishi has joined #zuul | 11:07 | |
| *** jcapitao is now known as jcapitao_lunch | 11:07 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 11:11 |
| *** fbo|off is now known as fbo | 11:13 | |
| *** brendangalloway has joined #zuul | 11:18 | |
| brendangalloway | Is there any mechanism by which a user can manually checkout/hold a nodepool node without having a job fail on it? We use a number of static nodes for our testing and would like a way to flag that some debugging is being done on them after a failed build so that nodepool doesn't reuse it until that is done | 11:25 |
| avass | brendangalloway: there's: https://review.opendev.org/#/c/679306/ | 11:30 |
| *** bhavikdbavishi has quit IRC | 11:30 | |
| avass | brendangalloway: but yeah I understand what you mean, we want it too. | 11:30 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 11:31 |
| brendangalloway | avass: ok, that's essentially having a sit and wait if it detects some sort of error? Like a hold, but also prepping the node to be logged into for debugging? | 11:34 |
| brendangalloway | avass: We're looking for something different - since our nodes are static we're less interested in being able to pause the job, more interested in stopping the next job from running on it. | 11:35 |
| avass | brendangalloway: yeah we have the same need, being able to offline a node to work on it directly. | 11:36 |
| avass | brendangalloway: and as far as I know there's no direct way to do that at the moment, but there are some workarounds | 11:37 |
| brendangalloway | avass: ok, thanks. Something like 'nodepool checkout' would be what we were thinking | 11:37 |
| avass | brendangalloway: nodepool.yaml is loaded automatically so if you can update that you can just comment out the node you need to offline | 11:38 |
| brendangalloway | avass: sure, but that's either a commit to the config repo or devs touching things I'd rather they didn't touch whenever they want to debug | 11:40 |
| avass | that's pretty much how we're doing it at the moment | 11:40 |
| avass | yup | 11:40 |
| brendangalloway | avass: ok, thanks. I can act as emergency sed mechanism in the meantime | 11:41 |
| avass | we're also automatically deploying our nodepool config so yeah, we're just pushing changes to gerrit with a quick review :) | 11:42 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: build tutorial image one time and use registry https://review.opendev.org/733005 | 11:45 |
| *** jpena is now known as jpena|lunch | 11:48 | |
| *** rfolco|rover has joined #zuul | 11:51 | |
| *** jcapitao_lunch is now known as jcapitao | 11:58 | |
| *** rpittau|bbl is now known as rpittau | 11:59 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 12:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 12:16 |
| *** rlandy has joined #zuul | 12:26 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 12:28 |
| masterpe | How to specify where the logs of generate-zuul-manifest and or upload-logs needed to saved, I see that in the docker the executor saves the logs in /srv/static/logs. | 12:34 |
| *** tosky_ has joined #zuul | 12:35 | |
| *** tosky has quit IRC | 12:37 | |
| *** tosky has joined #zuul | 12:39 | |
| *** tosky_ has quit IRC | 12:40 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add support for overriding parameters https://review.opendev.org/733809 | 12:41 |
| *** tosky_ has joined #zuul | 12:41 | |
| *** tosky has quit IRC | 12:43 | |
| *** bhavikdbavishi has joined #zuul | 12:46 | |
| *** tosky_ is now known as tosky | 12:50 | |
| *** saneax has quit IRC | 12:54 | |
| *** jpena|lunch is now known as jpena | 13:01 | |
| *** sgw has quit IRC | 13:04 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 13:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Add support for overriding parameters https://review.opendev.org/733809 | 13:11 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 13:35 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 13:40 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 13:43 |
| masterpe | found it, had to specifyed the variable zuul_logserver_root | 13:53 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 13:53 |
| *** bhavikdbavishi has quit IRC | 14:13 | |
| corvus | tobiash, frickler: i agree, dig lookup looks like something we could whitelist | 14:22 |
| *** sugaar has quit IRC | 14:26 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 14:38 |
| masterpe | If I set zuul_logserver_root: "/tmp" this works, if I set zuul_logserver_root: "/var/www/html" then it don't work. I have set the rights of the directory /var/www/html to 777. | 14:40 |
| masterpe | Why doesn't it work? | 14:40 |
| masterpe | is it becourse trusted_rw_paths is set to /tmp? | 14:41 |
| masterpe | in my zuul.conf? | 14:41 |
| *** sugaar has joined #zuul | 14:50 | |
| avass | masterpe: yeah that could be it, you need to add that path to trusted read/write | 14:51 |
| *** hashar has joined #zuul | 14:53 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 14:57 |
| masterpe | Current I have a vagrant setup with a openstack heat config, where I can deploy development openstack enviorment with multiple networks and multiple instances. | 14:58 |
| masterpe | In this vagrant I have a steppingstone where I have openstack-ansible installed. | 14:59 |
| masterpe | Now I want to use zuul to test the newest configs, but what setup, sould I use. | 15:00 |
| tobiash | corvus: are the yarn packages supposed to be mirrored in opendev? | 15:01 |
| masterpe | As I untherstand it the steppingstone sould be a zuul executor? | 15:01 |
| corvus | tobiash: i think we have a caching proxy for the npm registry | 15:07 |
| clarkb | corvus: tobiash ya we do. Not sure if we configure it in jobs by deafult though | 15:08 |
| tobiash | corvus: I had quite a long struggle with yarn to persuade it to actually use a mirror | 15:08 |
| tobiash | tldr, yarn ignores any mirror in presence of yarn.lock because reuses the urls that are defined there | 15:09 |
| corvus | masterpe: yes that's possible, but only trusted jobs (playbooks in config repos) can run arbitrary commands on an executor, so it may be somewhat difficult. you may want to define a static node or have some kind of dynamic stepping stone (maybe a k8s pod or something). | 15:09 |
| tobiash | I had patch the yarn.lock in order to force yarn to use the mirror | 15:09 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 15:11 |
| masterpe | Can zuul use heat? | 15:11 |
| clarkb | masterpe: nodepool's existing openstack driver does not, but you could implement aheat based driver if it would be useful to you | 15:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: WIP: Add terraform roles https://review.opendev.org/733675 | 15:14 |
| tobiash | corvus, clarkb: I didn't see something about an npm proxy in tox-py38 of zuul | 15:23 |
| corvus | tobiash: yeah, we may not be using it | 15:26 |
| tobiash | probably | 15:26 |
| tobiash | just wanted to note that if we were thinking we're using it we would be probably wrong | 15:27 |
| corvus | http://codesearch.openstack.org/?q=NODEPOOL_NPM_REGISTRY_PROXY&i=nope&files=&repos= | 15:27 |
| corvus | i'm not sure anything is using it | 15:27 |
| corvus | http://codesearch.openstack.org/?q=registry.npmjs%2F&i=nope&files=&repos= | 15:28 |
| corvus | possibly kolla | 15:28 |
| corvus | tobiash: http://codesearch.openstack.org/?q=nodepool_npmjs_proxy&i=nope&files=&repos= | 15:29 |
| corvus | tobiash: does that look plausible? | 15:29 |
| tobiash | checking | 15:30 |
| tobiash | kolla seems to be safe, I didn't find a yarn.lock file there | 15:30 |
| tobiash | if we want to use an npm mirror for zuul we'd need to do this before running yarn: sed -i 's#https://registry.yarnpkg.com#<LOCAL_REGISTRY>#' <PATH_TO>/yarn.lock | 15:32 |
| corvus | that's icky :( ^ mordred fyi | 15:33 |
| tobiash | it took me a while to find out why yarn ignored all my configured registry settings in various places | 15:33 |
| masterpe | But what do I need to do if I want to run ansible on other node then the zuul executer? | 15:33 |
| tobiash | related upstream issue: https://github.com/yarnpkg/yarn/issues/5892 | 15:33 |
| tobiash | masterpe: you could just run ansible in a shell task | 15:34 |
| masterpe | Ok | 15:34 |
| mordred | corvus, tobiash: *headdesk* | 15:36 |
| mordred | lock-treatment-tool | 15:36 |
| tobiash | mordred: yeah, did that the whole afternoon ;) | 15:36 |
| mordred | locktt --registry=https://npmregistryurl.com | 15:36 |
| mordred | *insanity ensues* | 15:37 |
| corvus | using that is going to be tricky.... it's fine for testing, but modifying the lock file in the git tree during release builds may be bad | 15:39 |
| corvus | (it might be okay, but it might not; we'd have to think about it) | 15:39 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 15:42 |
| tobiash | they really have a tool for that? | 15:43 |
| *** nils has quit IRC | 15:56 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 15:58 |
| *** brendangalloway has quit IRC | 16:03 | |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab - add driver documentation https://review.opendev.org/733880 | 16:08 |
| *** rpittau is now known as rpittau|afk | 16:10 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 16:25 |
| openstackgerrit | Fabien Boucher proposed zuul/zuul master: gitlab - add driver documentation https://review.opendev.org/733880 | 16:38 |
| *** jpena is now known as jpena|off | 16:40 | |
| *** jcapitao has quit IRC | 16:53 | |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: tutorial: Add "Use zuul jobs" test https://review.opendev.org/733520 | 16:58 |
| *** fbo is now known as fbo|off | 17:00 | |
| *** rfolco|rover is now known as rfolco|rover|brb | 17:22 | |
| *** sgw has joined #zuul | 17:24 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 17:30 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 17:42 |
| mordred | corvus: perhaps for release builds what we want to do is modify the lock file with the tool, do yarn install, then do git reset to unmodify the lock file, then do the actual build | 17:56 |
| mordred | corvus: in fact, maybe that's a decent pattern for "apply zuul mirrors to a javascript build" - we can actually run locktt with npx without adding it to the package.json (npx allows you to download and run javascript commands wihtout "installing" them permanently) | 17:57 |
| openstackgerrit | masterpe proposed zuul/zuul master: gitlab - add driver documentation https://review.opendev.org/733880 | 17:58 |
| mordred | so the zuul javascript roles coudl do "npx locktt --registery=https://mirror.example.com/npm ; yarn install ; git reset" just as part of normal life | 17:58 |
| mordred | tobiash: && | 17:58 |
| tobiash | mordred: sounds good | 18:00 |
| corvus | mordred: sgtm | 18:02 |
| mordred | for the record: npx -p lock-treatment-tool locktt | 18:02 |
| mordred | is how you run locktt with npx | 18:02 |
| mordred | and I have verified that npx -p lock-treatment-tool locktt --registry https://registry.example.com | 18:04 |
| mordred | did what we expect | 18:04 |
| mordred | we want to add "-s" - which tells it to skip integrity removal - since I would expect our mirroring to not introduce integrity issues | 18:05 |
| mordred | it also does ./package-lock.json if it's there- so it should be relatively safe for us to use across the board in our roles | 18:05 |
| *** hashar is now known as hasharAway | 18:14 | |
| *** brendangalloway has joined #zuul | 18:38 | |
| brendangalloway | Hi - our nodepool-launcher instance has gotten itself into a loop where it's spitting out a python exception due to kubernetes not being able to find a temporary file during the driver reset. Any idea what could be going wrong? | 18:39 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 18:47 |
| tristanC | brendangalloway: could you please paste the traceback to http://paste.openstack.org/ ? | 18:47 |
| avass | The terraform change is probably ready now, I keep coming up with things I need to add :) | 18:50 |
| tristanC | avass: the `terraform : List workspaces` is failing with `/bin/sh: 1: set: Illegal option -o pipefail` on debian | 18:53 |
| brendangalloway | tristanC: like this? http://paste.openstack.org/show/794414/ | 18:53 |
| tristanC | avass: perhaps a missing `executable: /bin/bash` to avoid dash | 18:53 |
| avass | tristanC: ah yep | 18:54 |
| tristanC | brendangalloway: yes thanks. Is there a `/tmp/tmp` string in your kube/config file? | 18:54 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 18:56 |
| tristanC | brendangalloway: e.g. could it be that your cluster ca or cert is a file reference to /tmp which is not available on the nodepool-launcher host? | 18:56 |
| brendangalloway | tristanc: The kube.conf file pointed to in sfconfig? | 18:56 |
| tristanC | brendangalloway: yes, which should be copied over to the `~nodepool/.kube/config` directory | 18:57 |
| brendangalloway | tristanc: it was copied, there's no file references in it though | 18:58 |
| tristanC | brendangalloway: looking at the stacktrace, it seems to be related to `self.key_file = FileOrData(self._user, 'client-key', ...` failing to load a file | 18:59 |
| tristanC | when loading the cluster info | 18:59 |
| brendangalloway | tristanc: the client-key-data and client-cert-data fields in the config file both have hashes | 19:00 |
| *** dustinc has joined #zuul | 19:01 | |
| tristanC | brendangalloway: in otherwords, here is the code raising an exception: https://github.com/kubernetes-client/python-base/blob/49ec06096ebc4e092c1ebc2291bf18ccaac935af/config/kube_config.py#L93-L121 | 19:03 |
| tristanC | brendangalloway: called from https://github.com/kubernetes-client/python-base/blob/49ec06096ebc4e092c1ebc2291bf18ccaac935af/config/kube_config.py#L518-L520 | 19:05 |
| tristanC | brendangalloway: which seems to indicate an invalid cluster configuration, perhaps a missing mandatory `client-key` attribute | 19:06 |
| *** rfolco|rover|brb is now known as rfolco|rover | 19:06 | |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Terraform roles and jobs. https://review.opendev.org/733675 | 19:09 |
| brendangalloway | tristanc: There's a client-key-data hash. Could the _create_temp_file_with_content call be failing? | 19:09 |
| tristanC | brendangalloway: i'm not familiar with kubernetes config, could you try running `kubectl --kubeconfig ~nodepool/.kube/config cluster-info` and see if the golang impl can read it? | 19:11 |
| brendangalloway | tristanc: It gives a cert error, but it reads it | 19:12 |
| tristanC | corvus: we are affected by a rather critical issue (one tenant configuration locked down the whole scheduler), the traceback is reported in: https://storyboard.openstack.org/#!/story/2007761 . Do you think we could add per tenant try/except in the manager.processQueue function? | 19:13 |
| tristanC | brendangalloway: it seems like `certificate-authority` and `client-certificate` are correctly loaded, would your `client-key` simply missing a `client-key-data` field? | 19:15 |
| corvus | tristanC: ack i'll take a look after lunch | 19:15 |
| brendangalloway | tristanc: there's a client-key-data field, but not a client-key field | 19:16 |
| tristanC | brendangalloway: that seems to be issue, not sure what it should be, but a cert error happening when using the kubectl cli directly may be an issue for nodepool python client | 19:18 |
| tristanC | corvus: thanks, some more details are also in the initial report: https://tree.taiga.io/project/morucci-software-factory/issue/3577 | 19:18 |
| brendangalloway | tristanc: Thanks, I will investigate that further | 19:21 |
| *** brendangalloway has quit IRC | 19:21 | |
| avass | tristanC: now it should be ready :) | 19:29 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "Use zuul jobs" https://review.opendev.org/732068 | 19:34 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "gate pipeline" https://review.opendev.org/732069 | 19:34 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "job secrets" https://review.opendev.org/732070 | 19:34 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: tutorial: Add "job dependencies" https://review.opendev.org/732071 | 19:34 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: DNM: build tutorial image one time and use registry https://review.opendev.org/733005 | 19:34 |
| *** hasharAway is now known as hashar | 19:57 | |
| corvus | tristanC: do you think you can collect the logs for that change right before the first time that traceback appears? | 20:45 |
| corvus | tristanC: and also one pipeline processing iteration before that | 20:46 |
| corvus | tristanC: i don't immediately see the sequence of events for how it got into that state | 20:47 |
| tristanC | corvus: looking for that, one moment | 20:49 |
| *** rfolco|rover has quit IRC | 20:50 | |
| tristanC | corvus: i've added the requested logs to the issue ( https://storyboard.openstack.org/#!/story/2007761 ) | 20:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: WIP: contain pipeline exceptions https://review.opendev.org/733917 | 20:58 |
| corvus | tristanC: ^ meanwhile, what do you think about that for containing the fallout? i added it to the status json too, so we can put an error banner on the status page | 20:58 |
| corvus | tristanC: does there happen to be two gerrit connections used in that tenant? | 21:04 |
| corvus | tristanC: is the pipeline config for that tenant publicly available? | 21:06 |
| corvus | tristanC: what i'm getting from the logs is that the gerrit pipeline reporter appears to be from a different gerrit connection than whence the change originated | 21:15 |
| tristanC | corvus: containing sounds like the best thing to do, and until the root caused is handled we had to manually dequeue the offending change, so adding an error banner is ideal | 21:17 |
| tristanC | corvus: it seems like the tenant is wrongly configured using different gerrits for triggering and reporting (there are actually 5 gerrits in that setup) | 21:19 |
| corvus | tristanC: ok, i think that confirms/explains the error; i bet we can detect and handle that better | 21:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Detect Gerrit gate pipelines with the wrong connection https://review.opendev.org/733929 | 21:44 |
| corvus | tristanC: i think that fixes the underlying issue ^ | 21:44 |
| *** rlandy has quit IRC | 21:58 | |
| *** dustinc has quit IRC | 22:01 | |
| tristanC | corvus: thanks you so much for the fast fix, i'll see if we can pick that on our affected system to confirm it fix the issue | 22:02 |
| corvus | tristanC: np, thanks for the report :) | 22:13 |
| *** rfolco|rover has joined #zuul | 22:15 | |
| *** hashar has quit IRC | 22:36 | |
| *** tosky has quit IRC | 23:56 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!