| *** rlandy has quit IRC | 00:10 | |
| *** mattw4 has quit IRC | 00:19 | |
| *** sshnaidm has quit IRC | 00:44 | |
| openstackgerrit | Paul Belanger proposed zuul/nodepool master: Add error handling when cleaning up resources https://review.opendev.org/661866 | 01:04 |
|---|---|---|
| pabelanger | clarkb: Shrews: we managed to wedge our cleanup handler in zuul.a.c, due to a provider outage, ^ is my attempt to fix it | 01:05 |
| clarkb | that fix makes sense. I'll have to vote on it in the mroning though. Currently making dinner | 01:07 |
| pabelanger | wfm | 01:07 |
| pabelanger | going to diable the provider for now | 01:07 |
| pabelanger | also, starlink goes over head tonight at 22:23 EST, hopefully get to see them | 01:13 |
| fungi | i get sms notifications on my phone when the iss is about to go overhead | 01:18 |
| *** sshnaidm has joined #zuul | 01:50 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: add host-key-checking toggle https://review.opendev.org/653679 | 02:38 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: static: enable using a single host with different user or port https://review.opendev.org/659209 | 02:38 |
| *** panda|ruck has quit IRC | 03:03 | |
| *** panda has joined #zuul | 03:03 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: model: add cleanup-run to the job configuration https://review.opendev.org/661880 | 03:07 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: wip: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 03:07 |
| *** threestrands has joined #zuul | 03:31 | |
| *** altlogbot_0 has quit IRC | 03:44 | |
| *** altlogbot_2 has joined #zuul | 03:46 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Report tenant and project specific resource usage stats https://review.opendev.org/616306 | 04:10 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Fix typo in docs https://review.opendev.org/661886 | 04:13 |
| *** raukadah is now known as chandankumar | 04:37 | |
| *** altlogbot_2 has quit IRC | 04:38 | |
| *** altlogbot_2 has joined #zuul | 04:42 | |
| *** pcaruana has joined #zuul | 05:15 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: model: add cleanup-run to the job configuration https://review.opendev.org/661880 | 05:42 |
| *** gtema has joined #zuul | 05:55 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: wip: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 06:22 |
| *** bjackman has joined #zuul | 06:30 | |
| *** saneax has joined #zuul | 06:49 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: wip: executor: run cleanup playbook on stop https://review.opendev.org/661881 | 06:59 |
| *** hashar has joined #zuul | 07:03 | |
| *** tosky has joined #zuul | 07:11 | |
| *** themroc has joined #zuul | 07:34 | |
| *** tosky has quit IRC | 07:40 | |
| *** tosky has joined #zuul | 07:42 | |
| *** flepied has joined #zuul | 07:55 | |
| *** jpena|off is now known as jpena | 07:58 | |
| openstackgerrit | Merged zuul/zuul master: Report tenant and project specific resource usage stats https://review.opendev.org/616306 | 08:21 |
| *** panda is now known as panda|ruck | 08:23 | |
| *** sshnaidm has quit IRC | 08:30 | |
| *** sshnaidm has joined #zuul | 08:31 | |
| openstackgerrit | Andreas Jaeger proposed x/pbrx master: Retire repo https://review.opendev.org/661912 | 08:34 |
| *** bjackman has quit IRC | 08:46 | |
| openstackgerrit | Slawek Kaplonski proposed zuul/zuul-jobs master: Add role to fetch journal log from test node https://review.opendev.org/643733 | 08:53 |
| *** MrCoder25 has joined #zuul | 09:06 | |
| jkt | I'm using the `command` Ansible module for launching my shell script which runs my project's tests. How does stdin of that shell script look like? | 09:08 |
| jkt | I'm asking because I'm debugging a random failure within a 3rd-party project that I'm testing. Apparently, it calls poll(stdin), and the result is revents=POLLHUP | 09:08 |
| jkt | this thing used to work before, whizh puzzles me. The only change I made was reinstalling the build containers (Fedora 29), but again, I used the same Ansible playbook for that installation as before | 09:09 |
| jkt | any clues? :) | 09:09 |
| *** bjackman has joined #zuul | 09:15 | |
| *** hashar has quit IRC | 09:28 | |
| *** saneax has quit IRC | 09:32 | |
| *** saneax has joined #zuul | 09:32 | |
| *** MrCoder25_ has joined #zuul | 09:39 | |
| MrCoder25_ | Hi, i'm using zuul 3.8.1 and trying to use the kubernetes nodepool driver and executing jobs | 09:40 |
| *** MrCoder25 has quit IRC | 09:40 | |
| MrCoder25_ | I managed to have nodepool create a pod but i'm having difficulties connecting to the pod from my zuul job | 09:40 |
| MrCoder25_ | I get the following error: 2019-05-29 11:37:07,587 DEBUG zuul.AnsibleJob: [build: 12a85cdb5bfa48459dd973d24a65e64b] Ansible output: b'fatal: [pod-centos]: FAILED! => {"changed": false, "module_stderr": "error: You must be logged in to the server (Unauthorized)\\n", "module_stdout": "", "msg": "MODULE FAILURE\\nSee stdout/stderr for the exact error", "rc": 1} | 09:41 |
| MrCoder25_ | I assume that the error message is from the kubectl client but I am not sure what i need to do to resolve it | 09:41 |
| MrCoder25_ | Is there an example job that shows how to run a job on a kubernetes pod created by nodepool? | 09:42 |
| MrCoder25_ | Or can anyone give me any pointers what i need to do to resolve it | 09:42 |
| *** threestrands has quit IRC | 09:54 | |
| *** electrofelix has joined #zuul | 10:06 | |
| *** gtema has quit IRC | 10:48 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** jangutter_ has joined #zuul | 11:32 | |
| *** jangutter has quit IRC | 11:36 | |
| *** bjackman has quit IRC | 12:27 | |
| *** bjackman has joined #zuul | 12:29 | |
| *** jpena|lunch is now known as jpena | 12:34 | |
| *** rlandy has joined #zuul | 12:35 | |
| *** jangutter has joined #zuul | 12:51 | |
| *** jangutter_ has quit IRC | 12:54 | |
| *** bjackman has quit IRC | 13:02 | |
| *** bjackman has joined #zuul | 13:03 | |
| *** dmellado has quit IRC | 13:25 | |
| *** dmellado has joined #zuul | 13:25 | |
| *** bjackman has quit IRC | 13:26 | |
| *** armstrongs has joined #zuul | 13:55 | |
| *** rf0lc0 has joined #zuul | 13:58 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 13:59 |
| *** rfolco has quit IRC | 14:00 | |
| *** rf0lc0 is now known as rfolco | 14:16 | |
| armstrongs | hey on the zuul executor is it possible to get ansible to use a custom ansible.cfg as i have some custom ansible library modules i need to use so would like to override the default one | 14:21 |
| armstrongs | i have it stored within my git repo | 14:21 |
| pabelanger | armstrongs: not today, ansible.cfg is hardcoded in zuul-executor and not exposed. I have often thought how we could expose it, since for ansible-network we need to setup different settings for network_cli connections | 14:25 |
| armstrongs | so theres no way to point at my custom library modules, that means i can't run my tests :( Is there any way to hack it? | 14:27 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 14:27 |
| SpamapS | armstrongs:modules are loaded in the playbook and role paths | 14:28 |
| SpamapS | armstrongs:you just can't load *plugins* | 14:28 |
| pabelanger | armstrongs: if the role is trusted, I believe zuul-executor will load it, if next to playbook, IRC | 14:28 |
| SpamapS | and yes, as pabelanger says, trusted context can run any module on the executor | 14:28 |
| SpamapS | (though I've kind of taken the position that executor-only jobs are more trouble than they're worth) | 14:28 |
| pabelanger | armstrongs: in this case, you likey want to ue nested ansible, zuul-executor ansible-playbook, runs ansible-playbook on the node for nodepool | 14:29 |
| armstrongs | ah so just put it under trusted rather than untrusted | 14:29 |
| pabelanger | this is a common way we do testing in ansible-network | 14:29 |
| armstrongs | so just move it in the main.yml | 14:29 |
| armstrongs | the repo that contains it | 14:30 |
| pabelanger | armstrongs: the down side with trusted, you won't get pre-merge testing | 14:30 |
| armstrongs | hmm ok so alternative is to setup a zuul nodepool node with ansible installed execute it there and it will load the cfg from the workspace? | 14:33 |
| pabelanger | yup, that is right | 14:33 |
| armstrongs | so a nested ansible call | 14:33 |
| armstrongs | ok | 14:33 |
| armstrongs | thanks again guys | 14:33 |
| armstrongs | will try that | 14:34 |
| armstrongs | :) | 14:34 |
| armstrongs | sounds better | 14:34 |
| fungi | it's probably the safer option at least, since that code is more thoroughly sandboxed by being on a separate, stateless machine | 14:35 |
| fungi | so the most which could happen by exploiting it (probably) would be compromise of the artifacts produced in that build | 14:36 |
| Shrews | https://duo.com/decipher/docker-bug-allows-root-access-to-host-file-system | 14:44 |
| *** saneax has quit IRC | 14:46 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 14:48 |
| Shrews | pabelanger: i don't see what your nodepool patch is fixing. that cleanup task is already wrapped in an exception handler: https://opendev.org/zuul/nodepool/src/branch/master/nodepool/launcher.py#L666 | 14:50 |
| clarkb | Shrews: it stops that loop from breaking out | 14:50 |
| clarkb | so if the failure is consistent you will only eveer get that far through the for loop before breaking out | 14:51 |
| Shrews | clarkb: oh, i see now | 14:51 |
| clarkb | I nees to leave a +2 once at computer | 14:52 |
| Shrews | i think a test case would be good though | 14:52 |
| pabelanger | yah, what clarkb said :) I can also do a test case | 14:52 |
| pabelanger | I believe we have something today to test leaked instances | 14:53 |
| *** chandankumar is now known as raukadah | 14:54 | |
| Shrews | pabelanger: yep. there should be an example there too of getting hold of the manager. should be easy to mock the failure. | 14:58 |
| pabelanger | +1 | 14:59 |
| Shrews | pabelanger: the difficult thing would be controlling the order of the provider loop to guarantee we proceed after the failure | 15:00 |
| Shrews | maybe a test isn't really necessary here anyway. | 15:02 |
| Shrews | pabelanger: ok, i think i talked myself out of that :) | 15:02 |
| pabelanger | hehe | 15:02 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 15:18 |
| *** mattw4 has joined #zuul | 15:24 | |
| tobiash | SpamapS: re pre-populate private keys, do you have an idea how we can get rid of pycrypto for seeding the private key generation? | 15:27 |
| clarkb | tobiash: zuul could fork to openssl/openssh if a seed/master key is provided? | 15:27 |
| clarkb | would probably be slow but its a one time cost (mostly) | 15:27 |
| *** mattw4 has quit IRC | 15:28 | |
| tobiash | is that something that openssl supports? | 15:28 |
| *** armstrongs has quit IRC | 15:33 | |
| SpamapS | Probably worth investigating pycryptodome .. also makes one wonder why cryptography doesn't allow the seed argument. Might be a relatively simple patch. | 15:33 |
| clarkb | tobiash: openssl genrsa takes a -rand argument which lists files to load into the seed | 15:33 |
| clarkb | tobiash: genrsa is apparently superceded by genpkey which does not document -rand as an arg | 15:33 |
| pabelanger | tobiash: I like the idea of a master key, I should test that out too | 15:34 |
| clarkb | can probably get away with genrsa until they delete it? | 15:34 |
| SpamapS | I'm also kind of wondering again (I think I wondered this when we started) why we don't use PGP | 15:34 |
| clarkb | (as an option with forking) | 15:34 |
| *** altlogbot_2 has quit IRC | 15:35 | |
| *** irclogbot_0 has quit IRC | 15:36 | |
| openstackgerrit | Andreas Jaeger proposed x/pbrx master: Retire repo https://review.opendev.org/661912 | 15:37 |
| *** altlogbot_1 has joined #zuul | 15:37 | |
| openstackgerrit | Merged zuul/zuul master: Fix typo in docs https://review.opendev.org/661886 | 15:38 |
| *** irclogbot_0 has joined #zuul | 15:38 | |
| *** mattw4 has joined #zuul | 15:40 | |
| fungi | SpamapS: if memory serves (it's been a while now so this may also have changed) there were no great options for openpgp python modules, and also it would have been more finicky to implement the command-line encryption tool | 15:41 |
| *** MrCoder25_ has quit IRC | 15:43 | |
| fungi | also, for the use case under discussion... couldn't we just symetrically encrypt the private rsa keys on the server? then all zuul needs to do is fetch the decryption key from vault or wherever and decrypt the private rsa keys into memory when it needs to decrypt a job secret with one | 15:44 |
| fungi | unless i'm misunderstanding the problem statement | 15:45 |
| tobiash | fungi: my use case is to not having to backup the private keys | 15:46 |
| *** themroc has quit IRC | 15:46 | |
| *** mattw4 has quit IRC | 15:46 | |
| fungi | oh, so you want the private keys to be two-party keys? | 15:47 |
| fungi | or you're wanting to store all the private keys somewhere central which does get backed up? | 15:48 |
| clarkb | fungi: tobiash uses a master key (seed) and can regenerate all keys from that | 15:48 |
| SpamapS | If you use a single random key as the seed for all the others, you can re-create all the others with the seed only. | 15:48 |
| fungi | what does it get salted with to keep them from all being the same key? an incremented serial? | 15:49 |
| clarkb | I'm guessnig project name | 15:50 |
| fungi | i guess that would work as long as you never rename a project | 15:50 |
| fungi | or you keep track of what the original project was to make sure you can regenerate the original key | 15:50 |
| tobiash | I have a master key in the deployment and deterministically generate all project specific keys from that | 15:50 |
| tobiash | That way zuul is stateless from this perspective | 15:51 |
| fungi | mostly wondering how you make the regeneration deterministic without making them all wind up with the exact same key | 15:51 |
| tobiash | I seed the prng from the private master key | 15:51 |
| fungi | using the same random seed multiple times would in theory produce the same key every time | 15:51 |
| tobiash | With the project name as salt | 15:52 |
| fungi | got it, that's the piece i was looking for | 15:52 |
| fungi | so do you never rename projects, or do you generate new keys for them when you rename them, or do you track their original names so as to still be able to regenerate their keys? | 15:52 |
| *** hashar has joined #zuul | 15:55 | |
| fungi | also i guess you don't have multiple tenants, or you use a different seed per tenant? | 15:55 |
| fungi | (otherwise two identically-named projects in different tenants would get the same key) | 15:55 |
| openstackgerrit | Merged zuul/zuul master: Create zuul/web/static on demand https://review.opendev.org/661498 | 15:56 |
| tobiash | Keys are only depending on the canoni al project name | 15:57 |
| tobiash | Not on the tenan6 | 15:57 |
| fungi | so if you had two projects with the same canonical name in different tenants they'd have the same key, i suppose it depends on how you're controlling access to configure and run jobs for those projects as to whether or not that's a concern | 16:00 |
| fungi | could be seen as a feature rather than a risk, if you wanted to reuse the same secret in jobs for the same project in different tenants | 16:01 |
| fungi | by "canonical name" i assume you mean including the hostname for the connection | 16:02 |
| tobiash | That's how it is in zuul generally atm | 16:03 |
| tobiash | And yes, renaming a project requires reencrypting the secrets of that project | 16:04 |
| tobiash | The canonical name in zuul is unique for any project | 16:05 |
| tobiash | (within one zuul deployment) | 16:05 |
| fungi | yep, the biggest risk i could see is if you had tenant project configuration under control of different groups, one tenant could add a project from another tenant and then be able to add jobs which expose the secrets in those projects | 16:06 |
| Shrews | does anyone know how to have tox not use a regex on the test name when specifying a test on the command line? | 16:10 |
| Shrews | maybe that's a stestr thing | 16:11 |
| clarkb | yes that is a test runner thing, has nothing to do with tox. python regexes may have a "this is a literal string" flag | 16:12 |
| clarkb | Shrews: re.escape() will escape a string for you. So you could run that then pass the result to tox/testr | 16:14 |
| Shrews | i just ran stestr directly with -n option | 16:14 |
| Shrews | that seems to work | 16:14 |
| openstackgerrit | Merged zuul/zuul master: encrypt_secret: display the full_url on error https://review.opendev.org/661134 | 16:17 |
| fungi | tobiash: oh, actually i think that concern is actually with zuul's implementation of rsa keys for secrets. the path on disk doesn't include the tenant, even though the api call to get the public key does... so the same project in different tenants will have the same key, but since the projects lists are in the central tenant configuration file and not split between configuration projects, the zuul admin has | 16:18 |
| fungi | control to prevent one tenant from adding a project for another tenant and adding a job to obtain secrets from it | 16:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:26 |
| fungi | i'm also not sure i've had enough caffeine today to work out a reasonable exploit where a config project maintainer could exfiltrate a secret from another project in its tenant anyway | 16:26 |
| fungi | maybe by manipulating a parent job in the config repo, it would be feasible, depending on how the inheriting job in the normal project repo is structured | 16:28 |
| fungi | SpamapS: digging in the pyca/cryptography documentation, it looks like maybe overriding the backend with a subclassed one might provide a means to inject specific seed data into openssl calls https://cryptography.io/en/latest/hazmat/backends/openssl/#os-random-engine | 16:46 |
| fungi | (the generate() method includes the ability to provide a backend parameter) | 16:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:47 |
| *** kmalloc_away is now known as kmalloc | 16:55 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 16:59 |
| *** jpena is now known as jpena|off | 17:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:27 |
| *** sshnaidm is now known as sshnaidm|off | 17:33 | |
| fungi | anybody know whether there's an outstanding change to add project key directory renames to https://opendev.org/openstack-infra/system-config/src/branch/master/playbooks/rename_repos.yaml ? | 17:35 |
| fungi | (for the zuul scheduler's filesystem) | 17:36 |
| fungi | also, do we need to stop the scheduler to do them? | 17:37 |
| fungi | though i guess we likely will regardless | 17:37 |
| corvus | fungi: should be safe to do with scheduler running, but yeah, we usually have it stopped anyway (since gerrit won't be able to merge changes) | 17:37 |
| corvus | fungi: i don't see such a change, though i thought we made one for the migration? | 17:39 |
| fungi | it was just hacked into the migration script instead | 17:45 |
| fungi | due to realizing at the last moment we needed to do that | 17:46 |
| corvus | apparently a not-uncommon occurance :) | 17:47 |
| fungi | heh | 17:52 |
| fungi | but yeah, it was squashed into 653138 | 17:52 |
| clarkb | oh did that not make it into my omnibus change? | 17:52 |
| *** hashar has quit IRC | 17:53 | |
| fungi | oh, and i just realized i meant to ask about it in #openstack-infra, apologies for the noise #zuul | 17:54 |
| fungi | though i suppose it was sort of related to the earlier topic | 17:54 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 17:55 |
| *** tosky__ has joined #zuul | 17:59 | |
| *** tosky has quit IRC | 18:00 | |
| *** tosky__ is now known as tosky | 18:00 | |
| *** electrofelix has quit IRC | 18:03 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 18:04 |
| *** ofosos has quit IRC | 18:08 | |
| Shrews | tobiash: btw, i think ^^ is necessary for the scale-out scheduler | 18:08 |
| tobiash | Shrews: yes, makes sense :) | 18:09 |
| Shrews | can't recall if that's called out in your spec | 18:09 |
| tobiash | nope, good point | 18:09 |
| clarkb | tobiash: I approved the usage stats chagne yseterday btw but didn't check if it ended up merging | 18:16 |
| tobiash | clarkb: thanks, it merged after one recheck due to a test timeout | 18:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 18:28 |
| *** panda|ruck has quit IRC | 18:35 | |
| *** panda has joined #zuul | 18:37 | |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 19:00 |
| mordred | Shrews: see! I *was* being helpful with my random out of the blue suggestion | 19:02 |
| openstackgerrit | David Shrewsbury proposed zuul/zuul master: WIP: Store hold requests in zookeeper https://review.opendev.org/661114 | 19:03 |
| Shrews | "helpful" is a relative term | 19:03 |
| Shrews | :) | 19:03 |
| Shrews | i now have therapy things to attend to. bbl | 19:04 |
| *** themroc has joined #zuul | 19:07 | |
| *** rlandy is now known as rlandy|brb | 19:47 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 19:53 |
| *** themroc has quit IRC | 19:58 | |
| *** rlandy|brb is now known as rlandy | 20:15 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:34 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 20:52 |
| *** pcaruana has quit IRC | 21:23 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 21:27 |
| *** tosky has quit IRC | 21:49 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: WIP: registry test job https://review.opendev.org/661327 | 22:04 |
| *** tjgresha has quit IRC | 22:07 | |
| *** rlandy is now known as rlandy|bbl | 22:22 | |
| *** ianychoi has quit IRC | 22:32 | |
| *** ianychoi has joined #zuul | 22:33 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!