| *** jamesmcarthur has joined #zuul | 00:20 | |
| *** saneax has quit IRC | 00:27 | |
| *** jamesmcarthur has quit IRC | 00:48 | |
| *** bhavikdbavishi has joined #zuul | 01:40 | |
| *** sgw has quit IRC | 01:46 | |
| *** jamesmcarthur has joined #zuul | 02:42 | |
| *** spsurya has joined #zuul | 02:52 | |
| *** bhavikdbavishi has quit IRC | 03:03 | |
| *** jamesmcarthur has quit IRC | 03:16 | |
| *** bhavikdbavishi has joined #zuul | 03:44 | |
| *** rfolco|rover has quit IRC | 04:04 | |
| *** bolg has joined #zuul | 04:09 | |
| *** bolg has quit IRC | 04:52 | |
| *** bolg has joined #zuul | 05:01 | |
| *** bstinson has quit IRC | 05:01 | |
| *** bstinson has joined #zuul | 05:15 | |
| *** igordc has joined #zuul | 05:58 | |
| *** igordc has quit IRC | 06:03 | |
| *** tosky has joined #zuul | 07:14 | |
| *** hashar has joined #zuul | 07:14 | |
| *** saneax has joined #zuul | 07:28 | |
| *** jangutter has quit IRC | 07:54 | |
| *** yolanda has joined #zuul | 07:58 | |
| *** jangutter has joined #zuul | 07:59 | |
| *** mhu has joined #zuul | 08:02 | |
| *** avass has joined #zuul | 08:05 | |
| *** mgoddard has joined #zuul | 08:05 | |
| *** pcaruana has joined #zuul | 09:02 | |
| *** yolanda__ has joined #zuul | 09:41 | |
| *** yolanda has quit IRC | 09:43 | |
| *** yolanda__ is now known as yolanda | 10:03 | |
| *** sshnaidm is now known as sshnaidm|afk | 10:11 | |
| *** fsvsbs has quit IRC | 10:13 | |
| *** hashar has quit IRC | 10:27 | |
| *** bolg has quit IRC | 10:31 | |
| *** pcaruana has quit IRC | 10:41 | |
| *** avass has quit IRC | 11:02 | |
| *** sshnaidm|afk is now known as sshnaidm|bbl | 11:36 | |
| openstackgerrit | Michal Pryc proposed zuul/nodepool master: WIP: Implement a Devnest nodepool driver https://review.opendev.org/689474 | 11:46 |
|---|---|---|
| *** hashar has joined #zuul | 11:51 | |
| *** avass has joined #zuul | 12:03 | |
| *** themroc has joined #zuul | 12:10 | |
| *** rlandy has joined #zuul | 12:17 | |
| *** rlandy_ has joined #zuul | 12:17 | |
| *** rlandy has quit IRC | 12:17 | |
| *** rlandy_ is now known as rlandy | 12:17 | |
| *** avass has quit IRC | 12:20 | |
| *** rfolco|rover has joined #zuul | 12:21 | |
| *** jamesmcarthur has joined #zuul | 12:47 | |
| *** pcaruana has joined #zuul | 12:53 | |
| *** sgw has joined #zuul | 13:02 | |
| *** jamesmcarthur has quit IRC | 13:11 | |
| *** jamesmcarthur has joined #zuul | 13:29 | |
| *** jamesmcarthur has quit IRC | 13:31 | |
| *** tosky has quit IRC | 13:38 | |
| *** tosky has joined #zuul | 13:39 | |
| *** sshnaidm|bbl is now known as sshnaidm | 13:57 | |
| *** jamesmcarthur has joined #zuul | 14:04 | |
| *** yolanda has quit IRC | 14:07 | |
| *** bolg has joined #zuul | 14:08 | |
| *** yolanda has joined #zuul | 14:14 | |
| *** swest has quit IRC | 14:15 | |
| *** saneax has quit IRC | 14:49 | |
| *** mattw4 has joined #zuul | 14:53 | |
| *** pcaruana has quit IRC | 14:57 | |
| *** mattw4 has quit IRC | 14:59 | |
| ofosos | Hey, if I want to use an Ansible lookup in the project config, to pass to a job as a variable, does this work in the natural way? | 15:04 |
| *** hashar has quit IRC | 15:04 | |
| ofosos | I.e., using vars: tox_environment: FOO: lookup('bla', 'foo') | 15:04 |
| corvus | ofosos: i think if you {{ }} quote it it will probably work | 15:05 |
| corvus | ofosos: zuul passes it through to ansible unchanged, and you can use {{ }} expressions when defining ansible variables | 15:05 |
| ofosos | corvus: Thanks :) | 15:05 |
| Shrews | SpamapS: i believe the issue that caused https://review.opendev.org/683205 to fail has been corrected if you want to re-apply your +A there to kick it off again | 15:11 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 15:28 |
| corvus | Shrews: any reason not to go ahead and do that? (either by leaving "reverify" or another +W?) | 15:30 |
| Shrews | corvus: nope | 15:31 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 15:32 |
| corvus | done | 15:32 |
| corvus | tobiash: can you re-review https://review.opendev.org/688645 with my comment? | 15:33 |
| corvus | bolg: ^ fyi | 15:33 |
| corvus | tobiash, clarkb, Shrews: i think https://review.opendev.org/690607 is what we want to do next for gerrit checks support | 15:34 |
| tobiash | corvus: thanks, that makes sense | 15:35 |
| corvus | yeah, i was scratching my head over that one :) | 15:35 |
| corvus | it's obviously right, but why? :) | 15:35 |
| clarkb | I see so the problem was it was leaving a vote and commenting together but we only want it leaving votes? | 15:36 |
| clarkb | and I guess inline file comments if present | 15:36 |
| corvus | clarkb: actually, right now we don't even want it to leave a vote | 15:37 |
| corvus | but historically there has been no reason to add a "success" reporter if you didn't want zuul to leave a message on gerrit (if you don't want that, just don't add the reporter) | 15:38 |
| corvus | but with the checks api there is a reason | 15:38 |
| clarkb | I see, we need a success reporter to report to the checks api, but don't want it to also comment | 15:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 15:38 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 15:38 |
| corvus | clarkb: exactly | 15:38 |
| corvus | (teensy pep8 fix ^) | 15:38 |
| *** themroc has quit IRC | 15:40 | |
| clarkb | I know "meth" is short for "method" but it is early and I totally read it differently at first | 15:43 |
| clarkb | corvus: one note inline that may help operators | 15:48 |
| corvus | clarkb: yeah, how about i make that a followup? | 15:49 |
| clarkb | sure | 15:49 |
| *** igordc has joined #zuul | 15:51 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 15:54 |
| *** mattw4 has joined #zuul | 16:01 | |
| ofosos | Hey, what can I do to make Zuul's Ansible find the SSM lookup plugin? | 16:07 |
| fungi | ofosos: something like https://review.opendev.org/662870 | 16:09 |
| fungi | at least up to this point, we've been auditing and then selectively whitelisting plugins, as an extra layer of safety. we've discussed dropping all that and just trusting bubblewrap to protect the executor from the ansible process | 16:11 |
| *** sshnaidm is now known as sshnaidm|afk | 16:22 | |
| ofosos | fungi: is there a list somewhere with the enabled plugins? | 16:24 |
| fungi | ofosos: i'm not finding it documented anywhere, but basically i think it's any .py files listed at https://opendev.org/zuul/zuul/src/branch/master/zuul/ansible/base/lookup which aren't symlinks, plus maybe any not shadowed there at all | 16:28 |
| fungi | but also note that, like with the https://review.opendev.org/662870 change, some functionality of the enabled plugins may be blocked if they were considered unsafe to perform on an executor | 16:28 |
| ofosos | Hmm, interesting there's an ssm and an aws_ssm module | 16:31 |
| fungi | also in some cases where it's just an interaction with some remote rest api, it may be simpler and safer to do that directly | 16:32 |
| fungi | highly dependent on the situation of course | 16:32 |
| ofosos | Ahh, ok, it's aws_ssm | 16:34 |
| ofosos | Now Ansible is complaining that boto3 and botocore are not installed. Can I fix this in the playbook? | 16:34 |
| *** tobiash has quit IRC | 16:35 | |
| clarkb | you can add those as extra depenedencies in the zuul ansible installs | 16:35 |
| fungi | if you're trying to use them directly on the executor, you need to add them to the list of python packages zuul should install into its ansible virtual envs (assuming you have zuul managing the ansible installations on executors) | 16:35 |
| ofosos | I have these variables in my job description, but I would be happy if these ran on the build node... Is the job description executed on the executor? | 16:36 |
| fungi | if the playbook specifies "localhost" as the host then that's the executor | 16:37 |
| *** tobiash has joined #zuul | 16:37 | |
| fungi | at least for ansible playbooks being executed by the executor's ansible | 16:37 |
| ofosos | fungi: it doesn't specify anything, it inherits from the zuul-jobs tox job | 16:38 |
| ofosos | Hmm, so it should run on the node? Then I can add these dependencies to the build node | 16:38 |
| fungi | if it is running on a remote node then yeah, add them as deps for the corresponding testenv in your tox.ini or whatever mechanism you like to get them available in the environments tox creates | 16:39 |
| corvus | ofosos: lookup plugins always run on the node where ansible is running (in this case, the zuul executor) | 16:39 |
| ofosos | corvus: I really would like to run this lookup on the build node, because I can attach privileges to that node, that are non-shared with other nodes. | 16:41 |
| ofosos | How do I do that? | 16:41 |
| fungi | you can run a "nested" ansible | 16:41 |
| *** jamesmcarthur has quit IRC | 16:41 | |
| fungi | have zuul's ansible invoke ansible installed on the build node | 16:41 |
| corvus | ofosos: that's an ansible design decision, they always run locally. so, yeah, a nested ansible (have zuul's ansible run another ansible) is the only way for that to happen | 16:41 |
| fungi | that also means not having to worry about whether specific lookup plugins are allowed/working on the executor's ansible | 16:43 |
| fungi | since the build node's ansible can safely be more full-featured | 16:43 |
| ofosos | I'm trying to set tox_environment ahead of the tox job. Can I somehow do a set_fact and let the tox job read it? Would that work? | 16:44 |
| ofosos | Or: is there some convenience wrapper around a sub-Ansible? | 16:45 |
| *** jamesmcarthur has joined #zuul | 16:45 | |
| clarkb | ofosos: we set it as a job var in the job definition iirc | 16:45 |
| *** hashar has joined #zuul | 16:45 | |
| corvus | ofosos: if you copy the zuul inventory file, you'll get the job vars too | 16:46 |
| clarkb | oh sub ansible | 16:46 |
| ofosos | clarkb: Yup, I would like to override it based on some value that the node obtains from the network. What's the best way to do this? | 16:46 |
| clarkb | so ya setting it like https://opendev.org/openstack/openstack-zuul-jobs/src/branch/master/zuul.d/jobs.yaml#L121 puts it in the inventroy which you can then grab it from | 16:47 |
| ofosos | the *build* node :) | 16:47 |
| ofosos | :( I'll have to experiment with that | 16:47 |
| ofosos | Any sample code/best practices for running sub-Ansibles? | 16:48 |
| ofosos | On the other hand it might be easier to just rip off the tox job and roll my own :/ | 16:49 |
| *** igordc has quit IRC | 16:50 | |
| corvus | ofosos: this playbook runs a nested ansible, but it does so for a very different purpose (it's simulating a non-zuul environment), so much of what you would need would be different i think: https://opendev.org/opendev/system-config/src/branch/master/playbooks/zuul/run-base.yaml | 16:51 |
| ofosos | If we go the nested Ansible route, can the node in some way access all the parent Ansible code? | 16:52 |
| ofosos | I mean, I can duplicate the tox job and put in a local call to `aws ssm get-parameter` and that'll work, without much ado. | 16:53 |
| corvus | ofosos: no, that's all on the executor, you'd be starting from scratch (including installing ansible). i would not recommend this solution. | 16:53 |
| ofosos | corvus: then I'll create a custom tox job. | 16:54 |
| corvus | ofosos: what would the custom tox job do? | 16:54 |
| ofosos | corvus: it would populate the tox_environment variable based on SSM Parameter Store values. | 16:54 |
| corvus | how are you going to get the ssm values? | 16:55 |
| ofosos | Call `aws ssm get-parameter` on the build node | 16:55 |
| corvus | that makes sense | 16:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded https://review.opendev.org/687435 | 17:08 |
| *** hwangbo has quit IRC | 17:19 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/nodepool master: Ensure both kubernetes and openshift token are b64decoded https://review.opendev.org/687435 | 17:31 |
| *** jamesmcarthur has quit IRC | 17:45 | |
| *** jamesmcarthur has joined #zuul | 17:46 | |
| corvus | zuul-quick-start jobs are failing on unable to load the libre2 .so file: https://zuul.opendev.org/t/zuul/build/99081a3b32ac4df397e7fee9c24a86a2/log/container_logs/executor.log | 17:47 |
| *** jamesmcarthur has quit IRC | 17:51 | |
| *** sshnaidm|afk is now known as sshnaidm | 17:51 | |
| corvus | i think it's because the python base image updated from stretch to bester? | 17:52 |
| clarkb | did it become a .so.2 or similar? | 17:52 |
| corvus | i think the package name just changed | 17:52 |
| corvus | i'm testing now | 17:53 |
| *** hashar has quit IRC | 17:54 | |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 17:55 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update libre2 dep for buster https://review.opendev.org/690701 | 17:55 |
| corvus | i think that's what we need ^ | 17:55 |
| *** hashar has joined #zuul | 17:58 | |
| *** spsurya has quit IRC | 17:59 | |
| Shrews | i'm so glad the lib name changes with each debian release | 18:03 |
| Shrews | s/lib/package/ | 18:03 |
| *** hashar is now known as hasharAway | 18:05 | |
| *** noorul has joined #zuul | 18:10 | |
| *** bhavikdbavishi has quit IRC | 18:10 | |
| *** noorul has quit IRC | 18:16 | |
| *** themroc has joined #zuul | 18:16 | |
| *** themroc has quit IRC | 18:20 | |
| *** jamesmcarthur has joined #zuul | 18:21 | |
| *** themroc has joined #zuul | 18:24 | |
| *** igordc has joined #zuul | 18:24 | |
| *** hasharAway is now known as hashar | 18:26 | |
| *** jamesmcarthur has quit IRC | 18:31 | |
| *** jamesmcarthur has joined #zuul | 18:32 | |
| *** igordc has quit IRC | 18:32 | |
| *** pcaruana has joined #zuul | 18:38 | |
| *** jamesmcarthur has quit IRC | 18:39 | |
| *** jamesmcarthur has joined #zuul | 18:39 | |
| *** pcaruana has quit IRC | 18:44 | |
| Shrews | corvus: well, i don't see any libre2 error in the executor now, but there is now an invalid ssh key error in the scheduler: https://zuul.opendev.org/t/zuul/build/aaf2704067374644949d3196fd9c0001/log/container_logs/scheduler.log#122 | 18:52 |
| Shrews | how odd | 18:53 |
| corvus | wow this just keeps on giving | 18:53 |
| corvus | (we may need to rethink how we depend on opendevorg/python-base) | 18:53 |
| corvus | like maybe, we need to tag much more often and intentionally bump tags | 18:53 |
| clarkb | or we can pin it back to 3.6 then roll up to 3.7 in a measured fashion? | 18:54 |
| corvus | there's a possibility a change in upstream gerrit broke this... | 18:57 |
| corvus | https://zuul.opendev.org/t/zuul/build/aaf2704067374644949d3196fd9c0001/log/container_logs/gerrit.log#131 | 18:57 |
| corvus | it's always complained about not being able to send an email, but maybe that's now a fatal error | 18:57 |
| corvus | if that's the case, we probably just need to properly configure "no email server" | 18:58 |
| corvus | (though, ideally, upstream gerrit's containers would work with no configuration) | 18:59 |
| corvus | git config -f /var/gerrit/etc/gerrit.config sendemail.enable false && \ | 19:00 |
| corvus | huh, that's there | 19:00 |
| corvus | also, gerrit:latest hasn't updated in 4 months | 19:02 |
| Shrews | is it possible that exception has always been there? | 19:02 |
| corvus | Shrews: yes, some form of it certainly always has | 19:02 |
| Shrews | https://zuul.opendev.org/t/zuul/build/1c04a113f7f14621b482a3136dd7ca0d/log/container_logs/gerrit.log#131 | 19:02 |
| corvus | i was wondering if it became more fatal recently, but that's looking unlikely | 19:02 |
| Shrews | that's from the 18th | 19:02 |
| corvus | cool, that looks the same | 19:03 |
| corvus | looking closer, it's an internal zuul/paramiko error | 19:05 |
| corvus | so gerrit isn't involved at all | 19:05 |
| Shrews | newer paramiko getting the key format wrong? | 19:05 |
| corvus | it may be more likely that this is indeed a python image change issue -- maybe we're missing some crypto lib | 19:05 |
| corvus | okay, i reproduced it manually | 19:11 |
| corvus | the error is from the ed25519key.py file, but it should be an rsa key | 19:13 |
| corvus | the private key file says -----BEGIN OPENSSH PRIVATE KEY----- | 19:16 |
| corvus | instead of -----BEGIN RSA PRIVATE KEY----- | 19:16 |
| corvus | so it's probably a different version of ssh-keygen that's causing this | 19:16 |
| *** jamesmcarthur has quit IRC | 19:17 | |
| *** jamesmcarthur has joined #zuul | 19:18 | |
| Shrews | ooh, i guess right. i get a cookie | 19:20 |
| Shrews | i could've swore we saw this in nodepool in the past | 19:21 |
| *** jamesmcarthur has quit IRC | 19:22 | |
| Shrews | maybe this? https://review.opendev.org/367381 | 19:24 |
| Shrews | but certainly doesn't explain current breakage | 19:24 |
| openstackgerrit | Clark Boylan proposed zuul/zuul-jobs master: Configure the npm mirror https://review.opendev.org/690718 | 19:24 |
| clarkb | something like ^ maybe? The potential problem with that is we assume the pypi mirror exists if mirror_fqdn is set but do we also assume an npm mirror is there at that path? | 19:25 |
| corvus | i'm not coming up with a working test case on my python:slim test | 19:25 |
| clarkb | I know the richer mirror config role is intended to address those concerns so maybe we wait for that? | 19:25 |
| corvus | clarkb: yeah, i'd rather spend effort on the new mirror | 19:25 |
| corvus | clarkb: i don't think using the new mirror config for npm would be blocked on anything, so could just go straight to that | 19:26 |
| clarkb | we'd need to add the ne wmirror config to jobs and switch over to it right? | 19:26 |
| clarkb | or maybe we can have it run alongside the old mirror config for a bit? | 19:27 |
| corvus | should be side-by-side compatible | 19:27 |
| corvus | i need to get lunch | 19:27 |
| corvus | if anyone wants to keep working on the ssh thing, you can repro with this: http://paste.openstack.org/show/785532/ | 19:28 |
| clarkb | the manpage for ssh-keygen says that -m PEM is the legacy format | 19:30 |
| clarkb | is it possible that paramiko updated and dropped support for hat format and requires the RFC4716 format? | 19:30 |
| clarkb | rereading scrollback I don't see a link to how it is failing? | 19:35 |
| clarkb | no recent paramiko releaes though | 19:35 |
| Shrews | gah, what package contains ssh-keygen? | 19:36 |
| clarkb | openssh | 19:36 |
| Shrews | apt-get doesn't like that on python:slim | 19:36 |
| clarkb | might be -server or -client | 19:37 |
| clarkb | but its the openssh suite | 19:37 |
| Shrews | ah | 19:37 |
| clarkb | looking at the scheduler logs I don't see an error? | 19:38 |
| Shrews | clarkb: https://zuul.opendev.org/t/zuul/build/aaf2704067374644949d3196fd9c0001/log/container_logs/scheduler.log#122 | 19:38 |
| clarkb | oh the other url must've been just to show this happened before in the gerrit log? doesn't show up in ca0d's scheduler.log | 19:39 |
| Shrews | hrm, not getting that error with that example from corvus | 19:41 |
| Shrews | clarkb: the ca0d log was showing we still had the gerrit exception before today | 19:42 |
| Shrews | but the problem is in zuul | 19:42 |
| clarkb | ya | 19:42 |
| Shrews | but i cannot recreate it | 19:43 |
| clarkb | I think https://github.com/paramiko/paramiko/issues/1226 is rleated | 19:43 |
| clarkb | previously the key type (RSA) was in the header but now it isn't | 19:43 |
| *** gouthamr has quit IRC | 19:43 | |
| clarkb | from that it seems the issue is specific to non rsa keys | 19:45 |
| clarkb | and adding -t rsa to the keygen would fix | 19:45 |
| clarkb | Shrews: can you check what type of key keygen generated for you? | 19:45 |
| Shrews | rsa | 19:45 |
| *** gouthamr has joined #zuul | 19:45 | |
| clarkb | ya so rsa would work according to that bug | 19:47 |
| *** igordc has joined #zuul | 19:47 | |
| clarkb | can you try ed25519? | 19:47 |
| clarkb | but ya I think the fix here is to use rasa | 19:48 |
| clarkb | *rsa | 19:48 |
| *** jamesmcarthur has joined #zuul | 19:48 | |
| Shrews | actually, it's possible my container is having issues connecting to my chosen ssh server | 19:53 |
| Shrews | so i may not be getting far enough to validate the test | 19:54 |
| *** gouthamr_ has joined #zuul | 19:54 | |
| Shrews | ok, with ed25519, i get "Authentication failed" (as expected). with RSA, i get "not a valid OPENSSH private key file" | 19:59 |
| Shrews | have to step a way for a few moments | 19:59 |
| *** hashar has quit IRC | 20:01 | |
| mordred | corvus: there is a python:3.7-slim-stretch image - we could pin opendevorg/python-base to stretch as well as 3.7 | 20:07 |
| corvus | clarkb, Shrews, mordred: i agree with Shrews's result -- should we change to ed25519 keys in quickstart? | 20:17 |
| corvus | (it's still really weird and disturbing that rsa doesn't work) | 20:18 |
| clarkb | corvus: ya especially since the bugs I found for paramiko imply that rsa should be the one that works | 20:20 |
| clarkb | I think the problem here is the s/rsa/openssh/ in the header though | 20:20 |
| clarkb | what does an ed25519 key look like with its headers? | 20:20 |
| *** jamesmcarthur has quit IRC | 20:20 | |
| corvus | clarkb: well the "-m PEM" should take care of that, right? | 20:20 |
| corvus | -----BEGIN OPENSSH PRIVATE KEY----- | 20:21 |
| corvus | that's ed255 | 20:21 |
| clarkb | maybe paramiko only expects the newer keytypes to have that type of header? | 20:21 |
| clarkb | and when combined with rsa it fails? | 20:21 |
| *** jamesmcarthur has joined #zuul | 20:21 | |
| corvus | clarkb: but "-t rsa -m PEM" fails too | 20:21 |
| clarkb | and that has the RSA PRIVATE KEY header? | 20:22 |
| corvus | that produces: -----BEGIN RSA PRIVATE KEY----- | 20:22 |
| Shrews | i'm still unclear on what is at fault: paramiko or ssh-keygen? | 20:22 |
| clarkb | weird | 20:22 |
| corvus | and paramiko.ssh_exception.SSHException: not a valid OPENSSH private key file | 20:22 |
| corvus | Shrews: so am i | 20:22 |
| mordred | didn't clarkb find something that said -m PEM was considered legacy? | 20:22 |
| clarkb | ya the PEM format is legacy | 20:22 |
| corvus | mordred: yeah, but it's the format we're currently using | 20:23 |
| clarkb | but paramiko bugs I found implied it wants that BEGIN RSA PRIVATE KEY header | 20:23 |
| mordred | my mind is just blown by PEM being legacy at all | 20:23 |
| *** gouthamr has quit IRC | 20:23 | |
| *** jamesmcarthur has quit IRC | 20:24 | |
| Shrews | without -m PEM, I still get: -----BEGIN OPENSSH PRIVATE KEY----- | 20:25 |
| Shrews | fyi | 20:25 |
| *** jamesmcarthur has joined #zuul | 20:26 | |
| corvus | yep, an "OPENSSH" key can be either rsa or ed255 | 20:26 |
| clarkb | and the bugs I found implied paramiko does not accept rsa in ^ that format | 20:26 |
| clarkb | I'm surprised it also fails with the other format | 20:26 |
| clarkb | perhaps someting to do with the encoding itself? | 20:27 |
| Shrews | so we can either restrict paramiko version, or just switch (and document) to ed255. I would think we would need to explicitly document it because someone else is definitely going to run into this | 20:28 |
| corvus | okay, i've tried on 3.7-slim and 3.7-slim-buster, and "-t rsa" fails on both | 20:28 |
| corvus | different errors, because one makes a "RSA PRIVATE" and the other makes "OPENSSH PRIVATE" key, but failures both ways | 20:29 |
| Shrews | well that's fun. what a wonderful rube goldberg machine you've discovered, corvus | 20:30 |
| Shrews | so many moving parts :) | 20:30 |
| corvus | i think that means we've identified the behavior change in the ssh-keygen encoding, but still more bisecting we can do to find out what caused paramiko to be unable to read either one | 20:30 |
| corvus | Shrews: ya | 20:30 |
| fungi | able to confirm on a debian/sid machine (my workstation) with source-built python 3.7.5, paramiko 2.6.0 and debian-packaged openssh-client 1:8.1p1-1 | 20:32 |
| clarkb | https://github.com/paramiko/paramiko/issues/1015#issuecomment-315130862 is apparently the way to fix this in paramiko | 20:33 |
| corvus | clarkb: i don't understand | 20:34 |
| *** jamesmcarthur has quit IRC | 20:35 | |
| fungi | and yeah, repeating the test with python 3.6.9 yields the same results | 20:35 |
| clarkb | corvus: aiui paramiko has its own pem file handling. The bug (and that specific comment there) says that paramiko should replace its home grown tooling for this with cryptography's tooling for this | 20:36 |
| clarkb | but this hasn't been done if I am reading the bug correctly | 20:36 |
| *** jamesmcarthur has joined #zuul | 20:36 | |
| corvus | clarkb: okay, but that doesn't really help me understand what has changed to cause something which did work to stop | 20:37 |
| clarkb | correct | 20:37 |
| clarkb | I'm just pointing out that is what paramiko has said will fix the problem. Unfortuantely there isn't a ton of data in there on what the problem is in the first place | 20:37 |
| clarkb | https://github.com/ansible/distro-test-containers/commit/59e2800df8cd7e32e68c67418e67af7ce8165b8c may be a clue | 20:38 |
| clarkb | does debian have openssh 8? | 20:38 |
| corvus | all the 3.7 images i'm using have 7.x (7.4 or 7.9) | 20:40 |
| clarkb | I wonder if they backported the pkcs8 pem switch or made it default if it waas already an option? | 20:40 |
| clarkb | is there an easy way to check what pem format it is? | 20:40 |
| fungi | 1:7.9p1-10+deb10u1 is the openssh-client package version in debian/stable (buster) | 20:41 |
| clarkb | my ssh keygen manpage says I have to set pm to PKCS8 to get that format | 20:41 |
| clarkb | but the ansible change implies some distros may not be doing that | 20:41 |
| corvus | i stuck an old rsa key into each of my test containers and they all work | 20:43 |
| fungi | that seems like a reasonable workaround, as long as the key is not valuable (obviously) | 20:44 |
| fungi | but only for testing | 20:44 |
| corvus | okay, i think i had a testing error earlier and one of my tests was invalid | 20:44 |
| corvus | a newly generated rsa key on debian 9 (which one is that?) does work | 20:45 |
| fungi | debian 9 is oldstable (stretch) | 20:45 |
| mordred | 9 is stretch | 20:45 |
| fungi | debian 10 is stable (buster) | 20:45 |
| corvus | okay, so if we add "-t rsa", i think it should work (because opendev/python-base is stretch) | 20:45 |
| corvus | oh wait no it's not | 20:46 |
| corvus | it's buster, but py3.7 | 20:46 |
| mordred | opendevorg/python-base is buster | 20:46 |
| mordred | yeah | 20:46 |
| mordred | we could pin it back to stretch though | 20:46 |
| corvus | so yeah, if we pin python-base to 3.7-slim-stretch and add "-t rsa" that will work | 20:46 |
| mordred | I've got a pin patch ready to go - I'll push it up | 20:46 |
| corvus | and that's basically what we had before last week, so that at least seems internally consistent. | 20:46 |
| mordred | yah | 20:47 |
| corvus | let me retry some -m PEM tests and see if there are other working configs | 20:47 |
| mordred | remote: https://review.opendev.org/690742 Pin base-python back to stretch | 20:47 |
| corvus | https://etherpad.openstack.org/p/vd3IZi1Vlq | 20:48 |
| fungi | and yeah, -t RSA (with any of a variety of -m values or the default) doesn't seem to solve this with keys generated by newer openssh | 20:48 |
| corvus | the testing error is that you must make sure the public key is actually present on the remote machine | 20:50 |
| corvus | because there's some really weird interaction with failed logins showing up as bad keys | 20:50 |
| clarkb | huh | 20:52 |
| *** jamesmcarthur has quit IRC | 20:52 | |
| *** jamesmcarthur has joined #zuul | 20:53 | |
| fungi | also making sure the pubkey, if present, actually corresponds to the privkey because for some reason we've seen paramiko get really weird when they don't correspond (even though the pubkey shouldn't even be needed for this purpose) | 20:53 |
| corvus | hey i got a success on buster | 20:53 |
| mordred | corvus: \o/ | 20:53 |
| fungi | just out of curiosity i repeated a variety of the failing scenarios with the pubkey deleted, but that didn't solve it | 20:53 |
| corvus | https://etherpad.openstack.org/p/vd3IZi1Vlq is current with the revised test method | 20:54 |
| corvus | how about instead of pinning, we go ahead and add "-t rsa -m PEM" since that should work universally? | 20:54 |
| fungi | i'm good with that | 20:55 |
| clarkb | ++ to that since it works universally and that lines up with what my paramiko bugs said works | 20:55 |
| fungi | and then we can drop the workaround when paramiko eventually grows support for other key formats | 20:55 |
| mordred | ++ | 20:56 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Update install for buster https://review.opendev.org/690701 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 21:03 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 21:03 |
| corvus | clarkb, mordred, fungi, Shrews: whew! i think 690701 should cover it | 21:03 |
| * corvus cleans up test server | 21:03 | |
| *** jamesmcarthur has quit IRC | 21:05 | |
| *** jamesmcarthur has joined #zuul | 21:08 | |
| *** themroc has quit IRC | 21:10 | |
| *** rfolco|rover has quit IRC | 21:31 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Test buildset registry with k8s and docker https://review.opendev.org/689280 | 21:45 |
| openstackgerrit | Merged zuul/zuul master: Update install for buster https://review.opendev.org/690701 | 21:57 |
| at_work | One comment -- the quickstart guide is partially broken with Gerrit 3.0.x on Centos 7.7, a newer git is required. | 22:11 |
| at_work | Now my question, is there anyway to use zuul without using webhooks? | 22:12 |
| clarkb | at_work: if using zuul with gerrit the ssh event stream works great for triggering jobs | 22:12 |
| at_work | The github application looks like it requires a webhook as well. | 22:12 |
| clarkb | if you want to use zuul with say github I believe the only way to get the necessary event data from github to zuul is via the webhooks | 22:13 |
| clarkb | re git and centos 7 what requires newer git? (there is a lot of git used and understanding the specific need may help us address that properly | 22:13 |
| at_work | We are actually a bitbucket shop, but I was going to use github for learning. Corporate will never open an incoming webhook, we can't even get that with our bitbucket server instance. | 22:14 |
| openstackgerrit | Merged zuul/zuul master: Missing labels is a subset of allow_needs https://review.opendev.org/688645 | 22:14 |
| clarkb | corvus: mordred fungi ^ I guess I wasn't crazy about my concern that gerrit seems to be pushing that direction impacting corporate firewall rules | 22:15 |
| clarkb | at_work: unfortunately I don't think there is another way to get that event data. Zuul could potentially poll I guess but the driver doesn't do that currently | 22:16 |
| mordred | yeah. corporate firewalls are indeed a thing | 22:16 |
| at_work | The githook in gerrit 3.0 uses new git features. I'll find the offending git command | 22:16 |
| clarkb | if you use gerrit it does use the ssh event stream by default | 22:17 |
| at_work | everything is about corporate security theater. | 22:17 |
| clarkb | and the bitbucket driver may actually poll too (does it poll? I seem to recall someone saying it does), but github does not | 22:17 |
| at_work | No gerrit here either, I was just following the demo. | 22:17 |
| mordred | yeah - I think the bitbucket driver is actually polling - adding webhook support is a TDL item | 22:18 |
| mordred | iirc | 22:18 |
| mordred | so - you know - maybe while adding webhook support to bitbucket - we should keep in mind that it might be a good idea to keep the polling version - and it might be good to add a polling version to the github driver | 22:19 |
| clarkb | re the git on centos issue we run gerrit in the quickstart in a container which is based on debian | 22:19 |
| clarkb | oh you mean the commit hook maybe? | 22:19 |
| clarkb | (I thought the server side hooks initially, but ya commit hook on the client side would do it) | 22:19 |
| at_work | commit-msg - hook | 22:19 |
| clarkb | got it | 22:19 |
| at_work | uses interpret-trailers | 22:19 |
| at_work | not available in centos git 1.8.xwhat_ever_old_version_it_is | 22:20 |
| mordred | yay old versions of things! | 22:21 |
| clarkb | I wonder if wecan convince gerrit to use the older commit hok | 22:22 |
| at_work | I was one of those bitbucket pests at ansiblefests. Is there a place to find some information about the bitbucket connector? | 22:23 |
| clarkb | at_work: https://review.opendev.org/#/c/657837/23 is the current base change for implementing it. ofosos and noorul have been doing much of the work and testing | 22:24 |
| *** jamesmcarthur has quit IRC | 22:24 | |
| at_work | I might also be able to convince someone to give me a playground on a bitbucket server instance. | 22:25 |
| clarkb | I think ofosos is using it in production (not sure which commit though) but noorul has been fiding rough edges that need fixing as it is tested | 22:25 |
| at_work | Thank you clarkb, I'll take a look. | 22:25 |
| fungi | at_work: i think i remember you coming to the zuul community booth at ansiblefest. welcome! | 22:26 |
| fungi | guessing you were the fellow i spoke with at length who was interested in possibly helping polish the bitbucket driver | 22:27 |
| at_work | hmmm, I usually limit my harassment to the breakout sessions. | 22:27 |
| fungi | ahh, then there may be several of you ;) | 22:27 |
| fungi | that bodes well | 22:27 |
| corvus | yeah, that'll come in handy | 22:27 |
| corvus | i think the bitbucket driver needs a rebase, and a few changes as suggested in review comments | 22:28 |
| corvus | once that happens, it's probably pretty close to mergable | 22:29 |
| corvus | and i agree -- supporting polling + webhooks might be desirable for most drivers | 22:30 |
| corvus | (or, of course, ssh stream in gerrit's case) | 22:30 |
| mordred | yeah | 22:30 |
| fungi | granted, in opendev we saw plenty of folks who couldn't connect to our gerrit ssh api because of draconian corporate firewalls blocking egress to "unknown" tcp ports | 22:31 |
| mordred | corvus: checks plugin + polling for gerrit might be a valuable combo for some cases | 22:31 |
| mordred | fungi: the egress blocking is so bonkers | 22:31 |
| clarkb | fungi: ya but at least you can often make a case that opening it for that is safe | 22:31 |
| clarkb | fungi: wheras opening "http into $internal network" is often not | 22:31 |
| mordred | like - I understand (although disagree with) the ingress blocking - but egress blocking is just insane | 22:31 |
| fungi | of you don't recognize the port number, it's probably hax0rz | 22:32 |
| at_work | Unfortunately, my time on this will have to move to dark work (undocumented), I doubt I can get real time allocated to work on this zuul for until I can demonstrate "value". | 22:32 |
| mordred | at_work: the ever fun "demonstrate value" ... I feel you | 22:32 |
| fungi | even just having more folks around to test and review the proposed driver helps | 22:32 |
| at_work | We are in a hackathon week, after that is it back to the over estimating agile cards to get some (hopefully) free time to do dark work. | 22:33 |
| openstackgerrit | Merged zuul/zuul master: Add 'comment' option to Gerrit reporter https://review.opendev.org/690607 | 22:34 |
| clarkb | corvus: we'll need to restart zuul for ^ then reenable the gerrit checks api pipeline? | 22:36 |
| at_work | I know there is tremendous value locked up in zuul and ansible, exposing it and putting it work is my dream. | 22:36 |
| at_work | clarkb, Might I message you and ask a question not suitable for public consumption? | 22:37 |
| clarkb | sure | 22:37 |
| corvus | clarkb: yep; i'm fading today and can do that tomorrow | 22:37 |
| at_work | thanks | 22:38 |
| corvus | (but if someone else wants to, great) | 22:38 |
| *** jamesmcarthur has joined #zuul | 22:38 | |
| corvus | at_work: if any of the material on zuul-ci.org helps with making the case for zuul, let us know -- also, if you can think of something we don't have there but can add to help you make the case, let us know too :) | 22:38 |
| fungi | yeah, we're always on the lookout for things we should be saying there but aren't yet | 22:40 |
| *** bolg has quit IRC | 22:45 | |
| *** bolg_ has joined #zuul | 22:45 | |
| *** armstrongs has joined #zuul | 22:56 | |
| openstackgerrit | Merged zuul/zuul master: Log an error on gerrit checks misconfiguration https://review.opendev.org/690616 | 22:58 |
| *** tosky has quit IRC | 22:59 | |
| *** armstrongs has quit IRC | 23:06 | |
| *** jamesmcarthur has quit IRC | 23:08 | |
| *** jamesmcarthur has joined #zuul | 23:10 | |
| *** jamesmcarthur has quit IRC | 23:14 | |
| openstackgerrit | Merged zuul/zuul master: Remove deprecated "checks_api" syntax https://review.opendev.org/690609 | 23:18 |
| *** igordc has quit IRC | 23:23 | |
| *** sgw has quit IRC | 23:32 | |
| at_work | clarkb, et al, another thing to consider when considering the enterprise of firewalls and other security theater, are tokens and https auth based cloning. | 23:35 |
| at_work | s/cloning/clone, poll, push, etc. operations. | 23:36 |
| *** mattw4 has quit IRC | 23:38 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!