Thursday, 2020-03-12

fungiSpamapS: agreed, digest auth by itself is only robust against sniffing, not traffic manipulation (like mitm)00:00
fungiclient certs issued by an internal ca would be robust against both00:01
clarkbI think tristanC has asserted sasl is useful for them in order to reject new connections with bad auth at connection time00:02
clarkbwithout sasl you can still connect and interact with the server00:02
fungiclient cert auth would also have that property00:03
fungiit's done during ssl/tls handshake00:03
clarkbcan't that still be delayed?00:03
clarkbah00:03
corvusyeah, when i botched my certs earlier, it got rejected with "unknown_ca" real quick00:03
fungiit can be delayed if you support "optional" authentication via a mechanism like starttls00:04
fungifor example this is how tls is negotiated on 25/tcp for opportunistic encryption between smtp peers00:04
fungior for an imap service to offer plaintext and encrypted sessions on the same socket00:05
fungibut if you do it like 443/tcp for https, you don't get past the tls handshake00:05
*** Defolos has quit IRC00:22
fungicircling back around, removing the version cap for python-daemon seems to have solved the tox-py38 failures00:31
fungiso 712489 and 712494 should be ready to go now00:32
*** openstackstatus has joined #zuul00:45
*** ChanServ sets mode: +v openstackstatus00:45
*** erbarr has quit IRC00:55
*** dangtrinhnt has joined #zuul01:16
*** dangtrinhnt has quit IRC01:31
*** dangtrinhnt has joined #zuul01:32
*** dangtrinhnt has quit IRC01:51
*** swest has quit IRC02:15
*** rlandy|bbl is now known as rlandy02:15
*** jamesmcarthur has joined #zuul02:18
*** swest has joined #zuul02:28
*** jamesmcarthur has quit IRC02:56
*** jamesmcarthur has joined #zuul02:58
*** jamesmcarthur has quit IRC03:03
*** bhavikdbavishi has joined #zuul03:03
*** jamesmcarthur has joined #zuul03:06
*** jamesmcarthur has quit IRC03:34
*** jamesmcarthur has joined #zuul03:35
*** jamesmcarthur has quit IRC03:40
*** rlandy has quit IRC03:44
*** jamesmcarthur has joined #zuul03:56
*** tobiash has quit IRC04:16
*** tobiash has joined #zuul04:17
*** jamesmcarthur has quit IRC04:23
*** jamesmcarthur has joined #zuul04:27
*** jamesmcarthur has quit IRC04:32
*** jamesmcarthur has joined #zuul04:33
*** tobiash has quit IRC04:43
*** tobiash has joined #zuul04:46
*** bolg has joined #zuul04:56
*** raukadah is now known as chandankumar05:16
*** zxiiro has quit IRC05:25
*** jamesmcarthur has quit IRC05:30
*** evrardjp has quit IRC05:35
*** evrardjp has joined #zuul05:36
*** reiterative has quit IRC05:40
*** reiterative has joined #zuul05:40
*** saneax has joined #zuul06:27
*** bolg has quit IRC06:50
*** smcginnis has quit IRC07:03
*** dpawlik has joined #zuul07:04
*** smcginnis has joined #zuul07:04
*** Defolos has joined #zuul07:17
openstackgerritAndreas Jaeger proposed zuul/zuul-jobs master: Use a fake zuul_return and an .ansible-lint file  https://review.opendev.org/71254707:30
*** avass has joined #zuul07:48
*** bolg has joined #zuul07:48
*** hashar has joined #zuul07:58
*** jcapitao has joined #zuul08:15
*** tosky has joined #zuul08:18
*** jpena|off is now known as jpena08:45
*** sshnaidm|afk is now known as sshnaidm09:39
openstackgerritFabien Boucher proposed zuul/zuul master: Declare support for Python3.8  https://review.opendev.org/71248009:45
fbofungi: ^ I've duplicated your change for zuul09:46
*** TomConte has joined #zuul09:51
*** avass has quit IRC10:44
*** TomConte has quit IRC10:59
*** jcapitao is now known as jcapitao_afk11:02
*** sshnaidm has quit IRC11:18
*** sshnaidm has joined #zuul11:19
openstackgerritFabien Boucher proposed zuul/zuul master: Declare support for Python3.8  https://review.opendev.org/71248011:20
*** sshnaidm has quit IRC11:22
*** sshnaidm has joined #zuul11:30
*** sshnaidm has quit IRC11:30
*** jpena is now known as jpena|lunch11:49
*** sshnaidm has joined #zuul11:52
*** sshnaidm has quit IRC11:53
*** sshnaidm has joined #zuul11:54
*** rlandy has joined #zuul12:01
*** avass has joined #zuul12:05
openstackgerritAlbin Vass proposed zuul/nodepool master: Enable setting label and instance name separately  https://review.opendev.org/71266612:15
*** avass has quit IRC12:25
*** jpena|lunch is now known as jpena12:32
*** jcapitao_afk is now known as jcapitao12:46
tristanCclarkb: i haven't tried TLS, it was not an option when i first implemented SASL. If it can be used in-place of SASL, then we should drop the initial implementation and replace it with TLS.12:51
*** avass has joined #zuul12:54
tristanCi mean, it seems like using a private ca to issue client cert should prevent anon connection too, if that's the case, i'm not sure why do we bother with sasl. Afterall, that's what we do for gearman, so why not just do the same with zk?13:07
tristanCcorvus: SpamapS: fungi: ^13:08
Shrewsfungi: Since we had the python-daemon cap for a reason, maybe we need to work to figure out which versions matching 2.1.0 and above do not work and have some != for those rather than just uncap?13:12
*** ianychoi has quit IRC13:13
Shrewscorvus: i think being able to support both sasl and tls is very reasonable. because as soon as we decide to not support one or the other, someone is going to request that we do.13:28
openstackgerritAlbin Vass proposed zuul/nodepool master: Enable setting label and instance name separately  https://review.opendev.org/71266613:32
openstackgerritAlbin Vass proposed zuul/nodepool master: Enable setting label and instance name separately  https://review.opendev.org/71266613:33
tristanCShrews: as an operator, i would wait for tls and not bother with sasl (assuming tls can also prevent anon connection). Perhaps we could ask on zuul-discuss if anyone would requires sasl?13:34
tristanCor maybe using the zuul survey to gather zookeeper usage (self hosted, managed service, other?) would be good info?13:45
avassanyone want to take a look at this ^ it's a quite small change :)13:47
*** kmalloc has joined #zuul13:47
*** TomConte has joined #zuul14:20
*** TomConte has quit IRC15:02
*** bolg has quit IRC15:12
*** hashar has quit IRC15:18
*** Goneri has joined #zuul15:18
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support buildkit backend when building docker images  https://review.opendev.org/71271615:35
openstackgerritDaniel Pawlik proposed zuul/zuul-jobs master: Add phoronix-test-suite job  https://review.opendev.org/67908215:39
openstackgerritMonty Taylor proposed zuul/zuul master: Use buildkit to mount instead of copy /output  https://review.opendev.org/71271715:41
fungifbo: why did it need duplicating? was the one i pushed for zuul incorrect?15:44
mordredavass: the change looks fine - but I'm curious as to why that's a desirable thing?15:45
*** rlandy is now known as rlandy|afk15:46
fbofungi: alright I haven't check you proposed one already for Zuul. Let me abandon the patch then.15:48
*** avass has quit IRC15:51
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253115:51
fungiyeah, sorry, i pushed the changes for nodepool and zuul at the same time15:52
fungiand tristanC's suggestion on your version is already met in my version15:52
fbofungi: sorry as well I though you did it only for nodepool w/o checking15:52
fungino need to apologize, i just thought i might have been missing something important there15:53
*** kmalloc has quit IRC15:57
corvusSpamapS: when you have a second, can you give https://review.opendev.org/712666 a look?  that seems good and safe to me, but i want to make sure that hard-coding the name tag wasn't an intentional decision.15:59
*** jamesmcarthur has joined #zuul16:06
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253116:11
*** avass has joined #zuul16:14
Shrewscorvus: last PS added a "with with" to the zookeeper.rst if you find another reason for another PS, but looking pretty good to me16:15
openstackgerritAlbin Vass proposed zuul/nodepool master: Enable setting label and instance name separately  https://review.opendev.org/71266616:15
Shrewsi like that entire doc being there16:17
Shrewscorvus: does the new quorum format really use ";2181" and not ":2181" in the normal case?16:18
*** jamesmcarthur has quit IRC16:20
corvusShrews: yep: https://zookeeper.apache.org/doc/r3.5.2-alpha/zookeeperReconfig.html16:22
corvusi think the semicolon separates the server from the client info16:23
Shrewshrm, how... odd (at least to the eye)16:23
corvus(because serverip:port:port;clientip:port is also valid)16:23
corvusyeah, if anything ever called out for a yaml file, this is it :)16:24
*** jamesmcarthur has joined #zuul16:24
*** dpawlik has quit IRC16:28
*** dpawlik has joined #zuul16:28
*** bolg has joined #zuul16:34
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253116:36
*** hashar has joined #zuul16:37
mordredcorvus: are you sure it shouldn't be TOML?16:39
corvus /kick mordred16:39
* mordred hasn't made a toml crack in months so figures his quota is built up high enough to be able to use one16:39
corvusnope16:39
*** jcapitao is now known as jcapitao_afk16:40
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support buildkit backend when building docker images  https://review.opendev.org/71271616:42
mordredcorvus: in the zuul dockerfile we're adding the stretch-backports repo - but python-base is buster now16:44
mordredcorvus: do we know what target version of bubblewrap and socat we were looking for?16:44
mordredI'm thinking we might be able to drop that section16:45
mordredbuster has bubblewrap 0.3.1-4 and socat 1.7.3.2-216:45
mordredstretch-backports has bubblewrap 0.3.1-4~bpo9+1 and socat 1.7.3.1-2+deb9u116:47
mordredso I'm going to say yes16:47
corvusmordred: that sounds plausible16:48
openstackgerritJames E. Blair proposed zuul/nodepool master: Add ZooKeeper TLS support  https://review.opendev.org/71273316:48
corvusShrews: ^ the other half16:48
corvuswith both of those, i think we can update quick-start to use it16:49
corvusshould be a nice test of containerized depends-on16:49
mordredcorvus: incidentally - debian buster ALSO has yarn and nodejs 10 in it16:49
mordredcorvus: so it's also conceivable we're at the point where we can express nodejs and yarn in bindep as well and not need install-js-tools in the dockerfile16:50
tristanCcorvus: i'm testing TLS with the operator, hopefully we can see it in action soon using two depends-on16:50
mnasershower thoughts: wouldn't it make life easier to put nodepool and zuul in the same codebase, esp with all of these changes that are happening now16:51
mnaser(and also, given that right now we don't have awareness of tenants in nodepool and there might be a whole lot of duplicated code if we were to ever think about that)16:51
corvusmnaser: well, zuul is pretty much designed to avoid needing a monorepo, so it'd be a bit of a disappointment to give up and pack it in on that one.16:52
mnasercorvus: yeah, but maybe we can think about zuul_lib or something to share the common zk code16:53
corvusyes16:53
corvussomeone suggests that just about every day16:53
mnaseroh, i see16:53
mnaserit's me today :)16:53
corvusit's not that much common code as it turns out :)16:53
corvusit's like 40 lines or something16:53
corvus(and those lines are not the lines that are changing)16:54
mnaserah, fair16:54
*** jamesmcarthur has quit IRC16:54
corvusi don't object to moving it into a lib, but i think we should do that after zuulv5, because heavy changes like that would make the zuulv4/v5 work take longer and about 2x as many changes16:56
*** jamesmcarthur has joined #zuul16:56
corvusmost of the upcoming work is going to be only in zuul, so we're already in a situation where we only need to change one repo for most of that16:57
mnaseri see16:57
openstackgerritMonty Taylor proposed zuul/zuul master: Remove stretch-backports from docker build  https://review.opendev.org/71273716:59
openstackgerritMonty Taylor proposed zuul/zuul master: Install javascript tools in docker via bindep  https://review.opendev.org/71273816:59
mordredfungi: ^^17:03
*** sshnaidm is now known as sshnaidm|afk17:07
*** jcapitao_afk is now known as jcapitao17:12
*** jamesmcarthur has quit IRC17:19
*** rlandy|afk is now known as rlandy17:19
*** hashar has quit IRC17:33
clarkbmordred: whats the risk with using experimental dockerfile syntax?17:34
*** evrardjp has quit IRC17:35
*** evrardjp has joined #zuul17:36
clarkbmordred: left a ocmment on https://review.opendev.org/#/c/712737/117:38
*** jamesmcarthur has joined #zuul17:47
*** paladox has quit IRC18:07
*** paladox has joined #zuul18:09
mordredclarkb: sholdn't be much risk these days - it just doesn't work with OOOOLD docker18:13
clarkbmordred: ok, I think we can definitely land the first two in the stack to enable buildkit but then the socat thing is probably worth an update18:14
mordredclarkb: kk. also - I thnk the javascript one is bong and I need to do a little more work there18:14
clarkbfwiw I kinda like being able to install from nodsource so that we use a consistent version of nodejs everywhere18:15
tristanCcorvus: running the zk-ca.sh and adding the ssl configuration to the zoo.cfg doesn't seem to be enough to make the container listen on 2281. Is there JAVA environment to set too?18:15
mordredclarkb: oh yeah? kk. I'm fine keeping that18:15
clarkbmordred: at one point we were using node 6,8 and 10 in different places :/18:15
mordredclarkb: which is not good18:16
tristanCcorvus: here is my current config: http://paste.openstack.org/show/790625/18:18
*** jcapitao has quit IRC18:20
corvustristanC: the main difference i see compared to my test config (i'll paste in a min) is maybe change the server.1 line to: server.1=0.0.0.0:2888:388818:20
corvustristanC: this is my config for the first node of a 3 node cluster: http://paste.openstack.org/show/790626/18:21
corvus(the ssl quorum is commented out just because i wasn't testing that in the most recent iteration)18:22
mordredclarkb: the zuul-jobs update doesn't seem to have actually resulted in the env var being passed - did I just do something stupid?18:23
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275918:24
clarkbmordred: https://review.opendev.org/#/c/712716/2/roles/build-docker-image/tasks/build.yaml that bit?18:25
mordredclarkb: oh - it's because I fixed the env thing before that built18:25
mordredI thnik18:26
mordredmaybe I didn't18:26
mordredclarkb: but yeah - that's the bit18:26
clarkbit looks like it should work to me18:27
corvustristanC: hrm, i changed my config to be the hostname too, and it still seems to work18:28
tristanCcorvus: are you using the docker.io/library/zookeeper image?18:29
corvustristanC: yes18:30
tristanCi had to disable the entrypoint because it tried to write to /conf which is not readonly as it is now provided as a volume18:30
tristanCperhaps it's silently failing because of the missing myid file?18:30
corvustristanC: yeah, that could be a problem; i think it gets upset if that is not there18:30
mordredclarkb: is it possible command doesn't do environment and you have to do shell?18:41
clarkbmordred: oh yup that could be it18:41
mordredclarkb: sigh18:42
clarkbansibles docs have command tasks examples using environment though18:42
clarkbbut it wouldn't surprise me if that was the issue18:42
openstackgerritMonty Taylor proposed zuul/zuul-jobs master: Support buildkit backend when building docker images  https://review.opendev.org/71271618:42
mordredclarkb: let's try it and see18:42
*** jpena is now known as jpena|off18:51
mnaserooou18:51
mnaserinstall-k8s role is currently "broken" because it looks like newer minikube is trying to download a preloaded image of k8s18:51
mnaserand then it fails to extract is cause its compressed with lz4 which is not on the image18:52
mordredmnaser: uhm.18:52
clarkbusing docker?18:52
mnaserso its just taking up 2 minutes for no reason (and slows down the eventual deployment too because its doing the pull twice, technically, i think)18:52
mnaseryes, see: https://a9558da425ae6df3e759-67674e607a46c69293c37e4affbebdc2.ssl.cf1.rackcdn.com/712763/1/check/lodgeit-helm-functional/a675357/job-output.txt18:52
mnaseri will push up a patch which should hopefully fix it18:52
clarkbwhy would docker compress things in a format it can't decompress18:52
mnasermaybe i misunderstood the error18:52
mnaserbut certainly it looks like it downloads something, fails to lz4 decompress, and then tries some other way18:53
mordredwhy wouldn't it just docker pull?18:53
mnaserlooks like this preload thing is new18:54
mnaserhttps://github.com/kubernetes/minikube/pull/672018:54
clarkboh this is out of band of docker18:56
clarkbnice18:56
fungiit knows what's best for you18:57
mordredI don't see anywhere that adds an option "please don't do that"18:57
mnaseryeah i'm looking at the code18:58
mnaserand im not finding that either18:58
mordredsame18:58
mnasermaybe a follow-up pr..18:58
fungiwhy would anyone want to turn off such a helpful feature?18:58
mordredfungi: because we hate freedom and bunnies and awesomeness18:58
clarkbinstalling lz4 seems fine for us fwiw18:58
clarkbits just weird to have a tool that handles all this then bypass it and break things18:58
*** erbarr has joined #zuul18:59
mnaserok let me push up a change18:59
fungiyeah, seems like with all their preferences for bundling stuff, they'd bundle the decompressor for their implementation18:59
mordredsure. it's just an annoying feature given we have caching proxy mirrors for docker and not for "some random object store bucket for a tarball"18:59
mnaseri _really_ hope lz4 is the same package name across different platforms18:59
clarkbmordred: ya that too18:59
mordredso - while I could see this being "better" for some people, it's going to result in more network bandwidth for us18:59
mordredand will be less reliable18:59
*** gmann is now known as gmann_lunch19:00
fungiand i agree, this does make caching things (if you're doing it over and over on different systems in the same environment) a pain19:00
fungithen again, they already designed dockerhub to be uncacheable, right? so not a regression19:00
clarkbmnaser: liblz4-tool on ubuntu I doubt its that on red hat distros :)19:00
avassmordred: You should be able to use command with env, I use it when running go here: https://review.opendev.org/#/c/691114/10/roles/go/tasks/main.yaml19:00
mnaserwell instal-kubernetes seems to be pretty oriented to debian-based distros...19:00
mnaserit uses apt_key/apt_repository/apt so19:01
avassmordred: just saw the update and I was a bit confused why you switched to shell19:01
mordredavass: I'm also confused ... mostly stabbing in the dark about why it didn't seem to pick up the env var19:03
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: install-kubernetes: install lz4 packages  https://review.opendev.org/71277119:04
mnaserclarkb, fungi, mordred: ^19:04
mordredavass: https://zuul.opendev.org/t/zuul/build/fbba2c5409644d0e86cd07bd4ce866ed/console - the failure there is what you get if you don't put DOCKER_BUILDKIT=1 in the env19:05
clarkbmordred: if it fails with shell you could update the zuul-jobs change to hardcode buildkit on then we'll see if its the jinja there19:05
mordredclarkb: good call19:06
tristanCcorvus: finally got an exception: `java.lang.UnsupportedOperationException: SSL isn't supported in NIOServerCnxn`19:10
tristanCusing `docker.io/library/zookeeper                   latest    bbebb888169c   2 days ago     229 MB`19:10
avassmordred, clarkb: the jinja seems to be correct19:12
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275919:13
clarkbavass: I agree19:13
*** jamesmcarthur has quit IRC19:15
mordredyah19:15
mordredclarkb, avass: it's possible I should just give up on this for now - this started as a nerdsniping this morning from looking at something else and remembering that I'd been meaning to see about trimming things down with mounts instead of copies ... but it seems like maybe the fates are not having it and I'm not sure its value is worth our continued effort19:16
openstackgerritMonty Taylor proposed zuul/zuul master: Remove stretch-backports from docker build  https://review.opendev.org/71273719:18
openstackgerritMonty Taylor proposed zuul/zuul master: Be explicit about source of base images  https://review.opendev.org/71277219:18
mordredclarkb: ^^ both of those should still be solid and not related to the buildkit stuff19:21
Shrewsdoes anyone know how to stop the licensing messages I'm seeing from a 'tox -epep8' run on nodepool now?  e.g., http://paste.openstack.org/show/790628/19:21
Shrewsthere are dozens more than what i pasted, so it's quite annoying19:22
mordreduhm19:22
mordredno - but let me go check19:22
*** bhavikdbavishi has quit IRC19:23
openstackgerritMerged zuul/zuul-jobs master: install-kubernetes: install lz4 packages  https://review.opendev.org/71277119:27
corvustristanC: wow, maybe i have a jvmflags setting of "-Dzookeeper.serverCnxnFactory=org.apache.zookeeper.server.NettyServerCnxnFactory" -- i'll look into that after lunch19:28
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: Add options to CLI info command  https://review.opendev.org/71253919:33
openstackgerritDavid Shrewsbury proposed zuul/nodepool master: WIP: Support uploads in 'info' CLI command  https://review.opendev.org/71277519:33
*** jamesmcarthur has joined #zuul19:35
mordredShrews: it's hacking doing it - I don't know what changed to cause this to start being printed out - ....19:36
mordredwhat in the ...19:37
mordredShrews: congratuations! you have just found mordred's afternoon rage-nerdsnipe19:37
Shrews\o/19:37
tristanCcorvus: that flags seems to fix the issue indeed19:38
Shrewsmordred: it's not like you could go watch ACC basketball anyway19:38
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275919:41
tristanCalso, it doesn't seems like any zookeeper operator/helm-char/ansible-roles are able to manage TLS19:44
mordredShrews, clarkb, corvus: remote:   https://review.opendev.org/712778 Remove hacking from requirements <--19:45
clarkbmordred: somehow that is in merge conflict19:45
mordreddib apparently has hacking in real requirments, not just test-requirements.txt.19:45
mordredhah19:45
clarkbmaybe you didn't update dib before committing?19:46
mordredupdated19:46
mordredyeah19:46
clarkbmordred: also maybe flake8 should be in requirements?19:46
mordredclarkb: no, I don't think it should be19:47
mordredI don't think someone installing diskimage-builder needs to install linting tools19:47
mordredthat's _way_ inappropriate19:47
corvustristanC: i just recreated everything without that env var, and it still works19:48
mordredclarkb: I could add a check in dib-lint and print an error message "you need to install flake8 to use dib-lint"19:48
clarkbmordred: ya that might be a good option, just so its clear what changed and why things have failed (if they do)19:49
corvustristanC: so just having it in zoo.cfg should be sufficient19:49
clarkbmordred: we run that linter on project-config iirc so may be able to test it there19:49
tristanCcorvus: oh, i had it commented from the zoo.cfg when i disabled the quorum setting19:50
openstackgerritMonty Taylor proposed zuul/nodepool master: Stop using hacking for anything  https://review.opendev.org/71278119:50
mordredclarkb: ++ will do19:50
mordredShrews: ^^ once we release a new dib with that patch applied, the license lines will disappear19:51
clarkbwhat does it think is wrong with the license?19:51
clarkb(I'm too lazy to run tox -re pep8)19:51
corvustristanC: ah.  i think that setting is really needed for either client or server.19:52
mnaserclarkb: i think this is what Shrews pasted - http://paste.openstack.org/show/790628/19:53
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253119:53
corvustristanC: ^ updated to clarify that19:53
clarkbseems like that is a legit bug in hacking? it shouldn't care about whitespace being ' ' or '\n'19:54
mordredclarkb: updated dib patch19:54
Shrewsmordred: woohoo!19:54
mordredclarkb: sdague decided that only chekcing the first 11 lines was ok19:54
mordredclarkb: apparently some of our files in nodepool have chosen to use 12 lines19:55
mordredclarkb: but what's worse is that it spews the message EVEN THOUGH WE IGNORE H19:55
mordredwhich is completely inappropriate19:55
clarkbya I guess my point is, lets stop running it inappropriately but maybe also we should file a bug or fix hacking?19:56
clarkbI dunno19:56
mordredclarkb: it is a legit bug in hacking ... but I think the first step is to de-hacking nodepool since it doesn't even actually use hacking ... but it turns out removing it from our test-requirements has no effect. also - it means our pin on the version to an old one was being ignored19:56
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275919:57
mordredso the first step is to make it POSSIBLE to stop using hacking in nodepool, which is currently impossible to do19:57
clarkb++19:57
corvusnot for nothin', but the literal text of the "how to apply" appendix of the actual apache license as published by the asf is 12 lines, so i feel like that's a not unreasonable choice.19:57
mordredsorry - I meant 12- https://opendev.org/openstack/hacking/src/branch/master/hacking/checks/comments.py#L137-L17019:59
clarkbya nodepool is doing it in 11 and tripping the error20:00
mordredbut still - in some of our files we're wrapped slightly differently ... and I don't think we should care :)20:00
clarkbbut changing a \n to a ' ' doesn't change the meaning of the license20:00
mordrednope20:00
mordredthis is mostly one of those "maybe let's get ourselves out of the path of needing to chase such things"20:00
corvusyes, i support the dehackification20:01
tristanCcorvus: alright thanks. i also add issue with running the zk-ca.sh script on fedora. it seems like it is loading /etc/pki/tls/openssl.cnf and failing because of a missing new_certs_dir option.20:01
*** gmann_lunch is now known as gmann20:01
tristanChad issues*20:01
corvuswe could bundle an openssl.conf, but it's huge20:03
openstackgerritMonty Taylor proposed zuul/nodepool master: Stop using hacking for anything  https://review.opendev.org/71278120:03
tristanCcorvus: here is what i wrote in the end: https://review.opendev.org/#/c/712759/4/roles/zuul-ensure-zookeeper-tls/tasks/main.yaml20:03
corvustristanC: in k8s, cert-manager may be helpful -- but ideally there would be a zk operator with cert-manager integration20:04
corvustristanC, mordred: we may need to decide soon whether we can rely on an external zk operator, or if we need to implement it in zuul-operator20:05
tristanCcorvus: i haven't find an existing thing that can deploy zookeeper with tls yet20:06
mordredcorvus: maybe, given all the work you and tristanC have been doing to get this right for how we're using it, it might be better to just do it in ours?20:07
corvusmaybe so?  at this point, just having zuul-operator also spin up 3 zk nodes is probably not crazy20:07
mordredyeah. and we could also take a parallel task to find a "good" operator and maybe work with them to add the support we need?20:08
tristanChow about creating a regular ansible role using the zk-ca.sh script, and then use that in the zuul-operator?20:08
mordredmaybe? but maybe it would be better to use cert-manager inside of k8s?20:09
corvustristanC: where does the ca end up in your operator patch?20:09
tristanCcorvus: in the operator work dir20:10
corvustristanC: and that's a persistent volume?20:11
corvus(so we'd only lose the ca and certs if the operator was deleted?)20:11
tristanCthe certs are saved as secrets20:12
corvusright, but the CA is a directory that contains a bunch of files20:12
tristanCwell, in that initial PS, i only saved the file needed by zk and zuul/nodepool20:13
corvusso if you issue a cert for a new host, you'd want to keep that around, otherwise you're going to end up with a different root cert20:13
tristanCright, at the moment the operator doesn't manage more than one zk pod20:13
corvustristanC: if it's easy to keep that around permanently, using the script long-term might be okay, but if we go too much further than that, we'd probably end up re-implementing cert-manager, so it might be worth a look to see if we can use it here.20:14
mordred++20:14
*** Goneri has quit IRC20:18
corvustristanC: in fedora 31 i see a new_certs_dir option20:19
corvustristanC: oh, it's because it sets this:  dir             = /etc/pki/CA20:20
tristanCcorvus: i understand. though i don't know cert-manager, are we going to require that for zuul?20:20
corvustristanC: i don't think zuul has any use for it20:20
tristanCand what about non k8s user, is there other tool we can use instead to setup zk tls?20:21
corvustristanC: i'm unaware of any which is why i wrote one20:21
mordredcorvus: well - unless we decided that it's better to use cert-manager than zk-ca.sh inside of k8s - at which point I'd think we should just require it20:21
mordred(for k8s)20:21
corvusmordred: yes... maybe i'm not following the conversation?20:21
mordredmaybe *I*'m not following the conversation20:22
corvusi thought the question was: "should zuul require cert-manager?" and zuul (the python application) doesn't use certs for anything...20:22
mordredright. I agree20:22
tristanCarg, well, `org.apache.zookeeper.common.X509Exception$SSLContextException: Failed to create KeyManager` in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_36d/712759/4/check/zuul-operator-functional-k8s/36d27e5/docker/k8s_zk_zuul-zk-0_default_5abf63cc-83a7-4b60-a1b5-985777c338ed_0.txt20:23
corvusi think the question of whether zuul-operator should require/use cert-manager is a good, relevant question and is open for discussion, and i don't know the answer.20:23
mordredI thought the question was asking if the zuul k8s operator was going to or should require cert-manager20:23
mordredyeah20:23
mordredI tink it's one of the options we could decide on20:23
corvustristanC: looks like20:23
corvusCaused by: java.io.IOException: Invalid keystore format20:23
corvusis the underlying error20:23
mordredI think the zuul-operator should have one and only one mechanism to manage certs for zk - and right now it seems like either an ansible role using zk-ca and using cert-manager are the two possibilities we've identified20:24
tristanCi guess we should check if cert-manager does work for zk certs.20:24
corvustristanC: i know that cert-manager can make self-signed certs.  what i don't know without more research is whether it can operate as a ca (so all the certs are signed by a single root cert) and how difficult it would be to get those either into a JKS or a pcks8 pem encoded format (with an encrypted key) -- those are the 2 formats zk can handle.20:26
tristanCcorvus: (failure to create KeyManager may be related to an invalid when resulting in a skip task)20:26
corvusthe first thing would be a requirement for us.  the second is probably something we could bridge in zuul-operator if we needed to.  we can fetch a pem cert and do the keytool import like zk-ca.sh is doing.20:26
mordredcorvus: yes to the first: https://cert-manager.io/docs/configuration/ca/20:27
mordredcorvus: experimental support for JKS has been added: https://github.com/jetstack/cert-manager/commit/98bc0d52f9f5c99ec58703b93d72af5b3440864120:28
corvusmordred, tristanC: then i would recommend that we try "porting" zk-ca.sh to zuul-operator as tasks that integrate with cert-manager.20:28
corvusit'd be cool if cert-manager could generate the key for us20:29
mordredyah20:29
corvusmordred: oh, you might be able to use a SelfSigned Issuer to get a key to then give to a CA Issuer20:29
mordredcorvus: oh yeah - good point20:30
corvussee 2nd pgraph of https://cert-manager.io/docs/configuration/selfsigned/20:30
mordredcorvus: there is also an article about doing this in openshift: https://developers.redhat.com/blog/2017/11/22/dynamically-creating-java-keystores-openshift/20:30
corvusthose are all steps in zk-ca.sh -- the first is create a keypair, the second is make a ca from the keypair, etc.  i think it may be pretty straightforward20:31
mordred++20:31
mordredcorvus: apparently in openshift there is a CA already20:31
corvushonestly, at the end of the day, it probably won't be much different than using zk-ca.sh, except that the resulting data are all k8s objects which may be more convenient for k8s users20:32
mordredyeah. may make debugging something more transparent to folks operating in that environment and stuff20:32
mordredand/or friendlier for things like doing a gerrit+zuul install in the same k8s with the gerrit also using the new zookeeper-based HA stuff20:33
mordred(obviously not our first concern here - but if we're using k8s primitives maybe that's a friendly future step)20:34
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275920:34
corvustristanC: fwiw, zk-ca.sh is expecting the default openssl.cnf:  https://github.com/openssl/openssl/blob/master/apps/openssl.cnf#L4520:35
tristanCbut is cert-manager already available, or is this something we now have to worry about installating and maintaining?20:35
corvustristanC: it looks like fedora has changed that; i'm not sure what we should do to better support fedora -- we may just have to bundle our own conf file?20:36
clarkbre k8s CA that will allow any k8s issued cert from that CA to talk to zk if doing client cert auth20:36
clarkb(potentially similar issue to using LE)20:37
*** hashar has joined #zuul20:37
mordredclarkb: yeah - that would be in teh openshift case of using the existing CA20:37
corvusthat might be a good reason to use cert-manager even in openshift20:38
tristanCcorvus: as said on the review, i think most (non-k8s) user are going to use zk-ca to configure their zookeeper. Thus if it doesn't work on some system, we should at least document that, or fix it20:38
mordredclarkb: I thnk for normal k8s cert-manager we'd be creating our own CA as a cert-manager resource20:38
mordredcorvus: yeah20:38
corvustristanC, mordred: i think we expected zuul-operator to interact with other operators.  i think the general approach is to expect someone to install cert-manager before installing zuul-operator.  but as long as zuul-operator has permissions to install cluster-wide objects (part of cert-manager is non-namespaced), i guess it could.20:38
corvusmostly, i've seen instructions like: run this kubectl to apply cert-manager, then run this kubectl to apply zuul-operator20:39
mordredcorvus: yah. I think there are some mechanisms where we can express that we want zuul-operator to install cert-manager - but I think as step one just having the instructions be "install cert-manager then install zuul-operator" should be fine20:39
mordredyeah20:40
mordredthat seems to be perfectly acceptable for folks20:40
corvustristanC: okay, i think we can put a copy of openssl.cnf in zuul/tools and have zk-ca.sh reference $TOOLDIR/openssl.cnf20:41
tristanCcorvus: i left a couple comment about usability issue in PS1020:45
corvustristanC: thanks, i took notes on cert-manager in https://review.opendev.org/71275920:47
tristanCcorvus: yeah, i'm trying to use cert-manager... it's a whole new world -_-20:48
tristanCcorvus: what keytool version are you using in your test?  using the one from java-11-openjdk-11.0.6.10-0.fc30.x86_64 seems to result in an invalid keystore20:52
corvustristanC: one important concept to note is that a ClusterIssuer works across namespaces, but since this is for zuul only, we can use regular Issurers which exist inside just the zuul namespace20:53
corvustristanC: openjdk-11-jre-headless from ubuntu20:54
corvus11.0.6+10-1ubuntu1~18.04.120:54
corvuswow.  those look they should really be the same.20:54
mordredopenjdk version "1.8.0_222" is the jdk that's in the zookeeper image20:55
corvusmordred: i ran zk-ca.sh locally and bind-mounted the results in20:55
mordredcorvus: and it's working for you but not for tristanC?20:55
corvusappears that way20:55
mordredo_O20:56
* mordred returns to being less useful20:56
corvusi may be able to repeat this on a fedora container20:56
clarkbmordred: AJaeger has a good comment on that dib change21:00
corvustristanC: the files i created under fedora work fine21:00
tristanCcorvus: arg, ok. thanks for confirming21:01
corvustristanC: see the comment i just left on 712759 -- that may be the problem21:03
tristanCcorvus: oh good catch, but i think that's only used client side21:05
tristanCi'm trying with a fqdn21:05
corvustristanC: oh yep, that's right.21:05
tristanCcorvus: does nodepool needs to know about EXPORT_PASSWORD=foobar ?21:14
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253121:15
corvustristanC: no, that's a temporary password for the pkcs12 export, which is no longer used after it's imported into the keystore21:16
tristanCthen i don't understand why there is an `Invalid keystore format` when using zk-ca and keytool from the operator-sdk image21:18
corvustristanC: i don't either, but it could be as simple as the file isn't there or has the wrong perms; do you have an environment where you can run it locally and log into the pod and check?21:20
corvus(i believe supplying the wrong password does give a different error, so it's probably not that)21:21
openstackgerritMonty Taylor proposed zuul/nodepool master: Be explicit about base image source  https://review.opendev.org/71279821:25
*** rlandy is now known as rlandy|afk21:32
*** hashar has quit IRC21:40
openstackgerritJames E. Blair proposed zuul/zuul master: Add TLS support for ZooKeeper  https://review.opendev.org/71253121:40
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275921:43
tristanCcorvus: indeed, so the issue was caused by the k8s secret using `stringData`, which doesn't work for jks file21:44
tristanCnow, onto testing cert-manager experimental jks support :)21:45
corvus\o/21:45
mordredwoot!21:45
mordredyay b64encode!21:45
corvusfwiw, i believe we can use a pem-encoded file instead of jks, but zk still wants the key to be encrypted, so i didn't see much of a reason to fully test that code path.  but if anything would be easier that way, we can try it.21:46
corvus(ie, a file with "--BEGIN ENCRYPTED PRIVATE KEY--")21:47
mnaseris it a good time for me to start the process of changing https://zuul-ci.org/docs/zuul-jobs/python-roles.html#rolevar-ensure-tox.tox_prefer_python2 to 'false' by default (im thinking propose a patch changing it, send an email to zuul-discuss, wait 2 weeks and then get it merged?)21:47
mnaseri ask because there's a lot of other changes in flight and in case we wanted to stagger changes21:47
clarkbmnaser: ++21:48
clarkbthat shouldn't really affect opendev. Our tox is already python3 most places and once we switch off preinstalling tox we'll set prefer_python2 to false (if not yet default) to repserve that behavior21:48
openstackgerritMohammed Naser proposed zuul/zuul-jobs master: ensure-tox: use python3 by default  https://review.opendev.org/71280421:50
mnasercorvus: if that seems sane to you, i can also shoot off the email and start the 2 week timer ^21:51
corvusmnaser: sounds like a plan21:57
*** etp has quit IRC22:15
*** etp has joined #zuul22:16
openstackgerritMerged zuul/nodepool master: Be explicit about base image source  https://review.opendev.org/71279822:25
*** jamesmcarthur has quit IRC22:30
openstackgerritTristan Cacqueray proposed zuul/zuul-operator master: Add missing zuul service volumes  https://review.opendev.org/71281122:44
*** rlandy|afk is now known as rlandy22:55
tristanCcorvus: found an issue with the scheduler config, which looks like that when a kazoo client connect without the use_ssl option: https://zuul.opendev.org/t/zuul/build/a4a3fe41171a4133a9a35bdc40836e4e/log/docker/k8s_zk_zuul-zk-0_default_2a009824-2213-4dac-a488-5665a91fd217_0.txt#6122:56
corvustristanC: what does zuul.conf look like?22:58
tristanCcorvus: i mean here: https://review.opendev.org/#/c/712531/13/zuul/cmd/scheduler.py@16222:59
corvustristanC: oh i see23:00
corvustristanC: the argument is set here: https://review.opendev.org/#/c/712531/13/zuul/zk.py23:01
corvusso if it's not being used, we should check the zuul.conf23:01
tristanCsearching for zuul.conf in https://zuul.opendev.org/t/zuul/build/a4a3fe41171a4133a9a35bdc40836e4e/log/docker/k8s_ansible_zuul-operator-5d8c859999-7mflj_default_d3c055c0-5f53-40a3-a6eb-fed418675921_0.txt#87 shows this content: http://paste.openstack.org/show/790652/23:02
tristanCarg, so it maybe something else23:02
*** Defolos has quit IRC23:06
*** Goneri has joined #zuul23:08
tristanCor perhaps the depends on zuul image didn't work, according to some bug report, that `not an SSL/TLS record: 00000` error usually means a non ssl connection attemps23:18
tristanCand running a minimal kazoo client works with these args {'hosts': 'zk:2281', 'use_ssl': True, 'keyfile': '/etc/zookeeper-tls/clientkey.pem', 'certfile': '/etc/zookeeper-tls/client.pem', 'ca': '/etc/zookeeper-tls/cacert.pem'}23:19
corvustristanC: yep, i think i missed zuul-operator in a recent change; 1 sec23:20
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Use explicit provides/requires for container jobs  https://review.opendev.org/71281623:21
*** jamesmcarthur has joined #zuul23:22
mordred++23:22
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Update to dhall lang v14  https://review.opendev.org/71064923:23
openstackgerritJames E. Blair proposed zuul/zuul-operator master: Add TLS configuration to ZooKeeper service  https://review.opendev.org/71275923:23
corvustristanC: ^ rebased your changes on that one; that should fix the depends-on.23:23
tristanCoh right, thanks!23:24
corvustristanC: i've been working on a change to quick-start and have run into "javax.net.ssl.SSLHandshakeException: no cipher suites in common".  i'm stumped there, and am going to stop for today and resume this tomorrow23:25
tristanCcorvus: i haven't seen that one. my latest result is "2020-03-12 23:17:14,865 [myid:1] - INFO  [nioEventLoopGroup-4-1:X509AuthenticationProvider@172] - Authenticated Id 'CN=client,OU=Org,O=Company Name,L=Oakland,ST=California,C=US' for Scheme 'x509'"23:26
openstackgerritJames E. Blair proposed zuul/zuul master: WIP: use ZK TLS in quickstart  https://review.opendev.org/71281723:27
corvustristanC: that looks correct :)23:27
*** jamesmcarthur has quit IRC23:27
corvusthere's the WIP on quickstart23:27
tristanCi'm not sure how zuul-operator upgrade should looks like, iiuc we need to tag and manage upgrade with the OLM or something, but in the end, i think that's great how the zuul-operator can manage things like zookeeper tls transparently for the user.23:29
corvusyeah, maybe we can not worry too much before we tag a 1.0 :)23:34
*** jamesmcarthur has joined #zuul23:37
mordredI agree with both of you23:42
ianwi just changed a bunch of node types in jobs @ https://review.opendev.org/712819 but it seems that zuul hasn't decided to run those jobs23:45
ianwthey all have specific files: matches, but i feel like this is something that should gate ... you could potentially break the job when it commits if it never ran?23:46
*** jamesmcarthur has quit IRC23:49
*** threestrands has joined #zuul23:50
mordredianw: I thought zuul would treat changes to a job as a reason to run the job23:51
ianwmordred: me too, i wonder if this is a corner case that slips through somehow23:51
fungipossible it doesn't consider node changes in that23:55
ianwhttps://opendev.org/zuul/zuul/src/branch/master/zuul/model.py#L2989 updatesJobConfig would be about where i'd expect it; if that returns it overrides the files matcher23:55
tristanCcorvus: mordred: not sure what's going on, but the depends-on doesn't show the same zuul registry tag name, see the references i commented in https://review.opendev.org/#/c/712816/123:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!