Friday, 2018-08-03

*** hongbin has joined #openstack-kuryr01:44
*** hongbin has quit IRC03:52
*** pmannidi has joined #openstack-kuryr05:13
*** janki has joined #openstack-kuryr05:44
*** pmannidi has quit IRC07:16
celebdorMorning07:25
dmelladomorning celebdor07:39
celebdordmellado: this patch is meant for you I think https://review.openstack.org/#/c/588222/07:43
dmelladolemme see07:43
dmelladohmm it timed out on the multinode xD07:45
dmelladodulek: did you check the cause of it?07:45
dmelladohttp://logs.openstack.org/22/588222/3/check/kuryr-kubernetes-tempest-multinode-octavia/8f72d78/compute1/logs/_.localrc_auto.txt07:45
dmelladokubelet was added there just right, but let's see07:45
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Namespace svc isolation tempest coverage  https://review.openstack.org/58777807:46
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-tempest-plugin master: Fix delete namespace resources function  https://review.openstack.org/58846307:46
celebdorltomasbo: I don't see us using sudo on the other places07:46
dmelladobtw celebdor, did you handle your yesterday's patch07:46
dmelladoI recall I saw some user permission issues07:46
dmelladoon docker07:47
dmelladobut didn't get back to the patch today07:47
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers  https://review.openstack.org/58068007:50
celebdordmellado: which?07:53
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add namespace isolation for services  https://review.openstack.org/58142107:53
celebdorthe devstack kuryr/demo thing?07:54
celebdorYes, it works already07:54
dmelladoyep07:54
celebdorand it is merged07:54
dmelladoawesome, makes my life even easier xD07:54
openstackgerritMerged openstack/kuryr-kubernetes master: fix a typo  https://review.openstack.org/56210807:54
dulekdmellado: Not yet, yesterday I've had issues with api_root not being set on the node.07:58
dulekdmellado: Let's see what happened this time!07:58
dmelladodulek: I left a comment there, I saw some SSL issues on the compute's kubelet07:58
dulekdmellado: Damn…07:59
dmelladodulek: reenable tls-proxy and let's see if it makes a differnece07:59
dmelladodifference07:59
dulekdmellado: Nah, it's not tls-proxy. It's kubernetes certs.07:59
dmelladoyeah, that was my 2nd thought07:59
dulekdmellado: I assumed this magically worked, but it was a while ago and we were connecting through magic localhost:8080 which is unauthorized and plain HTTP.08:00
dmelladoheh, which of course won't apply when multinode08:00
dulekdmellado: The issue is - if tls-proxy doesn't work in multinode due to certs synchronization, then I don't even know where to look for example…08:01
dulekI'll dig this.08:01
dmelladodulek: in any case it's been a while since I tried the tls-proxy thing in multinode tbh08:01
dmelladothere was an issue on orchestrate-devstack which prevented for it to work but tbh I'm not sure about its current state08:01
celebdorof course it is k8s certs08:02
celebdorthere's a reason I didn't do multinode08:02
celebdor:P08:02
celebdoryou need to generate a kubeconfig with certs for the other nodes08:02
dmelladocelebdor: you like botching up stuff xD08:02
celebdorwith openshift it is easy08:02
celebdorand my patch adds support for it08:03
celebdor(the one for the registry)08:03
dmelladoI'm pretty sure you'd generate a random number like this08:03
dmelladohttps://imgs.xkcd.com/comics/random_number.png08:03
dulekcelebdor: Oh come one, why don't we just hardcode a cert in DevStack? xD08:04
celebdordulek: I won't08:05
dmelladodulek: in fact08:15
dmelladolet's hardcode everything08:15
* dmellado thinking about the tests08:15
dmelladoreturn True08:15
dmelladojust like VW!08:15
ltomasbodulek, celebdor: besides the tempest tests/gates, this is the last remaining patch for the namespace isolation functionality: https://review.openstack.org/#/c/58142108:16
celebdorthanks ltomasbo08:16
celebdordulek: what surprises me if the scc addition is necessary08:16
ltomasbocelebdor, note it also fix some route when deploying with devstack so that host->pod connectivity is properly allowed08:16
celebdoris that the yaml I create the registry from is generated by oc adm08:17
celebdorone would think that it should just work08:17
dulekdmellado: Hey, VW did more like "if in_test_mode: result /= result". ;)08:20
ltomasbocelebdor, I'm giving it a try to your buildah patch, without much success08:21
ltomasbocp: cannot stat ‘/var/lib/containers/storage/overlay/9e10b3e7e28e3ea6c49ec3057428e550d4324a4e75d4ca7dc4468969f469505f/merged/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud’: No such file or directory08:22
celebdorltomasbo: distro?08:24
ltomasboCentOS Linux release 7.5.1804 (Core)08:24
celebdorinteresting08:25
celebdorlet me see08:25
celebdorltomasbo: run a find08:26
celebdorto see where the SIG-Cloud key was left08:26
ltomasboI think it is related to this: http://paste.openstack.org/show/727210/08:27
*** garyloug_ has joined #openstack-kuryr08:27
ltomasbothough find shows:08:28
*** garyloug has joined #openstack-kuryr08:28
ltomasbo -> /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud08:28
ltomasbo-> /var/lib/docker/overlay2/ea646937ed2a831a7c1ddad0413e52c508f07cf5af73b23fce359f651a093601/merged/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud08:28
ltomasbo-> /var/lib/docker/overlay2/589716330a09e8db6b81e4c7c253aae301ec56bb1db5509070139399328391db/diff/etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-Cloud08:28
ltomasboso, /var/lib/docker/overlay2 instead08:29
dulekdmellado: Hey, actually if I'll run Kuryr containerized, then K8s will take care of giving Kuryr pods the certs!08:30
dmelladoheh xD08:30
dmelladothat makes things much easier xD08:30
dulekdmellado: I mean https://review.openstack.org/#/c/588223 still failed, but probably due to different reason.08:30
dmelladowe can just go and run multinode only containerized for now xD08:30
dmelladodulek: I'll take a look while I wait xD08:31
dulekdmellado: Same reason, I did something wrong…08:31
*** garyloug has quit IRC08:33
*** garyloug_ has joined #openstack-kuryr08:33
*** garyloug has joined #openstack-kuryr08:34
ltomasbocelebdor, actually, I manually mounted the container with buildah, and directory scratchmnt seems to be empty08:34
celebdordafuq08:35
celebdorat that step it should already be mounted!08:35
celebdoryou running as sudo?08:35
ltomasboyep08:35
celebdortry to run the commands in the terminal08:35
celebdormanually08:35
celebdorsee where it goes to hell08:35
ltomasbothat is what I'm doing08:35
ltomasbosudo buildah from scratch08:36
ltomasbo sudo buildah mount working-container-508:36
ltomasbo-> /var/lib/containers/storage/overlay/5387bba9d90c7876793cfa1dff2216da6e4a561d8d5132647c2a43291668cb91/merged08:36
ltomasbobut then ls on that dir shows nothing08:36
celebdorman08:37
celebdorof course it is empty08:37
celebdor"from scratch"08:37
celebdorit will only be filled after you do the yum --installroot08:37
ltomasbook08:38
ltomasbothere you are08:38
ltomasbothen the problem will be with that yum install...08:39
celebdorprobably08:39
ltomasboError: Package: centos-release-storage-common-2-2.el7.centos.noarch (buildah_el7_extras)08:41
ltomasbo           Requires: centos-release >= 7-5.1804.el7.centos.208:41
ltomasbo           Installing: centos-release-7-5.1804.el7.centos.x86_64 (buildah_el7)08:41
ltomasbo               centos-release = 7-5.1804.el7.centos08:41
ltomasboyep, it complains and suggest either --skip-broken or rpm -Va --nofiles --nodigest08:41
ltomasboyou are testing this on fedora, right?08:41
celebdormmm08:41
ltomasboor a different centos... let me try yum update...08:42
celebdorltomasbo: yes08:42
celebdoron fedora08:42
celebdorI'm probably blind08:42
celebdorwhat's the diff between the requires and the installing08:42
celebdor?08:42
celebdorlooks like it's the same version to me08:42
ltomasboI don't see it either...08:42
ltomasbocentos.2 instead of centos.x8608:42
celebdoryeah...08:43
celebdorI know08:43
celebdorthat's weird08:43
celebdordulek: what a load of bollocks08:43
celebdorwith the scc addition then it works08:44
* celebdor taking a break for breakfast08:44
dulekcelebdor: ¯\_(ツ)_/¯08:45
celebdordulek: now to see how I can put that addition into a yaml08:48
celebdor:/08:48
celebdoroc get all08:48
celebdoris false advertising08:48
celebdorit misses most resources08:48
celebdorxD08:48
dulekcelebdor: Missing the non-vanilla-k8s resources?08:49
celebdorresources like securitycontextconstraints.security.openshift.io08:49
celebdorwhich is exactly what I need08:49
celebdor:/08:49
ltomasbocelebdor, I think I found the issue on your script, some repo is missing09:19
ltomasbocelebdor, if I remove the disablerepo, then it works09:19
ltomasbocelebdor, dmellado: this fix the issue with the namespace tempest test: https://review.openstack.org/#/c/58846309:22
ltomasboI checked it on https://review.openstack.org/#/c/58068009:22
ltomasboon that one, the kuryr-kubernetes-tempest-daemon-containerized-octavia-namespace gate fails due to me removing the regex to only execute the namespace test, but those passed09:23
celebdorltomasbo: repo can't be missing09:23
celebdorsince it works on my machine that does not have centos stuff at all09:23
celebdor:P09:23
celebdormost likely there's some collision between your centos repos and those that I put09:24
ltomasboprobably09:24
dulekOh crap, now there's an issue of Docker images missing on second node…09:26
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers  https://review.openstack.org/58068009:33
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Set namespace security group driver for namespace gate  https://review.openstack.org/58848709:33
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add namespace isolation for services  https://review.openstack.org/58142109:36
celebdordulek: that's a good one09:38
dulekcelebdor: I'm just working on forcing a build on subnode as well.09:39
dulekBut it would be best to have a registry on master. :)09:39
celebdordulek: either that or pulling from master09:39
celebdoryes09:39
celebdorthat would be the best09:39
dulekcelebdor: Can I pull from master without setting anything? Isn't Docker binding to 127.0.0.1 by default?09:40
celebdormmm09:44
celebdorunsure09:44
celebdordulek: IIRC you'll have to use docker save and docker load09:45
celebdorif you don't want to run a proper registry09:45
dulekcelebdor: Oh no, no, orchestrating SCP in multinode DevStack isn't something I'll do.09:45
openstackgerritMichał Dulko proposed openstack/kuryr-kubernetes master: WIP: Add HA gate  https://review.openstack.org/58822309:46
celebdordulek: I knew you'd like the idea09:46
celebdor:-)09:46
celebdorit's easier to run a registry probably09:46
celebdorxD09:47
celebdoror to run a python http server via devstack09:47
celebdorxD09:47
dulekcelebdor: Definitely. But even easier to add KURYR_FORCE_IMAGE_BUILD. ;)09:47
dulekcelebdor: I'm guessing there are no guarantees about synchronization of DevStack's progress on both nodes, so still it's hard.09:47
dulekdmellado: ^ ?09:47
dmelladodulek: just back here, let me read the backlog09:48
dulekdmellado: Nah, just the last question about synchronization. :)09:48
dmelladooh, no, that deff won't be synced09:49
dmelladoso I wouldn't count on that at all09:49
celebdordulek: xD09:54
celebdordulek: hey, wait09:54
celebdorthere is a better solution, maybe09:54
dulek?09:56
celebdordulek: http://logs.openstack.org/01/580201/11/check/kuryr-kubernetes-tempest-daemon-containerized-octavia/17f3a5f/job-output.txt.gz#_2018-08-02_16_45_00_34383609:58
celebdorwonder if that's somewhere we could push09:58
celebdorand pull from09:59
celebdor(probably not)09:59
dulekcelebdor: Ah, interesting. But I highly doubt we would get access.10:00
celebdoryeah10:00
celebdordulek: well, we could always use the ocp registry after my patch10:01
celebdorbut on k8s we still don't have it coded10:02
dulekYup!10:02
dmelladotbh I'm really looking forward for the openstack-infra registry...10:08
dmelladodulek: that's something that I'd like you to bring on the PTG, although I'm not that optimistic about it10:09
dulekdmellado: Me too!10:10
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: devstack: deploy openshift registry on origin envs  https://review.openstack.org/58020110:11
celebdordmellado: yes, I also look forward to things that will probably never come, like Catalan independence10:14
dmelladocelebdor: you shouldn't say that, man xD10:15
celebdorI'd put registry in about as likely as my son who doesn't like football ending up playing in the Ekstraklasa league10:15
dmelladowhat's that, the czech league?10:16
celebdorno, an even funnier one, source of multiple fail videos10:17
celebdorthe polish one10:17
celebdorhttps://www.reddit.com/r/soccer/comments/90agrj/today_is_the_day_the_best_league_in_the_world_is/10:19
celebdorin this case, Ekstraklasa is not a 'nomen omen'10:20
dmelladolol10:20
dmelladothat's even worse than Abreu10:20
celebdorindeed10:23
dulekcelebdor, dmellado: Oh, now it's even funnier - it's Europa League qualifications time!10:44
dulekThere's even a term for that - eurowpierdol.10:45
dmelladolol10:45
dmelladodulek: LOL10:45
dulekWhich kinda translates to eurobeating.10:45
dmelladodid you know that pierdo means 'I lost' in spanish xD10:45
dulekdmellado: Nah, seems like it's clever than I thought. :D10:45
ltomasbodulek, celebdor: this is fixing the namespace gate. Can you review it: https://review.openstack.org/#/c/588463/10:45
celebdorxD10:46
celebdorwill check10:47
celebdordid I ever tell you all how much I hate devstack plugin development?10:47
ltomasbolol10:47
ltomasboit rings a bell...10:47
dmelladocelebdor: since midonet?10:47
dmelladoI still recall about the devstack FORK10:47
dmelladowhich was almost impossible to keep sane10:47
celebdordmellado: cause it shouldn't have been forked10:49
celebdorthe TC should have just force moved to ansible10:49
celebdorfuck10:49
dmelladowell, that's funny10:49
dmelladoas we proposed that10:49
dmelladoI guess I'll tell you the storytale when we get to meet10:49
dmelladoxD10:49
dulekI'd say that main issue is rewriting all the DevStack plugins. There's quite a number of them out there.11:01
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Add namespace isolation for services  https://review.openstack.org/58142111:04
celebdordulek: indeed11:17
*** rh-jelabarre has joined #openstack-kuryr11:26
*** garyloug has quit IRC11:39
*** garyloug_ has quit IRC11:39
*** garyloug_ has joined #openstack-kuryr11:39
*** garyloug has joined #openstack-kuryr11:39
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: devstack: deploy openshift registry on origin envs  https://review.openstack.org/58020111:40
openstackgerritAntoni Segura Puimedon proposed openstack/kuryr-kubernetes master: devstack: deploy openshift registry on origin envs  https://review.openstack.org/58020111:45
celebdorltomasbo: dulek: dmellado: ^^ seems ready12:21
*** garyloug_ has quit IRC12:23
*** garyloug has quit IRC12:23
dmelladocelebdor: let's take a look12:28
dmelladocelebdor: do you know how much do I hate those devstack patches with bash, python and EOF hacks? xD12:29
celebdordmellado: not enough12:30
celebdorif you had to write them you'd hate them more12:30
dmelladocelebdor: you know what is even worse than having ISP issues?12:34
celebdortell me about ISP issues12:35
celebdorI'm having meetings on 4g cause the dsl here is crap12:35
dmelladohaving ISP issues on my parents in law place12:36
dmelladoI'm on 4g now...12:36
dmelladocalling the dreadeds movistar folks12:36
* dmellado sighs12:36
celebdordmellado: so you are in a similar situation12:37
celebdorbut 4g works really well12:38
dmelladosay goodby to video conferences, though xD12:38
dmelladogoodbye12:38
celebdordmellado: they work, if your data cap is high enough12:41
* celebdor -> weekend12:41
celebdorBlessed be 35h week in August12:41
dmelladoCelebdor12:44
dmelladoEnjoy!12:44
dmelladoNice weekend12:45
celebdorthanks12:45
celebdoryou too12:45
celebdorand merge my registry patch12:45
celebdorltomasbo: try it please12:45
ltomasbocelebdor, is you still here12:54
ltomasbo+2/+W this one: https://review.openstack.org/#/c/588463/12:54
ltomasbo(if you think it is ok ofc)12:55
dulekltomasbo: Don't worry, I'm still here.13:05
ltomasbo\o/13:05
ltomasboxD13:05
dulekltomasbo: And reviewing seems like a great activity when you're in passenger car seat. ;)13:05
ltomasbodulek, I have quite a few patch sets depending one on another13:05
ltomasboxD13:06
ltomasbowhere are you traveling to?13:06
dmelladoTo a foreign country, Spain xD13:07
dulekltomasbo: Gdańsk->Giżycko, my hometown. Ah, perks of remote work. :)13:08
dmelladoDulek, enjoy home food xD13:08
dulekltomasbo: Seems like namespaces gate still failed?13:09
dulekdmellado: I will. :)13:09
*** tzumainn has joined #openstack-kuryr13:13
dulekltomasbo: Ping, question above. ^13:29
celebdordulek: what a fishy coat of arms your town has13:29
celebdorbetween two lakes must have a good amount of mosquitos13:30
celebdor:P13:30
dulekcelebdor: Oh yes, there's this dull legend that a prince gave the… placement law (?) after being served three fishes on blue plate.13:34
*** garyloug has joined #openstack-kuryr13:34
*** garyloug_ has joined #openstack-kuryr13:34
dulekcelebdor: And yes, but after ~20 bites you don't feel any new.13:35
ltomasbodulek, sorry I was in a meeting13:38
ltomasbodulek, it does not fail, check it here: https://review.openstack.org/#/c/588487/13:39
dulekltomasbo: I'm convinced. :)13:40
ltomasbodulek, it is kind of a chicken/egg problem13:40
dulekltomasbo: Sure, sure, I get it.13:40
ltomasboI need to fix the kuryr-tempest-plugin, but as the first namespace isolation patch got merged, I need to enable the SG-driver (as in the second patch)13:41
ltomasbook13:41
openstackgerritVladimir Kuramshin proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI  https://review.openstack.org/51228113:46
openstackgerritVladimir Kuramshin proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler  https://review.openstack.org/51228013:47
ltomasbodulek, ... https://review.openstack.org/#/c/588487/13:57
ltomasbo:D13:57
ltomasbodulek, last one to review for today (in case you are still bored at the car)  https://review.openstack.org/#/c/58142114:15
openstackgerritVladimir Kuramshin proposed openstack/kuryr-kubernetes master: Add SR-IOV capabilities to VIF handler  https://review.openstack.org/51228014:15
dulekltomasbo: I'm already reading it. :)14:16
dulekltomasbo: What happens here: https://review.openstack.org/#/c/580680/8/.zuul.d/octavia.yaml ?14:16
ltomasbothat I will fix it later14:16
ltomasbothe problem there is that if we enable isolation, some of the current tests will fail14:16
ltomasboas they are expecting to have fip connectivity14:17
ltomasbowhich they will not have until the last namespace patch lands14:17
openstackgerritVladimir Kuramshin proposed openstack/kuryr-kubernetes master: Add SR-IOV binding driver to CNI  https://review.openstack.org/51228114:17
dulekltomasbo: Okay, so I'm leaving it out for now.14:17
ltomasboI can actually try to change the depends on and try14:17
ltomasboyep, important ones remaining are these two:14:17
ltomasbohttps://review.openstack.org/#/c/58142114:18
ltomasbohttps://review.openstack.org/#/c/587778/414:18
openstackgerritLuis Tomas Bolivar proposed openstack/kuryr-kubernetes master: Ensure OpenShift gate uses the namespace subnet/sg drivers  https://review.openstack.org/58068014:19
*** janki has quit IRC14:28
dulekltomasbo: Aww, you're removing worker_nodes_subnet option?14:30
ltomasboam I?14:30
ltomasbolet me see...14:31
dulekltomasbo: https://review.openstack.org/#/c/581421/21/kuryr_kubernetes/controller/drivers/nested_vif.py14:31
dulekltomasbo: Maybe you move it, haven't seen rest of the files yet.14:31
ltomasboahh, yes, I moved to the main one14:31
ltomasboas I needed https://review.openstack.org/#/c/581421/21/kuryr_kubernetes/config.py14:32
*** hongbin has joined #openstack-kuryr14:33
ltomasbodulek, I needed to have it accessible from the lbaasv2 driver, to add support for the non-native route, i.e., access from the nodes where the route pods run to the pods14:34
dulekltomasbo: Awesome!14:34
ltomasbomen, I wrote that code 2 weeks ago and it seems like it was 2 years... I completely forgot about that... xD14:35
dulekltomasbo: +2, now you need to convince celebdor. :)14:40
ltomasbothanks! I expected -1s!! xD14:41
*** kzaitsev_pi has quit IRC14:41
dulekltomasbo: Nah, I've already looked at it previously. :)14:46
dulekOkay, have a great weekend folks! :)14:46
ltomasboyou too!14:47
ltomasbothanks!14:47
*** janki has joined #openstack-kuryr14:51
openstackgerritMerged openstack/kuryr-tempest-plugin master: Fix delete namespace resources function  https://review.openstack.org/58846314:56
openstackgerritMerged openstack/kuryr-kubernetes master: devstack: deploy openshift registry on origin envs  https://review.openstack.org/58020115:12
*** janki has quit IRC15:29
*** openstackgerrit has quit IRC16:49
*** dmellado has quit IRC17:22
*** dmellado has joined #openstack-kuryr19:56
*** pcaruana has quit IRC21:38
*** rh-jelabarre has quit IRC21:56
*** hongbin has quit IRC22:19

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!