Wednesday, 2020-04-01

*** Tony31 has joined #openstack-kolla00:35
*** zhanglong has joined #openstack-kolla00:58
*** dave-mccowan has quit IRC01:01
*** Tony31 has quit IRC01:35
*** devfaz has quit IRC03:33
*** devfaz has joined #openstack-kolla03:33
*** Tengu has quit IRC03:34
*** Tengu has joined #openstack-kolla03:36
*** skramaja has joined #openstack-kolla04:01
*** ykarel|away is now known as ykarel04:10
*** evrardjp has quit IRC04:36
*** evrardjp has joined #openstack-kolla04:36
*** cah_link has joined #openstack-kolla04:49
*** JangwonLee_ has joined #openstack-kolla04:54
*** JangwonLee__ has quit IRC04:57
*** JangwonLee__ has joined #openstack-kolla05:07
*** JangwonLee_ has quit IRC05:10
*** JangwonLee_ has joined #openstack-kolla05:16
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic to Keystone service  https://review.opendev.org/71398605:16
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Keystone HAProxy traffic  https://review.opendev.org/71398605:18
*** JangwonLee__ has quit IRC05:19
*** vishalmanchanda has joined #openstack-kolla05:22
*** irclogbot_3 has quit IRC06:49
*** sorin-mihai_ has joined #openstack-kolla06:50
*** Reepicheep has quit IRC06:51
*** spotz has quit IRC06:51
*** Reepicheep has joined #openstack-kolla06:51
*** irclogbot_3 has joined #openstack-kolla06:52
*** irclogbot_3 has quit IRC06:53
*** sorin-mihai__ has quit IRC06:53
*** tonyb has quit IRC06:54
*** irclogbot_2 has joined #openstack-kolla06:54
*** dking_desktop has quit IRC06:58
*** JangwonLee__ has joined #openstack-kolla06:58
*** hrw has joined #openstack-kolla06:58
*** tonyb has joined #openstack-kolla06:59
*** priteau has joined #openstack-kolla07:00
priteauGood morning Kolla folks07:01
*** JangwonLee has joined #openstack-kolla07:01
*** dannins has joined #openstack-kolla07:01
*** JangwonLee_ has quit IRC07:02
priteauI just noticed that the cherry-pick of https://review.opendev.org/#/q/Ia73b5665cece523bb822f6a223335f6fae94fb6a to stable/stein only added /run/openvswitch to neutron-openvswitch-agent-xenapi, and not to neutron-openvswitch-agent.07:02
priteauhttps://review.opendev.org/#/c/713488/1/ansible/roles/neutron/defaults/main.yml07:03
patchbotpatch 713488 - kolla-ansible (stable/stein) - Fix native openvswitch firewall driver in neutron-... (MERGED) - 1 patch set07:03
*** JangwonLee__ has quit IRC07:03
*** dougsz has joined #openstack-kolla07:04
*** abdysn has joined #openstack-kolla07:06
*** jbadiapa has joined #openstack-kolla07:08
priteauThere's another unmerged patch fixing it: https://review.opendev.org/#/c/716178/07:12
patchbotpatch 716178 - kolla-ansible (stable/stein) - Fix ovs fw driver for the other ovs agent - 1 patch set07:12
*** bengates has joined #openstack-kolla07:13
*** bengates has quit IRC07:15
mnasiadkamorning07:15
mnasiadkapriteau: yes, yoctozepto noticed it yesterday - are you using native ovs firewall driver?07:15
mnasiadka(it's not the default)07:16
*** bengates has joined #openstack-kolla07:16
*** hrww has joined #openstack-kolla07:18
*** hrw has quit IRC07:19
*** hrw has joined #openstack-kolla07:20
*** hrww has quit IRC07:23
yoctozeptopriteau: yeah, we be blind07:25
*** Fl1nt has joined #openstack-kolla07:29
Fl1ntGood morning people!07:29
hrwmorning07:29
hrwno idea yet is it good or not07:29
Fl1ntIt's up to you for that day to be good or not ;-)07:31
*** e0ne has joined #openstack-kolla07:35
*** rpittau|afk is now known as rpittau07:37
priteaumnasiadka: I am using iptables_hybrid, so I don't think I am impacted. I was just inspecting the docker container after deploying with a newer kolla-ansible and noticed the missing line.07:38
mnasiadkapriteau: since neutron still uses ofctl commands in some cases, that mount point is needed - but won't be needed soon07:39
*** dougsz has quit IRC07:55
*** JangwonLee_ has joined #openstack-kolla07:55
*** cah_link has quit IRC07:56
*** JangwonLee__ has joined #openstack-kolla07:59
*** cah_link has joined #openstack-kolla07:59
*** JangwonLee has quit IRC07:59
*** negronjl has quit IRC08:00
*** JangwonLee_ has quit IRC08:02
cosmicsoundgood day08:03
*** negronjl has joined #openstack-kolla08:03
*** JangwonLee_ has joined #openstack-kolla08:04
*** JangwonLee__ has quit IRC08:07
*** abdysn has quit IRC08:20
*** abdysn has joined #openstack-kolla08:21
mrungegood morning, I have a q on centos CI jobs. Any idea if packages from centos-extras are available, or are there any known issues with caches?08:25
mrungethis is about https://review.opendev.org/#/c/715923/08:25
patchbotpatch 715923 - kolla - add dpdk_telemetry and logparser, use centos-relea... - 4 patch sets08:25
*** ykarel is now known as ykarel|lunch08:30
yoctozeptomrunge: yeah, centos 8 extras are enabled by this point08:33
mrungeI don't understand, why ci checks fail then08:34
mrungeit looks like the package can not be pulled from centos-extras08:34
mnasiadkahttps://zuul.opendev.org/t/openstack/build/7e7b02bed6df403e83ec89ae54cfcb90/log/kolla/build/000_FAILED_base.log#76208:35
mnasiadkathis shows the opstools are already known to dnf08:35
mrungethat's third party, coming from delorean or so08:35
mnasiadkaoh boy08:36
mrungeunable to find a match... 3 lines below08:36
mrungeand I see it knows centos-extras08:36
*** gfidente|afk is now known as gfidente08:40
*** JangwonLee__ has joined #openstack-kolla08:48
*** pbing19 has joined #openstack-kolla08:49
*** JangwonLee_ has quit IRC08:52
mnasiadkamrunge: weird, that package is in centos 8 extras when I just boot the centos:8 container from docker hub...09:03
*** priteau has quit IRC09:03
mnasiadkamrunge: blame infra - http://mirror.dfw.rax.opendev.org/centos/8/extras/x86_64/os/Packages/09:04
mnasiadkaseems mirror is not up to date09:05
*** zhanglong has quit IRC09:12
mrungeright09:14
mrungeany idea who to ping?09:14
yoctozeptomrunge: #opendev09:15
brtknrwhy is this failing folks: https://review.opendev.org/#/c/716339/09:25
patchbotpatch 716339 - kolla (stable/train) - Bump up magnum and manila versions - 2 patch sets09:25
*** abdysn has quit IRC09:26
*** jbadiapa is now known as jbadiapa|away09:27
*** abdysn has joined #openstack-kolla09:30
*** osmanlicilegi has quit IRC09:49
*** osmanlicilegi has joined #openstack-kolla09:50
*** k_mouza has joined #openstack-kolla09:53
*** ykarel|lunch is now known as ykarel09:59
*** Abdallahyas has joined #openstack-kolla10:10
*** abdysn has quit IRC10:13
*** abdysn has joined #openstack-kolla10:14
*** Abdallahyas has quit IRC10:15
*** rpittau is now known as rpittau|bbl10:16
*** Abdallahyas has joined #openstack-kolla10:18
*** abdysn has quit IRC10:21
mnasiadkabrtknr: well, easy - Cannot find a valid baseurl for repo: centos-nfs-ganesha2810:35
mnasiadka:)10:35
cosmicsounddid someone installed kolla oveer k8s?10:38
mnasiadkathere was a project that tried to do that - kolla-k8s, but it has been retired - there is although openstack-helm, which has the same goal - but doesn't use Kolla images10:40
brtknrmnasiadka: what does that have to do with my change10:42
brtknri'm guessing nothing :P10:42
mnasiadkabrtknr: probably nothing, just building images has a lot of external dependencies - and you just hit one being broken-ish :)10:43
mnasiadkabrtknr: rebase, recheck, be persistent unless you see the same error a couple of times :)10:43
brtknrmnasiadka: this is the second time i am seeing this issue10:43
mnasiadkabrtknr: well, I just even double checked the mirror - and it should work10:47
mnasiadkaoh boy, found a bug10:49
*** watersj has joined #openstack-kolla10:52
openstackgerritMichal Nasiadka proposed openstack/kolla master: CI: Add http[s] match for mirror.centos.org to template-overrides  https://review.opendev.org/71654810:55
mnasiadkabrtknr: ^^10:55
WellieHi guys, we are trying to add some custom roles to keystone policy.yaml file. As far as we know, we can overwrite the default policy (for keystone) by adding a folder keyston,  with content policy.yaml / policy.json, to kolla config dir and run a reconfigure. We read, that we need to overwrite the hole policy - not just the edited parts. In order to do this, we need the "default" keystone policy - which is actually in use. We10:58
Wellieallready tryed to generate it with oslo-polocy-generator within the keystone container, but the policy it generates, allways failes to deploy (we tryed it even without changes - just to test the deployment process itself – but didnt worked either). So – where to we get  the „default“ keystone policy from – cause oslopolicy-generator seems not beeing the right way? Or did we get something wrong ? --- We are running Openstack10:58
Wellie Train.10:58
mnasiadkaWellie: it seems keystone team generates it and pushes to docs - https://docs.openstack.org/keystone/latest/configuration/samples/policy-yaml.html, another option is to clone keystone repo from opendev/github and use tox for this (https://github.com/openstack/keystone/blob/stable/train/etc/README.txt)11:05
Welliehm.. we also generated the policy via tox -egenpolicy. But it seems to us - this is really only an example policy.json, cause almost everything in there is quoted out.11:15
*** abdysn has joined #openstack-kolla11:44
*** Abdallahyas has quit IRC11:47
*** zhanglong has joined #openstack-kolla11:52
*** ricolin has quit IRC11:54
*** zhanglong has quit IRC11:57
*** shyamb has joined #openstack-kolla11:58
*** zhanglong has joined #openstack-kolla11:58
*** jbadiapa|away is now known as jbadiapa11:59
*** rpittau|bbl is now known as rpittau12:07
*** spotz has joined #openstack-kolla12:07
*** skramaja has quit IRC12:07
*** skramaja has joined #openstack-kolla12:08
*** dmellado has quit IRC12:15
*** dmellado has joined #openstack-kolla12:23
*** shyamb has quit IRC12:24
openstackgerritKevin Zhao proposed openstack/kolla stable/train: CI: Use network mode = host  https://review.opendev.org/71657412:43
openstackgerritKevin Zhao proposed openstack/kolla stable/train: [DNM]Test Arm64 Job DNS - stable/train  https://review.opendev.org/71657512:43
openstackgerritKevin Zhao proposed openstack/kolla stable/train: [DNM]Test Arm64 Job DNS - stable/train  https://review.opendev.org/71657512:49
*** dave-mccowan has joined #openstack-kolla12:50
*** dave-mccowan has quit IRC12:54
mnasiadkaWellie: so might be keystone folks changed approach, and there's a default policy - which you just need to override12:59
*** ricolin_ has joined #openstack-kolla13:01
*** jbadiapa has quit IRC13:02
*** jbadiapa has joined #openstack-kolla13:04
openstackgerritMichal Nasiadka proposed openstack/kolla master: CI: Add http[s] match for mirror.centos.org to template-overrides  https://review.opendev.org/71654813:04
*** zhanglong has quit IRC13:21
*** rouk has joined #openstack-kolla13:27
roukanyone have any ideas where invalid utf8 could be coming from with fluentd to elastic (the default central logging setup)? suddenly started getting bad utf8 start bytes, and my logging has been hosed since.13:28
roukerrors just say what index... not what data.13:29
Fl1ntNo meeting today?13:38
*** ykarel is now known as ykarel|afk13:43
mnasiadkaFl1nt: no, just like Mark wrote on the ML13:49
Fl1ntok ok, I'm not registered to the ML sorry ^^13:51
hrwhm.         "ImportError: No module named queue" suggests stable/train being py3 but used with py213:54
hrwnope, it is not kolla's code13:58
hrw        "  File \"/opt/ansible/lib/python2.7/site-packages/openstack/utils.py\", line 13, in <module>",13:59
hrw        "    import queue",13:59
hrw        "ImportError: No module named queue"13:59
hrwso we got openstackclient py3 version on py2 env?13:59
cosmicsoundo/14:00
*** ykarel|afk is now known as ykarel14:02
hrwkolla_toolbox gets openstackclient and has py2...14:02
*** TrevorV has joined #openstack-kolla14:14
hrwreproduced locally ;D14:16
hrwupper-constraints.txt:python-openstackclient===4.0.014:17
hrwhm.14:18
*** cah_link1 has joined #openstack-kolla14:19
*** cah_link has quit IRC14:19
*** cah_link1 is now known as cah_link14:19
mnasiadkahrw: we have images with py2 in master?14:20
hrwthat's train14:20
hrwI start to suspect kolla_toolbox14:20
mnasiadkadeprecate kolla_toolbox14:22
mnasiadka:)14:22
hrwno, it is base we need to remove14:22
mnasiadkamakes sense14:23
hrwINFO:kolla.common.utils.kolla-toolbox:Ignoring pytest: markers 'python_version == "2.7"' don't match your environment14:24
hrwyeah14:24
hrwso we make image with py3 venv and then k-a uses py2 to use this venv?14:25
hrw"docker exec kolla_toolbox openstack"14:26
mnasiadkaoh well, god knows what was the reason14:28
mnasiadkaI hope we're not using py2 on kolla_toolbox in master :D14:28
hrwpy2 in master? die14:29
hrw" virtualenv --system-site-packages /opt/ansible" - we should add '-ppython{{distro python version}}14:32
yoctozeptois it just me or is CI quite red?14:32
mnasiadkayoctozepto: maybe it's related to centos 8 mirror infra uses :D14:33
yoctozeptomnasiadka: so much going on!14:33
mnasiadkayoctozepto: https://review.opendev.org/#/c/716548/ - that's also one thing, nfs ganesha 28 repo started using https://mirror.centos.org :)14:34
patchbotpatch 716548 - kolla - CI: Add http[s] match for mirror.centos.org to tem... - 2 patch sets14:34
mnasiadkapy38 job is failing because of missing mock, but that's non-voting14:36
hrwha! that's why my nova patch fails14:36
*** abdysn has quit IRC14:38
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: use proper Python version in virtualenv  https://review.opendev.org/71660914:39
yoctozeptopossibly kolla-relevant (for matters of support): https://etherpad.openstack.org/p/victoria-leaderless14:45
hrwyoctozepto: mgoddard applied for PTL in V14:47
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: CI: Add http[s] match for mirror.centos.org to template-overrides  https://review.opendev.org/71661614:50
yoctozeptohrw: I mean stuff we build14:51
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions  https://review.opendev.org/71633914:54
mnasiadkait's interesting to see that loci is dead14:55
osmanlicilegihi folks14:59
*** chason has joined #openstack-kolla14:59
hrwmnasiadka: not surprised15:00
mnasiadkawell, I have no clue who did consume it15:00
mnasiadkaI think openstack-helm moved to building their own images15:00
mnasiadkaso maybe that's why it's dead15:01
hrwI contributed a bit to loci15:01
hrwit was doomed as there was no deployment methods for it15:02
hrwkind of 'build and you are on your own'15:02
yoctozeptohi osmanlicilegi15:04
mnasiadkayoctozepto, osmanlicilegi, hrw: https://review.opendev.org/#/c/716548/ - willing to review?15:06
patchbotpatch 716548 - kolla - CI: Add http[s] match for mirror.centos.org to tem... - 2 patch sets15:06
*** chason_ has joined #openstack-kolla15:06
*** chason_ has quit IRC15:06
osmanlicilegiI hope you are all doing well while staying at home :) I'm trying to take myself out from depression. staying at home is not for me. I cannot sleep, I cannot focus... I'm taking some pills to turn back to life :)15:07
*** chason_ has joined #openstack-kolla15:07
*** chason_ has quit IRC15:08
hrw+215:08
*** chason has quit IRC15:08
mnasiadkaosmanlicilegi: I'm practicing social distancing for a couple of years now, just not being able to get out of my flat as often as I would like drives me crazy ;)15:09
yoctozeptomnasiadka: commented15:10
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: CI: Add http[s] match for mirror.centos.org to template-overrides  https://review.opendev.org/71661615:12
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions  https://review.opendev.org/71633915:12
yoctozeptomnasiadka: any way works15:19
*** ykarel is now known as ykarel|away15:22
kplantdo you guys have any idea what would cause: 'qemu-kvm: kvm_init_vcpu failed: cannot allocate memory' ? all of my compute nodes have >1T of ram available15:23
kplantthis is beating me up15:23
mnasiadkayoctozepto: yeah, just realized existing approach is less cryptic :D15:27
openstackgerritMichal Nasiadka proposed openstack/kolla master: CI: Add http[s] match for mirror.centos.org to template-overrides  https://review.opendev.org/71654815:27
mnasiadkayoctozepto: good thing is the infra change went through, so maybe tomorrow we'll have properly synced mirrors15:33
*** bline has joined #openstack-kolla15:39
*** jimcrowleyibm has joined #openstack-kolla15:40
blinetrying to troubleshoot 502 gateway error from barbican, can't find anything in the logs, anyone have ideas?15:40
*** noxoid has joined #openstack-kolla15:44
*** bengates has quit IRC15:50
* hrw out15:50
yoctozeptobline: is not docker ps showing barbican api as exited or restarting?16:02
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71654816:04
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71661616:05
openstackgerritRadosław Piliszek proposed openstack/kolla stable/train: CI: Add https match for mirror.centos.org to template-overrides  https://review.opendev.org/71661616:06
openstackgerritBharat Kunwar proposed openstack/kolla stable/train: Bump up magnum and manila versions  https://review.opendev.org/71633916:10
*** rpittau is now known as rpittau|afk16:15
*** skramaja has quit IRC16:16
*** jonaspaulo has joined #openstack-kolla16:21
*** gfidente has quit IRC16:27
*** evrardjp has quit IRC16:36
*** evrardjp has joined #openstack-kolla16:36
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Keystone HAProxy traffic  https://review.opendev.org/71398617:17
*** jbadiapa has quit IRC17:49
*** jbadiapa has joined #openstack-kolla17:50
openstackgerritAndreas Jaeger proposed openstack/kolla master: Update hacking for Python3  https://review.opendev.org/71669117:53
yoctozeptotrain ubuntu-source-upgrade fried on "ImportError: No module named queue" - due to deploy of stein (so ubuntu-source on stein probably too, argh)17:55
*** gmann is now known as gmann_lunch17:56
yoctozeptoso stein ubuntu source kolla images are fried (this is kolla-toolbox)17:58
yoctozeptoI just read your conversation about it17:58
*** k_mouza has quit IRC18:03
*** k_mouza has joined #openstack-kolla18:06
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix ovs fw driver for the other ovs agent  https://review.opendev.org/71617418:18
*** k_mouza has quit IRC18:19
yoctozeptomnasiadka, mgoddard: prio ^18:20
yoctozeptothanks mgoddard18:23
mgoddardnp18:23
mgoddardgood luck for TC vote yoctozepto18:23
mgoddardI wondered if you might go for it18:24
mgoddardI'm sure you'd do a fine job18:24
blineyoctozepto: It is not restarting and the logs, both /var/log/kolla/barbican and docker logs do not show anything at all18:27
*** gmann_lunch is now known as gmann18:28
blinenetstat shows it listening18:28
*** k_mouza has joined #openstack-kolla18:33
*** k_mouza has quit IRC18:34
*** k_mouza has joined #openstack-kolla18:35
*** e0ne has quit IRC18:38
*** e0ne has joined #openstack-kolla18:38
*** e0ne has quit IRC18:39
*** k_mouza has quit IRC18:39
blinemagnum failed with creating a cert for a cluster but then I was able to easily reproduce it with: openstack secret store --name mysecret --payload j4=]d2118:39
yoctozeptomgoddard: thanks!18:44
yoctozeptobline: well, those logs should have something18:45
yoctozeptobline: especially docker's18:45
yoctozeptobline: I don't mean directly related, just proof of aliveness18:45
yoctozeptobline: btw, we got this reported today https://bugs.launchpad.net/kolla-ansible/+bug/187008618:47
openstackLaunchpad bug 1870086 in kolla-ansible "key-manager-service(Barbican)deployment should use wsgi with Apache" [Undecided,New]18:47
*** dciabrin has quit IRC19:26
*** rouk has quit IRC19:35
*** cah_link1 has joined #openstack-kolla19:38
*** cah_link has quit IRC19:39
*** cah_link1 is now known as cah_link19:39
*** dciabrin has joined #openstack-kolla19:39
openstackgerritAndreas Jaeger proposed openstack/kolla master: Update hacking for Python3  https://review.opendev.org/71669119:42
*** k_mouza has joined #openstack-kolla19:48
*** k_mouza has quit IRC19:50
*** cah_link has quit IRC19:51
*** weshay is now known as weshay|ruck20:00
*** jbadiapa has quit IRC20:07
*** jbadiapa has joined #openstack-kolla20:08
*** e0ne has joined #openstack-kolla20:11
*** igordc has joined #openstack-kolla20:27
*** vishalmanchanda has quit IRC20:39
*** watersj has quit IRC20:46
*** watersj has joined #openstack-kolla20:46
*** cdearborn has joined #openstack-kolla20:52
*** dciabrin has quit IRC21:02
*** dciabrin has joined #openstack-kolla21:02
*** bline has quit IRC21:08
*** bline has joined #openstack-kolla21:09
*** jbadiapa has quit IRC21:14
openstackgerritMerged openstack/kolla-ansible master: Fix ovs fw driver for the other ovs agent  https://review.opendev.org/71617421:18
*** k_mouza has joined #openstack-kolla22:15
*** e0ne has quit IRC22:20
*** kcnomad has joined #openstack-kolla22:20
*** e0ne has joined #openstack-kolla22:21
*** jonaspaulo has quit IRC22:28
*** TrevorV has quit IRC22:31
*** jimcrowleyibm has quit IRC22:33
*** pbing19 has quit IRC23:00
*** e0ne has quit IRC23:13
*** k_mouza has quit IRC23:30
*** Fl1nt has quit IRC23:39

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!