Monday, 2020-11-16

« Sunday, 2020-11-15 Index Tuesday, 2020-11-17 »
*** zzzeek has quit IRC00:03
*** zzzeek has joined #openstack-kolla00:05
*** zzzeek has quit IRC00:28
*** zzzeek has joined #openstack-kolla00:30
*** LinPeiWen67 has joined #openstack-kolla00:53
*** dmsimard has quit IRC02:49
*** mchlumsky has quit IRC02:50
*** mchlumsky has joined #openstack-kolla02:50
*** dmsimard has joined #openstack-kolla02:50
*** skramaja has joined #openstack-kolla03:43
*** zzzeek has quit IRC03:45
*** zzzeek has joined #openstack-kolla03:47
*** zzzeek has quit IRC04:20
*** zzzeek has joined #openstack-kolla04:20
*** evrardjp has quit IRC05:33
*** evrardjp has joined #openstack-kolla05:33
*** wuchunyang has joined #openstack-kolla06:26
*** ricolin has quit IRC06:31
*** JamesBenson has quit IRC06:31
*** JamesBenson has joined #openstack-kolla06:32
oklhostMorning06:35
*** JamesBenson has quit IRC06:37
*** nikparasyr has joined #openstack-kolla06:41
*** cah_link has joined #openstack-kolla06:47
*** xinliang has joined #openstack-kolla06:55
wuchunyangmorning06:56
*** ricolin_ has joined #openstack-kolla07:18
*** ricolin_ has quit IRC07:20
*** kwazar has joined #openstack-kolla07:20
*** ricolin has joined #openstack-kolla07:21
*** jbadiapa has joined #openstack-kolla07:25
*** rpittau|afk is now known as rpittau07:26
*** xinliang has quit IRC07:38
*** brinzhang has joined #openstack-kolla07:42
*** rpittau is now known as rpittau|bbl07:52
*** zzzeek has quit IRC08:00
*** zzzeek has joined #openstack-kolla08:03
*** JamesBenson has joined #openstack-kolla08:04
*** e0ne has joined #openstack-kolla08:06
*** JamesBenson has quit IRC08:09
*** gfidente has joined #openstack-kolla08:14
*** bengates has joined #openstack-kolla08:18
*** stingrayza has quit IRC08:25
*** stingrayza has joined #openstack-kolla08:26
*** dougsz has joined #openstack-kolla08:31
*** zzzeek has quit IRC08:57
*** zzzeek has joined #openstack-kolla08:58
*** k_mouza has joined #openstack-kolla09:01
*** mgoddard has joined #openstack-kolla09:05
*** zzzeek has quit IRC09:06
*** k_mouza has quit IRC09:06
*** zzzeek has joined #openstack-kolla09:07
*** zzzeek has quit IRC09:24
*** zzzeek has joined #openstack-kolla09:24
mgoddardmorning09:28
mgoddardwelcome to the core team, wuchunyang :)09:29
hrwwuchunyang: guard the gate!09:32
wuchunyangthanks for all gays09:33
wuchunyanghaha09:33
wuchunyangit is glad to join you09:33
hrwsorry, but we run out of pins - https://photos.app.goo.gl/9GWaYCwkhkovj27R809:34
hrwI got it during SnowPenStack PTG09:35
wuchunyanghrw  an interesting pic09:47
*** k_mouza has joined #openstack-kolla10:01
*** sean-k-mooney2 has joined #openstack-kolla10:05
*** sean-k-mooney1 has quit IRC10:08
*** dougsz has quit IRC10:08
*** JamesBenson has joined #openstack-kolla10:15
*** JamesBenson has quit IRC10:20
*** dougsz has joined #openstack-kolla10:22
*** zzzeek has quit IRC10:23
*** zzzeek has joined #openstack-kolla10:24
*** sean-k-mooney has joined #openstack-kolla10:27
*** sean-k-mooney2 has quit IRC10:30
*** sean-k-mooney1 has joined #openstack-kolla10:32
*** sean-k-mooney has quit IRC10:35
openstackgerritMichal Nasiadka proposed openstack/kolla stable/victoria: Switch to centos-release-openstack-victoria package  https://review.opendev.org/76254610:36
*** zzzeek has quit IRC10:41
*** zzzeek has joined #openstack-kolla10:42
hrw+210:43
*** wuchunyang has quit IRC10:45
openstackgerritMark Goddard proposed openstack/kolla master: docs: add code reviews to contributor guide  https://review.opendev.org/76281710:55
*** zzzeek has quit IRC11:12
*** zzzeek has joined #openstack-kolla11:13
*** JamesBenson has joined #openstack-kolla11:23
*** zzzeek has quit IRC11:27
*** sean-k-mooney1 has quit IRC11:27
*** sean-k-mooney1 has joined #openstack-kolla11:27
*** sean-k-mooney1 is now known as sean-k-mooney11:28
*** JamesBenson has quit IRC11:28
*** rpittau|bbl is now known as rpittau11:30
*** zzzeek has joined #openstack-kolla11:30
openstackgerritMark Goddard proposed openstack/kolla master: WIP: revert to public package mirrors after build  https://review.opendev.org/76192811:32
openstackgerritMark Goddard proposed openstack/kolla-ansible master: WIP: CI: add experimental upgrade jobs  https://review.opendev.org/76008611:39
*** e0ne has quit IRC11:44
openstackgerritMark Goddard proposed openstack/kayobe-config master: Sync configs with kayobe  https://review.opendev.org/76152111:45
openstackgerritMark Goddard proposed openstack/kayobe-config master: Sync configs with kayobe @ 074024d63f9cb364ca16a7a7f0ac94d77ee9466b  https://review.opendev.org/76152111:50
openstackgerritMark Goddard proposed openstack/kayobe-config-dev master: Sync configs with kayobe @ 074024d63f9cb364ca16a7a7f0ac94d77ee9466b  https://review.opendev.org/76282611:52
*** e0ne has joined #openstack-kolla11:56
*** JamesBenson has joined #openstack-kolla12:00
openstackgerritMark Goddard proposed openstack/kolla master: docs: add code reviews to contributor guide  https://review.opendev.org/76281712:02
*** zzzeek has quit IRC12:14
*** zzzeek has joined #openstack-kolla12:16
hrwhttps://review.opendev.org/#/c/762546/3 waits for +W+212:22
patchbotpatch 762546 - kolla (stable/victoria) - Switch to centos-release-openstack-victoria package - 3 patch sets12:22
openstackgerritMerged openstack/kolla-ansible stable/ussuri: CI: add missing --fail argument to curl  https://review.opendev.org/76274612:25
*** JamesBenson has quit IRC12:31
*** JamesBenson has joined #openstack-kolla12:32
*** dcapone2004 has joined #openstack-kolla12:32
openstackgerritMerged openstack/kolla-ansible stable/victoria: Remove a CentOS IPv6 FIXME meant for Ussuri  https://review.opendev.org/76214012:33
openstackgerritMerged openstack/kolla-ansible stable/ussuri: [CI] Set 0 swap size  https://review.opendev.org/76214212:33
dcapone2004having an issue with an all in one deployment ... testing changing the IP address of the host ... i updated globals.yml with new vip to be in new subnet and ran reconfigure ... the process hangs "running nova bootstrap api" ... no output, the task just never s12:37
dcapone2004appears to never do anything12:37
*** brinzhang_ has joined #openstack-kolla12:48
*** brinzhang has quit IRC12:51
openstackgerritMerged openstack/kayobe stable/ussuri: Fix filtering of network names set to an empty string  https://review.opendev.org/76133012:51
openstackgerritMark Goddard proposed openstack/kolla master: WIP: revert to public package mirrors after build  https://review.opendev.org/76192812:52
*** Luzi has joined #openstack-kolla12:55
ug_egonmgoddard, i saw https://bugs.launchpad.net/kolla-ansible/+bug/1606143 was put on the wishlist. Is there anything i can do to get my pull request merged?13:01
openstackLaunchpad bug 1606143 in kolla-ansible "SSL termination in external loadbalancer" [Wishlist,New]13:01
openstackgerritMerged openstack/kayobe-config master: Sync configs with kayobe @ 074024d63f9cb364ca16a7a7f0ac94d77ee9466b  https://review.opendev.org/76152113:05
openstackgerritMerged openstack/kayobe stable/ussuri: docs: Add link back to Train CentOS 8 migration page  https://review.opendev.org/76132813:06
*** dcapone2004 has quit IRC13:25
*** kwazar has quit IRC13:27
openstackgerritMerged openstack/kolla-ansible stable/victoria: [CI] Set 0 swap size  https://review.opendev.org/76214113:27
openstackgerritMerged openstack/kolla-ansible master: Add override timeout for openstack exporter  https://review.opendev.org/76196613:27
openstackgerritMerged openstack/kolla master: doc: Add new-release tool usage to relmgmt  https://review.opendev.org/76117413:27
*** zzzeek has quit IRC13:31
*** zzzeek has joined #openstack-kolla13:34
*** Luzi has quit IRC13:35
openstackgerritPierre Riteau proposed openstack/kayobe-config-dev master: Sync configs with kayobe @ 074024d63f9cb364ca16a7a7f0ac94d77ee9466b  https://review.opendev.org/76282613:45
*** dcapone2004 has joined #openstack-kolla13:47
*** tbarron has joined #openstack-kolla13:47
openstackgerritMark Goddard proposed openstack/kolla master: WIP: revert to public package mirrors after build  https://review.opendev.org/76192813:55
openstackgerritPierre Riteau proposed openstack/kayobe-config stable/ussuri: Synchronise with kayobe for Ussuri  https://review.opendev.org/76284314:00
*** e0ne has quit IRC14:01
*** zzzeek has quit IRC14:02
kplantis dpdk completely unsupported on centos in kolla?14:06
*** zzzeek has joined #openstack-kolla14:07
*** wuchunyang has joined #openstack-kolla14:13
hrwkplant: there are no dpdk builds for centos 814:14
*** zzzeek has quit IRC14:14
hrwops. there are14:14
hrw~curse centos to cbs+mbox instead of one14:14
*** zzzeek has joined #openstack-kolla14:14
kplantis it supported in kolla though14:15
kplanti've seen verbiage that it is supported and verbiage that it's completely unsupported14:16
kplantand build.py seems to blacklist dpdk14:16
hrwdpdk != ovsdpdk14:18
kplantyou're right14:18
hrwand we do not support ovsdpdk14:19
kplantare there any avenues for dpdk in kolla?14:20
*** e0ne has joined #openstack-kolla14:21
kplantthe verbiage does state ovs-dpdk is supported14:22
hrwkplant: probably lack of interest14:22
kplanthttps://docs.openstack.org/kolla-ansible/latest/reference/networking/dpdk.html14:22
kplantthat says ovs-dpdk explicitly and centos 714:23
mnasiadkakplant: there's the image, which we probably don't build at all, and I think a separate role in k-a - but I doubt anybody has been running that with success in the last year or more :)14:23
kplantis there any OS where ovs-dpdk is supported in kolla?14:23
kplantso is the only way to escape ovs/linuxbridge limitations to use sr-iov? (in kolla)14:25
kplantor jam in something like tungsten14:25
mnasiadkawhat are the limitations you want to escape? :)14:26
kplantonly seeing around 300Mbps of throughput through a 10Gb nic14:26
kplantfrom the instance14:26
mnasiadkawell, dpdk will give you more, and will also keep your cpus quite busy14:26
kplantsr-iov is almost perfect scaling14:26
mnasiadkaso the easiest win is sr-iov, but then forget about live migration14:27
kplantyep14:27
*** wuchunyang has quit IRC14:28
mnasiadkakplant: but I think you should get somewhere closer to 10Gb with proper hw offloads on the NIC14:29
kplantyeah asap² works great with mellanox14:30
kplantproblem is we have some garbage intel x710s that we're forced to work with14:30
kplantand given $0 to get "more performance"14:30
kplant<3 leadership14:30
*** TrevorV has joined #openstack-kolla14:31
kplantthanks guys14:31
*** zzzeek has quit IRC14:33
mnasiadkakplant: well, now I think CentOS provides something for ovs-dpdk in the NFV repo, so if you'd be able to get this to work - it should be an easy win. But all the DPDK reqs like huge pages and other stuff give a headache on operations later on :)14:34
kplantdo you have a link to that nfv repo14:36
mnasiadkahttp://mirror.centos.org/centos/8/nfv/x86_64/openvswitch-2/Packages/o/14:36
*** zzzeek has joined #openstack-kolla14:36
mnasiadkabut no dpdk there14:37
kplantyeah :\14:37
mnasiadkait's in appstream, but I doubt openvswitch packages are built with --with-dpdk14:40
mnasiadkakplant: ask on #rdo, you might get better answers ;)14:40
*** zzzeek has quit IRC14:41
*** zzzeek has joined #openstack-kolla14:45
*** brinzhang0 has joined #openstack-kolla14:46
*** brinzhang_ has quit IRC14:48
*** nikparasyr has left #openstack-kolla14:50
*** skramaja has quit IRC14:54
rockeyopen question, is there any predicted release date of K-A 11.0 in the near vicinity?14:59
rockeyin the tracks of upgrading/reinstalling OS in the lab, would love haivng ussuri installed :)15:00
*** zzzeek has quit IRC15:11
*** zzzeek has joined #openstack-kolla15:14
*** dave-mccowan has joined #openstack-kolla15:34
mnasiadkakplant: thread for you: https://lists.centos.org/pipermail/centos-devel/2020-November/056129.html15:39
*** dave-mccowan has quit IRC15:45
*** zzzeek has quit IRC15:49
*** zzzeek has joined #openstack-kolla15:50
*** kevko has joined #openstack-kolla15:50
kevkohi, some ansible king ? :)15:51
kevkoi have very interisting use-case which i don't know how to solve :D15:51
*** k_mouza has quit IRC15:56
mgoddardug_egon: hi. Wishlist was applied because it is a mini feature request, rather than a bug15:57
mgoddardug_egon: if you want it to be merged, you should start with addressing the comments on the review: https://review.opendev.org/75836115:58
patchbotpatch 758361 - kolla-ansible - Closes-Bug: 1606143 - 1 patch set15:58
openstackbug 1606143 in kolla-ansible "SSL termination in external loadbalancer" [Wishlist,New] https://launchpad.net/bugs/160614315:58
mgoddardhttps://review.opendev.org/#/c/758361/15:58
patchbotpatch 758361 - kolla-ansible - Closes-Bug: 1606143 - 1 patch set15:58
*** zzzeek has quit IRC15:59
mgoddardrockey: should be in the next few weeks15:59
*** zzzeek has joined #openstack-kolla16:00
ug_egonThanks!16:06
*** zzzeek has quit IRC16:17
rockeymgoddard: thanks :)16:18
*** zzzeek has joined #openstack-kolla16:20
openstackgerritMark Goddard proposed openstack/kolla master: WIP: revert to public package mirrors after build  https://review.opendev.org/76192816:24
*** k_mouza has joined #openstack-kolla16:25
mgoddardseen in CI: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit\"16:33
mgoddardyoctozepto, hrw, mnasiadka: ^16:35
dcapone2004having an issue with an all in one deployment ... testing changing the IP address of the host ... i updated globals.yml with new vip to be in new subnet and ran reconfigure ... the process hangs "running nova bootstrap api" ... no output, the task just never appears to do anything...I cannot find any meaningful log output anywhere...any ideas on16:37
dcapone2004where/what log to check?16:37
yoctozeptomgoddard: k-a, no?16:37
mgoddardyoctozepto: yes16:37
yoctozeptomgoddard: fsck16:37
mgoddardyoctozepto: it may not normally hit us. It was in https://review.opendev.org/#/c/76008616:38
patchbotpatch 760086 - kolla-ansible - WIP: CI: add experimental upgrade jobs - 6 patch sets16:38
mgoddardwhich will pull more containers than usual16:38
dcapone2004also, if I just run deploy-containers, the containers deploy, but neutron_openvswitch_agent, nova_conductor, and nova_scheduler end up in a continue reboot state, but again no real good hints in the logs for any of those containers16:38
mgoddardbut probably we will hit it at some point16:38
dcapone2004I should rephrase that last part....there are errors in the restarting container logs, but the errors make sense with the reconfiguration not completing as the errors all reference the fact that there are incorrect endpoints, can't find IP X, etc....but nothing as to why the reconfigure would not run and complete16:41
*** bengates has quit IRC16:47
dcapone2004I should add that I attempted to use just "Deploy" as well and KA hangs at the same spot and task of running Nova API bootstrap container....and I should also note that despite KA being hung on that task, I can see that the nova_api container is running in docker16:50
*** kwazar has joined #openstack-kolla16:56
*** cah_link has quit IRC17:04
mnasiadkamgoddard: what if we authenticate as kolla user? Do we get unlimited pulls of our own images?:)17:05
mgoddardmnasiadka: it would give us 200 pulls, but that would be across all cloud providers17:07
mgoddardmnasiadka: probably better to have 100 pulls per IP17:07
mnasiadkamgoddard: I guess somebody needs to file for better limits - like an open source account?17:08
mgoddardmnasiadka: correct. infra were supposed to be looking at it17:09
mgoddardyoctozepto: did you speak to infra?17:09
yoctozeptomgoddard: not recently17:18
kevkohi, guys , how can i apply role to some groups ..iterating through with items17:25
kevko*to some group iterating over groups17:26
*** zzzeek has quit IRC17:37
*** zzzeek has joined #openstack-kolla17:38
*** rpittau is now known as rpittau|afk17:41
*** k_mouza has quit IRC17:43
*** zzzeek has quit IRC17:45
dcapone2004this is what I get in verbose mode as log output, and then everything just stops, I've let it stay for 24 hours with no change.... Same output whether I use deploy or reconfigure17:46
dcapone2004kolla17:46
dcapone2004/home/kolla/.ansible/tmp/ansible-tmp-1605548402.892203-1554347-216965824628485 `" ) && sleep 0'17:46
dcapone2004/home/kolla/.ansible/tmp/ansible-tmp-1605548402.892203-1554347-216965824628485/AnsiballZ_kolla_docker.py17:46
dcapone2004<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-lwcxcqfxgprwdxprzzpbxecbpvrjskyr ; /usr/libexec/platform-python /home/kolla/.ansible/tmp/ansible-tmp-1605548402.892203-1554347-216965824628485/AnsiballZ_kolla_docker.py'"'"' && sleep 0'17:46
*** zzzeek has joined #openstack-kolla17:48
*** dougsz has quit IRC17:48
*** k_mouza has joined #openstack-kolla17:51
*** k_mouza has quit IRC17:55
*** brinzhang_ has joined #openstack-kolla17:56
*** e0ne has quit IRC17:56
*** dking has joined #openstack-kolla17:57
*** brinzhang0 has quit IRC17:59
*** cah_link has joined #openstack-kolla18:03
*** mgoddard has quit IRC18:05
openstackgerritMerged openstack/kolla stable/victoria: Switch to centos-release-openstack-victoria package  https://review.opendev.org/76254618:11
*** cah_link has quit IRC18:22
*** gfidente is now known as gfidente|afk18:37
*** e0ne has joined #openstack-kolla18:39
*** e0ne has quit IRC18:45
*** e0ne has joined #openstack-kolla18:55
*** e0ne has quit IRC18:55
*** zzzeek has quit IRC18:59
*** zzzeek has joined #openstack-kolla19:00
*** cah_link has joined #openstack-kolla19:12
*** kevko has quit IRC19:20
*** cah_link has quit IRC19:38
*** dciabrin has quit IRC19:52
*** dciabrin has joined #openstack-kolla19:52
*** dcapone2004 has quit IRC19:58
*** dcapone2004 has joined #openstack-kolla20:01
*** jonaspaulo has joined #openstack-kolla20:31
*** k_mouza has joined #openstack-kolla20:44
*** jonaspaulo has quit IRC20:44
*** TrevorV has quit IRC20:47
dcapone2004for anyone that happens to come across my inquiry in the logs....it appears that changing an IP address is not supported (this may be an openstack "issue" moreso than a KA issue) because the IP address is stored in multiple connection strings in the mariadb...this prevents the container from properly launching20:48
*** k_mouza has quit IRC20:49
openstackgerritMerged openstack/kolla-ansible stable/train: [CI] Set 0 swap size  https://review.opendev.org/76232920:49
dcapone2004After going into the mariadb container and manual update several transport URL entries and other entries throughout the db, KA was able to redeploy the environment ... I guess this rules out a deployment scenario where a cluster is configured and tested for hardware validation on one network and shipping to a remote location and then only requiring20:50
dcapone2004IP addresses to be updated with KA20:50
*** dcapone2004 has quit IRC22:17
*** jbadiapa has quit IRC22:38
« Sunday, 2020-11-15 Index Tuesday, 2020-11-17 »

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!