Tuesday, 2020-10-06

*** pleimer has joined #openstack-kolla00:01
*** pleimer has quit IRC00:31
*** pleimer has joined #openstack-kolla00:31
*** zzzeek has quit IRC00:43
*** zzzeek has joined #openstack-kolla00:45
*** kevko_ has quit IRC00:46
*** pleimer has quit IRC00:53
*** ricolin_ has joined #openstack-kolla00:55
*** pleimer has joined #openstack-kolla00:57
*** ricolin_ has quit IRC01:01
*** ricolin_ has joined #openstack-kolla01:01
*** zzzeek has quit IRC01:49
*** zzzeek has joined #openstack-kolla01:51
*** maharg101 has joined #openstack-kolla01:53
*** maharg101 has quit IRC01:58
*** pleimer has quit IRC02:23
*** zzzeek has quit IRC03:17
*** zzzeek has joined #openstack-kolla03:19
*** shyamb has joined #openstack-kolla03:30
*** shyamb has quit IRC03:40
*** shyamb has joined #openstack-kolla03:40
*** brinzhang0 has joined #openstack-kolla03:42
*** brinzhang_ has quit IRC03:45
*** shyamb has quit IRC03:52
*** maharg101 has joined #openstack-kolla03:54
*** maharg101 has quit IRC03:58
*** jbalciunas has joined #openstack-kolla03:59
*** zzzeek has quit IRC04:01
*** zzzeek has joined #openstack-kolla04:05
*** jbalciunas has quit IRC04:09
*** kemopq has quit IRC04:27
*** kemopq has joined #openstack-kolla04:27
*** evrardjp has quit IRC04:33
*** evrardjp has joined #openstack-kolla04:33
*** shyamb has joined #openstack-kolla05:05
oklhostmorning all05:08
*** abdysn has joined #openstack-kolla05:26
*** shyam89 has joined #openstack-kolla05:36
*** shyamb has quit IRC05:38
*** nikparasyr has joined #openstack-kolla05:45
*** maharg101 has joined #openstack-kolla05:55
*** maharg101 has quit IRC06:00
*** _Cyclone_ has quit IRC06:05
*** gouthamr has quit IRC06:05
*** bbezak has quit IRC06:05
*** alanmeadows has quit IRC06:05
*** weshay|ruck has quit IRC06:05
*** mgoddard has quit IRC06:05
*** openstackgerrit has quit IRC06:05
*** dking has quit IRC06:05
*** rockey has quit IRC06:05
*** bbezak has joined #openstack-kolla06:05
*** _Cyclone_ has joined #openstack-kolla06:05
*** gouthamr has joined #openstack-kolla06:05
*** mgoddard has joined #openstack-kolla06:05
*** alanmeadows has joined #openstack-kolla06:05
*** weshay has joined #openstack-kolla06:06
*** v1k0d3n has quit IRC06:07
yoctozeptomorning06:07
*** v1k0d3n has joined #openstack-kolla06:08
*** dking has joined #openstack-kolla06:10
*** rockey has joined #openstack-kolla06:10
*** e0ne has joined #openstack-kolla06:37
*** brinzhang_ has joined #openstack-kolla06:56
*** brinzhang0 has quit IRC07:00
*** mnasiadka has joined #openstack-kolla07:12
mnasiadkamorning07:12
*** shyam89 has quit IRC07:18
*** shyamb has joined #openstack-kolla07:18
*** maharg101 has joined #openstack-kolla07:29
*** rpittau|afk is now known as rpittau07:45
yoctozeptomgoddard: I wonder if it's a chicken-and-egg problem with that ansible PR07:50
yoctozeptomaybe providing unit tests would help...07:51
yoctozepto(just thinking loud)07:51
mgoddardyoctozepto: maybe. Probably it needs someone to go and get involved on IRC or a meeting07:52
mgoddardgithub emails are easier to ignore07:52
*** brinzhang0 has joined #openstack-kolla08:04
*** brinzhang_ has quit IRC08:07
*** shyamb has quit IRC08:18
*** vishalmanchanda has joined #openstack-kolla08:18
*** pbing19 has joined #openstack-kolla08:23
mnasiadkayoctozepto: https://zuul.opendev.org/t/openstack/build/6a1271e587534343bea247cfd9c50294/log/primary/logs/ansible/deploy#21227 - interesting, right?:)08:35
yoctozeptomgoddard: true that; you want to try or want somebody else to?08:37
yoctozeptomnasiadka: what the...08:38
mnasiadkayoctozepto: only in ubuntu upgrade ;)08:38
yoctozeptomnasiadka: still weird08:39
yoctozeptoit's on the deploy step08:39
mnasiadkayoctozepto: yeah, even more weird, let's see if it happens again anywhere08:44
mnasiadkayoctozepto: not speaking that the module could be more... talkative08:44
yoctozeptomnasiadka: I would suspect the retry logic is wrong08:44
yoctozeptoas in the creation has happened08:45
yoctozeptoand have been retried08:45
yoctozeptoand failed08:45
*** Abdallahyas has joined #openstack-kolla08:46
mnasiadkayoctozepto: true, first error is keystoneauth1.exceptions.http.InternalServerError: Internal Server Error (HTTP 500)08:47
mnasiadkaso, error 500 but got created ;)08:47
*** kevko_ has joined #openstack-kolla08:47
*** cah_link has joined #openstack-kolla08:47
*** abdysn has quit IRC08:50
yoctozeptomnasiadka: well :-)08:51
yoctozeptomgoddard: https://review.opendev.org/754285 bad news arriving08:51
patchbotpatch 754285 - kolla-ansible - CI: octavia: create and test a load balancer - 18 patch sets08:51
*** priteau has joined #openstack-kolla08:54
yoctozeptomgoddard: so we basically need a non-neutron, IP-address-assigned, to-provider-network-connected local interface per octavia controller08:59
yoctozeptosounds fun08:59
yoctozeptoLOL08:59
yoctozeptojohnsom: please validate my logic when you are free09:00
kevko_mgoddard: Hi, any new comments regarding https://review.opendev.org/#/c/755775/ ? I've updated patchset...09:01
patchbotpatch 755775 - kolla-ansible - Use native fluent-logger instead of tail - Fluent ... - 3 patch sets09:01
*** jbadiapa has joined #openstack-kolla09:02
nikparasyrhello. using k-a here and i've noticed a difference between placement service and other services in terms of the wsgi config. The placement service does not use with_first_found and always uses the k-a template. we have a use-case where we want to overwrite the placement-api-wsgi config and currently we can only do it directly to k-a. I'm wondering if there is a reason why with_first_found is not used similarly with the other se09:03
nikparasyrrvices such as cinder, horizon, keystone etc. For example, placement: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/placement/tasks/config.yml#L71 keystone: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/keystone/tasks/config.yml#L147. if with_first_found can be used i could prep a patch for this, as this keeps it consistent with other services09:03
yoctozeptonikparasyr: it's a bug, please report at launchpad.net/kolla-ansible - we will handle it from there (and you can also submit a fixing patch to gerrit)09:05
nikparasyrwill do. thanks :)09:05
*** shyamb has joined #openstack-kolla09:09
*** gfidente has joined #openstack-kolla09:13
*** jbalciunas has joined #openstack-kolla09:14
*** born2bake has joined #openstack-kolla09:14
*** k_mouza has joined #openstack-kolla09:15
*** also_stingrayza has joined #openstack-kolla09:18
*** k_mouza has quit IRC09:19
*** k_mouza has joined #openstack-kolla09:19
*** stingrayza has quit IRC09:19
*** Tony31 has joined #openstack-kolla09:27
*** hjensas__ is now known as hjensas09:42
*** openstackgerrit has joined #openstack-kolla09:46
openstackgerritMerged openstack/kayobe master: Switch default docker storage driver to overlay2  https://review.opendev.org/65881509:46
*** priteau has quit IRC09:54
*** Tony31 has quit IRC09:54
*** shyamb has quit IRC09:56
*** shyamb has joined #openstack-kolla10:16
*** shyamb has quit IRC10:33
*** bbezak has quit IRC10:33
*** PrinzElvis has quit IRC10:33
*** shyamb has joined #openstack-kolla10:34
*** bbezak has joined #openstack-kolla10:34
*** PrinzElvis has joined #openstack-kolla10:34
*** vishalmanchanda has quit IRC10:48
mgoddardyoctozepto: correct regarding octavia interface. I meant to have a look at routing. How do we access floating IPs in CI?10:53
mgoddardah, found it10:55
yoctozeptook10:57
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/75428511:01
openstackgerritMark Goddard proposed openstack/kolla-ansible master: add octavia openrc file  https://review.opendev.org/74599711:02
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: octavia: create and test a load balancer  https://review.opendev.org/75428511:02
*** pbing19 has quit IRC11:06
*** pbing19 has joined #openstack-kolla11:07
*** shyamb has quit IRC11:08
*** ricolin_ has quit IRC11:10
*** ricolin has joined #openstack-kolla11:11
*** brinzhang0 has quit IRC11:12
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: enable designate in magnum CI job  https://review.opendev.org/75445611:14
*** Abdallahyas has quit IRC11:21
*** abdysn has joined #openstack-kolla11:22
*** shyamb has joined #openstack-kolla11:22
mgoddardkevko_: looks better with localhost. I'll do a review at some point but it will be a feature for the next release11:25
mgoddardkevko_: I'll comment11:26
mnasiadkayoctozepto: can you look into https://review.opendev.org/#/c/754691/ ?11:29
patchbotpatch 754691 - kolla-ansible - [baremetal]: Use $releasever in docker-ce repo - 2 patch sets11:29
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add log for cron script fernet-rotate.sh  https://review.opendev.org/75608311:41
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Fail on fluentd log parsing errors  https://review.opendev.org/74239911:51
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add log for cron script fernet-rotate.sh  https://review.opendev.org/75608311:55
*** shyamb has quit IRC12:01
*** zzzeek has quit IRC12:17
*** zzzeek has joined #openstack-kolla12:18
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change keepalived logging to syslog  https://review.opendev.org/74511112:19
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Change keepalived logging to syslog  https://review.opendev.org/74511112:19
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Update ironic iPXE configuration  https://review.opendev.org/75499712:21
*** pleimer has joined #openstack-kolla12:23
kevko_mgoddard: ok, for me it is ok , i will port it locally in our company ..for me is important that it WILL be in master ..12:24
yoctozeptomnasiadka: done12:26
*** Luzi has joined #openstack-kolla12:36
dswebbquick question, has anyone mentioned a memory leak with ovs-vswitchd and ovn on ussuri that you all know about?  we're having some serious issues with memory usage on 2.12.0.  seemingly upgrading ovs-vswitchd / db to 2.14 "fixes" the issue12:38
openstackgerritMark Goddard proposed openstack/kayobe master: docs: Add skeleton for scenario docs  https://review.opendev.org/74858212:39
dswebb16770 root      20   0   74.6g  70.9g  18520 S  28.1  14.1  89041:21 ovs-vswitchd12:40
*** pbing19 has quit IRC12:46
yoctozeptowow12:47
yoctozeptothat's huge12:47
yoctozeptomnasiadka: ^12:47
yoctozeptomgoddard: ansible folks confirmed they want us on their meeting ;d12:53
yoctozeptomgoddard: Thursday @ 15 UTC12:54
yoctozeptomgoddard: sounds like skipping kolla klub this time ;o12:54
mnasiadkadswebb: yes, I'm aware, is it centos?13:00
dswebbyep13:00
mnasiadkadswebb: RDO is trying to push update to 2.13 first in Victoria, then Ussuri - that should be enough13:01
mnasiadkadswebb: you could build your own containers and use CentoS NFV SIG repo13:02
dswebbwe built out own repo where we built some ovs packages and we're just patching the upstream containers13:03
dswebbwas just checking if this was a known issue13:03
mnasiadkadswebb: this is one, there are other issues if you use OVN13:04
mnasiadkadswebb: http://mirror.centos.org/centos/8/nfv/x86_64/openvswitch-2/ - this is the CentOS NFV SIG repo with 2.1313:05
dswebbso far this has been only this issue and distributed floating IP doesn't work that we've run into13:05
mnasiadkadswebb: and there is a SNAT issue, sending ARP packets to internet addresses :)13:05
dswebbthanks for the info @mnasiadka13:13
mnasiadkadswebb: it's already in Victoria, TripleO has to fix upgrade jobs, and then they will bump up the version in Ussuri as well13:19
openstackgerritMerged openstack/kolla-ansible master: [baremetal]: Use $releasever in docker-ce repo  https://review.opendev.org/75469113:32
*** TrevorV has joined #openstack-kolla13:39
*** Luzi has quit IRC13:43
*** vishalmanchanda has joined #openstack-kolla13:56
*** abdysn has quit IRC14:00
*** kevko_ has quit IRC14:04
openstackgerritMerged openstack/kayobe master: Disable ironic in the overcloud by default  https://review.opendev.org/75500814:18
openstackgerritMerged openstack/kayobe master: Disable overcloud networks for Ironic when unused  https://review.opendev.org/75501514:19
openstackgerritMerged openstack/kayobe master: dev: fix test scripts when ironic is disabled  https://review.opendev.org/75547414:26
*** e0ne has quit IRC14:52
*** oyrogerg has joined #openstack-kolla15:02
oyrogergRunning 'kolla-ansible -i inv prechecks' I am getting this error: FAILED! => {"msg": "'dict object' has no attribute 'neutron-ovn-metadata-agent'"}15:08
oyrogergA little earlier on I see `[WARNING]: Could not match supplied host pattern, ignoring: neutron-ovn-metadata-agent` and wonder if that is relevant.15:09
oyrogergDo I need to do anything to ensure we're using OVS and not OVN? This is with train.15:14
oyrogergIntersting - this fixed it: Add an empty group [neutron-ovn-metadata-agent:children] to the inv file.15:17
*** rpittau is now known as rpittau|afk15:48
*** k_mouza has quit IRC15:52
*** shyamb has joined #openstack-kolla15:53
*** k_mouza has joined #openstack-kolla16:02
*** pbing19 has joined #openstack-kolla16:02
*** shyamb has quit IRC16:10
openstackgerritRadosÅ‚aw Piliszek proposed openstack/kolla stable/train: Add collectd-libpod-stats to image  https://review.opendev.org/75587316:10
*** shyamb has joined #openstack-kolla16:12
openstackgerritMark Goddard proposed openstack/kolla-ansible master: CI: enable designate in magnum CI job  https://review.opendev.org/75445616:14
mgoddardoyrogerg: if you have upgraded you need to sync your inventory with the new release16:16
*** shyamb has quit IRC16:17
*** shyamb has joined #openstack-kolla16:18
*** shyamb has quit IRC16:26
*** kevko_ has joined #openstack-kolla16:38
*** gfidente is now known as gfidente|afk16:40
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Neutron API Server  https://review.opendev.org/75636717:01
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Add support for keystone auth_mellon config  https://review.opendev.org/75547517:07
*** k_mouza has quit IRC17:25
mnasiadkamgoddard: we could add a precheck to check if inventory has relevant groups? :)17:29
*** k_mouza has joined #openstack-kolla17:32
*** jonaspaulo has joined #openstack-kolla17:33
*** k_mouza has quit IRC17:35
*** maharg101 has quit IRC17:52
*** vishalmanchanda has quit IRC17:56
*** dking has quit IRC17:56
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Neutron API Server  https://review.opendev.org/75636718:01
*** cah_link has quit IRC18:08
mgoddardmnasiadka: already did18:25
mgoddarddoesn't work in every case18:26
mgoddardadded in ussuri I think18:26
*** jonaspaulo has quit IRC18:30
*** k_mouza has joined #openstack-kolla18:33
*** maharg101 has joined #openstack-kolla18:35
*** k_mouza has quit IRC18:40
*** kevko_ has quit IRC18:41
*** maharg101 has quit IRC18:44
*** k_mouza has joined #openstack-kolla18:47
openstackgerritNikolaos Parasyris proposed openstack/kolla-ansible master: Allow overwrite of placement-api wsgi config  https://review.opendev.org/75637918:48
*** k_mouza has quit IRC18:51
*** k_mouza has joined #openstack-kolla18:56
*** k_mouza has quit IRC19:00
*** nikparasyr has left #openstack-kolla19:01
*** kevko_ has joined #openstack-kolla19:02
*** k_mouza has joined #openstack-kolla19:05
*** k_mouza has quit IRC19:09
*** TrevorV has quit IRC19:20
*** k_mouza has joined #openstack-kolla19:23
*** k_mouza has quit IRC19:27
kplantall of my services are down due to rmq, connections are dropping with "missed heartbeats from client, timeout: 60s"19:40
kplanthas that been seen before?19:40
*** e0ne has joined #openstack-kolla19:44
*** k_mouza has joined #openstack-kolla19:51
kplantbringing the whole rmq cluster down and back up fixed it19:51
*** k_mouza has quit IRC19:55
*** gfidente|afk has quit IRC19:56
*** e0ne has quit IRC19:58
*** kevko_ has quit IRC20:08
*** k_mouza has joined #openstack-kolla20:09
*** pbing19 has quit IRC20:09
*** k_mouza has quit IRC20:13
*** zzzeek has quit IRC20:15
*** zzzeek has joined #openstack-kolla20:16
*** kevko has joined #openstack-kolla20:21
*** kevko_ has joined #openstack-kolla20:25
*** kevko has quit IRC20:29
*** maharg101 has joined #openstack-kolla20:41
*** maharg101 has quit IRC20:45
*** k_mouza has joined #openstack-kolla20:52
*** k_mouza has quit IRC20:56
*** k_mouza has joined #openstack-kolla21:05
*** k_mouza has quit IRC21:09
*** maharg101 has joined #openstack-kolla21:25
*** maharg101 has quit IRC21:29
*** kemopq has quit IRC21:49
*** kemopq_ has joined #openstack-kolla21:49
*** pleimer has quit IRC21:54
*** k_mouza has joined #openstack-kolla22:10
*** k_mouza has quit IRC22:15
*** k_mouza has joined #openstack-kolla22:20
*** k_mouza has quit IRC22:24
*** k_mouza has joined #openstack-kolla22:40
*** k_mouza has quit IRC22:45
*** born2bake has quit IRC22:49
*** maharg101 has joined #openstack-kolla23:26
*** maharg101 has quit IRC23:31
*** k_mouza has joined #openstack-kolla23:42
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend Neutron API Server  https://review.opendev.org/75636723:43
*** kevko__ has joined #openstack-kolla23:46
*** k_mouza has quit IRC23:47
*** kevko_ has quit IRC23:49

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!