Wednesday, 2020-03-18

*** tonythomas has quit IRC00:31
*** jbadiapa has quit IRC00:53
*** dave-mccowan has quit IRC00:55
*** boldo has quit IRC01:08
*** ivve has quit IRC01:27
*** xinliang has joined #openstack-kolla01:37
*** bold has joined #openstack-kolla01:41
boldTASK [kibana : Change kibana config to set index as defaultIndex] *************************************************************fatal: [controller1]: FAILED! => {"action": "uri", "changed": false, "connection": "close", "content": "{\"error\":{\"root_cause\":[{\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse\"}],\"type\":\"mapper_parsing_exception\",\"reason\":\"failed to parse\",\"caused_by\":{\"type\":\"i_o_exception\",\"reason\":\"U01:41
boldnexpected character (''' (code 39)): was expecting double-quote to start field name\\n at [Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@310629c; line: 1, column: 3]\"}},\"status\":400}", "content_length": "418", "content_type": "application/json; charset=UTF-8", "elapsed": 0, "json": {"error": {"caused_by": {"reason": "Unexpected character (''' (code 39)): was expecting double-quote to start field name\n at01:41
bold[Source: org.elasticsearch.common.bytes.BytesReference$MarkSupportingStreamInputWrapper@310629c; line: 1, column: 3]", "type": "i_o_exception"}, "reason": "failed to parse", "root_cause": [{"reason": "failed to parse", "type": "mapper_parsing_exception"}], "type": "mapper_parsing_exception"}, "status": 400}, "msg": "Status code was 400 and not [200, 201]: HTTP Error 400: Bad Request", "redirected": false, "status": 400, "url": "http://192.168.10.65:92001:41
bold0/.kibana/config/*"}01:41
boldhow to fix this error. it occurs when running kolla-ansible -i ./multinode deploy01:43
*** jimcrowleyibm has joined #openstack-kolla01:53
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827302:36
*** factor has quit IRC02:52
*** ricolin_ has joined #openstack-kolla03:00
*** diurnalist has joined #openstack-kolla03:17
*** diurnalist has quit IRC03:22
*** zhanglong has joined #openstack-kolla03:38
*** ykarel|away is now known as ykarel03:45
*** xinliang has quit IRC04:05
*** skramaja has joined #openstack-kolla04:15
*** negronjl has quit IRC04:29
*** jimcrowleyibm has quit IRC04:32
*** negronjl has joined #openstack-kolla04:32
*** jimcrowleyibm has joined #openstack-kolla04:37
*** jimcrowleyibm is now known as jimcrowleyibm[aw04:37
*** jimcrowleyibm has joined #openstack-kolla04:39
*** jimcrowleyibm is now known as jimcrowleyibm[aw04:39
*** jimcrowleyibm has joined #openstack-kolla04:41
*** jimcrowleyibm is now known as jimcrowleyibm[aw04:41
*** jimcrowleyibm[aw is now known as jimcrowleyibm04:41
*** jimcrowleyibm is now known as jimcrowleyibm[aw04:42
*** ricolin_ has quit IRC05:05
*** ricolin_ has joined #openstack-kolla05:05
*** ricolin_ has quit IRC05:05
*** k_mouza has joined #openstack-kolla05:32
*** evrardjp has quit IRC05:35
*** evrardjp has joined #openstack-kolla05:36
*** k_mouza has quit IRC05:36
*** suff has joined #openstack-kolla05:55
*** skramaja has quit IRC06:38
*** pbing19 has joined #openstack-kolla06:39
*** skramaja has joined #openstack-kolla06:46
openstackgerritMichal Nasiadka proposed openstack/kolla master: WIP: Add mariadb_clustercheck image  https://review.opendev.org/71021706:57
*** suff has quit IRC07:07
openstackgerritMichal Nasiadka proposed openstack/kolla master: neutron: Remove networking-ovn - part 2  https://review.opendev.org/71342207:07
*** suff has joined #openstack-kolla07:08
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684107:10
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed  https://review.opendev.org/70968907:14
bold{"msg": "The conditional check 'inventory_hostname in groups['ceph-mon'] or inventory_hostname in groups['cinder-api'] or inventory_hostname in groups['cinder-volume'] or inventory_hostname in groups['cinder-scheduler'] or inventory_hostname in groups['cinder-backup']' failed. The error was: error while evaluating conditional (inventory_hostname in groups['ceph-mon'] or inventory_hostname in groups['cinder-api'] or inventory_hostname in07:15
boldgroups['cinder-volume'] or inventory_hostname in groups['cinder-scheduler'] or inventory_hostname in groups['cinder-backup']): Unable to look up a name or access an attribute in template string ({% if inventory_hostname in groups['ceph-mon'] or inventory_hostname in groups['cinder-api'] or inventory_hostname in groups['cinder-volume'] or inventory_hostname in groups['cinder-scheduler'] or inventory_hostname in groups['cinder-backup'] %} True {% else07:15
bold%} False {% endif %}).\nMake sure your variable name does not contain invalid characters like '-': argument of type 'AnsibleUndefined' is not iterable\n\nThe error appears to be in '/usr/local/share/kolla-ansible/ansible/roles/cinder/tasks/deploy.yml': line 5, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem.\n\nThe offending line appears to be:\n\n\n- include_tasks: config.yml\n  ^ here\n"}07:15
boldhow to fix this?07:16
mnasiadkafirst use paste.openstack.org, then paste the output WITH playbook step name :-)07:23
*** cah_link has joined #openstack-kolla07:30
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Add ansible-lint to tox  https://review.opendev.org/69477907:32
mnasiadkaINFO:kolla.common.utils.horizon:    scss/src/scanner.h:17:10: fatal error: pcre.h: No such file or directory07:38
mnasiadkaugh07:38
yoctozeptowoot07:40
yoctozeptono day w/o failure ;o07:40
yoctozeptolooks like horizon finally switch pyscss to pyscss207:46
*** bold has quit IRC07:46
yoctozeptoand we have no binary wheels for it07:46
yoctozeptoand fail the compilation07:46
yoctozeptooh well07:46
*** boldo has joined #openstack-kolla07:47
yoctozepto"Retrying the build without the C extension now."07:47
yoctozeptoyeah, sure, and still try to compile C07:47
yoctozeptowell done07:47
*** boldo has quit IRC07:49
*** riuzen has joined #openstack-kolla08:06
riuzenwhen we deploy baremetal using Ironic, did kolla-ansible also provide dhcp-server for PXE or we should create first?08:08
*** bengates has joined #openstack-kolla08:09
*** pbing19 has quit IRC08:13
*** sorin-mihai has quit IRC08:13
*** pbing19 has joined #openstack-kolla08:13
*** yoctozepto9 has joined #openstack-kolla08:15
*** yoctozepto has quit IRC08:15
*** yoctozepto9 is now known as yoctozepto08:15
*** sorin-mihai has joined #openstack-kolla08:16
*** bold has joined #openstack-kolla08:16
openstackgerritRadosław Piliszek proposed openstack/kolla master: Fix Horizon build after switch to pyscss2 which lacks binary wheel  https://review.opendev.org/71358908:16
yoctozeptoriuzen: k-a provides it08:17
*** abdysn has joined #openstack-kolla08:27
openstackgerritRadosław Piliszek proposed openstack/kolla master: DNM: No binary wheels  https://review.opendev.org/71359108:31
openstackgerritRadosław Piliszek proposed openstack/kolla master: neutron: Remove networking-ovn - part 2  https://review.opendev.org/71342208:31
*** abdysn has quit IRC08:43
*** abdysn has joined #openstack-kolla08:43
*** rpittau|afk is now known as rpittau08:45
*** yoctozepto has quit IRC08:49
hrwmorning08:52
*** strigazi has joined #openstack-kolla08:57
*** yoctozepto has joined #openstack-kolla08:59
mnasiadkayoctozepto: just thinking, is infra going to rework linters job to not use tox? :)09:04
mnasiadkaah, they have a step to ensure tox is installed09:05
mnasiadka:)09:05
mnasiadkahrw, yoctozepto, mgoddard: anybody planning to fix missing pcre.h in horizon, or should I?09:06
hrwmnasiadka: 09:16 < openstackgerrit> Radosław Piliszek proposed openstack/kolla master: Fix Horizon build after switch to pyscss2 which lacks binary wheel  https://review.opendev.org/71358909:08
patchbotpatch 713589 - kolla - Fix Horizon build after switch to pyscss2 which la... - 1 patch set09:08
mnasiadkaah, ok - me blind as always09:08
*** ykarel is now known as ykarel|away09:08
hrwmnasiadka: so do not be lazy and +2+w09:09
mnasiadkahrw: will do, once kolla-build-centos8/debian jobs will pass in zuul :)09:09
*** zhanglong has quit IRC09:09
hrwI trust zuul more then ;d09:10
hrwhttps://review.opendev.org/#/c/710229/ anyone?09:10
patchbotpatch 710229 - kolla - version-check: handle more situations - 1 patch set09:10
mgoddardbold: hi. I think I have a fix for you09:12
mgoddardbold: which release/branch are you using?09:12
mgoddardbold: https://review.opendev.org/#/c/709090/09:13
patchbotpatch 709090 - kolla-ansible - Fix invalid JSON body in Elasticsearch API requests - 1 patch set09:13
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313509:20
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313509:22
hrwthis one removes installing tox as well09:22
mgoddardyoctozepto, mnasiadka, hrw: https://review.opendev.org/#/c/709090 may fix bold's issue - care to take a look?09:28
patchbotpatch 709090 - kolla-ansible - Fix invalid JSON body in Elasticsearch API requests - 1 patch set09:28
hrw+W09:30
hrwmnasiadka: https://review.opendev.org/#/c/713272/09:31
patchbotpatch 713272 - kolla-ansible - CentOS 8: Use master image tag by default - 2 patch sets09:31
mgoddardhrw: actually wait09:31
mgoddardpublish job failed this morning09:31
mgoddardso master is still centos709:31
hrwah.09:32
riuzen<yoctozepto> oke found it.09:32
mgoddardhrw: although, the other images were published. So we are in the strange position that all images except horizon are centos809:33
hrw;D09:33
hrwmgoddard: it's master. images are random state by default09:34
mnasiadkamgoddard: I guess it doesn't really hurt horizon is c7 :)09:34
mgoddardprobably not09:34
mgoddardI'm sure we'll soon find out09:34
*** jbadiapa has joined #openstack-kolla09:36
*** admin0 has quit IRC09:39
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: elasticsearch: use proper repo for debian/aarch64  https://review.opendev.org/71359809:43
hrwhttps://review.opendev.org/713135 + https://review.opendev.org/713598 == aarch64 working again09:43
patchbotpatch 713135 - kolla - CI: Avoid tox - 27 patch sets09:43
patchbotpatch 713598 - kolla - elasticsearch: use proper repo for debian/aarch64 - 1 patch set09:43
openstackgerritMark Goddard proposed openstack/kolla master: Revert "mark bifrost as unbuildable"  https://review.opendev.org/71350609:44
yoctozeptohrw: actually only the 2nd one as we now install tox which brings us setuptools as well09:44
hrwyoctozepto: anyway we can merge both09:45
hrwyoctozepto: INFO:kolla.common.utils.elasticsearch:Get:2 https://artifacts.elastic.co/packages/5.x/apt stable/main amd64 elasticsearch all 5.6.16 [33.8 MB]09:46
hrwyoctozepto: INFO:kolla.common.utils.elasticsearch:Setting up elasticsearch (5.6.16) ...09:46
hrwyoctozepto: INFO:kolla.common.utils.elasticsearch:Successfully tagged master1232/debian-source-elasticsearch:9.1.009:47
hrwyoctozepto: the [arch=amd64] is a trick09:47
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: Combined fluentd fixes  https://review.opendev.org/71351909:49
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684109:49
openstackgerritRadosław Piliszek proposed openstack/kolla master: DNM: No binary wheels  https://review.opendev.org/71359109:51
yoctozeptohrw: ah, I see you rebased to remove tox, well done09:52
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add framework to deploy user-defined containers on seed  https://review.opendev.org/70968909:52
hrwyoctozepto: avoid is avoid, right? :D09:53
hrw2020-03-18 09:51:44.443903 | primary | INFO:kolla.common.utils.elasticsearch:Successfully tagged kolla/debian-source-elasticsearch:9.1.0 ;D09:56
hrwworks on CI09:56
*** bjoernt_ has joined #openstack-kolla09:58
*** yoctozepto has quit IRC09:59
*** yoctozepto has joined #openstack-kolla10:03
*** bjoernt has quit IRC10:04
yoctozeptohrw: right10:05
yoctozeptosorry, folks, it seems my network connectivity is grumpy todya10:05
openstackgerritMark Goddard proposed openstack/kolla master: Revert "mark bifrost as unbuildable"  https://review.opendev.org/71350610:05
hrwyoctozepto: working from home?10:06
yoctozeptohrw: yeah, but with uni vpn10:08
yoctozeptohome isp is fine10:08
openstackgerritMark Goddard proposed openstack/kolla master: Revert "mark bifrost as unbuildable"  https://review.opendev.org/71350610:10
*** yoctozepto67 has joined #openstack-kolla10:11
yoctozepto67let's do it webchat10:12
*** ChanServ sets mode: +o yoctozepto10:13
hrwyoctozepto: in a need of videoconf you are?10:13
*** yoctozepto2 has joined #openstack-kolla10:13
*** yoctozepto has quit IRC10:13
*** yoctozepto2 is now known as yoctozepto10:13
yoctozepto67hrw: no, why?10:15
*** ChanServ sets mode: +o yoctozepto6710:15
*** yoctozepto67 sets mode: -b *senk*!*@*10:16
*** ChanServ sets mode: -o yoctozepto6710:16
hrwjoking10:16
yoctozepto67hrw: suche ;p10:17
*** tonythomas has joined #openstack-kolla10:19
hrwhttps://review.opendev.org/#/c/713492/ any brave cores?10:21
patchbotpatch 713492 - kolla (stable/stein) - kibana, nova-libvirt: handle it properly for non-x86 - 1 patch set10:21
hrwand https://review.opendev.org/#/c/71349110:22
patchbotpatch 713491 - kolla (stable/train) - kibana, nova-libvirt: handle it properly for non-x86 - 1 patch set10:22
yoctozepto67hrw: is this final? as in both merged?10:22
hrwfinal10:22
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313510:27
openstackgerritMark Goddard proposed openstack/kayobe master: Remove ceph block device support  https://review.opendev.org/70937610:28
openstackgerritRadosław Piliszek proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313510:29
openstackgerritMark Goddard proposed openstack/kolla stable/train: CI: Install tox  https://review.opendev.org/71361610:32
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CI: install tox  https://review.opendev.org/71361710:32
openstackgerritMark Goddard proposed openstack/kolla stable/stein: CI: Install tox  https://review.opendev.org/71361810:32
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/stein: CI: install tox  https://review.opendev.org/71361910:33
openstackgerritMark Goddard proposed openstack/kolla stable/rocky: CI: Install tox  https://review.opendev.org/71362010:35
hrwmgoddard: so you remember https://review.opendev.org/#/c/692996 patch?10:37
patchbotpatch 692996 - kolla (stable/stein) - Stein: support mariabackup for backups (MERGED) - 1 patch set10:37
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/rocky: CI: install tox  https://review.opendev.org/71362110:38
mgoddardhrw: yes10:38
hrwmgoddard: I have a feeling that it was incomplete. will have to dig more in stein commits10:39
mgoddardhrw: have you tried to use it?10:40
hrwmgoddard: https://bugs.launchpad.net/kolla/+bug/186775010:40
openstackLaunchpad bug 1867750 in kolla "Stein Debian MariaDB cluster failed to deploy on multi-node" [Undecided,Confirmed] - Assigned to Marcin Juszkiewicz (hrw)10:40
hrwmgoddard: no mariabackup on aarch64.10:40
mgoddardhrw: :(10:42
mgoddardis the mariadb package a different version?10:42
yoctozeptohrw: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_539/713591/2/check-arm64/kolla-build-debian-source-aarch64/539853b/kolla/build/000_FAILED_openvswitch-netcontrold.log10:42
hrwmgoddard: 10.3 should be on all archs10:42
yoctozeptolooks like this one went wheel in aarch6410:42
yoctozeptoas it did not fail previously10:42
hrwyoctozepto: this image is wrong on so many levels...10:43
yoctozeptosure, but the point is even on aarch64 some must have come from wheels10:43
hrwyoctozepto: or aarch64 was failing before we added that image10:44
hrwbuilding locally10:45
hrwINFO:kolla.common.utils.openvswitch-netcontrold:Collecting netcontrold10:47
hrwINFO:kolla.common.utils.openvswitch-netcontrold:  Downloading https://files.pythonhosted.org/packages/60/fc/171a207c50add7eaabb81cefb6c18b5cef82836db7184fd1de42b7fa9cb8/netcontrold-1.0.15-py3-none-any.whl10:47
hrwINFO:kolla.common.utils.openvswitch-netcontrold:Installing collected packages: netcontrold10:47
openstackgerritMerged openstack/kolla-ansible master: Fix invalid JSON body in Elasticsearch API requests  https://review.opendev.org/70909010:47
hrwyoctozepto: ok. I was not exact enough.10:48
hrwyoctozepto: on aarch64 we do binary wheels for 'arch:none' ones and build/compile all arch specific ones10:48
stingrayzahi all o/10:49
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Build bifrost image  https://review.opendev.org/71350910:54
yoctozepto67hrw: ack, thanks, makes sense11:01
*** Fl1nt has joined #openstack-kolla11:03
hrwyoctozepto67: arch:none are simply python text files11:04
*** k_mouza has joined #openstack-kolla11:04
yoctozepto67hrw: well, theoretically you could require some external preprocessor or something11:05
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: elasticsearch: use proper repo for debian/aarch64  https://review.opendev.org/71359811:09
hrwon top of horizon11:09
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera  https://review.opendev.org/71021311:12
mnasiadkahrw: everything today is on top of horizon :D11:26
hrwyesterday was on top of bifrost11:27
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684111:29
openstackgerritMark Goddard proposed openstack/kolla master: Revert "mark bifrost as unbuildable"  https://review.opendev.org/71350611:34
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362811:35
openstackgerritMerged openstack/kolla stable/stein: kibana, nova-libvirt: handle it properly for non-x86  https://review.opendev.org/71349211:37
*** k_mouza has quit IRC11:40
*** k_mouza has joined #openstack-kolla11:44
*** ivve has joined #openstack-kolla11:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684111:49
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362811:49
*** diurnalist has joined #openstack-kolla11:49
*** diurnalist has quit IRC11:54
mnasiadkayoctozepto67: https://bugs.launchpad.net/kolla-ansible/+bug/1731142 - I think we dumped clusterer in stein in favor of some better method?11:56
openstackLaunchpad bug 1731142 in kolla-ansible rocky "The extended control node is rabbitmq error reporting" [Low,Confirmed]11:56
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362811:58
*** rpittau is now known as rpittau|bbl12:01
hrwyoctozepto67: horizon will get binary wheels for pyscss2 soon12:05
*** pbing19 has quit IRC12:05
hrw    kolla-build-debian-source-aarch64 SUCCESS in 44m 58s (non-voting)12:05
hrwhttps://review.opendev.org/#/c/713598/ ready for +W :D12:05
patchbotpatch 713598 - kolla - elasticsearch: use proper repo for debian/aarch64 - 2 patch sets12:05
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684112:05
*** zhanglong has joined #openstack-kolla12:10
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362812:11
*** jbadiapa has quit IRC12:19
mnasiadkahrw: so just +w, you have 2 +2s ;)12:22
hrwmnasiadka: I tend to not +W own patches ;D12:23
mnasiadkaoh boy, +w'ed ;)12:23
hrwprobably wrong but that's me12:24
*** k_mouza has quit IRC12:32
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/train: Fix Horizon build after switch to pyscss2 which lacks binary wheel  https://review.opendev.org/71364412:39
*** yoctozepto67 has quit IRC12:46
*** k_mouza has joined #openstack-kolla12:47
openstackgerritMerged openstack/kolla master: Fix Horizon build after switch to pyscss2 which lacks binary wheel  https://review.opendev.org/71358912:50
openstackgerritJie Li proposed openstack/kolla-ansible stable/rocky: add region_name if enable barbican  https://review.opendev.org/71364612:53
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/stein: mariadb: install mariabackup on Debian  https://review.opendev.org/71364712:54
*** rpittau|bbl is now known as rpittau13:01
openstackgerritMerged openstack/kolla stable/train: kibana, nova-libvirt: handle it properly for non-x86  https://review.opendev.org/71349113:02
*** obre has quit IRC13:08
*** skramaja has quit IRC13:14
*** obre has joined #openstack-kolla13:15
*** vishalmanchanda has joined #openstack-kolla13:19
openstackgerritJie Li proposed openstack/kolla-ansible stable/rocky: add region_name if enable barbican  https://review.opendev.org/71364613:21
*** jbadiapa has joined #openstack-kolla13:24
mnasiadkayoctozepto: do we have any haproxy logs anywhere in CI? I see it's in theory configured to send syslog messages to {{ api_interface_address }}13:26
yoctozeptomnasiadka: yeah, but before fluentd fix it went nowhere13:30
yoctozeptomnasiadka: should be there now13:30
mnasiadkayoctozepto: where? logs/haproxy ?13:30
mnasiadkaah, we don't start fluentd in mariadb CI scenario13:31
*** zhanglong has quit IRC13:31
*** zhanglong has joined #openstack-kolla13:32
openstackgerritJie Li proposed openstack/kolla-ansible master: add region_name if enable barbican  https://review.opendev.org/71365213:34
*** TrevorV has joined #openstack-kolla13:36
*** umbSublime_ has joined #openstack-kolla13:42
openstackgerritMerged openstack/kolla master: neutron: Remove networking-ovn - part 2  https://review.opendev.org/71342213:42
openstackgerritMerged openstack/kolla stable/stein: CI: Install tox  https://review.opendev.org/71361813:42
openstackgerritMerged openstack/kolla stable/rocky: CI: Install tox  https://review.opendev.org/71362013:42
*** k_mouza has quit IRC13:43
*** spiette_ has joined #openstack-kolla13:43
*** spiette has quit IRC13:44
*** umbSublime has quit IRC13:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: OVN Support  https://review.opendev.org/69684113:46
mnasiadkanetworking-ovn begone13:46
*** k_mouza has joined #openstack-kolla13:48
*** jimcrowleyibm has joined #openstack-kolla13:53
*** jimcrowleyibm has joined #openstack-kolla13:54
yoctozeptomnasiadka: then we rather should13:55
*** zhanglong has quit IRC13:57
*** zhanglong has joined #openstack-kolla13:58
*** diurnalist has joined #openstack-kolla14:02
*** zhanglong has quit IRC14:02
*** pbing19 has joined #openstack-kolla14:08
*** vedup19 has joined #openstack-kolla14:10
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362814:11
*** yoctozepto68 has joined #openstack-kolla14:11
yoctozepto68arar14:11
*** yoctozepto1 has joined #openstack-kolla14:12
mnasiadkayoctozepto: yeah14:12
*** yoctozepto has quit IRC14:13
*** yoctozepto1 is now known as yoctozepto14:13
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362814:13
*** pbing19 has quit IRC14:14
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366314:16
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366314:19
mnasiadkayoctozepto68: I see it would make sense to handle per-job globals in a different way, e.g. changing tests/templates/globals-default.j2 does not trigger mariadb job in this case - maybe we should move that to Zuul vars?14:21
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: WIP: Custom haproxy script for monitoring galera  https://review.opendev.org/71021314:26
openstackgerritMerged openstack/kolla master: elasticsearch: use proper repo for debian/aarch64  https://review.opendev.org/71359814:27
yoctozepto68mnasiadka: yeah, I consider it part of a bigger problem of testing sometimes too much, sometimes not enough14:27
mnasiadkayoctozepto68: let's focus on ,,not enough''14:27
mnasiadka:)14:27
yoctozepto68mnasiadka: yeah, we can do like devstack and interpret some pattern from zuul vars14:28
yoctozepto68mnasiadka: would help in this particular case14:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366314:29
*** sluna has quit IRC14:29
mnasiadkayoctozepto68: I was rather thinking of including something like {{globals_extra}} in the template, and feed this var to zuul base job configs - I'll try to post something tomorrow14:29
*** sluna has joined #openstack-kolla14:29
yoctozepto68mnasiadka: good, I'm open for all ideas14:30
*** yoctozepto68 has quit IRC14:31
hrwhttps://review.opendev.org/#/c/713616/ +1 from zuul14:34
patchbotpatch 713616 - kolla (stable/train) - CI: Install tox - 1 patch set14:34
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: Avoid tox  https://review.opendev.org/71313514:35
hrwhorizon rebase14:35
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: DNM: No binary wheels  https://review.opendev.org/71359114:37
hrwyoctozepto: added make14:37
yoctozeptohrw: ++14:39
hrwyoctozepto: neutron-base needs binary wheels14:39
hrwtextfsm is wheel only. no tarbakk14:39
Fl1ntHum, I just stumble upon the following error deploying rally: standard_init_linux.go:211: exec user process caused “exec format error”14:41
Fl1ntI build the rally image14:41
Fl1ntfrom CentOS 7 binary package14:41
hrwFl1nt: wrong arch14:41
Fl1ntusing openstack-rally-1.1.0-2.el714:41
Fl1ntOo14:41
Fl1ntcan't be14:42
Fl1ntI build the image using x86_6414:42
hrwlog please14:42
yoctozeptohrw: lol, non-open-source? woot?14:42
hrwyoctozepto: https://github.com/google/textfsm14:44
Fl1nt@hrw, kolla-build log ?14:44
hrwFl1nt: yes14:44
Fl1ntok, hold on, I need to trigger a build (CICD stuff all around...).14:44
Fl1ntfrom kolla-build.conf base_arch = x86_6414:45
Fl1ntall other images such as elasticsearch nova etc are fine14:45
hrwFl1nt: you may skip setting base_arch. it is assumed to be native14:45
Fl1ntok, cool, another default to get ride of ^^ lovin it :D14:46
hrwFl1nt: you need distro/install_type and that's basically all. logs-dir is handy14:48
hrwFl1nt: https://github.com/hrw/docker-utils/blob/master/kolla-build-images.sh14:48
hrwFl1nt: clean_package_cache=false is good for CI builds. images are bigger but build faster14:48
yoctozeptohrw: I just realized we might be getting arbitrarily lower versions with wheels turned off, especially regarding transitive deps14:50
yoctozeptooh what a mess14:51
hrwyoctozepto: ? how14:51
Fl1nt@hrw, yep, already activated the clean package option as we have more than required room for our build and building faster is cool as for now it take us around 30mins per build.14:52
hrwwhich takes me to finishing patch for opendev ci doing same14:52
Fl1ntIs the OpenDev CI zuul ?14:54
hrwyes14:55
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: do not clean package cache if not publishing  https://review.opendev.org/71367014:55
yoctozeptohrw: if you can skip publishing tarball and publish only-wheel versions14:56
yoctozeptoand then run --no-binary14:56
hrwmgoddard mnasiadka hrw egonzalez yoctozepto rafaelweingartne cosmicsound osmanlicilegi: meeting soon14:56
Fl1ntAt some point, I'll have to look closer at zuul, it's a topic of interest but I never had time to look at it closer.14:57
hrwFl1nt: you use jenkins?14:57
Fl1ntyep14:57
hrwFl1nt: jenkins job builder or config-in-jenkins?14:57
Fl1ntJenkins pipeline multi-branch14:57
hrwyoctozepto: they have issue opened about lack of tarball14:58
hrwhm. three meetings to attend in 2 minutes14:58
yoctozeptohrw: sure, but any pypi project can skip tarball14:58
yoctozeptohrw: three? how so?14:58
hrwkolla, horizon and company...14:58
yoctozeptooh my14:58
yoctozeptohrw now horizon core?14:59
hrwyoctozepto: no. improved CI for pyscss2 today14:59
*** priteau has joined #openstack-kolla15:00
mgoddard#startmeeting kolla15:00
openstackMeeting started Wed Mar 18 15:00:23 2020 UTC and is due to finish in 60 minutes.  The chair is mgoddard. Information about MeetBot at http://wiki.debian.org/MeetBot.15:00
openstackUseful Commands: #action #agreed #help #info #idea #link #topic #startvote.15:00
*** openstack changes topic to " (Meeting topic: kolla)"15:00
openstackThe meeting name has been set to 'kolla'15:00
mgoddard#topic rollcall15:01
*** openstack changes topic to "rollcall (Meeting topic: kolla)"15:01
mgoddard\o15:01
osmanlicilegio/15:01
stackedsax\o15:01
hrwo/15:02
yoctozeptoo/15:02
*** vedup19 has quit IRC15:02
yoctozeptosorry for breaking the chain15:02
yoctozepto:D15:02
generalfuzzo/15:02
mgoddard#topic agenda15:02
*** openstack changes topic to "agenda (Meeting topic: kolla)"15:02
mgoddard* Roll-call15:02
mgoddard* Announcements15:02
mgoddard* Review action items from last meeting15:03
mgoddard* CI status15:03
mgoddard* Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig15:03
mgoddard* Ussuri release planning (kayobe)15:03
mgoddard* Ussuri release planning (kolla & kolla ansible)15:03
mgoddard* Stein/Train backports15:03
mgoddard  - https://bugs.launchpad.net/kolla/+bug/1867365 - on aarch64 we lack mariabackup in mariadb image. Train has it as we dropped Percona. Backporting to Stein == dropping Percona which makes a change on x86-64 or adding 'if !x86 then install mariadb-backup' block.15:03
openstackLaunchpad bug 1867365 in kolla train "nova-libvirt needs fixed libvirt on AArch64 servers with ThunderX cpu" [High,Fix committed] - Assigned to Marcin Juszkiewicz (hrw)15:03
mgoddard#topic announcements15:03
*** openstack changes topic to "announcements (Meeting topic: kolla)"15:03
hrwmariadb part handled by https://review.opendev.org/713647 basically15:03
patchbotpatch 713647 - kolla (stable/stein) - mariadb: install mariabackup on Debian - 1 patch set15:03
mgoddard#info Kolla CentOS 8 images now tagged as master on master15:03
yoctozeptoexcept horizon15:03
mgoddardtrue15:03
yoctozeptobut it looks worky15:04
mgoddardAny other announcements?15:04
mgoddard#topic Review action items from last meeting15:04
*** openstack changes topic to "Review action items from last meeting (Meeting topic: kolla)"15:04
mgoddardmgoddard to raise a bug on Ceph job fails15:04
mgoddardmnasiadka to look into Ceph job fails15:04
mgoddardI did not do mine15:04
yoctozeptobut mnasiadka did his15:05
mnasiadkaFixed15:05
yoctozeptoat least master got nice15:05
yoctozeptoworse for older branches still15:05
yoctozeptoso not fully15:05
mgoddardyeah, I think this was about kolla ceph not ceph-ansible?15:05
mnasiadkaYeah, need some more time on this15:05
yoctozeptosomething went wrong regarding our ceph deploys15:05
yoctozeptoyeah, mgoddard ++15:05
mnasiadkamgoddard: but c-a also broke :)15:05
mgoddardindeed15:05
yoctozeptoindeed :D15:05
mgoddard#action mgoddard to raise a bug on Ceph job fails15:06
mgoddard#action mnasiadka to look into Ceph job fails15:06
mgoddard#topic CI status15:06
*** openstack changes topic to "CI status (Meeting topic: kolla)"15:06
mgoddardWe had a horizon issue, has the fix merged yet?15:07
hrwyes15:07
hrwmaster done, train in queue15:07
yoctozeptotrain not broken15:07
yoctozeptoas we pin releases15:07
yoctozeptoso merge any time15:07
hrwhorizon merges pyscss2 into train...15:08
yoctozeptohrw: but we have pinned releases15:08
hrwso once we update we break15:08
yoctozeptohrw: we will be blocked from updating :-)15:08
mgoddardOtherwise we're good15:08
yoctozeptoseems so15:08
mgoddardKayobe seems green15:08
openstackgerritGrzegorz Sikorski proposed openstack/kolla-ansible stable/rocky: rabbitmq: update cluster version on reconfigure  https://review.opendev.org/71362815:09
hrwyoctozepto: right15:09
yoctozeptorocky nearing em15:09
yoctozeptobut folks sending patches ;D15:09
mgoddard#topic Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig15:09
*** openstack changes topic to "Kolla --SIG-- (aka Kolla Klub?) https://etherpad.openstack.org/p/kolla-sig (Meeting topic: kolla)"15:09
yoctozeptolet's kickstart this soon15:09
mgoddard+115:09
mgoddardI think we have around 20 people signed up now15:09
mgoddard#action mgoddard to spam people about kolla SIG15:10
mgoddardWill try to get a few more signed up15:10
mgoddardThen doodle poll for a slot for first meeting15:10
Fl1ntcan I join the SIG part of the meeting?15:11
hrwFl1nt: anyone can15:11
mgoddardFl1nt: anyone can join in the meeting15:11
Fl1ntI'm in for this part so.15:11
mgoddardgreat to hear15:11
cosmicsoundo/15:12
yoctozeptoFl1nt: any part of the meeting :-) as long as on topic15:12
Fl1ntOk, noticed that.15:12
mgoddardI don't think we have too much more to say on it at the moment - just need to get it started15:13
mgoddard#topic Ussuri release planning (kayobe)15:13
*** openstack changes topic to "Ussuri release planning (kayobe) (Meeting topic: kolla)"15:13
*** dougsz has joined #openstack-kolla15:14
mgoddardpriteau, dougsz: \o15:15
priteauo/15:15
dougsz\o/15:15
mgoddardwe are about 5-6 weeks from feature freeze15:15
openstackgerritMerged openstack/kolla-ansible stable/stein: Combined fluentd fixes  https://review.opendev.org/71351915:16
openstackgerritMerged openstack/kolla-ansible stable/rocky: Wait for MariaDB to be accessible via HAProxy  https://review.opendev.org/71350015:16
mgoddardso we need to be finishing things off fairly soon15:16
mgoddardwe have quite a few patches in flight needing reviews15:16
dougszsorry, are we talking kayobe or kolla here?15:16
mgoddardtopic: Ussuri release planning (kayobe)15:16
dougszwill look at review list later15:17
priteauI saw you flagged some patches as Review-Priority, I will try to review ASAP15:17
mgoddardthanks15:17
mgoddardright, just trying to provide some guidance on priority15:17
mgoddarddo either of you have features you would like to be included in ussuri?15:18
mgoddardwill there be more cells work in kayobe?15:18
priteauThe nclu patch might be nice to have15:18
mgoddard+115:18
dougsznot sure yet about cells, focus is on kolla-ansible patches15:19
yoctozepto( dougsz: yeah, I switched order to let you have more time in case we get too chatty )15:19
mgoddardgood thinking15:20
mgoddardI think jovial was going to look at customising kayobe commands, but he's not here15:21
*** jovial[m] has joined #openstack-kolla15:21
mgoddardwow, summoned by name15:21
mgoddardjovial[m]: hi, welcome back15:21
*** vedup19 has joined #openstack-kolla15:21
mgoddardare you still planning to look at customising kayobe commands?15:22
jovial[m]hi, apologies for missing the start, got carried away writing some docs!15:22
jovial[m]yes, I am still keen on looking at that :)15:23
mgoddardok. As I mentioned earlier we are about 5-6 weeks from feature freeze15:23
jovial[m]no progress to report yet though15:23
mgoddardwhich will no doubt go quite quickly15:23
mgoddardno problem, I know you've been busy15:23
mgoddardThanks, let's move onto kolla.15:24
mgoddard#topic Ussuri release planning (kolla & kolla ansible)15:24
*** openstack changes topic to "Ussuri release planning (kolla & kolla ansible) (Meeting topic: kolla)"15:24
mgoddardI'll kick off with CentOS 815:24
mgoddardI added some acceptance criteria to https://etherpad.openstack.org/p/kolla-centos815:24
mgoddardSo we'll have a better idea of when we're finished15:25
mgoddardPlease read through and amend or add as necessary15:25
mgoddardI think kolla is quite close now - just a few last images to tidy up15:25
mgoddardkolla ansible also not too far off15:26
*** abdysn has quit IRC15:26
mgoddardDocumentation and release notes still required15:26
mgoddardAnd some train backprots15:27
mgoddardThat's probably all I have to say on it15:27
mgoddardAny thoughts?15:28
mgoddardAnything missing from acceptance criteria?15:28
* yoctozepto thinking15:29
*** vishalmanchanda has quit IRC15:31
jovial[m]looks reasonable to me at first glance15:32
yoctozepto++15:32
yoctozeptono more ideas, I reworded some parts and switched order15:33
yoctozeptowe are looking good15:33
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Revert "openstack-base: pin setuptools < 46 to get horizon working"  https://review.opendev.org/71368115:33
mgoddardcool15:34
mgoddardAnyone interested in picking up any remaining CentOS 8 work?15:34
mgoddardI thought that would make it go quiet15:35
mgoddardok, let's move on15:35
mgoddardAny other Ussuri features we need to discuss?15:36
generalfuzzI would like to have a review of https://review.opendev.org/#/c/712005 - first pass at backend TLS implemented for keystone using uwsgi web server15:36
patchbotpatch 712005 - kolla-ansible - Add support for encrypting backend HAProxy traffic - 9 patch sets15:36
mgoddardthat's a reasonable request15:37
mgoddardwhat has been your experience of uwsgi so far generalfuzz ?15:38
generalfuzzmostly fine and straight forward. There are a number of peer reset exceptions always show in the log, but functionality seems to work15:38
generalfuzzthe "peer reset" seems to be a common issue with uwsgi and tls15:39
generalfuzzwe will need to make uwsgi available in tripleo builds15:40
mgoddardtripleo can provide config to remove the uwsgi package if they wish15:40
mgoddardwhat is the cause of the peer reset error?15:41
yoctozeptopeer reset sounds bad15:41
generalfuzzit does, but seems like many people hit this with no functionality issues15:41
generalfuzzstill looking into it15:41
generalfuzzit doesn't happen when using http - only https15:42
yoctozeptohmm15:43
generalfuzzfor example - https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_122/712005/9/check/kolla-ansible-ubuntu-source/12291c8/primary/logs/kolla/keystone/keystone-admin.txt15:43
mgoddardI was thinking we should send something to the ML about experience with uwsgi etc15:43
generalfuzz[uwsgi-http key:  client_addr: 192.0.2.1 client_port: 42150] hr_ssl_read(): Connection reset by peer [plugins/http/https.c line 393]15:43
yoctozeptoweird if only tls triggers it15:44
yoctozeptouwsgi bug?15:44
generalfuzzor possible configuration issue, but not an obvious one15:45
mgoddarddo we have a suitable package for uwsgi?15:45
yoctozeptowell, it blames peer15:45
yoctozeptoso could be clients doing something wrong with tls15:46
yoctozeptohttps://extras.getpagespeed.com/release-el8-latest.rpm <- this looks bad bad bad15:46
Fl1ntor the server doing something wrong and triggering the client to reset (something like improper handle and downgrade).15:46
stackedsaxwould the tripleo build issue prevent generalfuzz' work from getting accepted?15:47
yoctozeptoFl1nt: indeed15:47
generalfuzzyoctozepto: that is for tripleo15:47
mgoddardwhere does the package come from for non-tripleo?15:48
yoctozeptogeneralfuzz: for binary in general you mean15:48
yoctozeptomgoddard: for source from pypi15:48
yoctozeptohttps://review.opendev.org/#/c/710879/17/docker/keystone/keystone-base/Dockerfile.j215:48
patchbotpatch 710879 - kolla - Install uwsgi for Keystone - 17 patch sets15:48
yoctozeptoall look here15:48
generalfuzzyoctozepto: rhel 8 doesn't have uwsgi in epel, so this is the workaround15:49
mgoddardwe don't need to worry too much about that15:49
generalfuzzI will remove it then15:50
mgoddardwe don't really test RHEL on kolla-ansible15:50
yoctozeptomgoddard: I think generalfuzz also meant centos815:50
mgoddardif base_distro == 'rhel'15:50
Fl1nt@mgoddard, working with a restricted infra, such package is going to be a nightmare for companies with high security measures and containments. it won't pass any COTS.15:51
yoctozeptomgoddard: ah, missed15:51
mgoddardFl1nt: because it is from EPEL?15:51
mgoddardor do you mean https://extras.getpagespeed.com/release-el8-latest.rpm ?15:52
mgoddardwe can drop that15:52
Fl1ntbecause the epel repo installer is from getpagespeed.com15:52
yoctozepto😂15:53
Fl1ntit can be override, but still, having that kind of urls triggers a lot of alerts when screening the dockerfile template in here ^^ then I need to introspect them and either validate them with the SOC team etc.15:54
mgoddardI mean we will drop that from the patch15:54
Fl1ntoh ok cool ^^15:54
mgoddardit's only for RHEL, and we don't really test RHEL for kolla ansible15:54
mgoddardtripleo use httpd/mod_wsgi15:55
mgoddardwhich of course means we would need both in our images :(15:55
mgoddardany idea how big uwsgi and its dependencies are?15:55
yoctozeptoI'm a bit confused still15:55
Fl1ntI personally don't use tripleO but RHEL+kolla with some customers requiring professional agreements etc.15:55
yoctozeptoI thought we disabled epel by default15:55
mgoddardtrue, we do15:56
yoctozeptothen how come uwsgi gets installed 😂15:56
mgoddardcomes from delorean-master-testing15:57
yoctozeptoINFO:kolla.common.utils.keystone-base: uwsgi                    x86_642.0.18-4.el8                            delorean-master-testing     385 k15:57
mgoddardhttps://3f2e3a89d291731c913a-0da77774ba1e870a646f19d85e727a87.ssl.cf2.rackcdn.com/710879/17/check/kolla-build-centos8-binary/7520d7e/kolla/build/keystone-base.log15:57
yoctozeptomgoddard: yeah, I pasted it for all15:57
yoctozeptoso they do care about uwsgi15:57
mgoddardseems so15:57
yoctozeptojust rhel not having it *yet*15:58
mgoddardwonder why it is in RDO but not RHOSP15:58
yoctozeptorhosp always behind15:58
yoctozeptostill, it's delorean for ussuri15:58
hrwpyscss author responded ;D horizon developer and I may became comaintainers15:58
yoctozeptodlrn-master-rhel8-deps                          1.4 MB/s | 2.4 MB     00:0115:59
yoctozeptohrw: ++ but now that it moved to 2, are you going to move it back to no 2?15:59
hrwyoctozepto: yes15:59
yoctozeptolooks like rhel8 also uses delorean so should be aligned... hmm16:00
mgoddardrhel8 builds are broken generally atm16:00
mgoddardAnyway, we're out of time16:00
mgoddardThanks all16:00
mgoddard#action mgoddard to ask ML about uwsgi etc16:00
mgoddard#endmeeting16:00
*** openstack changes topic to "Please feel welcome to join Kolla SIG https://etherpad.openstack.org/p/kolla-sig | Kolla IRC meetings on Wednesdays @ 15:00 UTC - see agenda @ https://goo.gl/OXB0DL | Whiteboard: https://etherpad.openstack.org/p/KollaWhiteBoard | IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"16:00
openstackMeeting ended Wed Mar 18 16:00:50 2020 UTC.  Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4)16:00
openstackMinutes:        http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-03-18-15.00.html16:00
openstackMinutes (text): http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-03-18-15.00.txt16:00
openstackLog:            http://eavesdrop.openstack.org/meetings/kolla/2020/kolla.2020-03-18-15.00.log.html16:00
hrwthanks16:00
generalfuzzthanks16:01
osmanlicilegithanks16:01
openstackgerritRadosław Piliszek proposed openstack/kolla master: Install uwsgi for Keystone  https://review.opendev.org/71087916:01
yoctozeptolooking at no rhel override now to see where the problem pops up16:02
yoctozeptoalso, we probably want ubu/deb to have uwsgi too16:02
yoctozeptoprobably openstack-base should have uwsgi as it's going to be 'the server' to use16:02
yoctozeptomgoddard, generalfuzz16:02
yoctozeptoif you care to continue this discussion for a moment :-)16:03
stackedsaxI'm just a little concerned that the uwsgi work is moving the goalposts for generalfuzz's TLS work16:03
yoctozeptostackedsax: you mean extra delay?16:04
yoctozeptothat's true unfortunately16:04
stackedsaxyes.  can we not take things one step at a time?16:04
yoctozeptostackedsax: I agree, I am also worried about that peer reset, it does not sound nice at all16:05
stackedsaxseems like things were reasonable with the approach generalfuzz had with wsgi.16:05
yoctozeptostackedsax: apache is known to handle http(s) well ;p16:06
stackedsaxintroducing uwsgi at this point seems like it is destined to postpone the TLS story for a long time.16:06
*** dking_desktop has joined #openstack-kolla16:07
yoctozeptostackedsax: yeah, I'm willing to continue discussion but seems mgoddard and generalfuzz are gone16:07
stackedsaxit seems like a more practical approach would be to handle the movement to uwsgi as a separate story16:08
generalfuzzstill here :)16:08
Fl1nt@hrw, ok, so, I've just finished to rebuild and test the image, no error within the build logs, no error on push, but still the same issue.16:08
*** vedup19 has quit IRC16:08
yoctozeptoI would opt even for a quick call around tls, possibly including mnasiadka to get things orderly and flowing16:08
stackedsax+1 that16:09
generalfuzzdig it16:09
*** vedup19 has joined #openstack-kolla16:09
mnasiadkaorderly and flowing, sounds like a good marketing scheme16:13
mgoddardmnasiadka: https://bugs.launchpad.net/kolla-ansible/+bug/186794616:15
openstackLaunchpad bug 1867946 in kolla-ansible train "Ceph jobs failing on Train and Stein during reconfigure" [High,Triaged]16:15
mnasiadkayeah, looking into it now16:16
mnasiadkasince I need to wait for tomorrows images publish to continue with ovn16:16
mnasiadka:)16:16
mgoddardyoctozepto, generalfuzz, stackedsax: stepped away but back now16:17
mgoddardwe can have a call. When would be a good time?16:17
stackedsaxI mean, generalfuzz and I are usually free around the tie of this meeting most mornings.  I dare say we're free now, as well :D16:19
stackedsaxperhaps same time as the weekly meeting, but tomorrow?16:19
mgoddardif we're all free now that works for me16:20
generalfuzz+116:20
mgoddardyoctozepto, mnasiadka ?16:20
yoctozeptotomorrow 3 PM UTC ?16:21
mgoddardok16:21
mnasiadkayeah, why not16:21
mgoddardI'll setup a google meet16:21
yoctozepto++16:22
Fl1ntok, I even introspect the pulled image, it's a AMD64 one, nothing wrong within the image, don't get why docker doesn't want to run it...16:22
stackedsaxthanks, mgoddard16:22
hrwre16:23
hrwhttps://review.opendev.org/#/c/713135/ waits for reviews. zuul fine except c8-k-a16:23
patchbotpatch 713135 - kolla - CI: Avoid tox - 30 patch sets16:23
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible stable/train: DNM: Debug Ceph permission weirdness on reconfigure  https://review.opendev.org/71369116:24
hrwhttps://review.opendev.org/#/c/713647/ needs second core16:24
patchbotpatch 713647 - kolla (stable/stein) - mariadb: install mariabackup on Debian - 1 patch set16:24
yoctozeptohrw: let me review that coauthored patch which grew from one line to a few more :-)16:24
hrwso is https://review.opendev.org/#/c/713616/16:24
patchbotpatch 713616 - kolla (stable/train) - CI: Install tox - 1 patch set16:24
hrwyoctozepto: it was also rewrite of run.yml scripts ;D16:25
*** k_mouza has quit IRC16:27
yoctozeptohrw: yeah, that's your part :-)16:28
*** diurnalist has quit IRC16:29
*** k_mouza has joined #openstack-kolla16:30
*** diurnalist has joined #openstack-kolla16:35
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Avoid tox (and clean up gate setup)  https://review.opendev.org/71313416:36
mnasiadkayoctozepto: any idea why fluentd fails so miserably on add fluentd to mariadb CI change? https://zuul.opendev.org/t/openstack/build/ba032fb7155c4cdca354a078d67a0a0a/log/primary/logs/docker_logs/fluentd.txt ;-)16:38
yoctozeptomnasiadka: you reading my mind, but no idea so far :-)16:42
yoctozeptohttps://67bcfa89baec475c7d11-bb81120481bfc1a9d88d226642da11d8.ssl.cf5.rackcdn.com/713663/3/check/kolla-ansible-centos8-source-mariadb/ba032fb/primary/logs/kolla_configs/fluentd/input/00-global.conf16:42
yoctozeptohmm16:42
mgoddardmaybe because lots of things are disabled?16:43
yoctozepto  path {% for service, enabled in services if enabled | bool %}/var/log/kolla/{{ service }}/*.log{% if not loop.last %},{% endif %}{% endfor %}16:43
yoctozeptoyeah, all openstack services16:43
yoctozeptoso this ends up being empty16:43
mnasiadkaso I found a bug...16:44
yoctozeptomnasiadka: I'll handle this, we can treat it as bug16:44
mnasiadkayoctozepto: thanks :)16:44
*** bengates has quit IRC16:44
* hrw off16:49
openstackgerritMerged openstack/kolla stable/train: Fix Horizon build after switch to pyscss2 which lacks binary wheel  https://review.opendev.org/71364417:05
*** k_mouza has quit IRC17:23
Fl1nt\o/ VICTORY17:26
Fl1ntall right17:26
Fl1ntfinally found out what's going on with my images17:27
Fl1ntso17:27
Fl1ntwhen the base image (or any image) isn't able to curl dumb-init and end up with a 404 the process continue without marking the image as erroneous and so you end up with all your images being stale as the dumb-init called by the entrypoint isn't present17:29
yoctozeptoFl1nt: woop woop! what was it?17:29
yoctozepto:O17:29
yoctozeptothat's a bug, needs reporting17:29
Fl1ntAdditionnaly, why does the dockerfile.j2 template do a RUN CURL for RPM based where RDO get the packages and pip repositories too.17:32
*** evrardjp has quit IRC17:36
*** evrardjp has joined #openstack-kolla17:36
Fl1nt?17:36
Fl1ntsorry, was on a call while I typed the question ^^17:36
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix deployment of fluentd without any enabled OpenStack services  https://review.opendev.org/71370417:37
Fl1ntso, all in all, within the dockerfile.j2 once dumb_init is used it try to curl for the yelp dumb-init package, I think a more robust solution would be to rely on pip, does anyone have thoughts on this?17:38
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Enable fluentd in MariaDB jobs  https://review.opendev.org/71366317:41
*** priteau has quit IRC17:42
yoctozeptoFl1nt: which  branch is this?17:43
*** k_mouza has joined #openstack-kolla17:43
Fl1ntall, from 7.1.0 to master17:43
Fl1ntHere are for the tags: https://opendev.org/openstack/kolla/src/tag/7.1.0/docker/base/Dockerfile.j2 https://opendev.org/openstack/kolla/src/tag/8.0.2/docker/base/Dockerfile.j2 https://opendev.org/openstack/kolla/src/tag/9.0.1/docker/base/Dockerfile.j217:46
Fl1ntsame for all stable/x branch17:48
Fl1ntsame for all stable/x branches17:48
mgoddardcurl -f will fail properly. Should always be used in scripts17:51
yoctozeptoI thought we moved to pip's, maybe it was in ussuri...17:53
Fl1ntit may be related to CentOS not getting libc.so properly used with ld.17:54
Fl1nt@yoctozepto, where are the ussuri sources?17:54
yoctozeptoFl1nt: master plus what is proposed in gerrit17:55
yoctozeptobut no, that did not happen17:55
yoctozeptoI remember we must have at least discussed that at some point17:55
Fl1ntyeah, checked master earlier.17:55
yoctozeptofor now makes sense to raise a bug and fix curl calls17:56
Fl1ntok, I'll note that on my todo.17:58
Fl1ntare bugs filled with storyboard now?17:58
Fl1ntor directly within gitea?17:58
yoctozeptoFl1nt: nah, we are on launchpad17:58
yoctozeptohttps://bugs.launchpad.net/kolla17:59
Fl1nt>_< ouch17:59
Fl1ntok ^^17:59
Fl1ntI'll create all my required account this weekend, but gosh, the contributing process of OS is... challenging ^^17:59
yoctozeptoFl1nt: it's actually one account to rule them all18:00
yoctozeptoFl1nt: just not github18:00
yoctozeptobut ubuntu's18:01
Fl1ntyep, that's yet another account ^^18:10
Fl1ntbut anyway, I'll do necessary18:10
Fl1ntBTW18:10
Fl1ntthe problem is bigger than just curl18:10
Fl1ntwhen openstack-base fail due to the 404 Curl18:11
Fl1ntit's marked as successfully build18:11
Fl1ntBUT18:11
Fl1ntsome images detect that base parent is stale and so are maked as ERROR with the status: parent_error.18:12
Fl1ntfor now on my side I just override the dumb_init block.18:12
Fl1ntAll right dear people of this chan, that's it for me for today, going AFK.18:20
Fl1ntsee you18:20
*** suff has quit IRC18:20
*** k_mouza has quit IRC18:30
openstackgerritMerged openstack/kayobe master: Remove ceph block device support  https://review.opendev.org/70937618:35
*** sri__ has joined #openstack-kolla18:39
*** k_mouza has joined #openstack-kolla18:44
*** k_mouza has quit IRC18:48
*** k_mouza has joined #openstack-kolla18:49
*** k_mouza has quit IRC18:54
*** k_mouza has joined #openstack-kolla19:21
*** dougsz has quit IRC19:25
*** k_mouza has quit IRC19:26
*** diurnalist has quit IRC19:30
*** dougsz has joined #openstack-kolla19:40
*** rpittau is now known as rpittau|afk19:40
*** diurnalist has joined #openstack-kolla19:41
*** gmann is now known as gmann_afk19:57
*** gfidente is now known as gfidente|afk20:08
*** k_mouza has joined #openstack-kolla20:14
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: Fix deployment of fluentd without any enabled OpenStack services  https://review.opendev.org/71370420:18
*** k_mouza has quit IRC20:19
*** vedup19 has quit IRC20:19
*** gmann_afk is now known as gmann20:33
*** cah_link has quit IRC20:41
*** sri__ has quit IRC20:46
*** jbadiapa has quit IRC21:13
*** openstackstatus has joined #openstack-kolla21:16
*** ChanServ sets mode: +v openstackstatus21:16
*** Fl1nt has quit IRC21:45
*** dougsz has quit IRC21:50
*** suff has joined #openstack-kolla22:00
*** suff has quit IRC22:09
*** bersace has quit IRC22:12
*** k_mouza has joined #openstack-kolla22:15
*** k_mouza has quit IRC22:19
*** suff has joined #openstack-kolla22:31
*** suff has quit IRC22:38
*** TrevorV has quit IRC22:41
*** ivve has quit IRC22:50
*** threestrands has joined #openstack-kolla22:58
*** sean-k-mooney1 has joined #openstack-kolla23:02
*** sean-k-mooney has quit IRC23:04
openstackgerritMerged openstack/kolla stable/stein: mariadb: install mariabackup on Debian  https://review.opendev.org/71364723:19
*** jonaspaulo has quit IRC23:32

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!