Monday, 2020-02-24

cosmicsoundTASK [haproxy : Checking if kolla_internal_vip_address and kolla_external_vip_address are not pingable from any node] ***********************************************00:32
cosmicsoundok: [172.22.0.1] => (item={'address': '172.22.0.22', 'command': 'ping'})00:32
cosmicsoundok: [172.22.0.1] => (item={'address': '151.80.247.163', 'command': 'ping'})00:32
cosmicsoundKiller this one00:32
cosmicsoundAlways passes then it kills also admin/public and internal conections00:32
cosmicsoundAnother weird one00:33
cosmicsoundfatal: [172.22.0.30]: FAILED! => {"changed": false, "elapsed": 1, "msg": "Timeout when waiting for 172.22.0.30:3260 to stop."}00:33
*** k_mouza has joined #openstack-kolla01:03
*** k_mouza has quit IRC01:08
openstackgerritweichuancheng proposed openstack/kolla master: add o-hm0 interface ifcfg for octavia-worker and set it start when octavia-worker restart  https://review.opendev.org/70938001:21
openstackgerritweichuancheng proposed openstack/kolla-ansible master: add o-hm0 interface ifcfg for octavia-worker and keep it up when octavia-worker restart  https://review.opendev.org/70938101:23
*** ricolin has quit IRC01:32
*** zhanglong has joined #openstack-kolla01:36
openstackgerritweichuancheng proposed openstack/kolla-ansible master: add o-hm0 interface ifcfg for octavia-worker and keep it up when octavia-worker restart  https://review.opendev.org/70938101:38
*** verdurin has quit IRC02:04
*** zhanglong has quit IRC02:06
*** verdurin has joined #openstack-kolla02:19
*** hongbin has joined #openstack-kolla03:19
openstackgerritHongbin Lu proposed openstack/kolla-ansible master: [WIP] Zun: Add zun-cni-daemon to compute node  https://review.opendev.org/70821303:19
*** ricolin has joined #openstack-kolla03:36
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827303:43
*** skramaja has joined #openstack-kolla04:22
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827304:30
*** ricolin has quit IRC04:31
*** JangwonLee has quit IRC04:41
openstackgerritHongbin Lu proposed openstack/kolla master: Zun: add zun-cni-daemon image  https://review.opendev.org/70827305:02
*** hongbin has quit IRC05:04
openstackgerritMerged openstack/kolla-ansible stable/rocky: Change /run bind mount for neutron/openvswitch  https://review.opendev.org/70759805:19
*** cah_link has joined #openstack-kolla05:25
*** evrardjp has quit IRC05:34
*** evrardjp has joined #openstack-kolla05:35
*** cah_link has quit IRC05:42
*** sri_ has joined #openstack-kolla05:51
sri_cosmicsound, does someone here runs k-a and ceph-ansible together? 18:3405:53
sri_sri_, I've tested in lab a month ago05:54
*** zhanglong has joined #openstack-kolla05:59
*** shyamb has joined #openstack-kolla06:03
*** zhanglong has quit IRC06:07
*** happyhemant has joined #openstack-kolla06:13
*** zhanglong has joined #openstack-kolla06:13
*** abdysn has joined #openstack-kolla06:14
*** shyamb has quit IRC06:21
*** zhanglong has quit IRC06:22
*** zhanglong has joined #openstack-kolla06:24
*** shyamb has joined #openstack-kolla06:40
*** numans has joined #openstack-kolla06:51
cosmicsoundsri_ , how did it end up?07:01
sri_First I've install docker-ce and I've changed to ceph ansible to use docker-ce07:09
sri_cosmicsound, other then that it should be ok07:11
openstackgerritXinliang Liu proposed openstack/kolla-ansible stable/rocky: remove cinder v1 endpoint  https://review.opendev.org/70941707:13
sri_cosmicsound, i remember I had to remove the docker-py package before run the kolla-deploy07:21
*** jbadiapa has quit IRC07:23
*** jbadiapa has joined #openstack-kolla07:28
mnasiadkamorning07:30
xinliangmnasiadka: morning07:31
mnasiadkahi xinliang07:36
xinlianghi mnasiadka07:38
yoctozeptohi mnasiadka07:41
mnasiadkao/07:41
yoctozeptolooks like we got our deb source pretty b0rken in aio07:42
mnasiadkanice07:42
yoctozeptoanything coming to your mind?07:42
mnasiadkait's the neutron stuff, or something else?07:42
yoctozeptomnasiadka: guessing neutron07:45
mnasiadkayoctozepto: well, I'm not keen on digging in neutron code why it prepends sysctl command with ping :)07:45
mnasiadkayoctozepto: so just raise a bug in Neutron and let's see :)07:45
yoctozeptomnasiadka: ah, not this one07:45
yoctozeptomnasiadka: I mean deb general07:45
yoctozeptoas in ubuntu source07:46
yoctozeptohttps://review.opendev.org/70936907:46
patchbotpatch 709369 - kolla-ansible - CI: Test floating ip allocation and connectivity t... - 2 patch sets07:46
yoctozeptoplease read my monologue, I analysed job failures07:46
*** shyamb has quit IRC07:46
mnasiadkaconnection refused?07:48
mnasiadkamaybe we need a console log from the cirros instance?07:49
mnasiadkato be sure it booted properly?07:49
mnasiadkayoctozepto: https://zuul.opendev.org/t/openstack/build/9931ccf914e44711a974933992816e4f/log/primary/logs/kolla/libvirt/libvirtd.txt#407:51
mnasiadkayoctozepto: good morning!07:51
yoctozeptomnasiadka: and dnsmasq! :D07:52
yoctozeptointeresting that ping actually goes through07:53
mnasiadkayoctozepto: I think we need to grep -i error, libvirt log posts "error" instead of "ERROR"07:53
mnasiadka:)07:53
yoctozeptooh, yeah, we should07:54
mnasiadkasecond thing what is wrong with metadata agent? https://zuul.opendev.org/t/openstack/build/9931ccf914e44711a974933992816e4f/log/primary/logs/kolla/all-ERROR.txt07:54
yoctozeptowould save me some time yesterday07:54
yoctozeptomnasiadka: yeah, I mentioned that07:54
mnasiadkaseems deb family is broken, you're right07:54
mnasiadkabut now we have means to test it properly07:55
yoctozeptomnasiadka: but one other thing07:55
yoctozeptomnasiadka: multinode actually work07:55
yoctozeptoso this error message is extraneous07:55
yoctozeptooh how I love the mystery07:55
yoctozeptofor extra lolz07:56
yoctozeptodebian libvirtd is happy with ebtables07:56
yoctozepto;p07:56
yoctozeptoI added more lolz in comments08:03
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Cinder - remove glance_api_version and disable Cinder v2 API  https://review.opendev.org/70942208:05
*** bengates has joined #openstack-kolla08:12
mnasiadkayoctozepto: maybe we need to fine tune libvirt confs :)08:18
mnasiadkajust to be sure we don't have errors08:19
yoctozeptomnasiadka: no idea, I rather think it's not libvirt08:19
yoctozeptomnasiadka: yeah, for errors sure08:19
mgoddardmorning08:20
mgoddardI'm at the ironic midcycle today and tomorrow08:20
mgoddardwill be partially available08:20
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Cinder - remove glance_api_version and disable Cinder v2 API  https://review.opendev.org/70942208:23
mnasiadkamgoddard: have fun, remotely or travelled to CERN?08:23
mgoddardremote08:24
yoctozeptooh, nice08:28
yoctozeptohave fun08:28
*** zhanglong has quit IRC08:28
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Cinder - remove glance_api_version and disable Cinder v2 API  https://review.opendev.org/70942208:29
mnasiadkayoctozepto: can't do a reno - ERROR: Could not find a version that satisfies the requirement gitdb2===4.0.1 (from -c https://releases.openstack.org/constraints/upper/master (line 777)) (from versions: 2.0.0, 2.0.1, 2.0.2, 2.0.3, 2.0.4, 2.0.5, 2.0.6, 3.0.0, 3.0.1, 3.0.2, 3.0.3, 3.0.3.post1, 4.0.2)08:37
mnasiadka:D08:37
yoctozeptoeh08:37
yoctozeptoversional removal zawsze na propsie08:37
yoctozeptoversion*08:37
*** shyamb has joined #openstack-kolla08:45
*** ricolin has joined #openstack-kolla08:56
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Add addressing on external network  https://review.opendev.org/70936208:59
*** dougsz has joined #openstack-kolla09:02
mgoddardturns out ironic midcycle is tomorrow /facepalm09:11
*** cah_link has joined #openstack-kolla09:11
*** k_mouza has joined #openstack-kolla09:15
yoctozeptothankfully you chose the remote option ;-)09:21
mnasiadkamgoddard: right, I was curious there's some closed circle event today :D09:22
mgoddardyoctozepto: lol09:23
*** cah_link has quit IRC09:25
*** cah_link has joined #openstack-kolla09:28
*** gfidente has joined #openstack-kolla09:33
yoctozeptomgoddard, mnasiadka: https://review.opendev.org/70938909:36
patchbotpatch 709389 - requirements - Update gitdb to 4.0.1, gitdb2 to 4.0.2 - 5 patch sets09:36
yoctozeptoemergency situations need emergency solutions :O09:36
mnasiadkalol09:37
mnasiadkathe renaming flow looks a bit funny09:38
yoctozeptomerge  it like there is no tomorrow ;D09:38
*** dabukalam has joined #openstack-kolla09:47
mnasiadkayoctozepto: just thinking, service-ks-register would use an extension to delete old endpoints on upgrade (like cinder v1/v2 case)09:53
yoctozeptomnasiadka: we probably want to remove the ones we know about anyway09:54
yoctozeptonot just arbitrary09:54
mnasiadkamakes sense09:54
yoctozeptootoh, upgrade jobs could check if catalog is clean after09:54
yoctozeptoas we know its contents there09:54
mnasiadkayup09:58
cosmicsoundSomeone had a chance to check the TLS/SSL bug when using VIP External with FQDN09:58
cosmicsoundIt passes all prechecks and it never hits the ip when it comes to horizon or services09:59
*** shyamb has quit IRC10:01
*** shyamb has joined #openstack-kolla10:04
*** sorin-mihai has quit IRC10:14
openstackgerritMerged openstack/kolla-ansible stable/rocky: remove cinder v1 endpoint  https://review.opendev.org/70941710:21
*** shyamb has quit IRC10:27
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Test SSH connectivity to deployed instances  https://review.opendev.org/70760410:31
mgoddardyoctozepto: I think the existing fdb config would work10:33
yoctozeptomgoddard: well, it does not try to send it to oneself10:34
yoctozeptoreally never done same vni on the same host10:34
mgoddardhmm true10:34
mgoddardbridge+veth pair?10:35
mgoddardthat's the approach we use in kayobe for this kind of thing10:35
yoctozeptohmm, worth copying then; we can do some experiments though10:36
hrwmorning10:36
yoctozeptoI'm a little busy atm so feel free to propose in the meantime ;D10:36
mgoddardmorning hrw10:44
mnasiadkayoctozepto: https://bugs.launchpad.net/neutron/+bug/186418610:47
openstackLaunchpad bug 1864186 in neutron ""ping" prepended to ip netns commands" [Critical,Fix released] - Assigned to Lucian Petrut (petrutlucian94)10:47
*** hjensas has quit IRC10:48
mnasiadkamgoddard, yoctozepto: oh boy, the gitdb2 stuff breaks all pep8 and py36/37 jobs...10:49
mgoddardyup10:49
mgoddardmnasiadka: looks like we need to kill cluster_interface - it was only used by ceph10:50
mnasiadkamgoddard: makes sense - should I?10:50
mgoddardmnasiadka: go for it10:50
yoctozeptomnasiadka: ugly that in neutron, it truly looked nice at the first sight10:51
yoctozeptomnasiadka: does it mean only debian rebuilt with this bug10:51
yoctozepto?10:51
*** hjensas has joined #openstack-kolla10:52
yoctozeptowe should see breakage all over the place10:52
yoctozeptohttps://zuul.opendev.org/t/openstack/builds?job_name=kolla-publish-ubuntu-source&branch=master&job_name=kolla-publish-debian-source10:54
yoctozeptoodd10:54
yoctozeptomaybe it truly depends on python, or even distro10:54
mnasiadkayoctozepto: I think so, debian builds in check pipeline, while rest of k-a jobs pulls what kolla built in the morning?10:55
yoctozeptomnasiadka: does it?10:56
yoctozeptowe publish it nowadays10:56
*** ac7569 has joined #openstack-kolla10:57
*** shyamb has joined #openstack-kolla10:57
mnasiadkaright10:57
mnasiadkaso maybe it was python version10:57
mnasiadkabut it was a change to rootwrap filters10:57
mnasiadkaso should affect everybody10:57
*** sorin-mihai has joined #openstack-kolla10:58
yoctozeptowell, it only broke debian, so it must have triggered what others did not10:58
yoctozeptocould be anything10:58
ac7569hello, after installing kolla with kolla ansible the default conf for fluentd throw warn while parsing gnocchi-metricd log. I found this : https://bugs.launchpad.net/kolla/+bug/1828976 . Does someone know a way to fix this ? I tried modifying the fluentd conf without success10:58
openstackLaunchpad bug 1828975 in kolla-ansible "duplicate for #1828976 fluentd unable to parse gnocchi-metricd.log" [Undecided,New]10:58
mnasiadkaac7569: well, it seems that fluentd filters related to gnocchi need updating - feel free to fix them and contribute to kolla-ansible :)11:11
hrwceph--11:13
yoctozeptomgoddard: commented there, (vni, port) must be unique so it would be creating a separate topology there11:15
mgoddardok11:16
yoctozeptomgoddard: so we are left with current or some veth pair dance11:16
yoctozeptonot sure how ovs likes that11:16
ac7569mnasiadka thx for the answer i'll try that :)11:16
yoctozeptowill check kayobe later for inspiration11:16
mgoddardovs loves it11:16
yoctozepto^ that one sentence nobody said before11:17
mgoddardyours or mine?11:17
yoctozeptoyours11:17
mgoddard:)11:17
yoctozepto:D11:17
mgoddardcurrent approach is simpler11:17
mgoddardbut wouldn't work in production11:18
mgoddardnot sure how much it matters11:18
mnasiadkawe don't need it to work in production? we need it to work in CI11:24
mnasiadkaunless there are more crazy people (than me) to replicate upstream CI in a downstream workflow11:25
*** zhanglong has joined #openstack-kolla11:25
openstackgerritChristian Berendt proposed openstack/kolla-ansible stable/rocky: Add support for ovsdb conversion  https://review.opendev.org/70945111:29
yoctozeptomgoddard: if it works in CI, it will work in production :D11:29
yoctozeptothough it does not mean one should ever use it11:30
yoctozeptoanyways11:30
yoctozeptoanother mind teaser11:30
yoctozeptohttps://70a366f81cf63ddc4b03-580e84ccb0b17d3ff354ab4c1b56697e.ssl.cf1.rackcdn.com/709369/2/check/kolla-ansible-centos-source-linuxbridge/86fb4f0/primary/logs/ansible/test-core-openstack11:30
yoctozeptohttps://70a366f81cf63ddc4b03-580e84ccb0b17d3ff354ab4c1b56697e.ssl.cf1.rackcdn.com/709369/2/check/kolla-ansible-centos-source-linuxbridge/86fb4f0/primary/logs/system_logs/ip.txt11:30
yoctozeptoso this is lb testing11:30
yoctozepto"From 198.51.100.173 icmp_seq=1 Destination Host Unreachable"11:30
yoctozeptois quite contradictory with11:30
yoctozepto198.51.100.0/24 dev brq8e399d55-80 proto kernel scope link src 198.51.100.111:31
yoctozepto198.51.100.173 dev brq8e399d55-80 lladdr fa:16:3e:6c:ee:89 REACHABLE11:31
yoctozeptoany hints warmly welcome11:31
*** k_mouza has quit IRC11:34
mgoddardyoctozepto: iptables?11:42
mgoddarddocker forward drop rule + bridge-nftables?11:42
yoctozeptocould be, that could certainly play with lb and not ovs11:43
*** priteau has joined #openstack-kolla11:43
yoctozeptootoh, this message is for arp timing out11:43
yoctozeptoand it could not have since ip neigh shows it11:44
yoctozeptooh what a mystery11:44
mnasiadkanetworking is always a mystery :)11:47
openstackgerritMerged openstack/kolla-ansible stable/train: Configure Cinder to use lioadm on CentOS/RHEL 8  https://review.opendev.org/70241711:50
*** kplant has joined #openstack-kolla12:11
*** shyamb has quit IRC12:31
*** shyamb has joined #openstack-kolla12:35
*** shyam89 has joined #openstack-kolla12:39
*** shyamb has quit IRC12:42
*** happyhemant has quit IRC12:43
*** sorin-mihai has quit IRC12:46
*** Luzi has joined #openstack-kolla12:49
*** sorin-mihai has joined #openstack-kolla12:50
*** dougsz has quit IRC12:53
*** kplant has quit IRC12:55
*** factor has quit IRC12:58
*** factor has joined #openstack-kolla12:58
*** kplant has joined #openstack-kolla13:00
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Test floating ip allocation and connectivity to the instance  https://review.opendev.org/70936913:16
yoctozeptomnasiadka: sure, ubuntu source/binary still remains13:16
*** dave-mccowan has joined #openstack-kolla13:28
*** skramaja has quit IRC13:29
*** dougsz has joined #openstack-kolla13:39
*** dave-mccowan has quit IRC13:40
*** dave-mccowan has joined #openstack-kolla13:41
openstackgerritMark Goddard proposed openstack/kolla stable/train: nova: use proper uefi package names under CentOS/RHEL 8  https://review.opendev.org/70953013:48
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Add feature to support managing dynamic pollsters  https://review.opendev.org/67999913:54
*** shyam89 has quit IRC13:57
*** shyamb has joined #openstack-kolla13:58
*** shyamb has quit IRC14:15
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Support variable image tag suffix  https://review.opendev.org/70953414:22
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Deploy CentOS 8 containers  https://review.opendev.org/70953514:22
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70953614:22
openstackgerritMark Goddard proposed openstack/kolla stable/train: Remove tgtd and scsi-target-utils support in CentOS/RHEL 8  https://review.opendev.org/70242114:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Don't force tag in build jobs  https://review.opendev.org/70245214:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: base image fixes for RabbitMQ, fluentd & InfluxDB  https://review.opendev.org/70920114:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: disable repo_gpgcheck for grafana image  https://review.opendev.org/70920614:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Update packages in images  https://review.opendev.org/70920214:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Remove shellinabox from ironic-conductor  https://review.opendev.org/70920314:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Use upstream Ceph/master  https://review.opendev.org/70920414:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Allow SSH access to keystone_ssh and nova_ssh  https://review.opendev.org/70920514:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Relax ironic iPXE EFI bootloader location  https://review.opendev.org/70920714:24
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: base and openstack-base images  https://review.opendev.org/70953714:25
openstackgerritMark Goddard proposed openstack/kolla stable/train: CentOS 8: Add deploy jobs in CI  https://review.opendev.org/70953814:25
dougszcrikey!14:25
*** ac7569 has quit IRC14:26
yoctozeptooh my14:26
yoctozeptoreview devil14:26
yoctozeptomgoddard: required many adaptations / none / some? ;-)14:27
mgoddardyoctozepto: some14:28
mgoddardlittle fiddly bits14:28
mgoddardnothing huge14:28
mgoddardhaven't tried building all though14:28
yoctozeptomgoddard: anything to be wary of?14:28
mgoddardbut that's all backports done now I think. May need a revision14:28
*** Luzi has quit IRC14:29
mgoddardwe don't have repos disabled by default in train14:29
mgoddardthat caused a few diffs14:29
mgoddardI suppose the main difference is that it's more important that C7 still works14:30
*** sri_ has quit IRC14:33
*** abdysn has quit IRC14:38
hrwlet me run14:40
hrwmgoddard: --base-tag 8?14:42
mgoddardhrw: +114:42
hrw(venv3) 15:42 (0s) hrw@puchatek:kolla$ time ~/devel/moje/docker-utils/kolla-build-images.sh centos 'binary source' ' --base-tag 8' c8t14:42
mgoddard8+1=9?14:43
hrwmgoddard: no. 8114:43
mgoddard'8'+'1'='81'14:43
hrwhm. let me unabandon non-infra-base image change14:43
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: WIP: introduce non-infra-base image  https://review.opendev.org/70759914:44
hrwhm.., https://cbs.centos.org/koji/buildinfo?buildID=2856414:45
hrwceph naulitus built. just no repo yet14:45
cosmicsoundHow is OVN supported now in kolla?14:48
hrwhm. I am thinking of adding --noclean_package_cache option...14:50
*** lile has joined #openstack-kolla14:51
cosmicsoundfirewall_driver = openvswitch VS firewall_driver = neutron.agent.linux.iptables_firewall.OVSHybridIptablesFirewallDriver . What is best to use? Anyone have tested openvswitch14:57
*** TrevorV has joined #openstack-kolla15:00
*** ivve has joined #openstack-kolla15:04
noxoid_cosmicsound, i use openvswitch for my firewall driver but its still impacted by the following bug on certain branches: https://bugs.launchpad.net/neutron/+bug/173206715:04
openstackLaunchpad bug 1732067 in OpenStack Security Notes "openvswitch firewall flows cause flooding on integration bridge" [Undecided,New]15:04
cosmicsoundthanks noxoid_ for the hint15:05
noxoid_which is a major security concern for us. a large motivator for my upgrade to train is fixing it15:05
noxoid_but in general i like the ovs driver, no messing with host iptable rules (we like to manage them with salt)15:05
*** noxoid_ is now known as noxoid15:07
mgoddardcosmicsound: mnasiadka is working on OVN: https://review.opendev.org/#/c/696841/15:13
patchbotpatch 696841 - kolla-ansible - OVN Support - 96 patch sets15:13
*** sorin-mihai has quit IRC15:13
*** dave-mccowan has quit IRC15:14
*** dking_desktop has joined #openstack-kolla15:19
hrwmgoddard: https://pastebin.com/PPpVQcNc for centos/binary. some failures present15:20
mgoddardhrw: thanks for checking. etcd and barbican (uwsgi) failed on master. RDO added packages, I expect they need to do it again for train15:21
hrwmgoddard: https://review.opendev.org/#/c/700281/ needs to be added to the queue15:22
patchbotpatch 700281 - kolla - vitrage-base: honor distro_python_version (MERGED) - 2 patch sets15:22
mgoddardqdrouterd I have a fix for locally15:22
openstackgerritMark Goddard proposed openstack/kolla stable/train: vitrage-base: honor distro_python_version  https://review.opendev.org/70955315:23
mgoddardhrw: thanks15:23
mnasiadkamgoddard: vitrage doesn't work properly, so don't waste time on it:)15:24
mgoddarddoesn't hurt to cherry pick15:24
hrwmnasiadka: still listed as buildable...15:24
* hrw -> Z Nation15:24
mnasiadkahrw: well, correct - in Kolla it's buildable :)15:25
*** tonythomas has joined #openstack-kolla15:25
mnasiadkaoh boy, ceph-ansible deployment on python2 fails now15:27
mnasiadka virtualenv -p /usr/bin/python2 --system-site-packages /home/zuul/ceph-venv\r\nOSError: [Errno 2] No such file or directory: '/home/zuul/.local/share/virtualenv/seed-v1/2.7/wheels.lock'\r\n",15:27
hrwmnasiadka: I would force py3. even on c7 and then check does it work15:28
hrwif does then f. py215:28
mnasiadkahrw: it won't work, c7 is missing python selinux bindings for python315:28
*** dave-mccowan has joined #openstack-kolla15:28
openstackgerritMark Goddard proposed openstack/kolla master: Use StorageSIG repos for Ceph in CentOS8  https://review.opendev.org/70733815:28
mnasiadkaso I'm rather a fan of removing centos7 ceph-ansible job15:28
openstackgerritChristian Berendt proposed openstack/kolla master: Add a generic header  https://review.opendev.org/70875815:30
hrwany idea why all zuul jobs are queued?15:32
bjolowe are troubleshooting some weird ovs issues, so i want to try a newer OVS version in rocky15:32
bjolowhat would be the easiest way to get ovs 2.12 into the build?15:33
cosmicsoundbjolo , use tag?15:33
bjolohow do you mean?15:34
bjolo2.12 is not the the repos listed15:34
bjoloit is in the train repo, but i dont want to include that globally :)15:34
cosmicsoundWhen you build images in kolla you can use a build.conf that can contain the needed version of the image you wish, or its TAG / Version.15:34
*** dave-mccowan has quit IRC15:35
cosmicsoundI think someone can tell more on building custom I handled only once magnum15:35
mnasiadkabjolo: centos or ubuntu?15:35
bjolocentos15:35
bjolorocky 7.115:35
bjolosource15:35
mnasiadkause openvswitch tag train in kolla-ansible?15:35
cosmicsoundhttps://docs.openstack.org/kolla/latest/admin/image-building.html .15:36
mnasiadkabjolo: openvswitch_tag: "train"15:36
bjoloin kolla-build.conf15:37
mnasiadkabjolo: no, in kolla-ansible - I thought you are using images from docker hub15:37
cosmicsoundI was not either sure if you use premade images or custom15:37
cosmicsoundprocess differs15:37
mnasiadkabjolo: openvswitch in centos comes from RDO - so the only option to get centos with OVS 2.12 is to use RDO Train repo15:38
cosmicsoundwhat is the right docker_registry port 4000 or 5000 sometimes in docs its written as :4000 sometimes with :5000 . I was thinking keystone is on 5000 . no conflicting15:38
mnasiadkaif you use docker hub images - just do what I wrote - set openvswitch_tag: "train" in globals and go15:38
mnasiadkabjolo: if you build your own and push to your own registry - install kolla train somewhere, build openvswitch images, push them to the repo under train tag (or any other tag) and set openvswitch_tag to that tag15:39
mnasiadkaeasy as that15:39
mnasiadkapush them to registry I mean15:39
mnasiadkacosmicsound: keystone is 5000, that's why if you plan to deploy openstack on the same node you have a registry - use a different port than 500015:40
bjolobuilding train images15:43
bjoloi should only override the openvswitch containers right?15:43
mnasiadkabjolo: yup15:44
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Debug virtualenv on python2 - ceph-ansible  https://review.opendev.org/70955615:44
bjolomy thinking is that ovs is included in neutron_openvswitch_agent etc as well so those containers will have the older version. will that be an issue?15:44
cosmicsoundbjolo , should not be, yet again maybe other answers around15:45
bjoloi try and see what happens :)15:45
bjolotnx15:45
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: DNM: Debug virtualenv on python2 - ceph-ansible  https://review.opendev.org/70955615:45
mnasiadkabjolo: no, it's only ovs client - it will be compatible15:45
bjoloim gonna try the really ugly thing first. faster... just exec the ovs containers, change the repo, yum update --disable * --enable train15:50
bjoloif conversion needed, do it manually15:51
hrwmgoddard: same set of failures (except vitrage) for centos8/source15:51
*** zhanglong has quit IRC16:00
*** ivve has quit IRC16:07
openstackgerritMerged openstack/kolla-ansible master: CI: Remove useless pre task  https://review.opendev.org/70907116:07
openstackgerritMerged openstack/kolla-ansible master: [baremetal] Drop useless reboot task  https://review.opendev.org/70912716:14
openstackgerritMerged openstack/kolla-ansible stable/rocky: Add support for ovsdb conversion  https://review.opendev.org/70945116:14
openstackgerritMerged openstack/kolla-ansible master: Remove useless neutron-linuxbridge-agent env vars  https://review.opendev.org/70865816:25
openstackgerritMerged openstack/kolla-ansible master: Stop using deprecated stores and default_store in glance  https://review.opendev.org/70811416:25
*** bengates has quit IRC16:33
yoctozeptomnasiadka: ubu/deb aio are random - they have now timed out on ssh17:08
yoctozeptomnasiadka: it got stuck for an hour long17:08
yoctozeptolol17:08
yoctozeptopreviously they just could not connect17:08
yoctozeptomgoddard, mnasiadka: lb debugged, yay17:10
yoctozeptomgoddard, mnasiadka: so it failed to setup instance ip address in instance because it could not connect to provisioned dhcp server17:11
yoctozeptomgoddard, mnasiadka: host was doing forward because it is host handling the floating ip addr17:11
yoctozeptomgoddard, mnasiadka: hence it had the first hop arp resolved17:11
yoctozeptomgoddard, mnasiadka: yet message was as if it was not - that's because that same host could not resolve the other ip addr17:12
yoctozeptomgoddard, mnasiadka: maybe we want to disable docker taking over iptables, huh?17:12
noxoidthats what i do, it works well17:13
mgoddardprobably makes sense in most cases, unless someone is using non net=host17:13
mgoddard(for other containers)17:13
yoctozeptomgoddard, mnasiadka: nah, it's irrelevant -> docker does not set drop on that chain17:13
yoctozeptomgoddard, mnasiadka: must be zuul there17:14
*** jopdorp has joined #openstack-kolla17:14
jopdorphiya17:14
jopdorpcan anyone help me with a cloudkitty that doesn't seem to work?17:14
yoctozeptonot zuul, hmm, odd17:14
jopdorpit deployed fine17:14
jopdorpand i'm tring to get ratings from it17:14
jopdorpadded some mappings17:15
yoctozeptomgoddard, mnasiadka: ah, it's docker17:15
jopdorpbut in the cloudkitty-api.log I'm gettings this error17:15
jopdorphttps://pastebin.com/H5P8ySHG17:15
yoctozeptomgoddard, mnasiadka: I checked local deploy, and it seems to override this for controllers/network nodes17:15
yoctozeptobut not for computes17:15
yoctozeptoand seemingly not aio17:15
yoctozeptoweeird :D17:16
yoctozeptojopdorp: for cloudkitty I guess priteau is your folk https://www.stackhpc.com/tag/cloudkitty.html17:16
jopdorpyocozepto: I was able to deploy murano by the way, but I didn't succeed yet in building an image with the murano-agent pre installed. it seems diskimage-builder has a bug related to it17:17
yoctozeptojopdorp: glad to know it works17:17
jopdorpfrom source it worked murano17:17
yoctozeptojopdorp: btw, why do you want to use it?17:17
jopdorpyocozepto: i thought https://www.stackhpc.com/tag/cloudkitty.html is mgoddard17:18
priteaumgoddard is employed by StackHPC, but he's not the only one ;-)17:19
jopdorpI'm reading the web page now17:19
jopdorppriteau: I deployed kolla with gnocchi and gnocchi as the cloudkitty_collector_backend17:20
priteauWhat OpenStack version is this?17:20
jopdorptrain17:20
priteauCheck if your cloudkitty_processor services are running17:23
jopdorppriteau would i also have to deploy ceilometer?17:24
priteauI haven't had this problem but it looks like your API process is sending an RPC which doesn't receive any reply17:24
jopdorpyes17:24
priteauRe ceilometer, you *don't have to*, but that's generally the component that will give you the most metrics out of the box17:25
jopdorpi didn't deploy it because it thought that gnocchi would be anough17:25
jopdorpenough17:25
jopdorpmaybe i'm wrong and gnocchi just aggregates the stats from ceilometer?17:26
priteauGnocchi is just a timeseries database. If you don't have anything pushing data to it, it will be empty.17:26
jopdorpi think that's my problem right now17:26
jopdorpceilometer here we come17:26
jopdorpthanks for the help17:26
jopdorpoh i will check if cloudkitty_processor is running17:26
jopdorpI'm inside the cloudkitty_processor docker container with bash17:26
priteauIt's not the only problem, as you wouldn't have a backtrace like this withotu ceilometer running17:27
jopdorpshould i check some process here?17:27
priteauThere should be a cloudkitty-processor process17:27
priteauBut really check the logs17:27
jopdorpps -ef | grep  cloudkitty-processor17:27
jopdorpyeah17:28
jopdorpthey're there17:28
priteau /var/log/kolla/cloudkitty/cloudkitty-processor.log17:28
jopdorp2020-02-24 17:28:23.391 6 INFO cotyledon._service_manager [-] Forking too fast, sleeping17:28
priteauAlso there was a bug report for cloudkitty train recently: https://bugs.launchpad.net/kolla-ansible/+bug/186167917:28
openstackLaunchpad bug 1861679 in kolla-ansible "kolla/ubuntu-source-cloudkitty-processor:train won't start" [Undecided,In progress] - Assigned to Christopher Wellie (well100)17:28
jopdorp2020-02-24 17:28:23.390 6 INFO cotyledon._service_manager [-] Child 7037 exited with status 117:28
jopdorpthose errors repeat17:28
priteauCheck the above link17:30
jopdorpyou think it might be ralted?17:31
jopdorprelated17:31
jopdorpi don't get any errors during deploy17:31
jopdorpseems that his reproduction steps happen at deploy17:32
*** evrardjp has quit IRC17:34
*** evrardjp has joined #openstack-kolla17:35
yoctozeptomgoddard, mnasiadka: any idea why I got my controllers/network nodes with FORWARD ACCEPT?17:38
openstackgerritMerged openstack/kolla stable/train: nova: use proper uefi package names under CentOS/RHEL 8  https://review.opendev.org/70953017:39
yoctozeptolooks like ubu/deb source hang because of:17:44
yoctozeptodebug1: Next authentication method: password17:44
yoctozeptoso it did not transfer keypair properly17:44
yoctozeptowell, that makes sense17:45
mnasiadkayoctozepto: since when we distinct between controllers/network nodes and computes in CI? :)17:45
yoctozeptomnasiadka: we don17:45
yoctozeptomnasiadka: but i'm curious what makes such nodes fix themselves17:45
yoctozeptomnasiadka: I speak about my local deploy17:46
mnasiadkahave no clue17:46
mnasiadkaautohealing cloud17:46
yoctozeptomnasiadka: nah, jokster, it deployed like this17:47
yoctozeptomnasiadka: I suppose it could be neutron17:47
yoctozeptomnasiadka: but it's not observable in CI17:48
mnasiadkaneutron setting docker?17:48
yoctozeptomnasiadka: setting FORWARD to ACCEPT17:48
mnasiadkayoctozepto: closest to any docker engine config for any service is zun I think17:48
yoctozeptonah, I don't run zun in prod17:48
mnasiadkamight be neutron, who knows17:48
yoctozeptosomething undid what docker did there17:48
mnasiadkabut it doesn't mean we shouldn't control it in a better manner17:49
yoctozeptocompletely agreed17:49
yoctozeptoI'm fixing that ssh with -n17:49
yoctozeptoI don't want it to stall on password lol17:49
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Test floating ip allocation and connectivity to the instance  https://review.opendev.org/70936917:50
yoctozeptolet it fail, let it fail (you know the melody)17:51
yoctozeptohmm, and may need to increase attempts number18:07
yoctozeptossh got available only in 5th18:07
*** priteau has quit IRC18:07
yoctozeptowell, 5th counting from 0, so 6th18:07
yoctozeptoin 0th even ping did not pass there18:08
mnasiadkaincredible, a new virtualenv got released which fixes the problem with python2 usage by python3 virtualenv18:08
yoctozeptowhat? xD18:09
yoctozeptobtw, I find it ironic that centos is slow to deploy (compared to ubu/debb) but instance gets up much quicker18:11
kplantxfs >> ext418:12
kplanti think is the reason18:12
kplanthaven't tested deb9/10 on xfs to know for sure18:12
kplantbut you are 100% right about the debian installers being way faster than anaconda18:12
jopdorppriteau: I was able to get to the next error :p thanks for the help. I'm gonna continue tomorrow :)18:13
*** kozhukalov has joined #openstack-kolla18:14
*** tonythomas has quit IRC18:15
*** jopdorp has quit IRC18:17
openstackgerritRafael Weingärtner proposed openstack/kolla-ansible master: Add feature to support managing dynamic pollsters  https://review.opendev.org/67999918:18
yoctozeptokplant: it's not about the installer18:20
yoctozeptokplant: I am about deploy of k-a in CI18:20
yoctozeptokplant: it's observably faster than centos18:21
yoctozeptokplant: I am wiring up network connectivity testing now and while centos gets instances just right, ubu/dev are faily18:21
mnasiadkayoctozepto: after fixing old problems in virtualenv, they introduced a new one :)18:22
yoctozeptomnasiadka: does it affect us?18:22
mnasiadkayoctozepto: it affected ceph-ansible jobs on centos7, but it got fixed it seems18:23
yoctozeptomnasiadka: good18:23
yoctozepto:D18:23
mnasiadkaand centos7 won't live long anyway18:23
yoctozeptomnasiadka: killing stein/train? ;p18:23
mnasiadkayoctozepto: I meant in master :)18:23
yoctozeptomnasiadka: don't be shy, I'm dropping mariadb from time to time, you can drop c718:24
yoctozeptofwiw, it works very fine in general18:24
openstackgerritMichal Nasiadka proposed openstack/kayobe master: WIP: Allow setting pip_proxy  https://review.opendev.org/70958018:25
cosmicsoundIs there any reason that when deploying with k-a to be disconnected from ssh. when reaching to neutron playbook?18:28
*** k_mouza has joined #openstack-kolla18:30
mnasiadkacosmicsound: I guess setting neutron_external_interface to the one you use to connect via ssh18:31
cosmicsoundfatal: [control-1 -> control-1]: FAILED! => {"changed": true, "msg": "'Traceback (most recent call last):\\n  File \"/tmp/ansible_kolla_docker_payload_ZSGWCb/__main__.py\", line 1024, in main\\n    result = bool(getattr(dw, module.params.get(\\'action\\'))())\\n  File \"/tmp/ansible_kolla_docker_payload_ZSGWCb/__main__.py\", line 765, in18:32
cosmicsoundstart_container\\n    self.pull_image()\\n  File \"/tmp/ansible_kolla_docker_payload_ZSGWCb/__main__.py\", line 571, in pull_image\\n    repository=image, tag=tag, stream=True\\n  File \"/usr/local/lib/python2.7/dist-packages/docker/api/image.py\", line 415, in pull\\n    self._raise_for_status(response)\\n  File18:32
cosmicsound\"/usr/local/lib/python2.7/dist-packages/docker/api/client.py\", line 263, in _raise_for_status\\n    raise create_api_error_from_http_exception(e)\\n  File \"/usr/local/lib/python2.7/dist-packages/docker/errors.py\", line 31, in create_api_error_from_http_exception\\n    raise cls(e, response=response, explanation=explanation)\\nAPIError: 50018:32
cosmicsoundServer Error: Internal Server Error (\"Get https://registry-1.docker.io/v2/: net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)\")\\n'"}18:32
cosmicsoundAnyone seen something like this?18:32
mnasiadkacosmicsound: your docker cannot connect to Docker Hub18:32
cosmicsoundYes i wonder why...18:32
cosmicsoundIt killed net in whole server18:33
cosmicsoundweird18:33
mnasiadkafor that, you need to give alms to networking gods18:33
mnasiadka:)18:33
*** k_mouza has quit IRC18:34
*** igordc has joined #openstack-kolla18:36
yoctozeptonetworking gods seemingly require regular sacrifice18:39
yoctozeptousually observable in packet loss :<18:40
*** sean-k-mooney has quit IRC18:45
yoctozepto{"Payload":"Feb 24 14:50:33 haproxy[29]: 192.0.2.10:35240 [24/Feb/2020:14:50:33.182] nova_metadata/1: SSL handshake failure","log_level":"info","Hostname":"primary","programname":"haproxy"}18:48
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Cinder - remove glance_api_version and disable Cinder v2 API  https://review.opendev.org/70942218:54
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: Add missing release note for Glance multi store move.  https://review.opendev.org/70958619:03
mnasiadkaok, enough for today19:03
* mnasiadka off19:03
cosmicsoundyoctozepto , so much :D19:16
cosmicsoundwould be great if the bug i opened is not a bug and is just a one time occurance , yet to my dissapointment there is not a single scenario i havent used yet to internal external vip.19:26
*** igordc has quit IRC19:58
openstackgerritRadosław Piliszek proposed openstack/kolla-ansible master: CI: Test floating ip allocation and connectivity to the instance  https://review.opendev.org/70936920:26
*** kplant has quit IRC20:42
*** igordc has joined #openstack-kolla21:30
*** TrevorV has quit IRC21:37
*** ggolin has joined #openstack-kolla22:00
ggolinhey guys, kolla-ansible complains that it cannot find a tag for the fluentd image. im trying to use the prebuilt images. is it possible to override this particular tag? :train is available, for example.22:02
*** jbadiapa has quit IRC22:11
cosmicsoundggolin , yes22:14
cosmicsoundfluentd_tag: "train" in globals22:15
cosmicsoundAnd pick the version you want22:15
ggolinvery cool, thank you cosmicsound.22:15
cosmicsoundNot sure if all kolla containers are taggable22:15
cosmicsoundyet that is working for most of the containers.22:16
cosmicsoundyaw ggolin22:16
ggolinoh i should read that file more carefully22:17
cosmicsoundggolin , always better to start reading and then testing :D22:23
ggolinits been 20 years and im still doing it vice versa :)22:26
cosmicsoundsame here no worries22:27
cosmicsoundyet to remind to others and then to myself is best one can do22:27
*** spiette has quit IRC22:28
*** dave-mccowan has joined #openstack-kolla22:30
*** k_mouza has joined #openstack-kolla22:31
*** spiette has joined #openstack-kolla22:31
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [elasticsearch] Update config for 6.x  https://review.opendev.org/70962422:33
openstackgerritJason Anderson proposed openstack/kolla-ansible master: [elasticsearch] Add migration for Kibana 6.x index  https://review.opendev.org/70962522:35
*** k_mouza has quit IRC22:35
*** sean-k-mooney has joined #openstack-kolla23:08
*** lile has quit IRC23:14
*** cah_link1 has joined #openstack-kolla23:41
*** cah_link has quit IRC23:43
*** cah_link1 is now known as cah_link23:43
*** igordc has quit IRC23:58

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!