Tuesday, 2021-03-30

*** LinPeiWen has joined #openstack-kolla00:33
*** k_mouza has joined #openstack-kolla00:35
*** k_mouza has quit IRC00:39
*** kevinz has joined #openstack-kolla00:49
*** bengates has joined #openstack-kolla00:58
*** bengates has quit IRC01:05
*** ricolin_ has joined #openstack-kolla01:56
*** zzzeek has quit IRC02:18
*** zzzeek has joined #openstack-kolla02:22
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134002:25
openstackgerritLin PeiWen proposed openstack/kolla-ansible master: Use Docker healthchecks for memcached services  https://review.opendev.org/c/openstack/kolla-ansible/+/78377903:13
*** skramaja has joined #openstack-kolla03:29
*** ricolin_ has quit IRC03:47
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134003:59
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134004:38
*** ricolin has quit IRC04:47
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Add support for disabling amphora provider  https://review.opendev.org/c/openstack/kolla-ansible/+/78369905:26
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Disable amphora in OVN jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/78366905:26
*** cah_link has joined #openstack-kolla05:29
*** bengates has joined #openstack-kolla05:32
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Add --nocache option to container image build  https://review.opendev.org/c/openstack/kayobe/+/76756605:32
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: octavia: Add support for disabling amphora provider  https://review.opendev.org/c/openstack/kolla-ansible/+/78369905:37
*** bengates has quit IRC05:37
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134005:38
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: CI: Disable amphora in OVN jobs  https://review.opendev.org/c/openstack/kolla-ansible/+/78366905:44
yoctozeptomorning06:04
*** vishalmanchanda has joined #openstack-kolla06:19
*** jopdorp has joined #openstack-kolla06:19
mnasiadkaSo, I started seeing permission denied errors in octavia-api-error.log after switch to WSGI06:20
mnasiadkahttps://efaaf9dc66b562e2873d-28a40d3de53ae200fe2797286e214883.ssl.cf2.rackcdn.com/783699/2/check/kolla-ansible-ubuntu-source-ovn/0fec4d9/primary/logs/kolla/octavia/octavia-api-error.txt06:20
mnasiadkain theory WSGI is running as user/group=octavia06:21
mnasiadkaso what the heck?06:21
*** cah_link has quit IRC06:26
*** e0ne has joined #openstack-kolla06:30
*** k_mouza has joined #openstack-kolla06:40
*** k_mouza has quit IRC06:45
yoctozeptomnasiadka: checked the file permissions?06:46
yoctozeptoare they really set?06:46
mnasiadkawell, I'll deploy on my test env and will see06:46
mnasiadkawondering how the heck did we not see it in the check pipelines06:47
yoctozeptoprobably some technical debt, as usual06:48
*** cah_link has joined #openstack-kolla06:54
dardeleanwhich images are more stable in your experience, source or binary?06:59
dardeleanI had some issues with binary before so that's why I'm always using source06:59
*** bengates has joined #openstack-kolla07:02
*** bengates_ has joined #openstack-kolla07:06
*** bengates has quit IRC07:07
*** gfidente|afk is now known as gfidente07:13
*** bengates_ has quit IRC07:14
*** bengates has joined #openstack-kolla07:15
*** k_mouza has joined #openstack-kolla07:17
*** k_mouza has quit IRC07:22
*** Luzi has joined #openstack-kolla07:23
*** luksky has joined #openstack-kolla07:25
*** rpittau|afk is now known as rpittau07:29
mgoddardmnasiadka: probably because kolla-ansible merged but images are still using octavia user?07:42
mnasiadkamgoddard: might be, but it was merged 5 days ago (master)07:43
hrw5. we publish weekly07:51
*** ricolin has joined #openstack-kolla07:52
*** wuchunyang has joined #openstack-kolla07:55
*** k_mouza has joined #openstack-kolla07:56
mnasiadkahrw: in master as well?08:00
mnasiadkawelcome to hell08:00
mnasiadkabut still, it's been merged 5 days ago, I would assume we do publish somewhere around weekend or on Mondays08:02
hrwI lost track08:05
hrwmeh, why I even asked about aarch64 images08:06
yoctozeptomnasiadka, hrw, mgoddard: I will push with quay then08:10
yoctozeptogimme ffe on pacemaker+masakari08:10
yoctozepto:D08:10
mnasiadkastop swearing :)08:10
yoctozeptowhere is the swear?08:10
yoctozeptoffe - feature freeze exception08:11
*** rohit02 has joined #openstack-kolla08:12
*** bodgix has quit IRC08:14
*** hrw has quit IRC08:14
*** bodgix has joined #openstack-kolla08:14
*** hrw has joined #openstack-kolla08:14
hrwfatal: [jagular.local]: FAILED! => {"changed": true, "cmd": ["docker", "exec", "openvswitch_vswitchd", "ovs-vsctl", "set", "Open_vSwitch", ".", "external_ids:system-id=jagular"], "delta": "0:00:00.246980", "end": "2021-03-30 10:18:16.998172", "msg": "non-zero return code", "rc": 137, "start": "2021-03-30 10:18:16.751192", "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}08:18
hrwargh08:18
mnasiadkayoctozepto: pacemaker08:18
hrwcan I disable openvswitch in all-in-one?08:18
yoctozeptomnasiadka: lol, I exploded with laugh, thanks08:18
mnasiadkahrw: if you don't need networking or want linuxbridge - sure08:19
mnasiadkahrw: but I have a problem understanding why that command would fail :)08:19
hrwmnasiadka: that makes two of us08:20
yoctozeptorc 13708:20
yoctozeptois not that a timeout?08:20
yoctozeptoor forced stop?08:20
yoctozeptoSIGKILL08:21
yoctozeptothis08:21
yoctozeptosomething sent it SIGKILL08:21
hrwmeh.08:23
hrwI just want to deploy, init, create VM and destroy08:23
yoctozeptowell, it probably got killed by the container exiting08:26
yoctozeptoso need to check container logs08:26
hrwTASK [destroy : Removing Kolla images]08:29
hrwredeploying08:29
openstackgerritMark Goddard proposed openstack/kayobe stable/victoria: Avoid fact gathering for group_by  https://review.opendev.org/c/openstack/kayobe/+/78369208:33
*** kevko has joined #openstack-kolla08:50
kevkomorning \o/08:54
kevkoyoctozepto: could you check mariadb refactor again ? i have +1 from mgoddard finally :P , and also ..i've added reno to this tiny change ->  https://review.opendev.org/c/openstack/kolla-ansible/+/775627 , i think it's enough, isn't it ?08:55
hrw+++ modprobe openvswitch08:56
hrwmodprobe: ERROR: ../libkmod/libkmod.c:586 kmod_search_moddep() could not open moddep file '/lib/modules/5.10.25+/modules.dep.bin'08:56
hrwmodprobe: FATAL: Module openvswitch not found in directory /lib/modules/5.10.25+08:56
hrwyay08:56
bjolohowdy08:57
bjolovictoria is runing es och kibana 608:57
bjolois it possible to handle wallaby es+kibana containers with kolla-ansible victoria?08:58
bjoloanyone tried that?08:58
mnasiadkahrw: lost your kernel modules on the way? :)08:58
kevkobjolo: nope, but it should work08:58
mgoddardbjolo: there was a k-a patch to migrate to ES709:00
openstackgerritMark Goddard proposed openstack/kolla-ansible master: docs: Improve policy documentation  https://review.opendev.org/c/openstack/kolla-ansible/+/78380909:04
bjoloi just deployed es6 on victoria so not much to migrate09:05
bjoloi can just toss that and deploy wallaby containers instead09:06
bjoloi am a little concerned about fluentd not loggin properly though. almost all data i see in kibana is haproxy and fluentd09:07
bjolowhere is nova, neutron, etc?09:07
openstackgerritMark Goddard proposed openstack/kayobe stable/ussuri: Avoid fact gathering for group_by  https://review.opendev.org/c/openstack/kayobe/+/78369309:10
mgoddardbjolo: it's more about updating config than migrating data09:13
*** e0ne has quit IRC09:14
bjolook... my plan was to build wallaby es kibana, override image tag i globals, and deploy them with kolla-ansible victoria09:14
bjolobut they will get the wrong config then?09:15
mgoddardbjolo: yes09:15
bjoloso i need to override the config as well or is there another way?09:16
mgoddardbjolo: I think you proably need commit c2e08be414b45156b189c4923bb587596d07ef6b09:17
openstackgerritMark Goddard proposed openstack/kayobe master: zuul: add more irrelevant files  https://review.opendev.org/c/openstack/kayobe/+/78381909:21
openstackgerritMark Goddard proposed openstack/kayobe stable/train: Avoid fact gathering for group_by  https://review.opendev.org/c/openstack/kayobe/+/78369409:21
bjolook i try that09:32
bjolothank you :)09:32
hrwfatal: [localhost]: FAILED! => {"changed": false, "checksum": "81ae224fff596d018535c0c8799a09e0d93e3349", "msg": "Aborting, target uses selinux but python bindings (libselinux-python) aren't installed!"}09:34
hrwhm09:34
hrwpython3-selinux is installled on target09:38
hrwheh09:38
*** e0ne has joined #openstack-kolla09:41
*** e0ne has quit IRC09:41
hrwwhy kolla-ansible does not depend on selinux?09:42
hrwhttps://review.opendev.org/c/openstack/kolla/+/782386 - can someone +W?09:45
*** e0ne has joined #openstack-kolla09:45
mgoddardhrw: I'll wait for zuul09:46
*** k_mouza has quit IRC09:49
*** k_mouza has joined #openstack-kolla09:49
hrwok09:52
*** dking has joined #openstack-kolla10:00
*** wuchunyang has quit IRC10:14
openstackgerritPierre Riteau proposed openstack/kayobe-config master: Stop accessing unbound variables  https://review.opendev.org/c/openstack/kayobe-config/+/78383610:28
*** Luzi has quit IRC10:34
mnasiadkayoctozepto, hrw, mgoddard: https://zuul.openstack.org/builds?job_name=kolla-publish-centos8-source&project=openstack%2Fkolla&branch=master - am I blind, or the last publish was at 14th of March?10:36
yoctozeptomnasiadka: https://zuul.openstack.org/builds?job_name=kolla-publish-centos8s-source&project=openstack%2Fkolla&branch=master10:37
mnasiadkaah right10:37
kevkoyoctozepto: could you check my mariadb refactor again please ? I've need second +1 :/, and also I've added reno to https://review.opendev.org/c/openstack/kolla-ansible/+/775627 << could you check ?10:40
yoctozeptokevko: everything has its time, be patient, friend :-)10:41
kevkoyoctozepto: no problem, i've just waited for answer :P ..thanks10:41
*** ricolin_ has joined #openstack-kolla10:52
*** hjensas has quit IRC10:53
*** ricolin_ has quit IRC10:54
openstackgerritMerged openstack/kayobe stable/train: CI: Fix kayobe-tox-molecule job  https://review.opendev.org/c/openstack/kayobe/+/78361511:04
mnasiadkakevko: so, looking at that change - where are the docs?11:04
*** cah_link has quit IRC11:11
*** ricolin_ has joined #openstack-kolla11:16
*** hjensas has joined #openstack-kolla11:19
*** openstackgerrit has quit IRC11:21
*** ricolin_ has quit IRC11:24
*** rohit02 has quit IRC11:32
*** rohit02 has joined #openstack-kolla11:32
*** kevko has quit IRC11:51
*** e0ne has quit IRC11:56
*** Luzi has joined #openstack-kolla12:00
mnasiadkaso, octavia is broken in master12:04
mnasiadkaextend_start creates /var/log/kolla/octavia (as root), and we set chmod 755 on /var/log/kolla/octavia12:05
yoctozeptoso we should also set the user then12:08
mnasiadkayep, pushing a change12:14
mnasiadkato do it in the same way as we do in placement12:14
mnasiadkahttps://review.opendev.org/c/openstack/kolla/+/78384512:15
mnasiadkayoctozepto: I think we finally need to sort this out in a better way, e.g. add support for create: yes in kolla_set_configs permissions section...12:17
mnasiadkaand this https://review.opendev.org/c/openstack/kolla-ansible/+/78384612:19
mnasiadkaehh12:19
*** kevko has joined #openstack-kolla12:42
kevkomnasiadka: well, i don't know where to place doc ?12:44
kevkomnasiadka: to some existing docs ? advanced configuration ? or some separate docs for this tiny change ?12:45
mnasiadkakevko: under doc/source :) probably advanced-configuration.rst is fine12:47
kevkomnasiadka: and which change do you mean ? that haproxy weight ?12:48
*** k_mouza has quit IRC12:54
*** k_mouza_ has joined #openstack-kolla12:54
mnasiadkakevko: yep12:55
yoctozeptomgoddard: I have not seen the thread about registry credentials so I am assuming we are just going with the current approach for now?12:55
kevkomnasiadka: i tried to find some docs for haproxy itself ..but didn't find ..12:55
mnasiadkakevko: there's a part in advanced-configuration about disabling haproxy, or we can have a reference/ doc about haproxy as a starter12:56
mgoddardkevko: doc/source/reference/databases/12:56
mnasiadkamgoddard: haproxy weight in reference/databases?12:56
mgoddardyoctozepto: haven't emailed yet12:56
mgoddardmnasiadka, kevko: no, assumed it was mariadb12:56
kevkomgoddard: this tiny change -> https://review.opendev.org/c/openstack/kolla-ansible/+/77562712:57
mgoddardstill, we'll probably want a load-balancers section for proxysql12:58
mgoddard(in reference)12:58
kevkomgoddard: agree, but i think it should be provided in proxysql patch ..12:59
mgoddardmaybe13:00
mgoddardstill, this is more like reference info13:01
mgoddardadvanced-config is just a dumping ground13:01
mnasiadkaso kevko can start reference/haproxy13:03
mnasiadka:)13:03
yoctozepto15:01:16 <mgoddard> advanced-config is just a dumping ground13:03
yoctozeptoso true13:03
kevkohmm, ok, going to do13:03
mnasiadkakevko: around passwords... if I would tell some security officer we have multiple databases with the same root password - I think I would need to pack up :)13:04
kevkomnasiadka: yes, i know what u mean ..13:04
hrwhttps://review.opendev.org/c/openstack/kolla/+/782386 needs +W and with https://review.opendev.org/c/openstack/kolla/+/783681 will make ussuri debian aarch64 working again13:10
hrwthen https://review.opendev.org/c/openstack/kolla/+/783678 + https://review.opendev.org/c/openstack/kolla/+/783679 will fix Victoria13:11
hrwfor same target13:11
hrwhttps://review.opendev.org/c/openstack/kolla/+/783713 and https://review.opendev.org/c/openstack/kolla/+/783698 are for master13:12
mnasiadkayoctozepto: I think I found cephadm quirkness: Mar 30 13:01:21 primary bash[75459]: debug 2021-03-30T13:01:21.844+0000 7fa30431f700 -1 error: monitor data filesystem reached concerning levels of available storage space (available: 4% 1.8 GiB)13:22
*** spotz has joined #openstack-kolla13:30
*** Luzi has quit IRC13:31
*** wuchunyang has joined #openstack-kolla13:33
*** skramaja has quit IRC13:43
*** k-s-dean has joined #openstack-kolla13:44
yoctozeptomnasiadka: oooh13:50
yoctozeptowhat is that filesystem13:50
yoctozeptoI thought we had one big fs for everything :D13:51
yoctozeptoperhaps not?13:51
hrwok. deploying victoria aio...13:56
mnasiadkayoctozepto: maybe not so big, trying with a change to use 5G OSDs instead of 10G, but I don't know Octopus will like it :)13:58
mnasiadkaother option is to template out ceph.conf with disabling those crit checks13:59
yoctozeptomnasiadka: odd, we should have like 50G in there13:59
yoctozepto10G should not hurt13:59
yoctozeptobut meh13:59
mnasiadkawell, 1.8GiB available doesn't sound nice13:59
yoctozeptoindeed13:59
mnasiadkaand what is weird, it only happens sometimes13:59
yoctozeptoyup13:59
mnasiadkaso maybe one nodepool provider is special?13:59
yoctozeptoas if something else was filling the space14:00
yoctozeptoor that14:00
mnasiadkaI'll analyse list of failed jobs for some common denominator14:00
yoctozeptothanks14:01
*** jamesbenson has joined #openstack-kolla14:02
jamesbensonmorning all, has anyone re-configured from 3node controller/network to 1 node controller/network successfully?14:02
yoctozeptoI would dare to say it's a very uncommon thing to do14:10
*** vishalmanchanda has quit IRC14:19
jamesbensonWe have an issue with the networking that everything seems to be communicating but there is a break between br-ext and the floating IP's.  But absolutely no errors in nova/neutron.  Internal networking works still (ie I can ping vm to vm), but vm is dead in both directions in regard to public network.14:35
*** rohit02 has quit IRC14:36
jamesbensonWe have ussuri & enable_neutron_agent_ha.14:39
*** wuchunyang has quit IRC14:43
*** jamesbenson has quit IRC14:48
hrw2021-03-30 16:49:16.268 7 ERROR nova.compute.manager [instance: d7139cac-dbb1-4f17-af65-8a44bb481b8d] nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed14:52
hrwsomething different this time ;d14:52
*** jamesbenson has joined #openstack-kolla14:53
*** jamesbenson has quit IRC14:57
*** jamesbenson has joined #openstack-kolla14:57
*** k_mouza_ has quit IRC15:02
*** k_mouza has joined #openstack-kolla15:02
*** ysirndjuro has joined #openstack-kolla15:05
*** jamesbenson has quit IRC15:17
*** e0ne has joined #openstack-kolla15:19
*** rpittau is now known as rpittau|afk15:33
*** jamesbenson has joined #openstack-kolla15:39
*** e0ne has quit IRC15:57
*** jamesbenson1 has joined #openstack-kolla15:58
*** jamesbenson has quit IRC16:01
*** jamesbenson1 has quit IRC16:02
*** wuchunyang has joined #openstack-kolla16:05
*** wuchunyang has quit IRC16:14
*** jamesbenson has joined #openstack-kolla16:14
*** kevko_ has joined #openstack-kolla16:17
*** kevko_ has quit IRC16:22
*** kevko_ has joined #openstack-kolla16:23
*** kevko_ has quit IRC16:28
*** kevko_ has joined #openstack-kolla16:30
*** cah_link has joined #openstack-kolla16:32
*** kevko_ has quit IRC16:34
*** kevko_ has joined #openstack-kolla16:36
*** k_mouza has quit IRC16:38
*** kevko_ has quit IRC16:40
*** kevko_ has joined #openstack-kolla16:42
*** bengates has quit IRC16:43
*** kevko_ has quit IRC16:46
*** kevko_ has joined #openstack-kolla16:47
mnasiadkaok, heading for some days off, will be back after Easter.16:51
*** kevko_ has quit IRC16:51
*** kevko_ has joined #openstack-kolla16:52
*** kevko_ has quit IRC16:58
*** kevko_ has joined #openstack-kolla16:59
parallaxhave a nice one ;)17:00
*** kevko_ has quit IRC17:04
*** jamesbenson has quit IRC17:05
yoctozeptomnasiadka: enjoy your weekend17:05
*** kevko_ has joined #openstack-kolla17:11
*** bengates has joined #openstack-kolla17:13
*** gfidente is now known as gfidente|afk17:14
*** kevko_ has quit IRC17:16
*** bengates has quit IRC17:17
*** jamesbenson has joined #openstack-kolla17:23
*** jamesbenson has quit IRC17:27
*** cah_link has quit IRC17:29
*** cah_link has joined #openstack-kolla17:34
*** bengates has joined #openstack-kolla17:45
*** jamesbenson has joined #openstack-kolla17:48
*** jamesbenson has quit IRC17:49
*** bengates has quit IRC17:49
*** kevko has quit IRC18:03
*** spy is now known as antonym18:10
*** cah_link has quit IRC18:16
*** muhaha has joined #openstack-kolla18:38
*** k_mouza has joined #openstack-kolla18:39
*** k_mouza has quit IRC18:43
*** muhaha has quit IRC19:20
*** muhaha has joined #openstack-kolla19:24
*** iniazi has quit IRC19:28
*** muhaha has quit IRC20:09
*** kevko has joined #openstack-kolla20:12
*** k-s-dean has quit IRC20:19
*** Aethylred has joined #openstack-kolla20:52
*** kevko has quit IRC20:56
*** k_mouza has joined #openstack-kolla20:59
AethylredWould setting up Read-Only LDAP authentication be considered a post-install activity for Kolla? I'm trying to set it up in an initial deployment and the keystone bootstrapper keeps trying to create LDAP accounts (but it can't because it's read-only)21:03
*** k_mouza has quit IRC21:03
*** zzzeek has quit IRC21:13
*** zzzeek has joined #openstack-kolla21:15
*** k_mouza has joined #openstack-kolla21:27
*** k_mouza has quit IRC21:32
*** Fl1nt has quit IRC22:09
*** brinzhang_ has quit IRC22:35
*** brinzhang_ has joined #openstack-kolla22:36
*** osmanlicilegi has quit IRC22:42
*** calbers has quit IRC22:42
*** Jeffrey4l has quit IRC22:42
*** openstackstatus has quit IRC22:42
*** dciabrin_ has joined #openstack-kolla22:42
*** kklimonda_ has joined #openstack-kolla22:42
*** calbers_ has joined #openstack-kolla22:42
*** calbers_ is now known as calbers22:42
*** kklimonda has quit IRC22:42
*** openstack has joined #openstack-kolla22:45
*** ChanServ sets mode: +o openstack22:45
*** antonym has joined #openstack-kolla23:09
*** luksky has quit IRC23:10
*** openstackgerrit has joined #openstack-kolla23:22
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134023:22
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134023:39

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!