Monday, 2021-03-22

*** LinPeiWen has joined #openstack-kolla00:55
*** ivan_lin has quit IRC01:33
*** ivan_lin has joined #openstack-kolla01:34
*** ricolin has joined #openstack-kolla03:08
*** skramaja has joined #openstack-kolla04:30
*** vishalmanchanda has joined #openstack-kolla05:11
*** ricolin_ has joined #openstack-kolla05:52
*** Luzi has joined #openstack-kolla07:02
*** amoralej|off is now known as amoralej07:35
*** amoralej is now known as amoralej|off07:35
*** bengates has joined #openstack-kolla08:00
*** bengates has quit IRC08:04
*** bengates has joined #openstack-kolla08:05
*** bengates has quit IRC08:07
*** bengates has joined #openstack-kolla08:08
*** bengates has quit IRC08:11
*** bengates has joined #openstack-kolla08:12
*** andrewbonney has joined #openstack-kolla08:13
*** bengates has quit IRC08:17
*** bengates has joined #openstack-kolla08:17
*** rpittau|afk is now known as rpittau08:19
*** ug_egon has joined #openstack-kolla08:20
*** jbadiapa has joined #openstack-kolla08:59
mgoddardmorning09:00
*** amoralej|off is now known as amoralej09:04
*** e0ne has joined #openstack-kolla09:10
hrwmorng09:24
*** gfidente has joined #openstack-kolla09:24
hrwyoctozepto: argh.09:27
hrwlet me deploy something on local machine09:35
hrwhm. one month old images. time to build new ones09:36
mnasiadkamorning09:42
*** kevko has joined #openstack-kolla09:54
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/victoria: nova-cell: Stop printing ceph keys in output  https://review.opendev.org/c/openstack/kolla-ansible/+/78211810:03
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/ussuri: nova-cell: Stop printing ceph keys in output  https://review.opendev.org/c/openstack/kolla-ansible/+/78211910:03
openstackgerritMark Goddard proposed openstack/kolla-ansible stable/train: nova-cell: Stop printing ceph keys in output  https://review.opendev.org/c/openstack/kolla-ansible/+/78212010:04
openstackgerritMichal Nasiadka proposed openstack/kayobe master: Bump max Ansible version to 2.10  https://review.opendev.org/c/openstack/kayobe/+/78089810:05
mgoddardhrw: what's the status of https://review.opendev.org/c/openstack/kolla/+/781494 ?10:06
hrwmgoddard: need to check why it failed on k-a10:08
hrwmgoddard: other than that: mergeable10:08
mgoddardok10:08
hrwINFO:kolla.common.utils.nova-libvirt:Trying to push the image10:10
hrwERROR:kolla.common.utils.nova-libvirt:Get http://127.0.0.1:4000/v2/: dial tcp 127.0.0.1:4000: connect: connection refused10:10
hrwfeels CI issue10:10
kevkohrw: i know why you failed :)10:10
hrwkevko: where?10:10
kevkohrw: in https://review.opendev.org/c/openstack/kolla/+/78149410:10
hrwkevko: then share that knowledge10:11
kevkohrw: you have to rebase on top of my CI fix patch ->  https://review.opendev.org/c/openstack/kolla-ansible/+/78115110:11
hrwkevko: please rebase it then.10:12
hrwdid recheck - should take care of it10:12
hrwkevko: thanks10:12
kevkohrw: oh, it's kolla project ..so It should work against current kolla-ansible master because it is already merged10:13
kevkohrw: so, probably recheck will work10:13
hrwkevko: that's why recheck.10:13
kevko+110:13
hrwredeploying 0322-1 images on aarch64 to check do they work at all10:14
*** ricolin_ has quit IRC10:16
hrwok, destroyed. time for deploy10:28
hrwrabbitmq--10:35
hrw6efd5a69bb9   puchatek.lan:5000/master-0322-1/debian-source-rabbitmq:11.1.0               "dumb-init --single-…"   2 minutes ago   Restarting (0) 46 seconds ago             rabbitmq10:36
*** bengates has quit IRC10:37
hrwError: This RabbitMQ version cannot run on Erlang 22.1.7 (erts 10.5.5): minimum required version is 22.3 (erts 10.7)10:37
hrwyay10:37
*** bengates has joined #openstack-kolla10:38
hrwoh rmq, how I dislike you10:38
*** cah_link has joined #openstack-kolla10:38
kevko:D10:48
hrwit also shows that X cycle will start with merging https://review.opendev.org/c/openstack/kolla/+/772479 patch10:49
hrwto get rid of Buster10:49
openstackgerritMerged openstack/kolla-ansible master: Trivial fixes in grafana-guide doc.  https://review.opendev.org/c/openstack/kolla-ansible/+/78148810:51
*** brtknr has quit IRC10:56
hrwmgoddard: turns out that you and mnasiadka ignored Zuul failing on centos stream 8 binary.10:56
mgoddardhrw: doh10:56
mgoddardhrw: and you, to be fair :D10:57
hrwmgoddard: I did not noticed that too10:57
mgoddardhrw: now we have the root cause, thanks10:57
hrwI need to take some PTO or something10:57
hrwjust this @#%@^@%@ pandemic blocks me from going somewhere to rest a bit on such PTO10:58
mgoddardhrw: rest at home?10:58
hrwunable to10:59
hrwbbiab11:00
mnasiadkacentos binary? do we care? :)11:07
openstackgerritPiotr Parczewski proposed openstack/kolla-ansible master: Add Alertmanger metric target(s)  https://review.opendev.org/c/openstack/kolla-ansible/+/78221011:55
openstackgerritPiotr Parczewski proposed openstack/kolla-ansible master: Add Alertmanger metric target(s)  https://review.opendev.org/c/openstack/kolla-ansible/+/78221011:59
*** cah_link has quit IRC12:11
*** brtknr has joined #openstack-kolla12:13
*** cah_link has joined #openstack-kolla12:28
hrwok. I broke Debian builds for a while12:32
hrwerlang shit12:32
*** cah_link has quit IRC12:35
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: rabbitmq: bump Erlang requirements to v23  https://review.opendev.org/c/openstack/kolla/+/78224712:38
hrw(venv3) 13:32 (0s) hrw@jagular:kolla$ ~/devel/docker-utils/kolla-build-images.sh debian source '--push --registry puchatek.lan:5000 ' master-0322-212:40
hrwand another rebuild...12:40
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: ovn: make DVR work on VLAN tenant networks  https://review.opendev.org/c/openstack/kolla-ansible/+/78091913:02
openstackgerritMichal Nasiadka proposed openstack/kolla-ansible master: ovn: make DVR work on VLAN tenant networks  https://review.opendev.org/c/openstack/kolla-ansible/+/78091913:04
mnasiadkamgoddard, yoctozepto: Finally worked out octavia and OVN with CI - https://review.opendev.org/c/openstack/kolla-ansible/+/761872 - reviews please13:05
hrwkolla-ansible destroy++13:05
*** brinzhang has joined #openstack-kolla13:05
openstackgerritMerged openstack/kolla master: Build Masakari for Debian binary  https://review.opendev.org/c/openstack/kolla/+/78040913:13
openstackgerritMerged openstack/kolla-ansible stable/ussuri: docs: fix registry mirror example  https://review.opendev.org/c/openstack/kolla-ansible/+/78120713:13
openstackgerritMerged openstack/kolla-ansible stable/train: docs: fix registry mirror example  https://review.opendev.org/c/openstack/kolla-ansible/+/78120813:13
openstackgerritMerged openstack/kolla-ansible stable/victoria: docs: fix registry mirror example  https://review.opendev.org/c/openstack/kolla-ansible/+/78120613:14
openstackgerritMerged openstack/kolla-ansible stable/ussuri: nova-cell: Stop printing ceph keys in output  https://review.opendev.org/c/openstack/kolla-ansible/+/78211913:14
*** amoralej is now known as amoralej|lunch13:21
hrwso, who +2 https://review.opendev.org/c/openstack/kolla/+/78149413:30
hrwso we get centos stream 8 binary working?13:30
mgoddardhrw: centos binary job failed...13:35
*** sean-k-mooney has joined #openstack-kolla13:37
hrwha! I got you to look ;D13:38
hrwchecking locally why it failed13:40
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: centos: enable PowerTools repo in images which needs it  https://review.opendev.org/c/openstack/kolla/+/78149413:47
hrwmgoddard: fixed watcher and changed commit message13:48
hrwok, time to dig in nova13:56
yoctozeptomgoddard, mnasiadka, hrw: what to do about quay?13:59
hrwgo for it14:00
* hrw -> firefighting mode14:00
openstackgerritMartin Chlumsky proposed openstack/kolla-ansible master: Switch octavia-api to wsgi running under apache.  https://review.opendev.org/c/openstack/kolla-ansible/+/77989314:06
*** skramaja has quit IRC14:06
hrwyoctozepto: Debian/aarch64 will be fixed14:13
*** luksky has joined #openstack-kolla14:16
hrwforgot to bump libvirt-python ;(14:25
*** amoralej|lunch is now known as amoralej14:25
*** e0ne has quit IRC14:27
hrwmeh, linaro infra issues14:40
hrwso Debian stays broken for a while14:40
kevkomgoddard: around ? did u check mariadb-refactor ? if no, could you check if this is acceptable ? because if no.. i have reworked branch which i can push.. but i will be happy if you can check firstly current version14:41
mgoddardyoctozepto: discuss on wednesday14:43
*** Luzi has quit IRC14:55
yoctozeptomgoddard: ok15:01
yoctozeptofine with my time15:01
mgoddardyoctozepto: I've added it to the agenda15:05
openstackgerritMark Goddard proposed openstack/kayobe master: Ubuntu: AppArmor config for seed VM provisioning  https://review.opendev.org/c/openstack/kayobe/+/78227215:08
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227315:10
openstackgerritMark Goddard proposed openstack/kayobe master: CI: refactor zuul config to use templates  https://review.opendev.org/c/openstack/kayobe/+/78227615:17
*** schwicht has quit IRC15:32
*** schwicht has joined #openstack-kolla15:35
*** e0ne has joined #openstack-kolla15:39
openstackgerritJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133916:08
*** vishalmanchanda has quit IRC16:11
yoctozeptomgoddard: thanks16:20
openstackgerritMark Goddard proposed openstack/kayobe master: CI: refactor zuul config to use templates  https://review.opendev.org/c/openstack/kayobe/+/78227616:30
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Enable bare metal testing for Ubuntu  https://review.opendev.org/c/openstack/kayobe/+/78227316:35
*** amoralej is now known as amoralej|off16:46
*** ysirndjuro has joined #openstack-kolla17:04
*** rpittau is now known as rpittau|afk17:11
*** e0ne has quit IRC17:13
kevkomgoddard: did you check my new patchset ?17:14
mgoddardkevko: hi, not yet17:14
mgoddardkevko: will look later17:14
kevkomgoddard: i've changed few things .. locally I have reworked whole chain (mainly moved set_fact to defaults or groups vars)17:15
kevkobut not pushed yet ..17:15
kevkomgoddard: i have it in case you will not like my current +1 jenkins verified chain :)17:16
mgoddardkevko: looks like this is the only change: https://review.opendev.org/c/openstack/kolla-ansible/+/766952/43..47/ansible/roles/mariadb/tasks/recover_cluster.yml17:17
mgoddardkevko: this comment isn't really addressed: https://review.opendev.org/c/openstack/kolla-ansible/+/766952/42..47/ansible/roles/mariadb/tasks/loadbalancer.yml17:20
kevkomgoddard: this is one of changes i've locally changed .. but it's hard to rebase whole chain .. so i not pushed yet ..17:22
mgoddardok17:22
kevkomgoddard: wait a minute17:22
kevkomgoddard: this is diff what i currently have -> http://paste.openstack.org/show/803782/17:28
kevkomgoddard: but still testing whole chain17:28
kevkomgoddard: is it acceptable ?17:29
kevkomgoddard: i had to move it in group_vars/all as user can run  with --tags haproxy, or --tags mariadb and list of default shard/cluster members has to be accessible from both places ..17:31
kevkobecause tasks/loadbalancer.yml is running always ..if you are running --tags haproxy or --tags mariadb ..but of course ...you know it17:31
openstackgerritJames Kirsch proposed openstack/kolla master: Add LetsEncrypt images for cert request/renewal  https://review.opendev.org/c/openstack/kolla/+/74133917:32
kevko*and removed set_facts tasks17:32
mgoddardkevko: will need to review the whole patch, but it does look like a better approach. mariadb_default_database_shard_id is good, although I don't know if 'default' captures the intent17:33
mgoddardkevko: what does it mean to be in the default shard?17:34
mgoddardhaproxy load balancing? anything else?17:34
mgoddardmaybe mariadb_haproxy_shard_id is clearer?17:34
kevkomgoddard: i have to have id int because in proxysql configuration i need it to generate writer,writer_backup,reader_group etc17:35
kevkoit is identificator of mariadb_shard17:35
kevkoor let's say mariadb_cluster17:35
*** bengates has quit IRC17:36
kevkodefault shard means ..user didn't fill value in inventory ..so default it is 0, to work group_by grouping17:36
hrwyoctozepto, mnasiadka, osmanlicilegi: https://review.opendev.org/c/openstack/kolla/+/78149417:37
*** gfidente is now known as gfidente|afk17:40
hrwmgoddard: I am going to drop ubuntu/aarch64 from CI17:42
hrwmgoddard: it needs newer erlang and I do not have any interest in keeping it alive17:43
mgoddardkevko: yes, although by making it a variable it's configurable. We also need to document what happens for nodes in shard 017:43
mgoddardhrw: :( ok17:44
hrwlooks like no one uses ubuntu in kolla anyway17:44
yoctozeptolol17:44
hrwbecause no one noticed that we use Victoria UCA17:44
kevkomgoddard: as haproxy can LB only one mariadb shard/cluster (one tcp port) ..it is that one which will have VIP:3306 configured ..other shards/cluster will be just deployed17:45
yoctozeptobut but... should not we use Victoria UCA? :P17:45
kevkomgoddard: if proxysql will be turned on, there will be VIP:3306 for all shards/clusters and accessible via root_shard_ID17:45
yoctozeptowallaby still in devel17:46
kevkohrw: we are using ubuntu as baremetal ..but debian images ...17:46
kevko*debian docker images17:46
hrwyoctozepto: we ARE in wallaby. we USE wallaby packages IF they are available17:46
hrwyoctozepto: we use RDO wallaby.17:47
hrwwe used UCA develbranch in past17:47
hrwI was bumping it since train but in wallaby I was less involved in kolla than usual so no one notices17:48
hrwI was bumping it since train but in wallaby I was less involved in kolla than usual so no one noticed17:48
yoctozeptooj tam oj tam17:48
yoctozeptowe check on release17:48
hrwthat's why I suspect that ubuntu images were not used by anyone ;D17:48
kevkooj tam oj tam ? :D17:48
hrwkevko: Polish17:48
kevkono ved toto :) sa mi to zdalo :D17:49
kevkoyeah, understood :D17:49
hrwkevko: we joked few times in past that Kolla one day will switch to Polish17:49
kevkohrw: no problem, understand polish :D17:49
yoctozeptowe will call the project17:49
yoctozeptoklej17:49
yoctozeptoand klej-buduj17:50
kevkojoo :D17:50
yoctozeptojo na pewno :D17:50
kevkofunny language :)17:51
kevkono offense ..17:51
hrwyoctozepto: you forgot about KajObe17:51
yoctozeptohrw: kanabe17:52
kevkohttps://www.mojevideo.sk/video/287ec/pas_jest_kurwa_zapienty_(nastvany_polsky_kamionista).html17:52
kevko:D17:52
hrwkevko: we think similar about czechoslovak17:52
kevkoyeah, i know :)17:52
mgoddardkevko: I think there are 3 modes of usage17:52
yoctozeptoyeah, some of the dissimilarities make it funny to the other party17:53
kevkomgoddard: listening17:53
mgoddard1. old school, single default shard, haproxy17:53
* yoctozepto feels oldschool17:53
mgoddard2. new school, one or more shards, all proxysql17:53
* yoctozepto wants to join new school17:53
mgoddard3. cells, one old school haproxy balanced shard + multiple (single node?) non-load balanced shards17:54
yoctozeptowhy can't it use proxysql too?17:54
mgoddardyoctozepto: it could17:55
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: base: use UCA Wallaby  https://review.opendev.org/c/openstack/kolla/+/78230817:55
yoctozeptothen let's make it so17:55
mgoddardyoctozepto: but that would push all cell DB traffic through proxysql, which you might not want17:55
kevkomgoddard: oh, it's too complex, don't you think ?17:56
yoctozeptomgoddard: why not more proxysql instances?17:56
hrwhttps://review.opendev.org/c/openstack/kolla/+/782247 waits for brave cores wanting to help fix Debian/aarch6417:56
yoctozeptolater this week I will see to that stack17:56
mgoddardyoctozepto: out of scope :)17:56
mgoddard3 is the CERN cells model17:56
yoctozeptomgoddard: :-((((((((((((17:56
kevkomgoddard: well, honestly it is more easy to implement as it is now ..17:57
kevkomgoddard: i'm deleting mariadb haproxy cfg and stop/start haproxy to free 3306 to start proxysql if enabled17:58
mgoddardanyway, those modes were a suggestion17:58
mgoddardif it's just as simple as 1 or 2, then I think some logic could be simplified17:59
kevkomgoddard: if you are saying that you don't want to have all traffic via proxysql ... you probably want to have proxysql/haproxy on different hosts18:01
mgoddardwell, maybe18:01
kevkobut, in this case you have to also another VIP and another keepalived18:01
kevkoi have it written like that ...but you said on begginning it's too complex ..18:01
kevkoso, i've written it on one VIP/keepalived when haproxy OR proxysql can be turned on ..18:02
kevkobut, not saying it is not possible to implement in follow up patches ..18:03
mgoddardit's possible, but out of scope for now18:03
mgoddardanyway, I'm just trying to reverse engineer use cases from code18:03
kevkobut honestly my chain is growing, further, the harder it is to rebase18:03
mgoddardindeed18:04
kevkomgoddard: we are using it now as router to several database shards/clusters18:04
kevkomgoddard: nova, neutron, and others18:04
mgoddardok, you tell me how it's supposed to work18:05
mgoddardwould I ever mixe HAProxy & ProxySQL in front of mariadb?18:05
mgoddardI guess not due to VIP port clash18:05
kevkomgoddard: because connections to DB  ( nova(specially) and neutron was overkilled) ..and we had to split traffic ..18:06
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: ubuntu: drop not existing erlang repo on obs.linaro.org  https://review.opendev.org/c/openstack/kolla/+/78231118:06
mgoddardkevko: I get the pure proxysql use case18:06
mgoddardkevko: what I'm trying to work out, is what other combinations it's supposed to support18:07
kevkomgoddard: but, of course i wrote it as replacement for mariadb LB ..18:07
kevkomgoddard: better monitoring of galera (native), and what is cool ..you can see direct sql statement to DB, gather statistics about connections, sql statements, and ton of usefull informations18:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: CI: drop ubuntu-aarch64 job  https://review.opendev.org/c/openstack/kolla/+/78231218:08
kevkomgoddard: and you can also configuring proxysql without restart !18:09
kevkomgoddard: just configure in internal DB and load config ..18:09
*** andrewbonney has quit IRC18:09
mgoddardkevko: I don't need the proxysql sales pitch :D18:09
kevko:D18:09
mgoddardkevko: just want to know, if using haproxy, are multiple shards supported, and if so, what happens with them?18:10
mgoddard(I think that's the question I've been trying to ask18:10
mgoddardneed to go, child calls18:10
kevkomgoddard: well, we are currently running it in production and it's fine .. but my fear grows every day when it is not occupied in the master, because to carry such patches is not easy18:10
kevkomerged in master18:11
kevkomgoddard: if you are using haproxy, but define several shards in inventory ... you will have default shard behind VIP ..other shards only deployed ..without LB18:12
kevkomgoddard: of course, you can rework internal_haproxy_members to listen on port 3306+shard_id for example ..so you will have several ports maintained by haproxy ..doing LB for serveral shards18:13
kevkomgoddard: but from my perspective of view it is useless .. because you can just use proxysql .. and have only one 3306 port18:14
kevkomgoddard: as it is implemented now (with small changes)- it's possible to have haproxy for default shard ..and proxysql for others18:15
*** jonaspaulo has joined #openstack-kolla18:15
kevkomgoddard: small changes should be change proxysql to listen on different VIP:port for example 3307 ..and testing when: id != default_shard_id18:16
kevkomgoddard: on the end you will have mariadb role which is able to install several shards/clusters ...and you can also configure ids in inventory ..and based on that ids you can split that traffic this way18:17
kevkomgoddard: what is not very nice is ..that in this case every service which will have DB on another shard will have to regenerate config to point to VIP:330718:18
kevkoas it is now .. you just switch haproxy with proxysql ..and everything works ..nothing changed from services view18:19
kevkohrw: looks like no one uses ubuntu in kolla anyway <<< how did you noticed this ? is there some statistics ?18:41
yoctozeptokevko: he just assumed this because noone bumped the uca release for wallaby...18:55
kevkouca ?18:55
yoctozeptokevko: Ubuntu Cloud Archive18:55
yoctozeptoit's analogous to RDO in the CentOS world18:55
yoctozeptoor debian's per-openstack-release repos18:56
kevkobtw, guys, if you want to check current deb packages for OS -> you can check here -> https://debian.kevko.ultimum.cloud/18:56
kevkoah, ok18:56
*** hrw has quit IRC19:13
*** rockey has joined #openstack-kolla19:17
openstackgerritMark Goddard proposed openstack/kayobe master: CI: Fix baremetal SSH  https://review.opendev.org/c/openstack/kayobe/+/78232419:50
*** kevko has quit IRC20:24
*** suff has joined #openstack-kolla20:39
*** jonaspaulo has quit IRC20:43
*** kevko has joined #openstack-kolla20:47
suffHi guys! I want to add heartbeat variables for octavia to upstream. Before my MR, I need to create bug at launchpad?20:51
suffOr features?20:51
openstackgerritPiotr Parczewski proposed openstack/kolla-ansible master: Disable usage collection in Kibana  https://review.opendev.org/c/openstack/kolla-ansible/+/78233021:21
*** e0ne has joined #openstack-kolla21:25
openstackgerritPiotr Parczewski proposed openstack/kolla-ansible master: Disable usage collection in Kibana  https://review.opendev.org/c/openstack/kolla-ansible/+/78233021:28
openstackgerritDmitry Grachev proposed openstack/kolla-ansible master: Add octavia default heartbeats vars  https://review.opendev.org/c/openstack/kolla-ansible/+/78233221:42
*** hrw has joined #openstack-kolla21:44
yoctozeptosuff: it is very likely enough to propose the change with proper commit message, documentation etc.22:01
yoctozeptowe will let you know what's missing22:01
yoctozeptoalso, we are mostlly in European timezones here so don't feel bad if we don't reply THAT LATE :D22:02
*** kevko has quit IRC22:07
suffyoctozepto: Ok, thanks! I am in European timezone too :)22:15
*** e0ne has quit IRC22:28
*** e0ne has joined #openstack-kolla22:29
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134022:33
*** e0ne has quit IRC22:41
*** e0ne has joined #openstack-kolla22:46
*** e0ne has quit IRC22:51
*** e0ne has joined #openstack-kolla23:08
*** e0ne has quit IRC23:09
openstackgerritJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134023:21

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!