Monday, 2022-06-13

opendevreviewMark Goddard proposed openstack/kayobe-config-dev stable/yoga: Avoid rebooting after disabling SELinux  https://review.opendev.org/c/openstack/kayobe-config-dev/+/84539408:15
opendevreviewMark Goddard proposed openstack/kayobe-config-dev stable/xena: Avoid rebooting after disabling SELinux  https://review.opendev.org/c/openstack/kayobe-config-dev/+/84539508:15
opendevreviewMark Goddard proposed openstack/kayobe-config-dev stable/wallaby: Avoid rebooting after disabling SELinux  https://review.opendev.org/c/openstack/kayobe-config-dev/+/84539608:15
opendevreviewMark Goddard proposed openstack/kayobe-config-dev stable/wallaby: Avoid rebooting after disabling SELinux  https://review.opendev.org/c/openstack/kayobe-config-dev/+/84539608:15
Fl1ntHi everyone!08:17
Fl1ntI'm facing a weird situation with Ironic team. We enable pxe_filter==dnsmasq (default?), so inspector put the host port mac address tagged with pxe_enabled as whitelisted but once it finish inspecting the node,08:19
Fl1ntit add back ",ignore" to the end of the mac08:19
Fl1ntwhich prohibit the deploy step to correctly work08:19
Fl1ntas the node is rebooted but the inspector image can't be loaded as the DHCP_REQUESTS are filtered.08:20
Fl1ntis there a way to tell ironic-conductor to whitelist this node on dnsmasq?08:21
opendevreviewMark Goddard proposed openstack/kayobe master: ironic: revert to ironic's default drivers & interfaces  https://review.opendev.org/c/openstack/kayobe/+/83699908:54
Fl1ntmgoddard, did you ever faced this situation using IRONIC? I've just re-tested, each time the inspector finish to inspect a node, it then disallow it from DHCP requests, which prohibit you to deploy the host.09:10
kevko\o/09:58
Fl1nto/09:58
kevkohow do you feel about doing a review for me :p ? Fl1nt  ? :D 09:59
Fl1ntFine, send me your link :D09:59
kevko;D09:59
kevkoFl1nt: https://review.opendev.org/q/hashtag:proxysql+(status:open%20OR%20status:merged) < yeah, proxysql :D 10:00
Fl1ntLet's go :D10:01
Fl1nt7 of them?10:01
Fl1ntor a specific one?10:01
kevkowell, add proxysql support is the main one 10:04
Fl1ntoki dooki10:04
kevkoFl1nt: Edit services roles to support database sharding << this is just change defaults so proxysql-config can works ..and others are just CIs 10:05
Fl1ntlooking for the meta one 'Add proxysql'10:05
kevkoyeah, thanks, I am again preparing downstream branches ...and I think this is ready for merge ..so I am just trying to finally find reviewers so I can backport merged patch :P 10:06
Fl1ntFrom what I'm looking it seems to be ready, I've just a couple of comments but they're not blocking from my POV.10:07
kevkoask please 10:09
Fl1ntListing them.10:09
Fl1ntwhy do we set a monitor user to haproxy if proxysql isn't enabled? should be none or null/empty/whatever rather isn't?10:10
Fl1ntHow do you handle shards when users deploy using a All in one setup?10:12
Fl1nthum... adding a keepalived check for Haproxy while Haproxy is waiting for the keepalived VIP will end up in a race condition at some point (already get the issue two years ago).10:13
kevkoFl1nt: nope, haproxy user or let's say monitor user in mariadb cluster is used for several kolla-ansible releases - it's also used in mariadb_clustercheck container to check state of wsrep 10:13
Fl1ntok get it10:14
kevkoFl1nt: user is just replaced for better name - monitor - and with password 10:14
Fl1ntok, sounds good to me for user :D10:15
kevkoFl1nt: i am not sure if I understand your described situation 10:15
Fl1ntwhich one? All in one or Race condition?10:16
kevkoFl1nt: in keepalived container is only one check which is executing checks for services needed VIP .. proxysql, haproxy, etc... 10:16
kevkoFl1nt: if one of that check fail ..VIP is going away 10:16
Fl1nthum... we don't set any monitor password for Haproxy user?10:16
kevkoFl1nt: nope..that is current situation :D 10:17
Fl1ntkevko, yes, but at bootstrap time, you haproxy will wait for the VIP to be ready before answering keepalived for backend/frontend generation, if the VIP then wait for haproxy to be available to declare the VIP available you're on a dependency circle.10:18
Fl1ntI'll comment for the missing password on default haproxy monitor as it's kinda a security issue, but it's not a blocking point for your review IMHO10:19
hrwkevko: are there packages for proxysql for ubuntu 22.04 and centos stream 9?10:20
kevkoFl1nt: this is historical situation because of haproxy option  mysql-check user haproxy post-41 <<< this implementation of mysql check in haproxy worked only without password 10:21
Fl1ntah! argh :(10:21
Fl1nthum... this is really a security issue indeed, accessing the DB freely isn't really something you ever want ^^10:22
Fl1nteven if monitor user isn't supposed to get large rights.10:22
kevkoFl1nt: yeah, i think so, on the other side .. haproxy user don't have real rights ..10:22
Fl1ntWe will figure it out later on for this point as it's out of scope for your contribution.10:23
kevkoFl1nt: well, my patch is fixing this :) 10:23
kevkohrw: https://repo.proxysql.com/ProxySQL/proxysql-2.4.x/jammy/ << 22.0410:23
hrwkevko: https://repo.proxysql.com/ProxySQL/proxysql-2.4.x/centos/ no 9 yet ;(10:24
kevkohrw: yeah, i will add issue to github .. Author is very responsive ...10:24
hrwthanks10:25
Fl1ntok, LGTM to me10:29
Fl1ntBUT 10:29
Fl1ntwill need someone else to look at it too as I may have missed something.10:30
kevkoFl1nt: I still don't get above race condition ...because for now I thin it's working for haproxy in same way 10:30
kevkoFl1nt: yeah, thanks 10:30
Fl1ntyes, and it is causing RC actually :D10:30
Fl1ntbut it's a really rare situation.10:30
Fl1ntthe timeout window on both being large enough, it can still happens on slow servers or specific situation.10:31
mnasiadkahrw: https://review.opendev.org/c/openstack/kolla/+/844904 - missing +w for a reason? ;-)10:34
Fl1ntbe careful, openstacksdk\<0.99.0 <-- this release version is having issues with swift (missing headers).10:42
hrwmnasiadka: k-a part lacks second +210:45
kevkoanyone else for review guys :) ? 10:50
Fl1ntkevko, regarding the command within the config.json, your comment make sense, even if I have issue with the fact that we will have to rebuild the whole container to get a specific flag.10:51
kevkoFl1nt: specific flag ? 10:52
Fl1ntExemple: You've got a power outage, you need to reboot your mariadb cluster that didn't correctly marked the most advanced node, you'll just need to add a --new-wsrep-cluster flag, reboot node by node and then remove this flag again.10:52
Fl1ntwith this exemple I would need to:10:53
Fl1ntBuild specific image -> deploy -> redeploy the normal image.10:53
kevkoFl1nt: I can argument that you should use mariadb_recovery kolla-ansible playbook 10:54
kevkoFl1nt: then everything is done automatically 10:54
Fl1ntit doesn't work with power outage that prohibit you cluster to output a correct grwstate/grastate.10:54
Fl1ntoh ok, my bad, didn't see that it was updated10:56
Fl1ntok, I'm fine with that.10:57
kevko+110:57
Fl1ntDo you have any equivalent process to recover a proxysql cluster?10:57
kevkojust manually ...but i was satisfied with kolla-ansible approach11:00
kevkoFl1nt: i don't remember the reason ..but once i had a problem with something ...let me check 11:00
kevkoFl1nt: i had apatch for mariadb-arbitrator 11:02
kevkohttps://review.opendev.org/c/openstack/kolla-ansible/+/780811/45/ansible/roles/mariadb/tasks/recover_cluster.yml - line 79 - in some situations host which was reported  as host with largest seqno was not correct .. so i sorted numerically and get first 11:04
kevkoFl1nt: ^ 11:04
kevkobut i really don't remember what was the case .. empty seqno ..or -1 or something like that ..i don't know 11:05
opendevreviewMerged openstack/kolla-ansible master: Add keystone_authtoken.service_type  https://review.opendev.org/c/openstack/kolla-ansible/+/83403511:51
kevkohrw: what about you, would you like to review my patches also ?  12:46
hrwkevko: that's k-a and deployment. a bit outside of my knowledge12:47
kevkocommon , it's very simple :P 12:47
kevkobut ok :) 12:47
kevkocome-on :P 12:48
Fl1ntAnyone able to help me with this ironic issue? I've disabled the pxe_filter feature, it's now working fine, BTW I discovered that the current ironic-inspector.conf.j2 template don't get the appropriate shape to let you disable this feature.12:50
Fl1ntCan I patch that? I mean, at least get a way to disable the feature completly ?12:50
Fl1ntnevermind, victoria allow for noop driver in pxe_filter.12:53
opendevreviewMichal Arbet proposed openstack/kolla master: Change kolla_version LABEL to git sha-1  https://review.opendev.org/c/openstack/kolla/+/81872713:02
opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: CI: Switch upgrades xena->yoga to yoga->master  https://review.opendev.org/c/openstack/kolla-ansible/+/84490513:05
opendevreviewMichal Arbet proposed openstack/kolla master: Change kolla_version LABEL to git sha-1  https://review.opendev.org/c/openstack/kolla/+/81872713:06
opendevreviewMerged openstack/kolla-ansible master: Fix typo in endpoint influxdb_internal_endpoint variable  https://review.opendev.org/c/openstack/kolla-ansible/+/84492513:14
kevkomnasiadka: Why this is not merged, it looks like it's important to have cinder active-active properly configured - >  https://review.opendev.org/c/openstack/kolla-ansible/+/76301114:05
mnasiadkakevko: feel free to work on it, it's listed as a priority on the whiteboard14:06
kevkomnasiadka: i can, but where is a problem ? ( probably not clear what cinder-manage will do ? )14:06
mnasiadkakevko: Kolla whiteboard, L151 - everything is there14:07
Fl1ntoooh c'mon... another K/V store to install on CPlane just for cinder active/active coordination??? msg: "Please enable redis or etcd when using Cinder Ceph backend" ?? seriously?14:07
kevkomnasiadka: link please ? i promise that i am going to bookmark it :) 14:08
mnasiadkahttps://etherpad.opendev.org/p/KollaWhiteBoard14:08
Fl1ntCould we make the cinder active/active mode optional then?14:08
mnasiadkamake a feature out of a bug?14:10
mnasiadkaI think those questions are rather meant for weekly meeting, feel free to propose a topic14:11
Fl1ntI mean, I'm running our platforms over CEPH RBD on cinder, but we're definitely not willing to introduce ETCD on the stack and probably not redis neither for licenses issues.14:11
mnasiadkawhat's wrong with bsd 3 clause license? ;-)14:12
Fl1ntAnd I'm still having an hard time understanding what cinder is calling active/active? Dynamic Storage controller assignation for volumes in case a cinder-volume is down? 14:12
mnasiadkaYes, currently if your cinder-volume host goes down - all volumes served by it are unavailable for API operations (resize, etc)14:12
Fl1ntOk, so not really a big deal as you can live migrate any volumes from this host to another random host easily.14:13
Fl1ntmnasiadka, license management at job is out of my hand but I have to fill for a request each time we had a previously unknown software in the stack for review because even with BSD-3 there are commercial issues.14:15
mnasiadkaFl1nt: sounds like my IBM times14:15
Fl1ntyeah pretty much ^^14:15
Fl1ntbut anyway, if cinder need a k/v store for coordination why didn't they simply used memcached? it's exactly that.14:16
mnasiadkait uses tooz - if that supports memcached, then why not14:16
mnasiadkahttps://docs.openstack.org/tooz/latest/user/drivers.html#memcached14:17
Fl1ntseems so: https://docs.openstack.org/tooz/latest/_modules/tooz/drivers/memcached.html14:18
Fl1ntzookeeper aaarrghhh I almost pu**d on my mouth reading that... for god sake, those tools were created by a sick mind...14:21
Fl1ntzKserver.sh lol, ok I'm done with this service ^^14:21
Fl1nttooz is a really welcomed library tbh14:24
kevkohmm, if we are using coordination url, don't have host or backend_host or cluster defined ..we are already active-active, right ? 15:07
kevkodon't need to define cluster = something ..or yes ? 15:07
opendevreviewPierre Riteau proposed openstack/kolla master: Bump prometheus-openstack-exporter version to 1.6.0  https://review.opendev.org/c/openstack/kolla/+/84560115:34
Fl1ntkevko, Coordination url? you mean etcd endpoint?15:35
kevkoredis15:35
Fl1ntcluster is to specify which cluster to use in case of multibackend I think.15:36
kevkoFl1nt: # Name of this cluster. Used to group volume hosts that share the same backend15:36
kevko# configurations to work in HA Active-Active mode. (string value)15:36
kevko#cluster = <None>15:36
kevkoso, if we have only ceph ... and this is not set .. are we active-active ? 15:37
opendevreviewPierre Riteau proposed openstack/kolla master: Bump prometheus-openstack-exporter version to 1.6.0  https://review.opendev.org/c/openstack/kolla/+/84560115:40
Fl1ntkevko, nope, you're not ^^16:08
kevkohow it is possible ? 16:08
Fl1ntbut honestly, it's not that of a burden as this is just a matter of OPS actions not data flow.16:08
Fl1ntpriteau, I'm using 1.6.0 and unfortunately I'm no longer able to retrieve metrics on grafana because of a weird http error at exporter level.16:11
priteauI've used 1.5.0 without problem. I will test it anyway.16:19
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add proxysql support for database  https://review.opendev.org/c/openstack/kolla-ansible/+/77021517:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Edit services roles to support database sharding  https://review.opendev.org/c/openstack/kolla-ansible/+/77021617:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [CI] Test ProxySQL with shards in the nova cells scenario  https://review.opendev.org/c/openstack/kolla-ansible/+/77062117:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: [DNM] Trigger cells job  https://review.opendev.org/c/openstack/kolla-ansible/+/83891617:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use Docker healthchecks for mariadb-server service  https://review.opendev.org/c/openstack/kolla-ansible/+/80561617:22
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Use native fluent-logger instead of tail  https://review.opendev.org/c/openstack/kolla-ansible/+/75577517:37
opendevreviewJames Kirsch proposed openstack/kolla-ansible master: Add support for LetsEncrypt-managed certs  https://review.opendev.org/c/openstack/kolla-ansible/+/74134018:04
opendevreviewDr. Jens Harbott proposed openstack/kolla-ansible master: Further Keystone-related cleanups  https://review.opendev.org/c/openstack/kolla-ansible/+/84374818:31
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication  https://review.opendev.org/c/openstack/kolla-ansible/+/80595418:44
opendevreviewPiotr Parczewski proposed openstack/kolla-ansible master: Add support for configuring Elasticsearch replication  https://review.opendev.org/c/openstack/kolla-ansible/+/80595418:50

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!