Tuesday, 2022-05-24

*** mat_fechner is now known as matfechner07:28
guesswhatGuys? Any idea why horizon container ( Apache ) returns 302 ? It is actually working, but I am tryting to catch 401 response ( when someone is trying to use brute force ) code to implement fail2ban protection. Thanks07:28
hrwyoctozepto: reserved arm system, will play with it07:40
priteauHello. I have updated the white board to highlight that Kayobe's master branch is RED08:01
priteauBroken by python-novaclient 18.0.0 which requires Python 3.808:01
Fl1ntHi everyone!08:01
hrwpriteau: time to drop cs808:04
Fl1nthrw, woot??08:04
Fl1ntWhat could be the reason behind dropping support for a distribution that is officially supported until end of 2024 ??08:05
priteauPython 3.608:06
Fl1ntmeh? Oo08:06
hrwFl1nt: CS8 uses python 3.6, Zed+ requires 3.808:06
Fl1ntdnf install python39 works just fine...08:06
hrwand does not give you gazilions of python3* packages08:06
priteauAnyway Zed doesn't target c8s as a runtime, see https://governance.openstack.org/tc/reference/runtimes/zed.html08:07
hrwC7 had Python 3.6 but no one was insane enough to build whole set of python3 packages to be able to run openstack08:07
Fl1nthrw, what are you talking about, I was running STEIN/TRAIN just fine using PY3 over C7 images...08:08
Fl1nt then migrated to C8 and now using C8S and Debian which is also a pain in its own.08:08
hrwFl1nt: kolla images for c7 ended in train and used py208:08
Fl1ntthat's why we do overrides.08:09
hrwusers can do whatever they want. I say what project did08:09
Fl1ntbut anyway08:09
Fl1ntyep08:09
hrwtrain (and below) had c7/py2. train (and above to yoga) had c8|cs8/py3. zed (and above) needs cs908:10
Fl1ntsure08:10
hrwand we lack external repos for it08:10
Fl1ntwhat do you mean by that?08:11
hrwhttps://review.opendev.org/c/openstack/kolla/+/836664 exists already08:11
hrwFl1nt: no td-agent, elk stack, grafana08:11
Fl1ntyeah, because Openstack as a whole project is going to quickly for companies.08:11
hrwthey do not provide rhel9/cs9 packages/repos yet and use sha1 gpg keys which rhel9 does not support08:12
hrws+and+and/or+g08:12
Fl1ntWe in here literally struggle to follow the updates pace and we are around 20 but Managing a multi-geo cloud + adding features over time + updating OS Distribution/Openstack release/CEPH Release every 6 months or once a year is too much.08:13
Fl1nteven if we do have CICD that help a lot.08:13
hrwaware08:13
hrwand now TC decided to use useless naming to mark that you can skip one release08:14
hrwtock/tick is shitty name imho08:14
Fl1ntyeah well, I'm stuck at Victoria to Wallaby upgrade for now so TC can decide whatever, we will be more than one release away from tip release... And I'm pretty sure many teams/company struggle the same way.08:15
hrwwould be good to have a way to jump 2 or even 4 releases some way08:16
Fl1ntyeah a way for DB to do what is available with ruby/RAK meanning go from point A to Z just following .rak/.sql file history.08:17
Fl1ntBUT08:18
Fl1ntwithin OS there is INI config files that prohibit that too08:18
hrwusers can do V-W upgrade online. I wonder how many will be fine with V-X V-Z V-A requiring shutdown of whole cloud08:18
Fl1ntWe won't ^^08:18
hrwexactly08:18
Fl1ntshutting down a region isn't an option, there is defense systems running on it, can't do that.08:19
Fl1ntmigrating them to another region would take too much time.08:19
hrwdeploy new controllers, migrate controllers and then migrate computes?08:20
Fl1ntdoing it serially is OK tho08:20
Fl1nteach region is having 3 controllers, 3 aux servers and 8 network nodes.08:20
opendevreviewPierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict  https://review.opendev.org/c/openstack/kayobe/+/84311708:21
Fl1ntanyway, at least I can prepare a new migration to C9S for Zed now ^^08:21
Fl1ntBTW is anyone having issue with Glance and multi-backends on Victoria+ releases with glance complaining about swift_store_auth_address required when using swift as one of the available backends?08:22
hrwmeh. turns out that I got VM instead of baremetal ;(08:22
Fl1nthrw, our old system was doing just that, but the host to vm manager was internal and so not that flexible.?08:23
Fl1nt-?08:24
guesswhatGuys? Any idea why horizon container ( Apache ) returns 302 ? It is actually working, but I am tryting to catch 401 response ( when someone is trying to use brute force ) code to implement fail2ban protection. Thanks  ( sorry for double post, dc.. )08:24
Fl1ntguesswhat, plain HTTP endpoint doing permanent redirect to TLS.08:24
Fl1ntmaybe?08:24
guesswhatFl1nt:  I am not using TLS08:25
guesswhatInfact, I am using custom reverse proxy with SSL offloading08:25
Fl1ntCustom proxy? Nginx?08:25
Fl1ntvarnish?08:25
Fl1ntyour proxy is maybe throwing 302 when translating frontends to backends requests.08:26
guesswhatHAProxy.. Its HAProxy->HAProxy(kolla)->Apache(kolla)08:26
guesswhateven curl to horizon container returns 302 ...08:26
guesswhat*directly08:27
Fl1ntmen... this isn't custom proxy... it's the official kolla way to do it.08:27
Fl1ntok08:27
Fl1ntwhich endpoint?08:27
guesswhat . /auth/login08:27
Fl1ntok so keystone, yes it's normal, look at the horizon wsgi config08:27
guesswhatnot sure if is possible to use form data like curl -X POST -F username=admin -F password=foo /auth/login08:28
Fl1ntdepends on your authentication method08:28
Fl1ntbut I would rather use the keystone endpoint directly using correct API workflow08:28
guesswhatHAProxy is running on PFsense ( thats how I have Horizon exposed to the internet ), also using Crowdsec ( alternative to Fail2ban ) to evaluate "bad" behaviour08:31
Fl1ntin your case, as you probably want to filter out front requests, you'll have two options (that I can think of straight away), one is to catch requests at haproxy level and apply fail2ban from there or at apache level, however, at apache level means implementing rules for each endpoints.08:31
opendevreviewMark Goddard proposed openstack/kolla-ansible master: CI: Fix prometheus-efk scenario with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/84311908:32
opendevreviewMark Goddard proposed openstack/kolla-ansible master: DNM: test prometheus-efk with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/84312008:32
guesswhatIt can detect bad behaviour automatically, detect bruteforce, but the problem is that horizon returns 302.. ( but its working, so its confusing )08:33
Fl1ntyes, it's normal, look in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/keystone/templates/wsgi-keystone.conf.j208:34
Fl1nthorizon always redirect you to /auth/login or /<dashboard_section> depending on if you're logged in or not.08:36
Fl1ntwhoops, bad link but still, same idea08:36
Fl1nthrw, are you using multi-backend with glance at any chance?08:37
hrwFl1nt: I am that weird one who does not have any cloud08:38
Fl1ntarff :D08:39
Fl1ntok ok, no biggies :D08:39
guesswhatSo any idea how to do hardening then?08:40
Fl1ntguesswhat, do it at the endpoints themselves and limit requests on frontend (horizon) so you can't be abused.08:41
guesswhatrunning fail2ban/crowsec on keystone endpoint and use haproxy ratelimit for horizon, right?08:43
Fl1ntguesswhat, let me sniff my horizon traffic to see what's exactly is the behavior.08:43
Fl1ntfail2ban rules for every endpoints from haproxy logs as your traffic is incoming straight to this VIP.08:45
Fl1ntthen apply rules depending on the type of abuse, ratelimit/ban etc.08:46
Fl1ntTBN: Ban isn't legal for some countries ^^08:46
hrw# lscpu08:47
hrwArchitecture:        aarch6408:47
hrwByte Order:          Little Endian08:47
hrwCPU(s):              9608:47
hrwOn-line CPU(s) list: 0-9508:47
hrwThread(s) per core:  108:47
hrwCore(s) per socket:  1608:48
hrwSocket(s):           608:48
hrwNUMA node(s):        208:48
hrwok. now I can deploy openstack08:48
Fl1ntwait, 6 sockets? HW or SW ?08:50
Fl1nthrw, what kind of hw is that?08:50
hrwarm. Real config is 2 sockets, 48 cores per socket08:52
kevkohmm, intertisting that you want to ban IP when it's trying to do some brute force attack 08:52
kevkowhat if that IP is NAT and you block hundreds of regular users ? 08:53
Fl1ntkevko, that's exactly why it's illegal to strictly BAN for some countries.08:53
Fl1nthrw, interesting! is that a full SBSA hw ?08:54
hrwFl1nt: that's old ThunderX108:55
kevkoin previous company (mail platform, not openstack cloud) we were analyzing several fields from log (for example geo location, number of tries, if they are sending POSTS ..etc ..etc) ..and then we had some DB with "weights" of that users ..and if some user exceeded weight ..we've redirected him to dummy frontend :D 08:56
guesswhatMaybe ratelimit and 2FA would be better than banning shared IPs08:56
hrwiirc it was sbsa l308:56
kevko(russia, africa, afganistan ..etc ...they were trying to get into email boxes via bruteforce )08:57
opendevreviewMark Goddard proposed openstack/ansible-collection-kolla master: baremetal: refactor package installation into a separate role  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/82958608:57
kevkobecause if you ban some IP ..attacker will pay another IP if he wants ..and they want :) 08:57
guesswhati woud not need even use banning mechanims directly for each endpoint  then08:59
kevko2FA is good idea i think08:59
Fl1nthrw, so cool ^^09:04
Fl1ntguesswhat, yes, the proper way is to redirect to a HTTP Error page.09:04
Fl1ntor request 2FA09:04
Fl1ntoh and btw, just checked, any request to horizon end up with a 302 as it will necessarily redirect your request to either /auth/login or to a default location that differ from the request you made; so /auth/login once validatedf09:05
Fl1ntredirect to /project/overview09:05
Fl1ntetc09:05
guesswhat2FA will kill automation, like terraform or openstack cli I believ09:05
guesswhati like oauth for CLIs like in Azure and AWS cli...09:06
guesswhat*flow09:07
guesswhatbut its overkill, i have only few users, but the problem is that horizon is exposed to the internet09:07
hrwmake vpn, give them vpn entries09:08
hrwthis way those users have access and you have only one entry to guard09:09
guesswhatwhat about using http-response redirect in haproxy ?09:15
guesswhatFl1nt ^09:15
opendevreviewMerged openstack/kolla-ansible stable/yoga: Fix malformed OIDCMemCacheServers  https://review.opendev.org/c/openstack/kolla-ansible/+/84294009:17
guesswhati believe it would be correct behaviour to return response code, instead of 302 for each endpoint09:17
guesswhatand haproxy is "a central entrypoint" for all endpoinds so i would make a sense to be able to filter traffic there09:18
Fl1nthrw, this is so much pain for users, adding such technicality requirements is a dead shot for many end users09:32
kevkoguesswhat: what is incorrect to return 302 ? 09:32
Fl1ntguesswhat, I would personally filter out from HAPROXY yes, altho it isn't incorrect to return 302, you just need to follow the request till the end and adjust.09:33
kevkoguesswhat: horizon is always redirecting to login if not logged as every login based application 09:33
kevkoguesswhat: http://HORIZON_IP/auth/login/?next=/ is returning 200 ..09:34
Fl1ntguys, glance is driving me crazy... When you use swift as store backend, glance disable the store because it's REQUIRED to get swift_store_auth_address set but yet in the meantime the doc and configuration doc state that this directive is deprecated since few release and replaced by auth_address that I set on the glance-swift.conf file.09:35
Fl1ntis that thing even working???09:35
Fl1nteven if I put a swift_store_auth_address on the config file this isn't working.09:36
yoctozeptopriteau: might be you want https://review.opendev.org/c/openstack/kolla-ansible/+/842842 for kayobe09:38
yoctozeptofor the time being09:38
kevkoFl1nt: haha, how long you are doing with openstack to know that best documentation is reading the code :D :D 09:39
opendevreviewPierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict  https://review.opendev.org/c/openstack/kayobe/+/84311709:40
priteauThanks yoctozepto, I am testing to see what is covered by Zuul checkouts and what isn't09:41
kevkoFl1nt: do yo have multiple swift store accounts ? 09:44
opendevreviewMark Goddard proposed openstack/ansible-collection-kolla master: baremetal: refactor package installation into a separate role  https://review.opendev.org/c/openstack/ansible-collection-kolla/+/82958609:45
kevkoFl1nt: if you set swift_store_user, swift_store_key,swift_store_auth_address is not working ? 09:46
kevkoFl1nt: swift_store_config_file shouldn't be set if you are using single acc to swift ..09:47
opendevreviewMerged openstack/kolla-ansible stable/xena: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database  https://review.opendev.org/c/openstack/kolla-ansible/+/84293809:50
opendevreviewMerged openstack/kolla-ansible stable/yoga: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database  https://review.opendev.org/c/openstack/kolla-ansible/+/84293709:50
Fl1ntkevko, I'm reading the code right now and it confuse me further ^^ it's supposed to read the conf correctly.09:57
Fl1ntwhen setting everything under [glance_store] the store.py driver still complain not finding appropriate directive.09:57
Fl1ntwhen setting everything under [glance.store.swift.strore] same thing09:58
Fl1ntI'm using two backends for glance, one named: performance:rbd and a second one named cold:swift09:58
Fl1ntrbd is working like charm09:59
Fl1ntswift isn't09:59
Fl1nteven tried to get ride of the swift_store_config_file still same error09:59
Fl1ntbasically, I'm using a single account as the glance container is stored within the admin tenant.10:00
Fl1ntlet me paste you my conf ^^10:00
Fl1ntkevko, here is the glance-api.conf file: https://paste.opendev.org/show/bR28F6ipoxj6DJpkHit7/10:06
kevkoFl1nt: do you have some log ? 10:13
Fl1ntYEP10:19
Fl1ntlet me paste that10:19
Fl1nthttps://paste.opendev.org/show/bGeCJjSB1C5GwD6JYFvu/ here it is kevko 10:22
opendevreviewMerged openstack/kolla-ansible stable/wallaby: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database  https://review.opendev.org/c/openstack/kolla-ansible/+/84293910:25
opendevreviewMerged openstack/kolla-ansible stable/yoga: masakari: support libvirt SASL in instance monitor  https://review.opendev.org/c/openstack/kolla-ansible/+/84274010:42
opendevreviewMerged openstack/kolla-ansible stable/xena: masakari: support libvirt SASL in instance monitor  https://review.opendev.org/c/openstack/kolla-ansible/+/84274110:42
opendevreviewMerged openstack/kolla-ansible stable/wallaby: masakari: support libvirt SASL in instance monitor  https://review.opendev.org/c/openstack/kolla-ansible/+/84274210:42
opendevreviewMerged openstack/kolla-ansible master: Use the new image naming scheme  https://review.opendev.org/c/openstack/kolla-ansible/+/84270910:42
opendevreviewMerged openstack/kolla stable/yoga: Fix local sources of git repositories  https://review.opendev.org/c/openstack/kolla/+/84239110:42
opendevreviewMerged openstack/kolla stable/xena: Fix local sources of git repositories  https://review.opendev.org/c/openstack/kolla/+/84239210:42
opendevreviewMerged openstack/kolla stable/wallaby: Fix local sources of git repositories  https://review.opendev.org/c/openstack/kolla/+/84239310:42
opendevreviewMerged openstack/kolla stable/victoria: Fix local sources of git repositories  https://review.opendev.org/c/openstack/kolla/+/84239410:42
opendevreviewMerged openstack/kolla stable/ussuri: Fix local sources of git repositories  https://review.opendev.org/c/openstack/kolla/+/84239510:42
opendevreviewMerged openstack/kolla-ansible master: [CI] Move queue setting to project level  https://review.opendev.org/c/openstack/kolla-ansible/+/84228010:42
opendevreviewMerged openstack/kolla master: [CI] Move queue setting to project level  https://review.opendev.org/c/openstack/kolla/+/84227910:42
opendevreviewVerification of a change to openstack/kolla-ansible master failed: ovn: add network group to neutron-ovn-metadata-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/84236410:42
opendevreviewKyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84297411:11
opendevreviewKyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84297411:12
Fl1ntback11:23
Fl1ntkevko, tried to tell glance to use performance by default, works as expected, don't get it ^^11:26
guesswhatguys? would be apache aware of server proto if i set kolla_external_fqdn to https://foo.bar  ?11:34
opendevreviewMarcin Juszkiewicz proposed openstack/kolla-ansible master: genpwd: handle lack of password file nicer  https://review.opendev.org/c/openstack/kolla-ansible/+/84312911:34
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults  https://review.opendev.org/c/openstack/kolla-ansible/+/81319311:35
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers  https://review.opendev.org/c/openstack/kolla-ansible/+/84313011:35
kevkoFl1nt: sorry, i was rebasing some kolla stuff ..didn't check chat :/11:35
kevkogoing to check11:35
kevkoFl1nt: could you try to setup swift_store_config_file = /etc/glance/glance-swift.conf11:38
kevkocreate it 11:38
kevkoFl1nt: https://paste.opendev.org/show/bsTvqPOoCB3W7w1kRvqZ/11:40
kevkoand setup for some tenant ..11:40
kevkoand send a log 11:41
hrwhttps://paste.centos.org/view/e02d64c5 - kolla-ansible bootstrap-servers behaves weird...11:55
Fl1ntkevko, sure no problems :D Everyone is having its own job no worries ;) I'll test that11:59
kevkoFl1nt: w8 a minute12:01
Fl1ntyep?12:02
guesswhatFl1nt: seems that failed login, directly to horizon container ( 172.17.0.10:80 ) returns - - - [24/May/2022:12:00:48 +0000] "GET /header/ HTTP/1.1" 200 337 3294 "http://172.17.0.10/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36"12:02
guesswhat200 response, so there is no chance to make this happen12:02
kevkoFl1nt: you've sent this https://paste.opendev.org/show/bR28F6ipoxj6DJpkHit7/12:03
Fl1ntno, what you get here guesswhat is a 200 because a GET on /header redirected you to /auth/login as you're not login.12:03
Fl1ntkevko, yes12:03
guesswhatno, http://172.17.0.10/auth/login/ directly12:04
guesswhattheres not redirect involed i checked it with curl12:05
guesswhatits horizon container12:05
Fl1ntwhere is that coming tho?  "GET /header/ HTTP/1.1"12:05
kevkoFl1nt: use swift_store_user , swift_store_key , swift_store_auth_address and place it to  (wait i am checking)12:05
guesswhatNot sure, this one is correct 24/May/2022:12:00:48 +0000] "POST /auth/login/ HTTP/1.1" 200 3433 45809 "http://172.17.0.10/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.3612:06
Fl1ntyes, this post is 200 because the user is probably logged in12:06
guesswhatgot a new m1 mac and copy command is so confusing... ( linux user )12:06
Fl1ntcopy command?12:07
Fl1ntcmd+c ?12:07
guesswhati am not logged12:07
guesswhatyep, ctrl, cmd, opt is confusing for me :D12:07
Fl1ntthe POST is used when you send the keystone payload (a json formatted payload).12:07
Fl1ntguesswhat, I'm having few hiccups from time to time too as I constantly switch between windows,linux and mac ^^12:08
guesswhatyes, thats what i am saying, using form to login ( POST ) directly to horizon container, tailing apache logs..12:09
kevkoFl1nt: place it to both [glance.store.swift.store] and [glance_store] << did you try this ? 12:09
kevkoFl1nt: hmm, you've said that yes as I see above12:09
kevkoFl1nt: could you do it and send a log ? 12:09
guesswhatexactly.. , i am used to sync files via cloud and often switching linux, mac and window, what is a pain is alttab behaviour and hotkeys12:10
guesswhatand also syncing notes, todos, reminders is a pain too ( obsidian.md looks promising tho )12:10
Fl1ntkevko, Already tested and the log posted in here is the same as without those changes: https://paste.opendev.org/show/bGeCJjSB1C5GwD6JYFvu/12:11
kevkoFl1nt: glance version ? 12:11
Fl1ntlet me check12:11
kevkoFl1nt: i think you are facing this 12:14
kevkoFl1nt: if (not default_swift_reference) or (not self.auth_address):12:14
kevkoFl1nt: add both swift_store_auth_address and also auth_address 12:17
kevkoand try 12:17
Fl1ntthat where I've headed too 12:18
kevkotry both auth_addresses 12:18
Fl1ntMy glance version is 21.1.0-112:18
kevkocoud you try to define both ? 12:18
kevkoswift_store_user , swift_store_key , swift_store_auth_address, auth_address12:19
Fl1ntwill test12:19
Fl1ntTBN: [glance.store.swift.store] is legacy from single backend era of glance back in 201512:19
opendevreviewKyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84297412:20
hrwyoctozepto: that 'new naming' is kind of mess now12:20
hrwyoctozepto: sure, we recommend building own images but sometimes it is good to just grab whatever is on registry.12:21
hrwyoctozepto: 'kolla_base_distro_version' is undefined12:21
yoctozeptohrw: odd, it has to be12:22
hrwquay.io/openstack.kolla/debian-fluentd   master-aarch64   f465702fe39f   11 days ago    581MB12:24
Fl1ntkevko, same shit, no matter what, glance complain about the swift_store_auth_address12:25
hrwand if I copy openstack_tag from ansible/group_vars/all.yml into my globals.yml then undefined happens12:25
kevkowell, it has to be only that conditional i've mentioned above ..12:25
kevkogo inside container and add debug logs :D 12:25
hrwTASK [common : Find custom fluentd format config files] *****************************************************************12:27
hrw[WARNING]: Skipped '/home/marcin/devel/linaro/kolla-ansible/hrw/config/fluentd/format' path due to this access issue:12:27
hrw'/home/marcin/devel/linaro/kolla-ansible/hrw/config/fluentd/format' is not a directory12:27
hrwone day I will add --globals-yml-file arguiment12:27
hrw"manifest for quay.io/openstack.kolla/debian-fluentd:master-debian-bullseye-aarch64 not found: manifest unknown: manifest unknown"12:28
hrwfsck12:28
Fl1ntkevko, I'm already in debug level max from inside the container, hence why I ended up looking for the same code line than you ^^ what really bugs me is that I've put a verification on code and glance pass through it correctly, meaning it correctly load ini conf and directives :'(12:30
Fl1ntit's just like if the conditional test was somehow rotten12:31
hrwreinstalled, retrying12:31
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Control Masakari monitors deploy  https://review.opendev.org/c/openstack/kolla-ansible/+/84313712:31
kevkoFl1nt: well I meant change the code :D 12:31
opendevreviewMaksim Malchuk proposed openstack/kolla-ansible master: Control Masakari monitors deploy  https://review.opendev.org/c/openstack/kolla-ansible/+/84313712:31
Fl1nthum, not that silly yeah ^^12:32
Fl1ntthx12:32
hrwok, deployment goes on12:33
kevkoi am working on some my old patch and found that I should remove keystone admin endpoint 12:38
kevkowhat do you think / 12:38
kevkoit is removed from rocky :/12:38
fricklerkevko: iirc yoctozepto started a patch on that recently. earlier it didn't work because of legacy mess12:42
kevkofrickler: https://github.com/openstack/keystone/blob/master/keystone/server/wsgi.py12:43
kevkokeystone has same code for both public and admin ..12:43
fricklerkevko: keystone yes. client libs ... not all yet12:43
fricklerI've been trying to do the removal in devstack for years12:44
fricklerthe patch I had in mind only tackles the extra port, but that's a good start https://review.opendev.org/c/openstack/kolla-ansible/+/84089812:44
fricklerso you can have admin and public endpoint sharing the same URL, but without the admin endpoint registered in keystone, heat and other will fail. at least until xena, not sure when the fixed keystoneauth lib kicks in, yoga or zed12:47
kevkofrickler: probably i don't understand, some lib or client is going to do something which need admin endpoint, so it reads config ..send request to admin endpoint ..it goes to 35357 and open wsgi-admin file which is same as public :D 12:48
kevkofrickler: https://paste.opendev.org/show/bo9CduGr0AczqMKKVBOI/12:48
guesswhatFl1nt keystone has own locking mechanism https://docs.openstack.org/keystone/latest/admin/configuration.html#security-compliance ( Setting an account lockout threshold )12:51
guesswhatMaybe this would be the best way, without involving proxies ...12:51
Fl1ntbut it would not fix the issue of people abusing your HTTP frontend guesswhat 12:52
guesswhatyes, but thats easy with stick tables in haproxy12:53
guesswhati didnt know about account lock12:53
opendevreviewKyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84297413:22
hrwhttps://pastebin.com/fH9eU4vm - any ideas how to get those ok/fatal messages in human readable form?13:34
jingvarSomeone uses glance with cinder as backend. I have a issue with rootwrap.13:57
Fl1ntnope, sorry ^^14:01
Fl1ntkevko, so, I've mailed the discussion list as I'm out of idea, I'll let other peers review my conf and validate if it's fine. updating the code is a pain as I'm on a container...14:03
yoctozeptojingvar: not supported, vide https://review.opendev.org/c/openstack/kolla-ansible/+/71499914:03
yoctozeptoi.e. you can patch your code locally and it should work but it needs more work for an upstream, general solution14:07
priteauhrw: Have you tried using the yaml stdout_callback?14:12
priteauhttps://www.jeffgeerling.com/blog/2018/use-ansibles-yaml-callback-plugin-better-cli-experience14:12
hrwyep. have that in ~/.ansible.cfg and it feels ignored14:13
opendevreviewKyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84315014:26
opendevreviewKyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84315014:27
SvenKieskethis might be a stupid question, but I couldn't find it in the sources yet: does kolla-ansible do any snapshots or other backups from the elasticsearch deployment, specifically, from the log data?14:34
jingvarThanks. I was surpised that glance with cinder is unsupported, this boundle looks native for me14:34
yoctozeptojingvar: not really, it's not a common choice14:35
opendevreviewPierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict  https://review.opendev.org/c/openstack/kayobe/+/84311714:48
opendevreviewMerged openstack/kolla-ansible master: CI: Fix prometheus-efk scenario with TLS enabled  https://review.opendev.org/c/openstack/kolla-ansible/+/84311914:48
jingvaryoctozepto:  what is a reason for use it?14:57
jingvarnot use14:57
yoctozeptojingvar: better alternatives; image storage is usually different from volume storage14:58
jingvarwhat about  speed boot VMs14:59
yoctozeptojingvar: usually with ceph15:00
jingvarinstaed of copy, convert to raw on controller (cinder), mount, unmount - mount on hypervisor15:00
yoctozeptoraw images on ceph15:00
yoctozeptoboth glance and cinder on ceph15:00
jingvarhmm, glance to ceph, cinder to chep,  and it can create volume from image withot control part?15:02
yoctozeptoyeah15:04
yoctozeptoit's direct15:04
yoctozeptocopy-on-write in ceph15:04
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers  https://review.opendev.org/c/openstack/kolla-ansible/+/84313015:17
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults  https://review.opendev.org/c/openstack/kolla-ansible/+/81319315:17
opendevreviewKyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy  https://review.opendev.org/c/openstack/kolla-ansible/+/84315015:23
jingvaryoctozepto: thanks, my stage on FC15:34
yoctozeptokevko: replied on keystone admin port; I will try my idea today15:42
kevkoyoctozepto: i am trying :) 15:42
yoctozeptokevko: I have only now posted my idea so you are testing something else :D15:42
kevkoyoctozepto: i think we can keep haproxy  35357 and just point it to keystone 500015:42
kevkoyoctozepto: going to check :D 15:42
yoctozeptokevko: we also need to support haproxy-less configs15:43
yoctozeptothat's the trick15:43
kevkohaproxy-less ? 15:45
yoctozeptokevko: you can disable haproxy and not have ti15:51
yoctozeptoit*15:51
yoctozeptohaproxy-less == "without haproxy"15:51
kevkohmmm15:52
kevkowell, my idea was just copy all from /etc/kolla/keystone and create /etc/kolla/keystone-migration-whatewer and start two containers, regular keystone and keystone-migration-whatewer  with old config ..15:53
kevkoeverything done in keystone role ..15:53
kevkothen just provide cleanup playbook15:53
opendevreviewRadosław Piliszek proposed openstack/kolla-ansible master: [WIP] Do not use a different port for Keystone admin endpoint  https://review.opendev.org/c/openstack/kolla-ansible/+/84089815:59
yoctozeptokevko: ^16:02
kevkoyoctozepto: this will not work also 16:02
yoctozeptowhy?16:02
kevkoyoctozepto: because nova in upgrade task is checking "nova-status upgrade check" which is reading config inside container which is in old version (config is running right after that task ) ...16:04
kevkohmmm... maybe it will when i am thinking about it 16:04
yoctozepto:-)16:04
kevkoyoctozepto: yeah, it will pass 16:04
yoctozepto:D16:04
kevkoi've just tested it :D 16:05
yoctozeptokevko: nice, thanks; let's see CI anyways16:07
yoctozeptoand then document it properly16:08
kevkonova-status upgrade check just passed on my env 16:08
yoctozeptoah, yeah, it had to16:08
kevkobtw, i don't like when i am reconfiguring some service - glance,cinder etc ...and i have some local modifications for example for haproxy ... 16:09
kevkokolla-ansible with --tags cinder also reconfiguring haproxy 16:09
kevkoand i don't know if --skip-tags always working :/16:09
kevkoi have to try 16:09
yoctozeptoit will afair16:09
kevkowhen ? 16:09
yoctozeptowith --skip-tags, it should skip haproxy stuff16:10
kevkonope 16:10
yoctozeptothen that's sad16:10
kevkobecause tasks_from: service/tasks/loadbalancer.yml is mporting role haproxy-config 16:10
kevkoand that task has tags: always 16:10
yoctozeptoack, then I misremember16:11
kevkoand i don't think this is right behavious 16:13
kevko*r16:13
kevkobut probably yes if i am checking it 16:32
* yoctozepto off16:36
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers  https://review.opendev.org/c/openstack/kolla-ansible/+/84313016:42
opendevreviewMichal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults  https://review.opendev.org/c/openstack/kolla-ansible/+/81319316:42
opendevreviewVerification of a change to openstack/kolla-ansible master failed: ovn: add network group to neutron-ovn-metadata-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/84236417:31
opendevreviewPierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict  https://review.opendev.org/c/openstack/kayobe/+/84311720:14
guesswhatIs there any chance to change public endpoints URI to https without actually enabling TLS in kolla ?20:33
guesswhatmaybe public_protocol and kolla_external_fqdn20:34
priteauguesswhat: why do you want not to enable TLS in kolla?20:55
guesswhatpriteau: i want but in different reverse proxy21:06
guesswhatkolla does not support lets encrypt, at least dns challenge21:06
guesswhata http-01 is undocumented21:06
opendevreviewMerged openstack/kolla-ansible master: ovn: add network group to neutron-ovn-metadata-agent  https://review.opendev.org/c/openstack/kolla-ansible/+/84236422:41

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!