| *** mat_fechner is now known as matfechner | 07:28 | |
| guesswhat | Guys? Any idea why horizon container ( Apache ) returns 302 ? It is actually working, but I am tryting to catch 401 response ( when someone is trying to use brute force ) code to implement fail2ban protection. Thanks | 07:28 |
|---|---|---|
| hrw | yoctozepto: reserved arm system, will play with it | 07:40 |
| priteau | Hello. I have updated the white board to highlight that Kayobe's master branch is RED | 08:01 |
| priteau | Broken by python-novaclient 18.0.0 which requires Python 3.8 | 08:01 |
| Fl1nt | Hi everyone! | 08:01 |
| hrw | priteau: time to drop cs8 | 08:04 |
| Fl1nt | hrw, woot?? | 08:04 |
| Fl1nt | What could be the reason behind dropping support for a distribution that is officially supported until end of 2024 ?? | 08:05 |
| priteau | Python 3.6 | 08:06 |
| Fl1nt | meh? Oo | 08:06 |
| hrw | Fl1nt: CS8 uses python 3.6, Zed+ requires 3.8 | 08:06 |
| Fl1nt | dnf install python39 works just fine... | 08:06 |
| hrw | and does not give you gazilions of python3* packages | 08:06 |
| priteau | Anyway Zed doesn't target c8s as a runtime, see https://governance.openstack.org/tc/reference/runtimes/zed.html | 08:07 |
| hrw | C7 had Python 3.6 but no one was insane enough to build whole set of python3 packages to be able to run openstack | 08:07 |
| Fl1nt | hrw, what are you talking about, I was running STEIN/TRAIN just fine using PY3 over C7 images... | 08:08 |
| Fl1nt | then migrated to C8 and now using C8S and Debian which is also a pain in its own. | 08:08 |
| hrw | Fl1nt: kolla images for c7 ended in train and used py2 | 08:08 |
| Fl1nt | that's why we do overrides. | 08:09 |
| hrw | users can do whatever they want. I say what project did | 08:09 |
| Fl1nt | but anyway | 08:09 |
| Fl1nt | yep | 08:09 |
| hrw | train (and below) had c7/py2. train (and above to yoga) had c8|cs8/py3. zed (and above) needs cs9 | 08:10 |
| Fl1nt | sure | 08:10 |
| hrw | and we lack external repos for it | 08:10 |
| Fl1nt | what do you mean by that? | 08:11 |
| hrw | https://review.opendev.org/c/openstack/kolla/+/836664 exists already | 08:11 |
| hrw | Fl1nt: no td-agent, elk stack, grafana | 08:11 |
| Fl1nt | yeah, because Openstack as a whole project is going to quickly for companies. | 08:11 |
| hrw | they do not provide rhel9/cs9 packages/repos yet and use sha1 gpg keys which rhel9 does not support | 08:12 |
| hrw | s+and+and/or+g | 08:12 |
| Fl1nt | We in here literally struggle to follow the updates pace and we are around 20 but Managing a multi-geo cloud + adding features over time + updating OS Distribution/Openstack release/CEPH Release every 6 months or once a year is too much. | 08:13 |
| Fl1nt | even if we do have CICD that help a lot. | 08:13 |
| hrw | aware | 08:13 |
| hrw | and now TC decided to use useless naming to mark that you can skip one release | 08:14 |
| hrw | tock/tick is shitty name imho | 08:14 |
| Fl1nt | yeah well, I'm stuck at Victoria to Wallaby upgrade for now so TC can decide whatever, we will be more than one release away from tip release... And I'm pretty sure many teams/company struggle the same way. | 08:15 |
| hrw | would be good to have a way to jump 2 or even 4 releases some way | 08:16 |
| Fl1nt | yeah a way for DB to do what is available with ruby/RAK meanning go from point A to Z just following .rak/.sql file history. | 08:17 |
| Fl1nt | BUT | 08:18 |
| Fl1nt | within OS there is INI config files that prohibit that too | 08:18 |
| hrw | users can do V-W upgrade online. I wonder how many will be fine with V-X V-Z V-A requiring shutdown of whole cloud | 08:18 |
| Fl1nt | We won't ^^ | 08:18 |
| hrw | exactly | 08:18 |
| Fl1nt | shutting down a region isn't an option, there is defense systems running on it, can't do that. | 08:19 |
| Fl1nt | migrating them to another region would take too much time. | 08:19 |
| hrw | deploy new controllers, migrate controllers and then migrate computes? | 08:20 |
| Fl1nt | doing it serially is OK tho | 08:20 |
| Fl1nt | each region is having 3 controllers, 3 aux servers and 8 network nodes. | 08:20 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict https://review.opendev.org/c/openstack/kayobe/+/843117 | 08:21 |
| Fl1nt | anyway, at least I can prepare a new migration to C9S for Zed now ^^ | 08:21 |
| Fl1nt | BTW is anyone having issue with Glance and multi-backends on Victoria+ releases with glance complaining about swift_store_auth_address required when using swift as one of the available backends? | 08:22 |
| hrw | meh. turns out that I got VM instead of baremetal ;( | 08:22 |
| Fl1nt | hrw, our old system was doing just that, but the host to vm manager was internal and so not that flexible.? | 08:23 |
| Fl1nt | -? | 08:24 |
| guesswhat | Guys? Any idea why horizon container ( Apache ) returns 302 ? It is actually working, but I am tryting to catch 401 response ( when someone is trying to use brute force ) code to implement fail2ban protection. Thanks ( sorry for double post, dc.. ) | 08:24 |
| Fl1nt | guesswhat, plain HTTP endpoint doing permanent redirect to TLS. | 08:24 |
| Fl1nt | maybe? | 08:24 |
| guesswhat | Fl1nt: I am not using TLS | 08:25 |
| guesswhat | Infact, I am using custom reverse proxy with SSL offloading | 08:25 |
| Fl1nt | Custom proxy? Nginx? | 08:25 |
| Fl1nt | varnish? | 08:25 |
| Fl1nt | your proxy is maybe throwing 302 when translating frontends to backends requests. | 08:26 |
| guesswhat | HAProxy.. Its HAProxy->HAProxy(kolla)->Apache(kolla) | 08:26 |
| guesswhat | even curl to horizon container returns 302 ... | 08:26 |
| guesswhat | *directly | 08:27 |
| Fl1nt | men... this isn't custom proxy... it's the official kolla way to do it. | 08:27 |
| Fl1nt | ok | 08:27 |
| Fl1nt | which endpoint? | 08:27 |
| guesswhat | . /auth/login | 08:27 |
| Fl1nt | ok so keystone, yes it's normal, look at the horizon wsgi config | 08:27 |
| guesswhat | not sure if is possible to use form data like curl -X POST -F username=admin -F password=foo /auth/login | 08:28 |
| Fl1nt | depends on your authentication method | 08:28 |
| Fl1nt | but I would rather use the keystone endpoint directly using correct API workflow | 08:28 |
| guesswhat | HAProxy is running on PFsense ( thats how I have Horizon exposed to the internet ), also using Crowdsec ( alternative to Fail2ban ) to evaluate "bad" behaviour | 08:31 |
| Fl1nt | in your case, as you probably want to filter out front requests, you'll have two options (that I can think of straight away), one is to catch requests at haproxy level and apply fail2ban from there or at apache level, however, at apache level means implementing rules for each endpoints. | 08:31 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: CI: Fix prometheus-efk scenario with TLS enabled https://review.opendev.org/c/openstack/kolla-ansible/+/843119 | 08:32 |
| opendevreview | Mark Goddard proposed openstack/kolla-ansible master: DNM: test prometheus-efk with TLS enabled https://review.opendev.org/c/openstack/kolla-ansible/+/843120 | 08:32 |
| guesswhat | It can detect bad behaviour automatically, detect bruteforce, but the problem is that horizon returns 302.. ( but its working, so its confusing ) | 08:33 |
| Fl1nt | yes, it's normal, look in here: https://opendev.org/openstack/kolla-ansible/src/branch/master/ansible/roles/keystone/templates/wsgi-keystone.conf.j2 | 08:34 |
| Fl1nt | horizon always redirect you to /auth/login or /<dashboard_section> depending on if you're logged in or not. | 08:36 |
| Fl1nt | whoops, bad link but still, same idea | 08:36 |
| Fl1nt | hrw, are you using multi-backend with glance at any chance? | 08:37 |
| hrw | Fl1nt: I am that weird one who does not have any cloud | 08:38 |
| Fl1nt | arff :D | 08:39 |
| Fl1nt | ok ok, no biggies :D | 08:39 |
| guesswhat | So any idea how to do hardening then? | 08:40 |
| Fl1nt | guesswhat, do it at the endpoints themselves and limit requests on frontend (horizon) so you can't be abused. | 08:41 |
| guesswhat | running fail2ban/crowsec on keystone endpoint and use haproxy ratelimit for horizon, right? | 08:43 |
| Fl1nt | guesswhat, let me sniff my horizon traffic to see what's exactly is the behavior. | 08:43 |
| Fl1nt | fail2ban rules for every endpoints from haproxy logs as your traffic is incoming straight to this VIP. | 08:45 |
| Fl1nt | then apply rules depending on the type of abuse, ratelimit/ban etc. | 08:46 |
| Fl1nt | TBN: Ban isn't legal for some countries ^^ | 08:46 |
| hrw | # lscpu | 08:47 |
| hrw | Architecture: aarch64 | 08:47 |
| hrw | Byte Order: Little Endian | 08:47 |
| hrw | CPU(s): 96 | 08:47 |
| hrw | On-line CPU(s) list: 0-95 | 08:47 |
| hrw | Thread(s) per core: 1 | 08:47 |
| hrw | Core(s) per socket: 16 | 08:48 |
| hrw | Socket(s): 6 | 08:48 |
| hrw | NUMA node(s): 2 | 08:48 |
| hrw | ok. now I can deploy openstack | 08:48 |
| Fl1nt | wait, 6 sockets? HW or SW ? | 08:50 |
| Fl1nt | hrw, what kind of hw is that? | 08:50 |
| hrw | arm. Real config is 2 sockets, 48 cores per socket | 08:52 |
| kevko | hmm, intertisting that you want to ban IP when it's trying to do some brute force attack | 08:52 |
| kevko | what if that IP is NAT and you block hundreds of regular users ? | 08:53 |
| Fl1nt | kevko, that's exactly why it's illegal to strictly BAN for some countries. | 08:53 |
| Fl1nt | hrw, interesting! is that a full SBSA hw ? | 08:54 |
| hrw | Fl1nt: that's old ThunderX1 | 08:55 |
| kevko | in previous company (mail platform, not openstack cloud) we were analyzing several fields from log (for example geo location, number of tries, if they are sending POSTS ..etc ..etc) ..and then we had some DB with "weights" of that users ..and if some user exceeded weight ..we've redirected him to dummy frontend :D | 08:56 |
| guesswhat | Maybe ratelimit and 2FA would be better than banning shared IPs | 08:56 |
| hrw | iirc it was sbsa l3 | 08:56 |
| kevko | (russia, africa, afganistan ..etc ...they were trying to get into email boxes via bruteforce ) | 08:57 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: baremetal: refactor package installation into a separate role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/829586 | 08:57 |
| kevko | because if you ban some IP ..attacker will pay another IP if he wants ..and they want :) | 08:57 |
| guesswhat | i woud not need even use banning mechanims directly for each endpoint then | 08:59 |
| kevko | 2FA is good idea i think | 08:59 |
| Fl1nt | hrw, so cool ^^ | 09:04 |
| Fl1nt | guesswhat, yes, the proper way is to redirect to a HTTP Error page. | 09:04 |
| Fl1nt | or request 2FA | 09:04 |
| Fl1nt | oh and btw, just checked, any request to horizon end up with a 302 as it will necessarily redirect your request to either /auth/login or to a default location that differ from the request you made; so /auth/login once validatedf | 09:05 |
| Fl1nt | redirect to /project/overview | 09:05 |
| Fl1nt | etc | 09:05 |
| guesswhat | 2FA will kill automation, like terraform or openstack cli I believ | 09:05 |
| guesswhat | i like oauth for CLIs like in Azure and AWS cli... | 09:06 |
| guesswhat | *flow | 09:07 |
| guesswhat | but its overkill, i have only few users, but the problem is that horizon is exposed to the internet | 09:07 |
| hrw | make vpn, give them vpn entries | 09:08 |
| hrw | this way those users have access and you have only one entry to guard | 09:09 |
| guesswhat | what about using http-response redirect in haproxy ? | 09:15 |
| guesswhat | Fl1nt ^ | 09:15 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Fix malformed OIDCMemCacheServers https://review.opendev.org/c/openstack/kolla-ansible/+/842940 | 09:17 |
| guesswhat | i believe it would be correct behaviour to return response code, instead of 302 for each endpoint | 09:17 |
| guesswhat | and haproxy is "a central entrypoint" for all endpoinds so i would make a sense to be able to filter traffic there | 09:18 |
| Fl1nt | hrw, this is so much pain for users, adding such technicality requirements is a dead shot for many end users | 09:32 |
| kevko | guesswhat: what is incorrect to return 302 ? | 09:32 |
| Fl1nt | guesswhat, I would personally filter out from HAPROXY yes, altho it isn't incorrect to return 302, you just need to follow the request till the end and adjust. | 09:33 |
| kevko | guesswhat: horizon is always redirecting to login if not logged as every login based application | 09:33 |
| kevko | guesswhat: http://HORIZON_IP/auth/login/?next=/ is returning 200 .. | 09:34 |
| Fl1nt | guys, glance is driving me crazy... When you use swift as store backend, glance disable the store because it's REQUIRED to get swift_store_auth_address set but yet in the meantime the doc and configuration doc state that this directive is deprecated since few release and replaced by auth_address that I set on the glance-swift.conf file. | 09:35 |
| Fl1nt | is that thing even working??? | 09:35 |
| Fl1nt | even if I put a swift_store_auth_address on the config file this isn't working. | 09:36 |
| yoctozepto | priteau: might be you want https://review.opendev.org/c/openstack/kolla-ansible/+/842842 for kayobe | 09:38 |
| yoctozepto | for the time being | 09:38 |
| kevko | Fl1nt: haha, how long you are doing with openstack to know that best documentation is reading the code :D :D | 09:39 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict https://review.opendev.org/c/openstack/kayobe/+/843117 | 09:40 |
| priteau | Thanks yoctozepto, I am testing to see what is covered by Zuul checkouts and what isn't | 09:41 |
| kevko | Fl1nt: do yo have multiple swift store accounts ? | 09:44 |
| opendevreview | Mark Goddard proposed openstack/ansible-collection-kolla master: baremetal: refactor package installation into a separate role https://review.opendev.org/c/openstack/ansible-collection-kolla/+/829586 | 09:45 |
| kevko | Fl1nt: if you set swift_store_user, swift_store_key,swift_store_auth_address is not working ? | 09:46 |
| kevko | Fl1nt: swift_store_config_file shouldn't be set if you are using single acc to swift .. | 09:47 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database https://review.opendev.org/c/openstack/kolla-ansible/+/842938 | 09:50 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database https://review.opendev.org/c/openstack/kolla-ansible/+/842937 | 09:50 |
| Fl1nt | kevko, I'm reading the code right now and it confuse me further ^^ it's supposed to read the conf correctly. | 09:57 |
| Fl1nt | when setting everything under [glance_store] the store.py driver still complain not finding appropriate directive. | 09:57 |
| Fl1nt | when setting everything under [glance.store.swift.strore] same thing | 09:58 |
| Fl1nt | I'm using two backends for glance, one named: performance:rbd and a second one named cold:swift | 09:58 |
| Fl1nt | rbd is working like charm | 09:59 |
| Fl1nt | swift isn't | 09:59 |
| Fl1nt | even tried to get ride of the swift_store_config_file still same error | 09:59 |
| Fl1nt | basically, I'm using a single account as the glance container is stored within the admin tenant. | 10:00 |
| Fl1nt | let me paste you my conf ^^ | 10:00 |
| Fl1nt | kevko, here is the glance-api.conf file: https://paste.opendev.org/show/bR28F6ipoxj6DJpkHit7/ | 10:06 |
| kevko | Fl1nt: do you have some log ? | 10:13 |
| Fl1nt | YEP | 10:19 |
| Fl1nt | let me paste that | 10:19 |
| Fl1nt | https://paste.opendev.org/show/bGeCJjSB1C5GwD6JYFvu/ here it is kevko | 10:22 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: Use 'cloudkitty_influxdb_use_ssl' when creatign InfluxDB database https://review.opendev.org/c/openstack/kolla-ansible/+/842939 | 10:25 |
| opendevreview | Merged openstack/kolla-ansible stable/yoga: masakari: support libvirt SASL in instance monitor https://review.opendev.org/c/openstack/kolla-ansible/+/842740 | 10:42 |
| opendevreview | Merged openstack/kolla-ansible stable/xena: masakari: support libvirt SASL in instance monitor https://review.opendev.org/c/openstack/kolla-ansible/+/842741 | 10:42 |
| opendevreview | Merged openstack/kolla-ansible stable/wallaby: masakari: support libvirt SASL in instance monitor https://review.opendev.org/c/openstack/kolla-ansible/+/842742 | 10:42 |
| opendevreview | Merged openstack/kolla-ansible master: Use the new image naming scheme https://review.opendev.org/c/openstack/kolla-ansible/+/842709 | 10:42 |
| opendevreview | Merged openstack/kolla stable/yoga: Fix local sources of git repositories https://review.opendev.org/c/openstack/kolla/+/842391 | 10:42 |
| opendevreview | Merged openstack/kolla stable/xena: Fix local sources of git repositories https://review.opendev.org/c/openstack/kolla/+/842392 | 10:42 |
| opendevreview | Merged openstack/kolla stable/wallaby: Fix local sources of git repositories https://review.opendev.org/c/openstack/kolla/+/842393 | 10:42 |
| opendevreview | Merged openstack/kolla stable/victoria: Fix local sources of git repositories https://review.opendev.org/c/openstack/kolla/+/842394 | 10:42 |
| opendevreview | Merged openstack/kolla stable/ussuri: Fix local sources of git repositories https://review.opendev.org/c/openstack/kolla/+/842395 | 10:42 |
| opendevreview | Merged openstack/kolla-ansible master: [CI] Move queue setting to project level https://review.opendev.org/c/openstack/kolla-ansible/+/842280 | 10:42 |
| opendevreview | Merged openstack/kolla master: [CI] Move queue setting to project level https://review.opendev.org/c/openstack/kolla/+/842279 | 10:42 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: ovn: add network group to neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/kolla-ansible/+/842364 | 10:42 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/842974 | 11:11 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/842974 | 11:12 |
| Fl1nt | back | 11:23 |
| Fl1nt | kevko, tried to tell glance to use performance by default, works as expected, don't get it ^^ | 11:26 |
| guesswhat | guys? would be apache aware of server proto if i set kolla_external_fqdn to https://foo.bar ? | 11:34 |
| opendevreview | Marcin Juszkiewicz proposed openstack/kolla-ansible master: genpwd: handle lack of password file nicer https://review.opendev.org/c/openstack/kolla-ansible/+/843129 | 11:34 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults https://review.opendev.org/c/openstack/kolla-ansible/+/813193 | 11:35 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers https://review.opendev.org/c/openstack/kolla-ansible/+/843130 | 11:35 |
| kevko | Fl1nt: sorry, i was rebasing some kolla stuff ..didn't check chat :/ | 11:35 |
| kevko | going to check | 11:35 |
| kevko | Fl1nt: could you try to setup swift_store_config_file = /etc/glance/glance-swift.conf | 11:38 |
| kevko | create it | 11:38 |
| kevko | Fl1nt: https://paste.opendev.org/show/bsTvqPOoCB3W7w1kRvqZ/ | 11:40 |
| kevko | and setup for some tenant .. | 11:40 |
| kevko | and send a log | 11:41 |
| hrw | https://paste.centos.org/view/e02d64c5 - kolla-ansible bootstrap-servers behaves weird... | 11:55 |
| Fl1nt | kevko, sure no problems :D Everyone is having its own job no worries ;) I'll test that | 11:59 |
| kevko | Fl1nt: w8 a minute | 12:01 |
| Fl1nt | yep? | 12:02 |
| guesswhat | Fl1nt: seems that failed login, directly to horizon container ( 172.17.0.10:80 ) returns - - - [24/May/2022:12:00:48 +0000] "GET /header/ HTTP/1.1" 200 337 3294 "http://172.17.0.10/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36" | 12:02 |
| guesswhat | 200 response, so there is no chance to make this happen | 12:02 |
| kevko | Fl1nt: you've sent this https://paste.opendev.org/show/bR28F6ipoxj6DJpkHit7/ | 12:03 |
| Fl1nt | no, what you get here guesswhat is a 200 because a GET on /header redirected you to /auth/login as you're not login. | 12:03 |
| Fl1nt | kevko, yes | 12:03 |
| guesswhat | no, http://172.17.0.10/auth/login/ directly | 12:04 |
| guesswhat | theres not redirect involed i checked it with curl | 12:05 |
| guesswhat | its horizon container | 12:05 |
| Fl1nt | where is that coming tho? "GET /header/ HTTP/1.1" | 12:05 |
| kevko | Fl1nt: use swift_store_user , swift_store_key , swift_store_auth_address and place it to (wait i am checking) | 12:05 |
| guesswhat | Not sure, this one is correct 24/May/2022:12:00:48 +0000] "POST /auth/login/ HTTP/1.1" 200 3433 45809 "http://172.17.0.10/auth/login/" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/101.0.4951.64 Safari/537.36 | 12:06 |
| Fl1nt | yes, this post is 200 because the user is probably logged in | 12:06 |
| guesswhat | got a new m1 mac and copy command is so confusing... ( linux user ) | 12:06 |
| Fl1nt | copy command? | 12:07 |
| Fl1nt | cmd+c ? | 12:07 |
| guesswhat | i am not logged | 12:07 |
| guesswhat | yep, ctrl, cmd, opt is confusing for me :D | 12:07 |
| Fl1nt | the POST is used when you send the keystone payload (a json formatted payload). | 12:07 |
| Fl1nt | guesswhat, I'm having few hiccups from time to time too as I constantly switch between windows,linux and mac ^^ | 12:08 |
| guesswhat | yes, thats what i am saying, using form to login ( POST ) directly to horizon container, tailing apache logs.. | 12:09 |
| kevko | Fl1nt: place it to both [glance.store.swift.store] and [glance_store] << did you try this ? | 12:09 |
| kevko | Fl1nt: hmm, you've said that yes as I see above | 12:09 |
| kevko | Fl1nt: could you do it and send a log ? | 12:09 |
| guesswhat | exactly.. , i am used to sync files via cloud and often switching linux, mac and window, what is a pain is alttab behaviour and hotkeys | 12:10 |
| guesswhat | and also syncing notes, todos, reminders is a pain too ( obsidian.md looks promising tho ) | 12:10 |
| Fl1nt | kevko, Already tested and the log posted in here is the same as without those changes: https://paste.opendev.org/show/bGeCJjSB1C5GwD6JYFvu/ | 12:11 |
| kevko | Fl1nt: glance version ? | 12:11 |
| Fl1nt | let me check | 12:11 |
| kevko | Fl1nt: i think you are facing this | 12:14 |
| kevko | Fl1nt: if (not default_swift_reference) or (not self.auth_address): | 12:14 |
| kevko | Fl1nt: add both swift_store_auth_address and also auth_address | 12:17 |
| kevko | and try | 12:17 |
| Fl1nt | that where I've headed too | 12:18 |
| kevko | try both auth_addresses | 12:18 |
| Fl1nt | My glance version is 21.1.0-1 | 12:18 |
| kevko | coud you try to define both ? | 12:18 |
| kevko | swift_store_user , swift_store_key , swift_store_auth_address, auth_address | 12:19 |
| Fl1nt | will test | 12:19 |
| Fl1nt | TBN: [glance.store.swift.store] is legacy from single backend era of glance back in 2015 | 12:19 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/842974 | 12:20 |
| hrw | yoctozepto: that 'new naming' is kind of mess now | 12:20 |
| hrw | yoctozepto: sure, we recommend building own images but sometimes it is good to just grab whatever is on registry. | 12:21 |
| hrw | yoctozepto: 'kolla_base_distro_version' is undefined | 12:21 |
| yoctozepto | hrw: odd, it has to be | 12:22 |
| hrw | quay.io/openstack.kolla/debian-fluentd master-aarch64 f465702fe39f 11 days ago 581MB | 12:24 |
| Fl1nt | kevko, same shit, no matter what, glance complain about the swift_store_auth_address | 12:25 |
| hrw | and if I copy openstack_tag from ansible/group_vars/all.yml into my globals.yml then undefined happens | 12:25 |
| kevko | well, it has to be only that conditional i've mentioned above .. | 12:25 |
| kevko | go inside container and add debug logs :D | 12:25 |
| hrw | TASK [common : Find custom fluentd format config files] ***************************************************************** | 12:27 |
| hrw | [WARNING]: Skipped '/home/marcin/devel/linaro/kolla-ansible/hrw/config/fluentd/format' path due to this access issue: | 12:27 |
| hrw | '/home/marcin/devel/linaro/kolla-ansible/hrw/config/fluentd/format' is not a directory | 12:27 |
| hrw | one day I will add --globals-yml-file arguiment | 12:27 |
| hrw | "manifest for quay.io/openstack.kolla/debian-fluentd:master-debian-bullseye-aarch64 not found: manifest unknown: manifest unknown" | 12:28 |
| hrw | fsck | 12:28 |
| Fl1nt | kevko, I'm already in debug level max from inside the container, hence why I ended up looking for the same code line than you ^^ what really bugs me is that I've put a verification on code and glance pass through it correctly, meaning it correctly load ini conf and directives :'( | 12:30 |
| Fl1nt | it's just like if the conditional test was somehow rotten | 12:31 |
| hrw | reinstalled, retrying | 12:31 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Control Masakari monitors deploy https://review.opendev.org/c/openstack/kolla-ansible/+/843137 | 12:31 |
| kevko | Fl1nt: well I meant change the code :D | 12:31 |
| opendevreview | Maksim Malchuk proposed openstack/kolla-ansible master: Control Masakari monitors deploy https://review.opendev.org/c/openstack/kolla-ansible/+/843137 | 12:31 |
| Fl1nt | hum, not that silly yeah ^^ | 12:32 |
| Fl1nt | thx | 12:32 |
| hrw | ok, deployment goes on | 12:33 |
| kevko | i am working on some my old patch and found that I should remove keystone admin endpoint | 12:38 |
| kevko | what do you think / | 12:38 |
| kevko | it is removed from rocky :/ | 12:38 |
| frickler | kevko: iirc yoctozepto started a patch on that recently. earlier it didn't work because of legacy mess | 12:42 |
| kevko | frickler: https://github.com/openstack/keystone/blob/master/keystone/server/wsgi.py | 12:43 |
| kevko | keystone has same code for both public and admin .. | 12:43 |
| frickler | kevko: keystone yes. client libs ... not all yet | 12:43 |
| frickler | I've been trying to do the removal in devstack for years | 12:44 |
| frickler | the patch I had in mind only tackles the extra port, but that's a good start https://review.opendev.org/c/openstack/kolla-ansible/+/840898 | 12:44 |
| frickler | so you can have admin and public endpoint sharing the same URL, but without the admin endpoint registered in keystone, heat and other will fail. at least until xena, not sure when the fixed keystoneauth lib kicks in, yoga or zed | 12:47 |
| kevko | frickler: probably i don't understand, some lib or client is going to do something which need admin endpoint, so it reads config ..send request to admin endpoint ..it goes to 35357 and open wsgi-admin file which is same as public :D | 12:48 |
| kevko | frickler: https://paste.opendev.org/show/bo9CduGr0AczqMKKVBOI/ | 12:48 |
| guesswhat | Fl1nt keystone has own locking mechanism https://docs.openstack.org/keystone/latest/admin/configuration.html#security-compliance ( Setting an account lockout threshold ) | 12:51 |
| guesswhat | Maybe this would be the best way, without involving proxies ... | 12:51 |
| Fl1nt | but it would not fix the issue of people abusing your HTTP frontend guesswhat | 12:52 |
| guesswhat | yes, but thats easy with stick tables in haproxy | 12:53 |
| guesswhat | i didnt know about account lock | 12:53 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: DNM: Test TLS on openstack exporter behind haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/842974 | 13:22 |
| hrw | https://pastebin.com/fH9eU4vm - any ideas how to get those ok/fatal messages in human readable form? | 13:34 |
| jingvar | Someone uses glance with cinder as backend. I have a issue with rootwrap. | 13:57 |
| Fl1nt | nope, sorry ^^ | 14:01 |
| Fl1nt | kevko, so, I've mailed the discussion list as I'm out of idea, I'll let other peers review my conf and validate if it's fine. updating the code is a pain as I'm on a container... | 14:03 |
| yoctozepto | jingvar: not supported, vide https://review.opendev.org/c/openstack/kolla-ansible/+/714999 | 14:03 |
| yoctozepto | i.e. you can patch your code locally and it should work but it needs more work for an upstream, general solution | 14:07 |
| priteau | hrw: Have you tried using the yaml stdout_callback? | 14:12 |
| priteau | https://www.jeffgeerling.com/blog/2018/use-ansibles-yaml-callback-plugin-better-cli-experience | 14:12 |
| hrw | yep. have that in ~/.ansible.cfg and it feels ignored | 14:13 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/843150 | 14:26 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/843150 | 14:27 |
| SvenKieske | this might be a stupid question, but I couldn't find it in the sources yet: does kolla-ansible do any snapshots or other backups from the elasticsearch deployment, specifically, from the log data? | 14:34 |
| jingvar | Thanks. I was surpised that glance with cinder is unsupported, this boundle looks native for me | 14:34 |
| yoctozepto | jingvar: not really, it's not a common choice | 14:35 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict https://review.opendev.org/c/openstack/kayobe/+/843117 | 14:48 |
| opendevreview | Merged openstack/kolla-ansible master: CI: Fix prometheus-efk scenario with TLS enabled https://review.opendev.org/c/openstack/kolla-ansible/+/843119 | 14:48 |
| jingvar | yoctozepto: what is a reason for use it? | 14:57 |
| jingvar | not use | 14:57 |
| yoctozepto | jingvar: better alternatives; image storage is usually different from volume storage | 14:58 |
| jingvar | what about speed boot VMs | 14:59 |
| yoctozepto | jingvar: usually with ceph | 15:00 |
| jingvar | instaed of copy, convert to raw on controller (cinder), mount, unmount - mount on hypervisor | 15:00 |
| yoctozepto | raw images on ceph | 15:00 |
| yoctozepto | both glance and cinder on ceph | 15:00 |
| jingvar | hmm, glance to ceph, cinder to chep, and it can create volume from image withot control part? | 15:02 |
| yoctozepto | yeah | 15:04 |
| yoctozepto | it's direct | 15:04 |
| yoctozepto | copy-on-write in ceph | 15:04 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers https://review.opendev.org/c/openstack/kolla-ansible/+/843130 | 15:17 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults https://review.opendev.org/c/openstack/kolla-ansible/+/813193 | 15:17 |
| opendevreview | Kyle Dean proposed openstack/kolla-ansible master: talk TLS to openstack exporter via haproxy https://review.opendev.org/c/openstack/kolla-ansible/+/843150 | 15:23 |
| jingvar | yoctozepto: thanks, my stage on FC | 15:34 |
| yoctozepto | kevko: replied on keystone admin port; I will try my idea today | 15:42 |
| kevko | yoctozepto: i am trying :) | 15:42 |
| yoctozepto | kevko: I have only now posted my idea so you are testing something else :D | 15:42 |
| kevko | yoctozepto: i think we can keep haproxy 35357 and just point it to keystone 5000 | 15:42 |
| kevko | yoctozepto: going to check :D | 15:42 |
| yoctozepto | kevko: we also need to support haproxy-less configs | 15:43 |
| yoctozepto | that's the trick | 15:43 |
| kevko | haproxy-less ? | 15:45 |
| yoctozepto | kevko: you can disable haproxy and not have ti | 15:51 |
| yoctozepto | it* | 15:51 |
| yoctozepto | haproxy-less == "without haproxy" | 15:51 |
| kevko | hmmm | 15:52 |
| kevko | well, my idea was just copy all from /etc/kolla/keystone and create /etc/kolla/keystone-migration-whatewer and start two containers, regular keystone and keystone-migration-whatewer with old config .. | 15:53 |
| kevko | everything done in keystone role .. | 15:53 |
| kevko | then just provide cleanup playbook | 15:53 |
| opendevreview | Radosław Piliszek proposed openstack/kolla-ansible master: [WIP] Do not use a different port for Keystone admin endpoint https://review.opendev.org/c/openstack/kolla-ansible/+/840898 | 15:59 |
| yoctozepto | kevko: ^ | 16:02 |
| kevko | yoctozepto: this will not work also | 16:02 |
| yoctozepto | why? | 16:02 |
| kevko | yoctozepto: because nova in upgrade task is checking "nova-status upgrade check" which is reading config inside container which is in old version (config is running right after that task ) ... | 16:04 |
| kevko | hmmm... maybe it will when i am thinking about it | 16:04 |
| yoctozepto | :-) | 16:04 |
| kevko | yoctozepto: yeah, it will pass | 16:04 |
| yoctozepto | :D | 16:04 |
| kevko | i've just tested it :D | 16:05 |
| yoctozepto | kevko: nice, thanks; let's see CI anyways | 16:07 |
| yoctozepto | and then document it properly | 16:08 |
| kevko | nova-status upgrade check just passed on my env | 16:08 |
| yoctozepto | ah, yeah, it had to | 16:08 |
| kevko | btw, i don't like when i am reconfiguring some service - glance,cinder etc ...and i have some local modifications for example for haproxy ... | 16:09 |
| kevko | kolla-ansible with --tags cinder also reconfiguring haproxy | 16:09 |
| kevko | and i don't know if --skip-tags always working :/ | 16:09 |
| kevko | i have to try | 16:09 |
| yoctozepto | it will afair | 16:09 |
| kevko | when ? | 16:09 |
| yoctozepto | with --skip-tags, it should skip haproxy stuff | 16:10 |
| kevko | nope | 16:10 |
| yoctozepto | then that's sad | 16:10 |
| kevko | because tasks_from: service/tasks/loadbalancer.yml is mporting role haproxy-config | 16:10 |
| kevko | and that task has tags: always | 16:10 |
| yoctozepto | ack, then I misremember | 16:11 |
| kevko | and i don't think this is right behavious | 16:13 |
| kevko | *r | 16:13 |
| kevko | but probably yes if i am checking it | 16:32 |
| * yoctozepto off | 16:36 | |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Remove configuration related to api_workers https://review.opendev.org/c/openstack/kolla-ansible/+/843130 | 16:42 |
| opendevreview | Michal Arbet proposed openstack/kolla-ansible master: Add api_workers for each service to defaults https://review.opendev.org/c/openstack/kolla-ansible/+/813193 | 16:42 |
| opendevreview | Verification of a change to openstack/kolla-ansible master failed: ovn: add network group to neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/kolla-ansible/+/842364 | 17:31 |
| opendevreview | Pierre Riteau proposed openstack/kayobe master: [WIP] Use Yoga upper constraints to avoid Python version conflict https://review.opendev.org/c/openstack/kayobe/+/843117 | 20:14 |
| guesswhat | Is there any chance to change public endpoints URI to https without actually enabling TLS in kolla ? | 20:33 |
| guesswhat | maybe public_protocol and kolla_external_fqdn | 20:34 |
| priteau | guesswhat: why do you want not to enable TLS in kolla? | 20:55 |
| guesswhat | priteau: i want but in different reverse proxy | 21:06 |
| guesswhat | kolla does not support lets encrypt, at least dns challenge | 21:06 |
| guesswhat | a http-01 is undocumented | 21:06 |
| opendevreview | Merged openstack/kolla-ansible master: ovn: add network group to neutron-ovn-metadata-agent https://review.opendev.org/c/openstack/kolla-ansible/+/842364 | 22:41 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!