Thursday, 2023-03-02

opendevreviewMerged openstack/openstack-ansible-plugins master: Do not use openstack.osa.linear strategy plugin  https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/87442502:02
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Prepare service roles for separated haproxy config  https://review.opendev.org/c/openstack/openstack-ansible/+/87118903:58
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible stable/zed: Bump OpenStack-Ansible Zed  https://review.opendev.org/c/openstack/openstack-ansible/+/87602804:28
opendevreviewMerged openstack/openstack-ansible stable/zed: Do not run dstat by default  https://review.opendev.org/c/openstack/openstack-ansible/+/87560806:28
jrossermorning09:05
noonedeadpunko/09:12
noonedeadpunkregarding your comment on 871189 - I have no idea about LE part as haven't touched it09:13
jrosserthe comment is really about where the state comes from09:14
noonedeadpunkWith zuul being overloaded it's impossible to understand if any change makes performance impact or not....09:15
jrosserin the previous iteration of this, in a deployment with no horizon - horizon_all would be empty list. the certbot backend setup in the haproxy playbook would never be removed because there were no play targets09:15
noonedeadpunkmaybe we can change this condition then? https://review.opendev.org/c/openstack/openstack-ansible/+/871189/16/playbooks/os-horizon-install.yml#3109:16
jrosserin the new iteration, the haproxy role is always run against haproxy_all in it's own play, so there needs to be logic in that playbook to decide if the `certbot` backend should be present/absent, depending on horizon_all | length09:16
jrosserahha09:17
jrosserso it actually is conditional anyway then09:17
noonedeadpunkyep09:17
noonedeadpunkas we run against haproxy_all and not horizon_all09:17
jrosseryes indeed09:17
noonedeadpunkwe just add one group to another so it's pretty much the same09:18
noonedeadpunkand I'm not sure if we want or not to configure all backends regardless 09:18
noonedeadpunkI'd say not to save some execution time09:19
jrosserso maybe we should think about if it is good to have `haproxy_certbot_service` defined twice09:19
noonedeadpunkAlso I think we might want to try to speedup dynamic_inventory by introducing threads there09:19
noonedeadpunkWell. haproxy-service-config.yml is now a playbook, so we can call it whenever we want and provide it whatever we want09:20
jrosseri think i might prefer to have `haproxy_certbot_service` defined just once in group_vars/haproxy/ and not also in horizon group vars09:20
jrosserthen how an override works for that is super clear09:20
noonedeadpunkAnd have like haproxy_certbot_service_absent to call play one more time to remove it?09:21
noonedeadpunkor we can jsut |selectattr there09:22
jrosserwell there is an `enable` condition here https://review.opendev.org/c/openstack/openstack-ansible/+/871189/17/playbooks/haproxy-install.yml#5709:22
noonedeadpunkaha09:24
noonedeadpunkso basically we should run same thing twice09:24
noonedeadpunkI tend to leave that to you and damiandabrowski to fix to be frank :D09:24
jrosseromg now i dont understand why there is `haproxy_certbot_service` and also `haproxy_letsencrypt_service`09:26
jrosser /o\09:26
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump OpenStack-Ansible master  https://review.opendev.org/c/openstack/openstack-ansible/+/87604309:26
noonedeadpunkugh, now both yoga and xena seems to be broken due to rabbit on centos....09:32
jrosserthe other thing is that now the haproxy play can see vars from both haproxy_all and "{{ service_group }}" the patch can probably be divided in half again09:32
jrosserit's no longer necessary to move the vars around at the same time as make the other changes, as either location is good09:33
damiandabrowski"in the new iteration, the haproxy role is always run against haproxy_all in it's own play, so there needs to be logic in that playbook to decide if the `certbot` backend should be present/absent, depending on horizon_all | length"09:36
damiandabrowskii guess i need to spend some time trying to understand Dmitriy's changes as for now I have no idea what could have changed in terms of letsencrypt09:37
jrosser^ i had not noticed the when: horizon_all | length tbh09:37
damiandabrowskiok, so what problem do we have with LE for now? :D 09:37
jrosseri have no idea really its too hard to understand :(09:38
jrosserfor example why is there `haproxy_certbot_service` and also `haproxy_letsencrypt_service09:38
jrosserreally this is a super good example of why i kept asking for a series of short, easy to understand patches09:38
jrosserchanges need to be written for the reviewer really09:39
jrossereach one being the smallest possible step to achieve something well described in the commit message09:40
damiandabrowskibut we always had haproxy_certbot_service and haproxy_letsencrypt_service, it's not a new thing09:40
jrosser? https://codesearch.opendev.org/?q=haproxy_certbot_service&i=nope&literal=nope&files=&excludeFiles=&repos=09:40
damiandabrowskiah sorry, you may be right, so:09:42
noonedeadpunkwell, it's not always possible I should say... With changes like that where we need to update all playbooks I'm not sure how to split that in smaller chunks...09:42
damiandabrowskihaproxy_letsencrypt_service handles connections to certbot which listens on 888809:43
damiandabrowskiso if horizon is deployed, haproxy can redirect ./well-known requrests to haproxy_letsencrypt_service09:44
damiandabrowskibut if horizon is not deployed, haproxy_certbot_service listens on port 80 and redirects traffic to haproxy_letsencrypt_service09:45
damiandabrowskiafter horizon is deployed, haproxy_certbot_service is removed because it's no longer needed09:46
damiandabrowskimaybe services names are confusing...09:51
jrosseri think that we should rename `haproxy_horizon_service` and decouple the deployment of it completely from horizon because actually it is the handling for all http/https traffic on port 80/443 regardless of if we have horizon/LE/security.txt09:52
jrosserthat should just be one of the default services that haproxy deploys always09:53
jrosserand that the starts to open the door for making compute.example.com / dashboard.example.com etc etc rather than making all services on unique ports09:54
jrosserthis then turns into a simplification, because theres only ever one thing handling port 80/443 for all cases09:55
jrosser^ renaming the service makes a simplification09:55
damiandabrowskibut it's completely different approach, it doesn't go along with either point 1 or point 2 from my blueprint: https://opendev.org/openstack/openstack-ansible-specs/src/commit/30bbdf82c9df389c35c187aa9523e7e31d0c5b03/specs/antelope/separated-haproxy-service-config.rst09:59
damiandabrowskiwell... depending on how it's done10:01
damiandabrowskii can imagine that we may have something like "base" service, listening on 80/443 and forwarding traffic to other frontends based on URL etc.10:02
jrosserisnt that what i'm describing?10:02
jrosseranyway sorry i have meetings now10:02
damiandabrowskiprobably yes, i just needed a moment to process your msg :D 10:04
damiandabrowskisounds interesting, noonedeadpunk what do you think?10:04
noonedeadpunkthat sounds quite reasonable and we were discussing ability to have that kind of endpoints for a while now10:06
noonedeadpunkAnd blocker was partially on haproxy side10:06
noonedeadpunkBut I'm not really sure how this service should look like10:06
noonedeadpunkAs it should be able to understand when horizon/security.txt/certbot is available or needed and define ACLs based on that10:07
damiandabrowskiyeah, but it sounds doable. I guess it will be a mix of URL/path based routing10:14
damiandabrowskiit may help:10:14
damiandabrowskihttps://www.haproxy.com/blog/how-to-map-domain-names-to-backend-server-pools-with-haproxy/10:15
damiandabrowskihttps://www.haproxy.com/blog/path-based-routing-with-haproxy/10:15
damiandabrowskibut yeah, it may be tricky for horizon, because we can't redirect traffic to horizon backend if it doesn't exist yet :|10:18
damiandabrowskibut we can make the same trick as in there and just check in haproxy-install.yml if horizon is running: https://review.opendev.org/c/openstack/openstack-ansible/+/871189/17/playbooks/haproxy-install.yml#5010:36
jrosserdamiandabrowski: regarding the blueprint - it has always been that it's not 100% achievable10:39
damiandabrowskiwhen i mentioned the blueprint, i was thinking that you just want to move horizon service to haproxy_default_services10:41
damiandabrowskiso forget it :D 10:41
admin10hi damiandabrowski ..  i want your help in something i am struggling with ..   in haproxy :) 10:42
admin10and i know you are working day-in and out in it 10:42
admin10https://gist.githubusercontent.com/a1git/45b41d370f04af817cb5d592d52b307b/raw/40a1b024fc6c0406410b52ca88ce6692d89707f9/gistfile1.txt  --- this explains what i am trying to do 10:42
admin1i use a wildcard, and want to be able to do s3.domain.com on the same 443, but redirect it to ceph backend 10:44
admin1to take it even further, if this works, i will try to do id.example.com for keystone,   images.example.com etc images, volume.example.com for cinder etc 10:45
admin1so that restrictive firewalls from companies where only 80/443 is allowed can still access the apis 10:45
jrosseradmin1: you asked this before and i showed you how the letsencrypt ACL works10:47
jrosseryou should be able to use the same approach for other things as well10:47
damiandabrowskiyup, it should work IMO. I posted two links to haproxy blog above, they may help you10:48
jrosserthere is a place here that you can add ACLs for the horizon frontend https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L23910:49
admin1but where do i specify the actual SNI name ? is haproxy_service_name  the sni name ? 10:51
jrosseryou can also add arbitrary config to each frontend https://github.com/openstack/openstack-ansible-haproxy_server/blob/stable/zed/templates/service.j2#L86-L8810:51
jrosseryou are trying to add the config lines "lines to add" from your gist?10:52
admin1yes10:52
jrosserwhat release of OSA is this10:53
admin1the issue i see is the bind .. when 443 binds to  VIP address cloud.domain.com, would those acl work for   s3.domain.com 10:53
admin126.0.1 10:53
admin1the last tag we have .. zed10:53
jrosserdoes cloud.domain.com and s3.domain.com resolve to the same ip?10:54
admin1yes they do 10:54
jrosserso there is no problem with binding10:54
admin1ok10:54
jrosserbinding to a VIP is binding to an IP, not an FQDN10:54
admin1ok .. and  the ssl used is a *.domain.com wildcard10:55
jrosserso.....10:55
jrosserthis is where the horizon haproxy setup is defined, that handles port 80/443 https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L224-L24110:55
jrosserthat is what is generating `frontend horizon-front-1` i your gist - does that make sense?10:56
admin1looking into that, i need to add the 4 lines on haproxy_security_txt_acl as list ? 10:58
admin1i am looking into the wrong line10:59
jrossernow look here https://github.com/openstack/openstack-ansible/blob/master/inventory/group_vars/haproxy/haproxy.yml#L58710:59
jrosserthere is a variable already that you can use to merge new things into `haproxy_horizon_service `, it is called `haproxy_horizon_service_overrides`11:00
jrosserwhat you would then do is go and look in defaults/main.yml for the haproxy_server role, as that should be the documentation for what you can do11:01
jrosserand there we find a ready made example https://github.com/openstack/openstack-ansible-haproxy_server/blob/master/defaults/main.yml#L88-L8911:02
jrosserputting all of those things together would end up with something like this https://paste.opendev.org/show/b84UvW46rYtpYiZjmbo4/11:03
damiandabrowskijrosser:  noonedeadpunk i prepared a static haproxy PoC with a support for letsencrypt, security.txt, custom routing based on URL and fallback to horizon if no ACL is matched.11:46
damiandabrowskihttps://paste.openstack.org/show/bhZzGGKwrfm5AuS23WkG/11:46
damiandabrowskiso I actually like jrosser's idea, but considering that I have 2 more days before a vacation(I'm absent from 6th to 13th March) I'll focus now on recent "virtual groups" feature added by Dmitriy and come back to this PoC after vacation11:47
noonedeadpunkwe have quite nasty bug with systemd units being changed11:56
noonedeadpunkAnd it seems it was close to always like that (or until I've fixed another bug)11:57
noonedeadpunkSo, because we do provide "state: started" for systemd units, and don't listen on "systemd service changed" - services are not restarted if the only change is systemd unit11:58
noonedeadpunkGood/easy example is uwsgi role11:58
noonedeadpunkI think I'd prefer adding listen of `systemd service changed` to handlers over restarting services with systemd role...11:59
damiandabrowskibut we do listen on "systemd service changed", don't we? https://opendev.org/openstack/ansible-role-systemd_service/src/branch/master/handlers/main.yml12:04
damiandabrowski"services are not restarted if the only change is systemd unit" maybe you meant "systemd unit state"?12:05
damiandabrowskias i'm pretty sure that service will be restarted if /etc/systemd/system/*.service content changes12:05
noonedeadpunkdamiandabrowski: nope, it won't12:08
noonedeadpunkI've sumbitted a bug that explains it better https://bugs.launchpad.net/openstack-ansible/+bug/200902912:08
damiandabrowski'services_results.item.state is not defined' ah, i get it now12:10
noonedeadpunkActually... I'm thinking about third option12:16
noonedeadpunkChange condition to `'services_results.item.restart_changed | default(systemd_service_restart_changed) | bool' or 'services_results.item.state is not defined'` instead of AND12:16
noonedeadpunkAs for now I can't think of any bad consequences of this....12:20
noonedeadpunkthis condition was like that since role being established12:22
noonedeadpunkwell. except adding extra condition that state is not stopped if it's defined12:24
jrosserdamiandabrowski: i will work on a patch to make a `base` haproxy frontend for everything port 80/44312:36
damiandabrowskiok, great!12:36
jrosserwe can change that on master first and then your stuff to separate the configs will end up hopefully simpler12:37
damiandabrowskimakes sense12:38
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Restart changed services if state is started  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/87608313:18
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Restart changed services if state is started  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/87608313:23
jrosserdamiandabrowski: i am wondering about `frontend letsencrypt-front-1` - do we actually need that?13:28
jrosserthe LE acl redirects to a backend, and i'm not sure the frontend is actually useful13:29
jrosserperhaps this comes from before we had `haproxy_backend_only: true` available13:30
damiandabrowskiyup, i think we can safely drop it13:44
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Restart changed services if state is started  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/87608313:44
admin1jrosser, i ran the playbook with just this .. https://paste.opendev.org/show/b84UvW46rYtpYiZjmbo4/ . it made no changes  .. grep s3 does not show anything 13:46
admin1my bad 13:47
admin1i ran horizon playbook and not haproxy :D 13:47
noonedeadpunkDamn, 876083 is quite bad option13:50
noonedeadpunkDespite it looks damn easy13:50
noonedeadpunkMaybe we can target it for stable branches only as a fix...13:51
noonedeadpunkor close eyes on double restart13:54
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-systemd_service master: Restart changed services if state is started  https://review.opendev.org/c/openstack/ansible-role-systemd_service/+/87608314:16
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-haproxy_server master: Allow default_backend to be specified  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/87615714:23
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Split haproxy horizon config into 'base' frontend and 'horzion' backend  https://review.opendev.org/c/openstack/openstack-ansible/+/87616014:32
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Split haproxy horizon config into 'base' frontend and 'horzion' backend  https://review.opendev.org/c/openstack/openstack-ansible/+/87616014:33
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Split haproxy horizon config into 'base' frontend and 'horizon' backend  https://review.opendev.org/c/openstack/openstack-ansible/+/87616014:35
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to provide custom handler names  https://review.opendev.org/c/openstack/ansible-role-pki/+/87575714:59
opendevreviewDmitriy Rabotyagov proposed openstack/ansible-role-pki master: Allow to provide custom handler names  https://review.opendev.org/c/openstack/ansible-role-pki/+/87575715:00
noonedeadpunkthis systemd_service is rabbit hole....15:06
noonedeadpunkI regret I've spotted that bug15:09
admin1jrosser, it worked for 26.0.1 .. but not for 25.3.0  .. checking .. 15:09
* noonedeadpunk revises his plan to become a farmer15:09
jrosseradmin1: `haproxy_horizon_service_overrides` was only introduced in Zed15:11
admin1oh 15:11
admin1ok 15:11
jrosseryou will have to do the same thing a different way - thats why i asked you which release :)15:11
admin1it is ok .. i will upgrade this cluster to zed 15:11
admin1is there an easy way to do it pre zed ? 15:13
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Use 2.0.0 release for ansible-collections-openstack  https://review.opendev.org/c/openstack/openstack-ansible/+/87309215:14
jrosseradmin1: you would have to put the whole of https://github.com/openstack/openstack-ansible/blob/stable/yoga/inventory/group_vars/haproxy/haproxy.yml#L197-L213 into user_variables with those extra bits added15:15
jrosserso really not so bad15:15
admin1i think i will find upgrading easier then figuring that out :D15:15
jrosserreally just this i think https://paste.opendev.org/show/bf7o9MxUFSAf3KpRtBUX/15:17
jrosserurgh i think i find some wierd bug in python_venv_build as well16:21
jrosserwhen updating upper-constraints SHA/contents and rebuilding lets say utility venv, the correct versions don't appear until `rm /var/www/repo/os-releases/26.1.0.dev57/ubuntu-22.04-aarch64/requirements/utlity*` first16:23
jrossernoonedeadpunk: ahha look at this https://github.com/openstack/ansible-role-python_venv_build/blob/master/tasks/python_venv_wheel_build.yml#L12617:33
noonedeadpunkum?17:33
jrosserbut what is in `"{{ _venv_build_requirements_prefix }}-source-constraints.txt"` ?17:33
jrosserdoh `--constraint http://172.29.236.101:8181/constraints/upper_constraints_cached.txt`17:33
jrosserso that is never `changed`17:33
jrosserwe symlink something with a constant name to whatever u-c file is downloaded17:34
noonedeadpunkyup17:34
jrosserand i was just completely failing having updated to u-c SHA to build things with the right version17:34
jrosserand that is why17:34
noonedeadpunkaha17:35
jrosserthats really a bug17:35
noonedeadpunkI'm down in the rabbit hole with systemd_service....17:35
jrosseroh no :)17:35
noonedeadpunkWell. It's known things... We were having dynamic-SHA previously there, but decided to have constant thing17:36
jrosserdo you remember why?17:36
noonedeadpunkAs it was - fail unobviously why file is missing vs succeed with wrong venv17:36
jrosserhttps://github.com/openstack/openstack-ansible-repo_server/commit/a5df0d1a9bd4f4a24b578ae0596890f7b345381a17:36
noonedeadpunkAs to update u-c you'd need to run repo playbook17:36
noonedeadpunkwhich is unobvious17:36
noonedeadpunkwe indeed had couple of bugs and reports in IRC that things fail unobviously17:37
jrosseri will have a think about how to make that better17:37
noonedeadpunkAnd there's kind of close to no way to fail with good error message17:37
noonedeadpunkAnd we were out of idea how to make it better17:37
jrosserreally because every service can have it's own u-c version we should arrange it so that each service playbook ensures that the right version is present17:38
jrosserrather than do the wrong thing in the repo server playbook17:38
noonedeadpunkI kind of need help with systemd as it seems that now we're at unstable equilibrium of bugs 17:38
noonedeadpunkAnd fixing current one will lead to another one17:38
jrosserhmm17:39
noonedeadpunkWell, if you define glance_upper_constraints_url - it will be respected17:39
noonedeadpunkso each service habing own u-c kind of works as of today17:41
noonedeadpunkthe thing is that caching u-c on repo_host is kind of out of scope for pyhton_venv_build role which I kind of agree with17:41
noonedeadpunkSo about systemd it's https://bugs.launchpad.net/openstack-ansible/+bug/2009029 17:42
noonedeadpunkI went to dropping handlers from roles and instead notifying systemd_service role so that it take care about service restarts17:43
noonedeadpunkBut then I came to neutron and realised how bad idea is that17:43
noonedeadpunkSo we kind of should not fix systemd_service role as then we will get services restarted twice, and we can't leave service restart to systemd role as then we loose logic and flexability we need somewhere17:44
noonedeadpunkAnd we can't really restart services with service roles, as we don't know if service should be running at all or it should be remain stopped/disabled....17:45
noonedeadpunkAnd we can't filter that when generating variable as then we won't pass it to systemd_service and it won't be actually stopped/masked/disabled. So we need to do filtering right in handers and don't touch systemd_service role17:47
noonedeadpunkWhich sounds like terrible idea but I'm out of them17:47
admin1in a cluster, some hypervisors were removed ( and will not be put back) .. is there a good way to remove their entry from inventory so that next time ansible will not look for them or know about them 18:14
noonedeadpunkadmin1: inventoryy-manage.py does have a flag as of today that drops host from inventory18:18
noonedeadpunk-r or -d  can't really recall exactly. But --help should tell :)18:18
jrossernoonedeadpunk: i am trying to understand the systemd thing a bit18:22
jrosseris it that we need to conditionally notify inside the systemd_service role only sometimes?18:23
noonedeadpunkSo if we notify inside systemd_service AND we change like glance.conf - same service will be restarted twice18:33
noonedeadpunkalso we don't need to restart service when we've passed state: stopped or enabled: false or masked: true, for example18:34
noonedeadpunkcross out enabled here :D18:34
noonedeadpunkbut you got the gist18:34
jrosserits horrible but is `services_results` available outside the `systemd_service` role after it has run?18:40
jrosserjust wild handwaving about being able to suppress a second restart if you've already done it18:40
jrosseror do we need to make `systemd_service` have some more official "return value" in a set_fact that describes the state of what was done, and can be used later18:42
* jrosser has to travel18:44
noonedeadpunkWell, the thing is that systemd_service handler executes AFTER service role handler as service_role is usually triggered first (systemd_service is run almost at very end of the role)18:59
noonedeadpunkWe can listen for `systemd service changed` in role handlers - that's available and working, but yeah, then we kind of need to have same condition in each role19:00
noonedeadpunkAnd flushing handlers in systemd_service is not helpful either, just in case19:02
noonedeadpunkWe have more then enough control to be frank on the behaviour of systemd_service role. So it's not that it's doing smth unexpected.19:04
noonedeadpunkbut yeah... Will head out for today as well. Hopefully will be less frustrated in the morning...19:05
mgariepycya have a nice evening19:06
opendevreviewMatthew Thode proposed openstack/openstack-ansible-os_octavia master: Implement support for octavia-ovn-provider driver  https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/86846220:20
prometheanfirenoonedeadpunk: I'm able to use ^ to create a lb, but traffic has yet to get through (seems like an arp issue for the 'public' side of the LB)20:21
prometheanfireit's VERY dirty/hackish20:22
jrossernoonedeadpunk: https://paste.opendev.org/show/bpl3Nm0dfoWWG0ACiRq4/23:12

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!