| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Migrate ssl certificate generation to the PKI role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/830179 | 09:15 |
|---|---|---|
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Tidy IDP setup task files https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/830260 | 09:15 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: WIP: Enable TLS on haproxy VIPs and backends by default https://review.opendev.org/c/openstack/openstack-ansible/+/829937 | 09:56 |
| jrosser | noonedeadpunk: do you have any ideas about how we should build an http->https CI upgrade job? | 10:02 |
| noonedeadpunk | I'd say we likely should add some sed here https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L255 | 10:16 |
| noonedeadpunk | Considering that we don't want to make it default yet | 10:16 |
| noonedeadpunk | to replace some variables. or add some playbook which will do that | 10:17 |
| jrosser | yeah so we already enable internal vip ssl in xena, so currently a master branch upgrade cannot test http->https on the internal vip | 10:58 |
| noonedeadpunk | Well, we can override that behaviour for X in bootstrap-aio | 11:15 |
| noonedeadpunk | but yes, that all kind of messy... | 11:16 |
| jrosser | yeah, so i was thinking we need first to change something in master and backport to X to allow internal vip to be ssl or not | 11:17 |
| jrosser | alternatively, user_variables_<...>.yml are parsed in alphabetic order i think | 11:18 |
| jrosser | so we could drop an extra file that loads last to undo previous settings, but thats also nasty and fragile | 11:19 |
| *** dviroel|out is now known as dviroel | 11:21 | |
| noonedeadpunk | but I thought we enabled internal ssl somewhere in group_vars? | 11:25 |
| noonedeadpunk | So user_variables would have prescedence anyway? | 11:25 |
| jrosser | https://github.com/openstack/openstack-ansible/blob/stable/xena/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L268-L270 | 11:25 |
| noonedeadpunk | mmm | 11:25 |
| jrosser | so we drop that at the start of our upgrade jobs currently | 11:26 |
| noonedeadpunk | then we can just rollback that for X :) | 11:26 |
| jrosser | ok :) | 11:26 |
| jrosser | though it does very thoroughly test that the root CA stuff is working | 11:26 |
| noonedeadpunk | and handle setting these vars during upgrade (but not inside upgrade script I believe?) | 11:26 |
| jrosser | even though the backends are still http | 11:26 |
| noonedeadpunk | rabbitmq will still test root CA? | 11:27 |
| noonedeadpunk | as well as galera? | 11:27 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible/src/branch/master/inventory/group_vars/all/infra.yml#L20 (and L37) | 11:28 |
| noonedeadpunk | well yes, we can ofc add another upgrade job | 11:29 |
| noonedeadpunk | and just make a condition in https://github.com/openstack/openstack-ansible/blob/stable/xena/tests/roles/bootstrap-host/templates/user_variables.aio.yml.j2#L268-L270 based on scenario | 11:29 |
| jrosser | on X we could add 'ssl' scenario | 11:30 |
| noonedeadpunk | that's likely better option, except more tests to run | 11:30 |
| noonedeadpunk | or just check for upgrade? | 11:31 |
| noonedeadpunk | I wonder if that is parsed even and set properly... | 11:31 |
| noonedeadpunk | It should... | 11:31 |
| noonedeadpunk | ah, upgrade is not in scenario, it's action | 11:32 |
| noonedeadpunk | but we can implement bootstrap_host_action alike with https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/defaults/main.yml#L18 | 11:34 |
| noonedeadpunk | so that upgrade will always test SSL enablement, and non-upgrade will jsut deploy SSL from beginning | 11:35 |
| noonedeadpunk | Sounds really good | 11:35 |
| jrosser | oh, yes, nice | 11:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Improve bump patch readability https://review.opendev.org/c/openstack/openstack-ansible/+/830394 | 11:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/xena: Bump SHAs for Xena https://review.opendev.org/c/openstack/openstack-ansible/+/830398 | 12:11 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Improve bump patch readability https://review.opendev.org/c/openstack/openstack-ansible/+/830400 | 12:15 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/wallaby: Bump SHAs for Wallaby https://review.opendev.org/c/openstack/openstack-ansible/+/830406 | 12:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Improve bump patch readability https://review.opendev.org/c/openstack/openstack-ansible/+/830408 | 12:47 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible stable/victoria: Bump SHAs for Victoria https://review.opendev.org/c/openstack/openstack-ansible/+/830416 | 13:12 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: Add variable to bootstrap role to describe the test job action https://review.opendev.org/c/openstack/openstack-ansible/+/830430 | 14:14 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible master: In AIO and CI set the internal vip to https only for 'deploy' actions. https://review.opendev.org/c/openstack/openstack-ansible/+/830431 | 14:14 |
| jrosser | noonedeadpunk: like this? for master and backport to Xena | 14:15 |
| noonedeadpunk | yep, exactly | 14:54 |
| noonedeadpunk | likely needs backport to X though | 14:54 |
| jrosser | JamesGibo: do we have some sort of 'composite' patch were we bring together all the roles which can do backend SSL? | 14:57 |
| jrosser | for example we could add https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/830179 to that as well | 14:57 |
| opendevreview | James Gibson proposed openstack/openstack-ansible-os_glance master: Add support for TLS to Glance https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/821011 | 14:57 |
| JamesGibo | jrosser: no we do not have 'composite' patch, do you have an example of one and i can create? | 15:00 |
| jrosser | i think i mean one were we can 'depends-on' all the work-in-progress things and see if they all work together | 15:00 |
| jrosser | though it feels like we would need keystone specific changes to openstack-ansible too, just like you have for glance here https://review.opendev.org/c/openstack/openstack-ansible/+/821090 | 15:02 |
| JamesGibo | Sure will put a keystone patch together and create a composite' patch with all the patches we have so far | 15:03 |
| jrosser | might need a little re-arrangement of the depends-on things | 15:06 |
| jrosser | but it's basically right, the glance changes to openstack-ansible depends-on the changes to the glance role, which is all correct | 15:07 |
| jrosser | might be that the haproxy depends-on needs to move up to parent patch to openstack-ansible | 15:07 |
| *** dviroel is now known as dviroel|lunch | 15:11 | |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:12 |
| opendevmeet | Meeting started Tue Feb 22 15:12:14 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:12 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:12 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:12 |
| noonedeadpunk | #topic rollcall | 15:12 |
| noonedeadpunk | o/ | 15:12 |
| damiandabrowski[m] | hey! | 15:12 |
| noonedeadpunk | sorry, I moved between timezones, so got my alarm misbehaving) | 15:17 |
| noonedeadpunk | #topic office hours | 15:17 |
| jrosser | o/ hello | 15:18 |
| noonedeadpunk | damiandabrowski[m]: have you had chance to check out comments for tempest patches? | 15:18 |
| noonedeadpunk | as I believe one down the line blocks everything | 15:18 |
| damiandabrowski[m] | no sorry :/ but I remember about it | 15:19 |
| noonedeadpunk | also it's possible to de-couple them when things are not conflicting | 15:19 |
| noonedeadpunk | ok, gotcha | 15:19 |
| damiandabrowski[m] | yeah, i'll try to remove this huge relation chain | 15:19 |
| noonedeadpunk | so we're progressing on internal tls | 15:21 |
| noonedeadpunk | wip looks pretty fair to me atm | 15:21 |
| noonedeadpunk | except the redirect include is never placed on the host | 15:21 |
| NeilHanlon | o/ heyo folks | 15:23 |
| noonedeadpunk | \o/ hey there! | 15:24 |
| noonedeadpunk | how things are doing with Rocky?:) | 15:24 |
| NeilHanlon | Goodly! Some issues with the nodepool but I think those should be resolved now | 15:25 |
| noonedeadpunk | ok, great! | 15:25 |
| jrosser | noonedeadpunk: the redirect is a jinja include into service.j2 | 15:25 |
| jrosser | it's not another seperate template | 15:25 |
| noonedeadpunk | oh! | 15:26 |
| noonedeadpunk | I'm blind( | 15:26 |
| jrosser | :) | 15:26 |
| NeilHanlon | I still need to try another lxc install or two to check the patches jrosser put in last week | 15:26 |
| noonedeadpunk | I read that as haproxy include | 15:26 |
| jrosser | when do we think there will be a rocky node? | 15:26 |
| jrosser | i ws also worried about the epel stuff yesterday | 15:27 |
| jrosser | thats another repo we very specifically manage | 15:27 |
| noonedeadpunk | Just realized it's not possible there :D | 15:27 |
| jrosser | afaik JamesGibo has tested this in AIO | 15:27 |
| noonedeadpunk | well, I haven't (yet) | 15:27 |
| jrosser | unfortunatly i am double booked for meetings now | 15:28 |
| noonedeadpunk | But I'd say it's fair enough atm and we can always fix later | 15:28 |
| jrosser | but it would be great to all be agreed what the plan is for the TLS stuff | 15:28 |
| jrosser | for example "Y release will be mandatory transition to internal TLS" <- discuss | 15:29 |
| jrosser | "We support internal VIP on http or https" <- different discussion | 15:29 |
| noonedeadpunk | I'd say we should discuss that on PTG likely? | 15:30 |
| jrosser | and in the meantime we can carry on TLS'ing the roles and building an upgrade job | 15:30 |
| noonedeadpunk | As I'd say it can be fine to say that VIPs are only TLS now, but moving forcelly to internal TLS - dunno if it make sense for everybody, considering we don't really have process yet to rotate rootCA | 15:31 |
| noonedeadpunk | (or we do?) | 15:32 |
| NeilHanlon | jrosser: i'm hoping by the end of this week. looks like there was a dib release this morning so it should be ok to test the build now. https://review.opendev.org/c/zuul/nodepool/+/830345 | 15:32 |
| NeilHanlon | i'll ask someone in infra-root if they can unpause it | 15:32 |
| noonedeadpunk | btw regarding bug we discussed previous week - I pushed some patches to cover it https://review.opendev.org/q/topic:bug/1960587+status:open | 15:32 |
| noonedeadpunk | damiandabrowski[m]: btw related to the question you asked in the morning as well regarding 127.0.1.1 record ^ | 15:33 |
| damiandabrowski[m] | great! | 15:34 |
| damiandabrowski[m] | btw. do we have any other idea for fixing this? | 15:35 |
| damiandabrowski[m] | https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/829270 | 15:35 |
| damiandabrowski[m] | i'm not sure if we came up with any conclusion | 15:35 |
| noonedeadpunk | So with different operating systems, constraints file should still be the same? | 15:37 |
| noonedeadpunk | It may only happen with different OSA versions or when u-c version was updated and repo_server role wasn't run | 15:38 |
| noonedeadpunk | for last case that won't help | 15:38 |
| noonedeadpunk | and you actually shouldn't run different osa versions in same deployment? | 15:38 |
| noonedeadpunk | so my question here is more - are we sure we understood the reason why this was affecting us? | 15:39 |
| damiandabrowski[m] | in my case: i was using same osa version everywhere | 15:39 |
| noonedeadpunk | as patch sounds atm a bit unrelated to reall issue | 15:39 |
| * damiandabrowski[m] trying to find more info about it, give me 1min pls | 15:40 | |
| noonedeadpunk | as long as _constraints_file_slurp is a registered var for all hosts in the play, we shouldn't care for which host it was gathered | 15:40 |
| noonedeadpunk | in the meanwhile, I've adjusted a bit bump script to improve diff for openstack_services and also bumped shas for the release https://review.opendev.org/q/topic:bump_osa+status:open | 15:41 |
| damiandabrowski[m] | we hit this issue because 22.3.2 doesn't have pinned uWSGI version, that's why uWSGI version was different for bionic and focal in my case | 15:42 |
| noonedeadpunk | actually I will update master bump right after meeting as I also added command to bump collections versions | 15:42 |
| damiandabrowski[m] | i'm not sure if there may be more cases like this | 15:42 |
| noonedeadpunk | um.. and how that patch fixes that? | 15:43 |
| damiandabrowski[m] | if we're certainly sure that constraints should be the same for all supported operating systems, then we can abandon my change | 15:43 |
| noonedeadpunk | constraints are defined here https://opendev.org/openstack/openstack-ansible/src/branch/master/playbooks/defaults/repo_packages/openstack_services.yml#L33-L35 | 15:43 |
| damiandabrowski[m] | by slurping constraints for each host separately | 15:44 |
| noonedeadpunk | and they depend on osa version and openstack stable release... | 15:44 |
| noonedeadpunk | on top of that. they are cached on repo container early during setup | 15:44 |
| noonedeadpunk | https://opendev.org/openstack/openstack-ansible-repo_server/src/branch/master/tasks/repo_install_constraints.yml#L23-L28 | 15:45 |
| fungi | NeilHanlon: we need 830345 to merge so we get new nodepool-builder images, clarkb approved it a few minutes ago | 15:45 |
| noonedeadpunk | So I can kind of imagine that different repo containers had different constraints file becuase of different OS version on them... | 15:45 |
| fungi | once those images end up on dockerhub and get deployed to our servers, then we can unpause the rocky builds | 15:46 |
| noonedeadpunk | but that sounds like slightly different issue... | 15:46 |
| noonedeadpunk | damiandabrowski[m]: but still u-c are OS independant | 15:47 |
| damiandabrowski[m] | but are upper constraints == constraints slurped in my change? | 15:48 |
| NeilHanlon | fungi: gotcha. I see that workflow in the zuul cfg now | 15:49 |
| damiandabrowski[m] | `Slurp up the constraints file for later re-deployment` is delagated to `venv_build_host` so we are certainly sure they will be slurped from the "right" repo host | 15:50 |
| damiandabrowski[m] | but only when we disable `run_once` | 15:51 |
| noonedeadpunk | oh, hm | 15:51 |
| noonedeadpunk | I think you're actually right | 15:52 |
| noonedeadpunk | especially for cross-OS case | 15:53 |
| damiandabrowski[m] | well, at least in my case disabling `run_once` helped :D | 15:55 |
| noonedeadpunk | fungi: as you're here - can I ask you for review of https://review.opendev.org/c/openstack/project-config/+/829278 ? :) | 15:55 |
| noonedeadpunk | yeah, indeed, it won't be required if we had a repo_container that was a desitnation of sync from all build_hosts | 15:56 |
| fungi | you can ask, sure ;) | 15:56 |
| fungi | lgtm | 15:57 |
| noonedeadpunk | and now we have focal container wich is build_host for hostA and bionic container for hostB, which indeed doesn't work with run_once | 15:57 |
| noonedeadpunk | fungi: thanks! As I clean forgot about patch in dicusssion, as it wasn't on review board because of my mistake and missing repo ACL | 15:58 |
| noonedeadpunk | ok, awesome, we had some progress! :) | 15:59 |
| noonedeadpunk | jrosser: https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/829270 worth having another look indeed | 15:59 |
| noonedeadpunk | #endmeeting | 15:59 |
| opendevmeet | Meeting ended Tue Feb 22 15:59:50 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:59 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-22-15.12.html | 15:59 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-22-15.12.txt | 15:59 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-02-22-15.12.log.html | 15:59 |
| jrosser | noonedeadpunk: oh yes i aslo thought we were talking about u-c | 16:02 |
| noonedeadpunk | but it's about already built wheels in fact | 16:12 |
| *** dviroel|lunch is now known as dviroel | 16:15 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Bump SHAs for master https://review.opendev.org/c/openstack/openstack-ansible/+/830273 | 16:22 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: Add config to enable TLS on keystone https://review.opendev.org/c/openstack/openstack-ansible/+/830474 | 16:23 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: WIP: Add support for enabling TLS to keystone backends in OSA https://review.opendev.org/c/openstack/openstack-ansible/+/830474 | 16:28 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: WIP: Add support for enabling TLS to Glance backends in OSA https://review.opendev.org/c/openstack/openstack-ansible/+/821090 | 16:36 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Migrate ssl certificate generation to the PKI role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/830179 | 16:43 |
| opendevreview | James Gibson proposed openstack/openstack-ansible master: WIP: Enable TLS on haproxy VIPs and backends by default https://review.opendev.org/c/openstack/openstack-ansible/+/829937 | 16:43 |
| JamesGibo | jrosser: There is now https://review.opendev.org/c/openstack/openstack-ansible/+/829937 which can be used to test all patches in one, it enables TLS for internal VIPs, TLS to backends and seamless upgrade process. It depends on the haproxy patch for seamless HTTP to HTTPS upgrade and the per service patches to haproxy vars in openstack-ansible and the service roles that allow TLS to be enabled, but do not have it | 16:58 |
| JamesGibo | enabled. | 16:58 |
| jrosser | JamesGibo: excellent, thankyou | 17:00 |
| jrosser | we'll need to merge some patches to make the xena branch start in http mode for upgrade jobs, but those are fairly straightforward | 17:01 |
| JamesGibo | Yeah, i saw your patches starting work on that, thanks. Fingers crossed all test pass, will check in the morning | 17:04 |
| spatel | noonedeadpunk hey! | 17:10 |
| noonedeadpunk | o/ | 17:11 |
| spatel | I have blog out my HPC deployment on Openstack using infiniband fabric - https://satishdotpatel.github.io/HPC-on-openstack/ | 17:11 |
| noonedeadpunk | oh, nice read for todays evening:) | 17:11 |
| spatel | I am in process to buy storage so thinking i should use IPoIB for performance storage.. will see how that goes. | 17:13 |
| noonedeadpunk | that was mess for us I still regret using... | 17:39 |
| noonedeadpunk | know ppl who switched to ethernet mode | 17:40 |
| spatel | hmm | 17:42 |
| spatel | IPoIB is kind of ethernet correct? | 17:43 |
| spatel | Infiniband is physical layer and IP on top of it | 17:43 |
| noonedeadpunk | well, yes, kind of. But any card can support both IB and Ethernet modes | 17:57 |
| noonedeadpunk | IPoIB is eth implementation on top of Infiniband | 17:57 |
| noonedeadpunk | which has bunch of limitations as it's l3 and not l2 | 18:05 |
| spatel | I will do some test and see :) | 18:42 |
| spatel | Does HAProxy 2.x support UDP based load-balancing in persistence way? | 18:43 |
| spatel | I want to send UDP stream traffic to specific member based on IP dst/src | 18:43 |
| NeilHanlon | e.g. to make all udp traffic between a server and a client stay between those members? | 18:54 |
| spatel | I have 30 client sending UDP traffic to 5 servers so i want they maintain UDP stream with dst/src ip based | 19:00 |
| NeilHanlon | haproxy 2.3 introduced some udp proxying features with respect to syslog, but i'm not sure if it can be used for generic udp proxying, or how the balancing algorithm works. IMO load balancing udp introduces state into a protocol designed to be stateless (e.g., what happens if a backend goes down, how would the client and server know that data | 19:08 |
| NeilHanlon | wasn't received) | 19:08 |
| opendevreview | Merged openstack/ansible-role-python_venv_build master: Slurp constraints.txt separately for each host in a batch https://review.opendev.org/c/openstack/ansible-role-python_venv_build/+/829270 | 19:39 |
| *** tbarron is now known as Guest252 | 19:40 | |
| *** dviroel is now known as dviroel|brb | 21:23 | |
| *** dviroel|brb is now known as dviroel | 21:40 | |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_keystone master: Migrate ssl certificate generation to the PKI role https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/830179 | 22:42 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!