Wednesday, 2024-03-06

*** promethe- is now known as prometheanfire01:30
opendevreviewJimmy McCrory proposed openstack/openstack-ansible master: Add check_hostname option to db healthcheck tasks  https://review.opendev.org/c/openstack/openstack-ansible/+/91115002:56
noonedeadpunkjrosser_: yeah, so https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/901185 helped indeed: https://review.opendev.org/c/openstack/openstack-ansible-os_magnum/+/90118509:18
noonedeadpunk* https://review.opendev.org/c/openstack/openstack-ansible/+/91137709:18
noonedeadpunkalso... what a significant difference in runtime between 2023.2 upgrade and 2023.1....09:22
noonedeadpunkor maybe just not lucky...09:22
jrosser_noonedeadpunk: do we have a bug to raise for magnum?09:55
jrosser_for the two images == broken?09:55
noonedeadpunkwell. there should be a meaningful exception raised, as they do verify and want to supply ID in such cases. So it might be intended, in a way09:56
noonedeadpunkbut yes. there is some bug to raise09:56
noonedeadpunkas first this exception is catched somewhere and way less meaningful thing returned back09:56
noonedeadpunkand then - taking in account deactivated images is also wrong...09:57
noonedeadpunkI will report one shortly09:58
jrosser_great thankyou09:58
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Do not change mode of files recursively  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91142010:24
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Do not change mode of files recursively  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91142010:25
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Detect OVN VPNaaS installation  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91142110:27
g3t1nf0jrosser_: my idea and need to run opestack was that I want to offer to my custommers ready k8s clusters that people can hire and test/run in production their applications. 11:49
g3t1nf0is openstack something too heavy for what I need ? should I go the hypervisor way and just start the clusters this way?11:50
jrosser_you can definatly do that with openstack if you put in some work12:07
noonedeadpunkI think openstack offers quite good amount of tools and features to do that actually12:13
noonedeadpunknot sure if all of them are required if k8s will be the only service in portfolio12:14
jrosser_noonedeadpunk: should we ask for a unmaintained core group for OSA?12:17
g3t1nf0I want to go the openstack way because I will be able to do public offering on the openstack as well12:17
noonedeadpunkyes12:17
noonedeadpunkI should propose a patch for that12:17
* noonedeadpunk should do plenty of things12:18
fricklerjrosser_: noonedeadpunk: why not join openstack-unmaintained-core?12:18
jrosser_because i have really nothing to add on things outside osa12:18
noonedeadpunkfrickler: I'm not sure we have enough capacity to care about all unmaintained projects12:18
jrosser_i will bring no value there at all, and have the burden of expectation12:19
frickleryou don't have to do that, but you still don't need to add another group12:19
noonedeadpunkand openstack-unmaintained-core not going to care for osa either12:19
jrosser_this is human factors stuff12:19
jrosser_and feeling comfortable with what is being taken on12:19
jrosser_noonedeadpunk: i can make a patch if you like?12:22
noonedeadpunksure, feel free to12:22
jrosser_ok will look later, there is an ironic one to reference12:23
noonedeadpunk(and kolla one :D)12:24
jrosser_g3t1nf0: you should probably look at this for what will become the best approach for openstack/k8s-as-a-service https://github.com/openstack/openstack-ansible-ops/tree/master/mcapi_vexxhost/doc/source12:47
jrosser_magnum has an existing mechanism that uses openstack heat + lots of bash scripts to deploy k8s12:48
jrosser_thats not the best thing from an operator perspective, but some do manage to make it work12:48
jrosser_the future seems to lie with k8s "cluster API" instead12:49
jrosser_this is all also +/- there being two competing cluster api drivers for magnum, so you should keep an eye on this12:49
g3t1nf0oh no I don't want to do that I'm gonna install OKD they have installer for openstack12:50
g3t1nf0the use api calls to provision the whole k8s12:50
g3t1nf0network storage everything 12:51
jrosser_thats what magnum does12:51
jrosser_and magnum was on your list of required services yesterday?12:51
g3t1nf0yeap 12:51
noonedeadpunkthe bug report for magnum: https://bugs.launchpad.net/magnum/+bug/205632412:52
g3t1nf0keystone, nova, glance, horizon, swift, cinder, neutron, octavia, heat, ceilometer, cloudkitty, trove, magnum, sahara, manila, designate, barbican are the ones that I want 12:52
g3t1nf0from what I've calculated so far I'll have the 3 control planes with 48vcpu and 256G ram and 10 disks then 3 servers with 48vcpus 768G ram and 25 drives for the storage node with 256G ram reservation for ceph and nova, then 3 servers with 48vcpus 768G ram with 25 drives for compute nodes with 128G ram reservation for ceph osds and not sure about the network nodes if I can go just with one or I need another 3 i12:57
g3t1nf0256G ram for nova 12:57
opendevreviewJonathan Rosser proposed openstack/openstack-ansible-tests stable/zed: Update ansible-core collections to match integrated repo  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/91158013:36
opendevreviewMerged openstack/openstack-ansible master: Enable image rotation for Magnum  https://review.opendev.org/c/openstack/openstack-ansible/+/91137713:46
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Determine if upgrade source branch is stable/ or unmaintained/  https://review.opendev.org/c/openstack/openstack-ansible/+/91158313:56
*** tosky_ is now known as tosky13:57
opendevreviewJonathan Rosser proposed openstack/openstack-ansible stable/zed: Determine if upgrade source branch is stable/ or unmaintained/  https://review.opendev.org/c/openstack/openstack-ansible/+/91158414:03
opendevreviewJonathan Rosser proposed openstack/ansible-role-uwsgi stable/zed: Remove undefined bionic linters job  https://review.opendev.org/c/openstack/ansible-role-uwsgi/+/91018814:04
opendevreviewMerged openstack/openstack-ansible-tests stable/zed: Bump ansible-core to 2.12.8  https://review.opendev.org/c/openstack/openstack-ansible-tests/+/91106414:13
opendevreviewJonathan Rosser proposed openstack/openstack-ansible master: Determine if upgrade source branch is stable/ or unmaintained/  https://review.opendev.org/c/openstack/openstack-ansible/+/91158314:15
opendevreviewJonathan Rosser proposed openstack/openstack-ansible stable/zed: Determine if upgrade source branch is stable/ or unmaintained/  https://review.opendev.org/c/openstack/openstack-ansible/+/91158414:15
f0oHi, I'm trying to setup openstack-ansible with OVN and I'm scratching my head regarding network_interface_mappings values. Specifically I have the issue that my compute nodes have bond0 but my network nodes have mlagXY interfaces. How can I specify different mapping per host or grouping? Or ideally how can I specify a pre-created bridge instead of the physical interface?14:22
jrosser_f0o: you can find some general info about how to deal with that situation here https://docs.openstack.org/openstack-ansible/latest/user/prod/provnet_groups.html14:28
f0ojrosser_: I followed that but it doesnt seem to play well with OVN as I keep getting the error: "could not add network device br-vlan to ofproto (File exists)" from ovs-vsctl 14:34
jrosser_that documentation will likley predate OVN14:34
jrosser_but the `group_binds` concept that it is describing will almost certainly be relevant to a config for OVN14:35
f0ounfortunately that seems to be the case with a lot of the docs; a lot of the examples still use linuxbridge too - but OVN is the now default which makes things interesting to setup from the ground up14:35
f0omy biggest issue is that OVN does not like me creating the bridges beforehand (at least that's what I get from the error message)14:36
f0obut I have no clue how else I'm supposed to do this without creating another layer of dummy bridges to get all interface names identical14:36
jrosser_the names do not have to be identical14:36
jrosser_you've got two seperable issues14:36
jrosser_the first is how to describe different provider_networks setups for your different types of host14:37
jrosser_i beleive that `group_binds` is the way to do that14:37
jrosser_seperate, is how to specify the interface in a provider_networks section when using OVN14:37
f0obut those sort-of collapse to the same issue. If I can supply a named bridge (br-vlan in this case) to OVN as the backing physical interface then that will be identical for all hosts since I can offload the creation of that named bridge to the OS14:39
jrosser_but those the ansible will run seperately on those different hosts?14:40
f0oI tried doing that two ways; the first one was to just use openstack_user_config>global_overrides configs like I was used to with linuxbridge - that lead to above error; And then I tried to just user_variables override it and set it as network_interface_mappings - that lead to no erros but no traffic flow either (which I am actually uncertain if it's related or a whole new14:41
f0oerror)14:41
jrosser_did you look at this https://github.com/openstack/openstack-ansible/blob/master/etc/openstack_deploy/openstack_user_config.yml.aio.j2#L178-L19214:41
jrosser_the use of "network_interface" specifically14:41
f0oYep that caused the error I posted14:42
f0ogranted I named it br-vlan and not br-provider but the config (other than vlan ranges) is the same14:42
f0oI can retry with br-provider; maybe br-vlan is some reserved name now?14:43
noonedeadpunkSo.14:43
noonedeadpunkI'd suggest considering `provider_networks` jsut as a setup for containers14:43
noonedeadpunkand then use neutron_provider_networks for defining mappings for neutron/ovn14:44
f0onoonedeadpunk: while that does make sense, that would mean that I can specify two different values for network_interface_mappings depending if it's compute (bond0) or network/gateway (mlagXYZ)14:45
f0oor get it to work with a bridge-interface which I can freely name the same on all devices14:46
jrosser_you can define neutron_provider_networks in user_variables to make it global, and you only get to define it once14:47
jrosser_but if you were to define it in group_vars/<...> you can put it with different values in as many of those as you need14:48
noonedeadpunkneutron_provider_networks make prescedence anyway, so if it's defined - neutron won't care about provider_networks at all14:48
noonedeadpunkAlso, I think you can really leverage `group_binds` 14:48
f0ojrosser_: what's this magic you speak of, did I entirely miss someway to chuck per-group_binds overrides?14:48
noonedeadpunkBut also - there're no containers on net nodes or computes14:48
noonedeadpunkSo you need brdiges only on control plane14:49
jrosser_f0o: what i was trying to tell you earlier about `group_binds` was how you can make parts of provider_networks apply to specific ansible groups14:49
f0omy intial plan was to attempt to get network_mappings: "vlan:br-ext" + network_interface_mappings: "br-ext:br-vlan" to work but I couldnt get any traffic through, mainly I suspected arp_announce being an issue somewhere14:49
f0ojrosser_: apologies for my thickness there14:50
jrosser_so if you want to, you can essentially duplicate parts of provider_networks, but adjust the interface names14:50
jrosser_and have them apply *almost* the same config on different hosts, perhaps interfaces being named different14:50
jrosser_but as noonedeadpunk says you can bypass all of that by using the neutron_provider_networks variable, as an alternative14:51
jrosser_and if you do that, there are choices about where you define it14:51
jrosser_ /etc/openstack_deploy/user_variables.yml is global and maximum priority (wins over everything else)14:52
f0oI dont mind going neutron_provider_networks route, it does feel more intuitive than provider_networks to be honest. So now I just need to find a way to have ansible differ between two neutron_provider_networks blocks depending on group_binds or host-lists14:52
jrosser_but the normal rules of ansible variable precedence apply to anything you also define in /etc/openstack_deploy/group_vars/... and /etc/openstack_deploy/host_vars14:52
f0oguess I'll do more digging around14:52
f0ojrosser_: that's a great pointer thanks for clarifying it!14:53
jrosser_thats ok :)14:53
f0olet me just nuke the rack and have it pxe install some fresh ubuntu -I noticed after re-running ansible a few times the OS gets... tainted14:53
f0oin the meantime I will prepare /etc/openstack_deploy/host_vars for the network nodes14:54
jrosser_so that would need one file per host14:54
jrosser_which could be repetitous14:54
f0othis is really a good ansible refresher - last time I used it was years ago14:55
f0oI dont mind repetition here, it'll all just get chucked into git at the end14:55
jrosser_depending on how you have made your deployment, you can make something like /etc/openstack_deploy/group_vars/neutron_ovn_gateway.yml and put it in there once14:55
jrosser_though you do have your own choices there about what is / isnt a gateway node14:56
noonedeadpunkand you can also add extra groups if needed14:56
noonedeadpunkthis example with AZs might be helpful: https://docs.openstack.org/openstack-ansible/latest/reference/inventory/configure-inventory.html#example-defining-availability-zones14:57
noonedeadpunkand actually, env.d is even slightly optional...14:57
jrosser_yeah extra group definitions are basically free14:57
jrosser_and very useful for keeping the variable definitions sane14:58
jrosser_"compute_with_gpu" or whatever you need14:58
f0oI 100% believe I would've run into this sooner than later14:58
f0othanks for the link!14:59
noonedeadpunkwe have plenty of "hidden" things, unfortunatelly14:59
jrosser_hmm where did the upgrade jobs on Zed go15:01
jrosser_like none run here https://zuul.opendev.org/t/openstack/status#91158415:01
f0onoonedeadpunk:yeah I noticed; I posted a Q on launchpad about that as well. I will collect a bunch of experiences in a document and attempt to update the examples with OVN compatible ones15:02
noonedeadpunkthat would be super welcome frankly speaking, as we falling behind need of documentation update15:03
f0oquick Q; do host_vars take precedence over user_variables ?15:10
noonedeadpunkno15:10
f0oty15:11
noonedeadpunkuser_* are passed as extra_vars to asnible, so always have highest prescedence15:11
f0ook good to know15:11
opendevreviewMerged openstack/openstack-ansible-os_glance master: Add property protection configuration  https://review.opendev.org/c/openstack/openstack-ansible-os_glance/+/90982015:18
jrosser_f0o: there is a useful reference here https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_variables.html#understanding-variable-precedence15:25
jrosser_openstack-ansible user_variables.yml comes under no.22 there15:25
f0ojrosser_:already on it haha - yeah I wasnt sure where the user_variables are plugged in there since I guess they could've technically be added somewhere 10->1815:27
f0obut that would also be too late for my use. So now I hope that inference and defaults work15:28
f0onetwork_interface_mappings: "{{ zz_network_interface_mappings | default('br-ext:bond0') }}" # This is defined in the group/host_vars15:28
jrosser_openstack-ansible wrapper script converts user_*.yml into as many `-e @/etc/openstack_deploy/user_variables_blah.yml` as you have15:29
f0ohave to wait for the reinstall anyway so likely wont get to it today15:29
noonedeadpunkdamiandabrowski: can you kindly review this https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/910384 so we could backport and cut a release for 2023.2?16:06
noonedeadpunkthis what hit us internally as well fwiw16:07
jrosser_i still have no clue why https://review.opendev.org/c/openstack/openstack-ansible/+/911584 runs no upgrade jobs16:09
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-os_horizon master: Do not change mode of files recursively  https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/91142016:10
noonedeadpunkhuh16:12
noonedeadpunkfeel horizon is completely borked in CI16:12
noonedeadpunkafter master sha bump?16:13
jrosser_i was not able to find a useful log when i quickly looked at that earlier16:13
noonedeadpunklike https://review.opendev.org/c/openstack/openstack-ansible-os_horizon/+/910726 got broken overnight16:13
noonedeadpunkhttps://zuul.opendev.org/t/openstack/build/4d250f6b56fa46cca6b4b2af8f95923b/log/logs/openstack/aio1_horizon_container-59233b6b/apache2.service.journal-12-36-03.log.txt#416:13
jrosser_ /o\ doh there it is :)16:14
noonedeadpunkI guess it's it... But then I have some concerns about django version in u-c....16:14
noonedeadpunkBut I feel it being totally realted to the SHA bump16:27
noonedeadpunkwill spawn a sandbox for that...16:28
opendevreviewJonathan Rosser proposed openstack/openstack-ansible stable/zed: Revert "Drop upgrade jobs for Zed"  https://review.opendev.org/c/openstack/openstack-ansible/+/91160216:38
jrossernoonedeadpunk: we should also update the gerrit dashboard to have a unmaintained branches section17:07
noonedeadpunkugh17:08
noonedeadpunkwill do that17:08
jrosseri think i should be able to bring back the Y->Z upgrade jobs17:09
noonedeadpunkI was wondering if that should be done though17:09
jrosserand also fix it so thats more automatic as branch names change in the future17:09
noonedeadpunkas we were dropping them for EM just in case17:09
noonedeadpunkI do agree on the detection patch fully, as such transitions shouldn't block our CI17:10
noonedeadpunkwe can actually return Y for now, after bootstrap of Y will be fixed17:10
noonedeadpunkhttps://bugs.launchpad.net/openstack-ansible/+bug/205541717:11
jrosseryeah well thats my motivation really17:11
jrosserlast years user survey says most people are on Yoga17:13
jrosserso a route off is needed, which in out case starts usually with the most recent minor upgrade on Y17:13
jrossereven before a major version upgrade17:13
jrosseroh man like all these are just totally missing off the dashboard https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%5Eunmaintained/.*+status:open+17:16
noonedeadpunkok, fair enough17:17
opendevreviewJonathan Rosser proposed openstack/openstack-ansible stable/zed: Revert "Drop upgrade jobs for Zed"  https://review.opendev.org/c/openstack/openstack-ansible/+/91160217:21
noonedeadpunkjrosser: about that - maybe it's worth making more/less universal and backport from master? to be ready for Z unmaintained?17:33
noonedeadpunkBut I'm looking at https://opendev.org/openstack/openstack-ansible/src/branch/master/scripts/gate-check-commit.sh#L65-L67 now - and on master there's extra complexity present17:34
jrosserwhich one?17:34
jrosserah so i did two patches17:34
noonedeadpunkI was looking at https://review.opendev.org/c/openstack/openstack-ansible/+/911584/2/scripts/gate-check-commit.sh#5417:34
jrosseri also did this https://review.opendev.org/c/openstack/openstack-ansible/+/91158317:35
jrosseras it was clear it wouldnt backport to Z17:35
noonedeadpunkaha, and now I se efor master17:35
jrosseri gues i should make the change-id be the same, if we eventually backport it to all the branches17:36
noonedeadpunk1 thing I can think about - some period of time when unmaintained will be created, but stable/ not dropped17:36
noonedeadpunkbut this we can probably neglect...17:37
jrosseryeah well it is already a mess with needing to update .gitreview17:37
jrosserbut at the same time * else is broken too17:38
noonedeadpunkand naother thing17:38
jrosserso pretty ugly to unbreak it all17:38
noonedeadpunkyeah, so I guess it's fine17:38
noonedeadpunkjsut good to be aware that during some period we can be testing some crap until clean-up will be executed17:39
jrosserso i think my git branch -r | grep blah blah likley breaks if there are two things returned with $branchname in them17:40
noonedeadpunkah, true as well. add tail -n 1? :)17:42
noonedeadpunkas I guess unmaintained will be lower in the list17:42
jrossergit branch -r | grep origin | sort | grep yoga | tail -n 117:44
jrosserperhaps like that17:44
g3t1nf0so I need just 3 networks vlan 10 20 30 to deploy 17:58
opendevreviewMerged openstack/openstack-ansible-haproxy_server master: Use correct permissions for haproxy log mount  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/91038418:06
noonedeadpunkg3t1nf0: um, depends?18:16
noonedeadpunkI guess with the list of services you want - you need way more18:16
noonedeadpunkoctavia and trove need their own spearate vlans18:16
g3t1nf0https://docs.openstack.org/openstack-ansible/2023.2/user/prod/example.html they are missing as a service here but yeah I get what you mean18:21
noonedeadpunkyeah, and our docs also falling behind a bit some prorgress we made...18:22
noonedeadpunkThey are relevant for OVS/LXB neutron drivers, but inaccurate for OVN just in case18:23
noonedeadpunk(while OVN being default)18:23
g3t1nf0I'm looking at ovn yes18:23
noonedeadpunkwhile generic idea will be same, some configuration will differ a bit18:23
noonedeadpunkbut still - you'd need mgmt, tunnel, storage and external connectivity18:24
noonedeadpunkand then lbaas, dbaas...18:24
noonedeadpunkexternal br-ex can also be a vlan18:25
noonedeadpunkand then you can ignore flat networking18:25
g3t1nf0for certain it wouldn't be flat networking, I'm just checking the hardening documentation with apparmour18:29
g3t1nf0why app armour and not selinux if I may ask18:29
noonedeadpunkworth asking deb/canonical I assume... 18:35
noonedeadpunkAnd well. apparmor profiles are distributed with packages18:36
noonedeadpunkand we here mainly don't deal with any EL for quite a while, so no good skill with selinux18:36
noonedeadpunkBut any contributions are welcome :)18:36
noonedeadpunkSo if you decide to make things working with selinux and redy to make necessary changes - we will be glad to accept them18:37
opendevreviewDmitriy Rabotyagov proposed openstack/openstack-ansible-haproxy_server stable/2023.2: Use correct permissions for haproxy log mount  https://review.opendev.org/c/openstack/openstack-ansible-haproxy_server/+/91160318:38
opendevreviewMerged openstack/ansible-role-systemd_networkd stable/zed: Use OriginalName instead of Name in systemd.link  https://review.opendev.org/c/openstack/ansible-role-systemd_networkd/+/90881519:14
jrosserunmaintained/yoga is now working again https://review.opendev.org/c/openstack/openstack-ansible/+/911621?tab=change-view-tab-header-zuul-results-summary21:19
jrosserand we can bring everything back on zed as well https://review.opendev.org/q/parentproject:openstack/openstack-ansible+branch:%5Estable/zed+status:open+21:20

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!