| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install ZFS packages for bootstrap-host if needed https://review.opendev.org/c/openstack/openstack-ansible/+/865952 | 08:27 |
|---|---|---|
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Install ZFS packages for bootstrap-host if needed https://review.opendev.org/c/openstack/openstack-ansible/+/865952 | 08:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_container_create master: Replace systemd_service templates with role https://review.opendev.org/c/openstack/openstack-ansible-lxc_container_create/+/861394 | 08:28 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Do not install neutron venv if not needed. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/863546 | 08:40 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Set default plugin type to OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/865961 | 09:12 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Set default plugin type to OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/865961 | 09:19 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Set default plugin type to OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/865961 | 10:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Implement OVN inventory changes and deploy by default https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 10:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Add lxb jobs instead of ovn https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/865973 | 10:47 |
| * noonedeadpunk wonders what can posisbly can go wrong with these ^ | 10:48 | |
| *** dviroel|afk is now known as dviroel | 11:12 | |
| jrosser | noonedeadpunk: how does zookeeper end up in the scenario? i'm sure i'm missing something :) | 12:09 |
| noonedeadpunk | because current repo is also paresed and added to scenario | 12:09 |
| noonedeadpunk | so we take out of r"ansible-role-(.*)" | 12:10 |
| jrosser | we do also run an infra job on every integrated repo patch now | 12:10 |
| jrosser | becasue we broke them before when adjusting healthchecks | 12:10 |
| noonedeadpunk | But yes, zookeeper gets to be installed only when we run jobs against it's repo. Or designate... | 12:11 |
| noonedeadpunk | we do it here fwiw https://opendev.org/openstack/openstack-ansible/src/branch/master/zuul.d/playbooks/pre-gate-scenario.yml#L45-L48 | 12:11 |
| noonedeadpunk | well. we can add zookeeper to integrated as well I guess... | 12:12 |
| jrosser | it just needs to be for infra jobs | 12:12 |
| jrosser | ? | 12:12 |
| noonedeadpunk | So what I wanted to avoid - adding zookeeper everywhere as I'm not sure it must be in envirnment rather then it's worth being present | 12:13 |
| noonedeadpunk | but for infra I think we can add it | 12:13 |
| jrosser | i think that currently https://review.opendev.org/c/openstack/openstack-ansible/+/864750/19/etc/openstack_deploy/openstack_user_config.yml.aio.j2#246 this will only run it for designate & it's own repo jobs | 12:14 |
| *** frenzy_friday is now known as frenzy_friday|rover | 12:15 | |
| jrosser | and then this https://review.opendev.org/c/openstack/openstack-ansible/+/864750/19/etc/openstack_deploy/openstack_user_config.yml.aio.j2#250 only for infra jobs in it's own repo | 12:16 |
| jrosser | i think we should cover it with at least this https://github.com/openstack/openstack-ansible/blob/master/zuul.d/jobs.yaml#L263-L273 | 12:17 |
| noonedeadpunk | ok, will push update now then | 12:21 |
| noonedeadpunk | sounds quite fair | 12:21 |
| noonedeadpunk | or worth doing as follow up? | 12:21 |
| jrosser | i think it has to be a follow up or its circular patches | 12:22 |
| jrosser | also we should work on merging things :) | 12:22 |
| noonedeadpunk | yeah, for sure... | 12:26 |
| noonedeadpunk | not sure if it will be circular though... | 12:27 |
| noonedeadpunk | but yeah, since it's passing it's worth to make a follow-up I guess | 12:27 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Implement OVN inventory changes and deploy by default https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 12:31 |
| noonedeadpunk | damiandabrowski: would be great if you could spend some time on reviews today :) | 12:36 |
| noonedeadpunk | damn, I'm not sure how to add zookeeper to validate only :D | 12:38 |
| noonedeadpunk | I can add it to infra only regardless | 12:39 |
| noonedeadpunk | or well... | 12:39 |
| noonedeadpunk | need some bigger patch | 12:39 |
| jrosser | if we are happy to have it on all infra jobs then it should be ok | 13:12 |
| *** frenzy_friday|rover is now known as frenzy_friday|rover|food | 13:43 | |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_nova master: Fix default of neutron_plugin_type https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/866011 | 13:46 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Implement OVN inventory changes and deploy by default https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 13:49 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Explicitly define neutron_plugin_base for OVS https://review.opendev.org/c/openstack/openstack-ansible/+/866012 | 13:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_neutron master: Set default plugin type to OVN https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/865961 | 13:54 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Implement OVN inventory changes and deploy by default https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 13:55 |
| NeilHanlon | the latest rockylinux-9 image on the nodepool is on 9.1. I'll keep an eye on CI in case, but my testing seems OK | 13:59 |
| * jrosser tries to understand how we broke zfs jobs on rax - this used to work | 14:04 | |
| jrosser | NeilHanlon: i've not tried it yet but would you expect i should get on OK with rocky on arm systems? | 14:05 |
| NeilHanlon | yep, should be fine there | 14:05 |
| noonedeadpunk | jrosser: I already placed a fix | 14:06 |
| noonedeadpunk | https://review.opendev.org/c/openstack/openstack-ansible/+/865952 | 14:06 |
| noonedeadpunk | maybe it was pre-installed in infra images... | 14:07 |
| jrosser | noonedeadpunk: do you know how it broke? i remember writing the code in the first place to account for that | 14:07 |
| jrosser | oh hmm | 14:07 |
| noonedeadpunk | well, we covered in one place but not in another | 14:07 |
| jrosser | right i see - there are two paths for either a loopback or the extra device | 14:11 |
| jamesdenton | noonedeadpunk i'll be testing the ovn ssl patches today and hopefully have it reviewed later or tomorrow. I spent some time testing non-ssl -> ssl upgrade and it did not go as well as i'd hoped | 14:45 |
| NeilHanlon | OOP is probably really really important there, i'm guessing? | 14:54 |
| noonedeadpunk | #startmeeting openstack_ansible_meeting | 15:03 |
| opendevmeet | Meeting started Tue Nov 29 15:03:06 2022 UTC and is due to finish in 60 minutes. The chair is noonedeadpunk. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:03 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:03 |
| opendevmeet | The meeting name has been set to 'openstack_ansible_meeting' | 15:03 |
| noonedeadpunk | #topic rollcall | 15:03 |
| noonedeadpunk | sorry for being late - was on a short walk and realized that it's already time for meeting :) | 15:03 |
| noonedeadpunk | o/ | 15:03 |
| damiandabrowski | hi! | 15:03 |
| jamesdenton | o/ | 15:04 |
| NeilHanlon | o/ | 15:04 |
| noonedeadpunk | #topic office hours | 15:05 |
| noonedeadpunk | I don't think we have new and not addressed bugs | 15:05 |
| jamesdenton | i have 1 but have not submitted yet. not a big one | 15:06 |
| noonedeadpunk | Ok, good. | 15:06 |
| noonedeadpunk | Also there was one unsubmitted from ML but it was said that https://review.opendev.org/c/openstack/openstack-ansible-os_designate/+/865701 worked nicely | 15:06 |
| noonedeadpunk | I haven't checked failures though so dunno if they're related or not | 15:07 |
| jamesdenton | my designate deploy was pre-wallaby so did not run into that. | 15:07 |
| noonedeadpunk | failures seem unrelated at first glance | 15:11 |
| noonedeadpunk | will try to re-check | 15:11 |
| noonedeadpunk | so, we have 3 huge topics | 15:11 |
| noonedeadpunk | 1. osa/zookeeper 2. osa/ovn 3. osa-ironic-tidy | 15:12 |
| noonedeadpunk | regarding osa/zookeeper I think it's ready for review. I will add a follow-up patch to trigger zookeeper deployment for integrated repo as well (for validate job) | 15:12 |
| jrosser | o/ hello | 15:13 |
| jrosser | zookeeper looks ok i just found one tiny typo | 15:13 |
| jamesdenton | noonedeadpunk for later: https://bugs.launchpad.net/openstack-ansible/+bug/1998223 | 15:13 |
| noonedeadpunk | I tried to work on ovn, I have quite limited knowledge overall but I think it should work generaly | 15:15 |
| noonedeadpunk | I posted updates to jamesdenton patch to set default at the end (but it will be defined in neutron role only) and some upgrade path | 15:16 |
| jamesdenton | i installed the patches over a working install and it seemed to land OK, but i was really pushing a non->ssl upgrade and didn't get it. I'm doing a fresh deploy now and will kick the tires | 15:16 |
| jamesdenton | this is a 6-node environment so i'll flesh it out | 15:16 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/ansible-role-zookeeper master: Add SSL support for zookeeper https://review.opendev.org/c/openstack/ansible-role-zookeeper/+/865449 | 15:16 |
| noonedeadpunk | yeah, that is smth worth testing | 15:17 |
| spatel | love people started using OVN, can't wait to see it default on AIO | 15:18 |
| noonedeadpunk | and I still wasn't able to look into ironic - has this in my todo list but simply ENOTIME. Despite jrosser explained very good what the issue is still need to think what can be done not in rush | 15:18 |
| jamesdenton | i will try to revisit the osa/ovn default patches today, too. | 15:18 |
| jrosser | there are many things addressed in ironic patches | 15:19 |
| jrosser | mostly small but the consoles is the bog one | 15:19 |
| jrosser | *big | 15:19 |
| jrosser | though this is blocking a lot https://review.opendev.org/c/openstack/openstack-ansible-os_ironic/+/865566 | 15:19 |
| jamesdenton | is the console code still maintained? i can't recall | 15:19 |
| jrosser | for ironic or osa? | 15:20 |
| jamesdenton | ironic | 15:20 |
| jrosser | ipmi-sol seems to work fine, with also talk of adding some graphical stuff too | 15:20 |
| jamesdenton | cool. i guess i was thinking of something else | 15:20 |
| jamesdenton | shellinabox. | 15:21 |
| jrosser | ah yes | 15:21 |
| noonedeadpunk | Actually for 865566 I think there was some fix for centos8.... | 15:21 |
| noonedeadpunk | But I can't recall what it was | 15:22 |
| noonedeadpunk | oh, wow, it's failed on rally... | 15:23 |
| jrosser | oh rally - i thought it was tempest /o\ | 15:23 |
| jrosser | also ENOTIME here making keeping on top of all this pretty tricky | 15:23 |
| jamesdenton | we should ask santa nicely for more | 15:24 |
| noonedeadpunk | The tricky thing with all that that we kind of need to branch roles asap | 15:24 |
| noonedeadpunk | or well next week latest | 15:24 |
| damiandabrowski | i'll spend some time tomorrow on reviews | 15:25 |
| noonedeadpunk | So we need to review things that already passing CI and decide what we want to land for sure | 15:25 |
| noonedeadpunk | I think that internal TLS is out of scope at this point | 15:25 |
| noonedeadpunk | I really want to land ironic fixes as well | 15:25 |
| noonedeadpunk | the question is likely about ovn and if we feel comfortable to land that big change so late | 15:26 |
| jamesdenton | ovn default or ovn ssl or both? | 15:26 |
| noonedeadpunk | ovn default | 15:26 |
| mgariepy | ovn ssl please | 15:26 |
| noonedeadpunk | or both :D | 15:26 |
| mgariepy | also. | 15:26 |
| mgariepy | so we don't have to swap it live ! | 15:27 |
| noonedeadpunk | though I feel quite fine about migration path for ovn default | 15:27 |
| noonedeadpunk | I won't expect too much issues if ppl use our upgrade script or follow upgrade guide | 15:27 |
| jamesdenton | Well, i think that a "preflight" check for neutron could help mitigate operators forgetting to specify ml2.lxb. Just a check at the beginning of neutron playbook that looks for the var and fails if it isn't there. my biggest worry is that LXB folks would accidentally deploy ovn on top and break everything | 15:28 |
| damiandabrowski | i received a question from my company about OVS/OVN TLS. Do we plan to encrypt only management traffic or everything? | 15:28 |
| noonedeadpunk | The only issue I see is that we don't have proper ovn docs and all diagrams would refference only ovs/lxb | 15:28 |
| jamesdenton | :( docs are.. slow. lemme check where i'm at on that | 15:28 |
| noonedeadpunk | I think it covers only management traffic | 15:29 |
| noonedeadpunk | From what I understood | 15:29 |
| jrosser | it has never been about data plane? | 15:29 |
| damiandabrowski | ack, thanks | 15:29 |
| noonedeadpunk | well, we can land docs a bit later as well | 15:29 |
| jamesdenton | also, i just saw your 'define-neutron-plugin' playbook. sorry | 15:30 |
| spatel | we should put some check where people don't accidentally apply ovn playbook on lxb infra | 15:31 |
| noonedeadpunk | I checked that upgrade playbook both on ovs (on our sandbox) and lxb (on aio) and it looked quite fair | 15:33 |
| noonedeadpunk | I don't think we can/should do more rather then write release note and update upgrade docs to include step for defining variables if they're not yet present | 15:34 |
| spatel | Do we have successful path / dock of converting LXB to OVN? | 15:35 |
| jamesdenton | define... successful | 15:35 |
| noonedeadpunk | jamesdenton has a blog describing it - was shared in ML quite recently as well | 15:35 |
| noonedeadpunk | but yeah :D | 15:35 |
| spatel | Like playbook to convert running production to ovn :) | 15:36 |
| noonedeadpunk | oh, wow, quite ambitious :D | 15:36 |
| spatel | jamesdenton send me that link i would like to give it a try in my deployment env | 15:36 |
| jamesdenton | playbook? naw. but you could make one from the steps | 15:36 |
| jamesdenton | https://www.jimmdenton.com/migrating-lxb-to-ovn/ | 15:36 |
| spatel | +1 | 15:37 |
| spatel | jamesdenton no kidding :) - https://www.jimmdenton.com/assets/images/2022-08-31-migrating-lxb-to-ovn/walk-away.gif | 15:37 |
| noonedeadpunk | I can only imagine what playbook that would be | 15:37 |
| jamesdenton | the gifs are the best part | 15:37 |
| damiandabrowski | there was also a talk about it in Berlin: https://www.youtube.com/watch?v=O68Fzry50ic | 15:38 |
| jamesdenton | not a playbook for faint of heart | 15:38 |
| noonedeadpunk | it was from ovs though iirc | 15:38 |
| noonedeadpunk | migration from ovs is simple (comparing) | 15:38 |
| jamesdenton | yes, i think there may even be scripts for that | 15:38 |
| spatel | ovs to ovn is easy but lxb to ovn tricky | 15:38 |
| damiandabrowski | ahh sorry, i missed that we're talking about LXB | 15:39 |
| jamesdenton | spatel has volunteered to write the playbook | 15:39 |
| noonedeadpunk | Now it's in the meeting logs ;) | 15:39 |
| spatel | I wish.. trust me.. | 15:39 |
| damiandabrowski | \o/ | 15:39 |
| spatel | I am underwater last few month.. building new datacenter so migrating all my openstack cloud to new DC | 15:40 |
| jamesdenton | busy is good | 15:40 |
| spatel | not fun when you move your DC | 15:40 |
| noonedeadpunk | oh ,well. we have one more thing that is named sahara | 15:41 |
| jamesdenton | oh? | 15:41 |
| noonedeadpunk | so sahara is simply broken on Zed as upstream service | 15:41 |
| noonedeadpunk | And eventually it's not passing our tempest either (mostly becuase of jsonschema version) | 15:41 |
| noonedeadpunk | So kind of 2 things we can do - disable tempest tests for sahara or use Yoga u-c for it | 15:42 |
| noonedeadpunk | I even proposed some patch to fix it https://review.opendev.org/c/openstack/sahara/+/864728 but meh... | 15:43 |
| noonedeadpunk | it seems a bit more tricky then expected. | 15:43 |
| noonedeadpunk | (or better say complex) | 15:44 |
| jamesdenton | Hmm, well maybe just a release note to say.. don't upgrade to Zed until Sahara is fixed and move on? | 15:44 |
| jrosser | i wonder if anyone uses it | 15:45 |
| noonedeadpunk | yeah. but we won't be able to merge any patches to our repos | 15:45 |
| noonedeadpunk | or well, to os_sahara | 15:45 |
| noonedeadpunk | jrosser: that's good quetsion, but sahara has osa integration test in their pipeline as well | 15:46 |
| noonedeadpunk | so it's quite a rabbit hole tbh | 15:46 |
| damiandabrowski | (i have one thing to discuss regarding internal tls, so please let me know if we're done discussing other things) | 15:46 |
| jrosser | i would say tc maybe declare project dead if a working release is not exsting for Zed? | 15:46 |
| noonedeadpunk | I think I will go on and propose to comment out https://opendev.org/openstack/openstack-ansible/src/branch/master/tests/roles/bootstrap-host/templates/user_variables_sahara.yml.j2#L13-L14 | 15:47 |
| noonedeadpunk | jrosser: yup, that's smth we're on tbh | 15:47 |
| jrosser | good idea | 15:47 |
| noonedeadpunk | but it's not that easy either | 15:47 |
| noonedeadpunk | (and sahara not the only project) | 15:47 |
| noonedeadpunk | I think one of main issues that there was a PTL volunteer for sahara but we don't see much activity but want to give some chance... | 15:49 |
| noonedeadpunk | so ugh | 15:49 |
| noonedeadpunk | damiandabrowski: go on I guess :) | 15:49 |
| damiandabrowski | thanks! | 15:49 |
| damiandabrowski | As Dmitriy suggested, we are not aiming to implement internal TLS in Zed due to the lack of time. We'll implement it in Antelope instead. (but I'm not stopping working on it) | 15:49 |
| damiandabrowski | It gives us plenty of time so I came up with a slightly different idea of implementing it. | 15:50 |
| damiandabrowski | As you may know, the biggest challange is to provide a way for a smooth transition from unencrypted to encrypted traffic. | 15:50 |
| damiandabrowski | It's tricky because we configure all haproxy endpoints at once during haproxy-install.yml. | 15:50 |
| damiandabrowski | At the beginning we planned to implement a temporary feature to handle both HTTP & HTTPS traffic by haproxy. | 15:50 |
| damiandabrowski | Unfortunately it's quite complex and do not solve all our current issues(for ex. haproxy uses variables from different roles which may be not accessible in haproxy hostgroup). | 15:50 |
| damiandabrowski | So I was thinking about avoiding haproxy endpoint configuration directly when running haproxy-install.yml playbook and move it to the service roles. | 15:50 |
| damiandabrowski | In this case, for ex. os_glance would contain import_role pointing to the subset of tasks from haproxy-install.yml to configure endpoints for glance(the same as we do for pki role). | 15:50 |
| damiandabrowski | It may give us 2 main benefits: | 15:50 |
| damiandabrowski | What do you think? do you see any blockers? For sure we'll need to work on playbooks/common-tasks/haproxy-endpoint-manage.yml to avoid managing non-existent haproxy endpoint, but except this I don't see any real issues. | 15:50 |
| damiandabrowski | It would look pretty similar to kolla-ansible, except the fact I don't think we need a separate role for this. https://github.com/openstack/kolla-ansible/tree/master/ansible/roles/haproxy-config | 15:50 |
| damiandabrowski | 2. No more problems with trying to access unavailable variables when running haproxy-install.yml | 15:50 |
| damiandabrowski | 1. No need to worry about http->https transition for backends. | 15:50 |
| jrosser | wherever we have variables that must be accessed across roles those need to be in group_vars | 15:52 |
| jrosser | haproxy or not haproxy | 15:52 |
| noonedeadpunk | The one problem I see is that os_glance role is not running against haproxy_all hosts. While we do delegating for pki/python_venv_build we delegate only to 1 host | 15:52 |
| noonedeadpunk | And I don't think ansible can delegate to groups (but I never tried) | 15:52 |
| damiandabrowski | https://paste.openstack.org/show/bvi2iSjNMbJrCyM51bw1/ | 15:53 |
| damiandabrowski | you can use loop, i tested it yesterday | 15:53 |
| jrosser | :( we should use proper var scopes when we need | 15:54 |
| damiandabrowski | but it actually solves our issue, isn't it? | 15:54 |
| damiandabrowski | so according to my test, if os_glance role includes haproxy role - it has glance vars available | 15:54 |
| noonedeadpunk | another hassle I see is certs/let's encrypt. But maybe we can issue them during haproxy-install playbook and just somehow utilize later | 15:54 |
| noonedeadpunk | yeah, you can delegate_facts I guess or smth | 15:55 |
| noonedeadpunk | I kind of like the idea as it would solve some of our issues. I'm not sure it won't bring another ones though :D | 15:55 |
| jrosser | looping over delegate_to just feels like re-inveting the host loop of a playbook too | 15:55 |
| noonedeadpunk | But we wil lnever know until we try | 15:56 |
| noonedeadpunk | Well, even thinking about moving haproxy service configuration to the service playbook sounds like improvement, isn't it? | 15:56 |
| damiandabrowski | so to summarize: this idea may require some tweaks but you don't have anything against that and I can prepare some PoC? | 15:56 |
| damiandabrowski | for me it definitely sounds like an improvement :D | 15:57 |
| noonedeadpunk | My opinion is that I'm not sure if we will gain any profit and will be able not to make role even more complex, but it might be worth trying to see if it's ok | 15:58 |
| jrosser | there will be also some corner cases | 15:58 |
| noonedeadpunk | As it might result in quite a simplification as well | 15:58 |
| jrosser | LE is somehow tied to horizon | 15:58 |
| jrosser | and consoles are not tied to one service | 15:58 |
| jrosser | so there are some plusses, but it will also have tricky parts which are similar to the things that are tricky today | 15:59 |
| noonedeadpunk | I guess for LE we did some trick for keystone as well? Not sure though.... | 15:59 |
| jrosser | port 80 and 443 are there for the benefit of horizon | 15:59 |
| noonedeadpunk | ah, yes, true | 16:00 |
| jrosser | but it is not making much sense to deploy LE as part of horizon | 16:00 |
| jrosser | thats like step zero before everything else | 16:00 |
| damiandabrowski | okok thanks for your input! I made some notes and will look into that soon | 16:01 |
| noonedeadpunk | #endmeeting | 16:02 |
| opendevmeet | Meeting ended Tue Nov 29 16:02:55 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:02 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-29-15.03.html | 16:02 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-29-15.03.txt | 16:02 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/openstack_ansible_meeting/2022/openstack_ansible_meeting.2022-11-29-15.03.log.html | 16:02 |
| spatel | I would like to upgrade my production openstack running on wallaby to Xena or Yoga. Is there anything i should be worry or care about before i start upgrade? | 16:03 |
| spatel | I will read release notes before make any move but just curious if anyone notice anything outside doc | 16:04 |
| spatel | Any problem doing directly W->Y upgrade? | 16:05 |
| damiandabrowski | i've performed V->X upgrades in 4 regions this autumn, no big issues at all | 16:06 |
| noonedeadpunk | I wrote Marc some hassle with nova yestarday when jumping through releases. I haven't tried W->Y but we had that during V->X. So it can still apply to your path as well | 16:06 |
| noonedeadpunk | so you can check out logs :) | 16:06 |
| damiandabrowski | there was one minor thing but it's already merged: https://review.opendev.org/c/openstack/openstack-ansible-os_keystone/+/857749 | 16:06 |
| damiandabrowski | ah, and nova upgrade was a bit tricky | 16:07 |
| damiandabrowski | i had to upgrade all nova components at once and then execute `mysql -e "update nova.services set version = 57 where deleted = 0;"` | 16:07 |
| noonedeadpunk | Btw, any thoughts folks on https://review.opendev.org/c/openstack/openstack-ansible/+/863423 ? | 16:08 |
| damiandabrowski | (nova-conductor was refusing to start before applying this sql command) | 16:08 |
| noonedeadpunk | As I faced with issue of being hard to add compute node (or any other host) to inventory without running dynamic_inventory for real on production deploy host which is meh.... | 16:09 |
| damiandabrowski | at first glance it looks ok | 16:10 |
| jrosser | noonedeadpunk: what is the difference beweeen "running for real"? | 16:10 |
| cloudnull | đź‘‹ chai | 16:11 |
| cloudnull | **ohai | 16:11 |
| jamesdenton | mr. cloudnull | 16:11 |
| *** dviroel is now known as dviroel|lunch | 16:11 | |
| cloudnull | what's good ? | 16:12 |
| noonedeadpunk | jrosser: so I can place tox.ini in git repo with openstack_deploy folder | 16:12 |
| noonedeadpunk | and run tox locally on any machine | 16:12 |
| noonedeadpunk | without need to fork openstack-ansible repo, bootstrap it, etc | 16:12 |
| noonedeadpunk | Or maybe I'm missing some easy way for that? | 16:13 |
| jrosser | i think i was maybe not seeing the difference with `ansible-inventory -i ./inventory/dynamic_inventory.py --list` | 16:14 |
| jrosser | oh right yes but without bootstrap - i see | 16:15 |
| noonedeadpunk | yup | 16:15 |
| cloudnull | any chance I could get eyes on https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/855996 ? | 16:15 |
| cloudnull | Nothing super important, just a patch I'm running locally. | 16:15 |
| noonedeadpunk | and without direct clone of osa repo even (well - tox will do that kind of, but you don't need to clean it up) | 16:15 |
| noonedeadpunk | cloudnull: can you please tell more about reason behind the patch? | 16:16 |
| noonedeadpunk | (also from 60 to 640 is not doubling :D) | 16:17 |
| cloudnull | originally I doubled it, I'll update the message. As for the story, I replied in the comments. its just to make the error messages go away | 16:18 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Allow to set dnsmasq configuration options https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/864872 | 16:18 |
| noonedeadpunk | ah | 16:19 |
| noonedeadpunk | sorry I haven't checked comments | 16:19 |
| *** frenzy_friday|rover|food is now known as frenzy_friday|rover | 16:21 | |
| noonedeadpunk | tbh I'd rather made these setting configurable... Ultimately, add add some simple way to define extra options to config would be also great... | 16:21 |
| noonedeadpunk | I will also try to rebase your patch cloudnull for disabling net configuration for lxc | 16:22 |
| opendevreview | Kevin Carter proposed openstack/openstack-ansible-rabbitmq_server master: Update the heartbeat and handshake timeout https://review.opendev.org/c/openstack/openstack-ansible-rabbitmq_server/+/855996 | 16:22 |
| cloudnull | noonedeadpunk I think you had a better change for lxc networking? | 16:22 |
| noonedeadpunk | Yeah, it's merged but I didn't add condition to the patch | 16:23 |
| spatel | damiandabrowski good to know, then i will go directly to Yoga (hope its stable and production ready) | 16:23 |
| cloudnull | ah. | 16:23 |
| spatel | damiandabrowski what is this and for what? - i had to upgrade all nova components at once and then execute `mysql -e "update nova.services set version = 57 where deleted = 0;"` | 16:25 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-lxc_hosts master: Add option to disable lxc interface management https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/861676 | 16:27 |
| damiandabrowski | there was(or maybe still is) a bug in nova-conductor which prevented it to start even all nova services are upgraded | 16:27 |
| damiandabrowski | unfortunately i don't have any details, maybe noonedeadpunk recalls something | 16:27 |
| opendevreview | Jonathan Rosser proposed openstack/openstack-ansible-os_neutron master: Do not install neutron venv if not needed. https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/863546 | 16:28 |
| noonedeadpunk | spatel: https://meetings.opendev.org/irclogs/%23openstack-ansible/%23openstack-ansible.2022-11-28.log.html#t2022-11-28T14:43:45 | 16:28 |
| spatel | If its that critical then shouldn't be part of upgrade doc? | 16:29 |
| spatel | reading... | 16:29 |
| noonedeadpunk | soooo much typos in there.... | 16:29 |
| mgariepy | typos where ?:P | 16:30 |
| mgariepy | who cares.. | 16:30 |
| mgariepy | i surely dont hahah | 16:30 |
| spatel | noonedeadpunk very interesting issue. its its not a big deal to bump version then why put it in upgrade doc. | 16:33 |
| noonedeadpunk | cloudnull: please check if this will be fine for your usecase after update: https://review.opendev.org/c/openstack/openstack-ansible-lxc_hosts/+/861676 :) | 16:33 |
| spatel | I am glad i ask before started upgrading, imagine who didn't ask :D | 16:33 |
| noonedeadpunk | spatel: beguase upgrade trough releases is not supported? | 16:34 |
| noonedeadpunk | at least until Y | 16:34 |
| spatel | You are saying i won't see issue if i do W->X and X-Y right? | 16:34 |
| noonedeadpunk | Yup | 16:35 |
| spatel | i see | 16:35 |
| noonedeadpunk | I'm not sure you will for W->Y either. But you will for V->X | 16:35 |
| cloudnull | noonedeadpunk I think that works fine. | 16:35 |
| noonedeadpunk | As they've bumped rpc version somewhere for W I guess. So if you're already on W it might be fine | 16:36 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Implement OVN inventory changes and deploy by default https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 16:42 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: [doc] Document better requirement for keepalived vip_cidr https://review.opendev.org/c/openstack/openstack-ansible/+/866046 | 16:53 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Run zookeeper installation for validate job https://review.opendev.org/c/openstack/openstack-ansible/+/866047 | 17:03 |
| *** dviroel|lunch is now known as dviroel | 17:12 | |
| spatel | noonedeadpunk Thank you!!! i will let you know next week how my upgrade goes :) | 17:17 |
| opendevreview | Merged openstack/openstack-ansible-os_nova stable/yoga: Isolate vif for ovs backend by default https://review.opendev.org/c/openstack/openstack-ansible-os_nova/+/864984 | 17:57 |
| opendevreview | Merged openstack/openstack-ansible master: Do not duplicate vers in nfs mount options https://review.opendev.org/c/openstack/openstack-ansible/+/865805 | 18:06 |
| opendevreview | Merged openstack/openstack-ansible-os_heat stable/yoga: Install git into heat containers https://review.opendev.org/c/openstack/openstack-ansible-os_heat/+/865564 | 18:11 |
| noonedeadpunk | mmmm... Have anybody tried out ovn driver for octavia? :D | 18:17 |
| noonedeadpunk | also octavia does not use tooz for jobboard but has invented it's own thing... | 18:20 |
| mgariepy | I havent | 18:25 |
| johnsom | noonedeadpunk: What do you mean “invented it’s own”? It is using Taskflow code and drivers, which uses tooz. | 18:30 |
| noonedeadpunk | johnsom: well I looked briefly through code and haven't spotted tooz usage. Eventually there's even no tooz requirement, but instead in extras you can ask for redis/zookeeper independntly https://opendev.org/openstack/octavia/src/branch/master/setup.cfg#L114-L121 | 18:37 |
| johnsom | Yes, because tooz is a requirement of Taskflow and not Octavia directly. | 18:38 |
| johnsom | The extras let you pick your backend for tooz | 18:39 |
| noonedeadpunk | hm..... | 18:39 |
| noonedeadpunk | I don't see tooz requirement there either | 18:41 |
| noonedeadpunk | I think I thought why no tooz is used, becuase otherwise you won't need that pile of config variables, like jobboard_backend_hosts, jobboard_backend_port, jobboard_backend_namespace as the only thing that's needed is coordination_url that is just passed to tooz and parsed there | 18:42 |
| noonedeadpunk | Another point why I'm under impression that tooz is not used, because jobboard_zookeeper_ssl_options is not a thing in tooz for zookeeper - I've proposed https://review.opendev.org/c/openstack/tooz/+/865532/4 quite recently to cover that | 18:44 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Add coordination to octavia https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/866058 | 18:47 |
| noonedeadpunk | So basically that's usually all config when tooz is used - https://docs.openstack.org/designate/latest/admin/config.html#coordination | 18:48 |
| noonedeadpunk | but yeah, might be you needed smth that's simply not implemented there or it was hard to adopt or smth... | 18:49 |
| johnsom | Tooz is mentioned in the docs here: https://docs.openstack.org/taskflow/latest/user/engines.html is why I thought it was using it, but yeah, I don't see an import either. It could be that Taskflow existed before tooz did. | 18:50 |
| johnsom | What I can tell you for sure is Octavia didn't invent something unique for jobboard, we just used what Taskflow had setup. | 18:51 |
| johnsom | Josh was a key developer on both Taskflow and Tooz, but sadly he isn't around anymore to ask why.... | 18:52 |
| noonedeadpunk | Well in terms of integration with zookeeper way of octavia configuration is very different from cinder, designate and gnocchi at least. But anyway:) | 18:58 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-plugins master: Allow to define condition for DB creation https://review.opendev.org/c/openstack/openstack-ansible-plugins/+/866059 | 19:02 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Add coordination to octavia https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/866058 | 19:02 |
| noonedeadpunk | To be frank I'm not very familiar with taskflow overall.... | 19:04 |
| noonedeadpunk | johnsom: wait. am I reading that right https://docs.openstack.org/taskflow/latest/user/persistence.html#zookeeper can be used for persistence_connection url? | 19:06 |
| johnsom | Yes, I think so. I don't think Octavia tests with that, but it should work | 19:09 |
| noonedeadpunk | yeah, it tests with mysql for sure... | 19:10 |
| johnsom | Yeah, I just checked, it is using mysql | 19:10 |
| noonedeadpunk | But driver in octavia seems to be supporting only mysql somehow https://opendev.org/openstack/octavia/src/branch/master/octavia/controller/worker/v2/taskflow_jobboard_driver.py#L36 | 19:13 |
| noonedeadpunk | Though I'm not sure if zookeeper won't "jsut work" | 19:14 |
| noonedeadpunk | hm | 19:14 |
| noonedeadpunk | I need to play with that | 19:14 |
| johnsom | Scroll down to line 63 | 19:14 |
| noonedeadpunk | But it's flwo driver | 19:14 |
| johnsom | Ah, nope that is the "other" driver for taskflow | 19:14 |
| noonedeadpunk | not persistance | 19:14 |
| noonedeadpunk | I'm looking at taskflow persistance right now https://opendev.org/openstack/taskflow/src/branch/master/taskflow/persistence/backends/__init__.py | 19:15 |
| noonedeadpunk | And tbh I don't see why it won't work if pass it with zookeeper url... except missing some ssl-related kwargs | 19:16 |
| noonedeadpunk | ugh | 19:16 |
| noonedeadpunk | Yeah, likely I will leave that alone for now... | 19:16 |
| johnsom | Yeah, if you need a zookeeper persistence, maybe open a story for it. | 19:16 |
| johnsom | Ann or Greg would be best to address that | 19:17 |
| noonedeadpunk | Well, it would simplfy things? As you don't need another database, migrations for it... | 19:17 |
| noonedeadpunk | yup. done. https://storyboard.openstack.org/#!/story/2010455 | 19:22 |
| noonedeadpunk | Maybe will pick it up one day :D | 19:22 |
| johnsom | Thanks | 19:23 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Add coordination to octavia https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/866058 | 19:29 |
| noonedeadpunk | I probably still should try to provide zookeeper url yolo style in a sandbox.... | 19:30 |
| jamesdenton | mgariepy you around? | 19:33 |
| mgariepy | somewhat! | 19:33 |
| mgariepy | what can i do for you | 19:33 |
| jamesdenton | running the ovn ssl patches, and it looks like the certs are not being generated. I see your new "Create and install SSL certificates" task go by but nothing appears to happen | 19:34 |
| jamesdenton | <mnaio-compute2> Task "Create and install SSL certificates" has been omitted from the job because the conditional "['neutron_ovn_ssl', "'network_all' in group_names"]" was evaluated as "False" | 19:34 |
| jamesdenton | that could be why. lol | 19:35 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable octavia anti-affinity for AIO builds https://review.opendev.org/c/openstack/openstack-ansible/+/866061 | 19:35 |
| mgariepy | hmm should it be filtered on something else? | 19:35 |
| jamesdenton | well, a compute is not a network host | 19:36 |
| jamesdenton | the openstack controllers appear to have the keys | 19:36 |
| jamesdenton | and in an AIO, that would prob be OK | 19:36 |
| jamesdenton | so maybe filtering on the actual ovn groups would be better? northd and controller, or whatever they are | 19:37 |
| mgariepy | hmm yep i guess it would be better. | 19:38 |
| jamesdenton | i'll try it and see | 19:38 |
| jamesdenton | thanks | 19:39 |
| mgariepy | i remember when horizon network security group were not in the network tab.. | 19:39 |
| mgariepy | good times.. | 19:39 |
| mgariepy | i knew it wasn't perfect :D haah | 19:41 |
| jamesdenton | seems close! | 19:43 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-os_octavia master: Change defaults for octavia topology and affinity https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/866062 | 19:43 |
| noonedeadpunk | since everyone around - some review on https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/855074/3 would be helpful to unblock octavia | 19:44 |
| jamesdenton | done | 19:46 |
| noonedeadpunk | any idea wtf can be with https://zuul.opendev.org/t/openstack/build/6393f9a8f57e4857a9213ce793a6ca32/log/job-output.txt#17334 ? | 19:47 |
| noonedeadpunk | sounds like smth is missing in openstack_user_config or smth... | 19:48 |
| jamesdenton | hmm | 19:50 |
| noonedeadpunk | I think it's this task https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L56 | 19:50 |
| jamesdenton | hmm, might be container_bridge is undefined or null? | 19:50 |
| jamesdenton | looking | 19:51 |
| jamesdenton | did this only fail in jammy? | 19:52 |
| noonedeadpunk | nah, everywhere | 19:52 |
| noonedeadpunk | except upgrade jobs :D | 19:52 |
| noonedeadpunk | So upgrade hook looks like working | 19:52 |
| jamesdenton | hmm, strange. it never failed with this error before, though, IIRC | 19:52 |
| noonedeadpunk | well, I can assume that smth wrong is here specifically https://review.opendev.org/c/openstack/openstack-ansible/+/862924 | 19:53 |
| jamesdenton | agreed | 19:53 |
| noonedeadpunk | I haven't touched env.d/o_u_c from your patchset | 19:53 |
| noonedeadpunk | (as can hardly judge on them) | 19:54 |
| noonedeadpunk | Well, I've dropped ovn overrides for aio | 19:55 |
| noonedeadpunk | but I moved everything (I think) | 19:55 |
| jamesdenton | ok i know what it is | 19:57 |
| noonedeadpunk | ? | 19:58 |
| noonedeadpunk | As I can't find neutron_provider_networks being defined at all... | 19:59 |
| jamesdenton | provider bridge is setup for neutron-ovn-gateway group, but that task is failing on the neutron-ovn-controller group (since there's no provider bridge mapping) | 19:59 |
| jamesdenton | so, i think we may need to add an OR statement... to include both neutron-ovn-controller and neutron-ovn-gateway | 19:59 |
| jamesdenton | https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/760647 | 20:00 |
| jamesdenton | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L65 | 20:00 |
| jamesdenton | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L77 | 20:00 |
| jamesdenton | https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L84 | 20:00 |
| noonedeadpunk | you're saying we have circular dependency? | 20:00 |
| jamesdenton | not quite | 20:01 |
| jamesdenton | our current OVN setup assume an ovn controller (compute node) is also an ovn gateway node | 20:01 |
| jamesdenton | but we want that split out | 20:01 |
| noonedeadpunk | not sure I'm following why aio metal is affected though | 20:01 |
| jamesdenton | good question, i'm not quite sure either | 20:02 |
| noonedeadpunk | because it shouldn't matter kind of - all groups are there anyway | 20:02 |
| jamesdenton | but it's related to this, i think. | 20:02 |
| jamesdenton | agreed | 20:02 |
| jamesdenton | i'll spin up an AIO and see if i can fix this | 20:02 |
| noonedeadpunk | (and bridges) | 20:02 |
| noonedeadpunk | yes, would be awesome, thanks! | 20:02 |
| noonedeadpunk | (and it gets a bit late here) | 20:03 |
| noonedeadpunk | I have several dependancies for https://review.opendev.org/c/openstack/openstack-ansible/+/862924 (and relations chains in some of them) | 20:04 |
| noonedeadpunk | So basically it also depends on ovn ssl patch | 20:04 |
| jamesdenton | mgariepy https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/862403 | 20:07 |
| opendevreview | Merged openstack/openstack-ansible-openstack_hosts master: Revert "Use pam_env for su commands on Centos-9" https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/860658 | 20:07 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible-openstack_hosts stable/yoga: Revert "Use pam_env for su commands on Centos-9" https://review.opendev.org/c/openstack/openstack-ansible-openstack_hosts/+/865997 | 20:08 |
| jamesdenton | SSL patch successful w/ that change. VM accessible | 20:09 |
| opendevreview | Dmitriy Rabotyagov proposed openstack/openstack-ansible master: Disable fact variables https://review.opendev.org/c/openstack/openstack-ansible/+/778396 | 20:09 |
| * noonedeadpunk heads out for today | 20:10 | |
| jamesdenton | see ya | 20:10 |
| admin1 | is there a tag for all services to only add rabbitmq after a rabbitmq rebuild ? | 20:17 |
| opendevreview | Marc Gariépy proposed openstack/openstack-ansible-os_neutron master: add ovn ssl config https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/862403 | 21:19 |
| mgariepy | jamesdenton, done ^^ | 21:21 |
| jamesdenton | thanks! | 21:22 |
| mgariepy | you are welcome | 21:22 |
| mgariepy | do you have something to add to noonedeadpunk comment here https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/862403/comments/240adc70_5aa9d59f | 21:23 |
| mgariepy | not 100% sure about the handler stuff. | 21:25 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Adopt output structure to new collections version https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/855074 | 21:29 |
| opendevreview | Merged openstack/openstack-ansible-os_octavia master: Adding octavia_provider_network_mtu-parameter parameter https://review.opendev.org/c/openstack/openstack-ansible-os_octavia/+/864819 | 21:29 |
| *** dviroel is now known as dviroel|out | 22:00 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
