| opendevreview | Merged openstack/openstack-ansible-ceph_client master: Provide opportunity to define cluster_name https://review.opendev.org/c/openstack/openstack-ansible-ceph_client/+/852588 | 00:42 |
|---|---|---|
| *** chandankumar is now known as chkumar|ruck | 05:02 | |
| *** ysandeep|out is now known as ysandeep | 05:22 | |
| noonedeadpunk | damiandabrowski: pls take a look at https://review.opendev.org/c/openstack/openstack-ansible/+/855255 when possible | 06:55 |
| damiandabrowski | morning, done ;) | 07:00 |
| *** arxcruz is now known as arxcruz|rover | 07:12 | |
| anskiy | jamesdenton: I haven't been able to reproduce the problem with OVN cluster not upgrading, at least, on newly created OVN cluster, without ovn-controllers/openstack. I can possibly try to install test Xena cluster with more controller nodes and try to upgrade to Yoga, but I did it prior to "production" upgrade and haven't catched that error, at least with one controller. | 07:33 |
| anskiy | non-cluster setup seems to be upgrading just fine too | 07:33 |
| *** ysandeep is now known as ysandeep|afk | 09:49 | |
| *** ysandeep|afk is now known as ysandeep | 11:08 | |
| *** carloss is now known as carloss|afk | 11:28 | |
| jamesdenton | i believe you. i will try an OVN MNAIO and see - i don't think we have any specific "cluster upgrade" task, so if it's not rolled into an OVS/OVN service restart then we might be missing something | 11:32 |
| jamesdenton | noonedeadpunk: when you have a sec, please let me know about this: https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/855573 | 11:34 |
| noonedeadpunk | ah, that's good one | 11:42 |
| noonedeadpunk | thanks | 11:42 |
| *** carloss|afk is now known as carloss | 13:03 | |
| anskiy | jamesdenton: this environment on which I've tested upgrade is kinda similar to MNAIO, I just spawn it with Vagrant: one controller, one compute, one deployment host and a couple more VMs to help replicate "production" network and storage. "cluster upgrade" thing SHOULD be run on service restart, as it's backend in execstart script for ovsdb. Maybe, something was just more broken with my "production" setup | 13:41 |
| jamesdenton | FWIW i have done a 24.x -> 25.0.0 upgrade in the last few weeks and don't recall any OVN-related issue. | 13:42 |
| anskiy | maybe, even prior to X->Y upgrade, as this thing survived W->X upgrade already, but I do believe, that schema version in SB_Global DB was set to the version, that is used in OVN/OVS 20.03 | 13:42 |
| anskiy | yeah, that's what I'm talking: could be just some different problem on my end | 13:43 |
| jamesdenton | quite possible, good to know either way | 13:45 |
| jamesdenton | i am happy to say i know enough to know i don't know | 13:46 |
| anskiy | well, at least, I know how to fix this thing if it ever breaks :) | 13:46 |
| noonedeadpunk | well, for X our OVN implementation was a bit different as it was jsut adopted quite recently | 13:49 |
| noonedeadpunk | (back then) | 13:49 |
| jamesdenton | non-proper cluster implementation vs now? | 13:56 |
| anskiy | nah, I was carrying those spatel patches for proper cluster since the beginning in W | 14:01 |
| spatel | sorry i am not able to see offline messages.. just curious what is going on here? | 14:04 |
| spatel | i did upgrade from W -> X in OVN deployment without issue.. | 14:04 |
| spatel | because upgrade process just going to upgrade your distro binaries of ovn and openvswitch | 14:05 |
| noonedeadpunk | there was some issue that ovndb did not get upgraded when binaries were | 14:05 |
| opendevreview | Merged openstack/openstack-ansible-os_neutron master: Switch jobs from CentOS 8 Stream to CentOS 9 Stream https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/855573 | 14:05 |
| spatel | noonedeadpunk hmm | 14:06 |
| anskiy | spatel: so I had to run upgrade manually, and can't reproduce this thing now. That's it. | 14:08 |
| anskiy | could be just my broken environment, that highlight was more for appreciation of work :) | 14:09 |
| spatel | noonedeadpunk i am not seeing it required anything to update DB - https://docs.ovn.org/en/stable/intro/install/ovn-upgrades.html | 14:09 |
| spatel | anskiy How did you upgrade manually just curious? | 14:10 |
| anskiy | oh god... so, `upgrade_cluster` function is located in "library" (`/usr/share/openvswitch/scripts/ovs-lib`), which sourced in `/usr/share/ovn/scripts/ovn-ctl` | 14:12 |
| spatel | hmm interesting | 14:13 |
| anskiy | https://github.com/openvswitch/ovs/blob/024ba52575e3110118bbedc91f1c0d75eb60b8e8/utilities/ovs-lib.in | 14:13 |
| anskiy | I've either sourced it and called this thing directly, or fixed startup script `/usr/share/ovn/scripts/ovn-ctl` to always explicitly run upgrade, not quite sure now :) | 14:15 |
| jamesdenton | noted. will see if it happens again! :) | 14:16 |
| anskiy | this is how broken state looks in ovn-northd: https://paste.opendev.org/show/big24pUad9s4COBTREB1/ | 14:20 |
| anskiy | and all agents become dead, because neutron checks that https://paste.opendev.org/show/bXpHnsB7ohB6eQmqMl0b/ *_cfg's values are the same (or less, than one) in all databases (can't quite remember which ones are important, but during normal operation they should be the same | 14:22 |
| anskiy | and here, you can see the DB "version" | 14:23 |
| anskiy | which should somewhat replicate the OVN's package version | 14:24 |
| noonedeadpunk | that is veeeery interesting. | 14:28 |
| *** ysandeep is now known as ysandeep|out | 14:34 | |
| anskiy | noonedeadpunk: well, here is the code for you then! https://opendev.org/openstack/neutron/src/branch/master/neutron/plugins/ml2/drivers/ovn/agent/neutron_agent.py#L99-L111 | 14:34 |
| noonedeadpunk | yeah, I just did not have chance to get my hands into ovn | 14:35 |
| noonedeadpunk | so just writing down everything that might go wrong for the future me | 14:36 |
| *** dviroel is now known as dviroel|lunch | 15:13 | |
| noonedeadpunk | fwiw I already applied https://review.opendev.org/c/openstack/openstack-ansible/+/855184 from my AZ sandbox and overrides are soooo much cleaner now, so thank for contributing back anskiy :) | 15:40 |
| anskiy | noonedeadpunk: it was mostly your idea, but thanks :) | 15:59 |
| opendevreview | James Denton proposed openstack/openstack-ansible-os_neutron master: Dynamic ovn-encap-type configuration https://review.opendev.org/c/openstack/openstack-ansible-os_neutron/+/850639 | 16:09 |
| *** dviroel|lunch is now known as dviroel | 16:41 | |
| *** dviroel is now known as dviroel|out | 20:56 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!