| rdogerrit | rdo-trunk rdo-trunk proposed rdoinfo master: Promote CBS tags update for antelope-9s-testing https://review.rdoproject.org/r/c/rdoinfo/+/47785 | 00:05 |
|---|---|---|
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-openvswitch2.16-testing https://review.rdoproject.org/r/c/nfvinfo/+/48011 | 00:14 |
| rdogerrit | rdo-trunk rdo-trunk proposed nfvinfo master: Promote CBS tags update for nfvsig-openvswitch2.17-testing https://review.rdoproject.org/r/c/nfvinfo/+/48012 | 00:14 |
| rdogerrit | Takashi Kajinami proposed openstack/heat-agents-distgit rpm-master: Do not require puppet in RHOSP https://review.rdoproject.org/r/c/openstack/heat-agents-distgit/+/48002 | 01:26 |
| rdogerrit | Marios Andreou proposed rdo-jobs master: Add failed_when to nested make ansible for data plane adoption jobs https://review.rdoproject.org/r/c/rdo-jobs/+/47955 | 06:18 |
| rdogerrit | Merged rdo-infra/rdo-infra-playbooks master: Remove microshift role https://review.rdoproject.org/r/c/rdo-infra/rdo-infra-playbooks/+/47870 | 06:32 |
| *** amoralej|off is now known as amoralej | 07:26 | |
| rdogerrit | Merged rdo-jobs master: Remove rdoinfo-tripleo: -ussuri, -victoria, -xena, -yoga jobs https://review.rdoproject.org/r/c/rdo-jobs/+/48004 | 07:31 |
| rdogerrit | Daniel Pawlik proposed config master: Remove submit-logstash-job https://review.rdoproject.org/r/c/config/+/48013 | 07:32 |
| rdogerrit | Daniel Pawlik proposed config master: Remove submit-logstash-job https://review.rdoproject.org/r/c/config/+/48013 | 07:32 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Update ovs to 3.1 and ovn to 22.12 for Xena https://review.rdoproject.org/r/c/rdoinfo/+/48014 | 07:35 |
| rdogerrit | Merged nfvinfo master: Promote CBS tags update for nfvsig-openvswitch2.16-testing https://review.rdoproject.org/r/c/nfvinfo/+/48011 | 07:36 |
| rdogerrit | Alfredo Moralejo proposed rdoinfo master: Update ovs to 3.1 and ovn to 22.12 for Wallaby https://review.rdoproject.org/r/c/rdoinfo/+/48015 | 07:37 |
| amoralej | ykarel, ^ fyi | 07:37 |
| ykarel | amoralej, Thanks | 07:43 |
| rdogerrit | Joel Capitao proposed rdoinfo master: Remove CloudSIG GA repo definition for X and W https://review.rdoproject.org/r/c/rdoinfo/+/48016 | 07:49 |
| rdogerrit | Karolina Kula proposed rdo-jobs master: Remove tripleo jobs in RDO and rdoinfo https://review.rdoproject.org/r/c/rdo-jobs/+/47959 | 08:23 |
| rdogerrit | Jiří Podivín proposed openstack/validations-libs-distgit rpm-master: DNM this change is just to test review https://review.rdoproject.org/r/c/openstack/validations-libs-distgit/+/48017 | 08:25 |
| rdogerrit | Jiří Podivín proposed openstack/validations-libs-distgit wallaby-rdo: Removing check stage for the package build https://review.rdoproject.org/r/c/openstack/validations-libs-distgit/+/48018 | 08:37 |
| rdogerrit | Alfredo Moralejo proposed deps/ansible-core c9s-antelope-rdo: Rebuild and rename ansible-core with python 3.9 https://review.rdoproject.org/r/c/deps/ansible-core/+/48019 | 08:38 |
| rdogerrit | Joel Capitao proposed rdo-jobs master: WIP refactor rdoinfo dlrn script https://review.rdoproject.org/r/c/rdo-jobs/+/48020 | 08:42 |
| noonedeadpunk | Hey folks! We're seing a weird issue in the gates now, maybe you have some idea about that? https://zuul.opendev.org/t/openstack/build/f295470ef7fe4ad3ae25d5c985c3d304/log/job-output.txt#18951 | 08:52 |
| noonedeadpunk | basically openvswitch3.1 requires libmlx5.so.1(MLX5_1.24) which is melanox deliverable and is completely wrong to depend on that | 08:53 |
| rdogerrit | Merged rdoinfo master: Update ovs to 3.1 and ovn to 22.12 for Xena https://review.rdoproject.org/r/c/rdoinfo/+/48014 | 09:01 |
| noonedeadpunk | wonder if it also has smth to do with this ^ | 09:02 |
| noonedeadpunk | So now all releases down to Wallaby do require to have mellanox drivers installed? | 09:03 |
| rdogerrit | Marios Andreou proposed rdo-jobs master: Add failed_when to nested make ansible for data plane adoption jobs https://review.rdoproject.org/r/c/rdo-jobs/+/47955 | 09:06 |
| noonedeadpunk | I'm really not sure it's a good idea to upgrade stuff in that major way during the lifecycle. | 09:07 |
| noonedeadpunk | It's already breaking CI and I can hardly imagine consequences of already running deployments that do upgrade packages in a timely manner. | 09:07 |
| rdogerrit | Arx Cruz proposed rdo-jobs master: Run tempest on edpm based job https://review.rdoproject.org/r/c/rdo-jobs/+/48021 | 09:07 |
| rdogerrit | Merged config master: Fix filename for data-plane-adoption base-crc-standalone definition https://review.rdoproject.org/r/c/config/+/48006 | 09:11 |
| rdogerrit | Karolina Kula proposed rdo-jobs master: Remove tripleo master and zed jobs in rdoinfo https://review.rdoproject.org/r/c/rdo-jobs/+/47959 | 09:34 |
| rdogerrit | Karolina Kula proposed rdo-jobs master: Remove tripleo master and zed jobs in rdoinfo https://review.rdoproject.org/r/c/rdo-jobs/+/47959 | 09:53 |
| amoralej | noonedeadpunk, that is provided by libibverbs which is in centos baseos, what's the problem with it? | 10:16 |
| amoralej | i guess that coms from https://git.centos.org/rpms/openvswitch/blob/c9s-sig-nfv-openvswitch-3.1/f/SPECS/openvswitch3.1.spec#_309 | 10:18 |
| amoralej | mlx5 is in ENABLED_DRIVERS list | 10:18 |
| noonedeadpunk | Well, problem is that `nothing provides libmlx5.so.1(MLX5_1.24)`. But I've picked bad job example I guess as it's rocky | 10:20 |
| amoralej | that must be difference between centos stream and rocky | 10:21 |
| rdogerrit | Daniel Pawlik proposed rdo-jobs master: Add option to retry crc_storage creation https://review.rdoproject.org/r/c/rdo-jobs/+/48022 | 10:21 |
| amoralej | not reproducible in centos stream | 10:21 |
| noonedeadpunk | yeah... I just realized that centos fails differently :) | 10:21 |
| amoralej | mmm fails on centos? it worked in my test | 10:21 |
| noonedeadpunk | We just got all our centos gates broken in last 2-3 days and each time it fails kinda differently... | 10:22 |
| amoralej | mmm | 10:22 |
| amoralej | may i check logs on centos? | 10:22 |
| noonedeadpunk | First due to missing systemd-udev in bare minimum, while rocky had, next rocky failed with this rdo thing, but centos moved on and failed on temepst | 10:22 |
| noonedeadpunk | yeah, sec | 10:23 |
| noonedeadpunk | So centos fails due to this new ovs I'm pretty sure | 10:24 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/host/nova-compute.service.journal-02-32-13.log.txt#4156 | 10:24 |
| amoralej | neutron guys requested to bump ovs to 3.1 | 10:24 |
| amoralej | it's worthy to check what'd be the problem | 10:24 |
| amoralej | and we've run some ci on the update | 10:25 |
| noonedeadpunk | I'm jsut saying this bump is gonna break deployments from what I do see right now from our CI, that was fully green on Friday | 10:26 |
| amoralej | jobs are passing with same ovn and ovs packages in other gates as https://zuul.opendev.org/t/openstack/build/765be74feb184679979377d008af40a1/ | 10:32 |
| amoralej | 2023-03-29 02:31:40.646994+00:00 neutron-server.service 2023-03-29 02:31:40.646 117796 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-bb21a16b-fc5d-4b5a-a81b-6c8349e2aff0 req-667cfe1e-6c99-484f-86ad-f13211e0b6d6 4fc145aa63db41c8884644546f4b5220 57576c263b2e497a9c4a6c7c36f0476f - - default default] Refusing to bind port c6249ef3-801f-4615-8630-5dcaf2a7c3ee due to no OVN chassis for host: aio1 | 10:32 |
| amoralej | in | 10:33 |
| amoralej | https://5384e903e2d59c244ac8-1738d3a5d5585f54500f5c8600ed53a6.ssl.cf5.rackcdn.com/878115/3/check/openstack-ansible-deploy-aio_metal-centos-9-stream/64d8812/logs/host/neutron-server.service.journal-02-32-13.log.txt | 10:33 |
| amoralej | i guess that's the root cause | 10:33 |
| noonedeadpunk | The reason is probably this https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/host/ovn/ovn-northd.log.txt#32 | 10:34 |
| noonedeadpunk | I mean - we totally can figure that down and fix, main question is how this is going affect production deployments, also in terms of how smooth and expected such upgrade would be. As after `dnf update` unexpected consequences might rise | 10:36 |
| noonedeadpunk | amoralej: is reason why neutron folks asked for that written down anywhere? | 10:36 |
| noonedeadpunk | like some ptg etherpad? | 10:37 |
| amoralej | nop | 10:37 |
| dalvarez | amoralej o/ | 10:39 |
| amoralej | dalvarez, noonedeadpunk was asking about issues in openstack-ansible deployment after upgrade of ovs/ovn | 10:40 |
| amoralej | i guess the issue is in https://5384e903e2d59c244ac8-1738d3a5d5585f54500f5c8600ed53a6.ssl.cf5.rackcdn.com/878115/3/check/openstack-ansible-deploy-aio_metal-centos-9-stream/64d8812/logs/host/neutron-server.service.journal-02-32-13.log.txt | 10:40 |
| dalvarez | yeah i checked the logs, seems like there might be a problem with the hostname? (aio1) as per: https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/host/ovn/ovn-northd.log.txt | 10:40 |
| dalvarez | ykarel: does this ring a bell to you? | 10:41 |
| amoralej | we didn't hit this issue with other deployment tools, right? | 10:41 |
| dalvarez | amoralej: that seems like another issue... the 'aio1' issue is that the SBDB Chassis table does not contain an entry for that chassis | 10:41 |
| dalvarez | when ovn-controller registers itself it'll do it with the hostname in the local ovsdb | 10:42 |
| dalvarez | but i can't seem to find the ovs-vsctl list open . in the logs | 10:42 |
| amoralej | so that matchs the error in neutron-server | 10:42 |
| amoralej | 2023-03-29 02:31:40.627101+00:00 neutron-server.service 2023-03-29 02:31:40.626 117796 WARNING neutron.plugins.ml2.drivers.ovn.mech_driver.mech_driver [req-bb21a16b-fc5d-4b5a-a81b-6c8349e2aff0 req-667cfe1e-6c99-484f-86ad-f13211e0b6d6 4fc145aa63db41c8884644546f4b5220 57576c263b2e497a9c4a6c7c36f0476f - - default default] Refusing to bind port c6249ef3-801f-4615-8630-5dcaf2a7c3ee due to no OVN chassis for host: aio1 | 10:42 |
| dalvarez | amoralej: if we dont hit that in other deployment tools it's likely because this particular tool is not setting the hostname correctly | 10:42 |
| dalvarez | or it fails at some point (eg. becuase ovs hasn't started on time or something of that sort) | 10:42 |
| amoralej | you mean the hostname in ovsdb ? | 10:42 |
| dalvarez | amoralej: yeah exactly the hostname in the *local* ovsdb | 10:43 |
| dalvarez | like: ovs-vsctl list open . |grep hostname | 10:43 |
| dalvarez | would be the thing | 10:43 |
| amoralej | noonedeadpunk, ^ | 10:43 |
| noonedeadpunk | Yeah, i think we should be doing that, I'm trying to find the code | 10:44 |
| noonedeadpunk | Like that? https://opendev.org/openstack/openstack-ansible-os_neutron/src/branch/master/tasks/providers/setup_ovs_ovn.yml#L18 | 10:44 |
| dalvarez | noonedeadpunk: yeah that exactly | 10:44 |
| amoralej | afayk, there is any required task to update ovs/ovn from 2.17/22.06 to 3.1/22.12 ? | 10:44 |
| noonedeadpunk | this task ran at least, from what I can tell | 10:45 |
| dalvarez | that thing needs to be in place before this: | 10:45 |
| dalvarez | https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/host/ovn/ovn-controller.log.txt#52 | 10:45 |
| noonedeadpunk | Maybe there's some difference between hostname/fqdn/nodename, but given it jsut worked, I'd assume it should not be the case | 10:45 |
| dalvarez | if the files in that setup would include the actual ovs db files we could check | 10:45 |
| noonedeadpunk | well, it should not be hard to inlcude into logs gathering. | 10:48 |
| noonedeadpunk | do you have path to it somewhere handy ?:) | 10:49 |
| noonedeadpunk | or let me spawn env to check that | 10:52 |
| noonedeadpunk | but according to order I see in the log, setting up hostname is the first action we do | 10:55 |
| amoralej | that sounds good | 11:02 |
| *** amoralej is now known as amoralej|lunch | 11:03 | |
| karolinku[m] | amoralej, jcapitao: I'm chairing today's RDO meeting, but need to leave a little earlier, so I can't be at whole meeting. would do like to switch with me, and I will take next week one? | 11:18 |
| ykarel | dalvarez, conf.db collected https://8c14ddfe56056e4e35ff-0ca364809dbe3b9250075209fe309a8e.ssl.cf5.rackcdn.com/876851/8/check/openstack-ansible-deploy-aio_metal-centos-9-stream/3d37cb7/logs/etc/host/openvswitch/conf.db.txt | 11:25 |
| ykarel | {"_date":1680012190196,"Open_vSwitch":{"fd76f571-4dee-4883-923d-f6ef6e0b7f86":{"external_ids":["map",[["hostname","aio1.openstack.local"]]]}},"_is_diff":true,"_comment":"ovs-vsctl (invoked by /usr/bin/sh): ovs-vsctl --no-wait add Open_vSwitch . external-ids hostname=aio1.openstack.local"} | 11:25 |
| ykarel | noonedeadpunk, ^ | 11:26 |
| noonedeadpunk | aha... | 11:26 |
| noonedeadpunk | that is weird... | 11:26 |
| noonedeadpunk | BUT | 11:27 |
| noonedeadpunk | If you check couple of lines down | 11:27 |
| noonedeadpunk | {"_date":1680012236761,"Open_vSwitch":{"fd76f571-4dee-4883-923d-f6ef6e0b7f86":{"external_ids":["map",http://en.wikipedia.org/wiki/Special:Search?go=Go&search="hostname","aio1"],"next_cfg":1}},"_is_diff":true,"_comment":"ovs-vsctl (invoked by /usr/bin/python3): ovs-vsctl set open_vswitch . external-ids:hostname=aio1"} | 11:27 |
| noonedeadpunk | I assume that it should kind override? above one? | 11:28 |
| noonedeadpunk | As timestap is later at least | 11:28 |
| noonedeadpunk | but it happens after br-int /o\ | 11:29 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/etc/host/openvswitch/conf.db.txt#20 | 11:31 |
| noonedeadpunk | And I guess then on the restart, it's reseted? https://zuul.opendev.org/t/openstack/build/64d8812538584e4b9d2b95e4c0907299/log/logs/etc/host/openvswitch/conf.db.txt#64 | 11:32 |
| noonedeadpunk | Just in case - we have hostname != fqdn on purpose to catch such things, as we had issues with masakari for example, that always assumed that hostname == fqdn | 11:34 |
| noonedeadpunk | I'm still spawning the env. So basically this boils down on finding a way to prevent OVS resetting defined hostname | 11:36 |
| noonedeadpunk | I do hope it's smth in systemd unit | 11:37 |
| ykarel | yes the later one overrides | 11:40 |
| noonedeadpunk | and now we know which deployments end up broken after package upgrade on stable branches :) | 11:46 |
| jcapitao[m] | karolinku: sure I can chair the meeting today | 11:53 |
| karolinku[m] | thanks jcapitao! | 11:53 |
| *** amoralej|lunch is now known as amoralej | 12:19 | |
| amoralej | noonedeadpunk, so, what's doing the override? | 12:22 |
| amoralej | thanks ykarel++ ! | 12:22 |
| noonedeadpunk | amoralej: hostname is set on L8, L20 and L64. Playbooks do that on L20 only, L8 and L64 is smth new we're not aware about | 12:27 |
| noonedeadpunk | and eventually this smth new does things in a wrong way | 12:29 |
| noonedeadpunk | here's the paste with hostnamectl output https://ctxt.io/2/AACQoQeDEg | 12:31 |
| noonedeadpunk | So it should not be adding aio1.openstack.local as nova and neutron does discover compute service by their hostname, not fqdn. While nova hypervisor is discovered by fqdn in it's turn. | 12:32 |
| noonedeadpunk | And that boils down actually to how you define your /etc/hosts for 127.0.1.1 | 12:33 |
| noonedeadpunk | as results will depend only on that for such scenarios | 12:33 |
| amoralej | noonedeadpunk, apparently there is a config option you need to set if you don't want to use FQDN | 12:36 |
| amoralej | FULL_HOSTNAME | 12:36 |
| amoralej | by default is set to yes | 12:37 |
| amoralej | that forces hostname to be set to fqdn on each restart | 12:37 |
| amoralej | https://mail.openvswitch.org/pipermail/ovs-dev/2018-October/352688.html | 12:37 |
| noonedeadpunk | yeah, already reading that... | 12:38 |
| noonedeadpunk | this thread is from 2018... | 12:38 |
| noonedeadpunk | so I'd say smth else has changed now... | 12:39 |
| noonedeadpunk | I think event better option could be to jsut disable `RECORD_HOSTNAME` https://github.com/openvswitch/ovs/blob/master/utilities/ovs-ctl.in#L238 | 12:42 |
| noonedeadpunk | but yeah, thanks for helping out here | 12:44 |
| rdogerrit | Daniel Pawlik proposed config master: Remove submit-logstash-job https://review.rdoproject.org/r/c/config/+/48013 | 12:46 |
| amoralej | noonedeadpunk, if you want to compare with previous situation, the previous version of rdo-openvswitch wrapper is still in the repos so you can force to install it or exclude *rdo*3.1* in the repos config | 12:48 |
| amoralej | and test it | 12:48 |
| noonedeadpunk | I kind of wonder if `FULL_HOSTNAME=no` should be defaulted for opensatck usecase... | 12:53 |
| noonedeadpunk | btw, if check ubuntu conf.db, it does not trigger hostname change each time on restart somehow | 12:57 |
| noonedeadpunk | it's just l12 and l20 | 12:57 |
| noonedeadpunk | https://zuul.opendev.org/t/openstack/build/d2db8bf1cdeb44e0b5e2ab54f2e50bdd/log/logs/etc/host/openvswitch/conf.db.txt | 12:58 |
| amoralej | noonedeadpunk, changing FULL_HOSTNAME would change then hostname to uname -n | 13:01 |
| amoralej | that'd break installations running with fqdn | 13:01 |
| amoralej | maybe ubuntu changed the upstream behavior | 13:01 |
| noonedeadpunk | yeah, it's kinda stupid... | 13:06 |
| noonedeadpunk | Basically what should be done looking at the code - RECORD_HOSTNAME=no and then run it with `record-hostname-if-not-set` once on the first startup, like post-install trigger | 13:07 |
| noonedeadpunk | yes, it will change hostname on each package upgrade, but at least not after each service restart.... | 13:09 |
| amoralej | note that the package is not automatically starting the service, so we can't do in in post-install | 13:12 |
| amoralej | iirc | 13:13 |
| noonedeadpunk | yup, they do `--no-record-hostname` is systemd unit | 13:15 |
| noonedeadpunk | https://paste.openstack.org/show/bqcz2ijbDSKLZfgn7f9k/ | 13:15 |
| amoralej | that explains it | 13:16 |
| dalvarez | noonedeadpunk: amoralej ykarel https://mail.openvswitch.org/pipermail/ovs-dev/2020-May/370896.html | 13:16 |
| dalvarez | this patch i sent some time back has some insights | 13:16 |
| dalvarez | "I think it is better to add a new option to override it, such as "--no-hostname", so that existing environment won't get impacted" | 13:16 |
| amoralej | ah, that looks like the --no-record-hostname option | 13:17 |
| dalvarez | yeah | 13:17 |
| dalvarez | my patch added that option, perhaps it was renamed | 13:17 |
| dalvarez | + --no-hostname do not set external-id:hostname | 13:17 |
| noonedeadpunk | I don't see --no-hostname in 2.17 at least https://github.com/openvswitch/ovs/blob/branch-2.17/utilities/ovs-ctl.in#L419-L428 | 13:19 |
| noonedeadpunk | But having that by default in unit file makes much sense to me. IMO, it;s better not to set hostname at all rather then override it on each service restart by default | 13:20 |
| dalvarez | yeah i think thats precisely why i sent that patch | 13:20 |
| dalvarez | not sure if it was renamed or removed altogether | 13:20 |
| dalvarez | it's likely been renamed to " --no-record-hostname " | 13:21 |
| noonedeadpunk | at least it was renamed... | 13:21 |
| noonedeadpunk | amoralej: can you help me out on how to install older version in more details?:) as trying to isntall openvswitch2.17 doens't work obviously | 13:25 |
| amoralej | which repo are you using? how are you configuring repos? | 13:26 |
| ykarel | dalvarez, ack | 13:26 |
| noonedeadpunk | https://trunk.rdoproject.org/centos9-zed/deps/latest/ | 13:26 |
| noonedeadpunk | basically https://paste.openstack.org/show/bjpKKpx2PkAN5voZYb9r/ | 13:27 |
| amoralej | noonedeadpunk, add "exclude = *rdo-openvswitch*3.1*,*rdo-ovn*3.1* | 13:31 |
| amoralej | " | 13:31 |
| amoralej | or if you want to just test manually install it | 13:31 |
| amoralej | dnf install rdo-openvswitch-2.17 should just work | 13:31 |
| amoralej | i don't find any difference in systemd unit or ovs-ctl scripts between 2.17 and 3.1 | 13:33 |
| amoralej | i'm not sure what may have changed the behavior | 13:33 |
| noonedeadpunk | Yeah, I don't really see differences at first glance... | 13:34 |
| noonedeadpunk | Well, we have way forward by adjusting `/etc/sysconfig/openvswitch` at least... | 13:35 |
| amoralej | yes | 13:36 |
| noonedeadpunk | I'd kind of wish centos-nfv-openvswitch repo to patch systemd unit to be frank, but well... | 13:36 |
| amoralej | but look at this noonedeadpunk | 13:36 |
| amoralej | https://mail.openvswitch.org/pipermail/ovs-dev/2020-May/370915.html | 13:36 |
| amoralej | "since add doesn't overwrite existing values if the key is set[1]." | 13:36 |
| amoralej | that may be a bug | 13:36 |
| amoralej | in 3.1 ? | 13:37 |
| amoralej | With --if-exists... | 13:38 |
| noonedeadpunk | well. ubuntu has 3.1.1 atm, it's 3.1.0 in centos-nfv-openvswitch | 13:38 |
| amoralej | that'd be good to test with 2.17 and 3.1 | 13:38 |
| amoralej | https://github.com/openvswitch/ovs/commit/2221e8b034298bd77dc9282895c4b32fd11583e8 | 13:41 |
| noonedeadpunk | well, I see it as `add`... | 13:43 |
| amoralej | yes | 13:47 |
| noonedeadpunk | let me check if it works as intended... | 13:47 |
| noonedeadpunk | At least it does on 2.17 | 13:47 |
| noonedeadpunk | it's broken on 3.0 | 13:49 |
| noonedeadpunk | that's from 3.0 https://paste.openstack.org/show/bunzz47SWuAZ1sClVRhJ/ | 13:50 |
| noonedeadpunk | that's from 2.17 https://paste.openstack.org/show/bYtWgLL0sg4aXfk3sExA/ | 13:50 |
| amoralej | yep, i'm doing the same teest | 13:50 |
| amoralej | test | 13:50 |
| spotz__ | meeting in 10 | 13:51 |
| *** spotz__ is now known as spotz_ | 13:51 | |
| noonedeadpunk | yeah, add just overrides in 3.0 | 13:51 |
| noonedeadpunk | * `add` | 13:51 |
| noonedeadpunk | Will try to check on ubuntu for 3.1.1 just in case | 13:52 |
| noonedeadpunk | *3.0.1 | 13:54 |
| noonedeadpunk | it's working nicely | 13:54 |
| noonedeadpunk | https://paste.openstack.org/show/b5P2dovbfL1jSH4gPnCC/ | 13:54 |
| noonedeadpunk | So it's indeed just 3.1 that seems broken. And yeah, ubuntu using 3.0 so not affected yet | 13:56 |
| amoralej | noonedeadpunk, yes, i've just reproduced it, let's report a bugzilla | 13:59 |
| rdogerrit | Merged rdo-jobs master: Add option to retry crc_storage creation https://review.rdoproject.org/r/c/rdo-jobs/+/48022 | 13:59 |
| noonedeadpunk | you wanna me reporting it or do it yourself? | 14:00 |
| noonedeadpunk | As I'm not sure it's packaging or ovs issue | 14:01 |
| noonedeadpunk | (sounds like ovs) | 14:01 |
| amoralej | lemme find out what's the best way to report it for centos as the product i know is for rhel | 14:01 |
| amoralej | i'd say it's ovs issue | 14:01 |
| jcapitao[m] | it's meeting time | 14:01 |
| jcapitao[m] | #startmeeting RDO meeting - 2023-03-29 | 14:02 |
| opendevmeet | Meeting started Wed Mar 29 14:02:18 2023 UTC and is due to finish in 60 minutes. The chair is jcapitao[m]. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| opendevmeet | The meeting name has been set to 'rdo_meeting___2023_03_29' | 14:02 |
| jcapitao[m] | #topic roll call | 14:02 |
| jcapitao[m] | https://etherpad.opendev.org/p/RDO-Meeting | 14:02 |
| jcapitao[m] | you can add your topics in this pad | 14:02 |
| spotz_ | o/ | 14:04 |
| amoralej | o/ | 14:04 |
| jcapitao[m] | #chair spotz_ amoralej | 14:04 |
| opendevmeet | Current chairs: amoralej jcapitao[m] spotz_ | 14:04 |
| jcapitao[m] | karolinku: are you around ? | 14:05 |
| amoralej | (i'm in another meeting, so just i'm half here ... ) | 14:05 |
| jcapitao[m] | ask GPT-4 to answer on your behalf :) | 14:06 |
| jcapitao[m] | let's start with first topic | 14:07 |
| jcapitao[m] | #topic Antelope release preparation update | 14:07 |
| jcapitao[m] | #link https://issues.redhat.com/browse/RDO-84 | 14:07 |
| jcapitao[m] | #link https://issues.redhat.com/browse/RDO-84 | 14:07 |
| jcapitao[m] | #info we are waiting for the OpenStack puppet modules to be released | 14:08 |
| jcapitao[m] | #link https://review.opendev.org/c/openstack/releases/+/878731 | 14:08 |
| jcapitao[m] | #link https://review.opendev.org/c/openstack/releases/+/878731 | 14:08 |
| jcapitao[m] | the upstream patch is opened | 14:08 |
| jcapitao[m] | once merged, our bot will submit the patches automatically | 14:08 |
| jcapitao[m] | then we'll be able to promote the builds up to -release tag | 14:09 |
| jcapitao[m] | then the content will be available in https://mirror.stream.centos.org/SIGs/9-stream/cloud/x86_64/ | 14:10 |
| jcapitao[m] | it reminds be that we need to edit https://review.rdoproject.org/etherpad/p/antelope-release-announcement | 14:11 |
| jcapitao[m] | #link https://review.rdoproject.org/etherpad/p/antelope-release-announcement | 14:11 |
| spotz_ | Yeah there's a few things on the announcement, features, addressing the retirement of TripleO and then new and current contributors | 14:15 |
| spotz_ | I went through the release notes for a press release so have some points, not everyone had release notes those | 14:16 |
| jcapitao[m] | ah good! | 14:17 |
| jcapitao[m] | I think that's it for this topic | 14:18 |
| spotz_ | And I just added them to the etherpad so we can edit, delete, or add but have something to start with | 14:18 |
| spotz_ | yep | 14:18 |
| rdogerrit | Merged rdo-website master: Point Cloud SIG meeting link to sigs.c.o/cloud https://review.rdoproject.org/r/c/rdo-website/+/47872 | 14:19 |
| jcapitao[m] | I see them | 14:19 |
| jcapitao[m] | let's move to the second and last topic of the agenda | 14:20 |
| jcapitao[m] | #topic rdopkg RFE - should we output the repo provider ? | 14:20 |
| jcapitao[m] | #link https://issues.redhat.com/browse/RDO-115 | 14:20 |
| jcapitao[m] | I opened this US in order to describe the RFE properly | 14:21 |
| spotz_ | I'm assuming you're leaning to the main solution and not the distrepos one? | 14:22 |
| jcapitao[m] | yeah, but I prefer to get feedback before making the changes | 14:23 |
| spotz_ | I'm assuming more pros then cons of the other method?:) | 14:24 |
| amoralej | what's repo provider ? | 14:24 |
| jcapitao[m] | so when you have a chance, you can leave a comment on this US | 14:25 |
| jcapitao[m] | it's low prio though | 14:25 |
| amoralej | i'll check later, thanks jcapitao[m] for creating the ticket | 14:25 |
| jcapitao[m] | spotz_: yeah actually those two attributes are bound, so it's arbitrary choice | 14:27 |
| rdogerrit | Marios Andreou proposed rdo-jobs master: Adds network_isolation false for data-plane-adoption job https://review.rdoproject.org/r/c/rdo-jobs/+/48023 | 14:27 |
| jcapitao[m] | amoralej: trunk or cloudsig as repo provider (maybe not the right terminology here) | 14:28 |
| spotz_ | Ok I'll let you all doing the work decide then if it's equal vs reminding to choose the better/easier method:) | 14:28 |
| jcapitao[m] | yeah let's give some time :) | 14:29 |
| jcapitao[m] | moving to next topic | 14:29 |
| jcapitao[m] | #topic next chair | 14:29 |
| jcapitao[m] | #action karolinku to chair next week | 14:30 |
| jcapitao[m] | we've switched chairing today | 14:31 |
| jcapitao[m] | #topic open floor | 14:31 |
| spotz_ | I'll be PTO next week! | 14:32 |
| jcapitao[m] | me too ! :) | 14:32 |
| spotz_ | PTG is going on this week and so far all the sessions I've attended have been very productive | 14:32 |
| amoralej | i can take it | 14:32 |
| jcapitao[m] | is there some big news ? | 14:33 |
| amoralej | mmm next week is easter | 14:33 |
| jcapitao[m] | amoralej: karolinku will chair (we switch chairing today) | 14:34 |
| amoralej | i just saw, sorry, i was reading bottom-up | 14:34 |
| amoralej | :( | 14:34 |
| spotz_ | No real big news. Monday worked on the Diversity Survey and CoC revamp, then the TC/Leadership session. Yesterday went to the Nova Operators session and OopenStack-Ansible. Today planning on attending operator sessions | 14:35 |
| spotz_ | amoralej: I do the same thing when catching up:) | 14:35 |
| jcapitao[m] | no worries ! | 14:36 |
| spotz_ | The rest of the week for me will be TC sessions | 14:37 |
| jcapitao[m] | okk | 14:37 |
| jcapitao[m] | also, there is currently an effort to build OpenStack on Fedora | 14:37 |
| jcapitao[m] | https://github.com/LecrisUT/Fedora-openstack | 14:37 |
| jcapitao[m] | based on RDO distgit repos | 14:38 |
| lecris[m] | Help is appreciated on that btw :D | 14:39 |
| jcapitao[m] | yeah that's what I was about to say :) | 14:39 |
| amoralej | lecris[m], i see you used rpm-master branch | 14:39 |
| jcapitao[m] | I'll be on PTO next week, but I'll take a deeper look afterward | 14:40 |
| amoralej | note that branch is ready for dlrn, so there is no version and release | 14:40 |
| amoralej | i guess you want that, so i'd recomend to use antelope-rdo or zed-rdo branches | 14:40 |
| jcapitao[m] | +1 | 14:40 |
| lecris[m] | Indeed, I am going through these and adding the version and release so that it will work in packit workflow | 14:40 |
| lecris[m] | Also modernizing the spec files | 14:41 |
| spotz_ | I'm excited by the effort, let me know what I can do on the communication/community side of things. I'm also good at breaking things and documenting it:) | 14:42 |
| jcapitao[m] | will packit replace the version in the spec file on-the-fly ? | 14:42 |
| lecris[m] | Yes and maybe. If it is included in the upstream source, then it will automatically set to latest tagged version or git describe. In the dist-git method I am not sure, I am still trying to understand the workflow there. | 14:44 |
| lecris[m] | For automation, including .packit.yaml and webhooks would be preferred, but there needs more development to hook it with launchpad, gitea and gerrit. | 14:45 |
| jcapitao[m] | for sure | 14:46 |
| amoralej | our specs in stable branches hast last tagged version on each upstream stable branch too, so that should lead you to same releases | 14:46 |
| jcapitao[m] | yeah could be easier to use latest stable branch for bootstrap | 14:47 |
| jcapitao[m] | I'll close the mtg but we can continue off-mtg | 14:48 |
| lecris[m] | amoralej: Oh, it's replaced in `Version:`? | 14:48 |
| lecris[m] | Although it doesn't make much difference since I still want to go through them one-by-one to modernize to use %pyproject_buildrequires for example | 14:49 |
| jcapitao[m] | yeah in Version and Release to | 14:50 |
| jcapitao[m] | too | 14:50 |
| jcapitao[m] | #endmeeting | 14:50 |
| opendevmeet | Meeting ended Wed Mar 29 14:50:22 2023 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/rdo_meeting___2023_03_29/2023/rdo_meeting___2023_03_29.2023-03-29-14.02.html | 14:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/rdo_meeting___2023_03_29/2023/rdo_meeting___2023_03_29.2023-03-29-14.02.txt | 14:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/rdo_meeting___2023_03_29/2023/rdo_meeting___2023_03_29.2023-03-29-14.02.log.html | 14:50 |
| jcapitao[m] | thank you all for the discussion | 14:50 |
| amoralej | lecris[m], yes, see https://github.com/rdo-packages/openstackclient-distgit/blob/antelope-rdo/python-openstackclient.spec#L19 i.e | 14:51 |
| lecris[m] | Oh, didn't know it was a meeting. I was just bettlejuiced :D | 14:51 |
| amoralej | wrt modernization, we were planning to update the specs to use pyproject macros during bobcat release | 14:51 |
| amoralej | so, good to accept contributions :) | 14:51 |
| lecris[m] | bobcat is next one over? | 14:52 |
| amoralej | yes, it's the one we are just starting | 14:52 |
| rdogerrit | Joel Capitao proposed rdo-website master: Add meetings 03/22 and 03/29 https://review.rdoproject.org/r/c/rdo-website/+/48025 | 14:52 |
| amoralej | noonedeadpunk, https://bugzilla.redhat.com/show_bug.cgi?id=2182767 | 14:58 |
| NeilHanlon | amoralej: question re: libibverbs requirement. was there previously some Requires or Recomends for rdma-core? I am trying to hunt down why centos stream 9 pulls this in but not Rocky for the same instructions (OSA) | 15:03 |
| noonedeadpunk | aha, thanks | 15:03 |
| amoralej | NeilHanlon, likely it has been updated in centos stream after last rocky release ? | 15:07 |
| amoralej | note i built against centos stream | 15:08 |
| lecris[m] | Are the gpg sign crucial? E.g. why don't we use the opendev source for ansible-collections-openstack? | 15:12 |
| jcapitao[m] | it depends whether you want to ensure the data integrity or not | 15:14 |
| amoralej | NeilHanlon, 2.17 required libmlx5.so.1(MLX5_1.21)(64bit) while 3.1 depends on libmlx5.so.1(MLX5_1.24)(64bit) which may have been included in centos stream but not yet in last rhel | 15:15 |
| jcapitao[m] | lecris: ansible-collections-openstack is not under OpenStack governance | 15:15 |
| jcapitao[m] | it's not in https://tarballs.opendev.org/openstack/ | 15:15 |
| lecris[m] | Oh, but otherwise, everything that is should be gpg signed? | 15:16 |
| jcapitao[m] | so the tarball is not signed with the OpenStack key | 15:16 |
| jcapitao[m] | yeah, every package available in https://tarballs.opendev.org/openstack/ are signed by the OpenStack private key | 15:17 |
| NeilHanlon | amoralej: ah, yep. that's it.. Do you have any idea when the rdma-core update is coming to RHEL? | 15:17 |
| noonedeadpunk | amoralej: or alternative question - for how long ovs 2.17 is going to stay in repo? | 15:19 |
| amoralej | noonedeadpunk, we don't remove packages from repos | 15:19 |
| noonedeadpunk | so basically for Rocky we can use old OVS until it's in rhel at least | 15:19 |
| NeilHanlon | or I can go rebuild against rocky 🙃 | 15:20 |
| noonedeadpunk | yeah, or that ofc :) | 15:20 |
| amoralej | yes, that would be an option | 15:20 |
| NeilHanlon | I'd like to avoid that, as I don't particularly want to redo work already being done | 15:21 |
| amoralej | there may be chances to rebuild ovs in centos sigs with last rhel buildroot | 15:22 |
| amoralej | iirc there was that option in sigs | 15:23 |
| amoralej | would you like to join centos nfv sig and take care of that? :) | 15:23 |
| amoralej | it'd be good to build and test those builds in rocky or other rhel-like distro | 15:23 |
| NeilHanlon | Definitely agree! More than happy to assist where I can | 15:24 |
| NeilHanlon | meetings still first wednesday of the month in libera #centos-meeting ? | 15:24 |
| amoralej | NeilHanlon, https://pagure.io/centos-infra/issue/400 | 15:27 |
| amoralej | NeilHanlon, yes, join us there next week | 15:28 |
| NeilHanlon | iirc, it is an 'either or' situation with respect to the build roots. i'm not sure we can build against stream and rhel build roots | 15:28 |
| NeilHanlon | but.. i'll give it some thought. added to my calendar :) | 15:28 |
| amoralej | i'll ask centos infra team in the meanwhile | 15:29 |
| lecris[m] | `openstack-macros` contains stuff like `%{gpgverify}`? Can we replace it with `gnugpg2`? | 15:29 |
| lecris[m] | https://docs.fedoraproject.org/en-US/packaging-guidelines/#_verifying_signatures | 15:29 |
| amoralej | lecris[m], that's comgin from fedora macros iirc | 15:30 |
| NeilHanlon | amoralej - i was just about to ask fabian :) | 15:30 |
| amoralej | me too :) | 15:30 |
| NeilHanlon | i'll let you.. he's already heard from me today about mqtt cert issues :P | 15:31 |
| amoralej | let's ask in #centos-stream | 15:31 |
| NeilHanlon | good call | 15:31 |
| jcapitao[m] | lecris: it's using gpg2 https://git.centos.org/rpms/openstack-macros/blob/c9s-sig-cloud-openstack-xena/f/SOURCES/gpgverify#_97 | 15:33 |
| lecris[m] | Hmm why is this not packaged upstream? | 15:35 |
| jcapitao[m] | packaged on Fedora ? | 15:37 |
| lecris[m] | Both fedora and COS | 15:37 |
| jcapitao[m] | it's packaged on CloudSIG of COS | 15:39 |
| jcapitao[m] | https://cbs.centos.org/koji/buildinfo?buildID=34916 | 15:39 |
| lecris[m] | I mean the %{gpgverify} macro to be included in gnupg2 pacakge on these 2 so you only need BuildRequires: gunpg2 on both | 15:40 |
| jcapitao[m] | ah, I don't know the reason | 15:42 |
| jcapitao[m] | maybe they want to keep real content and macro separated | 15:43 |
| rdogerrit | Merged rdo-website master: Add meetings 03/22 and 03/29 https://review.rdoproject.org/r/c/rdo-website/+/48025 | 15:48 |
| rdogerrit | Marios Andreou proposed rdo-jobs master: Add failed_when to nested make ansible for data plane adoption jobs https://review.rdoproject.org/r/c/rdo-jobs/+/47955 | 15:48 |
| lecris[m] | What's the deal with redhat-openstack/ansible-pacemaker vs community.general.pacemaker_cluster module? | 15:50 |
| lecris[m] | tags and releases are so old there | 15:52 |
| rdogerrit | Sławek Kapłoński proposed openstack/tobiko-distgit rpm-master: Remove validations-lib dependency https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/48026 | 16:05 |
| lecris[m] | https://github.com/rdo-packages/ansible-collections-openstack-distgit/blob/045807193ede2f126cb972b2796d863b90237e8e/ansible-collections-openstack.spec#L2-L4 | 16:16 |
| lecris[m] | Lol, someone forgot the copy paste | 16:16 |
| rdogerrit | Alfredo Moralejo proposed deps/ansible-core c9s-antelope-rdo: Rebuild and rename ansible-core with python 3.9 https://review.rdoproject.org/r/c/deps/ansible-core/+/48019 | 16:17 |
| dalvarez | noonedeadpunk: https://mail.openvswitch.org/pipermail/ovs-dev/2023-March/403255.html this should do it | 16:23 |
| lecris[m] | jcapitao: Does COS use `redhat-rpm-config`? Fedora folks say it's defined there | 16:24 |
| amoralej | sorry, i was lost in conversation, still talking about {gpgverify} ? | 16:25 |
| lecris[m] | About this one, yes. %{gpgverify} is apparently defined there | 16:26 |
| amoralej | i think that comes from fedora iirc, we probably ported to openstack-macros to get it in previous versions of centos | 16:28 |
| amoralej | when the macro was not in centos yet | 16:29 |
| amoralej | we have used openstack-macros for that in the past | 16:29 |
| lecris[m] | They say it's in redhat-rpm-config, so shouldn't that be also in COS? | 16:29 |
| amoralej | it's probably not needed anymore as we are centos9 only | 16:29 |
| amoralej | yes | 16:29 |
| lecris[m] | Cool, a bit more tidier now then | 16:29 |
| amoralej | lecris[m], https://review.opendev.org/c/openstack/rpm-packaging/+/790902 | 16:30 |
| lecris[m] | How about %{autorelease} and equivalents? | 16:30 |
| amoralej | actually we added for centos8 compatibility | 16:31 |
| lecris[m] | coolbeans | 16:31 |
| noonedeadpunk | dalvarez: ah! this is great news | 16:32 |
| noonedeadpunk | really awesome you did track that down | 16:32 |
| dalvarez | \o/ | 16:35 |
| apevec | amoralej dalvarez MAD office connection? ;) | 16:42 |
| amoralej | :) | 16:43 |
| apevec | good job reporting and tracking down that ovs bug amoralej++ dalvarez++ | 16:43 |
| amoralej | and thanks noonedeadpunk for reporting | 16:43 |
| rdogerrit | Alfredo Moralejo proposed deps/ansible-core c9s-antelope-rdo: Rebuild and rename ansible-core with python 3.9 https://review.rdoproject.org/r/c/deps/ansible-core/+/48019 | 16:44 |
| rdogerrit | Merged openstack/tobiko-distgit rpm-master: Remove validations-lib dependency https://review.rdoproject.org/r/c/openstack/tobiko-distgit/+/48026 | 16:45 |
| *** amoralej is now known as amoralej|off | 16:49 | |
| rdogerrit | Alfredo Moralejo proposed deps/ansible-core c9s-antelope-rdo: Rebuild and rename ansible-core with python 3.9 https://review.rdoproject.org/r/c/deps/ansible-core/+/48019 | 19:50 |
| rdogerrit | Alfredo Moralejo proposed deps/ansible-core c9s-antelope-rdo: Rebuild and rename ansible-core with python 3.9 https://review.rdoproject.org/r/c/deps/ansible-core/+/48019 | 19:56 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!