| gmann | I am checking with sync mode also in case async tasks are creating issue here | 00:00 |
|---|---|---|
| gmann | and some logging for list all endpoints though endpoints seems fine in this case but will be helpful for other issues https://review.opendev.org/c/openstack/devstack/+/811246 | 00:01 |
| sean-k-mooney[m] | melwitt if i remember correctly its pulled in by cloud-init | 00:15 |
| sean-k-mooney[m] | just uninstall python-simplejson | 00:15 |
| sean-k-mooney[m] | you have had to do that on fedora in the past and python-yaml too form time to time | 00:16 |
| gmann | seeing same error with async disable too | 01:01 |
| gibi | good morning | 06:35 |
| gibi | I see you had a busy day while I was asleep | 06:36 |
| *** ianw is now known as ianw_pto | 07:21 | |
| bauzas | good morning Nova | 07:32 |
| gibi | bauzas: o/ | 07:32 |
| bauzas | yeah I saw this | 07:33 |
| bauzas | I'm glad Ian found the root cause | 07:33 |
| gibi | yepp | 07:35 |
| opendevreview | Merged openstack/nova master: tools: Ignore bot-generated branch creation patches https://review.opendev.org/c/openstack/nova/+/810285 | 09:27 |
| gibi | bauzas: I broke the old qos min bw support in Xena when I extended it with qos min pps support: https://bugs.launchpad.net/nova/+bug/1945310 So this bug is probably a Xena regression, but I will only be certain that it wasn't broken after I backported the repro tests to wallaby to see if the fault is not there | 10:09 |
| bauzas | ah zut | 10:09 |
| gibi | I have the fixs locally so I think we can still fit this in RC2 | 10:11 |
| gibi | but we have a broken gate so maybe not | 10:12 |
| bauzas | yep, I dunno what to say | 10:12 |
| bauzas | gibi: maybe we should accept the RC2 proposal for the moment and see whether the fix could be merged tomorrow for providing a RC3 | 10:13 |
| gibi | bauzas: good point | 10:13 |
| gibi | that is the safest bet I think | 10:13 |
| bauzas | gibi: add a comment for our meeting | 10:13 |
| gibi | so lets talk about this on the meeting today, accept the RC2 if nobody against it and I can push for RC3 | 10:13 |
| bauzas | yep | 10:13 |
| gibi | adding it to the meeting.. | 10:14 |
| gibi | done | 10:15 |
| bauzas | gibi: adding the xena-rc-potential tag to your bug report | 10:17 |
| bauzas | can you triage it ? | 10:18 |
| gibi | ack, there is a slight chance that it was similarly broken before xena too hence my hesitation in that | 10:18 |
| gibi | I will triage it | 10:19 |
| lyarwood | do we not have a bug for the Apache/Placement issue at the moment? | 10:24 |
| gibi | lyarwood: I don't see any | 10:25 |
| gibi | lyarwood: I only see the fix https://review.opendev.org/c/openstack/devstack/+/811303 | 10:25 |
| lyarwood | yeah same was just checking I hadn't missed something | 10:26 |
| gibi | partial fix, as grenade still needs love | 10:26 |
| lyarwood | yeah I can't test that until the devstack fix lands | 10:26 |
| gibi | lyarwood: I guess this is not even a placement bug but a devstack one | 10:26 |
| lyarwood | actually I'll just pull both in | 10:26 |
| gibi | lyarwood: wait, there are two grenade issues | 10:26 |
| lyarwood | oh wait sorry I see what you mean re grenade, we need it fixed on both branches | 10:26 |
| gibi | lyarwood: in the https://review.opendev.org/c/openstack/devstack/+/811303 grenade still fails with the placement issue | 10:26 |
| lyarwood | the devstack issue | 10:26 |
| lyarwood | right | 10:27 |
| lyarwood | yeah so we need to land on stable/xena first | 10:27 |
| lyarwood | then master | 10:27 |
| lyarwood | good old grenade | 10:27 |
| gibi | lyarwood: that was my guess but I wasn't sure | 10:27 |
| gibi | lyarwood: so grenade used the devstack from the stable branch? | 10:27 |
| lyarwood | gibi: yup I believe so | 10:28 |
| gibi | nice | 10:28 |
| elodilles | yes, grenade issues usually have to be fixed first at the oldest branch (where the problem exists) and merge "backward" | 10:30 |
| elodilles | is this such an issue? | 10:30 |
| lyarwood | yeah I think so if it's blocking nova-status checks during the initial install | 10:31 |
| lyarwood | I guess it needs to go back as far as that apache update landed | 10:31 |
| elodilles | :S | 10:32 |
| gibi | and the apache update is in focal | 10:32 |
| gibi | so it effect all our focal grenade jobs? | 10:32 |
| lyarwood | it could, I'm just watching the stable/xena job install stable/wallaby now | 10:33 |
| lyarwood | https://zuul.opendev.org/t/openstack/stream/540dea35d7a84633bb00d706fe02136d?logfile=console.log | 10:33 |
| elodilles | if it is only related to focal then we need to fix it first on victoria afaik | 10:33 |
| elodilles | (victoria is the first branch based on focal) | 10:34 |
| lyarwood | yeah it failed | 10:37 |
| lyarwood | fun times | 10:37 |
| gibi | lyarwood: could you drop a mail to the ML thread about this fun finding? | 10:38 |
| lyarwood | sure | 10:38 |
| * lyarwood finds the thread | 10:38 | |
| gibi | thanks | 10:41 |
| gibi | http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html | 10:42 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811394 | 10:59 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | 11:00 |
| lyarwood | gibi: I'm going to hold off posting to the thread, I think people are likely aware of the stable requirement here | 11:07 |
| lyarwood | gibi: but I'm slightly confused by the way forward, it seems that people want to add the trailing / but that's breaking keystone at least on stable/xena | 11:07 |
| gibi | lyarwood: ohh, I haven't checked the stable/xena result yet | 11:13 |
| *** bhagyashris_ is now known as bhagyashris|rover | 11:16 | |
| gibi | lyarwood: do you think a non trailing / version would pass? | 11:17 |
| lyarwood | gibi: it passed on master for the non-grenade jobs | 11:17 |
| gibi | hm, true | 11:17 |
| gibi | so then we have to go back and reason about the non trailing / case | 11:18 |
| lyarwood | yup I think so | 11:24 |
| opendevreview | Balazs Gibizer proposed openstack/nova master: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811396 | 12:13 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811405 | 12:19 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/xena: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811407 | 12:21 |
| gibi | bauzas: now I can confirm that https://bugs.launchpad.net/nova/+bug/1945310 is not fully a xena regression, at least parts of it is reproducible in wallaby too | 12:28 |
| bauzas | gibi: ack, then remove the xena regression tag | 12:29 |
| bauzas | gibi: can you backport the reproduction change to stable/wallaby so the CI would test it ? | 12:29 |
| gibi | I'm doing it right now | 12:30 |
| bauzas | cool | 12:30 |
| bauzas | so we don't need a RC3 for it | 12:30 |
| gibi | this is how I confirmed that it is a probelm in wallaby too | 12:30 |
| bauzas | but we will merge it for a 25.0.1 release | 12:30 |
| bauzas | huh | 12:30 |
| bauzas | 24.0.1 | 12:30 |
| gibi | :) | 12:30 |
| bauzas | gibi: thanks for working on it, very appreciated | 12:31 |
| gibi | I broke it :D | 12:31 |
| gibi | so hell yeah I have to work on it | 12:31 |
| jamesdenton_alt | Any ideas as to why Nova might return duplicates for hypervisors and services (same UUIDs)? https://paste.opendev.org/show/809646/ | 12:47 |
| *** jamesdenton_alt is now known as jamesdenton | 12:47 | |
| opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Fix NVMe volume attach https://review.opendev.org/c/openstack/nova/+/811413 | 13:16 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Reproduce bug 1945310 https://review.opendev.org/c/openstack/nova/+/811414 | 13:20 |
| gibi | bauzas: so on stable/wallaby we have 2/3 of the bug exists ^^ | 13:21 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | 13:21 |
| bauzas | gibi: over the phone for a meeting but ack | 13:22 |
| gibi | me too | 13:22 |
| opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Fix NVMe volume attach https://review.opendev.org/c/openstack/nova/+/811413 | 13:24 |
| opendevreview | Balazs Gibizer proposed openstack/nova stable/wallaby: Query ports with admin client to get resource_request https://review.opendev.org/c/openstack/nova/+/811416 | 13:31 |
| *** liuyulong__ is now known as liuyulong | 13:57 | |
| zigo | Looks like nova-api on Xena (over Debian Bullseye) doesn't like me... | 13:58 |
| zigo | # openstack server delete 26ee7dd9-75c2-41f5-86f4-c714ed36aab7 | 13:58 |
| zigo | Failed to delete allocations for consumer 26ee7dd9-75c2-41f5-86f4-c714ed36aab7. Error: {"errors": [{"status": 500, "title": "Internal Server Error", "detail": "The server has either erred or is incapable of performing the requested operation.\n\n unsupported callable ", "request_id": "req-34a9215b-9387-43b0-ae24-2d85c7d2b8f5"}]} (HTTP 409) (Request-ID: req-57d65ddf-f845-4fa4-9f13-4382ec19dc72) | 13:58 |
| zigo | When doing so, nova-api just spams its log with some policy deprecation stuff... | 13:58 |
| zigo | How am I supposed to generate the default policy file?!? | 13:59 |
| zigo | This looks like completely broken AGAIN... :( | 13:59 |
| * bauzas needs to disappear for getting his car from the garage | 14:00 | |
| zigo | Also, has the /healthcheck changed behavior?!? | 14:01 |
| zigo | Now it wants me to be authenticated ?!? | 14:02 |
| zigo | Seriously? | 14:02 |
| zigo | How am I supposed to do that with the standard haproxy thingy? | 14:03 |
| bauzas | afaik we haven't changed the paste.ini | 14:03 |
| bauzas | the whole policy thing is different | 14:03 |
| bauzas | but i need to go away or I'll be pedestrian | 14:04 |
| zigo | When I do a: curl -k https://<hostname>:8774/v2.1/healthcheck it's now telling me (in json) that I need to be authenticated ... :/ | 14:05 |
| opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Fix NVMe volume attach https://review.opendev.org/c/openstack/nova/+/811413 | 14:05 |
| zigo | I'm not even sure that can be done with haproxy. | 14:05 |
| zigo | It looks like its probably my bad, checking ... | 14:08 |
| gibi | lyarwood: does this ring a bell to you https://review.opendev.org/c/openstack/devstack/+/811399/1#message-941958791253e3555a648888281a5de6cbf3b1c4 ? | 14:08 |
| lyarwood | gibi: nope never seen that before | 14:12 |
| lyarwood | http://review.opendev.org/ dead for everyone? | 14:12 |
| gibi | works for me | 14:14 |
| kashyap | Yeah, same here | 14:14 |
| dansmith | same | 14:14 |
| kashyap | (As in, works) | 14:14 |
| dansmith | bauzas: is there a tentative ptg schedule yet? | 14:15 |
| gibi | dansmith: we have two cross project discussions scheduled to a certain slot but top of that we haven't assigned topics to slots yet | 14:17 |
| gibi | https://etherpad.opendev.org/p/nova-yoga-ptg | 14:17 |
| gibi | lyarwood: ack, then I recheck :) | 14:18 |
| dansmith | gibi: okay, it's always that arms race where everyone is trying to avoid others' sessions, but nobody has scheduled yet :) | 14:18 |
| gibi | I think it is also part of not having the PTG planning in focus until after Xena is out of the tood | 14:24 |
| gibi | door | 14:24 |
| zigo | When trying to delete a server that went into error, I get this in placement.log: | 14:24 |
| zigo | https://paste.opendev.org/show/809649/ | 14:24 |
| zigo | Any idea why? | 14:24 |
| dansmith | I dunno, I think the "I don't want to schedule X until I know of person Y is available, but person Y's schedule depends on other project Z" problem would exist either way | 14:25 |
| gibi | dansmith: true | 14:26 |
| gibi | zigo: interesting | 14:26 |
| zigo | Maybe I should try downgrading to oslo.policy 3.7.0 to see what's going on? | 14:26 |
| dansmith | that error can come from the json encoder/decoder, maybe there's something wrong with the policy file? | 14:26 |
| zigo | dansmith: What I'm surprised of, is that I'm using a yaml policy file in /etc/placement/policy.d/00_default_policy.yaml | 14:27 |
| dansmith | zigo: ah, good, that was going to be my next question | 14:27 |
| zigo | And there, everything was supposed to be commented out, but it's not: every DEPRECATED stuff is active ... | 14:27 |
| zigo | Same for Nova. | 14:28 |
| zigo | (I had to comment out everything...) | 14:28 |
| zigo | dansmith: So you can rule out a policy.json thingy, as I don't have one ! :) | 14:28 |
| * dansmith nods | 14:28 | |
| zigo | Same result with oslo.policy 3.7.0 | 14:30 |
| zigo | Ok, got it to work ... | 14:33 |
| zigo | Again, same stuff: oslopolicy-sample-generator is generating a default policy yaml file that DOES NOT WORK ! | 14:33 |
| zigo | When commenting everything out, it works... | 14:33 |
| gmann | zigo: I do not think oslopolicy-sample-generator add the deprecated rules | 14:34 |
| zigo | gmann: That's what it did for me ... | 14:34 |
| gmann | zigo: best way is not to have policy file if no rule overridden is needed | 14:34 |
| zigo | gmann: Well, in my package, I'm writing a /etc/placement/policy.d/00_default_policy.yaml with the expectation that this fille will only contain commented out directives ... | 14:35 |
| zigo | It worked this way for the last 2 or 3 releases. | 14:35 |
| zigo | Now that's broken. | 14:35 |
| gmann | zigo: commented out you mean except deprecated rules right? | 14:36 |
| zigo | gmann: Why would there be some special cases for the deprecated rules? | 14:36 |
| gmann | zigo: no, I mean everything was commented out ? because earlier you mentioned deprecated rules were not commented? | 14:37 |
| zigo | gmann: That's what I mean yes, in my packages, now, I have everything commented out but all what's deprecated ... :/ | 14:38 |
| gmann | I feel we should extend these tool oslopolicy-sample-generator and oslopolicy-policy-generator to add deprecated rule too with flag so that it does not break deployment not moved to new policies yet | 14:39 |
| zigo | Anyway, GOT MY FIRST Xena VM UP AND RUNNING AND PINGING !!! \o/ | 14:40 |
| gmann | zigo: can you paste your old (failing) policy file? | 14:40 |
| gibi | \o/ | 14:40 |
| gmann | zigo: perfect | 14:40 |
| bauzas | dansmith: I saw gibi answered you, not for the moment | 14:40 |
| dansmith | ack | 14:40 |
| *** liuyulong_ is now known as liuyulong | 14:46 | |
| gmann | zigo: and that policy file (failing ) one is not generated from oslopolicy-sample-generator tool right? is it old one of wallaby or something? | 14:47 |
| bauzas | dansmith: gibi: that said, I'll try to create an agenda for next week's meeting | 14:48 |
| bauzas | create a *PTG tentative agenda | 14:48 |
| gibi | bauzas: sounds good let me know if you need help | 14:49 |
| bauzas | gibi: will copy/think/paste :p | 14:49 |
| gmann | gibi: bauzas I added one more cross project topic oslo-nova about this ^^ policy file stuff L56 for now | 14:49 |
| bauzas | ack | 14:49 |
| gibi | bauzas: I can definitely help with the first and last step, not sure about the middle :P | 14:49 |
| gmann | this keep coming in every cycle especially when new policy are implemented, I feel we can find some better solution | 14:50 |
| gmann | zigo: FYI, in case you attend ptg, I have added this topic in PTG discussion. we will discuss if oslopolicy-sample-generator tool can be extended in way that it works for both deployment with old tokens + new tokens | 14:52 |
| zigo | gmann: I've put the files over here: http://shade.infomaniak.ch/policy-files/ | 14:53 |
| zigo | I wonder, should I add a sed in all my packages when generating the policy files then? | 14:53 |
| zigo | As in, commenting-out everything that not commented? | 14:53 |
| gmann | zigo: yeah or can you skip generating it (no policy file) ? I think you need to ship one in packaging ? | 14:55 |
| zigo | With something like: sed -i 's/^"/#"/ $(CURDIR)/debian/nova-common/etc/nova/policy.d/00_default_policy.yaml | 14:55 |
| gmann | zigo: ok, let me debug that today/tomorrow, having only deprecated rules in policy file should work | 14:56 |
| zigo | gmann: I do not *NEED* to generate one, but I think it is a service to the Debian users to provide them with a policy file that has all what's possible... | 14:56 |
| zigo | gmann: Well, maybe we should add an option in oslopolicy-sample-generator to comment out the DEPRECATED options ? | 14:57 |
| zigo | oslopolicy-sample-generator --commented-out-deprecated-options --output-file=$(CURDIR)/debian/nova-common/etc/nova/policy.d/00_default_policy.yaml --format yaml --namespace nova | 14:57 |
| gmann | zigo: oh does oslopolicy-sample-generator tool generated that file with depreacted rule not commented? | 14:57 |
| zigo | gmann: Yeah, that's what I'm saying since the begining ! :) | 14:58 |
| zigo | Sorry for not expressing myself correctly ... | 14:58 |
| gmann | humm | 14:58 |
| zigo | My package simply does: | 14:58 |
| zigo | mkdir -p $(CURDIR)/debian/nova-common/etc/nova/policy.d | 14:58 |
| zigo | PYTHONPATH=$(CURDIR)/debian/tmp/usr/lib/python3/dist-packages oslopolicy-sample-generator \ | 14:58 |
| zigo | --output-file=$(CURDIR)/debian/nova-common/etc/nova/policy.d/00_default_policy.yaml \ | 14:58 |
| zigo | --format yaml \ | 14:58 |
| zigo | --namespace nova | 14:58 |
| zigo | Nothing more ... | 14:58 |
| zigo | (note: the PYTHONPATH setting is there to use the installed python3-nova package as if it was in the system) | 14:59 |
| bauzas | lyarwood: seen this ? https://bugs.launchpad.net/nova/+bug/1945323 | 15:13 |
| bauzas | gibi: the reporter proposes ^ for a RC3 | 15:13 |
| bauzas | wondering whether it's a regression | 15:14 |
| bauzas | but this sounds valid as it's due to a recent osbrick release | 15:14 |
| gibi | bauzas: lyarwood: left a -1 on the fix, but I think his comments confirms that this needs to fixed | 15:15 |
| bauzas | ok, so let's do a RC2 for the moment and prepare a RC3 release if so | 15:16 |
| gibi | bauzas: based on the gate status we will go with RC2 :D | 15:17 |
| bauzas | gibi: hmmm, because of the pipeline ? | 15:18 |
| bauzas | gibi: I can only see a 2-hour delay which looks a bit busy but still ok-ish | 15:18 |
| gibi | bauzas: becuase i) of apache placement break needs to be fixed from victoria to master direction then ii) nova-granade-multinode still need to be fixed with trunk port | 15:19 |
| gibi | just to enable landing things on master :) | 15:19 |
| bauzas | hah | 15:19 |
| gibi | an we have 2 days left | 15:20 |
| bauzas | gibi: could we say to not use the latest osbrick release for Xena then ? | 15:22 |
| gibi | that also a patch to the requirements.txt to pin osbrick | 15:23 |
| gibi | which we need to land | 15:23 |
| gibi | we can say in the bug report this as a suggestion | 15:23 |
| bauzas | surely | 15:23 |
| gibi | and packagers can apply it | 15:23 |
| bauzas | I wasn't thinking of our reqs | 15:23 |
| bauzas | but rather documenting it for packagers | 15:23 |
| bauzas | in reno | 15:23 |
| gibi | yeah reno could work as that does not require full CI run | 15:24 |
| bauzas | we have a "bugs" reno section AFAICR | 15:24 |
| gibi | yepp you are right, this is a good idea | 15:24 |
| gibi | to doc as known bug | 15:24 |
| bauzas | let's discuss then it during the meeting | 15:24 |
| bauzas | I'll be off for the next 30 mins, I have some appointment coming at home | 15:25 |
| bauzas | but I'll be there for the start | 15:25 |
| bauzas | gibi: ^ (fyk) | 15:25 |
| gibi | bauzas: ack | 15:25 |
| zigo | Had to do this: https://salsa.debian.org/openstack-team/services/nova/-/commit/7c55fbf4356399f96906162accb3fa80d70f9d24 | 15:26 |
| zigo | :/ | 15:26 |
| zigo | Should that be reported to oslo.policy? | 15:26 |
| bauzas | gibi: grabbing the bug reports for the gate hold | 15:27 |
| bauzas | gibi: do you have them already somewhere ? | 15:27 |
| bauzas | I wanna write them for the meeting agenda | 15:27 |
| gibi | bauzas: we don't have bugreport for devstack about the apache placement break | 15:27 |
| bauzas | I've seen the apache issue in the ML, but I haven't seen a proper bug report yet | 15:27 |
| bauzas | dang | 15:27 |
| gibi | bauzas: also we don't have bugreport for the nova-granade-multinode issue either :/ | 15:28 |
| gibi | (we are lazy asses) | 15:28 |
| bauzas | merveilleux | 15:28 |
| gibi | but at leat we are discussing that in #openstack-neutron now in their CI meeting as we suspect they broke that for us | 15:29 |
| gmann | zigo: yes please, file a bug there and I will debug that | 15:32 |
| lyarwood | bauzas: I wouldn't hold rc for that fix FWIW | 15:41 |
| zigo | gmann: https://bugs.launchpad.net/oslo.policy/+bug/1945336 | 15:41 |
| * bauzas is back | 15:46 | |
| gibi | lyarwood: fyi, during the neutron ci meeting we talked about the nova-grenade-multinode issue and slaweq point at https://review.opendev.org/c/openstack/neutron/+/793141 that might be related | 15:51 |
| lyarwood | kk thanks | 15:52 |
| gibi | it is in xena, and grenade just started using devstack | 15:52 |
| gibi | s/devstac/xena devstack/ | 15:52 |
| ralonsoh | gibi, so far, I've seen three new extensions (between jobs executed 4 days ago and today) | 15:56 |
| ralonsoh | qos-pps | 15:56 |
| ralonsoh | dns-integration-domain-keywords | 15:56 |
| ralonsoh | ecmp_routes | 15:56 |
| bauzas | nova meeting in 3 mins | 15:58 |
| bauzas | *here* | 15:58 |
| gibi | ralonsoh: ack | 16:00 |
| bauzas | #startmeeting nova | 16:00 |
| opendevmeet | Meeting started Tue Sep 28 16:00:32 2021 UTC and is due to finish in 60 minutes. The chair is bauzas. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| opendevmeet | The meeting name has been set to 'nova' | 16:00 |
| bauzas | hola ! | 16:00 |
| bauzas | #link https://wiki.openstack.org/wiki/Meetings/Nova#Agenda_for_next_meeting | 16:01 |
| bauzas | thanks for being here | 16:01 |
| bauzas | who's around ? | 16:01 |
| * stephenfin is lurking | 16:01 | |
| elodilles | o/ | 16:01 |
| gibi | \o | 16:02 |
| bauzas | let's start | 16:02 |
| bauzas | #topic Bugs (stuck/critical) | 16:02 |
| bauzas | No Critical bug | 16:03 |
| bauzas | #link 13 new untriaged bugs (+0 since the last meeting): #link https://bugs.launchpad.net/nova/+bugs?search=Search&field.status=New | 16:03 |
| bauzas | thanks for people who triaged them :) | 16:03 |
| bauzas | one open bug marked with xena-rc-potential tag #link https://bugs.launchpad.net/nova/+bugs?field.tag=xena-rc-potential | 16:03 |
| gibi | bauzas: was it you ?:) | 16:03 |
| bauzas | gibi: shhhhhhhhtttt :p | 16:03 |
| bauzas | so, https://bugs.launchpad.net/nova/+bug/1945323 is proposed against RC* | 16:04 |
| gibi | :x | 16:04 |
| bauzas | tl;dr: last osbrick release is creating problems for NVMe volumes | 16:04 |
| bauzas | the fix is provided https://review.opendev.org/c/openstack/nova/+/811413 but not yet merged | 16:05 |
| bauzas | we'll discuss about RC releases in the next topic, but here the question is, should we try to merge ^ before Thursday as it's the deadline for RC | 16:05 |
| bauzas | ? | 16:05 |
| bauzas | lyarwood had concerns and I have an opinion | 16:06 |
| gibi | I think if the fix is valid then we should aim to land it | 16:06 |
| gibi | but the gate state might not allow that | 16:06 |
| bauzas | given the gate is bad at the moment (again, we will discuss this later), we could try to land it but at least we should provide a releasenotes documentation | 16:06 |
| bauzas | telling about the bug | 16:06 |
| bauzas | so, in case we can't land the fix, operators would still see the documentation and for example packagers wouldn't use the latest osbrick if they want to support NVMe | 16:07 |
| bauzas | thoughts ? | 16:07 |
| bauzas | remember that we have a "bugs" section in nova relnotes | 16:07 |
| bauzas | (this gives a "known bugs" section) | 16:07 |
| gibi | I agree to land a reno, regardless of the fix | 16:08 |
| bauzas | given I proposed, I need to write it | 16:08 |
| bauzas | but someone could write it if they want :) | 16:08 |
| bauzas | e0ne: around ? | 16:09 |
| bauzas | we're discussing about your open bug https://bugs.launchpad.net/nova/+bug/1945323 | 16:09 |
| e0ne | bauzas: hi | 16:09 |
| bauzas | and we're proposing you to write a reno change documenting the known bug | 16:09 |
| bauzas | so in case we can't merge the fix before thursday's deadline for RCs, operators would have a documentation explaining how to workaround | 16:10 |
| e0ne | ok | 16:10 |
| e0ne | fair enough | 16:10 |
| bauzas | e0ne: can you write it ? | 16:10 |
| e0ne | I understand that we found it too late | 16:10 |
| bauzas | I wish you would be better than me explaining the issue and how to workaround for an ops | 16:10 |
| e0ne | bauzas: ok, will do it a bit later today | 16:10 |
| bauzas | e0ne: the problem is not really about the time, this is more about the gate block we have atm | 16:11 |
| lyarwood | sorry back | 16:11 |
| gmann | on gate url issue - I think we need to go with no trailing '/' in both place but we need to push changes on stable branches first to make grenade job work. basically in reverse order. I will try after an hr or so, currently in another meeting. | 16:11 |
| bauzas | lyarwood: you had concerns on this bug, btw. take the mic | 16:11 |
| bauzas | gmann: we'll discuss the gate problems litterally after this one :) | 16:12 |
| gmann | k | 16:12 |
| lyarwood | bauzas: I don't even think it's a valid nova bug at the moment, it doesn't look like os-brick has actually changed it's return value with that fix | 16:12 |
| lyarwood | bauzas: and even if it had that's on os-brick not nova | 16:12 |
| bauzas | do we have a reproduction test ? | 16:12 |
| lyarwood | tbh this just smells like a poorly tested codepath | 16:12 |
| bauzas | e0ne: ^ | 16:12 |
| e0ne | lyarwood: there was an issue with CI. it's fixed now | 16:13 |
| bauzas | e0ne: you mean that CI wasn't using the latest osbrick hence not seeing the issue ? | 16:13 |
| e0ne | mellanox CI should fail with this issue on os-brick patches now | 16:13 |
| bauzas | ack | 16:14 |
| bauzas | fwiw, gibi also has another bugfix he wants to merge that's not a regression, so I guess we will provide a Xena stable release like 2 weeks after GA | 16:14 |
| e0ne | bauzas: when I merged that patch I was confused by mellanox CI Cinder-tgtadm | 16:15 |
| lyarwood | yup either way I don't think this needs to land in rc | 16:15 |
| gibi | bauzas: yupp this is a 1/3 xena regression and 2/3 existing bug in wallaby https://bugs.launchpad.net/nova/+bug/1945310 | 16:15 |
| gibi | bauzas: and I'm fine to include it in 24.0.1 later | 16:15 |
| bauzas | ok, then let's do this | 16:16 |
| e0ne | lyarwood: ok, I totally understand you. thanks for letting me discuss it | 16:16 |
| bauzas | e0ne: ping me when you're done with the reno patch | 16:16 |
| e0ne | bauzas: ok, will do | 16:16 |
| bauzas | e0ne: use the 'bugs' section for it please | 16:16 |
| bauzas | cool | 16:16 |
| bauzas | and for the fix itself, let's review it sooner than later in master | 16:17 |
| bauzas | so we could backport it to stable/xena as quickly as we can | 16:17 |
| bauzas | lyarwood: do you had time to review the fix itself ? | 16:17 |
| lyarwood | I already have several times | 16:18 |
| bauzas | nice | 16:18 |
| bauzas | so we need a second core volunteering for reviewing it :) | 16:18 |
| lyarwood | https://review.opendev.org/c/openstack/os-brick/+/800014/7/os_brick/initiator/connectors/nvmeof.py#199 and FWIW I think this is a regression in os-briclk | 16:18 |
| bauzas | hah | 16:18 |
| bauzas | so, no nova impact ? | 16:18 |
| e0ne | lyarwood: I'll discuss it in cinder meeting tomorrow too | 16:19 |
| bauzas | ok, then let's clarify the solution later | 16:19 |
| bauzas | the reno doc is the priority, the fix itself needs a bit more care I guess | 16:19 |
| bauzas | moving on | 16:19 |
| bauzas | #topic Gate status | 16:19 |
| bauzas | Nova gate bugs #link https://bugs.launchpad.net/nova/+bugs?field.tag=gate-failure | 16:19 |
| bauzas | we triaged a few other ones | 16:19 |
| bauzas | but, | 16:20 |
| bauzas | CI is blocked due to nova-grenade-multinode issue and apache placement issue | 16:20 |
| bauzas | gmann: your turn | 16:20 |
| gmann | frickler: ianw_pto figured it out on apache2 security bugs, you might have seen in email | 16:20 |
| gmann | we have two option 1. no trailing '/' 2. having trailing '/' in both url and target | 16:21 |
| gmann | later one seems causing keystone error so we might need to go with first one | 16:21 |
| bauzas | grabbing the link for the sake of understanding | 16:21 |
| gibi | that is the current fix proposal #link https://review.opendev.org/c/openstack/devstack/+/811399 | 16:21 |
| bauzas | #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html | 16:21 |
| gmann | but we need to do it on stable branches first due to grenade jobs , I will try to push changes in stable in reverse order and see if it all work till master. | 16:22 |
| bauzas | tl:dr: apache config is swallowing the placement endpoint URL | 16:22 |
| gmann | so in summary it may take time, hope today we get fixes in | 16:22 |
| gmann | gibi: yeah and ceph things on victoria is another thing | 16:23 |
| gibi | gmann: the stable/victoria fix failing with ceph issue now https://review.opendev.org/c/openstack/devstack/+/811399 | 16:23 |
| bauzas | gmann: any pointer for the nova-grenade-multinode issue ? | 16:23 |
| gibi | gmann: yepp, it is not relataed but it is blocking | 16:23 |
| gmann | true | 16:23 |
| bauzas | gmann: can't see any bug for either the apache issue or the grenade thing | 16:23 |
| gmann | bauzas: nova-grenade-multinode you mean for neutron-truck extension right? | 16:23 |
| bauzas | this is correct | 16:23 |
| gibi | gmann: right | 16:23 |
| gmann | yeah, I have lyarwood patch in list. let's get this apache things in then I will take a look into that | 16:24 |
| lyarwood | https://review.opendev.org/c/openstack/grenade/+/811117 is the fix for that | 16:24 |
| lyarwood | I can create a bug | 16:24 |
| gibi | yepp we need to fix the apache first anyhow | 16:24 |
| gmann | yeah ^^ | 16:24 |
| gmann | lyarwood: that will be good to track, please file | 16:24 |
| bauzas | #link https://review.opendev.org/c/openstack/grenade/+/811117 | 16:25 |
| bauzas | gmann: if you can do the same for the devstack bug, it would be appreciated | 16:25 |
| gmann | bauzas: sure, we have not logged yet :). will do | 16:25 |
| gmann | lyarwood: is nova-grenade-multinode breaking in stable too? | 16:25 |
| lyarwood | Yeah everything with focal | 16:26 |
| lyarwood | oh wait | 16:26 |
| bauzas | \o/ | 16:26 |
| lyarwood | I'm mixing things | 16:26 |
| gmann | we did extension cap in devstack stable/ but need to check if we have neutron-truck in the list there or no | 16:26 |
| lyarwood | no AFAICT that's just stable/xena to master I think | 16:26 |
| gmann | lyarwood: ok, stable/xena is not capped with extension yet | 16:27 |
| bauzas | we definitely need bug reports to understand the impact :) | 16:27 |
| gibi | I also think wallaby is not affecte by the trunk testing problem in grenade | 16:27 |
| bauzas | stable/xena is the top priority I'd say | 16:27 |
| lyarwood | gmann: would you have a link to the cap on stable/wallaby? | 16:27 |
| gmann | gibi: ok, need to check if that extension is enbaled or not may be we are just skipping it :) but I will check | 16:27 |
| bauzas | we can delay stable releases | 16:28 |
| bauzas | but we can't delay Xena GA | 16:28 |
| gmann | lyarwood: https://github.com/openstack/devstack/blob/stable/wallaby/lib/tempest#L669-L685 | 16:28 |
| gmann | #link https://github.com/openstack/devstack/blob/stable/wallaby/lib/tempest#L669-L685 | 16:28 |
| gibi | but we have to fix stable grenade first to fix master to have GA :D | 16:28 |
| bauzas | yeah, tangled. | 16:28 |
| bauzas | and this is not a Disney. | 16:28 |
| bauzas | anyway, I guess we can move on ? | 16:29 |
| gibi | with this apache update just now Murphy hit us hard | 16:29 |
| bauzas | with this last words from gibi full of hope, I guess we can move on | 16:29 |
| bauzas | :D | 16:29 |
| gmann | I will try to push extensions cap in xena too but anyways we have to get apache fix in first | 16:30 |
| bauzas | Placement periodic job status #link https://zuul.openstack.org/builds?project=openstack%2Fplacement&pipeline=periodic-weekly | 16:30 |
| bauzas | Please look at the gate failures, file a bug, and add an elastic-recheck signature in the opendev/elastic-recheck repo (example: #link https://review.opendev.org/#/c/759967) | 16:30 |
| bauzas | #topic Release Planning | 16:30 |
| bauzas | Release tracking etherpad #link https://etherpad.opendev.org/p/nova-xena-rc-potential | 16:30 |
| bauzas | RC2 proposals are up for approval #link https://review.opendev.org/c/openstack/releases/+/811092 and https://review.opendev.org/c/openstack/releases/+/811094 | 16:31 |
| bauzas | Remember to propose regression bugfixes for a new RC with nova-xena-rc-potential | 16:31 |
| bauzas | Final RC to be proposed before end of Thursday Oct 1st (only if needed) | 16:31 |
| bauzas | (gibi): Potential Xena regression #link https://bugs.launchpad.net/nova/+bug/1945310 | 16:31 |
| gibi | yepp | 16:31 |
| gibi | as I said about | 16:31 |
| bauzas | gibi: the floor is yours | 16:31 |
| gibi | it is 1/3 xena regression 2/3 bug in wallaby | 16:31 |
| gibi | fix is proposed | 16:31 |
| gibi | both on master, xena, wallaby | 16:32 |
| gibi | but I'm OK to delay this until 24.0.1 | 16:32 |
| bauzas | for this 1/3, and given the gate, can we assume we'll deliver the fix only as a stable release patch ? | 16:32 |
| bauzas | ok, sold | 16:32 |
| gibi | yepp | 16:32 |
| bauzas | I'll approve the RC2 patches tonight then | 16:32 |
| lyarwood | https://bugs.launchpad.net/grenade/+bug/1945346 for the neutron-trunk issue | 16:32 |
| gibi | it only affect min bw + live migration + non-admin token | 16:32 |
| bauzas | until someone has strong objections | 16:32 |
| bauzas | we *can* deliver a RC3 tomorrow or thursday if needed | 16:33 |
| gibi | bauzas: I'm OK to land RC2 both for nova and placement now | 16:33 |
| bauzas | keeping in mind we're in a bad shape for it tho | 16:33 |
| bauzas | but, crossed fingers | 16:33 |
| bauzas | gibi: ack, will tell the release folks we're good to go for the current proposals | 16:33 |
| bauzas | #topic Review priorities | 16:33 |
| elodilles | (bauzas: ack :)) | 16:33 |
| bauzas | https://review.opendev.org/q/status:open+(project:openstack/nova+OR+project:openstack/placement)+label:Review-Priority%252B1 | 16:34 |
| bauzas | I've used a bit this rp label | 16:34 |
| bauzas | I'll try to do reviews for the ones I've taken before end of this week | 16:34 |
| *** artom_ is now known as artom | 16:35 | |
| bauzas | if someone btw. has patches to review, please ping me on IRC so I can mark them with the Review-Priority flag | 16:35 |
| bauzas | also, avoid CC'ing nova-core in gerrit, this is useless | 16:35 |
| artom | bauzas, since you asked https://review.opendev.org/c/openstack/nova/+/808474/3 :) | 16:36 |
| bauzas | (until we discuss about what to do with this tag at the PTG) | 16:36 |
| bauzas | artom: your wish come true | 16:37 |
| bauzas | #topic PTG Planning | 16:37 |
| bauzas | every info is in the PTG etherpad #link https://etherpad.opendev.org/p/nova-yoga-ptg | 16:37 |
| artom | Wait, we're in the meeting, are we? | 16:37 |
| artom | Sorry, completely out of it | 16:37 |
| bauzas | If you see a need for a specific cross project section then please let me know | 16:37 |
| bauzas | artom: you're a terrible coworker | 16:38 |
| artom | I'm a terrible human being in general, but that's besides the point ;) | 16:38 |
| bauzas | fwiw, I can start to write up a proposed PTG agenda as a tentative next week | 16:38 |
| bauzas | at least we need to sync up with the oslo team about when to have a cross-project discussion for the policy thing (/cc gmann) | 16:39 |
| bauzas | #topic Stable Branches | 16:40 |
| bauzas | nova's stable/ussuri and stable/train are blocked (due to latest virtualenv uses latest setuptools which removed use_2to3) | 16:40 |
| bauzas | (last but not the least large topic) | 16:40 |
| bauzas | large discussion occurred on the ML #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024987.html | 16:40 |
| bauzas | looks a consensus is going forward with pinning virtualenv on stable branches # link https://review.opendev.org/c/openstack/nova/+/810461 | 16:40 |
| bauzas | tbc, we WON'T pin virtualenv (and setuptools) on master | 16:40 |
| bauzas | but we WILL pin virtualenv (and setuptools accordingly) on the required stable branches | 16:41 |
| bauzas | anyone disagreeing with this plan ? | 16:41 |
| gibi | (as a side note this only pins setuptools if you use tox, if you install the package by other means the this solution doesn't help) | 16:42 |
| bauzas | gibi: you're the mastermind of this plan, you could add things if you wish | 16:42 |
| bauzas | gibi: excellent detail indeed | 16:42 |
| bauzas | the pin is only done on tox.ini | 16:42 |
| gibi | currently we have issue with lower-constraints and tox, so it helps us | 16:42 |
| gibi | but on the ML there was other reports where devstack install failed on stable | 16:42 |
| bauzas | packagers are free to pick any dependency they want, for sure | 16:42 |
| gibi | so in the long run we might need additional fix for devstack too | 16:43 |
| bauzas | gibi: you mean, we should pin virtualenv in stable devstack releases ? | 16:43 |
| bauzas | (which makes sense) | 16:44 |
| gibi | bauzas: potentially yes. I haven't see such failure myself yet | 16:44 |
| gibi | so I'm not rushing with it | 16:44 |
| bauzas | interesting | 16:44 |
| bauzas | any distro impacted so far ? | 16:44 |
| bauzas | (for devstack) | 16:44 |
| gibi | I have no details | 16:44 |
| bauzas | awesome | 16:44 |
| elodilles | as I understood devstack is not hit with this issue in general (but I might be wrong) | 16:44 |
| bauzas | and I guess no bug report | 16:44 |
| bauzas | elodilles: what I guess is that it depends on the related OS | 16:45 |
| elodilles | bauzas: that sounds possible | 16:45 |
| bauzas | then, the one who claims there is a bug should explain which env they use | 16:45 |
| bauzas | so we could reproduce it | 16:46 |
| bauzas | for the moment, let's not try to boil the ocean | 16:46 |
| gibi | bauzas: as I said I saw reports on ML but I did not see it in nova yet, so probably the devstack part is not a nova issue right now | 16:46 |
| bauzas | gibi: ok, so let's move on with the tox change | 16:47 |
| gibi | ack | 16:47 |
| bauzas | but again, we'll probably hit the other gate issues | 16:47 |
| bauzas | anyway | 16:47 |
| bauzas | this release sounds so fun | 16:47 |
| bauzas | moving on | 16:47 |
| bauzas | #topic Stable Branches | 16:47 |
| bauzas | whoops | 16:48 |
| bauzas | placement branches are also impacted | 16:48 |
| bauzas | we waited with the wallaby release until xena RC2 gets released, now we could release from victoria and ussuri together with wallaby as well | 16:48 |
| elodilles | i can prepare the release patches for victoria and ussuri if needed | 16:48 |
| bauzas | elodilles: I guess we'll need some kind of smart stitching between all the concurrent patches so we can unblock things one after all | 16:49 |
| bauzas | elodilles: sure, let's discuss this tomorrow | 16:49 |
| elodilles | bauzas: ack | 16:50 |
| bauzas | other points on stable to discuss ? | 16:50 |
| elodilles | bauzas: actually, i'll be on pto tomorrow, but be back on thursday | 16:50 |
| elodilles | but anyway, nothing else from me :X | 16:50 |
| bauzas | elodilles: ok, then ping me on thursday so we can arrange a quick wrap-up about what to release and what's missing | 16:50 |
| bauzas | #topic Sub/related team Highlights | 16:50 |
| elodilles | bauzas: ++ | 16:50 |
| bauzas | Libvirt (bauzas) | 16:50 |
| bauzas | nothing to tell honestly | 16:51 |
| bauzas | btw. I will change the libvirt chairing to lyarwood for next week | 16:52 |
| bauzas | apologies, i forgot to modify it | 16:52 |
| bauzas | lyarwood: anything worth mentioning about libvirt ? | 16:52 |
| lyarwood | not yet :) | 16:53 |
| bauzas | hold your patchrds | 16:53 |
| bauzas | patches* | 16:53 |
| bauzas | #topic Open discussion | 16:53 |
| ganso | o/ | 16:53 |
| bauzas | (bauzas) : Call for Outreachy mentoring extended until Sep 29 #link http://lists.openstack.org/pipermail/openstack-discuss/2021-September/024992.html | 16:53 |
| bauzas | we have an opportunity for mentoring here | 16:53 |
| bauzas | we haven't brainstormed about the potential projects we could propose to outreachy students | 16:54 |
| bauzas | but I'm sure we have room for them | 16:54 |
| bauzas | if anyone has any idea, shoot | 16:54 |
| bauzas | I can propose to mentor | 16:54 |
| gibi | Dear student please fix the CI before GA, thanks. :P | 16:54 |
| gmann | :) | 16:55 |
| bauzas | lol | 16:55 |
| bauzas | I've noticed a bit late tho | 16:55 |
| bauzas | anyway | 16:56 |
| gibi | do we have ongoing work where easy to join? | 16:56 |
| gibi | mypy maybe ;) | 16:56 |
| bauzas | gibi: I was about to say functional testing but yeah | 16:57 |
| ganso | hi! sorry I wanted to ask this before the meeting ended. 2 weeks ago it was mentioned in this meeting that there would be a new tag release soon for all stable branches. Is there any more specific estimate on when the new tag release will happen? | 16:57 |
| gibi | bauzas: do we still have osc novaclient gaps? | 16:57 |
| bauzas | gibi: I guess so | 16:57 |
| gibi | bauzas: that would be also a good proposal | 16:57 |
| gibi | bauzas: or simply filling the osc gaps for placement, we have missed a list of microversions there | 16:57 |
| bauzas | interesting idea | 16:58 |
| bauzas | one alterative I had in mind was to see how to use some mdev testing framework for validating our GPUs in our CI | 16:58 |
| bauzas | but anyway, ganso has a question and we're 1 min | 16:59 |
| bauzas | ganso: we just discussed we're considering to tag stable releases end of this week if we can | 16:59 |
| ganso | great! | 16:59 |
| ganso | thanks! | 16:59 |
| bauzas | *if we can* as we have some blockers that need help | 17:00 |
| bauzas | but we're overtime | 17:00 |
| bauzas | thanks all | 17:00 |
| bauzas | #endmeeting | 17:00 |
| opendevmeet | Meeting ended Tue Sep 28 17:00:25 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:00 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/nova/2021/nova.2021-09-28-16.00.html | 17:00 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/nova/2021/nova.2021-09-28-16.00.txt | 17:00 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/nova/2021/nova.2021-09-28-16.00.log.html | 17:00 |
| gibi | gmann: do you say that we need something like https://github.com/openstack/devstack/blob/stable/wallaby/lib/tempest#L669-L685 to stable/xena in devstack to fix nova-grenade-multinode? | 17:01 |
| gibi | lyarwood: ^^ | 17:01 |
| opendevreview | Ivan Kolodyazhny proposed openstack/nova master: Add release note which descrube NVMe attach issue https://review.opendev.org/c/openstack/nova/+/811447 | 17:01 |
| gmann | gibi: yes but I saw neutron-truck extension is not enabled there so checking if that is correct or wrong | 17:01 |
| gibi | gmann: this change sets NETWORK_API_EXTENSIONS which we saw missing in the failing grenade jobs | 17:02 |
| gibi | gmann: as far as I see we never enabled trunk in grenade before | 17:02 |
| gibi | gmann: but we can try | 17:02 |
| gmann | gibi: yeah, as you mentioned in review, master is with 'all' which means everything is enabled | 17:02 |
| gibi | gmann: OK, thanks for pointing out the solution. We chased this for a day without success. | 17:04 |
| gmann | gibi: I should have done that while capping the microversion in stable/xena but just late by one day and then apache thing came up | 17:05 |
| gibi | gmann: I guess this capping is logically part of the devstack xena branch cut and the grenade switch from testing wallaby -> master to testing xena -> master | 17:05 |
| gibi | gmann: I see. | 17:06 |
| gibi | gmann: I should remember this | 17:06 |
| gmann | yeah, its just all these should go into sequence. I have added all of these stuff in QA release wiki but let me add it with depends-on things so that we avoid any issue in future | 17:07 |
| gibi | gmann: could you paste the link to that wiki, I want to learn more :) | 17:09 |
| gmann | gibi: this one https://wiki.openstack.org/wiki/QA/releases | 17:09 |
| gibi | thanks | 17:09 |
| gmann | lyarwood: can you remove WIP from https://review.opendev.org/c/openstack/grenade/+/811117 | 17:20 |
| lyarwood | gmann: I can if you don't want to fix this by capping things in tempest? | 17:21 |
| lyarwood | I haven't had a positive test run with this yet because of the other issues as well btw | 17:21 |
| gmann | lyarwood: capping is skipping the tests which is what happening in stable/wallaby so not broken :) | 17:21 |
| gmann | lyarwood: ok, let me enable this extension in stable/wallaby|xena and then we can test it with your fix | 17:22 |
| *** slaweq_ is now known as slaweq | 17:22 | |
| lyarwood | gmann: right but we should be skipping if the extension isn't installed right? | 17:22 |
| lyarwood | gmann: oh sorry tempest is hardcoded to assume these things are available | 17:23 |
| gmann | lyarwood: right but we do enable service in tempest multinode job | 17:23 |
| gmann | so basically grenade also should | 17:23 |
| gmann | yeah | 17:23 |
| lyarwood | right the issue is that grenade isn't derived from that | 17:23 |
| lyarwood | yeah | 17:23 |
| lyarwood | okay | 17:23 |
| lyarwood | let me remove the WIP now | 17:23 |
| admin1 | hi all ..suddenly nova-compute is connecting and disconnecting to rabbitmq .. OSError: Server unexpectedly closed connection .. and then reconnected .... in all 3 rabbitmq containers, netstat shows around 2000 connected sockets in each .. is this something that others have also seen ? | 19:30 |
| admin1 | did i hit a bug at rabbitmq | 19:30 |
| *** amodi_ is now known as amodi | 19:50 | |
| *** elodilles is now known as elodilles_pto | 20:52 | |
| gouthamr | hello seeing a devstack failure in some of the manila jobs; the "nova-status upgrade check" in devstack's post tasks fails the Placement API check... Details: Placement API does not seem to be running. | 21:27 |
| gouthamr | these jobs don't disable placement, and the placement-api logs don't show anything abnormal; example: https://zuul.opendev.org/t/openstack/build/727954069097463495e9e9d774101278/log/controller/logs/screen-placement-api.txt | 21:28 |
| tosky | gouthamr: there is an email iirc | 21:28 |
| tosky | gouthamr: isn't it http://lists.openstack.org/pipermail/openstack-discuss/2021-September/025101.html ? | 21:29 |
| gouthamr | ah! it is! | 21:29 |
| gouthamr | thanks tosky! | 21:29 |
| clarkb | the fixes are all in the gate right now | 21:30 |
| clarkb | we have to roll them forward from victoria to master due to grenade | 21:30 |
| gouthamr | great, thank you clarkb (/me watches https://review.opendev.org/c/openstack/devstack/+/811399 and the train) | 21:32 |
| gmann | gouthamr: as you are here, nova ceph jobs also broken which need fix in devstack-plugin-ceph. https://review.opendev.org/q/I7061f8d1491ff957452c9c777e40186a4e9c324e | 22:22 |
| gmann | I am still testing it on victoria and wallaby | 22:23 |
| gouthamr | ack, saw your change gmann | 22:23 |
| gmann | gouthamr: ok, I might need your help to merge it after testing | 22:23 |
| gouthamr | thanks for the patch! yep :) i'm on it | 22:23 |
| gmann | thanks | 22:24 |
| opendevreview | Ghanshyam proposed openstack/nova stable/wallaby: DNM: Testing nova-grenade-multinode with neutron-trunk https://review.opendev.org/c/openstack/nova/+/811513 | 23:20 |
| opendevreview | Ghanshyam proposed openstack/nova stable/xena: DNM: Testing nova-grenade-multinode with neutron-trunk https://review.opendev.org/c/openstack/nova/+/811491 | 23:22 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!