| *** jpena|off is now known as jpena | 06:54 | |
| *** rpittau|afk is now known as rpittau | 07:30 | |
| opendevreview | Rajat Dhasmana proposed openstack/grenade master: Unblock Gate: create encryption type with right params https://review.opendev.org/c/openstack/grenade/+/803317 | 07:41 |
|---|---|---|
| opendevreview | Rajat Dhasmana proposed openstack/grenade master: Unblock Gate: create encryption type with right params https://review.opendev.org/c/openstack/grenade/+/803317 | 08:01 |
| lyarwood | ^ can we get core grenade reviews on the above to unblock various gates? | 10:57 |
| *** dviroel|out is now known as dviroel | 11:18 | |
| *** jpena is now known as jpena|lunch | 11:20 | |
| *** jpena|lunch is now known as jpena | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:30 | |
| *** jpena|lunch is now known as jpena | 12:33 | |
| ykarel | review please https://review.opendev.org/c/openstack/devstack/+/803039 | 13:50 |
| gmann | office hour time | 14:00 |
| gmann | #startmeeting qa | 14:00 |
| opendevmeet | Meeting started Tue Aug 3 14:00:37 2021 UTC and is due to finish in 60 minutes. The chair is gmann. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:00 |
| opendevmeet | The meeting name has been set to 'qa' | 14:00 |
| gmann | who all are here today? | 14:00 |
| yoctozepto | \o/ | 14:01 |
| croelandt | o/ | 14:01 |
| opendevreview | Elod Illes proposed openstack/devstack-gate master: Remove neutron-lbaas from PROJECTS list https://review.opendev.org/c/openstack/devstack-gate/+/803346 | 14:02 |
| gmann | let's start | 14:03 |
| gmann | #link https://wiki.openstack.org/wiki/Meetings/QATeamMeeting#Agenda_for_next_Office_hours | 14:03 |
| gmann | today agenda ^^ | 14:03 |
| gmann | #topic Announcement and Action Item (Optional) | 14:04 |
| gmann | no announcement or AI from previous meeting | 14:04 |
| gmann | #topic Xena Priority Items progress | 14:04 |
| gmann | #link https://etherpad.opendev.org/p/qa-xena-priority | 14:04 |
| gmann | I have patch up for requirement to move tempest from u-c to blacklist file so that tempest master can be used for plugins job | 14:05 |
| gmann | #link https://review.opendev.org/c/openstack/requirements/+/787621 | 14:05 |
| jparoly | o/ | 14:05 |
| gmann | no other updates on rest of remaining Xena items | 14:06 |
| gmann | #topic OpenStack Events Updates and Planning | 14:06 |
| gmann | #link https://etherpad.opendev.org/p/qa-yoga-ptg | 14:06 |
| gmann | for PTG, please start adding the topics you would like to discuss | 14:06 |
| gmann | #topic Gate Status Checks | 14:08 |
| gmann | we have two failure to discuss today | 14:08 |
| gmann | one is grenade job failure | 14:08 |
| gmann | #link https://review.opendev.org/c/openstack/grenade/+/803317 | 14:08 |
| gmann | frickler raising the question of backward incompatible change. | 14:09 |
| abhishekk | this is blocking glance/cinder/nova and probably others as well | 14:09 |
| croelandt | yeah, this is blocking our RBAC work :/ | 14:09 |
| gmann | If i understand the original bug clearly, cinder change was done as bug fix instead of microversion | 14:09 |
| gmann | #link https://bugs.launchpad.net/cinder/+bug/1926630 | 14:10 |
| gmann | so the voluem create request fails later so new api checks are just an early failure? | 14:10 |
| whoami-rajat__ | grenade doesn't check the encryption flow with barbican and also not with the failing drivers so it will never see that failure but it is still wrong from cinder standpoint | 14:10 |
| whoami-rajat__ | cinder is just moving the async failure to a sync failure by moving the failing code from c-vol to c-api service | 14:11 |
| gmann | whoami-rajat__: you mean the volume create request can pass ? | 14:11 |
| whoami-rajat__ | for LVM, not for rbd nfs and probably other drivers | 14:11 |
| gmann | so back to frickler question, how they were passing before cinder change? | 14:11 |
| gmann | whoami-rajat__: yeah that is difference | 14:11 |
| gmann | I remember we did that change in tempest test too for that reason | 14:12 |
| gmann | I am ok for the change as it is changing 4/5xx->400 only | 14:12 |
| whoami-rajat__ | also how grenade is trying to create an encrypted volume type is not correct | 14:12 |
| whoami-rajat__ | so it needs to be corrected anyway | 14:13 |
| gmann | whoami-rajat__: anything else wrong other than passing cipher ? | 14:13 |
| whoami-rajat__ | gmann, nothing as i remember, i can go through it again but for now my patch passes glance gate so it seems good | 14:14 |
| opendevreview | yatin proposed openstack/devstack master: [DNM] Test c8-stream / fedora-latest with fixes https://review.opendev.org/c/openstack/devstack/+/803144 | 14:14 |
| gmann | whoami-rajat__: ack thanks | 14:14 |
| gmann | frickler: yoctozepto any comment on this how to proceed and unblock gate? | 14:14 |
| opendevreview | yatin proposed openstack/devstack master: [DNM] Test c8-stream with reduced tempest concurrency https://review.opendev.org/c/openstack/devstack/+/803144 | 14:16 |
| yoctozepto | still not understanding why it passed before | 14:16 |
| yoctozepto | no testing after request? | 14:16 |
| gmann | for lvm which is default in grenade does pass but not other backend like barbican | 14:16 |
| gmann | yoctozepto: in grenade yes, it just create it | 14:17 |
| yoctozepto | ok, makes sense | 14:17 |
| * yoctozepto approving | 14:17 | |
| gmann | thanks, seems like frickler is not online now so let's unblock the gate and if any strong objection from him then we can continue discussion | 14:18 |
| gmann | moving to 2nd gate issue | 14:18 |
| gmann | centos8 support and fix | 14:18 |
| gmann | #link https://review.opendev.org/q/I508eceb00d7501ffcfac73d7bc2272badb241494 | 14:18 |
| * yoctozepto approving rdo fix | 14:19 | |
| gmann | we have centos8 job fix and it work for victoria, you can see job green in victoria backport | 14:19 |
| gmann | yoctozepto: tricky thing here is how to fix centos8 in wallaby | 14:19 |
| yoctozepto | you mean stream-only? | 14:20 |
| gmann | yeah | 14:20 |
| gmann | comment from yatin: "For wallaby need to use c8-strem job devstack-platform-centos-8-stream as RDO wallaby released only for c8-stream so may not work for c8" | 14:20 |
| gmann | as there is no wallaby release for centos8 so question is 1. how to fix it or 2. should we change centos8 to centos8-stream in wallaby ? | 14:21 |
| yoctozepto | I would as it makes more sense support-wise | 14:22 |
| gmann | and wallaby testing runtime defined in TC is centos8 not centos8-stream so 2nd one is more tricky | 14:22 |
| tosky | just to clarify: when I've advocated for exceptions on the list thread, I was suggesting to move all centos linux 8 jobs to centos stream 8 whenever an openstack release is still alive after centos linux 8 EOL | 14:22 |
| yoctozepto | yeah | 14:22 |
| gmann | IMO, we should not change the stable runtime at this stage | 14:22 |
| ykarel | centos8-stream is replacement for centos8, so would need to switch atleast when c8 goes EOL at year end | 14:23 |
| ykarel | and would be better even before | 14:23 |
| gmann | ykarel: yes we did in Xena | 14:23 |
| gmann | but we should do in all stable branch or not that is bug question or work | 14:23 |
| ykarel | in puppet and tripleo have already switched to c8-stream | 14:23 |
| gmann | *big | 14:23 |
| gmann | ykarel: all stable also? | 14:23 |
| ykarel | gmann, for other stable c8 should work from RDO perspective as we support on both c8 and c8-stream | 14:24 |
| ykarel | but wallaby+ only c8-stream | 14:24 |
| ykarel | and this would change at year end when c8 goes eol | 14:24 |
| gmann | yeah so wallaby also changed right which is in questrion | 14:24 |
| tosky | ykarel: what will happen to victoria after c8 EOL? | 14:24 |
| gmann | tosky: it still work with old c8 | 14:25 |
| gmann | as worked in devstack patch | 14:25 |
| ykarel | tosky, i think repos will stale for C8 post EOL, i think it's currently being discussed | 14:25 |
| ykarel | so not have full context for that | 14:25 |
| tosky | gmann: right now, but December is close :) | 14:25 |
| ykarel | so atleast there will be no update post December | 14:25 |
| ykarel | and repos may exist for some time | 14:25 |
| gmann | so just curious, how c8 community migrate the customer using c8 for openstack victoria or wallaby? | 14:26 |
| gmann | ykarel: tosky that is risky as you are going to delete the old released/used repo ? | 14:26 |
| tosky | that's what I'm trying to understand | 14:26 |
| ykarel | c8 users will migrated to either c8-stream or rhel or any other rebuild of rhel | 14:26 |
| gmann | ykarel: that seems a contract break if anyone has their production running on c8 with old openstack where it used to work and now might stop ? | 14:27 |
| yoctozepto | yeah, I don't like the way it sounds | 14:28 |
| gmann | I do not know usage contract of c8 but still | 14:28 |
| yoctozepto | I am open to supporting Vicky on centos 8 stream though | 14:28 |
| gmann | yoctozepto: yeah, it seems very confusing for me and how migration is planned for old usage | 14:28 |
| ykarel | gmann, it should work with both c8 and c8-stream for supported releases | 14:28 |
| ykarel | there were some announcement for it | 14:28 |
| ykarel | need to fetch link | 14:28 |
| gmann | ykarel: but you are saying at some time when repo are deleted then it might not work like currently for stable/wallaby | 14:29 |
| tosky | but technically centos 8 stream is not really different from the base one - just gets the updates first instead of later | 14:29 |
| ykarel | gmann, for stable/wallaby repos were never published for c8 | 14:29 |
| ykarel | it was c8-stream from beginning | 14:29 |
| yoctozepto | gmann: as tosky and ykarel say, it mostly works by replacing the repos and doing an update | 14:29 |
| gmann | ykarel: ohk, then is it miss in our testing runtime? | 14:30 |
| ykarel | http://mirror.centos.org/centos/8-stream/cloud/x86_64/ | 14:30 |
| ykarel | vs http://mirror.centos.org/centos/8/cloud/x86_64/ | 14:30 |
| yoctozepto | gmann: it was announced late | 14:30 |
| gmann | ohk | 14:30 |
| gmann | how about this plan ?: 1. discuss in TC about updating the wallaby testing runtime to centos8-stream 2. accordingly QA proceed on changes. | 14:31 |
| yoctozepto | I agree | 14:33 |
| gmann | this is for stable/wallaby only but for victoria it is right fix and make c8 job green #link https://review.opendev.org/c/openstack/devstack/+/803072 | 14:33 |
| gmann | so I am ok to merge the victoria fix | 14:33 |
| gmann | and wallaby one we can continue discussion as per above plan | 14:33 |
| yoctozepto | gmann: as need be, I am open to doing the same to Vicky as c8 goes eol before Vicky does | 14:33 |
| gmann | yoctozepto: I think we should not change victoria testing as it is changing the history and support. wallaby is fine as there is no way and centos8-stream announced in wallaby time but little later | 14:34 |
| ykarel | gmann, that fix can be merged in wallaby too | 14:35 |
| yoctozepto | gmann: well, technically it's the same distro, except for the delivery model | 14:35 |
| yoctozepto | I'm just saying I'm fine with either going on stream | 14:35 |
| yoctozepto | it's a weird situation but it's not that hard for us to adapt now that we support stream in CI | 14:35 |
| gmann | ykarel: that does not fix anything in wallaby, there is no c8-stream job there and c8 still failing. I would like to see some job running with that fix | 14:36 |
| gmann | yoctozepto: humm, not sure still. anyways let's discuss in TC how much backport we can do for c8->c8-stream | 14:36 |
| ykarel | gmann, actually the fix will be needed for any c8 release | 14:36 |
| gmann | ykarel: confused. for wallaby c8 release? | 14:37 |
| yoctozepto | gmann: yeah, I would prefer we did not have to at all but life ain't easy this time of the century | 14:37 |
| gmann | sure. we can proceed as what TC decide on this. as QA we need to comply with defined runtime. | 14:38 |
| ykarel | gmann, so the patch is specific to how rpm name is computed for c8, it's doing it's work in computing the correct name for wallaby and installing the rpm | 14:38 |
| ykarel | but since the job there is c8, it's looking for wrong repo and fails | 14:39 |
| ykarel | but okk if it's squashed with c8-stream job addition | 14:39 |
| gmann | ykarel: yeah, let's wait for Thursday TC meet and then we can proceed on this. | 14:39 |
| ykarel | ok Thanks | 14:40 |
| gmann | ykarel: you would like to merge victoria one now? I am +2 on that and to merge before wallaby | 14:40 |
| ykarel | gmann, fine by me, it's unblocking some cinder jobs | 14:40 |
| ykarel | so should be ok to go with it | 14:41 |
| gmann | yeah please remove your -W | 14:41 |
| gmann | yoctozepto: what you say? https://review.opendev.org/c/openstack/devstack/+/803072 | 14:41 |
| ykarel | removing | 14:41 |
| gmann | though we are skipping wallaby backport and merging vicrotia first but I think it is fine in this siutation | 14:41 |
| yoctozepto | gmann: cherrypick id perhaps? | 14:41 |
| gmann | yoctozepto: ah yeah that can be updated. +1 | 14:41 |
| yoctozepto | I am ok with the skip | 14:41 |
| gmann | ykarel: ^^ | 14:41 |
| gmann | k | 14:42 |
| yoctozepto | just add the cherrypick metadata | 14:42 |
| ykarel | sure will update | 14:42 |
| gmann | thanks | 14:42 |
| yoctozepto | ok | 14:42 |
| gmann | any other gate blocker? | 14:42 |
| gmann | #topic Periodic jobs Status Checks | 14:42 |
| gmann | #link https://zuul.openstack.org/builds?job_name=tempest-full-victoria-py3&job_name=tempest-full-ussuri-py3&job_name=tempest-full-train-py3&pipeline=periodic-stable | 14:42 |
| gmann | #link https://zuul.openstack.org/builds?project=openstack%2Ftempest&project=openstack%2Fdevstack&pipeline=periodic | 14:43 |
| gmann | all green here | 14:43 |
| gmann | #topic Sub Teams highlights | 14:43 |
| gmann | any updates, highlights we need to discuss for today ? | 14:43 |
| gmann | from subteam | 14:43 |
| opendevreview | yatin proposed openstack/devstack stable/victoria: Fix Usage of rdo-release rpm https://review.opendev.org/c/openstack/devstack/+/803072 | 14:44 |
| gmann | #topic Gate Blocker Fix / Urgent Change | 14:44 |
| gmann | #link https://review.opendev.org/q/label:Review-Priority%253D%252B2+status:open+(project:openstack/tempest+OR+project:openstack/patrole+OR+project:openstack/devstack+OR+project:openstack/grenade+OR+project:openstack/hacking) | 14:44 |
| gmann | nothing here | 14:44 |
| gmann | I think we can merge this to about topic of gate status | 14:44 |
| gmann | as both are relevant and overlap | 14:44 |
| gmann | #action gmann merge the two gate check topic into one | 14:45 |
| gmann | #topic Open Discussion | 14:45 |
| gmann | anything else to discuss by anyone which is not in agenda ? | 14:45 |
| rosmaita | just want to say thanks for the work on getting centos-8 devstack build working in victoria | 14:46 |
| eliadcohen | +1 | 14:46 |
| gmann | rosmaita: +1, thanks to ykarel | 14:46 |
| rosmaita | ykarel: ++ | 14:46 |
| gmann | moving next | 14:47 |
| gmann | #topic Bug Triage | 14:47 |
| ykarel | also in noticed tempest tests random failures in master job, so trying out tempest concurrency reduce https://review.opendev.org/c/openstack/devstack/+/803144 | 14:47 |
| gmann | ykarel: ok, can you file a bug and how frequent it is? and which test | 14:48 |
| ykarel | gmann, sure will file | 14:48 |
| gmann | thanks | 14:49 |
| gmann | #link https://etherpad.opendev.org/p/qa-bug-triage-xena | 14:49 |
| gmann | bug report ^^ | 14:49 |
| gmann | no critical bug or other that what we already discussed | 14:49 |
| gmann | that is all from me today. any other bug we need to discuss? | 14:49 |
| yoctozepto | not from me | 14:49 |
| gmann | k | 14:50 |
| gmann | thanks all for joining. | 14:50 |
| gmann | #endmeeting | 14:50 |
| opendevmeet | Meeting ended Tue Aug 3 14:50:22 2021 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 14:50 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/qa/2021/qa.2021-08-03-14.00.html | 14:50 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/qa/2021/qa.2021-08-03-14.00.txt | 14:50 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/qa/2021/qa.2021-08-03-14.00.log.html | 14:50 |
| rosmaita | gmann: i was just reading through the scrollback | 14:50 |
| rosmaita | do you want to see a job run that uses https://review.opendev.org/c/openstack/devstack/+/803071 ? | 14:50 |
| *** ykarel is now known as ykarel|away | 14:52 | |
| gmann | rosmaita: we have devstack-platform-centos-8 in wallaby and if you see it is failing as this fix does not work for centos8 as no wallaby repo for centos8 | 14:52 |
| rosmaita | right, so i have a cinderclient job that uses centos-8-stream for a devstack-based job | 14:53 |
| gmann | rosmaita: plan is to first check if we can update the wallaby testing runtime from cewntos8 to centos8-stream and then we can change the job t use centos8-stream in this fix | 14:53 |
| gmann | https://governance.openstack.org/tc/reference/runtimes/wallaby.html | 14:53 |
| gmann | ^^ this still define centos8 and from Xena we started using centos8-stream | 14:54 |
| gmann | rosmaita: yeah, i know it work on centos8-stream as we see in master fix but which centos8 or centos8-stream to test in wllaby if in question | 14:54 |
| gmann | *is in | 14:54 |
| rosmaita | gotcha | 14:54 |
| rosmaita | ok, i will wait | 14:54 |
| gmann | in TC we will discuss about testing runtime doc update if we can do and then we can proceed | 14:55 |
| rosmaita | yeah, thanks for being willing to approve the victoria patch out-of-order | 14:55 |
| rosmaita | that will get us unblocked for cinderclient gate | 14:55 |
| gmann | +1 | 14:57 |
| *** jpena is now known as jpena|off | 15:06 | |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Add vgpu resize tests https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/790304 | 16:23 |
| opendevreview | yatin proposed openstack/devstack master: [DNM] Test c8-stream with reduced tempest concurrency https://review.opendev.org/c/openstack/devstack/+/803144 | 16:46 |
| opendevreview | Merged openstack/devstack master: Updated from generate-devstack-plugins-list https://review.opendev.org/c/openstack/devstack/+/803100 | 17:20 |
| *** dviroel is now known as dviroel|brb | 17:34 | |
| opendevreview | Huy Tran proposed openstack/tempest master: Remove container name from the list when container is deleted https://review.opendev.org/c/openstack/tempest/+/802388 | 18:05 |
| opendevreview | Merged openstack/grenade master: Unblock Gate: create encryption type with right params https://review.opendev.org/c/openstack/grenade/+/803317 | 18:38 |
| *** dviroel|brb is now known as dviroel | 18:42 | |
| clarkb | fwiw I think frickler is right with the -1 in ^ | 18:43 |
| clarkb | grenade exists to catch those problems and we should be careful overriding them | 18:43 |
| bluesonic | gmann and tempest cores: any chance I can get some reviews and possibly get approval for https://review.opendev.org/c/openstack/tempest/+/802388 | 19:07 |
| gmann | bluesonic: ack, will check today | 19:08 |
| bluesonic | gmann: thanks! really appreciate it. | 19:09 |
| gmann | clarkb: grenade does not test that. it only create and see if create is successful or not. but created volume type is not usable - https://bugs.launchpad.net/cinder/+bug/1926630 | 19:09 |
| gmann | what i understand from cinder change is - it is adding the api validation for early fail instead of current late fail so it is not actually changing any success case behavior | 19:10 |
| gmann | rosmaita: ^^ correct me if I am wrong | 19:10 |
| clarkb | gmann: ok usually we make things span a release so that grenade doesn't have to change on the old side only on the new side | 19:12 |
| clarkb | if you have to chagne something that will break gernade you deprecate it, then n-2 upgrade to n-1 works. Then in n-1 you switch grenade to doing the new thing. THen grenade can upgrade n-1 to n | 19:13 |
| clarkb | having failures like this indicates that this was not done and potentially will result in user upgrade problems | 19:13 |
| gmann | clarkb: this is not case of change, it is fixing later fail to early fail. grenade does not test created volume type is actually created or usable or not. | 19:14 |
| gmann | actually we did not test the broken case in grenade. | 19:14 |
| gmann | for general upgrade change yes we do follow the deprecation process or at least change API via microversion and old one keep working with old version | 19:15 |
| gmann | if greande check 1. create volume type 2. use that 3. use created volume with that volume type then we could see if we actually broke upgrade or not. and these check could have catch the actual bug cinder fixed now | 19:16 |
| gmann | just checking if volume type creation request is accepted or not is not complete upgrade testing | 19:18 |
| clarkb | its not but it does indicate a chagne in behavior htat people may trip over and we have processes to avoid that | 19:21 |
| clarkb | but I guess in this case you can't really deprecate the old behavior as it isn't something the user can opt itno | 19:22 |
| gmann | yeah that is why we can to change it as existing way of creating volume type is not any user has opted it as it fail later | 19:23 |
| gmann | so basically it is not backward incompatible behavior improving it to fast fail with api validation | 19:24 |
| opendevreview | Merged openstack/devstack-gate master: Remove neutron-lbaas from PROJECTS list https://review.opendev.org/c/openstack/devstack-gate/+/803346 | 19:30 |
| opendevreview | Merged openstack/devstack-gate master: Retire django-openstack-auth https://review.opendev.org/c/openstack/devstack-gate/+/800693 | 19:30 |
| rosmaita | clarkb: i think gmann has summarized the situation well | 19:52 |
| opendevreview | Ghanshyam proposed openstack/grenade master: WIP: testing https://review.opendev.org/c/openstack/grenade/+/803397 | 20:52 |
| *** dviroel is now known as dviroel|out | 20:54 | |
| gmann | bluesonic: thanks for explaining it in bug, now it is clear to me. | 21:05 |
| gmann | bluesonic: 1 comment to clear the list in class method delete_containers() https://review.opendev.org/c/openstack/tempest/+/802388/5/tempest/api/object_storage/base.py | 21:05 |
| bluesonic | gmann: thanks for the comments and suggestion. However, deleting in method delete_containers after L142 will not solve the race condition, and the change becomes unnecessary as the list is being re-initialized anyway. Can I make a copy of the list before L142, clear the list, and pass it in delete_containers() on L142? I can provide logs where the second test/method tearDown delete the container of first test/method. Hence, putting | 21:20 |
| bluesonic | the delete container name before all the condenses. | 21:20 |
| bluesonic | that is, pass the copy of the list to delete_containers() on L142. | 21:23 |
| opendevreview | Ghanshyam proposed openstack/grenade master: WIP: testing https://review.opendev.org/c/openstack/grenade/+/803397 | 22:54 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!