| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Update shared/dedicated cpu calculations https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/927855 | 01:03 |
|---|---|---|
| *** dtantsur_ is now known as dtantsur | 01:48 | |
| opendevreview | Amit Uniyal proposed openstack/whitebox-tempest-plugin master: Add provider.yaml tests https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/921755 | 06:01 |
| opendevreview | Lajos Katona proposed openstack/devstack stable/2023.1: Handle_tags and branches for unmaintained also https://review.opendev.org/c/openstack/devstack/+/927900 | 08:06 |
| opendevreview | Merged openstack/tempest master: Use stable constraint in tox to release Tempest new tag https://review.opendev.org/c/openstack/tempest/+/927831 | 08:20 |
| dtantsur | A tempest question: what could cause "The specified regex doesn't match with anything" on a seemingly innocent change to ironic-tempest-plugin? | 13:36 |
| dtantsur | My immediate guess was an import error but both files I'm modifying can be imported without problems | 13:37 |
| dtantsur | Change: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/927928 | 13:37 |
| *** whoami-rajat_ is now known as whoami-rajat | 14:00 | |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Update shared/dedicated cpu calculations https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/927855 | 14:24 |
| JayF | gmann: or anyone else on QA team, we need urgent assistance if possible | 14:36 |
| TheJulia | Yeah, we're seeing it on fresh changes to master branch it seems | 14:36 |
| JayF | I wonder if until https://review.opendev.org/c/openstack/tempest/+/927832 is landed we're in bad shape | 14:37 |
| JayF | TheJulia: do you even know where/how to start troubleshooting this? I am not seeing anything useful at all in logs | 14:39 |
| JayF | frickler: kopecmartin: Either of you around, perhaps? We need urgent assistance troubleshooting tempest | 14:40 |
| TheJulia | All I can think is just wait out 2024.1 for a few minutes | 14:41 |
| TheJulia | and go from there | 14:41 |
| TheJulia | comparing what might have changed in devstack or tempest logic | 14:42 |
| frickler | JayF: sorry, I need to step away for a bit, but the above patch should land soon. we can look into force-merging it if it keeps failing CI | 14:42 |
| JayF | frickler: We have a CVE right now, being announced in 2 hours, all patches related are stuck in the gate | 14:43 |
| JayF | frickler: after we validated Ironic gate was happy as recent as two days ago :( | 14:43 |
| JayF | frickler: so whatever you can do to unstick, please do all of those whatevers :) | 14:43 |
| TheJulia | https://zuul.opendev.org/t/openstack/stream/8e4588458fa7459b9ddfb98d73508a98?logfile=console.log | 14:43 |
| TheJulia | is same job on stable/2024.1 | 14:43 |
| JayF | so stables are happy; it's just master? That reinforces the hypothesis that tempest 927832 needs to land | 14:44 |
| JayF | oh, it's still in setup. dunno if it's happy yet, I guess | 14:45 |
| JayF | TheJulia: fwiw, Ironic tempest is passing on that 927832 change in tempest; I wonder if we should both put our patches depends-on that so we can at least get them verified? I'm unsure if tempest is loaded from git though | 14:47 |
| JayF | that's a silly question, given I think merging something to git will fix ironic... | 14:48 |
| dtantsur | I'm quite sure tempest is loaded from source | 14:48 |
| TheJulia | hmm, why is tempest task trying to install tempest with 2024.1 constraints on a job for master branch | 14:49 |
| JayF | look at the patch I've been pointing at | 14:49 |
| JayF | they changed it to 2024.1 to cut a release | 14:49 |
| JayF | the patch to change it back to master is not landed | 14:49 |
| JayF | that's why we're broken | 14:49 |
| TheJulia | #*@#*#!(#@$(!@$ | 14:49 |
| JayF | couldn't have said it better myself | 14:49 |
| TheJulia | seriously, don't break master branch | 14:49 |
| JayF | Hopefully frickler comes back soon and can drop-kick that patch into the repo so we can get at least one of these landed before we have to announce | 14:50 |
| TheJulia | WTF | 14:50 |
| TheJulia | WHY is it that way on mater brancj | 14:50 |
| TheJulia | master | 14:51 |
| * TheJulia screams | 14:51 | |
| JayF | I don't know, and I don't care right now except for getting it fixed | 14:51 |
| JayF | https://zuul.opendev.org/t/openstack/status#927832 all voting jobs on that change are passing | 14:51 |
| JayF | but it has to do the double-run through | 14:51 |
| JayF | fungi: honestly, if you could do something to make 927832 land faster, that'd be the most helpful thing; but I don't know if I have any authority to ask you to do things for QA repos | 14:52 |
| kopecmartin | if anyone has force push rights, feel free to force push https://review.opendev.org/c/openstack/tempest/+/927832 .. it was supposed to land right after https://review.opendev.org/c/openstack/tempest/+/927831/ but the CI is not very stable .. | 14:53 |
| fungi | catching up | 14:54 |
| fungi | i'm out of town dealing with a family emergency, so it'll take me a few minutes | 14:56 |
| fungi | trying to wrangle this stuff on a netbook over borrowed internet | 14:56 |
| opendevreview | Merged openstack/tempest master: Switch to the master constraint in tox.ini https://review.opendev.org/c/openstack/tempest/+/927832 | 15:00 |
| fungi | kopecmartin: ^ done | 15:01 |
| kopecmartin | thanks fungi | 15:01 |
| JayF | thank you very much everyone :) | 15:04 |
| JayF | we've been wound up for weeks waiting for today, so a little high strung trying to get things done this morning | 15:04 |
| kopecmartin | i'm sorry for that, we've hit "the perfect" moment for a mistake while cutting a release :/ :D | 15:05 |
| fungi | it happens far more often than you probably realize | 15:17 |
| frickler | kopecmartin: but if using 2024.1 reqs is breaking things for master, the intermediate release will only work for 2024.1 or older, not for 2024.2, right (cf. what ralonsoh wrote in the release channel) | 15:24 |
| frickler | JayF: TheJulia: sorry about the bad timing, I had forgotten about your CVE issue. feel free to ping if you need further stuff to be pushed into gate or beyond | 15:25 |
| JayF | I think we're past the tempest tests being blocking at this point, I'll ping ya if we need something else | 15:25 |
| kopecmartin | frickler: i guess .. does it mean we have to do this dance again in tempest? at this point i only care about the release | 15:29 |
| frickler | kopecmartin: we would need a commit that contains the 2024.2 u-c ref if we want to make a release from that, yes | 15:31 |
| gmann | JayF: dtantsur it is done everytime we release tempest and plugin jobs stop working. this is bcz tox recreate venv and uninstall plugins when there is any change in constraints. | 17:37 |
| gmann | we revert the constraint change immediate right after the constraints change. we approve both together. it is matter of few hrs when revert get merged | 17:38 |
| gmann | and 2024.1 or 2024.2 constraints does not break this is the process we need to do for every release to release tempest with compatible constraints | 17:39 |
| gmann | so all good now. using 2024.1 constraint is right thing for tempest new release and as 927832 is merged all plugins are good too | 17:40 |
| gmann | kopecmartin: maybe we could have avoided this intermediate release as final release anyways cmg in a month or so. That could have reduced this situation | 17:44 |
| gmann | but not sure if anything urgent was required by anyone. | 17:44 |
| gmann | JayF: dtantsur frickler one thing we can try next time is to do these steps during Friday late or weekend with a email notification on ML. I will make a note of it in QA process. | 17:53 |
| gmann | kopecmartin: ^^ | 17:53 |
| JayF | I'm sure there's a good reason, and we just got unlucky. | 18:00 |
| JayF | Might be worth evaluating if VMT process should contain a nod to someone in QA saying "don't break the gate" at embaro time or similar | 18:00 |
| frickler | well actually I knew about the embargo, but didn't connect the dots properly | 18:01 |
| fungi | also, if "don't break the gate" notes worked... | 18:02 |
| frickler | I'm still wondering if it would be worthwhile to teach zuul to only merge such a stack of two changes together | 18:02 |
| fungi | but yeah, we can put qa representatives on the embargo-notice ml explicitly too, we've done that for requirements, release management and stable branch management in the past, also deployment project representatives | 18:03 |
| fungi | frickler: zuul does have functionality for approximated atomic merges/circular dependencies, but it requires explicit configuration | 18:03 |
| fungi | and needs a different kind of workflow (i.e. gerrit's "merge whole topic" feature) | 18:04 |
| frickler | fungi: yes, I know, but does that work for two changes in the same repo? like we have A is parent of B, and I say in A "depende-on: B"? | 18:04 |
| gmann | frickler: +1, yeah that could help if we have merge together things | 18:05 |
| gmann | anyways i added it in QA release process in a bold red color text and hope we will handle it more smoothly in future https://wiki.openstack.org/wiki/QA/releases#Tempest | 18:05 |
| fungi | frickler: i think that's what gerrit's "merge whole topic" feature is for, and then zuul considers all the changes in the same topic as being circularly interdependent | 18:06 |
| gmann | one thing in past I tried to force tox not to recreate venv if constraint change but there is no way to do and it could be wrong for many cases so we went for this solution of 'constraints-update + revert and try to merge both together' | 18:15 |
| opendevreview | Ihar Hrachyshka proposed openstack/devstack master: Show cloned git commit message instead of oneliner with a hash https://review.opendev.org/c/openstack/devstack/+/928082 | 19:47 |
| opendevreview | Sreelakshmi Menon Kovili proposed openstack/whitebox-tempest-plugin master: test cpu state https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/927641 | 22:21 |
| kopecmartin | gmann: writing emails to the ML won't help , the problem was that the revert change wasn't merged quick enough, i'm not gonna do any extra administrative steps just to make a simple release, the only way forward is to merge such changes asap, e.g. force push as it was handled today | 22:27 |
| kopecmartin | but doing these commits during Fridays or weekends is a good idea | 22:32 |
| kopecmartin | also, i don't see any email about so called embargo on the ML | 22:34 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!