| opendevreview | Ghanshyam Mann proposed openstack/tempest master: Optimize the resource creation and status waiting in scenario tests https://review.opendev.org/c/openstack/tempest/+/889207 | 01:22 |
|---|---|---|
| opendevreview | Maxim Sava proposed openstack/tempest master: Add image task client and image tests task APIs. https://review.opendev.org/c/openstack/tempest/+/888755 | 07:26 |
| opendevreview | Merged openstack/tempest master: Split up ServerActionsTestJSON for better mixing https://review.opendev.org/c/openstack/tempest/+/889109 | 10:56 |
| dansmith | kopecmartin: if you're willing, there are several more patches in that series which reduce our wall clock time by making resource allocation more overlap-friendly: https://review.opendev.org/q/topic:split-sat | 13:51 |
| dansmith | kopecmartin: analysis here: https://review.opendev.org/c/openstack/tempest/+/889200 | 16:18 |
| *** jgwentworth is now known as melwitt | 16:18 | |
| dansmith | kopecmartin: I really want to get the next patch in the series landed, so please let me know ASAP if I should reorder these so we can get that underway | 16:19 |
| kopecmartin | dansmith: thanks, i'm ok with the changes, it makes total sense to wait for multiple states at the same time, i just noticed that the time of the test wasn't shorter and i wasn't sure if the shorter exec time was the primary goal ... i see it wasn't, we can merge it as is because it is a good idea | 16:22 |
| dansmith | kopecmartin: exec time *is* the primary goal, it just varies a *ton* by the speed of the worker | 16:22 |
| dansmith | if you compare other volume-based tests in the two jobs you see they're all way slower in the one run | 16:23 |
| dansmith | waiting for two things at once will be more efficient in all cases, you just can't compare absolute numbers to each other since workers vary so much in perofrmance | 16:23 |
| dansmith | in the latest recheck, that modified test is 4% of overall execution time, vs. 8% in the unmodified case you quoted | 16:24 |
| dansmith | even though the unmodified one was on a very fast worker and it took less overall time to run the test as a result, it was still slower in adjusted time | 16:25 |
| kopecmartin | you're right | 16:26 |
| kopecmartin | i oversimplified it .. all makes sense now | 16:26 |
| dansmith | kopecmartin: so this is the juicy one: https://review.opendev.org/c/openstack/tempest/+/889202/1 | 16:29 |
| dansmith | the test was creating an extra server.. one to not use and then another with volume-backed | 16:29 |
| dansmith | that's *obviously* better to avoid :) | 16:29 |
| dansmith | gmann: I noticed you rechecked a docs timeout recently | 17:29 |
| dansmith | I saw this just now: python tools/generate-tempest-plugins-list.py taking like 30 minutes | 17:29 |
| dansmith | perhaps that's fetching things from remote and getting stuck? | 17:29 |
| gmann | dansmith: oh is it, 30 min is too long for that script. I need to check if it search in non OpenStack namespace too ? | 17:33 |
| dansmith | idk, I haven't looked at what it's doing, just saw that it's the offender for my most recent docs timeout | 17:33 |
| gmann | ok, I will check | 17:34 |
| gmann | all those timeout things in various places and fixes stopping me to think on doc job timeout. sorry about not checking it | 17:35 |
| gmann | yeah, it is more than just openstack/ namespace search I will suggest 4 years of that release can be good amount (1.5 years in Maintained phase + 2.5 years in Unmaintained phase). | 17:36 |
| gmann | This way we will keep (counting from the time of latest 2023.1 release): | 17:36 |
| gmann | Maintained: | 17:36 |
| gmann | master | 17:36 |
| gmann | stable/2023.1 (< 6 months of release) | 17:36 |
| gmann | stable/zed (6 months of release) | 17:36 |
| gmann | stable/yoga (12 months of release) | 17:36 |
| gmann | Unmaintained: | 17:36 |
| gmann | unmaintained/xena (1.5 years of release) | 17:36 |
| gmann | unmaintained/wallaby (2 years of release) | 17:36 |
| gmann | unmaintained/victoria (2.5 years of release) | 17:36 |
| gmann | unmaintained/ussuri (3 years of release) | 17:36 |
| gmann | unmaintained/train (3.5 years of release) | 17:36 |
| gmann | unmaintained/stein (4 years of release) | 17:36 |
| gmann | unmaintained/rocky -> EOL ( > 4 years of release) | 17:36 |
| gmann | or we can make it 5 if that timeline seems good. | 17:36 |
| gmann | ohh, my bad | 17:36 |
| gmann | this is what I wanted to paste https://github.com/openstack/tempest/blob/master/tools/generate-tempest-plugins-list.py#L129-L145 | 17:36 |
| dansmith | gmann: maybe we just need to log something from there to see if we're getting stuck on one or they're just so slow that we're taking too long? | 17:40 |
| dansmith | seems like some of our workers are hiiiightly io constrained these days, like some take double the time of others to run the same job | 17:40 |
| dansmith | so maybe this is just a very slow worker? | 17:41 |
| gmann | maybe, as i have not see it much but let me do logs and check locally also | 17:42 |
| gmann | dansmith: total 1348 projects are searched in <2 min locally | 17:49 |
| gmann | may be some other issue on worker/node | 17:49 |
| dansmith | yeah | 17:50 |
| dansmith | kopecmartin pointed out one of these test optimization runs where it was 100% slower on one worker than another he found | 17:51 |
| dansmith | and I looked at several tests in each and the difference was staggering | 17:51 |
| opendevreview | Ghanshyam Mann proposed openstack/tempest master: Add logs in generate-tempest-plugins-list.py https://review.opendev.org/c/openstack/tempest/+/889333 | 18:07 |
| dansmith | ++ | 18:07 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Enable LM back and forth flag for SRIOV https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/889338 | 18:39 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Enable LM back and forth flag for SRIOV https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/889338 | 18:42 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Enable LM back and forth flag for SRIOV https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/889338 | 18:43 |
| opendevreview | James Parker proposed openstack/whitebox-tempest-plugin master: Enable LM back and forth flag for SRIOV https://review.opendev.org/c/openstack/whitebox-tempest-plugin/+/889338 | 18:44 |
| JayF | I'm trying to get a devstack setup based on an Ironic CI job, and it's repeatedly failing starting etcd: https://gist.github.com/jayofdoom/b7020b52d8c69d719627d3d27d4231cc | 20:04 |
| JayF | This is a freshly installed jammy VM with 16GB of ram and a couple of vcpus | 20:04 |
| JayF | Configuration is https://zuul.opendev.org/t/openstack/build/cde0eec7274b46f69788d62f08856f1e/logs controller/logs/_.localrc_auto.txt tossed into localrc with ERROR_ON_CLONE=true removed | 20:05 |
| JayF | aha, I think I need to nuke SERVICE_HOST too | 20:15 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 20:37 |
| dansmith | gmann: sorry meant to mention https://54345f3748db63fac266-f6d343514834eb14212ec1823288b4de.ssl.cf5.rackcdn.com/889196/2/check/tempest-slow-py3/298c784/testr_results.html in here | 21:12 |
| * dansmith is juggling too many places at the moment | 21:12 | |
| gmann | dansmith: let's wait for ykarel response once he is online as it is failing in serial run also and if he does not have any quick finding then we can skip it | 21:13 |
| gmann | ykarel: you remember we talked in neutron channel about test_port_security_macspoofing_port test failing failing in parallel run but it is failing in same frequency in serial run also | 21:14 |
| opendevreview | Goutham Pacha Ravi proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 21:51 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!