| *** mjturek has quit IRC | 04:20 | |
| *** ricolin has joined #openstack-multi-arch | 04:41 | |
| ricolin | ianw, hey around? | 07:01 |
|---|---|---|
| ianw | ricolin: hi, yep for a little | 07:01 |
| ricolin | current arm64 nodes seems to be suffering from large number of unit test request, and leave small gap for devstack tempest job | 07:02 |
| ianw | ricolin: yeah, over in #opendev we've been working with kevinz, the clouds has been very unhappy | 07:02 |
| ricolin | is there way to limit number of nodes or any resource unit for UT? | 07:02 |
| ianw | do you mean in general, or hte last few days? | 07:02 |
| ricolin | kevinz and I discuss this morning, about devstack jobs beem waiting for UTs in last few days | 07:04 |
| ianw | yeah, the cloud has basically been offline for the last few days | 07:04 |
| ianw | i think things are moving along again now. today he added another compute node i believe | 07:05 |
| ianw | also, we are close to having about 15 more nodes from OSU coming online | 07:05 |
| ianw | https://review.opendev.org/c/opendev/system-config/+/785813 | 07:05 |
| ricolin | physical nodes? | 07:05 |
| ricolin | sweet | 07:05 |
| ianw | vm's, not sure what they're backed by but hopefully even faster than linaro | 07:05 |
| ianw | we also have others seeing what resources they might be able to find for us :) | 07:06 |
| ianw | but yeah, there is a huge backup of jobs, but they should be moving now | 07:07 |
| ricolin | so do you think we don't need to separate/limit nodes for UT jobs? | 07:08 |
| ianw | we can't really do that at any rate, zuul doesn't really know the difference between jobs | 07:09 |
| ianw | but i think that generally we're not running at capacity | 07:09 |
| ianw | it might also be a zuul/nodepool issue in this case. the jobs should have been timing out, giving clearer feedback that nodes couldn't be allocated | 07:10 |
| ricolin | indeed | 07:10 |
| ianw | going into multi-day queues isn't the intended behaviour | 07:10 |
| ianw | but i think this might have been a corner case of the type of failures returned by linaro's cloud | 07:10 |
| ricolin | I think after rebuild for centos8s image everything solved? | 07:10 |
| ianw | yes there was also that problem last week as well, compounding issues | 07:12 |
| ianw | but yeah, i believe that is all solved now and we have new images done | 07:12 |
| ricolin | ianw, I'm adding some performance improve suggestions on https://review.opendev.org/c/openstack/devstack/+/708317 like PIP_EXTRA_INDEX_URL | 07:15 |
| ianw | ok | 07:16 |
| ricolin | also if it's landed, some devstack works should be parallel now | 07:16 |
| ianw | i mentioned in a email response, but i found nested boot to be *really* slow | 07:16 |
| ianw | like "i'm not sure if i did something wrong" slow ... | 07:17 |
| ricolin | ok, didn't read ML yet, will take a look | 07:17 |
| ricolin | haha | 07:17 |
| ianw | anyway, i was starting the vm by hand using qemu, maybe nova (via devstack) launches things with the right sets of flags to be usable | 07:17 |
| ricolin | backup is also same level sloooow too;( | 07:17 |
| ricolin | I mean backup/restore volume | 07:18 |
| ianw | i don't think we can do nested virt? | 07:18 |
| ianw | apparently the OSU resources will be faster ... we will certainly be able to stress test them :) | 07:18 |
| ricolin | let's find out! | 07:19 |
| *** rh-jelabarre has joined #openstack-multi-arch | 12:17 | |
| *** mjturek1 has joined #openstack-multi-arch | 12:17 | |
| *** mjturek1 is now known as mjturek | 12:42 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!