| opendevreview | Merged openstack/tempest master: Add test for assisted volume snapshot https://review.opendev.org/c/openstack/tempest/+/864839 | 00:21 |
|---|---|---|
| opendevreview | Maxim Sava proposed openstack/tempest master: Revert "Add image task client and image tests task APIs." https://review.opendev.org/c/openstack/tempest/+/890498 | 10:11 |
| opendevreview | Alfredo Moralejo proposed openstack/devstack master: Remove wget + rpm workaround to manage repos install in CentOS https://review.opendev.org/c/openstack/devstack/+/890644 | 12:16 |
| opendevreview | Merged openstack/tempest master: Skip scenario tests early to avoid unnecessary setup https://review.opendev.org/c/openstack/tempest/+/890573 | 13:08 |
| opendevreview | Katarina Strenkova proposed openstack/tempest master: Skip failing tests affected by minimum password age https://review.opendev.org/c/openstack/tempest/+/890653 | 13:30 |
| opendevreview | Katarina Strenkova proposed openstack/tempest master: Skip failing tests affected by minimum password age https://review.opendev.org/c/openstack/tempest/+/890653 | 13:33 |
| dansmith | gmann: so I rechecked the concurrency revert all weekend. I hit *one* OOM in all of that, but the majority of them all passed, with only a few hitting known issues | 13:42 |
| dansmith | in comparison, I and others rechecked a few other patches and hit several OOMs over the same time period (but with fewer runs) | 13:42 |
| dansmith | if you still think it's too early to tell, would you be willing to drop concurrency on some specific jobs and see how that goes? | 13:43 |
| dansmith | also, no timeouts, FWIW, but that could be because overall load was light being the weekend | 13:44 |
| dansmith | actually, the one OOM it did hit had 13 (!) qemu processes running | 13:47 |
| dansmith | so perhaps something went haywire and we created too many servers, or we got unlucky and ran four tests at the same time that create like three servers each somehow? | 13:48 |
| opendevreview | Dan Smith proposed openstack/tempest master: Skip test_image_tasks_create() for bug 2030527 https://review.opendev.org/c/openstack/tempest/+/890687 | 16:31 |
| dansmith | gmann: kopecmartin: ^ per the skip decision.. I haven't seen any other bug or attempt to skip.. blocking nova's gate now | 16:32 |
| gmann | dansmith: thanks, I was asking author if they have filed any bug or not. | 16:39 |
| gmann | this test passed in tempest-full job but not sure why it is failing in import job or other | 16:40 |
| gmann | dansmith: on concurrency, I agree with the idea of reducing it in some job. let me try on tempest full (integrated job) jobs which have the most parallel tests | 16:41 |
| sean-k-mooney | that specific test does not appare to creating servers so i assuem the 13 are form somehting else? | 16:42 |
| sean-k-mooney | oh i guess that a diffent topic | 16:42 |
| sean-k-mooney | i was looking at the test_image_tasks_create patch | 16:43 |
| sean-k-mooney | but there was a sperate concurancy change ? | 16:43 |
| dansmith | gmann: ++ on concurrency | 16:43 |
| dansmith | gmann: I saw the query and no response and found no bug, so I figured I should do it quick since it's so bad | 16:44 |
| gmann | dansmith: ++, thanks, 1 comment on using skip-because | 16:46 |
| dansmith | I just saw | 16:47 |
| sean-k-mooney | oh wow 'default_concurrency: "{{ num_cores|int - 2 }}"' ya that would break alot of job | 16:47 |
| sean-k-mooney | it would go form 4 to 6 | 16:48 |
| dansmith | sean-k-mooney: it doesn't break everything, but it results in us ending up with a lot of qemu processes which OOMs mysql a lot | 16:48 |
| dansmith | right, it went to 6 as a result | 16:48 |
| sean-k-mooney | right so most of the jobs are turned to expect 4 | 16:48 |
| sean-k-mooney | and OOM is exaclty what i woudl expect | 16:48 |
| sean-k-mooney | for jobs that dont boot vms | 16:49 |
| dansmith | https://imgur.com/a/VNHnNNi | 16:49 |
| sean-k-mooney | then sure 6 might work | 16:49 |
| sean-k-mooney | should i be concern that imgur asked if it was over 18 :P | 16:49 |
| dansmith | it does that all the time for screenshots now, I have no idea why | 16:50 |
| opendevreview | Dan Smith proposed openstack/tempest master: Skip test_image_tasks_create() for bug 2030527 https://review.opendev.org/c/openstack/tempest/+/890687 | 16:50 |
| sean-k-mooney | i mean that graph is a bit spice you wanot wnat to offfend the ci gods | 16:50 |
| dansmith | gmann: sorry I should have known about skip_because ^ | 16:50 |
| gmann | dansmith: +W, thanks | 16:51 |
| dansmith | gmann: thanks | 16:52 |
| sean-k-mooney | for what its worth increaseing the default concurancy is not nessiarly a bad idea | 16:52 |
| sean-k-mooney | i think we just need to clamp some of the jobs in nova and perhaps other porjects first | 16:53 |
| dansmith | sean-k-mooney: we just did a bunch of work to pack the workers tighter so we make better use of the concurrency we have, which made a big difference for the timeouts | 16:53 |
| dansmith | sean-k-mooney: I suggested a tempest semaphore for server boots, so we can say "no more than 4 at a time" or something | 16:53 |
| dansmith | without dynamic worker loading in testr that will likely generate some long waits though | 16:53 |
| sean-k-mooney | hum thats interesting it might work although it could also cause timeouts | 16:54 |
| sean-k-mooney | unless we tagged the tests that create servers | 16:54 |
| sean-k-mooney | i.e. dont allow the test to start until there is space | 16:54 |
| dansmith | yup, but that's really what we want.. to limit the number of servers in parallel.. test concurrency is just a guess/race/approximation | 16:54 |
| dansmith | and it results in us wasting time we could be using more cpus otherwise | 16:54 |
| dansmith | right, we'd need to say "this test needs 3 from the semaphore" | 16:55 |
| sean-k-mooney | ya which we could probaly do with a decorator | 16:55 |
| dansmith | but still, that test will wait and not do any more useful work on that worker until it gets it without dynamic scheduling | 16:55 |
| sean-k-mooney | although som times the creation happens in setup | 16:55 |
| sean-k-mooney | so we might need to decorate the class in that case | 16:55 |
| dansmith | we'd have to handle it in class setup | 16:55 |
| sean-k-mooney | ya | 16:55 |
| sean-k-mooney | the test scheduling is static correct. i.e. we can tweak the allcoation to workser when we first invoke tempest but cant dynmaically schedule after that? | 16:57 |
| sean-k-mooney | im kind of wondeiring if we could tag all tests that create vms and have them handled by a singel worker until the other works finish all the ohter test. and then only start allocating the vm test to them at that point | 16:58 |
| dansmith | mtreinish has an old PR to make it dynamic | 16:58 |
| dansmith | but I think it's a ways off | 16:58 |
| dansmith | we can also try upping the swap to 8G like the ceph job.. the ceph job is restricted to concurrency=3 and has swap=8G | 17:00 |
| sean-k-mooney | ya the memory pressure shoudl have reduced now that we are using the qemu cache size setting | 17:00 |
| dansmith | I worry about more swap taking more IO which we're also *definitely* hitting our limit on, but n=4 and swap=8G might be better than what we had before for the odd OOM | 17:00 |
| sean-k-mooney | but more swap wont hurt | 17:00 |
| sean-k-mooney | well on that front | 17:01 |
| sean-k-mooney | i have been meaning to look into adding zswap or zram | 17:01 |
| sean-k-mooney | to ci | 17:01 |
| dansmith | yeah, I was wondering about KSM too, since we boot a bunch of instances that are the exact same | 17:01 |
| sean-k-mooney | thats trading cpu for disk ioi but i thikn zswap could help mask that | 17:01 |
| sean-k-mooney | ksm is interesting in principal your right we shoudl eb able to get good deduplicaiton | 17:02 |
| sean-k-mooney | i have not used it personally | 17:02 |
| sean-k-mooney | so in practice im not sure if we would see a noticibale effect | 17:02 |
| dansmith | it was designed for this sort of workload | 17:03 |
| dansmith | I think it didn't really pan out for general usage as much as expected, but I think that's because over time identical guests diverge too much | 17:03 |
| sean-k-mooney | yes althouguh mainly on a host i dont knwo if the same works when we have nested virt | 17:03 |
| dansmith | but our guests are short-lived and a lot less busy than normal | 17:03 |
| sean-k-mooney | provided we dont have ksm also on the host of the zuul vm | 17:03 |
| sean-k-mooney | i would expect it to be worth a try | 17:04 |
| dansmith | KSM works independent of virtualization | 17:04 |
| sean-k-mooney | it shoudl i would just be workied of the host and our vm both deduping and spliting the same page | 17:04 |
| sean-k-mooney | still worth a try and see | 17:05 |
| dansmith | idk why that would matter so much really, but yeah | 17:05 |
| sean-k-mooney | i had code for zswap somewhere in the past i started pocing it as part of https://review.opendev.org/c/openstack/devstack/+/828639. i can give that a try on thursday or next week and let you know when its up for review if you like | 17:08 |
| sean-k-mooney | when i got devstack running on my m1 macbook air i was using zswap on debian bookworm + the venv patches cobbled together locally | 17:10 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Setting Tempest run concurrency to 4 for a few jobs https://review.opendev.org/c/openstack/tempest/+/890689 | 17:12 |
| sean-k-mooney | i think i just did that by hand however rather then coding it up in devstack. | 17:12 |
| gmann | dansmith: sean-k-mooney ^^ setting it to 4 for highly loaded jobs | 17:12 |
| dansmith | kopecmartin: ^ | 17:13 |
| dansmith | gmann: thanks | 17:13 |
| dansmith | vfat change is very close to merging | 17:14 |
| dansmith | just running the last scenario tests in the scope test | 17:14 |
| sean-k-mooney | vfat? | 17:14 |
| sean-k-mooney | as in for configdrive? | 17:14 |
| dansmith | sean-k-mooney: for the stamp test thing, to try to work around mke2fs occasionally going off the rails | 17:14 |
| sean-k-mooney | oh hehe ok | 17:15 |
| sean-k-mooney | im not sure why vfat would be any faster but if it is cool | 17:16 |
| dansmith | mkfs.vfat writes a lot less to the disk to make it usable for us to write 32 bytes to for the test | 17:16 |
| sean-k-mooney | i guess no journal | 17:16 |
| dansmith | it's way faster | 17:16 |
| dansmith | and doesn't use any barriers for normal writes | 17:16 |
| dansmith | yeah | 17:16 |
| sean-k-mooney | ya so like ext2 | 17:16 |
| dansmith | we'll see .. if it doesn't help then meh, but if it does then cool | 17:16 |
| dansmith | I think it's substantially lighter weight than ext2 as well | 17:16 |
| sean-k-mooney | cool well we still use it optionally for config drive instead of iso | 17:17 |
| dansmith | I thought about not even using a filesystem and just writing some bytes to the block device ;) | 17:17 |
| sean-k-mooney | so its not like vfat is going anywhere | 17:17 |
| JayF | fat is a filesystem built on hope so that's pretty close to the same thing Dan ;) | 17:17 |
| dansmith | yup | 17:17 |
| gmann | dansmith: anything I have not reviewed the one we discussed last week. I might be online for another 1 hr only. its holiday here and I am not supposed to be working :) | 17:17 |
| JayF | just mark on the sheet where you put the data and we'll trust you ;) | 17:17 |
| dansmith | gmann: go away then! | 17:17 |
| dansmith | JayF: well, compared to zfs, ext4 isn't much better in that department :) | 17:18 |
| sean-k-mooney | sameish but i arragend 2 meetign for today because i forgot so taking friday off instead. | 17:18 |
| gmann | :) finishing up a few downstream things | 17:18 |
| dansmith | gmann: sorry if my pings earlier drug you in. I didn't know you were supposed to be off | 17:19 |
| gmann | no, while doing that I thought of checking here and gate things. specially task test | 17:20 |
| dansmith | ack | 17:20 |
| dansmith | I spent way too much time this weekend rechecking and looking at logs for sure | 17:22 |
| sean-k-mooney | dansmith: it looks liek ksm is ksmtuned on ubuntu, debian and centos 9 stream and its in the main/appstream repos | 17:24 |
| sean-k-mooney | so i can include isntalling that in my zswap patch | 17:24 |
| dansmith | ksm is a kernel function, but maybe enabled by those? | 17:24 |
| opendevreview | Merged openstack/tempest master: Use vfat for timestamp https://review.opendev.org/c/openstack/tempest/+/890350 | 17:25 |
| dansmith | ah yeah, enabled and managed by ksmtuned | 17:25 |
| sean-k-mooney | ya | 17:25 |
| sean-k-mooney | i dont really know how to configre it so i assume its default will just work | 17:26 |
| sean-k-mooney | but i can take a look | 17:26 |
| dansmith | there are thresholds that control how much it tries I think | 17:26 |
| sean-k-mooney | i assume like vm.swapieness | 17:26 |
| dansmith | enabling it is probably better than nothing, but we would likely benefit from some aggressive tunes | 17:26 |
| dansmith | yeah | 17:26 |
| sean-k-mooney | so fro zswap ther is a great block on how they got ubuntu desktop to work on a 2G rasberry pi | 17:27 |
| sean-k-mooney | https://www.omgubuntu.co.uk/2022/01/ubuntu-on-raspberry-pi-4-2gb-zswap and and https://ubuntu.com//blog/how-low-can-you-go-running-ubuntu-desktop-on-a-2gb-raspberry-pi-4 | 17:27 |
| sean-k-mooney | not sure if tose still load but i was goign to use those as reffence for turnign zswap ill see if there is somehting simialr for ksm | 17:28 |
| dansmith | maybe we should have zmem, zswap, and ksm all as separate knobs and spread out the testing a bit :) | 17:28 |
| sean-k-mooney | i can add them as diffent commtis/nobs ya and then have some dnm patche on top that turns them on | 17:29 |
| sean-k-mooney | dansmith: truns out ksm is already partly enabled in devstack | 18:36 |
| sean-k-mooney | https://github.com/openstack/devstack/commit/f85e0ba353c71fc5c8bacebe006e68bebf2af024 | 18:36 |
| sean-k-mooney | that enabels ksm but without the tuning deamon | 18:36 |
| dansmith | ah, okay | 18:46 |
| sean-k-mooney | if im reading the comment correclty it would only help if the ap mmaped the paages as mergabel | 18:46 |
| sean-k-mooney | and i dont know if anything will do that by default | 18:46 |
| sean-k-mooney | which iassume is what ksmtuned can do | 18:47 |
| dansmith | idk, I didn't think you needed to do that.. any readonly (or not yet CoW'd) page should be a candidate I'd think | 18:50 |
| sean-k-mooney | perhaps in any case its enabeld out of the box since 2017 with devstack | 18:51 |
| dansmith | ack | 18:51 |
| opendevreview | sean mooney proposed openstack/devstack master: [WIP] add support for zswap and ksmtuned https://review.opendev.org/c/openstack/devstack/+/890693 | 19:38 |
| sean-k-mooney | dansmith: ^ i still have that running locally but im going to finish for today. | 19:39 |
| sean-k-mooney | that is based on top of the venv change just becasue that is what im using locally to deploy | 19:39 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 20:03 |
| dansmith | ugh, that skip fix failed again | 20:36 |
| opendevreview | Merged openstack/devstack-plugin-ceph master: [CI] Drop overrides in cephfs-nfs job https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/882625 | 22:01 |
| opendevreview | sean mooney proposed openstack/devstack master: [WIP] add support for zswap and ksmtuned https://review.opendev.org/c/openstack/devstack/+/890693 | 22:06 |
| opendevreview | sean mooney proposed openstack/devstack master: [WIP] add support for zswap and ksmtuned https://review.opendev.org/c/openstack/devstack/+/890693 | 22:29 |
| opendevreview | sean mooney proposed openstack/devstack master: [WIP] add support for zswap and ksmtuned https://review.opendev.org/c/openstack/devstack/+/890693 | 22:31 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!