| opendevreview | Merged openstack/devstack master: Always set image_uuid_alt in configure_tempest() https://review.opendev.org/c/openstack/devstack/+/888953 | 02:34 |
|---|---|---|
| opendevreview | Merged openstack/tempest master: Fix test_schedule_to_all_node to support installation with ironic https://review.opendev.org/c/openstack/tempest/+/887493 | 05:41 |
| opendevreview | Jakub Skunda proposed openstack/devstack master: git: git checkout for a commit hash combinated with depth argument https://review.opendev.org/c/openstack/devstack/+/889012 | 08:22 |
| sean-k-mooney | hi o/ | 11:03 |
| sean-k-mooney | i have not really been following the "install devstack in a single venv" topic since last september | 11:03 |
| sean-k-mooney | but i think i saw some comment in my email that its close to working end to end again? | 11:04 |
| sean-k-mooney | i need to create a devstack env to test something but i was wondering if it was worth testing that while i was at it | 11:04 |
| sean-k-mooney | https://review.opendev.org/c/openstack/devstack/+/558930 | 11:07 |
| sean-k-mooney | based on the zuul cofnig it looks liek its disabeld on rpm distos but i can test it out on ubuntu 22.04 right | 11:07 |
| frickler | sean-k-mooney: yes, I didn't get it to work on rpm, if you want to look into that, that would be great. ubuntu and debian should work fine, see also https://review.opendev.org/c/openstack/devstack/+/887547 | 11:25 |
| sean-k-mooney | honestly i dont really care about fedora/centos testing normally. i will ocationally fix things for coworkers that use them like this https://review.opendev.org/c/openstack/devstack/+/878718 review welcome :P but i use ubunut for almost all my dev | 11:28 |
| sean-k-mooney | next time i deploy a rpm dev env i can give it a try but currenly installing 22.04 | 11:29 |
| frickler | sean-k-mooney: ok, thx | 11:34 |
| sean-k-mooney | my underdstandign is the intent of debian/ubuntu to break sudo pip install | 11:35 |
| sean-k-mooney | is not yet something that fedora have looked at | 11:36 |
| sean-k-mooney | so for now it not an issue | 11:36 |
| frickler | sean-k-mooney: well we could simplify some code if we stop supporting the non-venv variant, but I agree it is not urgent | 11:39 |
| sean-k-mooney | yep i think we shoudl move in that direction | 11:40 |
| sean-k-mooney | but that i think can wait till the C release or at least till after this inital path is merged | 11:40 |
| frickler | ack | 11:45 |
| sean-k-mooney | hum | 11:55 |
| sean-k-mooney | so it look like devstack is assuming that /opt/stack/logs or opt stack in general is owned by the current user | 11:56 |
| sean-k-mooney | which it almost always will be | 11:57 |
| sean-k-mooney | i normally either use "sudo ./tools/create-stack-user.sh" or add the account i log in with with passwordless sudo in my devstack vms | 11:58 |
| sean-k-mooney | this time i did both and it did not like it because my current user did not create /opt/stack as its the stack user home dir | 11:58 |
| sean-k-mooney | ill just swap to the stack user instead | 11:59 |
| frickler | I always run devstack as the default user of the cloud image I'm using, that avoids any such issue. is there still any installer that only creates a root account? if not, we might drop some of this confusing extra user stuff | 12:09 |
| sean-k-mooney | yep | 12:09 |
| sean-k-mooney | and when i do that on my home openstack thats what i do | 12:09 |
| sean-k-mooney | i installed the vm on my laptop form an iso this time | 12:10 |
| sean-k-mooney | i dont think there is any disto that currently has an option to give passwordless sudo rights to the current user | 12:11 |
| sean-k-mooney | * to the new user at install time | 12:11 |
| sean-k-mooney | but we could jsut add a script file to do that in devstack | 12:11 |
| sean-k-mooney | and drop the stack user stuff | 12:11 |
| * frickler likes that idea | 12:12 | |
| sean-k-mooney | that or move the stack users home dir to /home ... | 12:13 |
| sean-k-mooney | and keep /opt/stack for devstack | 12:14 |
| sean-k-mooney | that actully the root of the isuse i hit sharing /opt/stack as teh dest dir and home dir | 12:14 |
| frickler | maybe the CI setup also gets simpler if we do everything as zuul user. I'll take a look at that | 12:20 |
| sean-k-mooney | ya possibly. do we currently use the stack user in ci | 12:21 |
| sean-k-mooney | i assumed it was using the zuul user already | 12:21 |
| sean-k-mooney | also even with a password if you do sudo ls before you do ./stack.sh | 12:22 |
| sean-k-mooney | devstack normally works fine | 12:22 |
| sean-k-mooney | so as it stand today if you dont run the create stack user script | 12:22 |
| sean-k-mooney | it works perfectly fine with any user with the right permissiosn | 12:23 |
| frickler | https://opendev.org/openstack/devstack/src/branch/master/roles/setup-stack-user | 12:26 |
| frickler | and a different tempest user | 12:26 |
| sean-k-mooney | ========================= | 12:47 |
| sean-k-mooney | DevStack Component Timing | 12:47 |
| sean-k-mooney | (times are in seconds) | 12:47 |
| sean-k-mooney | ========================= | 12:47 |
| sean-k-mooney | wait_for_service 13 | 12:47 |
| sean-k-mooney | async_wait 364 | 12:47 |
| sean-k-mooney | osc 167 | 12:47 |
| sean-k-mooney | apt-get 202 | 12:47 |
| sean-k-mooney | test_with_retry 6 | 12:47 |
| sean-k-mooney | dbsync 27 | 12:47 |
| sean-k-mooney | pip_install 123 | 12:47 |
| sean-k-mooney | apt-get-update 0 | 12:47 |
| sean-k-mooney | run_process 27 | 12:47 |
| sean-k-mooney | git_timed 119 | 12:47 |
| sean-k-mooney | ------------------------- | 12:47 |
| sean-k-mooney | Unaccounted time 173 | 12:47 |
| sean-k-mooney | ========================= | 12:47 |
| sean-k-mooney | Total runtime 1221 | 12:48 |
| sean-k-mooney | i have not done a normal install on this laptop | 12:48 |
| sean-k-mooney | but that does not seam bad for a clean vm | 12:48 |
| sean-k-mooney | so a.) 20 mins is about on par with what i expect so the venv didnt seam to slow anything down | 12:49 |
| sean-k-mooney | b.) it mostly worked "first time" on a clean vm | 12:49 |
| sean-k-mooney | im going to actully boot a vm and then ill reveiw the vnev patch but at first galnce its looking promising | 12:50 |
| sean-k-mooney | has this been tried with any devstack plugins? | 12:50 |
| sean-k-mooney | like devstack-plugin-ceph | 12:50 |
| frickler | I tested with neutron and designate, but not ceph I think, that's a good addition | 13:41 |
| sean-k-mooney | if the devstack plugins were suing the helper furnciotn in function and functions-common form devstack | 13:54 |
| sean-k-mooney | they likely dont need to do anything special | 13:55 |
| sean-k-mooney | i.e. if they were using the pip helpers | 13:55 |
| sean-k-mooney | things like cpeh proably are fine since they are configuring an external system | 13:55 |
| sean-k-mooney | but a devstack pluging dat integrated a out of tree driver ectra would need to ensure its moduels were put in the venv | 13:56 |
| yusufgungor | Hi everyone, our wallaby cluster has v2.88 nova api version. But tempest latest version tempest 35.0.1.dev17 does have v2.79. tempest.api.compute.admin.test_live_migration.LiveMigrationTest.test_volume_backed_live_migration test fails because of schema validation when listing migrations. Is it ok to set min_microversion = 2.79 on tempest and ignore this situation? In real life will our users still use v.288? Thanks | 14:32 |
| yusufgungor | fail error is | 14:32 |
| yusufgungor | validators.py", line 1292, in validate | 14:32 |
| yusufgungor | raise error | 14:32 |
| yusufgungor | jsonschema.exceptions.ValidationError: Additional properties are not allowed ('project_id', 'user_id' were unexpected) | 14:32 |
| dansmith | gmann: what would you think about trying to change the names of some of the tempest classes to distribute them amongst the workers better? | 14:33 |
| dansmith | AFAICT, tons of stuff starts with S(erver) and T(est) and that seems to pile it up in the middle | 14:34 |
| dansmith | Could we pick some meaty classes and just try making them ZServerRescueTest and ATestSnapshot ? | 14:34 |
| dansmith | I mean I know it's silly, but... | 14:34 |
| opendevreview | Rodolfo Alonso proposed openstack/tempest master: [WIP][DNM] Test fix for LP#2027605 https://review.opendev.org/c/openstack/tempest/+/888770 | 15:21 |
| dansmith | or maybe we break up ServerActionsTest as that seems to be the single largest test and maybe breaking it in half would be worth the overhead | 15:25 |
| dansmith | hmm, I guess it's not partitioning based on class name but I thought that's how it worked | 15:26 |
| opendevreview | Dan Smith proposed openstack/tempest master: WIP: Try splitting up ServerActionsTestJSON https://review.opendev.org/c/openstack/tempest/+/889109 | 15:32 |
| opendevreview | Eric Harney proposed openstack/devstack-plugin-ceph master: Drop usage of python3_enabled https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/889110 | 15:44 |
| opendevreview | Merged openstack/devstack stable/2023.1: The AZ filter is deprecated and planned for removal this cycle https://review.opendev.org/c/openstack/devstack/+/888648 | 15:51 |
| gmann | dansmith: I am not sure class name will make any difference but splitting the long tests into separate class can be benefit. But we need to be careful on non-admin test class renaming as they break interop and other users. | 16:14 |
| gmann | this has all the checks/things we need to care when we refactor the tests https://docs.openstack.org/tempest/latest/REVIEWING.html#test-removal-and-refactoring | 16:15 |
| gmann | Honestly saying I am waiting for this and at least it will increase the worker and the load at some level- https://review.opendev.org/c/openstack/tempest/+/887220 | 16:15 |
| dansmith | gmann: ugh okay I didn't know we couldn't move (some) tests :( | 16:46 |
| dansmith | I thought the uuid was the key there not the name/location | 16:46 |
| gmann | yeah, that was supposed to be that way use uuid as test name/location but it did not happen and interop still using test class/test name | 16:47 |
| dansmith | that kinda keeps us stuck unfortunately | 16:47 |
| dansmith | gmann: where's the actual list of interop tests? will codesearch show them to me? | 16:50 |
| dansmith | ah, in interop guidelines json I guess? | 16:51 |
| gmann | dansmith: yes, codesearch should show if that search in openinfra/ namespace also | 16:51 |
| dansmith | yeah, I see | 16:52 |
| gmann | dansmith: here are those guidliens including the test location/name https://opendev.org/openinfra/interop/src/branch/master/guidelines | 16:52 |
| gmann | next.json and a few lalest one fo 2022/21 year we can care of | 16:52 |
| gmann | we have a voting job in tempest gate verify that if location of test in tempest is changed form what interop has | 16:53 |
| gmann | interop-tempest-consistency | 16:53 |
| dansmith | okay I see it yeah | 17:05 |
| opendevreview | Dan Smith proposed openstack/tempest master: WIP: Try splitting up ServerActionsTestJSON https://review.opendev.org/c/openstack/tempest/+/889109 | 18:07 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: [WIP][DNM] Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 18:12 |
| dansmith | gmann: I assume we don't run stackviz on failed/timedout runs because we don't get a clear subunit file, but we do have the partial one | 18:41 |
| dansmith | I wonder if it would still work and would show us that a worker got stuck on something useful? | 18:41 |
| gmann | dansmith: we run i think, it is not showing? | 18:42 |
| dansmith | gmann: not that I see: https://zuul.opendev.org/t/openstack/build/3f40c45857ab409aaef1583c258727ac/logs | 18:43 |
| gmann | dansmith: ah for job timeout, yes we do not run as that does not proceed to that role except logs collection | 18:45 |
| gmann | for test failing/ test timeout we do run it | 18:45 |
| dansmith | ah okay | 18:46 |
| dansmith | gmann: I think splitting the non-interop tests out of ServerActionTestJSON is working locally.. will look at the spread of time, but I think that's got to be worth doing based on the interleaving of tests among the workers that I'm seeing | 19:01 |
| dansmith | versus running that whole test always on one worker | 19:01 |
| gmann | ok | 19:43 |
| opendevreview | Dan Smith proposed openstack/tempest master: WIP: Try splitting up ServerActionsTestJSON https://review.opendev.org/c/openstack/tempest/+/889109 | 20:25 |
| dansmith | gmann: this ^ runs in 1/3 the clock time locally | 20:37 |
| dansmith | obviously that's just running the one class, which perfectly distributes between the workers, which won't always happen in a full run, but I bet it will help | 20:38 |
| dansmith | it's almost 15 minutes worth of wall time when it's running sequentially | 20:38 |
| dansmith | and just over five when run in that split | 20:39 |
| dansmith | melwitt: ^ since you were talking about worker distribution too | 20:39 |
| dansmith | ....and doesn't fail the interop test \o/ | 20:39 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Run slow api tests in parallel https://review.opendev.org/c/openstack/tempest/+/889129 | 20:42 |
| gmann | dansmith: ack. | 20:43 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Do not set the concurrency for tempest run https://review.opendev.org/c/openstack/tempest/+/887220 | 20:43 |
| dansmith | man, 66% performance improvement and just an ack, huh? :P | 20:43 |
| dansmith | gmann: I left a comment on the tests that are moving that I see in skip lists according to codesearch, so I dunno what to do about those, but I left it WIP as a result | 20:44 |
| gmann | dansmith: :) sorry I need to check, I was busy in increasing the number of worker :) | 20:44 |
| dansmith | haha, I'm just giving you a heard time :P | 20:45 |
| gmann | splitting test at least in server action tests is definitely good idea | 20:45 |
| dansmith | there are a couple others that are pretty chunky, but not sure they will be as much gain | 20:46 |
| gmann | skip list of tripleO ? or pther project | 20:46 |
| dansmith | I'll wait to see what the stackviz looks like on this | 20:46 |
| dansmith | I don't remember | 20:46 |
| dansmith | mostly this I think: https://opendev.org/x/trio2o/src/branch/master/trio2o/tempestplugin/tempest_compute.sh | 20:46 |
| dansmith | I dunno what that is | 20:46 |
| dansmith | openstack-operators/tempest_skip.yml | 20:48 |
| gmann | I think that is vexxhost one ? | 20:48 |
| gmann | I remember they have one such skip list | 20:48 |
| dansmith | okay | 20:49 |
| gmann | but we can either puch a quick patch to them to modify the location but not to wait for them to merge | 20:49 |
| gmann | if it is openstack then we can wait in case we break any project CI | 20:50 |
| dansmith | ack, if this looks good on stackviz I will revise to remove WIP and those comments and make a list of all the references | 20:50 |
| gmann | cool. I will check that. going out now but will be back in ~2 hrs | 20:56 |
| dansmith | o/ | 20:56 |
| opendevreview | Ashley Rodriguez proposed openstack/devstack-plugin-ceph master: [WIP][DNM] Remote Ceph with cephadm https://review.opendev.org/c/openstack/devstack-plugin-ceph/+/876747 | 21:00 |
| opendevreview | Dan Smith proposed openstack/tempest master: WIP: Try splitting up ServerActionsTestJSON https://review.opendev.org/c/openstack/tempest/+/889109 | 21:04 |
| melwitt | dansmith: cool idea | 21:16 |
| dansmith | melwitt: still around? | 22:51 |
| melwitt | yes | 22:52 |
| dansmith | okay so, here's another tempest-py3 run from earlier today: | 22:52 |
| dansmith | https://5fb6cfbc0c5c1a5be079-29e820dbac3fa779e4aa716d6c5c5850.ssl.cf1.rackcdn.com/887857/5/check/tempest-full-py3/929ccbf/controller/logs/stackviz/index.html#/stdin/timeline?test=tempest.api.compute.servers.test_server_actions.ServerActionsTestJSON.test_shelve_paused_server | 22:52 |
| dansmith | here's the one from the split: | 22:52 |
| dansmith | https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_113/889109/4/check/tempest-full-py3/113b3e8/controller/logs/stackviz/index.html#/stdin/timeline?test=tempest.scenario.test_security_groups_basic_ops.TestSecurityGroupsBasicOps.test_cross_tenant_traffic | 22:52 |
| dansmith | now, notice the scenario tests are always going to be serial at the end I think, | 22:52 |
| dansmith | so the odd gap in worker0 there is because we finish all the base tests and then do scenario | 22:53 |
| dansmith | but I think that looks like better packing | 22:53 |
| melwitt | oh, huh. I've never seen this graph before | 22:53 |
| melwitt | yeah it does look better | 22:53 |
| dansmith | I think it's actually better than it looks too, | 22:55 |
| dansmith | on the other one, the gaps between the tests appear shorter, because it's more zoomed out | 22:56 |
| melwitt | this will be sweet if we can get better load balancing on the workers | 22:57 |
| dansmith | yeah if you drag the window on the bottom overview to cover everything other than the scenario block at the end, it looks even more densely packed in the parallel section | 22:57 |
| dansmith | like two of the workers end up doing nothing for half the time, and one for 1/3 the time | 22:57 |
| melwitt | oh yeah, I see. this is cool | 22:58 |
| dansmith | but with the broken up test, it's much more even | 22:58 |
| melwitt | yeah. I never thought about it before to look at the size of the classes, but it makes sense | 22:59 |
| dansmith | yeah | 23:00 |
| dansmith | I occasionally have good ideas | 23:00 |
| dansmith | I'm also waiting for a nova DNM based on this to see the ceph job, | 23:01 |
| dansmith | because that one does only three workers | 23:01 |
| melwitt | lol | 23:01 |
| dansmith | I'm not sure if that's going to be better or worse for this | 23:01 |
| opendevreview | Dan Smith proposed openstack/tempest master: Split up ServerActionsTestJSON for better mixing https://review.opendev.org/c/openstack/tempest/+/889109 | 23:05 |
| dansmith | de-WIP'd ^ | 23:05 |
| dansmith | well, the nova ceph job completed in 1.5 hours, which is pretty fast | 23:06 |
| dansmith | worker distribution doesn't look super awesome but I need to compare to another one: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_14a/889130/1/check/nova-ceph-multistore/14a078d/controller/logs/stackviz/index.html#/stdin/timeline | 23:07 |
| dansmith | yeah, about the same: https://23747c8ea869a495f1ca-a232ce3bdc50fca913ceba9a1c600c62.ssl.cf2.rackcdn.com/879499/10/check/nova-ceph-multistore/1d0369b/controller/logs/stackviz/index.html#/stdin/timeline | 23:07 |
| dansmith | actually, looks like most of these tests don't run under the ceph job anyway, so that makes sense | 23:08 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Run test_port_security_macspoofing_port slow tests serially https://review.opendev.org/c/openstack/tempest/+/889129 | 23:35 |
| opendevreview | Ghanshyam proposed openstack/tempest master: Do not set the concurrency for tempest run https://review.opendev.org/c/openstack/tempest/+/887220 | 23:58 |
| melwitt | dansmith: did you consider doing 'stestr run --concurrency N --random' rather than the class thing? I was just looking at this which says they're sorted alphabetically in the absence of historical timing data https://stestr.readthedocs.io/en/latest/MANUAL.html#test-scheduling | 23:58 |
| melwitt | I don't think we have or keep historical timing data between job runs but maybe I'm wrong | 23:59 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!