| *** ociuhandu has joined #openstack-nova | 00:31 | |
| *** ociuhandu has quit IRC | 00:35 | |
| *** brinzhang has quit IRC | 00:48 | |
| *** gbarros has quit IRC | 00:58 | |
| *** ccamacho has quit IRC | 01:09 | |
| *** slaweq_ has joined #openstack-nova | 01:11 | |
| *** slaweq_ has quit IRC | 01:16 | |
| *** factor has quit IRC | 01:23 | |
| *** factor has joined #openstack-nova | 01:25 | |
| *** factor has quit IRC | 01:28 | |
| *** factor has joined #openstack-nova | 01:28 | |
| *** factor has quit IRC | 01:31 | |
| *** factor has joined #openstack-nova | 01:31 | |
| *** brinzhang has joined #openstack-nova | 01:31 | |
| *** lbragstad has quit IRC | 01:39 | |
| *** markvoelker has joined #openstack-nova | 01:53 | |
| *** markvoelker has quit IRC | 01:58 | |
| *** factor has quit IRC | 02:03 | |
| *** factor has joined #openstack-nova | 02:04 | |
| *** factor has quit IRC | 02:07 | |
| *** factor has joined #openstack-nova | 02:07 | |
| *** larainema has joined #openstack-nova | 02:08 | |
| *** dannins has joined #openstack-nova | 02:09 | |
| *** factor has quit IRC | 02:10 | |
| *** factor has joined #openstack-nova | 02:10 | |
| *** dklyle has quit IRC | 02:11 | |
| *** david-lyle has joined #openstack-nova | 02:11 | |
| *** factor has quit IRC | 02:14 | |
| *** factor has joined #openstack-nova | 02:15 | |
| openstackgerrit | Merged openstack/nova master: Deprecate CONF.workarounds.enable_numa_live_migration https://review.opendev.org/640021 | 02:17 |
|---|---|---|
| *** lbragstad has joined #openstack-nova | 02:18 | |
| *** yaawang has quit IRC | 02:45 | |
| *** idlemind has quit IRC | 02:46 | |
| *** yedongcan has joined #openstack-nova | 02:53 | |
| *** rcernin has quit IRC | 02:59 | |
| *** yaawang has joined #openstack-nova | 03:00 | |
| *** mkrai has joined #openstack-nova | 03:06 | |
| openstackgerrit | ya.wang proposed openstack/nova master: Fix typor of cpu model when check CPU compatibility https://review.opendev.org/682267 | 03:06 |
| *** slaweq_ has joined #openstack-nova | 03:11 | |
| *** slaweq_ has quit IRC | 03:15 | |
| *** markvoelker has joined #openstack-nova | 03:24 | |
| openstackgerrit | Luyao Zhong proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 03:25 |
| *** markvoelker has quit IRC | 03:29 | |
| *** psachin has joined #openstack-nova | 03:31 | |
| *** factor has quit IRC | 03:37 | |
| *** factor has joined #openstack-nova | 03:38 | |
| *** factor has quit IRC | 03:41 | |
| *** factor has joined #openstack-nova | 03:41 | |
| *** factor has quit IRC | 03:42 | |
| *** factor has joined #openstack-nova | 03:43 | |
| *** factor has quit IRC | 03:45 | |
| *** factor has joined #openstack-nova | 03:46 | |
| *** rcernin has joined #openstack-nova | 03:46 | |
| *** jhesketh has joined #openstack-nova | 03:50 | |
| openstackgerrit | Boxiang Zhu proposed openstack/nova master: Make evacuation respects anti-affinity rule https://review.opendev.org/649963 | 03:56 |
| *** udesale has joined #openstack-nova | 04:11 | |
| *** factor has quit IRC | 04:49 | |
| *** factor has joined #openstack-nova | 04:50 | |
| *** factor has quit IRC | 04:52 | |
| *** factor has joined #openstack-nova | 04:53 | |
| *** factor has quit IRC | 04:56 | |
| *** factor has joined #openstack-nova | 04:57 | |
| *** Luzi has joined #openstack-nova | 04:59 | |
| *** jawad_axd has joined #openstack-nova | 04:59 | |
| *** macz has joined #openstack-nova | 05:09 | |
| *** pcaruana has joined #openstack-nova | 05:11 | |
| *** etp has joined #openstack-nova | 05:14 | |
| *** macz has quit IRC | 05:17 | |
| *** slaweq_ has joined #openstack-nova | 05:36 | |
| *** ratailor has joined #openstack-nova | 05:44 | |
| *** HagunKim has joined #openstack-nova | 05:44 | |
| *** nnsingh has joined #openstack-nova | 05:45 | |
| *** boxiang has joined #openstack-nova | 05:45 | |
| nnsingh | HI all, i have one doubt, why this file name is .yaml.txt "https://github.com/openstack/placement/blob/master/etc/placement/README-policy.yaml.txt"? what the reason behind this. | 05:46 |
| *** zhubx has quit IRC | 05:47 | |
| openstackgerrit | garyk proposed openstack/nova master: Deconstruct the mother of all locks https://review.opendev.org/682242 | 05:56 |
| openstackgerrit | Boxiang Zhu proposed openstack/nova master: Fix live migration break group policy simultaneously https://review.opendev.org/651969 | 05:59 |
| *** janki has joined #openstack-nova | 06:01 | |
| *** factor has quit IRC | 06:02 | |
| *** mkrai has quit IRC | 06:11 | |
| *** Garyx has quit IRC | 06:22 | |
| *** Garyx has joined #openstack-nova | 06:22 | |
| *** owalsh has quit IRC | 06:23 | |
| *** logan- has quit IRC | 06:23 | |
| *** owalsh has joined #openstack-nova | 06:23 | |
| *** markvoelker has joined #openstack-nova | 06:24 | |
| *** logan- has joined #openstack-nova | 06:26 | |
| *** markvoelker has quit IRC | 06:30 | |
| *** janki has quit IRC | 06:39 | |
| *** rha has joined #openstack-nova | 06:44 | |
| *** trident has quit IRC | 06:48 | |
| *** slaweq_ is now known as slaweq | 06:52 | |
| *** mjozefcz|away has joined #openstack-nova | 06:54 | |
| *** rpittau|afk is now known as rpittau | 06:55 | |
| *** trident has joined #openstack-nova | 06:57 | |
| *** damien_r has joined #openstack-nova | 06:58 | |
| *** janki has joined #openstack-nova | 07:00 | |
| *** trident has quit IRC | 07:03 | |
| *** trident has joined #openstack-nova | 07:12 | |
| *** mkrai has joined #openstack-nova | 07:17 | |
| *** macz has joined #openstack-nova | 07:18 | |
| *** macz has quit IRC | 07:23 | |
| *** mjozefcz|away is now known as mjozefcz | 07:26 | |
| openstackgerrit | Bhagyashri Shewale proposed openstack/nova master: Ignore root_gb for BFV in simple tenant usage API https://review.opendev.org/612626 | 07:27 |
| *** ivve has joined #openstack-nova | 07:32 | |
| *** ralonsoh has joined #openstack-nova | 07:33 | |
| *** jangutter has joined #openstack-nova | 07:34 | |
| *** FlorianFa has quit IRC | 07:36 | |
| *** ociuhandu has joined #openstack-nova | 07:41 | |
| *** FlorianFa has joined #openstack-nova | 07:41 | |
| *** ociuhandu has quit IRC | 07:42 | |
| *** ttsiouts has joined #openstack-nova | 07:47 | |
| *** panda has quit IRC | 07:56 | |
| *** janki has quit IRC | 07:57 | |
| *** janki has joined #openstack-nova | 07:57 | |
| *** panda has joined #openstack-nova | 07:58 | |
| *** xek_ has joined #openstack-nova | 08:05 | |
| *** ricolin has joined #openstack-nova | 08:09 | |
| *** mkrai has quit IRC | 08:10 | |
| *** lpetrut has joined #openstack-nova | 08:10 | |
| *** xek_ has quit IRC | 08:19 | |
| *** ociuhandu has joined #openstack-nova | 08:20 | |
| *** markvoelker has joined #openstack-nova | 08:27 | |
| *** luksky has joined #openstack-nova | 08:30 | |
| *** markvoelker has quit IRC | 08:32 | |
| *** nnsingh has quit IRC | 08:34 | |
| *** derekh has joined #openstack-nova | 08:38 | |
| *** ociuhandu has quit IRC | 08:51 | |
| *** panda is now known as panda|ruck | 08:59 | |
| *** kaisers has quit IRC | 09:05 | |
| *** kaisers has joined #openstack-nova | 09:08 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Allow specify user to reset password https://review.opendev.org/682302 | 09:12 |
| *** mgoddard has quit IRC | 09:26 | |
| *** mgoddard has joined #openstack-nova | 09:28 | |
| *** dtantsur|afk is now known as dtantsur | 09:32 | |
| *** yedongcan has quit IRC | 09:37 | |
| openstackgerrit | Arthur Dayne proposed openstack/nova-specs master: Proposal for a safer noVNC console with password authentication https://review.opendev.org/623120 | 09:39 |
| *** jaosorior has joined #openstack-nova | 09:45 | |
| *** ricolin has quit IRC | 09:52 | |
| *** ociuhandu has joined #openstack-nova | 10:04 | |
| *** xek_ has joined #openstack-nova | 10:06 | |
| *** ttsiouts has quit IRC | 10:10 | |
| *** ttsiouts has joined #openstack-nova | 10:10 | |
| *** ttsiouts has quit IRC | 10:14 | |
| donnyd | sean-k-mooney: that was exactly what I was looking for. Thanks a bunch | 10:22 |
| *** markvoelker has joined #openstack-nova | 10:28 | |
| *** udesale has quit IRC | 10:28 | |
| *** ociuhandu has quit IRC | 10:33 | |
| *** markvoelker has quit IRC | 10:33 | |
| sean-k-mooney | donnyd: you should be aware that it does not always work. on some distors there is a scrip that shuts down any running vms on host reboot. if the compute agent is not stopped first it might notice this and mark the vm as shutdown. the livbrt vm shutdown service file is there to prevent filesystem curroption by gracefully shuting down the vms instead of sig killing them | 10:34 |
| sean-k-mooney | so if it does cause you issue then you have to consider if you should disable the serivce file or not. | 10:35 |
| sean-k-mooney | there is also another config option in nova to disabel reporting of the vm state in the db | 10:37 |
| sean-k-mooney | i think disabling that will also prevent this issue butthen if the guest does a poweroff it wont be reflected in nova status | 10:38 |
| sean-k-mooney | un less they do it via that api. | 10:38 |
| *** osmanlicilegi has joined #openstack-nova | 10:40 | |
| openstackgerrit | Merged openstack/nova master: Parse vpmem related flavor extra spec https://review.opendev.org/678456 | 10:40 |
| sean-k-mooney | pmem is almost done... the PCUP can merge and all the pending feature should be landed | 10:43 |
| *** ociuhandu has joined #openstack-nova | 10:45 | |
| *** ociuhandu has quit IRC | 10:50 | |
| *** tesseract has joined #openstack-nova | 10:51 | |
| *** ttsiouts has joined #openstack-nova | 10:54 | |
| *** artom has joined #openstack-nova | 10:59 | |
| *** ociuhandu has joined #openstack-nova | 11:01 | |
| *** ociuhandu has quit IRC | 11:03 | |
| *** ociuhandu has joined #openstack-nova | 11:04 | |
| *** tesseract has quit IRC | 11:09 | |
| *** ociuhandu has quit IRC | 11:10 | |
| *** avolkov has joined #openstack-nova | 11:10 | |
| *** ociuhandu has joined #openstack-nova | 11:12 | |
| osmanlicilegi | greetings. i'm having problem with nova-conductor, it gives "errno 111 econnrefused" while trying to connecto to rabbitmq. i know it's not a network/firewall issue because all other rabbitmq related services work like a charm. i had the same problem with nova-api and the root cause was monkey patching but this time it's not because nova-conductor does not use monkey patching. anybody had a similar | 11:14 |
| osmanlicilegi | issue before? | 11:14 |
| aspiers | in case anyone didn't get a chance to review my draft SEV blog post last week, here is a link which will last 24 hours (I think) https://blog.adamspiers.org/?p=1871&preview=1&_ppp=dbc2fbd3ce | 11:19 |
| sean-k-mooney | the conductor shuold be useing monkey patching | 11:19 |
| artom | osmanlicilegi, you'll have bette luck in #openstack, operators hang out there, this channel is for development | 11:19 |
| artom | osmanlicilegi, see /topic :) | 11:19 |
| sean-k-mooney | osmanlicilegi: e.g. it should be using eventlet | 11:19 |
| sean-k-mooney | but i have not seen that so i dont know how to help | 11:19 |
| *** macz has joined #openstack-nova | 11:19 | |
| *** mjozefcz has quit IRC | 11:20 | |
| *** mjozefcz has joined #openstack-nova | 11:20 | |
| lyarwood | artom: https://review.opendev.org/#/c/672595/ - do you think you'll have time today to work on this and potentially break out the per n-cpu service connection setting into another change? | 11:20 |
| lyarwood | artom: if not I should be able to get to it this afternoon | 11:20 |
| artom | lyarwood, so... Yes, but! | 11:21 |
| lyarwood | But! | 11:21 |
| lyarwood | Nothing good ever follows but ;) | 11:22 |
| artom | But! https://review.opendev.org/#/c/681060/ is in the gate | 11:22 |
| sean-k-mooney | we might want to hold of doing any upstream work until https://review.opendev.org/#/q/topic:bp/cpu-resources+(status:open+OR+status:merged) lands | 11:22 |
| artom | lyarwood, which does the same thing, right? At least for the hostname part | 11:22 |
| artom | For the connection I need to play around with it some more | 11:22 |
| artom | lyarwood, wait, I misunderstood you, didn't I? | 11:24 |
| *** macz has quit IRC | 11:24 | |
| lyarwood | artom: it does, I'd just rather do this in a helper method in the fixture instead of in a loop per test as in that series | 11:24 |
| artom | You just need the heterogeneous computes helper | 11:24 |
| lyarwood | artom: and again I really need something I can backport for a few fixes | 11:24 |
| lyarwood | artom: yeah pretty much | 11:24 |
| artom | lyarwood, well, I have to stack on top of https://review.opendev.org/#/c/681060/11 regardless | 11:25 |
| sean-k-mooney | aspiers: the assertion you cant snopp if you have physical acess is a little strong. sev and mktme both do not encyrpt cache content so if you have phyical acess to the server and snoop cache you can see the unencypted state of what ever the vm is doing | 11:25 |
| artom | Otherwise we'll conflict | 11:25 |
| artom | But yeah | 11:25 |
| *** dtantsur is now known as dtantsur|bbl | 11:26 | |
| sean-k-mooney | aspiers: you cant read its ram but you could constuct a view of it by recodering the reads/writes to cache | 11:26 |
| lyarwood | sean-k-mooney: sorry re you're earlier comment are you also suggesting avoding POSTing new stuff at all still? I've been holding off for the last week anyway but thought we were passed the bulk of it now. | 11:27 |
| artom | I think what sean-k-mooney's saying is that the SEV work was *completely* pointless >;) | 11:27 |
| lyarwood | your* | 11:27 |
| sean-k-mooney | lyarwood: i personally am holding off posting new stuff until stephens changes land | 11:27 |
| sean-k-mooney | or we decide to punt it | 11:28 |
| sean-k-mooney | i dont want to take gate time form it | 11:28 |
| lyarwood | ack I'll continue to hold off then, wasn't sure if you were just talking about landing new stuff or posting. | 11:28 |
| sean-k-mooney | if its small sure. if you have 10 pending patchs for different thing i dont know | 11:28 |
| sean-k-mooney | today is the cut off to not need an FFE i think | 11:29 |
| lyarwood | kk | 11:29 |
| sean-k-mooney | artom: :) not quite but as a person who previously worked at a hardware companiy im very carful of claims of hardware releatd security features | 11:31 |
| sean-k-mooney | artom: also at this point any limitation of the technology are well within amd's court to go fix aspiers work is quite good | 11:32 |
| sean-k-mooney | aspiers: the only real feedback i would give is the images are a little hard to read without clicking on them. the commandline ones more then anything else | 11:34 |
| sean-k-mooney | the content looks good to me | 11:34 |
| *** jawad_ax_ has joined #openstack-nova | 11:37 | |
| *** ociuhandu has quit IRC | 11:38 | |
| *** ccamacho has joined #openstack-nova | 11:39 | |
| *** jawad_axd has quit IRC | 11:40 | |
| openstackgerrit | ya.wang proposed openstack/nova master: Fix typor of cpu model when check CPU compatibility https://review.opendev.org/682267 | 11:42 |
| *** udesale has joined #openstack-nova | 11:45 | |
| *** ttsiouts has quit IRC | 11:58 | |
| *** xek_ has quit IRC | 12:00 | |
| *** ociuhandu has joined #openstack-nova | 12:00 | |
| *** ociuhandu has quit IRC | 12:00 | |
| *** ociuhandu has joined #openstack-nova | 12:01 | |
| *** ttsiouts has joined #openstack-nova | 12:05 | |
| *** luksky has quit IRC | 12:09 | |
| *** dave-mccowan has joined #openstack-nova | 12:13 | |
| *** mjozefcz has quit IRC | 12:21 | |
| *** mjozefcz has joined #openstack-nova | 12:22 | |
| bauzas | FWIW, I'm a bit on and off today, working back on the placement audit command | 12:26 |
| *** lbragstad has quit IRC | 12:32 | |
| *** ratailor has quit IRC | 12:34 | |
| *** rcernin has quit IRC | 12:35 | |
| *** etp has quit IRC | 12:40 | |
| efried | o/ nova! | 12:43 |
| efried | ready for a fun-filled day of rechecks?! | 12:43 |
| sean-k-mooney | at least we are down to 2 series | 12:45 |
| sean-k-mooney | there is 1 patch need for vpmem, + https://review.opendev.org/#/q/topic:bp/cpu-resources+(status:open+OR+status:merged) + 2 pathes for forbiden aggreates i think | 12:46 |
| efried | I'm starting to think there may actually be something wrong. | 12:46 |
| efried | forbidden aggs is in. | 12:46 |
| sean-k-mooney | oh ok | 12:47 |
| sean-k-mooney | with the pmem stuff | 12:47 |
| efried | And i think there's one for numalm, but not an important one | 12:47 |
| efried | pmem and cpu-resources | 12:47 |
| efried | are the big ones | 12:47 |
| sean-k-mooney | i was ingoring the funcitonal tests for numalm | 12:47 |
| efried | right | 12:47 |
| sean-k-mooney | the actully featre has lannded just tests are leaft | 12:47 |
| * efried counts... | 12:47 | |
| efried | 13 rechecks on the top cpu-resources patch | 12:48 |
| efried | cascading effect of -2s in the gate from a prior patch failing. | 12:48 |
| efried | mostly | 12:48 |
| sean-k-mooney | ya that was what i was about to say | 12:49 |
| efried | I think we have a race in select_destinations though | 12:49 |
| efried | been seeing quite a few like this https://storage.gra1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_778/674895/42/check/openstack-tox-py36/7787cd5/testr_results.html.gz | 12:49 |
| sean-k-mooney | maybe or that test is just not deterministic | 12:51 |
| efried | yeah, either way | 12:52 |
| efried | it's not just that one test | 12:52 |
| efried | I saw several similar failures with that same select_destinations unequal | 12:52 |
| efried | I didn't dig all the way into them (lack of appropriate tooling on my phone) | 12:52 |
| sean-k-mooney | all in the fallback tests | 12:52 |
| efried | but thinking maybe I should at this point. | 12:52 |
| efried | no, I remember seeing one in a pre-existing test (I think) | 12:53 |
| efried | i.e. regression | 12:53 |
| sean-k-mooney | we might not be mocking some global state in the tests? | 12:53 |
| sean-k-mooney | that said the fallback code is very new so there might be something there | 12:54 |
| stephenfin | efried: I'm thinking we should stop rechecking the two few patches for now | 12:56 |
| stephenfin | *top | 12:56 |
| efried | whyzat? | 12:56 |
| efried | oo, I have a local repro | 12:56 |
| efried | running the whole test class | 12:57 |
| efried | nova.tests.unit.scheduler.test_scheduler.SchedulerManagerTestCase | 12:57 |
| efried | sean-k-mooney: ^ | 12:57 |
| sean-k-mooney | ok then it is likely either a shared state issue or a bug in the code | 12:57 |
| stephenfin | On account of having to recheck again if anything lower gets in. Better just get those lower ones in | 12:57 |
| sean-k-mooney | but if that repoduces it locally that makes it eaiser to figure out | 12:57 |
| *** luksky has joined #openstack-nova | 12:57 | |
| stephenfin | Hmm, I've seen that as well, actually | 12:58 |
| stephenfin | Okay, that sounds like a bug. That's testing the retry logic | 12:58 |
| * stephenfin pulls down to try recreate | 12:58 | |
| efried | stephenfin: if it's not affecting queue times -- which it probably isn't at this point -- then it's better to have the top ones kicked out by merge-failed -2s than to have to wait for them to pass the check pipeline again once the lower ones are in, imho | 12:59 |
| *** rcernin has joined #openstack-nova | 12:59 | |
| stephenfin | that's a fair point | 12:59 |
| sean-k-mooney | efried: im just about to grab lunch but ill dig into it when i get back if ye have not figured it out by then | 12:59 |
| efried | stephenfin, sean-k-mooney: I definitely remember seeing the failure on a patch lower in the series at some point. | 12:59 |
| efried | I don't remember which one now | 12:59 |
| efried | where was that fallback introduced, again? | 12:59 |
| stephenfin | efried: https://review.opendev.org/#/c/671801/ | 13:00 |
| sean-k-mooney | ya thats the patch that updated those tests too | 13:00 |
| stephenfin | intermittently failing tests are the _best_ | 13:00 |
| sean-k-mooney | brb | 13:01 |
| *** markvoelker has joined #openstack-nova | 13:04 | |
| *** shilpasd has joined #openstack-nova | 13:04 | |
| shilpasd | Eric: finally after so many rechecks isoalte agg patches are merged, tnx for your extended support | 13:05 |
| shilpasd | Gibi: tnx for followup patch, review on isolate agg patches | 13:06 |
| efried | shilpasd: Your work on this feature is really appreciated. Please pass our compliments on to your colleagues as well. | 13:06 |
| *** mriedem has joined #openstack-nova | 13:08 | |
| shilpasd | dansmith: stephenfin: takashin: thanks for review | 13:08 |
| shilpasd | efried: sure thanks | 13:08 |
| *** markvoelker has quit IRC | 13:09 | |
| efried | stephenfin: First thing that jumps out is test_select_destination_with_pcpu_fallback_disabled, which is a test, is calling test_select_destination_with_4_3_client, which is also a test. I.e. both are going to run "simultaneously", one being a subset of the other. | 13:13 |
| efried | ...which should be fine, as long as neither is doing anything global | 13:13 |
| stephenfin | they don't _look_ like they're doing anything global | 13:14 |
| kashyap | print domain_caps[arch][machine_type]._os.loader.enums | 13:18 |
| kashyap | Oops | 13:18 |
| *** eharney has joined #openstack-nova | 13:18 | |
| kashyap | (Disregard that, please) | 13:18 |
| *** KeithMnemonic has joined #openstack-nova | 13:18 | |
| *** macz has joined #openstack-nova | 13:20 | |
| * efried adds to kashyap's fbi file | 13:21 | |
| kashyap | efried: Mired in parsing some XML bits and testing :D | 13:21 |
| efried | suuuure | 13:21 |
| kashyap | Not my day today... | 13:21 |
| efried | stephenfin: I agree. I'm looking through the other tests in that class; the problem could just as easily be caused by one of *them* doing something global. | 13:23 |
| *** macz has quit IRC | 13:25 | |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org will be offline briefly starting at 14:00 UTC (that's roughly 30 minutes from now) for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 13:29 | |
| *** beekneemech is now known as bnemec | 13:30 | |
| *** xek has joined #openstack-nova | 13:35 | |
| *** Luzi has quit IRC | 13:38 | |
| *** tbachman has joined #openstack-nova | 13:40 | |
| *** jangutter has quit IRC | 13:40 | |
| *** jangutter has joined #openstack-nova | 13:42 | |
| mriedem | gibi: a few comments in this bug fix of yours https://review.opendev.org/#/c/666857/ | 13:44 |
| stephenfin | efried: something is messing with flags | 13:44 |
| efried | stephenfin: Because CONF is global mebbe? | 13:44 |
| stephenfin | quite possibly, yeah | 13:45 |
| *** pcaruana has quit IRC | 13:45 | |
| stephenfin | 'CONF.workarounds.disable_fallback_pcpu_query' is intermittently 'False' despite me setting it to 'True' in the test | 13:45 |
| efried | stephenfin: I can reproduce reliably locally when I skip test discovery, and reliably *not* reproduce when I let discovery happen (which runs the tests slower). So definitely racy. | 13:48 |
| *** rcernin has quit IRC | 13:49 | |
| *** jangutter has quit IRC | 13:53 | |
| efried | stephenfin: Yeah, self.flags is just CONF.set_override | 13:54 |
| stephenfin | and CONF is global | 13:54 |
| efried | stephenfin: use oslo_config.fixture.Config | 13:55 |
| stephenfin | This is hardly the first time we've got positive and negative tests for something config opt-driven though, is it? | 13:55 |
| sean-k-mooney | well | 13:56 |
| sean-k-mooney | you wont have two tests running at the same time in the same process | 13:56 |
| sean-k-mooney | so its fine to set things with set flags | 13:56 |
| sean-k-mooney | as long as you alreay set the correct state up in the test | 13:56 |
| efried | sean-k-mooney: same process yes, same thread no, right? | 13:56 |
| sean-k-mooney | no | 13:56 |
| artom | Wait, has oslo_config.fixture.Config finally fixed the global CONF problem? I guess the fixture is not per-test, so no. | 13:57 |
| sean-k-mooney | tox runes tests in multiple processes not threads | 13:57 |
| stephenfin | well, stestr | 13:57 |
| efried | artom: ...no, it looks like it still by default uses global CONF. So much for that idea. | 13:57 |
| sean-k-mooney | yes point is each worker is tis own process that runs its own python interpreter | 13:58 |
| sean-k-mooney | and only 1 test is running in each worker at a time | 13:58 |
| sean-k-mooney | so test only share global state with other test that ran in that worker | 13:58 |
| stephenfin | we're using ConfFixture in our base test class in nova/test.py | 13:58 |
| sean-k-mooney | but only one test is running at a time | 13:58 |
| stephenfin | then wtf is happening here? :-D | 14:00 |
| sean-k-mooney | im just back form lunch so i can start runing it in a debugger | 14:00 |
| sean-k-mooney | ye have narrorwed it down to the config option yes? | 14:01 |
| stephenfin | That's what I'm seeing anyway | 14:01 |
| stephenfin | It should be True but it's False instead | 14:01 |
| sean-k-mooney | and ye are seeing it only when the test class is run | 14:01 |
| *** gbarros has joined #openstack-nova | 14:01 | |
| sean-k-mooney | rather then the test directly | 14:02 |
| *** markvoelker has joined #openstack-nova | 14:02 | |
| stephenfin | yeah, I'm seeing it intermittently but efried notes that it's consistent if you run without test discovery | 14:02 |
| efried | which gels with the problematic result, which is expecting from GET /a_c {normal results} and is getting {normal results + fallback results} | 14:02 |
| stephenfin | so 'tox -e py27 -- -n nova/tests/unit/scheduler/test_scheduler.py' | 14:02 |
| *** jawad_ax_ has quit IRC | 14:02 | |
| efried | stephenfin, sean-k-mooney: fwiw I'm running it from within the venv as | 14:03 |
| efried | stestr run -n nova.tests.unit.scheduler.test_scheduler.SchedulerManagerTestCase | 14:03 |
| efried | hitting it every time | 14:03 |
| sean-k-mooney | ya i can run the full file in the debugger too although im not sure if that will show anything specifically | 14:03 |
| efried | ^^ is just the one test class | 14:03 |
| -openstackstatus- NOTICE: The Gerrit service on review.opendev.org is offline briefly for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html | 14:05 | |
| *** ChanServ changes topic to "The Gerrit service on review.opendev.org is offline briefly for maintenance: http://lists.openstack.org/pipermail/openstack-discuss/2019-September/009064.html" | 14:05 | |
| sean-k-mooney | yep it failed for me too | 14:05 |
| mriedem | i haven't been following but is there a new gate failure? | 14:05 |
| mriedem | and if so, do we have a bug and e-r query tracking it? | 14:05 |
| sean-k-mooney | no | 14:05 |
| sean-k-mooney | its a intermitent failure of one of the tests in the PCPU series | 14:06 |
| *** pcaruana has joined #openstack-nova | 14:07 | |
| *** ttsiouts has quit IRC | 14:08 | |
| *** ttsiouts has joined #openstack-nova | 14:09 | |
| *** janki has quit IRC | 14:09 | |
| *** ttsiouts has quit IRC | 14:09 | |
| *** ttsiouts has joined #openstack-nova | 14:09 | |
| efried | stephenfin: So I've narrowed it down, by commenting out all the tests in the suite *except* test_select_destination_with_pcpu_fallback_disabled and test_select_destination_with_pcpu_fallback I still get the repro; commenting out either of those and I'm fine. | 14:10 |
| efried | I also factored out the 4_3 thing so the former ^ only runs the one test | 14:10 |
| efried | I also commented out the self.flags in the latter ^ because it's setting the default -- and still get the repro | 14:10 |
| efried | so those two tests are banging on each other somehow. | 14:10 |
| *** markvoelker has quit IRC | 14:11 | |
| sean-k-mooney | why are we calling a test method directly by the way instead fo factoring out the common code into a helper method | 14:11 |
| *** markvoelker has joined #openstack-nova | 14:12 | |
| efried | sean-k-mooney: that was the first thing I mentioned. But it doesn't make a difference, cause it's also the first thing I tried :P | 14:12 |
| stephenfin | sean-k-mooney: Because the theory was that the test end result and assertions should be identical under both circumstances | 14:13 |
| stephenfin | ditto | 14:13 |
| sean-k-mooney | yes but i was wonddering if we where sharing mocks by not doing it | 14:13 |
| stephenfin | possibly. I tried reseting the 'select_destinations' mock to no avail | 14:14 |
| artom | Why does discovery affect it, though? Discovery doesn't run any tests, does it? | 14:15 |
| stephenfin | I think it's just ordering | 14:15 |
| efried | no, it just makes the tests run much more quickly | 14:15 |
| efried | yeah | 14:15 |
| artom | Ah, so a timing issue | 14:15 |
| efried | it seems as though discovery even slows down the running of the tests themselves | 14:15 |
| efried | yes | 14:16 |
| sean-k-mooney | well its not a timeing issue when we are leaking shared state | 14:16 |
| artom | It causes thing to not run at the same time | 14:16 |
| sean-k-mooney | no | 14:16 |
| sean-k-mooney | but it may change the order | 14:16 |
| efried | I would have thought it would do all discovery, come up with a list of tests, and then run 'em, and by the time you got to that last thing it was the same as if you did no discovery. But clearly that's not how it's happening. | 14:16 |
| efried | yeah, or possibly ordering I guess | 14:16 |
| * efried tries... | 14:16 | |
| stephenfin | efried: I've noticed that all the other mock assertions pass too | 14:16 |
| stephenfin | so maybe it's not the global conf :-\ | 14:17 |
| *** markvoelker has quit IRC | 14:17 | |
| artom | stephenfin, but... you said you've observed an option being False when it should be True... | 14:18 |
| efried | stephenfin: waitwait | 14:18 |
| efried | you're saying GET /a_c is called the appropriate number of times?? | 14:18 |
| stephenfin | That's what I'm seeing. Put the 'select_destinations.assert_called_once_with' to the end | 14:19 |
| stephenfin | in select_destinations.assert_called_once_with | 14:19 |
| stephenfin | sorry, test_select_destination_with_4_3_client | 14:19 |
| sean-k-mooney | i have it open in the debuger now so ill check | 14:19 |
| stephenfin | assert_called_once_with checks that things are called exactly once, right/ | 14:19 |
| sean-k-mooney | yes its called once | 14:20 |
| sean-k-mooney | stephenfin: it check its called exactly once and has the correct args | 14:21 |
| sean-k-mooney | so yes | 14:21 |
| efried | I put a call_count assertion just in case | 14:21 |
| *** mjozefcz has quit IRC | 14:22 | |
| stephenfin | moving the mocks inline and using context managers instead of function decorators doesn't help | 14:23 |
| sean-k-mooney | if i put the select_destinations.assert_called_once_with( call at the end all the rest pass | 14:24 |
| sean-k-mooney | its failing in assert_called_with when its comparing the args | 14:27 |
| stephenfin | efried, sean-k-mooney: got it | 14:32 |
| efried | tell | 14:32 |
| *** BjoernT has joined #openstack-nova | 14:33 | |
| stephenfin | https://review.opendev.org/#/c/671801/50/nova/scheduler/manager.py@191 | 14:33 |
| stephenfin | we're using extend, which modifies a list in place | 14:33 |
| stephenfin | and our mock is returning 'fakes.ALLOC_REQS' | 14:33 |
| efried | oy vay | 14:34 |
| stephenfin | so that's getting modified by the non-disabled fallback test | 14:34 |
| efried | stephenfin: I think I saw those globals only used by this one test suite? | 14:34 |
| stephenfin | correct | 14:34 |
| efried | so make 'em instance vars, for future safety | 14:35 |
| stephenfin | wdym? | 14:35 |
| stephenfin | I was doing to do 'fakes.ALLOC_REQS[:]' | 14:35 |
| stephenfin | though I could add a 'get_fake_alloc_reqs' helper too | 14:36 |
| efried | I mean, you could deepcopy 'em to fix this problem, but it's just going to bite us in the ass again later, somewhere else. | 14:36 |
| efried | global test artifacts bad | 14:36 |
| stephenfin | v bad. | 14:36 |
| efried | first 205 lines of fakes, bad. | 14:36 |
| *** ChanServ changes topic to "Current runways: https://etherpad.openstack.org/p/nova-runways-train -- This channel is for Nova development. For support of Nova deployments, please use #openstack." | 14:37 | |
| -openstackstatus- NOTICE: The Gerrit outage portion of the current maintenance is complete and the service is back on line, however reindexing for renamed repositories is still underway and some Zuul job fixes are in the process of being applied | 14:37 | |
| *** macz has joined #openstack-nova | 14:37 | |
| efried | stephenfin: I guess for now you could just make it a helper method that returns a fresh new copy every time | 14:37 |
| efried | but if it were me, I would make it completely fresh | 14:38 |
| efried | def get_fake_alloc_reqs(): | 14:38 |
| efried | return { $everything } | 14:38 |
| efried | rather than | 14:38 |
| efried | EVIL_GLOBAL | 14:38 |
| efried | def get_fake_alloc_reqs(): | 14:38 |
| efried | return some_unreliable_copy_method(EVIL_GLOBAL) | 14:38 |
| stephenfin | gotcha | 14:39 |
| stephenfin | coming right up | 14:39 |
| efried | this is gonna be partway down the series, yah? | 14:39 |
| stephenfin | yeah, just before that patch. I'll do it separately | 14:39 |
| efried | stephenfin: you're going to have to fix the patch anyway, so might as well do it in place, nah? | 14:39 |
| efried | don't try to fix the race afterward | 14:40 |
| stephenfin | I meant add the helper function in a precursor patch and modify the intermittently failing patch to use it | 14:40 |
| efried | tbc, the patch in question is "Add support for translating CPU policy extra specs, image meta" | 14:40 |
| stephenfin | but I can combine too | 14:40 |
| efried | yeah, just combine, help me justify a fast approve | 14:41 |
| stephenfin | ack | 14:41 |
| *** TxGirlGeek has joined #openstack-nova | 14:47 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add support for translating CPU policy extra specs, image meta https://review.opendev.org/671801 | 14:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: fakelibvirt: Make 'Connection.getHostname' unique https://review.opendev.org/681060 | 14:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: libvirt: Mock 'libvirt_utils.file_open' properly https://review.opendev.org/681061 | 14:47 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Add reshaper for PCPU https://review.opendev.org/674895 | 14:47 |
| stephenfin | efried: ^ | 14:47 |
| stephenfin | I'll remove the rest of those global fakes now (separate patch) | 14:48 |
| efried | ++ | 14:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize volume create code during boot from volume https://review.opendev.org/682378 | 14:48 |
| mriedem | ^ is a simple refactor split off from https://review.opendev.org/#/c/541420/ which has been around since february of 2018, | 14:48 |
| mriedem | and is important if y'all ever want to drop the legacy volume attachment compat code | 14:49 |
| efried | stephenfin: +A, and re+W up the pile. Nice work, thank you. | 14:50 |
| *** belmoreira has joined #openstack-nova | 14:55 | |
| *** belmoreira has quit IRC | 14:55 | |
| *** belmoreira has joined #openstack-nova | 14:58 | |
| *** shilpasd has quit IRC | 14:58 | |
| stephenfin | mriedem: done | 14:59 |
| *** mlavalle has joined #openstack-nova | 14:59 | |
| efried | mriedem: one adjustment requested pls | 15:00 |
| mriedem | doing it | 15:00 |
| efried | alex_xu: yt? | 15:00 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Centralize volume create code during boot from volume https://review.opendev.org/682378 | 15:06 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | 15:06 |
| *** Sundar has joined #openstack-nova | 15:07 | |
| *** openstackgerrit has quit IRC | 15:08 | |
| *** xek has quit IRC | 15:08 | |
| efried | mriedem: are we merging stuff like that ^ at this point or waiting for ussuri to fork? | 15:09 |
| mriedem | which one? the refactor is trivial and i've added the latter to https://etherpad.openstack.org/p/nova-train-release-todo | 15:09 |
| *** BjoernT_ has joined #openstack-nova | 15:10 | |
| mriedem | as i said, it's been around forever without much core review outside melwitt | 15:10 |
| mriedem | the mox->mock stuff in the tests blew it all up | 15:10 |
| *** BjoernT has quit IRC | 15:10 | |
| mriedem | but if we ever want to migrate off the legacy volume attach code, we need to be creating all volumes with the new style attachment stuff | 15:10 |
| mriedem | iow, the longer we wait, the bigger the data migration is going to be | 15:10 |
| mriedem | e.g. https://review.opendev.org/#/c/549130/ | 15:11 |
| mriedem | i don't expect to get ^ into train at this point | 15:11 |
| mriedem | nor is it probably the only way to skin that cat | 15:11 |
| mriedem | at some point in the future we can add a nova-status upgrade check and fail to start if you haven't migrated old bdm records | 15:12 |
| mriedem | and then drop all that compat code | 15:12 |
| mriedem | melwitt: do you want to send this through? https://review.opendev.org/#/c/677736/ | 15:15 |
| melwitt | I do. thanks | 15:16 |
| *** ivve has quit IRC | 15:20 | |
| *** dtantsur|bbl is now known as dtantsur | 15:23 | |
| *** damien_r has quit IRC | 15:23 | |
| *** openstackgerrit has joined #openstack-nova | 15:26 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Follow up for the bandwidth series https://review.opendev.org/682389 | 15:26 |
| gibi | mriedem, efried: two smallish nits for the bandwidth series. I have two other testing enhancements on my TODO list that I plan to propose this week. None of these is critical, just nice to have. | 15:26 |
| mriedem | gibi: i got the fup | 15:27 |
| *** damien_r has joined #openstack-nova | 15:27 | |
| gibi | mriedem: thanks | 15:27 |
| *** damien_r has quit IRC | 15:27 | |
| gibi | mriedem: and thanks for the review on the bug fix https://review.opendev.org/#/c/666857/ I will respin that | 15:27 |
| bauzas | gibi: I can help you :) | 15:36 |
| *** lbragstad has joined #openstack-nova | 15:41 | |
| *** luksky has quit IRC | 15:44 | |
| *** gyee has joined #openstack-nova | 15:47 | |
| *** ccamacho has quit IRC | 15:52 | |
| *** TxGirlGeek has quit IRC | 15:52 | |
| gibi | bauzas: thanks | 15:56 |
| *** xek has joined #openstack-nova | 15:57 | |
| *** ccamacho has joined #openstack-nova | 15:58 | |
| *** ttsiouts has quit IRC | 15:58 | |
| *** cfriesen has joined #openstack-nova | 16:03 | |
| *** mjozefcz has joined #openstack-nova | 16:05 | |
| *** belmoreira has quit IRC | 16:08 | |
| *** markvoelker has joined #openstack-nova | 16:13 | |
| *** larainema has quit IRC | 16:14 | |
| *** lpetrut has quit IRC | 16:16 | |
| *** lpetrut has joined #openstack-nova | 16:16 | |
| *** slaweq has quit IRC | 16:17 | |
| *** slaweq has joined #openstack-nova | 16:17 | |
| *** markvoelker has quit IRC | 16:18 | |
| *** damien_r has joined #openstack-nova | 16:20 | |
| *** ccamacho has quit IRC | 16:27 | |
| *** lpetrut has quit IRC | 16:31 | |
| *** ociuhandu has quit IRC | 16:31 | |
| *** udesale has quit IRC | 16:32 | |
| *** markvoelker has joined #openstack-nova | 16:36 | |
| *** markvoelker has quit IRC | 16:40 | |
| *** ociuhandu has joined #openstack-nova | 16:45 | |
| *** jmlowe has quit IRC | 16:46 | |
| *** ozzzo has joined #openstack-nova | 16:47 | |
| *** ociuhandu has quit IRC | 16:50 | |
| *** TxGirlGeek has joined #openstack-nova | 16:53 | |
| *** JamesBenson has joined #openstack-nova | 16:56 | |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| *** derekh has quit IRC | 17:00 | |
| *** slaweq has quit IRC | 17:01 | |
| *** jmlowe has joined #openstack-nova | 17:04 | |
| *** gbarros has quit IRC | 17:05 | |
| *** slaweq has joined #openstack-nova | 17:08 | |
| *** dtantsur is now known as dtantsur|afk | 17:14 | |
| *** ralonsoh has quit IRC | 17:16 | |
| *** xek_ has joined #openstack-nova | 17:16 | |
| *** gbarros has joined #openstack-nova | 17:18 | |
| *** xek has quit IRC | 17:18 | |
| *** psachin has quit IRC | 17:25 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: objects: use all_things_equal from objects.base https://review.opendev.org/681397 | 17:26 |
| *** TxGirlGe_ has joined #openstack-nova | 17:26 | |
| *** mjozefcz has quit IRC | 17:27 | |
| *** TxGirlGeek has quit IRC | 17:28 | |
| openstackgerrit | Merged openstack/nova master: Deprecate the XenAPIDriver https://review.opendev.org/680732 | 17:39 |
| *** ccamacho has joined #openstack-nova | 17:53 | |
| *** jmlowe has quit IRC | 17:57 | |
| *** gbarros has quit IRC | 18:13 | |
| openstackgerrit | Merged openstack/nova master: libvirt: Fix service-wide pauses caused by un-proxied libvirt calls https://review.opendev.org/677736 | 18:20 |
| *** luksky has joined #openstack-nova | 18:24 | |
| sean-k-mooney | cool that ^ will make mdbooth happy | 18:25 |
| sean-k-mooney | by the way can i get a second +2+w on https://review.opendev.org/#/c/670585/ | 18:27 |
| sean-k-mooney | i really want to ensure that is deprecated in train so we can remove it in Ussuri | 18:27 |
| sean-k-mooney | as in like 2 weeks if not sooner | 18:27 |
| *** ociuhandu has joined #openstack-nova | 18:28 | |
| sean-k-mooney | dansmith: mriedem: melwitt: could one of ye take a look when ye have time ^ | 18:29 |
| *** ociuhandu has quit IRC | 18:36 | |
| *** markvoelker has joined #openstack-nova | 18:37 | |
| *** munimeha1 has joined #openstack-nova | 18:41 | |
| *** markvoelker has quit IRC | 18:42 | |
| *** xek has joined #openstack-nova | 18:42 | |
| *** xek_ has quit IRC | 18:45 | |
| *** openstackgerrit has quit IRC | 18:52 | |
| *** ozzzo has quit IRC | 18:56 | |
| *** nweinber has joined #openstack-nova | 18:56 | |
| *** Sundar has quit IRC | 19:07 | |
| *** zhubx has joined #openstack-nova | 19:10 | |
| *** boxiang has quit IRC | 19:12 | |
| *** ozzzo has joined #openstack-nova | 19:22 | |
| *** ccamacho has quit IRC | 19:31 | |
| *** mmethot_ has quit IRC | 19:41 | |
| *** mmethot_ has joined #openstack-nova | 19:42 | |
| *** TxGirlGe_ has quit IRC | 19:45 | |
| *** gbarros has joined #openstack-nova | 19:45 | |
| *** igordc has joined #openstack-nova | 19:51 | |
| *** TxGirlGeek has joined #openstack-nova | 20:02 | |
| mriedem | sean-k-mooney: commented, | 20:03 |
| mriedem | and added moshe and adrianc to it | 20:03 |
| *** mmethot_ has quit IRC | 20:04 | |
| sean-k-mooney | thanks. if stephen does not adress the docs bugs ill do them tomorrow | 20:04 |
| *** mmethot_ has joined #openstack-nova | 20:04 | |
| sean-k-mooney | mriedem: assuming no objection from moshe or adrianc are you generally ok with this? | 20:05 |
| mriedem | i can't say i'm very familiar with the pci passthrough whitelist dev name stuff | 20:06 |
| mriedem | or how much it's used | 20:06 |
| sean-k-mooney | its an alternitive to useing the pci addres of vendor id and product id | 20:07 |
| sean-k-mooney | the issue with is if you restart the compute agnet while its passed to a vm the device wont be found | 20:07 |
| sean-k-mooney | we recently prevented the compute agent from removeing in use device from the db | 20:08 |
| efried | mriedem: guessing that az fail is yet another global test var race. We should go on a crusade to murder all of those. | 20:08 |
| sean-k-mooney | but we didnt always, also the name can change after you release teh VF/PF form teh vm | 20:08 |
| sean-k-mooney | so the main issue is its not realible and we keep getting bug downstream that we cant fix | 20:09 |
| sean-k-mooney | so we just want to stop supproting it | 20:09 |
| mriedem | efried: is this a recent failure? b/c i'm not sure what is global about this one and it's been around awhile (the test that is) | 20:10 |
| efried | mriedem: from what I could tell, it started hitting on 9/10 | 20:10 |
| efried | mriedem: I'm looking at the delta in test_aggregates.py for I9ab9d7d65378be564b3731b5227ede8cece71bef | 20:10 |
| efried | https://review.opendev.org/#/c/671075/21/nova/tests/functional/test_aggregates.py | 20:11 |
| *** mmethot_ has quit IRC | 20:12 | |
| *** mmethot_ has joined #openstack-nova | 20:12 | |
| mriedem | hmm, yeah probably related to that series | 20:13 |
| artom | sean-k-mooney, https://review.opendev.org/#/c/682435/3 | 20:13 |
| efried | never mind, that one only merged yesterday. | 20:14 |
| artom | Which mean that upstream whitebox has now "caught up" and can be used in tests | 20:14 |
| efried | and its predecessor the day before | 20:14 |
| efried | so that's not the culprit. | 20:14 |
| mriedem | efried: so from the error it looks like the actual failure is not really the issue, | 20:14 |
| mriedem | it's that we failed to put the instance on the host | 20:14 |
| mriedem | b/c of the az filter | 20:14 |
| efried | yup | 20:14 |
| artom | Next step: update https://review.opendev.org/#/c/656890/ to move whitebox under openstack-qa/ | 20:14 |
| sean-k-mooney | artom: ok you didnt need to merge them for me to test with them | 20:14 |
| sean-k-mooney | but ok | 20:14 |
| * artom runs off to pick up daughter | 20:14 | |
| mriedem | the boot server function is just waiting for the server to exit BUILD status, which it does when it changes to ERROR status | 20:14 |
| *** luksky has quit IRC | 20:16 | |
| *** luksky has joined #openstack-nova | 20:17 | |
| mriedem | efried: so if i had to guess, | 20:23 |
| mriedem | changing the base class on the test to ProviderUsageBaseTestCase but not cleaning up a bunch of the duplicate setup means we have multiple fixtures and multiple copies of the same conductor/api/scheduler services, and the api syncs informatoin to the scheduler about aggregates, which is probably not multi-scheduler aware in the functional tests, | 20:24 |
| mriedem | so it's probably intermittent b/c sometimes we hit hte scheduler process that knows about the az metadata and sometimes we don't | 20:24 |
| efried | mm. so why hitting since the 10th | 20:25 |
| mriedem | i can put up a patch to cleanup that duplicate setup | 20:25 |
| mriedem | efried: the first change to hit it in logstash was the one you pointed at | 20:25 |
| *** nweinber has quit IRC | 20:25 | |
| mriedem | 671075 | 20:25 |
| mriedem | and then it was just a recheck grind | 20:26 |
| efried | yeah... | 20:26 |
| mriedem | see how many times someone named Eric Fried blindly rechecked it :) | 20:26 |
| mriedem | i'll put up a patch | 20:26 |
| efried | If only I had full debuggability from my phone. | 20:26 |
| mriedem | or make shilpa responsible for investigating the failures and rechecking them | 20:27 |
| mriedem | but whatever, done is done | 20:27 |
| *** nweinber has joined #openstack-nova | 20:28 | |
| efried | mriedem: looks to me like everything (except the super()) through the self.computes = {} can be removed, yah? | 20:30 |
| mriedem | yup, just did it, running the tests | 20:30 |
| efried | k, just prepping for review :P | 20:31 |
| efried | +2 "that's how I would have done it" | 20:31 |
| *** TxGirlGeek has quit IRC | 20:35 | |
| *** ociuhandu has joined #openstack-nova | 20:36 | |
| efried | tests pass locally for me | 20:37 |
| *** openstackgerrit has joined #openstack-nova | 20:37 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475 | 20:37 |
| efried | mriedem: two more lines | 20:38 |
| *** markvoelker has joined #openstack-nova | 20:38 | |
| *** pcaruana has quit IRC | 20:38 | |
| mriedem | ack | 20:38 |
| efried | mriedem: why only partial-bug? | 20:39 |
| mriedem | b/c i don't know if it's the root fix | 20:40 |
| mriedem | hence the "at least rule it out" comment | 20:40 |
| mriedem | if we see the hits drop off in e-r by next week we can close the bug | 20:40 |
| efried | this way requires us to remember to go back at the bug :) | 20:40 |
| efried | but okay | 20:40 |
| mriedem | i will when i cleanup old e-r queries | 20:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove redundancies from AggregateRequestFiltersTest.setUp https://review.opendev.org/682475 | 20:41 |
| *** ociuhandu has quit IRC | 20:41 | |
| efried | thanks for the quick fix mriedem <high five> | 20:42 |
| mriedem | <down low> | 20:43 |
| *** markvoelker has quit IRC | 20:43 | |
| mriedem | i wonder if the fake messaging driver doesn't do rpc fanout cast properly, if we should have some sort of fixture that blows up if you try to start more than one non-nova-compute service in a functional test | 20:45 |
| mriedem | since that's twice in 2 weeks that we've had some issue from functional test inheritance doubling up on fixtures and stuff | 20:45 |
| sean-k-mooney | are you suggestion on;ly allowing 1 comptue service in the fucntional tests | 20:46 |
| efried | no | 20:46 |
| sean-k-mooney | or havign a fixture to detect when you have more then one but have not set things up correctly | 20:46 |
| efried | he's suggesting allowing only one *non* nova-compute service. | 20:46 |
| *** BjoernT_ has quit IRC | 20:46 | |
| efried | we clearly need multiple compute services for many tests. | 20:46 |
| efried | but we should never need more than one superconductor | 20:47 |
| sean-k-mooney | oh so only one conductor or scheduler | 20:47 |
| efried | right | 20:47 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove SchedulerReportClient from AggregateRequestFiltersTest https://review.opendev.org/682480 | 20:47 |
| *** trident has quit IRC | 20:47 | |
| mriedem | correct, one controller service | 20:47 |
| mriedem | one of each | 20:47 |
| *** markvoelker has joined #openstack-nova | 20:49 | |
| mriedem | i think multiple apis and conductors are probably ok since those are stateless | 20:50 |
| mriedem | but the scheduler has some stateful crap in the HostManager | 20:51 |
| mriedem | like aggregate info | 20:51 |
| *** xek has quit IRC | 20:52 | |
| mriedem | but yeah the test was definitely starting 2 schedulers | 20:52 |
| mriedem | b'2019-09-16 15:10:19,491 INFO [nova.service] Starting scheduler node (version 19.1.0)' | 20:52 |
| mriedem | b'2019-09-16 15:10:20,156 INFO [nova.service] Starting scheduler node (version 19.1.0)' | 20:52 |
| *** eharney has quit IRC | 20:53 | |
| *** trident has joined #openstack-nova | 20:59 | |
| *** markvoelker has quit IRC | 21:00 | |
| efried | mriedem: what was the other bug with the duplicated services? | 21:01 |
| mriedem | duplicated placement fixture | 21:01 |
| mriedem | sec | 21:01 |
| mriedem | https://github.com/openstack/nova/commit/5e1b096894f6de4cfbca254cf74dcfcf56358ea5#diff-5befe429f14247314e6ca487aa4e13bd | 21:01 |
| efried | thank you sir | 21:02 |
| efried | I'm posting a poison patch now | 21:02 |
| mriedem | heh i was just writing my commit message ofr one | 21:02 |
| mriedem | *for | 21:02 |
| efried | oh, no bug for I057a07c8d0b880c8d09fc2e618ce1f7fc885beda ? | 21:02 |
| mriedem | guess not | 21:02 |
| efried | hm, if that one was placement fixture, my fix isn't going to hit it. | 21:03 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Do not allow mutiple nova-scheduler workers https://review.opendev.org/682485 | 21:03 |
| mriedem | efried: ^ is what i was thinking of | 21:04 |
| efried | whoah | 21:04 |
| efried | Mine is way dumber, but more complete, and faster. | 21:04 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Only allow one non-compute service in tests https://review.opendev.org/682486 | 21:05 |
| efried | mriedem: ^ | 21:05 |
| *** JamesBenson has quit IRC | 21:06 | |
| *** lpetrut has joined #openstack-nova | 21:06 | |
| mriedem | you're going to at least have failures on tests that intentionally start multiple api fixtures with different projects and roles (admin vs non-admin, project1 and projec2 for filtering, etc) | 21:09 |
| sean-k-mooney | if we have multi cell tests then those would also fail correct | 21:10 |
| *** panda|ruck has quit IRC | 21:10 | |
| efried | yeah, I guess I thought the cell conductors would have different names, but that doesn't make sense. | 21:11 |
| sean-k-mooney | so we need a way to allow it | 21:11 |
| mriedem | the api/conductor/scheduler services don't get registered in the cell dbs | 21:11 |
| mriedem | so i don't think that matters here | 21:11 |
| *** nweinber has quit IRC | 21:11 | |
| mriedem | but i'm also not sure how sophisticated our fixture stuff is to know where to create those services (they should go into cell0 but might not) | 21:12 |
| mriedem | the CellDatabase fixture has a default of cell1 so the stuff probably just gets created there | 21:12 |
| mriedem | sean-k-mooney: also btw i added https://review.opendev.org/#/c/670585/2 to https://review.opendev.org/#/c/670585/2 https://etherpad.openstack.org/p/nova-train-release-todo | 21:12 |
| mriedem | so we don't forget | 21:12 |
| sean-k-mooney | ok cool thanks. it would not be the end of the world if it sliped but im hopeing that we can drop supprot for this downstream before our next LTS | 21:13 |
| sean-k-mooney | so in 24-18 months when people start using it we dont have to support it | 21:14 |
| *** lpetrut has quit IRC | 21:14 | |
| * sean-k-mooney is surprised how many people are still running newton and wont upgrade | 21:14 | |
| * sean-k-mooney and expect it to work and get new feature | 21:15 | |
| artom | sean-k-mooney, are you really though? :P | 21:15 |
| sean-k-mooney | only by the expect it to work and get new features bit | 21:15 |
| sean-k-mooney | they can choose one but not both if they want something that old | 21:16 |
| *** panda has joined #openstack-nova | 21:16 | |
| mriedem | they expect you to make it work and provide new features when they shovel out the $$$ to your sales guys | 21:17 |
| mriedem | and gals | 21:17 |
| mriedem | and then the shit rolls down hill to you, the developer | 21:17 |
| sean-k-mooney | ya well we try to do both but when you cant backport api,db or object changes addinf features that arent bug fixs to old release is | 21:18 |
| sean-k-mooney | challanging | 21:18 |
| efried | mriedem: I don't see anywhere we start cell conductor services. What would that look like? | 21:19 |
| efried | Or do we do it implicitly when we start a compute service in a specific cell? | 21:20 |
| sean-k-mooney | if we are starting only one conductor the we are running it in the non super conductor toplogy | 21:20 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Only allow one scheduler service in tests https://review.opendev.org/682486 | 21:21 |
| sean-k-mooney | i dont know if we create multiple conductors in the current fucntional test but the cross cell migration code might have some? | 21:21 |
| efried | we don't have any cross-cell migration code | 21:22 |
| efried | tssssssss | 21:22 |
| sean-k-mooney | i ment there might be case in mriedem series | 21:23 |
| *** gbarros has quit IRC | 21:23 | |
| sean-k-mooney | we might be relying on https://github.com/openstack/nova/blob/master/nova/tests/functional/integrated_helpers.py#L118 | 21:24 |
| mriedem | efried: we don't do the whole super conductor / cell conductor thing in functional tests | 21:24 |
| mriedem | it's all just one | 21:24 |
| sean-k-mooney | to start the conductor in the functional test | 21:24 |
| mriedem | efried: meaning we get away with shit in functional tests that we wouldn't in devstack | 21:25 |
| mriedem | e.g. we do stuff like have the CheatingSerializer so it all looks like one RPC | 21:25 |
| sean-k-mooney | i think devstack has the abiltiy to deploy in the legacy mode too but i dont know if thats tested still so it could be broken | 21:25 |
| mriedem | which has been a problem in my cross-cell series with functional tests | 21:25 |
| mriedem | sean-k-mooney: all grenade jobs are non-superconductor | 21:26 |
| sean-k-mooney | ah ok. is that still true for the new zuulv3 version? | 21:26 |
| mriedem | i would expect so but haven't looked | 21:27 |
| sean-k-mooney | i tried using that to do a greade version of my numa jobs but i found i could not set local.conf diffrently for new/old nodes | 21:27 |
| sean-k-mooney | localrc yes | 21:27 |
| sean-k-mooney | but not local.conf | 21:27 |
| sean-k-mooney | so i could not do the config overrides i needed | 21:27 |
| sean-k-mooney | to be fair i dont know if normal greande supports that but it was something i found when i treid to use it | 21:28 |
| *** markvoelker has joined #openstack-nova | 21:35 | |
| sean-k-mooney | mriedem: by the way im going to spend some time this week trying to refine the nfv job into something we can have more permentaly and aslo creating an ovs-dpdk job. is this something we would consider merging before RC1 or shoudl i target U with the hope of maybe backporting. | 21:37 |
| sean-k-mooney | initally i want to run them as periodic jobs/via experimental but if i get multiple provider i would like to consier addign them to check if they seam stable | 21:38 |
| mriedem | i would say don't get the cart (backports) before the horse (actually getting something working on master) | 21:38 |
| sean-k-mooney | well i had a dpdk job working w while ago then it broke because fedora | 21:39 |
| sean-k-mooney | https://review.opendev.org/#/c/656580/ | 21:39 |
| *** Sundar has joined #openstack-nova | 21:39 | |
| sean-k-mooney | and the nfv job works. but i want to get them working properly before going anywher near nova check | 21:40 |
| sean-k-mooney | hence start with a nightly periodic job | 21:40 |
| *** markvoelker has quit IRC | 21:45 | |
| efried | mriedem: any chance of nailing down test_walk_versions this week? | 21:46 |
| *** rcernin has joined #openstack-nova | 21:47 | |
| mriedem | efried: i lost track of that | 21:47 |
| mriedem | weren't you trying to pull the mysql logs or something for debug? | 21:48 |
| efried | yeah, I think I eventually succeeded in doing that but they didn't show anything interesting. | 21:48 |
| efried | mriedem: https://review.opendev.org/#/c/678051/ | 21:48 |
| sean-k-mooney | look like the pcpu reshap went into merge conflict with somehting | 21:49 |
| efried | mriedem: the bad one has this additional message https://zuul.opendev.org/t/openstack/build/f9b92d66ade145a195b996708cd66c28/log/mysql/error.log#202 | 21:50 |
| efried | and a few more of the final 'Aborted connection' lines | 21:50 |
| efried | but otherwise they look the same to me. | 21:50 |
| efried | sean-k-mooney: what makes you say that? The yellow dot? | 21:50 |
| efried | oh | 21:50 |
| sean-k-mooney | no the "Patch in Merge Conflict" | 21:51 |
| efried | well wtf | 21:51 |
| *** TxGirlGeek has joined #openstack-nova | 21:51 | |
| efried | nothing has merged lately | 21:51 |
| sean-k-mooney | we merged a few patches like an hour ago | 21:51 |
| mriedem | i've noticed sometimes gerrit will say something is in merge conflict and then i have a triviall conflict-free local rebase | 21:51 |
| sean-k-mooney | proably the service wide pause thing | 21:52 |
| sean-k-mooney | ya | 21:52 |
| sean-k-mooney | shall i pull it down and rebase against master? | 21:52 |
| efried | no | 21:53 |
| sean-k-mooney | ok | 21:53 |
| efried | then we lose all of our +Vs on the rest of the series. | 21:53 |
| efried | Next time a bottom one fails in the gate and -2s the whole pile, maybe we can do that. | 21:53 |
| sean-k-mooney | i was just about to check that | 21:53 |
| efried | mriedem: So should we be getting all those "Aborted connection" things in the first place? | 21:54 |
| mriedem | efried: that i don't know | 21:54 |
| efried | does that mean we're not shutting down the db gracefully? | 21:54 |
| efried | and regardless, should that matter? | 21:54 |
| mriedem | the wacky names are just the random db names | 21:54 |
| sean-k-mooney | there are not patches in the gate for nova. they are all in check. but ill leave it wait till the morning | 21:54 |
| efried | is this a job for zzzeek? | 21:54 |
| mriedem | idk about that, but zzzeek might be able to tell us what to turn on for debugging | 21:55 |
| efried | sean-k-mooney: right, about 12 of the 15 patches have cleared the check queue at this point. The bottom one is waiting; once it completes, the whole swatch will get shoved into the gate pipeline. | 21:55 |
| mriedem | we're also using some deprecated opportunistic db test fixture stuff from oslo.db and i don't know if moving off the deprecated stuff would help us | 21:55 |
| efried | mriedem: well, couldn't hurt. What are you talking about specifically? | 21:55 |
| mriedem | it's been awhile since i looked, sec | 21:56 |
| sean-k-mooney | ya ok then let hold off untill the patches below it have merged | 21:57 |
| efried | sean-k-mooney: and tbh I'm not super worried about a merge conflict on that 15th patch. We've got at least a dozen rechecks to go before we get there. | 21:57 |
| sean-k-mooney | ... hopefully not but we do need to kill some of the random failures | 21:58 |
| *** slaweq has quit IRC | 21:58 | |
| efried | well hopefully we killed two of them today | 21:58 |
| sean-k-mooney | ya and mriedem killed 1?2? yesterday | 21:58 |
| *** eharney has joined #openstack-nova | 21:58 | |
| efried | I took a couple of samples this morning and determined that the Nth patch takes ~N rechecks to merge. | 21:58 |
| sean-k-mooney | firday i cant rememebr | 21:58 |
| sean-k-mooney | :( | 21:59 |
| efried | mriedem: test_fixtures.PostgresqlOpportunisticFixture and test_fixtures.MySQLOpportunisticFixture ? | 21:59 |
| mriedem | efried: was fixed already https://review.opendev.org/#/c/609352/ | 22:00 |
| *** gbarros has joined #openstack-nova | 22:00 | |
| efried | okay | 22:01 |
| sean-k-mooney | mriedem: by the way i got an error running the sql migate unit tests http://paste.openstack.org/show/776829/ | 22:02 |
| sean-k-mooney | given the rest passed im going to push this soon but shoudl tox not install that | 22:03 |
| sean-k-mooney | i guess it could have moved | 22:03 |
| sean-k-mooney | i assume ibm_db_sa is a sqlalcheme pluging for ibm db backend and its just not installled by tox for some reason | 22:04 |
| mriedem | umm, that should just be skipped | 22:05 |
| mriedem | well, or, | 22:06 |
| mriedem | tox should have pulled it in https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py2.txt#L1 | 22:06 |
| sean-k-mooney | the py27 version allso requries me to install the mysql hearder to compile the client... | 22:06 |
| mriedem | https://github.com/openstack/sqlalchemy-migrate/blob/master/test-requirements-py3.txt#L1 | 22:06 |
| sean-k-mooney | i was runnnig py36] | 22:06 |
| sean-k-mooney | which also should install it | 22:07 |
| mriedem | right | 22:07 |
| mriedem | i don't have that problem locally | 22:07 |
| mriedem | are you hitting some pypi mirror? | 22:07 |
| sean-k-mooney | no | 22:07 |
| sean-k-mooney | but i can recreat the env and see what happens | 22:08 |
| *** spatel has joined #openstack-nova | 22:08 | |
| sean-k-mooney | the rest of the test pass fine | 22:08 |
| sean-k-mooney | i assume they just use sqlite | 22:08 |
| sean-k-mooney | thats interesting its not in pip freeze in the tox env | 22:10 |
| sean-k-mooney | oh there is not explcit py36 target | 22:11 |
| sean-k-mooney | and it wont look for requirement-py3.txt by defualt | 22:11 |
| sean-k-mooney | i might update the tox file in a sperate patch. | 22:12 |
| *** spatel has quit IRC | 22:12 | |
| efried | mriedem: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/testr_results.html.gz looks like the root cause is some kind of timeout. If we could figure out where, we could probably extend it a bit and make the problem go away...? | 22:14 |
| efried | mmph, it's a socket read, probably a real deadlock. | 22:15 |
| mriedem | sean-k-mooney: do you have this? https://review.opendev.org/#/c/659388/ | 22:17 |
| efried | actually, it's the test timeout | 22:17 |
| mriedem | efried: right, | 22:17 |
| mriedem | that's a red herring | 22:17 |
| sean-k-mooney | mriedem: apparently not but i cloned this on friday | 22:18 |
| mriedem | i think we hit some db failure, switch eventlet context, and then never fail the test outright but it times out | 22:18 |
| mriedem | sean-k-mooney: that migrate patch merged awhile back | 22:18 |
| sean-k-mooney | mriedem: oh i cloned it form github so look like github sync for this is broken | 22:18 |
| mriedem | sean-k-mooney: did you clone from github or opendev? | 22:18 |
| efried | the time stamps are pretty funky on the log messages leading up to the exception. | 22:18 |
| mriedem | efried: yup | 22:18 |
| mriedem | sean-k-mooney: yeah you need opendev | 22:19 |
| efried | nearly 12 minutes elapse | 22:19 |
| sean-k-mooney | ya ok ill rebase | 22:19 |
| sean-k-mooney | am should i still clean up the tox file seperatly | 22:19 |
| mriedem | efried: b/c https://github.com/openstack/nova/blob/master/nova/tests/unit/db/test_migrations.py#L72 | 22:19 |
| sean-k-mooney | we done actully still support py 26 or py 33/34 right | 22:19 |
| sean-k-mooney | *don't | 22:19 |
| mriedem | sean-k-mooney: how about you clone from the current repo before asking questoins | 22:20 |
| efried | 4 * 160 = 11m40s, yup, that's right on. | 22:20 |
| mriedem | the timeouts on those were bumped a couple of years ago for a valid reason at the time (slow nodes i think) | 22:20 |
| *** mlavalle has quit IRC | 22:20 | |
| mriedem | https://github.com/openstack/nova/commit/71d6333f855e139894f497fc120487895a1d66ce | 22:20 |
| mriedem | but the thread switch thing kills us | 22:21 |
| *** gbarros has quit IRC | 22:22 | |
| efried | this is always on limestone-regionone?? | 22:23 |
| efried | that... seems like a thing | 22:25 |
| efried | is it because limestone has slower nodes consistently? | 22:25 |
| efried | or because of some particular config over there? | 22:25 |
| sean-k-mooney | are the limesotone jobs gernerally slower or is it just this one thing that is failing | 22:25 |
| efried | could we... increase the timeout s'more and see if it goes away? Not the greatest answer, but overall gate time wasted would be less | 22:25 |
| sean-k-mooney | of the over all job is slow then it coudl be that its just slower in general | 22:26 |
| efried | sean-k-mooney: I dunno, before digging into that I was going to see if anyone knew off the top. | 22:26 |
| efried | not sure how I would look into that, other than clicking through a zillion different builds | 22:26 |
| mriedem | yes it's always limestone | 22:26 |
| mriedem | idk if they are slower or what, you'd have to ask in infra | 22:26 |
| *** markvoelker has joined #openstack-nova | 22:26 | |
| mriedem | there is probably some way to find average job time per node provider and nova openstack-tox-py36 job or something | 22:27 |
| sean-k-mooney | infra have many grapha dashborads that track things but i dont know if they yave abuild time per regoin one | 22:27 |
| sean-k-mooney | you could proably figure it out form logstash | 22:27 |
| mriedem | i think awhile back i had wondered if those nodes used some different mysql binary or something | 22:27 |
| mriedem | for everything to be not crazy though it should be the same ubuntu 18.04 image and such regardless of node provider | 22:28 |
| sean-k-mooney | they should all use the nodepool image and then hit the infra mirros which are avaible via afs in all regions right | 22:28 |
| sean-k-mooney | so it shoudl be the same | 22:28 |
| efried | http://zuul.openstack.org/builds?job_name=openstack-tox-py27&branch=master&project=openstack%2Fnova but this table doesn't show the node provider... | 22:29 |
| sean-k-mooney | i think infra has been uploading its own image nightly since the zuul v2 days to not have random things break in diffrent clouds | 22:29 |
| logan- | o/ from limestone | 22:30 |
| efried | this slow bastard happened on rax https://zuul.opendev.org/t/openstack/build/38181c3314284545943b38b52c72f8d7/log/job-output.txt | 22:30 |
| sean-k-mooney | logan- they are trying to debug a test that seam to be hitting a timeout | 22:31 |
| sean-k-mooney | logan-: but only (mostly?) happens in limestone-regionone | 22:32 |
| alex_xu | efried: I'm here | 22:33 |
| sean-k-mooney | logan-: so we were wondering if limesotne is generally slower the other providre and hiting the timeout more often or if it could be a config/could issue | 22:33 |
| sean-k-mooney | logan-: e.g. is there something differnet that could make it fail more often on limestone or it that just a red herring | 22:33 |
| efried | alex_xu: Can you, or Rui, or luyao, or someone please formulate a bullet for vpmem for the train cycle highlights in a followon to https://review.opendev.org/#/c/681943/ ? | 22:33 |
| alex_xu | efried: got it, will do today | 22:34 |
| efried | alex_xu: If you seed the message, I'll scrub the grammar etc. Thanks. | 22:34 |
| alex_xu | efried: thanks | 22:34 |
| logan- | i'm not sure if our testcloud is slower than other nodepool providers. certainly the image would be the same as any other nodepool node. looking thru the log.. i'm curious if the job hangs somewhere specific, or if its just slower in general. and if our nodes are just showing up slower on average i'd like to know so we can figure out if it is h/w or config related. | 22:35 |
| efried | logan-: The thing we're trying to nail down is http://logstash.openstack.org/#/dashboard/file/logstash.json?query=message:%5C%22sqlalchemy.exc.InterfaceError:%20(pymysql.err.InterfaceError)%20(0,%20'')%5C%22%20AND%20tags:%5C%22console%5C%22%20AND%20voting:1&from=864000s | 22:36 |
| efried | we consistently see the test suite time out on the same 1-3 tests | 22:36 |
| *** macz has quit IRC | 22:37 | |
| *** macz has joined #openstack-nova | 22:38 | |
| efried | logan-: and if you punch the `node_provider` checkbox on the left-hand side next to the result list, you see it's *always* limestone-regionone | 22:39 |
| *** avolkov has quit IRC | 22:40 | |
| logan- | https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/logs_51/678051/4/check/nova-tox-collect-py37/f9b92d6/mysql/error.log are any of these messages at the bottom pertinent? | 22:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | 22:42 |
| mriedem | logan-: we're not sure, we need zzzeek's help for that | 22:42 |
| mriedem | those are definitely related to the opportunistic db tests that fail, | 22:43 |
| mriedem | the db is a random db name created by the test fixture, | 22:44 |
| mriedem | openstack_citest is setup by a script on the node | 22:44 |
| mriedem | https://github.com/openstack/nova/blob/master/tools/test-setup.sh does the mysql/postgresql setup | 22:44 |
| efried | logan-: this might be what you were already looking at, but I did a compare of the mysql error log on a "good" run and a "bad" one, and didn't see significant differences. | 22:45 |
| *** munimeha1 has quit IRC | 22:45 | |
| efried | logan-: see bottom-most comment on https://review.opendev.org/#/c/678051/ | 22:45 |
| logan- | yep, that's what i was wondering, thanks | 22:45 |
| efried | although the bad one is taking way longer... | 22:47 |
| efried | but dunno if that's cause or effect | 22:47 |
| logan- | to answer the general question -- I don't know of anything significantly different on our nodes. they're pretty standard dual proc intel systems with ssds. I wonder if there's something in particular with this test that is wrecking the node somehow. | 22:47 |
| efried | mhm, mriedem here's an interesting data point: in the "good" one, all those "abort" messages come through in under a minute. In the "bad" one, the first 16 or so are under a minute, then we start getting long delays. | 22:48 |
| efried | now, IIUC, everything before those "abort" messages is just setup, and the "abort" things are happening during the actual meat of the test. So I guess that's not really surprising. | 22:49 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Create volume attachment during boot from volume in compute https://review.opendev.org/541420 | 22:50 |
| efried | It would be painful, but maybe if we knew what operations were being run, it would give us a clue as to what's happening when it starts to grind down. | 22:53 |
| *** tkajinam has joined #openstack-nova | 22:54 | |
| efried | anybody know how to turn on like trace logs to see the sql statements themselves as they're issued? | 22:54 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: DNM: Stop using volume_api.initialize_connection https://review.opendev.org/682508 | 22:54 |
| mriedem | efried: i want to say https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_debug or | 22:55 |
| mriedem | https://docs.openstack.org/nova/latest/configuration/config.html#database.connection_trace | 22:55 |
| alex_xu | efried: https://review.opendev.org/682509 | 22:57 |
| efried | alex_xu: cool, thank you. | 22:58 |
| alex_xu | efried: np! | 22:58 |
| efried | alex_xu: since this is market-y, we probably don't want to mention the generic resource framework | 22:58 |
| alex_xu | efried: ah, got it | 22:58 |
| efried | that's a thing only devs would care about, at least until it reaps perf benefits via placement, which is a ways off. | 22:58 |
| efried | alex_xu: perhaps kill that sentence and instead explain a bit more what vpmem is | 22:59 |
| alex_xu | efried: sev mentioned that, that is why I follow | 22:59 |
| efried | oh? looking... | 22:59 |
| efried | alex_xu: yeah, I'm looking for something corresponding to "to protect users against attackers or rogue administrators snooping on | 23:00 |
| efried | their workloads when using the libvirt compute driver" | 23:00 |
| alex_xu | efried: got it | 23:00 |
| alex_xu | mriedem: you are super fast | 23:00 |
| efried | So like "Added $feature for $benefit" | 23:00 |
| * mriedem holsters gun | 23:01 | |
| mriedem | "Added VPMEMs for Intel to sell hardware." | 23:01 |
| mriedem | did it for you :) | 23:01 |
| sean-k-mooney | add vpmem so the nsa can keep allyour data in "ram" to track you faster | 23:02 |
| mriedem | "added vpmems so alex can continue working upstream" | 23:02 |
| mriedem | i kid i kid | 23:03 |
| * mriedem has already made alex angry by 7am | 23:03 | |
| mriedem | i think as long as you have HPC in there somewhere it will hit enough marketing bells to satisfy the entry | 23:04 |
| mriedem | "Support virtual persistent memory devices for HPC workloads when using the libvirt driver.": | 23:04 |
| mriedem | something like that | 23:04 |
| efried | "edge" | 23:04 |
| sean-k-mooney | the primary usecase is really something like "vPMEM support was intoduced to allow big data workloads to retain more data in persitent memroy reduceing the total cost of onership for big data clouds" | 23:04 |
| sean-k-mooney | ya hpc is the other vertical that will likely use it most | 23:05 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | 23:05 |
| openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | 23:05 |
| efried | mriedem, logan-: Howza ^ | 23:05 |
| mriedem | "other vertical"? someone get this guy a c-suite office | 23:05 |
| mriedem | efried: it's dinner o'clock for me | 23:06 |
| efried | me too | 23:06 |
| sean-k-mooney | its tuesday for me so im going to sleep soon o/ | 23:06 |
| openstackgerrit | Eric Fried proposed openstack/nova master: DNM: Try to repro bug 1823251 with mysql logs https://review.opendev.org/678051 | 23:07 |
| openstack | bug 1823251 in OpenStack Compute (nova) "Spike in TestNovaMigrationsMySQL.test_walk_versions/test_innodb_tables failures since April 1 2019 on limestone-regionone" [High,Confirmed] https://launchpad.net/bugs/1823251 | 23:07 |
| *** mriedem has quit IRC | 23:07 | |
| alex_xu | whatever that is worth to celebrate :) | 23:11 |
| *** hoonetorg has quit IRC | 23:11 | |
| *** tbachman has quit IRC | 23:14 | |
| *** Sundar has quit IRC | 23:14 | |
| *** hoonetorg has joined #openstack-nova | 23:25 | |
| *** eharney has quit IRC | 23:25 | |
| *** gbarros has joined #openstack-nova | 23:34 | |
| *** gbarros has quit IRC | 23:37 | |
| *** tbachman has joined #openstack-nova | 23:48 | |
| *** gbarros has joined #openstack-nova | 23:49 | |
| *** luksky has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!