| *** takashin has joined #openstack-placement | 00:06 | |
| *** nicolasbock has quit IRC | 00:45 | |
| *** nicolasbock has joined #openstack-placement | 00:58 | |
| *** fanzhang has joined #openstack-placement | 01:02 | |
| *** openstackgerrit has quit IRC | 01:06 | |
| *** Nel1x has joined #openstack-placement | 01:13 | |
| *** lei-zh has joined #openstack-placement | 01:52 | |
| *** openstackgerrit has joined #openstack-placement | 02:09 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Fix service list for disabled compute using MC driver https://review.openstack.org/583936 | 02:09 |
|---|---|---|
| *** lei-zh has quit IRC | 02:31 | |
| *** lei-zh has joined #openstack-placement | 02:31 | |
| *** takashin has left #openstack-placement | 03:02 | |
| *** Nel1x has quit IRC | 03:12 | |
| *** lei-zh has quit IRC | 04:16 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Delete instance_id_mappings record in instance_destroy https://review.openstack.org/591558 | 05:45 |
| *** lei-zh has joined #openstack-placement | 05:56 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: Fix service list for disabled compute using MC driver https://review.openstack.org/583936 | 06:15 |
| *** lei-zh1 has joined #openstack-placement | 06:33 | |
| *** lei-zh has quit IRC | 06:36 | |
| *** e0ne has joined #openstack-placement | 07:18 | |
| *** tssurya has joined #openstack-placement | 07:54 | |
| *** openstackstatus has quit IRC | 08:12 | |
| *** cdent has joined #openstack-placement | 08:43 | |
| openstackgerrit | Merged openstack/nova master: placement: ignore policy scope check failures if not enforcing scope https://review.openstack.org/590445 | 09:19 |
| openstackgerrit | Chris Dent proposed openstack/nova master: DNM: Add placement perf info gathering to end of nova-next https://review.openstack.org/591367 | 09:20 |
| openstackgerrit | Chris Dent proposed openstack/nova stable/pike: [placement] Retry allocation writes server side https://review.openstack.org/590745 | 09:28 |
| cdent | efried, jaypipes : I was able to get aggregate concurrency issues to happen in mysql too: https://bugs.launchpad.net/nova/+bug/1786703 | 09:35 |
| openstack | Launchpad bug 1786703 in OpenStack Compute (nova) "Placement duplicate aggregate uuid handling during concurrent aggregate create insufficiently robust" [Medium,New] | 09:35 |
| cdent | that was while doing placeload in the nova-next job | 09:35 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen support for delete instance allocations https://review.openstack.org/591597 | 09:35 |
| jaypipes | cdent: mornin. | 09:36 |
| jaypipes | cdent: different problem I believe. see note on bug report. | 09:41 |
| *** openstackstatus has joined #openstack-placement | 09:41 | |
| *** ChanServ sets mode: +v openstackstatus | 09:41 | |
| jaypipes | cdent: I can push up a patch for you to apply and see if it makes any different. | 09:42 |
| *** lei-zh1 has quit IRC | 09:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: WIP: Force refresh instance info_cache during heal https://review.openstack.org/591607 | 10:02 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: order multi-row INSERT by PK https://review.openstack.org/591609 | 10:05 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: placement: order multi-row INSERT by PK https://review.openstack.org/591609 | 10:15 |
| cdent | jaypipes: yah, wasn't trying to suggest it was exactly the same, just that slamming aggregates can results of 500s and thus there's stuff to do | 10:24 |
| jaypipes | cdent: can you apply the above patch and check to see if that addresses the issue? if not, I'll try the single-INSERT statement thing. | 10:26 |
| jaypipes | cdent: how repeatable is this, BTW? | 10:27 |
| jaypipes | every run? | 10:27 |
| cdent | jaypipes: I don't know. That report was the first time I've seen, but that is also the first time I've run placeload against mysql | 10:27 |
| cdent | I saw it in the results of https://review.openstack.org/#/c/591367/ after (finally) getting that to actually do a run | 10:28 |
| jaypipes | ack | 10:28 |
| cdent | so testing your patch will require some adjustments to my test setup | 10:28 |
| jaypipes | k, no worries. thx for looking into this. | 10:28 |
| cdent | however in the placeload run it happend many times | 10:28 |
| cdent | in the run currently running it should provide a bit tidier output | 10:30 |
| cdent | jaypipes, efried, edleafe: https://review.openstack.org/#/c/591367/ is basically working now save the issues from bug 1786703 but it likely needs some tidying up, so please provide your suggestions | 10:58 |
| openstack | bug 1786703 in OpenStack Compute (nova) "Placement duplicate aggregate uuid handling during concurrent aggregate create insufficiently robust" [Medium,In progress] https://launchpad.net/bugs/1786703 - Assigned to Jay Pipes (jaypipes) | 10:58 |
| openstackgerrit | Chen proposed openstack/nova master: Typo fixes on notifications doc https://review.openstack.org/591621 | 10:58 |
| cdent | I reckon things like this happening without us (or at least me) knowing is good: https://www.openstack.org/summit/berlin-2018/summit-schedule/global-search?t=placement | 11:12 |
| *** s10 has joined #openstack-placement | 11:27 | |
| *** nicolasbock has quit IRC | 11:49 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen support for put allocations https://review.openstack.org/591647 | 12:33 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen support for put allocations https://review.openstack.org/591647 | 12:35 |
| cdent | gibi: can you have a quick look at https://review.openstack.org/#/c/591367/ and say whether you think it is a good first stab that we can cleanup later, or should be cleaned up now? I need to take the DNM off at some point but have left it there because I assume we don't want to merge it too quickly without some non-placement people looking | 12:42 |
| gibi | cdent: looking | 12:42 |
| cdent | thanks | 12:42 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova list when a cell is down https://review.openstack.org/567785 | 13:00 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova service-list when a cell is down https://review.openstack.org/584829 | 13:00 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Add get_by_cell_and_project() method to InstanceMappingList https://review.openstack.org/591656 | 13:00 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: API microversion bump for handling-down-cell https://review.openstack.org/591657 | 13:00 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova master: Return a minimal construct for nova show when a cell is down https://review.openstack.org/591658 | 13:00 |
| gibi | cdent: left couple of notes inline. I think we can easily merge this patch | 13:02 |
| cdent | thanks | 13:02 |
| cdent | gibi: thanks for that excellent feedback, responded with some questions | 13:15 |
| gibi | cdent: looking | 13:15 |
| cdent | efried, thanks, question in my response to you | 14:10 |
| cdent | efried: also if you have any thoughts on the microversion discussion going on between gibi and I on that review | 14:14 |
| efried | cdent: I looked at that discussion, and I don't care. | 14:14 |
| efried | Does the microversion in that output actually correspond to the microversion you're using to do the query? | 14:14 |
| cdent | efried isa nihilist | 14:14 |
| cdent | which "microversion in that output"? | 14:15 |
| efried | in the placement-perf.txt.gz | 14:15 |
| cdent | where is says 1.29 in http://logs.openstack.org/67/591367/5/check/nova-next/96e7d94/logs/placement-perf.txt.gz is the max versiono of the service | 14:15 |
| cdent | where it says "placement 1.28" that is the actual header of the request(s) | 14:15 |
| efried | + /opt/stack/new/nova/gate/post_test_perf_check.sh:check_placement:39 : curl -s -H 'x-auth-token: admin' -H 'openstack-api-version: placement 1.28' 'https://158.69.81.174/placement/allocation_candidates? | 14:15 |
| cdent | that's the actual command | 14:16 |
| efried | I assume that's actually the version you're passing in to the command. | 14:16 |
| efried | Yeah. | 14:16 |
| efried | So as long as that's true, I don't really care which microversion you use for the command. | 14:16 |
| efried | Is the contention about "Placement is 1.29" at the top? | 14:16 |
| cdent | I'm not certain, but I think gibi was confused about why 1.28 was being used | 14:16 |
| efried | I have zero problem with that. It would be better, I suppose, if it could say "Placement max is 1.29". | 14:17 |
| efried | Yeah, and I agree that 1.28 isn't a match for anything where specific GET /a_c function was added - but I really don't care. | 14:17 |
| efried | ^ this is the part I really don't care about. | 14:17 |
| cdent | I was trying (perhaps lamely) to indicate that the 1.29 is what placement's version is, but that's perhaps not a convention we really have. Changing that would require changing placeload, which we can worry about some other time if we want. I'll update the hook script to use 1.21, update the commit message and ... | 14:18 |
| cdent | efried: where/how did you want the error message? | 14:19 |
| efried | I'm composing that response inline. | 14:19 |
| efried | but in two other meetings at the same time :) | 14:19 |
| efried | cdent: Done | 14:20 |
| efried | cdent: TL;DR: echo | 14:20 |
| cdent | you poor thing | 14:20 |
| efried | cdent: btw, it wouldn't be wildly out of the question for setup failure to fail the whole CI job. | 14:20 |
| cdent | I think that would be overstepping for this kind of thing. I do think that having a smash placement to see if it breaks CI job would be a good thing. But this is not that. Combining the two would be confusing and messy | 14:22 |
| gibi | sorry, I was pulled to local brainstorming | 14:23 |
| gibi | cdent: I | 14:24 |
| gibi | cdent: I replied | 14:24 |
| cdent | yeah, reading, thank you! | 14:25 |
| efried | cdent: I'm not saying we should have a perf threshold and fail the job for that. Just so we're clear. | 14:25 |
| cdent | efried: yeah, I get ya | 14:25 |
| cdent | but since I can fail the setup right now, every single time (that's what A500 means), I don't want to build that in here | 14:26 |
| efried | cdent: But yes, I agree long-term it would be nice to have a separate CI job that just does placement load. | 14:26 |
| cdent | because of https://bugs.launchpad.net/nova/+bug/1786703 | 14:26 |
| openstack | Launchpad bug 1786703 in OpenStack Compute (nova) "Placement duplicate aggregate uuid handling during concurrent aggregate create insufficiently robust" [Medium,In progress] - Assigned to Jay Pipes (jaypipes) | 14:26 |
| cdent | (i broke in on mysql too, not just postgresql) | 14:27 |
| cdent | s/in/it/ | 14:27 |
| jaypipes | cdent: able to reproduce that mysql one? | 14:27 |
| cdent | jaypipes: haven't had a chance to try your patch, been fixing other things. but if fails consistently on master | 14:29 |
| jaypipes | cdent: no pressure. was just curious :) | 14:32 |
| cdent | if/when we merge this change to nova-next we'll see how it goes on your change. Unless I get to it before then | 14:33 |
| efried | jaypipes: where's your patch? | 14:34 |
| efried | https://review.openstack.org/591609 ? | 14:34 |
| efried | cdent: Can't we just rebase ^ onto yours? | 14:34 |
| cdent | yeah sure, but I tend to avoid it because I get mentally trained by too many stacks. I guess these are sufficiently related though. | 14:35 |
| cdent | drained | 14:36 |
| cdent | it's a bad day for typing | 14:36 |
| efried | I figured it out, but it took me a sec. | 14:36 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Add placement perf info gathering hook to end of nova-next https://review.openstack.org/591367 | 14:43 |
| openstackgerrit | Chris Dent proposed openstack/nova master: placement: order multi-row INSERT by PK https://review.openstack.org/591609 | 14:46 |
| efried | cdent: +2 | 14:46 |
| cdent | rad | 14:47 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Add regression test for bug 1550919 https://review.openstack.org/591733 | 15:10 |
| openstack | bug 1550919 in OpenStack Compute (nova) "[Libvirt]Evacuate fail may cause disk image be deleted" [Medium,In progress] https://launchpad.net/bugs/1550919 - Assigned to Matthew Booth (mbooth-9) | 15:10 |
| *** e0ne has quit IRC | 15:16 | |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: WIP: Wait for network-vif-plugged before starting live migration https://review.openstack.org/591735 | 15:17 |
| *** e0ne has joined #openstack-placement | 15:20 | |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Move static _get_power_off_values to compute_utils https://review.openstack.org/578822 | 15:21 |
| openstackgerrit | Matthew Booth proposed openstack/nova master: Don't delete disks on shared storage during evacuate https://review.openstack.org/578846 | 15:21 |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/591761 | 15:30 |
| jaypipes | efried: I like https://review.openstack.org/#/c/591510. If you want to squash it into mine, that's cool with me. | 15:30 |
| efried | jaypipes: Cool, will do. | 15:31 |
| efried | jaypipes: I do want to get a look from the folks on that ML thread. | 15:31 |
| openstackgerrit | Dan Smith proposed openstack/nova stable/pike: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/591762 | 15:32 |
| efried | But I suppose we can merge this and improve on it later. | 15:32 |
| openstackgerrit | Eric Fried proposed openstack/nova master: [placement] split gigantor SQL query, add logging https://review.openstack.org/591510 | 15:35 |
| efried | jaypipes, cdent, gibi: ^ | 15:35 |
| cdent | roger | 15:36 |
| efried | ahdang, I used the wrong change-id, wound up squashing backwards. | 15:36 |
| efried | Probably doesn't matter, but I guess I would rather have the history/discussion from the original... will fix. | 15:37 |
| openstackgerrit | Eric Fried proposed openstack/nova master: [placement] split gigantor SQL query, add logging https://review.openstack.org/590041 | 15:37 |
| efried | jaypipes, cdent, gibi: This is the right one ^ The other is now abandoned. | 15:39 |
| jaypipes | ack | 15:40 |
| openstackgerrit | xhzhf proposed openstack/nova master: [vmware] add missing os types in vSphere sdk 6.5 https://review.openstack.org/591771 | 15:51 |
| *** e0ne has quit IRC | 15:52 | |
| *** tssurya has quit IRC | 15:54 | |
| cdent | i'll be back later | 15:58 |
| *** cdent has quit IRC | 15:58 | |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775 | 16:19 |
| *** s10 has quit IRC | 16:28 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Remove blacklisted py3 xen tests https://review.openstack.org/591419 | 16:34 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen: remove_provider_from_instance_allocation https://review.openstack.org/591784 | 16:57 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Remove obsolete func test_ironic_ocata_to_pike https://review.openstack.org/591785 | 17:05 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ironic: report 0 for vcpus/memory_mb/disk_gb resources https://review.openstack.org/565841 | 17:47 |
| openstackgerrit | Dan Smith proposed openstack/nova stable/queens: DNM: Debug patch to test live migration waiting https://review.openstack.org/591775 | 17:48 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen support for put allocations https://review.openstack.org/591647 | 18:15 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Consumer gen: remove_provider_from_instance_allocation https://review.openstack.org/591784 | 18:16 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: support claim_resources https://review.openstack.org/583667 | 18:16 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: move_allocations https://review.openstack.org/591810 | 18:16 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: consumer gen: more tests for delete allocation cases https://review.openstack.org/591811 | 18:16 |
| openstackgerrit | Merged openstack/nova stable/rocky: Revert "libvirt: slow live-migration to ensure network is ready" https://review.openstack.org/591275 | 18:19 |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ironic: report 0 for vcpus/memory_mb/disk_gb resources https://review.openstack.org/565841 | 18:44 |
| *** e0ne has joined #openstack-placement | 19:23 | |
| *** e0ne has quit IRC | 19:26 | |
| *** e0ne has joined #openstack-placement | 19:27 | |
| *** cdent has joined #openstack-placement | 19:53 | |
| cdent | jaypipes: doesn't look like your ordering thing helped: http://logs.openstack.org/09/591609/3/check/nova-next/7bdd800/logs/screen-placement-api.txt.gz#_Aug_14_16_34_29_060039 | 19:54 |
| jaypipes | cdent: well, poop. at least it was worth a shot. | 19:55 |
| jaypipes | cdent: I'll push a patch that changes that to single-shot INSERTs then and see if that works. | 19:55 |
| cdent | is that the same error? I didn't look closely, just saw that it A500 | 19:55 |
| jaypipes | yep, same error. | 19:55 |
| efried | cdent: Okay, I'm mystified now. I just checked the results and they looked okay to me. | 20:01 |
| efried | cdent: I thought http://logs.openstack.org/09/591609/3/check/nova-next/7bdd800/logs/placement-perf.txt.gz would show no timing results if we got that failure. | 20:02 |
| cdent | not _that_ failure | 20:02 |
| cdent | efried: all rp_count is counting is resource providers | 20:03 |
| efried | Right, but I thought we would bail out of creating resource providers if we got that error. | 20:03 |
| cdent | all the errors were A500 meaning aggregate failures, not R500 meaing rp failures | 20:03 |
| cdent | so 1001 rp's exist | 20:03 |
| cdent | but some of them don't have aggregates | 20:03 |
| cdent | which is why the pirate talks in the log | 20:04 |
| cdent | so you can know that things went wrong | 20:04 |
| efried | the A500 doesn't cause the whole thing to bail? | 20:04 |
| cdent | under "normal" circumstances there will never be R, I, A, only r, i, a | 20:04 |
| efried | okaaay... | 20:04 |
| cdent | As I said: placeload is a tool being used to create stuff, it is _not_ the thing doing the testing | 20:04 |
| efried | I was considering saying this before. May still get shot down, but I'll say it: | 20:05 |
| cdent | as far as placeload is concerned it creates resource providers, inventory, and aggregates in an async fashion | 20:05 |
| cdent | the reason it is async is so it is fast, but it also has the nice side effect of finding now 3 different bugs under load | 20:05 |
| cdent | that's why it was created. that it is useful for the perf test is happenstance | 20:05 |
| efried | What about echo "How to read this log file ...<words here about what crack pirate means etc.>..." >> $LOG | 20:05 |
| cdent | if you like, now that it is ready to merge we (perhaps you) can do it as a followup | 20:07 |
| cdent | it's easy to change, now that it exist | 20:07 |
| efried | or at least "To understand this output, see <link to docstring in placeload>" | 20:07 |
| cdent | that would require placeload (which was created in the space a very small number of hours) to have such info it | 20:08 |
| cdent | you wanted a rush job, I did a rush job | 20:08 |
| cdent | but now it exists, we can make it pretty | 20:08 |
| efried | Just trying to lower the barriers to other people wanting to use this. | 20:09 |
| cdent | efried: I get that, and I'm saying: please feel free to do so. As I said when I was volunteered to do this work I didn't really think it was particularly useful but I was willing to put in the work to get _something_ in place. | 20:14 |
| cdent | Now that we have something concrete to compare against, and an understanding of how it can be horked in, it can be improved. | 20:14 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: DNM: test possible deadlock cause https://review.openstack.org/591845 | 20:42 |
| jaypipes | cdent: total shot in the dark ^ | 20:42 |
| jaypipes | cdent: I'll still do the single-shot INSERT patch. just curious if the above makes a diff. | 20:43 |
| cdent | interestig | 20:43 |
| cdent | you'll wanna base that on the nova-next change won't you? or did that merge? | 20:44 |
| jaypipes | cdent: ah, yeah... | 20:44 |
| cdent | efried: I'll go ahead and add a crack pirate decoder ring patch, unless you're already doing something | 20:49 |
| efried | cdent: I'm not. I really needed to get some IBMish work done today, so trying to ignore other distractions for a bit. | 20:49 |
| cdent | good and just | 20:49 |
| openstackgerrit | Chris Dent proposed openstack/nova master: Add explanatory prefix to post_test_perf output https://review.openstack.org/591850 | 21:15 |
| openstackgerrit | Chris Dent proposed openstack/nova master: DNM: test possible deadlock cause https://review.openstack.org/591845 | 21:19 |
| openstackgerrit | Dmitry Sutyagin proposed openstack/nova master: Implement disabling KSM via flavor's extra specs https://review.openstack.org/591854 | 21:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM: update_provider_tree() (compatible) https://review.openstack.org/468560 | 21:30 |
| *** s10 has joined #openstack-placement | 21:30 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova master: Ironic: report 0 for vcpus/memory_mb/disk_gb resources https://review.openstack.org/565841 | 21:32 |
| *** cdent has quit IRC | 21:59 | |
| *** s10 has quit IRC | 22:23 | |
| *** s10 has joined #openstack-placement | 22:23 | |
| *** s10 has quit IRC | 22:24 | |
| *** s10 has joined #openstack-placement | 22:24 | |
| *** s10 has quit IRC | 22:24 | |
| *** s10 has joined #openstack-placement | 22:25 | |
| *** s10 has quit IRC | 22:25 | |
| *** s10 has joined #openstack-placement | 22:26 | |
| *** s10 has quit IRC | 22:26 | |
| *** s10 has joined #openstack-placement | 22:26 | |
| *** s10 has quit IRC | 22:27 | |
| *** e0ne has quit IRC | 22:39 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: [placement] Regex consts for placement schema https://review.openstack.org/591863 | 22:40 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!