| *** cdent has quit IRC | 00:36 | |
| *** mriedem has quit IRC | 01:26 | |
| openstackgerrit | Merged openstack/placement master: Update master for stable/stein https://review.openstack.org/645001 | 01:31 |
|---|---|---|
| openstackgerrit | Merged openstack/placement stable/stein: Update .gitreview for stable/stein https://review.openstack.org/644999 | 01:31 |
| openstackgerrit | Merged openstack/placement stable/stein: Update UPPER_CONSTRAINTS_FILE for stable/stein https://review.openstack.org/645000 | 01:31 |
| openstackgerrit | Merged openstack/placement master: Revert "Centralize registration of logging options" https://review.openstack.org/645014 | 01:46 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:29 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:29 |
| *** e0ne has joined #openstack-placement | 06:06 | |
| *** e0ne has quit IRC | 06:07 | |
| *** e0ne has joined #openstack-placement | 07:45 | |
| *** ttsiouts has joined #openstack-placement | 08:06 | |
| *** tssurya has quit IRC | 08:12 | |
| *** openstackgerrit has quit IRC | 08:17 | |
| *** tssurya has joined #openstack-placement | 08:24 | |
| *** helenafm has joined #openstack-placement | 08:35 | |
| *** e0ne has quit IRC | 08:48 | |
| *** e0ne has joined #openstack-placement | 08:51 | |
| *** cdent has joined #openstack-placement | 10:08 | |
| cdent | mornin' | 10:08 |
| cdent | gibi: I put some thoughts on testing the fixture stuff on the nova-placement cross project etherpad for ptg, you might want to add your thinking there: https://etherpad.openstack.org/p/ptg-train-xproj-nova-placement | 10:09 |
| gibi | cdent: thanks | 10:10 |
| gibi | cdent: looking | 10:10 |
| *** e0ne has quit IRC | 10:14 | |
| *** e0ne has joined #openstack-placement | 10:17 | |
| cdent | efried, mriedem: I backported the PlacementFixture fix ( https://review.openstack.org/#/c/645105/ ). I assume that was the right thing to do since placement has already cut an rc1? | 10:36 |
| *** e0ne has quit IRC | 10:58 | |
| *** e0ne has joined #openstack-placement | 10:59 | |
| *** ttsiouts has quit IRC | 11:37 | |
| *** ttsiouts has joined #openstack-placement | 11:37 | |
| *** ttsiouts has quit IRC | 11:42 | |
| *** cdent has quit IRC | 12:04 | |
| *** ttsiouts has joined #openstack-placement | 12:13 | |
| *** mriedem has joined #openstack-placement | 12:38 | |
| mriedem | . | 12:39 |
| *** e0ne has quit IRC | 12:51 | |
| *** edleafe_ has quit IRC | 12:51 | |
| *** irclogbot_3 has quit IRC | 13:07 | |
| *** irclogbot_3 has joined #openstack-placement | 13:09 | |
| *** cdent has joined #openstack-placement | 13:20 | |
| *** altlogbot_2 has quit IRC | 13:23 | |
| *** e0ne has joined #openstack-placement | 13:23 | |
| *** altlogbot_1 has joined #openstack-placement | 13:24 | |
| *** openstackgerrit has joined #openstack-placement | 13:27 | |
| openstackgerrit | Chris Dent proposed openstack/placement stable/stein: Add register_opts param to PlacementFixture https://review.openstack.org/645105 | 13:27 |
| *** takashin has joined #openstack-placement | 13:27 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Re-propose cross-cell-resize spec for Train https://review.openstack.org/642807 | 13:31 |
| *** e0ne has quit IRC | 13:37 | |
| *** altlogbot_1 has quit IRC | 13:39 | |
| *** altlogbot_0 has joined #openstack-placement | 13:41 | |
| *** irclogbot_3 has quit IRC | 13:45 | |
| *** e0ne has joined #openstack-placement | 13:45 | |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add request-filter-image-types spec https://review.openstack.org/644625 | 13:46 |
| *** jaypipes has quit IRC | 13:46 | |
| *** irclogbot_2 has joined #openstack-placement | 13:47 | |
| openstackgerrit | Dan Smith proposed openstack/nova-specs master: Add request-filter-image-types spec https://review.openstack.org/644625 | 13:56 |
| efried | cdent: you want to be in nova meeting? | 14:21 |
| efried | are your ears burning? | 14:22 |
| cdent | oops, forgot about it | 14:22 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Initial structure for in-tree specs https://review.openstack.org/645195 | 14:32 |
| *** altlogbot_0 has quit IRC | 14:35 | |
| *** altlogbot_2 has joined #openstack-placement | 14:36 | |
| *** ttsiouts has quit IRC | 14:37 | |
| *** irclogbot_2 has quit IRC | 14:38 | |
| *** ttsiouts has joined #openstack-placement | 14:38 | |
| *** irclogbot_3 has joined #openstack-placement | 14:39 | |
| *** ttsiouts has quit IRC | 14:41 | |
| *** ttsiouts has joined #openstack-placement | 14:41 | |
| *** takashin has left #openstack-placement | 15:01 | |
| mriedem | cdent: i know you've done some scale/performance type testing with placement and posted blogs about it, can you link me those so i can pass them on to some folks internally that are asking me about placement performance? | 15:17 |
| mriedem | i get random questions like, "but what issues are we going to have with locks on a centralized data store?!" | 15:17 |
| mriedem | i think probably most related is https://anticdent.org/placement-scale-fun.html | 15:19 |
| cdent | mriedem: yeah, that one is a bit old, but still fairly relevant | 15:22 |
| cdent | there's also https://anticdent.org/profiling-wsgi-apps.html | 15:22 |
| mriedem | yup found that one as well | 15:22 |
| mriedem | my answer so far has been placement scaling isn't the problem with nova scheduling, it's nova scheduling | 15:22 |
| mriedem | i.e. we still have lots of python filters and such, | 15:23 |
| mriedem | and the issues of not properly modeling data in placement so we have issues like where cern limts the number of allocation candidates and only gets back disabled computes | 15:23 |
| mriedem | and cern limits the number of candidates to deal with the huge number of hosts they have, and because they don't have 2-level scheduling anymore like with cells v1 | 15:24 |
| cdent | I've got on my to do list extending the perfload job to do many parallel GET /allocation_candidate requests to see how that looks | 15:26 |
| cdent | right not it does parallel writes, but then single GET | 15:26 |
| cdent | but, yeah, in general most of the problems happen on the nova side when trying to work with too-much-data from palcement | 15:26 |
| openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support server power state update through external event https://review.openstack.org/636132 | 15:33 |
| cdent | mriedem: and in case it's not obvious already, the best way to deal with placement scale issues (in placement) itself is to have more severs, and the easiest way I've found to do that is stick it in kubernetes with a horizontal load balancer. I've got a helm chart that does that, in placedock, if somebody wants a refernce | 15:34 |
| mriedem | i think these guys are more concerned about the db than the api | 15:36 |
| mriedem | but, they also pack the shit out of their computes rather than spread, | 15:36 |
| mriedem | so allocations are going to be hitting the same providers concurrently | 15:36 |
| mriedem | but i've tried to explain that nova-scheduler will at least be retrying that during scheduling now rather than the expensive reschedule loop with the compute as we had before when an RT claim failed | 15:37 |
| mriedem | so end to end server create time should be better | 15:37 |
| mriedem | even if they are just focused on how long scheduling takes | 15:37 |
| cdent | yeah, lots of allocations on the same rp, has shown some issues, but I think rado provided a reasonable fix for that: a randomized backoff | 15:37 |
| cdent | (on retry). When it was not random, it would just keep colliding | 15:37 |
| mriedem | yeah https://review.openstack.org/#/c/567506/ | 15:38 |
| cdent | but again, that's nova-side. placement seems really happy to say 409 over and over again without sweat if that's what needs to happen | 15:38 |
| mriedem | heh yeah | 15:38 |
| cdent | I vaguely remember somebody was going to do some investigation (this was maybe 6 months or more ago) into lock contention potential in various places in openstack. I thought they were huawei people, but not sure? | 15:40 |
| mriedem | doesn't ring a bell | 15:40 |
| *** belmoreira has quit IRC | 15:58 | |
| openstackgerrit | Chris Dent proposed openstack/placement master: Initial structure for in-tree specs https://review.openstack.org/645195 | 15:58 |
| cdent | edleafe: if you could do the honors on https://wiki.openstack.org/wiki/First_Contact_SIG that would teh ossum | 16:01 |
| edleafe | cdent: will do | 16:01 |
| cdent | thanks | 16:01 |
| edleafe | updating the liaison page to reference API-SIG | 16:02 |
| bauzas | folks, how to ask devstack to use osc-placement ? | 16:02 |
| bauzas | jut pip it ? | 16:02 |
| bauzas | errr, nvm | 16:02 |
| bauzas | https://github.com/openstack-dev/devstack/blob/master/lib/placement#L149 | 16:04 |
| edleafe | cdent: done | 16:06 |
| cdent | huzzah | 16:06 |
| openstackgerrit | Merged openstack/placement stable/stein: Add register_opts param to PlacementFixture https://review.openstack.org/645105 | 16:09 |
| *** e0ne has quit IRC | 16:10 | |
| *** ttsiouts has quit IRC | 16:30 | |
| *** ttsiouts has joined #openstack-placement | 16:31 | |
| *** tssurya has quit IRC | 16:34 | |
| *** ttsiouts has quit IRC | 16:35 | |
| efried | cdent: So yeah, no testing for that register_opts thing? | 16:45 |
| efried | that kinda got blown past | 16:45 |
| cdent | I was asleep | 16:45 |
| cdent | I think any unit test there would have just been confirm that python conditionals work | 16:46 |
| efried | I would have thought there's a way to detect that conf has or hasn't been loaded. | 16:46 |
| efried | sorry, I mean opts have or haven't been registered | 16:47 |
| efried | like 'debug' in CONF | 16:47 |
| cdent | one sec | 16:47 |
| efried | I mean yeah, it's slightly trivial, but still seems like a good idea. | 16:47 |
| cdent | i'm not disputing that but it's done now, we can address it later if you like but I think that may as well be in concert with addressing the bigger issue of "it's hard to _really_ test the PlacementFixture" | 16:48 |
| cdent | I made it clear before I went to be last night that the patch should be tweaked however people needed before it was merged. and then it was merged... | 16:49 |
| *** rubasov has quit IRC | 16:49 | |
| efried | cdent: Yeah, I was asleep when it was pushed, and my question wasn't answered by anyone. Let me dork with it a sec... | 16:50 |
| cdent | it's also the case that nova's use of the fix was the most robust was of testing the fix | 16:50 |
| cdent | I'm sure we can add a test but I really don't see why _now_ matters | 16:50 |
| efried | heh, it doesn't matter any more than getting that real fix in _now_ mattered. | 16:50 |
| cdent | that's entirely untrue | 16:50 |
| cdent | the logging revert fixed one symptom of a larger problem | 16:51 |
| cdent | which was identified by the logging opts problem but not limited to it | 16:51 |
| cdent | the larger fixed fixed it, and works and was tested by the thing that was broken | 16:52 |
| cdent | having an additional test simply buys a bit of insurance that we won't regress it, but even with that I'm not all that convinced of the value, as each time this happened it has been ... weird | 16:52 |
| cdent | so we can buy that insurance next week if we prefer | 16:53 |
| cdent | but if you're feeling like you have nothing better to do, go for it | 16:53 |
| openstackgerrit | Chris Dent proposed openstack/placement master: Initial structure for in-tree specs https://review.openstack.org/645195 | 16:54 |
| *** helenafm has quit IRC | 16:58 | |
| *** mriedem is now known as mriedem_grooming | 17:16 | |
| openstackgerrit | Eric Fried proposed openstack/placement master: Start unit tests for PlacementFixture https://review.openstack.org/645255 | 17:22 |
| *** rubasov has joined #openstack-placement | 17:43 | |
| *** e0ne has joined #openstack-placement | 18:05 | |
| openstackgerrit | Eric Fried proposed openstack/placement master: Start unit tests for PlacementFixture https://review.openstack.org/645255 | 18:08 |
| *** e0ne has quit IRC | 18:12 | |
| bauzas | stupid question, but where uwsgi placement logs are in a devstack ? | 18:13 |
| efried | bauzas: You don't mean journalctl -u devstack@placement-api.service do you? | 18:17 |
| bauzas | I'm fuck*** tired | 18:17 |
| bauzas | I was journalctl -f | 18:17 |
| efried | glad you bleeped out the "ing" there bauzas | 18:18 |
| bauzas | devstack returned a 503 when testing whether placement-api was starting | 18:18 |
| cdent | bauzas: likely you are seeing a problem where apache2 is looking in the wrong place for the running uwsgi process | 18:19 |
| bauzas | actually, I think it's my issue | 18:20 |
| bauzas | because it runs uwsgi now | 18:20 |
| bauzas | and no longer httpd | 18:20 |
| bauzas | but I still have httpd placement files | 18:21 |
| cdent | you need both | 18:21 |
| cdent | httpd uses mod_proxy_uwsgi to talk to uwsgi processes managed by systemd | 18:21 |
| bauzas | oh you're right | 18:21 |
| bauzas | anyway, I think I found | 18:21 |
| bauzas | I yet have both nova-placement and placement ini files | 18:22 |
| bauzas | I guess I have to delete the nova-placement one | 18:22 |
| bauzas | for some reason, devstack reinstalled it | 18:22 |
| cdent | it may not have been removed | 18:26 |
| cdent | cleaning is not very reliable | 18:27 |
| bauzas | yeah i know | 18:27 |
| efried | cdent: hate to burn a whole email with basically a +1, but crickets-as-assent can be disquieting. | 18:30 |
| cdent | efried: which email? I seem to send so many | 18:31 |
| efried | cdent: placement specs process | 18:31 |
| cdent | oh aye. cool. good. | 18:31 |
| * cdent dines | 18:31 | |
| *** mriedem_grooming is now known as mriedem | 18:53 | |
| openstackgerrit | melanie witt proposed openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | 19:01 |
| *** cdent has quit IRC | 19:13 | |
| *** cdent has joined #openstack-placement | 19:15 | |
| *** e0ne has joined #openstack-placement | 19:30 | |
| *** e0ne has quit IRC | 19:49 | |
| *** e0ne has joined #openstack-placement | 19:52 | |
| *** e0ne has quit IRC | 20:04 | |
| cdent | i think I'm done for today | 20:23 |
| cdent | 'night all | 20:24 |
| *** cdent has quit IRC | 20:24 | |
| *** e0ne has joined #openstack-placement | 20:28 | |
| openstackgerrit | Merged openstack/nova-specs master: Re-propose count quota usage from placement https://review.openstack.org/645302 | 21:00 |
| *** e0ne has quit IRC | 21:17 | |
| openstackgerrit | Eric Fried proposed openstack/placement master: Start unit tests for PlacementFixture https://review.openstack.org/645255 | 21:36 |
| *** irclogbot_3 has quit IRC | 22:05 | |
| *** mriedem is now known as mriedem_afk | 22:50 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!