| *** rcernin has joined #openstack-nova | 00:00 | |
| *** mingyu has quit IRC | 00:01 | |
| *** Swami has quit IRC | 00:03 | |
| *** mriedem has quit IRC | 00:03 | |
| *** mriedem has joined #openstack-nova | 00:04 | |
| *** harlowja has quit IRC | 00:04 | |
| openstackgerrit | Eric Brown proposed openstack/nova: Add RPC version aliases for Ocata https://review.openstack.org/425958 | 00:07 |
|---|---|---|
| mriedem | looks like that grenade change is happy now | 00:07 |
| mriedem | at least for single-node | 00:07 |
| *** liangy has quit IRC | 00:09 | |
| *** dtp has quit IRC | 00:10 | |
| *** Swami has joined #openstack-nova | 00:10 | |
| *** dtp has joined #openstack-nova | 00:10 | |
| *** hongbin has quit IRC | 00:16 | |
| mriedem | dansmith: so the multinode grenade job passed with your change https://review.openstack.org/#/c/424730/10/projects/60_nova/resources.sh but i don't see this in the logs | 00:17 |
| mriedem | echo "Expecting $expected computes" | 00:17 |
| mriedem | oh hrm http://logs.openstack.org/30/424730/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/9a57ab4/logs/grenade.sh.txt.gz#_2017-01-26_23_58_11_661 | 00:19 |
| mriedem | Expecting 0 computes | 00:19 |
| mriedem | Found 2 computes | 00:19 |
| *** dimtruck is now known as zz_dimtruck | 00:21 | |
| *** tblakes has joined #openstack-nova | 00:22 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/os-vif: Fix broken Link https://review.openstack.org/425994 | 00:23 |
| mriedem | http://logs.openstack.org/30/424730/10/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/9a57ab4/logs/grenade_save/grenade_db.ini | 00:24 |
| mriedem | nova_compute_count = 0 | 00:24 |
| *** mlavalle has quit IRC | 00:27 | |
| *** nic has quit IRC | 00:27 | |
| openstackgerrit | Jaivish Kothari(janonymous) proposed openstack/os-vif: Removing Deprecated hacking Check https://review.openstack.org/425996 | 00:36 |
| *** chyka has quit IRC | 00:38 | |
| *** chyka has joined #openstack-nova | 00:41 | |
| *** jose-phillips has quit IRC | 00:43 | |
| *** chyka has quit IRC | 00:43 | |
| *** chyka has joined #openstack-nova | 00:43 | |
| *** thorst_ has joined #openstack-nova | 00:45 | |
| dansmith | mriedem: greeat | 00:46 |
| *** hfu has joined #openstack-nova | 00:47 | |
| *** claudiub has quit IRC | 00:47 | |
| mriedem | it's weird b/c it finds the 2 in the verify step | 00:48 |
| mriedem | but not before it | 00:48 |
| mriedem | but we're creating resources so we should have 2 computes | 00:48 |
| *** chyka has quit IRC | 00:48 | |
| *** hfu has quit IRC | 00:49 | |
| *** rcernin has quit IRC | 00:51 | |
| alex_xu | mriedem: I'm already in vacation. But I think I have better to give you a update. The servers filter BP already merged the filter and sort part. For the policy rule change didn't merge yet, johnthetubaguy pointed out that changing a rule enforcement from a hard to soft should bump microversion. | 00:52 |
| alex_xu | mriedem: and gmann works on it https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/add-whitelist-for-server-list-filter-sort-parameters | 00:53 |
| *** rcernin has joined #openstack-nova | 00:54 | |
| alex_xu | mriedem: that probably needs two patch, one is about the rename of the policy rule, then change the hard to soft by microversion. I'm not sure whether it can get FFE. That should be your call | 00:54 |
| alex_xu | mriedem: or you want something detail, I can write a email | 00:55 |
| *** thorst_ has quit IRC | 00:55 | |
| *** rcernin has quit IRC | 00:55 | |
| *** rcernin has joined #openstack-nova | 00:55 | |
| gmann | mriedem: as alex_xu said. i will try to push microversion patch (max over weekend) also and then you can take call on that | 00:55 |
| alex_xu | gmann: thanks | 00:56 |
| alex_xu | gmann: mriedem another thing is worth to track is this bug https://launchpad.net/bugs/1658571 | 00:57 |
| openstack | Launchpad bug 1658571 in OpenStack Compute (nova) "Microversion 2.37 break 2.32 usage" [High,In progress] - Assigned to Artom Lifshitz (notartom) | 00:57 |
| gmann | alex_xu: mriedem i updates the spec with latest discusion, if you guys can have a look - https://review.openstack.org/#/c/425533/ | 00:57 |
| alex_xu | gmann: let me check now | 00:59 |
| gmann | alex_xu: about your comment on https://review.openstack.org/#/c/415330/25/nova/policy.py | 00:59 |
| alex_xu | gmann: yea | 00:59 |
| gmann | alex_xu: that if condition is when policy file is updated right? for example say if old deployer has policy file(with old rule overridden) and do not change file then also it hit if condition ? | 01:00 |
| *** zz_dimtruck is now known as dimtruck | 01:00 | |
| mriedem | alex_xu: gmann: a ML thread on the sort/filters policy thing is probably best | 01:01 |
| mriedem | i'm leaving soon for dinner | 01:01 |
| alex_xu | gmann: it only happened when the policy file changed. the olso.policy will check timestamp of policy file, then it will reload the file. Then you will pass that if condition | 01:01 |
| alex_xu | mriedem: got it | 01:02 |
| mriedem | for https://launchpad.net/bugs/1658571 we said we'd fix it after FF as a bug fix, but that was before i was reminded that we only have 1 week until rc1 | 01:02 |
| openstack | Launchpad bug 1658571 in OpenStack Compute (nova) "Microversion 2.37 break 2.32 usage" [High,In progress] - Assigned to Artom Lifshitz (notartom) | 01:02 |
| gmann | mriedem: +1 | 01:03 |
| alex_xu | mriedem: 1 week from now? | 01:03 |
| mriedem | alex_xu: yeah | 01:03 |
| alex_xu | ok, got it | 01:03 |
| mriedem | so, that one might end up waiting until pike | 01:03 |
| mriedem | since it was already broken in newton | 01:03 |
| mriedem | we might just have to document it for ocata | 01:03 |
| alex_xu | ok | 01:03 |
| mriedem | i've tagged the bug with ocata-rc-potential though so we don't forget about it | 01:04 |
| alex_xu | cool | 01:04 |
| gmann | it looks pretty close if we can get in Ocata only but yea let's not hurry on that may be | 01:04 |
| *** tblakes has quit IRC | 01:04 | |
| alex_xu | gmann: +1 | 01:04 |
| gmann | alex_xu: but for old deployer timestamp will not be changed right? | 01:05 |
| *** ijw has quit IRC | 01:05 | |
| gmann | alex_xu: where we can see those warning so that we can verify the same | 01:05 |
| alex_xu | gmann: emm...they should get a warning after upgrade, due to 'saved_file_rules' is empty | 01:05 |
| gmann | alex_xu: ah yea, thats is nice point | 01:07 |
| *** mtanino has quit IRC | 01:10 | |
| *** browne has quit IRC | 01:10 | |
| *** tblakes has joined #openstack-nova | 01:12 | |
| artom | mriedem, is one week really too little? | 01:16 |
| artom | I've addressed gmann's and alex_xu's feedback :) | 01:16 |
| gmann | artom: +1, ll try to check today. m sure that is very close | 01:17 |
| artom | The patch itself is fairly straightforward, if discussion is needed I think it's more about the strategy | 01:17 |
| mriedem | artom: w/o looking at the change i can't really say | 01:18 |
| mriedem | artom: we should have a spec for the thing too since it's a new microversion right? | 01:18 |
| artom | My understanding is that a new microversion is the prefered approach | 01:18 |
| mriedem | yes it is | 01:18 |
| artom | mriedem, a spec for that seems... pulled by the hair, as we say in Quebec | 01:18 |
| *** dimtruck is now known as zz_dimtruck | 01:18 | |
| artom | But if folks want, I'll hack one up :) | 01:19 |
| mriedem | artom: let them eat process | 01:19 |
| artom | mriedem, although if we do a spec, there's no chance of it landing in ocata? | 01:20 |
| mriedem | there is | 01:20 |
| * artom is confused... what happened to eating process? | 01:20 | |
| mriedem | artom: fwiw, | 01:21 |
| *** zz_dimtruck is now known as dimtruck | 01:21 | |
| mriedem | https://specs.openstack.org/openstack/nova-specs/specs/newton/implemented/response-for-invalid-status.html was for a bug fix that required a microversion in newton | 01:21 |
| mriedem | it should be a relatively small and straight-forward spec | 01:21 |
| mriedem | just like ^ | 01:21 |
| artom | mriedem, PRECEDENT!!11 | 01:21 |
| mriedem | yar | 01:23 |
| *** mriedem is now known as mriedem_afk | 01:23 | |
| *** dtp has quit IRC | 01:23 | |
| *** ijw has joined #openstack-nova | 01:26 | |
| *** thorst_ has joined #openstack-nova | 01:27 | |
| *** raunak has quit IRC | 01:28 | |
| edleafe | mriedem_afk: almost done with the fixes | 01:29 |
| *** ijw has quit IRC | 01:30 | |
| *** edmondsw has joined #openstack-nova | 01:31 | |
| *** edmondsw has quit IRC | 01:43 | |
| *** liangy has joined #openstack-nova | 01:44 | |
| *** Swami_ has joined #openstack-nova | 01:45 | |
| *** Swami_ has quit IRC | 01:46 | |
| *** Swami has quit IRC | 01:48 | |
| *** edmondsw has joined #openstack-nova | 01:54 | |
| *** tbachman has quit IRC | 01:54 | |
| *** Sukhdev_ has quit IRC | 01:54 | |
| *** tbachman has joined #openstack-nova | 01:55 | |
| *** tbachman has quit IRC | 01:55 | |
| *** tbachman has joined #openstack-nova | 01:55 | |
| openstackgerrit | Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 01:56 |
| edleafe | ^^ and there they are | 01:56 |
| *** tbachman has quit IRC | 01:59 | |
| diana_clarke | I have what is probably a stupid question... when I do a `ps aux | grep nova-api` I don't see the nova-api-metadata service, but I can still successfully query the metadata service... why? | 02:03 |
| *** tbachman has joined #openstack-nova | 02:04 | |
| *** kaisers has quit IRC | 02:05 | |
| *** hfu has joined #openstack-nova | 02:05 | |
| diana_clarke | (asking b/c I'm looking at the # of processes started when you change osapi_compute_workers vs. metadata_workers in the nova conf) | 02:06 |
| *** tbachman_ has joined #openstack-nova | 02:08 | |
| dansmith | diana_clarke: http://logs.openstack.org/30/424730/10/check/gate-grenade-dsvm-neutron-ubuntu-xenial/e3caf96/logs/etc/nova/nova.conf.txt.gz | 02:08 |
| dansmith | diana_clarke: runs them both in a single process | 02:08 |
| dansmith | (he said authoritatively, despite being a little fuzzy on that stuff) | 02:09 |
| dansmith | diana_clarke: enabled_apis = osapi_compute,metadata | 02:09 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 02:09 |
| *** tbachman has quit IRC | 02:09 | |
| *** tbachman_ is now known as tbachman | 02:09 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 02:12 |
| *** dims_ has joined #openstack-nova | 02:14 | |
| *** dims has quit IRC | 02:14 | |
| *** thorst_ has quit IRC | 02:18 | |
| *** thorst_ has joined #openstack-nova | 02:20 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extentions https://review.openstack.org/425876 | 02:22 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Fix tag attribute disappearing in 2.33 and 2.37 https://review.openstack.org/424759 | 02:22 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 02:23 |
| diana_clarke | dansmith: sorry for quick absence (was trying to get baby to bed), scrolling back | 02:24 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add new policy for server list/detail with all_tenants https://review.openstack.org/415330 | 02:24 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add new policy for server list/detail with all_tenants https://review.openstack.org/415330 | 02:25 |
| *** sapcc-bot1 has quit IRC | 02:27 | |
| *** sapcc-bot has joined #openstack-nova | 02:27 | |
| *** yamahata has quit IRC | 02:29 | |
| *** tblakes has quit IRC | 02:29 | |
| *** dimtruck is now known as zz_dimtruck | 02:30 | |
| diana_clarke | dansmith: My hunch is that a new nova-api process is started for each metadata_worker, but /usr/bin/nova-api-metadata exists too (but isn't a running process). I gather it can be run independently though in some cases. I dunno... | 02:34 |
| *** baoli has joined #openstack-nova | 02:35 | |
| dansmith | it's used when you need them separate | 02:35 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 02:35 |
| *** zz_dimtruck is now known as dimtruck | 02:36 | |
| *** Apoorva has quit IRC | 02:39 | |
| diana_clarke | dansmith: hmmm... I think I get it now. Kind of. I'm not sure if the # of nova-api processes is osapi_compute_workers + metadata_workers + 1 or max(osapi_compute_workers, metadata_workers) + 1 | 02:41 |
| diana_clarke | dansmith: and maybe maybe nproc'd, but I think I'm getting closer. Thanks! | 02:41 |
| diana_clarke | oops, Jacob Two-Two | 02:42 |
| *** edmondsw has quit IRC | 02:42 | |
| *** edmondsw has joined #openstack-nova | 02:43 | |
| *** edmondsw has quit IRC | 02:43 | |
| *** edmondsw has joined #openstack-nova | 02:43 | |
| *** baoli has quit IRC | 02:45 | |
| *** dims_ has quit IRC | 02:47 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 02:47 |
| *** d-bark has joined #openstack-nova | 02:47 | |
| *** dims has joined #openstack-nova | 02:50 | |
| *** edmondsw has quit IRC | 02:52 | |
| *** ijw has joined #openstack-nova | 02:55 | |
| openstackgerrit | Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 02:55 |
| *** tbachman has quit IRC | 02:58 | |
| *** ijw has quit IRC | 02:59 | |
| *** hongbin has joined #openstack-nova | 03:01 | |
| *** edmondsw has joined #openstack-nova | 03:01 | |
| *** raunak has joined #openstack-nova | 03:06 | |
| *** baoli has joined #openstack-nova | 03:08 | |
| *** edmondsw has quit IRC | 03:09 | |
| *** edmondsw has joined #openstack-nova | 03:11 | |
| *** yamahata has joined #openstack-nova | 03:13 | |
| *** edmondsw has quit IRC | 03:14 | |
| *** edmondsw has joined #openstack-nova | 03:14 | |
| *** edmondsw has quit IRC | 03:14 | |
| *** edmondsw has joined #openstack-nova | 03:15 | |
| *** mtanino has joined #openstack-nova | 03:15 | |
| *** mtanino has quit IRC | 03:21 | |
| mriedem_afk | dansmith: cells is now 21 in the gate huh | 03:30 |
| *** mriedem_afk is now known as mriedem | 03:30 | |
| mriedem | not sure there is really anything else i'm going to work on tonight | 03:30 |
| mriedem | there was an old shitty steve segal movie on the other night and i'm hoping for some similar magic tonight | 03:30 |
| mriedem | *steven - we're not on an informal name basis | 03:30 |
| dansmith | heh | 03:31 |
| dansmith | mriedem: yeah, it's moving slow | 03:31 |
| dansmith | mriedem: I think this attempt on the grenade patch is going to work | 03:31 |
| dansmith | just like I thought the last 12 were going to work | 03:31 |
| dansmith | I'm about done for the night though | 03:31 |
| *** yarkot has joined #openstack-nova | 03:31 | |
| *** pbandark has joined #openstack-nova | 03:32 | |
| mriedem | ooo a new one eh | 03:36 |
| *** guchihiro has joined #openstack-nova | 03:37 | |
| *** tbachman has joined #openstack-nova | 03:37 | |
| diana_clarke | mriedem: I'm sad you've worked so much from mexico ;( I submitted one patch from the airport, but then fully embraced beach life. | 03:39 |
| mriedem | i can only swim so much in the same 3 pools and single pacific ocean | 03:39 |
| *** thorst_ has quit IRC | 03:40 | |
| diana_clarke | My favorite part was ditching the car for golf-cart transportation. But I was in a private city, on a private street, so it wasn;t really mexico... | 03:40 |
| *** hongbin has quit IRC | 03:41 | |
| *** hongbin has joined #openstack-nova | 03:41 | |
| *** amotoki has quit IRC | 03:43 | |
| *** gouthamr has quit IRC | 03:43 | |
| mriedem | edleafe: thanks for the updates, i'll review in detail in the morning | 03:45 |
| mriedem | diana_clarke: i successfully ordered a cheese, and only cheese, quesadilla for my daughter using all spanish words - i made sure to gloat to my wife | 03:46 |
| mriedem | high school spanish still works | 03:46 |
| *** liangy has quit IRC | 03:47 | |
| dansmith | mriedem: well, the race between the subnode and the scheduler must not be the problem, | 03:55 |
| dansmith | because we're not having to wait at all for that to happen | 03:55 |
| diana_clarke | mriedem: Nice! I really regret not having tried to learn some spanish beforehand (I took german in school). I'm now doing 10 minutes per day for when I go back. | 03:55 |
| diana_clarke | (hoping to go back, I'm also not posting the pics I got of the mexican president online, surreal town was where I was at, lol) | 04:00 |
| *** dimtruck is now known as zz_dimtruck | 04:00 | |
| mriedem | dansmith: ok - i haven't wrapped my head around the grenade change yet and how that fixes the scheduler thing, or is meant to, but i think i get it, basically wait for as many enabled nova-compute services to show up after the upgrade as before, because the new compute service will go down and come back up so we'd wait for that | 04:03 |
| mriedem | anyway i'm calling it a night | 04:04 |
| diana_clarke | mriedem: If you find yourself online tomorrow, let me know if there is something I can do, so you can swim, etc. cheers | 04:06 |
| *** zz_dimtruck is now known as dimtruck | 04:07 | |
| *** dimtruck is now known as zz_dimtruck | 04:08 | |
| *** zz_dimtruck is now known as dimtruck | 04:09 | |
| *** edmondsw has quit IRC | 04:17 | |
| *** edmondsw has joined #openstack-nova | 04:17 | |
| *** nicolasbock has quit IRC | 04:17 | |
| gmann | alex_xu: mriedem johnthetubaguy - its ready now, can you check - https://review.openstack.org/#/c/415330/27 | 04:20 |
| *** edmondsw has quit IRC | 04:21 | |
| *** mtreinish has quit IRC | 04:22 | |
| *** mtreinish has joined #openstack-nova | 04:23 | |
| *** psachin has joined #openstack-nova | 04:24 | |
| *** zerda3 has joined #openstack-nova | 04:29 | |
| *** zerda2 has quit IRC | 04:29 | |
| *** mdnadeem has joined #openstack-nova | 04:30 | |
| *** ayogi has joined #openstack-nova | 04:50 | |
| *** baoli_ has joined #openstack-nova | 04:53 | |
| *** baoli has quit IRC | 04:55 | |
| *** hongbin has quit IRC | 04:57 | |
| *** udesale has joined #openstack-nova | 04:58 | |
| *** yamahata has quit IRC | 05:01 | |
| *** armax has quit IRC | 05:05 | |
| *** adisky_ has joined #openstack-nova | 05:11 | |
| *** prateek has joined #openstack-nova | 05:11 | |
| *** ratailor has joined #openstack-nova | 05:12 | |
| *** raunak has quit IRC | 05:26 | |
| *** sree has joined #openstack-nova | 05:27 | |
| *** dimtruck is now known as zz_dimtruck | 05:27 | |
| *** tbachman has quit IRC | 05:29 | |
| *** raunak has joined #openstack-nova | 05:29 | |
| *** artom has quit IRC | 05:42 | |
| *** david-lyle has quit IRC | 05:46 | |
| *** artom has joined #openstack-nova | 05:46 | |
| *** jaypipes_ has quit IRC | 05:53 | |
| *** karthiks has joined #openstack-nova | 05:53 | |
| *** kaisers has joined #openstack-nova | 05:54 | |
| *** kaisers has quit IRC | 05:55 | |
| *** Jack_V has joined #openstack-nova | 05:56 | |
| *** thorst_ has joined #openstack-nova | 05:56 | |
| *** thorst_ has quit IRC | 06:01 | |
| *** jaypipes_ has joined #openstack-nova | 06:02 | |
| *** jaypipes_ has quit IRC | 06:08 | |
| *** karthiks has quit IRC | 06:09 | |
| *** jaypipes_ has joined #openstack-nova | 06:11 | |
| *** sridharg has joined #openstack-nova | 06:22 | |
| *** vsaienko has joined #openstack-nova | 06:25 | |
| *** claudiub has joined #openstack-nova | 06:25 | |
| *** lpetrut has joined #openstack-nova | 06:26 | |
| *** claudiub|2 has joined #openstack-nova | 06:28 | |
| *** claudiub has quit IRC | 06:31 | |
| *** Sukhdev has joined #openstack-nova | 06:34 | |
| *** vsaienko has quit IRC | 06:37 | |
| *** avolkov has joined #openstack-nova | 06:40 | |
| *** vsaienko has joined #openstack-nova | 06:42 | |
| *** unicell1 has quit IRC | 06:46 | |
| *** moshele has joined #openstack-nova | 06:48 | |
| *** kaisers has joined #openstack-nova | 06:55 | |
| *** nkrinner_afk is now known as nkrinner | 06:56 | |
| *** moshele has quit IRC | 06:57 | |
| *** karthiks has joined #openstack-nova | 06:58 | |
| *** mjura has joined #openstack-nova | 07:01 | |
| *** vsaienko has quit IRC | 07:03 | |
| openstackgerrit | Pawel Koniszewski proposed openstack/nova: Mark live_migration_downtime_steps/delay as deprecated for removal https://review.openstack.org/408002 | 07:06 |
| *** amotoki has joined #openstack-nova | 07:07 | |
| *** hfu has quit IRC | 07:09 | |
| *** diga has joined #openstack-nova | 07:09 | |
| *** Sukhdev has quit IRC | 07:10 | |
| *** diga has quit IRC | 07:12 | |
| *** diga has joined #openstack-nova | 07:12 | |
| *** hfu has joined #openstack-nova | 07:17 | |
| *** kaisers has quit IRC | 07:21 | |
| *** rcernin has quit IRC | 07:24 | |
| *** diga has quit IRC | 07:26 | |
| *** andreas_s has joined #openstack-nova | 07:26 | |
| *** adisky_ has quit IRC | 07:29 | |
| *** kaisers has joined #openstack-nova | 07:36 | |
| *** mfeoktistov|2 has joined #openstack-nova | 07:45 | |
| *** ralonsoh has joined #openstack-nova | 07:50 | |
| *** sahid has joined #openstack-nova | 07:56 | |
| *** thorst_ has joined #openstack-nova | 07:57 | |
| *** markus_z has joined #openstack-nova | 07:59 | |
| *** vsaienko has joined #openstack-nova | 08:00 | |
| *** takashin has left #openstack-nova | 08:00 | |
| *** thorst_ has quit IRC | 08:01 | |
| *** amotoki has quit IRC | 08:05 | |
| *** diga has joined #openstack-nova | 08:11 | |
| *** tanee is now known as tanee_away | 08:13 | |
| *** jpena|off is now known as jpena | 08:19 | |
| *** vsaienko has quit IRC | 08:28 | |
| *** vsaienko has joined #openstack-nova | 08:28 | |
| *** gszasz has joined #openstack-nova | 08:31 | |
| *** jpena is now known as jpena|off | 08:32 | |
| *** owalsh_ is now known as owalsh | 08:36 | |
| *** adisky_ has joined #openstack-nova | 08:36 | |
| *** sree has quit IRC | 08:36 | |
| *** sree has joined #openstack-nova | 08:37 | |
| *** vsaienko has quit IRC | 08:38 | |
| *** jpena|off is now known as jpena | 08:39 | |
| *** sree has quit IRC | 08:41 | |
| *** vsaienko has joined #openstack-nova | 08:41 | |
| *** guchihiro has left #openstack-nova | 08:44 | |
| *** zzzeek has quit IRC | 09:00 | |
| *** rmart04 has joined #openstack-nova | 09:01 | |
| *** zzzeek has joined #openstack-nova | 09:01 | |
| *** tanee_away is now known as tanee | 09:05 | |
| *** claudiub has joined #openstack-nova | 09:08 | |
| *** gszasz has quit IRC | 09:09 | |
| *** claudiub|2 has quit IRC | 09:11 | |
| *** karimb has joined #openstack-nova | 09:16 | |
| *** gszasz has joined #openstack-nova | 09:17 | |
| *** Cristina_ has quit IRC | 09:20 | |
| *** efoley has joined #openstack-nova | 09:22 | |
| *** gszasz has quit IRC | 09:23 | |
| *** moshele has joined #openstack-nova | 09:26 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova: Make policy 'all_tenants_visible' a soft enforcement rule https://review.openstack.org/426128 | 09:28 |
| *** jaypipes_ has quit IRC | 09:31 | |
| *** lucas-afk is now known as lucasagomes | 09:32 | |
| *** amotoki has joined #openstack-nova | 09:35 | |
| *** karthiks has quit IRC | 09:35 | |
| *** derekh has joined #openstack-nova | 09:36 | |
| *** amotoki has quit IRC | 09:45 | |
| *** hfu has quit IRC | 09:47 | |
| andreykurilin | amrith: hi! | 09:48 |
| *** hfu has joined #openstack-nova | 09:48 | |
| *** hfu has quit IRC | 09:49 | |
| *** hfu has joined #openstack-nova | 09:49 | |
| *** hfu has quit IRC | 09:49 | |
| *** CristinaPauna has joined #openstack-nova | 09:51 | |
| *** hfu has joined #openstack-nova | 09:51 | |
| *** hfu has quit IRC | 09:51 | |
| *** karthiks has joined #openstack-nova | 09:51 | |
| *** baoli_ has quit IRC | 09:51 | |
| *** claudiub has quit IRC | 09:52 | |
| *** gabor_antal has joined #openstack-nova | 09:52 | |
| johnthetubaguy | bauzas: looking at the patch of doom, eek | 09:54 |
| johnthetubaguy | whats going on there :S | 09:54 |
| *** claudiub has joined #openstack-nova | 09:54 | |
| bauzas | johnthetubaguy: in a train to Paris so probably a bit lagging | 09:55 |
| bauzas | johnthetubaguy: but lemme explain the problem | 09:55 |
| johnthetubaguy | bauzas: I remember you say that, safe journey | 09:55 |
| johnthetubaguy | looks like we get the wrong min version? | 09:55 |
| bauzas | johnthetubaguy: exactly | 09:56 |
| bauzas | we get 16 as minimum | 09:56 |
| bauzas | so we call the placement API | 09:56 |
| johnthetubaguy | super interesting if thats broken | 09:56 |
| bauzas | but we should have 15 given we have an old subnode | 09:56 |
| * johnthetubaguy nods | 09:56 | |
| bauzas | what I'd love is some way to introspect more | 09:57 |
| bauzas | like LOG.debugging the DB call itself | 09:57 |
| johnthetubaguy | bauzas: we could add a line in there that lists all the nova compute service records | 09:57 |
| bauzas | I can throw a patch but I have very limited access | 09:57 |
| johnthetubaguy | I can do something if that fails | 09:57 |
| johnthetubaguy | I assume we are still trying to get this in ASAP | 09:58 |
| johnthetubaguy | its kinda release critical in my book | 09:58 |
| bauzas | yup, mriedem said he granted a FFE | 09:58 |
| *** thorst_ has joined #openstack-nova | 09:58 | |
| johnthetubaguy | cool | 09:58 |
| johnthetubaguy | I guess this is the only one so far right? | 09:58 |
| *** sree has joined #openstack-nova | 09:58 | |
| bauzas | yeah, my one-day trip to Paris is terrible :/ | 09:58 |
| johnthetubaguy | it seems a safe date to pick | 09:58 |
| bauzas | mriedem said he'd love also custom-rc | 09:58 |
| johnthetubaguy | that rest day after FFEE | 09:58 |
| johnthetubaguy | oh, not been tracking those patches | 09:59 |
| johnthetubaguy | that would be good, for iroinc | 09:59 |
| bauzas | yup | 09:59 |
| bauzas | tbh, me neither | 09:59 |
| johnthetubaguy | bauzas: so if you can't push a patch, let me know, and I can try something, but I am just going to do some log digging on your patch | 10:00 |
| bauzas | johnthetubaguy: I provided the line to look at | 10:00 |
| bauzas | johnthetubaguy: in the gerrit comment | 10:01 |
| johnthetubaguy | yeah, seen that | 10:01 |
| bauzas | johnthetubaguy: the one matching the req-id that fails | 10:01 |
| bauzas | you think it'd be better to introspect within the DB method, or outside of that? | 10:01 |
| bauzas | like in the caller? | 10:02 |
| bauzas | sec | 10:02 |
| bauzas | I think of something | 10:02 |
| johnthetubaguy | I would go for the fool proof thing, and just log out a list of services | 10:02 |
| *** thorst_ has quit IRC | 10:02 | |
| bauzas | that's a remotable method, right? | 10:02 |
| bauzas | so we expect to have it run by the conductor | 10:03 |
| bauzas | but | 10:03 |
| bauzas | it's within the scheduler | 10:03 |
| bauzas | where the indirection api is not set | 10:03 |
| bauzas | so we directly lookup the DB | 10:03 |
| johnthetubaguy | bauzas: is it just me, or do the old and new nodes both have the same CONF.host setting? | 10:03 |
| bauzas | johnthetubaguy: it's not | 10:03 |
| bauzas | johnthetubaguy: or it shouldn't | 10:04 |
| bauzas | but I briefly looked once at the subnode n-cpu log, and I saw it having the right name | 10:04 |
| johnthetubaguy | oops, I was on the wrong log | 10:04 |
| bauzas | that test is actually pretty dummy | 10:04 |
| bauzas | it just takes all the computes, and force an instance to boot against each one | 10:05 |
| johnthetubaguy | http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-sch.txt.gz#_2017-01-27_03_42_11_626 | 10:05 |
| johnthetubaguy | http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/subnode-2/old/screen-n-cpu.txt.gz#_2017-01-27_03_31_28_948 | 10:05 |
| bauzas | so it's passing on the new node, but not on the newton subnode | 10:05 |
| johnthetubaguy | do we truncate the hostname? | 10:05 |
| bauzas | I don't think so | 10:06 |
| bauzas | oh wait | 10:06 |
| bauzas | ergh, I'm almost arriving to Paris | 10:06 |
| bauzas | I should loose some connectivity soon | 10:07 |
| bauzas | let's add a bit of logging before my connection drops | 10:07 |
| * bauzas rushes | 10:07 | |
| openstackgerrit | Sylvain Bauza proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 10:13 |
| bauzas | johnthetubaguy: I'm loosing connectivity, but ^ | 10:14 |
| bauzas | johnthetubaguy: ttyl once I'm arriving at the office | 10:14 |
| bauzas | we won't get a lot of details by that call but hopefully, we would get enough info to see if the service is found | 10:15 |
| johnthetubaguy | +1 thats the hack I was thinking about | 10:15 |
| *** d-bark has quit IRC | 10:17 | |
| *** karimb_ has joined #openstack-nova | 10:17 | |
| *** karimb has quit IRC | 10:18 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova: Add release note and docs for filter/sort whitelist https://review.openstack.org/421760 | 10:20 |
| gmann | johnthetubaguy: alex_xu mriedem : all_tenant things are ready now - https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:bp/add-whitelist-for-server-list-filter-sort-parameters | 10:21 |
| *** lpetrut has quit IRC | 10:21 | |
| gmann | johnthetubaguy: alex_xu mriedem : updated spec and new version bump with 'no 403' on new rule | 10:21 |
| gmann | johnthetubaguy: alex_xu mriedem : this one with spec patch also - https://review.openstack.org/#/q/status:open+branch:master+topic:bp/add-whitelist-for-server-list-filter-sort-parameters | 10:22 |
| *** vsaienko has quit IRC | 10:22 | |
| *** cdent has joined #openstack-nova | 10:23 | |
| *** karimb_ has quit IRC | 10:24 | |
| *** sree_ has joined #openstack-nova | 10:24 | |
| *** sree_ is now known as Guest20358 | 10:25 | |
| *** sree has quit IRC | 10:26 | |
| johnthetubaguy | gmann: thanks for fixing that up, I will try get to that | 10:27 |
| *** karthiks has quit IRC | 10:27 | |
| johnthetubaguy | just chasing the approved FFEs | 10:27 |
| gmann | johnthetubaguy: thanks, ll check tomorrow on those if any comments | 10:28 |
| cdent | I'm available to be directed to whatever matters by whomever. | 10:36 |
| *** karthiks has joined #openstack-nova | 10:39 | |
| *** pkoniszewski has quit IRC | 10:41 | |
| *** pkoniszewski has joined #openstack-nova | 10:42 | |
| johnthetubaguy | bauzas: http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-cpu.txt#_2017-01-27_03_42_12_029 | 10:44 |
| johnthetubaguy | cdent: I think the priorities are the custom resource patches and bauzas's call placement patch | 10:44 |
| johnthetubaguy | seems like we get the min_service version wrong somehow | 10:45 |
| johnthetubaguy | but the plot thickens | 10:45 |
| cdent | eeeph | 10:45 |
| johnthetubaguy | bauzas: then later you see this: http://logs.openstack.org/61/417961/32/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/bee8a26/logs/new/screen-n-cpu.txt#_2017-01-27_03_42_14_659 | 10:45 |
| johnthetubaguy | I mean if the min_service version stuff is broken, thats a big deal for other things | 10:48 |
| johnthetubaguy | so it could be a good find, or a total red herring, but thats life I guess | 10:48 |
| *** guchihiro has joined #openstack-nova | 10:50 | |
| openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | 10:55 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Implement new attach/detach Cinder flow https://review.openstack.org/330285 | 10:56 |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova: Remove check_attach https://review.openstack.org/335358 | 10:56 |
| *** ociuhandu has quit IRC | 10:58 | |
| openstackgerrit | Andy McCrae proposed openstack/nova: Allow placement endpoint interface to be set https://review.openstack.org/426163 | 10:59 |
| *** udesale has quit IRC | 11:01 | |
| *** karimb has joined #openstack-nova | 11:03 | |
| *** mvk has quit IRC | 11:06 | |
| cdent | johnthetubaguy: It looks like edleafe has addressed mriedem's concers on https://review.openstack.org/#/c/404472/ and I just gave it read through and nothing lept out as crazy, and the voiting bits of jenkins are happy, so it might be gtg | 11:12 |
| johnthetubaguy | cdent: ah, sweet, I should look into that one | 11:12 |
| *** hfu has joined #openstack-nova | 11:13 | |
| *** sambetts|afk is now known as sambetts | 11:14 | |
| johnthetubaguy | cdent: edleafe: I wonder if we should add a reno note for that patch, not a blocker as it can be a follow on, but worth squaring that away | 11:18 |
| cdent | johnthetubaguy: perhaps, but if we're looking for an excuse to why not, this change has no visible change on end behavior (you have to look in the db or placement api with specific intent to know). | 11:20 |
| johnthetubaguy | cdent: good point | 11:21 |
| johnthetubaguy | "hey that thing you didn't know way rubbish, we fixed it, yey us" | 11:21 |
| cdent | heh | 11:22 |
| *** lpetrut has joined #openstack-nova | 11:23 | |
| cdent | bauzas, johnthetubaguy : the current strategy on the scheduler changes is to watch the debug output and from that figure out what's next? | 11:24 |
| johnthetubaguy | cdent: yeah | 11:24 |
| johnthetubaguy | I tralled through the code and logs a bit this morning, haven't seem any more clues as such, just more things that confuse me | 11:25 |
| johnthetubaguy | I haven't double checked that we correctly have the min_version caching working, that was one thread I haven't double checked | 11:25 |
| cdent | is here a chance that the old version can be cached, leaving the cache wrong? that is, the scheduler starts up while a node is old and caches the bad version? | 11:27 |
| cdent | we could test that by turning off the caching? | 11:27 |
| cdent | that does seem possible | 11:30 |
| cdent | assuming the cache is local to the current process | 11:30 |
| *** yassine has joined #openstack-nova | 11:31 | |
| *** yassine is now known as Guest21909 | 11:32 | |
| *** sree has joined #openstack-nova | 11:32 | |
| cdent | johnthetubaguy: yeah, I'm pretty sure the above is possible, because unlike the conductor, the scheduler isn't required to listen to a hup to reset the cache. This is the review that put the cache in https://review.openstack.org/#/c/265326/ | 11:34 |
| *** Guest20358 has quit IRC | 11:34 | |
| *** lpetrut has quit IRC | 11:34 | |
| *** lpetrut has joined #openstack-nova | 11:35 | |
| johnthetubaguy | cdent: so in this case we want to ensure we get the older version | 11:35 |
| *** tbachman has joined #openstack-nova | 11:35 | |
| johnthetubaguy | cdent: so I was expecting the scheduler to help really | 11:35 |
| johnthetubaguy | I mean the cache to help | 11:35 |
| cdent | oh. hmm. | 11:35 |
| cdent | I guess that means we have two different problems :) | 11:36 |
| cdent | because now we still need to make sure the cache becomes right after the final compute node becomes modern? | 11:36 |
| *** tbachman has quit IRC | 11:41 | |
| johnthetubaguy | thats always going to require a restart anyways | 11:46 |
| johnthetubaguy | for the compute RPC api version | 11:46 |
| cdent | johnthetubaguy: yesterday we had a lot of discussions about what happens when people are "accidentally" upgrading stuff | 11:47 |
| cdent | lots of test failures in in unit tests and things on that latest run of 417961, since bauzas is on a train should I get that stuff? | 11:48 |
| *** smatzek has joined #openstack-nova | 11:52 | |
| *** moshele has quit IRC | 11:53 | |
| johnthetubaguy | cdent: I think its due to a hack, those failures | 11:58 |
| * cdent is unsure of what to do :) | 11:58 | |
| *** thorst_ has joined #openstack-nova | 11:59 | |
| johnthetubaguy | bummer | 11:59 |
| johnthetubaguy | ServersTestManualDisk-1134536160] Bauzas: all services: ServiceList(objects=[Service(1),Service(2),Service(3),Service(4),Service(5),Service(6),Service(7),Service(8)]) _get_all_host_states | 11:59 |
| johnthetubaguy | thats not what we wanted there | 11:59 |
| cdent | ugh, yeah, that's not very helpful | 12:00 |
| *** guchihiro has quit IRC | 12:00 | |
| johnthetubaguy | maybe fix up str(service) ? | 12:00 |
| johnthetubaguy | then we should find out the versions its seeing | 12:01 |
| johnthetubaguy | http://logs.openstack.org/61/417961/33/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/a2da154/logs/new/screen-n-sch.txt.gz#_2017-01-27_10_48_09_048 | 12:01 |
| cdent | Is there some way we can model some of this without having to use the gate? | 12:01 |
| *** BobBall_AWOL is now known as BobBall | 12:02 | |
| *** sree has quit IRC | 12:02 | |
| johnthetubaguy | if you have an old compute node running, and a new compute node running, and something to query the DB using objects, that should give us the same thing | 12:02 |
| johnthetubaguy | just at this point, I really don't get whats happening, so its hard to know what we could fake | 12:02 |
| *** thorst_ has quit IRC | 12:03 | |
| johnthetubaguy | its tempting to also log service versions in the service updates, just to be sure | 12:03 |
| johnthetubaguy | I can do those changes, one sec | 12:03 |
| *** sree has joined #openstack-nova | 12:03 | |
| *** catintheroof has joined #openstack-nova | 12:04 | |
| *** mvk has joined #openstack-nova | 12:04 | |
| *** sree has quit IRC | 12:04 | |
| *** sree has joined #openstack-nova | 12:04 | |
| *** nicolasbock has joined #openstack-nova | 12:04 | |
| *** sree has quit IRC | 12:04 | |
| *** khushbu has joined #openstack-nova | 12:05 | |
| *** diga has quit IRC | 12:16 | |
| *** jpena is now known as jpena|lunch | 12:17 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 12:18 |
| *** bvanhav has joined #openstack-nova | 12:18 | |
| cdent | I hope dong.wenjuan is having a nice vacation | 12:18 |
| johnthetubaguy | cdent: I think these should help: https://review.openstack.org/#/c/417961/33..34/ | 12:18 |
| johnthetubaguy | cdent: if you could check for fat fingers, that would be awesome | 12:18 |
| * cdent is looking | 12:18 | |
| *** sridharg has quit IRC | 12:19 | |
| cdent | johnthetubaguy++ | 12:20 |
| johnthetubaguy | I just thought, I should add something in the service object save too | 12:20 |
| johnthetubaguy | to make sure it gets across RPC OK | 12:21 |
| *** lucasagomes is now known as lucas-hungry | 12:21 | |
| johnthetubaguy | cdent: thanks for checking, fingers crossed on this while I get some lunch | 12:22 |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 12:22 |
| * johnthetubaguy bravely runs away | 12:23 | |
| * cdent sings | 12:23 | |
| johnthetubaguy | heh | 12:24 |
| *** moshele has joined #openstack-nova | 12:27 | |
| *** moshele has quit IRC | 12:29 | |
| *** sdague has joined #openstack-nova | 12:40 | |
| *** thorst__ has joined #openstack-nova | 12:43 | |
| *** ociuhandu has joined #openstack-nova | 12:46 | |
| *** vladikr has quit IRC | 12:50 | |
| *** dakmetov has joined #openstack-nova | 12:52 | |
| *** sdague_ has joined #openstack-nova | 12:52 | |
| dakmetov | greetings everyone! | 12:53 |
| *** tbachman has joined #openstack-nova | 12:54 | |
| *** andreas_s has quit IRC | 12:55 | |
| *** cdent has left #openstack-nova | 13:01 | |
| *** cdent has joined #openstack-nova | 13:01 | |
| *** psachin has quit IRC | 13:02 | |
| *** ratailor has quit IRC | 13:03 | |
| dakmetov | 2 mbooth: thank you for the great review! | 13:03 |
| *** cdent has quit IRC | 13:04 | |
| dakmetov | 2 mbooth: could you please clarify one thing: https://review.openstack.org/#/c/399679/5/nova/virt/libvirt/driver.py | 13:04 |
| *** cdent has joined #openstack-nova | 13:04 | |
| dakmetov | 2 mbooth: what do you mean under 'source info'? | 13:04 |
| *** catintheroof has quit IRC | 13:05 | |
| cdent | johnthetubaguy: something weird: http://logs.openstack.org/61/417961/35/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/7e52701/logs/new/screen-n-sch.txt.gz#_2017-01-27_12_55_20_946 | 13:05 |
| cdent | min is showing as 16, but a compute is 15! | 13:05 |
| wznoinsk | sfinucan, hi | 13:08 |
| sfinucan | wznoinsk: hey | 13:09 |
| sfinucan | what's up? | 13:09 |
| bhagyashris | cdent: Hi, Is the create resource_provider available for the user or it's only for the internal api for service? | 13:09 |
| cdent | bhagyashris: It's possible for anything with the admin role to create resource providers, but at the moment there's not a lot of reason to do so. Long term the idea is that resource providers for things like shared disk farms will exist | 13:11 |
| bhagyashris | cdent: And Is it necessary that the resource_provider name should be the host name? | 13:12 |
| wznoinsk | sfinucan, I saw your chat with vladikr yesterday, I'm guessing he has some hardware dependent tests, was that passthrough and alike or more of networking like sriov work? | 13:13 |
| *** khushbu has quit IRC | 13:13 | |
| cdent | bhagyashris: the name of the resource provider is there for humans to remember and filter on as needed. The resource providers that are automatically created for compute nodes use the hostname. | 13:14 |
| *** catintheroof has joined #openstack-nova | 13:16 | |
| *** tlbr has joined #openstack-nova | 13:16 | |
| bhagyashris | cdent: Thank you for useful information. :) | 13:17 |
| cdent | you're welcome | 13:18 |
| *** lucas-hungry is now known as lucasagomes | 13:19 | |
| *** ayogi has quit IRC | 13:20 | |
| dakmetov | guys, is there any way to get user name and tenant name using userid and tenantid inside nova? | 13:21 |
| *** edmondsw has joined #openstack-nova | 13:21 | |
| dakmetov | without external lookups to keystone, I mean | 13:21 |
| *** pradk has joined #openstack-nova | 13:21 | |
| *** prateek has quit IRC | 13:22 | |
| *** jpena|lunch is now known as jpena | 13:22 | |
| *** edmondsw_ has joined #openstack-nova | 13:23 | |
| *** mdnadeem has quit IRC | 13:23 | |
| *** sdague has quit IRC | 13:25 | |
| *** edmondsw has quit IRC | 13:25 | |
| *** pbandark has quit IRC | 13:26 | |
| johnthetubaguy | cdent: crud, I think I know the problem... | 13:26 |
| cdent | johnthetubaguy: oh good, because I've been staring at that trying to figure it out to no avail. do tell? | 13:26 |
| johnthetubaguy | adding a comment, one sec | 13:27 |
| sfinucan | wznoinsk: So yeah, he's looking to test the exposing of VLAN tag through the metadata service | 13:27 |
| sfinucan | We're thinking that will require physical hardware. Not sure if it's something the Intel PCI CI could validate or not | 13:27 |
| johnthetubaguy | cdent: does my comment make sense? | 13:27 |
| johnthetubaguy | going to fix up now, and push it | 13:28 |
| cdent | oh hell | 13:28 |
| cdent | i hate it when it is something like that | 13:28 |
| *** gszasz has joined #openstack-nova | 13:29 | |
| cdent | I just assumed that was correct because all this service version stuff is new to me | 13:29 |
| cdent | savagely educated | 13:29 |
| johnthetubaguy | I read that three times and checked it | 13:29 |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 13:30 |
| johnthetubaguy | I like that phrase | 13:30 |
| johnthetubaguy | its a bit like used in anger | 13:30 |
| cdent | "bugger" is the best comment ever | 13:30 |
| johnthetubaguy | its the correct british way, I feel | 13:30 |
| cdent | quite | 13:30 |
| johnthetubaguy | one doesn't want to offend | 13:30 |
| cdent | unless one really needs to | 13:31 |
| johnthetubaguy | heh | 13:31 |
| cdent | we were nearly there | 13:31 |
| johnthetubaguy | so I left the debugging in, just in case | 13:31 |
| *** pradk has quit IRC | 13:31 | |
| cdent | good choice | 13:31 |
| cdent | edleafe: you surfaced yet? | 13:35 |
| markus_z | dansmith: If you find time to review: https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton+topic:%22bug+1455252%22 | 13:36 |
| *** khushbu_ has joined #openstack-nova | 13:37 | |
| *** gszasz has quit IRC | 13:40 | |
| *** jheroux has joined #openstack-nova | 13:40 | |
| markus_z | ^ or any other stable core, I'm not picky ;) | 13:42 |
| *** thorst__ is now known as thorst_ | 13:42 | |
| *** rwmjones has quit IRC | 13:43 | |
| *** gouthamr has joined #openstack-nova | 13:53 | |
| *** edleafe is now known as figleaf | 13:54 | |
| figleaf | cdent: I'm here | 13:54 |
| cdent | figleaf: just wanted to know if you're up to date on the above | 13:55 |
| figleaf | cdent: still reading... | 13:55 |
| figleaf | cdent: want to give me the summary? | 13:55 |
| cdent | figleaf: the only thing that real matters is: https://review.openstack.org/#/c/417961/35/nova/scheduler/filter_scheduler.py | 13:56 |
| cdent | (john's discovery) | 13:56 |
| figleaf | just reading the 'bugger' comment :) | 13:56 |
| figleaf | looks like johnthetubaguy has taken care of it | 13:57 |
| cdent | yes | 13:58 |
| figleaf | and I have a merge conflict on the last of the RP series https://review.openstack.org/#/c/414769/ | 13:58 |
| * johnthetubaguy still has fingers crossed for a test pass | 13:58 | |
| cdent | the debugging stuff that's right around bauzas patch is still in there for the current gate run, but if it happys will be cleared out | 13:58 |
| figleaf | cool | 13:59 |
| cdent | we're not trying to get nested rps in ocata, or are we? | 13:59 |
| figleaf | cdent: yes, but they won't be doing very much at all | 14:00 |
| cdent | then I guess my rp update 9 email is wrong, could you extend it? | 14:01 |
| *** xyang1 has joined #openstack-nova | 14:01 | |
| jroll | cdent: figleaf: safe to assume we didn't get the extra_specs=CUSTOM_FOO_RESOURCE_CLASS=1 thing done, right? | 14:02 |
| johnthetubaguy | the ironic custom stuff, I guess we don't query that either yet, but if its in this cycle we can query it next cycle? | 14:02 |
| johnthetubaguy | is looking at: https://review.openstack.org/#/c/404472 | 14:02 |
| jroll | johnthetubaguy: I think we have similar questions :) | 14:02 |
| cdent | jroll: the handling of extra specs, no, but the recording of inventory is still trying to get in | 14:02 |
| johnthetubaguy | jroll: heh, yup | 14:03 |
| *** bvanhav has quit IRC | 14:03 | |
| figleaf | cdent: dammit, you type too fast | 14:03 |
| jroll | cdent: right, so operators cannot change their flavors until pike, so we can't cut over how we schedule until q :( | 14:03 |
| *** kfarr has joined #openstack-nova | 14:03 | |
| jroll | with 'cut over' ~= 'drop cpu/ram/disk-based scheduling' | 14:04 |
| figleaf | jroll: wait - why can't they update their flavors? The new stuff will be ignored, but still... | 14:04 |
| cdent | jroll: I guess so? :( It's been really hard to track some of this | 14:04 |
| jroll | figleaf: because a flavor with cpu/ram/disk won't work in the new world, where ironic resource providers don't have cpu/ram/disk | 14:05 |
| jroll | it would require CUSTOM_FOO *and* 2 vcpu and 16384 rams, etc | 14:05 |
| *** pradk has joined #openstack-nova | 14:05 | |
| johnthetubaguy | so I am reading through, it looks like each nova-compute node can only look at one ironic resource_class, is that correct? | 14:06 |
| jroll | the old flavors are incompatible with the new | 14:06 |
| figleaf | jroll: ah, so they could update their flavors, but the flavors will have both the resource class name *and* the vm-ish stuff too | 14:06 |
| figleaf | johnthetubaguy: that's the idea | 14:07 |
| jroll | figleaf: yeah, something like that | 14:07 |
| johnthetubaguy | jroll: are we OK with nova-compute to resource class 1:1 mapping? | 14:08 |
| figleaf | jroll: the scheduler/placement will just ignore the former for now, and at some point, switch to ignoring the latter | 14:08 |
| *** vsaienko has joined #openstack-nova | 14:08 | |
| figleaf | johnthetubaguy: to be clear, there can be multiple nova-computes that are mapped to the same resource class | 14:08 |
| *** ratailor has joined #openstack-nova | 14:09 | |
| jroll | figleaf: I'm skeptical, how would it know to ignore ram/cpu for that flavor (but not for a virt flavor)? | 14:09 |
| jroll | johnthetubaguy: that seems... wrong. are we talking nova-compute service or compute_node entry? a compute service should handle more than one resource class fine | 14:10 |
| *** cleong has joined #openstack-nova | 14:10 | |
| jroll | there should be n:1 node:resource_class, and m:n service:resource_class | 14:10 |
| johnthetubaguy | jroll: that was my assumption | 14:11 |
| *** ratailor has quit IRC | 14:11 | |
| *** sdague_ is now known as sdague | 14:12 | |
| figleaf | jroll: there will probably be an "if _is_ironic():" check added to that code to look for the resource class | 14:12 |
| jroll | johnthetubaguy: I also don't see that n:1 service:resource_class in the code | 14:12 |
| johnthetubaguy | jroll: the resource class request's presence just always dominates any other request, that seems fine | 14:12 |
| cdent | Is it normal for us to be as confused about stuff as we've been? | 14:13 |
| johnthetubaguy | figleaf: I guess we can make it nested, once we have nested? | 14:14 |
| *** mdrabe has joined #openstack-nova | 14:14 | |
| johnthetubaguy | cdent: thats probably yes, I prefer the "can we do better at this" question, I hope thats a yes too | 14:14 |
| * cdent nods | 14:14 | |
| cdent | just checking | 14:15 |
| johnthetubaguy | jroll: I am looking here: https://review.openstack.org/#/c/404472/23/nova/compute/resource_tracker.py@525 | 14:15 |
| johnthetubaguy | cdent: its good to check these things | 14:15 |
| cdent | I'm trying to decide whether it is reasonable to be chill or panicky | 14:15 |
| *** owalsh is now known as owalsh-afk | 14:15 | |
| jroll | johnthetubaguy: yeah, so that's for a single ironic node, and that code will run once per ironic node | 14:15 |
| * cdent defaults to chill | 14:16 | |
| *** markvoelker has joined #openstack-nova | 14:16 | |
| openstackgerrit | Andrey Volkov proposed openstack/nova: WIP Update pci device fields from deleted object https://review.openstack.org/426243 | 14:16 |
| figleaf | cdent: it's more the overlapping terminology that confuses me | 14:16 |
| johnthetubaguy | jroll: doh, I see the loop now | 14:16 |
| figleaf | "compute" can mean multiple things | 14:17 |
| johnthetubaguy | figleaf: sorry, I see what you were trying to tell me now | 14:18 |
| jroll | figleaf: I've learned to always say compute *service* (nova-compute process) or compute *node* (the hypervisor/hardware), it seems to help | 14:18 |
| cdent | jroll: if you said compute service to me, I'd assume you meant all of nova | 14:18 |
| cdent | language is hard | 14:19 |
| jroll | cdent: ahem, "nova-compute service" | 14:19 |
| jroll | heh | 14:19 |
| *** bvanhav has joined #openstack-nova | 14:19 | |
| *** owalsh-afk has quit IRC | 14:19 | |
| * johnthetubaguy renames compute node to grape, and compute service to vine, now where does the wine come in? | 14:20 | |
| openstackgerrit | Sam Betts proposed openstack/nova: Change order of _cleanup_deploy and _unprovision in Ironic virt https://review.openstack.org/422678 | 14:21 |
| jroll | through an IV | 14:21 |
| cdent | jroll++ | 14:23 |
| artom | Intra-vineous :D | 14:26 |
| figleaf | johnthetubaguy: in that case, let's make some wine! | 14:26 |
| *** tblakes has joined #openstack-nova | 14:27 | |
| * figleaf notes that it *is* Friday, after all | 14:27 | |
| *** esberglu has joined #openstack-nova | 14:27 | |
| *** owalsh-afk has joined #openstack-nova | 14:28 | |
| openstackgerrit | Andrey Volkov proposed openstack/nova: WIP Update pci device fields from deleted object https://review.openstack.org/426243 | 14:28 |
| *** takedakn has joined #openstack-nova | 14:28 | |
| *** baoli has joined #openstack-nova | 14:30 | |
| *** karthiks has quit IRC | 14:30 | |
| *** esberglu_ has joined #openstack-nova | 14:30 | |
| *** esberglu has quit IRC | 14:33 | |
| johnthetubaguy | figleaf: best idea I heard all day | 14:36 |
| *** efoley has quit IRC | 14:37 | |
| *** Dinesh_Bhor has quit IRC | 14:37 | |
| *** lpetrut has quit IRC | 14:37 | |
| *** efoley has joined #openstack-nova | 14:38 | |
| *** efoley has quit IRC | 14:39 | |
| BlackDex | hello again. I'm having trouble with nova-consoleauth | 14:39 |
| *** tlian has joined #openstack-nova | 14:39 | |
| BlackDex | i have an ha setup, with hacluster and a vip | 14:39 |
| BlackDex | 3 systems | 14:39 |
| *** efoley has joined #openstack-nova | 14:39 | |
| BlackDex | if i use horizon to view the console i have a 1 in 3 times luck to get the console to view | 14:40 |
| *** dakmetov has quit IRC | 14:40 | |
| * figleaf has to step out for a doctor appt - back in a few. | 14:40 | |
| BlackDex | i have configured memcached, but nothing is getting into memcached, i can connect to memcached from the nova-controle nodes | 14:40 |
| *** mjura has quit IRC | 14:42 | |
| *** dansmith is now known as superdan | 14:47 | |
| *** sahid has quit IRC | 14:47 | |
| superdan | that scheduler change failed on the old side due to volumes and neutron | 14:47 |
| cdent | johnthetubaguy, dansmith : looks like the latest multinode grenade failed on timeouts this time | 14:47 |
| cdent | jinx | 14:47 |
| superdan | heh | 14:47 |
| * johnthetubaguy shakes fist | 14:47 | |
| *** vladikr has joined #openstack-nova | 14:48 | |
| johnthetubaguy | is that a real something is broken timeout, or a you hit an unlucky node timeout? | 14:48 |
| johnthetubaguy | cdent: did we get the correct min_version back now? | 14:49 |
| cdent | no logs for the new side | 14:49 |
| superdan | johnthetubaguy: we didn't get far enough to look | 14:49 |
| johnthetubaguy | I was worried you would say that | 14:50 |
| * cdent gives johnthetubaguy more wine | 14:50 | |
| johnthetubaguy | heh | 14:50 |
| mriedem | superdan: congrats https://review.openstack.org/319379 | 14:50 |
| * superdan takes a bow | 14:50 | |
| mriedem | now we just cleanup the release notes for cells v2 and maybe upgrade docs and we're good for ocata i think | 14:51 |
| jroll | what's the best way to talk to the placement service in devstack? just grab a keystone token and use curl? | 14:51 |
| mriedem | jroll: yeah, althouh mlavalle has an osc plugin somewhere | 14:51 |
| jroll | mriedem: cool, thanks | 14:52 |
| cdent | jroll: a) probably yes b) i'm not sure if mlavalle's thing is "done" c) I use gabbi yaml files, run throuh gabbi-run | 14:52 |
| johnthetubaguy | figleaf: cdent: I am struggling to understand this line: https://review.openstack.org/#/c/404472/23/nova/compute/resource_tracker.py@548 | 14:52 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876 | 14:52 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Fix tag attribute disappearing in 2.33 and 2.37 https://review.openstack.org/424759 | 14:52 |
| johnthetubaguy | figleaf: cdent: I feel I am missing something big and obvious | 14:53 |
| jroll | cdent: thanks | 14:53 |
| cdent | jroll: in case you're curious: https://gist.github.com/cdent/a9590764fbc7402d450fa36df14f35e0 | 14:53 |
| jroll | cdent: I need to poll for resources in our devstack plugin, curl seems like the right thing for now | 14:53 |
| *** sahid has joined #openstack-nova | 14:53 | |
| cdent | jroll: for that, yeah | 14:53 |
| cdent | (although, just because we're almost on the topic, gabbi can poll) | 14:54 |
| *** owalsh-afk is now known as owalsh | 14:54 | |
| cdent | johnthetubaguy: the cleanup or that the exception happens? | 14:54 |
| johnthetubaguy | cdent: I am wondering if we need cleanup in other cases, like when an ironic node is there, but there is no allocation | 14:55 |
| johnthetubaguy | cdent: maybe the API just does that for us already? | 14:55 |
| johnthetubaguy | ah, I think it does | 14:55 |
| cdent | I don't think I'm sufficiently contextualized to really follow what you're saying? | 14:56 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 14:56 |
| *** mlavalle has joined #openstack-nova | 14:56 | |
| *** prateek has joined #openstack-nova | 14:56 | |
| *** armax has joined #openstack-nova | 14:56 | |
| johnthetubaguy | cdent: sorry, I was worried about us not always deleting the old style ironic inventories, but I didn't realise the API would do that for us when we put the inventories | 14:57 |
| mriedem | superdan: cdent: jaypipes: bauzas: johnthetubaguy: melwitt: fyi i'm starting to make a list of things to do before rc1 https://etherpad.openstack.org/p/nova-ocata-rc1-todos | 14:57 |
| johnthetubaguy | its just when we have allocations we need to do better | 14:57 |
| johnthetubaguy | mriedem: perfect | 14:57 |
| mriedem | jroll: https://github.com/miguellavalle/python-placementclient | 14:57 |
| jroll | mriedem: cool, I'll wait til that's more "official" to depend on it in devstack, I think | 14:58 |
| *** khushbu_ has quit IRC | 14:59 | |
| cdent | mriedem: do we want to be concerned with the traffic breadth from the resource tracker that I wrote about in: http://lists.openstack.org/pipermail/openstack-dev/2017-January/110953.html ? There are two bug fixes in progress related to that. | 14:59 |
| mriedem | cdent: maybe? throw it into the etherpad, i haven't read that thread in detail yet | 15:01 |
| mriedem | i know we're super chatty | 15:01 |
| openstackgerrit | Jordan Pittier proposed openstack/nova: Fix unspecified bahavior on GET /servers/detail?tenant_id=X as admin https://review.openstack.org/426259 | 15:01 |
| cdent | k, will do | 15:01 |
| mriedem | the RT is chatty i mean when using the report client | 15:01 |
| mriedem | i think i pointed something out about that in the resource classes series that's still up | 15:02 |
| *** sahid has quit IRC | 15:02 | |
| *** sahid has joined #openstack-nova | 15:02 | |
| mriedem | but now i can't remember what it was, i think something about updating inventory twice when we didn't need to | 15:02 |
| *** vladikr has quit IRC | 15:02 | |
| *** tbachman has quit IRC | 15:02 | |
| cdent | mriedem: yeah | 15:03 |
| mriedem | but thanks for doing an audit, it's easy to not care about performance when you're just trying to get something functional | 15:03 |
| cdent | I think in order to maintain our self-healing we still need to report more than we want | 15:03 |
| cdent | s/want/could/ | 15:03 |
| cdent | but at the moment it is very redundant, and some if it just because of old bugs | 15:03 |
| cdent | mriedem: was worried about what would happen when this suddenly landed on a big deployment | 15:04 |
| *** tbachman has joined #openstack-nova | 15:04 | |
| superdan | cdent: so one thing to consider, while we see how this goes, | 15:04 |
| superdan | cdent: is are we likely to make more trouble by doing what we're doing than we have now.. | 15:04 |
| superdan | because currently, | 15:04 |
| superdan | we do this, but over rpc and into the same database as everything else, | 15:04 |
| superdan | with placement, it's over http, probably a little more compact, and to a potentially separate database | 15:05 |
| *** efoley_ has joined #openstack-nova | 15:05 | |
| superdan | and probably more efficient horizontally-scaled under apache | 15:05 |
| superdan | so we could assume it's at least a wash for ocata and see how it goes with some early testers | 15:05 |
| cdent | certainly the horizontal scale capabilities are more gooderer | 15:05 |
| mriedem | and we're still doing the rpc to single db thing | 15:05 |
| superdan | you didn't identify anything that is worse than today right? | 15:06 |
| mriedem | will be nice in the future when we can drop that load from rpc traffic | 15:06 |
| cdent | so yeah, we are continuing the existing, and adding more on top | 15:06 |
| superdan | yeah | 15:06 |
| superdan | yeah, I guess I forgot we're doing both, but, theoretically to different places | 15:06 |
| superdan | but yeah | 15:06 |
| cdent | the thing with object comparison is something we should probably just fix as it is clear and simple | 15:07 |
| *** takedakn has quit IRC | 15:07 | |
| *** rwmjones has joined #openstack-nova | 15:07 | |
| mriedem | if only we had a performance working group or something | 15:07 |
| cdent | I'm not sure how much impact it has on the http side, but maybe some on the rpc side | 15:07 |
| cdent | the thing with tracked_instances is just weird | 15:07 |
| superdan | I haven't looked at the two fixes | 15:07 |
| cdent | as the way it works now is that it doesn't at all do what it says it is going to do | 15:07 |
| superdan | I'll try to do that and make a judgment | 15:07 |
| cdent | thanks, that would be awesome, just doing that analysis felt like I had a significant leap forward on understanding the resource tracker, but then right at the end I felt like I fell way backwards :( | 15:08 |
| cdent | in the "why does this even work" sense | 15:08 |
| superdan | well, I'm not a RT expert by any means | 15:08 |
| superdan | probably jay would be better, but.. | 15:08 |
| *** efoley has quit IRC | 15:09 | |
| johnthetubaguy | figleaf: cdent: this ironic resource provider change, I could be missing something again, but I feel like we are missing something around updating new instance allocations? well possibly anyways. | 15:09 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: Fix backwards compatibility for InstanceNUMACell https://review.openstack.org/396184 | 15:09 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: Combine obj_make_compatible tests https://review.openstack.org/426261 | 15:09 |
| cdent | johnthetubaguy: figleaf has gone to the docs, I think, and knows more about that than me, but I'll listen... | 15:10 |
| mriedem | johnthetubaguy: replied in https://review.openstack.org/#/c/404472/ - | 15:10 |
| mriedem | johnthetubaguy: none of that fancy new code is hit unless node.resource_class is set | 15:10 |
| mriedem | which it's not today in our CI | 15:10 |
| mriedem | which kind of blows | 15:10 |
| mriedem | if you look at the logs for the ironic job on that patch it's still setting ram/cpu/disk | 15:10 |
| mriedem | would be nice if we could have a one-off DNM patch to set node.resource_class and test that stuff | 15:11 |
| mriedem | jroll: ^ | 15:11 |
| mriedem | johnthetubaguy: and i think the ironic-specific stuff is mostly contained in the RT | 15:11 |
| mriedem | which is why in earlier patches i had asked that long-term we subclass the ResourceTracker to do the ironic-specific bits | 15:11 |
| *** yamahata has joined #openstack-nova | 15:12 | |
| mriedem | because i don't like the "if ironic" spaghetti code | 15:12 |
| mriedem | i'll review the latest on that patch when i get back in an hour or so | 15:12 |
| *** nkrinner is now known as nkrinner_afk | 15:12 | |
| cdent | I may be misremembering, but I thought part of the rt+resosurce class stack was to change the resource tracker so that there is only one of them (for everything it tracks), not one for each node it cares about? | 15:13 |
| cdent | that might be somewhere else... | 15:14 |
| *** mnestratov|2 has joined #openstack-nova | 15:14 | |
| melwitt | cdent: fwiw the tracked_instances clear every update_available_resource period is known and intentional. it surprised me too when I first realized it. I don't recall why it's that way, I guess to prevent stale things from hanging around potentially | 15:14 |
| *** jamesdenton has joined #openstack-nova | 15:14 | |
| cdent | melwitt: I took it out and no tests failed... | 15:14 |
| *** mnestratov has quit IRC | 15:15 | |
| cdent | which made me sad and confused | 15:15 |
| mriedem | cdent: that's the same | 15:15 |
| mriedem | but not what i'm talking about | 15:15 |
| mriedem | bbiab | 15:15 |
| *** mtanino has joined #openstack-nova | 15:16 | |
| melwitt | yeah, I don't think tests would fail from it. I have mentioned it to jaypipes in the past and he knew about it | 15:16 |
| *** raunak has quit IRC | 15:16 | |
| cdent | melwitt: hmm. well, thanks for providing some background. I wonder if it is worth revisiting since now it is spinning off redudant http requests? | 15:17 |
| melwitt | bauzas: do you know anything about why tracked_instances is cleared each update_available_resource period? | 15:17 |
| *** vladikr has joined #openstack-nova | 15:18 | |
| *** vsaienko has quit IRC | 15:18 | |
| *** prateek has quit IRC | 15:19 | |
| *** gouthamr has quit IRC | 15:19 | |
| johnthetubaguy | cdent: so the bit I am stuck with, is where we make the new style ironic resource_class allocations | 15:19 |
| melwitt | cdent: I think in general it's worth considering since it seemed excessive even in its original pre placement api form. just don't know what's involved in doing that or what gotchas there are | 15:20 |
| *** stvnoyes has quit IRC | 15:20 | |
| *** claudiub|2 has joined #openstack-nova | 15:20 | |
| *** stvnoyes has joined #openstack-nova | 15:21 | |
| jroll | mriedem: I'm working on devstack support right now, do you want a throwaway hack in the meantime? | 15:21 |
| superdan | melwitt: cdent it's just to try to always stab the latest info into the database in case it's stale, right? | 15:21 |
| melwitt | that's what I'm guessing | 15:22 |
| cdent | if the node is authoritative for itself, and it keeps a cache of what it successfully wrote last time, how does staleness happen? | 15:22 |
| cdent | (where last time could be 60s or 60 days ago) | 15:23 |
| *** claudiub|3 has joined #openstack-nova | 15:23 | |
| *** hieulq_ has joined #openstack-nova | 15:23 | |
| cdent | (mostly thinking about inventory here, not allocations) | 15:23 |
| cdent | johnthetubaguy: i'm rereading the code, to bring it back into my brain | 15:23 |
| *** claudiub has quit IRC | 15:23 | |
| *** bhagyashris has quit IRC | 15:24 | |
| *** jaosorior has joined #openstack-nova | 15:25 | |
| *** claudiub|2 has quit IRC | 15:25 | |
| cfriesen | cdent: refreshing tracked_instances would allow it to recover from errors where it missed an event that would cause the instance to be added/removed from the list. But it should be possible to handle that in a different way. | 15:26 |
| johnthetubaguy | cdent: you probably want to ignore any changes to the "service" field in the compute_node as well, as service will keep getting updated every DB service update interval, unsure if thats related (re https://review.openstack.org/#/c/424305) | 15:26 |
| johnthetubaguy | cdent: ignore that | 15:27 |
| melwitt | cdent: looking back at the code to refresh my memory ... I think without the periodic clear, the only thing that removes instances from tracked_instances is the vm_state DELETED or SHELVED_OFFLOADED and if for some reason (errors) an instance doesn't get its state saved that way, it would never be removed | 15:29 |
| cdent | cfriesen: is missing events of that nature a thing happens a) often, b) sometimes, c) rarely? | 15:29 |
| *** eglynn has joined #openstack-nova | 15:29 | |
| *** sneti_ has joined #openstack-nova | 15:30 | |
| *** sneti_ has quit IRC | 15:32 | |
| cfriesen | cdent: I would expect rarely | 15:32 |
| cfriesen | melwitt: abort_instance_claim() also removes it, by calling _update_usage_from_instance(is_removed=True) | 15:33 |
| melwitt | yeah, sorry. I forgot to mention that too | 15:34 |
| cfriesen | melwitt: cdent: but I think melwitt is correct that there would be cases we'd need to add explicit removal | 15:34 |
| *** zz_dimtruck is now known as dimtruck | 15:34 | |
| cdent | at least over the long run, making things more explicit would probably be better | 15:34 |
| cfriesen | cdent: agreed, and it would make the resource tracking react faster to freeing up of resources. | 15:35 |
| *** rwmjones has quit IRC | 15:36 | |
| *** satyar has joined #openstack-nova | 15:37 | |
| johnthetubaguy | cdent: at somepoint doesn't nova-conductor create the instance resource allocations, and retries if the generation id of the resource provider doesn't match, (as generation id increments on a related allocation, or something like that)? So the resource tracker is more just checking for rogue instances with missing allocations? | 15:40 |
| cfriesen | so hardware.numa_usage_from_instances() expects a list, but we only ever call it with a single instance. Do we plan on ever using it with a list or should we maybe clean it up to expect a single instance? | 15:41 |
| *** gouthamr has joined #openstack-nova | 15:41 | |
| *** rwmjones has joined #openstack-nova | 15:41 | |
| cdent | johnthetubaguy: when you say "allocations" do you mean allocations in the placement api, or something more general? because if the former, the conductor doesn't talk to the placement api at all. | 15:42 |
| *** mnestratov|2 has quit IRC | 15:43 | |
| johnthetubaguy | cdent: I mean placement API, and when I say conductor, that happens in nova-scheduler today | 15:43 |
| *** Guest21909 has quit IRC | 15:43 | |
| cdent | johnthetubaguy: as far as I know, right now, the only thing that writes allocations is the resource tracker | 15:45 |
| cdent | and the only thing that writes inventories is the resource tracker | 15:45 |
| cdent | when nova-scheduler talks to placement API it is read only | 15:45 |
| *** marst has joined #openstack-nova | 15:47 | |
| johnthetubaguy | cdent: I am thinking about the (not too distant) future, as the sooner we add the allocations, the sooner other queries "notice" them | 15:47 |
| johnthetubaguy | but thats probably just a distraction right now | 15:48 |
| cdent | oh you mean "at some point won't it be the cast that...". | 15:48 |
| cdent | Yes, at some point the scheduler will request and claim resources (thus make allocations) in one go | 15:48 |
| cdent | probably with a POST to /allocations of something resembling a request spec | 15:48 |
| cdent | instead of the curerrent GET /resoruce_provdiers (which then passing into existing filter scheduler) | 15:49 |
| cdent | when that time comes, the resource tracker will change quite a bit, presumably | 15:49 |
| *** hieulq_ has quit IRC | 15:49 | |
| cdent | but I suspect that that time is probably further off than we think | 15:49 |
| cdent | s/think/hope/ | 15:49 |
| *** amotoki has joined #openstack-nova | 15:51 | |
| *** mnestratov|2 has joined #openstack-nova | 15:56 | |
| *** amotoki has quit IRC | 15:56 | |
| *** Guest21909 has joined #openstack-nova | 15:58 | |
| *** mvk has quit IRC | 15:58 | |
| *** dave-johnston has joined #openstack-nova | 15:59 | |
| *** ijw has joined #openstack-nova | 16:00 | |
| johnthetubaguy | cdent: fair point, just wondering if we can make baby steps in that direction, while we still have the middle ground | 16:03 |
| *** efoley__ has joined #openstack-nova | 16:06 | |
| cdent | johnthetubaguy: I'm not super familiar with the existing claim handling, so I'm too clear on what the options are for steps | 16:06 |
| *** lpetrut has joined #openstack-nova | 16:07 | |
| openstackgerrit | Chris Dent proposed openstack/nova: Add more debug logging on RP inventory update failures https://review.openstack.org/414230 | 16:08 |
| openstackgerrit | Chris Dent proposed openstack/nova: Add more debug logging on RP inventory delete failures https://review.openstack.org/426290 | 16:09 |
| *** efoley_ has quit IRC | 16:09 | |
| *** efoley__ has quit IRC | 16:12 | |
| *** Oku_OS is now known as Oku_OS-away | 16:12 | |
| *** liangy has joined #openstack-nova | 16:16 | |
| superdan | sdague: are you back today? | 16:16 |
| sdague | superdan: yeh | 16:16 |
| superdan | sdague: you wanna look at the mess I've made here? https://review.openstack.org/#/c/424730/ | 16:17 |
| superdan | I think the wait for services is probably a good thing we want to avoid a race with the subnode checking back in, | 16:17 |
| superdan | although I don't think it's related to the problem we were trying to solve | 16:18 |
| superdan | and I figure that should be pulled out | 16:18 |
| superdan | to a separate patch | 16:18 |
| superdan | but, the configuring/starting of the placement stuff would be good to validate if you could | 16:18 |
| *** eharney has quit IRC | 16:20 | |
| sdague | yeh, it seems like that should be separate, and I do agree, based on how long all these phases take, I seriously doubt we'll have a practical issue there | 16:21 |
| cdent | anyone: probably too late but: CORS for placement had a +2 on it a while back and there was some support for getting it in as part of ocata (probably not a huge deal if it doesn't happen): https://review.openstack.org/#/c/392891/ | 16:21 |
| sdague | superdan: I saw this patch briefly the other day, how about we bite the bullet now and make a 55_placement top level construct and do the placement service enablement and start there | 16:21 |
| superdan | sdague: it would specifically catch the situation where we introduce a thing that broke the subnode entirely and call it out, instead of just trying to decipher it from the failed tempest run, at least | 16:21 |
| *** burgerk has joined #openstack-nova | 16:21 | |
| sdague | superdan: sure, but has that shown up yet? | 16:22 |
| *** tbachman has quit IRC | 16:22 | |
| superdan | sdague: meaning have we ever broken the subnode? | 16:22 |
| sdague | superdan: in this flow right now | 16:23 |
| superdan | sdague: at one point the service version code had a bug where the subnode couldn't check back in because its service record had timed out and it was told it couldn't check back in and regress the minimum version, yeah | 16:23 |
| superdan | sdague: and it took a while to realize that the subnode was running, but never being sent instances | 16:23 |
| sdague | I get the idea that it can happen, but lets not mix up problems being solved here | 16:24 |
| superdan | sdague: unrelated to this flow, although this flow can definitely race with the subnode checking back in after the upgrade, I just haven't seen it happen yet | 16:24 |
| sdague | we can always put a check like this back in later if it's happening at any frequency, and even backport it | 16:24 |
| superdan | cdent: mriedem: scheduling patch just passed multinode grenade | 16:24 |
| superdan | johnthetubaguy: ^ | 16:25 |
| cdent | amazeballs | 16:25 |
| *** tbachman has joined #openstack-nova | 16:25 | |
| superdan | sdague: it just doesn't seem responsible to not check for and catch a thing we know can happen when it's trivial to do, but.. the fight is all gone from me this week | 16:25 |
| sdague | superdan: I just want to fix one thing at a time here and get people unblocked. We can fix these other things later. | 16:26 |
| superdan | sdague: I'm not saying that has to be in front of this necessarily | 16:26 |
| mriedem | superdan: how is that possible | 16:26 |
| superdan | mriedem: cosmic rays | 16:26 |
| sdague | because it's not like you'll get the wrong test results if it happens, it's just that figuring out what went wrong takes a little longer | 16:27 |
| superdan | mriedem: cdent johnthetubaguy I will get a bite to eat and then do the grenade refactor sdague is asking for | 16:27 |
| superdan | there's also pep8 and unit fails in the nova side patch | 16:27 |
| johnthetubaguy | superdan: cool, I can roll back the patch so the hacks are removed, and the tests are passing? | 16:27 |
| superdan | sdague: it does mean we run another 50m of tests that we know will fail instead of just bailing early, but yeah | 16:27 |
| mriedem | jroll: throwaway hack yes please | 16:27 |
| superdan | johnthetubaguy: remove the logging and such, but keep the *correct* min ver check yeah :) | 16:28 |
| johnthetubaguy | cdent: sorry, I meant to ask you, did you have any more thoughts on this one: https://review.openstack.org/#/c/404472 | 16:28 |
| johnthetubaguy | superdan: thats my *aim* anyways | 16:28 |
| sdague | superdan: sure, but that makes it an optimization we can fix later, even deep in the freeze, once we see how often it happens | 16:28 |
| superdan | johnthetubaguy: I wish you luck sir | 16:28 |
| cdent | superdan, johnthetubaguy is the issue with making sure the service cache gets updated after each compute node update suitably okay? are we assuming the scheduler will be restarted every time some uprades a compute node? | 16:29 |
| *** david-lyle has joined #openstack-nova | 16:29 | |
| cdent | johnthetubaguy: sorry, I started looking and got distracted by a merge conflict on some logging fixes and lost my place. Is there a summary of what we're trying to resolve? | 16:30 |
| johnthetubaguy | cdent: I was thinking you needed to SIG_HUP for the RPC API change anyways, but it might be the first time | 16:30 |
| johnthetubaguy | cdent: the -1 comment on the patch should do it | 16:30 |
| superdan | cdent: that's the assumption that my grenade service check ensures is safe, and which sdague wants to wait on since we haven't seen it yet, even though it's 100% possible | 16:30 |
| cdent | sig hup won't do it, as there's no reset handler? | 16:30 |
| superdan | cdent: so I'll pull that out and do it separately and if we start to break the world after this merges, we'll have it ready | 16:30 |
| johnthetubaguy | cdent: basically, seems like the instance allocations will fail for ironic nodes that have resource_classes defined | 16:30 |
| superdan | cdent: correct, there's no reset handler, which I did ask for, but we can do after as well, and won't help the gate anyway | 16:31 |
| johnthetubaguy | cdent: ah, I missed that | 16:31 |
| johnthetubaguy | superdan: +1, but lets keep that separate | 16:31 |
| cdent | superdan: k, I wasn't think in terms of the gate, but you know, like real people ;) | 16:31 |
| *** ralonsoh has quit IRC | 16:31 | |
| superdan | johnthetubaguy: yeah, I was going to pull it out regardless, but I was going to keep it in the path of this set, but I won't now | 16:32 |
| superdan | cdent: yes, which is why I asked for the reset thing initially | 16:33 |
| cdent | johnthetubaguy: tacking back to that allocations thing now | 16:34 |
| *** mfeoktistov|2 has quit IRC | 16:34 | |
| mriedem | is someone fixing up the nova side scheduler change? because if they are, can they address the exception wording comment in https://review.openstack.org/#/c/425806/ ? | 16:36 |
| johnthetubaguy | cdent: ack | 16:37 |
| mriedem | snikitin: are you saying the diagnostics patches were all approved yesterday before nit fixing? | 16:38 |
| mriedem | or just had +2s on them? | 16:38 |
| *** bvanhav has quit IRC | 16:38 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova: DNM: hack ironic with resource providers https://review.openstack.org/426296 | 16:38 |
| jroll | mriedem: enjoy | 16:38 |
| * figleaf is back, but in need of coffee | 16:39 | |
| *** annegentle has joined #openstack-nova | 16:39 | |
| mriedem | jroll: cool hopefully i can load that sometime today | 16:41 |
| jroll | heh | 16:41 |
| *** chyka has joined #openstack-nova | 16:41 | |
| *** mdrabe has quit IRC | 16:41 | |
| *** tbachman has quit IRC | 16:43 | |
| sdague | cdent: there are many ways that things can go wrong. My biggest concern is that we don't have a failed software stack that provides a passing result. As long as we don't have that, our cost is debug time and time until test completion. | 16:48 |
| sdague | because we have a specific test that attempts to schedule to all nodes, and a min node count, we have one test that points in the right direction already, so debug time isn't hugely negatively impacted | 16:49 |
| sdague | and test run time until failure is basically +15m here | 16:49 |
| cdent | sdague: I'm not sure which of several context to put that response in? | 16:49 |
| *** mdrabe has joined #openstack-nova | 16:49 | |
| figleaf | johnthetubaguy: can you summarize the sticking points in https://review.openstack.org/#/c/404472 for your -1? | 16:50 |
| cdent | johnthetubaguy: are you concerned about the situation of old style ironic allocations being deleted but then nothing allocation the new style? I'm still confused (apparently like figleaf too) | 16:50 |
| sdague | cdent: about why I said don't both with this - https://review.openstack.org/#/c/424730/12/projects/60_nova/resources.sh@136 at the moment | 16:50 |
| cdent | sdague: I hadn't made any comments with respect to that? | 16:51 |
| sdague | cdent: ok, well a bunch of the superdan cross comments were about that | 16:51 |
| johnthetubaguy | cdent: figleaf: the bit I don't understand, is who make allocations for ironic instances, making use of the new resource class | 16:51 |
| sdague | anyway, lunch, biab | 16:51 |
| cdent | johnthetubaguy: nobody | 16:51 |
| cdent | which is why the old allocations will never be cleaned up | 16:52 |
| cdent | because InventoryInUse will never get raised | 16:52 |
| mriedem | jroll: CUSTOM_LOL eh | 16:52 |
| johnthetubaguy | cdent: figleaf: it looks like we still trying to make the old style allocations in the resource tracker | 16:52 |
| johnthetubaguy | cdent: figleaf: which should fail I assume, as that resource class isn't present for that resource provider any more | 16:53 |
| superdan | sdague: if you mean cross in the "unhappy" sense, you're misinterpreting my exhaustion | 16:53 |
| cdent | I think it was "cross" as in "interleaved" | 16:53 |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 16:53 |
| superdan | sdague: I think that check should be in there given the requirement we're adding that it be right at scheduler start, but I'm okay to wait until later if you think it's best | 16:53 |
| superdan | cdent: okay, cool | 16:53 |
| cdent | johnthetubaguy: why are they not there anymore? (not asserting they are, just tracing your thought process) | 16:54 |
| openstackgerrit | Stephen Finucane proposed openstack/nova: console: Move proxies to 'console/proxy' https://review.openstack.org/408192 | 16:55 |
| johnthetubaguy | cdent: because the code now only creates inventory entires for the new custom resource class | 16:55 |
| *** unicell has joined #openstack-nova | 16:55 | |
| *** unicell has quit IRC | 16:55 | |
| *** unicell has joined #openstack-nova | 16:55 | |
| figleaf | johnthetubaguy: existing ironic nodes won't have the custom resource class set | 16:56 |
| cdent | johnthetubaguy: so if a new ironic thing comes into existence, with a defined, resource class, that's when we run into trouble? | 16:56 |
| figleaf | johnthetubaguy: so they'll get handled like they were vms | 16:56 |
| *** bvanhav has joined #openstack-nova | 16:56 | |
| johnthetubaguy | cdent: yes | 16:56 |
| figleaf | johnthetubaguy: ah | 16:56 |
| figleaf | johnthetubaguy: so your concern is that we don't have the ironic-aware bits to select a resource provider? | 16:56 |
| mriedem | andreykurilin: your upper-constraints dreams have come true https://review.openstack.org/#/c/426116/5/upper-constraints.txt@332 | 16:57 |
| andreykurilin | mriedem: \o/ | 16:57 |
| openstackgerrit | Sarafraj Singh proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 16:57 |
| johnthetubaguy | figleaf: ish, its basically that this method will fail (looking for link) | 16:57 |
| johnthetubaguy | figleaf: ah, so this will fail, when there is a new style ironic resource provider being used by a new instance that is created: https://github.com/openstack/nova/blob/master/nova/scheduler/client/report.py#L664 | 16:58 |
| johnthetubaguy | figleaf: at least I think it will | 16:58 |
| openstackgerrit | Sarafraj Singh proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 16:58 |
| sfinucan | Interesting problem here - I've backported a bugfix that's exposed a latent Python 3 issue in the code https://review.openstack.org/#/c/425087 | 16:58 |
| johnthetubaguy | figleaf: cdent: maybe a quick google hangout would make this easier to talk through? | 16:59 |
| sfinucan | Wonder if I should fix the Python 3 issue or drop the test? | 16:59 |
| *** markus_z has quit IRC | 17:00 | |
| figleaf | johnthetubaguy: maybe, but I'm switching context to follow the code through the different parts | 17:01 |
| figleaf | johnthetubaguy: I can't keep it all straight while working on something else :) | 17:02 |
| *** sahid has quit IRC | 17:02 | |
| johnthetubaguy | figleaf: no worries, it took me a good while to trace all that back too | 17:02 |
| *** slaweq has joined #openstack-nova | 17:02 | |
| *** raunak has joined #openstack-nova | 17:03 | |
| superdan | sdague: I have to merge something in devstack to make grenade consider a new projects/XX_thing plugin? | 17:05 |
| cdent | johnthetubaguy: will objects.InstanceList.get_by_host_and_node return ironic things? | 17:06 |
| johnthetubaguy | cdent: I believe it does | 17:06 |
| superdan | cdent: it'll return ironic instances yeah | 17:07 |
| cdent | k, thanks johnthetubaguy and superdan, trying to properly trace the code (as figleaf said, is meandering...) | 17:07 |
| superdan | sdague: oh, nm I think i see | 17:09 |
| *** dtp has joined #openstack-nova | 17:09 | |
| *** yamahata has quit IRC | 17:11 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 17:12 |
| openstackgerrit | John Garbutt proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | 17:12 |
| *** nic has joined #openstack-nova | 17:12 | |
| *** slaweq has quit IRC | 17:12 | |
| mriedem | johnthetubaguy: ah i see you updated it :) | 17:13 |
| mriedem | was just leaving comments, will finish that up on the older PS | 17:13 |
| johnthetubaguy | mriedem: hopefully, that was a rebase, but I think it was superdan 's change that failed to merge, but never mind | 17:13 |
| mriedem | johnthetubaguy: just posted comments in PS36 | 17:15 |
| jroll | mriedem: sure why not :P | 17:16 |
| johnthetubaguy | mriedem: I thought I had removed all that :S | 17:16 |
| mriedem | johnthetubaguy: want to update the wording on https://review.openstack.org/#/c/425806/4/nova/exception.py ? | 17:17 |
| mriedem | while you're at it | 17:17 |
| johnthetubaguy | mriedem: aye | 17:18 |
| figleaf | johnthetubaguy: cdent: so update_instance_allocation() should only be called for "old-style" allocations. create_provider_allocations() should be called for "new-style" (i.e., ironic with custom RC) | 17:18 |
| superdan | mriedem: johnthetubaguy I'm going to avoid rechecking the nova scheduler patch until I have reasonable faith in the refactor to move placement into its own top-level | 17:19 |
| superdan | just FYI | 17:19 |
| jroll | More than one endpoint exists with the name 'placement'. <- not cool, devstack, not cool | 17:19 |
| figleaf | johnthetubaguy: cdent: what it looks like is that we need to make sure that the update_instance_allocation() calls are all checked to make sure that they aren't called for new ironic nodes | 17:19 |
| mriedem | johnthetubaguy: i'll go through PS38 too | 17:20 |
| johnthetubaguy | figleaf: but there are no new calls right now, so surely we need someone to claim the new resource class? | 17:20 |
| figleaf | johnthetubaguy: I only see two places where update_instance_allocations() is called in the RT. | 17:21 |
| johnthetubaguy | figleaf: I would rather update_instance_allocation() was able to know from the instance what resources it should be claiming, and it just claims the correct ones, for now thats a bit hard coded I guess, but eventually its the same logic the scheduler has to do when requesting the correct resource class | 17:21 |
| figleaf | johnthetubaguy: yeah, this is all transitional code | 17:22 |
| figleaf | johnthetubaguy: we wanted to get ironic handled before we are able to handle all types of RPs | 17:22 |
| figleaf | johnthetubaguy: can you think of a test that would fail now with ironic? | 17:23 |
| johnthetubaguy | figleaf: add resource class on ironic node, boot instance that will land there, resource claim fails | 17:23 |
| johnthetubaguy | figleaf: at least, thats the way it looks to me | 17:23 |
| cdent | figleaf, johnthetubaguy my read of the code in the rt ironic inventory patch is that johnthetubaguy is right: it is possible to reache update_instance_allocation with a compute node and an instance and depending on what "compute_node" is, the allocations will either fail, or apply to the nova-compute node, incorrectly | 17:23 |
| cdent | because the data at that point is about resource usages (vcpu etc) | 17:24 |
| *** Jeffrey4l_ has quit IRC | 17:24 | |
| *** unicell has quit IRC | 17:24 | |
| johnthetubaguy | yeah, instead of allocating for vcpu, it should be for the custom resource class, I think | 17:24 |
| *** Jeffrey4l_ has joined #openstack-nova | 17:24 | |
| johnthetubaguy | oh balls... | 17:25 |
| cdent | which should mean that the ironic "instance" is not include in the loop that is calling update_instance_allocation | 17:25 |
| cdent | because it isn't really an instance | 17:25 |
| cdent | (in that context) | 17:25 |
| johnthetubaguy | if we merge the code to query placement, we will never land on an iroinc node in the first place, unless it is reporting VCPU resources | 17:25 |
| figleaf | johnthetubaguy: that's a known issue | 17:26 |
| cdent | "oh balls" means we've passed the "bugger" state of affairs | 17:26 |
| johnthetubaguy | cdent: heh, ack | 17:26 |
| figleaf | johnthetubaguy: which is why ironic has to keep reporting vm-type resources until Pike | 17:26 |
| johnthetubaguy | figleaf: but this code doesn't do that, I thought | 17:26 |
| cdent | brb | 17:27 |
| figleaf | johnthetubaguy: digging into the claim stuff - haven't looked at that in a while | 17:27 |
| johnthetubaguy | figleaf: actually, I am changing my mind on the fix here, I think its probably just that we should still be reporting VCPUs, we can't drop those till we schedule based on the new resources | 17:28 |
| *** annegentle has quit IRC | 17:29 | |
| figleaf | johnthetubaguy: yes, that's been the plan | 17:29 |
| *** rmart04 has quit IRC | 17:29 | |
| johnthetubaguy | figleaf: but the code only reports the custom resource classes, or did I read that wrongly? | 17:30 |
| *** annegentle has joined #openstack-nova | 17:30 | |
| cdent | johnthetubaguy, figleaf: can we restate the problem again (I know we keep doing that, but I think it is useful): If we released this code today without any change it would be wrong because: | 17:30 |
| figleaf | cdent: ...once an ironic node reports resources with a custom resource class, all the VCPU etc. stuff gets removed | 17:31 |
| figleaf | cdent: ...and we have no way to then select that ironic node | 17:31 |
| figleaf | johnthetubaguy: cdent: Does that sound right? | 17:32 |
| openstackgerrit | John Garbutt proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | 17:32 |
| cdent | "reports resources" == writes inventory? | 17:32 |
| figleaf | yeah | 17:32 |
| *** sree has joined #openstack-nova | 17:33 | |
| *** slaweq has joined #openstack-nova | 17:33 | |
| *** sree has joined #openstack-nova | 17:33 | |
| figleaf | cdent: the _cleanup_ironic_legacy_allocations() method removes all the old-style inventory | 17:34 |
| cdent | (if there was any) | 17:34 |
| mriedem | the funny thing is, ironic + placement in newton doesn't work anyway | 17:35 |
| figleaf | cdent: well, unless the node is newly-added, there will be | 17:35 |
| mriedem | i.e. https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:stable/newton+topic:bug/1647316 | 17:35 |
| cdent | figleaf: right, but when a node is newly added the same problem exists: there's no way to allocate or claim inventory? | 17:35 |
| figleaf | mriedem: yeah - "funny" | 17:35 |
| *** tbachman has joined #openstack-nova | 17:35 | |
| johnthetubaguy | mriedem: its almost like we need some ironic testing | 17:36 |
| mriedem | johnthetubaguy: we have it in ocata | 17:36 |
| mriedem | just not in newton where placement was optional | 17:36 |
| johnthetubaguy | mriedem: ah | 17:36 |
| mriedem | ironic + placement ci was broken for a few days in master | 17:36 |
| figleaf | cdent: if a node is added and has a custom RC, its inventory will be added new-style | 17:36 |
| mriedem | which is why i'm backporting that series of fixes | 17:36 |
| cdent | figleaf: yes, but you can't allocate against it | 17:37 |
| jroll | bah, no '-' in custom resource classes? :| | 17:37 |
| figleaf | cdent: and we have no way to select that node now | 17:37 |
| cdent | and that too | 17:37 |
| cdent | feh | 17:37 |
| mriedem | jroll: _ is far superior | 17:37 |
| cdent | jroll: just upperletters and numbers | 17:37 |
| jroll | and _ | 17:37 |
| *** sree has quit IRC | 17:38 | |
| jroll | kinda lame IMO, but alas | 17:38 |
| cdent | jroll: yeah, but that's just to keep you conused so best to just leave it out ... | 17:38 |
| * jroll continues trying to make this work in devstack | 17:38 | |
| johnthetubaguy | figleaf: cdent: so the summary, either we have VCPUs, claim them, and schedule on them, OR we have to schedule on custom_resource classes and claim them, current state: schedule VCPU, report custom, claim VCPU (except for existing instances that we tidy up to claim the new resource class) | 17:38 |
| cdent | I _think_ the constraint came into play when we didn't even have custom resource classes, not sure. | 17:38 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | 17:39 |
| *** Apoorva has joined #openstack-nova | 17:40 | |
| *** liangy has quit IRC | 17:41 | |
| jroll | well, unless I did something awful wrong, the ironic code gives me 2017-01-27 17:39:56.654 TRACE nova.compute.manager AttributeError: 'SchedulerClient' object has no attribute 'set_provider_inventory' | 17:41 |
| *** Apoorva has quit IRC | 17:41 | |
| *** Apoorva has joined #openstack-nova | 17:41 | |
| jroll | nope, I'm at b6339a876a440831903445bd4b10afb732e91335 which is latest here | 17:42 |
| johnthetubaguy | hmm, interesting | 17:42 |
| *** mriedem is now known as mriedem_beach | 17:42 | |
| cdent | this sounds like a job for functional tests | 17:43 |
| figleaf | jroll: there are some places I found where SchedulerClient and SchedulerClient.reportclient are treated as the same | 17:43 |
| figleaf | It's in reportclient | 17:43 |
| *** ociuhandu has quit IRC | 17:43 | |
| jroll | figleaf: just reporting this code is broken :) | 17:44 |
| jroll | cdent: I'm working on integration stuff atm | 17:44 |
| figleaf | jroll: heh, can relate to that | 17:45 |
| *** vladikr has quit IRC | 17:45 | |
| cdent | jroll: which is great, but there's something in the middle: more functional testing while we are developing the features (just setting a bit for future efforts, not moaning) | 17:46 |
| *** jpena is now known as jpena|off | 17:46 | |
| jroll | cdent: sure, don't disagree | 17:46 |
| * jroll way more familiar with an actual deployment than nova's functests | 17:46 | |
| cdent | jroll: shit bro, it's not you who I think should have done functional tests on this stuff. It's me and figleaf and jaypipes | 17:47 |
| jroll | cdent: heh. it's all of us. I just want to help get this working | 17:48 |
| cdent | figleaf: are you working on dealing with the traceback jroll's reported, or should I do that? | 17:49 |
| jroll | figleaf: okay, that worked, now I should see an inventory record that is mapped to that custom resource class, yeah? | 17:49 |
| figleaf | jroll: sorry, I'm looking at 6 different bits of code, trying to keep them straight :( | 17:49 |
| figleaf | jroll: what was it that you were doing when you hit the bug | 17:50 |
| figleaf | ? | 17:50 |
| cdent | traceback on the review | 17:50 |
| jroll | figleaf: so I've got devstack. ironic stuff has resource classes. using this code. | 17:50 |
| jroll | fixed traceback as mentioned (leaving a comment on that now) | 17:50 |
| jroll | looks like things go through but I don't see anything in inventory table that references the custom RP | 17:51 |
| jroll | er, s/RP/RC/ | 17:51 |
| *** Swami has joined #openstack-nova | 17:51 | |
| * jroll debugging now | 17:53 | |
| *** ijw has quit IRC | 17:54 | |
| *** unicell has joined #openstack-nova | 17:54 | |
| *** ijw has joined #openstack-nova | 17:54 | |
| *** lucasagomes is now known as lucas-afk | 17:55 | |
| *** derekh has quit IRC | 17:55 | |
| *** yamahata has joined #openstack-nova | 17:56 | |
| *** ijw_ has joined #openstack-nova | 17:56 | |
| *** ijw has quit IRC | 17:56 | |
| figleaf | jroll: ugh - looks like there are more of those all over the place | 17:56 |
| *** owalsh is now known as owalsh-afk | 18:00 | |
| cdent | figleaf, jroll, johnthetubaguy: sorry guys I gotta go. I haven't eaten enough and starting to fade badly. If there's something I can do on sunday or in the UK morning monday, let me know, either by email to os-dev or direct to me. | 18:01 |
| *** eharney has joined #openstack-nova | 18:01 | |
| jroll | cdent: no worries man, I'll just leave anything I find on gerrit | 18:02 |
| *** harlowja has joined #openstack-nova | 18:02 | |
| jroll | looks like it submits inventory correctly | 18:02 |
| johnthetubaguy | cdent: figleaf: I am going to run soon too, similar comments about monday morning | 18:02 |
| *** hfu has quit IRC | 18:02 | |
| cdent | best of luck | 18:02 |
| *** cdent has quit IRC | 18:02 | |
| johnthetubaguy | jroll: well, not sure, once we get bauzas's change in, you will no longer be able to pick iroinc nodes if the current proposal merges | 18:02 |
| johnthetubaguy | figleaf: I attempted to add a summary comment about what I think we should do | 18:02 |
| jroll | johnthetubaguy: ... O_o | 18:03 |
| figleaf | johnthetubaguy: cdent: ok, thanks for opening up this can of worms! (Seriously! It would have really sucked to let this out like this) | 18:03 |
| * jroll will read that | 18:03 | |
| johnthetubaguy | there may be 40% nonsense in my current suggestion, but lets see if that makes things clearer or not | 18:03 |
| *** owalsh-afk has quit IRC | 18:04 | |
| *** vladikr has joined #openstack-nova | 18:05 | |
| jroll | johnthetubaguy: lgtm, I think | 18:05 |
| johnthetubaguy | figleaf: are you on board with the plan I sketched out in https://review.openstack.org/#/c/404472? | 18:06 |
| *** owalsh-afk has joined #openstack-nova | 18:06 | |
| *** sambetts is now known as sambetts|afk | 18:07 | |
| *** owalsh-afk is now known as owalsh | 18:07 | |
| *** jose-phillips has joined #openstack-nova | 18:08 | |
| *** vsaienko has joined #openstack-nova | 18:10 | |
| *** tbachman has quit IRC | 18:10 | |
| *** tbachman has joined #openstack-nova | 18:12 | |
| *** weshay is now known as weshay_brb | 18:12 | |
| *** ijw_ has quit IRC | 18:12 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 18:13 |
| figleaf | johnthetubaguy: sorry, working on a different fix atm. Looking now... | 18:13 |
| johnthetubaguy | figleaf: so as I understood it, this is one of the two remaining FFEs, hence my focus on it | 18:14 |
| jroll | fwiw I'm just getting lost in this code at this point | 18:14 |
| * jroll bows out and takes a break for a bit | 18:15 | |
| figleaf | johnthetubaguy: I was thinking similarly - getting rid of the cleanup of old-style inventory | 18:15 |
| figleaf | jroll: welcome to the club!! | 18:15 |
| figleaf | johnthetubaguy: and trying to ensure that both old- and new-style can coexist | 18:16 |
| figleaf | johnthetubaguy: and then add back the cleanup in Pike, remove it in Queens | 18:16 |
| johnthetubaguy | figleaf: cool, yeah, I think we just report the new resources, for the first step | 18:16 |
| *** mvk has joined #openstack-nova | 18:16 | |
| johnthetubaguy | figleaf: yeah, the clean up can only be added after we start scheduling and claiming using the new resource class, I think, we can probably drop reporting the old ones at that point, but not 100% sure | 18:17 |
| johnthetubaguy | actually the scheduling can happen first, then we can start claiming the new style, then we drop reporting the legacy resource classes | 18:18 |
| johnthetubaguy | I think | 18:18 |
| *** psachin has joined #openstack-nova | 18:19 | |
| *** tbachman has quit IRC | 18:20 | |
| *** JoseMello has joined #openstack-nova | 18:21 | |
| *** vsaienko has quit IRC | 18:22 | |
| openstackgerrit | Dan Smith proposed openstack/nova: Avoid redundant call to update_resource_stats from RT https://review.openstack.org/424305 | 18:23 |
| *** tbachman has joined #openstack-nova | 18:24 | |
| *** jose-phillips has quit IRC | 18:26 | |
| *** yhvh has quit IRC | 18:27 | |
| *** khushbu has joined #openstack-nova | 18:28 | |
| *** mnestratov|2 has quit IRC | 18:38 | |
| *** khushbu has quit IRC | 18:38 | |
| *** david-lyle has quit IRC | 18:39 | |
| openstackgerrit | Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 18:41 |
| figleaf | ^^ This just fixes the client bugs jroll found | 18:42 |
| *** dharinic is now known as dharinic|lunch | 18:42 | |
| *** slaweq has quit IRC | 18:45 | |
| *** dave-johnston has quit IRC | 18:45 | |
| *** owalsh has quit IRC | 18:47 | |
| *** slaweq has joined #openstack-nova | 18:47 | |
| *** owalsh has joined #openstack-nova | 18:47 | |
| *** slaweq has quit IRC | 18:52 | |
| *** slaweq has joined #openstack-nova | 18:52 | |
| *** bvanhav_ has joined #openstack-nova | 18:53 | |
| *** bvanhav has quit IRC | 18:54 | |
| *** ijw has joined #openstack-nova | 18:56 | |
| *** beagles is now known as beagles-biab | 18:58 | |
| *** liangy has joined #openstack-nova | 19:00 | |
| *** dtp is now known as dtp-afk | 19:04 | |
| *** weshay_brb is now known as weshay | 19:06 | |
| *** unicell has quit IRC | 19:12 | |
| *** hongbin has joined #openstack-nova | 19:12 | |
| openstackgerrit | John Garbutt proposed openstack/nova: Added instance.reboot.error to the legacy notifications https://review.openstack.org/411816 | 19:15 |
| *** Sukhdev has joined #openstack-nova | 19:18 | |
| *** slaweq has quit IRC | 19:19 | |
| *** tbachman has quit IRC | 19:26 | |
| *** dharinic|lunch is now known as dharinic | 19:27 | |
| bauzas | superdan: johnthetubaguy: I'm now in the train back to Grenoble, what's the current situation for the scheduler change? | 19:29 |
| superdan | bauzas: sdague wants the grenade change refactored, so I'm working on that | 19:30 |
| superdan | bauzas: the scheduler thing passed multinode grenade before I started, johnthetubaguy was going to fix up the pep8 and unit tests | 19:30 |
| bauzas | okay, thanks for both you | 19:31 |
| bauzas | the pep8 issue is really easy | 19:31 |
| sdague | superdan: yeh, the placement add patch seems good now, pending tests | 19:32 |
| bauzas | not sure I understand an unittest problem | 19:32 |
| superdan | sdague: I think it's still not working yet, but getting closer | 19:32 |
| *** Matias has quit IRC | 19:32 | |
| sdague | superdan: ok | 19:32 |
| superdan | bauzas: see all the red FAILED on the patch | 19:32 |
| *** Matias has joined #openstack-nova | 19:33 | |
| superdan | I have to run out to meet someone in a little bit, but I will pick it up when I get back | 19:33 |
| superdan | this person says they have a nice minimum wage job flipping burgers and I am SUPER INTERESTED | 19:33 |
| figleaf | superdan: much less stress, for sure | 19:35 |
| superdan | aye | 19:35 |
| *** unicell has joined #openstack-nova | 19:42 | |
| *** eglynn has quit IRC | 19:42 | |
| *** jheroux has quit IRC | 19:43 | |
| *** tbachman has joined #openstack-nova | 19:47 | |
| *** ijw_ has joined #openstack-nova | 19:47 | |
| bauzas | oh man, the internet connection in the my train is so terrible :( | 19:48 |
| *** jose-phillips has joined #openstack-nova | 19:49 | |
| *** ijw has quit IRC | 19:49 | |
| *** tbachman has quit IRC | 19:52 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876 | 19:52 |
| superdan | well, it upgraded placement without exploding, we'll see if it actually works | 19:52 |
| *** d-bark has joined #openstack-nova | 19:53 | |
| *** david-lyle has joined #openstack-nova | 19:53 | |
| bauzas | superdan: so you modified grenade by adding a new placement service? heh, okay | 19:53 |
| *** david-lyle has quit IRC | 19:53 | |
| bauzas | I thought it was within nova, but fine by me | 19:54 |
| *** david-lyle has joined #openstack-nova | 19:54 | |
| bauzas | FWIW, I'd need to rebase the top patch since matt wrote a new PS for the bottom one | 19:55 |
| *** unicell has quit IRC | 19:55 | |
| openstackgerrit | Artom Lifshitz proposed openstack/nova: Pass APIVersionRequest to extensions https://review.openstack.org/425876 | 19:55 |
| openstackgerrit | Artom Lifshitz proposed openstack/nova-specs: Fix tag attribute disappearing https://review.openstack.org/426030 | 19:57 |
| *** vsaienko has joined #openstack-nova | 19:58 | |
| *** Jack_V has quit IRC | 19:58 | |
| *** beagles-biab is now known as beagles | 20:00 | |
| *** psachin has quit IRC | 20:00 | |
| *** david-lyle has quit IRC | 20:01 | |
| *** david-lyle has joined #openstack-nova | 20:01 | |
| *** tbachman has joined #openstack-nova | 20:01 | |
| *** lpetrut has quit IRC | 20:03 | |
| openstackgerrit | Jim Rollenhagen proposed openstack/nova: DNM: hack ironic with resource providers https://review.openstack.org/426296 | 20:10 |
| jroll | figleaf: rebased that on your better patchset ^ | 20:10 |
| figleaf | jroll: working better? | 20:10 |
| *** dtp-afk is now known as dtp | 20:12 | |
| jroll | figleaf: I couldn't get it going for some reason, and dealing with other stuff since | 20:13 |
| jroll | saw you uploaded that so figured I'd kick my test patch off at least | 20:13 |
| openstackgerrit | Alex Szarka proposed openstack/nova: Transform instance.add_fixed_ip notification https://review.openstack.org/332876 | 20:27 |
| *** slaweq has joined #openstack-nova | 20:28 | |
| *** jose-phillips has quit IRC | 20:29 | |
| *** owalsh has quit IRC | 20:35 | |
| *** owalsh has joined #openstack-nova | 20:35 | |
| *** vnovakov has joined #openstack-nova | 20:36 | |
| *** bvanhav_ has quit IRC | 20:37 | |
| *** vsaienko has quit IRC | 20:43 | |
| *** vsaienko has joined #openstack-nova | 20:47 | |
| *** pradk has quit IRC | 20:47 | |
| *** claudiub has joined #openstack-nova | 20:47 | |
| *** armax has quit IRC | 20:49 | |
| *** claudiub|3 has quit IRC | 20:49 | |
| *** lpetrut has joined #openstack-nova | 20:50 | |
| *** dtp_ has joined #openstack-nova | 20:51 | |
| *** jose-phillips has joined #openstack-nova | 20:52 | |
| *** dtp has quit IRC | 20:54 | |
| *** sdague has quit IRC | 20:56 | |
| *** jose-phillips has quit IRC | 20:58 | |
| *** smatzek has quit IRC | 20:58 | |
| *** smatzek has joined #openstack-nova | 20:59 | |
| *** liangy has quit IRC | 21:03 | |
| *** haplo37_ has quit IRC | 21:05 | |
| *** sdague has joined #openstack-nova | 21:07 | |
| *** satyar has quit IRC | 21:08 | |
| *** tblakes has quit IRC | 21:12 | |
| *** cleong has quit IRC | 21:13 | |
| *** Apoorva has quit IRC | 21:15 | |
| *** haplo37_ has joined #openstack-nova | 21:16 | |
| *** raunak has quit IRC | 21:17 | |
| *** raunak has joined #openstack-nova | 21:19 | |
| *** dtp_ has quit IRC | 21:22 | |
| *** smatzek has quit IRC | 21:22 | |
| *** karimb has quit IRC | 21:23 | |
| *** tblakes has joined #openstack-nova | 21:25 | |
| *** armax has joined #openstack-nova | 21:26 | |
| superdan | sdague: even though I did a bunch of work on that grenade patch are you cool with me +W when we've got a good run from the nova patch? | 21:28 |
| sdague | superdan: the one I've got a +2 on? | 21:30 |
| sdague | if so, yes | 21:30 |
| superdan | sdague: yup | 21:30 |
| sdague | yep | 21:30 |
| superdan | thanks | 21:31 |
| *** tbachman has quit IRC | 21:31 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** Apoorva has joined #openstack-nova | 21:40 | |
| *** dimtruck is now known as zz_dimtruck | 21:40 | |
| *** vladikr has quit IRC | 21:40 | |
| *** Jeffrey4l_ has joined #openstack-nova | 21:41 | |
| *** pradk has joined #openstack-nova | 21:41 | |
| *** tbachman has joined #openstack-nova | 21:42 | |
| *** catintheroof has quit IRC | 21:44 | |
| *** catintheroof has joined #openstack-nova | 21:45 | |
| *** unicell has joined #openstack-nova | 21:46 | |
| *** unicell has quit IRC | 21:46 | |
| *** catintheroof has quit IRC | 21:49 | |
| *** unicell has joined #openstack-nova | 21:50 | |
| *** ijw_ has quit IRC | 21:51 | |
| *** raunak has quit IRC | 21:51 | |
| *** raunak has joined #openstack-nova | 21:54 | |
| *** ducttape_ has joined #openstack-nova | 21:54 | |
| *** JoseMello has quit IRC | 21:55 | |
| diana_clarke | superdan: What do you mean by "unmapped due to normally transient state"? https://github.com/openstack/nova/blob/e57714f9f162f60e77222b1ae8e1e78d2cb10a10/nova/cmd/manage.py#L1348 | 21:59 |
| diana_clarke | superdan: asking as I try to respond to this review feedback: https://review.openstack.org/#/c/421436/ | 22:00 |
| *** Guest21909 has quit IRC | 22:00 | |
| *** breitz has quit IRC | 22:01 | |
| *** breitz has joined #openstack-nova | 22:02 | |
| *** pradk has quit IRC | 22:02 | |
| *** rmart04 has joined #openstack-nova | 22:03 | |
| *** rmart04 has left #openstack-nova | 22:04 | |
| superdan | diana_clarke: like before it's scheduled | 22:09 |
| diana_clarke | superdan: So return code 2 will eventually fix itself, correct? It has an instance mapping, just no cell mapping. | 22:11 |
| *** jose-phillips has joined #openstack-nova | 22:11 | |
| superdan | diana_clarke: yeah | 22:13 |
| diana_clarke | So perhaps that should read: "Returns 0 when the instance is successfully mapped to a cell, 1 if the instance is not mapped to a cell, and 2 if the instance is in the process of being mapped to a cell." | 22:13 |
| superdan | it's a really small window | 22:13 |
| superdan | sure | 22:13 |
| *** vishwanathj has joined #openstack-nova | 22:13 | |
| diana_clarke | so return code 1 is an error case, right? | 22:13 |
| superdan | I mean if you like less awesome comments | 22:13 |
| diana_clarke | superdan: not changing your comment, trying to write the man page | 22:14 |
| diana_clarke | ;) | 22:14 |
| superdan | they both mean that the instance isn't functional, | 22:14 |
| *** vishwanathj has quit IRC | 22:14 | |
| superdan | and 1 could be a transient case as well I think, let me think about it | 22:14 |
| superdan | I know, I meant "awesome" in the snarky way | 22:14 |
| *** vishwanathj has joined #openstack-nova | 22:15 | |
| openstackgerrit | Dan Smith proposed openstack/nova: Scheduler calling the Placement API https://review.openstack.org/417961 | 22:15 |
| openstackgerrit | Dan Smith proposed openstack/nova: Block starting compute unless placement conf is provided https://review.openstack.org/425806 | 22:15 |
| *** thorst_ has quit IRC | 22:16 | |
| diana_clarke | Or asking differently (like johnthetubaguy), what action should be taken for return codes 1 & 2? | 22:16 |
| superdan | diana_clarke: okay, so from here forward, 1 is always an error, but not one we expect to happen, because: | 22:16 |
| *** zz_dimtruck is now known as dimtruck | 22:16 | |
| superdan | in the past, we didn't have instance mappings, so if you have an instance in the database and no mapping, it was created on an older release and you didn't do your homework to patch up all the mappings yet | 22:17 |
| *** vishwanathj has quit IRC | 22:17 | |
| superdan | so the point of that tool is really to verify that you created instance mappings for all of them | 22:17 |
| superdan | however, if you run that while things are going on, you could get retval=2 due to a transient luck-based event | 22:17 |
| superdan | but doing the online migrations should fix both cases up for any non-booting instances | 22:18 |
| superdan | so the answer to "what to do" is run the map command | 22:18 |
| superdan | map_instances for 1 and map_cell_and_hosts for 2 (modulo transientness) | 22:19 |
| superdan | clear as mud? | 22:19 |
| diana_clarke | superdan: I going to re-read that 10 times in a row, and then hopefully it will be ;) much appreciated! | 22:21 |
| superdan | let me say it again linearly now that I have it in my head... | 22:21 |
| superdan | If you get retval=1, that means you need to run map_instances as some instances (which were created with a previous version) don't have instance mappings yet | 22:22 |
| superdan | if you get retval=2, that may be because you're racing with instances that are booting, or it may mean that you are missing cell mappings, which means you need to run map_cell_and_hosts | 22:22 |
| *** edmondsw_ has quit IRC | 22:23 | |
| superdan | someone could modify that code to not return 2 if the instance has a building-related task state such that we know it's the former case for retval=2 | 22:23 |
| *** edmondsw has joined #openstack-nova | 22:23 | |
| diana_clarke | nice, thx! | 22:24 |
| *** gouthamr has quit IRC | 22:27 | |
| *** armax has quit IRC | 22:27 | |
| *** edmondsw has quit IRC | 22:28 | |
| *** baoli has quit IRC | 22:28 | |
| *** slaweq_ has quit IRC | 22:30 | |
| *** esberglu_ has quit IRC | 22:36 | |
| *** esberglu has joined #openstack-nova | 22:36 | |
| *** esberglu has quit IRC | 22:41 | |
| *** jaosorior has quit IRC | 22:42 | |
| openstackgerrit | Ed Leafe proposed openstack/nova: placement: RT now adds proper Ironic inventory https://review.openstack.org/404472 | 22:43 |
| figleaf | johnthetubaguy: cdent: ^^ adds the workarounds for the problem we identified earlier. | 22:44 |
| *** xyang1 has quit IRC | 22:44 | |
| *** ijw has joined #openstack-nova | 22:47 | |
| *** ijw has quit IRC | 22:47 | |
| *** ijw has joined #openstack-nova | 22:47 | |
| mriedem_beach | hola | 22:48 |
| *** mriedem_beach is now known as mriedem | 22:48 | |
| mriedem | donde esta el progresso? | 22:48 |
| superdan | where is the progress? | 22:49 |
| mriedem | si | 22:49 |
| mriedem | grenade patch is +W | 22:49 |
| superdan | is it? | 22:49 |
| mriedem | i thought you were going to +W it | 22:49 |
| superdan | I will when I see the nova one yeah | 22:49 |
| mriedem | oh | 22:49 |
| superdan | I think the nova patch was running against a stale version of the grenade one, | 22:50 |
| superdan | so it's going again now | 22:50 |
| superdan | john fixed up a bunch of the unit and pep8 issues | 22:50 |
| mriedem | yeah i see the series must have been rebased | 22:50 |
| mriedem | i'll review that all again | 22:50 |
| *** burgerk has quit IRC | 22:50 | |
| *** esberglu has joined #openstack-nova | 22:50 | |
| *** gouthamr has joined #openstack-nova | 22:51 | |
| superdan | still getting NoValidHost on 11 tests on regular grenade ... | 22:51 |
| superdan | jeez | 22:51 |
| superdan | compute is still not starting because of detecting the placement config | 22:52 |
| mriedem | i thought things were peachy this morning? | 22:52 |
| mriedem | or was that not valid b/c of bad deps? | 22:52 |
| superdan | yeah, I dunno now, I thought we were past that | 22:52 |
| *** kfarr has quit IRC | 22:53 | |
| *** Swami_ has joined #openstack-nova | 22:53 | |
| mriedem | maybe changing the grenade patch messed something up? | 22:53 |
| superdan | PS3 worked. but that was the last time | 22:53 |
| openstackgerrit | Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 22:54 |
| superdan | but I see no reason why, there's been almost no change since | 22:54 |
| *** esberglu has quit IRC | 22:54 | |
| *** lpetrut has quit IRC | 22:55 | |
| superdan | yeah no placement config in nova.conf | 22:56 |
| superdan | grr | 22:56 |
| *** Swami has quit IRC | 22:57 | |
| mriedem | hmm, | 22:58 |
| mriedem | we're now deploying placement before nova in grenade right? | 22:58 |
| superdan | yeah | 22:58 |
| *** vsaienko has quit IRC | 22:58 | |
| mriedem | which should configure nova.conf on the primary node, | 22:58 |
| mriedem | which would actually be newton at that point | 22:58 |
| mriedem | since it's pre-upgrade | 22:58 |
| mriedem | 55 is updating a newton compute i mean | 22:58 |
| *** jamesdenton has quit IRC | 22:58 | |
| superdan | but we still call the configure bit | 22:58 |
| mriedem | sure, | 22:59 |
| *** takedakn has joined #openstack-nova | 22:59 | |
| mriedem | and that shouldn't get wiped out or anything by the 60_nova part | 22:59 |
| openstackgerrit | Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 22:59 |
| *** Swami_ has quit IRC | 23:00 | |
| mriedem | huh, so now both of the nova changes depend on the grenade change | 23:00 |
| mriedem | doesn't hurt anything | 23:00 |
| superdan | I did that because it ran against a stale grenade patch for some reason | 23:01 |
| superdan | so when I rebased I just added it on | 23:01 |
| superdan | didn't know why | 23:01 |
| *** tbachman has quit IRC | 23:01 | |
| superdan | ahhh | 23:01 |
| superdan | I wonder if we still need to do configure_placement_nova_compute in the 60_nova bit | 23:01 |
| superdan | not sure why it wouldn't survive, but that's the part that gets us placemetn config | 23:02 |
| superdan | also, what about this? http://logs.openstack.org/06/425806/6/check/gate-grenade-dsvm-neutron-ubuntu-xenial/fb281f1/logs/grenade.sh.txt.gz#_2017-01-27_20_02_44_142 | 23:02 |
| mriedem | https://review.openstack.org/#/c/424730/15/projects/55_placement/from-newton/upgrade-placement@17 | 23:02 |
| mriedem | ^ should configure nova-compute on the primary node for placement | 23:02 |
| superdan | I know | 23:02 |
| superdan | but | 23:03 |
| superdan | after the run it's not set | 23:03 |
| *** ducttape_ has quit IRC | 23:03 | |
| superdan | oh wait | 23:03 |
| superdan | it doesn't have $NOVA_CONF in the run | 23:03 |
| *** ducttape_ has joined #openstack-nova | 23:03 | |
| mriedem | because we didn't source the nova stuff yet? | 23:04 |
| *** slaweq_ has joined #openstack-nova | 23:04 | |
| superdan | yeah | 23:04 |
| mriedem | teehee | 23:04 |
| mriedem | i believe i called this ~6 hours ago | 23:04 |
| superdan | yes you did | 23:04 |
| superdan | it's definitely weird doing this out of body but still part of nova | 23:05 |
| mriedem | superdan: so do we no longer need the thing you added where it saves off the number of computes before and checks for that after? | 23:05 |
| superdan | mriedem: that is being challenged, so I split it out | 23:06 |
| superdan | with this set we're introducing a very specific race because the scheduler checks the min vers at startup | 23:06 |
| superdan | which is why I think we need it | 23:06 |
| *** annegentle has quit IRC | 23:06 | |
| superdan | but we haven't hit it yet, so ... | 23:06 |
| superdan | mriedem: this is the standalone patch: https://review.openstack.org/#/c/426310/ | 23:07 |
| superdan | which I see I have tabs in.. gdi | 23:08 |
| *** figleaf is now known as edleafe | 23:09 | |
| *** tblakes has quit IRC | 23:10 | |
| *** greghaynes has quit IRC | 23:10 | |
| *** mdrabe has quit IRC | 23:14 | |
| mriedem | ok the 55_placement thing looks good now | 23:17 |
| mriedem | now we play the waiting game | 23:19 |
| *** marst has quit IRC | 23:20 | |
| *** tbachman has joined #openstack-nova | 23:21 | |
| superdan | cool | 23:21 |
| superdan | personally, | 23:21 |
| superdan | I hate the effing waiting game | 23:21 |
| superdan | I've been playing it all week and I'm done with it | 23:21 |
| *** ducttape_ has quit IRC | 23:24 | |
| *** greghaynes has joined #openstack-nova | 23:37 | |
| openstackgerrit | Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 23:40 |
| *** jamesdenton has joined #openstack-nova | 23:41 | |
| openstackgerrit | Pushkar Umaranikar proposed openstack/nova: Adopts keystoneauth with glance client. https://review.openstack.org/412634 | 23:42 |
| *** edmondsw has joined #openstack-nova | 23:43 | |
| *** Swami has joined #openstack-nova | 23:43 | |
| *** jamesden_ has joined #openstack-nova | 23:44 | |
| *** Swami has quit IRC | 23:44 | |
| *** Swami has joined #openstack-nova | 23:44 | |
| *** jamesdenton has quit IRC | 23:45 | |
| mriedem | diana_clarke: just went through https://review.openstack.org/#/q/status:open+project:openstack/nova+branch:master+topic:man - some comments inline, | 23:48 |
| mriedem | i think there are only a few that need some tweaks | 23:48 |
| *** Swami_ has joined #openstack-nova | 23:48 | |
| mriedem | like discover_hosts should come after create_cell because you can't map the hosts to a cell w/o a cell | 23:48 |
| openstackgerrit | Matt Riedemann proposed openstack/nova: DNM: hack ironic with resource providers https://review.openstack.org/426296 | 23:50 |
| *** Swami has quit IRC | 23:52 | |
| *** smatzek has joined #openstack-nova | 23:52 | |
| *** greghaynes has quit IRC | 23:52 | |
| diana_clarke | mriedem: Yes, thanks a ton! I'll circle back and add more detail etc monday at the latest. | 23:52 |
| mriedem | superdan: passed http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/5fb6f74/ | 23:53 |
| *** smatzek_ has joined #openstack-nova | 23:53 | |
| superdan | like, yay | 23:53 |
| mriedem | http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-ubuntu-xenial/5fb6f74/logs/new/screen-n-sch.txt.gz#_2017-01-27_23_44_07_736 | 23:54 |
| mriedem | 2017-01-27 23:44:07.736 12381 DEBUG nova.scheduler.filter_scheduler [req-f76d4b01-c248-4de4-8675-a4cd3cb6d4cb tempest-AttachInterfacesTestJSON-881960388 tempest-AttachInterfacesTestJSON-881960388] Skipping call to placement, as upgrade in progress. _get_all_host_states /opt/stack/new/nova/nova/scheduler/filter_scheduler.py:179 | 23:54 |
| superdan | woot | 23:54 |
| *** smatzek_ has quit IRC | 23:54 | |
| *** tbachman has quit IRC | 23:55 | |
| *** smatzek_ has joined #openstack-nova | 23:55 | |
| mriedem | wonder what the whiny ass grenade + live migration job is complaining about | 23:55 |
| *** ducttape_ has joined #openstack-nova | 23:56 | |
| mriedem | heh wtf http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8c363cd/logs/devstack-gate-post_test_hook.txt.gz | 23:56 |
| mriedem | tdurakov: ^ | 23:57 |
| mriedem | grenade + live migration apparently never passes | 23:57 |
| superdan | lol | 23:57 |
| *** smatzek has quit IRC | 23:57 | |
| mriedem | http://logs.openstack.org/61/417961/40/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/8c363cd/logs/new/tempest_conf.txt.gz | 23:57 |
| mriedem | block_migration_for_live_migration = False live_migration = False | 23:57 |
| melwitt | I've noticed that before. when all tests are skipped, it doesn't consider that a success | 23:58 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!