| *** catintheroof has joined #openstack-nova | 00:03 | |
| sbezverk | mriedem: correct about individual containers | 00:03 |
|---|---|---|
| sbezverk | mriedem: in paralel I have a job running newton images and it passes wihtout issues | 00:03 |
| sbezverk | mriedem: I will continue debugging to confirm connectivity issue.. curl should tell me that | 00:04 |
| Sukhdev | mriedem : I see lots of time out errors in n-cpu.logs - here is a sample - http://paste.openstack.org/show/603037/ | 00:04 |
| *** iceyao has joined #openstack-nova | 00:07 | |
| *** tbachman_ has joined #openstack-nova | 00:08 | |
| *** claudiub has quit IRC | 00:09 | |
| *** iceyao has quit IRC | 00:09 | |
| mriedem | Sukhdev: do you have any customizations to the local.conf you used to stack devstack? can you post your local.conf in a paste? | 00:09 |
| mriedem | seems you're having rpc issues, which would mean investigating the oslo.messaging configuration values in nova.conf | 00:09 |
| *** tbachman has quit IRC | 00:10 | |
| *** tbachman_ is now known as tbachman | 00:10 | |
| *** yingjun has joined #openstack-nova | 00:11 | |
| Sukhdev | mriedem : here is local.conf - http://paste.openstack.org/show/603039/ | 00:13 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | 00:13 |
| Sukhdev | mriedem : here is /etc/nova/nova.conf - http://paste.openstack.org/show/603040/ | 00:15 |
| *** catintheroof has quit IRC | 00:16 | |
| *** sbezverk has quit IRC | 00:18 | |
| *** zz_dimtruck is now known as dimtruck | 00:19 | |
| *** ssurana has quit IRC | 00:19 | |
| *** sbezverk has joined #openstack-nova | 00:20 | |
| Sukhdev | mriedem : can you spot anything suspicious in either of those configs? | 00:22 |
| *** sbezverk has quit IRC | 00:24 | |
| *** Jeffrey4l has quit IRC | 00:25 | |
| mriedem | they look ok, | 00:28 |
| mriedem | they are all for ironic though, and i've never setup an ironic devstack instance, so i'm not sure what to be looking for, | 00:28 |
| mriedem | you might want to ask in the #openstack-ironic channel | 00:28 |
| *** Sukhdev_ has joined #openstack-nova | 00:30 | |
| openstackgerrit | Tony Breeds proposed openstack/os-vif master: Add Constraints support https://review.openstack.org/413325 | 00:31 |
| *** amotoki has joined #openstack-nova | 00:37 | |
| Sukhdev | mriedem : this does not go to ironic at all - as soon as nova boot is issued, it fails in nova itself. So, there is no log in the Ironic side | 00:38 |
| *** gouthamr has joined #openstack-nova | 00:38 | |
| *** amotoki has quit IRC | 00:41 | |
| mriedem | Sukhdev: is ironic running OK? | 00:42 |
| mriedem | the ironic virt driver in the nova-compute service will be getting resource information from the ironic API service | 00:42 |
| Sukhdev | mriedem : yes | 00:42 |
| mriedem | that resource information is fed into the placement service, | 00:42 |
| mriedem | and the filter scheduler uses that information for scheduling decisions | 00:43 |
| Sukhdev | mriedem : I did not look into Ironic API logs - let me take a peek | 00:44 |
| mriedem | Sukhdev: you could also compare to the local.conf from the ironic CI job that runs on nova changes here http://logs.openstack.org/01/446601/1/check/gate-tempest-dsvm-ironic-ipa-wholedisk-bios-agent_ipmitool-tinyipa-ubuntu-xenial-nv/4630b48/logs/local.conf.txt.gz | 00:46 |
| mriedem | it could be the flavors you have defined don't fit the nodes available | 00:47 |
| Sukhdev | mriedem : excellent Idea - thanks for the pointer - let me check if some of this config is causing the issue. | 00:47 |
| Sukhdev | mriedem : One quick question (for my clarification) I read somewhere while looking for help on Google it stated that in Ocata release Nova mendate cell configuration - is it true? Do I need to add anything in local.conf in this regard> | 00:53 |
| *** sbezverk has joined #openstack-nova | 00:53 | |
| mriedem | Sukhdev: you don't need anything in local.conf for cells v2 in ocata | 00:54 |
| Sukhdev | mriedem : cool - thanks - was worried if that has anything to do with this :-) | 00:54 |
| mriedem | i don't think it would, the scheduler in ocata didn't take cells into consideration | 00:55 |
| Sukhdev | Great - thanks | 00:55 |
| sbezverk | mriedemЬ curl returns 401 Unauthorized | 00:55 |
| sbezverk | mriedem: it does not look like network issue then.. | 00:56 |
| mriedem | sbezverk: did you provide a token in the curl request to / ? | 00:57 |
| *** dikonoor has joined #openstack-nova | 00:58 | |
| sbezverk | mriedem: cannot do it as it is a gate job | 01:00 |
| *** crusio has quit IRC | 01:00 | |
| melwitt | mriedem: argh, dangit (test trace) | 01:01 |
| *** crusio has joined #openstack-nova | 01:01 | |
| sbezverk | mriedem: do not have much flexebility there | 01:01 |
| melwitt | oh, you took care of it. thanks | 01:01 |
| mriedem | melwitt: yup | 01:01 |
| *** catintheroof has joined #openstack-nova | 01:02 | |
| *** catintheroof has quit IRC | 01:02 | |
| mriedem | sbezverk: i'm not sure what you mean. you're scripting this into the CI job right? if you've added something that makes a curl request to the root of the placement endpoint from the nova-scheduler container, can't you also get a token and include that in the request, from within the script/CI job? | 01:02 |
| *** catintheroof has joined #openstack-nova | 01:02 | |
| mriedem | OS_TOKEN=$(openstack token issue -f value -c id) | 01:02 |
| sbezverk | mriedem: cool thanks, let me try with it | 01:03 |
| *** phuongnh has joined #openstack-nova | 01:04 | |
| *** Apoorva has quit IRC | 01:06 | |
| dtp-afk | jroll are you still here? | 01:08 |
| *** dtp-afk is now known as dtp | 01:09 | |
| *** kevinz has joined #openstack-nova | 01:10 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | 01:10 |
| *** sbezverk has quit IRC | 01:12 | |
| *** sbezverk has joined #openstack-nova | 01:12 | |
| *** armax has quit IRC | 01:13 | |
| *** baoli has joined #openstack-nova | 01:14 | |
| openstackgerrit | Dan Peschman proposed openstack/nova master: Option to always create network metadata information as static https://review.openstack.org/336293 | 01:18 |
| *** unicell has quit IRC | 01:19 | |
| *** unicell has joined #openstack-nova | 01:21 | |
| *** dikonoor has quit IRC | 01:26 | |
| *** unicell has quit IRC | 01:31 | |
| *** liangy has joined #openstack-nova | 01:36 | |
| sbezverk | mriedem: ok so the process is alive, there must be something with scheduler preventing it finding it.. | 01:39 |
| sbezverk | mriedem: http://paste.openstack.org/show/603043/ | 01:39 |
| *** Sukhdev has quit IRC | 01:40 | |
| *** Sukhdev_ is now known as Sukhdev | 01:40 | |
| mriedem | sbezverk: try restarting nova-scheduler, i think we cache the failed attempt and don't try again for like 10 minutes | 01:40 |
| *** wangqun has joined #openstack-nova | 01:43 | |
| dtp | mriedem, if i have other ideas on implementation of https://blueprints.launchpad.net/nova/+spec/service-hyper-pci-uuid-in-api, should i just keep them to myself? | 01:44 |
| sbezverk | mriedem: well, it is at the gate. once failed the job is failed/gone.. | 01:44 |
| mriedem | sbezverk: you can't reproduce locally? | 01:45 |
| mriedem | dtp: umm, idk | 01:45 |
| *** amotoki has joined #openstack-nova | 01:45 | |
| mriedem | dtp: we could talk tomorrow, i've got to call it a day | 01:45 |
| dtp | sounds good | 01:45 |
| sbezverk | mriedem: when scheduler starts it tries to communicate with placement right away? | 01:46 |
| mriedem | sbezverk: i don't think so no | 01:46 |
| mriedem | only once a request to create an instance comes in | 01:46 |
| sbezverk | mriedem: ok so ordering would not matter much | 01:46 |
| sbezverk | I mean process start up | 01:46 |
| mriedem | sbezverk: placement should come before the scheduler, but i'd have to dig into details, which i can't right now | 01:47 |
| mriedem | sbezverk: check that nova devref doc on placement that i linked earlier, there might be something i'm forgeting | 01:47 |
| sbezverk | mriedem: ok I will, also will try to repro in local test bed. thank you for your help | 01:47 |
| mriedem | yw | 01:48 |
| *** amotoki has quit IRC | 01:50 | |
| *** Jeffrey4l has joined #openstack-nova | 01:53 | |
| *** jamielennox is now known as jamielennox|away | 01:56 | |
| *** rfolco_ has joined #openstack-nova | 01:57 | |
| *** rfolco_ has quit IRC | 01:57 | |
| gmann | alex_xu: sfinucan mriedem can you check this. updated enum value format (api-ref for sort/filter) - https://review.openstack.org/#/c/421760/ | 01:59 |
| *** zioproto has quit IRC | 02:01 | |
| *** jamielennox|away is now known as jamielennox | 02:01 | |
| *** dtp has quit IRC | 02:03 | |
| *** dimtruck is now known as zz_dimtruck | 02:06 | |
| *** tanee is now known as tanee_away | 02:06 | |
| *** tanee_away is now known as tanee | 02:07 | |
| *** kaisers__ has quit IRC | 02:09 | |
| *** crusio has quit IRC | 02:12 | |
| *** crusio has joined #openstack-nova | 02:12 | |
| *** Jeffrey4l has quit IRC | 02:14 | |
| *** Jeffrey4l has joined #openstack-nova | 02:14 | |
| alex_xu | gmann: thanks, I will check it soon | 02:23 |
| gmann | mriedem: alex_xu johnthetubaguy sfinucan snikitin : i doubt cpu id should be integer, i replied on https://review.openstack.org/#/c/445855/1 and https://review.openstack.org/#/c/434745/3 | 02:24 |
| gmann | need your opinion | 02:24 |
| gmann | alex_xu: thanks | 02:24 |
| *** masber has joined #openstack-nova | 02:26 | |
| gmann | libvirt driver(current diagnostics info provider) seems not appending the CPU id | 02:26 |
| *** Shunli has joined #openstack-nova | 02:27 | |
| *** Shunli has quit IRC | 02:29 | |
| *** Shunli has joined #openstack-nova | 02:29 | |
| *** fragatin_ has joined #openstack-nova | 02:34 | |
| *** yamahata has quit IRC | 02:35 | |
| *** Sukhdev has quit IRC | 02:36 | |
| *** jichen has joined #openstack-nova | 02:36 | |
| *** tbachman has quit IRC | 02:36 | |
| *** catintheroof has quit IRC | 02:36 | |
| *** fragatina has quit IRC | 02:37 | |
| *** tbachman has joined #openstack-nova | 02:39 | |
| *** fragatin_ has quit IRC | 02:39 | |
| *** crusio has quit IRC | 02:39 | |
| *** crusio has joined #openstack-nova | 02:40 | |
| *** Jeffrey4l has quit IRC | 02:41 | |
| *** Jeffrey4l has joined #openstack-nova | 02:41 | |
| *** hongbin_ has joined #openstack-nova | 02:50 | |
| *** gouthamr has quit IRC | 03:01 | |
| *** vladikr has quit IRC | 03:03 | |
| *** mirrorbox has quit IRC | 03:04 | |
| *** mirrorbox has joined #openstack-nova | 03:04 | |
| *** sree has joined #openstack-nova | 03:14 | |
| *** sree_ has joined #openstack-nova | 03:16 | |
| *** sree has quit IRC | 03:17 | |
| *** sree_ is now known as Guest86435 | 03:17 | |
| *** crushil has joined #openstack-nova | 03:20 | |
| *** zz_dimtruck is now known as dimtruck | 03:27 | |
| *** liangy has quit IRC | 03:33 | |
| *** nicolasbock has quit IRC | 03:43 | |
| *** amotoki has joined #openstack-nova | 03:46 | |
| *** ijw has quit IRC | 03:50 | |
| *** ijw has joined #openstack-nova | 03:50 | |
| *** amotoki has quit IRC | 03:51 | |
| *** ijw has quit IRC | 03:52 | |
| *** markvoelker_ has quit IRC | 03:53 | |
| *** crusio has quit IRC | 03:56 | |
| *** crusio has joined #openstack-nova | 03:57 | |
| *** yuntongjin has joined #openstack-nova | 03:58 | |
| *** udesale has joined #openstack-nova | 04:00 | |
| *** dikonoor has joined #openstack-nova | 04:04 | |
| *** yamahata has joined #openstack-nova | 04:13 | |
| *** alex_xu has quit IRC | 04:17 | |
| *** Daisy has joined #openstack-nova | 04:19 | |
| *** owalsh_ has joined #openstack-nova | 04:20 | |
| *** Daisy has quit IRC | 04:20 | |
| *** alex_xu has joined #openstack-nova | 04:23 | |
| *** owalsh has quit IRC | 04:23 | |
| *** gongysh has joined #openstack-nova | 04:27 | |
| *** kaisers_ has joined #openstack-nova | 04:27 | |
| *** kaisers_ has quit IRC | 04:28 | |
| *** psachin has joined #openstack-nova | 04:29 | |
| *** ayogi has joined #openstack-nova | 04:30 | |
| *** fragatina has joined #openstack-nova | 04:31 | |
| *** vks1 has joined #openstack-nova | 04:31 | |
| *** fragatina has quit IRC | 04:32 | |
| *** fragatina has joined #openstack-nova | 04:33 | |
| *** dimtruck is now known as zz_dimtruck | 04:35 | |
| gmann | BobBall: can we use VM_metrics.get_VCPUs_utilisation to fetch the CPU utilization ? - https://review.openstack.org/#/c/399613/29 | 04:37 |
| *** hongbin_ has quit IRC | 04:39 | |
| *** unicell has joined #openstack-nova | 04:41 | |
| *** nic has quit IRC | 04:44 | |
| *** yuntongjin has quit IRC | 04:51 | |
| *** jwcroppe has joined #openstack-nova | 04:55 | |
| *** ratailor has joined #openstack-nova | 04:57 | |
| *** unicell1 has joined #openstack-nova | 05:11 | |
| *** unicell has quit IRC | 05:11 | |
| *** crushil has quit IRC | 05:12 | |
| *** yuntongjin has joined #openstack-nova | 05:20 | |
| *** ssurana has joined #openstack-nova | 05:22 | |
| *** Jack_Iv has joined #openstack-nova | 05:24 | |
| *** kaisers_ has joined #openstack-nova | 05:29 | |
| *** ssurana has quit IRC | 05:29 | |
| *** sree has joined #openstack-nova | 05:30 | |
| *** Guest86435 has quit IRC | 05:34 | |
| *** kaisers_ has quit IRC | 05:34 | |
| *** erlon has quit IRC | 05:35 | |
| *** NikhilS has joined #openstack-nova | 05:36 | |
| *** fragatina has quit IRC | 05:40 | |
| *** ecelik has joined #openstack-nova | 05:50 | |
| *** ecelik has left #openstack-nova | 05:50 | |
| *** trinaths has joined #openstack-nova | 05:51 | |
| *** ijw has joined #openstack-nova | 05:51 | |
| *** gcb has joined #openstack-nova | 05:54 | |
| *** ijw_ has joined #openstack-nova | 05:54 | |
| *** ijw_ has quit IRC | 05:56 | |
| *** ijw_ has joined #openstack-nova | 05:56 | |
| *** ijw has quit IRC | 05:58 | |
| *** ijw_ has quit IRC | 06:01 | |
| *** Jeffrey4l has quit IRC | 06:02 | |
| *** jith has quit IRC | 06:07 | |
| *** tuanluong has joined #openstack-nova | 06:11 | |
| *** avolkov has joined #openstack-nova | 06:11 | |
| *** jith has joined #openstack-nova | 06:16 | |
| *** Jeffrey4l has joined #openstack-nova | 06:17 | |
| *** lpetrut has joined #openstack-nova | 06:19 | |
| *** zhurong has joined #openstack-nova | 06:21 | |
| *** phuongnh has quit IRC | 06:25 | |
| *** sree has quit IRC | 06:26 | |
| *** ltomasbo|away is now known as ltomasbo | 06:26 | |
| *** phuongnh has joined #openstack-nova | 06:26 | |
| *** satyar has joined #openstack-nova | 06:31 | |
| *** ayogi has quit IRC | 06:32 | |
| *** sridharg has joined #openstack-nova | 06:37 | |
| *** belmoreira has joined #openstack-nova | 06:38 | |
| *** ayogi has joined #openstack-nova | 06:47 | |
| alex_xu | gmann: may I say 'valid value' instead of 'enum value' at https://review.openstack.org/#/c/421760/15/api-ref/source/parameters.yaml@421 | 06:49 |
| alex_xu | gmann: due to from the schema, it just a field accept any string | 06:49 |
| alex_xu | gmann: you won't get error when you input something out of 'auto' and 'manual' | 06:49 |
| gmann | alex_xu: ah, nice point. ll update. | 06:52 |
| alex_xu | gmann: thanks | 06:52 |
| gmann | alex_xu: please -1 :) | 06:52 |
| alex_xu | gmann: ah, yea, got one point | 06:52 |
| *** andreas_s has joined #openstack-nova | 06:53 | |
| alex_xu | gmann: done, thanks :) | 06:53 |
| *** gcb has quit IRC | 06:54 | |
| alex_xu | gmann: I can update the patch online | 06:54 |
| alex_xu | do you want me do that, and I can +2 directly | 06:54 |
| gmann | alex_xu: updating | 06:54 |
| alex_xu | gmann: thanks | 06:54 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova master: Add api-ref for filter/sort whitelist https://review.openstack.org/421760 | 06:56 |
| gmann | alex_xu: ^^ | 06:57 |
| alex_xu | gmann: thanks, done | 06:57 |
| gmann | alex_xu: thanks. | 06:58 |
| *** nkrinner_afk is now known as nkrinner | 07:00 | |
| *** claudiub has joined #openstack-nova | 07:01 | |
| *** unicell1 has quit IRC | 07:04 | |
| *** unicell has joined #openstack-nova | 07:04 | |
| *** unicell has quit IRC | 07:04 | |
| *** gongysh has quit IRC | 07:07 | |
| *** gongysh has joined #openstack-nova | 07:10 | |
| *** slaweq has quit IRC | 07:10 | |
| *** gcb has joined #openstack-nova | 07:10 | |
| *** markus_z has joined #openstack-nova | 07:12 | |
| *** jaosorior has joined #openstack-nova | 07:16 | |
| *** lpetrut has quit IRC | 07:18 | |
| *** JayF has quit IRC | 07:20 | |
| *** mrhillsman has quit IRC | 07:20 | |
| *** jbernard has quit IRC | 07:23 | |
| *** jbernard has joined #openstack-nova | 07:23 | |
| *** zz_dimtruck is now known as dimtruck | 07:26 | |
| markus_z | bauzas: lyarwood: claudiub: Please add this to your review backlog: https://review.openstack.org/#/c/406898/ It's a backport to stable/newton. mriedem's questions there are answered but he's busy with other items. | 07:27 |
| *** kaisers_ has joined #openstack-nova | 07:30 | |
| *** JayF has joined #openstack-nova | 07:33 | |
| *** mrhillsman has joined #openstack-nova | 07:33 | |
| *** yuntongjin has quit IRC | 07:34 | |
| *** tesseract has joined #openstack-nova | 07:34 | |
| *** lpetrut has joined #openstack-nova | 07:35 | |
| *** phuongnh has quit IRC | 07:35 | |
| *** kaisers_ has quit IRC | 07:36 | |
| *** dimtruck is now known as zz_dimtruck | 07:36 | |
| *** phuongnh has joined #openstack-nova | 07:36 | |
| *** kaisers_ has joined #openstack-nova | 07:41 | |
| *** yuntongjin has joined #openstack-nova | 07:41 | |
| *** pcaruana has joined #openstack-nova | 07:43 | |
| *** kaisers_ has quit IRC | 07:45 | |
| *** amotoki has joined #openstack-nova | 07:48 | |
| *** sree has joined #openstack-nova | 07:48 | |
| *** voelzmo has joined #openstack-nova | 07:48 | |
| *** Jack_Iv has quit IRC | 07:52 | |
| *** amotoki has quit IRC | 07:52 | |
| *** Jack_Iv has joined #openstack-nova | 07:52 | |
| *** sree has quit IRC | 07:52 | |
| *** Jack_Iv has quit IRC | 07:56 | |
| *** zzzeek has quit IRC | 08:00 | |
| *** zzzeek has joined #openstack-nova | 08:00 | |
| *** hferenc has quit IRC | 08:01 | |
| *** Jack_Iv has joined #openstack-nova | 08:01 | |
| *** karthiks has quit IRC | 08:03 | |
| *** lpetrut has quit IRC | 08:04 | |
| *** lpetrut has joined #openstack-nova | 08:04 | |
| *** jpena|off is now known as jpena | 08:05 | |
| *** jpena is now known as jpena|off | 08:18 | |
| *** zioproto has joined #openstack-nova | 08:18 | |
| *** karthiks has joined #openstack-nova | 08:19 | |
| *** Jack_Iv has quit IRC | 08:22 | |
| *** Jack_Iv has joined #openstack-nova | 08:23 | |
| *** jpena|off is now known as jpena | 08:25 | |
| *** Jack_Iv has quit IRC | 08:27 | |
| *** zhurong has quit IRC | 08:27 | |
| *** yuntongjin1 has joined #openstack-nova | 08:29 | |
| *** yuntongjin has quit IRC | 08:29 | |
| *** amoralej|off is now known as amoralej | 08:38 | |
| *** ygl has joined #openstack-nova | 08:38 | |
| *** karthiks has quit IRC | 08:39 | |
| *** sree has joined #openstack-nova | 08:39 | |
| *** udesale has quit IRC | 08:41 | |
| *** udesale has joined #openstack-nova | 08:41 | |
| *** kaisers_ has joined #openstack-nova | 08:42 | |
| *** aarefiev_afk is now known as aarefiev | 08:43 | |
| *** lpetrut has quit IRC | 08:44 | |
| *** kaisers_ has quit IRC | 08:46 | |
| *** hferenc has joined #openstack-nova | 08:47 | |
| *** karthiks has joined #openstack-nova | 08:51 | |
| *** gcb has quit IRC | 08:51 | |
| *** ducnc has quit IRC | 08:52 | |
| *** crusio has quit IRC | 08:52 | |
| *** crusio has joined #openstack-nova | 08:52 | |
| *** yamamoto has quit IRC | 08:57 | |
| *** belmorei_ has joined #openstack-nova | 08:58 | |
| *** belmoreira has quit IRC | 08:59 | |
| *** lpetrut has joined #openstack-nova | 08:59 | |
| *** arne_r has joined #openstack-nova | 08:59 | |
| *** vks1 has quit IRC | 08:59 | |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Fix wrong unit test about config option enabled_apis https://review.openstack.org/446898 | 09:02 |
| *** hshiina has joined #openstack-nova | 09:03 | |
| openstackgerrit | 龚肖 proposed openstack/nova master: Modify api-ref information https://review.openstack.org/446902 | 09:09 |
| *** gcb has joined #openstack-nova | 09:09 | |
| *** diga has joined #openstack-nova | 09:12 | |
| *** david-lyle_ has joined #openstack-nova | 09:18 | |
| *** david-lyle has quit IRC | 09:18 | |
| *** jahsis has joined #openstack-nova | 09:19 | |
| *** belmorei_ has quit IRC | 09:21 | |
| *** karimb has joined #openstack-nova | 09:24 | |
| *** yuntongjin1 has quit IRC | 09:24 | |
| *** yuntongjin has joined #openstack-nova | 09:26 | |
| *** trinaths has left #openstack-nova | 09:27 | |
| BobBall | gmann: VM_metrics is the old way of doing things and I think doesn't work correctly. The best way is to obtain metrics using the RRDs with VM.query_data_source. | 09:28 |
| *** ayogi has quit IRC | 09:28 | |
| BobBall | gmann: I think (but I'm not certain) that VM_metrics only stores info on the first vCPU | 09:29 |
| *** belmoreira has joined #openstack-nova | 09:29 | |
| BobBall | gmann: querying the RRD datasource is the most correct way of getting the details from XenAPI | 09:29 |
| BobBall | gmann: See also https://git.openstack.org/cgit/openstack/ceilometer/commit/ceilometer/compute/virt/xenapi/inspector.py?id=1374cf7d229ed0b8fca0fbdc6477703d9fff0ecc | 09:32 |
| *** Shunli has quit IRC | 09:32 | |
| gmann | BobBall: ah i see. | 09:33 |
| gmann | BobBall: query_data_source will fetch all latest stats | 09:34 |
| *** yamamoto has joined #openstack-nova | 09:35 | |
| BobBall | Indeed | 09:35 |
| gmann | BobBall: thanks, very useful information :) | 09:36 |
| BobBall | Happy to help :) | 09:36 |
| gmann | BobBall: so all info in VM_metrics is not guaranteed ? | 09:37 |
| BobBall | I would recommend against using it as it will likely be removed in future versions | 09:37 |
| *** yamamoto has quit IRC | 09:38 | |
| *** kaisers_ has joined #openstack-nova | 09:38 | |
| BobBall | If you can get information from it then it is probably right (I think) but it doesn't have some information and is likely to have less moving forward | 09:38 |
| *** yuntongjin has quit IRC | 09:38 | |
| *** yamamoto has joined #openstack-nova | 09:38 | |
| gmann | BobBall: i see. +1 to remove. it give much clarity | 09:38 |
| gmann | thanks again | 09:39 |
| *** belmoreira has quit IRC | 09:39 | |
| openstackgerrit | Michael Henkel proposed openstack/nova master: Implements: blueprint opencontrail-dpdk-support https://review.openstack.org/441183 | 09:40 |
| kashyap | lyarwood: When you're about, I saw mention of a Gate failure on detach. Is there a bug for it? [Me searches in the scrollback meanwhile] | 09:41 |
| *** ayogi has joined #openstack-nova | 09:41 | |
| *** kaisers_ has quit IRC | 09:42 | |
| *** yamamoto has quit IRC | 09:42 | |
| *** karthiks is now known as karthiks_afk | 09:42 | |
| *** wangqun has quit IRC | 09:43 | |
| *** Akhil has quit IRC | 09:43 | |
| * johnthetubaguy is not believing there was no chat after 7pm... znc what have you been up to | 09:44 | |
| *** amotoki has joined #openstack-nova | 09:48 | |
| *** belmoreira has joined #openstack-nova | 09:50 | |
| *** lpetrut has quit IRC | 09:52 | |
| *** amotoki has quit IRC | 09:53 | |
| *** saju_m has joined #openstack-nova | 09:54 | |
| saju_m | i am getting this error https://bugs.launchpad.net/nova/+bug/1669924 in nova | 09:55 |
| openstack | Launchpad bug 1669924 in OpenStack Compute (nova) "InstanceUpdateConflict: Conflict updating instance Expected: {'vm_state': ['building', None]}. Actual: {'vm_state': u'error'}" [Undecided,New] | 09:55 |
| saju_m | i am using nova master branch | 09:55 |
| saju_m | any clue ? | 09:55 |
| saju_m | because of this, nova-scheduler shows no host found error | 09:57 |
| *** mriedem has quit IRC | 09:58 | |
| *** mriedem has joined #openstack-nova | 09:58 | |
| *** mriedem has quit IRC | 10:03 | |
| *** rcernin has joined #openstack-nova | 10:04 | |
| lyarwood | I've asked this before but does anyone have a way of unwrapping comments in launchpad as you can do in bugzilla? | 10:05 |
| * lyarwood has had enough of reading fsck'd up backtraces | 10:05 | |
| *** abalutoiu has quit IRC | 10:05 | |
| *** kevinz has quit IRC | 10:08 | |
| *** nicolasbock has joined #openstack-nova | 10:11 | |
| mdbooth | lyarwood: johnthetubaguy pointed out in review that we don't have a user for bdm_get_by_attachmend_id (or whatever) | 10:13 |
| mdbooth | lyarwood: which got me looking for a user of attachment_id in the following 3 patches | 10:13 |
| mdbooth | lyarwood: What did I miss? | 10:13 |
| *** yingjun has quit IRC | 10:15 | |
| lyarwood | mdbooth: Are you asking why there isn't a user of attachment_id yet? | 10:17 |
| lyarwood | mdbooth: these patches are just moving things around to make it easier to introduce the new detach flow that will use it in the following patches | 10:17 |
| lyarwood | mdbooth: while also keeping the cinderv2 code around | 10:17 |
| mdbooth | lyarwood: I mean the detach flow refactor doesn't seem related to the attachment_id thing | 10:18 |
| lyarwood | mdbooth: it's not directly, I just wanted to land it before we started to add code that used it, thus the ordering | 10:18 |
| mdbooth | johnthetubaguy is saying don't add a db function we don't call, and mriedem is saying don't therefore add an index that's only required by that function | 10:18 |
| *** hshiina has quit IRC | 10:19 | |
| mdbooth | And I'm saying that that function and that index are the reason attachment_id exists | 10:19 |
| mdbooth | So if it's not being used, don't add it yet | 10:19 |
| lyarwood | mdbooth: not at all | 10:19 |
| lyarwood | mdbooth: that's just the lookup right | 10:19 |
| lyarwood | mdbooth: we are still going to use bdm.attachment_id | 10:19 |
| lyarwood | mdbooth: we just don't need to lookup bdms by attachment_id | 10:19 |
| lyarwood | mdbooth: we already have instance and volume lookups for that part | 10:20 |
| mdbooth | Where are we using that in the series already posted? | 10:20 |
| *** satyar has quit IRC | 10:20 | |
| lyarwood | mdbooth: using what sorry? | 10:20 |
| mdbooth | bdm.attachment_id | 10:20 |
| lyarwood | mdbooth: again, we are not but I wanted to land the refactor _before_ we started using it in the new detach flow | 10:21 |
| lyarwood | mdbooth: jgriffith posted a PoC that we were going to rebase on top of the refactor for the new detach flow | 10:21 |
| mdbooth | Right, that makes sense to me | 10:21 |
| mdbooth | So why not move the refactor to the head of the series? | 10:21 |
| mdbooth | It doesn't have to depend on the attachment_id patches | 10:21 |
| mdbooth | Now that I've noticed that there's no user for attachment_id, not even just posted for review, I don't think we should merge it | 10:22 |
| lyarwood | the db and model changes were the priority at the time so I didn't want the refactor changes getting in the way | 10:22 |
| *** jianghuaw has quit IRC | 10:22 | |
| mdbooth | I don't understand why they would be a priority if we're not writing data there, or reading from it | 10:23 |
| lyarwood | how does that matter? we know we are going to need it | 10:23 |
| mdbooth | Not really. I think, for example, that you'll discover that bdm lookup by attachment id is essential, and you'll need a second db migration | 10:24 |
| mdbooth | But we have no solid data because there's no code. | 10:25 |
| lyarwood | haha, I can't win either way | 10:25 |
| mdbooth | lyarwood: I hadn't realised there was no user in the following patches. I'd just assumed there was. | 10:26 |
| mdbooth | The patches as posted are good, imho. There's no reason I see why they can't merge quickly when they have a reason to be in the code. | 10:26 |
| mdbooth | I just think it's counter-productive to merge them now. | 10:27 |
| mdbooth | s/counter-productive/premature/ | 10:27 |
| lyarwood | yup understood | 10:27 |
| * lyarwood assumes mdbooth is writing that up into a review comment | 10:28 | |
| mdbooth | lyarwood: Yeah, I will. | 10:28 |
| lyarwood | thanks | 10:28 |
| mdbooth | lyarwood: Is there a merge conflict with the detach patches, btw? | 10:28 |
| mdbooth | If not, why not just separate them into their own independent series anyway? | 10:29 |
| lyarwood | mdbooth: yes, they are based on jgriffith's own db & object change | 10:29 |
| mdbooth | Then they can both proceed at their own pace. | 10:29 |
| lyarwood | mdbooth: ah | 10:29 |
| lyarwood | mdbooth: wait you mean my refactor patches | 10:29 |
| mdbooth | Yeah | 10:29 |
| lyarwood | mdbooth: no, not with the db and object changes | 10:29 |
| lyarwood | iirc | 10:29 |
| mdbooth | I think you could pull https://review.openstack.org/#/c/440693/ to the top of its own series | 10:30 |
| lyarwood | mdbooth: let me grab some more coffee and spin them out | 10:30 |
| lyarwood | mdbooth: sure | 10:30 |
| mdbooth | And that wouldn't compromise either those patches or the attachment_id patches in any way | 10:30 |
| *** tanee is now known as tanee_away | 10:30 | |
| *** tanee_away is now known as tanee | 10:31 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: do not include context to exception notification https://review.openstack.org/446948 | 10:31 |
| *** tanee is now known as tanee_away | 10:31 | |
| *** tanee_away is now known as tanee | 10:32 | |
| *** cdent has joined #openstack-nova | 10:38 | |
| *** kaisers_ has joined #openstack-nova | 10:38 | |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | 10:39 |
| lyarwood | mdbooth: ^ that change should be flying solo now | 10:40 |
| *** yamamoto has joined #openstack-nova | 10:41 | |
| *** kaisers_ has quit IRC | 10:43 | |
| mdbooth | lyarwood: Cool, check in a minute. | 10:43 |
| *** tuanluong has quit IRC | 10:44 | |
| *** vks1 has joined #openstack-nova | 10:45 | |
| *** rcernin has quit IRC | 10:46 | |
| *** udesale has quit IRC | 10:48 | |
| *** phuongnh has quit IRC | 10:49 | |
| *** smatzek has joined #openstack-nova | 10:52 | |
| *** yamahata has quit IRC | 10:54 | |
| *** jaosorior has quit IRC | 10:55 | |
| *** gongysh has quit IRC | 10:56 | |
| *** jaosorior has joined #openstack-nova | 10:56 | |
| *** rcarrillocruz has quit IRC | 11:00 | |
| gmann | mdbooth: johnthetubaguy kashyap : 1 query on swap volume | 11:01 |
| gmann | mdbooth: johnthetubaguy kashyap : when we swap volume with nova volume update, we loss the thin provisioning. | 11:01 |
| gmann | any way to keep the thin provisioning when libvirt rebase during swap volume | 11:02 |
| johnthetubaguy | gmann: what are you aiming to do with the swap volume? | 11:02 |
| gmann | did not find any flag on virDomainBlockRebase | 11:02 |
| gmann | johnthetubaguy: i had thin provisioned volume attached to instance and then i want to swap that volume but new volume is not thin provisioned one its all full size as used | 11:03 |
| *** karimb has quit IRC | 11:04 | |
| gmann | allocated size is all volume size not actual consumed sized | 11:04 |
| *** ygl has left #openstack-nova | 11:04 | |
| johnthetubaguy | gmann: but why did you swap to a new volume, I am just curious? migrating between volume types? | 11:05 |
| *** karimb has joined #openstack-nova | 11:06 | |
| johnthetubaguy | ( gmann: honestly, I think its the same problem we have using stroage pools for migration, it doesn't zip copying zeros, but mdbooth and kashyap will know more about that than me) | 11:06 |
| johnthetubaguy | s/zip/skip/ | 11:06 |
| gmann | johnthetubaguy: yea same issue, it copy all zero block too | 11:07 |
| gmann | johnthetubaguy: yea use case is migrating volume to new type | 11:07 |
| johnthetubaguy | gmann: cool, that makes sense | 11:07 |
| johnthetubaguy | so we have to copy everything, but copying zeros is stupid | 11:07 |
| kashyap | gmann: Sorry, occupied for a bit with something, bbiab | 11:08 |
| gmann | only thing i had is to do fstrim or blkdiscard | 11:08 |
| johnthetubaguy | sounds consistent with the current problems we have with all that | 11:08 |
| johnthetubaguy | gmann: yeah, that shouldn't be needed for sure. | 11:08 |
| gmann | yea | 11:08 |
| gmann | kashyap: sure take your time. | 11:08 |
| mdbooth | gmann: That could be tough | 11:09 |
| mdbooth | gmann: I think qemu would have to do explicit zero detection during the rebase operation | 11:10 |
| *** tbachman has quit IRC | 11:10 | |
| mdbooth | gmann: When we do this for ephemeral storage, we get sparseness from metadata | 11:11 |
| mdbooth | kashyap: Do you know qemu supports doing a block rebase with explicit zero detection? | 11:11 |
| lyarwood | gmann: which volumebackend is this with btw? | 11:12 |
| mdbooth | And if libvirt exposes that? | 11:12 |
| kashyap | mdbooth: I saw some 'detect zeroes' stuff fly by on the QEMU list. Don't recall top off my head, need some digging | 11:12 |
| *** lpetrut has joined #openstack-nova | 11:12 | |
| *** zhurong has joined #openstack-nova | 11:13 | |
| *** edmondsw has quit IRC | 11:14 | |
| gmann | mdbooth: kashyap ok. please let us know if something going to be supported or already there. it will really help | 11:16 |
| gmann | lyarwood: lvm | 11:16 |
| mdbooth | gmann: So, for ephemeral storage block migration with qcow2 we use VIR_MIGRATE_NON_SHARED_INC | 11:17 |
| mdbooth | This basically means we copy the backing file | 11:17 |
| *** crusio has quit IRC | 11:17 | |
| mdbooth | Then the live migration copies just the 'overlay' | 11:17 |
| *** crusio has joined #openstack-nova | 11:18 | |
| mdbooth | IOW qemu is relying on qcow2 metadata | 11:18 |
| gmann | k | 11:19 |
| mdbooth | For volume copy we don't have this | 11:19 |
| gmann | can it be done with VIR_DOMAIN_BLOCK_REBASE_SHALLOW when same backing file (not sure it can) ? | 11:20 |
| mdbooth | https://libvirt.org/html/libvirt-libvirt-domain.html#virDomainBlockRebaseFlags | 11:20 |
| mdbooth | gmann: There's no backing file for bolumes | 11:20 |
| mdbooth | volumes | 11:21 |
| mdbooth | gmann: It doesn't look to me like any of the above options support zero-detection during copy | 11:21 |
| gmann | humm | 11:21 |
| gmann | yea | 11:21 |
| mdbooth | gmann: I'd be moderately confident that this can't be done right now | 11:22 |
| kashyap | mdbooth: gmann: I was quickly scanning QEMU block schema | 11:22 |
| kashyap | I see a 'detect_zeroes' option that could be specificed when _configuring_ a block device | 11:23 |
| mdbooth | Incidentally, what *could* be done is this: | 11:23 |
| kashyap | And a QEMU block developer confirms: If an area is allocated nowhere in the backing chain, block-stream won't allocate it | 11:23 |
| mdbooth | 1. cinder tells nova to make an atomic local snapshot of the volume | 11:24 |
| mdbooth | 2. Nova makes atomic snapshot and starts writing to snapshot | 11:24 |
| mdbooth | 3. Cinder does fast backend-specific copy of volume from A to B | 11:24 |
| mdbooth | 4. Cinder tells Nova to rebase its snapshot on to B, which is identical to A at the point Nova snapshotted it | 11:25 |
| mdbooth | 5. Nova commits its snapshot to B | 11:25 |
| mdbooth | 6. Profit | 11:25 |
| mdbooth | That's a way different control flow, though | 11:26 |
| *** tbachman has joined #openstack-nova | 11:26 | |
| mdbooth | kashyap: That's qcow2 thought, right? | 11:27 |
| mdbooth | though | 11:28 |
| gmann | mdbooth: humm, it is on what johnthetubaguy proposal on cinder nova flow ? | 11:28 |
| kashyap | mdbooth: Yep. Still talking to Max. Let me paste my conv. snippet | 11:28 |
| mdbooth | kashyap: So this is a cinder volume: lvm exported over iscsi | 11:28 |
| mdbooth | qemu just sees a single, flat block device | 11:28 |
| johnthetubaguy | that swap volume stuff has to follow after multi-attach I feel | 11:29 |
| mdbooth | johnthetubaguy: Yep. | 11:29 |
| johnthetubaguy | cool, sounds like we are on the same page there then | 11:29 |
| kashyap | mdbooth: Noted | 11:29 |
| kashyap | So, here we go, 8 lines of paste: | 11:29 |
| kashyap | 12:21 < mreitz> Well, if an area is allocated nowhere in the backing chain, block-stream won't allocate it | 11:29 |
| kashyap | 12:21 * kashyap does notice 'detect_zeroes' flag as an option for @BlockDeviceInfo | 11:29 |
| kashyap | 12:22 < mreitz> At least from what I can tell from the code | 11:30 |
| kashyap | 12:22 < mreitz> Yeah, you can open the target BDS with detect-zeroes=on... | 11:30 |
| kashyap | 12:24 < mreitz> But as far as I can tell, copy-on-read actually always uses zero detection and converts requests accordingly | 11:30 |
| kashyap | 12:24 < mreitz> And block-stream performs its writes through the copy-on-read mechanic | 11:30 |
| kashyap | 12:24 < kashyap> mreitz: Ah, so, in that case (when the target block dev is opened with 'detect-zeroes=on'), thin provisioning will be retained? | 11:30 |
| kashyap | 12:24 < mreitz> So I'd think that block-stream should always do zero detection | 11:30 |
| * kashyap tries a little experiment | 11:31 | |
| mdbooth | kashyap: Ah, ok. So there's no detect zeroes option because it's always done. | 11:31 |
| *** hshiina has joined #openstack-nova | 11:31 | |
| mdbooth | So we'd just need to ensure we're doing it on the output? | 11:31 |
| kashyap | mdbooth: I think so. The experiment is: | 11:32 |
| mdbooth | gmann: Anyway, ^^^ is orthogonal to cinderv3. | 11:32 |
| kashyap | Build a backing chain of empty images, stream and see whether something gets allocated. | 11:32 |
| mdbooth | As johnthetubaguy says. | 11:33 |
| mdbooth | And I don't see how we could support swap_volume with multi_attach in any case. | 11:33 |
| mdbooth | kashyap: Never mind the backing chain. Create a sparse flat file with some data allocated in the middle and stream that. | 11:34 |
| mdbooth | Can you get it to create an output file which is also sparse? | 11:34 |
| kashyap | Need to try it out. | 11:34 |
| kashyap | mdbooth: You have a quick way to write zeroes to a qcow2 file? | 11:35 |
| *** ratailor has quit IRC | 11:35 | |
| mdbooth | kashyap: To be a good test, I would explicitly not use qcow2 at any point. | 11:35 |
| mdbooth | Should be flat at both source and dest. | 11:35 |
| kashyap | [Maybe dd if=/dev/zero...] | 11:35 |
| mdbooth | However, creating sparse qcow2 is trivial, right? | 11:35 |
| mdbooth | Also, libguestfs if you want to write data into it. | 11:36 |
| kashyap | mdbooth: Not use QCOW2 because, to remove its metadata complications? | 11:36 |
| mdbooth | Yeah, the interesting test is whether sparse detection works *without* qcow2 metadata. | 11:36 |
| * mdbooth is confident that it works with qcow2 metadata. | 11:36 | |
| kashyap | Oh, yes, 'guestfish'. In the past I did use it, instead of 'dd' | 11:36 |
| *** kaisers_ has joined #openstack-nova | 11:39 | |
| *** Jack_Iv has joined #openstack-nova | 11:40 | |
| *** Jack_Iv has quit IRC | 11:40 | |
| *** Jack_Iv has joined #openstack-nova | 11:40 | |
| kashyap | mdbooth: Sparse qcow2 is the default, no? (When you do `qemu-img create -f qcow2 foo.qcow2 10M`) | 11:42 |
| * kashyap bbiab; lunch | 11:42 | |
| *** kaisers_ has quit IRC | 11:44 | |
| gmann | kashyap: mdbooth seems interesting to me. but i have to go home now(its late). ll read log and catch you guys later. thanks for help btw | 11:44 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/nova master: Updated from global requirements https://review.openstack.org/446777 | 11:45 |
| *** gszasz has joined #openstack-nova | 11:48 | |
| *** amotoki has joined #openstack-nova | 11:49 | |
| *** arne_r has quit IRC | 11:53 | |
| *** amotoki has quit IRC | 11:53 | |
| *** ijw has joined #openstack-nova | 11:54 | |
| *** jichen has quit IRC | 11:54 | |
| *** ijw has quit IRC | 11:58 | |
| *** ijw_ has joined #openstack-nova | 12:00 | |
| *** jaosorior is now known as jaosorior_brb | 12:01 | |
| *** ijw_ has quit IRC | 12:04 | |
| *** aysyd has joined #openstack-nova | 12:06 | |
| *** satyar has joined #openstack-nova | 12:07 | |
| *** edmondsw has joined #openstack-nova | 12:09 | |
| *** tblakes has joined #openstack-nova | 12:09 | |
| *** vks1 has quit IRC | 12:13 | |
| *** Jack_Iv_ has joined #openstack-nova | 12:15 | |
| *** bkopilov has quit IRC | 12:16 | |
| *** Jack_Iv has quit IRC | 12:18 | |
| *** alexpilotti has quit IRC | 12:18 | |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Add json style checking for sample notifications https://review.openstack.org/443677 | 12:18 |
| *** alexpilotti has joined #openstack-nova | 12:18 | |
| openstackgerrit | Gábor Antal proposed openstack/nova master: Add json style checking for sample notifications https://review.openstack.org/443677 | 12:20 |
| *** Oku_OS is now known as Oku_OS-away | 12:24 | |
| *** Oku_OS-away is now known as Oku_OS | 12:24 | |
| *** erlon has joined #openstack-nova | 12:33 | |
| *** Jack_Iv_ has quit IRC | 12:33 | |
| *** markvoelker has joined #openstack-nova | 12:35 | |
| cdent | edleafe: not sure if this will be helpful on your functional test explorations but I was just reminded that I had this that I started before jay's big functional test he did: https://review.openstack.org/#/c/427309/ | 12:36 |
| *** fandi has joined #openstack-nova | 12:39 | |
| *** kaisers_ has joined #openstack-nova | 12:40 | |
| *** kaisers_ has quit IRC | 12:44 | |
| *** felipemonteiro_ has joined #openstack-nova | 12:48 | |
| *** jaypipes is now known as leakypipes | 12:49 | |
| *** sree_ has joined #openstack-nova | 12:52 | |
| *** sree_ is now known as Guest52417 | 12:52 | |
| *** Jack_Iv has joined #openstack-nova | 12:52 | |
| *** sree has quit IRC | 12:54 | |
| *** mriedem has joined #openstack-nova | 12:55 | |
| edleafe | kashyap: are you around for some questions? | 12:55 |
| *** yamamoto has quit IRC | 12:55 | |
| dims | cdent : could use a "## Need Volunteers/help" section :) thanks for the updates! | 12:55 |
| *** sree has joined #openstack-nova | 12:55 | |
| *** yamamoto has joined #openstack-nova | 12:56 | |
| edleafe | cdent: my problem was the driver mocks. Started on it, but on a client call now | 12:56 |
| openstackgerrit | ChangBo Guo(gcb) proposed openstack/nova master: Remove old oslo.messaging transport aliases https://review.openstack.org/446990 | 12:56 |
| kashyap | edleafe: Hiya, yes. I can try :-) | 12:56 |
| cdent | dims: that's a good idea but I would be half inclined to put that as the top level heading and indent everything else :) | 12:56 |
| dims | :) | 12:57 |
| *** Guest52417 has quit IRC | 12:57 | |
| *** awaugama has joined #openstack-nova | 12:58 | |
| *** gcb has quit IRC | 12:58 | |
| edleafe | kashyap: it's the same problem that mriedem was talking to you about - volume migration failures | 12:58 |
| *** litao has quit IRC | 12:59 | |
| edleafe | kashyap: on a call with the client now - give me a few | 12:59 |
| kashyap | edleafe: Sure. Just to note: I wasn't following the entire issue from Nova POV, mdbooth is more in the trenches there. I was mostly following it from libvirt / QEMU perspective | 13:00 |
| *** sree has quit IRC | 13:01 | |
| *** sree has joined #openstack-nova | 13:02 | |
| *** psachin has quit IRC | 13:03 | |
| *** baoli has quit IRC | 13:03 | |
| edleafe | kashyap: well, there are two different errors: one from libvirt, and one from QEMU :) | 13:03 |
| *** kevinz has joined #openstack-nova | 13:03 | |
| kashyap | edleafe: One of them is the memory corruption issue? | 13:04 |
| kashyap | (The one from libvirt) | 13:04 |
| kashyap | And do you also have a log for the other one from QEMU? | 13:06 |
| *** sree has quit IRC | 13:06 | |
| edleafe | kashyap: I can get them | 13:06 |
| mriedem | kashyap: edleafe isn't talking about the thing from yesterday | 13:06 |
| mriedem | he's talking about volume swap | 13:06 |
| *** sree has joined #openstack-nova | 13:06 | |
| edleafe | kashyap: the error is libvirtError: Requested operation is not valid: pivot of disk 'vdb' requires an active copy job | 13:06 |
| kashyap | mriedem: Ah, a different issue | 13:06 |
| edleafe | This is on a newer libvirt, using Mitaka with Matt's patches | 13:07 |
| mdbooth | edleafe: That's an old one | 13:07 |
| edleafe | mdbooth: I saw the bugzilla reports | 13:07 |
| edleafe | mdbooth: their libvirt should have the fixes for that | 13:07 |
| kashyap | mriedem: Meanwhile, for the Gate issue mdbooth was looking yesterday, please see the comment here when you get a moment -- https://bugs.launchpad.net/nova/+bug/1673483/comments/3 | 13:08 |
| openstack | Launchpad bug 1673483 in OpenStack Compute (nova) "libvirt: test_attach_volume_shelved_or_offload_server times out waiting for device detach (which fails)" [Undecided,Confirmed] | 13:08 |
| mdbooth | edleafe: The latest Nova should work with old libvirt on that one | 13:08 |
| *** mdrabe has joined #openstack-nova | 13:08 | |
| kashyap | mriedem: I also notified folks on #ubuntu-packaging IRC on Freenode -- so they can pull in the missing patch | 13:08 |
| edleafe | mriedem: so would your patches make their code "the lates Nova"? | 13:08 |
| edleafe | at least as far as this issue is concerned | 13:08 |
| mdbooth | kashyap: Ubuntu have engineers? I thought they just had marketing. | 13:09 |
| kashyap | mdbooth: :-) C'mon, don't be so mean | 13:09 |
| * mdbooth is yet to meet one | 13:09 | |
| *** karimb has quit IRC | 13:09 | |
| kashyap | Credit where it's due: Serge Hallyn is a real machine | 13:09 |
| mriedem | edleafe: i had backported to mitaka the fix i thought was related to this pivot issue that lyarwood worked on | 13:09 |
| *** catintheroof has joined #openstack-nova | 13:10 | |
| kashyap | mdbooth: He does most of the work. And once when I met him at a conference, and asked him how on earth is he handling it all | 13:10 |
| edleafe | mriedem: yeah, that's what I thought | 13:10 |
| *** catintheroof has quit IRC | 13:10 | |
| kashyap | He said: "Yeah, my SO complains..." | 13:10 |
| mriedem | edleafe: this https://review.openstack.org/#/c/433793/ | 13:10 |
| *** catintheroof has joined #openstack-nova | 13:10 | |
| *** sree has quit IRC | 13:10 | |
| kashyap | But that's just one person. I admittedly don't know any others | 13:11 |
| kashyap | mriedem: Why did you abandon it? | 13:11 |
| *** kevinz has quit IRC | 13:11 | |
| *** kevinz has joined #openstack-nova | 13:12 | |
| *** jpena is now known as jpena|lunch | 13:12 | |
| mdbooth | mriedem edleafe: Landed in Newton: | 13:13 |
| mdbooth | https://review.openstack.org/#/c/427290/ | 13:13 |
| * mdbooth thought there was a Mitaka fix, but maybe that was downstream only | 13:13 | |
| *** esberglu has joined #openstack-nova | 13:13 | |
| *** SzarkaAlex has joined #openstack-nova | 13:14 | |
| *** mat128 has quit IRC | 13:14 | |
| mriedem | kashyap: mitaka | 13:15 |
| mriedem | kashyap: it's not justified for mitaka at this point | 13:15 |
| kashyap | Ah! | 13:15 |
| kashyap | For a moment, lost track of release status | 13:15 |
| kashyap | Says, it's "Legacy Release" phase https://releases.openstack.org/ | 13:16 |
| kashyap | edleafe: That fix mriedem pointed out is what you're looking for. | 13:16 |
| edleafe | kashyap: just checked their system, and they have those changes applied | 13:17 |
| kashyap | edleafe: And they still see the same pivot failure? | 13:17 |
| *** karimb has joined #openstack-nova | 13:18 | |
| SzarkaAlex | @gibi Hi! Are you here? | 13:18 |
| *** bkopilov has joined #openstack-nova | 13:18 | |
| mdbooth | edleafe: Can you do me a favour. There's an edge case in that code that we knowingly glossed over, namely that we can't detect if the job fails. | 13:20 |
| *** ociuhandu has quit IRC | 13:20 | |
| mdbooth | edleafe: Can you correlate the Nova log with the libvirt logs and look for a failure of the rebase job? | 13:21 |
| edleafe | mdbooth: sec... | 13:21 |
| edleafe | mdbooth: like this? http://paste.openstack.org/show/603131/ | 13:23 |
| *** ayogi has quit IRC | 13:23 | |
| mdbooth | edleafe: That's the nova log. | 13:23 |
| edleafe | ah | 13:23 |
| mdbooth | Can you look in the libvirt log? | 13:23 |
| *** gouthamr has joined #openstack-nova | 13:23 | |
| mdbooth | /var/log/libvirt/whatever probably | 13:23 |
| edleafe | looking... | 13:24 |
| mdbooth | kashyap: 'requires an active copy job' | 13:24 |
| mdbooth | Is that the same as the pivot-too-early problem? | 13:25 |
| kashyap | mdbooth: Oh, I think vaguely recall, I think that edge case you discussed in one of the reviews... | 13:25 |
| *** liangy has joined #openstack-nova | 13:25 | |
| kashyap | mdbooth: Yes, it's the same problem | 13:25 |
| kashyap | Wait, let me think again | 13:26 |
| kashyap | mdbooth: No, I was wrong: | 13:26 |
| mdbooth | _("disk '%s' not ready for pivot yet"), | 13:26 |
| kashyap | Pivot too early implies that there _is_ an active block operation in progress | 13:26 |
| mdbooth | if (disk->mirrorState != VIR_DOMAIN_DISK_MIRROR_STATE_READY) { | 13:26 |
| mdbooth | virReportError(VIR_ERR_BLOCK_COPY_ACTIVE, | 13:26 |
| mdbooth | _("disk '%s' not ready for pivot yet"), | 13:26 |
| mdbooth | disk->dst); | 13:26 |
| mdbooth | vs: | 13:26 |
| mdbooth | if (!disk->mirror) { | 13:27 |
| mdbooth | virReportError(VIR_ERR_OPERATION_INVALID, | 13:27 |
| mdbooth | _("pivot of disk '%s' requires an active copy job"), | 13:27 |
| mdbooth | I think there's no copy job at all | 13:27 |
| kashyap | Requires an active job means: there's no block operation at all | 13:27 |
| mdbooth | edleafe: Yeah, those libvirt logs would be really interesting | 13:27 |
| kashyap | s/no/no _live_/ | 13:27 |
| *** ratailor has joined #openstack-nova | 13:28 | |
| mdbooth | Incidentally, if the issue is that the job failed I think we *have* to switch to consuming events from libvirt. | 13:29 |
| mdbooth | I think we're out of workarounds | 13:29 |
| kashyap | edleafe: Those logs would be useful, assuming you had log filters enabled in /etc/libvirt/libvirtd.conf: | 13:29 |
| kashyap | log_filters="1:libvirt 1:qemu 1:conf 1:security 3:event 3:json 3:file 3:object 1:util" | 13:29 |
| kashyap | log_outputs="1:file:/var/log/libvirt/libvirtd.log" | 13:29 |
| *** amotoki has joined #openstack-nova | 13:29 | |
| *** rfolco has joined #openstack-nova | 13:29 | |
| kashyap | (Restart libvirtd; repeat your test; _then_ post the log.) | 13:30 |
| *** dave-mccowan has joined #openstack-nova | 13:30 | |
| mdbooth | kashyap: Actually looking at that code in qemuDomainBlockPivot(), I think the Nova code is wrong when it does: | 13:31 |
| mdbooth | if status is None: | 13:31 |
| mdbooth | return True | 13:31 |
| mdbooth | Looks like it should raise an exception at that point, as it will never succeed. | 13:31 |
| edleafe | mdbooth: I see the log; it's 3GB, but its last entry is about 8 hours ago?!? | 13:31 |
| mdbooth | edleafe: Did it get rolled somewhere? | 13:32 |
| kashyap | Oh dear | 13:32 |
| edleafe | mdbooth: kashyap: let me restart libvirt, verify that it's logging, repeat the test, and then I'll post the logs and re-ping you | 13:32 |
| kashyap | edleafe: If possible to repeat the test | 13:32 |
| *** smatzek has quit IRC | 13:32 | |
| kashyap | First, please take a backup of that 3G log, _if_ you need it, and then empty it out ($ > /var/log/libvirt/libvirt.log), so we can start from a clean slate. | 13:33 |
| edleafe | mdbooth: it has a current timestamp, but last entry is 8 hours ago | 13:33 |
| kashyap | Then, enable the log filters, repeat the test, etc. | 13:33 |
| edleafe | kashyap: I was going to roll over the log, eyah | 13:33 |
| edleafe | yeah | 13:33 |
| *** jianghuaw__ has quit IRC | 13:33 | |
| *** baoli has joined #openstack-nova | 13:33 | |
| *** amotoki has quit IRC | 13:33 | |
| edleafe | but first another pot of coffee is a higher priority :) | 13:34 |
| *** jianghuaw__ has joined #openstack-nova | 13:34 | |
| kashyap | Go for it | 13:34 |
| *** bkopilov has quit IRC | 13:34 | |
| kashyap | mdbooth: Sigh, it's like a landmine. Not sure how your eyes don't bleed looking at that | 13:34 |
| *** diga has quit IRC | 13:38 | |
| *** eharney has joined #openstack-nova | 13:38 | |
| *** weshay_pto is now known as weshay_sick | 13:40 | |
| *** kaisers_ has joined #openstack-nova | 13:41 | |
| kashyap | mdbooth: And yeah, didn't miss your quote above, from qemuDomainBlockPivot() function, it is looking for the libvirt event READY to decide whether the job is ready for pivot or not... | 13:41 |
| mdbooth | kashyap: But before that it'll raise a different error if the job doesn't exist | 13:42 |
| mdbooth | Specifically the error that edleafe is seeing | 13:42 |
| kashyap | mdbooth: You're referring to: | 13:42 |
| kashyap | if (!disk->mirror) { | 13:42 |
| kashyap | virReportError(VIR_ERR_OPERATION_INVALID, | 13:42 |
| kashyap | _("pivot of disk '%s' requires an active copy job"), | 13:42 |
| kashyap | disk->dst); | 13:42 |
| mdbooth | Yes | 13:43 |
| kashyap | ? | 13:43 |
| openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | 13:43 |
| *** ratailor has quit IRC | 13:43 | |
| *** andreas_s has quit IRC | 13:43 | |
| kashyap | mdbooth: In that case, either the job was already finished (maybe not gracefully); or it was not started. Hence, it whines "no block operation for me work on" | 13:44 |
| kashyap | Let's wait for edleafe's logs (hopefully small, with that specific test) | 13:45 |
| mdbooth | Either way, based on that code I belive the pivot will fail | 13:45 |
| *** kaisers_ has quit IRC | 13:45 | |
| *** aysyd has quit IRC | 13:46 | |
| *** SzarkaAlex has quit IRC | 13:47 | |
| *** bkopilov has joined #openstack-nova | 13:48 | |
| *** ociuhandu has joined #openstack-nova | 13:50 | |
| *** jaosorior_brb is now known as jaosorior | 13:50 | |
| *** yingjun has joined #openstack-nova | 13:52 | |
| *** ratailor has joined #openstack-nova | 13:53 | |
| *** crushil has joined #openstack-nova | 13:53 | |
| *** ijw has joined #openstack-nova | 13:55 | |
| *** ijw has quit IRC | 13:55 | |
| *** zhurong has quit IRC | 13:56 | |
| *** smatzek has joined #openstack-nova | 13:57 | |
| *** jianghuaw__ has quit IRC | 14:03 | |
| *** tbachman has quit IRC | 14:05 | |
| *** gongysh has joined #openstack-nova | 14:06 | |
| *** hongbin has joined #openstack-nova | 14:08 | |
| *** jpena|lunch is now known as jpena | 14:08 | |
| *** brault_ is now known as brault|away | 14:09 | |
| openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | 14:10 |
| leakypipes | mhenkel: :) | 14:11 |
| kashyap | mdbooth: So, if you want to hear the results (credit: Max from QEMU did the tests & reported) of earlier experiment with qcow2, I can write here. | 14:11 |
| mhenkel | hey leakypipes | 14:11 |
| mdbooth | kashyap: Shoot | 14:11 |
| kashyap | mdbooth: When I asked about the format, he says it doesn't really matter: Detection whether the source contains zeroes -- That is independent of the format, the code actually checks the data read from the source whether it is zero. | 14:11 |
| mdbooth | s/with/without/ | 14:11 |
| *** tbachman has joined #openstack-nova | 14:11 | |
| kashyap | mdbooth: Okay, here we go: | 14:12 |
| *** tbachman has quit IRC | 14:12 | |
| mhenkel | leakypipes: is the commit msg looking ok now? | 14:12 |
| kashyap | The experiment being: build a backing chain, of depth one, of empty images, perform 'block-stream' and see whether something gets allocated. Three cases: | 14:12 |
| kashyap | (1) With empty images nothing is allocated at all; | 14:12 |
| kashyap | (2) With zeros (using 'qemu-io -c 'write -P 0 0 64M' base.qcow2') written to the base, the qcow2 file for some reason (there's one, but not sure what) grows by a cluster (64 kB). | 14:13 |
| kashyap | (3) With non-zero data (using 'qemu-io -c 'write -P 42 0 64M' base.qcow2'), all data is written to the overlay. | 14:13 |
| kashyap | * * * | 14:13 |
| openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | 14:13 |
| dansmith | so mriedem, we supposedly have tests for this ^ and they pass, but the two cases they check don't work in real life | 14:14 |
| kashyap | mdbooth: Max tells (& I can confirm by testing as I have the QMP env. ready) running the same tests on a raw image should yield the same results. | 14:14 |
| dansmith | mriedem: thoughts on how to test that change? | 14:14 |
| openstackgerrit | Roman Dobosz proposed openstack/nova-specs master: Provide detailed error information for placement API https://review.openstack.org/418393 | 14:14 |
| mdbooth | kashyap: Is this dependent on how the target disk is initially created? | 14:14 |
| kashyap | mdbooth: Hmm, that is a good question. The target image in this specific test is a simple qcow2 overlay | 14:15 |
| mdbooth | So, in the case of volume migration the source and destination are both block devices from the pov of qemu | 14:16 |
| edmondsw | mriedem, what is the plan/status on setting send_service_user_token conf setting to True by default (i.e. no longer experimental)? | 14:16 |
| *** brault|away is now known as brault | 14:16 | |
| mriedem | dansmith: ooo i smell a customer bug | 14:16 |
| mriedem | dansmith: tests where? novaclient? | 14:16 |
| mdbooth | kashyap: I'd be interested to see the result if you create 2 sparse files with loopback devices, and copy between the loopback devices | 14:17 |
| dansmith | mriedem: hang on, linking | 14:17 |
| mdbooth | Is the result sparse? | 14:17 |
| mriedem | edmondsw: we need some scale testing first, | 14:17 |
| kashyap | Always asks tricky questions on a Friday, mdbooth... | 14:17 |
| mdbooth | kashyap: Hopefully it's just a little reconfiguration of the test you've already got | 14:17 |
| mdbooth | I think you've already done the hard work :) | 14:18 |
| kashyap | mdbooth: I will let you know, once I come back from my bike ride. My eyes are aching | 14:18 |
| mriedem | edmondsw: raj_sing- is the lead on that stuff i believe. we have service user tokens enabled in the nova-next job, but as part of the original spec we had talked about the need for some scale testing to stress it to force the usage of the service user tokens, and to stress keystone since it has to do double duty token-wise per request now. | 14:18 |
| dansmith | mriedem: added a comment with links | 14:18 |
| mriedem | edmondsw: https://specs.openstack.org/openstack/nova-specs/specs/ocata/approved/use-service-tokens.html#performance-impact | 14:18 |
| mriedem | "Rally benchmark tests will be ran with and without the “service_user” config options set to compare the results for long running tasks like snapshot or live migration." | 14:18 |
| mriedem | edmondsw: if you have some people familiar with scale testing and rally, it would be helpful to pitch in on the testing there | 14:19 |
| mriedem | i'm not sure if raj_sing- or the osic people are focusing on that right now | 14:19 |
| lbragstad | aunnam around? | 14:19 |
| mriedem | dansmith: i wouldn't trust the unit tests at all, | 14:20 |
| *** ijw has joined #openstack-nova | 14:20 | |
| mriedem | dansmith: i'd probably just write a functional test in the novaclient tree that does what you need for how this is supposed to work | 14:20 |
| edmondsw | mriedem that all makes sense. We don't use rally but we do some scale testing. I can ask about trying this there | 14:20 |
| dansmith | mriedem: clearly not, the same tests pass before and after this | 14:20 |
| mriedem | see it break on the bug, and then show it passing with the fix | 14:20 |
| kashyap | mdbooth: Quick clarification, before I head out for a few: IIUC, you want the loopback device to be the overlay, yes? | 14:20 |
| mriedem | dansmith: like what i've been doing with the functional regression tests in nova lately | 14:20 |
| mdbooth | kashyap: I want both source and destination to be loopback devices backed by sparse files | 14:21 |
| mdbooth | kashyap: They should both be the same size | 14:21 |
| mdbooth | kashyap: The source should contain some data, but the dest should initially be blank | 14:21 |
| mriedem | edmondsw: cool. i'm thinking a long-running volume-backed migration where the token will expire and have to be revalidated for cinder and neutron, | 14:21 |
| dansmith | mriedem: okay I forgot about the functional ones in here.. looks like we don't have any for aggregates | 14:21 |
| mriedem | edmondsw: or a long-running snapshot of a large instance (note you'll need the unmerged glance patches for that) | 14:21 |
| *** saju_m has quit IRC | 14:22 | |
| *** prateek has quit IRC | 14:22 | |
| mriedem | dansmith: at first glance i'm not seeing anyway | 14:22 |
| mriedem | but they are pretty simple to add | 14:22 |
| mriedem | like https://github.com/openstack/python-novaclient/blob/master/novaclient/tests/functional/v2/legacy/test_extended_attributes.py | 14:22 |
| dansmith | yeah, I'll work on it | 14:22 |
| mriedem | specify the base 2.1 version, | 14:23 |
| edleafe | kashyap: mdbooth: still trying to get the repro for the failure. Didn't give up (yet) :) | 14:23 |
| kashyap | mdbooth: Okay, noted. Just to restate: The backing file(s) must be sparse; and the overlays are loopback devices -- then try a copy among the overlays and see if sparseness is preserved | 14:23 |
| mriedem | extend base class, and that gives you most of what you need | 14:23 |
| mriedem | self.nova() runs the command | 14:23 |
| mdbooth | kashyap: Bingo | 14:23 |
| kashyap | edleafe: Don't give up! One simply does not give up in #openstack-nova channel! | 14:23 |
| mdbooth | kashyap: I think that accurately represents the issue as described | 14:23 |
| kashyap | Okay, with that clarity, bbiab | 14:23 |
| edleafe | kashyap: heh, I figured it was taking me so long you might have thought that I just went back to bed or something | 14:24 |
| ijw | The Nova team has beds now? | 14:24 |
| ijw | Luxury. | 14:24 |
| edleafe | Indeed! | 14:24 |
| mriedem | i have a dugout in the dirt | 14:24 |
| mriedem | it serves my needs | 14:24 |
| markus_z | mriedem: AFAIK I provided all information for your concerns on https://review.openstack.org/#/c/406898/ Can I do anything else there? | 14:27 |
| *** mlavalle has joined #openstack-nova | 14:30 | |
| mriedem | markus_z: i've removed the -1 | 14:30 |
| markus_z | thanks | 14:30 |
| *** zz_dimtruck is now known as dimtruck | 14:31 | |
| mriedem | markus_z: so basically there are some things backported, like the console refactor patches, which were not functionally needed, but made the target bug fix backport easier | 14:31 |
| *** xinliang has quit IRC | 14:32 | |
| mriedem | and all of those groundwork backports are already merged by now anyway | 14:32 |
| markus_z | mriedem: The refactor patches were not backported as they weren't functionally needed. That's why there are conflicts. | 14:33 |
| *** vks1 has joined #openstack-nova | 14:33 | |
| markus_z | *if* we would have backported the "groundwork" patches, then there wouldn't have been conflicts. | 14:34 |
| mriedem | sure, but that refactor series also introduced a regression in ocata, | 14:34 |
| mriedem | the one about xen and libvirtd | 14:34 |
| mriedem | *virtlogd | 14:34 |
| *** amoralej is now known as amoralej|lunch | 14:34 | |
| markus_z | ah, damn, you're right | 14:35 |
| mriedem | right, so we don't want those backports | 14:35 |
| mriedem | for the refactor patches | 14:35 |
| mriedem | or did we backport that to newton? | 14:35 |
| markus_z | sec, I have to check the fix again, need a minute | 14:35 |
| mriedem | looking at the topic branch for newton i don't think we did https://review.openstack.org/#/q/topic:%22bug+1455252%22 | 14:35 |
| mriedem | i think that was https://review.openstack.org/#/c/385367/ | 14:36 |
| mriedem | hmm maybe not | 14:36 |
| markus_z | That was your fix: https://review.openstack.org/#/c/442550/ | 14:38 |
| *** gongysh has quit IRC | 14:39 | |
| markus_z | I think it was this one: https://review.openstack.org/#/c/407450/3/nova/virt/libvirt/driver.py | 14:40 |
| markus_z | yep, pretty sure. That's where I screwed up the log_path. | 14:41 |
| mriedem | yup | 14:41 |
| markus_z | That's not backported. | 14:41 |
| mriedem | ok | 14:41 |
| *** kaisers_ has joined #openstack-nova | 14:41 | |
| *** rmart04 has joined #openstack-nova | 14:42 | |
| raj_sing- | mriedem: edmondsw Performance testing with service token enabled is part of our plan for Pike. I am not sure yet how to do that, but will focus once glance patch land. Currently we are debugging some errors on that patch. | 14:43 |
| *** tbachman has joined #openstack-nova | 14:43 | |
| mriedem | markus_z: +W my german friend | 14:43 |
| mriedem | sorry for the delay | 14:43 |
| edmondsw | raj_sing- great, tx | 14:43 |
| *** yamamoto has quit IRC | 14:43 | |
| *** xinliang has joined #openstack-nova | 14:44 | |
| mriedem | mtreinish: sdague: can you take a look at these ocata backports? fixes the retry regression: https://review.openstack.org/#/q/topic:bug/1671648+branch:stable/ocata | 14:44 |
| markus_z | mriedem: Thanks a lot my american hero ;P | 14:45 |
| gibi | mriedem, mdbooth: I looked into the circular reference bug you found yesterday. Here is the easiest way to solve it https://review.openstack.org/#/c/446948/ | 14:45 |
| *** yamamoto has joined #openstack-nova | 14:45 | |
| sdague | mriedem: mtreinish is in singapore, so it's sat already | 14:45 |
| mriedem | mtreinish: what was the anime new york hero guy again? | 14:45 |
| mriedem | damn | 14:45 |
| mriedem | bauzas: can you take a pass through these backports? https://review.openstack.org/#/q/topic:bug/1671648+branch:stable/ocata | 14:45 |
| *** kevinz has quit IRC | 14:46 | |
| markus_z | frickler: Got a second about https://review.openstack.org/#/c/444164/3 ? That's the VNC setup patch. | 14:46 |
| *** kaisers_ has quit IRC | 14:47 | |
| Dinesh_Bhor | Hi all, Can someone take a look at these two small patches: https://review.openstack.org/#/c/445338/ , https://review.openstack.org/#/c/445782/ Both are having one +2. | 14:49 |
| *** raj_sing- is now known as raj_singh | 14:49 | |
| *** yamamoto has quit IRC | 14:50 | |
| *** dikonoor has quit IRC | 14:50 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 14:52 |
| sdague | mriedem / melwitt how about that wording ^^^^ | 14:53 |
| *** fandi has quit IRC | 14:53 | |
| mriedem | sdague: typos | 14:55 |
| mriedem | plus other unaddressed stuff from PS1 | 14:55 |
| mriedem | otherwise the new wording is good, and the example of why it's useful | 14:55 |
| sdague | mriedem: I'll fix the typos, I think the adminstrator only adds are just consistent with other fields that have it | 14:55 |
| sdague | so are fine | 14:55 |
| mriedem | i'm pretty sure nothing else in the os-hypervisors api ref says "this field is only for admins" | 14:56 |
| *** Dinesh_Bhor has quit IRC | 14:56 | |
| mriedem | https://developer.openstack.org/api-ref/compute/?expanded=show-hypervisor-details-detail#show-hypervisor-details | 14:56 |
| mriedem | so if we say that for hypervisor_hostname in that API ref, then say it for all, | 14:56 |
| mriedem | or none | 14:56 |
| mriedem | if you're going to chew gum in class, you have to bring enough for everyone | 14:57 |
| *** stvnoyes has joined #openstack-nova | 14:57 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 14:57 |
| sdague | damn, the inline editor doesn't let you make 2 edits as once | 14:58 |
| *** marst has quit IRC | 14:58 | |
| *** nkrinner is now known as nkrinner_afk | 14:58 | |
| openstackgerrit | Sean Dague proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 14:59 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 14:59 |
| mriedem | jesus | 14:59 |
| mriedem | at some point you just have to pull it down sean :() | 14:59 |
| sdague | because changesets are expensive? | 15:00 |
| openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | 15:00 |
| mriedem | sdague: because it keeps getting messed up | 15:01 |
| mriedem | you fix one thing but not another, and then vice-versa | 15:01 |
| sdague | mriedem: ok, oh is it reverting the old ones? | 15:01 |
| sdague | gah, ok | 15:01 |
| mriedem | i guess? | 15:01 |
| mriedem | idk man, just pull it down | 15:01 |
| mriedem | i'm enjoying my -1 stat padding though | 15:01 |
| sdague | heh | 15:01 |
| sdague | this is one of those areas where github is *so* much better than gerrit | 15:02 |
| *** markus_z has quit IRC | 15:02 | |
| *** tbachman has quit IRC | 15:02 | |
| *** sree has joined #openstack-nova | 15:03 | |
| jgriffith | mriedem johnthetubaguy hey... just want to check with the two of you. For the client.get_highest_microversion call; would you prefer that we ONLY call that when a call for a method that requires/wants the version info is requested? | 15:03 |
| *** sbezverk has quit IRC | 15:03 | |
| *** marst has joined #openstack-nova | 15:04 | |
| jgriffith | johnthetubaguy I think that might be the point you were trying to make in the review comments, so rather than ALWAYS do it on init, provide an explicit method to query that info, and then also use that mechanism when calling things like the new attach/detach methods | 15:04 |
| openstackgerrit | Sean Dague proposed openstack/nova master: Improve descriptions for hostId, host, and hypervisor_hostname https://review.openstack.org/446761 | 15:04 |
| *** marst has quit IRC | 15:05 | |
| *** marst has joined #openstack-nova | 15:05 | |
| leakypipes | mhenkel: yup! | 15:05 |
| johnthetubaguy | jgriffith: so if you get your detach patch, and try use the logic, I think that will clear things up. That needs 3.27 or it fails with "the world is upside down" error. Only attach needs the explcit call to check if 3.27 is supported and chose the old or new path. | 15:05 |
| melwitt | that api-ref wording is a hot topic I see! | 15:06 |
| mriedem | +2 | 15:06 |
| mriedem | finally | 15:06 |
| mriedem | welcome new contributor | 15:06 |
| johnthetubaguy | melwitt: looking at what dansmith has done to your quota patches, I guess you are working through that? | 15:06 |
| jgriffith | johnthetubaguy ok | 15:06 |
| jgriffith | johnthetubaguy thanks | 15:06 |
| dansmith | johnthetubaguy: yes, I made a mess, but I knew it was going to be | 15:07 |
| *** tbachman has joined #openstack-nova | 15:07 | |
| melwitt | johnthetubaguy: yes, sorry about that. I'm going to finish fixing those up today. we split them into smaller chunks for easier review | 15:07 |
| dansmith | melwitt: I can help if you want, but probably not very parallelizeable work | 15:07 |
| melwitt | johnthetubaguy: on the one you reviewed today, that "destroy all by project" is just a plain destroy. that patch is just making Quotas object read/write to API DB | 15:07 |
| melwitt | dansmith: nah, it's okay, I have got it | 15:08 |
| *** NikhilS has quit IRC | 15:08 | |
| melwitt | johnthetubaguy: I think what you were thinking of was an older version of the quota counting patch where I was using a destroy with limit/offset for addressing races | 15:09 |
| melwitt | I took that out | 15:09 |
| *** mriedem is now known as mriedem_afk | 15:10 | |
| *** dansmith is now known as superdan | 15:10 | |
| *** armax has joined #openstack-nova | 15:12 | |
| openstackgerrit | Silvan Kaiser proposed openstack/nova master: Removes getfattr from Quobyte Nova driver https://review.openstack.org/428646 | 15:12 |
| melwitt | johnthetubaguy: nevermind my earlier statement. I think I don't understand what you mean by "Its the delete something and the sync logic brings it back to life problem." | 15:13 |
| *** amoralej|lunch is now known as amoralej | 15:13 | |
| melwitt | everybody was +2 on this patch PS8. all I did was update a few code comments based on leakypipes feedback | 15:14 |
| *** dave-mccowan has quit IRC | 15:15 | |
| leakypipes | melwitt: the Quota object patch you mean? | 15:15 |
| *** READ10 has joined #openstack-nova | 15:16 | |
| melwitt | leakypipes: yeah, https://review.openstack.org/#/c/410945/ | 15:16 |
| edleafe | kashyap: mdbooth: Sorry, wasn't able to grab the logs from the client machine - it's behind several layers of firewalls, so I'll need someone from their location to send them to me | 15:16 |
| edmondsw | mriedem fyi just marked https://bugs.launchpad.net/nova/+bug/1656056 as a dup of https://bugs.launchpad.net/nova/+bug/1657774 where the fix is already proposed | 15:16 |
| openstack | Launchpad bug 1657774 in OpenStack Compute (nova) "duplicate for #1656056 Nova does not re-raise 401 Unauthorized received from Neutron for admin users" [Undecided,Confirmed] - Assigned to Roman Podoliaka (rpodolyaka) | 15:16 |
| openstack | Launchpad bug 1657774 in OpenStack Compute (nova) "Nova does not re-raise 401 Unauthorized received from Neutron for admin users" [Undecided,Confirmed] - Assigned to Roman Podoliaka (rpodolyaka) | 15:16 |
| edleafe | They're in Japan, so it won't be until Monday | 15:16 |
| edmondsw | mriedem you had marked the original high, so you probably want to do the same on the other now that the original is a dup | 15:16 |
| edmondsw | fix proposed lgtm | 15:17 |
| melwitt | leakypipes: I think I addressed all your comments, the one you wanted most was the single db transaction thing and I posted a comment that when I tried that, I got a "transaction already started" because I think the decorator already starts and maintains a transaction? | 15:17 |
| kashyap | edleafe: No worries. When you have it, can take a look. But please don't forget to enable the log filters I noted in the scrollback | 15:17 |
| leakypipes | melwitt: right, I think I had re-commented "ok, yes, makes sense" on that (or similar) :) | 15:18 |
| melwitt | leakypipes: oh, hm, lemme check again | 15:18 |
| edmondsw | mriedem_afk ^ I was tagging mriedem... | 15:18 |
| melwitt | leakypipes: you didn't reply on that, the last reply I see from you was about the lack of a parent-child relationship of the tables | 15:19 |
| leakypipes | doh, melwitt it was a draft comment never submitted... fuug | 15:19 |
| *** Jack_Iv has quit IRC | 15:20 | |
| edleafe | kashyap: yes, I checked, and they were exactly as you said | 15:20 |
| edleafe | oops, my manners | 15:21 |
| *** edleafe is now known as figleaf | 15:21 | |
| *** Jack_Iv has joined #openstack-nova | 15:21 | |
| * kashyap nods | 15:22 | |
| melwitt | leakypipes: a-ha! | 15:22 |
| johnthetubaguy | melwitt: yeah, I was trying to remember what I mean in there too | 15:25 |
| melwitt | heh :) | 15:25 |
| johnthetubaguy | melwitt: I think it was fine before, just trying to remember what I was worried before | 15:26 |
| *** dave-mccowan has joined #openstack-nova | 15:28 | |
| *** jwcroppe has quit IRC | 15:28 | |
| *** jwcroppe has joined #openstack-nova | 15:29 | |
| *** yingjun has quit IRC | 15:31 | |
| *** armax has quit IRC | 15:32 | |
| *** jwcroppe has quit IRC | 15:33 | |
| *** Apoorva has joined #openstack-nova | 15:34 | |
| *** felipemonteiro_ has quit IRC | 15:35 | |
| *** kenperkins has joined #openstack-nova | 15:36 | |
| *** ijw has quit IRC | 15:37 | |
| *** ijw has joined #openstack-nova | 15:39 | |
| *** ijw_ has joined #openstack-nova | 15:41 | |
| *** ijw_ has quit IRC | 15:42 | |
| openstackgerrit | Thomas Stewart proposed openstack/nova master: Add lan9118 as valid nic for hw_vif_model property for qemu https://review.openstack.org/393489 | 15:42 |
| *** mriedem_afk is now known as mriedem | 15:42 | |
| *** ijw_ has joined #openstack-nova | 15:42 | |
| *** ijw has quit IRC | 15:43 | |
| *** jwcroppe has joined #openstack-nova | 15:43 | |
| *** crushil has quit IRC | 15:45 | |
| *** jsheeren has joined #openstack-nova | 15:45 | |
| jsheeren | hi all | 15:45 |
| jsheeren | i'm seeing this in my nova-compute log since today: INFO oslo.messaging._drivers.impl_rabbit [-] A recoverable connection/channel error occurred, trying to reconnect: Socket close | 15:46 |
| jsheeren | the nova-compute service get's marked down by the controllers for a minute or two, then it recovers | 15:46 |
| jsheeren | has anyone seen this behaviour before, | 15:46 |
| jsheeren | ? | 15:46 |
| *** ZZelle has joined #openstack-nova | 15:47 | |
| *** ratailor has quit IRC | 15:48 | |
| leakypipes | mhenkel: looks like you still need to address those unit test failures, though. let us know if you need assistance. | 15:48 |
| *** yamamoto has joined #openstack-nova | 15:48 | |
| *** Jack_Iv has quit IRC | 15:49 | |
| sneti | johnthetubaguy, I've updated the base patch for policy. can you please check this: https://review.openstack.org/#/c/434842/ | 15:50 |
| *** READ10 has quit IRC | 15:51 | |
| *** aarefiev is now known as aarefiev_afk | 15:51 | |
| *** rmart04 has quit IRC | 15:52 | |
| *** sbezverk has joined #openstack-nova | 15:52 | |
| mriedem | superdan: some easy test fixups in https://review.openstack.org/#/c/447009/ | 15:52 |
| efried | mriedem (aka Swamp Thing) your influence has spread: https://review.openstack.org/#/c/445667/2/nova_powervm/tests/virt/powervm/volume/test_iscsi.py@53 | 15:53 |
| mriedem | superdan: our next release of novaclient is going to be a major version bump, so we'll want to backport that bug fix to stable/ocata | 15:53 |
| *** voelzmo has quit IRC | 15:53 | |
| *** rmart04 has joined #openstack-nova | 15:54 | |
| mriedem | efried: cool | 15:54 |
| mriedem | superdan: but then after the bug fix i think we should just remove those deprecated arguments | 15:54 |
| efried | mriedem BTW, yesterday I looked through nova-powervm to see what it would take to retrofit the whole project accordingly. Yeeeaaaahhh, prolly not gonna do that wholesale any time soon. | 15:54 |
| mriedem | retrofit for what? autospec? | 15:55 |
| efried | yeah. | 15:55 |
| efried | But at least we can implement it moving forward. | 15:55 |
| efried | And the in-tree driver will be clean. | 15:55 |
| mriedem | warning on autospec, claudiub has experience with that https://review.openstack.org/#/c/339232/ | 15:55 |
| mriedem | it does take a performance hit on the test run | 15:56 |
| superdan | mriedem: I guess I missed the point at which we started having stable branches for clients | 15:56 |
| *** aysyd has joined #openstack-nova | 15:56 | |
| mriedem | superdan: years ago | 15:56 |
| mriedem | so we could backport the fix to ocata and release 7.1.1 | 15:56 |
| efried | That's interesting - the docs on autospec claim it's lazy out of the box. | 15:56 |
| *** yamamoto has quit IRC | 15:56 | |
| superdan | mriedem: regardless of when, I guess I missed it | 15:56 |
| mriedem | so your customer can pick that up w/o the backward incompatible changes that will be in 8.0.0 in pike | 15:57 |
| mriedem | we needed stable branches on all of the clients and libraries just to get fixes back, plus we needed it at times to unscrew the gate back before upper-constraints | 15:57 |
| claudiub | ooh, autospec | 15:59 |
| claudiub | i'm actually working on it | 15:59 |
| claudiub | I want it to become a permanent thingy | 15:59 |
| *** rmart04 has quit IRC | 16:00 | |
| claudiub | mriedem: something like this should do the trick: https://github.com/testing-cabal/mock/pull/394 | 16:00 |
| claudiub | there's still an issue with mock.patch autospecing, on which i'm working on. | 16:00 |
| mriedem | cool | 16:00 |
| efried | claudiub Second sentence here claims it's already lazy: https://docs.python.org/3/library/unittest.mock.html#autospeccing | 16:01 |
| efried | claudiub So you've discovered that's a false claim under certain conditions? | 16:02 |
| claudiub | it is a false claim. :) | 16:02 |
| claudiub | efried: https://github.com/testing-cabal/mock/issues/393 | 16:02 |
| efried | claudiub Down a page or so, more explicitly: "Because the speccing is done “lazily” (the spec is created as attributes on the mock are accessed) you can use it with very complex or deeply nested objects (like modules that import modules that import modules) without a big performance hit." | 16:02 |
| efried | Imagine my disappointment. | 16:02 |
| claudiub | i know, right? :) | 16:03 |
| *** vishwanathj has joined #openstack-nova | 16:03 | |
| claudiub | anyways, the only thing that specing / autospecing does at the moment, is ensure that your mock only have the same attributes / members as the spec itself, raising an AttributeError if non-existent attributes are accesed. | 16:04 |
| claudiub | so, there's at least that. | 16:04 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Move detach logic from manager into driver BDM https://review.openstack.org/439520 | 16:04 |
| openstackgerrit | Lee Yarwood proposed openstack/nova master: compute: Only destroy BDMs after successful detach call https://review.openstack.org/440693 | 16:04 |
| *** nicolasbock has quit IRC | 16:04 | |
| efried | claudiub That issue doesn't look related to performance/laziness | 16:04 |
| lyarwood | mdbooth: ^ remove_volume_connection behaviour should remain the same now | 16:04 |
| *** nic has joined #openstack-nova | 16:04 | |
| melwitt | claudiub, efried: well, it's expensive when you go and autospec every possible attribute, which is what you're doing right? | 16:05 |
| claudiub | efried: but, you can give my PR i've linked above a try. it fixes this issue. check the added unit tests. There are still some unit tests failing for pypy, because the unittests themselves are faulty. | 16:05 |
| mdbooth | lyarwood: Thanks, will look. | 16:05 |
| *** Oku_OS is now known as Oku_OS-away | 16:05 | |
| claudiub | melwitt: actually, the way mock works, is create a child mock when a mock's attrbute is accessed, and it wasn't accessed before, and then caches it. | 16:05 |
| claudiub | so, what I've done, is ensure that that child mock is properly spec'ed when created. | 16:06 |
| claudiub | so, it won't autospec everything, it will only autospec accessed attributes / methods. | 16:06 |
| melwitt | claudiub: okay, but you don't use every child mock in every test, right? or did you only spec the childs that are going to be asserted in tests? | 16:07 |
| openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | 16:08 |
| claudiub | right, only spec the childs that are going to be asserted in the tests. | 16:09 |
| *** jahsis has quit IRC | 16:09 | |
| *** stewie925 has joined #openstack-nova | 16:10 | |
| claudiub | anyways. will try out some new experiments on nova with mock autospecs. will see then the performance impact. | 16:10 |
| claudiub | and if it's a reasonable impact, autospecs should be used pretty much everywhere in openstack. :) | 16:11 |
| melwitt | okay, cool | 16:11 |
| cfriesen | so going back to the issue I raised yesterday about RPC timeouts during live migration waiting for large images to download from glance to the compute node...is this just a case of "we don't support that"? | 16:11 |
| mriedem | cfriesen: did you see my reply? | 16:12 |
| *** sree has quit IRC | 16:12 | |
| *** Apoorva has quit IRC | 16:12 | |
| cfriesen | mriedem: yes, though that was dealing with a service token. in my case it's an RPC timeout that's the issue, not authentication | 16:12 |
| cfriesen | if it takes more than 60 seconds to download the glance image, pre_live_migration() times out | 16:13 |
| figleaf | cdent: regarding https://review.openstack.org/#/c/427309 - it seems like you are also doing a lot of stuff manually, like telling RT to update usages, etc. Do you think it would be possible to more directly simulate the actual processes in order to ensure that the right code paths get executed? | 16:13 |
| mriedem | cfriesen: ok so https://docs.openstack.org/developer/nova/api_microversion_history.html#id31 won't help | 16:13 |
| openstackgerrit | Dan Smith proposed openstack/python-novaclient master: Fix aggregate_update name and availability_zone clash https://review.openstack.org/447009 | 16:14 |
| cfriesen | mriedem: nope. | 16:14 |
| mriedem | cfriesen: i'd ask tdurakov, | 16:14 |
| cburgess | cfriesen Worse if *anything* takes more then 60 seconds. lets says you have an instance with 10 volumes all using iscsi + multipath, that can take 2 or 3 minutes to build all the luns. | 16:14 |
| mriedem | i'd think that we could make conductor rpc cast to the compute to do pre-live migration, and have the compute cast back to conductor when it's done | 16:14 |
| cburgess | mriedem That might work. The problem right now is we have to set the RPC timeout to a very large value, like 5-10 minutes in some AZs to work around this. Honestly I would like to see that become async like you just described or have a seperate timeout for just that operation. | 16:15 |
| cdent | figleaf: my intent was to start narrow and then get more broad | 16:15 |
| figleaf | cdent: understood, but it feels sorta unit test-y | 16:16 |
| mriedem | cburgess: i'd have to dig into the oslo.messaging internals to find out how easy it is to adjust the rpc timeout on the fly like that for a single operation | 16:16 |
| cfriesen | mriedem: it's easy | 16:16 |
| cburgess | mriedem Just pass it as an arg | 16:16 |
| cburgess | I think we might have to expand one nova wrapper class to support the arg but otherwise its just an arg to the call to change the timeout for that specific call. | 16:17 |
| cfriesen | mriedem: but the thing is that you don't necessarily know up front what a reasonable value is since it depends on the IO load on your computes and glance-api | 16:17 |
| cfriesen | and other stuff like cburgess pointed out | 16:17 |
| *** vishwanathj has quit IRC | 16:17 | |
| cburgess | mriedem cfriesen But a change to have conductor/compute cast to/from each other and simply take action once the call comes in rather then waiting on it and forcing it to take some fixed amount of time is a way way better solution. | 16:17 |
| cburgess | cfriesen Right... which is why I think moving everything to cast calls to/from conductor is way better. | 16:18 |
| cfriesen | cburgess: mriedem: agreed...would we need to change the handling of the overall migration completion timeout? (checking code) | 16:18 |
| cburgess | cfriesen Not sure if that timer starts during pre flight. | 16:18 |
| mriedem | the timer is while it's moving data right? | 16:19 |
| cfriesen | cburgess: mriedem: looks like the migration completion timeout is just a libvirt thing | 16:19 |
| mriedem | to kill it if it gets stuck | 16:19 |
| cburgess | mriedem I thought that timer was going away? | 16:19 |
| mriedem | cburgess: i believe it is | 16:19 |
| cfriesen | cburgess: the progress timer is going away | 16:19 |
| mriedem | johnthetubaguy and the osic fok are looking at that | 16:19 |
| cburgess | Ahhh | 16:19 |
| mriedem | *folk | 16:19 |
| *** slaweq_ has quit IRC | 16:19 | |
| johnthetubaguy | sorry, I missed all that chat there | 16:19 |
| *** dillaman has joined #openstack-nova | 16:20 | |
| cburgess | johnthetubaguy Welcome to the party! | 16:20 |
| cburgess | johnthetubaguy Wait its after 4PM on a Friday there... why are you at the pub? | 16:20 |
| *** slaweq has joined #openstack-nova | 16:20 | |
| cfriesen | my concern about just moving to cast is that if one of those gets lost we're stuck with a migration in progress | 16:20 |
| cburgess | Sorry why aren't you | 16:20 |
| cburgess | cfriesen Yes thats the rub. | 16:20 |
| johnthetubaguy | heh, I am around for a few hours yet | 16:21 |
| johnthetubaguy | got band rehersal before I get close to a pub anyways! | 16:21 |
| cburgess | cfriesen We would have to be able to cancel it. Could the migration records accomplish that? | 16:21 |
| cdent | figleaf I think that's inevitable unless we want solely use the external interface of the compute manager (which might be fine), but even with there would still need to be something to causing the period job to run, and to call the external interface | 16:21 |
| mriedem | cfriesen: cburgess: so i just looked at oslo.messaging, and yeah it looks trivial to pass a new timeout in when constructing the client | 16:21 |
| cfriesen | I wonder if the existing live migration abort API would be sufficient | 16:21 |
| cburgess | mriedem I don't even think you have to do it when constructing the client. I think with an existing client its just an arg you can pass on call. | 16:21 |
| figleaf | cdent: yeah, I guess I'm just too idealistic :) | 16:21 |
| johnthetubaguy | mriedem: I thought that was a per call thing we can tweak? | 16:22 |
| mriedem | cburgess: oh that's even better b/c we cache the clients | 16:22 |
| johnthetubaguy | yeah, what cburgess said | 16:22 |
| cdent | figleaf: no, you just are trying to apply principles that might work elsewhere, here | 16:22 |
| cburgess | cfriesen Right thats what I'm thinking. If we know the state is "we made the cast to run pre live migration" then we should be able to cancel that by making the right call to abort it. | 16:22 |
| johnthetubaguy | cburgess: I am +1 increasing the timeout for the lots of volumes case | 16:22 |
| figleaf | cdent: like I said, too idealistic | 16:22 |
| superdan | mriedem: no caching of the client anymore right? | 16:23 |
| superdan | well, for compute anyway | 16:23 |
| cburgess | johnthetubaguy Some type of equation you thjink? Before call is made look at size of BDM and make some type of auto-adjust? Or just a new config value for pre-live timeout? | 16:23 |
| cfriesen | mriedem: I think there's an example of the per-call timeout in nova/cells/rpcapi.py | 16:23 |
| superdan | well, sometimes in compute actually | 16:23 |
| johnthetubaguy | cburgess: just a large one I think | 16:23 |
| *** bnemec is now known as beekneemech | 16:23 | |
| mriedem | yeah i see the timeout kwarg to the client.prepare method | 16:23 |
| johnthetubaguy | cburgess: its only there as a "oh dear, you must have given up" kind of timeout | 16:23 |
| johnthetubaguy | mriedem: yeah, thats the one | 16:23 |
| cburgess | johnthetubaguy Just an arbitrary "This call can suck, wait 10 minutes" type thing? | 16:24 |
| johnthetubaguy | mriedem: I thought we used that already for something, I can't remember now | 16:24 |
| *** hshiina has quit IRC | 16:24 | |
| superdan | it'd be nice if we had a per-call heartbeat mechanism, | 16:24 |
| superdan | which o.msg could provide for us | 16:24 |
| superdan | send a "still working" reply to the reply queue which would reset the timer | 16:24 |
| cburgess | johnthetubaguy The timeout would be a stop gap really. I think mriedem and cfriesen idea about moving to cast is the right solution but probably going to take more effort. | 16:24 |
| johnthetubaguy | superdan: thats the "task" framework in my head | 16:24 |
| cburgess | superdan OMG yes | 16:24 |
| superdan | johnthetubaguy: no, I mean something at the o.msg layer | 16:24 |
| superdan | johnthetubaguy: we could solve it in nova with a task too | 16:24 |
| johnthetubaguy | superdan: yeah, check if job is progressing, and you know the owner | 16:24 |
| mriedem | def ping(self, context, arg, timeout=None): | 16:24 |
| mriedem | arg_p = jsonutils.to_primitive(arg) | 16:24 |
| mriedem | cctxt = self.client.prepare(timeout=timeout) | 16:24 |
| mriedem | we do have something using it | 16:24 |
| superdan | but it'd be nice for lots of things if o.msg would heartbeat a particular call | 16:25 |
| johnthetubaguy | superdan: oh I see what you mean now | 16:25 |
| cfriesen | mriedem: you can set it in the client init but also in the cctxt.call() | 16:25 |
| superdan | johnthetubaguy: you could just send something to the reply queue like "this isn't the reply, but I ain't dead yet" | 16:25 |
| *** belmoreira has quit IRC | 16:25 | |
| cburgess | mriedem cfriesen johnthetubaguy superdan: So 1 bug and 1 spec here? bug = set a "large" timeout for the pre_live call today, spec = move to casting? | 16:25 |
| mriedem | cfriesen: i don't actually see the timeout kwarg in the rpc client call() method in oslo.messaging | 16:25 |
| johnthetubaguy | superdan: yeah, like a "continue" or whatever | 16:25 |
| mriedem | it's in the prepare() method | 16:25 |
| *** mdrabe has quit IRC | 16:26 | |
| superdan | johnthetubaguy: or lease renewal | 16:26 |
| mriedem | cfriesen: http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/rpc/client.py#n178 | 16:26 |
| *** mdrabe has joined #openstack-nova | 16:26 | |
| mriedem | cfriesen: call() doesn't use a timeout kwarg http://git.openstack.org/cgit/openstack/oslo.messaging/tree/oslo_messaging/rpc/client.py#n151 | 16:26 |
| cfriesen | mriedem: we're using it in nova/cells/rpcapi.py, line 332 | 16:27 |
| johnthetubaguy | superdan: yeah, reset the timeout timer | 16:27 |
| mriedem | cfriesen: that looks like a mistake | 16:27 |
| cburgess | mriedem cfriesen Yeah the kwargs just gets pased all the way down and I think that one gets dropped. | 16:28 |
| mriedem | cfriesen: the oslo.messaging rpc client call() method takes any **kwargs | 16:28 |
| mriedem | right, the call() method doesn't use the timeout | 16:28 |
| *** mlavalle has quit IRC | 16:28 | |
| mriedem | the prepare() method does | 16:28 |
| *** kaisers_ has joined #openstack-nova | 16:28 | |
| mriedem | which is nova/cells/rpcapi.py L326 | 16:28 |
| *** mlavalle has joined #openstack-nova | 16:28 | |
| cburgess | mriedem Right. | 16:28 |
| *** marst has quit IRC | 16:28 | |
| cburgess | mriedem I still suspect its doable... we just have to modify things a bit more to make it work. | 16:29 |
| cfriesen | mriedem: I think maybe the call() routine used to take a timeout | 16:29 |
| mriedem | maybe | 16:29 |
| mriedem | anyway yeah this is easy | 16:29 |
| mriedem | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L720 | 16:29 |
| mriedem | you use your nova config pre_lie_migration_timeout value there | 16:29 |
| cburgess | https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L720 | 16:29 |
| cburgess | LOL yeah | 16:30 |
| cburgess | mriedem Just found that same line. | 16:30 |
| mriedem | default to None to get the default from oslo.messaging | 16:30 |
| mriedem | bump to 10 minutes if you need to | 16:30 |
| cburgess | mriedem Yup | 16:30 |
| cburgess | mriedem So fixed value or config option? | 16:30 |
| mriedem | config | 16:30 |
| cburgess | Where is sdague to yell at me about config options? | 16:30 |
| cburgess | mriedem Bug? or Spec? | 16:30 |
| mriedem | this is completely operator side config | 16:30 |
| mriedem | this is a bug i think | 16:30 |
| mriedem | the spec is changing the blocking call to two casts | 16:31 |
| cburgess | mriedem Right | 16:31 |
| cburgess | mriedem OK... I shall file bug with code since this is easy and allow me to get ATC again. :P | 16:31 |
| mriedem | cburgess: ATC isn't a thing anymore | 16:31 |
| mriedem | remember | 16:31 |
| mriedem | your ATC pass is your PTG $100 | 16:31 |
| *** dtp has joined #openstack-nova | 16:32 | |
| openstackgerrit | Chris Friesen proposed openstack/nova master: remove unused parameter in rpc call https://review.openstack.org/447055 | 16:32 |
| openstackgerrit | Sarafraj Singh proposed openstack/nova-specs master: Spec to add per instance timeout for LM https://review.openstack.org/442809 | 16:33 |
| cburgess | mriedem ATC = voting rights still | 16:33 |
| *** kaisers_ has quit IRC | 16:33 | |
| cburgess | mriedem Semi-related... any idea who to contact about my ATC pass? I never got that email with details for Boston. | 16:33 |
| *** jsheeren has quit IRC | 16:33 | |
| *** kaisers_ has joined #openstack-nova | 16:34 | |
| cburgess | mriedem I can't vote in elections for TC, or nova unless I'm ATC. Or unless a PTL goes through the special ATC nomination process to make someone an ATC. | 16:34 |
| dtp | johnthetubaguy and jroll, are you guys around? | 16:34 |
| raj_singh | johnthetubaguy: I tried to address your concerns on https://review.openstack.org/#/c/442809/. Can you please have another look, when you get a chance? | 16:35 |
| johnthetubaguy | dtp: I am around | 16:35 |
| cfriesen | cburgess: I had an issue too...ended up emailing summit@openstack.org | 16:35 |
| johnthetubaguy | raj_singh: cool | 16:35 |
| cburgess | cfriesen Cool thanks. | 16:35 |
| dtp | johnthetubaguy - would have have a look at https://blueprints.launchpad.net/nova/+spec/conf-network-metadata-always-static-ip when you have a few minutes, please? | 16:35 |
| cburgess | cfriesen Weird because I recall them even asking me to confirm I was going to boston and my email when I checked in at PTG but nothing. | 16:36 |
| mriedem | cburgess: i don't remember anything about ATC passes for boston | 16:36 |
| mriedem | true for the elections though | 16:36 |
| *** sridharg has quit IRC | 16:36 | |
| mriedem | cburgess: did you land any code changes in ocata? | 16:36 |
| cfriesen | cburgess: whoops, make that summitreg@openstack.org | 16:36 |
| mriedem | cburgess: otherwise there is an ATC nomination process but i think we're past that | 16:36 |
| cburgess | mriedem No I haven't landed actual code in... a while... | 16:36 |
| cburgess | mriedem I don't get to code anything. :( | 16:37 |
| sdague | cburgess: we can totally find you some easy bugs to fix :) | 16:37 |
| cburgess | I last landed code on 2/8/2016 | 16:37 |
| sdague | cburgess: nova api-ref is in tree now, I'm sure you can tell us a ton of things we lie about with the API :) | 16:37 |
| cburgess | sdague I just found one. It involves a new config option, you will hate me. | 16:37 |
| cburgess | sdague LOL | 16:38 |
| mriedem | dtp: btw, now that you're around, we should talk about your alternative for https://blueprints.launchpad.net/nova/+spec/service-hyper-pci-uuid-in-api | 16:38 |
| *** kaisers_ has quit IRC | 16:38 | |
| dtp | sure, you want to do that here or privately? | 16:38 |
| mriedem | dtp: here | 16:38 |
| mriedem | unless it involves profanity | 16:38 |
| mriedem | but then still here | 16:38 |
| superdan | since when does that matter? | 16:39 |
| mriedem | superdan: it doesn't | 16:39 |
| dtp | hehe | 16:39 |
| dtp | my main concern is [Then when a request to delete a service is made, if the uuid is provided we can simply iterate cells until we find the service, or error with a 404.] | 16:39 |
| dtp | i would expect to not have to iterate when given an id | 16:39 |
| dtp | or guid | 16:39 |
| superdan | dtp: no, we do | 16:39 |
| superdan | dtp: unless we index services in the api db, which wouldn't be worth the trouble, IMHO | 16:40 |
| superdan | plus the syncing problem | 16:40 |
| *** crushil has joined #openstack-nova | 16:40 | |
| dtp | if i understand correctly, the problem w/ id is we don't know which cell to point to, right? | 16:40 |
| mriedem | yes | 16:41 |
| superdan | dtp: the problem with id is that it can be duplicated in multiple cells | 16:41 |
| superdan | not that we don't know what cell it's in | 16:41 |
| mriedem | dtp: we'd need a ServiceMapping in the api db | 16:41 |
| mriedem | like we have for HostMapping and InstanceMapping | 16:41 |
| superdan | -2 on that | 16:41 |
| superdan | please no | 16:41 |
| mriedem | i agree, no to that | 16:41 |
| dtp | so i was thinking we could embed the cell id w/ the object id | 16:41 |
| mriedem | the int? | 16:41 |
| superdan | you mean change id to "%(cell_uuid)s-%(service_uuid)s" ? | 16:42 |
| dtp | yes | 16:42 |
| dtp | something like that | 16:42 |
| *** kaisers_ has joined #openstack-nova | 16:42 | |
| mriedem | id is an int primary key right now, | 16:42 |
| mriedem | and expressed that way in the api | 16:42 |
| mriedem | so we can't change that | 16:42 |
| superdan | mriedem: he means in the api after a microversion | 16:43 |
| superdan | which we could do | 16:43 |
| mriedem | unless you're talking about rather than return service uuid in the microversion, just put the cell id in that | 16:43 |
| superdan | however, | 16:43 |
| mriedem | ok | 16:43 |
| superdan | I'd rather not expose cell details out of the api, even subversively | 16:43 |
| superdan | and I don't think this is anywhere near a big enough problem to warrant that | 16:43 |
| mriedem | that is something that exists with cellsv1 right? | 16:43 |
| mriedem | the cell namespace? | 16:43 |
| superdan | mriedem: yeah, and it has problems, and we vowed not to expose cell details out of the api with v2 | 16:43 |
| mriedem | ok, yes, it does feel too special snowflake for a problem that doesn't warrant it | 16:44 |
| superdan | yeah | 16:44 |
| mriedem | because if we did that for services/computes/pci_devices, we'd have to start thinking about doing it for other things too, for consistency, which we don't want to do | 16:44 |
| mriedem | you could restrict an id like that to admin-only type resources, but still | 16:45 |
| superdan | there's just no need for it, IMHO | 16:45 |
| mriedem | dtp: just curious, how many cellsv1 cells are you guys running in production? | 16:45 |
| superdan | we can parallelize the queries anyway, to lower the latency at some point | 16:46 |
| superdan | mriedem: cern has 50 | 16:46 |
| mriedem | because initial deployments that weren't ever cellsv1 to begin with will just be a single cell | 16:46 |
| dtp | we don't have 50 | 16:46 |
| mriedem | so there is no looping | 16:46 |
| superdan | mriedem: service delete has to be an infrequent thing | 16:46 |
| mriedem | superdan: i know | 16:46 |
| dtp | but hypervisor lookup? | 16:46 |
| mriedem | i was just thinking years down the road when having hundreds of cells could cause performance pain in the API | 16:47 |
| superdan | mriedem: if we end up with actual pain, then we add a cache or mapping | 16:47 |
| mriedem | dtp: we have the HostMapping for that | 16:47 |
| mriedem | superdan: yeah | 16:47 |
| *** bmace has quit IRC | 16:48 | |
| dtp | ok, got it. thanks guys | 16:48 |
| *** bmace has joined #openstack-nova | 16:48 | |
| mriedem | np. and thanks for thinking up alternatives | 16:48 |
| dtp | yw | 16:48 |
| mriedem | i can work that into the alternatives section of the spec | 16:48 |
| mriedem | "dtp sez..." | 16:48 |
| dtp | lol | 16:48 |
| superdan | dtp: you should /nick deeteepee | 16:49 |
| superdan | because: friday | 16:49 |
| mriedem | superdan: +2 on the novaclient change | 16:49 |
| superdan | thanks | 16:49 |
| *** dtp is now known as deeteepee | 16:50 | |
| superdan | much better | 16:50 |
| *** Jack_Iv has joined #openstack-nova | 16:50 | |
| *** jianghuaw-m has joined #openstack-nova | 16:52 | |
| *** Jack_Iv has quit IRC | 16:53 | |
| *** tbachman has quit IRC | 16:54 | |
| *** jianghuaw-m has quit IRC | 16:55 | |
| *** jaosorior has quit IRC | 16:55 | |
| *** vks1 has quit IRC | 16:59 | |
| *** deeteepee is now known as deeteepee-afk | 17:00 | |
| openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | 17:01 |
| *** armax has joined #openstack-nova | 17:01 | |
| *** dikonoor has joined #openstack-nova | 17:01 | |
| *** Apoorva has joined #openstack-nova | 17:05 | |
| *** voelzmo has joined #openstack-nova | 17:14 | |
| *** haplo37 has quit IRC | 17:15 | |
| *** ijw has joined #openstack-nova | 17:18 | |
| *** pcaruana has quit IRC | 17:18 | |
| *** ayogi has joined #openstack-nova | 17:20 | |
| *** ijw has quit IRC | 17:21 | |
| *** ijw_ has quit IRC | 17:22 | |
| jroll | heh | 17:23 |
| jroll | deeteepee-afk: you were looking for me last night, what's up? | 17:23 |
| *** haplo37 has joined #openstack-nova | 17:24 | |
| *** saju_m has joined #openstack-nova | 17:31 | |
| *** voelzmo has quit IRC | 17:33 | |
| openstackgerrit | Christopher Brown proposed openstack/nova master: Add lan9118 as valid nic for hw_vif_model property for qemu https://review.openstack.org/393489 | 17:33 |
| cburgess | cfriesen Did you already have a bug open about the pre live timeouts on glance? | 17:34 |
| *** jpena is now known as jpena|off | 17:34 | |
| *** ltomasbo is now known as ltomasbo|away | 17:35 | |
| *** kaisers_ has quit IRC | 17:39 | |
| deeteepee-afk | hey jroll! do you mind if i add your comments from yesterday regarding https://blueprints.launchpad.net/nova/+spec/conf-network-metadata-always-static-ip ? or if you have any other comments, i would appreciate them | 17:41 |
| jroll | leakypipes: having a conversation about ironic capabilities^Wtraits, reading the spec, it sounds like standard traits go through os-traits, custom go through the api. does queens sound like a reasonable timeframe for us to move ironic capabilities into that so people can stop using the ComputeCapabilitiesFilter thing? | 17:41 |
| jroll | leakypipes: (which also makes me wonder, should we go ahead and cover a flavor extra spec for custom traits when we do the custom resource class extra spec?) | 17:42 |
| *** gabor_antal_ has joined #openstack-nova | 17:42 | |
| *** gabor_antal has quit IRC | 17:42 | |
| *** Jack_Iv has joined #openstack-nova | 17:42 | |
| jroll | deeteepee-afk: yeah, go for it :) | 17:42 |
| deeteepee-afk | thanks | 17:42 |
| leakypipes | jroll: yes and yes | 17:42 |
| *** voelzmo has joined #openstack-nova | 17:43 | |
| jroll | leakypipes: cool - also question from yesterday, I put custom RC allocations as a dependency for the flavor changes, do you want to do that work, and do you want to write a separate spec or put it in this one? https://review.openstack.org/#/c/446570/2/specs/pike/approved/custom-resource-classes-in-flavors.rst | 17:44 |
| *** yamahata has joined #openstack-nova | 17:44 | |
| *** fragatina has joined #openstack-nova | 17:46 | |
| *** voelzmo has quit IRC | 17:48 | |
| openstackgerrit | John Garbutt proposed openstack/nova master: Add and use ProcessedNetworkRequest object https://review.openstack.org/418025 | 17:50 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Pass objects into _validate_requested_network_ids https://review.openstack.org/418027 | 17:50 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Pass objects into _create_ports_for_instance https://review.openstack.org/418512 | 17:50 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Stop passing network when you only need dns_domain https://review.openstack.org/419577 | 17:51 |
| openstackgerrit | John Garbutt proposed openstack/nova master: Pass objects into _update_ports_for_instance https://review.openstack.org/418513 | 17:51 |
| *** burt has quit IRC | 17:51 | |
| openstackgerrit | John Garbutt proposed openstack/nova master: Extract preallocate_for_instance https://review.openstack.org/418876 | 17:51 |
| *** macsz_and_cheese has joined #openstack-nova | 17:52 | |
| *** crushil has quit IRC | 17:55 | |
| *** nicolasbock has joined #openstack-nova | 17:55 | |
| *** gszasz has quit IRC | 17:56 | |
| *** Sukhdev has joined #openstack-nova | 17:58 | |
| *** gyee has joined #openstack-nova | 17:59 | |
| *** ayogi has quit IRC | 18:00 | |
| *** Sukhdev has quit IRC | 18:00 | |
| *** zioproto has quit IRC | 18:01 | |
| *** mnestratov has quit IRC | 18:05 | |
| *** lpetrut has quit IRC | 18:06 | |
| *** armax has quit IRC | 18:10 | |
| *** Sukhdev has joined #openstack-nova | 18:11 | |
| *** armax has joined #openstack-nova | 18:14 | |
| *** ociuhandu has quit IRC | 18:14 | |
| openstackgerrit | Maciej Szankin proposed openstack/nova master: conf: rename qemu_enable_gluster_storage_driver https://review.openstack.org/447091 | 18:16 |
| *** satyar has quit IRC | 18:17 | |
| *** kaisers_ has joined #openstack-nova | 18:19 | |
| *** Sukhdev has quit IRC | 18:21 | |
| *** mdrabe has quit IRC | 18:22 | |
| openstackgerrit | John Garbutt proposed openstack/nova-specs master: Spec to add per instance timeout for LM https://review.openstack.org/442809 | 18:23 |
| johnthetubaguy | cfriesen: cburgess: thats that live-migrate API spec that raj_singh is working on ^ | 18:23 |
| *** crushil has joined #openstack-nova | 18:23 | |
| johnthetubaguy | there is also the dependent spec thats interesting | 18:24 |
| cburgess | johnthetubaguy Thanks | 18:25 |
| *** kaisers_ has quit IRC | 18:25 | |
| *** liverpooler has quit IRC | 18:31 | |
| *** liverpooler has joined #openstack-nova | 18:31 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 18:32 |
| *** mdrabe has joined #openstack-nova | 18:34 | |
| *** annegentle has joined #openstack-nova | 18:41 | |
| *** david-lyle_ is now known as david-lyle | 18:43 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 18:47 |
| *** tbachman has joined #openstack-nova | 18:47 | |
| *** ar1el has joined #openstack-nova | 18:49 | |
| *** tesseract has quit IRC | 18:50 | |
| openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | 18:51 |
| *** kaisers has quit IRC | 18:54 | |
| *** unicell has joined #openstack-nova | 18:56 | |
| *** annegentle has quit IRC | 18:56 | |
| *** tbachman has quit IRC | 18:56 | |
| *** unicell has quit IRC | 18:57 | |
| *** annegentle has joined #openstack-nova | 18:57 | |
| *** catintheroof has quit IRC | 18:59 | |
| *** Sukhdev has joined #openstack-nova | 19:00 | |
| *** unicell has joined #openstack-nova | 19:01 | |
| *** IRCFReAK has joined #openstack-nova | 19:01 | |
| *** IRCFReAK has quit IRC | 19:01 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: WIP - add some functional tests for placement https://review.openstack.org/446123 | 19:05 |
| figleaf | cdent: if you're bored ^^ | 19:06 |
| *** catintheroof has joined #openstack-nova | 19:06 | |
| *** IRCFReAK has joined #openstack-nova | 19:07 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Add helper method to add additional data about policy rule. https://review.openstack.org/434842 | 19:08 |
| *** lpetrut has joined #openstack-nova | 19:08 | |
| *** amoralej is now known as amoralej|off | 19:09 | |
| *** IRCFReAK has quit IRC | 19:09 | |
| *** kaisers has joined #openstack-nova | 19:11 | |
| *** ar1el has quit IRC | 19:12 | |
| *** IRCFReAK has joined #openstack-nova | 19:17 | |
| *** lpetrut has quit IRC | 19:17 | |
| *** stvnoyes has quit IRC | 19:17 | |
| *** IRCFReAK has quit IRC | 19:18 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in agents.py https://review.openstack.org/445208 | 19:19 |
| *** Jack_Iv has quit IRC | 19:19 | |
| *** tbachman has joined #openstack-nova | 19:20 | |
| openstackgerrit | Anusha Unnam proposed openstack/nova master: Add keypairs field to InstancePayload https://review.openstack.org/419730 | 19:21 |
| *** Jack_Iv has joined #openstack-nova | 19:23 | |
| *** stvnoyes has joined #openstack-nova | 19:24 | |
| *** ar1el has joined #openstack-nova | 19:25 | |
| cfriesen | cburgess: sorry, just noticed your message. I don't have a bug open about the pre live timeouts on glance. | 19:25 |
| cburgess | cfriesen k | 19:25 |
| cfriesen | cburgess: mriedem: incidentally, for converting the pre-live-timeout to RPC cast, it's actually coming from the source nova-compute, not from nova-conductor. | 19:27 |
| cfriesen | gah, pre_live_migration, not timeout | 19:27 |
| mriedem | yeah i wondered about that | 19:27 |
| mriedem | because the computes are talking to each other rather than having coordinator orchestrate | 19:27 |
| cburgess | cfriesen LOL well its fairly accurate in some situtations. | 19:27 |
| cfriesen | lol | 19:27 |
| mriedem | which is something tdurakov was looking into | 19:27 |
| mriedem | the timeout hack bug fix doesn't really change though right? | 19:28 |
| mriedem | if it's the source compute or conductor as the rpc client | 19:28 |
| cburgess | mriedem Right... the source caller needs a configurable timeout. | 19:28 |
| cburgess | mriedem Because today is just blocks, hits rpc_timeout, and goes "oh well" | 19:29 |
| *** psachin has joined #openstack-nova | 19:29 | |
| mriedem | markus_z left too early https://www.youtube.com/watch?v=bGXaJzgtvS8 | 19:29 |
| mriedem | i have to assume markus loves scorpions | 19:30 |
| cfriesen | mriedem: the live migration is aready a cast from nova-conductor to the source nova-compute | 19:30 |
| cfriesen | mriedem: so this would be a cast from nova-compute to other nova-compute | 19:30 |
| mriedem | sorry i've moved topics | 19:31 |
| * cfriesen has to go do work | 19:31 | |
| cfriesen | I'm running into a case where running "virsh list" on a compute node hangs, trying to figure out why libvirtd isn't responding | 19:32 |
| erhudy | cfriesen: i had a problem a while back where libvirt got really cranky with nwfilters and it would hang like that | 19:34 |
| erhudy | rebooting made it happy again, but it was frustrating not being able to find a root cause | 19:34 |
| mriedem | superdan: leakypipes: sdague: can one of you push this through? i forgot it was still sitting out here https://review.openstack.org/#/c/432471/3 | 19:39 |
| superdan | yeah | 19:40 |
| mriedem | danke | 19:40 |
| leakypipes | mriedem: done | 19:40 |
| mriedem | that was fast | 19:40 |
| mriedem | ...maybe too fast | 19:40 |
| mriedem | i should have put an easter egg in there | 19:40 |
| leakypipes | heh | 19:40 |
| *** kaisers_ has joined #openstack-nova | 19:43 | |
| *** kaisers_ has quit IRC | 19:48 | |
| mriedem | ffs is compute_nodes.pci_stats just a json blob? | 19:50 |
| mriedem | ah i suppose it is but it's turned into PciDevicePoolList | 19:50 |
| mriedem | this is awful https://github.com/openstack/nova/blob/master/nova/api/openstack/compute/pci.py#L62 | 19:51 |
| mriedem | we literally take the pci_stats blob out of the db for a compute node, convert it to the object, then convert it back to a json dict and spew it out of the REST API | 19:51 |
| mriedem | so, | 19:51 |
| mriedem | if we ever were to add new fields to PciDevicePool, those automatically go out of the REST API | 19:52 |
| *** xyang1 has joined #openstack-nova | 19:52 | |
| mriedem | https://bugs.launchpad.net/nova/+bug/1673869 | 19:55 |
| openstack | Launchpad bug 1673869 in OpenStack Compute (nova) "api-ref: os-pci API is not documented at all" [Low,Confirmed] | 19:55 |
| mriedem | maybe we should just delete the API before anyone ever knew it existed | 19:55 |
| alex_xu | mriedem: more bad thing, actually the os-pci API isn't enable in the nova rest api | 19:57 |
| alex_xu | mriedem: https://github.com/openstack/nova/blob/master/setup.cfg#L130 | 19:57 |
| *** dimtruck is now known as zz_dimtruck | 19:58 | |
| *** lpetrut has joined #openstack-nova | 19:59 | |
| mriedem | alex_xu: oh, well that is actually good | 20:00 |
| alex_xu | mriedem: yea, that's ture for defect in the api | 20:01 |
| mriedem | so, if the API was added in v3, but never exposed or supported | 20:01 |
| mriedem | this is really just all dead code isn't it? | 20:01 |
| mriedem | and we would have to decide if we want to expose it out of the REST API in a microversion | 20:01 |
| mriedem | which i'd think we don't actually want to do that | 20:01 |
| alex_xu | mriedem: actually there is plan to do that. it is jsut the people, who commit to do that, disappeared | 20:02 |
| mriedem | sure, | 20:03 |
| mriedem | but that was a long time ago | 20:03 |
| alex_xu | yes, super long time ago | 20:03 |
| mriedem | i'm not sure we need to be exposing pci details out of the compute REST API | 20:03 |
| alex_xu | why? currently we really have no way to query what pci device on the host | 20:04 |
| *** awaugama has quit IRC | 20:04 | |
| efried | Should that be a get_inventory() thing? | 20:06 |
| mriedem | efried: unrelated | 20:07 |
| *** dave-mccowan has quit IRC | 20:07 | |
| mriedem | alex_xu: i'm sure there are other tools available to get that information from the host system | 20:07 |
| mriedem | actual inventory reporting tools | 20:07 |
| mriedem | alex_xu: i sort of consider this like https://docs.openstack.org/developer/nova/policies.html?highlight=metrics#metrics-gathering | 20:08 |
| mriedem | years ago we thought it was a great idea for nova to be a metrics gatherer, but in newton we decided we're not so great at that | 20:08 |
| mriedem | and shouldn't be in the business of doing it | 20:08 |
| *** cdent has quit IRC | 20:09 | |
| *** ssurana has joined #openstack-nova | 20:09 | |
| mriedem | i'll post to the ML about the os-pci api | 20:09 |
| *** psachin has quit IRC | 20:10 | |
| alex_xu | mriedem: thanks, i probably get your point. ++ for a email | 20:10 |
| *** IRCFReAK has joined #openstack-nova | 20:11 | |
| *** dikonoor has quit IRC | 20:12 | |
| *** IRCFReAK has quit IRC | 20:13 | |
| *** Sukhdev has quit IRC | 20:15 | |
| mriedem | http://lists.openstack.org/pipermail/openstack-dev/2017-March/114261.html | 20:19 |
| mriedem | superdan: leakypipes: sdague: ^ you'd probably like to weigh in | 20:20 |
| sdague | mriedem: if you need "... with fire!" sure | 20:21 |
| *** Sukhdev has joined #openstack-nova | 20:21 | |
| mriedem | acks are always good | 20:22 |
| mriedem | so i don't just look like a maniac | 20:22 |
| superdan | I thought it was actually kindof important for the pci use case, | 20:23 |
| superdan | but if it's not enabled by default, hopefully that is not true | 20:23 |
| mriedem | apparently not important enough since icehouse for anyone to care to expose it | 20:23 |
| mriedem | or they did via and out of tree extension/patch | 20:23 |
| mriedem | cfriesen: i'm looking at you | 20:23 |
| sdague | superdan: it's not just not enabled, it was never plumbed back in at all | 20:24 |
| superdan | well, it'd be a packaging change, right? but yeah | 20:24 |
| mriedem | it'd be a change to setup.cfg to enable it | 20:24 |
| mriedem | plus a microversion at this point | 20:24 |
| superdan | just removing that comment out is all you'd need to do right? | 20:24 |
| mriedem | if you don't care about microversions, sure | 20:24 |
| superdan | it's not enabled in our git tree, I'd have to check the actual packages to make sure it's not enabled there, | 20:25 |
| superdan | but it'd be better to just get vladik or someone to chime in | 20:26 |
| mriedem | i'm not following your point about packaging | 20:26 |
| mriedem | like enabled in the rpm spec? | 20:26 |
| mriedem | by uncommenting that line in the setup.cfg? | 20:26 |
| superdan | yeah, something like that | 20:26 |
| mriedem | that would be terribad | 20:26 |
| sdague | so, I kind of feel like that would be a big bad on the packaging side | 20:26 |
| superdan | I dunno, I have no idea, I'm just saying we should ask some pci-knowing person to make sure | 20:27 |
| sdague | well, it is on the ML | 20:27 |
| mriedem | "sorry developers, you didn't know it but we were supporting this b/c of code in the rpm spec all along!" | 20:27 |
| superdan | yep | 20:27 |
| sdague | I kind of feel like that's in the category of "you are running a fork" | 20:27 |
| superdan | I'll point some people at it to make sure it gets seen | 20:27 |
| sdague | even though the change looks small, it would light up a ton of dark code that has no active testing | 20:28 |
| superdan | I think that it being commented out in setup.cfg would make people just assume it's not on by default | 20:28 |
| superdan | but that there's not a problem enabling it | 20:28 |
| superdan | does turning that on make it show up in 2.1 | 20:28 |
| superdan | ? | 20:28 |
| mriedem | i explain a problem with enabling it | 20:28 |
| mriedem | i'm not sure | 20:29 |
| mriedem | as noted in the email, we pass the json blob straight out of the db via the rest api | 20:29 |
| superdan | because if so, that's a bit on us too | 20:29 |
| mriedem | no whitelist | 20:29 |
| openstackgerrit | Eric Fried proposed openstack/nova master: PowerVM Driver: spawn/delete #1: no-ops https://review.openstack.org/438119 | 20:29 |
| superdan | mriedem: we have other (or have had other) apis that do the same thing, so that in itself doesn't mean much :) | 20:29 |
| mriedem | but we shouldn't be doing that, and i'm hoping we are actively trying to clamp done on doing anything like that, | 20:29 |
| mriedem | like the diagnostics API rewrite | 20:30 |
| mriedem | *down | 20:30 |
| *** zz_dimtruck is now known as dimtruck | 20:30 | |
| openstackgerrit | christopher.uhler proposed openstack/nova master: Libvirt volume driver for Veritas HyperScale https://review.openstack.org/443951 | 20:30 |
| superdan | definitely | 20:30 |
| sdague | I also think we definitively had said API extensions are deprecated and not supported any more | 20:31 |
| *** IRCFReAK has joined #openstack-nova | 20:31 | |
| mriedem | we did, in newton | 20:32 |
| mriedem | this would have to come back in as a microversion | 20:32 |
| mriedem | like server tags | 20:32 |
| mriedem | which was 2.26 | 20:32 |
| sdague | yep | 20:32 |
| *** IRCFReAK has quit IRC | 20:32 | |
| mriedem | that's why cyeoh removed it and opened a bug | 20:32 |
| sdague | anyway, it's on the mailing list | 20:32 |
| *** crushil has quit IRC | 20:32 | |
| *** deeteepee-afk is now known as deeteepee | 20:33 | |
| *** smatzek has quit IRC | 20:34 | |
| *** Jack_Iv has quit IRC | 20:34 | |
| *** cuhler has joined #openstack-nova | 20:36 | |
| *** slaweq has quit IRC | 20:37 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Adding auto_disk_config field to InstancePayload https://review.openstack.org/419185 | 20:37 |
| superdan | mriedem: sdague I think we're good | 20:38 |
| sdague | superdan: \o/ | 20:38 |
| mriedem | yay | 20:39 |
| superdan | I'd let it simmer on the list for a bit, at least | 20:39 |
| superdan | it would also be a candidate for interfacing with ops in boston, | 20:39 |
| superdan | at least to explain it if anyone speaks up | 20:39 |
| mriedem | i did post the same thing to the dev and ops list btw | 20:39 |
| mriedem | but yeah i'm not removing it today | 20:39 |
| superdan | okay | 20:39 |
| superdan | well, we should remove it regardless of the response | 20:40 |
| *** lpetrut has quit IRC | 20:41 | |
| cuhler | Hi. I submitted a new Nova volume driver and the os-brick connector on which it depends for review for Pike. How can I make the volume driver unit tests run before the os-brick connector is approved and merged to master? Some way to configure tox/testr? | 20:42 |
| *** xyang1 has quit IRC | 20:42 | |
| cuhler | https://review.openstack.org/#/c/443951/ | 20:42 |
| cuhler | https://review.openstack.org/#/c/442754/ | 20:42 |
| *** crushil has joined #openstack-nova | 20:42 | |
| *** kaisers_ has joined #openstack-nova | 20:44 | |
| *** IRCFReAK has joined #openstack-nova | 20:45 | |
| *** IRCFReAK has quit IRC | 20:45 | |
| *** xyang1 has joined #openstack-nova | 20:46 | |
| *** Jack_Iv has joined #openstack-nova | 20:47 | |
| *** tblakes has quit IRC | 20:48 | |
| *** kaisers_ has quit IRC | 20:49 | |
| *** IRCFReAK has joined #openstack-nova | 20:50 | |
| *** IRCFReAK has quit IRC | 20:50 | |
| sbezverk | mriedem: ping | 20:53 |
| *** lpetrut has joined #openstack-nova | 20:54 | |
| mriedem | sbezverk: pong | 20:55 |
| *** Sukhdev has quit IRC | 20:56 | |
| openstackgerrit | Matthew Edmonds proposed openstack/nova master: fix os-volume_attachments policy checks https://review.openstack.org/389314 | 21:00 |
| *** tbachman has quit IRC | 21:01 | |
| *** eharney has quit IRC | 21:03 | |
| *** crushil has quit IRC | 21:03 | |
| *** aysyd has quit IRC | 21:05 | |
| *** ar1el has quit IRC | 21:10 | |
| mriedem | superdan: we might have a problem with the services uuid in api thin | 21:11 |
| mriedem | *thing | 21:11 |
| mriedem | https://developer.openstack.org/api-ref/compute/#disable-scheduling-for-a-compute-service | 21:11 |
| mriedem | the action APIs on the services records don't take an id | 21:11 |
| *** annegentle has quit IRC | 21:11 | |
| mriedem | they take the host/binary | 21:11 |
| mriedem | would those be enough to uniquely identify a service within a cell? | 21:11 |
| superdan | yeah | 21:11 |
| superdan | should be | 21:11 |
| superdan | if you have two hosts with the same hostname you have scheduling problems anyway | 21:12 |
| superdan | I mean, it'd be better if they used a uuid, but.. | 21:12 |
| mriedem | sure | 21:12 |
| mriedem | ok, i'll make a note of it in here | 21:12 |
| *** tbachman has joined #openstack-nova | 21:13 | |
| *** catintheroof has quit IRC | 21:16 | |
| openstackgerrit | Michael Henkel proposed openstack/nova master: contrail: add vrouter VIF plugin type support https://review.openstack.org/441183 | 21:18 |
| openstackgerrit | Sarafraj Singh proposed openstack/nova-specs master: Add live-migration-force-after-timeout spec https://review.openstack.org/438467 | 21:21 |
| openstackgerrit | Sarafraj Singh proposed openstack/nova-specs master: Spec to add per instance timeout for LM https://review.openstack.org/442809 | 21:22 |
| *** sbezverk has quit IRC | 21:23 | |
| *** edmondsw has quit IRC | 21:24 | |
| *** alex_xu has quit IRC | 21:25 | |
| *** cuhler has quit IRC | 21:26 | |
| *** alex_xu has joined #openstack-nova | 21:28 | |
| *** edmondsw has joined #openstack-nova | 21:29 | |
| *** edmondsw has quit IRC | 21:31 | |
| *** gouthamr has quit IRC | 21:33 | |
| *** Jack_Iv has quit IRC | 21:34 | |
| *** edmondsw has joined #openstack-nova | 21:34 | |
| *** Jack_Iv has joined #openstack-nova | 21:35 | |
| *** Jeffrey4l has quit IRC | 21:35 | |
| *** Jeffrey4l has joined #openstack-nova | 21:36 | |
| *** Jack_Iv has quit IRC | 21:39 | |
| *** edmondsw has quit IRC | 21:39 | |
| *** jwcroppe has quit IRC | 21:40 | |
| *** esberglu has quit IRC | 21:41 | |
| *** esberglu has joined #openstack-nova | 21:41 | |
| *** baoli has quit IRC | 21:42 | |
| *** leakypipes has quit IRC | 21:43 | |
| *** esberglu has quit IRC | 21:46 | |
| openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in admin_actions. https://review.openstack.org/440829 | 21:46 |
| *** xyang1 has quit IRC | 21:48 | |
| *** tbachman has quit IRC | 21:49 | |
| *** karimb has quit IRC | 21:51 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Use uuids in services and os-hypervisors APIs (spec) https://review.openstack.org/447149 | 21:52 |
| mriedem | superdan: finally got it done ^ | 21:52 |
| mriedem | detailed as all get out | 21:52 |
| mriedem | i'll have to pick your brain a bit at some point about online data migrations and what we want to do there | 21:52 |
| deeteepee | is that the same one i'm working on? | 21:52 |
| openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in admin_actions. https://review.openstack.org/440829 | 21:52 |
| mriedem | deeteepee: were you actually writing it up? | 21:52 |
| deeteepee | yes | 21:53 |
| mriedem | crap | 21:53 |
| mriedem | i didn't know that | 21:53 |
| mriedem | sorry | 21:53 |
| deeteepee | it's ok | 21:53 |
| mriedem | i threw our name as a co-contributor in ^ so you should read it over | 21:53 |
| openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in admin_actions. https://review.openstack.org/440829 | 21:53 |
| deeteepee | k | 21:53 |
| superdan | deeteepee: and you now get to be uuber picky :) | 21:54 |
| openstackgerrit | Sujitha proposed openstack/nova master: Add description to policies in assisted_volume_snapshots.py https://review.openstack.org/445238 | 21:54 |
| mriedem | i'm pretty sure this is flawless | 21:54 |
| *** kaisers_ has joined #openstack-nova | 21:54 | |
| cburgess | lol | 21:54 |
| *** tbachman has joined #openstack-nova | 21:54 | |
| *** Swami has joined #openstack-nova | 21:55 | |
| *** gouthamr has joined #openstack-nova | 21:57 | |
| *** kaisers_ has quit IRC | 21:59 | |
| *** kenperkins has quit IRC | 22:03 | |
| *** yamahata has quit IRC | 22:10 | |
| *** sbezverk has joined #openstack-nova | 22:12 | |
| *** lpetrut has quit IRC | 22:15 | |
| *** mdrabe has quit IRC | 22:15 | |
| *** jdurgin has quit IRC | 22:40 | |
| *** gongysh has joined #openstack-nova | 22:40 | |
| *** Swami has quit IRC | 22:43 | |
| *** timello has quit IRC | 22:44 | |
| *** Sukhdev has joined #openstack-nova | 22:44 | |
| *** liangy has quit IRC | 22:45 | |
| *** timello has joined #openstack-nova | 22:47 | |
| *** dimtruck is now known as zz_dimtruck | 22:51 | |
| *** figleaf is now known as edleafe | 22:52 | |
| *** ociuhandu has joined #openstack-nova | 22:54 | |
| *** tbachman has quit IRC | 22:55 | |
| *** ssurana has quit IRC | 22:57 | |
| *** superdan is now known as dansmith | 22:57 | |
| openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies to availability_zone https://review.openstack.org/440836 | 22:58 |
| *** ociuhandu has quit IRC | 22:59 | |
| *** ssurana has joined #openstack-nova | 23:00 | |
| *** ssurana has quit IRC | 23:02 | |
| *** liverpooler has quit IRC | 23:02 | |
| *** ociuhandu has joined #openstack-nova | 23:05 | |
| *** kaisers_ has joined #openstack-nova | 23:09 | |
| *** ociuhandu has quit IRC | 23:09 | |
| *** Swami has joined #openstack-nova | 23:10 | |
| *** jamesdenton has joined #openstack-nova | 23:12 | |
| openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to volumes_attachments.py https://review.openstack.org/445252 | 23:14 |
| *** kaisers_ has quit IRC | 23:14 | |
| *** erlon has quit IRC | 23:15 | |
| *** jamesden_ has joined #openstack-nova | 23:16 | |
| *** tbachman has joined #openstack-nova | 23:17 | |
| *** jamesdenton has quit IRC | 23:19 | |
| openstackgerrit | Anusha Unnam proposed openstack/nova master: Add description to policies in volumes.py https://review.openstack.org/445637 | 23:20 |
| *** nicolasbock has quit IRC | 23:22 | |
| openstackgerrit | Anusha Unnam proposed openstack/nova master: Add descripiton to policies in virtual_interfaces.py https://review.openstack.org/445652 | 23:24 |
| *** claudiub has quit IRC | 23:25 | |
| *** markvoelker has quit IRC | 23:26 | |
| *** Swami has quit IRC | 23:26 | |
| *** ijw has joined #openstack-nova | 23:26 | |
| *** Swami has joined #openstack-nova | 23:26 | |
| *** zz_dimtruck is now known as dimtruck | 23:29 | |
| *** unicell has quit IRC | 23:31 | |
| *** stewie925 has quit IRC | 23:32 | |
| *** edleafe- has joined #openstack-nova | 23:33 | |
| *** dimtruck is now known as zz_dimtruck | 23:33 | |
| *** edleafe has quit IRC | 23:35 | |
| *** zz_dimtruck is now known as dimtruck | 23:37 | |
| *** abalutoiu has joined #openstack-nova | 23:45 | |
| *** abalutoiu has quit IRC | 23:45 | |
| *** marst has joined #openstack-nova | 23:50 | |
| *** crushil has joined #openstack-nova | 23:52 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!