| *** gbarros has joined #openstack-nova | 00:01 | |
| *** cdent has quit IRC | 00:02 | |
| *** yamamoto has quit IRC | 00:02 | |
| *** ijw has joined #openstack-nova | 00:08 | |
| *** ijw has quit IRC | 00:08 | |
| *** ijw has joined #openstack-nova | 00:08 | |
| mriedem | mtreinish: does sourcing /opt/stack/new/devstack/lib/nova in grenade not also source the /opt/stack/new/devstack/local.conf? | 00:09 |
|---|---|---|
| mtreinish | mriedem: probably not, in devstack the assumption is likely that is sourced well before lib/nova is called | 00:11 |
| mriedem | alright | 00:12 |
| mriedem | hmm, so can i even source this thing from within a post_test_hook? | 00:12 |
| mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/new/local.conf.txt.gz | 00:12 |
| mriedem | i need to get the CELLSV2_SETUP value | 00:13 |
| *** hemna__ has quit IRC | 00:14 | |
| *** mingyu has joined #openstack-nova | 00:14 | |
| *** aloga has quit IRC | 00:14 | |
| mriedem | seems like the only thing available to me is the ansible env vars from the d-g setup in http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/logs/devstack-gate-setup-host.txt.gz | 00:15 |
| mtreinish | mriedem: local.conf should be sourced by grenade already: https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 | 00:15 |
| mriedem | so i can get like GRENADE_OLD_BRANCH | 00:15 |
| *** aloga has joined #openstack-nova | 00:16 | |
| mriedem | yeah but this is a script running in the post_test_hook | 00:16 |
| mriedem | would that carry over? | 00:17 |
| mriedem | http://logs.openstack.org/71/508271/1/check/gate-grenade-dsvm-neutron-multinode-live-migration-nv/659a2cd/console.html#_2017-10-04_14_36_43_557726 | 00:18 |
| mtreinish | ah, ok. Yeah it won't be sourced there | 00:18 |
| melwitt | mtreinish: still failing after installing python3-rados, even though it shows up in py3 pip freeze on the job. was not expecting that. and when I try locally I can import rados in py3 http://logs.openstack.org/63/509663/3/experimental/gate-tempest-dsvm-py35-full-devstack-plugin-ceph-ubuntu-xenial-nv/2361fe2/logs/screen-g-api.txt.gz?level=ERROR | 00:18 |
| mriedem | ok, so i need to do https://github.com/openstack-dev/grenade/blob/03de9e0fc7f4fc50a00db5d547413e26cf0780dd/grenade.sh#L228 from within the script | 00:18 |
| *** crushil has joined #openstack-nova | 00:19 | |
| melwitt | back to the drawing board | 00:19 |
| mtreinish | melwitt: what about rbd on py3. Looking at the glance_store code it if either rados or rbd isn't present it'll set both to None | 00:20 |
| mtreinish | melwitt: https://github.com/openstack/glance_store/blob/master/glance_store/_drivers/rbd.py#L37-L42 | 00:20 |
| melwitt | mtreinish: hm, good point. lemme go down that road | 00:21 |
| melwitt | yeah you're right | 00:21 |
| melwitt | thanks | 00:21 |
| melwitt | need to get py3 rbd separately | 00:21 |
| mtreinish | sure, np | 00:21 |
| mtreinish | mriedem: probably | 00:22 |
| mtreinish | mriedem: or something like that to source the vars you need from the conf file | 00:22 |
| *** namnh has quit IRC | 00:22 | |
| *** crushil has quit IRC | 00:23 | |
| *** crushil has joined #openstack-nova | 00:25 | |
| *** hemna__ has joined #openstack-nova | 00:25 | |
| *** crushil has quit IRC | 00:26 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix live migration grenade ceph setup https://review.openstack.org/508271 | 00:29 |
| mriedem | mtreinish: check out this crazy shit ^ | 00:29 |
| *** Swami has quit IRC | 00:32 | |
| *** claudiub has quit IRC | 00:33 | |
| *** vladikr has joined #openstack-nova | 00:40 | |
| *** Nel1x has quit IRC | 00:44 | |
| *** yamamoto has joined #openstack-nova | 00:45 | |
| *** vladikr has quit IRC | 00:45 | |
| *** yamamoto has quit IRC | 00:49 | |
| *** baoli has joined #openstack-nova | 00:52 | |
| *** crushil has joined #openstack-nova | 01:01 | |
| *** coreywright has quit IRC | 01:08 | |
| *** Apoorva has quit IRC | 01:11 | |
| *** jogo has joined #openstack-nova | 01:13 | |
| *** openstackstatus has quit IRC | 01:14 | |
| *** openstackstatus has joined #openstack-nova | 01:16 | |
| *** ChanServ sets mode: +v openstackstatus | 01:16 | |
| *** hongbin has joined #openstack-nova | 01:20 | |
| *** penick has joined #openstack-nova | 01:20 | |
| *** coreywright has joined #openstack-nova | 01:26 | |
| *** mriedem has quit IRC | 01:26 | |
| *** zigo has quit IRC | 01:27 | |
| *** baoli has quit IRC | 01:27 | |
| *** gyee has quit IRC | 01:28 | |
| *** dikonoor has joined #openstack-nova | 01:29 | |
| *** zigo has joined #openstack-nova | 01:31 | |
| *** esberglu has joined #openstack-nova | 01:31 | |
| *** esberglu has quit IRC | 01:32 | |
| *** esberglu has joined #openstack-nova | 01:32 | |
| *** tbachman has quit IRC | 01:34 | |
| *** chyka has joined #openstack-nova | 01:36 | |
| *** esberglu has quit IRC | 01:36 | |
| *** penick has quit IRC | 01:37 | |
| *** penick has joined #openstack-nova | 01:38 | |
| *** chyka has quit IRC | 01:40 | |
| *** tetsuro has joined #openstack-nova | 01:41 | |
| *** tbachman has joined #openstack-nova | 01:46 | |
| *** yamamoto has joined #openstack-nova | 01:47 | |
| *** tpatil has joined #openstack-nova | 01:47 | |
| *** ijw has quit IRC | 01:47 | |
| *** chyka has joined #openstack-nova | 01:49 | |
| *** markvoelker has joined #openstack-nova | 01:52 | |
| *** yamamoto has quit IRC | 01:53 | |
| *** chyka has quit IRC | 01:53 | |
| *** yamahata has quit IRC | 01:55 | |
| *** sdake has quit IRC | 02:04 | |
| *** zhenq has quit IRC | 02:05 | |
| *** Tom has joined #openstack-nova | 02:05 | |
| *** mingyu has quit IRC | 02:06 | |
| *** andymccr has quit IRC | 02:06 | |
| *** rha has quit IRC | 02:06 | |
| *** lyarwood has quit IRC | 02:06 | |
| *** mingyu has joined #openstack-nova | 02:06 | |
| *** mtreinish has quit IRC | 02:08 | |
| *** lyarwood has joined #openstack-nova | 02:08 | |
| *** jistr has quit IRC | 02:08 | |
| *** john5223_ has quit IRC | 02:08 | |
| *** stephenfin has quit IRC | 02:09 | |
| *** phuongnh has joined #openstack-nova | 02:09 | |
| *** jistr has joined #openstack-nova | 02:10 | |
| *** Tom has quit IRC | 02:10 | |
| *** mtreinish has joined #openstack-nova | 02:10 | |
| *** stephenfin has joined #openstack-nova | 02:11 | |
| *** sergek__ has joined #openstack-nova | 02:11 | |
| *** sergek_ has quit IRC | 02:12 | |
| *** logan- has quit IRC | 02:12 | |
| *** sergek__ is now known as sergek_ | 02:12 | |
| *** sdake has joined #openstack-nova | 02:13 | |
| *** andymccr has joined #openstack-nova | 02:13 | |
| *** rha has joined #openstack-nova | 02:13 | |
| *** rha has quit IRC | 02:13 | |
| *** rha has joined #openstack-nova | 02:13 | |
| *** sdake is now known as Guest2849 | 02:13 | |
| *** logan- has joined #openstack-nova | 02:15 | |
| *** TuanLA has joined #openstack-nova | 02:16 | |
| *** crushil has quit IRC | 02:17 | |
| *** mingyu has quit IRC | 02:18 | |
| *** litao__ has joined #openstack-nova | 02:22 | |
| *** hughsaunders has quit IRC | 02:22 | |
| *** crushil has joined #openstack-nova | 02:22 | |
| *** mingyu has joined #openstack-nova | 02:22 | |
| *** McNinja has quit IRC | 02:23 | |
| *** toan has quit IRC | 02:23 | |
| *** mgagne has quit IRC | 02:23 | |
| *** comstud has quit IRC | 02:24 | |
| *** mgagne has joined #openstack-nova | 02:24 | |
| *** McNinja has joined #openstack-nova | 02:24 | |
| *** mgagne is now known as Guest66098 | 02:24 | |
| *** hughsaunders has joined #openstack-nova | 02:25 | |
| *** markvoelker has quit IRC | 02:27 | |
| *** toan has joined #openstack-nova | 02:28 | |
| *** takashin has quit IRC | 02:29 | |
| *** takashin has joined #openstack-nova | 02:29 | |
| *** jwcroppe has joined #openstack-nova | 02:30 | |
| *** litao___ has joined #openstack-nova | 02:30 | |
| *** diegows_ has joined #openstack-nova | 02:32 | |
| *** dave-mccowan has quit IRC | 02:33 | |
| *** litao__ has quit IRC | 02:35 | |
| *** diegows has quit IRC | 02:35 | |
| *** litao___ is now known as litao__ | 02:35 | |
| *** sapd has joined #openstack-nova | 02:38 | |
| *** sapd has quit IRC | 02:38 | |
| *** adreznec has quit IRC | 02:38 | |
| *** ericyoung has quit IRC | 02:38 | |
| *** comstud has joined #openstack-nova | 02:39 | |
| *** vivsoni has joined #openstack-nova | 02:39 | |
| *** mhenkel has quit IRC | 02:39 | |
| *** adreznec has joined #openstack-nova | 02:40 | |
| *** mhenkel has joined #openstack-nova | 02:40 | |
| *** ericyoung has joined #openstack-nova | 02:41 | |
| vivsoni | i am trying to attach FC, then attach ISCSI volume to nova instance.. while attaching it create a proper vlun entries in /dev/disk/by-pth/ | 02:41 |
| vivsoni | but after detaching the iscsi volume some of the lun entries are instact in /dev/disk/by-path directory | 02:41 |
| vivsoni | it would be great if someone can point me to code, from where the lun entries are created in /dev/disk/by-path | 02:44 |
| vivsoni | i suspect some 'iscsiadm' or similar 'rescan' command is triggered and due to which the lun entries are create in /dev/disk/by-path... please correct me if i am worng | 02:45 |
| vivsoni | thanks !!! | 02:45 |
| *** baoli has joined #openstack-nova | 02:46 | |
| *** Tom_ has joined #openstack-nova | 02:49 | |
| *** crushil has quit IRC | 02:52 | |
| *** crushil has joined #openstack-nova | 02:53 | |
| *** nicolasbock_ has quit IRC | 03:00 | |
| *** nicolasbock has quit IRC | 03:01 | |
| *** tpatil has quit IRC | 03:03 | |
| *** edmondsw has joined #openstack-nova | 03:04 | |
| *** edmondsw has quit IRC | 03:09 | |
| *** felipemonteiro_ has joined #openstack-nova | 03:12 | |
| *** markvoelker has joined #openstack-nova | 03:24 | |
| *** takashin has quit IRC | 03:25 | |
| *** links has joined #openstack-nova | 03:32 | |
| *** mriedem has joined #openstack-nova | 03:35 | |
| mriedem | jaypipes: dansmith: i did it! http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | 03:37 |
| mriedem | reproduced that 409 during claim resources in the scheduler, creating 1000 instances at once | 03:37 |
| *** udesale has joined #openstack-nova | 03:38 | |
| *** udesale has quit IRC | 03:38 | |
| *** penick has quit IRC | 03:38 | |
| mriedem | retried 18 times across that 1000 | 03:40 |
| mriedem | one of those poor saps just couldn't hack it | 03:40 |
| mriedem | need to run with https://review.openstack.org/#/c/507705/2/nova/scheduler/client/report.py to find out which one i guess | 03:40 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Log consumer uuid when retrying claims in the scheduler https://review.openstack.org/509961 | 03:41 |
| *** mriedem has quit IRC | 03:42 | |
| *** gouthamr has quit IRC | 03:42 | |
| *** udesale has joined #openstack-nova | 03:42 | |
| *** manasm has joined #openstack-nova | 03:45 | |
| *** hongbin has quit IRC | 03:45 | |
| *** chyka has joined #openstack-nova | 03:46 | |
| *** chyka has quit IRC | 03:50 | |
| *** crushil has quit IRC | 03:52 | |
| *** baoli has quit IRC | 03:53 | |
| *** crushil has joined #openstack-nova | 03:53 | |
| *** gbarros has quit IRC | 03:53 | |
| *** markvoelker has quit IRC | 03:57 | |
| *** gjayavelu has joined #openstack-nova | 04:02 | |
| *** penick has joined #openstack-nova | 04:05 | |
| *** aloga has quit IRC | 04:07 | |
| *** yamamoto has joined #openstack-nova | 04:15 | |
| *** bnemec has quit IRC | 04:16 | |
| *** yamahata has joined #openstack-nova | 04:19 | |
| *** jaosorior has joined #openstack-nova | 04:21 | |
| *** armax has quit IRC | 04:23 | |
| *** claudiub has joined #openstack-nova | 04:31 | |
| *** trungnv has joined #openstack-nova | 04:42 | |
| openstackgerrit | melanie witt proposed openstack/nova master: Improve the CellDatabases test fixture and usage https://review.openstack.org/508432 | 04:43 |
| openstackgerrit | melanie witt proposed openstack/nova master: Target context for build notification in conductor https://review.openstack.org/509967 | 04:44 |
| openstackgerrit | melanie witt proposed openstack/nova master: Elevate existing RequestContext to get bandwidth usage https://review.openstack.org/509968 | 04:44 |
| *** felipemonteiro_ has quit IRC | 04:49 | |
| *** edmondsw has joined #openstack-nova | 04:52 | |
| *** markvoelker has joined #openstack-nova | 04:54 | |
| *** udesale has quit IRC | 04:55 | |
| *** edmondsw has quit IRC | 04:57 | |
| *** psachin has joined #openstack-nova | 05:05 | |
| *** lpetrut_ has joined #openstack-nova | 05:06 | |
| *** esberglu has joined #openstack-nova | 05:09 | |
| *** hanish has joined #openstack-nova | 05:10 | |
| hanish | one of my compute node is disabled due to 10 vm launch failing, how can i recover that node | 05:11 |
| *** esberglu has quit IRC | 05:13 | |
| *** jwcroppe has quit IRC | 05:15 | |
| *** ratailor has joined #openstack-nova | 05:16 | |
| *** udesale has joined #openstack-nova | 05:18 | |
| Tengu | you must re-enable nova agent, hanish | 05:20 |
| *** takashin has joined #openstack-nova | 05:21 | |
| hanish | @Tengu: i restarted nova-compute agent on compute node, but still i facing the issue | 05:25 |
| *** markvoelker has quit IRC | 05:27 | |
| *** penick has quit IRC | 05:27 | |
| Tengu | hmmm nope, not via systemctl | 05:29 |
| Tengu | there's an openstack command for that, in order to re-enable it at openstack level | 05:29 |
| takashin | 05:30 | |
| Tengu | hanish: there's something like nova service-list - you should see it's disabled for your host. | 05:32 |
| Tengu | then you have nova service-enable | 05:32 |
| Tengu | hanish: I don't remember the "openstack unified" command for those. | 05:32 |
| *** elod has joined #openstack-nova | 05:37 | |
| *** crushil has quit IRC | 05:37 | |
| hanish | tengu: thanks | 05:38 |
| *** gjayavelu has quit IRC | 05:47 | |
| *** sree has joined #openstack-nova | 05:56 | |
| *** spectr has joined #openstack-nova | 05:56 | |
| *** vvargaszte has joined #openstack-nova | 05:58 | |
| *** jwcroppe has joined #openstack-nova | 06:03 | |
| *** trinaths has joined #openstack-nova | 06:06 | |
| *** lpetrut_ has quit IRC | 06:06 | |
| *** hemna_ has joined #openstack-nova | 06:07 | |
| *** mdnadeem has joined #openstack-nova | 06:09 | |
| *** Oku_OS-away is now known as Oku_OS | 06:09 | |
| *** hemna__ has quit IRC | 06:11 | |
| *** jwcroppe has quit IRC | 06:14 | |
| *** clayton has quit IRC | 06:16 | |
| Tengu | hanish: did it do the trick? | 06:17 |
| *** rcernin has joined #openstack-nova | 06:17 | |
| *** clayton has joined #openstack-nova | 06:18 | |
| hanish | Tengu: thanks, yes it worked. | 06:20 |
| *** lajoskatona has joined #openstack-nova | 06:21 | |
| Tengu | hanish: good :). | 06:23 |
| Tengu | I had that kind of issue earlier with tripleO. | 06:23 |
| *** markvoelker has joined #openstack-nova | 06:24 | |
| *** pcaruana has joined #openstack-nova | 06:24 | |
| *** karthiks has joined #openstack-nova | 06:28 | |
| *** udesale has quit IRC | 06:31 | |
| *** udesale has joined #openstack-nova | 06:33 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova-specs master: Abort Cold Migration https://review.openstack.org/334732 | 06:36 |
| *** vvargaszte has quit IRC | 06:36 | |
| *** dtantsur|afk has quit IRC | 06:40 | |
| *** dtantsur has joined #openstack-nova | 06:40 | |
| *** rcernin has quit IRC | 06:41 | |
| *** links has quit IRC | 06:42 | |
| *** rcernin has joined #openstack-nova | 06:43 | |
| *** trinaths has quit IRC | 06:44 | |
| *** trinaths has joined #openstack-nova | 06:44 | |
| *** avolkov has joined #openstack-nova | 06:45 | |
| takashin | gmann: cdent: I have modified the spec for "Abort cold migration" function. Would you review https://review.openstack.org/#/c/334732/ again? | 06:50 |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | 06:50 |
| *** vks1 has joined #openstack-nova | 06:52 | |
| *** sahid has joined #openstack-nova | 06:54 | |
| *** swamireddy has quit IRC | 06:55 | |
| *** esberglu has joined #openstack-nova | 06:57 | |
| *** markvoelker has quit IRC | 06:58 | |
| *** links has joined #openstack-nova | 06:59 | |
| *** esberglu has quit IRC | 07:02 | |
| *** swamireddy has joined #openstack-nova | 07:04 | |
| *** jwcroppe has joined #openstack-nova | 07:05 | |
| *** zhenq has joined #openstack-nova | 07:10 | |
| *** zhenq has quit IRC | 07:10 | |
| *** andreas_s has joined #openstack-nova | 07:21 | |
| *** chyka has joined #openstack-nova | 07:22 | |
| *** tesseract has joined #openstack-nova | 07:25 | |
| *** chyka has quit IRC | 07:26 | |
| *** aloga has joined #openstack-nova | 07:27 | |
| *** jwcroppe has quit IRC | 07:27 | |
| *** hanish has quit IRC | 07:29 | |
| *** lajoskatona has quit IRC | 07:31 | |
| *** lajoskatona has joined #openstack-nova | 07:36 | |
| *** links has quit IRC | 07:46 | |
| *** jpena|off is now known as jpena | 07:50 | |
| *** esberglu has joined #openstack-nova | 07:52 | |
| *** esberglu has quit IRC | 07:52 | |
| *** esberglu has joined #openstack-nova | 07:53 | |
| *** gszasz has joined #openstack-nova | 07:54 | |
| *** markus_z has joined #openstack-nova | 07:54 | |
| *** markvoelker has joined #openstack-nova | 07:54 | |
| *** mlakat has quit IRC | 07:56 | |
| *** esberglu has quit IRC | 07:57 | |
| *** aloga has quit IRC | 07:58 | |
| *** links has joined #openstack-nova | 07:59 | |
| zioproto | hello :) Working on Newton I have a region in my cloud where openstack usage list goes in stacktrace. Instance not found | 08:24 |
| zioproto | looking at the nova bugs I did not find anything usefull | 08:24 |
| zioproto | tracking my logs it looks like this is broken since I upgraded to Newton | 08:25 |
| zioproto | dansmith: usually you like these database stories :) | 08:25 |
| *** bauzas is now known as bauwser | 08:26 | |
| bauwser | zioproto: stacktrace ? | 08:26 |
| bauwser | zioproto: by Newton, we began to use the API DB | 08:27 |
| zioproto | https://pastebin.com/gtJbutvi | 08:27 |
| zioproto | it is funny because this happens only in 1 production region | 08:28 |
| *** mlakat has joined #openstack-nova | 08:28 | |
| zioproto | I have the same setup on dev/staging/prod and there it works | 08:28 |
| *** markvoelker has quit IRC | 08:28 | |
| zioproto | my feeling is that there is a broken database entry, or something specific to that region that breaks it | 08:28 |
| *** edmondsw has joined #openstack-nova | 08:28 | |
| zioproto | of course openstack server show 72afef44-1a4b-46e5-8cbc-7bd4f0eb31ff gives me a instance not found as well | 08:29 |
| zioproto | what other tables should I dig to look for this uuid ? | 08:29 |
| *** lucas-afk is now known as lucasagomes | 08:29 | |
| gmann | takashin: thanks. i will check soon. | 08:32 |
| *** edmondsw has quit IRC | 08:33 | |
| takashin | gmann: Thanks in advance. | 08:33 |
| *** psachin has quit IRC | 08:34 | |
| *** takashin has left #openstack-nova | 08:34 | |
| *** jangutter_ has quit IRC | 08:35 | |
| *** tetsuro has quit IRC | 08:36 | |
| bauwser | zioproto: are you aware of those commands when you deploy a Newton cloud ? https://docs.openstack.org/nova/latest/cli/nova-manage.html#man-page-cells-v2 | 08:39 |
| zioproto | bauwser: I think I did this when I upgraded to Mitaka | 08:40 |
| zioproto | to split the nova db into two DBs | 08:40 |
| bauwser | zioproto: if I were you, I'd be looking at the nova_api DB for the instance_mappings table | 08:41 |
| bauwser | zioproto: and check which cell is for the instance UUID | 08:41 |
| zioproto | I go have a look | 08:41 |
| bauwser | zioproto: then looking at host_mappings if we have a cell record for each host in it | 08:42 |
| *** derekh has joined #openstack-nova | 08:46 | |
| *** jwcroppe has joined #openstack-nova | 08:49 | |
| *** aloga has joined #openstack-nova | 08:51 | |
| *** vks1 has quit IRC | 08:52 | |
| zioproto | bauwser: my host_mappings table is empty, is that a bad sign ? | 08:52 |
| bauwser | zioproto: indeed, how many hosts do you have? | 08:53 |
| zioproto | you mean compute hosts ? | 08:53 |
| zioproto | like hundreds | 08:53 |
| zioproto | looks like we upgraded to newton without doing this cells housekeeping | 08:56 |
| zioproto | this was mandatory at this point ? | 08:56 |
| zioproto | to have a cell0 database ? | 08:56 |
| *** jwcroppe has quit IRC | 08:58 | |
| bauwser | zioproto: I don't exactly remember when we used the api DB for getting the instance list | 09:01 |
| bauwser | zioproto: but you should definitely map the hosts | 09:02 |
| *** gszasz has quit IRC | 09:04 | |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | 09:06 |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | 09:07 |
| *** vks1 has joined #openstack-nova | 09:10 | |
| openstackgerrit | zhangyangyang proposed openstack/nova master: Move libvirts qemu-img support to privsep https://review.openstack.org/507848 | 09:15 |
| *** sambetts_ is now known as sambetts | 09:16 | |
| *** ociuhandu has joined #openstack-nova | 09:18 | |
| *** psachin has joined #openstack-nova | 09:18 | |
| *** sahid has quit IRC | 09:18 | |
| *** sahid has joined #openstack-nova | 09:19 | |
| *** ociuhandu has quit IRC | 09:21 | |
| *** markvoelker has joined #openstack-nova | 09:25 | |
| *** spectr has quit IRC | 09:26 | |
| *** spectr has joined #openstack-nova | 09:34 | |
| *** spectr has quit IRC | 09:34 | |
| *** spectr has joined #openstack-nova | 09:36 | |
| *** spectr has quit IRC | 09:36 | |
| *** spectr has joined #openstack-nova | 09:37 | |
| *** gszasz has joined #openstack-nova | 09:37 | |
| *** spectr has quit IRC | 09:38 | |
| *** manasm has quit IRC | 09:38 | |
| *** spectr has joined #openstack-nova | 09:40 | |
| *** spectr has quit IRC | 09:40 | |
| *** spectr has joined #openstack-nova | 09:41 | |
| *** spectr has quit IRC | 09:43 | |
| *** spectr has joined #openstack-nova | 09:43 | |
| *** lajoskatona has quit IRC | 09:48 | |
| *** lajoskatona has joined #openstack-nova | 09:49 | |
| *** spectr has quit IRC | 09:56 | |
| *** spectr has joined #openstack-nova | 09:57 | |
| *** spectr has quit IRC | 09:57 | |
| *** spectr has joined #openstack-nova | 09:58 | |
| *** markvoelker has quit IRC | 09:58 | |
| *** manasm has joined #openstack-nova | 10:04 | |
| *** trungnv has quit IRC | 10:04 | |
| *** stephenfin is now known as finucannot | 10:05 | |
| *** TuanLA has quit IRC | 10:08 | |
| *** phuongnh has quit IRC | 10:09 | |
| *** edmondsw has joined #openstack-nova | 10:16 | |
| *** trinaths has quit IRC | 10:17 | |
| *** edmondsw has quit IRC | 10:21 | |
| *** mingyu has quit IRC | 10:21 | |
| *** andreas_s has quit IRC | 10:24 | |
| *** andreas_s has joined #openstack-nova | 10:24 | |
| *** lajoskatona has quit IRC | 10:25 | |
| *** sdague has joined #openstack-nova | 10:28 | |
| *** andreas_s has quit IRC | 10:29 | |
| *** Dinesh_Bhor has quit IRC | 10:35 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Transform instance.exists notification https://review.openstack.org/403660 | 10:35 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add sample test for instance audit https://review.openstack.org/480955 | 10:35 |
| *** jwcroppe has joined #openstack-nova | 10:39 | |
| *** andreas_s has joined #openstack-nova | 10:40 | |
| *** jwcroppe has quit IRC | 10:47 | |
| *** hieulq has quit IRC | 10:47 | |
| *** tbachman has quit IRC | 10:49 | |
| *** yamamoto has quit IRC | 10:53 | |
| *** yamamoto has joined #openstack-nova | 10:54 | |
| *** yamamoto has quit IRC | 10:54 | |
| *** markvoelker has joined #openstack-nova | 10:56 | |
| *** chyka has joined #openstack-nova | 10:58 | |
| *** ijw has joined #openstack-nova | 11:00 | |
| *** chyka has quit IRC | 11:02 | |
| openstackgerrit | Merged openstack/nova master: Blacklist test_extend_attached_volume from cells v1 job https://review.openstack.org/509907 | 11:03 |
| *** ijw has quit IRC | 11:04 | |
| *** nicolasbock has joined #openstack-nova | 11:10 | |
| *** nicolasbock_ has joined #openstack-nova | 11:10 | |
| *** smatzek has joined #openstack-nova | 11:11 | |
| *** nicolasbock_ has quit IRC | 11:15 | |
| *** nicolasbock has quit IRC | 11:15 | |
| gibi | hm https://review.openstack.org/509907 has been merged so I guess it is recheck time | 11:17 |
| *** dave-mccowan has joined #openstack-nova | 11:20 | |
| *** mingyu has joined #openstack-nova | 11:22 | |
| *** yamamoto has joined #openstack-nova | 11:25 | |
| zioproto | bauwser: in nova-manage cell_v2 simple_cell_setup [--transport-url <transport_url>] how does <transport_url> look like ? | 11:25 |
| *** mingyu has quit IRC | 11:26 | |
| *** vks1 has quit IRC | 11:26 | |
| *** nicolasbock has joined #openstack-nova | 11:27 | |
| *** nicolasbock_ has joined #openstack-nova | 11:27 | |
| zioproto | ok I found it here https://docs.openstack.org/nova/latest/user/cells.html | 11:28 |
| *** nicolasbock_ has quit IRC | 11:29 | |
| *** nicolasbock has quit IRC | 11:29 | |
| *** diga has quit IRC | 11:29 | |
| *** markvoelker has quit IRC | 11:29 | |
| *** takedakn has joined #openstack-nova | 11:34 | |
| *** efried is now known as fried_rice | 11:35 | |
| *** mingyu has joined #openstack-nova | 11:38 | |
| *** vks1 has joined #openstack-nova | 11:38 | |
| *** cdent has joined #openstack-nova | 11:41 | |
| *** lajoskatona has joined #openstack-nova | 11:41 | |
| *** baoli has joined #openstack-nova | 11:43 | |
| fried_rice | jaypipes yt? Wanted to brainstorm a couple of edge cases. | 11:43 |
| *** artom has quit IRC | 11:43 | |
| *** baoli has quit IRC | 11:48 | |
| *** lucasagomes is now known as lucas-hungry | 11:56 | |
| *** tbachman has joined #openstack-nova | 11:58 | |
| *** spectr has quit IRC | 12:02 | |
| *** tbachman has quit IRC | 12:03 | |
| *** spectr has joined #openstack-nova | 12:03 | |
| *** edmondsw has joined #openstack-nova | 12:05 | |
| *** jpena is now known as jpena|lunch | 12:05 | |
| gibi | could a second core look at this code removal patch? https://review.openstack.org/#/c/505164/ | 12:06 |
| *** tbachman has joined #openstack-nova | 12:09 | |
| *** edmondsw has quit IRC | 12:09 | |
| *** vks1 has quit IRC | 12:11 | |
| *** baoli has joined #openstack-nova | 12:12 | |
| *** zhenq has joined #openstack-nova | 12:15 | |
| *** zhenq has quit IRC | 12:15 | |
| gibi | bauwser: hi! you can still support my embarrassment in https://review.openstack.org/#/c/509750 if you would like to :) | 12:18 |
| * cdent gives fried_rice some soy sauce | 12:19 | |
| *** litao__ has quit IRC | 12:19 | |
| fried_rice | Gluten free, if you please. | 12:19 |
| fried_rice | cdent Perhaps you'd be willing to give some feedback on these edge cases that came to me in my sleep (or lack thereof) last night. | 12:20 |
| * cdent swaps out for some tamari | 12:20 | |
| cdent | I can try, but these cases keep confusing me, but since it will probably help to talk about it, shoot. | 12:21 |
| fried_rice | If I ask for CUSTOM_FOO:4, is it *ever* legal for placement to give me 3 CUSTOM_FOOs from one RP and 1 from a different RP? | 12:21 |
| *** MVenesio has joined #openstack-nova | 12:21 | |
| *** nicolasbock has joined #openstack-nova | 12:22 | |
| cdent | In the fundament case of resource providers and inventory, no | 12:22 |
| cdent | the request is a for a chunk of size 4 | 12:22 |
| fried_rice | Here's the thing: it makes total sense for the answer to be "yes" for something like VFs on separate PFs, all other traits being equal. Because e.g. what if I ask for 4 VFs and I've only got 2 on each PF? | 12:22 |
| cdent | right, I was just going to say “vfs make that weird" | 12:23 |
| fried_rice | But even there, if I'm asking for e.g. bandwidth inventory along with my VF count, how do I split that up? | 12:23 |
| cdent | and also why I said “fundament[al]” because I think nested makes some of these decisions less clear | 12:23 |
| cdent | (btw, it is great that you are exploring this stuff) | 12:24 |
| fried_rice | And it makes *no* sense for something like DISK_GB. If I ask for 4, I don't want you giving me 1GB from each of 4 providers. | 12:24 |
| * cdent nods | 12:24 | |
| *** markus_z has quit IRC | 12:24 | |
| *** esberglu has joined #openstack-nova | 12:24 | |
| *** esberglu has quit IRC | 12:24 | |
| fried_rice | So let's put that to bed and say Nay. | 12:24 |
| *** esberglu has joined #openstack-nova | 12:24 | |
| *** esberglu has quit IRC | 12:24 | |
| cdent | this issue may be why in some conversations VFs have been proposed as resource providers | 12:24 |
| fried_rice | eek | 12:24 |
| cdent | ikr | 12:25 |
| fried_rice | I mean, I guess you could do that, in the "pre-create" case like current VF passthrough does. | 12:25 |
| *** pchavva has joined #openstack-nova | 12:25 | |
| fried_rice | No good in the "create VF dynamically" case of the future. | 12:25 |
| fried_rice | cdent Okay, so next thing: | 12:25 |
| fried_rice | Back to the VF scenario, if I ask for VF:2,BANDWIDTH:20000 | 12:26 |
| fried_rice | I'm answering my own question. | 12:26 |
| zioproto | bauwser: still around here ? | 12:26 |
| *** markvoelker has joined #openstack-nova | 12:26 | |
| fried_rice | Those are total chunks on the RP. | 12:26 |
| zioproto | it comes out that the uuid from the stacktrace is pretty unique | 12:26 |
| fried_rice | Placement doesn't care how they're going to be split up | 12:26 |
| zioproto | it is the only instance in the cloud that matches this query | 12:27 |
| zioproto | select * from instances where vm_state="shelved_offloaded" | 12:27 |
| fried_rice | That's up to the virt driver once it gets that information (which we still don't have a way of doing yet - discussion Monday) | 12:27 |
| cdent | yup | 12:27 |
| fried_rice | So I guess the op would have to assume each VF will get 10000. And if they want a different split, they can specify them as separate request numbers (per the spec I'm composing). | 12:27 |
| fried_rice | Cool cool. | 12:28 |
| *** jwcroppe has joined #openstack-nova | 12:28 | |
| fried_rice | cdent Thanks for sounding-boarding. | 12:28 |
| cdent | you’re welcome | 12:28 |
| fried_rice | (Hopefully it wasn't too much like water-boarding) | 12:28 |
| cdent | not today | 12:28 |
| *** spectr has quit IRC | 12:28 | |
| *** Mr_Smurf has left #openstack-nova | 12:29 | |
| fried_rice | jaypipes I don't need you anymore. | 12:29 |
| *** jdwidari has joined #openstack-nova | 12:30 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add snapshot id to the snapshot notifications https://review.openstack.org/453077 | 12:30 |
| *** markvoelker has quit IRC | 12:31 | |
| fried_rice | cdent Oh, BUT if I ask for CUSTOM_FOO:3 and CUSTOM_BAR:2, placement *could* give me those guys from different RPs | 12:31 |
| *** markvoelker has joined #openstack-nova | 12:31 | |
| fried_rice | That is, 3 CUSTOM_FOOs from RP1 and 2 CUSTOM_BARs from RP2 | 12:31 |
| cdent | fried_rice: it depends on how you are forming your request and what sort of sharing or nested relationship there is between rp1 and rp2 | 12:32 |
| *** baoli has quit IRC | 12:32 | |
| jaypipes | fried_rice: heh | 12:32 |
| *** manasm has quit IRC | 12:33 | |
| cdent | in a pre-nested universe rp1 and rp2 would need to associated in the same aggregate | 12:33 |
| *** jaypipes is now known as leakypipes | 12:33 | |
| fried_rice | leakypipes I take it back; feel free to weigh in on the latest craziness | 12:33 |
| * leakypipes reads back | 12:33 | |
| fried_rice | cdent Same aggregate, interesting. | 12:33 |
| fried_rice | What if there are no aggregate associations? | 12:33 |
| cdent | fried_rice: same _placement_ aggregate (which is not the same as a nova aggregate) | 12:34 |
| fried_rice | Oh, I get it, because that's the only way you don't get VCPU from one host and MEM_MB from another. | 12:34 |
| * cdent nods | 12:34 | |
| fried_rice | Does RT implicitly create aggregates today? | 12:34 |
| cdent | not yet, shared is only implemented on the placement side so far, not the nova side | 12:35 |
| cdent | and it got punted down the priority stack | 12:35 |
| fried_rice | Well. | 12:36 |
| fried_rice | That's going to get interesting. | 12:36 |
| *** jwcroppe has quit IRC | 12:36 | |
| fried_rice | So *with* nested but *without* shared, we would have to enforce that semantic as "same root provider". | 12:37 |
| * cdent nods | 12:37 | |
| fried_rice | FYI, the semantic I'm wrestling with is how resources are bound together (or not) with this numbered syntax thingy. | 12:38 |
| *** ratailor has quit IRC | 12:38 | |
| *** spectr has joined #openstack-nova | 12:39 | |
| fried_rice | And what I've been leaning towards is: When you use the (existing) "un-numbered" deal, the rule is "same root provider". But when you use a "numbered" deal, the rule is "same *provider*". | 12:39 |
| fried_rice | cdent leakypipes ^ | 12:39 |
| cdent | that’s kind of been my assunption too | 12:39 |
| *** nicolasbock has quit IRC | 12:39 | |
| fried_rice | We've gotta have a way to express that second thing; otherwise I might wind up with my VF inventory from one PF and my bandwidth inventory from another. | 12:40 |
| fried_rice | But | 12:40 |
| fried_rice | Do we need to be able to do that first thing at all? | 12:40 |
| fried_rice | Just thinking out loud here. I believe the answer is "yes". Because it's the most flexible and simplest UX. | 12:41 |
| cdent | I think “yes” is correct, because we may not care, we just want some stuff | 12:41 |
| *** erlon has joined #openstack-nova | 12:41 | |
| fried_rice | Cool deal. And I can't think of a case you couldn't cover between those two options. | 12:41 |
| fried_rice | Except the ones for which we need aggregates. | 12:41 |
| fried_rice | Whereupon the "same root provider" rule extends to "same root provider *or* aggregate" | 12:42 |
| fried_rice | Hum, (how) does aggregate-ness propagate around a tree? | 12:42 |
| * cdent doesn’t know | 12:43 | |
| fried_rice | Was gonna say, if it applies to a whole tree, the above reduces to "same aggregate". But that doesn't cover the case where we didn't actually declare any aggregates. | 12:44 |
| fried_rice | I'm inclined to think of an aggregate as kind of a special case of a trait. | 12:44 |
| *** alexchadin has joined #openstack-nova | 12:45 | |
| cdent | I’m not entirely following that logic? | 12:45 |
| fried_rice | In which case, aggregates should propagate downward (from parent to child) like traits. | 12:45 |
| fried_rice | Yeah, it's not fully formed in my bean. Something like... | 12:46 |
| leakypipes | fried_rice: ack | 12:46 |
| *** takedakn has quit IRC | 12:47 | |
| fried_rice | An aggregate is an "implicit trait" that you don't actually ask for, but that we kinda add to the request as we go along. That is, once we pick a RP for one piece of the request, we implicitly add its invisible-aggregate-trait for purposes of the rest of the request, so we only get the rest of the inventory from RPs with that same invisible-aggregate-trait. | 12:47 |
| fried_rice | leakypipes What were you acking? | 12:48 |
| leakypipes | fried_rice: ack on thing above about when you use a numbered deal, that means same provider. | 12:48 |
| fried_rice | leakypipes Cool | 12:48 |
| leakypipes | fried_rice: aggregates don't have traits. aggregates are nothing but grouping mechanisms | 12:48 |
| fried_rice | yah, I get that, but am I off base logically thinking of them as described ^x4? | 12:49 |
| *** baoli has joined #openstack-nova | 12:49 | |
| *** nicolasbock has joined #openstack-nova | 12:51 | |
| cdent | leakypipes: am I right that the nested stack awaits resolutino of the no-orm stack (want to mention that in the rp update if it is in fact true)? | 12:51 |
| *** zhenq has joined #openstack-nova | 12:52 | |
| fried_rice | cdent That's what I have been led to understand. | 12:54 |
| cdent | ✔ | 12:54 |
| fried_rice | It also occurred to me that I hadn't seen the code on the placement side that handles tree-ness for GET /allocation_candidates | 12:54 |
| fried_rice | Like, the SQL magic that does downward trait propagation and suchlike. | 12:55 |
| fried_rice | At least, I don't *think* I've seen that. | 12:55 |
| cdent | doesn’t exist yet as far as I’m aware? | 12:55 |
| fried_rice | okay. | 12:55 |
| cdent | parts of it will be in https://review.openstack.org/#/q/topic:bp/nested-resource-providers+status:open | 12:55 |
| fried_rice | cdent Yeah, I reviewed that stack and don't recall seeing the bits I'm talking about. | 12:56 |
| *** takedakn has joined #openstack-nova | 12:58 | |
| *** READ10 has joined #openstack-nova | 12:59 | |
| *** jpena|lunch is now known as jpena | 13:00 | |
| *** edleafe is now known as figleaf | 13:00 | |
| *** tbachman has quit IRC | 13:02 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/os-vif stable/ocata: Updated from global requirements https://review.openstack.org/490256 | 13:02 |
| *** crushil has joined #openstack-nova | 13:02 | |
| *** vladikr has joined #openstack-nova | 13:03 | |
| *** psachin has quit IRC | 13:05 | |
| *** nicolasbock has quit IRC | 13:05 | |
| *** udesale has quit IRC | 13:05 | |
| *** nicolasbock has joined #openstack-nova | 13:05 | |
| *** takedakn has quit IRC | 13:05 | |
| *** takedakn has joined #openstack-nova | 13:07 | |
| *** manasm has joined #openstack-nova | 13:10 | |
| *** eharney has joined #openstack-nova | 13:12 | |
| *** bnemec has joined #openstack-nova | 13:12 | |
| *** priteau has joined #openstack-nova | 13:13 | |
| openstackgerrit | Matthew Booth proposed openstack/nova-specs master: Virtual instance rescue with stable disk devices https://review.openstack.org/510106 | 13:14 |
| *** lbragstad has joined #openstack-nova | 13:15 | |
| mdbooth | lyarwood: ^^^ | 13:15 |
| mdbooth | lyarwood: Although it's your spec with only request REST API changes | 13:16 |
| *** ygl has joined #openstack-nova | 13:16 | |
| *** takedakn has quit IRC | 13:18 | |
| cdent | https://thebritishdrea.com/?text=Nested+providers+will+fix+That | 13:18 |
| *** yamamoto has quit IRC | 13:21 | |
| *** mingyu has quit IRC | 13:21 | |
| *** gbarros has joined #openstack-nova | 13:21 | |
| mdbooth | cdent: Lol | 13:24 |
| *** mingyu has joined #openstack-nova | 13:25 | |
| *** cdent has quit IRC | 13:26 | |
| *** ygl has quit IRC | 13:27 | |
| *** dansmith is now known as superdan | 13:28 | |
| *** jmlowe_ has quit IRC | 13:29 | |
| *** mingyu has quit IRC | 13:30 | |
| *** jmlowe has joined #openstack-nova | 13:30 | |
| *** jmlowe_ has joined #openstack-nova | 13:33 | |
| *** trinaths has joined #openstack-nova | 13:33 | |
| *** spectr has quit IRC | 13:35 | |
| *** jmlowe has quit IRC | 13:35 | |
| *** yamamoto has joined #openstack-nova | 13:35 | |
| *** lbragstad has quit IRC | 13:36 | |
| *** lbragstad has joined #openstack-nova | 13:36 | |
| *** spectr has joined #openstack-nova | 13:36 | |
| *** artom has joined #openstack-nova | 13:38 | |
| *** trinaths1 has joined #openstack-nova | 13:38 | |
| *** mriedem has joined #openstack-nova | 13:39 | |
| *** gouthamr has joined #openstack-nova | 13:39 | |
| *** artom_ has joined #openstack-nova | 13:39 | |
| andreykurilin | mriedem: hi! Do you know anything merged recently which could affect pagination? | 13:40 |
| superdan | andreykurilin: pagination of instances? | 13:40 |
| andreykurilin | yes | 13:40 |
| superdan | andreykurilin: what are you seeing? | 13:40 |
| *** trinaths has quit IRC | 13:41 | |
| mriedem | duh duh duh | 13:41 |
| andreykurilin | superdan: marker doesn't work in some cases | 13:41 |
| superdan | andreykurilin: heh, any more detail than that? :) | 13:41 |
| mriedem | andreykurilin: have a failed job log? | 13:42 |
| mriedem | novaclient or rally? | 13:42 |
| andreykurilin | superdan: sure, just need to collect links:) | 13:42 |
| andreykurilin | give me a sec | 13:42 |
| *** artom has quit IRC | 13:42 | |
| *** lucas-hungry is now known as lucasagomes | 13:44 | |
| andreykurilin | rally gates are failing due to an issue with pagination. we use limit=-1 option from novaclient. It is designed to make an inf loop changing the marker until the response will include an empty list. For some reasons API ignores the market in some cases. I copy-pasted the code from novaclient and added some debug messages | 13:44 |
| andreykurilin | here is a ok execution - http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_16_892620 | 13:44 |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Add error notification for instance.interface_attach https://review.openstack.org/506643 | 13:45 |
| andreykurilin | and just after several seconds, there is one more execution (another iteration of the workload) and it stucks | 13:45 |
| andreykurilin | http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_19_441231 | 13:45 |
| andreykurilin | here is a code which I'm using for dedbugging purpose - https://review.openstack.org/#/c/509783/3/rally/plugins/openstack/scenarios/nova/utils.py | 13:45 |
| superdan | andreykurilin: is that implying that you get a page with a marker and you get back a page with the marker in it? | 13:46 |
| andreykurilin | it is equal to what we have in novaclient but with some debug message as I already mentione | 13:46 |
| andreykurilin | superdan: yes | 13:46 |
| andreykurilin | i think so | 13:46 |
| superdan | hrm | 13:47 |
| andreykurilin | but it happens not regulary. In some cases the page includes the marker, in others - no | 13:47 |
| superdan | what sort key are you using? | 13:47 |
| andreykurilin | superdan: due to migration to Zull v3 I cannot say when it had happend actually, but can assume that 2 days ago | 13:47 |
| andreykurilin | no sort keys | 13:47 |
| superdan | andreykurilin: yeah I know what changed, so no question there | 13:48 |
| andreykurilin | suyuperdan: here is a query http://logs.openstack.org/83/509783/3/check/gate-rally-dsvm-neutron-existing-users-rally/1a480ab/console.html#_2017-10-05_23_02_19_452283 | 13:48 |
| superdan | andreykurilin: okay so default sort | 13:48 |
| andreykurilin | just marker in query, nothing more | 13:48 |
| *** links has quit IRC | 13:48 | |
| *** manasm has quit IRC | 13:48 | |
| superdan | andreykurilin: are these instances from a num_instances=N type create operation? | 13:48 |
| *** lajoskatona has quit IRC | 13:48 | |
| superdan | andreykurilin: such that they probably have very similar create times? | 13:49 |
| *** awaugama has joined #openstack-nova | 13:49 | |
| andreykurilin | no | 13:49 |
| superdan | andreykurilin: no meaning they were created one at a time in a client loop? | 13:49 |
| andreykurilin | yes | 13:50 |
| andreykurilin | sec | 13:50 |
| superdan | and how many(ish)? | 13:50 |
| andreykurilin | superdan: there are 2 instances which acre created in one time (~1 sec), but from different threads and with different names | 13:50 |
| *** cdent has joined #openstack-nova | 13:50 | |
| superdan | andreykurilin: so you're literally paging through two instances? | 13:51 |
| andreykurilin | yes. just need to mention, that there are 2 cases and both failed. first one boot_and_list actions are performed twice in the same time. the second: list action performed once after both vms are booted | 13:52 |
| *** edmondsw has joined #openstack-nova | 13:53 | |
| *** lajoskatona has joined #openstack-nova | 13:53 | |
| superdan | andreykurilin: okay so limit=1 then? | 13:53 |
| *** jdwidari has quit IRC | 13:54 | |
| superdan | andreykurilin: and both instances are ACTIVE right? | 13:54 |
| *** jwcroppe has joined #openstack-nova | 13:55 | |
| *** READ10 has quit IRC | 13:57 | |
| bauwser | zioproto: sorry, I have a huge internal backlog to do | 13:57 |
| *** edmondsw has quit IRC | 13:57 | |
| *** smatzek has quit IRC | 13:59 | |
| *** sree has quit IRC | 13:59 | |
| andreykurilin | superdan: so there are 2 cases. The shared logging relates to the first case, but the behaviour of nova the same and for the second case. Let me dedscribe it more details. there are 2 threads which perfroms boot_and_list actions. the listing is performed right after the vm become active. Both threads are using the same user and tenant | 13:59 |
| zioproto | bauwser: no worries ! | 14:00 |
| *** sree has joined #openstack-nova | 14:00 | |
| *** smatzek has joined #openstack-nova | 14:00 | |
| andreykurilin | superdan: in this case the first thread performs list action successfully (with using limit=-1 option of novaclient) and the second thread fails | 14:00 |
| *** ratailor has joined #openstack-nova | 14:00 | |
| superdan | andreykurilin: and what does limit=-1 mean to novaclient? | 14:00 |
| mriedem | page until there is nothing returned i think | 14:01 |
| andreykurilin | yes | 14:01 |
| superdan | right but with what limit to the api? | 14:01 |
| superdan | no limit= default? | 14:01 |
| andreykurilin | no limit | 14:01 |
| mriedem | so default limit of 1000 | 14:01 |
| andreykurilin | yes | 14:01 |
| superdan | okay, so this really should get both instances in the first page, | 14:02 |
| superdan | try another with result[-1] and get an empty page, yes? | 14:02 |
| andreykurilin | `marker = result[-1]` gives the same page as previously with marker in it | 14:03 |
| superdan | right, I was describing what _should_ be happening | 14:03 |
| *** hongbin has joined #openstack-nova | 14:03 | |
| andreykurilin | yes | 14:03 |
| superdan | okay | 14:03 |
| superdan | I might have an idea of what is going on, but I need to do some experimentation | 14:04 |
| superdan | andreykurilin: in the meantime, can you alter that loop a bit just to see if it helps? | 14:04 |
| gibi | cburgess: hi! Is there any next step about https://blueprints.launchpad.net/nova/+spec/libvirt-virtio-set-queue-sizes I can look at / help with? | 14:04 |
| andreykurilin | superdan: sure | 14:04 |
| superdan | andreykurilin: can you set the sort_keys=['uuid'] | 14:04 |
| *** catintheroof has joined #openstack-nova | 14:05 | |
| *** sree has quit IRC | 14:05 | |
| *** slaweq has quit IRC | 14:05 | |
| superdan | although that really shouldn't matter since we're only iterating instances in a single cell db here | 14:05 |
| superdan | andreykurilin: and I can throw up a nova patch you can depends-on right? | 14:05 |
| superdan | andreykurilin: got a bug number for this yet? | 14:06 |
| andreykurilin | superdan: yes, we can do depends-on to check the fix. no, I do not have a bug report | 14:06 |
| *** catinthe_ has joined #openstack-nova | 14:06 | |
| *** catintheroof has quit IRC | 14:07 | |
| andreykurilin | superdan: superdan: made a patch to check sort_keys, but based on the queue of the zuul the results will be in several hours | 14:08 |
| *** archit has joined #openstack-nova | 14:09 | |
| andreykurilin | superdan: should I create a bug report? | 14:11 |
| superdan | andreykurilin: yeah please create a bug and I'll some debugging | 14:11 |
| superdan | sorry, I'm stuck on a call atm | 14:11 |
| andreykurilin | thanks | 14:11 |
| *** bnemec is now known as beekneemech | 14:11 | |
| *** spectr has quit IRC | 14:11 | |
| *** namnh has joined #openstack-nova | 14:12 | |
| *** alexchadin has quit IRC | 14:15 | |
| *** gouthamr has quit IRC | 14:16 | |
| *** felipemonteiro_ has joined #openstack-nova | 14:21 | |
| andreykurilin | superdan: https://bugs.launchpad.net/nova/+bug/1721791 | 14:21 |
| openstack | Launchpad bug 1721791 in OpenStack Compute (nova) "Pagination of instances works incorrect" [Undecided,New] | 14:21 |
| *** alexchadin has joined #openstack-nova | 14:22 | |
| *** felipemonteiro__ has joined #openstack-nova | 14:22 | |
| superdan | andreykurilin: thanks, I'm not sure how this is happening, but I'm really distracted on this call | 14:22 |
| superdan | andreykurilin: are you going to be around for a while? | 14:22 |
| andreykurilin | np, I'll planning to be there :) | 14:23 |
| *** alexchadin has quit IRC | 14:23 | |
| *** gyee has joined #openstack-nova | 14:24 | |
| *** artom_ is now known as artom | 14:25 | |
| *** felipemonteiro_ has quit IRC | 14:26 | |
| *** armax has joined #openstack-nova | 14:26 | |
| openstackgerrit | Merged openstack/nova master: Remove dest node allocations during live migration rollback https://review.openstack.org/507687 | 14:30 |
| superdan | oooh, I might have a recreate | 14:33 |
| cdent | must be because you’re super | 14:34 |
| superdan | andreykurilin: do you do a regular unpaged list after the fail at all? | 14:35 |
| superdan | andreykurilin: I kinda feel like one of the instances has to be in ERROR state, in cell0 to make this happen | 14:35 |
| andreykurilin | superdan: while doing unpaged list, both instances are returned. After making a boot request, we are fetching the status of VM and do not continue until it become ACTIVE. Both VMs returned ACTIVE status | 14:38 |
| andreykurilin | so I'm pretty sure that they are not in ERROR while listing | 14:39 |
| superdan | andreykurilin: okay they should be sorted by created_at,id which is stable if you only have one database. Unless you have multiple cells here, or instances in cell0, I'm not sure how you could end up with unstable sort | 14:40 |
| *** beekneemech has quit IRC | 14:40 | |
| andreykurilin | superdan: it is dsvm job with a single node. it doesn't have any special configs | 14:41 |
| superdan | yeah | 14:42 |
| openstackgerrit | Dan Smith proposed openstack/nova master: WIP Always put 'uuid' into sort_keys for stabile instance lists https://review.openstack.org/510140 | 14:42 |
| superdan | andreykurilin: can you try with this in place ^ ? | 14:42 |
| *** bnemec has joined #openstack-nova | 14:42 | |
| superdan | if you revert the functional part of that change, the test added fails in the same way | 14:43 |
| andreykurilin | ok, will make a depends on patch | 14:43 |
| andreykurilin | superdan: btw, performance of list action is quite good. before I added a limit to the loop, debug messages flooded the log file by 10gb of text (until jenkins kicked the job by temout) :D | 14:51 |
| superdan | andreykurilin: hah, cool | 14:51 |
| *** yamamoto has quit IRC | 14:52 | |
| *** yamamoto has joined #openstack-nova | 14:53 | |
| *** yamamoto has quit IRC | 14:53 | |
| *** mdnadeem has quit IRC | 14:56 | |
| *** spectr has joined #openstack-nova | 15:04 | |
| superdan | andreykurilin: hmm, actually, that test isn't fully stable, so I need to keep working on it | 15:04 |
| *** spectr has quit IRC | 15:04 | |
| mriedem | fried_rice: merry friday https://review.openstack.org/#/c/488137/22 | 15:05 |
| mriedem | i didn't -1, but i'm sort of inlined to | 15:05 |
| fried_rice | mriedem ack, looking. | 15:05 |
| *** gouthamr has joined #openstack-nova | 15:07 | |
| *** links has joined #openstack-nova | 15:07 | |
| *** tbachman has joined #openstack-nova | 15:07 | |
| *** bnemec has quit IRC | 15:08 | |
| *** ratailor has quit IRC | 15:09 | |
| *** bnemec has joined #openstack-nova | 15:09 | |
| *** mriedem is now known as ronlund | 15:09 | |
| *** Oku_OS is now known as Oku_OS-away | 15:11 | |
| cdent | ronlund: nobody wants to love on your doc fix? https://review.openstack.org/#/c/502168/ | 15:13 |
| *** READ10 has joined #openstack-nova | 15:15 | |
| *** MVenesio has quit IRC | 15:16 | |
| cdent | fried_rice: you still have https://review.openstack.org/#/c/499826/ in your mind? what do we need to do to resolve that? | 15:17 |
| fried_rice | ... | 15:17 |
| cdent | i hear that | 15:18 |
| fried_rice | cdent I actually keep forgetting to put it on the "stuck reviews" list for the nova meetings. | 15:18 |
| fried_rice | It really seems like overkill to put up a whole microversion for that change. | 15:18 |
| cdent | yah | 15:19 |
| fried_rice | But the process nazis would freak out if we slid it into some unrelated change that's doing a legit microversion bump. | 15:19 |
| fried_rice | And I *totally* have no problem doing it without a microversion bump. If the rules forbid such a change, the rules are silly. | 15:19 |
| *** rcernin has quit IRC | 15:19 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: update live migration to use v3 cinder api https://review.openstack.org/463987 | 15:19 |
| fried_rice | But I clearly don't get to make that call. | 15:20 |
| * cdent shrugs | 15:20 | |
| fried_rice | cdent Makes it harder that we cut a release since the original splitup was done. | 15:20 |
| * cdent nods | 15:21 | |
| fried_rice | you know, the one that didn't cut a new microversion when it changed the API in a similar (but more extensive) way than this. | 15:21 |
| fried_rice | cdent Guess I'll add it to "stuck reviews" now while I'm thinking about it, and we can discuss it next Thursday. | 15:22 |
| cdent | an astute plan | 15:22 |
| fried_rice | cdent Thanks for the reminder | 15:22 |
| cdent | was doing my weekly cruise of placement tagged bugs | 15:22 |
| *** jaosorior has quit IRC | 15:24 | |
| *** shvepsy has quit IRC | 15:25 | |
| *** shvepsy has joined #openstack-nova | 15:25 | |
| cdent | gibi: you seen https://bugs.launchpad.net/nova/+bug/1721652 ? references a change you made as the potential cause | 15:25 |
| openstack | Launchpad bug 1721652 in OpenStack Compute (nova) "Evacuate cleanup fails at _delete_allocation_for_moved_instance" [Undecided,New] | 15:25 |
| gibi | cdent: looking... | 15:27 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Add alternate hosts https://review.openstack.org/486215 | 15:28 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Add Selection objects https://review.openstack.org/499239 | 15:28 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Return Selection objects from the scheduler driver https://review.openstack.org/495854 | 15:28 |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/510159 | 15:28 |
| *** jwcroppe has quit IRC | 15:28 | |
| *** jwcroppe has joined #openstack-nova | 15:29 | |
| *** smatzek has quit IRC | 15:31 | |
| *** smatzek has joined #openstack-nova | 15:32 | |
| *** jwcroppe has quit IRC | 15:33 | |
| *** baoli has quit IRC | 15:34 | |
| *** baoli has joined #openstack-nova | 15:35 | |
| *** smatzek has quit IRC | 15:36 | |
| *** jwcroppe has joined #openstack-nova | 15:36 | |
| gibi | cdent: I can confirm that bug based on looking at the code. It seems that the functional test did not catched it somehow | 15:37 |
| cdent | gibi: cool, I figured you would know what was going on | 15:38 |
| gibi | cdent: I felt save becuause of the functional coverage, but it seems we need a better test for it | 15:39 |
| gibi | s/save/safe/ | 15:39 |
| *** edmondsw has joined #openstack-nova | 15:41 | |
| bauwser | leakypipes: maybe tracking all the changes for https://review.openstack.org/#/c/509025/ and above would be better if we have a specless BP ? | 15:42 |
| bauwser | leakypipes: of course, not needing a spec | 15:42 |
| *** cdent has quit IRC | 15:44 | |
| *** gszasz has quit IRC | 15:44 | |
| *** lucasagomes is now known as lucas-afk | 15:44 | |
| *** dtantsur is now known as dtantsur|afk | 15:44 | |
| *** ildikov is now known as coffee_cat | 15:45 | |
| *** edmondsw has quit IRC | 15:45 | |
| *** gjayavelu has joined #openstack-nova | 15:45 | |
| *** yamahata has quit IRC | 15:46 | |
| *** chyka has joined #openstack-nova | 15:46 | |
| ronlund | claudiub: want to send this in? https://review.openstack.org/#/c/509766/ | 15:50 |
| *** smatzek has joined #openstack-nova | 15:50 | |
| *** yamamoto has joined #openstack-nova | 15:53 | |
| *** Apoorva has joined #openstack-nova | 16:01 | |
| *** Apoorva has quit IRC | 16:02 | |
| *** yamamoto has quit IRC | 16:02 | |
| *** Apoorva has joined #openstack-nova | 16:02 | |
| finucannot | leakypipes superdan: Could you take a look at this? I don't think we need 'obj_make_compatible' functions because we're not transferring these objects over the wire, but it's gone to be sure to be sure https://review.openstack.org/#/c/508498/ | 16:08 |
| *** bnemec has quit IRC | 16:08 | |
| superdan | finucannot: I'm in the middle of something deep right now, but we're registering those objects which means they can go over the wire, which means they need to have the make_compat routine | 16:09 |
| superdan | I'm sure leakypipes can speak to the over-the-wire-ness of now and future | 16:09 |
| openstackgerrit | Merged openstack/nova master: stabilize test_resize_server_error_and_reschedule_was_failed https://review.openstack.org/509750 | 16:10 |
| finucannot | superdan: Ta. Holding for leakypipes | 16:10 |
| *** gjayavelu has quit IRC | 16:10 | |
| *** pcaruana has quit IRC | 16:13 | |
| *** jwcroppe has quit IRC | 16:13 | |
| *** mingyu has joined #openstack-nova | 16:15 | |
| *** andreas_s has quit IRC | 16:16 | |
| *** xyang1 has joined #openstack-nova | 16:16 | |
| *** xyang1 has quit IRC | 16:16 | |
| *** jwcroppe has joined #openstack-nova | 16:16 | |
| *** xyang1 has joined #openstack-nova | 16:17 | |
| *** fried_rice is now known as fried_rice_afk | 16:18 | |
| andreykurilin | superdan: the first patch, where I put sort_key=["uuid"] fixed an issue. the second patch(the check for a fix at nova's side) still waits for a resources at CI | 16:19 |
| superdan | andreykurilin: okay | 16:19 |
| *** mingyu has quit IRC | 16:20 | |
| *** penick has joined #openstack-nova | 16:23 | |
| openstackgerrit | Balazs Gibizer proposed openstack/nova master: Reproduce bug 1721652 in the functional test env https://review.openstack.org/510176 | 16:26 |
| openstack | bug 1721652 in OpenStack Compute (nova) pike "Evacuate cleanup fails at _delete_allocation_for_moved_instance" [High,Confirmed] https://launchpad.net/bugs/1721652 | 16:26 |
| *** penick has quit IRC | 16:28 | |
| *** dikonoor has quit IRC | 16:28 | |
| *** cdent has joined #openstack-nova | 16:28 | |
| gibi | cdent, ronlund: I started looking into the bug 1721652 but I run out of time for today. I will continue on Monday if nobody feels the urge to take it over. | 16:30 |
| openstack | bug 1721652 in OpenStack Compute (nova) pike "Evacuate cleanup fails at _delete_allocation_for_moved_instance" [High,Confirmed] https://launchpad.net/bugs/1721652 | 16:30 |
| *** penick has joined #openstack-nova | 16:30 | |
| ronlund | gibi: ok, thanks | 16:31 |
| *** Tom_ has quit IRC | 16:33 | |
| *** baoli has quit IRC | 16:34 | |
| *** baoli has joined #openstack-nova | 16:34 | |
| melwitt | what's ronlund? | 16:36 |
| *** baoli has quit IRC | 16:36 | |
| *** baoli has joined #openstack-nova | 16:37 | |
| *** dikonoor has joined #openstack-nova | 16:38 | |
| *** Tom has joined #openstack-nova | 16:44 | |
| superdan | mmmmm, yeeeeeeaahhhh | 16:44 |
| sean-k-mooney | anyone know if we were to abandon https://review.openstack.org/#/c/373293/7 would it prevent the proposal bot updating it in the future. | 16:44 |
| superdan | that would be greeeeeeaaaat, mmmmkayyy? | 16:44 |
| superdan | we're just going to go ahead and have to ask you to move your desk to the basement, mmmmmkayy? thaaaaaaankss... | 16:45 |
| openstackgerrit | Merged openstack/os-vif master: Add Port Profile info to VIF objects Linux Bridge plugin https://review.openstack.org/490829 | 16:46 |
| ronlund | melwitt: ron lund is a powerful name | 16:47 |
| ronlund | and is a name you can trust | 16:47 |
| ronlund | for all your retirement investment needs | 16:47 |
| ronlund | in the greater tristate area | 16:47 |
| melwitt | lol ahh | 16:47 |
| sean-k-mooney | finucannot: stephen is that you? | 16:47 |
| ronlund | ron lund is a man that speaks in the 3rd person, loves turtle necks and has a great dirty blonde mustache | 16:48 |
| *** Tom has quit IRC | 16:48 | |
| *** Swami has joined #openstack-nova | 16:48 | |
| superdan | oh I guess I'm wrong | 16:48 |
| melwitt | yours is bill lumbergh | 16:49 |
| superdan | ohhh, damn, right | 16:49 |
| melwitt | at first I thought maybe ron lund was ron swanson from parks and rec but it wasn't | 16:50 |
| melwitt | that's the only ron I could think of | 16:50 |
| *** derekh has quit IRC | 16:52 | |
| ronlund | i also personally know a ron bruns | 16:53 |
| ronlund | also a powerful name | 16:53 |
| ronlund | really anyone named "ron" shouldn't be fucked with | 16:53 |
| *** baoli has quit IRC | 16:54 | |
| ronlund | ask maya, she'll tell you | 16:54 |
| melwitt | I dunno anyone named ron in real life | 16:55 |
| ronlund | ron bruns is a cattleman from baltic, south dakota | 16:56 |
| ronlund | i think he even sold insurance on the side... | 16:56 |
| melwitt | an enterprising fellow | 16:56 |
| cdent | sounds like ron doesn’t like taxes | 16:56 |
| *** esberglu has joined #openstack-nova | 16:56 | |
| *** esberglu has quit IRC | 16:56 | |
| sean-k-mooney | cdent: someone likes taxes? | 16:57 |
| sean-k-mooney | people tolerate taxes in retrun for services but i have never meet anyone who actully likes them | 16:58 |
| cdent | well presumably anyone who cares for their fellow person in society has some small amount of like for taxes | 16:58 |
| *** bnemec has joined #openstack-nova | 16:59 | |
| *** gbarros has quit IRC | 16:59 | |
| cdent | but ron, being a strong ron, sounds like the sort that might reject the federal gov’s right to tax | 17:00 |
| *** trinaths1 has quit IRC | 17:01 | |
| *** baoli has joined #openstack-nova | 17:02 | |
| penick | Aww whaaaat "JunoMan signed on at October 4, 2017 at 9:40:36 PM PDT and has been idle for 1 day, 12 hours, 21 minutes, " | 17:02 |
| *** penick is now known as MrJuno | 17:02 | |
| openstackgerrit | Elod Illes proposed openstack/nova master: WIP: Transform scheduler.select_destinations notification https://review.openstack.org/508506 | 17:02 |
| ronlund | sdague: the "should use allow passing user_data to rebuild" thread has taken a weird path, | 17:02 |
| *** melwitt is now known as jgwentworth | 17:02 | |
| ronlund | sdague: effectively putting me on the fence about whether or not we should add that when removing personality from rebuild | 17:03 |
| superdan | MrJuno: bravo | 17:03 |
| MrJuno | Gotta own it | 17:03 |
| sean-k-mooney | ronlund: how so? i have not been folowing it | 17:03 |
| ronlund | sdague: it's not like it would be hard to add, and we are saying that user_data replaces personality files, and people do love their rebuild | 17:03 |
| ronlund | sean-k-mooney: just the amount of love for rebuild | 17:04 |
| sdague | ronlund: I'd be fine with that | 17:04 |
| MrJuno | rybridges I think I shall decree that all members of the OpenStack team at Oath wear the scarlet letter J until we're on Ocata | 17:04 |
| ronlund | i originally assumed that personality files were added to rebuild b/c they aren't persisted like user_data is, but looking at the change that added personality files to rebuild, there was no explanation of why in the commit message | 17:04 |
| ronlund | it predated gerrit so i wasn't surprised by that | 17:05 |
| jgwentworth | yeah, ppl not using floating ips like keeping their ip and volumes stay attached and all that jazz | 17:05 |
| *** openstackstatus has quit IRC | 17:05 | |
| *** openstack has joined #openstack-nova | 17:07 | |
| *** ChanServ sets mode: +o openstack | 17:07 | |
| cdent | ronlund: so ron bruns probably feeks like he’d be even happier without that thieving corporation tax | 17:08 |
| ronlund | i don't know his actual feelings on taxes. he's genuinely a nice guy, so i doubt it bothers him that much. | 17:08 |
| cdent | a weak ron | 17:09 |
| *** MrJuno has quit IRC | 17:09 | |
| openstackgerrit | Merged openstack/nova master: Add error notification for instance.interface_attach https://review.openstack.org/506643 | 17:09 |
| *** penick has joined #openstack-nova | 17:09 | |
| *** yamahata has joined #openstack-nova | 17:13 | |
| jgwentworth | uh oh, I'm seeing on some gate runs of the py27 unit test job it's not running all the unit tests, only the os profiler test http://logs.openstack.org/66/509766/1/check/gate-nova-python27-ubuntu-xenial/794e1a9/testr_results.html.gz | 17:13 |
| jgwentworth | this is bad | 17:13 |
| *** jpena is now known as jpena|off | 17:19 | |
| jgwentworth | I think maybe it's only happening on stable | 17:19 |
| *** sambetts is now known as sambetts|afk | 17:20 | |
| *** bnemec is now known as beekneemech | 17:20 | |
| jgwentworth | I see it on stable/pike and stable/ocata | 17:21 |
| jgwentworth | stable/newton looks okay | 17:22 |
| sean-k-mooney | jgwentworth: is it on master | 17:24 |
| jgwentworth | sean-k-mooney: I'm not seeing it on master | 17:24 |
| *** Tom has joined #openstack-nova | 17:25 | |
| sean-k-mooney | jgwentworth: thats not so bad then because master should prevent anything getting backported if the unit tests fail | 17:25 |
| sean-k-mooney | though i guess it would not catch dependecy issues | 17:25 |
| jgwentworth | yeah, definitely not as bad as on master | 17:25 |
| superdan | sean-k-mooney: help, but not prevent.. a backport could assert something that is true on master and not on stable and we'll think it's okay to merge | 17:26 |
| sean-k-mooney | i guess that has something to do with the zuul v2->v3->v2 changes in the last few weeks | 17:26 |
| jgwentworth | ronlund: I noticed on stable/pike and stable/ocata there's something wrong with our unit test jobs and they're running the os test profiler test instead of all of the unit tests ^ | 17:26 |
| ronlund | jgwentworth: sounds like an issue for mtreinish | 17:27 |
| ronlund | did some stestr stuff get mixed up in stable? | 17:28 |
| sean-k-mooney | jgwentworth: that should be defiend by this job spec correct https://github.com/openstack-infra/project-config/blob/master/jenkins/jobs/python-jobs.yaml#L109-L131 | 17:28 |
| *** Tom has quit IRC | 17:29 | |
| *** edmondsw has joined #openstack-nova | 17:29 | |
| jgwentworth | I dunno, I'm not familiar with how this works but that looks like probably | 17:29 |
| *** smatzek has quit IRC | 17:30 | |
| ronlund | the zuulv3 jobs were defined elsewhere | 17:30 |
| ronlund | in openstack-zuul-jobs | 17:30 |
| ronlund | but that's for zuulv3, the zuulv2 stuff should be as before | 17:30 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Deprecate allowed_direct_url_schemes and nova.image.download.modules https://review.openstack.org/510195 | 17:30 |
| ronlund | but i'm no expert | 17:30 |
| *** gjayavelu has joined #openstack-nova | 17:30 | |
| ronlund | i would be suspect of something with stestr but we shouldn't be using that in stable | 17:30 |
| *** smatzek has joined #openstack-nova | 17:30 | |
| ronlund | cdent: btw i got a recreate of the scheduling 409 failure on https://review.openstack.org/#/c/507918/ | 17:31 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | 17:32 |
| ronlund | it doesn't have a logging patch in that run which logs which instance failed and prompted retries | 17:32 |
| *** links has quit IRC | 17:32 | |
| ronlund | but maybe not necessary to debug? | 17:32 |
| jgwentworth | newton: https://review.openstack.org/#/c/509441/ ocata: https://review.openstack.org/#/c/509440/ pike: https://review.openstack.org/#/c/509439/ | 17:32 |
| *** smatzek has quit IRC | 17:32 | |
| *** smatzek has joined #openstack-nova | 17:32 | |
| sean-k-mooney | looking at https://review.openstack.org/#/c/509766/ its reporting as jenkins not zuul so i guess the python job is running as zuul v2.5 not v3 | 17:32 |
| jgwentworth | newton job is fine, ocata and pike are messed up | 17:33 |
| ronlund | well i guess we should know because the instance uuid is logged right before it | 17:33 |
| *** edmondsw has quit IRC | 17:33 | |
| jgwentworth | sean-k-mooney: yeah, this is the old jenkins stuff that I'm looking at | 17:33 |
| jgwentworth | mtreinish we need you | 17:35 |
| *** gszasz has joined #openstack-nova | 17:36 | |
| *** ijw has joined #openstack-nova | 17:36 | |
| ronlund | hmm and we only ever put RP inventory once - which is what i expected since it's the fake driver and inventory doesn't change | 17:37 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_27_24_496283 | 17:37 |
| jgwentworth | I know that ostestr uses stestr underneath starting in a specific version. so if our ostestr version is sufficiently new, we could be getting stestr behavior | 17:37 |
| ronlund | so what else can cause "Inventory changed while attempting to allocate: Another thread concurrently updated the data." if not updating the RP inventory? | 17:37 |
| ronlund | leakypipes: ^ any ideas? | 17:37 |
| ronlund | other allocations on the same resource provider at the same time i suppose | 17:38 |
| sean-k-mooney | ronlund: should there not be a db lock on the inventory while a transaction is in flight that would prevent multiple concurent updates | 17:39 |
| ronlund | we're not actually updating the inventory | 17:40 |
| ronlund | besides the first time when the RP is created | 17:40 |
| *** MVenesio has joined #openstack-nova | 17:40 | |
| ronlund | we are making allocations in a loop in the scheduler | 17:40 |
| ronlund | one by one, this isn't concurrent as far as i know | 17:41 |
| sean-k-mooney | unless you have 2+ schduers doing this at the same time | 17:41 |
| sean-k-mooney | just looking at http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_27_24_496283 so there we are creating the inventory right? | 17:42 |
| sean-k-mooney | or is the payload of the put an update | 17:43 |
| jgwentworth | okay, so it's running all the tests, just showing the results of the os profiler run only, so it looks like this is just a display problem | 17:43 |
| *** armax has quit IRC | 17:44 | |
| jgwentworth | like, it's picking up the wrong results to show in the testr_results html | 17:44 |
| sean-k-mooney | jgwentworth: thats good because looking at https://raw.githubusercontent.com/openstack-infra/project-config/master/zuul.d/projects.yaml and the job definition everything looks correct | 17:44 |
| ronlund | sean-k-mooney: it's only 1 scheduler | 17:44 |
| *** oomichi_afk is now known as oomichi | 17:44 | |
| openstackgerrit | Merged openstack/nova master: api-ref: note that project_id filter only works with all_tenants https://review.openstack.org/509650 | 17:44 |
| *** avolkov has quit IRC | 17:45 | |
| *** MVenesio has quit IRC | 17:45 | |
| *** yamamoto has joined #openstack-nova | 17:45 | |
| ronlund | sean-k-mooney: and yes, PUT /placement/resource_providers/c8d3d366-c0a0-481d-b7e7-b3e31b8b73e8/inventories is updating the inventory for the compute node resource provider with uuid c8d3d366-c0a0-481d-b7e7-b3e31b8b73e8 | 17:45 |
| ronlund | that happens when nova-compute starts up and creates the compute node | 17:45 |
| *** penick is now known as MrJuno | 17:45 | |
| ronlund | gotta run to get my license renewed, bbiab | 17:46 |
| sean-k-mooney | ronlund: my point was its not safe to decorment an inventory by doing a put with the new value if you have 2 schduler that will create a race | 17:46 |
| ronlund | sean-k-mooney: the PUT has a generatoin id in it | 17:46 |
| ronlund | like an etag | 17:46 |
| ronlund | and we're only doing it once anyway | 17:47 |
| ronlund | so i don't think that's the issue | 17:47 |
| sean-k-mooney | oh ok so if it does not match the current generation then the scecond put will fail and retry | 17:47 |
| ronlund | the 2nd put would fail and the client would have to fetch the latest generation and update their requet | 17:47 |
| *** namnh has quit IRC | 17:47 | |
| ronlund | the server doesn't do it automatically | 17:47 |
| ronlund | the consumer allocation is what's failing with the 409 | 17:47 |
| ronlund | there is no generation id on the consumer allocation | 17:47 |
| andreykurilin | superdan: as expected, your fix works :) | 17:47 |
| jgwentworth | I'm guessing it has something to do with the newer ostestr being stestr underneath and somehow it's messing up the result gathering | 17:48 |
| sean-k-mooney | ya that workflow is fine | 17:48 |
| ronlund | sean-k-mooney: this https://developer.openstack.org/api-ref/placement/#update-allocations | 17:48 |
| superdan | andreykurilin: okay it won't actually work with multiple cells, but I'm polishing off the full fix | 17:48 |
| superdan | andreykurilin: will certainly want you to test that as well | 17:48 |
| andreykurilin | superdan: I do not have multiple cells installation, so will able to test only on regular one | 17:49 |
| superdan | andreykurilin: yep that's fine | 17:49 |
| *** gszasz has quit IRC | 17:50 | |
| jgwentworth | .tox/py27/bin/testr last --subunit vs .tox/py27/bin/stestr last --subunit | 17:50 |
| *** lajoskatona has quit IRC | 17:50 | |
| sean-k-mooney | jgwentworth: so on master the tox config for py27 is https://github.com/openstack/nova/blob/master/tox.ini#L33 with stestr and on newton we delegate to pretty tox script | 17:52 |
| leakypipes | bauwser, superdan, ronlund: reading back... just got back in. | 17:53 |
| sean-k-mooney | on pike we use ostestr | 17:53 |
| *** yamamoto has quit IRC | 17:53 | |
| jgwentworth | sean-k-mooney: yeah, I think we're gonna need mtreinish to look at this. especially now that we know it's not urgent, it's running all the tests, just showing the wrong results in the html page | 17:56 |
| jgwentworth | and if the run fails the unit tests, the os profiler thing won't even run, so I think in a fail case it would show the fail results | 17:57 |
| sean-k-mooney | jgwentworth: looking at stable ocata i think we are overriding the result because we do this https://github.com/openstack/nova/blob/c2aa30b102808882c85d3d3f53d531c4510218cd/tox.ini#L28-L32 | 17:57 |
| leakypipes | bauwser, superdan: https://blueprints.launchpad.net/nova/+spec/de-orm-resource-providers | 17:58 |
| sean-k-mooney | i think we are running all the test first then running the osprofiler tests | 17:59 |
| jgwentworth | sean-k-mooney: yes, it's been like that for a long time. it's like that on master too. it's just I don't know how the results are picked out of that | 17:59 |
| jgwentworth | out of the fact that we have two separate runs I mean | 17:59 |
| leakypipes | finucannot: we are indeed going to be transferring these objects over the wire in short order. Probably good to get the obj_make_compatible() stuff done sooner or later. | 17:59 |
| sean-k-mooney | yes but on master we are not using pretty tox anymore we are using stestr | 18:00 |
| leakypipes | cdent: sorry, I had to leave before answering your question. yes, I'd like to add the nested resource providers series on to the end of that de-orm series because it makes handling superdan's request to make root_provider_uuid and parent_provider_uuid into root_provider_id and parent_provider_id. | 18:01 |
| leakypipes | ronlund: ok, now looking into your issue | 18:01 |
| * leakypipes just realized mriedem is ronlund | 18:02 | |
| *** vladikr has quit IRC | 18:02 | |
| *** gbarros has joined #openstack-nova | 18:03 | |
| leakypipes | ronlund: technically, any change to either inventory or traits of a resource provider would cause that concurrent update error. that said, I don't believe we are yet setting traits on a resource provider (other than in functional DB tests...) | 18:04 |
| sean-k-mooney | jgwentworth: basically my assertion is that ostestr(pike) and python setup.py testr(ocata) probably overrite the results stestr(master) is appending? | 18:04 |
| leakypipes | ronlund: grasping at straws, but maybe cdent's patch that reduced the number of times we update aggregates might be playing into this.. cdent, does updating aggs change the generation? /me goes to check | 18:06 |
| jgwentworth | sean-k-mooney: maybe. I know almost nothing about how the unit test jobs work so I couldn't tell you :) | 18:06 |
| sean-k-mooney | jgwentworth: https://github.com/openstack/nova/blob/353db2d1932965b6502e002b8be510440ff529c0/tox.ini#L33 yes just checked stestr docs thats what the --combine does http://stestr.readthedocs.io/en/latest/MANUAL.html#combining-test-results | 18:07 |
| sean-k-mooney | jgwentworth: on newton we doe not run os profiler which is why that works | 18:08 |
| jgwentworth | ahhh, nice sleuthing | 18:08 |
| *** chyka has quit IRC | 18:09 | |
| leakypipes | ronlund: that's a negative on the set aggregates changing the resource provider generation. | 18:09 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Always put 'uuid' into sort_keys for stable instance lists https://review.openstack.org/510140 | 18:11 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance_get_by_sort_filters() for multiple sort keys https://review.openstack.org/510203 | 18:11 |
| superdan | ronlund: andreykurilin ^ | 18:11 |
| *** priteau has quit IRC | 18:16 | |
| *** lbragstad has quit IRC | 18:16 | |
| sean-k-mooney | jgwentworth: we might be able to use the --partial flag for testr to get similar behavior ill propose a patch to stable/ocata to test it | 18:16 |
| jgwentworth | sounds cool | 18:17 |
| sean-k-mooney | the doc of what --partial actully does is kindof vagure but it mentioned that it was useed with --failing to make sure un run failures where not lost if interupted so im guessing it will prevent whiping the resuts of previous runs | 18:19 |
| *** baoli has quit IRC | 18:22 | |
| *** baoli has joined #openstack-nova | 18:22 | |
| *** lbragstad has joined #openstack-nova | 18:25 | |
| *** READ10 has quit IRC | 18:27 | |
| *** psachin has joined #openstack-nova | 18:29 | |
| * figleaf has to run out for a bit | 18:29 | |
| *** ijw has quit IRC | 18:30 | |
| *** claudiub has quit IRC | 18:35 | |
| ronlund | leakypipes: note we aren't setting traits or doing aggregate anything in this recreate | 18:37 |
| ronlund | RP inventory is only set once when the compute node is created | 18:37 |
| leakypipes | ronlund: yeah, weird indeed. | 18:37 |
| ronlund | the only thing we're doing is trying to create 1000 instances in a single request, so we're processing those instances in order and making the consumer allocation requests against the same RP | 18:38 |
| ronlund | leakypipes: which i assume means the error comes from the amount consumed changing | 18:38 |
| leakypipes | ronlund: yes. | 18:38 |
| ronlund | which, fine, but it's happening in serial | 18:38 |
| ronlund | unless that just means i'm hitting some limit, but then i'd expect a different error? | 18:38 |
| ronlund | there were also 18 retries logged in that recreate until the failure, so something is giving the 409 and saying, retry, and it's working for some | 18:39 |
| leakypipes | ronlund: it could be the periodic task on the compute node kicking in, noticing the generation has updated, and pulling inventory and generation info again. | 18:39 |
| ronlund | so it's not a limit issue it seems | 18:39 |
| ronlund | leakypipes: the inventory isn't getting updated though | 18:39 |
| leakypipes | ronlund: but it shouldn't be setting inventory to a different set of values... :( | 18:39 |
| ronlund | right | 18:39 |
| ronlund | i see only 1 PUT /resource_providers/<uuid>/inventories in the placement api logs | 18:40 |
| ronlund | mayhap i need a patch to dump a stacktrace when we hit the 409 in the placement api | 18:40 |
| ronlund | to see where it's coming from | 18:40 |
| cdent | if you get the inventory, update the local generation on the local node, will an inflight allocation rap out? | 18:40 |
| ronlund | also, i like to say mayhap | 18:40 |
| cdent | crap , not rap | 18:41 |
| cdent | but who knows, allocations may like to rhyme | 18:41 |
| ronlund | leakypipes: specless bp approved btw | 18:42 |
| leakypipes | ronlund: donkey shane. | 18:43 |
| *** ijw has joined #openstack-nova | 18:43 | |
| cdent | ronlund: the 409 is being received in the scheduler? and the scheduler is definitely not eventlet-ing? | 18:44 |
| cdent | (i believe you said that before, just confirming) | 18:45 |
| *** Tom has joined #openstack-nova | 18:45 | |
| ronlund | we don't have multiple scheduler workers | 18:46 |
| cdent | ronlund: i know that, but if somehow eventlet is involved and socket is patched, things might go awry | 18:46 |
| ronlund | well, there was something weird | 18:48 |
| ronlund | sec | 18:48 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance_get_by_sort_filters() for multiple sort keys https://review.openstack.org/510203 | 18:48 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Always put 'uuid' into sort_keys for stable instance lists https://review.openstack.org/510140 | 18:48 |
| ronlund | so the error in the scheduler is here | 18:48 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-sch.txt.gz#_Oct_04_18_32_08_753794 | 18:48 |
| ronlund | right before that, it says, | 18:48 |
| ronlund | Attempting to claim resources in the placement API for instance af059052-6221-4685-8151-6f450e4dc97d {{(pid=29557) _claim_resources | 18:48 |
| ronlund | but if you look at req-16b6ac82-6274-4300-bb65-cc94a26648fd in the placement logs, | 18:48 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_32_08_696647 | 18:49 |
| ronlund | PUT /placement/allocations/da3c3f24-227b-4b3a-8b1d-43a62c637e44 | 18:49 |
| ronlund | that's a different consumer | 18:49 |
| * cdent nods | 18:49 | |
| * cdent looks at code | 18:49 | |
| *** Tom has quit IRC | 18:49 | |
| cdent | ronlund: since the scheduler is started via nova/cmd, the monkey patch in __init__.py is called, right? | 18:50 |
| cdent | So any expectations of linear may be wrong, if a socket at some point decides to wait | 18:51 |
| ronlund | nova/cmd/scheduler.py calls utils.monkey_patch() but i think that's a different thing | 18:51 |
| cdent | yues | 18:52 |
| ronlund | oh in nova/cmd/__init__ yes | 18:52 |
| cdent | but if you are in nova/cmd, __init__ is run | 18:52 |
| ronlund | oh nvm the logging thing | 18:53 |
| ronlund | Unable to submit allocation for instance da3c3f24-227b-4b3a-8b1d-43a62c637e44 (409 {"errors": [{"status": 409, "request_id": "req-16b6ac82-6274-4300-bb65-cc94a26648fd", "detail": "There was a conflict when trying to complete your request.\n\n Inventory changed while attempting to allocate: Another thread concurrently updated the data. Please retry your update ", "title": "Conflict"}]}) | 18:53 |
| ronlund | so da3c3f24-227b-4b3a-8b1d-43a62c637e44 was the instance it was requesting on | 18:53 |
| ronlund | the "Attempting to claim resources in the placement API for instance af059052-6221-4685-8151-6f450e4dc97d" was a red herring, | 18:54 |
| ronlund | if we log ^ but no error, it just means the claim was successful | 18:55 |
| cdent | ronlund: I’m not sure that’s the case? | 18:55 |
| ronlund | we don't log anything for a successful claim | 18:55 |
| cdent | so in that case we have an attempt for instance X, then a fail for instance Y, where is the attempt for Y? | 18:55 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_32_08_825897 is that other instance | 18:56 |
| cdent | so why are we getting the warn about the failure after the debug about the other attempt | 18:56 |
| cdent | they _are_ out of order | 18:56 |
| cdent | which is what we would expect in a monkey patched eventlet scene | 18:57 |
| *** Jose____ has joined #openstack-nova | 18:59 | |
| cdent | bbs, gotta do the dishes | 18:59 |
| ronlund | i'm seeing something f'ed in the scheduler code, | 18:59 |
| ronlund | it's doing the same scheduling request 3 times | 18:59 |
| ronlund | "Starting to schedule for" | 18:59 |
| ronlund | 3 seconds apart | 19:00 |
| ronlund | well, 1 second apart, 3 times | 19:00 |
| ronlund | all with the same request id | 19:00 |
| ronlund | leakypipes: well that answers that i think ^ | 19:01 |
| ronlund | the 409 is probably because we already posted allocations for the same consumer | 19:01 |
| Jose____ | Hi! i found this in nova placement api log: "Placement API returning an error response: JSON does not validate: 0 is less than the minimum of 1", can this be affecting with the image creation service? | 19:01 |
| *** pchavva has quit IRC | 19:01 | |
| ronlund | Jose____: no | 19:02 |
| leakypipes | ronlund: hmm, that's odd indeed. | 19:02 |
| cdent | ronlund: is that request id the local one or the one passed up from nova-api | 19:03 |
| * cdent really does the dishes | 19:03 | |
| ronlund | cdent: the api http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-api.txt.gz#_Oct_04_18_30_21_221501 | 19:04 |
| ronlund | {"server": {"name": "test-alloc-conflict", "imageRef": "947823a9-8c15-420a-95d8-f35cd2f024b9", "flavorRef": "1", "max_count": 1000, "min_count": 1000, "networks": "none"}} | 19:04 |
| ronlund | i see a successful allocation for the failed instance 2 seconds after the final 409 retry failure | 19:04 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-placement-api.txt.gz#_Oct_04_18_34_13_854776 | 19:04 |
| ronlund | wth, why would we have 3 greenthreads? | 19:05 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: de-ORM ResourceProvider.get_by_uuid() https://review.openstack.org/509025 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: Move RP._get|set_aggregates() to module scope https://review.openstack.org/509026 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: Remove RP.get_traits() method https://review.openstack.org/509027 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: move RP._set_traits() to module scope https://review.openstack.org/509028 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove CRUD operations on Inventory class https://review.openstack.org/509029 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: streamline InventoryList.get_all_by_rp_uuid() https://review.openstack.org/509030 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove dead code in Allocation._create_in_db() https://review.openstack.org/509031 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove ability to delete 1 allocation record https://review.openstack.org/509032 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: fix up AllocList.get_by_resource_provider_uuid https://review.openstack.org/509033 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: rework AllocList.get_all_by_consumer_id() https://review.openstack.org/509035 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: remove _HasAResourceProvider mixin https://review.openstack.org/509036 | 19:08 |
| openstackgerrit | Jay Pipes proposed openstack/nova master: rp: break functions out of _set_traits() https://review.openstack.org/509908 | 19:08 |
| ronlund | aha | 19:09 |
| ronlund | http://logs.openstack.org/18/507918/6/check/gate-tempest-dsvm-neutron-full-ubuntu-xenial/d4f175d/logs/screen-n-super-cond.txt.gz#_Oct_04_18_32_07_052398 | 19:09 |
| ronlund | select_destinations took too long, so oslo.messaging retried | 19:09 |
| ronlund | well that explains that i think | 19:09 |
| ronlund | i'm going to bump up the rpc timeout | 19:09 |
| cdent | that doesn’t sound safe | 19:10 |
| cdent | (dishwasher not done yet) | 19:10 |
| ronlund | huh, given we don't rate limit min_count at all, | 19:11 |
| ronlund | you could totally bork some stuff up with making a huge request | 19:11 |
| ronlund | if the server's rpc timeout isn't configured high enough to handle it | 19:11 |
| cfriesen | ronlund: I think we check quota fairly early, no? | 19:11 |
| ronlund | this isn't quota | 19:11 |
| ronlund | although, sure you wouldn't hit this for real if you had default quota of 10 | 19:12 |
| sean-k-mooney | jgwentworth: https://github.com/openstack-infra/project-config/blob/dbdef981de7cb56e9cd44514a41102270bfc9bac/jenkins/scripts/run-tox.sh#L32-L48 this is why its failing we only grab the last result dir | 19:12 |
| ronlund | i've disabled quota | 19:12 |
| ronlund | if you did allow a tenant 1000 instances to burst at once, | 19:12 |
| ronlund | then you are going to have to deal with big ass rpc timeouts | 19:12 |
| leakypipes | ronlund: there's a crapload of timeouts on the MQ in that log file... | 19:13 |
| ronlund | yeah i know | 19:14 |
| ronlund | and by default oslo.messaging retries twice | 19:14 |
| superdan | so it just adds to the load? | 19:14 |
| superdan | because it's starting extra scheduling runs for the same set of stuff? | 19:15 |
| superdan | seems like we'd be breaking hard on that already anyway, pre-placement | 19:15 |
| ronlund | yeah this is definitely user error on my part :) | 19:17 |
| *** gbarros_ has joined #openstack-nova | 19:17 | |
| ronlund | maybe it's a decent simulator of sorts, but probably not | 19:17 |
| *** edmondsw has joined #openstack-nova | 19:17 | |
| ronlund | "how to dos your devstack" | 19:17 |
| superdan | user error why? | 19:18 |
| superdan | just because all the limits are removed? | 19:18 |
| ronlund | yeah | 19:18 |
| ronlund | we could definitely hammer scheduler/placement with concurrent requests, | 19:18 |
| cdent | in a perfect world it would at least fail gracefully rather than confusedly | 19:18 |
| ronlund | but within a single tenant, default quota is 10 so that'd be your max | 19:18 |
| superdan | well, we probably really should never retry a call to the scheduler like this after a timeout, I'm thinking | 19:18 |
| ronlund | well, the good news is it totally does the allocation cleanup properly | 19:19 |
| ronlund | and everything is put into error state and shoved in cell0 | 19:19 |
| superdan | because you could hit that timeout for other reasons | 19:19 |
| *** gbarros has quit IRC | 19:19 | |
| superdan | well, I guess it has already created the instance records, so maybe not a huge deal I guess, | 19:19 |
| superdan | I expect there is a case where you could do a boot, fail to hear from scheduler, never send boots to compute, | 19:20 |
| superdan | but the scheduler made allocations | 19:20 |
| *** MrJuno is now known as penick | 19:20 | |
| superdan | I guess we just live with that and assume they're cleaned up at delete, but technically it's holding space for those dead ones | 19:20 |
| superdan | actually, maybe we wouldn't clean up allocations on delete in that case since instance.host=None? | 19:21 |
| *** edmondsw has quit IRC | 19:22 | |
| ronlund | superdan: well i think if instance.host == None we assume the allocations are already gone | 19:23 |
| ronlund | either it failed to schedule, | 19:23 |
| ronlund | or it was shelved offloaded | 19:23 |
| superdan | right, my point | 19:23 |
| ronlund | and we remove allocations when shelve offloading | 19:23 |
| superdan | we call to scheduler, timeout, | 19:23 |
| superdan | scheduler has made allocations | 19:23 |
| superdan | we just delete from db because it never scheduled | 19:23 |
| ronlund | instance goes to error | 19:24 |
| ronlund | but has allocations | 19:24 |
| superdan | yes | 19:24 |
| ronlund | heh yeah | 19:24 |
| *** tbachman has quit IRC | 19:25 | |
| ronlund | checking for something like that on every delete kind of sucks if it's a super edge case | 19:26 |
| superdan | but no healing, so.. leaking capacity will anger people and rightly so :) | 19:26 |
| ronlund | right | 19:26 |
| *** tbachman has joined #openstack-nova | 19:26 | |
| ronlund | plus a delete request for allocations that never existing should be fast | 19:26 |
| ronlund | *existed | 19:26 |
| superdan | yes | 19:27 |
| ronlund | i know huawei customers love nfv, i need to see what their instance quota limit is quick... :) | 19:27 |
| *** cdent has quit IRC | 19:29 | |
| ronlund | ha | 19:29 |
| ronlund | @utils.retry_select_destinations | 19:29 |
| ronlund | that's what's causing the retry | 19:29 |
| ronlund | it's by design | 19:29 |
| ronlund | it's not oslo.messaging, it's nova | 19:29 |
| ronlund | https://github.com/openstack/nova/blob/353db2d1932965b6502e002b8be510440ff529c0/nova/scheduler/utils.py#L599 | 19:31 |
| *** tesseract has quit IRC | 19:31 | |
| superdan | yeah | 19:31 |
| *** hemna_ has quit IRC | 19:31 | |
| ronlund | so yeah, now that we're doing claims in the scheduler, that seems like a bad idea... | 19:31 |
| ronlund | it does it up to max_attempts-1, so by default 2 retries | 19:32 |
| superdan | that doesn't fix the allocation leak, mind you, | 19:32 |
| *** gbarros has joined #openstack-nova | 19:32 | |
| superdan | but yeah, seems like if you fail talking to it, you're just going to hurt things by adding to the load with a retry | 19:32 |
| *** fried_rice_afk is now known as fried_rice | 19:32 | |
| ronlund | i wonder if we double up the 2nd allocation request for the same consumer | 19:33 |
| ronlund | maybe not if there is only 1 rp uuid in the request | 19:33 |
| ronlund | note the 2nd time through the scheduler on the retry, we could likely target a completely different host :) | 19:33 |
| ronlund | thus totally fucking up things for everything | 19:33 |
| superdan | yeah | 19:33 |
| ronlund | huh, well this is fun | 19:34 |
| *** gbarros_ has quit IRC | 19:35 | |
| *** Tom___ has joined #openstack-nova | 19:36 | |
| *** Tom___ has quit IRC | 19:40 | |
| *** chyka has joined #openstack-nova | 19:42 | |
| *** baoli has quit IRC | 19:44 | |
| *** namnh has joined #openstack-nova | 19:48 | |
| leakypipes | fried_rice: putting: "blueprint: XXXX" does the same thing. | 19:50 |
| fried_rice | leakypipes Coolio. Is there a Source Of Truth for these taggy things? | 19:51 |
| *** eharney has quit IRC | 19:52 | |
| *** namnh has quit IRC | 19:53 | |
| leakypipes | fried_rice: meh, https://wiki.openstack.org/wiki/GitCommitMessages | 19:54 |
| leakypipes | fried_rice: but it only mentions using Implements: blueprint XXX | 19:54 |
| fried_rice | mm | 19:54 |
| leakypipes | fried_rice: that's not necessary though. the word "blueprint" followed by a tag-like thing is all that's needed to link the patch with the blueprint on Launchpad. | 19:55 |
| *** baoli has joined #openstack-nova | 19:55 | |
| fried_rice | leakypipes Including having whatever bot add the URL to the whiteboard on the bp? | 19:55 |
| fried_rice | Cause that seems to be a thing. | 19:55 |
| leakypipes | fried_rice: correct, that's what I mean. | 19:56 |
| fried_rice | k, thought you were just talking about gerrit turning it into a nice hyperlink to the LP page. | 19:56 |
| fried_rice | Anyway, I dig it. | 19:56 |
| *** jwcroppe has quit IRC | 19:57 | |
| ronlund | bp also works i think | 19:58 |
| ronlund | maybe not | 19:58 |
| *** jwcroppe has joined #openstack-nova | 19:58 | |
| *** baoli has quit IRC | 19:58 | |
| mtreinish | ronlund, jgwentworth: do you have a link to the thing you're seeing? | 19:59 |
| mtreinish | there is a lot of backscroll, but I couldn't see a link to what I should be looking at | 19:59 |
| *** jwcroppe_ has joined #openstack-nova | 20:00 | |
| *** baoli has joined #openstack-nova | 20:00 | |
| jgwentworth | sec | 20:00 |
| jgwentworth | mtreinish: this is happening on stable/ocata and stable/pike only http://logs.openstack.org/39/509439/1/check/gate-nova-python27-ubuntu-xenial/e456c8f/testr_results.html.gz | 20:00 |
| jgwentworth | I think it's just a display issue, showing the os profiler result instead of the unit tests result. in the console you can see that both ran | 20:01 |
| mtreinish | jgwentworth: ok, yeah that's because you have 2 test runs in the tox command | 20:01 |
| mtreinish | for the post processing to generate that we run testr last --subunit and pipe that into subunit2html to generate that page | 20:01 |
| jgwentworth | it shows the right thing on master and stable/newton for some reason even though we have 2 runs | 20:01 |
| mtreinish | but testr doesn't let you combine the results | 20:01 |
| mtreinish | so it's just showing the results from the second one | 20:02 |
| jgwentworth | yeah, that's what sean-k-mooney was saying | 20:02 |
| jgwentworth | well, I think testr is showing the first one. os profiler always runs last | 20:02 |
| mtreinish | it works on master because stestr has a --combine flag that treats the 2 commands as a single run | 20:02 |
| *** jwcroppe has quit IRC | 20:02 | |
| *** john5223_ has joined #openstack-nova | 20:02 | |
| ronlund | gah 2017-10-06 04:30:20.654 | /opt/stack/new/devstack/inc/meta-config: line 209: /opt/stack/new/devstack/.localrc.auto: Permission denied | 20:03 |
| jgwentworth | mtreinish: oh, on stable/newton we're not running the os profiler thing | 20:03 |
| jgwentworth | that's why that one shows up correctly | 20:04 |
| mtreinish | yep | 20:04 |
| mtreinish | this was the same thing I was seeing on openstack-health, which is why I pushed https://review.openstack.org/#/c/501842/ before doing the stestr migration | 20:04 |
| jgwentworth | weird. it seems like this would always have been happening before stestr but I could have sworn I had seen full lists of the unit tests on a pass run prior to stestr | 20:04 |
| jgwentworth | maybe I dreamed it | 20:05 |
| mtreinish | jgwentworth: this was a longstanding issue, I just don't think anyone noticed it | 20:05 |
| mtreinish | before the osprofiler tests were added it wasn't an issue | 20:05 |
| jgwentworth | right | 20:05 |
| jgwentworth | oh well, sorry for the noise. I thought something had changed | 20:06 |
| *** Tom___ has joined #openstack-nova | 20:06 | |
| mtreinish | no worries, I'm just glad other people are looking at this stuff :) | 20:06 |
| mtreinish | jgwentworth: fwiw, if you want to fix that output you could probably backport 501842 | 20:08 |
| mtreinish | or just rip the osprofiler test out on the stable branches, tbh I'm not sure what value it adds | 20:08 |
| jgwentworth | yeah, I like running profiler last locally so tests I'm working on run and fail first without having to wait for profiler | 20:10 |
| openstackgerrit | Matt Riedemann proposed openstack/nova stable/pike: Set regex flag on ostestr command for osprofiler tests https://review.openstack.org/510226 | 20:10 |
| ronlund | let's see | 20:10 |
| *** Tom___ has quit IRC | 20:10 | |
| *** smatzek has quit IRC | 20:12 | |
| *** gouthamr has quit IRC | 20:21 | |
| *** artom has quit IRC | 20:24 | |
| *** catinthe_ has quit IRC | 20:28 | |
| openstackgerrit | Ildiko Vancsa proposed openstack/nova master: update live migration to use v3 cinder api https://review.openstack.org/463987 | 20:29 |
| *** spectr has joined #openstack-nova | 20:34 | |
| *** spectr has quit IRC | 20:36 | |
| *** david-lyle has quit IRC | 20:38 | |
| *** beekneemech has quit IRC | 20:45 | |
| *** archit has quit IRC | 20:47 | |
| *** crushil has quit IRC | 20:48 | |
| *** dikonoor has quit IRC | 20:52 | |
| *** awaugama has quit IRC | 20:57 | |
| *** Jose____ has quit IRC | 21:01 | |
| *** jwcroppe_ has quit IRC | 21:03 | |
| *** edmondsw has joined #openstack-nova | 21:05 | |
| *** jwcroppe has joined #openstack-nova | 21:05 | |
| *** sahid has quit IRC | 21:06 | |
| *** Tom has joined #openstack-nova | 21:07 | |
| *** zhenq has quit IRC | 21:07 | |
| superdan | ronlund: I think those two patches are good to go | 21:07 |
| superdan | ronlund: cool if you want to wait for andreykurilin to run a test, but he confirmed on one of the earlier iterations, I have a repro test in there, etc | 21:08 |
| andreykurilin | superdan: Does the new revision ready for testing? I can recheck the patch now | 21:09 |
| superdan | andreykurilin: oh yeah, I poked you earlier but no response :) | 21:09 |
| andreykurilin | oh. | 21:09 |
| superdan | andreykurilin: would be awesome if you could, and maybe point me at the review so I can watch? | 21:09 |
| *** edmondsw has quit IRC | 21:10 | |
| *** Tom has quit IRC | 21:11 | |
| *** lyan has joined #openstack-nova | 21:11 | |
| andreykurilin | superdan: heh, sure. https://review.openstack.org/#/c/510144 gate-rally-dsvm-neutron-existing-users-rally job. grep tag-to-search to see debug messages which actually shows that everything works or not | 21:11 |
| superdan | andreykurilin: doesn't it just fail if it fails? | 21:13 |
| superdan | andreykurilin: I don't see that job in the first run of that patch... | 21:15 |
| *** crushil has joined #openstack-nova | 21:16 | |
| ronlund | ok, writing up a draft spec for discussion on this max_count rate limiting thing | 21:16 |
| andreykurilin | superdan: no :( I tried to make this change as soon as possible, so i have not removed a hack to limit the loop. I'll do it now. the first results were not published, it looks like you posted a new revision before the whole jobs were finished | 21:16 |
| superdan | ah okay | 21:16 |
| *** ijw has quit IRC | 21:17 | |
| *** david-lyle has joined #openstack-nova | 21:19 | |
| andreykurilin | superdan: so originally the listing doesn't fail. it was just a inf loop. To add debug messages which will not flood the log file, I added a limit in the loop(10). That is why it stopped failing. Now I resubmitted a patch which fails in case of 10 iterations of the loop. so you do not need to check the logs | 21:19 |
| *** xyang1 has quit IRC | 21:20 | |
| superdan | andreykurilin: if len(log) == 10g -> fail? :) | 21:20 |
| superdan | andreykurilin: awesome thanks | 21:20 |
| andreykurilin | ha | 21:20 |
| *** felipemonteiro__ has quit IRC | 21:21 | |
| *** ijw has joined #openstack-nova | 21:23 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova-specs master: Limit instance create max_count (spec) https://review.openstack.org/510235 | 21:29 |
| ronlund | superdan: cfriesen: leakypipes: i know it's late so probably just ignore until next week, but ^ has some thoughts on the concurrent scheduling issue | 21:29 |
| *** ronlund is now known as mriedem | 21:29 | |
| superdan | ack | 21:30 |
| mriedem | now lemme take a looksie at these crazy patches | 21:31 |
| *** gbarros has quit IRC | 21:31 | |
| *** ijw has quit IRC | 21:37 | |
| *** ijw has joined #openstack-nova | 21:39 | |
| *** dklyle has joined #openstack-nova | 21:39 | |
| *** claudiub has joined #openstack-nova | 21:39 | |
| *** david-lyle has quit IRC | 21:41 | |
| *** dklyle has quit IRC | 21:41 | |
| *** dklyle has joined #openstack-nova | 21:42 | |
| *** lbragstad has quit IRC | 21:43 | |
| mriedem | wow rally runs a lot of jobs | 21:44 |
| mriedem | that's got to be a challenge to both tempest and trove | 21:44 |
| mtreinish | mriedem: I think htat's more than tempest | 21:45 |
| *** namnh has joined #openstack-nova | 21:49 | |
| *** dklyle has quit IRC | 21:52 | |
| *** namnh has quit IRC | 21:54 | |
| cfriesen | mriedem: did we ever fix the issues with multi-boot where scheduling more than min_count but less than max_count would result in instances in the "error" state? | 21:56 |
| *** jwcroppe has quit IRC | 21:58 | |
| mriedem | cfriesen: i'd need more details than that | 21:58 |
| mriedem | why would that put them in error state? | 21:59 |
| *** jwcroppe has joined #openstack-nova | 21:59 | |
| mriedem | if i request min=5 max=10 but there are only 7 ports available in my port quota, the api changes max=7 | 21:59 |
| mriedem | you wrote that patch | 21:59 |
| *** Apoorva_ has joined #openstack-nova | 22:00 | |
| *** Apoorva has quit IRC | 22:01 | |
| *** Apoorva_ has quit IRC | 22:02 | |
| *** Apoorva has joined #openstack-nova | 22:02 | |
| jgwentworth | zzzeek: are you around? I'm looking into something with oslo.db and was wondering, given a TransactionContextManager, how does it switch from reader to writer mode and vice versa between separate transactions? | 22:02 |
| *** Apoorva has quit IRC | 22:03 | |
| *** jwcroppe has quit IRC | 22:03 | |
| *** Apoorva has joined #openstack-nova | 22:03 | |
| mriedem | superdan: done https://review.openstack.org/#/c/510203/ | 22:07 |
| *** chyka has quit IRC | 22:07 | |
| mriedem | superdan: 2 issues, (1) the tests aren't using the args passed in and (2) apparently we have to handle boolean sort keys differently | 22:07 |
| *** jgwentworth is now known as melwitt | 22:08 | |
| mriedem | superdan: presumably because not all db backends model booleans the same | 22:08 |
| mriedem | some are bools, some are ints | 22:08 |
| cfriesen | mriedem: I'm thinking about the case where we request min=5 and max=10 and the scheduler only finds hosts for 7 of them. | 22:08 |
| superdan | mriedem: ah I didn't think we had any booleans, but I guess we do | 22:08 |
| superdan | mriedem: hah, sorry, last minute cleanup on the tests and I didn't remember to plumb those through | 22:09 |
| melwitt | zzzeek: it looks like it's controlled by the "independent" attribute, but I don't see that we use that | 22:09 |
| cfriesen | mriedem: I think the ones without hosts might get left in the BUILD state | 22:10 |
| mriedem | cfriesen: if you get fewer hosts than requested instances, it's NoValidHost https://github.com/openstack/nova/blob/master/nova/scheduler/filter_scheduler.py#L86 | 22:13 |
| mriedem | and conductor will put them all into ERROR state and into cell0 yes | 22:13 |
| cfriesen | mreidem: yeah, was just looking at that | 22:13 |
| cfriesen | mriedem: but that doesn't make sense if the scheduler was able to schedule at least min_count instances. | 22:13 |
| mriedem | cfriesen: then don't request max? | 22:14 |
| mriedem | but yeah that's weird | 22:14 |
| mriedem | cfriesen: see check_num_instances_quota | 22:15 |
| mriedem | if you have quota for max, then that's what we say to build | 22:15 |
| mriedem | we don't pass anything to the scheduler about what minimum number should be built if possible | 22:16 |
| mriedem | as far as i can tell | 22:16 |
| mriedem | wonder how this behaved in ocata | 22:16 |
| cfriesen | mriedem: if I remember right the quota part is okay, but the scheduler part is a bit wonky. we added a local patch to pass "min_num_instances" in the spec_obj | 22:16 |
| cfriesen | mriedem: it's been broken since forever | 22:16 |
| mriedem | did you try upstreaming that patch ever? | 22:17 |
| *** lyan has quit IRC | 22:17 | |
| cfriesen | mriedem: I think so, and it was suggested to just get rid of multiboot, but that got objections | 22:17 |
| cfriesen | I can try reviving it. :) | 22:17 |
| cfriesen | the patch, I mean | 22:18 |
| mriedem | yeah so multicreate isn't going away | 22:18 |
| mriedem | just like rebuild isn't going away | 22:18 |
| mriedem | so yeah would probably be good to revive that | 22:18 |
| cfriesen | would that count as an API change and need a spec? | 22:19 |
| cfriesen | or is it incorrectly returning an error now and we can just correct it | 22:19 |
| *** ijw has quit IRC | 22:19 | |
| zzzeek | jgwentworth:. Can you email me at mike@zzzcomputing | 22:23 |
| *** ijw has joined #openstack-nova | 22:23 | |
| *** armax has joined #openstack-nova | 22:23 | |
| zzzeek | .com and I'll get back to you tomorrow? Not at a computer right now | 22:23 |
| melwitt | zzzeek: okay, will do that if I can't figure this out. thanks | 22:24 |
| *** gouthamr has joined #openstack-nova | 22:24 | |
| *** esberglu has joined #openstack-nova | 22:26 | |
| *** esberglu has quit IRC | 22:26 | |
| fried_rice | Be there existing docs describing the resources:<resource_class>=<count> syntax in flavor extra specs? | 22:26 |
| * fried_rice couldn't find any | 22:27 | |
| *** ijw has quit IRC | 22:27 | |
| *** armax has quit IRC | 22:28 | |
| *** baoli has quit IRC | 22:29 | |
| cfriesen | mriedem: hah, I thought I had a bug open: https://bugs.launchpad.net/nova/+bug/1458122 | 22:30 |
| openstack | Launchpad bug 1458122 in OpenStack Compute (nova) "nova shouldn't error if we can't schedule all of max_count instances at boot time" [Wishlist,Opinion] - Assigned to Chris Friesen (cbf123) | 22:30 |
| cfriesen | there's also this one: https://bugs.launchpad.net/nova/+bug/1623809 | 22:30 |
| openstack | Launchpad bug 1623809 in OpenStack Compute (nova) "Quota exceeded when spawning instances in server group" [Wishlist,Opinion] | 22:30 |
| *** erlon has quit IRC | 22:30 | |
| cfriesen | that latter one is really about min_count/max_count and quota_server_group_members | 22:30 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Fix instance_get_by_sort_filters() for multiple sort keys https://review.openstack.org/510203 | 22:31 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Always put 'uuid' into sort_keys for stable instance lists https://review.openstack.org/510140 | 22:31 |
| mtreinish | mriedem, jgwentworth: http://logs.openstack.org/26/510226/1/check/gate-nova-python27-ubuntu-xenial/939736e/testr_results.html.gz | 22:31 |
| *** shvepsy_ has joined #openstack-nova | 22:31 | |
| superdan | mriedem: good catches, thanks | 22:31 |
| *** armax has joined #openstack-nova | 22:32 | |
| *** armax has quit IRC | 22:33 | |
| *** shvepsy has quit IRC | 22:33 | |
| *** NightKhaos has quit IRC | 22:33 | |
| *** NightKhaos has joined #openstack-nova | 22:37 | |
| mriedem | YES! | 22:39 |
| mriedem | i need to get out of here and get some pizza | 22:40 |
| mriedem | and later some cookies maybe | 22:40 |
| mriedem | i won't be 160 by christmas with pizza and cookies though | 22:40 |
| *** crushil has quit IRC | 22:41 | |
| *** leakypipes has quit IRC | 22:43 | |
| * cfriesen hasn't been 160 since he was 13 | 22:44 | |
| *** mingyu has joined #openstack-nova | 22:48 | |
| mriedem | it's all that poutin you eat | 22:51 |
| mriedem | i heard canadian babies are bottle fed poutin | 22:51 |
| mriedem | aye | 22:51 |
| *** gbarros has joined #openstack-nova | 22:51 | |
| *** mingyu has quit IRC | 22:52 | |
| cfriesen | mmmm...poutine. | 22:53 |
| cfriesen | hard to find the good stuff though....lots of crappy versions | 22:53 |
| *** edmondsw has joined #openstack-nova | 22:53 | |
| cfriesen | hot crispy fries, squeeky cheese curds, and boiling hot gravy | 22:54 |
| cfriesen | great...now I want poutine and I'm pretty sure there's going to be something healthy for supper. | 22:55 |
| mriedem | superdan: ok comments in those 2 changes | 22:55 |
| mriedem | but really leaving this time | 22:56 |
| superdan | mriedem: just spiteful on that bug fix eh? | 22:56 |
| superdan | I think doing pagination without a requested sort order is kinda reckless, but I'll capitulate | 22:57 |
| *** edmondsw has quit IRC | 22:58 | |
| openstackgerrit | Merged openstack/nova master: Note TrustedFilter deprecation in docs https://review.openstack.org/509931 | 22:58 |
| openstackgerrit | Dan Smith proposed openstack/nova master: Always put 'uuid' into sort_keys for stable instance lists https://review.openstack.org/510140 | 22:59 |
| *** ijw has joined #openstack-nova | 23:03 | |
| *** gbarros has quit IRC | 23:04 | |
| *** crushil has joined #openstack-nova | 23:05 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Granular Resource Request Syntax https://review.openstack.org/510244 | 23:06 |
| *** gouthamr has quit IRC | 23:10 | |
| *** Apoorva has quit IRC | 23:16 | |
| *** Apoorva has joined #openstack-nova | 23:17 | |
| *** esberglu has joined #openstack-nova | 23:20 | |
| *** hongbin has quit IRC | 23:21 | |
| *** artom has joined #openstack-nova | 23:21 | |
| *** crushil has quit IRC | 23:24 | |
| *** jgriffith is now known as jgriffith_ | 23:24 | |
| *** esberglu has quit IRC | 23:24 | |
| *** crushil has joined #openstack-nova | 23:38 | |
| *** Swami has quit IRC | 23:38 | |
| *** claudiub has quit IRC | 23:40 | |
| *** david-lyle has joined #openstack-nova | 23:42 | |
| *** harlowja has quit IRC | 23:45 | |
| *** namnh has joined #openstack-nova | 23:50 | |
| *** mingyu has joined #openstack-nova | 23:53 | |
| *** yamamoto has joined #openstack-nova | 23:53 | |
| *** markvoelker has quit IRC | 23:54 | |
| *** namnh has quit IRC | 23:55 | |
| openstackgerrit | Ed Leafe proposed openstack/nova master: Change RPC for select_destinations() https://review.openstack.org/510159 | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!