| *** mrjk has quit IRC | 00:05 | |
| *** brinzhang has quit IRC | 00:05 | |
| *** mrjk has joined #openstack-nova | 00:06 | |
| *** brinzhang has joined #openstack-nova | 00:06 | |
| *** erlon has quit IRC | 00:20 | |
| *** wolverineav has quit IRC | 00:29 | |
| *** tbachman has joined #openstack-nova | 00:32 | |
| *** ileixe has joined #openstack-nova | 00:59 | |
| *** sapd1_x has joined #openstack-nova | 01:03 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (4) https://review.openstack.org/574106 | 01:07 |
|---|---|---|
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (5) https://review.openstack.org/574110 | 01:07 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (6) https://review.openstack.org/574113 | 01:07 |
| *** mlavalle has quit IRC | 01:10 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (7) https://review.openstack.org/574974 | 01:22 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (8) https://review.openstack.org/575311 | 01:22 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (9) https://review.openstack.org/575581 | 01:22 |
| *** wolverineav has joined #openstack-nova | 01:26 | |
| openstackgerrit | zhaixiaojun proposed openstack/nova master: Remove flavor id and name validation code https://review.openstack.org/638150 | 01:45 |
| *** tiendc has joined #openstack-nova | 01:54 | |
| *** tetsuro has joined #openstack-nova | 01:56 | |
| *** cfriesen has quit IRC | 01:57 | |
| *** hongbin has joined #openstack-nova | 02:05 | |
| *** burt has quit IRC | 02:07 | |
| *** ricolin has joined #openstack-nova | 02:07 | |
| *** sapd1_x has quit IRC | 02:15 | |
| *** irclogbot_2 has quit IRC | 02:19 | |
| *** edleafe has quit IRC | 02:22 | |
| *** edmondsw has quit IRC | 02:22 | |
| *** wolverineav has quit IRC | 02:35 | |
| *** wolverineav has joined #openstack-nova | 02:38 | |
| *** wolverineav has quit IRC | 02:39 | |
| *** ileixe has quit IRC | 02:45 | |
| *** dannins has joined #openstack-nova | 02:50 | |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:01 |
| openstackgerrit | Ghanshyam Mann proposed openstack/nova-specs master: Spec for API policy updates https://review.openstack.org/547850 | 03:05 |
| *** psachin has joined #openstack-nova | 03:17 | |
| *** lpetrut has joined #openstack-nova | 03:50 | |
| *** edmondsw has joined #openstack-nova | 04:01 | |
| *** udesale has joined #openstack-nova | 04:02 | |
| *** hongbin has quit IRC | 04:17 | |
| *** lpetrut has quit IRC | 04:19 | |
| *** takashin has left #openstack-nova | 04:24 | |
| *** takashin has joined #openstack-nova | 04:26 | |
| *** jaosorior has joined #openstack-nova | 04:39 | |
| *** wolverineav has joined #openstack-nova | 04:40 | |
| *** wolverineav has quit IRC | 04:46 | |
| *** psachin has quit IRC | 05:09 | |
| *** psachin has joined #openstack-nova | 05:15 | |
| openstackgerrit | Merged openstack/os-vif master: Update master for stable/stein https://review.openstack.org/644036 | 05:20 |
| *** ivve has joined #openstack-nova | 05:22 | |
| *** whoami-rajat has joined #openstack-nova | 05:31 | |
| *** ratailor has joined #openstack-nova | 05:32 | |
| *** sidx64 has joined #openstack-nova | 05:36 | |
| *** dpawlik has joined #openstack-nova | 05:38 | |
| *** jaosorior has quit IRC | 05:46 | |
| *** jaosorior has joined #openstack-nova | 05:47 | |
| *** psachin has quit IRC | 05:57 | |
| *** sidx64 has quit IRC | 05:59 | |
| *** psachin has joined #openstack-nova | 06:05 | |
| *** elbragstad has quit IRC | 06:07 | |
| *** slaweq has joined #openstack-nova | 06:19 | |
| *** janki has joined #openstack-nova | 06:21 | |
| *** sridharg has joined #openstack-nova | 06:24 | |
| openstackgerrit | zhaixiaojun proposed openstack/nova master: Remove flavor id and name validation code https://review.openstack.org/638150 | 06:27 |
| *** sapd1_x has joined #openstack-nova | 06:36 | |
| *** ratailor_ has joined #openstack-nova | 06:38 | |
| *** ratailor has quit IRC | 06:40 | |
| *** sapd1_x has quit IRC | 06:41 | |
| *** krypto has joined #openstack-nova | 06:49 | |
| *** ccamacho has quit IRC | 06:53 | |
| *** lpetrut has joined #openstack-nova | 06:57 | |
| *** rcernin has quit IRC | 06:58 | |
| *** pcaruana has joined #openstack-nova | 06:58 | |
| *** pcaruana has quit IRC | 07:02 | |
| *** pcaruana has joined #openstack-nova | 07:02 | |
| *** psachin has quit IRC | 07:08 | |
| *** awalende has joined #openstack-nova | 07:08 | |
| *** awalende has quit IRC | 07:09 | |
| *** awalende has joined #openstack-nova | 07:10 | |
| *** awalende has joined #openstack-nova | 07:11 | |
| *** tosky has joined #openstack-nova | 07:13 | |
| *** awalende has quit IRC | 07:15 | |
| *** psachin has joined #openstack-nova | 07:16 | |
| *** psachin has quit IRC | 07:16 | |
| *** awalende has joined #openstack-nova | 07:22 | |
| *** ccamacho has joined #openstack-nova | 07:22 | |
| *** luksky has joined #openstack-nova | 07:23 | |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| *** tesseract has joined #openstack-nova | 07:24 | |
| *** sidx64 has joined #openstack-nova | 07:30 | |
| *** ralonsoh has joined #openstack-nova | 07:31 | |
| *** kashyap has joined #openstack-nova | 07:33 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (10) https://review.openstack.org/576017 | 07:43 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (11) https://review.openstack.org/576018 | 07:43 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (12) https://review.openstack.org/576019 | 07:43 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (13) https://review.openstack.org/576020 | 07:44 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (14) https://review.openstack.org/576027 | 07:44 |
| openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances virt driver https://review.openstack.org/648912 | 07:44 |
| openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/648913 | 07:44 |
| openstackgerrit | Yongli He proposed openstack/nova master: Clean up orphan instances https://review.openstack.org/627765 | 07:48 |
| *** Dinesh_Bhor has quit IRC | 07:59 | |
| openstackgerrit | ya.wang proposed openstack/nova-specs master: expose auto converge in rest api https://review.openstack.org/648619 | 08:01 |
| *** Dinesh_Bhor has joined #openstack-nova | 08:03 | |
| *** jangutter has joined #openstack-nova | 08:04 | |
| *** phasespace has joined #openstack-nova | 08:05 | |
| *** zbr is now known as zbr|pto | 08:08 | |
| *** tssurya has joined #openstack-nova | 08:08 | |
| *** cdent has joined #openstack-nova | 08:15 | |
| *** ttsiouts has joined #openstack-nova | 08:19 | |
| *** sidx64 has quit IRC | 08:23 | |
| *** sidx64 has joined #openstack-nova | 08:24 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Change the default value of swap in the show flavor details API https://review.openstack.org/648919 | 08:24 |
| *** wolverineav has joined #openstack-nova | 08:30 | |
| *** sidx64 has quit IRC | 08:33 | |
| *** tkajinam has quit IRC | 08:34 | |
| *** wolverineav has quit IRC | 08:34 | |
| *** dtantsur|afk is now known as dtantsur | 08:35 | |
| *** derekh has joined #openstack-nova | 08:37 | |
| openstackgerrit | Brin Zhang proposed openstack/nova-specs master: Change the default value of swap in the show flavor details API https://review.openstack.org/648919 | 08:59 |
| *** davidsha has joined #openstack-nova | 09:07 | |
| openstackgerrit | Zhenyu Zheng proposed openstack/nova master: WIP fix live migration violate affinity groups. https://review.openstack.org/648929 | 09:09 |
| *** xek has joined #openstack-nova | 09:10 | |
| openstackgerrit | Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | 09:12 |
| *** med_ has quit IRC | 09:15 | |
| *** cdent has quit IRC | 09:28 | |
| *** sapd1_x has joined #openstack-nova | 09:32 | |
| openstackgerrit | Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | 09:40 |
| openstackgerrit | Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | 09:51 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (15) https://review.openstack.org/576031 | 09:51 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (16) https://review.openstack.org/576299 | 09:51 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (17) https://review.openstack.org/576344 | 09:51 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (18) https://review.openstack.org/576673 | 09:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Stub out privsep modules https://review.openstack.org/648747 | 09:52 |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Stub out os_vif https://review.openstack.org/648748 | 09:52 |
| openstackgerrit | Boxiang Zhu proposed openstack/nova-specs master: Add host and hypervisor_hostname flag to create server https://review.openstack.org/645458 | 09:57 |
| *** cdent has joined #openstack-nova | 10:12 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: [stable-only] Add report_ironic_standard_resource_class_inventory option https://review.openstack.org/620111 | 10:13 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: Fail to live migration if instance has a NUMA topology https://review.openstack.org/629597 | 10:13 |
| openstackgerrit | Stephen Finucane proposed openstack/nova stable/queens: fix up numa-topology live migration hypervisor check https://review.openstack.org/648939 | 10:13 |
| stephenfin | bauzas: Any chance you could look at these two trivial fixes? https://review.openstack.org/#/c/648748/ https://review.openstack.org/#/c/648747/ | 10:16 |
| kashyap | stephenfin: If you have time this week: https://review.openstack.org/#/c/645814/ (Add "CPU selection with hypervisor consideration" spec) | 10:16 |
| *** luksky has quit IRC | 10:17 | |
| kashyap | Also appreciate a critical eye from the "native English speaker" point of view. /me already spent far too much time (is there such a thing?) with words. :-) | 10:17 |
| sean-k-mooney | kashyap: wordsmithing | 10:18 |
| kashyap | sean-k-mooney: Yeah, I enjoy it :-) | 10:19 |
| kashyap | Also your feedback is appreciated, too. If you have time. | 10:19 |
| *** ttsiouts has quit IRC | 10:21 | |
| *** sapd1_x has quit IRC | 10:28 | |
| *** wolverineav has joined #openstack-nova | 10:31 | |
| kashyap | Unrelated: Is it really frowned upon to rename a (previously-approved) blueprint (with a spec), when re-proposing? | 10:33 |
| kashyap | (I didn't file the blueprint, but the name bothers me.) | 10:34 |
| kashyap | This is the BP: https://blueprints.launchpad.net/nova/+spec/cpu-model-selection | 10:34 |
| kashyap | And my suggested "diff" is: | 10:34 |
| kashyap | - cpu-model-selection | 10:34 |
| kashyap | + allow-a-list-of-cpu-models | 10:34 |
| sean-k-mooney | i dont think so if the new name makes more sense | 10:34 |
| sean-k-mooney | on the other hand it goes slightly beyond https://github.com/openstack/nova-specs#previously-approved-specifications | 10:35 |
| *** wolverineav has quit IRC | 10:35 | |
| *** ivve has quit IRC | 10:35 | |
| kashyap | Probably I'll quit editing it. As I don't want to break others bookmarks | 10:35 |
| kashyap | I don't really dislike it _that_ much, and can slightly grudgingly live with it. :D | 10:36 |
| * kashyap --> lunch | 10:36 | |
| sean-k-mooney | you could jsut file a new blueprint and update the old one to say its superceed by the new one | 10:36 |
| *** tbachman has quit IRC | 10:37 | |
| *** sridharg has quit IRC | 10:38 | |
| *** efried has quit IRC | 10:54 | |
| *** ttsiouts has joined #openstack-nova | 10:57 | |
| *** efried has joined #openstack-nova | 11:01 | |
| *** ttsiouts has quit IRC | 11:02 | |
| *** sridharg has joined #openstack-nova | 11:02 | |
| *** ttsiouts has joined #openstack-nova | 11:03 | |
| *** cdent has quit IRC | 11:03 | |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (19) https://review.openstack.org/576676 | 11:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (20) https://review.openstack.org/576689 | 11:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (21) https://review.openstack.org/576709 | 11:05 |
| openstackgerrit | Takashi NATSUME proposed openstack/nova master: Remove mox in unit/network/test_neutronv2.py (22) https://review.openstack.org/576712 | 11:05 |
| *** erlon has joined #openstack-nova | 11:13 | |
| *** tiendc has quit IRC | 11:15 | |
| *** takashin has quit IRC | 11:16 | |
| *** udesale has quit IRC | 11:22 | |
| *** ivve has joined #openstack-nova | 11:25 | |
| sean-k-mooney | efried: fyi this https://blueprints.launchpad.net/nova/+spec/consistent-use-of-network-events is the blueprint to track teh nova side of the neutron spec https://review.openstack.org/#/c/645173/ | 11:26 |
| *** luksky has joined #openstack-nova | 11:26 | |
| *** zhubx has quit IRC | 11:37 | |
| *** zhubx has joined #openstack-nova | 11:37 | |
| *** erlon has quit IRC | 11:39 | |
| *** tbachman has joined #openstack-nova | 11:45 | |
| *** erlon has joined #openstack-nova | 11:46 | |
| *** ratailor_ has quit IRC | 11:46 | |
| openstackgerrit | Surya Seetharaman proposed openstack/nova-specs master: Support adding the reason behind a server lock https://review.openstack.org/638629 | 11:48 |
| *** cdent has joined #openstack-nova | 11:55 | |
| *** brinzhang has quit IRC | 12:00 | |
| *** weshay|rover is now known as weshay | 12:03 | |
| *** leakypipes is now known as jaypipes | 12:07 | |
| *** priteau has joined #openstack-nova | 12:12 | |
| *** tetsuro has quit IRC | 12:18 | |
| bauzas | stephenfin: I'll take a look | 12:19 |
| *** rcernin has joined #openstack-nova | 12:19 | |
| *** avolkov has joined #openstack-nova | 12:20 | |
| *** elbragstad has joined #openstack-nova | 12:31 | |
| *** wolverineav has joined #openstack-nova | 12:32 | |
| *** markvoelker has quit IRC | 12:32 | |
| *** wolverineav has quit IRC | 12:36 | |
| *** cdent has quit IRC | 12:36 | |
| *** nicolasbock has joined #openstack-nova | 12:45 | |
| *** nicolasbock has quit IRC | 12:47 | |
| *** nicolasbock has joined #openstack-nova | 12:47 | |
| *** cdent has joined #openstack-nova | 12:48 | |
| *** mvkr has quit IRC | 12:49 | |
| *** ricolin has quit IRC | 12:50 | |
| *** priteau has quit IRC | 12:51 | |
| *** zhubx has quit IRC | 12:51 | |
| *** zhubx has joined #openstack-nova | 12:52 | |
| *** priteau has joined #openstack-nova | 12:54 | |
| *** jmlowe has quit IRC | 12:55 | |
| *** mriedem has joined #openstack-nova | 12:55 | |
| *** whoami-rajat has quit IRC | 13:00 | |
| *** elbragstad is now known as lbragstad | 13:02 | |
| *** whoami-rajat has joined #openstack-nova | 13:05 | |
| *** munimeha1 has joined #openstack-nova | 13:07 | |
| openstackgerrit | sean mooney proposed openstack/nova master: make lower-constraints env use lower-constraints https://review.openstack.org/645392 | 13:09 |
| efried | sean-k-mooney: ack, thanks. | 13:09 |
| efried | sean-k-mooney: Are you wanting it targeted for train? | 13:09 |
| sean-k-mooney | the blueprint i think so yes but that will likely depend on the conversation at the PTG. | 13:10 |
| sean-k-mooney | we (nova and neutorn) could decide that it was not a priority for train but it is techdebt i would like to resovle eventually | 13:11 |
| sean-k-mooney | efried: by the way im hoping we can merge either cdents lower constratints fix https://review.openstack.org/#/c/622972 or mine https://review.openstack.org/645392 sooner rather then later | 13:12 |
| efried | sean-k-mooney: I'll mark the bp series goal for train then. | 13:12 |
| cdent | sean-k-mooney: what's the essential difference? | 13:12 |
| cdent | (between mine and yours) | 13:12 |
| sean-k-mooney | cdent: i thihk at this point they are more or less the same although you remove a test i didnt need too | 13:14 |
| *** edleafe has joined #openstack-nova | 13:14 | |
| sean-k-mooney | cdent: also your's has a better commit message but it does not have the close bug line "Closes-Bug: #1822575" for the bug i opened to track it | 13:14 |
| openstack | bug 1822575 in OpenStack Compute (nova) "lower-constraints are not used in gate job" [Undecided,In progress] https://launchpad.net/bugs/1822575 - Assigned to sean mooney (sean-k-mooney) | 13:15 |
| sean-k-mooney | cdent: im fine to abandon min in favor of yours as long as one of them is merged | 13:15 |
| *** mvkr has joined #openstack-nova | 13:16 | |
| efried | They do seem to differ a fair bit in the updates to l-c | 13:18 |
| sean-k-mooney | they do | 13:19 |
| efried | and requirements.txt | 13:19 |
| sean-k-mooney | i tried to make as few chagnes as i coudl initally but ended up pulling in some of the changes cdent had made. | 13:19 |
| efried | sean-k-mooney: You're not thinking to backport this to stein, are you? | 13:19 |
| sean-k-mooney | efried: no. we could but if we did the lower constraitn version change would have to be different | 13:20 |
| sean-k-mooney | well we proably should backport somthing but it cant be exactly the same | 13:20 |
| efried | backporting requirements changes is messy under the best of circumstances; I would think we should avoid doing it at all unless there's a really good reason. | 13:21 |
| sean-k-mooney | efried: well what i was planning to do was backport the tox change only and see if it worked on stable* | 13:21 |
| sean-k-mooney | the older the branch the more likely it will | 13:22 |
| efried | but not stein | 13:22 |
| sean-k-mooney | stein need some of the version bumps or the unitest wont work | 13:22 |
| sean-k-mooney | for example oslo.messaging 7.0.0 is need for the mocking of RPC calls to work properly | 13:23 |
| *** ttsiouts has quit IRC | 13:24 | |
| sean-k-mooney | were as branches that test with ubuntu-xenial dont ned the psycopg2 bump as i think they wont have postgresql 10.3 | 13:24 |
| *** cdent has quit IRC | 13:25 | |
| *** ttsiouts has joined #openstack-nova | 13:25 | |
| *** irclogbot_1 has joined #openstack-nova | 13:26 | |
| efried | sean-k-mooney: I'm asking thode about it in -requirements. | 13:27 |
| efried | cdent also ^ | 13:27 |
| efried | sean-k-mooney: meanwhile, would you please ack https://review.openstack.org/#/c/647512/ for me? | 13:27 |
| efried | mriedem: have you started looking at the lm job yet? | 13:27 |
| mriedem | efried: i didn't see any bug reported for it | 13:28 |
| efried | Yeah, I haven't opened one or looked into it at all, just noticed (on my bug mail on my phone) that it's failing 100% since like Friday. | 13:28 |
| sean-k-mooney | if by ack you mean +1 ill review it now but yes i think this is the more complete fix for the patch i abandoned | 13:28 |
| efried | just didn't want to start digging if you were already in there | 13:28 |
| mriedem | efried: first morning back from vacation so no i'm not on it | 13:28 |
| *** cfriesen has joined #openstack-nova | 13:29 | |
| efried | sean-k-mooney: Yes, that's what I'm looking for, since you had already started to think about the solution previously. | 13:29 |
| *** rcernin has quit IRC | 13:29 | |
| efried | mriedem: ack. I could actually use a brief tutorial on how you do the triage here. You seem to have a system for identifying when failures started and registering the signature of the problem somewhere (e-r?). | 13:30 |
| efried | if you have time. | 13:30 |
| mriedem | efried: i'd need a link to a failing job first | 13:30 |
| efried | teach me to fish, in other words | 13:30 |
| efried | mriedem: stand by | 13:31 |
| efried | (but any patch from the past three days) | 13:31 |
| mriedem | ok http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/ | 13:31 |
| efried | https://review.openstack.org/#/c/648653/ | 13:31 |
| mriedem | looks like this is the problem http://logs.openstack.org/00/641200/7/check/nova-live-migration/d55c8ab/job-output.txt.gz#_2019-04-01_11_12_47_685033 | 13:31 |
| mriedem | Multiple possible networks found, use a Network ID to be more specific. (HTTP 409) (Request-ID: req-6c6a7e2b-6e5a-4ff4-ba6a-a3aaa8d8f83a) | 13:31 |
| mriedem | which probably means https://github.com/openstack-dev/devstack/commit/0d4c9c9fa37c6f47523359044e5f65629321ff6d | 13:32 |
| mriedem | so if i take "Multiple possible networks found, use a Network ID to be more specific" from that nova-live-migration console log and dump it in logstash, i bet it starts around 3/29 | 13:33 |
| *** cdent has joined #openstack-nova | 13:34 | |
| efried | clearly I don't know how to use logstash | 13:35 |
| mriedem | http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22Multiple%20possible%20networks%20found%2C%20use%20a%20Network%20ID%20to%20be%20more%20specific%5C%22%20AND%20tags%3A%5C%22console%5C%22%20AND%20build_name%3A%5C%22nova-live-migration%5C%22&from=7d | 13:35 |
| mriedem | so the message filter is part of the message above, excluding anything unique like a request id | 13:36 |
| mriedem | or timestamp | 13:36 |
| mriedem | the 'tags' is the file the message is in, so 'console' in this case | 13:36 |
| mriedem | and i threw build_name in there, which we don't normally do, but figured i would to avoid noise from falst negatives in functional jobs or something | 13:36 |
| mriedem | *false | 13:36 |
| efried | how did you get it to span the correct time frame? | 13:37 |
| *** yan0s has joined #openstack-nova | 13:37 | |
| mriedem | anyway, my guess is https://review.openstack.org/#/c/601433/ because (1) devstack creates the networks we're using here and (2) it doesn't fail for tempest because tempest always provides a specific nic when creating a server, but in the evacuate tests in nova-live-migration we don't | 13:37 |
| mriedem | used the dropdown at the top right | 13:37 |
| mriedem | https://wiki.openstack.org/wiki/BootstrappingHour/Debugging_Gate is old but might be relevant still | 13:38 |
| efried | Cool. So now a bug needs to be opened, and then registered in e-r? | 13:38 |
| mriedem | yeah | 13:38 |
| *** jroll has quit IRC | 13:39 | |
| efried | how did you get from the error message to the patch? Did you just happen to know about that devstack change? | 13:39 |
| *** eharney has joined #openstack-nova | 13:39 | |
| *** jroll has joined #openstack-nova | 13:39 | |
| mriedem | i didn't, but i know that devstack creates the network we use in testing, | 13:40 |
| mriedem | so assumed that because we're now failing on ambiguous network errors, someone started creating a 2nd network in devstack | 13:41 |
| mriedem | lo and behold someone did | 13:41 |
| efried | do you want me to open the bug? | 13:41 |
| mriedem | sure | 13:41 |
| yonglihe | orphan patch was splited: https://review.openstack.org/#/q/topic:bug/1820802+(status:open+OR+status:merged) nova-live-migration test fail, but the test result said all test passed... | 13:42 |
| efried | against nova or against devstack? | 13:42 |
| mriedem | yonglihe: we're talking about that nova-live-migration failure right now | 13:42 |
| mriedem | efried: i guess start with nova | 13:42 |
| yonglihe | ooh, got you. | 13:42 |
| mriedem | it's really a behavior change in devstack, but it looks like it's only breaking nova-live-migration right now | 13:42 |
| sean-k-mooney | efried: one nit inline but yep it looks good to me. i left a +1 to recored the same | 13:43 |
| mriedem | i guess our options are (1) make nova-live-migration pick a network if there are multiple or (2) add a variable to devstack to toggle whether or not it creates a shared network | 13:44 |
| mriedem | #1 is more flexible to env changes, but i can see myself being really annoyed without #2 in local devstack testing if i have to always specify a network now | 13:45 |
| efried | mriedem: https://bugs.launchpad.net/nova/+bug/1822605 | 13:45 |
| openstack | Launchpad bug 1822605 in OpenStack Compute (nova) "nova-live-migration fails 100% with "Multiple possible networks found, use a Network ID to be more specific"" [Undecided,New] | 13:45 |
| *** mrjk has quit IRC | 13:45 | |
| mriedem | although i guess i always disable tempest in my devstack setup local.conf so i won't hit #2 | 13:46 |
| efried | mriedem: okay, so how do I register this with e-r? | 13:46 |
| mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html | 13:47 |
| *** mrjk has joined #openstack-nova | 13:48 | |
| mriedem | https://docs.openstack.org/infra/elastic-recheck/readme.html#queries specifically | 13:48 |
| sean-k-mooney | mriedem: isnt there already a tempest config option to specify the network to use | 13:48 |
| mriedem | sean-k-mooney: see https://review.openstack.org/#/c/615425/ | 13:48 |
| mriedem | but what's failing isn't tempest | 13:48 |
| *** awaugama has joined #openstack-nova | 13:50 | |
| *** burt has joined #openstack-nova | 13:51 | |
| *** mdbooth_ is now known as mdbooth | 13:52 | |
| *** tetsuro has joined #openstack-nova | 13:53 | |
| sean-k-mooney | mriedem: sorry thats too much of a context switch to figure out if the review you linked helps or not. | 13:53 |
| sean-k-mooney | i need either more or less coffee but not the amount i have currently consumed | 13:53 |
| mdbooth | kaisers: Just found a bug in https://review.openstack.org/#/c/648093/ which just merged. Nothing catastrophic, but definitely worthy of another follow-up. | 13:53 |
| *** jmlowe has joined #openstack-nova | 13:55 | |
| mriedem | efried: i should have a patch up shortly | 13:55 |
| sean-k-mooney | mriedem: #2 sound resonable just in general as a way to auto create an env with a shared network | 13:55 |
| sean-k-mooney | althouhg i often just do that via a local.sh file which you could also do in the gate job | 13:57 |
| *** tbachman has quit IRC | 13:57 | |
| *** takashin has joined #openstack-nova | 13:58 | |
| efried | mriedem: https://review.openstack.org/#/c/649034/ ? | 13:58 |
| *** sridharg has quit IRC | 13:59 | |
| *** cdent has quit IRC | 13:59 | |
| *** erlon has quit IRC | 14:00 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Pass --nic when creating servers in evacuate integration test script https://review.openstack.org/649036 | 14:00 |
| *** awalende has quit IRC | 14:01 | |
| *** awalende has joined #openstack-nova | 14:01 | |
| sean-k-mooney | https://bugs.launchpad.net/tempest/+bug/1821007 is ment to be fixed right? | 14:02 |
| openstack | Launchpad bug 1821007 in tempest "The field 'environment' has an invalid value, which includes an undefined variable. The error was: 'gabbi_tempest_path' is undefined" [Critical,Fix released] - Assigned to Paul Belanger (pabelanger) | 14:02 |
| mriedem | efried: +W | 14:02 |
| mriedem | sean-k-mooney: yes http://status.openstack.org/elastic-recheck/#1821007 | 14:02 |
| sean-k-mooney | mriedem: im still hitting it in my thirdparty ci job and just trying to figure out why | 14:03 |
| *** priteau has quit IRC | 14:03 | |
| *** tetsuro has quit IRC | 14:04 | |
| *** mrhillsman_wkend is now known as mrhillsman | 14:04 | |
| *** awalende has quit IRC | 14:06 | |
| *** belmoreira has joined #openstack-nova | 14:08 | |
| sean-k-mooney | for some reason it appears to be using a commit form 7 days ago which was the commit just before that merged. | 14:11 |
| efried | sean-k-mooney: prometheanfire doesn't care which l-c fix as long as it passes. I'm going to take cdent's because a) older, b) more comprehensive commit message. Cool? | 14:11 |
| *** yan0s has quit IRC | 14:11 | |
| sean-k-mooney | efried: yep that is fine with me | 14:12 |
| sean-k-mooney | efried: can we land it in stien or no? | 14:12 |
| efried | sean-k-mooney: That I don't know. Sounds hairy to me. | 14:13 |
| efried | sean-k-mooney: What would be the motivation to do that? | 14:13 |
| mriedem | we don't need an rc2 for lower-constraints fixes right? | 14:13 |
| efried | Is it going to break distros or something? | 14:13 |
| *** tbachman has joined #openstack-nova | 14:13 | |
| mriedem | also, how latent is this? | 14:13 |
| sean-k-mooney | mriedem: it has always been broken | 14:13 |
| sean-k-mooney | since we introduced lower-constarits | 14:14 |
| sean-k-mooney | i think | 14:14 |
| mriedem | then it's definitely not an rc2 issue | 14:14 |
| sean-k-mooney | mriedem: correcct | 14:14 |
| sean-k-mooney | i was wondering if we want to backport it after rc2 | 14:14 |
| sean-k-mooney | e.g. do we want to fix lower-constratins on stable branches or not in general | 14:14 |
| sean-k-mooney | if the answer is no we proably shoudl disable the job on the stable branches | 14:15 |
| *** tbachman has quit IRC | 14:20 | |
| *** zhubx has quit IRC | 14:22 | |
| *** cdent has joined #openstack-nova | 14:22 | |
| *** phasespace has quit IRC | 14:22 | |
| sean-k-mooney | mriedem: oh i realised i have my own version of devstack-tempest because i cant import jobs from tempest without needing to pull in amost all of openstack into my zuul configruation. so ya i need to copy the same fix. thanks for confiming its fixed upstream | 14:27 |
| *** rchurch has joined #openstack-nova | 14:27 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: tests: Full stub out os_vif https://review.openstack.org/648748 | 14:30 |
| *** mlavalle has joined #openstack-nova | 14:33 | |
| *** med_ has joined #openstack-nova | 14:39 | |
| *** derekh has quit IRC | 14:40 | |
| *** derekh has joined #openstack-nova | 14:40 | |
| *** mrch_ has quit IRC | 14:41 | |
| *** wolverineav has joined #openstack-nova | 14:41 | |
| *** yan0s has joined #openstack-nova | 14:42 | |
| *** dpawlik has quit IRC | 14:44 | |
| *** wolverineav has quit IRC | 14:46 | |
| *** janki has quit IRC | 14:47 | |
| *** janki has joined #openstack-nova | 14:47 | |
| *** belmoreira has quit IRC | 14:47 | |
| *** itlinux_ has quit IRC | 14:50 | |
| *** belmoreira has joined #openstack-nova | 14:53 | |
| *** dpawlik has joined #openstack-nova | 14:56 | |
| mriedem | efried: looks like https://review.openstack.org/#/c/649036/ fixes the problem, nova-live-migration job is past that issue now, but will need to recheck the change | 14:56 |
| efried | ack | 14:57 |
| efried | mriedem: grenade failure spurious? | 14:57 |
| efried | ssh auth error | 14:58 |
| bauzas | I just feel we have enough consensus with https://review.openstack.org/#/c/647512/4 | 14:59 |
| bauzas | efried: alex_xu: ^ | 14:59 |
| mriedem | efried: yes | 15:00 |
| efried | bauzas: Yes. Let's wait for the gate fix before pushing this into the queue. | 15:00 |
| *** dpawlik has quit IRC | 15:00 | |
| *** takashin has left #openstack-nova | 15:01 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Do not persist RequestSpec.ignore_hosts https://review.openstack.org/647512 | 15:01 |
| efried | bauzas: I rebased on the gate fix and pushed ^ | 15:01 |
| efried | mriedem: any reason to wait to recheck? | 15:02 |
| mriedem | efried: i'm not sure it will do anything while zuul is still running | 15:03 |
| bauzas | cool, mriedem: planning to propose the backports or want me doing it ? | 15:03 |
| mriedem | bauzas: for the RequestSpec.ignore_hosts thing? i'll handle it. | 15:03 |
| *** ttsiouts has quit IRC | 15:04 | |
| *** ttsiouts has joined #openstack-nova | 15:04 | |
| *** lpetrut has quit IRC | 15:04 | |
| bauzas | mriedem: ok :) | 15:05 |
| *** ivve has quit IRC | 15:06 | |
| *** _alastor_ has joined #openstack-nova | 15:06 | |
| *** priteau has joined #openstack-nova | 15:08 | |
| efried | answer appears to be no, the recheck doesn't kick the running job out. | 15:08 |
| *** ttsiouts has quit IRC | 15:09 | |
| dansmith | you have to change the patch to kick it out of any queue | 15:09 |
| dansmith | no vote or comment will do it | 15:09 |
| *** ttsiouts has joined #openstack-nova | 15:11 | |
| *** owalsh has quit IRC | 15:11 | |
| *** dpawlik has joined #openstack-nova | 15:15 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Add a script for counting blueprints https://review.openstack.org/581914 | 15:16 |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Add a process to abandon a spec https://review.openstack.org/648800 | 15:16 |
| *** dpawlik has quit IRC | 15:19 | |
| openstackgerrit | Merged openstack/nova stable/queens: [Stable Only] hardware: Handle races during pinning https://review.openstack.org/647831 | 15:21 |
| *** luksky has quit IRC | 15:23 | |
| openstackgerrit | Chris Dent proposed openstack/nova master: Add placement as required project to functional py36 and 37 https://review.openstack.org/649068 | 15:23 |
| *** belmoreira has quit IRC | 15:25 | |
| *** belmoreira has joined #openstack-nova | 15:25 | |
| *** sapd1_x has joined #openstack-nova | 15:25 | |
| *** owalsh has joined #openstack-nova | 15:26 | |
| *** owalsh has quit IRC | 15:27 | |
| bauzas | efried: sorry, just saw your question, well, like dansmith said, you need to provide a new PS to put off the check or gate pipeline | 15:31 |
| bauzas | just use the right words, I wasn't understanding your question | 15:32 |
| efried | nod. | 15:32 |
| *** tbachman has joined #openstack-nova | 15:33 | |
| bauzas | but let's ask jaypipes to provide a follow-up post from http://www.joinfu.com/2014/01/understanding-the-openstack-ci-system/ :p | 15:33 |
| *** owalsh has joined #openstack-nova | 15:33 | |
| bauzas | or just digest https://zuul-ci.org/docs/zuul/user/gating.html :p | 15:34 |
| bauzas | huh, rather https://zuul-ci.org/docs/zuul/user/concepts.html | 15:34 |
| *** ircuser-1 has joined #openstack-nova | 15:36 | |
| *** janki has quit IRC | 15:37 | |
| *** ttsiouts has quit IRC | 15:41 | |
| *** tssurya has quit IRC | 15:45 | |
| *** itlinux has joined #openstack-nova | 15:47 | |
| *** tbachman has quit IRC | 15:47 | |
| *** priteau has quit IRC | 15:48 | |
| *** yan0s has quit IRC | 15:54 | |
| jaypipes | bauzas: erm... | 15:57 |
| jaypipes | bauzas: yeah, "Just Use Zuul CI" | 15:57 |
| *** _alastor_ has quit IRC | 15:57 | |
| *** _alastor_ has joined #openstack-nova | 15:57 | |
| *** wolverineav has joined #openstack-nova | 16:00 | |
| *** awalende has joined #openstack-nova | 16:01 | |
| *** wolverineav has quit IRC | 16:04 | |
| *** awalende has quit IRC | 16:07 | |
| bauzas | jaypipes: FWIW, I give you credits on helping some French folk who was working on a Stackforge project named "Climate" to know why Jenkins was telling him -1 sometimes | 16:07 |
| *** dtantsur is now known as dtantsur|afk | 16:07 | |
| bauzas | then, 5 months after, he was working on Nova | 16:07 |
| bauzas | err, 4 | 16:08 |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Tools & docs for backlog & abandoned spec process https://review.openstack.org/648800 | 16:08 |
| *** tbachman has joined #openstack-nova | 16:09 | |
| openstackgerrit | Merged openstack/nova master: tests: Stub out privsep modules https://review.openstack.org/648747 | 16:10 |
| *** rpittau is now known as rpittau|afk | 16:13 | |
| *** temka has quit IRC | 16:14 | |
| *** cdent has quit IRC | 16:14 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Abandon flavor-classes backlog spec https://review.openstack.org/649095 | 16:18 |
| mnaser | hmm | 16:26 |
| *** wolverineav has joined #openstack-nova | 16:27 | |
| *** sapd1_x has quit IRC | 16:30 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Abandon instance-tasks backlog spec https://review.openstack.org/649100 | 16:30 |
| *** wolverineav has quit IRC | 16:32 | |
| *** erlon has joined #openstack-nova | 16:32 | |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Abandon parallel-scheduler backlog spec https://review.openstack.org/649101 | 16:34 |
| openstackgerrit | Eric Fried proposed openstack/nova-specs master: Abandon same-instances-scheduling backlog spec https://review.openstack.org/649102 | 16:36 |
| *** Sundar has joined #openstack-nova | 16:39 | |
| *** belmoreira has quit IRC | 16:42 | |
| *** wolverineav has joined #openstack-nova | 16:44 | |
| openstackgerrit | Stephen Finucane proposed openstack/nova master: Raise Exception instead of Exception method call https://review.openstack.org/482200 | 16:48 |
| *** wolverineav has quit IRC | 16:49 | |
| *** wolverin_ has joined #openstack-nova | 16:49 | |
| *** wolverin_ has quit IRC | 16:51 | |
| *** wolverineav has joined #openstack-nova | 16:52 | |
| *** wolverineav has quit IRC | 16:53 | |
| *** wolverineav has joined #openstack-nova | 16:53 | |
| *** derekh has quit IRC | 17:02 | |
| *** davidsha has quit IRC | 17:04 | |
| *** erlon has quit IRC | 17:10 | |
| *** Sundar has quit IRC | 17:18 | |
| *** eandersson has joined #openstack-nova | 17:22 | |
| *** Sundar has joined #openstack-nova | 17:25 | |
| *** zigo has quit IRC | 17:37 | |
| *** phasespace has joined #openstack-nova | 17:38 | |
| *** wolverineav has quit IRC | 17:39 | |
| *** wolverineav has joined #openstack-nova | 17:40 | |
| *** wolverin_ has joined #openstack-nova | 17:43 | |
| *** wolverineav has quit IRC | 17:44 | |
| *** wolverin_ has quit IRC | 17:47 | |
| mnaser | just to confirm | 17:58 |
| *** tesseract has quit IRC | 17:58 | |
| mnaser | nova-consoleauth is no longer technically needed as of stein | 17:58 |
| mnaser | assuming all the "cleanup" of remaining things was killed off in rocky? | 17:59 |
| *** mvkr has quit IRC | 18:01 | |
| *** wolverineav has joined #openstack-nova | 18:05 | |
| melwitt | mnaser: right, not technically needed. only if you needed already existing console auths to work during a rolling upgrade, or if you have long TTL and didn't want things invalidated immediately | 18:12 |
| melwitt | what do you mean by "cleanup" of remaining things? | 18:12 |
| mnaser | melwitt: yeah I meant that if you didn't want things to be smooth and consoles not be invalidated | 18:14 |
| mnaser | now curiously I have this other issue where the calculated token hash doesn't seem to match the one in the db.. | 18:14 |
| mnaser | I get token id `dc574410-2c5c-4762-ae0d-f1689372337f` .. the sha256 is b7cc4998dea5dd0a3b663fcbe8df7c487246b3d3e52ec3361530b091477e5b28 which matches the db entry | 18:16 |
| mnaser | but when I do a print(token_hash) in the novncproxy codebase.. I get e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855 | 18:16 |
| *** wolverineav has quit IRC | 18:16 | |
| melwitt | hm, that's strange. I'm not aware of anything that could have changed there | 18:17 |
| *** wolverineav has joined #openstack-nova | 18:17 | |
| mnaser | ok, looks like the function is being called with token == '' | 18:17 |
| melwitt | hm... empty string should mean that the GET url didn't have the token in it | 18:19 |
| mnaser | yeah url im hitting has a token, im following it up the stack, im up to _get_connect_info_database() and that gets a token == none | 18:19 |
| melwitt | that's where the token comes from (GET request param or cookie) in the novncproxy | 18:20 |
| *** wolverineav has quit IRC | 18:22 | |
| *** wolverineav has joined #openstack-nova | 18:23 | |
| melwitt | mnaser: https://github.com/openstack/nova/blob/master/nova/console/websocketproxy.py#L195 | 18:23 |
| mnaser | melwitt: that gets called when we call /websockify which doesn't seem to have a token | 18:23 |
| mnaser | print('path', self.path) in here shows | 18:23 |
| mnaser | ('path', '/websockify') | 18:23 |
| mnaser | oh crap, I have a dr appointment in 5 minutes gotta run, ill dig into this after if you don't keep looking | 18:25 |
| mnaser | but looks like unless its not sending cookies with a token.. we can't do anything | 18:25 |
| melwitt | ok. I'll keep looking. I'm not familiar with the /websockify path so I need to find what that is | 18:25 |
| melwitt | because usually the path should be like this http://ip:port/?token=xyz | 18:26 |
| *** jonher has quit IRC | 18:29 | |
| *** jonher has joined #openstack-nova | 18:29 | |
| mnaser | melwitt: indeed and xyz is there but it looks like the token lookup seems to happen in the websockify part | 18:33 |
| * mnaser is on mobile while waiting | 18:33 | |
| mnaser | Weird thing is that code has been there for 5 years so.. | 18:34 |
| *** igordc has joined #openstack-nova | 18:35 | |
| mnaser | Unless the behaviour to forward token from cookie was removed with more recent novnc versions | 18:35 |
| melwitt | yeah, I know. I'm puzzled as to what's going on here | 18:35 |
| mnaser | We pull master novnc in OSA so mahbe its recent behaviour change | 18:35 |
| melwitt | I'll take a look in novnc | 18:36 |
| mnaser | Okay | 18:37 |
| melwitt | I thought the cookie thing was a backup, if the token is not in the GET request params, but forwarding via cookie does sound really familiar | 18:37 |
| mnaser | melwitt: we use vnc_lite.html in OSA. https://github.com/novnc/noVNC/commit/51f9f0098d306bbc67cc8e02ae547921b6f6585c | 18:37 |
| melwitt | I had dug into this stuff back when I worked on the db backend for console auths, back when we were thinking we could pass the instance uuid in the GET request too (turns out bc of novnc limitations, more than one request param would not work) | 18:38 |
| melwitt | (it only forwards the first request param) | 18:38 |
| mnaser | So it always probably used a cookie but that seems to have been pulled out from vnc_lite.html which is for some historic reason what’s being used in OSA | 18:39 |
| melwitt | huh, interesting | 18:40 |
| mnaser | The worrying thing is I can’t find any token references in novnc | 18:40 |
| melwitt | / If a token variable is passed in, set the parameter in a cookie. | 18:40 |
| melwitt | / This is used by nova-novncproxy. | 18:40 |
| mnaser | Where do you see this melwitt ? | 18:40 |
| melwitt | seeing that all in red being removed. "used by nova-novncproxy" :P | 18:40 |
| melwitt | mnaser: in the commit you linked https://github.com/novnc/noVNC/commit/51f9f0098d306bbc67cc8e02ae547921b6f6585c#diff-1d6838e3812778e95699b90d530543a1L173 | 18:41 |
| mnaser | ugh | 18:41 |
| mnaser | What does devstack use | 18:41 |
| mnaser | I wonder how it hasn’t been affected | 18:42 |
| melwitt | must not be using lite? | 18:42 |
| melwitt | bc this was removed from the vnc_lite.html file, unless I'm missing something | 18:43 |
| * melwitt looks at a recent devstack run | 18:43 | |
| mnaser | Of course | 18:43 |
| mnaser | NOVNC_FROM_PACKAGE defined in jobs | 18:43 |
| *** itlinux has quit IRC | 18:44 | |
| melwitt | http://logs.openstack.org/65/627765/12/check/tempest-full-py3/b00b2ab/controller/logs/devstacklog.txt.gz#_2019-04-01_08_01_25_051 | 18:44 |
| mnaser | Yikes well i guess we can ask novnc to revert | 18:45 |
| *** mvkr has joined #openstack-nova | 18:48 | |
| melwitt | mnaser: I don't understand, how do you use lite vs not? | 18:49 |
| mnaser | melwitt: vnc_lite seems to be what vnc_auto was a while ago | 18:50 |
| mnaser | Searching the code doesn’t give any results for token | 18:50 |
| melwitt | mnaser: and which one you use depends on the url you use? | 18:50 |
| melwitt | yeah, ok. looks like we have to ask for a revert of at least that part | 18:51 |
| mnaser | melwitt: I assume we’re using the lite which handled the token stuff and did the cookie handoff | 18:51 |
| melwitt | mnaser: yeah, I'm just saying, how do you choose whether to use lite or not | 18:51 |
| mnaser | dunno :P | 18:51 |
| melwitt | haha ok | 18:52 |
| mnaser | It seems like there’s something called token plugin | 18:52 |
| melwitt | at first I was thinking, can we use non-lite but then you pointed out only lite was handling token | 18:52 |
| *** wolverineav has quit IRC | 18:52 | |
| melwitt | so my thought was moo | 18:52 |
| mnaser | Don’t quote me on this stuff I’m going over it on my phone heh | 18:52 |
| *** wolverineav has joined #openstack-nova | 18:53 | |
| melwitt | :) | 18:53 |
| melwitt | I do wonder why we have to rely on the cookie forwarding. I wonder why it doesn't just show up as a GET request param by the time we get it | 18:54 |
| melwitt | from the way our code is written, it looks like the expectation was that the cookie was a fall back | 18:54 |
| *** itlinux has joined #openstack-nova | 18:55 | |
| *** tbachman has quit IRC | 18:55 | |
| melwitt | but in practice, we were always falling back | 18:55 |
| *** BjoernT has joined #openstack-nova | 18:55 | |
| *** ralonsoh has quit IRC | 18:56 | |
| melwitt | going to see if I can find out | 18:56 |
| *** wolverineav has quit IRC | 18:58 | |
| *** erlon has joined #openstack-nova | 18:59 | |
| *** wolverineav has joined #openstack-nova | 19:03 | |
| *** wolverineav has quit IRC | 19:07 | |
| *** eharney has quit IRC | 19:09 | |
| *** erlon has quit IRC | 19:11 | |
| *** erlon_ has joined #openstack-nova | 19:11 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | 19:15 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Correct lower-constraints.txt and the related tox job https://review.openstack.org/622972 | 19:16 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Mock time.sleep() in unit tests https://review.openstack.org/648762 | 19:17 |
| *** tbachman has joined #openstack-nova | 19:18 | |
| *** jmlowe has quit IRC | 19:22 | |
| mnaser | melwitt: so I think the problem here is the /websockify HTTP request comes afterwards | 19:25 |
| mnaser | so its GET /vnc_lite.html?token=foobar .. and then CONNECT /websockify | 19:26 |
| mnaser | so you can't really reliably tie that GET request and the CONNECT request | 19:26 |
| mnaser | UNLESS we sent a Set-Cookie: token when you visit vnc_lite.html | 19:26 |
| mnaser | that would workaround it | 19:26 |
| melwitt | mnaser: I see. well, from some googling it looks like something like this might work? GET /vnc_lite.html?path=/websockify?token=foobar | 19:27 |
| openstackgerrit | Eric Fried proposed openstack/nova master: Add placement as required project to functional py36 and 37 https://review.openstack.org/649068 | 19:28 |
| melwitt | second ? should probably be a & | 19:28 |
| mnaser | I think it would be cleaner to set-cookie | 19:28 |
| melwitt | or maybe not | 19:28 |
| melwitt | mnaser: ok. I can see if I can figure out how to do that, unless you want to | 19:30 |
| mnaser | melwitt: im looking around right now, just need to figure out where the HTTP requests are handled in nova-novncconsole | 19:32 |
| mnaser | ill welcome all help :P | 19:32 |
| mnaser | *novncproxy | 19:32 |
| mnaser | think it relies on upstream this https://github.com/novnc/websockify/blob/master/websockify/websocketproxy.py#L267 | 19:34 |
| mnaser | self.token_plugin = kwargs.pop('token_plugin', None) | 19:34 |
| mnaser | oh I wonder what that is | 19:34 |
| melwitt | mnaser: I looked at that and it doesn't seem like something that would help us. but I could be wrong https://github.com/novnc/websockify/wiki/Token-based-target-selection | 19:35 |
| *** luksky has joined #openstack-nova | 19:35 | |
| *** spsurya has quit IRC | 19:36 | |
| *** artom has joined #openstack-nova | 19:36 | |
| mnaser | maybe we somehow override the html portion | 19:37 |
| mnaser | https://github.com/novnc/websockify/blob/master/websockify/websocketserver.py#L35-L52 | 19:37 |
| melwitt | mnaser: I think handling of HTTP (GET) requests might be this https://github.com/novnc/websockify/blob/6e09ec25486f431e2acc463f994c4eb4bebc6ddd/websockify/websockifyserver.py#L268 | 19:37 |
| mnaser | oh that's even better | 19:37 |
| melwitt | one of those, I don't really understand it yet | 19:38 |
| efried | This one came across my desk again just now, can I please get a couple of pairs of core eyeballs on it? https://review.openstack.org/#/c/635533/ | 19:39 |
| mnaser | I'm thinking https://github.com/novnc/websockify/wiki/Token-based-target-selection#custom-plugins | 19:41 |
| *** wolverineav has joined #openstack-nova | 19:41 | |
| melwitt | mnaser: hm. so instead of validating the token in new_websocket_client we'd do it in a new lookup method? | 19:43 |
| melwitt | 'lookup' | 19:43 |
| mnaser | melwitt: yeah, if we write a custom plugin, we can validate the token in it and then give it back to novnc | 19:43 |
| efried | mriedem: This looks like a deadlock starting n-api? http://logs.openstack.org/36/649036/1/check/tempest-full-py3/a8c7223/controller/logs/screen-n-api.txt.gz | 19:43 |
| *** wolverineav has quit IRC | 19:43 | |
| melwitt | mnaser: it says lookup has to return a host:port tuple? | 19:43 |
| *** wolverineav has joined #openstack-nova | 19:43 | |
| mnaser | or none, none (if we don't validate I guess) | 19:44 |
| efried | (maybe not, I guess the locks are being released quickly) | 19:44 |
| mnaser | and the host:port tuple business can be grabbed from the db | 19:44 |
| mnaser | the example module there seems to be reasonable | 19:46 |
| mnaser | ugh but even then | 19:47 |
| mnaser | I don't think anything will pass ?token=foo | 19:47 |
| melwitt | mnaser: ok, I was trying to say when you said "give the token back to novnc" I don't think you can do that in lookup() | 19:48 |
| mnaser | oooo | 19:48 |
| melwitt | mnaser: earlier I was saying maybe we could do the entire validation in lookup() and then no longer need the token after that? | 19:48 |
| mnaser | melwitt: https://github.com/novnc/websockify/blob/6e09ec25486f431e2acc463f994c4eb4bebc6ddd/websockify/websocketproxy.py#L164-L175 | 19:49 |
| melwitt | yeah | 19:49 |
| mnaser | so we get lookup called with the token, grab the stuff from the db, and return it back, and we can probably drop all that extra code | 19:49 |
| melwitt | what do you mean by return it back? | 19:49 |
| mnaser | so in the token plugin, we check the db, see if the var is still valid, and return (host, port) | 19:50 |
| mnaser | and technically... I think none will just do the rest of the issues | 19:50 |
| melwitt | ok. I thought you meant return the token | 19:50 |
| melwitt | ok. that's what I was trying to say earlier | 19:50 |
| mnaser | ah no I meant return the host/port, sorry distracted | 19:50 |
| mnaser | people keep walking into my office >:( | 19:51 |
| melwitt | how dare they! | 19:51 |
| mnaser | let me try to hack at this | 19:52 |
| melwitt | sounds good. I will review it when it's ready :) | 19:54 |
| mnaser | honestly I think this could completely eliminate the websocketproxy.py | 19:54 |
| mnaser | a lot of that stuff exists in webproxify but I dunno how much of this can be back portable at this point.. | 19:54 |
| melwitt | maybe. there is some extra stuff in there though | 19:55 |
| *** awalende has joined #openstack-nova | 20:03 | |
| *** awalende has quit IRC | 20:07 | |
| *** jmlowe has joined #openstack-nova | 20:09 | |
| *** jmlowe has quit IRC | 20:09 | |
| mnaser | melwitt: well, I got a nice refactor out of it, but still I think it expects I to be there | 20:19 |
| melwitt | mnaser: yeah, you'd have to change https://github.com/openstack/nova/blob/master/nova/cmd/baseproxy.py#L68 too | 20:21 |
| *** xek has quit IRC | 20:22 | |
| mnaser | ok well to get it just to work I found a gross fix.. I think | 20:23 |
| mnaser | let me test it against the original code | 20:23 |
| mnaser | vnc_lite.html?path=?token=<token> seems to be calling my token plugin | 20:23 |
| mnaser | But I think then the original code works | 20:25 |
| mnaser | We would just need to tweak the default value :/ | 20:25 |
| melwitt | yeah, shouldn't need path= right | 20:25 |
| melwitt | default value of what? | 20:25 |
| *** jmlowe has joined #openstack-nova | 20:26 | |
| mnaser | The nova configuration value | 20:27 |
| mnaser | The one pointing at vnc_lite or whatever | 20:27 |
| mnaser | To not longer suggest ?token=foo is the way but ?path=?token=foo | 20:28 |
| melwitt | I thought earlier you told me including path= wasn't as clean :P | 20:28 |
| mnaser | Yeah it’s just BLAH | 20:28 |
| mnaser | why people gotta rip code out | 20:28 |
| melwitt | I guess, I thought the token plugin would let us use the old default url | 20:29 |
| mnaser | Same. | 20:29 |
| mnaser | But once it was done I realized the web socket thing was only checking it at the web socket stage | 20:29 |
| melwitt | because if we include path in the default, we could fix this without the token plugin, I think, as I mentioned earlier | 20:29 |
| mnaser | Yeah then we won’t be falling back to cookies every time | 20:29 |
| melwitt | right | 20:30 |
| *** whoami-rajat has quit IRC | 20:30 | |
| mnaser | I guess we can propose a revert and then I’ll just pin on the old commit :/ | 20:31 |
| mriedem | efried: you're talking about get_or_set_cached_cell_and_set_connections right? | 20:31 |
| efried | mriedem: yeah | 20:32 |
| efried | I didn't go back and look whether that's "normal". | 20:32 |
| mriedem | i think that's nova.compute.api.API being constructed per controller per worker | 20:32 |
| efried | k | 20:32 |
| mriedem | which is a lot since it's not a singleton | 20:32 |
| efried | Last week I noticed pretty frequent failures sprayed across the long-running jobs. Never more than one per patch, and never the same one twice in a row. | 20:33 |
| efried | but I didn't dig into any of them. | 20:33 |
| mriedem | i assume the get_or_set_cached_cell_and_set_connections target_cell is due to a query on the services table when constructing the compute RPCAPI with [upgrade_levels]/compute=auto | 20:33 |
| mriedem | https://bugs.launchpad.net/nova/+bug/1815697 | 20:33 |
| openstack | Launchpad bug 1815697 in OpenStack Compute (nova) "[upgrade_levels]compute=auto grinds the API response times when a cell is down" [Medium,Confirmed] | 20:33 |
| mnaser | I honestly don’t have that much time to dig more into this. I can report a bug and add a known issue (if needed) but I don’t have time to deal with getting this merged into novnc :/ | 20:33 |
| mriedem | efried: also https://bugs.launchpad.net/nova/+bug/1815692 | 20:33 |
| openstack | Launchpad bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged] | 20:34 |
| melwitt | mnaser: or we could change our default nova conf value? or do you think asking for a revert is a better route? | 20:34 |
| mnaser | melwitt: I think the amount of downstream stuff that will break (because everyone configures this in their compute nodes) is gonna be .. a lot | 20:35 |
| mnaser | cause now tripleo/osa/etc all have to change our paths | 20:35 |
| mnaser | to something sketchy none the less :P | 20:35 |
| melwitt | mnaser: yeah, I see. ok | 20:35 |
| mnaser | I wouldn't revert it all | 20:36 |
| mnaser | maybe just the small part to set the cookie | 20:36 |
| melwitt | yeah | 20:36 |
| mriedem | efried: i guess bug 1815692 won't help because we don't cache the minimum compute service version until we get one, meaning until a nova-compute starts up, which in devstack is after the api | 20:41 |
| openstack | bug 1815692 in OpenStack Compute (nova) "minimum nova-compute service version is not cached in nova-api when running under wsgi" [Low,Triaged] https://launchpad.net/bugs/1815692 | 20:41 |
| efried | is making the thing a singleton an option? | 20:42 |
| mriedem | making nova.compute.api.API or nova.compute.rpcapi.ComputeAPI a singleton might help | 20:42 |
| efried | Is there any evidence that that's actually a problem though? | 20:42 |
| mnaser | melwitt: https://bugs.launchpad.net/nova/+bug/1822676 | 20:42 |
| openstack | Launchpad bug 1822676 in OpenStack Compute (nova) "novnc no longer sets token inside cookie" [Undecided,New] | 20:42 |
| melwitt | mnaser: you know, while I was looking at the websockify source, I was thinking maybe in code we could append to our self.path earlier in the call path. I can try something out | 20:42 |
| efried | the devstack log just says n-api doesn't start. | 20:42 |
| efried | I guess that means "within a certain amount of time" | 20:42 |
| melwitt | mnaser: thanks for the bug report | 20:42 |
| mriedem | efried: it's a timeout | 20:42 |
| mnaser | melwitt: all yours :) I don't think I can take more time on this :X | 20:42 |
| mnaser | I will "work-around" it in osa | 20:43 |
| melwitt | mnaser: yeah, I'll try this out and let you know | 20:43 |
| mriedem | efried: the non-singleton nature with locks can definitely cause slowness ala https://review.openstack.org/#/c/623246/ | 20:43 |
| melwitt | mnaser: if what I'm thinking doesn't end up working, I can ask the novnc folks if they can revert the nova part | 20:43 |
| *** itlinux has quit IRC | 20:58 | |
| *** itlinux has joined #openstack-nova | 20:59 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Hacking N362: Don't abbrev/alias privsep import https://review.openstack.org/649188 | 21:00 |
| *** ivve has joined #openstack-nova | 21:02 | |
| efried | this f'in thing is never going to merge. Now it's failed py37 | 21:02 |
| *** erlon_ has quit IRC | 21:05 | |
| *** Sundar has quit IRC | 21:08 | |
| *** ivve has quit IRC | 21:09 | |
| *** owalsh_ has joined #openstack-nova | 21:12 | |
| *** awaugama has quit IRC | 21:13 | |
| mriedem | dansmith: you know how on nova-api startup we check for the minimum nova-compute service version in order to set the rpc pin if [upgrade_levels]/compute=auto https://github.com/openstack/nova/blob/master/nova/compute/rpcapi.py#L408 ? | 21:15 |
| *** owalsh has quit IRC | 21:15 | |
| mriedem | the comment above that says we don't cache it because "we will get a better answer next time." - but next time would have to be on restart of the service right? because we only call _determine_version_cap once in order to construct the ClientRouter | 21:16 |
| *** owalsh has joined #openstack-nova | 21:17 | |
| *** owalsh_ has quit IRC | 21:17 | |
| *** pcaruana has quit IRC | 21:17 | |
| openstackgerrit | Michael Still proposed openstack/nova master: Style corrections for privsep usage. https://review.openstack.org/648615 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.path. https://review.openstack.org/648601 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs. https://review.openstack.org/648602 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Improve test coverage of nova.privsep.fs, continued. https://review.openstack.org/648603 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Add test coverage for nova.privsep.libvirt. https://review.openstack.org/648616 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Privsepify ipv4 forwarding enablement. https://review.openstack.org/635431 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Remove unused FP device creation and deletion methods. https://review.openstack.org/635433 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Privsep the ebtables modification code. https://review.openstack.org/635435 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move adding vlans to interfaces to privsep. https://review.openstack.org/635436 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move iptables rule fetching and setting to privsep. https://review.openstack.org/636508 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move dnsmasq restarts to privsep. https://review.openstack.org/639280 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move router advertisement daemon restarts to privsep. https://review.openstack.org/639281 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move calls to ovs-vsctl to privsep. https://review.openstack.org/639282 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move setting of device trust to privsep. https://review.openstack.org/639283 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Move final bridge commands to privsep. https://review.openstack.org/639580 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Cleanup the _execute shim in nova/network. https://review.openstack.org/639581 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: We no longer need rootwrap. https://review.openstack.org/554438 | 21:18 |
| openstackgerrit | Michael Still proposed openstack/nova master: Cleanup no longer required filters and add a release note. https://review.openstack.org/639826 | 21:18 |
| *** wolverineav has quit IRC | 21:27 | |
| *** wolverineav has joined #openstack-nova | 21:31 | |
| *** samueldmq has joined #openstack-nova | 21:36 | |
| *** wolverineav has quit IRC | 21:36 | |
| *** lbragstad is now known as lbragstad_dad_mo | 21:40 | |
| *** lbragstad_dad_mo is now known as lbragstad | 21:40 | |
| *** itlinux has quit IRC | 21:43 | |
| *** ccamacho has quit IRC | 21:43 | |
| *** wolverineav has joined #openstack-nova | 21:44 | |
| *** itlinux has joined #openstack-nova | 21:44 | |
| *** itlinux has quit IRC | 21:44 | |
| *** slaweq has quit IRC | 21:45 | |
| *** mrhillsman is now known as mrhillsman_afk | 21:46 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Make nova.compute.rpcapi.ComputeAPI.router a singleton https://review.openstack.org/649197 | 21:47 |
| *** eharney has joined #openstack-nova | 21:52 | |
| *** munimeha1 has quit IRC | 22:01 | |
| *** BjoernT has quit IRC | 22:01 | |
| *** eharney has quit IRC | 22:04 | |
| *** mlavalle has quit IRC | 22:14 | |
| *** krypto has quit IRC | 22:16 | |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Fix ProviderUsageBaseTestCase._run_periodics for multi-cell https://review.openstack.org/641179 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Improve CinderFixtureNewAttachFlow https://review.openstack.org/639382 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add functional recreate test for bug 1818914 https://review.openstack.org/641521 | 22:17 |
| openstack | bug 1818914 in OpenStack Compute (nova) "Hypervisor resource usage on source still shows old flavor usage after resize confirm until update_available_resource periodic runs" [Low,In progress] https://launchpad.net/bugs/1818914 - Assigned to Matt Riedemann (mriedem) | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Remove unused context parameter from RT._get_instance_type https://review.openstack.org/641792 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Update usage in RT.drop_move_claim during confirm resize https://review.openstack.org/641806 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Migration.cross_cell_move and get_by_uuid https://review.openstack.org/614012 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add InstanceAction/Event create() method https://review.openstack.org/614036 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add Instance.hidden field https://review.openstack.org/631123 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add TargetDBSetupTask https://review.openstack.org/627892 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellMigrationTask https://review.openstack.org/631581 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Execute TargetDBSetupTask https://review.openstack.org/633853 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add can_connect_volume() compute driver method https://review.openstack.org/621313 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_dest compute method https://review.openstack.org/633293 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtDestTask https://review.openstack.org/627890 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add prep_snapshot_based_resize_at_source compute method https://review.openstack.org/634832 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add nova.compute.utils.delete_image https://review.openstack.org/637605 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add PrepResizeAtSourceTask https://review.openstack.org/627891 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Refactor ComputeManager.remove_volume_connection https://review.openstack.org/642183 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Revert cross-cell resize from the API https://review.openstack.org/638048 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Confirm cross-cell resize while deleting a server https://review.openstack.org/638268 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add CrossCellWeigher https://review.openstack.org/614353 | 22:17 |
| openstackgerrit | Matt Riedemann proposed openstack/nova master: Add cross-cell resize policy rule and enable in API https://review.openstack.org/638269 | 22:17 |
| *** mriedem has quit IRC | 22:21 | |
| openstackgerrit | Eric Fried proposed openstack/nova master: Mock time.sleep() in unit tests https://review.openstack.org/648762 | 22:34 |
| *** betherly has joined #openstack-nova | 22:34 | |
| *** nikrin is now known as knikolla | 22:34 | |
| *** betherly has quit IRC | 22:39 | |
| *** rcernin has joined #openstack-nova | 22:40 | |
| *** tkajinam has joined #openstack-nova | 22:56 | |
| *** itlinux has joined #openstack-nova | 23:23 | |
| *** tosky has quit IRC | 23:28 | |
| *** wolverineav has quit IRC | 23:39 | |
| *** wolverineav has joined #openstack-nova | 23:39 | |
| *** wolverineav has quit IRC | 23:44 | |
| *** BjoernT has joined #openstack-nova | 23:50 | |
| *** lbragstad has quit IRC | 23:54 | |
| *** BjoernT has quit IRC | 23:54 | |
| *** openstackgerrit has quit IRC | 23:56 | |
| *** BjoernT has joined #openstack-nova | 23:57 | |
| *** Vek has quit IRC | 23:57 | |
| *** Vek has joined #openstack-nova | 23:57 | |
| *** antonym has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!