| *** markwash has joined #openstack-neutron | 00:05 | |
| *** markwash has quit IRC | 00:08 | |
| *** clev has joined #openstack-neutron | 00:10 | |
| *** SumitNaiksatam has joined #openstack-neutron | 00:13 | |
| *** markwash has joined #openstack-neutron | 00:15 | |
| *** matsuhashi has joined #openstack-neutron | 00:19 | |
| *** ijw_ has quit IRC | 00:22 | |
| *** markwash has quit IRC | 00:30 | |
| marun | salv-orlando: pretty please merge? https://review.openstack.org/#/c/43776/ | 00:39 |
|---|---|---|
| *** markwash has joined #openstack-neutron | 00:45 | |
| *** roeyc has joined #openstack-neutron | 00:46 | |
| *** roeyc has quit IRC | 00:46 | |
| *** harlowja_ has quit IRC | 01:09 | |
| *** ljjjustin has joined #openstack-neutron | 01:16 | |
| *** gdubreui has quit IRC | 01:19 | |
| *** yamahata has joined #openstack-neutron | 01:24 | |
| *** markwash has quit IRC | 01:35 | |
| *** Jianyong has joined #openstack-neutron | 01:40 | |
| *** ljjjustin has quit IRC | 01:42 | |
| *** dzyu has joined #openstack-neutron | 01:46 | |
| *** matsuhashi has quit IRC | 01:57 | |
| *** matsuhashi has joined #openstack-neutron | 02:00 | |
| openstackgerrit | Jay Pipes proposed a change to openstack/neutron: Corrects broken format strings in check_i18n.py https://review.openstack.org/63858 | 02:00 |
| *** clev has quit IRC | 02:25 | |
| *** mayu has joined #openstack-neutron | 02:33 | |
| *** ljjjustin has joined #openstack-neutron | 02:33 | |
| *** mayu has quit IRC | 02:40 | |
| *** ljjjustin has quit IRC | 02:46 | |
| *** alexpilotti has quit IRC | 02:46 | |
| *** harlowja has joined #openstack-neutron | 02:48 | |
| *** nati_ueno has joined #openstack-neutron | 02:50 | |
| *** nati_ueno has quit IRC | 02:56 | |
| *** harlowja has quit IRC | 03:03 | |
| *** harlowja has joined #openstack-neutron | 03:04 | |
| *** nati_ueno has joined #openstack-neutron | 03:06 | |
| *** rwsu has quit IRC | 03:06 | |
| *** nati_ueno has quit IRC | 03:11 | |
| *** vkozhukalov has joined #openstack-neutron | 03:27 | |
| *** WackoRobie has joined #openstack-neutron | 03:36 | |
| *** liudong_ has joined #openstack-neutron | 03:53 | |
| liudong_ | exit | 03:53 |
| *** liudong_ has left #openstack-neutron | 03:53 | |
| *** otherwiseguy has joined #openstack-neutron | 04:12 | |
| *** Jianyong has quit IRC | 04:13 | |
| *** markwash has joined #openstack-neutron | 04:21 | |
| *** markwash has quit IRC | 04:46 | |
| *** ashaikh has joined #openstack-neutron | 04:59 | |
| *** harlowja is now known as harlowja_away | 05:00 | |
| *** matsuhashi has quit IRC | 05:02 | |
| *** matsuhashi has joined #openstack-neutron | 05:03 | |
| *** __amotoki__ is now known as amotoki | 05:18 | |
| *** clev has joined #openstack-neutron | 05:20 | |
| *** clev has quit IRC | 05:26 | |
| *** WackoRobie has quit IRC | 05:29 | |
| *** yfried has quit IRC | 05:35 | |
| *** markwash has joined #openstack-neutron | 05:41 | |
| *** markwash has quit IRC | 05:45 | |
| *** ljjjustin has joined #openstack-neutron | 06:01 | |
| *** SumitNaiksatam has quit IRC | 06:05 | |
| *** otherwiseguy has quit IRC | 06:05 | |
| *** yfried has joined #openstack-neutron | 06:21 | |
| *** matsuhashi has quit IRC | 06:35 | |
| *** matsuhashi has joined #openstack-neutron | 06:36 | |
| openstackgerrit | Jenkins proposed a change to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/63877 | 06:37 |
| *** matsuhashi has quit IRC | 06:41 | |
| *** matsuhashi has joined #openstack-neutron | 06:45 | |
| *** amritanshu_RnD has joined #openstack-neutron | 06:49 | |
| *** fatguylittlecoat has joined #openstack-neutron | 06:51 | |
| *** vkozhukalov has quit IRC | 06:53 | |
| *** garyk has joined #openstack-neutron | 07:01 | |
| *** ashaikh has quit IRC | 07:11 | |
| *** amuller has joined #openstack-neutron | 07:12 | |
| *** bvandenh has joined #openstack-neutron | 07:18 | |
| *** gongysh has joined #openstack-neutron | 07:20 | |
| *** amuller has quit IRC | 07:30 | |
| *** zhhuabj has joined #openstack-neutron | 07:33 | |
| *** irenab_ has joined #openstack-neutron | 07:34 | |
| *** matsuhashi has quit IRC | 07:40 | |
| *** matsuhashi has joined #openstack-neutron | 07:41 | |
| *** matsuhashi has quit IRC | 07:47 | |
| *** matsuhashi has joined #openstack-neutron | 07:49 | |
| *** skraynev has quit IRC | 07:53 | |
| *** amuller has joined #openstack-neutron | 08:07 | |
| *** fatguylittlecoat has quit IRC | 08:12 | |
| *** matsuhashi has quit IRC | 08:16 | |
| *** amuller has quit IRC | 08:17 | |
| *** matsuhashi has joined #openstack-neutron | 08:17 | |
| *** fouxm has joined #openstack-neutron | 08:19 | |
| *** networkstatic has joined #openstack-neutron | 08:23 | |
| *** vkozhukalov has joined #openstack-neutron | 08:30 | |
| *** SumitNaiksatam has joined #openstack-neutron | 08:32 | |
| *** amuller has joined #openstack-neutron | 08:37 | |
| *** nati_ueno has joined #openstack-neutron | 08:38 | |
| yfried | nati_ueno: ping | 08:46 |
| yfried | nati_ueno: Do you have time to test your patch today? I would like to assist you if possible | 08:49 |
| *** fatguylittlecoat has joined #openstack-neutron | 08:49 | |
| *** nati_ueno has quit IRC | 08:50 | |
| *** nati_ueno has joined #openstack-neutron | 08:50 | |
| *** ygbo has joined #openstack-neutron | 08:50 | |
| yfried | nati_ueno: are you here? | 08:51 |
| *** zhhuabj has quit IRC | 09:00 | |
| *** zhhuabj has joined #openstack-neutron | 09:00 | |
| *** safchain has joined #openstack-neutron | 09:02 | |
| *** yongli is now known as yongli_away | 09:06 | |
| *** nati_ueno has quit IRC | 09:10 | |
| safchain | gongysh, ping | 09:11 |
| gongysh | safchain: pong | 09:11 |
| safchain | I submitted a new version of the patch about the l3 ext. net. | 09:12 |
| safchain | gongysh, and I would like to know what is the exact limitation with the br-ex | 09:12 |
| safchain | gongysh, I mean the limitation is one ext net per agent or one ext for neutron | 09:14 |
| gongysh | safchain: one ext net per agent. | 09:14 |
| safchain | gongysh, So I will submit a patch to remove the rpc call from the agent | 09:15 |
| gongysh | safchain: and you have to specify the external network id if you wants multiple ones. | 09:15 |
| gongysh | safchain: I don't think so. | 09:15 |
| gongysh | safchain: we still need to keep it if user does not specify the external network id in l3 agent and have external bridge set. | 09:16 |
| safchain | gongysh, Is it not enough to check the number of network regarding the routers in the agent | 09:17 |
| gongysh | safchain: what do u mean by ' the number of network regarding the routers in the agent'? | 09:18 |
| safchain | gongysh, I mean raise an exception if a new router is processed with a new network and the agent use br-ex | 09:19 |
| safchain | gongysh, checking if all routers use the same ext net. | 09:20 |
| gongysh | safchain: in this case, there may be multiple external networks on the neutron side, but this agent happens to have the routers on one external network. | 09:21 |
| safchain | gongysh, correct that why I was asking you to the real limitation | 09:22 |
| safchain | gongysh, if the limitation is one net. per agent, I think this patch could be ok | 09:23 |
| *** armax has joined #openstack-neutron | 09:26 | |
| gongysh | safchain: when there are multiple external networks with br-ex, the user must know the br-ex connects to right network. | 09:26 |
| *** armax has quit IRC | 09:26 | |
| gongysh | safchain: so we have the external_network_id in the l3-agent.ini. | 09:27 |
| safchain | gongysh, so yes you're right | 09:29 |
| *** yamahata has quit IRC | 09:30 | |
| safchain | gongysh, btw could you review https://review.openstack.org/#/c/59359/ | 09:31 |
| *** matsuhashi has quit IRC | 09:40 | |
| openstackgerrit | mouad benchchaoui proposed a change to openstack/neutron: Make the metadata namespace proxy transparent https://review.openstack.org/28137 | 09:46 |
| *** networkstatic has quit IRC | 09:47 | |
| *** networkstatic has joined #openstack-neutron | 09:48 | |
| *** matsuhashi has joined #openstack-neutron | 09:58 | |
| gongysh | safchain: what happend to jenkin -1? | 10:04 |
| *** ljjjustin has quit IRC | 10:05 | |
| obondarev | hi folks! This patch fixes lbaas bug which blocks scenario tests patches for lbaas in tempest: https://review.openstack.org/#/c/61673/ | 10:05 |
| obondarev | core dev review will be much appreciated | 10:05 |
| openstackgerrit | Dazhao Yu proposed a change to openstack/neutron: Calculate stateless IPv6 address https://review.openstack.org/56184 | 10:13 |
| safchain | gongysh, thanks for the review, I'll check the jenkin -1 | 10:20 |
| *** majopela has joined #openstack-neutron | 10:24 | |
| *** dzyu has quit IRC | 10:27 | |
| *** gongysh has quit IRC | 10:29 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks https://review.openstack.org/59359 | 10:29 |
| *** amuller_ has joined #openstack-neutron | 10:31 | |
| *** fouxm has quit IRC | 10:32 | |
| *** ijw has joined #openstack-neutron | 10:33 | |
| *** fouxm has joined #openstack-neutron | 10:33 | |
| *** amuller has quit IRC | 10:34 | |
| openstackgerrit | András Gyácsok proposed a change to openstack/neutron: Files without code should not contain copyright notices https://review.openstack.org/63901 | 10:40 |
| *** yamahata has joined #openstack-neutron | 10:47 | |
| openstackgerrit | Aleks Chirko proposed a change to openstack/neutron: Bugfix and refactoring for ovs_lib flow mgnt methods https://review.openstack.org/58533 | 10:48 |
| *** amuller_ has quit IRC | 11:01 | |
| *** amuller_ has joined #openstack-neutron | 11:01 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: Dhcp agent now allows multiple dnsmasq_dns_server https://review.openstack.org/62002 | 11:03 |
| *** zhhuabj has quit IRC | 11:12 | |
| *** zhhuabj has joined #openstack-neutron | 11:13 | |
| *** matsuhashi has quit IRC | 11:15 | |
| *** heyongli has joined #openstack-neutron | 11:22 | |
| *** ijw has quit IRC | 11:26 | |
| *** ijw has joined #openstack-neutron | 11:27 | |
| *** networkstatic has quit IRC | 11:36 | |
| *** amuller_ has quit IRC | 11:39 | |
| *** amuller_ has joined #openstack-neutron | 12:02 | |
| openstackgerrit | Édouard Thuleau proposed a change to openstack/neutron: [ML2] l2-pop MD handle multi create/delete ports https://review.openstack.org/63913 | 12:22 |
| openstackgerrit | Édouard Thuleau proposed a change to openstack/neutron: [ML2] l2-pop MD handle multi create/delete ports https://review.openstack.org/63913 | 12:24 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: Dhcp agent now allows multiple dnsmasq_dns_server https://review.openstack.org/62002 | 12:36 |
| openstackgerrit | Édouard Thuleau proposed a change to openstack/neutron: OVS lib defer apply doesn't handle concurrency https://review.openstack.org/63917 | 12:44 |
| openstackgerrit | enikanorov proposed a change to openstack/neutron: Fix race in get_network(s) in OVS plugin https://review.openstack.org/63918 | 12:53 |
| *** vkozhukalov has quit IRC | 12:59 | |
| *** fouxm has quit IRC | 13:05 | |
| *** Jianyong has joined #openstack-neutron | 13:10 | |
| openstackgerrit | Jay Pipes proposed a change to openstack/neutron: Corrects broken format strings in check_i18n.py https://review.openstack.org/63858 | 13:15 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Agent can handle many external networks https://review.openstack.org/59359 | 13:15 |
| *** irenab_ has quit IRC | 13:15 | |
| jaypipes | obondarev: thx for the heads up on the Closes-Bug: link... always forget the format of that... :) | 13:19 |
| *** pcm_ has joined #openstack-neutron | 13:22 | |
| *** pcm_ has quit IRC | 13:25 | |
| *** pcm_ has joined #openstack-neutron | 13:26 | |
| *** evgenyf has quit IRC | 13:31 | |
| *** evgenyf has joined #openstack-neutron | 13:31 | |
| *** WackoRobie has joined #openstack-neutron | 13:31 | |
| *** WackoRobie has quit IRC | 13:38 | |
| *** b3nt_pin has joined #openstack-neutron | 13:54 | |
| *** salv-orlando_ has joined #openstack-neutron | 13:54 | |
| *** yfried has quit IRC | 13:56 | |
| *** mestery_ has joined #openstack-neutron | 13:56 | |
| *** fcoj_ has joined #openstack-neutron | 13:57 | |
| *** yhe_ has joined #openstack-neutron | 13:58 | |
| *** yhe_ is now known as yongli | 13:59 | |
| *** rdo_ has joined #openstack-neutron | 13:59 | |
| *** ygbo has quit IRC | 13:59 | |
| *** ygbo has joined #openstack-neutron | 14:01 | |
| *** evgenyf has quit IRC | 14:06 | |
| *** heyongli has quit IRC | 14:06 | |
| *** zhhuabj has quit IRC | 14:06 | |
| *** mestery has quit IRC | 14:06 | |
| *** beagles has quit IRC | 14:06 | |
| *** salv-orlando has quit IRC | 14:06 | |
| *** ekarlso has quit IRC | 14:06 | |
| *** sc68cal has quit IRC | 14:06 | |
| *** dave_tucker_zzz has quit IRC | 14:06 | |
| *** rdo has quit IRC | 14:06 | |
| *** fcoj has quit IRC | 14:06 | |
| *** salv-orlando_ is now known as salv-orlando | 14:06 | |
| *** evgenyf has joined #openstack-neutron | 14:10 | |
| *** ekarlso has joined #openstack-neutron | 14:10 | |
| *** sc68cal has joined #openstack-neutron | 14:10 | |
| *** dave_tucker_zzz has joined #openstack-neutron | 14:10 | |
| *** Jianyong has quit IRC | 14:16 | |
| *** WackoRobie has joined #openstack-neutron | 14:19 | |
| *** rossella_s has joined #openstack-neutron | 14:23 | |
| *** yamahata has quit IRC | 14:26 | |
| *** yamahata has joined #openstack-neutron | 14:27 | |
| *** aymenfrikha has joined #openstack-neutron | 14:27 | |
| *** annegentle_ has quit IRC | 14:27 | |
| *** clev has joined #openstack-neutron | 14:28 | |
| *** rossella_s has quit IRC | 14:30 | |
| *** russellb is now known as russellb_see_you | 14:33 | |
| *** russellb_see_you is now known as russellb_gonetil | 14:33 | |
| *** russellb_gonetil is now known as russellb_vacatio | 14:33 | |
| *** russellb_vacatio is now known as russellb_gone | 14:34 | |
| *** dims has quit IRC | 14:43 | |
| openstackgerrit | Evgeny Fedoruk proposed a change to openstack/neutron: New SSL extension https://review.openstack.org/63510 | 14:48 |
| *** markwash has joined #openstack-neutron | 14:54 | |
| *** dims has joined #openstack-neutron | 14:55 | |
| *** mrsnivvel has quit IRC | 15:03 | |
| *** fatguylittlecoat has quit IRC | 15:04 | |
| *** yfried has joined #openstack-neutron | 15:06 | |
| *** yongli has quit IRC | 15:06 | |
| *** markwash has quit IRC | 15:12 | |
| *** amuller_ is now known as amuller | 15:20 | |
| *** alagalah has joined #openstack-neutron | 15:21 | |
| *** amritanshu_RnD has quit IRC | 15:24 | |
| *** aymenfrikha has quit IRC | 15:25 | |
| openstackgerrit | András Gyácsok proposed a change to openstack/neutron: Removed copiright notices https://review.openstack.org/63901 | 15:31 |
| *** harlowja_away is now known as harlowja | 15:38 | |
| anteaya | at the same time 1253896 decreased (nice work, salv-orlando) 1254890 increased | 15:45 |
| anteaya | enikanorov__: have you had an opportunity to chat with dims about testing https://review.openstack.org/#/c/63304/ ? | 15:46 |
| anteaya | it appears that 1254890 is now top spot in the gate blocking bugs category :/ | 15:46 |
| dims | anteaya, 63304 does not work. need to find something else | 15:47 |
| anteaya | okay, can we get some comments on the patch and the bug report to that effect? | 15:48 |
| anteaya | anyone bored today who wants to work with enikanorov__ on addressing 1254890? | 15:50 |
| anteaya | if we don't make some progress with it, it will just keep coming back to haunt us during break | 15:50 |
| enikanorov | you're welcome :) | 15:51 |
| anteaya | :D | 15:51 |
| anteaya | enikanorov: any new thoughts? | 15:51 |
| anteaya | since testing has demonstrated 63304 is not a good direction, any sense of what direction might be better, enikanorov | 15:52 |
| enikanorov | anteaya: i'm thinking around adding some logging aroud the commands that lead to the issue. | 15:52 |
| anteaya | that would be a great help | 15:52 |
| enikanorov | well, i think there could be some workarounds | 15:52 |
| anteaya | do share the url of the patch with increased logging so we can get it in before we are officially a ghost town | 15:52 |
| * anteaya listens | 15:53 | |
| enikanorov | but in fact it could be better to get to the bottom of it | 15:53 |
| *** zzelle has joined #openstack-neutron | 15:53 | |
| enikanorov | to the root cause which seems unclear currently | 15:53 |
| enikanorov | dims: | 15:54 |
| anteaya | well traffic will be light the next 2 weeks | 15:54 |
| dims | right | 15:54 |
| enikanorov | so i have some observation | 15:54 |
| anteaya | now might be a good time to get to the bottom of it, or at least make the attempt | 15:54 |
| enikanorov | dims: i don't know whether it is connected to the issue or not | 15:54 |
| enikanorov | the following issue i have seen: typically you have a number of network namespaces when running devstack | 15:55 |
| enikanorov | for the router, dhcp and lbaas | 15:55 |
| enikanorov | if there some haproxies ronning in lbaas namespaces | 15:55 |
| enikanorov | and you do unstack | 15:55 |
| enikanorov | where all router and dhcp namespaces are cleaned up | 15:56 |
| enikanorov | you'll see that lbaas namespaces are not deleted (because there are processes that use them) | 15:56 |
| enikanorov | but also router and dhcp namespaces are not delted as well | 15:56 |
| enikanorov | eventhough there are no resources that use them | 15:56 |
| enikanorov | and you actually can't delete them untill you release those which belong to lbaas | 15:57 |
| enikanorov | dims: i dont know the reason for it, and i'm just guessing if something could affect nbd unmounitng in the same way | 15:57 |
| enikanorov | (because unmount is what actually used when deleting network namespaces) | 15:58 |
| dims | enikanorov, makes sense | 15:58 |
| enikanorov | also, there is a bug on neutron about this | 15:58 |
| enikanorov | quite an old one, and no fix yet | 15:58 |
| enikanorov | https://bugs.launchpad.net/neutron/+bug/1052535 | 15:59 |
| enikanorov | another thought is that we need a fresh look on the difference between neutron and neutron-pg jobs | 16:02 |
| enikanorov | the issue doesn't appear on the former and seems to olny apper on the latter | 16:02 |
| enikanorov | i don't know how postgres can affect this... | 16:03 |
| *** markwash has joined #openstack-neutron | 16:05 | |
| anteaya | some good thoughts enikanorov | 16:06 |
| anteaya | can we create a plan of action? | 16:06 |
| anteaya | what do you think is a good first step? | 16:06 |
| enikanorov | i think we need another pair of eyes from nova side. I'm not a nova expert and also I'd say we have not seen an evidence that neutron itself has something to do with the issue | 16:07 |
| anteaya | okay great | 16:07 |
| anteaya | dims do you have nova expertise? or can you suggest someone? | 16:08 |
| anteaya | is arosen around? | 16:08 |
| *** garyk has quit IRC | 16:08 | |
| anteaya | I know marun was moving towards more neutron-nova knowledge | 16:08 |
| anteaya | I wonder if he has anything to offer | 16:08 |
| dims | anteaya, phil/hp has had many patches as well | 16:09 |
| anteaya | great, is phil his irc nick? | 16:10 |
| anteaya | not sure i have met him | 16:10 |
| anteaya | have we met phil? | 16:10 |
| dims | anteaya, i have not seen him on irc | 16:10 |
| anteaya | ah | 16:10 |
| anteaya | going to make him hard to reach then | 16:10 |
| anteaya | carl_baldwin isn't online atm, he might know phil | 16:11 |
| anteaya | dims: can I get some updated comments on https://bugs.launchpad.net/nova/+bug/1254890 regarding https://review.openstack.org/#/c/63304/ | 16:12 |
| anteaya | it will make it easier for me if I am shopping for some additional nova eyes | 16:12 |
| *** chandankumar has quit IRC | 16:20 | |
| openstackgerrit | dekehn proposed a change to openstack/python-neutronclient: Adds delete of extra-dhcp-opt to the client https://review.openstack.org/49166 | 16:21 |
| *** ijw has quit IRC | 16:25 | |
| anteaya | dims: thank you | 16:26 |
| anteaya | just put out a call in -nova | 16:28 |
| anteaya | hopefully we get a volunteer | 16:28 |
| *** ygbo has quit IRC | 16:31 | |
| anteaya | enikanorov: while we are waiting to have a broader discussion, do you think it is worthwhile to add the additional logging you had mentioned? | 16:32 |
| *** markwash has quit IRC | 16:40 | |
| *** evgenyf has quit IRC | 16:42 | |
| *** garyk has joined #openstack-neutron | 16:43 | |
| *** harlowja is now known as harlowja_away | 16:44 | |
| geekinutah | anteaya enikanorov: what do you guys want to know from a nova perspective? I can probably help out | 16:44 |
| anteaya | hello geekinutah | 16:47 |
| geekinutah | hi :-) | 16:48 |
| anteaya | geekinutah: have you had a chance to read the backscroll ^^ | 16:48 |
| geekinutah | reading it.... | 16:48 |
| anteaya | enikanorov outlines some of this thoughts | 16:48 |
| anteaya | thanks | 16:48 |
| geekinutah | I'm looking at some of the logs from dims' review | 16:50 |
| geekinutah | see if I can come up with anything... | 16:50 |
| anteaya | thank you | 16:51 |
| anteaya | we are grateful for any help | 16:51 |
| geekinutah | do you know what OS these VMs are? | 17:01 |
| geekinutah | err | 17:01 |
| geekinutah | distro | 17:01 |
| geekinutah | ubuntu I'm guessing? | 17:01 |
| anteaya | the devstack vms are all precise | 17:04 |
| anteaya | except for the python2.6 tests which run on centos | 17:04 |
| anteaya | geekinutah: ^ | 17:04 |
| geekinutah | okay cool, thx | 17:06 |
| *** alexpilotti has joined #openstack-neutron | 17:06 | |
| geekinutah | well, the claim was successful according to n-cpu | 17:17 |
| geekinutah | so it should have gone active... looking at conductor now | 17:17 |
| anteaya | k | 17:21 |
| geekinutah | oh wait... I'm wrong | 17:21 |
| *** garyk has quit IRC | 17:21 | |
| anteaya | do tell | 17:21 |
| geekinutah | I just got confused, I'm used to looking at a lot of folsom code | 17:21 |
| geekinutah | the claim only happens on success there | 17:21 |
| anteaya | feel free to paste anything we can look at together | 17:21 |
| anteaya | ah | 17:21 |
| *** safchain has quit IRC | 17:22 | |
| *** garyk has joined #openstack-neutron | 17:23 | |
| *** mlavalle has joined #openstack-neutron | 17:23 | |
| geekinutah | well, I'm back to where dims started I think | 17:26 |
| geekinutah | need to figure out why qemu-nbd never suceeds | 17:26 |
| *** csd has joined #openstack-neutron | 17:26 | |
| anteaya | geekinutah: okay | 17:27 |
| geekinutah | so some extra logging is probably in order | 17:27 |
| anteaya | alright | 17:28 |
| anteaya | enikanorov: looks like we are back to extra logging | 17:28 |
| geekinutah | from dims patch I gather that the problems are generally around mounting an nbd device or unmounting it? | 17:28 |
| *** nati_ueno has joined #openstack-neutron | 17:28 | |
| anteaya | that would be my assumption as well | 17:28 |
| anteaya | geekinutah: are you in the mood to cobble together a patch putting in some extra logging? | 17:28 |
| geekinutah | yeah, I was just about to | 17:29 |
| anteaya | enikanorov may have hit the end of his day | 17:29 |
| anteaya | geekinutah: awesome, thank you | 17:29 |
| anteaya | do post the url once you have something | 17:29 |
| geekinutah | will do | 17:29 |
| anteaya | thanks | 17:29 |
| dims | geekinutah, yep. once we see this | 17:29 |
| dims | 2013-12-24 00:07:51.617 DEBUG nova.virt.disk.vfs.localfs [req-35c764fe-5778-4c64-a4b0-e6fb9c71ec43 tempest.scenario.manager-tempest-547235891-user tempest.scenario.manager-tempest-547235891-tenant] Failed to remove /tmp/openstack-vfs-localfsvPPNuZ: [Errno 16] Device or resource busy: '/tmp/openstack-vfs-localfsvPPNuZ' teardown /opt/stack/new/nova/nova/virt/disk/vfs/localfs.py:98 | 17:29 |
| dims | then the next mount on the same nbd hangs | 17:30 |
| anteaya | geekinutah: also if you have anything to add to the bug report, please add, even if it only confirming dims findings, since that helps | 17:30 |
| geekinutah | dims: any suspicions on what could be using it? | 17:31 |
| geekinutah | my initial thought is to see if there is anything in 'dm_setup ls' or lsof | 17:32 |
| dims | geekinutah, take a peek at the commands that use the /tmp/openstack-vfs-localfsvPPNuZ in my example - http://paste.openstack.org/show/56703/ | 17:33 |
| *** WackoRobie has quit IRC | 17:34 | |
| *** WackoRobie has joined #openstack-neutron | 17:35 | |
| geekinutah | dims: did umount ever give a useful error message before you added the -l option? | 17:39 |
| dims | geekinutah, nope | 17:39 |
| geekinutah | I assume it gave Device or resource busy also? | 17:40 |
| *** WackoRobie has quit IRC | 17:40 | |
| dims | geekinutah, nope. "Failed to remove" was when deleting the directory not from umount | 17:41 |
| geekinutah | ahh, okay | 17:42 |
| geekinutah | so umount actually would work just fine? | 17:43 |
| dims | as far as we can tell from logs | 17:44 |
| geekinutah | so from a kernel perspective, there are no references to the device | 17:44 |
| geekinutah | so probably an nbd bug :-( | 17:44 |
| dims | y possible | 17:46 |
| anteaya | in ubuntu? | 17:46 |
| geekinutah | there are only two places in the nbd code that return EBUSY which is the errno for "Device or resource busy" | 17:47 |
| geekinutah | but even if it is a bug in nbd that I can fix, would it be better just to work around it in nova? | 17:48 |
| dims | y, trying to figure that out. no luck yet | 17:49 |
| anteaya | https://launchpad.net/nbd one contributor and no bugs filed that I can see | 17:49 |
| geekinutah | anteaya: yeah, it's really old even | 17:51 |
| *** clev has quit IRC | 17:51 | |
| geekinutah | it was basically contributed then never really maintained from what I remember | 17:52 |
| *** clev has joined #openstack-neutron | 17:52 | |
| dims | geekinutah, back in a bit | 17:52 |
| enikanorov__ | i wonder when this issue has appeared and why doesn't it happen with nova jobs | 17:52 |
| anteaya | geekinutah: is there an option to use something other than nbd? | 17:53 |
| anteaya | using old unmaintained code doesn't help us much going forward | 17:53 |
| anteaya | enikanorov__: good point | 17:53 |
| geekinutah | anteaya: nova tries to use libguestfs first, nbd is a last resort really | 17:53 |
| geekinutah | but you are right, it's kinda crappy to use nbd code at all | 17:54 |
| geekinutah | in fact, it is crappy to locally mount filesystems and inject crap into them, but we are stuck supporting that at least until j | 17:54 |
| geekinutah | I think it is getting deprecated in icehouse.... can't remember what was ultimately decided there | 17:55 |
| geekinutah | be back in a bit | 17:55 |
| anteaya | k | 17:56 |
| *** amuller has quit IRC | 18:01 | |
| *** clev has quit IRC | 18:02 | |
| geekinutah | dims: FWIW I think this is the EBUSY we are hitting https://github.com/torvalds/linux/blob/master/drivers/block/nbd.c#L704 | 18:05 |
| *** markwash has joined #openstack-neutron | 18:19 | |
| *** salv-orlando has quit IRC | 18:25 | |
| *** clev has joined #openstack-neutron | 18:26 | |
| *** mlavalle_ has joined #openstack-neutron | 18:40 | |
| *** mlavalle has quit IRC | 18:41 | |
| *** mlavalle_ is now known as mlavalle | 18:41 | |
| *** WackoRobie has joined #openstack-neutron | 18:47 | |
| *** markwash has quit IRC | 18:47 | |
| *** WackoRobie has quit IRC | 18:51 | |
| *** fatguylittlecoat has joined #openstack-neutron | 18:52 | |
| anteaya | geekinutah: so where are we now? still agreed more logging is in order? | 19:00 |
| *** mlavalle has quit IRC | 19:09 | |
| dims | geekinutah, would that cause a infinite loop? (after a EBUSY failure in deleting a directory, next ndb mount is hanging indefinitely) | 19:20 |
| *** markwash has joined #openstack-neutron | 19:24 | |
| *** ashaikh has joined #openstack-neutron | 19:25 | |
| *** alexpilotti has quit IRC | 20:07 | |
| *** clev has quit IRC | 20:14 | |
| *** markmcclain has quit IRC | 20:19 | |
| *** markwash has quit IRC | 20:23 | |
| *** alexpilotti has joined #openstack-neutron | 20:34 | |
| *** fatguylittlecoat has quit IRC | 20:34 | |
| *** pcm_ has quit IRC | 20:34 | |
| *** markwash has joined #openstack-neutron | 20:37 | |
| *** markwash has joined #openstack-neutron | 20:40 | |
| *** markwash has quit IRC | 20:40 | |
| *** ijw has joined #openstack-neutron | 20:44 | |
| *** ijw has quit IRC | 20:48 | |
| *** ijw has joined #openstack-neutron | 21:00 | |
| *** ijw has quit IRC | 21:05 | |
| *** ashaikh has quit IRC | 21:16 | |
| *** ijw has joined #openstack-neutron | 21:22 | |
| *** ijw_ has joined #openstack-neutron | 21:24 | |
| *** ijw has quit IRC | 21:26 | |
| *** JoeHazzers has quit IRC | 21:31 | |
| *** JoeHazzers has joined #openstack-neutron | 21:31 | |
| *** ijw_ has quit IRC | 21:32 | |
| *** clev has joined #openstack-neutron | 21:38 | |
| *** clev has quit IRC | 21:39 | |
| geekinutah | sorry just got back | 21:40 |
| geekinutah | dims: I wonder if the kernel thread or qemu-nbd is in D state | 21:40 |
| geekinutah | looking at the nbd code I don't see an obvious deadlock or anything, but the fact that somehow either a socket doesn't get cleaned up or a kernel thread is still running makes me think there are probably other bugs hanging around | 21:41 |
| geekinutah | anyway, we should log that, log 'lsof +f /mountpoint' and probably log 'dm_setup ls' when we encounter a failure | 21:42 |
| geekinutah | this will at least give us good infos | 21:42 |
| geekinutah | anteaya: ^ | 21:44 |
| geekinutah | dims: http://logs.openstack.org/60/63960/1/experimental/check-tempest-dsvm-neutron-pg/b389650/logs/screen-n-cpu.txt.gz | 21:49 |
| geekinutah | it's hanging on /dev/nbd14 again? | 21:49 |
| *** dims has quit IRC | 21:58 | |
| *** jaypipes has quit IRC | 22:07 | |
| *** dims has joined #openstack-neutron | 22:12 | |
| geekinutah | dims enikanorov anteaya: https://lkml.org/lkml/2013/11/12/315 | 22:14 |
| geekinutah | is there anything that the neutron tests do that might cause the condition described in this thread to happen? | 22:14 |
| geekinutah | it seems unlikely, but we would just have to interrupt connectivity for a short amount of time for this to happen | 22:15 |
| *** aymenfrikha has joined #openstack-neutron | 22:17 | |
| *** ashaikh has joined #openstack-neutron | 22:25 | |
| geekinutah | mmm, actually scratch that theory, we would get the kernel stack trace in this version of kernel | 22:29 |
| *** mlavalle has joined #openstack-neutron | 22:29 | |
| *** markwash has joined #openstack-neutron | 22:32 | |
| *** mlavalle_ has joined #openstack-neutron | 22:33 | |
| *** mlavalle has quit IRC | 22:36 | |
| *** mlavalle_ is now known as mlavalle | 22:36 | |
| anteaya | geekinutah: thanks for coming back | 22:55 |
| anteaya | lots of good ideas here | 22:55 |
| anteaya | can you add any of them to the bug report? | 22:55 |
| anteaya | I am going to go offline for the night and don't expect to be around much the next 2 days | 22:56 |
| anteaya | any progress is good progress but take a break too, it is time to enjoy a bit of a break | 22:56 |
| geekinutah | well, I'm just about to leave | 22:56 |
| anteaya | geekinutah dims enikanorov__ thanks for all your help on this | 22:56 |
| geekinutah | I will add what I find if anything to the bug report | 22:56 |
| * anteaya nods | 22:56 | |
| anteaya | thank you | 22:56 |
| *** ashaikh has quit IRC | 22:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!