| *** nati_ueno has quit IRC | 00:00 | |
| *** nati_ueno has joined #openstack-neutron | 00:02 | |
| pcm_ | Folks familiar with L3 services, I have some questions about vendor plugins. Can anyone field them? | 00:02 |
|---|---|---|
| *** nati_ueno has quit IRC | 00:02 | |
| *** nati_ueno has joined #openstack-neutron | 00:03 | |
| *** armax has joined #openstack-neutron | 00:04 | |
| *** Mierdin has joined #openstack-neutron | 00:04 | |
| *** dims has joined #openstack-neutron | 00:07 | |
| *** Ric_ has quit IRC | 00:13 | |
| *** WackoRobie has joined #openstack-neutron | 00:15 | |
| *** banix has quit IRC | 00:15 | |
| *** zzelle_ has quit IRC | 00:15 | |
| *** WackoRobie has quit IRC | 00:19 | |
| *** clev has joined #openstack-neutron | 00:20 | |
| *** pcm_ has quit IRC | 00:22 | |
| *** Mr_W has joined #openstack-neutron | 00:23 | |
| *** matsuhashi has joined #openstack-neutron | 00:26 | |
| *** Ric_ has joined #openstack-neutron | 00:27 | |
| *** clev has quit IRC | 00:29 | |
| *** clev has joined #openstack-neutron | 00:37 | |
| *** Ric_ has quit IRC | 00:38 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: Allow multiple DNS forwarders for dnsmasq https://review.openstack.org/62002 | 00:38 |
| *** xianghui has joined #openstack-neutron | 00:40 | |
| *** clev has quit IRC | 00:42 | |
| *** thuc has joined #openstack-neutron | 00:42 | |
| *** thuc has quit IRC | 00:43 | |
| *** thuc has joined #openstack-neutron | 00:44 | |
| *** Sukhdev has joined #openstack-neutron | 00:46 | |
| anteaya | this patch is in the gate: https://review.openstack.org/#/c/67475/ | 00:46 |
| anteaya | how did it get there? | 00:47 |
| *** clev has joined #openstack-neutron | 00:49 | |
| *** jecarey has quit IRC | 00:51 | |
| *** nati_uen_ has joined #openstack-neutron | 00:51 | |
| *** MM_at_HP has quit IRC | 00:52 | |
| *** nati_ueno has quit IRC | 00:54 | |
| *** ijw has quit IRC | 00:55 | |
| *** julim has joined #openstack-neutron | 01:04 | |
| *** nati_uen_ has quit IRC | 01:07 | |
| *** nati_ueno has joined #openstack-neutron | 01:07 | |
| *** Guest11536 has quit IRC | 01:09 | |
| *** sn6i23a has quit IRC | 01:15 | |
| *** thuc has quit IRC | 01:29 | |
| *** thuc has joined #openstack-neutron | 01:30 | |
| openstackgerrit | A change was merged to openstack/neutron: Fix the migration adding a UC to agents table https://review.openstack.org/61663 | 01:33 |
| *** thuc has quit IRC | 01:33 | |
| *** yamahata has joined #openstack-neutron | 01:40 | |
| *** IanGovett has quit IRC | 01:41 | |
| *** clev has quit IRC | 01:42 | |
| *** banix has joined #openstack-neutron | 01:42 | |
| *** clev has joined #openstack-neutron | 01:44 | |
| *** mestery_ has joined #openstack-neutron | 01:45 | |
| *** markwash has joined #openstack-neutron | 01:48 | |
| *** mestery has quit IRC | 01:48 | |
| *** Coma_ has joined #openstack-neutron | 01:58 | |
| Coma_ | Hello. | 01:58 |
| *** clev has quit IRC | 02:04 | |
| *** Mr_W has quit IRC | 02:08 | |
| *** WackoRobie has joined #openstack-neutron | 02:09 | |
| *** sbalukoff has quit IRC | 02:11 | |
| anteaya | nati_ueno: so isloated jobs haven't been fixed yet, is someone babysitting https://review.openstack.org/#/c/65034/ to snipe it out of the gate if it starts failing tests? | 02:16 |
| anteaya | if no and it fails tests we just create ill will with the rest of the projects | 02:17 |
| anteaya | I am at a conference and can't babysit it myself | 02:17 |
| marun | Is there a reason not to kick it out of the gate preemptively? | 02:18 |
| marun | It doesn't seem like it should be competing with patches that have to land. | 02:18 |
| anteaya | we can do that | 02:18 |
| anteaya | and I agree | 02:18 |
| anteaya | since it just went in 20 minutes ago, I want to give nati_ueno an opportunity to respond | 02:19 |
| marun | I'm betting nachi missed your plea not to approve | 02:20 |
| *** vkozhukalov has joined #openstack-neutron | 02:20 | |
| anteaya | no, he is aware | 02:20 |
| marun | really? then why did he do it? | 02:20 |
| marun | I mean, didn't we agree not to send things to the gate that hadn't passed the check job? | 02:21 |
| marun | I say kick it out now. | 02:21 |
| marun | huh, so it actually did pass the check job. does the +1 for that get wiped when the gate job is triggered? | 02:23 |
| nati_ueno | hi | 02:23 |
| marun | I'm confused, was the check job passing an anomoly? | 02:23 |
| marun | or are things moving again? | 02:24 |
| nati_ueno | anteaya: sorry I found your ping now | 02:24 |
| nati_ueno | marun: anteaya: what's up? | 02:25 |
| marun | nati_ueno: why did you approve https://review.openstack.org/#/c/65034/ | 02:25 |
| marun | nati_ueno: I'm not clear on why it should have gate priority vs all the critical patches that have been held up | 02:25 |
| nati_ueno | marun: ah sorry something wrong happening? Jenkins was OK, so I thought it can be merged | 02:26 |
| marun | nati_ueno: (i'm also confused as to how the checks are passing given that https://review.openstack.org/#/c/67537/ hasn't merged yet) | 02:26 |
| nati_ueno | marun: may be, i'm missing to read community mail | 02:27 |
| nati_ueno | marun: so we shouldn't approve patches now? | 02:27 |
| anteaya | I am on crappy conf wifi, sorry | 02:27 |
| *** aymenfrikha has left #openstack-neutron | 02:28 | |
| marun | nati_ueno: that was my understanding for the past week | 02:28 |
| anteaya | the tests passing was an anomoly | 02:28 |
| marun | nati_ueno: did something change? | 02:28 |
| anteaya | you will see that the builds failed prior to that one set of passing tests | 02:28 |
| nati_ueno | marun: ah sorry, i was pretty buzy, so I didn't know the situation | 02:28 |
| marun | anteaya: so we're still stuck with the isolated job failures due to kernel panics? | 02:28 |
| nati_ueno | anteaya: you mean jenkins on that patch? | 02:29 |
| nati_ueno | anteaya: last jenkins job looks succeeded for that job | 02:29 |
| anteaya | we are still experiencing isolated job failures, yes | 02:29 |
| marun | nati_ueno: afaik there are a bunch of patches that should land before approval of non-critical patches should resume | 02:29 |
| *** clev has joined #openstack-neutron | 02:29 | |
| anteaya | the working theory is that kernel panics play a role, yes | 02:29 |
| marun | nati_ueno: and we're waiting on jenkins stability first | 02:29 |
| nati_ueno | marun: sure I put -2 on that patch | 02:30 |
| nati_ueno | marun: I'm not sure this stop merging but let's try | 02:30 |
| anteaya | but that theory may not be bourne out by early returns today, since what little I saw says the experiemtnal job logs remove kernel failures but the failure itself still persits | 02:30 |
| marun | anteaya: is -2'ing the job enough to pull it out of the gate queue? | 02:30 |
| anteaya | salv-orlando has a better understanding on this one | 02:30 |
| anteaya | no, we must submit another patchset | 02:31 |
| anteaya | -2'ing the patch just prevents it from merging, it doestn' remove it from the gate queue | 02:31 |
| anteaya | nati_ueno: yes the most recent jenkins checks succeeded for that patch | 02:31 |
| anteaya | but the jobs prior to that failed | 02:31 |
| nati_ueno | anteaya: so I should push new patchset for this one? | 02:32 |
| anteaya | yes please | 02:32 |
| *** yamahata has quit IRC | 02:32 | |
| nati_ueno | anteaya: the jobs prior to that failed <-- is this matters? | 02:33 |
| anteaya | yes | 02:33 |
| anteaya | it demonstrates that our testing is unstable in teh gate | 02:33 |
| anteaya | neutron testing being unstable in the gate is the reason we have a hold on approvals | 02:33 |
| anteaya | and I am keeping an eye and removing patches from the gate queue that get in on reverify | 02:34 |
| openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: Make metaplugin be used with a router service plugin https://review.openstack.org/65034 | 02:34 |
| anteaya | thank you nati_ueno | 02:34 |
| nati_ueno | anteaya: I just changed the commit message, is this works? | 02:34 |
| anteaya | Queue lengths: 4 events, 28 results. | 02:34 |
| openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: Make metaplugin be used with a router service plugin https://review.openstack.org/65034 | 02:34 |
| anteaya | when events reachs 0 the patch should be removed | 02:34 |
| nati_ueno | I just revert commit message | 02:34 |
| anteaya | okay | 02:35 |
| anteaya | yes, when I remove patches from the gate, I just edit the commit message | 02:35 |
| anteaya | something along the lines of this patch has been removed from the gate because <reason> | 02:35 |
| nati_ueno | anteaya: ok | 02:36 |
| anteaya | thanks | 02:36 |
| nati_ueno | anteaya: could you share mailing list thread for discussing this issues? | 02:36 |
| anteaya | this one has not been approved but is still in the gate, I will snipe it out | 02:36 |
| anteaya | I just use the irc logs | 02:36 |
| anteaya | I have just been discussing in irc | 02:37 |
| anteaya | I didn't want to take it to the ml, because of profile | 02:37 |
| nati_ueno | anteaya: Ah Ok so there is no announcement for DO NOT APPROVE NOW ? | 02:37 |
| anteaya | we have some good cred right now with our hard work with the code sprint | 02:37 |
| anteaya | I didn't want to undo that with a ml discussion | 02:37 |
| anteaya | no | 02:37 |
| *** yamahata has joined #openstack-neutron | 02:37 | |
| anteaya | just in irc | 02:37 |
| nati_ueno | anteaya: in this chat room? | 02:38 |
| anteaya | if we can deal with this in irc, that is my preference | 02:38 |
| anteaya | yes | 02:38 |
| nati_ueno | I don't think everyone read all messages | 02:38 |
| anteaya | channel logs: http://eavesdrop.openstack.org/irclogs/%23openstack-neutron/ | 02:38 |
| anteaya | they don't | 02:38 |
| anteaya | that is part of the problem | 02:38 |
| marun | nati_ueno: there was an announcement, | 02:38 |
| nati_ueno | marun: ah OK, could you share the mail thread? | 02:39 |
| nati_ueno | marun: I wanna read it | 02:39 |
| *** Jabadia has quit IRC | 02:39 | |
| marun | nati_ueno: pretty sure it was core only, not public ml | 02:39 |
| nati_ueno | marun: my bad,,, I haven't read it.. | 02:39 |
| marun | nati_ueno: ah, it was related to check jobs only :( | 02:39 |
| *** Jabadia has joined #openstack-neutron | 02:39 | |
| nati_ueno | marun: could you share the subject? | 02:40 |
| marun | nati_ueno: Patchset Approvals | 02:40 |
| marun | went to nachi at ntti3 | 02:40 |
| nati_ueno | marun: I found the mail. Thanks | 02:40 |
| nati_ueno | marun: so even if jenkins is OK, we shouldn't approve it | 02:41 |
| marun | nati_ueno: not at present, no | 02:41 |
| nati_ueno | marun: anyway, I missed to read this one | 02:41 |
| marun | nati_ueno: and given the backlog, when jenkins does stabilize, critical patches will need to be prioritized. if a bug hasn't been triaged as critical, it probably shouldn't be approved at first. | 02:41 |
| openstackgerrit | Anita Kuno proposed a change to openstack/neutron: Avoid unnecessarily checking the existence of a device https://review.openstack.org/67475 | 02:42 |
| * marun wishes that gerrit could post bug severity automatically | 02:42 | |
| nati_ueno | marun: OK Thanks for letting me know | 02:42 |
| nati_ueno | That's good idea | 02:42 |
| marun | nati_ueno: i'm starting to use the qgerrit cli tool | 02:42 |
| anteaya | we are working on a gerrit upgrade, I am uncertain of the exact feature set | 02:43 |
| marun | nati_ueno: it should be easy enough to cross-reference with launchpad to sort by severity | 02:43 |
| nati_ueno | marun: http://status.openstack.org/reviews/#neutron This one has priority | 02:43 |
| nati_ueno | marun: anteaya: Anyway, Thank you for pinging me this one. I should go now. TTYL | 02:44 |
| marun | nati_ueno: neat. too bad it's not sortable by the severity but that can't be so hard to add | 02:44 |
| anteaya | marun, you asked if the +1 goes away if gate jobs start, the answer is yes | 02:44 |
| marun | anteaya: good to know | 02:44 |
| anteaya | nati_ueno: thanks for your understanding, ttyl | 02:44 |
| *** iwamoto has joined #openstack-neutron | 02:45 | |
| *** markwash has quit IRC | 02:45 | |
| *** nati_uen_ has joined #openstack-neutron | 02:45 | |
| *** Sukhdev has quit IRC | 02:46 | |
| *** salv-orlando has quit IRC | 02:46 | |
| *** nati_ueno has quit IRC | 02:48 | |
| *** nati_uen_ has quit IRC | 02:50 | |
| *** snowblind_ has joined #openstack-neutron | 02:52 | |
| *** Coma_ has quit IRC | 02:52 | |
| *** shivharis has quit IRC | 02:55 | |
| *** clev has quit IRC | 02:56 | |
| openstackgerrit | Armando Migliaccio proposed a change to openstack/neutron: Tweak version nvp/nsx version validation logic for router operations https://review.openstack.org/70043 | 02:59 |
| *** clev has joined #openstack-neutron | 02:59 | |
| *** emagana has quit IRC | 03:05 | |
| *** dave_tucker has left #openstack-neutron | 03:08 | |
| *** pheadron1 has joined #openstack-neutron | 03:10 | |
| *** WackoRobie has quit IRC | 03:11 | |
| *** WackoRobie has joined #openstack-neutron | 03:11 | |
| *** pheadron has quit IRC | 03:13 | |
| *** WackoRob_ has joined #openstack-neutron | 03:15 | |
| *** WackoRobie has quit IRC | 03:15 | |
| *** WackoRob_ has quit IRC | 03:16 | |
| *** WackoRobie has joined #openstack-neutron | 03:16 | |
| *** clev has quit IRC | 03:19 | |
| *** markwash has joined #openstack-neutron | 03:19 | |
| *** godara has quit IRC | 03:21 | |
| *** markmcclain has quit IRC | 03:21 | |
| *** mestery_ is now known as mestery | 03:21 | |
| *** WackoRobie has quit IRC | 03:21 | |
| *** clev has joined #openstack-neutron | 03:21 | |
| *** markmcclain has joined #openstack-neutron | 03:21 | |
| *** sgordon has quit IRC | 03:22 | |
| *** matsuhashi has quit IRC | 03:29 | |
| *** Ric_ has joined #openstack-neutron | 03:31 | |
| *** jp_at_hp has quit IRC | 03:35 | |
| *** packet has quit IRC | 03:36 | |
| *** chandankumar_ has joined #openstack-neutron | 03:36 | |
| *** xianghui has quit IRC | 03:37 | |
| *** markwash has quit IRC | 03:38 | |
| *** chandankumar_ has quit IRC | 03:40 | |
| *** chandankumar_ has joined #openstack-neutron | 03:41 | |
| *** markwash has joined #openstack-neutron | 03:45 | |
| *** chandankumar_ has quit IRC | 03:46 | |
| *** chandankumar__ has joined #openstack-neutron | 03:47 | |
| *** aveiga has quit IRC | 03:49 | |
| *** chandankumar__ has left #openstack-neutron | 03:49 | |
| *** Ric_ has quit IRC | 03:50 | |
| *** alex_klimov has joined #openstack-neutron | 03:57 | |
| *** clev has quit IRC | 04:00 | |
| *** clev has joined #openstack-neutron | 04:01 | |
| *** markwash has quit IRC | 04:02 | |
| *** matsuhashi has joined #openstack-neutron | 04:02 | |
| *** alex_klimov has quit IRC | 04:02 | |
| *** clev has quit IRC | 04:11 | |
| *** clev has joined #openstack-neutron | 04:16 | |
| *** shashank_ has quit IRC | 04:17 | |
| *** Ric_ has joined #openstack-neutron | 04:21 | |
| *** pheadron has joined #openstack-neutron | 04:25 | |
| *** WackoRobie has joined #openstack-neutron | 04:27 | |
| *** pheadron1 has quit IRC | 04:28 | |
| *** pheadron1 has joined #openstack-neutron | 04:30 | |
| *** WackoRobie has quit IRC | 04:31 | |
| *** Ric_ has quit IRC | 04:32 | |
| *** pheadron has quit IRC | 04:33 | |
| *** markmcclain has quit IRC | 04:34 | |
| openstackgerrit | Armando Migliaccio proposed a change to openstack/neutron: Tweak version nvp/nsx version validation logic for router operations https://review.openstack.org/70043 | 04:46 |
| *** Ric_ has joined #openstack-neutron | 04:46 | |
| *** alex_klimov has joined #openstack-neutron | 04:48 | |
| *** ykaneko has joined #openstack-neutron | 04:56 | |
| *** yfried has joined #openstack-neutron | 04:59 | |
| *** Jabadia has quit IRC | 05:06 | |
| *** Jabadia has joined #openstack-neutron | 05:06 | |
| *** matsuhashi has quit IRC | 05:14 | |
| *** yfried has quit IRC | 05:16 | |
| *** matsuhas_ has joined #openstack-neutron | 05:17 | |
| *** banix has quit IRC | 05:19 | |
| *** thuc_ has joined #openstack-neutron | 05:21 | |
| *** pheadron has joined #openstack-neutron | 05:27 | |
| *** rwsu has quit IRC | 05:27 | |
| *** pheadron1 has quit IRC | 05:30 | |
| *** clev has quit IRC | 05:32 | |
| *** irenab has joined #openstack-neutron | 05:38 | |
| *** shashank_ has joined #openstack-neutron | 05:45 | |
| openstackgerrit | Yohei Matsuhashi proposed a change to openstack/python-neutronclient: Enable to select specific network service type https://review.openstack.org/54534 | 05:56 |
| openstackgerrit | oda-g proposed a change to openstack/neutron: Enhance GET networks performance of metaplugin https://review.openstack.org/70060 | 06:03 |
| *** thuc has joined #openstack-neutron | 06:13 | |
| *** pheadron1 has joined #openstack-neutron | 06:14 | |
| *** thuc_ has quit IRC | 06:17 | |
| *** thuc has quit IRC | 06:17 | |
| *** pheadron has quit IRC | 06:17 | |
| *** vkozhukalov has quit IRC | 06:18 | |
| *** gdubreui has quit IRC | 06:22 | |
| *** yjiang5 has joined #openstack-neutron | 06:27 | |
| openstackgerrit | Jenkins proposed a change to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/66501 | 06:28 |
| *** yjiang5_1 has quit IRC | 06:29 | |
| *** nati_ueno has joined #openstack-neutron | 06:43 | |
| *** Sukhdev has joined #openstack-neutron | 06:47 | |
| *** nati_ueno has quit IRC | 06:48 | |
| *** amuller has joined #openstack-neutron | 07:01 | |
| *** otherwiseguy has quit IRC | 07:06 | |
| *** armax has quit IRC | 07:10 | |
| *** Sukhdev has quit IRC | 07:15 | |
| *** nati_ueno has joined #openstack-neutron | 07:15 | |
| *** gpza has quit IRC | 07:22 | |
| *** nati_ueno has quit IRC | 07:26 | |
| *** humbolt1 has joined #openstack-neutron | 07:27 | |
| *** humbolt has quit IRC | 07:29 | |
| *** ijw has joined #openstack-neutron | 07:37 | |
| *** emagana has joined #openstack-neutron | 07:40 | |
| *** ijw has quit IRC | 07:41 | |
| *** jlibosva has joined #openstack-neutron | 07:44 | |
| *** akamyshnikova has quit IRC | 07:46 | |
| *** sbalukoff has joined #openstack-neutron | 07:47 | |
| *** snowblind_ has quit IRC | 07:48 | |
| *** vkozhukalov has joined #openstack-neutron | 07:57 | |
| *** jistr has joined #openstack-neutron | 08:02 | |
| *** djoreilly has joined #openstack-neutron | 08:09 | |
| *** vkozhukalov has quit IRC | 08:10 | |
| *** Mierdin has quit IRC | 08:15 | |
| *** harlowja is now known as harlowja_away | 08:18 | |
| *** vkozhukalov has joined #openstack-neutron | 08:19 | |
| *** jpich has joined #openstack-neutron | 08:20 | |
| *** evgenyf has joined #openstack-neutron | 08:24 | |
| *** emagana has quit IRC | 08:25 | |
| openstackgerrit | IWAMOTO Toshihiro proposed a change to openstack/neutron: LBaaS: implements LVS driver (WORKINPROGRESS) https://review.openstack.org/67405 | 08:26 |
| *** shashank_ has quit IRC | 08:31 | |
| *** iwamoto has quit IRC | 08:32 | |
| *** simon-AS559 has joined #openstack-neutron | 08:33 | |
| *** simon-AS559 has quit IRC | 08:36 | |
| *** ijw has joined #openstack-neutron | 08:40 | |
| *** samuelbercovici has joined #openstack-neutron | 08:42 | |
| *** vkozhukalov has quit IRC | 08:43 | |
| openstackgerrit | Evgeny Fedoruk proposed a change to openstack/neutron: Extending quota support for neutron LBaaS entities https://review.openstack.org/69803 | 08:44 |
| openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 model (WIP) https://review.openstack.org/61721 | 08:47 |
| *** amritanshu_RnD has joined #openstack-neutron | 08:50 | |
| *** amritanshu_RnD is now known as Guest31820 | 08:50 | |
| *** ijw has quit IRC | 08:52 | |
| *** Guest31820 has quit IRC | 08:53 | |
| *** ygbo has joined #openstack-neutron | 08:53 | |
| *** ijw has joined #openstack-neutron | 08:53 | |
| *** amrit_ has joined #openstack-neutron | 08:53 | |
| *** vkozhukalov has joined #openstack-neutron | 08:57 | |
| *** zzelle has joined #openstack-neutron | 08:58 | |
| *** amuller has quit IRC | 09:03 | |
| *** sweston has joined #openstack-neutron | 09:03 | |
| openstackgerrit | Yves-Gwenael Bourhis proposed a change to openstack/neutron: Make dnsmasq aware of all names https://review.openstack.org/52930 | 09:05 |
| *** safchain has joined #openstack-neutron | 09:09 | |
| *** AlexF_ has joined #openstack-neutron | 09:12 | |
| *** jpich has quit IRC | 09:12 | |
| *** simon-AS559 has joined #openstack-neutron | 09:13 | |
| *** yfried has joined #openstack-neutron | 09:16 | |
| *** simon-AS5591 has joined #openstack-neutron | 09:18 | |
| *** samuelbercovici has quit IRC | 09:19 | |
| *** simon-AS559 has quit IRC | 09:20 | |
| *** rossella_s has joined #openstack-neutron | 09:28 | |
| *** jlibosva has quit IRC | 09:32 | |
| *** matsuhas_ has quit IRC | 09:33 | |
| *** matsuhashi has joined #openstack-neutron | 09:39 | |
| *** yamahata has quit IRC | 09:40 | |
| *** jlibosva has joined #openstack-neutron | 09:48 | |
| *** sweston has quit IRC | 09:49 | |
| *** matsuhashi has quit IRC | 09:50 | |
| *** jlibosva has quit IRC | 09:51 | |
| *** jlibosva has joined #openstack-neutron | 09:51 | |
| *** sweston has joined #openstack-neutron | 09:55 | |
| *** jistr has quit IRC | 10:08 | |
| *** jp_at_hp has joined #openstack-neutron | 10:14 | |
| *** amrit_ has quit IRC | 10:17 | |
| *** ijw has quit IRC | 10:17 | |
| *** Mr_W has joined #openstack-neutron | 10:23 | |
| *** ijw has joined #openstack-neutron | 10:23 | |
| *** Mr_W has quit IRC | 10:25 | |
| *** Mr_W has joined #openstack-neutron | 10:25 | |
| *** AlexF_ has quit IRC | 10:37 | |
| *** ykaneko has quit IRC | 10:38 | |
| *** jistr has joined #openstack-neutron | 10:38 | |
| *** rohit404 has joined #openstack-neutron | 10:39 | |
| *** akamyshnikova has joined #openstack-neutron | 10:39 | |
| *** yamahata has joined #openstack-neutron | 10:41 | |
| *** networkstatic has quit IRC | 10:47 | |
| *** vkozhukalov has quit IRC | 10:48 | |
| *** garyachy has joined #openstack-neutron | 10:48 | |
| *** vkozhukalov has joined #openstack-neutron | 11:00 | |
| openstackgerrit | Trinath Somanchi proposed a change to openstack/neutron: implements: blueprint fsl-sdn-os-mech-driver https://review.openstack.org/69838 | 11:07 |
| *** amuller has joined #openstack-neutron | 11:20 | |
| *** Ric_ has quit IRC | 11:22 | |
| *** Ric_ has joined #openstack-neutron | 11:26 | |
| openstackgerrit | Trinath Somanchi proposed a change to openstack/neutron: implements: blueprint fsl-sdn-os-mech-driver https://review.openstack.org/70089 | 11:28 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Metering label as shared https://review.openstack.org/70090 | 11:34 |
| *** evgenyf has quit IRC | 11:36 | |
| *** amritanshu_RnD has joined #openstack-neutron | 11:37 | |
| *** amritanshu_RnD is now known as Guest45771 | 11:37 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/python-neutronclient: Add shared parameter for metering labels https://review.openstack.org/70091 | 11:38 |
| *** jpich has joined #openstack-neutron | 11:38 | |
| *** pheadron1 has quit IRC | 11:39 | |
| *** sweston has quit IRC | 11:45 | |
| *** HenryG has quit IRC | 11:48 | |
| *** pheadron has joined #openstack-neutron | 11:54 | |
| *** shausy has joined #openstack-neutron | 12:09 | |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: Fix wrap target in iptables_manager https://review.openstack.org/67556 | 12:11 |
| *** rohit404 has quit IRC | 12:17 | |
| *** rohit404 has joined #openstack-neutron | 12:19 | |
| *** garyachy has quit IRC | 12:23 | |
| *** ijw has quit IRC | 12:25 | |
| *** afazekas has joined #openstack-neutron | 12:27 | |
| *** IanGovett has joined #openstack-neutron | 12:30 | |
| *** bashok has joined #openstack-neutron | 12:33 | |
| *** aburaschi has joined #openstack-neutron | 12:37 | |
| irenab | hi, asking for review: https://review.openstack.org/#/c/67189/ | 12:44 |
| *** evgenyf has joined #openstack-neutron | 12:49 | |
| *** dave_tucker_zzz has joined #openstack-neutron | 12:53 | |
| *** dave_tucker_zzz is now known as dave_tucker | 12:53 | |
| *** mfink has quit IRC | 12:55 | |
| *** WackoRobie has joined #openstack-neutron | 12:56 | |
| *** pcm_ has joined #openstack-neutron | 13:00 | |
| openstackgerrit | Ofer Barkai proposed a change to openstack/neutron: Enhance floating IP router lookup https://review.openstack.org/55987 | 13:03 |
| *** HenryG has joined #openstack-neutron | 13:03 | |
| *** vkozhukalov has quit IRC | 13:03 | |
| ihrachys | trying to run 'tox' against neutron, getting: No distributions matching the version for psutil>=0.6.1,<1.0 (from -r /home/ihrachys/proj/neutron/requirements.txt (line 19)) | 13:07 |
| ihrachys | any ideas how to fix this? | 13:07 |
| *** evgenyf has quit IRC | 13:15 | |
| *** Mr_W has quit IRC | 13:15 | |
| *** shausy has quit IRC | 13:15 | |
| *** shausy has joined #openstack-neutron | 13:16 | |
| *** yamahata has quit IRC | 13:16 | |
| HenryG | ihrachys: quick (temporary) fix: edit requirements.txt and change to psutil>1.1.0 | 13:17 |
| *** vkozhukalov has joined #openstack-neutron | 13:17 | |
| HenryG | ihrachys: real fix: rebase on https://review.openstack.org/68993 | 13:18 |
| *** yamahata has joined #openstack-neutron | 13:20 | |
| *** WackoRobie has quit IRC | 13:20 | |
| *** Jabadia has quit IRC | 13:27 | |
| *** giulivo has joined #openstack-neutron | 13:30 | |
| *** mestery_ has joined #openstack-neutron | 13:35 | |
| *** Jabadia has joined #openstack-neutron | 13:35 | |
| giulivo | guys, I'd like the dhcp agent (dnsmasq) to pass some dhcp option to the clients | 13:37 |
| *** mestery has quit IRC | 13:37 | |
| giulivo | can I configure that somehow when creating the dhcp or shall I hack into the per-tenant dnsmasq config files? | 13:37 |
| *** Jabadia has quit IRC | 13:39 | |
| enikanorov_ | giulivo: it should be possible with extra dhcp options extension | 13:44 |
| *** evgenyf has joined #openstack-neutron | 13:45 | |
| *** IanGovett has quit IRC | 13:45 | |
| *** yamahata has quit IRC | 13:46 | |
| giulivo | enikanorov, thanks but I'm not much into it I think ... is that something I can use from cli commands when creating the dhcp? | 13:46 |
| *** yamahata has joined #openstack-neutron | 13:47 | |
| giulivo | enikanorov, a variable I can pass to subnet-create maybe? | 13:47 |
| *** thuc has joined #openstack-neutron | 13:49 | |
| giulivo | enikanorov, http://docs.openstack.org/api/openstack-network/2.0/content/extra-dhcp-opt-ext-create.html ? | 13:50 |
| *** markwash has joined #openstack-neutron | 13:55 | |
| enikanorov_ | giulivo: yes | 13:58 |
| giulivo | enikanorov, thanks | 14:01 |
| *** akamyshnikova has quit IRC | 14:02 | |
| *** yfried has quit IRC | 14:03 | |
| *** samuelbercovici has joined #openstack-neutron | 14:03 | |
| *** beagles is now known as beagles_brb | 14:04 | |
| *** vkozhukalov has quit IRC | 14:06 | |
| *** aymenfrikha has joined #openstack-neutron | 14:08 | |
| *** yamahata has quit IRC | 14:09 | |
| *** yamahata has joined #openstack-neutron | 14:09 | |
| *** yamahata has quit IRC | 14:10 | |
| *** yamahata has joined #openstack-neutron | 14:10 | |
| *** dims has quit IRC | 14:15 | |
| *** dims has joined #openstack-neutron | 14:15 | |
| *** clev has joined #openstack-neutron | 14:18 | |
| *** vkozhukalov has joined #openstack-neutron | 14:19 | |
| *** yamahata has quit IRC | 14:19 | |
| dkehn | safchain: tripleo has a need for HA at the routers and was looking at using https://review.openstack.org/#/c/64553/8, would you more folks testing on it? | 14:20 |
| safchain | dkehn, still working on it, will submit the agent side by the of the week | 14:21 |
| *** yamahata has joined #openstack-neutron | 14:21 | |
| dkehn | safchain: thats great news, within a devstack env, usning a single vm is there any obvious changes to the config (i.e. localrc) that are required to get it to run? | 14:22 |
| safchain | dkehn, you can't test it without the agent side | 14:23 |
| *** ivoks has quit IRC | 14:23 | |
| dkehn | safchain: understand | 14:23 |
| *** rha has quit IRC | 14:23 | |
| safchain | dkehn, I'll ping you when I'ill submit the agent patch | 14:23 |
| *** ivoks has joined #openstack-neutron | 14:23 | |
| dkehn | safchain: thanks tons | 14:24 |
| dkehn | safchain: we are looking forward to it | 14:24 |
| *** rha has joined #openstack-neutron | 14:25 | |
| giulivo | enikanorov, pardon one more question, that seems to be something I need to apply on a neutron "port" ... so should that be the port of the dhcp agent on a given subject ... or the port of each and every instance connected? | 14:28 |
| giulivo | *given subnet | 14:28 |
| *** thuc has quit IRC | 14:29 | |
| *** thuc has joined #openstack-neutron | 14:30 | |
| *** peristeri has joined #openstack-neutron | 14:30 | |
| *** IanGovett has joined #openstack-neutron | 14:32 | |
| *** thuc has quit IRC | 14:34 | |
| *** yamahata has quit IRC | 14:38 | |
| *** changbl has quit IRC | 14:39 | |
| *** yamahata has joined #openstack-neutron | 14:40 | |
| enikanorov_ | giulivo: probably dkehn could help you better than i | 14:41 |
| giulivo | again, thanks for helping | 14:42 |
| giulivo | dkehn, basically I'd like to pass some dhcp option to all clients so I wanted to customize the dnsmasq file for the dhcp agent | 14:42 |
| giulivo | I was now looking into http://docs.openstack.org/api/openstack-network/2.0/content/extra-dhc-opt-ext-update.html which seems to apply on a per-port basis, so that should be for each client on the subnet? | 14:43 |
| *** afazekas has quit IRC | 14:43 | |
| dkehn | giulivo: yes, its on a per port | 14:43 |
| dkehn | giulivo: the way each opts file write out is usually by network | 14:44 |
| giulivo | dkehn, I see thanks and any chance I could instad apply it to all client via some "overriding" or similar in another file? | 14:45 |
| dkehn | giulivo: I would think that as you create the networks and the underlying ports keeping the same dhcp_options would be the way | 14:45 |
| *** Jabadia has joined #openstack-neutron | 14:46 | |
| *** salv-orlando has joined #openstack-neutron | 14:46 | |
| giulivo | dkehn, not sure I understand that ... aren't the ports created/delete with every instance created/deleted? how do I "keep" the underlying ports? | 14:47 |
| dkehn | giulivo: you don't I was thinking progmatically, if you were in a loop creating networks and the ports assocaited to it. | 14:48 |
| giulivo | dkehn, oh sorry, okay | 14:48 |
| dkehn | giulivo: np, I need to explain better | 14:49 |
| giulivo | anyway thanks for helping | 14:49 |
| giulivo | so is it worth a bug or something? I see dnsservers for instance are "global" | 14:49 |
| giulivo | maybe there is chance to have something similar for the other options too? | 14:49 |
| *** bashok has quit IRC | 14:50 | |
| *** Jabadia has quit IRC | 14:50 | |
| *** banix has joined #openstack-neutron | 14:50 | |
| dkehn | giulivo: we are looking into options on a per-subnet basis, but there are a lot of options that can be applied at differenet items (i.e. ports, subnets, etc.) | 14:51 |
| giulivo | heh, understand that, thanks | 14:51 |
| *** Jabadia has joined #openstack-neutron | 14:52 | |
| *** mestery_ is now known as mestery | 14:56 | |
| dkehn | mestery: snowin up there again? | 14:57 |
| *** Jabadia has quit IRC | 14:57 | |
| * mestery nods. | 14:57 | |
| mestery | dkehn: Getting lots of snow right now! | 14:57 |
| dkehn | mestery: ours is starting tonight , snowin in the high country now, if you didn't get up there to go skiing you missed it | 14:58 |
| *** Lill__ has joined #openstack-neutron | 14:58 | |
| Lill__ | Hi, Nova has a force_dhcp_release directive, what's the equivalent in Neutron? What's the default in Neutron? | 14:58 |
| *** alex_klimov has quit IRC | 14:58 | |
| mestery | dkehn: Ha! We get cold temps following this tonight I hear. Should make for some fun driving later today. :) | 14:59 |
| dkehn | mestery: wonder how markmcclain is dealing, but then again, hard to get a snow day when you work remote | 15:00 |
| *** iwamoto has joined #openstack-neutron | 15:01 | |
| *** IanGovett has quit IRC | 15:01 | |
| mestery | dkehn: Yeah, those pics from Atlanta made it look like the world was ending down there. :) | 15:01 |
| enikanorov_ | samuelbercovici: that's the same as with members and pools relationship. not much profit from sharing members between pools | 15:01 |
| *** clev_ has joined #openstack-neutron | 15:01 | |
| obondarev | enikanorov_: agree | 15:02 |
| dkehn | mestery: fear and loathing, cats sleeping with dogs, yep, I'll bet the sale of SUVs is up next year | 15:02 |
| *** s3wong has joined #openstack-neutron | 15:02 | |
| mestery | dkehn: :) | 15:02 |
| samuelbercovici | enikanorov_: in the past i have alreadyproposed that member definition will be shared between multiple pool but the current state is acceptable | 15:02 |
| samuelbercovici | enikanorov_: obondarev: in the case of pools it is different | 15:03 |
| dkehn | mestery: also the number of snow removal vehicles will probably double | 15:03 |
| samuelbercovici | we have setups where the same pools are associated to different vips to genereate a distributed topology | 15:03 |
| enikanorov_ | how is it different? | 15:03 |
| enikanorov_ | but pools can be cloned | 15:03 |
| *** clev has quit IRC | 15:04 | |
| samuelbercovici | enikanorov_: with what i understand now about lb instance, this will not be possible | 15:04 |
| samuelbercovici | enikanorov_: if we clone than we need to manage adding and removing members to all the clones | 15:04 |
| enikanorov_ | samuelbercovici: right | 15:04 |
| samuelbercovici | the whole idea of the lbaas model was that it is logical and not physical | 15:05 |
| *** IanGovett has joined #openstack-neutron | 15:05 | |
| enikanorov_ | and it is | 15:05 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Metering label as shared https://review.openstack.org/70090 | 15:05 |
| samuelbercovici | enikanorov_: what it looks is that the backend implementation gets reflected back into the api | 15:06 |
| *** bada has joined #openstack-neutron | 15:06 | |
| enikanorov_ | i don't see it is | 15:07 |
| samuelbercovici | enikanorov_: anyway, will look again on the patch to understand it better | 15:07 |
| enikanorov_ | sure | 15:07 |
| enikanorov_ | right now i see lb instance solve whole list of problems when multiple vips and pools are considered | 15:08 |
| enikanorov_ | both for user and for developer | 15:08 |
| *** Lill__ has quit IRC | 15:09 | |
| samuelbercovici | enikanorov_: the fact that it looks like you would expect that to use a pool in a l7 policy it must be associated to the same lb instance whcih in ha proxy case is also an ha proxy service is exactly that | 15:09 |
| iwamoto | I'd like to see https://wiki.openstack.org/wiki/Neutron/LBaaS/LoadbalancerInstance get updated after this discussion | 15:09 |
| enikanorov_ | yes, in order to use a pool in l7 case, it should belong to the same instance | 15:10 |
| enikanorov_ | iwamoto: ok, will do | 15:10 |
| iwamoto | enikanorov_: thanks | 15:11 |
| samuelbercovici | enikanorov_:i don't like this limitation | 15:12 |
| *** shausy has quit IRC | 15:12 | |
| *** shausy has joined #openstack-neutron | 15:13 | |
| enikanorov_ | not sure how else that could be. the pool which is on another device? | 15:13 |
| samuelbercovici | it is similar to the health monitor model | 15:14 |
| enikanorov_ | ok, you're refering to the pool definition shared between devices | 15:15 |
| samuelbercovici | enikanorov_: correct | 15:15 |
| *** jecarey has joined #openstack-neutron | 15:15 | |
| samuelbercovici | and it can then aslo be shared in the device if it needs to ditribute to several underlying backends | 15:15 |
| * iwamoto (0am here. see you later...) | 15:16 | |
| *** iwamoto has quit IRC | 15:16 | |
| *** amuller has quit IRC | 15:16 | |
| *** otherwiseguy has joined #openstack-neutron | 15:17 | |
| enikanorov_ | i think we could improve flexibility further eventually, based on demand | 15:17 |
| *** dims has quit IRC | 15:18 | |
| samuelbercovici | enikanorov_: i will look at the proposed change again and will let you know if i c additional issues | 15:19 |
| *** dave_tucker is now known as dave_tucker_zzz | 15:20 | |
| enikanorov_ | sure! | 15:20 |
| *** shausy has quit IRC | 15:20 | |
| *** rohit404 has left #openstack-neutron | 15:21 | |
| *** dims has joined #openstack-neutron | 15:21 | |
| *** rwsu has joined #openstack-neutron | 15:24 | |
| *** jgrimm has joined #openstack-neutron | 15:25 | |
| *** markwash has quit IRC | 15:29 | |
| *** evgenyf has quit IRC | 15:30 | |
| *** dave_tucker_zzz is now known as dave_tucker | 15:35 | |
| *** s3wong has quit IRC | 15:35 | |
| *** alagalah has quit IRC | 15:35 | |
| * roaet waves. | 15:36 | |
| roaet | Oh hey. this isn't nova. | 15:37 |
| *** terrylhowe has joined #openstack-neutron | 15:37 | |
| *** aymenfrikha has quit IRC | 15:38 | |
| *** emagana has joined #openstack-neutron | 15:39 | |
| *** emagana has quit IRC | 15:39 | |
| *** emagana has joined #openstack-neutron | 15:39 | |
| *** markmcclain has joined #openstack-neutron | 15:41 | |
| *** zzelle has quit IRC | 15:45 | |
| *** pheadron1 has joined #openstack-neutron | 15:45 | |
| *** clev_ has quit IRC | 15:46 | |
| *** AlexF_ has joined #openstack-neutron | 15:48 | |
| *** pheadron has quit IRC | 15:48 | |
| *** tongli has joined #openstack-neutron | 15:50 | |
| *** MM_at_HP has joined #openstack-neutron | 15:50 | |
| *** evgenyf has joined #openstack-neutron | 15:51 | |
| *** beagles_brb is now known as beagles | 15:54 | |
| *** aburaschi has quit IRC | 15:54 | |
| *** terryw has joined #openstack-neutron | 15:55 | |
| *** aburaschi has joined #openstack-neutron | 15:55 | |
| *** WackoRobie has joined #openstack-neutron | 15:56 | |
| *** otherwiseguy has quit IRC | 15:58 | |
| openstackgerrit | Berezovsky Irena proposed a change to openstack/neutron: Change tenant network type usage for IB Fabric https://review.openstack.org/67189 | 16:00 |
| *** matrohon has quit IRC | 16:00 | |
| *** simon-AS5591 has quit IRC | 16:01 | |
| *** simon-AS559 has joined #openstack-neutron | 16:01 | |
| openstackgerrit | Berezovsky Irena proposed a change to openstack/neutron: Change tenant network type usage for IB Fabric https://review.openstack.org/67189 | 16:04 |
| *** clev has joined #openstack-neutron | 16:05 | |
| *** thedodd has joined #openstack-neutron | 16:05 | |
| *** unicell1 has joined #openstack-neutron | 16:05 | |
| *** unicell1 has joined #openstack-neutron | 16:05 | |
| *** simon-AS559 has quit IRC | 16:06 | |
| *** carl_baldwin has joined #openstack-neutron | 16:06 | |
| *** irenab has quit IRC | 16:09 | |
| *** alagalah has joined #openstack-neutron | 16:10 | |
| *** pheadron has joined #openstack-neutron | 16:10 | |
| *** pheadron1 has quit IRC | 16:13 | |
| *** evgenyf has quit IRC | 16:15 | |
| *** alagalah has left #openstack-neutron | 16:17 | |
| *** mlavalle has joined #openstack-neutron | 16:30 | |
| *** thuc has joined #openstack-neutron | 16:30 | |
| *** thuc has quit IRC | 16:31 | |
| *** jprovazn has joined #openstack-neutron | 16:31 | |
| *** thuc has joined #openstack-neutron | 16:31 | |
| *** thuc has quit IRC | 16:32 | |
| *** thuc has joined #openstack-neutron | 16:32 | |
| openstackgerrit | Avishay Balderman proposed a change to openstack/neutron: LBaaS L7 model (WIP) https://review.openstack.org/61721 | 16:34 |
| *** networks_ has joined #openstack-neutron | 16:34 | |
| *** armax has joined #openstack-neutron | 16:36 | |
| *** Guest45771 has quit IRC | 16:38 | |
| *** sweston has joined #openstack-neutron | 16:40 | |
| *** changbl has joined #openstack-neutron | 16:41 | |
| *** sweston has quit IRC | 16:41 | |
| *** yfried has joined #openstack-neutron | 16:42 | |
| *** sweston has joined #openstack-neutron | 16:43 | |
| openstackgerrit | András Gyácsok proposed a change to openstack/neutron: Removed copyright from empty files https://review.openstack.org/63901 | 16:44 |
| openstackgerrit | András Gyácsok proposed a change to openstack/neutron: Removed copyright from empty files https://review.openstack.org/63901 | 16:52 |
| *** morganfainberg|z is now known as morganfainberg | 16:52 | |
| *** dave_tucker is now known as dave_tucker_zzz | 16:56 | |
| *** dave_tucker_zzz is now known as dave_tucker | 16:57 | |
| *** afazekas has joined #openstack-neutron | 16:58 | |
| *** dave_tucker is now known as dave_tucker_zzz | 17:00 | |
| *** armax has left #openstack-neutron | 17:06 | |
| *** aymenfrikha has joined #openstack-neutron | 17:21 | |
| *** afazekas has quit IRC | 17:21 | |
| *** networks_ has quit IRC | 17:22 | |
| *** aymenfrikha has quit IRC | 17:22 | |
| *** networks_ has joined #openstack-neutron | 17:23 | |
| *** networks_ is now known as networkstatic | 17:23 | |
| *** julim has quit IRC | 17:23 | |
| *** ygbo has quit IRC | 17:24 | |
| *** dave_tucker_zzz is now known as dave_tucker | 17:24 | |
| mestery | salv-orlando: Regarding https://bugs.launchpad.net/neutron/+bug/1273386, I am going to see if I can reproduce this locally or not. | 17:26 |
| mestery | To help diagnose and maybe fix it. | 17:26 |
| salv-orlando | thanks mastery. Only info I have for you is that we've never seen that happening the our CI which runs 3.2.0-39, we've seen it the first time with 3.2.0-54 and it became a lot worse with 3.2.0-58. We also keep seeing it in 3.11 | 17:27 |
| salv-orlando | s/mastery/mestery | 17:27 |
| *** shashank_ has joined #openstack-neutron | 17:27 | |
| mestery | salv-orlando: OK, those are good data points! I'll see what I can do to reproduce it locally, even without OpenStack running. | 17:28 |
| mestery | salv-orlando: If it's a namespace issue itself which is causing this, should be possible. | 17:28 |
| *** francois_eleouet has quit IRC | 17:28 | |
| salv-orlando | mestery you should be able to write a simple script that uses neutron.agent.linux.external_process to launch processes in namespaces | 17:28 |
| salv-orlando | this should stress things out | 17:29 |
| salv-orlando | Running tox can be a tedious process. I did this for 4 hours in the past two days without seeing any failures. | 17:29 |
| mestery | salv-orlando: Thanks! | 17:29 |
| mestery | You mean you ran tox or the script? | 17:29 |
| salv-orlando | see also instructions for crash dump smb left among comments for bug 1273386 | 17:29 |
| * mestery nods. | 17:30 | |
| salv-orlando | nope just run the script. With 'run tox' I actually meant 'run tempest' | 17:30 |
| mestery | Thanks salv-orlando, and thanks for all the work you've done on this one! | 17:30 |
| mestery | Got it. | 17:30 |
| salv-orlando | mestery: I just wish it was useful. Unfortunately I've spending 5 days on this without any result. | 17:31 |
| openstackgerrit | Sylvain Afchain proposed a change to openstack/neutron: L3 Metering label as shared https://review.openstack.org/70090 | 17:34 |
| *** jpich has quit IRC | 17:37 | |
| *** dave_tucker is now known as dave_tucker_zzz | 17:41 | |
| *** AlexF_ has quit IRC | 17:42 | |
| *** networkstatic has quit IRC | 17:44 | |
| mestery | salv-orlando: I'll noodle around and keep you posted here. | 17:44 |
| *** vkozhukalov has quit IRC | 17:44 | |
| salv-orlando | mestery: sure I will be around | 17:45 |
| *** armitage81 has joined #openstack-neutron | 17:47 | |
| *** armitage81 has quit IRC | 17:49 | |
| *** armitage81 has joined #openstack-neutron | 17:49 | |
| *** jistr has quit IRC | 17:53 | |
| *** clev has quit IRC | 17:54 | |
| *** emagana has quit IRC | 17:58 | |
| *** emagana has joined #openstack-neutron | 17:58 | |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/neutron: Fix passing keystone token to neutronclient instance https://review.openstack.org/70178 | 18:03 |
| openstackgerrit | Ihar Hrachyshka proposed a change to openstack/neutron: Remove psutil dependency https://review.openstack.org/68993 | 18:03 |
| *** bjornar has joined #openstack-neutron | 18:03 | |
| *** dave_tucker_zzz is now known as dave_tucker | 18:04 | |
| *** markwash has joined #openstack-neutron | 18:05 | |
| pcm_ | pcm | 18:05 |
| *** clev has joined #openstack-neutron | 18:10 | |
| *** sweston has quit IRC | 18:17 | |
| *** nati_ueno has joined #openstack-neutron | 18:17 | |
| *** rossella_s has quit IRC | 18:17 | |
| *** ktbenton has quit IRC | 18:22 | |
| *** julim has joined #openstack-neutron | 18:22 | |
| *** sn6i23a has joined #openstack-neutron | 18:23 | |
| *** networkstatic has joined #openstack-neutron | 18:24 | |
| *** sweston has joined #openstack-neutron | 18:24 | |
| *** jlibosva has quit IRC | 18:27 | |
| *** harlowja_away is now known as harlowja | 18:28 | |
| *** HenryG has quit IRC | 18:30 | |
| *** pcm has joined #openstack-neutron | 18:32 | |
| *** HenryG has joined #openstack-neutron | 18:33 | |
| *** aveiga has joined #openstack-neutron | 18:33 | |
| *** sweston has quit IRC | 18:34 | |
| *** sputnik13net has joined #openstack-neutron | 18:35 | |
| *** pcm_ has quit IRC | 18:35 | |
| *** thedodd has quit IRC | 18:37 | |
| *** safchain has quit IRC | 18:40 | |
| *** sbalukoff has quit IRC | 18:42 | |
| *** alex_klimov has joined #openstack-neutron | 18:44 | |
| marun | markmcclain: ping | 18:49 |
| markmcclain | marun: pong | 18:50 |
| marun | markmcclain: we have a volunteer to start work on migrating to oslo.messaging | 18:51 |
| marun | markmcclain: https://blueprints.launchpad.net/neutron/+spec/oslo-messaging | 18:51 |
| marun | markmcclain: is there anything the blueprint is missing or could you approve and target for icehouse-3? | 18:51 |
| *** jp_at_hp has quit IRC | 18:52 | |
| *** zzelle has joined #openstack-neutron | 18:53 | |
| *** clev has quit IRC | 18:55 | |
| markmcclain | marun: approved | 18:55 |
| marun | markmcclain: awesome | 18:55 |
| marun | markmcclain: also, see https://review.openstack.org/#/c/70178/1 | 18:55 |
| markmcclain | the blueprint is accurate | 18:56 |
| marun | markmcclain: unless I'm missing something, this is a pretty important catch | 18:56 |
| markmcclain | yeah will definitely cause some issues | 18:57 |
| marun | markmcclain: i'm afraid I haven't been doing triaging. would this bug be high or critical? | 18:58 |
| *** nati_uen_ has joined #openstack-neutron | 19:00 | |
| markmcclain | high | 19:00 |
| *** rkukura has quit IRC | 19:00 | |
| markmcclain | the code still works | 19:00 |
| *** nati_ueno has quit IRC | 19:03 | |
| *** aveiga has quit IRC | 19:03 | |
| marun | markmcclain: ah, so critical is 'broken' and high is 'works but suboptimally'? | 19:03 |
| *** nati_uen_ has quit IRC | 19:03 | |
| markmcclain | usually medium is works, but suboptimally | 19:04 |
| markmcclain | in this case I think it deserves a bump to high | 19:04 |
| markmcclain | because hammering keystone can kill some deployments | 19:04 |
| *** nati_ueno has joined #openstack-neutron | 19:04 | |
| marun | so what is high then? code works, but can break things under the right conditions? | 19:05 |
| markmcclain | basically the code breaks frequently or makes using the cloud tough | 19:05 |
| marun | markmcclain: gotcha | 19:05 |
| markmcclain | critical are items where things are basically always broken | 19:06 |
| markmcclain | either in real life or in the gate | 19:06 |
| marun | markmcclain: gotcha. | 19:06 |
| *** sputnik13net has quit IRC | 19:06 | |
| *** nati_ueno has quit IRC | 19:07 | |
| *** sn6i23a has quit IRC | 19:08 | |
| *** sputnik13net has joined #openstack-neutron | 19:09 | |
| salv-orlando | mestery: just in case you don't have yet a test script -> https://gist.github.com/salv-orlando/8715991 | 19:15 |
| *** rkukura has joined #openstack-neutron | 19:18 | |
| *** nati_ueno has joined #openstack-neutron | 19:19 | |
| *** nati_ueno has quit IRC | 19:19 | |
| *** vkozhukalov has joined #openstack-neutron | 19:19 | |
| *** armitage81 has quit IRC | 19:19 | |
| *** nati_ueno has joined #openstack-neutron | 19:20 | |
| *** pcm has quit IRC | 19:21 | |
| marun | salv-orlando: does the problem still occur on the updated kernel? | 19:21 |
| salv-orlando | marun: I'm afraid it does | 19:21 |
| marun | salv-orlando: :( | 19:22 |
| marun | salv-orlando: and does it reproduce consistently? | 19:23 |
| salv-orlando | nope. There are at least 4 people now trying to reproduce. | 19:23 |
| salv-orlando | So far I have not seen the crash happening outside of the gate. | 19:23 |
| marun | salv-orlando: nuts. well, i'll give it a go. | 19:24 |
| salv-orlando | marun: so that'll make 5 of us | 19:25 |
| *** carl_baldwin has quit IRC | 19:25 | |
| marun | salv-orlando: is the unix domain socket involved in this test? | 19:28 |
| marun | salv-orlando: nmind | 19:28 |
| *** sbalukoff has joined #openstack-neutron | 19:34 | |
| mestery | salv-orlando: Thanks! I almost have mine done too, looking at yours now. :) | 19:35 |
| *** Mierdin has joined #openstack-neutron | 19:36 | |
| openstackgerrit | Brian Haley proposed a change to openstack/neutron: Change metadata-agent to spawn multiple workers https://review.openstack.org/70204 | 19:44 |
| *** thuc has quit IRC | 19:46 | |
| *** thuc has joined #openstack-neutron | 19:46 | |
| *** thedodd has joined #openstack-neutron | 19:46 | |
| *** rohit404 has joined #openstack-neutron | 19:47 | |
| *** thuc_ has joined #openstack-neutron | 19:48 | |
| *** thuc has quit IRC | 19:48 | |
| *** thuc_ has quit IRC | 19:49 | |
| *** irenab has joined #openstack-neutron | 19:49 | |
| *** thuc has joined #openstack-neutron | 19:49 | |
| marun | is brian haley here? | 19:50 |
| *** pcm has joined #openstack-neutron | 20:02 | |
| *** aveiga has joined #openstack-neutron | 20:02 | |
| *** nati_ueno has quit IRC | 20:02 | |
| marun | jaypipes: ping | 20:04 |
| jaypipes | marun: pong | 20:06 |
| jaypipes | marun: how may I be of assistance, kind sir? | 20:06 |
| marun | jaypipes: i went ahead and responded on the review... | 20:07 |
| marun | jaypipes: there's a reason we don't run the agent in the namespace | 20:08 |
| *** vkozhukalov has quit IRC | 20:10 | |
| jaypipes | marun: so it's purely a security reason then? | 20:10 |
| marun | jaypipes: yes | 20:10 |
| jaypipes | marun: in what way could a tenant gain access to the network namespace? /me curious.... | 20:10 |
| marun | jaypipes: network traffic transits the namespace | 20:10 |
| *** RajeshMohan has quit IRC | 20:10 | |
| marun | jaypipes: so the namespace shouldn't have access to the management network | 20:11 |
| marun | jaypipes: arg | 20:11 |
| jaypipes | marun: right, but how would a tenant's network traffic enable the tenant to access the management network? sorry if I'm being thick :( | 20:11 |
| *** RajeshMohan has joined #openstack-neutron | 20:12 | |
| *** RajeshMohan has quit IRC | 20:14 | |
| * jaypipes must have asked a very stupid question :( | 20:15 | |
| *** nati_ueno has joined #openstack-neutron | 20:15 | |
| marun | jaypipes: sadly, on phone | 20:15 |
| marun | sorry :( | 20:15 |
| *** RajeshMohan has joined #openstack-neutron | 20:16 | |
| jaypipes | marun: lol, no worries at all mate. | 20:16 |
| marun | jaypipes: markmcclain is the person to talk to, I'm clearly a bit fuzzy on the details | 20:23 |
| sc68cal | Hey - has anyone tried to build a new tox environment lately? | 20:23 |
| sc68cal | I'm having trouble finding a match for psutil | 20:23 |
| sc68cal | http://paste.openstack.org/show/62196/ | 20:23 |
| markmcclain | sc68cal: that's expected | 20:24 |
| sc68cal | you know one of these days I'm not going to ask stupid questions | 20:24 |
| markmcclain | newer versions of pip require special flags to use packages that are not hosted by mypi | 20:24 |
| markmcclain | *pypi | 20:24 |
| markmcclain | otherwiseguy has made a fix to remove psutil from our deps | 20:25 |
| markmcclain | sc68cal: https://review.openstack.org/#/c/68993/ | 20:25 |
| markmcclain | we can't merge it until we get the kernel issues solved | 20:25 |
| sc68cal | weeee | 20:26 |
| sc68cal | :) | 20:26 |
| markmcclain | jaypipes: which process running in a namespace? | 20:26 |
| *** sn6i23a has joined #openstack-neutron | 20:27 | |
| jaypipes | markmcclain: see here: https://review.openstack.org/#/c/70204/ | 20:27 |
| markmcclain | right this is the generic proxy that handles request from multiple tenants | 20:28 |
| markmcclain | when we run in a namespace the agent does not have a device in the mgt network | 20:28 |
| markmcclain | s/in/plugged/ | 20:29 |
| jaypipes | markmcclain: understood. I was asking marun why there is a security concern about having the meta agent run in the namespaces... | 20:29 |
| markmcclain | the security concern is that any tenant can declare any IP range | 20:30 |
| markmcclain | to avoid issues with overlapping ranges and other fun they can have we run the agent outside of their space | 20:31 |
| jaypipes | k, thx! | 20:32 |
| markmcclain | also allows the deployer to run the nova metadata service on a network that is not reachable via the public network | 20:32 |
| markmcclain | I can considered enabling requests to exit out of the tenant network's external interface, but that wouldn't work in the case where a tenant does not have a router | 20:33 |
| markmcclain | our fall back for that scenario is the the dhcp agent can run the proxy | 20:33 |
| *** sputnik13net has quit IRC | 20:35 | |
| *** alex_klimov has quit IRC | 20:38 | |
| *** sweston has joined #openstack-neutron | 20:42 | |
| marun | salv-orlando: script is running, running, running… no failures yet. | 20:45 |
| salv-orlando | marun: same here | 20:50 |
| salv-orlando | markmcclain goes going to add some network traffic | 20:50 |
| salv-orlando | through the namespace | 20:50 |
| mestery | salv-orlando: I've got my script looping and creating namespaces, launching dnsmasq processes, waiting, and killing them, then deleting namespaces. | 21:01 |
| mestery | And doing it again | 21:01 |
| mestery | and again :) | 21:01 |
| mestery | No crashes so far. | 21:01 |
| mestery | Will try to add more punch to this thing in hopes of triggering something. | 21:01 |
| salv-orlando | same here | 21:01 |
| mestery | OK, at least we're consistent. :P | 21:01 |
| markmcclain | I'm wondering if there is something unique about the gate we're not replicating correctly | 21:01 |
| openstackgerrit | Brian Haley proposed a change to openstack/neutron: Change metadata-agent to have a configurable backlog https://review.openstack.org/70214 | 21:02 |
| mestery | markmcclain: I agree on that thinking. | 21:02 |
| salv-orlando | my script has created over 1,000 namespaces so far | 21:02 |
| salv-orlando | perhaps | 21:02 |
| salv-orlando | markmcclain perhaps | 21:02 |
| mestery | salv-orlando: I'm going to rock my script a bit more to add some rockets and grenades :P | 21:03 |
| markmcclain | nice | 21:03 |
| salv-orlando | right, I'll go for a different strategy, trying to get have the reproduce the failure being cunning and subtle ;) | 21:04 |
| openstackgerrit | lifeless proposed a change to openstack/python-neutronclient: Don't reuse 'body' for response data https://review.openstack.org/69110 | 21:05 |
| mestery | salv-orlando: :P | 21:05 |
| *** aburaschi has quit IRC | 21:06 | |
| *** HenryG has quit IRC | 21:06 | |
| *** aveiga has quit IRC | 21:06 | |
| *** changbl has quit IRC | 21:07 | |
| *** samuelbercovici has quit IRC | 21:13 | |
| *** irenab has quit IRC | 21:22 | |
| *** godara has joined #openstack-neutron | 21:23 | |
| *** markwash has quit IRC | 21:24 | |
| *** julim has quit IRC | 21:26 | |
| *** haleyb has quit IRC | 21:26 | |
| *** haleyb has joined #openstack-neutron | 21:27 | |
| mestery | salv-orlando: Bingo! | 21:27 |
| mestery | salv-orlando: I have a script which crashes my Ubuntu box. | 21:28 |
| salv-orlando | great | 21:28 |
| mestery | Now to enable crashdumps to verify it's the same thing. | 21:28 |
| salv-orlando | have you been creating vms as well? | 21:28 |
| * mestery knew it would be the rockets and grenades | 21:28 | |
| mestery | Nope | 21:28 |
| mestery | The combination appears to be creating a mapping of 1 OVS port to 1 dnsmasq process | 21:28 |
| mestery | Once I hit 1000, deleteing them caused the crash | 21:28 |
| sdague | hey, what's up with this? | 21:28 |
| sdague | 2014-01-30 21:16:00 /opt/stack/new/devstack/lib/neutron: line 428: /usr/local/bin/neutron-db-manage: No such file or directory | 21:28 |
| salv-orlando | sdague: does that happen consistently? sounds like a setup issue | 21:29 |
| sdague | it's failing a bunch of nova stuff in the gate right now | 21:29 |
| mestery | salv-orlando: Hold on a minute, the box hasn't crashed, it's just gotten incredibly slow, though it appears to be heading to it's own death. | 21:29 |
| salv-orlando | mestery: I feel like a kid who's being taken away his christmas present | 21:30 |
| salv-orlando | markmcclain: seem what sdague is reporting needs some priority | 21:30 |
| mestery | salv-orlando: Just wait, there may be something behind the chair in the corner. | 21:31 |
| salv-orlando | neutron-db-manage not found - there must be something failing in install | 21:31 |
| markmcclain | yeah.. setup must be failing | 21:31 |
| mestery | salv-orlando sdague: Since no changes have merged recently, wonder what's going on? | 21:31 |
| markmcclain | because that script is installed via console-scripts | 21:31 |
| markmcclain | sdague: ^^^ | 21:31 |
| sdague | 14-01-30 20:30:38.307 | 2014-01-30 20:30:38 Downloading/unpacking psutil>1.1.0 (from neutron==2014.1.dev3.gba12cc8) | 21:32 |
| sdague | 2014-01-30 20:30:38.311 | 2014-01-30 20:30:38 Could not find a version that satisfies the requirement psutil>1.1.0 (from neutron==2014.1.dev3.gba12cc8) (from versions: 0.6.1, 0.7.1) | 21:32 |
| markmcclain | sdague: we have a fix, but are stuck because nothing will pass the gate to due to kernel issues | 21:33 |
| markmcclain | sdague: https://review.openstack.org/#/c/68993/ | 21:33 |
| sdague | hmmmm | 21:34 |
| sdague | let me go figure out why were are tripping up on this now | 21:35 |
| markmcclain | that's one of the 2 testing related fixes we can't merge | 21:35 |
| *** pcm has quit IRC | 21:36 | |
| sdague | so the kernel bug isn't hitting every run | 21:38 |
| sdague | any idea what's different between them | 21:39 |
| sdague | like on that review, 3 out of 4 tempest runs worked | 21:39 |
| *** nati_uen_ has joined #openstack-neutron | 21:39 | |
| enikanorov_ | sdague: could it be that some jenkins slaves fail more often than others? | 21:39 |
| markmcclain | it's the isolated job that really induces it | 21:40 |
| sdague | markmcclain: that passed the isolated jobs | 21:40 |
| sdague | enikanorov: they are all single use | 21:40 |
| enikanorov_ | i see | 21:40 |
| mestery | sdague: There are theories we have, but nothing has been confirmed yet. | 21:40 |
| *** nati_ueno has quit IRC | 21:42 | |
| sdague | ok | 21:42 |
| *** sn6i23a has quit IRC | 21:43 | |
| *** sputnik13net has joined #openstack-neutron | 21:43 | |
| haleyb | ihrachys: maru said you and jakub had been looking at metadata-agent performance? i just posted two patches for this, both linked from https://bugs.launchpad.net/neutron/+bug/1274536 if you want to take a look. thanks. | 21:47 |
| *** markwash has joined #openstack-neutron | 21:50 | |
| *** IanGovett has quit IRC | 21:51 | |
| *** emagana has quit IRC | 21:53 | |
| *** Sukhdev has joined #openstack-neutron | 21:53 | |
| *** emagana has joined #openstack-neutron | 21:54 | |
| *** aveiga has joined #openstack-neutron | 21:58 | |
| marun | haleyb: they're in europe so likely asleep now | 21:59 |
| *** rohit404 has quit IRC | 21:59 | |
| marun | haleyb: one of the reasons the metadata agent is so slow is due to the bug fixed by this patch: https://review.openstack.org/#/c/70178/ | 22:00 |
| marun | haleyb: another is that the agent keeps contacting the neutron service via the api | 22:00 |
| haleyb | marun: thanks, i see you added one of them to the review anyways | 22:00 |
| haleyb | and with the two patches i proposed and the right tuning, the agent behaves much better | 22:01 |
| marun | haleyb: running multiple workers is likely to cause problems, to be honest | 22:01 |
| *** WackoRobie has quit IRC | 22:01 | |
| marun | haleyb: what kind of load have you seen on the neutron service when you run multiple agents? | 22:02 |
| *** jprovazn is now known as jprovazn_afk | 22:02 | |
| haleyb | marun: i disagree regarding multiple worker threads, we have run it with extremely high loads without a problem | 22:03 |
| marun | haleyb: with no adverse results to the keystone or neutron services? | 22:03 |
| haleyb | you do have to increase the nova side as well | 22:03 |
| *** simon-AS559 has joined #openstack-neutron | 22:04 | |
| haleyb | marun: the metadata-agent doesn't seem to be a problem at scale with those changes, it's certainly not consuming much more cpu time. without them there were many problems | 22:06 |
| marun | haleyb: I can't argue with your results, I'm just trying to point out that there is the potential for overload in other services when one service is scaled up without consideration. | 22:07 |
| *** simon-AS5591 has joined #openstack-neutron | 22:07 | |
| marun | haleyb: from my understanding, the neutron and keystone services would be impacted if the metadata agent was scaled out to handle more requests | 22:07 |
| haleyb | marun: yes, you also have to increase the number of nova worker threads, else the problem moves there | 22:07 |
| *** pheadron1 has joined #openstack-neutron | 22:08 | |
| marun | haleyb: so you haven't seen anything strange wrt keystone? | 22:08 |
| *** simon-AS559 has quit IRC | 22:09 | |
| haleyb | marun: not yet, no | 22:09 |
| marun | haleyb: unless that patch is applied, each agent request to the neutron service needs to be auth'd, rather than caching | 22:09 |
| marun | haleyb: the token for reuse | 22:09 |
| *** WackoRobie has joined #openstack-neutron | 22:09 | |
| marun | haleyb: also, the agent has to make multiple api calls to neutron where it would make more sense to use rpc and get it in a single request | 22:09 |
| *** bjornar has quit IRC | 22:09 | |
| jaypipes | marun: ++ on both counts. | 22:10 |
| marun | haleyb: lastly, the mapping of network/local ip to instance id could probably be cached for a limited amount of time… it's not likely (and probably should not) be changing during a cloud-init run | 22:10 |
| *** pheadron has quit IRC | 22:10 | |
| haleyb | marun: 70178 - yes that should help, and other changes could help too | 22:11 |
| *** pheadron1 is now known as pheadron | 22:11 | |
| *** nati_uen_ has quit IRC | 22:11 | |
| *** nati_ueno has joined #openstack-neutron | 22:12 | |
| marun | haleyb: For your use case simply scaling out might be sufficient. For a resource-constrained environment like the gate, though, I think we need to look at the root of the problem too. | 22:12 |
| marun | haleyb: I'm glad to have you attention on this, it's funny to think it's only been noticed now! | 22:13 |
| haleyb | marun: we noticed it earlier, i'm just slow at pushing this patch upstream :) | 22:14 |
| marun | haleyb: ah, makes sense. | 22:14 |
| haleyb | the way i see it, this isn't too different from the api server, which has both of those tuning knobs | 22:15 |
| *** sputnik13net has quit IRC | 22:15 | |
| marun | haleyb: it would be helpful to file a bug next time, even if you don't get a chance to submit a patch. | 22:15 |
| marun | haleyb: otherwise people go off and solve the same problem without at least checking if you've started in on it | 22:15 |
| haleyb | marun: we had talked to mark about it at the summit, but things got in the way | 22:16 |
| salv-orlando | woot | 22:16 |
| salv-orlando | hit the bug | 22:16 |
| *** peristeri has quit IRC | 22:16 | |
| marun | salv-orlando: do tell! | 22:16 |
| markmcclain | how many runs? | 22:16 |
| salv-orlando | disable FORCE_CONFIG_DRIVE=FALSE | 22:17 |
| salv-orlando | and you won't see it anymore | 22:17 |
| marun | ? | 22:17 |
| salv-orlando | it happens if you concurrently do nbd mount of the image AND ip netns operations | 22:17 |
| markmcclain | wow | 22:18 |
| markmcclain | sdague: what options do we have to workaround ^^^ | 22:18 |
| salv-orlando | basically we can't reproduce it locally because or devstack default to config drive | 22:18 |
| salv-orlando | while localrc on gate tests has FORCE_CONFIG_DRIVE=FALSE | 22:19 |
| salv-orlando | and btw, vm boot with config drive is way faster | 22:19 |
| anteaya | salv-orlando: do you want fungi to hold a test vm for you to confirm it? | 22:19 |
| salv-orlando | we should need a little change to openstack-infra/config | 22:19 |
| mestery | salv-orlando: Nice work man! | 22:19 |
| salv-orlando | anteaya: i have a crashed VM in front of me now | 22:19 |
| anteaya | yay | 22:19 |
| salv-orlando | with a nice crash dumpt | 22:19 |
| markmcclain | salv-orlando: so we need to turn off config drive in gate? | 22:19 |
| anteaya | awesome work | 22:19 |
| marun | wait, doesn't FORCE_CONFIG_DRIVE=FALSE turn off config drive? i.e. it's not being used in the gate? | 22:20 |
| *** networkstatic has quit IRC | 22:20 | |
| anteaya | marun: that is how I am reading it | 22:21 |
| *** sputnik13net has joined #openstack-neutron | 22:21 | |
| anteaya | so I am wondering if the fix is FORCE_CONFIG_DRIVE=TRUE | 22:21 |
| mestery | My reading is the same as yours anteaya | 22:21 |
| marun | but how would that cause a problem? | 22:22 |
| marun | and if we were to enable config drive, wouldn't that preclude testing of the metadata proxy/agent? | 22:22 |
| mestery | Spot on marun. :) | 22:22 |
| salv-orlando | the key is injected | 22:22 |
| *** aveiga has quit IRC | 22:22 | |
| *** networkstatic has joined #openstack-neutron | 22:22 | |
| salv-orlando | believe it or not, the key is injected in the gate | 22:22 |
| marun | injected by what? | 22:23 |
| marun | we've caught metadata proxy issues a couple of times due to gate failure | 22:23 |
| salv-orlando | by inject_data in nova/virt/disk/api.py | 22:23 |
| marun | i'm totally confused now | 22:23 |
| marun | so the gate is not currently testing the metadata agent/proxy? | 22:23 |
| sdague | salv-orlando: nice | 22:24 |
| sdague | markmcclain: this only shows up on pg jobs? | 22:24 |
| salv-orlando | sdague: can you confirm we are injecting the key during nova boot into the vm filesystem rather than retrieving it using metadata? | 22:24 |
| markmcclain | sdague: nearly all of the isolated jobs | 22:24 |
| sdague | right, but that's not the matrix issue | 22:25 |
| markmcclain | and most even the non-iso ones too | 22:25 |
| salv-orlando | btw, I now have 8 kernel failures | 22:25 |
| sdague | the issue is metadata server vs. config drive | 22:25 |
| markmcclain | sdague: so pg jobs run with key injection and the others don't? | 22:26 |
| marun | i apologize if it's clear to everyone else, but if the problem exhibits when config drive is used, and the gate is not using config drive, then ?? | 22:26 |
| salv-orlando | marun: i have not been clear | 22:26 |
| salv-orlando | the problem appears if you DO NOT use config drive, and therefore the key is injected via NBD mount | 22:27 |
| salv-orlando | when the failure happens | 22:27 |
| marun | salv-orlando: ah, ok | 22:27 |
| salv-orlando | metadata server is NOT being used | 22:27 |
| marun | salv-orlando: is that be design? | 22:27 |
| salv-orlando | sdague was probably saying that there was a matrix of jobs which use injection vs jobs which use metadata server | 22:27 |
| marun | salv-orlando: it was always my impression that we were testing the metadata service when we were logging into a vm with an ssh kye | 22:27 |
| sdague | ok, so we added a thing to turn off config drive on some jobs, which was the default, because it meant meta data server wasn't getting tested | 22:27 |
| sdague | and a break in the neutron meta data server hit tripleo | 22:28 |
| salv-orlando | sdague, but FORCE_CONFIG_DRIVE=false | 22:28 |
| *** simon-AS5591 has quit IRC | 22:28 | |
| *** simon-AS559 has joined #openstack-neutron | 22:28 | |
| salv-orlando | alone, triggers the nbd mounting for injection in nova if inject_key is not disabled as well | 22:28 |
| sdague | salv-orlando: yeh, there are a bunch of double negatives here, let me dig for a couple of links | 22:28 |
| marun | ah, so it's a configuration issue... | 22:29 |
| salv-orlando | well marun there is also a kernel issue underlying | 22:29 |
| salv-orlando | and I think our goal should be now to tweak configs so it does not appear anymore | 22:29 |
| marun | salv-orlando: at least for the neutron jobs, I can't think of a good reason to use key injection | 22:30 |
| *** IanGovett has joined #openstack-neutron | 22:30 | |
| sdague | marun: so I think there is an issue that if you don't use config drive, that's the only way to get the keys in the guests | 22:30 |
| marun | i'm not entirely sure that tempest should be testing key injection, anyway | 22:30 |
| marun | sounds more like a functional test for nova | 22:30 |
| sdague | marun: we need to log into guests | 22:31 |
| marun | sdague: uh | 22:31 |
| marun | sdague: i guess that's true if the guest isn't using the metadata service | 22:31 |
| sdague | marun: even if it is | 22:31 |
| marun | sdague: which doesn't have to be true for a guest in the gate, at least | 22:31 |
| marun | sdague: ?? | 22:31 |
| marun | sdague: doesn't cloud-init set the keys? | 22:31 |
| sdague | there might be something that needs to change | 22:31 |
| sdague | I don't know | 22:31 |
| sdague | I know existing behavior, if yuo turn off config drive, the keys get injected directly | 22:32 |
| sdague | but that's at a macro level | 22:32 |
| sdague | I don't know each part in how it goes wrong | 22:32 |
| marun | sdague: I'm pretty sure cloud-init and configdrive serve the same purpose, so I would be surprised if key injection is something that cloud init cannot do | 22:32 |
| *** gdubreui has joined #openstack-neutron | 22:32 | |
| salv-orlando | sdague: by directly do you mean mounting the instance drive? | 22:32 |
| sdague | marun: ok, well someone should trace that, because it was there for a reason | 22:32 |
| *** simon-AS559 has quit IRC | 22:33 | |
| marun | sdague: I think it's a 3rd path | 22:33 |
| marun | sdague: configdrive OR cloud-init OR injection | 22:33 |
| marun | sdague: all serve essentially the same purpose | 22:33 |
| sdague | marun: maybe, but our systems don't seem to function that way | 22:33 |
| marun | russellb: ping | 22:33 |
| sdague | anyway, someone should dive on that, let me go figure out the right vars to turn on config drive everywhere again | 22:33 |
| marun | sdague: I'll look into it | 22:34 |
| salv-orlando | sdague: I've taken a look at devstack-gate.yaml | 22:35 |
| salv-orlando | DEVSTACK_GATE_CONFIGDRIVE is always set to 0 | 22:35 |
| salv-orlando | sorry sdague, that was dumb me talking | 22:36 |
| salv-orlando | I forgot to consider that it default to 1 in devstack-gate | 22:36 |
| marun | I don't think it should be true for neutron jobs :( | 22:37 |
| marun | Don't we want to disable file injection and configdrive? | 22:37 |
| marun | I mean, at least run a job and see if it passes with that config? | 22:37 |
| salv-orlando | marun: I tend to agree with you, but I'm a bit scared of enabling key retrieval via metadata now. One can of worms at a time | 22:37 |
| marun | salv-orlando: ?! | 22:38 |
| salv-orlando | because this means we've actually never tests it :/ | 22:38 |
| sdague | salv-orlando: https://review.openstack.org/70235 | 22:38 |
| salv-orlando | and therefore we should assume it to be broken? | 22:38 |
| marun | salv-orlando: no, it means we haven't tested it recently, in the gate | 22:38 |
| sdague | marun: we've never tested it in the gate | 22:38 |
| marun | salv-orlando: but most developers are using it, because it's the default | 22:38 |
| marun | sdague: really? never? | 22:38 |
| sdague | there are 2 modes | 22:38 |
| sdague | from what I can see | 22:38 |
| marun | the question is - has that always been the case? | 22:39 |
| *** WackoRobie has quit IRC | 22:39 | |
| *** dims has quit IRC | 22:40 | |
| sdague | marun: at least for 6 months | 22:40 |
| marun | sdague: that makes more sense | 22:41 |
| sdague | but I have no reason to believe that it ever was the case | 22:41 |
| marun | sdague: i'm pretty sure I fixed a bug in May related to the metadata proxy | 22:41 |
| russellb | marun: pong | 22:41 |
| marun | russellb: apparently we're woefully ignorant of configdrive/cloud-init/file injection | 22:41 |
| marun | russellb: cloud init can set ssh keys right? | 22:41 |
| russellb | it can yes | 22:42 |
| marun | sdague: ^^ | 22:42 |
| russellb | sdague: i see a config drive failure in an older patch though | 22:42 |
| russellb | while looking through logs in the top bug, "waiting for thing to become ACTIVE", i saw some config drive failures in a job for https://review.openstack.org/#/c/68753/ | 22:43 |
| russellb | http://logs.openstack.org/53/68753/8/check/check-tempest-dsvm-postgres-full/5a631f2/logs/screen-n-cpu.txt.gz?level=TRACE | 22:43 |
| russellb | so you sure it's not enabled? | 22:43 |
| sdague | russellb: so we spread config drive across some jobs | 22:43 |
| russellb | ok, so on for some, off for some? | 22:43 |
| sdague | yes | 22:43 |
| russellb | k | 22:43 |
| sdague | https://review.openstack.org/#/c/70235/ | 22:43 |
| marun | russellb: is it possible to disable key injection and config drive? | 22:44 |
| marun | from the commit message on that review, it sounds like not | 22:44 |
| russellb | yes | 22:44 |
| marun | which would be surprising | 22:44 |
| russellb | well | 22:44 |
| russellb | you can have config drive off, file injection off, yes | 22:45 |
| russellb | and rely on cloud-init to get ssh keys in there | 22:45 |
| russellb | and it would do that by grabbing keys from the metadata server | 22:45 |
| sdague | so it might require devstack tweaks then | 22:45 |
| marun | (which is what I thought we were doing for at least the past year) | 22:45 |
| marun | what kind of tweaks? | 22:45 |
| sdague | marun: yeh, I don't think we were ever doing that | 22:45 |
| marun | oy | 22:45 |
| russellb | i consider cloud-init to be the most common deployment mode | 22:46 |
| marun | well, that would explain why the gate hasn't revealed that the metadata agent has serious performance issues :/ | 22:46 |
| russellb | cloud-init + metadata i mean | 22:46 |
| *** RajeshMohan has quit IRC | 22:46 | |
| russellb | sdague: so, for file injection, installing libguestfs will make nova not use nbd | 22:46 |
| russellb | libguestfs is preferred to nbd anyway | 22:46 |
| sdague | russellb: ok, so I think there are probably a series of cleanups out of this that need to happen. | 22:47 |
| russellb | perhaps | 22:47 |
| russellb | my brain is tired for today | 22:47 |
| sdague | yeh, same | 22:47 |
| *** RajeshMohan has joined #openstack-neutron | 22:47 | |
| russellb | surprised libguestfs isn't being installed already, bah | 22:47 |
| russellb | only on rpm distros right now | 22:48 |
| russellb | weak | 22:48 |
| lifeless | hi | 22:48 |
| lifeless | or we can disable file injection | 22:48 |
| lifeless | which I'm about to push the libvirt driver patch for | 22:49 |
| sdague | ok, I got to punt, hopefully enough folks here to sort out what is needed | 22:49 |
| marun | lifeless: I vote for disabling file injection | 22:49 |
| lifeless | I've put the config entry thats needed up in the review | 22:49 |
| marun | lifeless: and ensuring that the metadata path is actually tested | 22:49 |
| lifeless | marun: we're heavily dependent on it | 22:50 |
| russellb | sdague: https://review.openstack.org/70237 | 22:50 |
| *** godara has quit IRC | 22:50 | |
| marun | lifeless: so I hear. Glad that somebody is on it :) | 22:50 |
| russellb | sdague: thanks! | 22:51 |
| sdague | russellb: I'll check back later, if it passes tests I'll +A it in | 22:51 |
| russellb | sdague: awesome. yeah, it's going to run some new code paths .... | 22:52 |
| russellb | sdague: but should eliminate nbd | 22:52 |
| * russellb &\ | 22:52 | |
| *** jecarey has quit IRC | 22:55 | |
| *** dims has joined #openstack-neutron | 22:56 | |
| lifeless | marun: russellb: https://review.openstack.org/70239 | 22:56 |
| lifeless | I haven't checked for unit test fallout yet | 22:56 |
| *** nati_uen_ has joined #openstack-neutron | 23:00 | |
| *** nati_uen_ has quit IRC | 23:01 | |
| *** nati_uen_ has joined #openstack-neutron | 23:01 | |
| *** jprovazn_afk has quit IRC | 23:01 | |
| *** SumitNaiksatam has joined #openstack-neutron | 23:02 | |
| *** nati_ueno has quit IRC | 23:03 | |
| *** shivh has joined #openstack-neutron | 23:05 | |
| *** emagana has quit IRC | 23:06 | |
| *** emagana has joined #openstack-neutron | 23:07 | |
| *** emagana has quit IRC | 23:07 | |
| openstackgerrit | Mohammad Banikazemi proposed a change to openstack/neutron: Adds the new IBM SDN-VE plugin https://review.openstack.org/66453 | 23:09 |
| dkehn | salv-orlando: excellent | 23:13 |
| *** emagana has joined #openstack-neutron | 23:14 | |
| salv-orlando | I vote for whatever gets the neutron job working again! | 23:16 |
| *** jgrimm has quit IRC | 23:18 | |
| *** networkstatic has quit IRC | 23:19 | |
| markmcclain | +1000 | 23:19 |
| *** djoreilly has quit IRC | 23:19 | |
| openstackgerrit | Nachi Ueno proposed a change to openstack/neutron: API Extension for SSL-VPN services https://review.openstack.org/58897 | 23:25 |
| openstackgerrit | Nachi Ueno proposed a change to openstack/python-neutronclient: SSLVPN Client support https://review.openstack.org/70254 | 23:26 |
| *** IanGovett has quit IRC | 23:32 | |
| *** IanGovett has joined #openstack-neutron | 23:32 | |
| *** nati_uen_ has quit IRC | 23:32 | |
| *** godara has joined #openstack-neutron | 23:32 | |
| *** godara_ has joined #openstack-neutron | 23:34 | |
| salv-orlando | http://logs.openstack.org/37/70237/1/check/check-devstack-dsvm-neutron/01c37e0/logs/screen-n-cpu.txt.gz#_2014-01-30_23_16_25_346 | 23:34 |
| salv-orlando | russellb: ^^^^ from logs for patch 70237 | 23:35 |
| *** nati_ueno has joined #openstack-neutron | 23:36 | |
| *** godara has quit IRC | 23:37 | |
| *** godara_ is now known as godara | 23:37 | |
| sdague | salv-orlando: https://review.openstack.org/#/c/70237/ is failing all the tempest jobs | 23:49 |
| sdague | but it's good to know that guestfs is completely untested :) | 23:49 |
| salv-orlando | sdague: yes, it looks an easy fix at first glance, but I'm too tired now to attempt a nova patch. Perhaps russellb is still around. | 23:50 |
| sdague | yep | 23:50 |
| sdague | salv-orlando: as soon as the last test result comes in, I'm going to pull it from the gate | 23:51 |
| sdague | also, again, nice job on getting to the bottom of this | 23:52 |
| *** Sukhdev has quit IRC | 23:53 | |
| marun | so if fixing file injection isn't an option, can we look at merging lifeless's patch to simply disable it? | 23:54 |
| marun | nati_ueno: yikes | 23:55 |
| nati_ueno | marun: what's up? | 23:55 |
| marun | nati_ueno: saw the email re: arp | 23:55 |
| nati_ueno | marun: yeah, thanks | 23:56 |
| sdague | marun: that's not injection, that's guestfs | 23:56 |
| sdague | injection is a different patch | 23:56 |
| marun | sdague: i know, i meant that if libguestfs isn't an option at this time to safely enable file injection, can we consider disabling file injection and using the metadata agent? | 23:57 |
| marun | nati_ueno: so, question | 23:57 |
| openstackgerrit | Sean M. Collins proposed a change to openstack/neutron: Create new IPv6 attributes for Subnets https://review.openstack.org/52983 | 23:57 |
| marun | nati_ueno: would a viable 3rd option (at least for ovs) be adding a flow rule to drop arp traffic? | 23:57 |
| marun | sdague: sorry, ignore me. | 23:58 |
| marun | sdague: good > optimal in the short term as I so easily forget | 23:58 |
| nati_ueno | marun: we can only use it for ovs agent. and the implementation looks like same difficulties as option2 | 23:58 |
| marun | nati_ueno: *sigh* | 23:59 |
| * marun asks the internet what allowed address pair is | 23:59 | |
| nati_ueno | marun: That's the bp merged in havana | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!