| marun | salv-orlando: ping | 00:02 |
|---|---|---|
| salv-orlando | good morning marun | 00:02 |
| marun | salv-orlando: hiya :) | 00:02 |
| marun | salv-orlando: I am hoping to convince you to approve the dhcp patch | 00:02 |
| salv-orlando | the one I reviewed today? | 00:03 |
| salv-orlando | or yesterday for you | 00:03 |
| marun | salv-orlando: yes. | 00:03 |
| *** mlavalle has quit IRC | 00:03 | |
| marun | salv-orlando: on the first point, the patch continues to ensure that an agent that is marked as disabled (admin_state_up = False) does not receive notifications | 00:03 |
| dkehn | marun: do you know who or whom wrote the devstack/lib/floating_ips, I've got a few question | 00:04 |
| marun | salv-orlando: the changes are that notifications are sent to enabled agents regardless of whether they are active (as per timely heartbeat), and that an error is logged if a notification cannot be sent. | 00:04 |
| salv-orlando | yes, I was just wondering if that could be part of the db query (the admin_state filter) but is a minor point | 00:04 |
| *** HenryG has quit IRC | 00:05 | |
| *** otherwiseguy has quit IRC | 00:05 | |
| marun | salv-orlando: i aimed for a simple patch. the agent scheduler helper methods don't separate enabled from active and I didn't want to unfocus to cleanup. | 00:05 |
| salv-orlando | marun: got the change I was just wondering if the logic should be "send always to all enabled agents" or "send to active agents if any, otherwise to all enabled" | 00:05 |
| salv-orlando | seems you've implemented the second | 00:05 |
| marun | salv-orlando: ah, I get it | 00:06 |
| marun | dkehn: I think nati_ueno is the person to talk to. | 00:06 |
| salv-orlando | at first glance it looks simpler to always send to all enabled agents and warn if none if active | 00:06 |
| dkehn | marun: thx | 00:06 |
| nati_ueno | what's up? | 00:06 |
| salv-orlando | but I don't know if there's something I'm missing | 00:06 |
| marun | salv-orlando: No, that's a good call. | 00:07 |
| salv-orlando | k | 00:07 |
| marun | salv-orlando: There is no reason to avoid sending to inactive agents. | 00:07 |
| marun | salv-orlando: the only question is whether to log if none or active or some are inactive. | 00:07 |
| marun | salv-orlando: ? | 00:07 |
| dkehn | nati_ueno: we are timeing out on the https://github.com/openstack-dev/devstack/blob/master/exercises/floating_ips.sh#L143 | 00:07 |
| dkehn | nati_ueno: the instance is active, and pingable from just a general ping not via the netns | 00:08 |
| salv-orlando | I think you want to emit a warning if no agent is active. If there is at least one active agent I don't see a need to warn. Perhaps a log with debug level wit the number of active agents might help for debugging purposes. | 00:08 |
| dkehn | nati_ueno: this is just running the devstack locally | 00:08 |
| nati_ueno | dkehn: did you enable namespace in local? | 00:09 |
| marun | salv-orlando: I'm not sure we want still more debug logging :( | 00:09 |
| dkehn | nati_ueno: from the devstack format wouldn't that be in the floating_ips? | 00:09 |
| marun | salv-orlando: we really need a more granular logging system. too much noise in debug logging right now | 00:09 |
| nati_ueno | dkehn: Default is namespace enabled. | 00:10 |
| dkehn | nati_ueno: when I do a ip netns list that ns shows | 00:10 |
| nati_ueno | dkehn: it is wired if we can ping to the IP without namespace | 00:10 |
| marun | salv-orlando: as per your second comment, there has been a policy change wrt copyright notices - they are no longer required for empty files. | 00:10 |
| nati_ueno | s/wired/strange/ | 00:11 |
| marun | salv-orlando: http://docs.openstack.org/developer/hacking/#openstack-licensing | 00:11 |
| dkehn | nati_ueno: yes I can ping it directly "ping 10.1.0.4" returns ping respomnses | 00:11 |
| marun | salv-orlando: Now that I think about it, I'll file a bug for that. | 00:11 |
| *** dave_tucker is now known as dave_tucker_zzz | 00:11 | |
| nati_ueno | dkehn: The network is private. so It shouldn't be pinggable from outside | 00:12 |
| nati_ueno | dkehn: Could you try devstack with clean VM? | 00:12 |
| dkehn | nati_ueno: you mean from the host | 00:12 |
| nati_ueno | dkehn: Even if it is from host, it shouldn't be pinggable | 00:12 |
| dkehn | sure, but I've done that more times than I can count | 00:12 |
| dkehn | nati_ueno: ok | 00:13 |
| nati_ueno | dkehn: so, it get timeout on devstack on clean vm? | 00:13 |
| salv-orlando | marun: summarising are you going to push a new patch set for always notified all enabled agents? | 00:13 |
| dkehn | nati_ueno: yes | 00:13 |
| marun | salv-orlando: yes. | 00:13 |
| *** banix has joined #openstack-neutron | 00:13 | |
| nati_ueno | dkehn: Are we facing same issue in the gating? | 00:13 |
| salv-orlando | marun: cool. I am around for another hour and can +2 right away if needed. | 00:14 |
| dkehn | nati_ueno: check most of the grenade failures | 00:14 |
| marun | salv-orlando: ok, cool. | 00:14 |
| salv-orlando | otherwise, I'll be up again in 6 hours | 00:14 |
| dkehn | nati_ueno: upper | 00:14 |
| dkehn | nati_ueno: yepper | 00:14 |
| *** layer427expert has joined #openstack-neutron | 00:14 | |
| nati_ueno | dkehn: sounds like some degration | 00:14 |
| nati_ueno | dkehn: could you share the result of ovs-vsctl show ? | 00:15 |
| dkehn | nati_ueno: http://paste.openstack.org/show/55399/ | 00:15 |
| anteaya | dkehn: I don't see a devstack/lib/floating_ips file | 00:16 |
| dkehn | anteaya: https://github.com/openstack-dev/devstack/blob/master/exercises/floating_ips.sh | 00:16 |
| nati_ueno | dkehn: Thx. Could you also share neturon port-list -c id -c device_owner | 00:16 |
| anteaya | the exercises | 00:16 |
| anteaya | yes okay | 00:17 |
| *** dave_tucker_zzz is now known as dave_tucker | 00:18 | |
| *** layer427expert has joined #openstack-neutron | 00:18 | |
| dkehn | nati_ueno: http://paste.openstack.org/show/55400/ | 00:18 |
| anteaya | layer427expert: did you get your use of git review sorted out to your satisfaction? | 00:18 |
| nati_ueno | dkehn: Thanks. "neturon port-list -c id -c device_owner " <-- could you add -c id -c device_owner ? | 00:19 |
| dkehn | nati_ueno: sure, sorry | 00:19 |
| layer427expert | Kind of I did but I am still dealing with PEP8 issues.... Local testing is claiming things are good but remote does not | 00:19 |
| nati_ueno | dkehn: np | 00:19 |
| layer427expert | trying to sort that out but everything else is good. | 00:19 |
| anteaya | layer427expert: do you have a link to your patch? | 00:19 |
| layer427expert | Thanks! | 00:19 |
| anteaya | I can look at the pep8 | 00:20 |
| layer427expert | Yahttp://logs.openstack.org/64/62464/8/check/gate-neutron-pep8/9dff757/console.html | 00:20 |
| dkehn | nati_ueno: http://paste.openstack.org/show/55401/ | 00:20 |
| anteaya | thanks | 00:20 |
| layer427expert | https://review.openstack.org/#/c/62464/ | 00:20 |
| dkehn | nati_ueno: http://paste.openstack.org/show/55402/ just in case | 00:21 |
| nati_ueno | dkehn: Thanks. It looks like neutron debug command is broken. | 00:21 |
| dkehn | nati_ueno: what lead use to that conclusion? | 00:21 |
| nati_ueno | dkehn: The port Port "tap8d47266e-2d"'s vlan id is 4095 | 00:21 |
| anteaya | layer427expert: okay for starters https://review.openstack.org/#/c/62464/ is over 2000 lines in the patch | 00:21 |
| nati_ueno | dkehn: it should be 1 | 00:22 |
| salv-orlando | neutron core devs -> the patch https://review.openstack.org/#/c/58860/ was already approved by I blocked it because it failed on the gate queue for bug 1253896. I looked at logs to exclude any relation with such bug and performed several rechecks, which were all successfull. If you can approve again I reckon it's safe to merge. | 00:22 |
| anteaya | layer427expert: okay so this is a patch for https://blueprints.launchpad.net/neutron/+spec/a10networks-lbaas-driver | 00:23 |
| layer427expert | yesyes | 00:23 |
| dkehn | nati_ueno: in the devstack flow where does that get built up | 00:23 |
| anteaya | which you submitted, approved and assigned to yourself | 00:23 |
| anteaya | has anyone else reviewed this blueprint? | 00:23 |
| nati_ueno | dkehn: here https://github.com/openstack-dev/devstack/blob/master/lib/neutron#L847 | 00:23 |
| salv-orlando | layer427expert: perhaps your commit message refers only to changes in the last patch set? or do 2,000 lines just fix pep8 errors (being ironic here) | 00:24 |
| layer427expert | Not in the community... It was late I can assign it to who ever. | 00:24 |
| anteaya | and over 1000 lines in your other patch | 00:24 |
| anteaya | lets talk a bit | 00:24 |
| layer427expert | I reformated the files based on pep | 00:24 |
| layer427expert | However I think the commit file was not edited properly. | 00:24 |
| anteaya | we try to go with small changes to the code base | 00:25 |
| anteaya | 200 line of code per patch or less is optimal for a reviewer | 00:25 |
| dkehn | nati_ueno: so taking the Q_USE_DEBUG_COMMAND=True out of the localrc would stop that? | 00:25 |
| nati_ueno | dkehn: yes. | 00:25 |
| anteaya | unless there are rare circumstances | 00:26 |
| anteaya | you can make one patch dependant on another | 00:26 |
| layer427expert | anteaya:so I should pull the patches and then rebase then start from scratch? | 00:26 |
| layer427expert | abandon | 00:26 |
| layer427expert | is what I meant | 00:26 |
| dkehn | nati_ueno: hmm, good to know bu I'm pretty sure this is how tgrenade sets it up | 00:26 |
| anteaya | so that you can control the order the patches are merged | 00:26 |
| anteaya | layer427expert: no you don't need to abandon | 00:26 |
| anteaya | let's just talk a minute | 00:26 |
| anteaya | what is your motivation for the blueprint? | 00:26 |
| nati_ueno | dkehn: I'm new to granade, so could you share some docs on that? | 00:27 |
| anteaya | what do you want to see happen and how does that improve the project? | 00:27 |
| nati_ueno | dkehn: so where we get this error? | 00:27 |
| layer427expert | Will it enables our customer to utilize our platform with out a modification of Neutron. | 00:27 |
| anteaya | okay that's fair | 00:28 |
| dkehn | anteaya: so I'm basically running devstack-vm-gate.sh which runs the exercises.sh --> which runs the floating_ips.sh, this is basically what grenade testing does | 00:28 |
| anteaya | do you see anyone else using this code, or do you think this code is pretty unique to your own needs? | 00:28 |
| anteaya | dkehn: I was wondering if that is what it was | 00:28 |
| dkehn | nati_ueno: https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh | 00:28 |
| nati_ueno | dkehn: is this granade? | 00:29 |
| layer427expert | Well considering it is specifically for A10 then it will most likely be utilized by A10 Customers. | 00:29 |
| anteaya | and that is awesome, I have a patch up to turn on exercises (non-voting) for our master and havana patches: https://review.openstack.org/#/c/62930/4 | 00:29 |
| anteaya | dkehn: would be glad of a review if you have the time | 00:29 |
| anteaya | layer427expert: okay | 00:29 |
| anteaya | well let's look at it with that in mind | 00:30 |
| anteaya | salv-orlando: can you offer any suggestions | 00:30 |
| dkehn | nati_ueno: yes this is what grenade is it runs the for mentioned scripts with the https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L285 set | 00:30 |
| anteaya | does layer427expert's direction so far seem like he will reach his goal? | 00:30 |
| dkehn | anteaya: huh | 00:31 |
| dkehn | anteaya: I'm about ready to head off for dinner, or at least cooking it, if it can wait till afterwords | 00:31 |
| anteaya | dkehn: sure | 00:33 |
| dkehn | anteaya: nati_ueno that probably won;t work with https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L75 | 00:33 |
| salv-orlando | 2,595 lines diff won't make review any shorter. In the past we had similar discussion with driver/plugin implementors regarding splitting the patch in smaller chunks, but in most cases the answer was that it was very difficult or not feasible at all and in general led to patches which were not self-contained anyway. I don't know if this is layer247expert case as well. | 00:33 |
| anteaya | I would shudder at having to review 1000 lines of code in one sitting | 00:34 |
| dkehn | nati_ueno: let me try it without DEBUG on and then I'll localize around the DEBUG sound fair enough | 00:34 |
| dkehn | nati_ueno: or reasonable | 00:34 |
| layer427expert | What was implemented is the smallest ~amount of code that can be utilized to enable our devices and virtual appliances. | 00:34 |
| salv-orlando | I also have to check with markmcclain where we stand for 3rd party testing for drivers. I do not recall the policy now. | 00:34 |
| salv-orlando | layer427expert: I was kind of sure that was the answer. I guess then the core team has just to swallow it and start reviewing it. | 00:35 |
| anteaya | layer427expert: right, we are just discussing format for patches | 00:35 |
| nati_ueno | dkehn: let's me confirmed. It is not working with granade? so devstack without grande works? | 00:35 |
| layer427expert | salv-orlando: Mark said he want 3rd party testing to occur before review. That is why I was placing the patches in draft | 00:35 |
| layer427expert | Is there a better way? | 00:36 |
| anteaya | grenade doesn't work for neutron right now | 00:36 |
| nati_ueno | dkehn: It won't work without Debug command | 00:36 |
| anteaya | jlibovar is trying to get it working | 00:36 |
| dkehn | nati_ueno: I see the confusion, no I'm testing with jsut devstack and I'll post my lcoalrc | 00:36 |
| salv-orlando | work in progress rather than draft maybe. I think the draft makes the patch 'private' as well. | 00:36 |
| nati_ueno | dkehn: so there is no relation with granade for this issue? | 00:36 |
| *** dims_ is now known as dims | 00:37 | |
| anteaya | stay away from draft | 00:37 |
| anteaya | it creates so many problems later in the life of the patch | 00:37 |
| nati_ueno | dkehn: if you have a faild gating run with this issue, could you share the url? | 00:37 |
| dkehn | nati_ueno: ythere is a reationship in that this is hat grenade is going to do is run devstack.sh | 00:37 |
| anteaya | use work in progress | 00:37 |
| dkehn | nati_ueno: http://logs.openstack.org/63/61663/2/check/check-grenade-dsvm-neutron/5f49167/console.html.gz | 00:37 |
| dkehn | nati_ueno: basically any neutron jenkins run | 00:38 |
| dkehn | nati_ueno: http://logs.openstack.org/63/61663/2/check/check-grenade-dsvm-neutron/5f49167/ | 00:38 |
| nati_ueno | dkehn: check-tempest-dsvm-neutron looks success. but check-grenade-dsvm-neutron/ fails | 00:38 |
| anteaya | dkehn: yes | 00:38 |
| nati_ueno | dkehn: so this issue happens when we run granade + neutron | 00:39 |
| anteaya | that is why I am turning on exercises for neutron for master and havana | 00:39 |
| anteaya | so that this patch can be debugged: https://review.openstack.org/#/c/58695/ | 00:39 |
| anteaya | it is hitting the same error | 00:39 |
| dkehn | nati_ueno: yes, I devstack-vm-gate.sh basically to get my devstack setup up | 00:39 |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 import alphabetical order https://review.openstack.org/62464 | 00:39 |
| anteaya | exercises don't work with havana so they were turned off, but they need to work for grenade | 00:40 |
| dkehn | dkehn: goes for food | 00:43 |
| anteaya | jog0: did you say you were going to change the priority of this bug: https://bugs.launchpad.net/neutron/+bug/1244255 | 00:44 |
| jog0 | anteaya: I wasn't clear on that, I said that the priority could be changed, its not a gate issue at the moment | 00:45 |
| jog0 | but think high is appropriate being setting api workers to 4 shouldn't do this | 00:46 |
| anteaya | would you mind adding that as a comment? | 00:46 |
| nati_ueno | dkehn: This line should use neutron https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L76 | 00:47 |
| jog0 | anteaya: done | 00:47 |
| anteaya | jog0: thanks | 00:47 |
| anteaya | I think I'll sign off for the night | 00:47 |
| jog0 | o/ night | 00:48 |
| anteaya | layer427expert: you don't have to abandon your code | 00:48 |
| anteaya | you can make your patch smaller if that works and link patches as dependances | 00:49 |
| anteaya | you can unabandon anytime if you want | 00:49 |
| layer427expert | I I just realized that we can put it into a work in progress and that would | 00:49 |
| layer427expert | eliminate the review | 00:49 |
| anteaya | no it won't | 00:49 |
| layer427expert | until it is ready | 00:49 |
| anteaya | work in progress is a button | 00:49 |
| anteaya | just press it | 00:50 |
| anteaya | if you want it out of work on progress just submit a new patchset | 00:50 |
| anteaya | don't lose the history of comments on your work | 00:50 |
| layer427expert | So pressing the button will put it into a non-review state until it is ready to be reviewed? | 00:50 |
| anteaya | that just confuses people | 00:50 |
| anteaya | work in progress means that people can review it if they want | 00:51 |
| layer427expert | Ok | 00:51 |
| anteaya | but you are letting them know that you are still working on the patch, it isn't ready to be merged | 00:51 |
| layer427expert | ok that is the objective... | 00:51 |
| anteaya | you can reset the work in progress setting with each new patchset | 00:51 |
| *** krast has quit IRC | 00:51 | |
| layer427expert | Ok | 00:51 |
| layer427expert | Thanks | 00:51 |
| anteaya | yes, it just allows reviewers to make the most productive use of their time | 00:51 |
| anteaya | you are welcome | 00:52 |
| anteaya | glad you are here for the chat | 00:52 |
| anteaya | feel free to ask questions | 00:52 |
| anteaya | we will try to help as best we can | 00:52 |
| layer427expert | Thanks | 00:52 |
| anteaya | you are welcome | 00:52 |
| layer427expert | Appreciate it. | 00:52 |
| anteaya | my pleasure | 00:52 |
| anteaya | and stay away from draft | 00:53 |
| anteaya | it can cause so many issues later when you try to merge | 00:53 |
| anteaya | just use the work in progress button | 00:53 |
| anteaya | okay now signing off | 00:53 |
| anteaya | night and see you tomorrow | 00:53 |
| *** ashaikh has quit IRC | 00:55 | |
| openstackgerrit | Micheal Thompson proposed a change to openstack/neutron: Fixed PEP8 import alphabetical order https://review.openstack.org/62464 | 00:56 |
| *** otherwiseguy has joined #openstack-neutron | 00:56 | |
| *** safchain has joined #openstack-neutron | 00:59 | |
| *** safchain has quit IRC | 01:00 | |
| marun | ok, i am now pretty concerned at our use of mock | 01:02 |
| marun | there is no concept of passthrough, so mocking things like logging to check the call count can end up hiding errors (e.g. string substitution errors) | 01:04 |
| dkehn | back | 01:10 |
| dkehn | nati_ueno: you still around | 01:12 |
| nati_ueno | dkehn: yes | 01:12 |
| nati_ueno | dkehn: This line should use neutron https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L76 | 01:12 |
| dkehn | nati_ueno: not sure I understand your ^^^^^ remark | 01:12 |
| nati_ueno | dkehn: MY_ENABLED_SERVICES=$MY_ENABLED_SERVICES,quantum <-- adding quantum | 01:12 |
| dkehn | nati_ueno: instead of quantum | 01:12 |
| dkehn | nati_ueno: hmm, loads the neutron server and what not ok, is there something else dependent upon the ENABLED_SERVICES | 01:13 |
| nati_ueno | dkehn: https://github.com/openstack-dev/devstack/blob/master/lib/neutron#L130 | 01:14 |
| nati_ueno | dkehn: so some script's using is_service_enabled neutron | 01:14 |
| nati_ueno | dkehn: so neutron should be in the service list | 01:14 |
| nati_ueno | dkehn: it should be MY_ENABLED_SERVICES=$MY_ENABLED_SERVICES,neutron | 01:16 |
| dkehn | nati_ueno: let me amke a quick run at that | 01:16 |
| *** safchain has joined #openstack-neutron | 01:16 | |
| salv-orlando | neutron core devs -> the patch https://review.openstack.org/#/c/58860/ was already approved by I blocked it because it failed on the gate queue for bug 1253896. I looked at logs to exclude any relation with such bug and performed several rechecks, which were all successfull. If you can approve again I reckon it's safe to merge. | 01:16 |
| nati_ueno | dkehn: Thanks. may be I'm wrong, | 01:17 |
| *** suresh12 has quit IRC | 01:17 | |
| nati_ueno | dkehn: but it should worth try | 01:17 |
| dkehn | nati_ueno: let me try it , it would be wonderfull if that all it is | 01:17 |
| *** suresh12 has joined #openstack-neutron | 01:17 | |
| dkehn | nati_ueno: but at minumum there is a labelling issue to resolve | 01:18 |
| nati_ueno | dkehn: yeah, I agree | 01:18 |
| *** rwsu has quit IRC | 01:18 | |
| *** suresh12 has quit IRC | 01:18 | |
| *** safchain has quit IRC | 01:19 | |
| *** thansen has quit IRC | 01:19 | |
| *** suresh12 has joined #openstack-neutron | 01:19 | |
| openstackgerrit | Maru Newby proposed a change to openstack/neutron: Send DHCP notifications regardless of agent status https://review.openstack.org/61168 | 01:19 |
| *** rwsu has joined #openstack-neutron | 01:19 | |
| marun | nati_ueno: I added a blueprint as you requested for pm debug, feel free to +2 ;) https://review.openstack.org/#/c/43776/ | 01:21 |
| *** yfujioka has joined #openstack-neutron | 01:22 | |
| nati_ueno | marun: Thanks! +2ed | 01:23 |
| marun | nati_ueno: much appreciated | 01:23 |
| nati_ueno | marun: This patch is super cool :) | 01:23 |
| marun | nati_ueno: will be even cooler when I don't have to keep manually applying, heh. | 01:24 |
| *** thansen has joined #openstack-neutron | 01:24 | |
| nati_ueno | marun: ha ha true | 01:24 |
| *** pcm_ has quit IRC | 01:25 | |
| marun | anyone from radware here? | 01:26 |
| layer427expert | no but A10 :) | 01:28 |
| marun | markmcclain: Are there any guidelines for how 3rd party testing should interact with gerrit? Seeing what appears to be gerrit messages from 3rd party jobs makes me think letting jobs spew without restriction is going to be a problem. | 01:29 |
| *** Jianyong has joined #openstack-neutron | 01:32 | |
| openstackgerrit | Maru Newby proposed a change to openstack/neutron: Send DHCP notifications regardless of agent status https://review.openstack.org/61168 | 01:33 |
| dkehn | nati_ueno: pretty much the same outcomde | 01:36 |
| nati_ueno | dkehn: hmm sorry | 01:36 |
| dkehn | nati_ueno: timeout 90 sh -c 'while ! sudo /usr/local/bin/neutron-rootwrap /etc/neutron/rootwrap.conf ip netns exec qprobe-4cb65bcd-ff8\ | 01:36 |
| dkehn | 3-492c-830a-85cebff4670f ping -w 1 -c 1 10.1.0.4; do sleep 1; done' | 01:36 |
| *** dzyu has joined #openstack-neutron | 01:36 | |
| dkehn | nati_ueno: is what its failing on ,same place | 01:36 |
| dkehn | nati_ueno: will put together a past | 01:37 |
| dkehn | s/past/paste | 01:37 |
| nati_ueno | dkehn: thanks | 01:37 |
| *** yamahata has joined #openstack-neutron | 01:39 | |
| dkehn | nati_ueno: http://paste.openstack.org/show/55409/ | 01:40 |
| nati_ueno | dkehn: yeah, pretty much same | 01:41 |
| layer427expert | check-tempest-dsvm-neutron-pg FAILURE in 33m 02s | 01:44 |
| layer427expert | check-tempest-dsvm-neutron-isolated FAILURE in 36m 41s | 01:44 |
| layer427expert | check-tempest-dsvm-neutron-pg-isolated FAILURE in 31m 58s | 01:44 |
| layer427expert | I got an error from jenkins with failures on | 01:44 |
| layer427expert | the above | 01:44 |
| layer427expert | https://bugs.launchpad.net/bugs/1254890 | 01:45 |
| layer427expert | BadRequest: Unable to find security_group with name 'secgroup_general--tempest-1401508404' (HTTP 400) (Request-ID: req-5d1419ef-4045-40e5-be58-ddd1abce533f) | 01:45 |
| layer427expert | this was for check-tempest-dsvm-neutron-isolated | 01:46 |
| dkehn | nati_ueno: not sure if this helps but here's the setup_neutron_debug bash debug: http://paste.openstack.org/show/55410/ | 01:48 |
| nati_ueno | dkehn: could you mail me zip of /etc/neutron ? my mail address is nachi@ntti3.com | 01:49 |
| *** julim has quit IRC | 01:49 | |
| dkehn | nati_ueno: ok | 01:50 |
| *** harlowja has quit IRC | 01:51 | |
| openstackgerrit | Dazhao Yu proposed a change to openstack/neutron: Calculate stateless IPv6 address https://review.openstack.org/56184 | 01:55 |
| dkehn | nati_ueno: check your mail | 01:57 |
| nati_ueno | dkehn: Thanks! I got email | 01:58 |
| nati_ueno | dkehn: hmm config looks good.. | 02:00 |
| nati_ueno | dkehn: Which version the granade updates from ? | 02:01 |
| dkehn | thre is really no grenade involved in my test I'm pretty much using devstack.sh | 02:01 |
| dkehn | with devstack-vm-gate.sh , that is modified to drive it | 02:02 |
| dkehn | which is what grenade is using by by default | 02:02 |
| *** layer427expert has quit IRC | 02:02 | |
| nati_ueno | dkehn: hmm we should identify what's different with neutron gating without granade script and with granade script | 02:03 |
| dkehn | the reason why this is done this way is to minimize the effects of gate issue during development | 02:03 |
| *** harlowja has joined #openstack-neutron | 02:03 | |
| nati_ueno | dkehn: so how to run devstack.sh should be different | 02:03 |
| nati_ueno | dkehn: for instance localrc's | 02:04 |
| dkehn | its not differenet, but it deos run the exercises which devstack.sh does not | 02:04 |
| dkehn | sso the gate will use https://github.com/openstack-infra/devstack-gate with different sets of parameters is variouos phase of the gating process | 02:05 |
| *** clev has joined #openstack-neutron | 02:06 | |
| dkehn | during grenade testing as demonstrated from failed tests you see that exercises are run as part of that | 02:06 |
| *** layer427expert has joined #openstack-neutron | 02:11 | |
| *** networkstatic_zZ is now known as networkstatic | 02:24 | |
| *** clev has quit IRC | 02:25 | |
| nati_ueno | dkehn: so you are executing this line? https://github.com/openstack-infra/devstack-gate/blob/master/devstack-vm-gate.sh#L246 | 02:26 |
| nati_ueno | dkehn: running devstack-gate with DEVSTACK_GATE_GRENADE=1 | 02:27 |
| dkehn | nati_ueno: no I am not | 02:27 |
| nati_ueno | dkehn: so you are running devstack-gate with DEVSTACK_GATE_GRENADE=0 ? | 02:28 |
| dkehn | nati_ueno: that is correct | 02:28 |
| dkehn | nati_ueno: what I'm doing is basically running stack.sh and then the exercise.sh and thats about it | 02:29 |
| nati_ueno | dkehn: so what you are doing is same with check-tempest-dsvm-neutron ? | 02:29 |
| dkehn | nati_ueno: is there a dependancy upon the exercises and grenade? | 02:29 |
| dkehn | nati_ueno: I'm not that familiar with that but its done locally | 02:30 |
| *** krast has joined #openstack-neutron | 02:30 | |
| *** layer427expert has quit IRC | 02:30 | |
| nati_ueno | dkehn: check-tempest-dsvm-neutron runs the devstack-gate script without granade | 02:31 |
| nati_ueno | dkehn: it looks it is working in the gating env | 02:31 |
| nati_ueno | dkehn: this is an example http://logs.openstack.org/46/21946/26/check/check-tempest-dsvm-neutron/01a9c21/ | 02:31 |
| dkehn | nati_ueno: ok, then I gues what would be different | 02:31 |
| nati_ueno | dkehn: This is localrc http://logs.openstack.org/46/21946/26/check/check-tempest-dsvm-neutron/01a9c21/logs/localrc.txt.gz | 02:32 |
| nati_ueno | dkehn: could you share your localrc ? (may be I shared before..) | 02:32 |
| *** clev has joined #openstack-neutron | 02:33 | |
| dkehn | nati_ueno: the log I'm at the log http://logs.openstack.org/46/21946/26/check/check-tempest-dsvm-neutron/01a9c21/console.html does appear to be running exercise.sh | 02:34 |
| dkehn | nati_ueno: am I looking in the wrong place | 02:34 |
| dkehn | nati_ueno: will post it | 02:34 |
| nati_ueno | dkehn: Thanks | 02:35 |
| *** WackoRobie has joined #openstack-neutron | 02:35 | |
| nati_ueno | dkehn: check-grenade-dsvm-neutron is with granade | 02:35 |
| dkehn | nati_ueno: http://paste.openstack.org/show/55415/ | 02:36 |
| dkehn | nati_ueno: yes understand but I don't see the execution of the exercise.sh | 02:36 |
| nati_ueno | d, | 02:38 |
| nati_ueno | dkehn: Ah you are right.. I didn't know exercise.sh isn't running in check-tempest-dsvm-neutron | 02:39 |
| dkehn | nati_ueno: are you going to be around tomorrow? getting late here | 02:39 |
| nati_ueno | dkehn: sure | 02:40 |
| dkehn | nati_ueno: what tz are you in | 02:40 |
| nati_ueno | dkehn: pst | 02:40 |
| dkehn | nati_ueno: ok, I'm in MDT | 02:40 |
| dkehn | nati_ueno: close | 02:40 |
| nati_ueno | dkehn: oh you are hard worker :) | 02:40 |
| dkehn | nati_ueno: u too | 02:40 |
| nati_ueno | dkehn: not so late in PST now | 02:41 |
| dkehn | nati_ueno: thanks tons, really, will contact you tomorrow | 02:41 |
| nati_ueno | dkehn: sure TL! | 02:41 |
| dkehn | l8r | 02:41 |
| *** nati_ueno has quit IRC | 02:48 | |
| *** dave_tucker is now known as dave_tucker_zzz | 02:49 | |
| *** vkozhukalov has joined #openstack-neutron | 03:06 | |
| *** WackoRobie has quit IRC | 03:07 | |
| *** nati_ueno has joined #openstack-neutron | 03:35 | |
| *** WackoRob_ has joined #openstack-neutron | 03:38 | |
| *** WackoRob_ is now known as WackoRobie | 03:51 | |
| *** ashaikh has joined #openstack-neutron | 03:52 | |
| *** WackoRobie has quit IRC | 03:52 | |
| *** WackoRobie has joined #openstack-neutron | 03:52 | |
| *** julim has joined #openstack-neutron | 03:54 | |
| *** suresh12 has quit IRC | 03:57 | |
| *** julim has quit IRC | 03:57 | |
| openstackgerrit | Kevin Benton proposed a change to openstack/neutron: BigSwitch: Fixes floating IP backend updates https://review.openstack.org/63047 | 04:00 |
| *** nati_ueno has quit IRC | 04:01 | |
| *** WackoRobie has quit IRC | 04:04 | |
| *** WackoRobie has joined #openstack-neutron | 04:04 | |
| openstackgerrit | Kevin Benton proposed a change to openstack/neutron: BigSwitch: Fixes floating IP backend updates https://review.openstack.org/63047 | 04:06 |
| *** yamahata has quit IRC | 04:11 | |
| *** yamahata has joined #openstack-neutron | 04:11 | |
| *** yamahata__ has quit IRC | 04:12 | |
| *** yamahata__ has joined #openstack-neutron | 04:13 | |
| *** yamahata__ has quit IRC | 04:13 | |
| *** yamahata_ has joined #openstack-neutron | 04:13 | |
| *** banix has quit IRC | 04:28 | |
| *** banix has joined #openstack-neutron | 04:34 | |
| *** banix has quit IRC | 04:38 | |
| *** banix has joined #openstack-neutron | 04:46 | |
| *** nati_ueno has joined #openstack-neutron | 04:46 | |
| *** banix has quit IRC | 04:47 | |
| *** rohit404 has joined #openstack-neutron | 04:58 | |
| *** clev has quit IRC | 04:59 | |
| *** yfried has quit IRC | 05:00 | |
| *** chandankumar has joined #openstack-neutron | 05:02 | |
| *** SumitNaiksatam has quit IRC | 05:09 | |
| *** WackoRobie has quit IRC | 05:11 | |
| *** SumitNaiksatam has joined #openstack-neutron | 05:25 | |
| *** Jianyong has quit IRC | 05:33 | |
| *** harlowja is now known as harlowja_away | 05:40 | |
| *** WackoRobie has joined #openstack-neutron | 05:41 | |
| *** Jianyong has joined #openstack-neutron | 05:46 | |
| *** mlavalle has joined #openstack-neutron | 05:48 | |
| *** mlavalle has quit IRC | 05:48 | |
| *** WackoRobie has quit IRC | 05:50 | |
| *** bashok has joined #openstack-neutron | 05:52 | |
| *** suresh12 has joined #openstack-neutron | 05:52 | |
| *** otherwiseguy has quit IRC | 05:53 | |
| *** garyk has joined #openstack-neutron | 05:53 | |
| *** harlowja_away is now known as harlowja | 05:59 | |
| *** nati_ueno has quit IRC | 06:05 | |
| *** yfried has joined #openstack-neutron | 06:06 | |
| *** majopela has joined #openstack-neutron | 06:08 | |
| *** WackoRobie has joined #openstack-neutron | 06:16 | |
| *** jlibosva has joined #openstack-neutron | 06:18 | |
| *** WackoRobie has quit IRC | 06:22 | |
| *** zz_ajo is now known as ajo | 06:28 | |
| *** ajo is now known as zz_ajo | 06:29 | |
| *** zz_ajo is now known as ajo | 06:29 | |
| *** ajo is now known as zz_ajo | 06:33 | |
| *** nati_ueno has joined #openstack-neutron | 06:37 | |
| openstackgerrit | Jenkins proposed a change to openstack/neutron: Imported Translations from Transifex https://review.openstack.org/63056 | 06:37 |
| *** SushilKM has joined #openstack-neutron | 06:47 | |
| *** WackoRobie has joined #openstack-neutron | 06:48 | |
| *** Abhishek_ has joined #openstack-neutron | 06:48 | |
| *** amritanshu_RnD has joined #openstack-neutron | 06:52 | |
| *** vkozhukalov has quit IRC | 06:52 | |
| *** WackoRobie has quit IRC | 06:53 | |
| *** alex_klimov has joined #openstack-neutron | 06:54 | |
| *** SushilKM has quit IRC | 07:02 | |
| openstackgerrit | Maru Newby proposed a change to openstack/neutron: Send DHCP notifications regardless of agent status https://review.openstack.org/61168 | 07:08 |
| *** ashaikh has quit IRC | 07:11 | |
| *** Abhishek_ has quit IRC | 07:13 | |
| *** irenab_ has joined #openstack-neutron | 07:14 | |
| *** WackoRobie has joined #openstack-neutron | 07:18 | |
| *** WackoRobie has quit IRC | 07:23 | |
| *** suresh12 has quit IRC | 07:29 | |
| *** SushilKM has joined #openstack-neutron | 07:39 | |
| *** fouxm has joined #openstack-neutron | 07:49 | |
| *** nati_ueno has quit IRC | 07:54 | |
| *** SushilKM__ has joined #openstack-neutron | 08:03 | |
| *** SushilKM has quit IRC | 08:04 | |
| *** harlowja is now known as harlowja_away | 08:04 | |
| openstackgerrit | berlin proposed a change to openstack/neutron: add router_id to response for CRU on fw/vip objs https://review.openstack.org/59129 | 08:12 |
| openstackgerrit | Sridar Kandaswamy proposed a change to openstack/neutron: Remove FWaaS Noop driver as default and move to unit tests dir https://review.openstack.org/63066 | 08:19 |
| kruskakli | anteaya: thanks, it is a bit late for me, but if I'm awake I'll try to join | 08:22 |
| *** Mierdin has quit IRC | 08:24 | |
| openstackgerrit | jun xie proposed a change to openstack/neutron: Change from eager load to lazy load https://review.openstack.org/61027 | 08:27 |
| *** vkozhukalov has joined #openstack-neutron | 08:29 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Process port_update notifications in the main agent loop https://review.openstack.org/61964 | 08:39 |
| *** djoreilly has joined #openstack-neutron | 08:45 | |
| openstackgerrit | Simon Pasquier proposed a change to openstack/neutron: neutron-rootwrap-xen-dom0 handles data from stdin https://review.openstack.org/62346 | 08:51 |
| *** enikanorov has quit IRC | 08:53 | |
| *** marun has quit IRC | 08:53 | |
| *** rohit404 has quit IRC | 08:53 | |
| *** WackoRobie has joined #openstack-neutron | 08:57 | |
| *** jroovers has joined #openstack-neutron | 08:57 | |
| *** jroovers has quit IRC | 09:01 | |
| *** ygbo has joined #openstack-neutron | 09:02 | |
| *** jroovers has joined #openstack-neutron | 09:06 | |
| *** safchain has joined #openstack-neutron | 09:06 | |
| *** yfujioka has quit IRC | 09:07 | |
| openstackgerrit | Akihiro Motoki proposed a change to openstack/neutron: Use base.BaseTestCase in NVP config test https://review.openstack.org/63074 | 09:08 |
| *** yongli is now known as yongli_away | 09:08 | |
| *** rohit404 has joined #openstack-neutron | 09:09 | |
| *** amuller has joined #openstack-neutron | 09:09 | |
| *** gongysh has joined #openstack-neutron | 09:10 | |
| gongysh | salv-orlando: ping | 09:10 |
| gongysh | amotoki: ping | 09:13 |
| amotoki | gongysh: hi | 09:15 |
| gongysh | amotoki: I found that neutron net-list cannot list the external networks now. | 09:15 |
| gongysh | amotoki: I mean with very non-admin user. | 09:16 |
| gongysh | amotoki: If it is the case, how can I create a router with gateway interface? | 09:17 |
| amotoki | gongysh: I am now searching bugs. I saw a related bug to this. | 09:17 |
| amotoki | gongysh: in that case non-admin fails to set external gateway. | 09:17 |
| gongysh | amotoki: I remember that should work before. I mean non-admin could list external networks before. | 09:19 |
| amotoki | gongysh: according to my experience we can see ext_net first but at some timing ext_net disappears from net-list. | 09:20 |
| gongysh | amotoki: will u file a bug or send it on maillist? | 09:23 |
| amotoki | gongysh: https://bugs.launchpad.net/neutron/+bug/1251982 | 09:24 |
| amotoki | gongysh: in addition https://bugs.launchpad.net/neutron/+bug/1254555 may be related. | 09:24 |
| amotoki | gongysh: are you seeing this issue now? The priority was changed to low a few weeks ago. If you see it frequently please update the bug priority. | 09:27 |
| *** jpich has joined #openstack-neutron | 09:27 | |
| amotoki | gongysh: i haven't experienced this recently. | 09:27 |
| gongysh | amotoki: ok, I will have an explore | 09:27 |
| gongysh | amotoki: thanks | 09:27 |
| amotoki | gongysh: it is really strange behavior.... | 09:28 |
| jroovers | hi enikanorov__ | 09:36 |
| *** jroovers has quit IRC | 09:38 | |
| *** jorisroovers has joined #openstack-neutron | 09:38 | |
| *** rohit404 has joined #openstack-neutron | 09:41 | |
| salv-orlando | gongysh: how can I help you? | 09:42 |
| gongysh | salv-orlando: <amotoki> gongysh: hi | 09:47 |
| gongysh | <gongysh> amotoki: I found that neutron net-list cannot list the external networks now. | 09:47 |
| gongysh | <gongysh> amotoki: I mean with very non-admin user. | 09:47 |
| gongysh | <gongysh> amotoki: If it is the case, how can I create a router with gateway interface? | 09:47 |
| gongysh | <amotoki> gongysh: I am now searching bugs. I saw a related bug to this. | 09:47 |
| gongysh | <amotoki> gongysh: in that case non-admin fails to set external gateway. | 09:47 |
| gongysh | <gongysh> amotoki: I remember that should work before. I mean non-admin could list external networks before. | 09:47 |
| gongysh | <amotoki> gongysh: according to my experience we can see ext_net first but at some timing ext_net disappears from net-list. | 09:47 |
| gongysh | <gongysh> amotoki: will u file a bug or send it on maillist? | 09:47 |
| gongysh | <amotoki> gongysh: https://bugs.launchpad.net/neutron/+bug/1251982 | 09:47 |
| gongysh | <amotoki> gongysh: in addition https://bugs.launchpad.net/neutron/+bug/1254555 may be related. | 09:47 |
| gongysh | <amotoki> gongysh: are you seeing this issue now? The priority was changed to low a few weeks ago. If you see it frequently please update the bug priority. | 09:47 |
| gongysh | * jpich (~jpich@217.173.99.61) has joined #openstack-neutron | 09:48 |
| gongysh | <amotoki> gongysh: i haven't experienced this recently. | 09:48 |
| gongysh | <gongysh> amotoki: ok, I will have an explore | 09:48 |
| gongysh | <gongysh> amotoki: thanks | 09:48 |
| gongysh | <amotoki> gongysh: it is really strange behavior.... | 09:48 |
| salv-orlando | I think enikanorov was working on this. The reason is that context.is_admin uses a policy to validate what it means to be an admin. To do so it loads the policy engine. The policy engine has also a rule for allowing everyone to see external networks. This rule needs to evaluate the router:external field and verify it matches to True (as a boolean value); to do so it uses the converter which is specified in the RESOURCE_ATTRIBUTE_MAP. Ther | 09:51 |
| salv-orlando | are some cases where this happens before all the extensions are loaded, and this would lead to skipping the conversion, with the result that the policy evaluation fails. | 09:51 |
| salv-orlando | This happens usually when a plugin does db operations on initialization | 09:52 |
| salv-orlando | there are several ways to fix it, and one would be doing the conversion at every evaluation, which is a bit expensive but perhaps negligible. | 09:52 |
| salv-orlando | I don't know if enikanorov is still actively working on this bug. You can ask him. | 09:53 |
| salv-orlando | gongysh^ | 09:53 |
| gongysh | salv-orlando: ok | 09:53 |
| gongysh | enikanorov_ ping | 09:54 |
| *** rohit404 has quit IRC | 09:56 | |
| salv-orlando | neutron core devs: please have a look at https://review.openstack.org/#/c/58860/, needs to be re-approved only. It failed the gate and we verified the reason was not the patch itself. We also did a few rechecks to be sre. | 09:56 |
| *** rohit404 has joined #openstack-neutron | 09:59 | |
| *** yfried is now known as yfried_lunch | 10:01 | |
| *** enikanorov has joined #openstack-neutron | 10:08 | |
| gongysh | amotoki: ping | 10:18 |
| *** jp_at_hp has joined #openstack-neutron | 10:20 | |
| *** dzyu_ has joined #openstack-neutron | 10:25 | |
| *** dzyu has quit IRC | 10:26 | |
| *** dzyu_ is now known as dzyu | 10:26 | |
| *** gongysh has quit IRC | 10:27 | |
| *** networkstatic has quit IRC | 10:29 | |
| *** dzyu has quit IRC | 10:36 | |
| *** rossella_s has joined #openstack-neutron | 10:38 | |
| *** gizmoguy has quit IRC | 10:39 | |
| *** gizmoguy has joined #openstack-neutron | 10:41 | |
| *** Jianyong has left #openstack-neutron | 10:50 | |
| *** krast has quit IRC | 10:51 | |
| openstackgerrit | berlin proposed a change to openstack/neutron: Add session persistence support for NVP advanced LBaaS https://review.openstack.org/59146 | 10:56 |
| *** Abhishek_ has joined #openstack-neutron | 10:57 | |
| *** SushilKM__ has quit IRC | 10:59 | |
| *** rohit404 has quit IRC | 11:18 | |
| openstackgerrit | Isaku Yamahata proposed a change to openstack/neutron: Implement service vm framework(WIP): https://review.openstack.org/56892 | 11:20 |
| *** djoreilly has quit IRC | 11:24 | |
| *** yamahata has quit IRC | 11:28 | |
| *** pcm_ has joined #openstack-neutron | 11:28 | |
| *** pcm_ has quit IRC | 11:29 | |
| *** pcm_ has joined #openstack-neutron | 11:30 | |
| *** yfried_lunch is now known as yfried | 11:30 | |
| *** SushilKM__ has joined #openstack-neutron | 11:34 | |
| *** rohit404 has joined #openstack-neutron | 11:40 | |
| *** pcm_ has quit IRC | 11:46 | |
| *** rkukura has quit IRC | 11:48 | |
| *** WackoRobie has quit IRC | 11:50 | |
| *** rpodolyaka has joined #openstack-neutron | 12:02 | |
| *** bashok has quit IRC | 12:06 | |
| *** yamahata has joined #openstack-neutron | 12:14 | |
| *** alex_klimov has quit IRC | 12:15 | |
| *** alex_klimov has joined #openstack-neutron | 12:16 | |
| *** jorisroovers has quit IRC | 12:20 | |
| *** WackoRobie has joined #openstack-neutron | 12:20 | |
| *** jroovers has joined #openstack-neutron | 12:21 | |
| *** rpodolyaka has left #openstack-neutron | 12:28 | |
| *** jroovers has quit IRC | 12:31 | |
| *** jroovers has joined #openstack-neutron | 12:31 | |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Improve handling of security group updates https://review.openstack.org/63100 | 12:41 |
| *** SushilKM__ has quit IRC | 12:51 | |
| *** pcm_ has joined #openstack-neutron | 12:51 | |
| *** HenryG has joined #openstack-neutron | 13:00 | |
| *** clev has joined #openstack-neutron | 13:00 | |
| *** rossella_s has quit IRC | 13:14 | |
| *** aymenfrikha has joined #openstack-neutron | 13:26 | |
| *** Abhishek_ has quit IRC | 13:29 | |
| *** heyongli has joined #openstack-neutron | 13:31 | |
| *** aveiga has joined #openstack-neutron | 13:35 | |
| *** SushilKM__ has joined #openstack-neutron | 13:36 | |
| *** rossella_s has joined #openstack-neutron | 13:38 | |
| *** chandankumar_ has joined #openstack-neutron | 13:44 | |
| *** chandankumar_ has quit IRC | 13:44 | |
| openstackgerrit | Akihiro Motoki proposed a change to openstack/neutron: Return request-id in API response https://review.openstack.org/58270 | 13:49 |
| *** WackoRobie has quit IRC | 13:53 | |
| *** WackoRobie has joined #openstack-neutron | 13:53 | |
| *** ygbo has quit IRC | 13:53 | |
| *** heyongli has quit IRC | 13:54 | |
| *** ygbo has joined #openstack-neutron | 13:55 | |
| *** markwash has joined #openstack-neutron | 13:58 | |
| *** yamahata has quit IRC | 13:58 | |
| *** yamahata has joined #openstack-neutron | 13:59 | |
| *** irenab_ has quit IRC | 14:01 | |
| *** jpich has quit IRC | 14:02 | |
| *** rkukura has joined #openstack-neutron | 14:04 | |
| yfried | pasquier-s: posted a comment on bug https://bugs.launchpad.net/tempest/+bug/1262613 | 14:05 |
| yfried | pasquier-s: this is due to tenant_isolation issue. I already pushed a fix for this | 14:06 |
| *** chandankumar has quit IRC | 14:06 | |
| *** peristeri has joined #openstack-neutron | 14:25 | |
| *** aymenfrikha has quit IRC | 14:28 | |
| *** julim has joined #openstack-neutron | 14:30 | |
| *** markmcclain has quit IRC | 14:30 | |
| *** jecarey has quit IRC | 14:31 | |
| *** mattymo has quit IRC | 14:32 | |
| *** mattymo has joined #openstack-neutron | 14:32 | |
| *** rkukura has quit IRC | 14:33 | |
| *** rkukura has joined #openstack-neutron | 14:34 | |
| *** banix has joined #openstack-neutron | 14:39 | |
| *** WackoRobie has quit IRC | 14:44 | |
| *** openstack has joined #openstack-neutron | 14:49 | |
| anteaya | kruskakli: thanks for considering it | 14:49 |
| *** amritanshu_RnD has quit IRC | 14:50 | |
| *** yfried has quit IRC | 14:50 | |
| dkehn | anteaya: didn't get around to reviewing https://review.openstack.org/#/c/62930/4, I see sdague's comment , which is the best way to handle the non-voting? | 14:51 |
| anteaya | marun: the third party jobs that are running elsewhere return a verified or unverified and a +1/-1 vote with logs to output from the test run | 14:53 |
| anteaya | marun: this gives the reviewers of the patch information and the freedom to merge or not merge as they see fit | 14:53 |
| anteaya | marun: the important part perhaps is in the naming of the test providing the comment so that folks can follow up to find out more if they have questions | 14:54 |
| anteaya | marun: if it becomes a problem of too much information then we can have a conversation when it happens | 14:55 |
| anteaya | marun: jeblair made the point that he has never heard a dev say "my patch has been tested too throughly" | 14:55 |
| *** jecarey has joined #openstack-neutron | 14:55 | |
| anteaya | dkehn: thanks, I haven't seen that yet, just getting rolling, I will amend | 14:56 |
| dkehn | anteaya: no prob, me too on the rolling part | 14:56 |
| dkehn | anteaya: when you look at it , just curious which is the better way to handle it | 14:57 |
| anteaya | dkehn: I have no personal opinion and if sdague only wants it running on check, I shall offer a patch that conveys such | 14:57 |
| anteaya | I just need to get jlibosva some information so he can keep working, check info is better than what we have now | 14:58 |
| dkehn | anteaya: thx, I'm figuring the are so many way to skin the cat, so to speak, but the cat must be skinned | 14:58 |
| anteaya | the cat must be skinned, aye | 14:58 |
| dkehn | anteaya: something worth raising with fungi | 14:58 |
| *** markmcclain has joined #openstack-neutron | 15:01 | |
| *** julim has quit IRC | 15:03 | |
| *** clev has quit IRC | 15:04 | |
| *** carl_baldwin has joined #openstack-neutron | 15:04 | |
| anteaya | dkehn: nah, he will see the patch and my money is he will agree with sean and upvote it | 15:04 |
| anteaya | I just want it merged so we can get some logs on it | 15:05 |
| EmilienM | rkukura: hi, i face a problem with security group api when using ml2, could I ask you to have a look at https://bugs.launchpad.net/puppet-neutron/+bug/1262678 please ? | 15:06 |
| *** julim has joined #openstack-neutron | 15:07 | |
| pasquier-s | yfried, thanks for the feedback | 15:09 |
| rkukura | EmilienM: I'll get back to you momentarily. | 15:10 |
| EmilienM | rkukura: thank you | 15:11 |
| *** mestery_ has joined #openstack-neutron | 15:12 | |
| openstackgerrit | Carl Baldwin proposed a change to openstack/neutron: Remove release_lease from the DHCP driver interface https://review.openstack.org/56285 | 15:12 |
| *** WackoRobie has joined #openstack-neutron | 15:15 | |
| *** djschaap has joined #openstack-neutron | 15:15 | |
| EmilienM | enikanorov: could you give me an update about SSL termination in LBaaS ? I'm not sure you spoke about that during the meeting today | 15:16 |
| *** mestery has quit IRC | 15:16 | |
| *** otherwiseguy has joined #openstack-neutron | 15:16 | |
| enikanorov | EmilienM: we did. evgenyf is working on the proposal and is going to post the code soon | 15:16 |
| *** ashaikh has joined #openstack-neutron | 15:17 | |
| markmcclain | enikanorov: evgenyf and I were just discussing it | 15:17 |
| markmcclain | I'm hoping he'll jump in here shortly | 15:17 |
| enikanorov | cool | 15:18 |
| *** evgenyf has joined #openstack-neutron | 15:18 | |
| markmcclain | so the problem we've got to tackle is how we protect the certs and passphrases | 15:19 |
| evgenyf | Actually we need protection for private key only, or am I mistaken? | 15:19 |
| markmcclain | right private key | 15:20 |
| * markmcclain needs more coffee | 15:20 | |
| EmilienM | enikanorov: thank you for the update | 15:20 |
| *** mestery has joined #openstack-neutron | 15:22 | |
| rkukura | EmilienM: I've been recommending setting firewall_driver=dummy_value_to_enable_security_groups_in_server in ml2_conf.ini, as long as the agent has the real firewall driver configured in an agent-specific file. | 15:22 |
| rkukura | See http://openstack.redhat.com/Modular_Layer_2_%28ML2%29_Plugin | 15:22 |
| EmilienM | rkukura: thank you, I have to update openstack manuals and puppet-neutron. I can see some people missing these informations. | 15:24 |
| rkukura | EmilienM: I consider it a bug that the server-side of the security group RPC code uses the firewall_driver config variable to determine whether the API is enabled. I thought I had filed a bug, but couldn't find it. | 15:24 |
| *** aymenfrikha has joined #openstack-neutron | 15:24 | |
| *** mestery_ has quit IRC | 15:25 | |
| *** aymenfrikha has quit IRC | 15:27 | |
| *** aymenfrikha1 has joined #openstack-neutron | 15:27 | |
| EmilienM | rkukura: so I guess I have to configure both ml2 & OVS config files with firewall_driver (ml2 is for enable it and ovs with right driver) right ? | 15:30 |
| rkukura | EmilienM: I think so. Not sure about other distros, but with RDO and RHOS, the init scripts for the L2 agents give them their own config files, not ml2_conf.ini. | 15:32 |
| rkukura | EmilienM: So ovs_neutron_plugin.ini should have the real firewall driver, and ml2_conf.ini should have a dummy value to enable the API. | 15:34 |
| EmilienM | rkukura: far enough, thank you a lot Robert, I'm going to update documentation and puppet stuffs. | 15:35 |
| *** aymenfrikha1 has quit IRC | 15:35 | |
| *** alex_klimov has quit IRC | 15:36 | |
| EmilienM | and also in default config file i think. | 15:36 |
| EmilienM | rkukura: is it useful ? ^ | 15:36 |
| rkukura | EmilienM: I don't think firewall_driver needs to be set in neutron.conf, if that's what you mean. | 15:37 |
| EmilienM | rkukura: no i was wondering about ovs config file | 15:38 |
| *** vkozhukalov has quit IRC | 15:40 | |
| rkukura | EmilienM: It does need to be set in ovs_neutron_plugin.ini if security groups are being used. Beware that there is work going on to implement an ovs-firewall-driver that uses flow rules rather than iptables rules, so in icehouse there may be a choice of which driver to use for openvswitch-agent. | 15:41 |
| markmcclain | evgenyf: so pickup the convo.. we need to make sure that the private keys are secured at all times | 15:43 |
| markmcclain | that included when the keys are in transit between processes via RPC | 15:44 |
| markmcclain | one of my concerns in that the infrastructure we'll end up creating will overlap with the services of barbican | 15:45 |
| markmcclain | I'd rather contribute the resources necessary to barbican ready for incubation vs creating our own infrastructure | 15:45 |
| evgenyf | I'm not familiar with barbican, is it including RPC security? | 15:46 |
| EmilienM | rkukura: ack. thx again for precious informations. | 15:47 |
| rkukura | EmilienM: No problem. Let me know if you've got any other questions, and I'll keep an eye out for the reviews. | 15:48 |
| markmcclain | evgenyf: no barbican is a secure store | 15:48 |
| EmilienM | rkukura: awesome, I appreciate your help | 15:48 |
| markmcclain | currently it is an ecosystem project that is trying for incubation | 15:48 |
| markmcclain | the team is working on harmonizing the tech stack with the rest of OpenStack and growing their team | 15:48 |
| markmcclain | barbican can provide a secure store | 15:49 |
| evgenyf | markmcclain:Yes, but still, with a key stored in a secure store, sending it to teh back-end system should be secured to and this is a second problem | 15:50 |
| *** aymenfrikha has joined #openstack-neutron | 15:52 | |
| markmcclain | well at that point in time the reference can passed | 15:54 |
| markmcclain | and the backend driver can properly retrieve and prepare the request for the appliance | 15:54 |
| *** rossella_s has quit IRC | 15:54 | |
| evgenyf | What do you mean by "back-end driver"? | 15:57 |
| markmcclain | the agent | 16:00 |
| *** julim has quit IRC | 16:01 | |
| markmcclain | enikanorov: any update on https://bugs.launchpad.net/neutron/+bug/1254890 | 16:02 |
| evgenyf | Ok. And this is for the case when secure store is in place. And what about the case then key is not stored in secure place, but is trancient. Is there a way to make the driver-agent communication secured? | 16:02 |
| *** julim has joined #openstack-neutron | 16:02 | |
| markmcclain | I've been seeing this outside of the large ops tests | 16:02 |
| markmcclain | http://logstash.openstack.org/#eyJzZWFyY2giOiJcIlRpbWVkIG91dCB3YWl0aW5nIGZvciB0aGluZ1wiIiwiZmllbGRzIjpbXSwib2Zmc2V0IjowLCJ0aW1lZnJhbWUiOiIxNzI4MDAiLCJncmFwaG1vZGUiOiJjb3VudCIsInRpbWUiOnsidXNlcl9pbnRlcnZhbCI6MH0sInN0YW1wIjoxMzg3NDY4ODg5MTUxfQ== | 16:02 |
| markmcclain | 85 hits in last 48hrs out of 256 for the last week | 16:02 |
| *** SushilKM__ has quit IRC | 16:05 | |
| markmcclain | evgenyf: right now oslo messaging does not support secure messages | 16:06 |
| markmcclain | they're working on it | 16:06 |
| markmcclain | so we'd have to contribute to that effort | 16:07 |
| evgenyf | markmcclain: ic, anyhow the feature should wait for secured messaging or secured store, right? | 16:08 |
| markmcclain | I think so | 16:08 |
| markmcclain | but I also think this is an area where we can work with those projects | 16:09 |
| markmcclain | and help further the development vs waiting | 16:09 |
| markmcclain | also having a real problem to solve is a good use case of the projects | 16:10 |
| evgenyf | sure, I have to get familiar with those projects, thanks Mark | 16:11 |
| *** amuller has quit IRC | 16:17 | |
| *** Mierdin has joined #openstack-neutron | 16:17 | |
| *** SushilKM__ has joined #openstack-neutron | 16:18 | |
| evgenyf | markmcclain: Have a nice holidays | 16:26 |
| *** dims has quit IRC | 16:26 | |
| markmcclain | evgenyf: thanks! | 16:26 |
| markmcclain | evgenyf: if you got any questions feel free to ping the mailing list | 16:26 |
| *** dims has joined #openstack-neutron | 16:30 | |
| *** HenryG_ has joined #openstack-neutron | 16:38 | |
| *** HenryG has quit IRC | 16:41 | |
| *** garyk1 has joined #openstack-neutron | 16:45 | |
| *** evgenyf has quit IRC | 16:45 | |
| *** dims_ has joined #openstack-neutron | 16:46 | |
| *** mlavalle has joined #openstack-neutron | 16:47 | |
| mlavalle | mtreinish: ping | 16:48 |
| enikanorov | markmcclain: not yet. I'm digging in logs. checked few failed test runs so far. Didn't find any specific suspicious errors on nova side. I think I'll have to add more logging to nova | 16:51 |
| markmcclain | ok | 16:51 |
| *** dims has quit IRC | 16:52 | |
| *** mattymo has quit IRC | 16:52 | |
| *** peristeri has quit IRC | 16:52 | |
| *** garyk has quit IRC | 16:52 | |
| *** amotoki has quit IRC | 16:52 | |
| *** garyk1 has quit IRC | 16:55 | |
| *** mattymo has joined #openstack-neutron | 16:55 | |
| *** amotoki has joined #openstack-neutron | 16:55 | |
| *** SushilKM__ has quit IRC | 16:56 | |
| *** mattymo has quit IRC | 16:58 | |
| *** amotoki has quit IRC | 16:58 | |
| *** SushilKM__ has joined #openstack-neutron | 17:02 | |
| *** thedodd has joined #openstack-neutron | 17:03 | |
| *** peristeri has joined #openstack-neutron | 17:04 | |
| *** mattymo has joined #openstack-neutron | 17:04 | |
| *** amotoki has joined #openstack-neutron | 17:04 | |
| *** jorisroovers has joined #openstack-neutron | 17:06 | |
| *** clev has joined #openstack-neutron | 17:07 | |
| *** jroovers has quit IRC | 17:09 | |
| *** peristeri has quit IRC | 17:09 | |
| *** mattymo has quit IRC | 17:09 | |
| *** amotoki has quit IRC | 17:09 | |
| *** rohit404 has quit IRC | 17:11 | |
| *** amotoki has joined #openstack-neutron | 17:12 | |
| *** networkstatic has joined #openstack-neutron | 17:13 | |
| *** mattymo has joined #openstack-neutron | 17:15 | |
| *** jp_at_hp has quit IRC | 17:15 | |
| *** WackoRobie has quit IRC | 17:16 | |
| *** markmcclain has quit IRC | 17:17 | |
| anteaya | <- afk for probably the remainder of the day | 17:19 |
| *** peristeri has joined #openstack-neutron | 17:24 | |
| *** mlavalle has quit IRC | 17:25 | |
| *** marun has joined #openstack-neutron | 17:30 | |
| *** mlavalle has joined #openstack-neutron | 17:35 | |
| *** ygbo has quit IRC | 17:36 | |
| *** SushilKM__ has left #openstack-neutron | 17:39 | |
| *** vkozhukalov has joined #openstack-neutron | 17:46 | |
| *** dims_ is now known as dims | 17:50 | |
| marun | salv-orlando: ping | 18:02 |
| *** garyk has joined #openstack-neutron | 18:03 | |
| salv-orlando | hi marun | 18:03 |
| marun | hi salvatore | 18:03 |
| marun | I've been thinking about how to guarantee (eventual) state consistency between the dhcp agent and db... | 18:03 |
| *** jorisroovers has quit IRC | 18:03 | |
| *** jlibosva1 has joined #openstack-neutron | 18:04 | |
| marun | salv-orlando: For the case of a single rpc server, using time-based sequencing might be ok. | 18:04 |
| marun | salv-orlando: I'm less sure this scales, though. Having a global sequence id (monotonically increasing, not necessarily contiguous) might be necessary. | 18:04 |
| *** harlowja_away is now known as harlowja | 18:04 | |
| *** jlibosva2 has joined #openstack-neutron | 18:05 | |
| marun | salv-orlando: have you given this any thought? I'm assuming the other agents have a similar requirement | 18:05 |
| openstackgerrit | Carl Baldwin proposed a change to openstack/neutron: Use information from the dnsmasq hosts file to call dhcp_release https://review.openstack.org/56263 | 18:05 |
| ijw | marun: I was looking at your discussions there | 18:05 |
| *** jlibosva has quit IRC | 18:06 | |
| marun | ijw: and? | 18:06 |
| ijw | Seems like it wants a multicast for updates and, for emergencies like a restart, a call back to request a complete update. The 'it's dead' spotting right now just doesn't seem to serve a purpose, because (a) we don't take corrective action and (b) | 18:06 |
| ijw | (b) we actually make things worse... | 18:07 |
| *** jlibosva2 has quit IRC | 18:07 | |
| *** jlibosva1 has quit IRC | 18:08 | |
| ijw | The question I have (and this might be because I don't understand the semantics of AMQP) is, why do we stop sending messages when the DHCP server isn't responding? | 18:08 |
| ijw | I'm used to protocols where you send a message, it gets lost, and you fix it when you notice. | 18:08 |
| marun | ijw: what do you mean by 'stop sending messages'? | 18:11 |
| marun | ijw: and what do you mean by DHCP server? Are you talking about the DHCP agent? dnsmasq? or...? | 18:11 |
| *** insanida1e is now known as insanidade | 18:13 | |
| marun | ijw: the larger issue I'm grappling with is how to ensure | 18:14 |
| marun | ijw: that full sync and incremental sync (triggered by notifications) don't step on one another | 18:14 |
| marun | ijw: right now there are no guarantees of even eventual consistency, we're just hoping things work out | 18:15 |
| *** networkstatic has quit IRC | 18:17 | |
| *** evgenyf has joined #openstack-neutron | 18:20 | |
| *** markmcclain has joined #openstack-neutron | 18:22 | |
| *** suresh12 has joined #openstack-neutron | 18:23 | |
| marun | markmcclain: ping | 18:27 |
| *** yfried has joined #openstack-neutron | 18:28 | |
| markmcclain | marun: pong | 18:28 |
| marun | markmcclain: Hi Mark | 18:28 |
| *** fouxm has quit IRC | 18:28 | |
| markmcclain | hey.. still in Asia or in US again? | 18:28 |
| marun | back in the us at last | 18:29 |
| marun | quell la jet-lag! | 18:29 |
| *** fouxm has joined #openstack-neutron | 18:29 | |
| markmcclain | jet lag isn't fun | 18:29 |
| marun | israel was bad enough, but asia? | 18:29 |
| *** fouxm has quit IRC | 18:30 | |
| marun | i end up sleeping only 3-4 hours/night and then crashing in the afternoon | 18:30 |
| markmcclain | yeah I've fallen into that pattern before | 18:30 |
| marun | markmcclain: You travel way more than me, and after this time I don't think I can envy you anymore. :) | 18:32 |
| *** SumitNaiksatam has quit IRC | 18:32 | |
| marun | markmcclain: so, a couple of questions... | 18:32 |
| marun | markmcclain: i'm seeing 3rd party jobs start to come online and spew review comments | 18:32 |
| enikanorov | yeah... | 18:33 |
| marun | markmcclain: i'm thinking this is going to be a real problem - automated comments obscuring peoples' comments - once we have all the 3rd party jobs running | 18:33 |
| marun | markmcclain: Tempest reviews are starting to see the same thing | 18:33 |
| marun | markmcclain: I don't think there's any easy solution, but I would suggest that cleaner gerrit integration is going to be necessary. | 18:34 |
| marun | markmcclain: that's all, I'll raise the same question in infra. | 18:34 |
| marun | markmcclain: the other thing, working with the dhcp agent has resulted in me being concerned that we can't guarantee even eventual consistency at present | 18:35 |
| marun | markmcclain: I'm not sure the other agents could be different | 18:35 |
| marun | markmcclain: at least with the dhcp agent, there is no way to determine if a notification is stale after a full sync has occured. | 18:35 |
| markmcclain | marun: 2 mins.. got pulled into a local conf | 18:36 |
| marun | markmcclain: np | 18:36 |
| markmcclain | marun: def agree that 3rd party testing integration is going to be something that work out as more systems come online | 18:38 |
| markmcclain | our project is the first to really add it a large scale | 18:39 |
| markmcclain | I'm happy to coordinate with -infra if we need to modify anything | 18:39 |
| markmcclain | marun: as for DHCP | 18:40 |
| markmcclain | I think we have to assume that the notification is valid and when in doubt do a full sync | 18:40 |
| markmcclain | even if it is horribly stale | 18:40 |
| marun | markmcclain: there is no guarantee that we don't get out of sync | 18:40 |
| markmcclain | the other option is to coalesce all of the pending messages for a queue | 18:40 |
| marun | markmcclain: again, not useful | 18:40 |
| markmcclain | and say if we get X delta updates just convert to a full sync | 18:41 |
| markmcclain | that way data isn't lost and we can process faster | 18:41 |
| marun | there is no way at present to prevent stale notifications from introducing inconsistency in the local cache | 18:41 |
| markmcclain | why is converting teh deltas not useful? | 18:41 |
| marun | because if we get notifications during sync, we have no way to know whether they are stale | 18:42 |
| markmcclain | we could move to the model where the notifications are interpreted as signals to update that cache in a more consistent way | 18:42 |
| markmcclain | the messages have timestamps | 18:42 |
| markmcclain | so where the systems are using ntp | 18:42 |
| marun | markmcclain: that almost works | 18:43 |
| markmcclain | we have the ability to determine the age of a message | 18:43 |
| marun | markmcclain: but the timestamps occur outside of transaction boundaries | 18:43 |
| markmcclain | if is an update we have the delta we have the ability to compare with the cache | 18:43 |
| markmcclain | if it agrees we can ignore the message right? | 18:43 |
| marun | markmcclain: that isn't an error case, though | 18:44 |
| marun | markmcclain: it's if they differ - who wins? | 18:44 |
| markmcclain | if they differ the tie goes to the db | 18:44 |
| markmcclain | and we init a full sync of that net | 18:44 |
| marun | markmcclain: that doesn't help | 18:44 |
| marun | *sigh* | 18:44 |
| markmcclain | why not.. the db is the owner of the record right? | 18:44 |
| markmcclain | I don't know of another source of truth | 18:44 |
| *** clev has quit IRC | 18:45 | |
| marun | the db is the source of truth, yes | 18:45 |
| marun | it's how we access it that's the problem. | 18:45 |
| marun | when we do a sync, we get read isolation | 18:45 |
| markmcclain | right | 18:45 |
| marun | during that transaction, changes can happen | 18:45 |
| markmcclain | right | 18:46 |
| marun | we need to know which happened before (stale) and after (not stale) the read transaction started | 18:46 |
| markmcclain | but those changes should generate a new notification | 18:46 |
| markmcclain | which should get processed after we've read the db | 18:46 |
| markmcclain | in a normal env | 18:46 |
| markmcclain | we'll finish the sync | 18:46 |
| markmcclain | the process that notificaiton | 18:46 |
| markmcclain | which will update the cache via applying a delta or full sync | 18:46 |
| markmcclain | it's when stuff goes wrong that you're concerned right? | 18:47 |
| marun | so we should get close if we make sure the timestamp is as close as possible to the start of the read transaction of the sync, and that notifications are as close as possible to the end of the write transaction | 18:47 |
| markmcclain | ie lost notification | 18:47 |
| marun | yes, when things go wrong | 18:47 |
| *** rohit404 has joined #openstack-neutron | 18:47 | |
| marun | not just lost notifications, though | 18:48 |
| markmcclain | ok that was the original intent of the periodic full resync | 18:48 |
| marun | i don't think we should need periodic full resync | 18:48 |
| markmcclain | to ensure to proactively rebuild | 18:48 |
| marun | we should resync when we detect a problem | 18:48 |
| marun | (which is what we actually do right now anyway) | 18:48 |
| markmcclain | short of building a true presence tracking you ahve to have it | 18:48 |
| marun | we don't have it | 18:49 |
| *** evgenyf has quit IRC | 18:49 | |
| marun | the resync only happens when something goes wrong | 18:49 |
| markmcclain | yeah we do fall back when we're out of sync | 18:49 |
| markmcclain | but it's not perfect | 18:49 |
| marun | we're just not doing a great job of detecting errors | 18:49 |
| markmcclain | which is why there's the full resync janitor thread | 18:49 |
| marun | all it does is check whether a resync is required | 18:49 |
| marun | but anyway. | 18:49 |
| markmcclain | oh the resync thread just used to set that flag to true | 18:50 |
| markmcclain | must have been lost | 18:50 |
| marun | it's a mess right now | 18:50 |
| markmcclain | the other approach is to further integrate DHCP resolution with out db | 18:50 |
| marun | i'm working on moving to a single event loop so that notification processing and resync is coordinated | 18:50 |
| markmcclain | but dnsmasq does not allow us to do that | 18:50 |
| markmcclain | we used to have a single loop | 18:50 |
| marun | hmm, without db? | 18:50 |
| markmcclain | s/out/our/ | 18:51 |
| marun | markmcclain: yeah, i don't know what happened. your original intent has been lost, I think. | 18:51 |
| markmcclain | so we did single loop, but for largish networks that actually caused a backlog | 18:51 |
| marun | (re: single loop) | 18:51 |
| markmcclain | talk with arosen on the perf problems they encountered | 18:52 |
| marun | markmcclain: no reason not to spin greenthreads per-network | 18:52 |
| markmcclain | I know they ran into same bad issues with it | 18:52 |
| *** evgenyf has joined #openstack-neutron | 18:52 | |
| marun | but right now it's per-event | 18:52 |
| marun | event -> notification | 18:52 |
| markmcclain | yeah | 18:52 |
| markmcclain | think having a thread per network is interesting | 18:52 |
| markmcclain | might be a good balance | 18:52 |
| marun | i think that could work | 18:53 |
| marun | so, I can work on timestamp-based discard. | 18:53 |
| markmcclain | ok | 18:53 |
| * markmcclain brb | 18:53 | |
| marun | np | 18:53 |
| *** SumitNaiksatam has joined #openstack-neutron | 18:54 | |
| *** banix has quit IRC | 18:58 | |
| *** mlavalle has quit IRC | 19:01 | |
| *** markwash has quit IRC | 19:01 | |
| *** banix has joined #openstack-neutron | 19:03 | |
| *** evgenyf has quit IRC | 19:05 | |
| *** jroovers has joined #openstack-neutron | 19:09 | |
| markmcclain | marun: sorry back | 19:16 |
| ijw | marun: hey | 19:18 |
| ijw | Sorry, I was trying to suggest an approach like BGP - incremental updates to DHCP agents (and an understanding that actually one network may want multiple DHCP agents for redundancy) from the server, full resync if DHCP agent resets or if it sees a message go astray | 19:19 |
| ijw | But I'm talking based on what I've seen go past in discussion, I haven't read the code (yet, but it's on my v6 joblist) | 19:20 |
| lifeless | ijw: so neutron already has support for redundant active-active DHCP agents ;) | 19:22 |
| ijw | lifeless: I wasn't sure if it was in, I did think it was anticipated at least, I know we've talked about it | 19:23 |
| *** WackoRobie has joined #openstack-neutron | 19:31 | |
| *** rossella_s has joined #openstack-neutron | 19:34 | |
| enikanorov | markmcclain: i have an update for bug 1254890 | 19:35 |
| markmcclain | enikanorov: awesome.. what's the update? | 19:36 |
| rossella_s | beagles: ping | 19:36 |
| enikanorov | apparently in each failed test run an instance got stuck with the task of mounting the fs | 19:36 |
| enikanorov | i don't see it related to neutron anyhow. but my experience there is very limited | 19:36 |
| beagles | rossella_s: hi.. 1s on phone | 19:36 |
| enikanorov | if not to say none | 19:36 |
| rossella_s | beagles: NP | 19:37 |
| enikanorov | all of network operations seem to succeed for the instance being spawned. but 'mount' cmd never returns | 19:37 |
| markmcclain | good to know this information | 19:38 |
| markmcclain | can update hte bug with your findings? | 19:38 |
| enikanorov | yes, I was going to add it. i'm currently trying to catch the failure with few additional log lines that i've added to nova. | 19:39 |
| *** otherwiseguy has quit IRC | 19:40 | |
| markmcclain | ok.. also looks like dims added some stuff to the report too | 19:40 |
| enikanorov | indeed | 19:44 |
| markmcclain | enikanorov: I'm going to mark the bug as invalid for Neutron | 19:45 |
| markmcclain | but do add your info so that the others working it will have your findings | 19:45 |
| enikanorov | I see DIMS has added pretty verbose explanation. I have nothing to add except that i confirm his findings. | 19:48 |
| *** HenryG_ has quit IRC | 19:49 | |
| markmcclain | taht's useful data | 19:49 |
| markmcclain | thank you for digging into this | 19:49 |
| dims | enikanorov, thanks for confirming | 19:49 |
| marun | markmcclain: ping | 19:53 |
| markmcclain | marun: pong | 19:53 |
| marun | so, potential problem with time-based discard. | 19:53 |
| marun | i think there might be a race between getting the timestamp after commit of a write transaction (that sends a notification) and getting the timestamp before initiating a read transaction (for a sync). | 19:54 |
| marun | even if the transactions are aligned such that the write transaction is committed before the read transaction, the timestamps could suggest that the write transaction didn't happen until after the read transaction began | 19:56 |
| marun | markmcclain: does that make sense? | 19:57 |
| markmcclain | it does | 19:58 |
| markmcclain | but in that case the we would read again would we not? | 19:58 |
| marun | markmcclain: why? | 19:58 |
| marun | markmcclain: i think the result would be processing a stale notification for the write transaction | 19:59 |
| *** networkstatic has joined #openstack-neutron | 19:59 | |
| marun | markmcclain: i'm actually fuzzy about the implications, can you think of a case where processing a stale notification would corrupt local state? | 20:00 |
| markmcclain | there could be a gap between finalizing the transaction and reading the system tiem | 20:00 |
| marun | markmcclain: yes, that's the nature of the problem | 20:00 |
| marun | markmcclain: and presumably why systems that need to coordinate on db state use sequence ids of some sort | 20:01 |
| marun | and a gap between reading the system time and starting the read transaction | 20:01 |
| beagles | rossella_s: hi | 20:02 |
| rossella_s | beagles: hi | 20:02 |
| beagles | rossella_s: how are you? | 20:02 |
| *** networkstatic has quit IRC | 20:03 | |
| rossella_s | Better, almost ok. Thanks | 20:03 |
| markmcclain | marun: might ahve to consider payload in addtion to timestamp determine if something is stale | 20:03 |
| *** networkstatic has joined #openstack-neutron | 20:03 | |
| rossella_s | I've read the log of your meeting on Tuesday | 20:03 |
| marun | markmcclain: it's not really an issue when we only have a single rpc-processing server. but we're going to need multi-process if not multi-host to scale | 20:03 |
| rossella_s | beagles: are you meeting again today at 22 UTC, right? | 20:04 |
| marun | markmcclain: uh, that sounds like guesswork | 20:04 |
| beagles | rossella_s: not today actually, most of us weren't available, but we are tomorrow | 20:04 |
| markmcclain | marun: we can already run dhcp on multiple network hosts | 20:05 |
| marun | markmcclain: i dont' mean the agent | 20:05 |
| marun | markmcclain: i mean the server | 20:05 |
| *** carlp has joined #openstack-neutron | 20:05 | |
| rossella_s | beagles: ok, better for me actually. Let's sync tomorrow, is it at 22 UTC? | 20:05 |
| markmcclain | marun: which server? dnsmasq or neutron? | 20:05 |
| marun | markmcclain: the timing race is only a problem if we have more than a single process processing rpc | 20:05 |
| marun | markmcclain: neutron | 20:06 |
| rossella_s | beagles: where are you based? What's your time zone? | 20:06 |
| beagles | rossella_s: yes.. I suggested that maybe if that time was problematic for anyone we could try a spontaneous sync up earlier | 20:06 |
| carlp | Greetings! I'm trying to figure out how to use the vlan type with ML2. I have the config in place so that physnet1 maps to VLANs 1-100. I'm trying to figure out the neutron invocation which will let me create that provider network and attach a physical NIC. Can someone help me out? | 20:06 |
| marun | markmcclain: if we only have a single rpc server, then only one thing runs at a time and there is no possibility of a race | 20:06 |
| beagles | rossella_s: I am in Newfoundland so it is UTC -3:30 | 20:06 |
| marun | markmcclain: but if we have multiple processes (on the same host or distributed), then a race if possible | 20:07 |
| markmcclain | multi-threaded API server already exists | 20:07 |
| beagles | rossella_s: how about you? | 20:07 |
| markmcclain | marun: so the transactions could commit at different times already | 20:07 |
| marun | markmcclain: not real threads | 20:07 |
| marun | markmcclain: oh, crap | 20:07 |
| rossella_s | beagles: I am in Spain usually, in Italy right now. UTC +1 | 20:07 |
| marun | markmcclain: sorry to be dumb. so, yeah. we have a potential race condition, period. | 20:08 |
| *** djschaap has quit IRC | 20:08 | |
| markmcclain | marun: yes we do | 20:08 |
| marun | markmcclain: so, sequence ids? | 20:08 |
| markmcclain | except what generates those? | 20:09 |
| rossella_s | beagles: let's try a spontaneous sync up earlier then. | 20:09 |
| beagles | rossella_s: cool.. so it might make sense to try and sync up a little earlier | 20:09 |
| beagles | ha | 20:09 |
| beagles | rossella_s: you beat me to it :) | 20:09 |
| marun | markmcclain: well, I'm sure there are distributed systems people that could answer that better than i | 20:09 |
| marun | markmcclain: my naive approach might be db-based (since that's the point of synchronization) via an update trigger | 20:09 |
| markmcclain | marun: that's why I think we have process stale notifications | 20:09 |
| rossella_s | beagles: :) | 20:09 |
| markmcclain | marun: and assume that we should re-sync state | 20:10 |
| marun | markmcclain: 'spray-and-pray' | 20:10 |
| beagles | marun markmcclain you guys have tweaked my attention | 20:10 |
| markmcclain | marun: not the greatest solution for now | 20:10 |
| beagles | is this something that is summarizable in a one liner? | 20:11 |
| marun | markmcclain: there is no guarantee that a resync fixes the problem | 20:11 |
| * beagles scrolls | 20:11 | |
| markmcclain | beagles: basically we're discussing how to ensure the DHCP network state is consistent with the db state | 20:11 |
| *** networkstatic has quit IRC | 20:12 | |
| markmcclain | there are a few races in the process that make it possible for the DHCP cached state to not reflect the proper logical state | 20:12 |
| marun | beagles: state synchronization between server and agent -> notifications can't always guarantee consistency, but when a full sync is triggered, there is no way at present to determine if a subsequently seen notification is stale | 20:12 |
| *** Abhishek has joined #openstack-neutron | 20:14 | |
| *** banix has quit IRC | 20:14 | |
| marun | markmcclain: My takeaway from this is that an rdms is a poor choice for coordinating a distributed system. | 20:15 |
| marun | rdms -> rdbms | 20:15 |
| markmcclain | relational databases and eventual consistency don't always play well | 20:16 |
| *** yfried1 has joined #openstack-neutron | 20:16 | |
| *** networkstatic has joined #openstack-neutron | 20:16 | |
| markmcclain | the crazy thing about the whole thing is that we only care that the are in sync a really tiny moment in time | 20:16 |
| marun | markmcclain: ok, so I'll give some thought to what are the implications of processing stale notifications and see if we can work around | 20:16 |
| markmcclain | we only want them to be insync when the DHCPDISCOVER is processed | 20:17 |
| markmcclain | oops DHCPOFFER | 20:17 |
| markmcclain | otherwise the sync state does not matter | 20:17 |
| ijw | markmclain: It's a timing thing - we need to be ready for it, really | 20:18 |
| *** yfried has quit IRC | 20:18 | |
| marun | markmcclain: sure, it's all super simple ;) | 20:18 |
| markmcclain | ijw: being ready does help make things run faster | 20:18 |
| markmcclain | dnsmasq is not the only solution for providing DHCP services | 20:18 |
| beagles | memcached? | 20:18 |
| markmcclain | so it might make sense to consider alternatives | 20:19 |
| marun | markmcclain: what are you thinking of? | 20:19 |
| markmcclain | beagles: memcahed presents a problem because dnsmasq reads the data from it's configuration files | 20:19 |
| marun | I'm not sure the inherent complexity of state synchronization can really be done away with | 20:19 |
| *** networkstatic has quit IRC | 20:19 | |
| *** networkstatic has joined #openstack-neutron | 20:20 | |
| beagles | markmcclain: oh.. I thought we were poking it with data, but I guess the problem remains that there needs to be an actor to update it | 20:21 |
| markmcclain | marun: ISC DHCP is another option that has been tossed around for years but no one has explored it | 20:21 |
| marun | markmcclain: although…. if we stored the complete dhcp state in memcache (or something like it), regenerated it on every change, and sent notifications on change... | 20:21 |
| markmcclain | during Folsom there was a little talk of it as an alternative | 20:22 |
| *** networkstatic has quit IRC | 20:22 | |
| *** networks_ has joined #openstack-neutron | 20:22 | |
| *** networks_ is now known as networkstatic | 20:22 | |
| markmcclain | beagles: correct | 20:22 |
| marun | markmcclain: would using ISC remove the burden of state synchronization? | 20:22 |
| markmcclain | possibly.. the ISC does have an API | 20:28 |
| markmcclain | not sure if we're trading one problem for another | 20:28 |
| ijw | Two ways to work this - we pass the state over, or DHCP calls back for it | 20:30 |
| markmcclain | ijw: correct | 20:30 |
| ijw | If we pass it over we need to revisit that sync protocol. | 20:30 |
| marun | markmcclain: i think there is less chance to get out of sync if we have an incremental api | 20:31 |
| *** networkstatic has quit IRC | 20:31 | |
| markmcclain | incremental APIs can easily get out of sync | 20:31 |
| markmcclain | especially in systems where were messages can be lost | 20:31 |
| ijw | markmcclain: they do, but they're also a solved problem | 20:32 |
| markmcclain | ijw: right | 20:32 |
| *** otherwiseguy has joined #openstack-neutron | 20:32 | |
| ijw | You always need the 'help I lost a message' and 'help I need a full table' options | 20:32 |
| markmcclain | I think long term a callback mech is really the road to go down | 20:32 |
| marun | markmcclain: do tell? | 20:32 |
| ijw | Yeah, but then I'll turn this around - if we can't get it right here we have the same problem with everything else we're keeping in sync (or not) | 20:32 |
| markmcclain | ijw: DHCP is kind of special child because we're basically replicating the entire DB | 20:33 |
| markmcclain | the other sub children of the prob only replicate smaller parts of the db | 20:34 |
| ijw | markmcclain: depends on how you view it - with OVS you're putting together a subset of the l2 table and then syncing it to the agent; it's the same issue, and what happens there if you lose a message? | 20:34 |
| marun | markmcclain: I think you're right that stale notifications are probably not so bad for the race in question. the incidence should be low enough to only be a single add/update/delete for a given network, and we can just filter it out. | 20:35 |
| markmcclain | ijw: the OVS l2 agent has it's troubles too | 20:35 |
| marun | ijw: losing a message isn't really the problem | 20:36 |
| ijw | markmcclain: Case in point. If this is a recurring problem, can we solve this once and for everything? | 20:36 |
| markmcclain | ijw: something I'm working on | 20:36 |
| marun | ljw: failure on either end will mean a restart, and a sync will ensure the message is not lost | 20:36 |
| marun | not lost -> doesn't matter | 20:37 |
| ijw | marun: If I understand your comments in the bug, the issue is that we stop sending messages when a DHCP agent appears to die. | 20:37 |
| markmcclain | internally we've been looking at multi-process, multi-threaded approach | 20:37 |
| marun | ljw: there is a patch under review that fixes that | 20:37 |
| markmcclain | it also coalesces events to be more intelligent about how it asks the db questions when updating | 20:38 |
| *** jecarey has quit IRC | 20:38 | |
| *** rohit404 has quit IRC | 20:38 | |
| markmcclain | still working making it work in proper HA mode | 20:38 |
| marun | markmcclain: that helps things scale, but I don't think that addresses the issue of state synchronization | 20:39 |
| *** rossella_s has quit IRC | 20:39 | |
| marun | i.e reliability | 20:39 |
| markmcclain | marun: actually it does | 20:39 |
| ijw | OK, so could we have an incrementing state version? That's an old favourite for making sure we know whether a message is relevant or not | 20:39 |
| marun | markmcclain: do tell? | 20:39 |
| markmcclain | because the event processing and syncs are handled in a predictable order | 20:39 |
| marun | markmcclain: how do you coordinate them? | 20:39 |
| markmcclain | at worst it might sync an extra time or two if message is that stale | 20:39 |
| marun | markmcclain: How do you detect that an extra sync is necessary? | 20:40 |
| markmcclain | based on the pending messages for a network | 20:40 |
| marun | markmcclain: or rather, how are you discarding stale notifications? | 20:40 |
| *** networkstatic has joined #openstack-neutron | 20:41 | |
| markmcclain | we look at the queue for network | 20:41 |
| marun | markmcclain: uh | 20:41 |
| markmcclain | and actually process the messages mostly in order | 20:41 |
| markmcclain | but occasionally process them out of order | 20:41 |
| markmcclain | because you can infer that certain ones will fail | 20:41 |
| markmcclain | or are redundant | 20:42 |
| marun | markmcclain: what a headache | 20:42 |
| markmcclain | we're planning to share for more feedback once we work out the HA side of it | 20:43 |
| marun | markmcclain: i'm having trouble sorting out if the complexity we're talking about is inherent or accidental | 20:43 |
| marun | markmcclain: so is this modifications to neutron? | 20:43 |
| markmcclain | it's drop in replacement | 20:43 |
| markmcclain | for the agent | 20:43 |
| marun | uh | 20:43 |
| marun | and you're only going to show it at the end? | 20:43 |
| markmcclain | well we want something works :) | 20:44 |
| marun | that seems problematic, especially coming from the ptl | 20:44 |
| marun | if 3rd parties try to do that, we smack them | 20:44 |
| markmcclain | not going to drop something at the end | 20:44 |
| markmcclain | because it is a PoC that solves on particular use case | 20:44 |
| *** mlavalle has joined #openstack-neutron | 20:44 | |
| marun | i get it - its good for dreamhost | 20:44 |
| *** garyk has quit IRC | 20:45 | |
| marun | but giving the project a drop-in replacement for something significant is problematic from a support/maintenance perspective, as i'm sure you're away | 20:45 |
| marun | aware | 20:45 |
| markmcclain | we are but is supports a system which is unique to us | 20:45 |
| *** networkstatic has quit IRC | 20:45 | |
| marun | i mean, we ask for blueprints and designs for significant work to educate and engage stakeholders | 20:46 |
| markmcclain | so there will always be bits that where we have to tie to legacy | 20:46 |
| *** garyk has joined #openstack-neutron | 20:46 | |
| marun | I get why you're doing it - a business case needs solving. | 20:46 |
| markmcclain | if it proves workable then it also could serve as a blueprint to solve the more general case | 20:47 |
| marun | I'm not sure the approach is consistent with how Neutron is supposed to run, though. | 20:47 |
| marun | If it were anyone but you, I think it would be a big problem. | 20:47 |
| markmcclain | the interesting thing about this particular solution is that it increases complexity in a few places | 20:47 |
| markmcclain | and you know how I generally feel about doing that :) | 20:48 |
| marun | Fair enough | 20:48 |
| markmcclain | debugging the thing is not always the easiest either | 20:48 |
| marun | I still think doing this in the open would be preferable, though, for all the same reasons I've yelled at other contributors for taking a similar approach. | 20:48 |
| markmcclain | yeah we're close to opening it | 20:49 |
| marun | How soon? | 20:49 |
| markmcclain | it's the legacy ties that prevent | 20:49 |
| markmcclain | I'm hoping before the end of the year | 20:49 |
| ijw | markmcclain: I don't suppose you have a design you could share rather than code? Firstly it would be easier to critique and secondly (and given your IP issues) it might be more useful | 20:50 |
| marun | We still need to support havana and a drop-in replacement probably won't be backportable, and if there are lessons to be learned from this work in maintaining the existing stuff it would be good to see it sooner than later. | 20:50 |
| marun | btw, I'm going to be completely offline dec 25->jan 5 | 20:50 |
| * ijw is completely offline now. | 20:50 | |
| markmcclain | ijw: no designs yet | 20:51 |
| *** jroovers has quit IRC | 20:51 | |
| *** vkozhukalov has quit IRC | 20:51 | |
| markmcclain | this is somethign we've been wanting to talk about as you know we prefer to work in the open | 20:51 |
| markmcclain | just have to get all fo the proper signoffs | 20:51 |
| ijw | I'm trying to picture what you've said, actually, haven't really got it yet | 20:52 |
| markmcclain | it's very similar to marun's threaded approach | 20:53 |
| markmcclain | *event-loop threaded | 20:53 |
| markmcclain | but is also multiprocess because well eventlet like to deadlock alot | 20:53 |
| * ijw could wish we started with Twisted | 20:54 | |
| markmcclain | ijw: developing with concurrency in mind from the beginning is much better than the current way we do it | 20:54 |
| *** networkstatic has joined #openstack-neutron | 20:55 | |
| marun | i'm glad we don't have twisted | 20:55 |
| ijw | Indeed, but I'm less worried about concurrency in a process and more worried about how we deal with all the distributed system failures, cos they're the ones that generally bite | 20:56 |
| marun | but i kinda wish we didn't have eventlet | 20:56 |
| ijw | marun: you see, that's your problem, you don't like to live dangerously | 20:56 |
| ijw | node.js ftw | 20:56 |
| markmcclain | marun: we can't get rid of eventlet until oslo.messaging supports alternative concurrency libs | 20:56 |
| * beagles bites tongue | 20:56 | |
| ijw | I have never uysed node.js, I'd like to point out. On the other hand, I have programmed hard realtime systems so I'm a proper grownup when it comes to events and tasks | 20:57 |
| beagles | :) | 20:57 |
| * marun goes to work on oslo.messaging | 20:57 | |
| ijw | May God have mercy on your soul | 20:58 |
| *** pcm_ has quit IRC | 20:58 | |
| marun | the fact that oslo.messaging's default rpc dispatcher is so brain-dead doesn't really bode well | 20:58 |
| beagles | oh god now you are just taunting me | 20:58 |
| marun | or maybe it simply wasn't intended to support state synchronization? | 20:58 |
| marun | (though what else would we need it for?) | 20:58 |
| ijw | marun: everything in a distributed system can be reduced to a state sync problem, I would argue | 20:59 |
| marun | ijw: that's my feeling too | 20:59 |
| marun | just like ai is search, distributed is sync | 20:59 |
| ijw | I think the original issue is that Openstack was designed with a 'messages always arrive' mentality - Nova used to be awful in Essex and Folsom, Neutron started later and has got to that point | 20:59 |
| beagles | the evolution seems to be along the lines: this always works, this sometimes doesn't work so we need to retry, then to oh shit when we retry we don't know if the first thing actually happened, HELP | 21:04 |
| beagles | then finally you realize you have to make your data interchange pessimistic but simple and not load everything down with extra chattiness and complexity | 21:04 |
| beagles | some folks go the latter route.. and we never see them again | 21:05 |
| *** networkstatic has quit IRC | 21:05 | |
| marun | wait, isn't that us? :p | 21:06 |
| beagles | so am I to take it that eventlet pools and work queues are on a few people's menus? | 21:06 |
| beagles | hehe.. we aren't quite terminal yet I think | 21:06 |
| beagles | by menu, I mean approaches to tackling some of the issues? | 21:07 |
| * beagles twiddles thumbs | 21:08 | |
| beagles | markmcclain: wake up ^^^^ | 21:09 |
| markmcclain | sorry was tracking the nova meeting :) | 21:10 |
| marun | beagles: we're approaching it by way of trying to fix races | 21:10 |
| beagles | :) | 21:10 |
| marun | beagles: or deadlocks, as the case may be | 21:10 |
| beagles | I've got it scrolling in another window | 21:11 |
| markmcclain | yeah I don't like eventlet for a lot of reasons | 21:11 |
| markmcclain | and would love to move away from it | 21:11 |
| markmcclain | eventlet also has some Py3 considerations too | 21:11 |
| * beagles nods at marun | 21:15 | |
| beagles | I'm trying to think of a way to put what I'm thinking as succinctly as possible. | 21:21 |
| marun | that sounds like it could be good | 21:24 |
| beagles | The word spawn appears 68 times in neutron - filtering out the tests and openstack directroy, some is just delegation to another method with spawn in and some are third party plugins... | 21:25 |
| beagles | so.. not all 68 are particularly relevant... | 21:25 |
| beagles | greenthreads or no, spawn asynchronous tasks without proper organization of the work they are performing is a recipe for disaster... | 21:26 |
| beagles | the dhcp thing I pointed out some weeks ago is an example... | 21:27 |
| beagles | I wonder if there is value in picking through each place and imagining ways it could go horribly wrong | 21:28 |
| beagles | (as opposed to me simply asserting that there is definitely stuff going wrong in most of those places ;)) | 21:29 |
| marun | there might be value, but most of the problems are actually db-related | 21:30 |
| beagles | damn, I shoud've filtered out the async_process cuz that word is all over that thing | 21:30 |
| marun | i worked pretty hard to make sure it's sane, and anyway it's impact is just the l2 agent | 21:31 |
| beagles | dhcp you mean? or the async_process thing? | 21:31 |
| marun | async_process | 21:35 |
| beagles | ah okay.. I'm assuming it is cool :) | 21:35 |
| beagles | I kind of figured what it does wouldn't be that dangerous | 21:36 |
| dkehn | markmcclain: https://review.openstack.org/#/c/59858/ please | 21:37 |
| beagles | are the db contention issues the db-specific kind like table or page level locks or are more mundane like locking things in different orders | 21:37 |
| * beagles wanders off an maybe looks it up in the existing info like he should | 21:39 | |
| marun | tbd | 21:40 |
| *** aymenfrikha has quit IRC | 21:40 | |
| marun | boot instances at a high enough rate and nova starts timing out. then investigate why | 21:40 |
| *** suresh12 has quit IRC | 21:44 | |
| *** suresh12 has joined #openstack-neutron | 21:44 | |
| *** suresh12 has quit IRC | 21:49 | |
| *** roeyc has joined #openstack-neutron | 21:51 | |
| *** hichihara has joined #openstack-neutron | 21:52 | |
| *** rkukura has quit IRC | 21:56 | |
| markmcclain | dkehn: approved | 21:57 |
| dkehn | markmcclain: thx tons | 21:57 |
| *** clev has joined #openstack-neutron | 21:59 | |
| *** HenryG has joined #openstack-neutron | 22:07 | |
| *** pcm_ has joined #openstack-neutron | 22:07 | |
| *** pcm_ has quit IRC | 22:09 | |
| *** julim has quit IRC | 22:09 | |
| *** pcm_ has joined #openstack-neutron | 22:09 | |
| ijw | OK, we may actually have RAs this century | 22:10 |
| *** WackoRobie has quit IRC | 22:10 | |
| ijw | marun: how far have you explored the boot timeouts? | 22:26 |
| marun | ljw: ? | 22:27 |
| *** roeyc has quit IRC | 22:27 | |
| ijw | high rate startup boot timeouts | 22:29 |
| ijw | Also, seriously mate, that's an 'i' | 22:29 |
| *** hichihara has left #openstack-neutron | 22:30 | |
| *** yamahata has quit IRC | 22:37 | |
| *** nati_ueno has joined #openstack-neutron | 22:38 | |
| *** carlp has quit IRC | 22:46 | |
| openstackgerrit | A change was merged to openstack/neutron: Remove start index 0 in range() https://review.openstack.org/61100 | 22:46 |
| *** safchain has quit IRC | 22:48 | |
| *** clev has quit IRC | 22:59 | |
| jog0 | new critical bug https://bugs.launchpad.net/neutron/+bug/1262906 | 23:03 |
| openstackgerrit | Salvatore Orlando proposed a change to openstack/neutron: Improve handling of security group updates https://review.openstack.org/63100 | 23:04 |
| jog0 | markmcclain: ^ | 23:05 |
| *** thedodd has quit IRC | 23:06 | |
| markmcclain | jog0: looking | 23:07 |
| *** clev has joined #openstack-neutron | 23:10 | |
| *** harlowja is now known as harlowja_away | 23:12 | |
| *** b3nt_pin has joined #openstack-neutron | 23:13 | |
| jog0 | markmcclain: going to talk about it in -qa | 23:13 |
| openstackgerrit | Emilien Macchi proposed a change to openstack/neutron: Configure security group when using ML2 plugin https://review.openstack.org/63240 | 23:14 |
| *** beagles has quit IRC | 23:14 | |
| *** b3nt_pin is now known as beagles | 23:14 | |
| *** Abhishek has quit IRC | 23:15 | |
| *** yfried1 has quit IRC | 23:17 | |
| openstackgerrit | A change was merged to openstack/neutron: Do not trigger agent notification if bindings do not change https://review.openstack.org/58860 | 23:18 |
| *** peristeri has quit IRC | 23:18 | |
| *** beagles has quit IRC | 23:19 | |
| openstackgerrit | dekehn proposed a change to openstack/python-neutronclient: Adds delete of extra-dhcp-opt to the client https://review.openstack.org/49166 | 23:26 |
| *** b3nt_pin has joined #openstack-neutron | 23:26 | |
| *** otherwiseguy has quit IRC | 23:27 | |
| *** b3nt_pin is now known as beagles | 23:33 | |
| *** gdubreui has joined #openstack-neutron | 23:34 | |
| *** nati_uen_ has joined #openstack-neutron | 23:34 | |
| *** nati_uen_ has quit IRC | 23:34 | |
| *** nati_uen_ has joined #openstack-neutron | 23:35 | |
| *** nati_ueno has quit IRC | 23:37 | |
| *** clev has quit IRC | 23:38 | |
| *** networkstatic has joined #openstack-neutron | 23:39 | |
| *** networkstatic has quit IRC | 23:43 | |
| *** harlowja_away is now known as harlowja | 23:57 | |
| *** mlavalle has quit IRC | 23:58 | |
| *** yamahata has joined #openstack-neutron | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!