Tuesday, 2016-06-07

*** apearson has quit IRC00:01
*** Ashana has quit IRC00:01
*** Ashana has joined #openstack-powervm00:02
*** thorst has quit IRC00:04
*** Ashana has quit IRC00:07
*** Ashana has joined #openstack-powervm00:08
*** Ashana has quit IRC00:13
*** Ashana has joined #openstack-powervm00:16
*** Ashana has quit IRC00:20
*** Ashana has joined #openstack-powervm00:22
*** Ashana has quit IRC00:27
*** Ashana has joined #openstack-powervm00:28
*** Ashana has quit IRC00:32
*** dwayne_ has joined #openstack-powervm00:33
*** Ashana has joined #openstack-powervm00:33
*** Ashana has quit IRC00:38
*** Ashana has joined #openstack-powervm00:39
*** dwayne_ has quit IRC00:42
*** Ashana has quit IRC00:44
*** Ashana has joined #openstack-powervm00:45
*** Ashana has quit IRC00:49
*** Ashana has joined #openstack-powervm00:51
*** Ashana has quit IRC00:55
*** thorst has joined #openstack-powervm00:56
*** jwcroppe has quit IRC00:56
*** Ashana has joined #openstack-powervm00:57
*** thorst has quit IRC00:58
*** thorst has joined #openstack-powervm00:58
*** dwayne_ has joined #openstack-powervm00:59
*** Ashana has quit IRC01:01
*** Ashana has joined #openstack-powervm01:03
*** thorst has quit IRC01:07
*** Ashana has quit IRC01:07
*** Ashana has joined #openstack-powervm01:09
*** dwayne_ has quit IRC01:12
*** Ashana has quit IRC01:13
*** Ashana has joined #openstack-powervm01:15
*** Ashana has quit IRC01:19
*** Ashana has joined #openstack-powervm01:20
*** Ashana has quit IRC01:25
*** Ashana has joined #openstack-powervm01:26
*** Ashana has quit IRC01:31
*** Ashana has joined #openstack-powervm01:32
*** Ashana has quit IRC01:36
*** Ashana has joined #openstack-powervm01:38
*** Ashana has quit IRC01:42
*** Ashana has joined #openstack-powervm01:45
*** dwayne_ has joined #openstack-powervm01:47
*** Ashana has quit IRC01:49
*** Ashana has joined #openstack-powervm01:50
*** Ashana has quit IRC01:55
*** Ashana has joined #openstack-powervm01:56
*** dwayne_ has quit IRC01:58
*** Ashana has quit IRC02:01
*** Ashana has joined #openstack-powervm02:02
*** thorst has joined #openstack-powervm02:05
*** Ashana has quit IRC02:07
*** Ashana has joined #openstack-powervm02:08
*** dwayne_ has joined #openstack-powervm02:12
*** thorst has quit IRC02:12
*** Ashana has quit IRC02:12
*** Ashana has joined #openstack-powervm02:14
*** Ashana has quit IRC02:19
*** Ashana has joined #openstack-powervm02:20
*** Ashana has quit IRC02:24
*** Ashana has joined #openstack-powervm02:26
*** dwayne_ has quit IRC02:28
*** Ashana has quit IRC02:30
*** Ashana has joined #openstack-powervm02:32
*** Ashana has quit IRC02:36
*** Ashana has joined #openstack-powervm02:37
*** Ashana has quit IRC02:42
*** Ashana has joined #openstack-powervm02:45
*** Ashana has quit IRC02:49
*** Ashana has joined #openstack-powervm02:51
*** Ashana has quit IRC02:56
*** Ashana has joined #openstack-powervm02:57
*** Ashana has quit IRC03:01
*** Ashana has joined #openstack-powervm03:03
*** seroyer has joined #openstack-powervm03:03
*** Ashana has quit IRC03:07
*** Ashana has joined #openstack-powervm03:08
*** thorst has joined #openstack-powervm03:10
*** Ashana has quit IRC03:13
*** Ashana has joined #openstack-powervm03:14
*** thorst has quit IRC03:17
*** tlian has quit IRC03:17
*** Ashana has quit IRC03:19
*** seroyer has quit IRC03:20
*** Ashana has joined #openstack-powervm03:20
*** Ashana has quit IRC03:25
*** Ashana has joined #openstack-powervm03:26
*** Ashana has quit IRC03:31
*** Ashana has joined #openstack-powervm03:32
*** Ashana has quit IRC03:36
*** Ashana has joined #openstack-powervm03:38
*** Ashana has quit IRC03:42
*** Ashana has joined #openstack-powervm03:45
*** Ashana has quit IRC03:50
*** Ashana has joined #openstack-powervm03:51
*** Ashana has quit IRC03:55
*** Ashana has joined #openstack-powervm03:57
*** Ashana has quit IRC04:01
*** Ashana has joined #openstack-powervm04:03
*** Ashana has quit IRC04:07
*** Ashana has joined #openstack-powervm04:09
*** Ashana has quit IRC04:13
*** Ashana has joined #openstack-powervm04:14
*** thorst has joined #openstack-powervm04:15
*** Ashana has quit IRC04:19
*** Ashana has joined #openstack-powervm04:20
*** thorst has quit IRC04:22
*** Ashana has quit IRC04:25
*** Ashana has joined #openstack-powervm04:26
*** Ashana has quit IRC04:31
*** Ashana has joined #openstack-powervm04:32
*** mdrabe has joined #openstack-powervm04:33
*** Ashana has quit IRC04:37
*** Ashana has joined #openstack-powervm04:38
*** Ashana has quit IRC04:43
*** Ashana has joined #openstack-powervm04:45
*** Ashana has quit IRC04:50
*** Ashana has joined #openstack-powervm04:51
*** mdrabe has quit IRC04:52
*** Ashana has quit IRC04:56
*** Ashana has joined #openstack-powervm04:57
*** Ashana has quit IRC05:02
*** Ashana has joined #openstack-powervm05:03
*** Ashana has quit IRC05:07
*** Ashana has joined #openstack-powervm05:09
*** Ashana has quit IRC05:13
*** Ashana has joined #openstack-powervm05:14
*** Ashana has quit IRC05:19
*** Ashana has joined #openstack-powervm05:20
*** thorst has joined #openstack-powervm05:22
*** Ashana has quit IRC05:25
*** Ashana has joined #openstack-powervm05:26
*** thorst has quit IRC05:28
*** Ashana has quit IRC05:30
*** Ashana has joined #openstack-powervm05:32
*** Ashana has quit IRC05:36
*** Ashana has joined #openstack-powervm05:38
*** Ashana has quit IRC05:42
*** Ashana has joined #openstack-powervm05:46
*** Ashana has quit IRC05:50
*** Ashana has joined #openstack-powervm05:52
*** Ashana has quit IRC05:56
*** Ashana has joined #openstack-powervm05:58
*** Ashana has quit IRC06:02
*** Ashana has joined #openstack-powervm06:04
*** Ashana has quit IRC06:08
*** Ashana has joined #openstack-powervm06:10
*** Ashana has quit IRC06:14
*** Ashana has joined #openstack-powervm06:15
*** Ashana has quit IRC06:20
*** Ashana has joined #openstack-powervm06:21
*** thorst has joined #openstack-powervm06:25
*** Ashana has quit IRC06:26
*** Ashana has joined #openstack-powervm06:27
*** Ashana has quit IRC06:31
*** thorst has quit IRC06:32
*** Ashana has joined #openstack-powervm06:33
*** Ashana has quit IRC06:38
*** Ashana has joined #openstack-powervm06:39
*** Ashana has quit IRC06:43
*** Ashana has joined #openstack-powervm06:46
*** Ashana has quit IRC06:51
*** Ashana has joined #openstack-powervm06:52
*** Ashana has quit IRC06:57
*** Ashana has joined #openstack-powervm06:58
*** Ashana has quit IRC07:02
*** Ashana has joined #openstack-powervm07:04
*** Ashana has quit IRC07:09
*** Ashana has joined #openstack-powervm07:10
*** Ashana has quit IRC07:14
*** Ashana has joined #openstack-powervm07:16
*** Ashana has quit IRC07:20
*** Ashana has joined #openstack-powervm07:21
*** Ashana has quit IRC07:26
*** Ashana has joined #openstack-powervm07:27
*** thorst has joined #openstack-powervm07:30
*** Ashana has quit IRC07:32
*** Ashana has joined #openstack-powervm07:33
*** thorst has quit IRC07:37
*** Ashana has quit IRC07:37
*** Ashana has joined #openstack-powervm07:39
*** Ashana has quit IRC07:43
*** Ashana has joined #openstack-powervm07:47
*** Ashana has quit IRC07:51
*** Ashana has joined #openstack-powervm07:53
*** Ashana has quit IRC07:57
*** Ashana has joined #openstack-powervm07:59
*** Ashana has quit IRC08:03
*** Ashana has joined #openstack-powervm08:05
*** k0da has joined #openstack-powervm08:07
*** Ashana has quit IRC08:10
*** Ashana has joined #openstack-powervm08:11
*** Ashana has quit IRC08:15
*** Ashana has joined #openstack-powervm08:17
*** Ashana has quit IRC08:22
*** Ashana has joined #openstack-powervm08:23
*** Ashana has quit IRC08:27
*** Ashana has joined #openstack-powervm08:29
*** Ashana has quit IRC08:34
*** Ashana has joined #openstack-powervm08:35
*** thorst has joined #openstack-powervm08:35
*** Ashana has quit IRC08:39
*** Ashana has joined #openstack-powervm08:41
*** thorst has quit IRC08:42
*** Ashana has quit IRC08:45
*** Ashana has joined #openstack-powervm08:47
*** Ashana has quit IRC08:51
*** Ashana has joined #openstack-powervm08:53
*** Ashana has quit IRC08:57
*** Ashana has joined #openstack-powervm08:59
*** Ashana has quit IRC09:03
*** Ashana has joined #openstack-powervm09:05
*** Ashana has quit IRC09:09
*** thorst has joined #openstack-powervm09:39
*** thorst has quit IRC09:47
*** thorst has joined #openstack-powervm10:47
*** smatzek has joined #openstack-powervm10:52
*** thorst has quit IRC10:53
*** thorst_ has joined #openstack-powervm11:46
openstackgerritDrew Thorstensen proposed openstack/nova-powervm: DNM: CI Check  https://review.openstack.org/29593512:02
*** smatzek has quit IRC12:26
*** thorst_ is now known as thorst12:31
*** kylek3h has joined #openstack-powervm12:32
*** Ashana has joined #openstack-powervm12:38
*** Ashana has quit IRC12:38
*** Ashana has joined #openstack-powervm12:38
*** jwcroppe has joined #openstack-powervm12:42
*** smatzek has joined #openstack-powervm12:46
*** kylek3h has quit IRC12:47
*** smatzek_ has joined #openstack-powervm12:50
*** jwcroppe has quit IRC12:54
*** smatzek has quit IRC12:54
*** kylek3h has joined #openstack-powervm12:57
*** smatzek_ has quit IRC12:58
*** edmondsw has joined #openstack-powervm13:01
*** mdrabe has joined #openstack-powervm13:21
*** arnoldje has joined #openstack-powervm13:25
*** esberglu has joined #openstack-powervm13:28
*** miltonm has quit IRC13:36
*** lmtaylor1 has joined #openstack-powervm13:38
*** miltonm has joined #openstack-powervm13:38
*** lmtaylor1 has left #openstack-powervm13:40
*** apearson has joined #openstack-powervm13:43
*** tlian has joined #openstack-powervm13:54
*** smatzek_ has joined #openstack-powervm13:59
*** kriskend has joined #openstack-powervm14:08
*** jwcroppe has joined #openstack-powervm14:13
*** lmtaylor1 has joined #openstack-powervm14:27
*** Ashana has quit IRC14:29
*** Ashana has joined #openstack-powervm14:30
thorstesberglu: ping me when I can try another CI run (and have the logs post)14:34
esbergluI already kicked off a few sandbox ones and they connected to the logserver fine. I started a single nova-powervm one too just to make sure14:35
thorstawesome.  Looks like the one I did today had 3 failures...really curious to get into debugging those14:37
thorstI'm sure efried is too  :-)14:37
esbergluThat and I would really like to figure out the 404s trying to get project info on the controller14:38
thorstesberglu: Yeah...that is super weird to me14:39
thorstbecause I just don't see that.14:39
thorstin any env's I deploy14:39
thorstthat's why I was asking if it was updating the code properly14:39
esberglu@thorst: If you go on the dashboard and go to system overview it has one project but it’s listed as deleted.14:49
esbergluBut when I list the projects with openstack project list it has that project id for admin14:50
esbergluSo it’s just trying to get info from a deleted project14:50
thorstweird14:58
*** mdrabe has quit IRC15:00
*** mdrabe has joined #openstack-powervm15:14
*** Ashana has quit IRC15:22
*** Ashana has joined #openstack-powervm15:24
*** k0da has quit IRC15:36
*** smatzek_ has quit IRC15:52
*** Ashana has quit IRC16:06
openstackgerritDrew Thorstensen proposed openstack/nova-powervm: Blueprint for OVS and LB agent support  https://review.openstack.org/32660616:08
*** smatzek_ has joined #openstack-powervm16:09
*** smatzek_ has quit IRC16:10
*** Ashana has joined #openstack-powervm16:21
thorstesberglu efried: Odd path, but we have CI results again16:23
thorsthttp://184.172.12.213/$LOG_PATH/16:23
thorstesberglu: What change was that run done against?  What openstack review?16:23
esbergluOh I just kicked it off from jenkins that makes sense16:24
esbergluDuh16:24
efriedOne failure ain't bad.16:25
efriedesberglu, you on the hook for tracking that down?16:25
*** Ashana has quit IRC16:26
*** Ashana has joined #openstack-powervm16:28
*** Ashana has quit IRC16:32
*** Ashana has joined #openstack-powervm16:34
*** Ashana has quit IRC16:38
*** Ashana has joined #openstack-powervm16:40
*** Ashana has quit IRC16:44
*** Ashana has joined #openstack-powervm16:46
*** Ashana has quit IRC16:51
thorstanyone else relieved we only had one failure in the CI tests after being offline this long?16:51
efriedthorst, me!16:52
*** Ashana has joined #openstack-powervm16:52
*** Ashana has quit IRC16:56
*** Ashana has joined #openstack-powervm16:58
*** Ashana has quit IRC17:03
*** Ashana has joined #openstack-powervm17:05
*** apearson has quit IRC17:58
*** apearson has joined #openstack-powervm18:02
*** esberglu has quit IRC18:05
*** esberglu has joined #openstack-powervm18:10
*** esberglu has quit IRC18:10
*** Ashana has quit IRC18:31
*** Ashana has joined #openstack-powervm18:41
*** apearson has quit IRC18:43
*** Ashana has quit IRC18:44
*** Ashana has joined #openstack-powervm18:45
efriedthorst, got any ideas as to how PowerVM vNIC will play into SRIOV in OpenStack?18:49
efriedPerhaps we add a conf option mapping physical port labels to network names?18:49
efriedOr perhaps we employ a naming convention whereby the physical port labels can be associated with the network names on the fly.  'cept that wouldn't work cause we have no control over the network names, eh?18:50
thorstefried: So its not clear to me.  If we give the API a set of ports...will it pick a port to use?  Or do we need to map it to a very specific port on our own?18:50
*** erlarese has joined #openstack-powervm18:50
efriedthorst, we at least have to be able to associate particular pports with specific routes.18:50
thorstefried: so at a minimum, we need something (like we have for NPIV) where we have ports assigned to 'fabrics'18:51
thorstor as neutron calls them 'physical networks'18:52
efriedRight, the pports have a 'label' field, which we can treat like fabric names.18:52
thorstooo, fancy18:52
thorsterlarese: If we have that...do we just map the physical_network from neutron to that fabric name?18:52
efriedIt's gotta be up to the user to preconfigure the physical ports with labels in such a way that the driver can know which physical networks they're physically cabled to.18:52
efriedThen we can just pick pports (intelligently, distributing across cards & VIOSes as much as possible, while avoiding saturation) associated with that physical network.18:53
efriedThe question is, how do we map from pport label to phys net name/ID?18:53
thorstefried: I'm not so sure  :-)18:54
thorstcan the label == the phys net?18:54
*** jwcroppe has quit IRC18:54
efriedThe labels can be registered beforehand, but networks can be created on the fly.18:54
efriedor I would say yeah.18:54
thorstwell18:54
thorstremember, a physical network != neutron network18:54
efriedThough I suppose it's the same admin handling both, so...18:54
thorsta neutron network is a virtual network that sits on top of a physical network18:54
efriedIs that 1:1?18:54
thorstso many neutron networks can sit on top of a single physical network18:54
efriedmm18:55
thorstmany to 118:55
efriedWhere the separate neutron networks allow you to do what, control QoS, routing, filtering, etc?18:55
efriedNAT, DNS, IP ranges...18:55
thorstyep18:56
efriedAnd today when a user spawns, does she choose a neutron network or a physical network?18:56
thorstneutron network18:56
efriedWhen/how are the neutron networks created?18:56
thorstand when an operator makes a neutron network, if they pick a physical network type (flat or vlan) it must map to a physical network18:57
thorstoverlays don't require physical networking (ex. gre, vxlan or geneve)18:57
thorstneutron networks are created 'whenevs'18:57
thorstIf using VLANs, kinda rarely (one neutron network is shared with many tenants)18:57
thorstif using overlays, often.  Each user gets their own network usually.18:58
efriedSo our neutron (networking-powervm) agent would have to remain responsible for maintaining a mapping of physical networks to pport labels.  Whenever a neutron network is created, it is (either implicitly or explicitly) associated with that physical network.  Then whenever a nova spawn happens and a neutron network is selected, we use that mapping to determine which pports to create the vNIC on.18:59
*** apearson has joined #openstack-powervm19:00
thorstefried: Well, so what I don't know...  The networking guide for KVM says that the neutron agent is completely optional when using SR-IOV19:00
thorstso...how does that work?19:00
thorsthow is this problem solved in the KVM space?19:00
efriedPresumably you would have to say which phys net you want to be on in that case.19:01
openstackgerritDrew Thorstensen proposed openstack/nova-powervm: Initial LB VIF Type  https://review.openstack.org/30244719:01
thorstyeah...so...I think we need to figure that out19:01
efriedWhich is fine - skips the step of going through the neutron network.19:01
efriedToday in the spawn dialog in horizon, it's not clear to me exactly which paths the networking options go through.19:01
thorstthey're honestly all over19:02
thorstthey want the experience to be 'I pick a network' and the end user not knowing the thousand steps (or two steps) it takes to complete that request19:02
thorstsee https://review.openstack.org/#/c/326606/ for an example how19:03
efriedKVM doesn't have the same vNIC semantic we do, though, right?19:04
thorstvNIC no.  But this isn't really a vNIC issue.  You have 4 ports on a card.  Two are on one fabric, the other two are on another fabric19:04
efriedTo them, vNIC == { VF | macvtap }19:05
thorsthow do I assign my VFs19:05
thorstI *think* vnic is similar to macvtap, but I'm not sure19:05
thorstwe both have VFs19:05
efriedYeah, I think macvtap is where the VFs are owned by the "hypervisor" (VIOS in our case) and somehow presented through to the VM.19:05
thorstefried: that's the simplist explanation of macvtap I've heard19:06
thorsthopefully accurate  :-D19:06
efriedHah, lots of hand-waving via "somehow"19:06
efriedBut I'm guessing we can't just override19:06
efriedneutron port-create <net-id> --binding:vnic-type macvtap19:06
efriedWas thinking we would need our own vnic-type?19:07
thorstunless we bind macvtap to our vnic tech19:07
efriedright.  Is that an option?19:07
efriedIt's sort of... disingenuous19:07
efriedCause even if they work kinda the same way, our tech is NOT macvtap.19:07
efriedSure would make the UX simpler, though...19:08
thorstefried: agree its disingenuous.  Will need to explore19:08
thorstbetter understand macvtap19:08
efriedCurrent valid vnic-types are: <direct | direct-physical | macvtap | normal | baremetal>19:08
efried...at least assuming the neutron CLI is up to date.19:09
thorstnot really sure what each of those means19:09
thorstdirect vs. direct-physical19:09
thorstmaybe that's the line for us?19:09
efriedyeah, vs baremetal19:09
thorstbaremetal is for ironic I think19:09
thorstnot our use cases19:09
efriedSo THEY get their own keyword.  Why can't we?19:09
thorstthey're 100% open source19:09
efried(Cause we're oot)19:09
thorstand we're not in tree19:09
*** esberglu has joined #openstack-powervm19:10
thorstour hypervisor isn't in tree yet I think is the biggest issue19:10
efried'direct' sounds like direct VF-to-VM; 'direct-physical' sounds like pport-to-VM, which sounds like promiscuous mode, which I think is dedicated/non-SRIOV.19:11
efriedNo idea what 'normal' is.19:11
thorstnormal would be like ovs, sea, or linux bridge19:12
thorstthat stuff I'm doing19:12
efriedHow does the user map networks to SEAs today?19:12
efriedRephrase: How does the user map networks (physical and/or neutron) to NetBridges/SEAs today?19:13
thorstusing the physical_networks attribute when defining the neutron network19:13
thorstwhich is only valid if you're using Flat or VLAN networks19:13
efriedhow are physical_networks mapped to SEAs?19:14
thorstand then the SEA agent maps a physical network to a given SEA via the conf file19:14
efriedbingo.19:14
efriedIt may be possible to use the exact same ethos for (PowerVM's incarnation of) vNIC.19:14
efriedBecause a SEA is the same thing, logically, as a "label".19:15
efriedI.e. a glom of physical ports pointing to the same physical network.19:15
thorsthttps://uproxx.files.wordpress.com/2014/11/zapp-brannigan-checkmate.jpg?quality=9019:15
efriedAhh, where's /giphy when you need it?19:16
thorston Twitter19:16
efriedI don't twit.19:16
thorstI know you love twitter19:16
efriedNo opinion - never used it.19:16
thorstooo, lets change that19:17
thorstthe whole world needs to hear our musings19:17
openstackgerritDrew Thorstensen proposed openstack/nova-powervm: Blueprint for OVS and LB agent support  https://review.openstack.org/32660619:18
*** apearson has quit IRC19:18
efriedI assume you are not serious.  So back to the topic at hand:19:19
*** apearson has joined #openstack-powervm19:19
efriedMethinks SRIOV is potentially even simpler than SEA, because you can have multiple SEAs on the same phys net.19:20
efriedLooks like the config for SEA is19:21
efriedbridge_mappings = <physical network>:<vio partition name>:<sea device>,<physical network>:<vio partition name>:<sea device>,...19:21
thorstyou generally only want one network bridge (up to two SEAs) on a given physical network19:21
efriedConceivably SRIOV could be:19:21
efriedsriov_mappings = <phsyical_network>:<port_label>19:21
efriedsriov_mappings = <phsyical_network>:<port_label>,<phsyical_network>:<port_label>,...19:22
efriedPerhaps optional if network name matches pport label.19:22
efriedSo19:23
efriedsriov_mappings = production:production,dev:dev,test:test19:23
efriedcould be omitted entirely.19:23
*** Ashana has quit IRC19:24
thorstefried: yep...but again, I really want to know how KVM does this19:25
thorstI've fed that comment to svenkat several times, something I'd like to see in the blueprint19:25
efriedPreconfig would comprise simply19:25
efriedpvmctl sriov update -i loc_code=xyz -s label=production19:25
efriedor similar.19:25
efriedthorst, any tips for the most efficient way of finding out "how KVM does X"?19:26
efriedRead source code?  Ask on an IRC channel?  Find docs?19:27
thorstefried: uhh, lots of reading19:27
thorstof source19:27
thorstthat's what I do19:27
thorstand testing it out on my laptop19:27
thorst(but can't really test SR-IOV on a laptop VM)19:27
thorstefried: Do me a favor...pause thinking about that at some point and review this:19:28
thorsthttps://review.openstack.org/#/c/302447/19:28
efriedthorst, Yeah, sure.  Been assuming I should first have some background understanding of things like "VIF" and "Linux Bridge" and "Open VSwitch".19:29
efriedWhich I don't.19:29
thorstsee blueprint :-D19:29
efriedRoger wilco.19:29
*** Ashana has joined #openstack-powervm19:33
efriedthorst, should the change set have19:33
efriedImplements: blueprint xxxx19:33
efried?19:33
thorstefried: seems, reasonable  :-)19:35
*** lmtaylor2 has joined #openstack-powervm19:49
*** lmtaylor1 has quit IRC19:50
*** jwcroppe has joined #openstack-powervm20:04
thorstugh...I do not like having to restart the conductor to do LPM20:08
thorstadreznec: you there?20:12
efriedthorst, https://review.openstack.org/#/c/302447/ reviewed, fwiw.20:15
thorstbah, you hater20:15
efriedHatin on self.20:16
efriedoh, you mean volume of comments?  Probably mostly bogus.20:16
*** lmtaylor2 has quit IRC20:17
*** lmtaylor1 has joined #openstack-powervm20:17
*** lmtaylor2 has joined #openstack-powervm20:36
*** lmtaylor2 has left #openstack-powervm20:37
*** lmtaylor1 has quit IRC20:40
*** k0da has joined #openstack-powervm20:46
*** kylek3h has quit IRC20:54
*** jwcroppe has quit IRC20:54
*** kriskend has quit IRC21:05
mdrabeefried: Could there be any possibility of a race condition with the build_map in the vscsi driver?21:05
mdrabehttps://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/slot.py#L89-L10121:05
mdrabehttps://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/volume/vscsi.py#L25021:06
efriedmdrabe, the symptom would be one of the rebuilt mappings using "next available slot"?21:07
mdrabesymptom would be VIOS 1 getting slot X in one "connect_volume_to_vio" execution's build_map and VIOS 2 also getting slot X in another "connect_volume_to_vio" execution21:09
mdrabeThen getting that 500 error21:09
efriedBecause both VIOS FeedTasks executing at once.21:09
efriedMm.21:09
efriedI thought the algo was deterministic.21:10
efriedAh, but no, I remember now.21:10
efriedIt ain't.21:10
mdrabeThe thing is, the PlugVifs task always runs before in spawn....21:10
efriedSo yeah, the build_map property should be synchronized.21:10
*** Ashana has quit IRC21:10
mdrabeI've been trying to reproduce this issue, got logs in place to catch the race but just can't hit it21:10
efriedNevertheless, build_map should be synchronized.21:11
mdrabeDo you think that should be something that's backported to mitaka?21:12
*** thorst has quit IRC21:12
efriedAnd init_recreate_map should be private, and ideally even somehow prevented from ever running twice.21:12
efriedYes, mitaka.  Especially if you can prove that it's the cause of your bugs.21:13
*** thorst has joined #openstack-powervm21:13
efriedI think the fact that two separate init_recreate_map()s can be running at once - both writing to the same self._build_map - is enough to screw up vifs, storage, anything.21:14
mdrabeThe only argument against is that the PlugVifs task runs before the volume connection tasks and calls build_map21:16
mdrabeSo in theory the _build_map should not be None by that point21:16
efriedSure, PlugVifs is running in *both* threads at the same time.21:16
efriedBoth threads enter the init_recreate_map at the same time.21:16
efriedIt's the map that gets effed up.21:16
*** Ashana has joined #openstack-powervm21:17
efriedSubsequent accesses to it (for storage attach etc.) are hitting the same _build_map21:17
efriedIt just happens to be an effed up _build_map21:17
*** thorst has quit IRC21:17
efriedBecause multiple conflicting init_recreate_map()s ran at the same time.21:17
mdrabeOh okay, I was under the impression that PlugVifs completes entirely before the volume connection tasks21:17
efriedYes, it does.  You're still not getting me.21:18
efriedPlugVifs is running in two separate places, isn't it?21:18
efriedOr maybe it isn't?21:18
mdrabehttps://github.com/openstack/nova-powervm/blob/master/nova_powervm/virt/powervm/driver.py#L415-L41721:18
efrieddah, okay, my bad.21:20
*** Ashana has quit IRC21:21
*** Ashana has joined #openstack-powervm21:22
*** Ashana has quit IRC21:27
efriedmdrabe, I'm convinced you can't get there twice at the same time.  Nevertheless, a lock would be appropriate.21:28
*** Ashana has joined #openstack-powervm21:28
mdrabeefried: Should I open a bug?21:29
efriedGuess it depends how far from your current defects it is.21:29
efriedIf you're still having trouble debugging, I would add a dump of the rebuild slot map and ask for a recreate.21:29
*** Ashana has quit IRC21:32
*** Ashana has joined #openstack-powervm21:34
*** thorst has joined #openstack-powervm21:38
*** Ashana has quit IRC21:38
*** Ashana has joined #openstack-powervm21:40
*** thorst has quit IRC21:42
*** Ashana has quit IRC21:44
*** Ashana has joined #openstack-powervm21:46
*** Ashana has quit IRC21:50
*** Ashana has joined #openstack-powervm21:55
*** esberglu has quit IRC21:55
*** Ashana has quit IRC21:59
*** Ashana has joined #openstack-powervm22:00
*** Ashana has quit IRC22:05
*** Ashana has joined #openstack-powervm22:06
*** Ashana has quit IRC22:11
*** Ashana has joined #openstack-powervm22:12
*** Ashana has quit IRC22:16
*** Ashana has joined #openstack-powervm22:18
*** Ashana has quit IRC22:22
*** Ashana has joined #openstack-powervm22:24
*** Ashana has quit IRC22:28
*** Ashana has joined #openstack-powervm22:30
*** arnoldje has quit IRC22:30
*** efried has quit IRC22:34
*** Ashana has quit IRC22:34
*** Ashana has joined #openstack-powervm22:36
*** Ashana has quit IRC22:40
*** mdrabe has quit IRC22:41
*** Ashana has joined #openstack-powervm22:42
*** Ashana has quit IRC22:46
*** edmondsw has quit IRC22:47
*** Ashana has joined #openstack-powervm22:47
*** Ashana has quit IRC22:52
*** Ashana has joined #openstack-powervm22:55
*** k0da has quit IRC22:56
*** Ashana has quit IRC23:00
*** Ashana has joined #openstack-powervm23:01
*** Ashana has quit IRC23:05
*** Ashana has joined #openstack-powervm23:07
*** Ashana has quit IRC23:11
*** Ashana has joined #openstack-powervm23:13
*** Ashana has quit IRC23:17
*** Ashana has joined #openstack-powervm23:19
*** Ashana has quit IRC23:23
*** Ashana has joined #openstack-powervm23:24
*** Ashana has quit IRC23:29
*** Ashana has joined #openstack-powervm23:30
*** Ashana has quit IRC23:34
*** Ashana has joined #openstack-powervm23:36
*** Ashana has quit IRC23:40
*** Ashana has joined #openstack-powervm23:42
*** Ashana has quit IRC23:46
*** Ashana has joined #openstack-powervm23:48
*** Ashana has quit IRC23:52
*** Ashana has joined #openstack-powervm23:56
*** thorst has joined #openstack-powervm23:56
*** thorst has quit IRC23:56
*** thorst has joined #openstack-powervm23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!