| *** samueldmq has quit IRC | 00:08 | |
| *** gyee has quit IRC | 00:31 | |
| *** zhangfei has joined #openstack-ironic | 01:17 | |
| *** zhangfei has quit IRC | 01:23 | |
| *** zhangfei has joined #openstack-ironic | 01:25 | |
| *** hwoarang has quit IRC | 01:46 | |
| *** hwoarang has joined #openstack-ironic | 01:48 | |
| *** dsneddon has quit IRC | 02:50 | |
| *** dsneddon has joined #openstack-ironic | 02:50 | |
| *** dsneddon has quit IRC | 02:57 | |
| *** irclogbot_1 has quit IRC | 03:05 | |
| *** irclogbot_3 has joined #openstack-ironic | 03:08 | |
| *** edleafe has quit IRC | 03:10 | |
| *** dsneddon has joined #openstack-ironic | 03:15 | |
| *** dsneddon has quit IRC | 03:19 | |
| *** dsneddon has joined #openstack-ironic | 03:32 | |
| *** dsneddon has quit IRC | 03:39 | |
| *** w14161_1 has quit IRC | 03:47 | |
| *** w14161_1 has joined #openstack-ironic | 03:48 | |
| *** dsneddon has joined #openstack-ironic | 05:20 | |
| *** dsneddon has quit IRC | 05:24 | |
| *** whoami-rajat has joined #openstack-ironic | 06:15 | |
| *** dsneddon has joined #openstack-ironic | 06:20 | |
| *** dsneddon has quit IRC | 06:25 | |
| *** e0ne has joined #openstack-ironic | 06:26 | |
| *** e0ne has quit IRC | 06:28 | |
| *** e0ne has joined #openstack-ironic | 06:36 | |
| *** e0ne has quit IRC | 06:38 | |
| *** e0ne has joined #openstack-ironic | 06:38 | |
| *** rpittau|afk is now known as rpittau | 06:52 | |
| rpittau | good morning ironic! o/ | 06:52 |
|---|---|---|
| *** pcaruana has joined #openstack-ironic | 07:12 | |
| *** yolanda_ has joined #openstack-ironic | 07:24 | |
| *** e0ne has quit IRC | 07:37 | |
| *** e0ne has joined #openstack-ironic | 07:43 | |
| *** e0ne has quit IRC | 07:55 | |
| *** mbeierl has quit IRC | 08:03 | |
| *** e0ne has joined #openstack-ironic | 08:04 | |
| *** e0ne has quit IRC | 08:07 | |
| *** gkadam has joined #openstack-ironic | 08:07 | |
| *** e0ne has joined #openstack-ironic | 08:15 | |
| *** dsneddon has joined #openstack-ironic | 08:21 | |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Truncate node text fields when too long https://review.openstack.org/650307 | 08:23 |
| *** whoami-rajat has quit IRC | 08:24 | |
| *** dsneddon has quit IRC | 08:26 | |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Truncate node text fields when too long https://review.openstack.org/650307 | 08:27 |
| *** dsneddon has joined #openstack-ironic | 08:28 | |
| *** dsneddon has quit IRC | 08:33 | |
| *** e0ne has quit IRC | 08:35 | |
| *** dsneddon has joined #openstack-ironic | 08:37 | |
| openstackgerrit | Digambar proposed openstack/ironic master: Modify the iDRAC driver to use realtime RAID creation https://review.openstack.org/634903 | 08:41 |
| *** e0ne has joined #openstack-ironic | 08:41 | |
| *** dsneddon has quit IRC | 08:42 | |
| *** e0ne has quit IRC | 08:43 | |
| *** whoami-rajat has joined #openstack-ironic | 08:55 | |
| *** andrein has joined #openstack-ironic | 08:55 | |
| *** andrein has quit IRC | 08:56 | |
| *** diga has joined #openstack-ironic | 08:59 | |
| *** dsneddon has joined #openstack-ironic | 09:22 | |
| *** dsneddon has quit IRC | 09:27 | |
| openstackgerrit | Varsha Verma proposed openstack/ironic master: Removes `hash_distribution_replicas` configuration option https://review.openstack.org/650912 | 09:41 |
| *** andrein has joined #openstack-ironic | 09:54 | |
| *** pcaruana has quit IRC | 10:17 | |
| *** e0ne has joined #openstack-ironic | 10:18 | |
| *** andrein has quit IRC | 10:23 | |
| *** dsneddon has joined #openstack-ironic | 11:03 | |
| *** e0ne has quit IRC | 11:04 | |
| *** dsneddon has quit IRC | 11:07 | |
| *** andrein has joined #openstack-ironic | 11:40 | |
| *** Lucas_Gray has joined #openstack-ironic | 11:42 | |
| *** dsneddon has joined #openstack-ironic | 12:02 | |
| *** whoami-rajat has quit IRC | 12:05 | |
| *** dsneddon has quit IRC | 12:06 | |
| *** Lucas_Gray has quit IRC | 12:08 | |
| *** e0ne has joined #openstack-ironic | 12:08 | |
| *** Lucas_Gray has joined #openstack-ironic | 12:10 | |
| openstackgerrit | Digambar proposed openstack/ironic master: Modify the iDRAC driver to use realtime RAID creation https://review.openstack.org/634903 | 12:29 |
| *** e0ne has quit IRC | 12:41 | |
| *** e0ne has joined #openstack-ironic | 12:42 | |
| *** dsneddon has joined #openstack-ironic | 13:03 | |
| *** Lucas_Gray has quit IRC | 13:07 | |
| *** dsneddon has quit IRC | 13:08 | |
| *** mjturek has joined #openstack-ironic | 13:11 | |
| *** dims has quit IRC | 13:21 | |
| *** e0ne has quit IRC | 13:23 | |
| *** edleafe has joined #openstack-ironic | 13:29 | |
| *** whoami-rajat has joined #openstack-ironic | 13:31 | |
| *** e0ne has joined #openstack-ironic | 13:36 | |
| *** dims has joined #openstack-ironic | 13:48 | |
| *** dsneddon has joined #openstack-ironic | 13:52 | |
| *** baha has joined #openstack-ironic | 13:53 | |
| *** dsneddon has quit IRC | 13:57 | |
| *** diga has quit IRC | 14:04 | |
| *** hjensas has joined #openstack-ironic | 14:19 | |
| *** dims has quit IRC | 14:20 | |
| *** e0ne has quit IRC | 14:21 | |
| *** e0ne has joined #openstack-ironic | 14:26 | |
| *** Lucas_Gray has joined #openstack-ironic | 14:27 | |
| *** dsneddon has joined #openstack-ironic | 14:32 | |
| *** dims has joined #openstack-ironic | 14:35 | |
| *** dsneddon has quit IRC | 14:37 | |
| NobodyCam | Good Morning Ironic'ers | 14:40 |
| NobodyCam | and ofc | 14:40 |
| rpittau | hey NobodyCam :) | 14:40 |
| NobodyCam | TGIF! | 14:40 |
| NobodyCam | hey hey rpittau :) happy Friday | 14:40 |
| rpittau | :) | 14:40 |
| *** e0ne has quit IRC | 14:43 | |
| *** e0ne has joined #openstack-ironic | 14:52 | |
| *** dims has quit IRC | 14:53 | |
| *** e0ne has quit IRC | 14:53 | |
| *** dims has joined #openstack-ironic | 14:56 | |
| *** dims has quit IRC | 15:01 | |
| *** Goneri has joined #openstack-ironic | 15:01 | |
| *** e0ne has joined #openstack-ironic | 15:03 | |
| -openstackstatus- NOTICE: Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html | 15:03 | |
| *** ChanServ changes topic to "Gerrit is offline for several hours starting at 15:00 UTC to perform the opendev migration; see http://lists.openstack.org/pipermail/openstack-discuss/2019-April/005011.html" | 15:03 | |
| *** gyee has joined #openstack-ironic | 15:06 | |
| *** dims has joined #openstack-ironic | 15:07 | |
| *** dsneddon has joined #openstack-ironic | 15:31 | |
| *** dims has quit IRC | 15:35 | |
| *** dsneddon has quit IRC | 15:36 | |
| *** zhangfei has quit IRC | 15:41 | |
| *** dims has joined #openstack-ironic | 15:44 | |
| *** Lucas_Gray has quit IRC | 15:44 | |
| *** dsneddon has joined #openstack-ironic | 15:49 | |
| *** dsneddon has quit IRC | 15:54 | |
| *** e0ne has quit IRC | 15:57 | |
| *** e0ne has joined #openstack-ironic | 16:09 | |
| *** mgoddard has quit IRC | 16:34 | |
| *** mgoddard has joined #openstack-ironic | 16:35 | |
| *** e0ne has quit IRC | 16:36 | |
| rpittau | bye all! have a great long weekend! o/ | 16:42 |
| *** rpittau is now known as rpittau|afk | 16:42 | |
| *** mgoddard has quit IRC | 16:44 | |
| *** mgoddard has joined #openstack-ironic | 16:45 | |
| *** gyee has quit IRC | 16:48 | |
| *** andrein has quit IRC | 16:48 | |
| *** gyee has joined #openstack-ironic | 16:49 | |
| *** gkadam has quit IRC | 16:58 | |
| TheJulia | \o/ friday | 17:25 |
| eandersson | o/ | 18:03 |
| eandersson | We are finally able to use port groups | 18:03 |
| eandersson | but we are seeing one issue during pxe | 18:03 |
| eandersson | ironic only configures one port during pxe. Which makes sense because only one port has pxe enabled. | 18:03 |
| eandersson | The problem is that both switches needs to be configured for lacp to not freak out | 18:04 |
| *** e0ne has joined #openstack-ironic | 18:19 | |
| *** dims has quit IRC | 18:26 | |
| *** jaypipes_ has joined #openstack-ironic | 18:28 | |
| *** dims has joined #openstack-ironic | 18:29 | |
| *** jaypipes has quit IRC | 18:30 | |
| *** e0ne has quit IRC | 18:46 | |
| *** dims has quit IRC | 18:48 | |
| *** dims has joined #openstack-ironic | 18:51 | |
| *** baha has quit IRC | 18:56 | |
| *** Goneri has quit IRC | 18:57 | |
| *** ijw has joined #openstack-ironic | 19:17 | |
| TheJulia | eandersson: static switch side? | 19:18 |
| *** hjensas has quit IRC | 19:19 | |
| TheJulia | eandersson: the issue if the other Port is online is the switch tries to send the packets down the second link and you then have magical packet loss :/ | 19:20 |
| *** jaypipes_ is now known as jaypipes | 19:26 | |
| *** hjensas has joined #openstack-ironic | 19:32 | |
| TheJulia | eandersson: I know some switches have defaults that can be set for lacp behavior, but it has been ages since I've done anything like that with fully featured switches. | 19:40 |
| *** whoami-rajat has quit IRC | 19:40 | |
| *** andrein has joined #openstack-ironic | 19:52 | |
| *** jcoufal has joined #openstack-ironic | 20:10 | |
| ccstone | eandersson has summoned me because I forgot that I'm on IRC! I work with him and have been working on PXE boot also. What I ended up doing was configuring port groups for all nodes, setting the first port PXE-enabled and the second PXE-disabled, but I had to go onto the switch and set 'switchport trunk allowed vlan <provisioning_net_vlan>' | 20:32 |
| ccstone | ... on the second switch, so it would pass through that VLAN ID on both ports in the channel (sorry, early-ENTER) | 20:32 |
| ccstone | One fix may be to have the IPA image only bring up eth0 and never eth1 - but from looking at docs, it looks like that element also pulls in an element that configures DHCP on all available interfaces. | 20:35 |
| *** hjensas has quit IRC | 20:41 | |
| *** andrein has quit IRC | 20:49 | |
| TheJulia | Will be in front of a computer in 15 | 20:50 |
| *** macintoshme has joined #openstack-ironic | 20:50 | |
| *** jcoufal has quit IRC | 20:50 | |
| *** andrein has joined #openstack-ironic | 20:56 | |
| TheJulia | ccstone: But is that not the best way to be summoned to irc? :) | 21:01 |
| TheJulia | ccstone: so... I guess there are two ways people have done it that I'm aware of. The first being static portgroup configurations with LACP configured such that once the OS begins to announce LACP, the LAG is established, but it also sounds like your portgrouping across switches... | 21:04 |
| *** Emine has quit IRC | 21:04 | |
| TheJulia | ccstone: The other way that I know peopel have do it, and I know cisco had been pushing was with the ML2 drivers to pass the portgroup configuration so the port group is dynamically configured. | 21:05 |
| *** mjturek has quit IRC | 21:05 | |
| TheJulia | I feel like a better descriptive phrase for what seems to be your configuration is cross switch bonding/balancing | 21:06 |
| ccstone | Yeah, all of our port groups are across multiple switches. In this case there should be no case where the port channel differs between the two switches (it either doesn't work, or intermittently works in this case), but if we only set one of the two ports to pxe_enabled, it never configures the port channel on the partner switch | 21:10 |
| ccstone | A workaround was to set that VLAN ID on the second switch. It allows provisioning, and networking-cisco later configures both ports correctly. We're then left in a case where SW1 PO has only one vlan (the desired vlan) and SW2 PO has both the desired vlan and the provisioning vlan (since networking-cisco doesn't remove existing config). And then a manual step to remove that provisioning vlan from the second switch. | 21:12 |
| ccstone | But for security, that provisioning vlan must be removed from the second switch, or there's a risk of the provisioned node being able to access resources on the provisioning VLAN | 21:13 |
| *** dsneddon has joined #openstack-ironic | 21:15 | |
| TheJulia | ccstone: Is that logic burried in networking-cisco? | 21:23 |
| TheJulia | specifically the refusal to configure the second switch based upon pxe_enabled? | 21:23 |
| * TheJulia wonders if networking-ansible could one day be useful | 21:24 | |
| ccstone | Yeah - it seems that the switch only gets configured during the provisioning phase if pxe_enabled on the port is true - so it sort of feels like it's treated as if there's no port group at all during the provisioning phase, these were the same issues we were having before we could use port groups (nova-mitaka). (After provisioning it works exactly as expected) | 21:29 |
| *** Lucas_Gray has joined #openstack-ironic | 21:31 | |
| TheJulia | Okay, that is partial logic that is likely causing that in ironic which we could likely patch or make changable | 21:31 |
| TheJulia | ccstone: neutron network interface right? :) | 21:32 |
| TheJulia | interesting... | 21:34 |
| *** ijw has quit IRC | 21:34 | |
| ccstone | TheJulia: Yep :D Ideally we would have a port group with two ports, both with switch+portchannel config, one pxe_enabled, one pxe_disabled. It should configure both port channels (because those should always be the same) but since IPA treats them as separate interfaces, we need to only give DHCP to one of them. Otherwise we have 2x nics with different IPs behind the same port channel. | 21:36 |
| TheJulia | Well, it is interesting because when we sort portgroup data, pxe_enabled should be set | 21:36 |
| *** ijw has joined #openstack-ironic | 21:36 | |
| TheJulia | but I'm only looking at part of the path | 21:36 |
| * TheJulia is still looking | 21:36 | |
| ccstone | so neutron should only create a DHCP lease for either the pxe_enabled port, or (ideally) the port address on the port group | 21:36 |
| *** ianw_pto is now known as ianw | 21:37 | |
| ccstone | Or at least this is how it seems from my tinkering last night and how we were able to get around it :D | 21:38 |
| TheJulia | rocky right? | 21:38 |
| eandersson | Yep | 21:39 |
| TheJulia | so I guess, we basically still need to plug the port, but not do PXE config.... is that correct? | 21:40 |
| *** ijw has quit IRC | 21:41 | |
| TheJulia | what if both ports are labeled with pxe_enabled? | 21:41 |
| eandersson | That almost works | 21:41 |
| TheJulia | howso? | 21:42 |
| TheJulia | and how not, trying to compare/contrast with the code on my other monitor :) | 21:42 |
| TheJulia | eandersson: ccstone: I guess my impression is the attempt is to throw all provisioning traffic to one side of the switching fabric? | 21:55 |
| ccstone | Yeah - first attempt was with pxe_enabled on both, which does configure the switch exactly the way we want. At that point the problem is IPA and Neutron - Neutron supplies DHCP info for each port MAC address individually, so we have two independent configs on the same port channel. So one fix would be for neutron to still only provide one DHCP interface, or for IPA to never bring up eth1 | 21:57 |
| ccstone | Yeah, since it's very unlikely we can do LACP within the IPA image, we need to make sure eth1 doesn't interfere with anything (don't come up, absolutely don't ask for a DHCP address, etc) | 21:58 |
| ccstone | A hacky fix that I was thinking of was just putting a dummy MAC on the 2nd NIC of the port group. There are no cases where we want neutron to give a unique IP to that second NIC. We either want it to not work, or to be configured in a port group post-provision. | 21:59 |
| ccstone | And the port group will ignore the dummy MAC we gave it | 21:59 |
| ccstone | So neutron will think it configured it, but the DHCP address is for the wrong MAC and it never comes up | 22:00 |
| TheJulia | Its not a bad idea, except there is a bunch of predication on keeping that up to date | 22:03 |
| * TheJulia ponders | 22:03 | |
| TheJulia | ccstone: what if an IP is assigned but DHCP is not offered? | 22:05 |
| ccstone | TheJulia: in our case that would work (both are pxe_enabled, both DHCP only offered to one MAC?) That would configure both sides of the port channel correctly. eth1 would attempt to come up but fail. Since we have 'no lacp suspend-individual' set on ports (which allows them to be standalone), as long as eth1 fails to become active then we're good. | 22:07 |
| TheJulia | oh | 22:08 |
| TheJulia | shoot | 22:08 |
| ccstone | We run into an issue only when a: the port channel config on both switches differs, or b: both interfaces on the host come up independently because neutron gave them different DHCP IPs, but are supposed to be part of a single port channel | 22:08 |
| TheJulia | so dhcp at all is going to bring up link carrier on eth1 | 22:08 |
| ccstone | Yeah.. Honestly feels like something we could fix in IPA if we just never brought eth1 up. That's how PXE works IIRC, it never tries eth1 | 22:09 |
| TheJulia | then again, well behaved dhcp clients also down the port after failing.... | 22:09 |
| TheJulia | That is correct | 22:09 |
| ccstone | Yeah, an active port with no IP will not interfere with us from what I've been noticing. | 22:09 |
| ccstone | So as long as eth1 never gets an IP from DHCP, it's still OK that it's link up | 22:10 |
| *** ijw has joined #openstack-ironic | 22:10 | |
| ccstone | (again, this is just our cisco hardware, this could be different elsewhere) | 22:10 |
| TheJulia | ccstone: interesting, I wonder if that is a safety measure in the switch... | 22:10 |
| *** ijw has quit IRC | 22:13 | |
| TheJulia | ccstone: what if we admin state down'ed it? | 22:14 |
| *** ijw has joined #openstack-ironic | 22:14 | |
| TheJulia | the second port that is? | 22:14 |
| *** rpioso is now known as rpioso|afk | 22:14 | |
| ijw | I sense networking | 22:14 |
| TheJulia | ijw: indeed | 22:15 |
| ijw | Ah, the old PXE/bond problem? | 22:15 |
| *** rh-jelabarre has quit IRC | 22:15 | |
| TheJulia | ijw: indeed! | 22:15 |
| TheJulia | cross-switch as well! | 22:15 |
| ijw | Yup | 22:15 |
| TheJulia | I don't think we can tell neutron not to actually do a dhcp entry for an individual port :\ | 22:16 |
| ijw | Break the port group (and you can leave both ports up if you like), and things work, which it seems you have... | 22:16 |
| ijw | No, I think you can only disable DHCP for the whole network | 22:16 |
| * TheJulia wonders what exactly admin state down does | 22:16 | |
| ijw | If only there were some sort of gathering of teams where you could ask people for that sort of thing | 22:16 |
| TheJulia | heh | 22:17 |
| ijw | It is literally port-down no-link, I think, and as such you don't have to remove the DHCP entry | 22:17 |
| TheJulia | Only if there is beer or whisky | 22:17 |
| ijw | In Denver there is much beer | 22:17 |
| TheJulia | ++ | 22:17 |
| TheJulia | Guess it depends on the ml2 driver if it would honor the port admin state down bit | 22:18 |
| ijw | Sorry, the 'cisco' bit pinged me or I would have never spotted this, but seems to me that you really want both interfaces to have DHCP entries in the right circumstances, and for the same address | 22:18 |
| ijw | (spitballing a bit) | 22:18 |
| TheJulia | Oh wow, there IS a "don't allocate an ip" flag | 22:18 |
| ijw | Perhaps one option would be not to use Neutron's DHCP for the provisioning network since it doesn't do what you want. You could run your own | 22:19 |
| ijw | That does work - ports don't have to have IPs at all (and the only reason the flag is there is because it was so easy to do it by accident) | 22:19 |
| TheJulia | ijw: ccstone's issue is a little different than that because they are doing cross-chassis lacp and if only one side gets configured bad things happen | 22:20 |
| ijw | Yeah, that would probably end badly | 22:20 |
| ijw | But bonding is bad, you want ECMP :) | 22:21 |
| TheJulia | Eh, I'm an OSPF (with ECMP...) girl. | 22:21 |
| *** sleterrier has joined #openstack-ironic | 22:25 | |
| sleterrier | Hey there. I am not sure if this is the best place to ask but I figured there must be a good pool of cloud-init experts around here. | 22:28 |
| sleterrier | I am trying to overwrite the config of growpart cloud-init, but without having to pass any user_data | 22:28 |
| TheJulia | sleterrier: One warning, most people are off/away today, but we do have some folks here | 22:28 |
| ijw | Also #cloud-init is full of friendly folks | 22:29 |
| TheJulia | sleterrier: if memory serves, it is static in the code path only controlled by user-data :\ | 22:29 |
| TheJulia | sleterrier: but yeah, #cloud-init is likely your best bet for ideas. | 22:29 |
| sleterrier | Could I drop a file somewhere in /var/lib/cloud/ instead of passing the following user_data: | 22:30 |
| TheJulia | sleterrier: The other option may be to actually bake the config in the images.... the default config includes the flag to growpart if memory serves | 22:30 |
| TheJulia | sleterrier: That would totally be a #cloud-init question :\ | 22:30 |
| TheJulia | but if you can influence the config by dropping a file, you can inject files | 22:30 |
| TheJulia | I'm 95% sure it unpacks files first | 22:30 |
| sleterrier | http://paste.openstack.org/show/749557/ | 22:31 |
| sleterrier | Ok thanks TheJulia ! | 22:31 |
| *** Goneri has joined #openstack-ironic | 22:42 | |
| TheJulia | ccstone: ijw might be able to provide something of a quick sanity check, but I suspect something like https://gist.github.com/juliakreger/4b56b0c67c2776b54815dfcc02847c04 might work... (missing a config entry that would need to be created.) | 22:44 |
| ijw | So if you're looking for Neutron DHCP to leave those ports alone - which is what I imagine you're after - then that looks right to me | 22:48 |
| ccstone | That looks like it'd be good to me too. I've still got a couple of nodes to try deploying to shortly and can try patching that and see how it works. | 22:55 |
| TheJulia | ccstone: adding config option to the gist | 22:58 |
| TheJulia | ccstone: done, running unit tests locally | 22:58 |
| TheJulia | current gist revision passes unit tests, so if your conductor setting is flipped.... it _should_ work for creating ports. Digging through the other paths now | 23:00 |
| TheJulia | remove_ports_from_network needs logic too | 23:01 |
| TheJulia | updated again, unit tests running. I'll need to write tests and upload this into review once it is back up | 23:05 |
| TheJulia | python3 unit tests pass, pep8 passes, so... should be mostly harmless in the default state, true will change things of course | 23:09 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!