| *** lcastell has quit IRC | 00:08 | |
| *** lcastell has joined #starlingx | 00:17 | |
| *** ericho has quit IRC | 00:19 | |
| *** lcastell has quit IRC | 00:19 | |
| *** lcastell has joined #starlingx | 00:20 | |
| *** lcastell has quit IRC | 00:24 | |
| *** cheng1 has joined #starlingx | 01:12 | |
| *** cheng1 has quit IRC | 01:27 | |
| *** cheng1 has joined #starlingx | 01:38 | |
| *** ijolliffe has quit IRC | 02:07 | |
| *** cheng1 has quit IRC | 02:37 | |
| *** cheng1 has joined #starlingx | 02:38 | |
| *** jsun3 has quit IRC | 03:06 | |
| *** jsun3 has joined #starlingx | 03:07 | |
| *** cheng1 has quit IRC | 03:24 | |
| *** cheng1 has joined #starlingx | 04:03 | |
| *** sgw has joined #starlingx | 04:07 | |
| *** zhubx007 has quit IRC | 04:57 | |
| *** zhubx has joined #starlingx | 05:05 | |
| *** OutBackDingo has quit IRC | 05:45 | |
| *** OutBackDingo has joined #starlingx | 06:03 | |
| *** cheng1 has quit IRC | 06:10 | |
| *** cheng1 has joined #starlingx | 06:20 | |
| *** sdin has joined #starlingx | 07:26 | |
| *** sdinescu has quit IRC | 07:28 | |
| *** sgw has quit IRC | 07:50 | |
| *** OutBackDingo has quit IRC | 07:50 | |
| *** OutBackDingo has joined #starlingx | 07:51 | |
| *** changcheng has quit IRC | 07:59 | |
| *** zhubx has quit IRC | 09:24 | |
| *** zhubx has joined #starlingx | 09:24 | |
| *** cheng1 has quit IRC | 09:30 | |
| *** cheng1 has joined #starlingx | 09:45 | |
| *** hoonetorg has quit IRC | 10:39 | |
| *** zhubx has quit IRC | 10:41 | |
| *** zhubx has joined #starlingx | 10:42 | |
| *** hoonetorg has joined #starlingx | 10:52 | |
| *** zhubx has quit IRC | 10:57 | |
| *** zhubx has joined #starlingx | 10:57 | |
| *** cheng1 has quit IRC | 11:21 | |
| *** ijolliffe has joined #starlingx | 12:22 | |
| *** ijolliffe has quit IRC | 12:26 | |
| *** ijolliffe has joined #starlingx | 12:27 | |
| *** ijolliffe has quit IRC | 12:58 | |
| *** ijolliffe has joined #starlingx | 13:03 | |
| *** ijolliffe has quit IRC | 13:36 | |
| *** sgw has joined #starlingx | 13:39 | |
| *** ijolliffe has joined #starlingx | 13:51 | |
| *** OutBackDingo has quit IRC | 14:33 | |
| sgw | Morning all | 14:50 |
|---|---|---|
| sgw | Back to my libvirt question from yesterday, I did some more testing debug, it seems that both nodes can ping the host (I installed a rescueOS on node 1), and tcpdump (running on the host) shows the ICMP request from either node, but not the reply, so it seems the host is not bridging the traffic correct, I thought that gets setup automagically. | 14:55 |
| dpenney | sgaw: you should send an email to starlingx-discuss for that one | 14:56 |
| *** OutBackDingo has joined #starlingx | 14:58 | |
| dtroyer | sgw: do you have forwarding set in the host kernel? | 15:00 |
| dtroyer | I know, like asking if it is plugged in, but I've forgotten that so many times myself... | 15:00 |
| sgw | dtroyer: you are talking about net.ipv4.ip_forward = 1, right? | 15:04 |
| sgw | dpenney: I figured this might require a more interactive debug that IRC provides | 15:05 |
| dtroyer | yup | 15:05 |
| mthebeau | with libvirt from stx-tools, the only thing I'm using forwarding for is to move packets from OAM network to external | 15:06 |
| mthebeau | My first guess would be that the controller is configured to expect requests on one interface/bridge, but the XMl has other nodes putting them out on another. | 15:07 |
| *** ericbarrett has joined #starlingx | 15:08 | |
| mthebeau | the question from me is "what is the output of system host-if-list controller-0, and does that match the boot info in xml in vms/ folder | 15:09 |
| sgw | [wrsroot@controller-0 ~(keystone_admin)]$ system host-if-list controller-0 | 15:11 |
| sgw | +--------------------------------------+-------+----------+----------+---------+--------------+----------+-------------+---------------------------+---------------+ | 15:11 |
| sgw | | uuid | name | class | type | vlan id | ports | uses i/f | used by i/f | attributes | data networks | | 15:11 |
| sgw | +--------------------------------------+-------+----------+----------+---------+--------------+----------+-------------+---------------------------+---------------+ | 15:11 |
| sgw | | 76838bc8-f472-458b-9774-e9b1d7915629 | ens6 | platform | ethernet | None | [u'ens6'] | [] | [] | MTU=1500 | [] | | 15:11 |
| sgw | | 78a16ca5-37c4-4f87-8946-353ee5e9aa57 | ens7 | platform | ethernet | None | [u'ens7'] | [] | [] | MTU=1500 | [] | | 15:11 |
| sgw | | dabf57f0-ce03-4353-a5c4-cb2e34a85fcd | data1 | data | ethernet | None | [u'eth1001'] | [] | [] | MTU=1500,accelerated=True | [u'physnet1'] | | 15:11 |
| sgw | | ee7f946b-d9a4-4fe9-a800-21c1a76d8aa5 | data0 | data | ethernet | None | [u'eth1000'] | [] | [] | MTU=1500,accelerated=True | [u'physnet0'] | | 15:11 |
| sgw | +--------------------------------------+-------+----------+----------+---------+--------------+----------+-------------+---------------------------+---------------+ | 15:11 |
| dpenney | sgw: I figured most of the folks who'd be able to help you are not on IRC | 15:17 |
| mthebeau | lol, host-if-list used to show "oam" and "mgmt". try: for i in ens6 ens7; do { system host-if-show controller-0 $i | grep -q "networks.*mgmt" && echo $i && break; }; done | 15:18 |
| mthebeau | which is to say, which interface is configured as mgmt | 15:18 |
| sgw | ens7, I know that without the script | 15:19 |
| mthebeau | "I know, like asking if it is plugged in" | 15:20 |
| dpenney | you can also check sysinv.log, as there's some interaction between dnsmasq and sysinv | 15:21 |
| mthebeau | after that I'd double check libvirt dump-xml $node to make sure the xml is what it is supposed to be. | 15:22 |
| mthebeau | sorry virsh dumpxml | 15:22 |
| sgw | dpenney: I think it's more basic issue that interaction, I can't even ping between the 2 vms yet | 15:23 |
| dpenney | oh, ok | 15:23 |
| sgw | mthebeau: ok, so I am looking at the 2 bridges on the 2 hosts and the target devs look correct relative to the output of brctl virbr2 links vnet1 and vnet5 which are the ens7 devices inside the vms | 15:26 |
| sgw | the xml matches this. | 15:27 |
| mthebeau | two questions: "2 hosts" may be a misunderanding for me: 1 virt host, two controllers? | 15:29 |
| mthebeau | q2: where is "virbr" coming from; that sounds like vbox instead of libvirt | 15:29 |
| sgw | 2 controllers on my baremetal server, yes virbr is coming from libvirt (in the xml) | 15:30 |
| mthebeau | what does virsh list --all look like | 15:31 |
| sgw | Id Name State | 15:31 |
| sgw | ---------------------------------------------------- | 15:31 |
| sgw | 22 ctn-duplex-controller-0 running | 15:31 |
| sgw | 28 ctn-duplex-controller-1 running | 15:31 |
| mthebeau | you may not be using the stx-tools defaults, the bridges look like stxbr1 | 15:31 |
| sgw | I am not, your correct | 15:32 |
| mthebeau | but you are using stx-tools/ implementation; what is your head commit? | 15:33 |
| sgw | I am not using the stx-tools depolyment, this is a generic libvirt question. | 15:34 |
| mthebeau | ok; one bridge for mgmt network and both controllers have their host-side tun/tap (vnetx) on that bridge? | 15:35 |
| mthebeau | you didn't configure vlans | 15:37 |
| mthebeau | before I start using host for libvirt/qemu virtualization I check that there are no iptables rules related to my configuration | 15:45 |
| sgw | no vlans, the only iptables are for 10.10.10.0 and DOCKER changes, I wonder if I am missing an iptables entry for 192.168.204.0 (the mgnt network) | 15:48 |
| mthebeau | The virt host doesn't usually know about the the mgmt network, doesn't need to do anything with packets, so no iptables. THe virt host usually doesn't have an ip address on the mgmt network | 15:50 |
| mthebeau | the virt host bridge for mgmt is like a simple hub | 15:51 |
| mthebeau | for bridge and tun/taps are all "ifconfig $1 up"? | 15:52 |
| sgw | Yes, they are all up | 15:58 |
| mthebeau | it's been a really long time, and I don't remember the cause, but I have observed a condition where the packet shows up on tun/tap but not the bridge, or it shows up on the guest's interface, but not the host's tun/tap. | 16:00 |
| mthebeau | if you put out a ping on controller-0 mgmt interface, you see it on the bridge? | 16:03 |
| mthebeau | did you copy-paste the mac addresses for the controller's xml by chance? | 16:04 |
| sgw | yes, I can see a ping from controller-0 on the bridge (via tcpdump on the host) | 16:07 |
| sgw | I can get the mac address easily enough | 16:08 |
| mthebeau | and you already said you watched the controller-1 put out pxe/dhcp request | 16:08 |
| sgw | yup and I put a resuceOS on controller-1 and tried to ping controller-0, which I also saw on the bridge via tcpdump | 16:09 |
| mthebeau | I think I'm about out of guesses. I have a controller storage (2 work nodes) running based on 20190325T013001Z . | 16:25 |
| mthebeau | I followed the wiki: https://wiki.openstack.org/wiki/StarlingX/Containers/InstallationOnStandard | 16:28 |
| sgw | mthebeau: I am trying to reset and start over. | 16:30 |
| mthebeau | what interface type did you use in xml for the first two interfaces? It shouldn't matter, but I'm curious; the stx-tools has e1000 and they present as enp7s? on the controllers | 16:33 |
| sgw | e1000 for ens7, I see its enp7 with a different config | 16:37 |
| mthebeau | lspci | grep Ethernet gives me Intel Corporation 82540EM Gigabit Ethernet Controller (rev 03) | 16:41 |
| *** kenyis709 has joined #starlingx | 17:27 | |
| *** Samiam1999DTP has joined #starlingx | 17:39 | |
| *** Samiam1999 has quit IRC | 17:41 | |
| *** rcw has joined #starlingx | 18:04 | |
| *** kenyis709 has quit IRC | 18:46 | |
| *** zhubx has quit IRC | 20:01 | |
| *** zhubx has joined #starlingx | 20:02 | |
| *** rcw has quit IRC | 20:12 | |
| *** sgw has quit IRC | 21:06 | |
| *** ijolliffe has quit IRC | 21:37 | |
| *** jsun3 has quit IRC | 22:49 | |
| *** ericbarrett has quit IRC | 22:49 | |
| *** jsun3 has joined #starlingx | 22:49 | |
| *** bwensley_ has joined #starlingx | 22:54 | |
| *** jsun3 has quit IRC | 22:57 | |
| *** bwensley has quit IRC | 22:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!