| *** dave-mccowan has joined #openstack-kolla | 00:17 | |
| *** dcapone2004 has joined #openstack-kolla | 01:03 | |
| *** gbatir_ has quit IRC | 01:25 | |
| *** dave-mccowan has quit IRC | 01:32 | |
| *** Tony31 has joined #openstack-kolla | 01:39 | |
| *** k_mouza has joined #openstack-kolla | 01:39 | |
| *** k_mouza has quit IRC | 01:43 | |
| *** dave-mccowan has joined #openstack-kolla | 01:56 | |
| *** xinliang has joined #openstack-kolla | 02:11 | |
| Tony31 | Does kolla mean to create `br-ex` or is it named differently? | 02:28 |
|---|---|---|
| goldyfruit | Tony31, https://github.com/openstack/kolla-ansible/blob/d8300d5ae77fdb28bbb749ed5bfb8ca1f4471af0/ansible/group_vars/all.yml#L885 | 02:30 |
| goldyfruit | by default it will be br-ex except if the agent is vmware | 02:31 |
| Tony31 | ok, so i now need to figure out why I dont have br-ex | 02:31 |
| Tony31 | thanks | 02:31 |
| Tony31 | I get this in ml2 conf `[ovs] | 02:31 |
| goldyfruit | Tony31, make sure that the interface set in neutron_external_interface is correct | 02:37 |
| Tony31 | I have 2 x interfaces in there `eth0-ovs` and `eth5-ovs`. Still not sure why there are two | 02:38 |
| Tony31 | flat neutron network is not a provider network right? | 02:39 |
| goldyfruit | https://docs.openstack.org/kolla-ansible/latest/reference/networking/neutron.html | 02:41 |
| Tony31 | no mention of "flat" on that page | 02:42 |
| goldyfruit | But mention of provider network | 02:43 |
| goldyfruit | The thing I don't understand is the "-ovs" in your interface name | 02:43 |
| Tony31 | it's coming from Kayobe. So I am trying to figure out the kolla stuff and work backwards 🙈 | 02:44 |
| goldyfruit | Ohh, Kayobe guys are mostly based in London :/ | 02:44 |
| Tony31 | i'll enable `kolla_enable_neutron_provider_networks: yes ` and re-deploy and see what happens | 02:45 |
| goldyfruit | At least UK, so for me it's the night | 02:45 |
| Tony31 | yes this is my problem. I'm on the other side of the planet and struggle all day until I can chat with them, while I'm at home trying to make dinner and stuff :) | 02:45 |
| Tony31 | Time is `10.45am` | 02:45 |
| goldyfruit | For me it is 9:45pm | 02:46 |
| Tony31 | So you're in Florida or somewhere close? | 02:46 |
| goldyfruit | Montreal | 02:46 |
| Tony31 | ah. Hi from Australia | 02:46 |
| goldyfruit | Ohhh yeah, very far! | 02:47 |
| Tony31 | thanks for your help earlier. I'll have torn all my hair out by end of today I think :) | 02:51 |
| goldyfruit | arf | 02:51 |
| goldyfruit | Good luck | 02:51 |
| Tony31 | nope - still not working | 03:27 |
| Tony31 | cant even ARP for the virtual openstack router | 03:28 |
| Tony31 | will need to wait for the experts | 03:28 |
| sorin-mihai | for some reason, venv/share/kolla-ansible/ansible/roles/keystone/tasks/register.yml fails whenever i try to run deploy, regardless of ansible config and different settings in /etc/kolla/globals.yml. any idea? i can't run it on http only | 03:34 |
| *** vmixor has joined #openstack-kolla | 03:52 | |
| *** Tony31 has quit IRC | 03:57 | |
| *** vmixor has quit IRC | 04:05 | |
| *** dave-mccowan has quit IRC | 04:13 | |
| *** Tony31 has joined #openstack-kolla | 04:24 | |
| Tony31 | sorin-mihai - when it fails, it should give you some output to the terminal as to why it had failed, or at least something for you to go on even if it does not make sense initially. | 04:31 |
| sorin-mihai | you prefer any pastebin? | 04:31 |
| Tony31 | are you unable to read it because of the colours | 04:33 |
| Tony31 | ? | 04:33 |
| Tony31 | what i do is copy the terminal text to notepad++ and then i can see it. | 04:34 |
| sorin-mihai | https://pastebin.com/raw/FS6yuy8k | 04:36 |
| Tony31 | `'certificate verify failed'` | 04:40 |
| Tony31 | Are you installing a certificate? | 04:40 |
| Tony31 | Sounds like it is not correct in terms of CA cert / root cert / server cert | 04:41 |
| Tony31 | or rather `CA cert | (root cert / server cert)` | 04:41 |
| sorin-mihai | using the default paths in globals.yml and running 'kolla-ansible certificates' isn't enough to generate "correct" self signed certificates? when i inspect them they look fine though. also, i have 'host_key_checking=False' in ansible's config, i thought that should be it | 04:52 |
| Tony31 | First - I've never used kolla ansible, I am using kayobe which uses kolla. But I am completely new to this. I will try and help as best I can. | 04:53 |
| Tony31 | So I am not sure about the `kolla-ansible certificates` but I expect it should be correct | 04:53 |
| Tony31 | host_key_checking I think relates to the SSH connection | 04:54 |
| Tony31 | is it possible that you could try and deploy without SSL cert and then after successful deployment, reconfigure with SSL cert? | 04:55 |
| sorin-mihai | the thing is, it used to work yesterday, this changed when i pulled in changes from git | 04:55 |
| sorin-mihai | without ssl it works | 04:55 |
| Tony31 | ah | 04:55 |
| Tony31 | then I am sorry but I think you will need to speak to the experts in about 1 or 2 hours when they come online | 04:56 |
| Tony31 | do you see the certificate files at all? | 04:56 |
| Tony31 | you can manually validate them for any issues | 04:56 |
| sorin-mihai | yes, they are generated and on a quick look they seem fine. i could also generate certs with certbot, but i'm not sure which file should be where, or if i could just simlink them. not sure how the permissions will work out, i see that the permissions of the files is being changed when i try to deploy | 04:59 |
| Tony31 | the error is validation failed, so maybe the CA cert is not there or has a different key to the root and server cert? How are you taking a quick look? :) | 05:06 |
| Tony31 | I am using Kayobe and I commented out a variable I Wanted to remove, and I re-deployed 3 x but that variable kept being used. In the end, I deleted it entirely and then re-deployed and problem fixed. . . | 05:07 |
| sorin-mihai | openssl x509 -in /etc/kolla/certificates/private/internal/internal.crt -text | 05:15 |
| sorin-mihai | Subject: C = US, ST = NC, L = RTP, OU = kolla, CN = 10.2.1.254 | 05:15 |
| sorin-mihai | so, as self signed certificate, it's fine | 05:15 |
| Tony31 | certificate has 2 parts, the cert itself as well as the CA that signed it | 05:16 |
| Tony31 | so there are 2 certs minimum for it to be valid | 05:16 |
| Tony31 | https://www.google.com/search?client=firefox-b-d&q=openssl+validate+certificate+chain | 05:17 |
| sorin-mihai | the ca is set correctly: kolla_internal_fqdn_cacert: "{{ node_config }}/certificates/ca/haproxy-internal.crt" | 05:19 |
| Tony31 | most likely.... but the error you are getting is cert validation failed. If I were in your place, I would check the CA cert and server cert chain, to make sure it validates. Then I'd rule that out as a cause and move on to the next thing | 05:20 |
| *** dcapone2004 has quit IRC | 05:21 | |
| sorin-mihai | but, should this whole checking part just be ignored if there is host_key_checking=False in ansible's config? | 05:21 |
| Tony31 | host key checking is for the ssh connection I believe, so it's not related to SSL HTTPS cert | 05:21 |
| sorin-mihai | i meant validate_certs=False | 05:22 |
| Tony31 | well from what I understand is failing here, the cert is trying to be installed during the deploy. The validate certs = false is for when a `client` is making a https connection and the root/intermediate/CA certs are not `trusted` from the client cert store, and normally the client would get a browser warning. So this validate_certs is to ignore | 05:23 |
| Tony31 | the warning and continue | 05:23 |
| Tony31 | But in this case, the error you have is saying there is something amiss with the certs themselves and they are not matching in terms of validation, so I recommend checking it manually. You can simply copy the cert text to your local system and run openssl command on them to check. | 05:25 |
| sorin-mihai | acording to openssl /etc/kolla/certificates/haproxy-internal.pem: OK | 05:25 |
| Tony31 | you need to validate them both together as a chain | 05:26 |
| Tony31 | a bit more about certs (because no one told me this before and I used to have a hard time understanding it before I figured this out) | 05:26 |
| sorin-mihai | yup | 05:26 |
| sorin-mihai | openssl verify -CAfile /etc/kolla/certificates/ca/haproxy-internal.crt /etc/kolla/certificates/haproxy-internal.pem | 05:26 |
| sorin-mihai | yeah, i never went this path with certs, they usually work. lol | 05:27 |
| Tony31 | I think that verifiy command is right... give me a moment I can do this on my local system with some certs I have | 05:28 |
| *** evrardjp has quit IRC | 05:34 | |
| Tony31 | ok - when the cert verifies fine, I just get `cert: OK` with openssl | 05:34 |
| *** evrardjp has joined #openstack-kolla | 05:34 | |
| Tony31 | then I picked a different cert that was not signed by the CA and I get `verification failed` | 05:34 |
| sorin-mihai | indeed. but i get OK with openssl and then failed only for keystone | 05:52 |
| *** skramaja has joined #openstack-kolla | 05:52 | |
| Tony31 | where is the cert stored for keystone? | 05:54 |
| sorin-mihai | should be the same location as mentioned in globals.yml, right? | 05:56 |
| *** goldyfruit has quit IRC | 06:00 | |
| *** goldyfruit has joined #openstack-kolla | 06:00 | |
| Tony31 | I am not sure about it :/ | 06:01 |
| Tony31 | kayobe guys - does `internal_net_name:` and `external_net_names:` need to match the same network? | 06:06 |
| *** vmixor has joined #openstack-kolla | 06:13 | |
| *** vmixor has quit IRC | 06:27 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add CentOS 8 ceph-ansible job https://review.opendev.org/706886 | 06:32 |
| Tony31 | kolla guys - is it possble to get sudo access in the containers? | 06:41 |
| *** hu_berlin_kalle has joined #openstack-kolla | 06:59 | |
| *** rgogunskiy has joined #openstack-kolla | 07:00 | |
| Tony31 | I have `neutron_external_interface: "p-eth0-ovs,p-eth5-ovs"` and `p-eth5-ovs` is not working. Are there any bugs for this? | 07:00 |
| *** cah_link has joined #openstack-kolla | 07:07 | |
| *** sri_ has joined #openstack-kolla | 07:17 | |
| sri_ | Tony31, Hey to get root access docker exec -it -u root <container> | 07:18 |
| Tony31 | thanks sri_ | 07:20 |
| *** sluna has joined #openstack-kolla | 07:28 | |
| openstackgerrit | Xinliang Liu proposed openstack/kolla-ansible stable/rocky: Use become for kill command https://review.opendev.org/707772 | 07:30 |
| mnasiadka | yoctozepto: https://zuul.opendev.org/t/openstack/build/d50bd5f73aca49518fe577edf4b4ced0/log/primary/logs/ansible/deploy#70421 - any idea how to get rid of that? make haproxy check wsrep status? | 07:36 |
| mnasiadka | yoctozepto: or fine tune gmcast.peer_timeout and let's see if that helps | 07:37 |
| Tony31 | Kayobe guys - something amiss with external networking and I cant figure it out. Basically, flat network doesnt work | 07:39 |
| *** shyamb has joined #openstack-kolla | 07:41 | |
| Tony31 | when I look at a working config on a different system, I have br-ex and within that I have the real system ethernet "port" eth5. On the Kayobe deployment, I dont get any real ethX interfaces in the bridge. But I do have `phy-eth5-ovs` | 07:42 |
| mnasiadka | and what's wrong with having a virtual interface in the ovs bridge? | 07:45 |
| mnasiadka | please post brctl show and ip link command outputs to paste.openstack.org | 07:45 |
| Tony31 | I'm also confused why the controller/network node has a bridge for eth0 which is the interface used for admin_oc_net_name: / oob_oc_net_name: / oob_wl_net_name: / provision_wl_net_name: / storage_mgmt_net_name: / ceph_storage_mgmt_net_name: | 07:46 |
| Tony31 | thx will do now | 07:47 |
| *** sorin-mihai has quit IRC | 07:47 | |
| Tony31 | which container do I need to do this within? | 07:47 |
| Tony31 | neutron_openvswitch_agent ? | 07:47 |
| *** shyamb has quit IRC | 07:47 | |
| hu_berlin_kalle | tony31, i think it's neutron_openvswitchd (if not it was indeed the agent) | 07:49 |
| Tony31 | no such command brctl show | 07:49 |
| Tony31 | does not recognise brctl | 07:49 |
| hu_berlin_kalle | tony31, ahh brctl should just work on the controller itself | 07:49 |
| hu_berlin_kalle | the ovs specific stuff is in neutron_openvswitchd | 07:50 |
| Tony31 | `[root@juc-kcont1-prd ~]# brctl show | 07:50 |
| Tony31 | docker :) | 07:50 |
| hu_berlin_kalle | and about that bridge for eth0 - why shouldn't it be on the controller. the controller needs access to the admin_oc_net | 07:52 |
| Tony31 | mnasiadka is this right? http://paste.openstack.org/show/789558/ | 07:53 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707776 | 07:54 |
| *** ivve has joined #openstack-kolla | 07:56 | |
| yoctozepto | 08:36:17 <mnasiadka> yoctozepto: https://zuul.opendev.org/t/openstack/build/d50bd5f73aca49518fe577edf4b4ced0/log/primary/logs/ansible/deploy#70421 - any idea how to get rid of that? make haproxy check wsrep status? | 07:56 |
| yoctozepto | internets knowledge is to ditch haproxy | 07:56 |
| yoctozepto | in favor of a specialistic proxy | 07:56 |
| yoctozepto | 08:37:56 <mnasiadka> yoctozepto: or fine tune gmcast.peer_timeout and let's see if that helps | 07:57 |
| Tony31 | hu_berlin_kalle - no, I mean I have configured the external network for eth5 and it's not working. I dont really care about eth0 at this point other than I can't understand why there's a bridge there | 07:57 |
| yoctozepto | hmm, but the issue seems quite real, it's usually accompanied by other network-looking issues | 07:57 |
| *** xinliang has quit IRC | 07:58 | |
| hu_berlin_kalle | tony31 can you also paste the relvant config files (networks.yml and whatever groupvars or hostvars you have set and if relevant changes to controllers.yml) | 07:58 |
| Tony31 | yes I can | 07:59 |
| yoctozepto | mnasiadka: you think something held networking for more than 3 seconds? | 07:59 |
| Tony31 | what I have done now is make `external_net_names:` and `public_net_name` be the same network because I was not sure if they need to be the same. | 07:59 |
| *** pbing19 has joined #openstack-kolla | 08:00 | |
| hu_berlin_kalle | they don't (but it shouldn't be a problem either | 08:00 |
| Tony31 | ok good, thank you for confirming | 08:00 |
| Tony31 | networks.yml: http://paste.openstack.org/show/789559/ | 08:00 |
| *** bengates has joined #openstack-kolla | 08:01 | |
| Tony31 | group_vars for controllers: http://paste.openstack.org/show/789560/ | 08:01 |
| Tony31 | I believe that's it. There's no host_vars at all for kayobe | 08:02 |
| Tony31 | I've run tcpdump on eth5 and I can see ARP requests coming in from the network for the virtual router but there is not any response. The interface IP addresses for eth5 are pingable from the network | 08:05 |
| Tony31 | so seems there is some issue linking that interface eth5 to the bridge and into openstack | 08:06 |
| Tony31 | the virtual router has IP `192.168.20.252` and the interface IPs are in the `192.168.20.0/24` range. I've also tried by changing `networks.yml` to comment out the section which provides the IP addresses to eth5 and then reconfigure / redeploy but the same issue occurs | 08:07 |
| hu_berlin_kalle | tony31, hmm I think you need to configure the external network as a bridge. (https://docs.openstack.org/kayobe/latest/configuration/network.html Section "Neutron Networking") | 08:08 |
| *** sluna has quit IRC | 08:08 | |
| hu_berlin_kalle | at least that's what i did to resolve this problem | 08:09 |
| *** sluna has joined #openstack-kolla | 08:09 | |
| Tony31 | hu_berlin_kalle please could you explain some more? I understand what a bridge is but I dont quite understand what you mean just yet | 08:09 |
| hu_berlin_kalle | if the external network should link be able to reach the outside world via eth5 you'll need a device were eth5 and the ovs-stuff from Openstack con both plug into | 08:11 |
| hu_berlin_kalle | so instead of using eth5 directly you provide a bridge an plug eth5 into it | 08:12 |
| Tony31 | I understood that is the purpose of `networks.yml` `external_net_names:` ? | 08:12 |
| hu_berlin_kalle | osext_interface: brext | 08:12 |
| hu_berlin_kalle | osext_bridge_ports: | 08:12 |
| hu_berlin_kalle | - eth5 | 08:12 |
| Tony31 | I can try it :) thanks for the tip | 08:13 |
| Tony31 | so how does this fit with ovs ? | 08:15 |
| hu_berlin_kalle | p-eth5-phy will also be plugged into the brext | 08:16 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla-ansible master: Add support for encrypting nova/heat api https://review.opendev.org/707131 | 08:16 |
| mnasiadka | yoctozepto: well, it's nodepool - you don't know what is running there and how often networking fails :) | 08:16 |
| mnasiadka | yoctozepto: finetuning that in CI would be probably easier than moving to a specialistic proxy | 08:17 |
| hu_berlin_kalle | tony31, btw there is a problem with this config and vlans but i haven't tried your specific way to configure it (vlan config in networks.yml) | 08:17 |
| yoctozepto | mnasiadka: due to the way galera works I believe this may require tuning other timeouts in galera and plain mariadb as well | 08:18 |
| yoctozepto | mnasiadka: but +1 from my side | 08:18 |
| mnasiadka | Tony31: not really, eth0/5-ovs is a bridge, and I think it should be p-eth0/5-ovs as the interface in ovs (it's a veth connected to that bridge) | 08:19 |
| mnasiadka | yoctozepto: I found some similar bug/commit in charms, they only changed that and the problem in CI in theory went away | 08:19 |
| yoctozepto | mnasiadka: the truth is I can at least get these come from real networking problems, the thing with keepalived is not that nice | 08:19 |
| yoctozepto | mnasiadka: oh, nice, maybe it just requires 3.1 seconds from time to time | 08:20 |
| mnasiadka | yoctozepto: I'm starting to think that cpu and i/o in CI is very constrained :) | 08:20 |
| Tony31 | mnasiadka - so this is the confusing part :) | 08:20 |
| yoctozepto | could be io sure | 08:20 |
| Tony31 | mnasiadka `ovs` has `bridge_mappings = physnet1:eth0-ovs,physnet2:eth5-ovs` | 08:21 |
| Tony31 | so should it be `bridge_mappings = physnet1:eth0-ovs,physnet2:phy-eth5-ovs` | 08:21 |
| Tony31 | `phy-eth5-ovs` | 08:21 |
| Tony31 | OR should I add a bridge config to the host_vars/group_vars | 08:22 |
| mnasiadka | Tony31: one moment, show output of "brctl show" | 08:24 |
| Tony31 | mnasiadka from the controller host or within a container? | 08:24 |
| mnasiadka | from the host | 08:24 |
| Tony31 | `[root@juc-kcont1-prd ~]# brctl show | 08:24 |
| mnasiadka | yoctozepto: when we're into timeouts - I need to bump up ceph-ansible docker pull timeout :) | 08:25 |
| yoctozepto | mnasiadka: why so? | 08:25 |
| mnasiadka | yoctozepto: it failed to docker pull ceph-daemon in 300s :) | 08:25 |
| yoctozepto | mnasiadka: 5 minutes? | 08:26 |
| yoctozepto | mnasiadka: is that a full-fledged os there? | 08:26 |
| yoctozepto | mnasiadka: wow, it's up to 500M | 08:27 |
| mnasiadka | yeah | 08:27 |
| yoctozepto | lolz | 08:27 |
| yoctozepto | I guess double that timeout and we are done, more than that is just silly | 08:27 |
| mnasiadka | yeah, doubled - let's see | 08:29 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Add CentOS 8 ceph-ansible job https://review.opendev.org/706886 | 08:29 |
| mnasiadka | Tony31: I think you pasted only first line with the command :) | 08:29 |
| Tony31 | mnasiadka I think you mean I should be seeing some other bridge there, right? | 08:30 |
| Tony31 | All there is, is the docker0 bridge | 08:30 |
| mnasiadka | yeah, you should | 08:32 |
| mnasiadka | paste your network interfaces config on paste.openstack.org - I mean the kayobe one | 08:32 |
| Tony31 | I added this in group_vars: `[root@juc-kcont1-prd ~]# brctl show | 08:32 |
| Tony31 | p-brext-phy | 08:32 |
| Tony31 | mnasiadka http://paste.openstack.org/show/789562/ <- current group_vars config for the controller node (network node) | 08:33 |
| hrw | morning | 08:33 |
| hrw | I found one nasty thing related to UNBUILDABLE_IMAGES... | 08:33 |
| yoctozepto | mnasiadka: https://review.opendev.org/707776 | 08:34 |
| patchbot | patch 707776 - kolla-ansible - Fix Cinder Backup access to kernel modules (iscsi_... - 1 patch set | 08:34 |
| hrw | it can not be changed without patching code ;( | 08:34 |
| yoctozepto | hrw: nasty! | 08:34 |
| mnasiadka | Tony31: well, you don't have external_net_interface set - so I guess it won't do the networking config properly :) | 08:34 |
| Tony31 | mnasiadka - yes I do it's called `osext_interface:` and it used to be `osext_interface: eth5` | 08:35 |
| *** xinliang has joined #openstack-kolla | 08:36 | |
| mnasiadka | Tony31: and from where Kayobe should now, that it's the external_net_interface? | 08:36 |
| *** tonythomas has joined #openstack-kolla | 08:36 | |
| mnasiadka | Tony31: unless you override this https://github.com/openstack/kayobe/blob/aea7117764ee3ced0ccb7477c5c962569fa1c926/ansible/group_vars/all/network#L32 (which I don't think is a good idea) | 08:37 |
| Tony31 | mnasiadka from `networks.yml` I have set `external_net_names: osext` and in the `group_vars` I had `osext_interface: eth5` which was not working. | 08:37 |
| Tony31 | mnasiadka - no, not overrriding (we had this chat yesterday :) ) | 08:37 |
| mnasiadka | Tony31: I haven't done it this way in my life, so I can't tell you what might be wrong - just use the normal names :) | 08:38 |
| Tony31 | whats the normal names? 🙈 | 08:39 |
| mnasiadka | hrw: a good rewrite is waiting? | 08:39 |
| mnasiadka | Tony31: set internal_net_interface: eth0, external_net_interface: eth5 ? | 08:40 |
| hrw | mnasiadka: yamling would be a possible way | 08:40 |
| Tony31 | mnasiadka ok - i'll re-do the networks.yml and the group_vars... maybe I misunderstood this whole thing. | 08:41 |
| mnasiadka | yoctozepto: lot of polish +1s :) | 08:42 |
| yoctozepto | mnasiadka: add yours +2 | 08:42 |
| yoctozepto | hrw: https://review.opendev.org/707776 you too | 08:42 |
| patchbot | patch 707776 - kolla-ansible - Fix Cinder Backup access to kernel modules (iscsi_... - 1 patch set | 08:42 |
| yoctozepto | Poland taking over, woop woop | 08:42 |
| Tony31 | mnasiadka - the docs say that `admin_oc_net_name: ` means a prefix and I have specified the prefix of `inside` so then I am using `[prefix]_cidr:` etc. So I have it as per the docs explain it :/ | 08:43 |
| Tony31 | so `admin_oc_net_name: inside` means I then need to configure the cidr etc like `inside_cidr: 192.168.7.0/24` | 08:44 |
| mnasiadka | Tony31: as I said, I have never used it this way, don't get me wrong - it might work, but if it doesn't I don't know how to help you :) | 08:45 |
| Tony31 | mnasiadka I understand that, but is the documentation wrong | 08:45 |
| mnasiadka | Tony31: well, feel free to raise a change to kayobe docs :) | 08:45 |
| hu_berlin_kalle | tony31, sorry to barge in again: the kayobe network configuration guide really explicitly states you should a bridge there. And I think the issue you are facing is the exact reason for it. (It's in this guide since somewhere arround Stein and I just stumbeled across it last week facing the exact same issue.) | 08:46 |
| Tony31 | https://docs.openstack.org/kayobe/train/configuration/network.html#neutron-networking | 08:46 |
| cosmicsound | good day | 08:47 |
| Tony31 | mnasiadka - happy to do it, but so far, not confirmed any issue with doc :) | 08:47 |
| Tony31 | hu_berlin_kalle - I am with you on this. I was assuming that this bridge was managed in the container and I couldnt run any commands there because of missing sudo for last 2 days. But now I know how to do that :) | 08:47 |
| mnasiadka | hrw: got a moment for https://review.opendev.org/#/c/707603/ ? ;-) | 08:47 |
| patchbot | patch 707603 - kolla - Remove kolla Ceph container images - 5 patch sets | 08:47 |
| Tony31 | hu_berlin_kalle what name did you call your bridge? I called it brext and I'm not sure if that is mapping correctly | 08:48 |
| hrw | give me few minutes. kibana got prio | 08:49 |
| hrw | mnasiadka: -1 | 08:50 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Remove kolla Ceph container images https://review.opendev.org/707603 | 08:51 |
| *** shyamb has joined #openstack-kolla | 08:51 | |
| mnasiadka | hrw: thanks, updated :) | 08:51 |
| hrw | yoctozepto: +2+w | 08:51 |
| yoctozepto | hrw: thx | 08:51 |
| hrw | mnasiadka: I would do s/kolla cli/kolla/ but let it be | 08:51 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: Remove kolla Ceph container images https://review.opendev.org/707603 | 08:52 |
| mnasiadka | hrw: updated again :D | 08:52 |
| yoctozepto | lol, I was just about to mention cli | 08:53 |
| hrw | mnasiadka: repos.yml | 08:53 |
| hrw | mnasiadka: it has ceph repos which may not be no longer needed | 08:53 |
| yoctozepto | hmm, worth checking but they all probably provide client code | 08:53 |
| hrw | mnasiadka: but no. they are needed for deps for clients | 08:54 |
| hrw | +2 | 08:54 |
| yoctozepto | and off it goes | 08:54 |
| mnasiadka | yeah, was writing they are needed | 08:54 |
| mnasiadka | yoctozepto: and N years of ceph history in kolla goes to trash ;) | 08:55 |
| yoctozepto | mnasiadka: the one that worked for our users :_) | 08:55 |
| mnasiadka | yeah | 08:55 |
| mnasiadka | well, now we could get rid of two keys for nova | 08:55 |
| mnasiadka | but I guess it doesn't hurt for them to stay for some time | 08:57 |
| hrw | I love days when in the morning I get bug report, reply 'report upstream please' and then it gets RESOLVED INVALID as fix was found by reporter | 08:57 |
| Tony31 | mnasiadka - OK this is working now. Thanks a lot for all your help and time on this. I have spent 3 days on it approx 🤣 I can now ping the virtual router `Reply from 192.168.20.252: bytes=32 time=1ms TTL=63` The fix was to add the bridge config in the 'group_vars' like hu_berlin_kalle explained and then 1) configure host and 2) kayobe | 09:01 |
| Tony31 | reconfigure | 09:01 |
| Tony31 | ARP on the core router `192.168.20.252 00:00:01 fa16.3ea1.ac76 Vlan20 | 09:01 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: kibana: enable for non-x86 on Debian/Ubuntu https://review.opendev.org/707787 | 09:02 |
| hrw | Tony31: interesting emoji ;D | 09:02 |
| hrw | yoctozepto, mnasiadka: your turn :D | 09:02 |
| Tony31 | Thanks again for all the help you guys gave | 09:02 |
| Tony31 | now I need to talk about openstack router design.... | 09:03 |
| hrw | Tony31: can you take a look at docs does it needs fixing? | 09:04 |
| Tony31 | Could I voice it here? | 09:04 |
| Tony31 | hrw yes it could be improved. I had taken it that the purpose of `networks.yml` `external_net_names:` was to achieve this. Because in networks.yml it explains: `# List of names of networks used to provide external network access via# Neutron.` | 09:05 |
| mnasiadka | hrw: you want to say that it works on ppc64le as well? :) | 09:05 |
| hrw | mnasiadka: should | 09:05 |
| hrw | mnasiadka: it would even work on x86-64! ;D | 09:06 |
| mnasiadka | ERROR: Could not install packages due to an EnvironmentError: HTTPSConnectionPool(host='opendev.org', port=443): Max retries exceeded with url: /openstack/requirements/raw/branch/master/upper-constraints.txt (Caused by ResponseError('too many 500 error responses',))" | 09:07 |
| mnasiadka | awesome | 09:07 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla stable/train: kibana: enable for non-x86 on Debian/Ubuntu https://review.opendev.org/707789 | 09:13 |
| hrw | manual cherrypick was needed ;( | 09:13 |
| openstackgerrit | Kevin Zhao proposed openstack/kolla-ansible stable/stein: Haproxy: fix haproxy_cmd for Debian https://review.opendev.org/707790 | 09:18 |
| hrw | vote for ^^ | 09:22 |
| yoctozepto | hrw: 🤣 | 09:27 |
| yoctozepto | hrw: why backport without a merge? | 09:28 |
| *** Tony31 has quit IRC | 09:29 | |
| *** gfidente|afk is now known as gfidente | 09:29 | |
| hrw | yoctozepto: which one you mean? | 09:30 |
| yoctozepto | hrw: kibana | 09:30 |
| hrw | yoctozepto: I go for vacations in few hours. and my coworkers will need it while I am away ;( | 09:31 |
| hrw | yoctozepto: and unbuildable are not overrideable | 09:32 |
| yoctozepto | hrw: ok | 09:32 |
| *** shyamb has quit IRC | 09:33 | |
| *** shyamb has joined #openstack-kolla | 09:40 | |
| *** vmixor has joined #openstack-kolla | 09:41 | |
| *** aleccoder has joined #openstack-kolla | 09:46 | |
| *** klippo has joined #openstack-kolla | 09:46 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Junos switch: update ncclient to 0.6.7+ https://review.opendev.org/707796 | 09:47 |
| mgoddard | morning | 09:48 |
| hrw | hi mgoddard | 09:50 |
| *** shyamb has quit IRC | 09:52 | |
| *** bengates has quit IRC | 09:56 | |
| *** vmixor has quit IRC | 10:02 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: [community goal]: Add contributor and PTL guide https://review.opendev.org/707800 | 10:04 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla master: [community goal]: Add contributor and PTL guide https://review.opendev.org/707800 | 10:06 |
| hrw | bbl | 10:11 |
| *** vmixor has joined #openstack-kolla | 10:12 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Switch from shade to openstacksdk https://review.opendev.org/707689 | 10:20 |
| *** Tony31 has joined #openstack-kolla | 10:22 | |
| *** sri_ has quit IRC | 10:24 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Use become for kill command https://review.opendev.org/707772 | 10:30 |
| Tony31 | How are you guys providing floating IPs ? I think I misunderstand the concept of how this is supposed to work | 10:33 |
| mnasiadka | yoctozepto, mgoddard: https://review.opendev.org/#/c/706886/ | 10:34 |
| patchbot | patch 706886 - kolla-ansible - CI: Add CentOS 8 ceph-ansible job - 26 patch sets | 10:34 |
| yoctozepto | mnasiadka: green, nice | 10:36 |
| *** shyamb has joined #openstack-kolla | 10:36 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Allow setting gmcast.peer_timeout value https://review.opendev.org/707817 | 10:43 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 | 10:45 |
| *** bengates has joined #openstack-kolla | 10:48 | |
| mnasiadka | Tony31: floating ip is a NAT between external network and internal tenant (VM) network - 1:1 | 10:49 |
| mnasiadka | Tony31: other option is to use provider networks - direct vlan attachment for VMs | 10:50 |
| Tony31 | mnasiadka - This one is hard for me to explain what I am confused about. I have floating IPs working at the moment but it uses proxy arp and a network VLAN interface with a secondary address. | 10:50 |
| Tony31 | I was looking for a simpler way. | 10:50 |
| Tony31 | So I thought, one virtual router in openstack with IP `192.168.20.250` then on the network switch side I can add a single route for the floating IP network, like `route 192.168.23.0/24 192.168.20.250` | 10:51 |
| Tony31 | basically, sending 192.168.23.0/24 to the openstack virtual router | 10:51 |
| Tony31 | but I think I would need to do this for each project... so not that great | 10:52 |
| Tony31 | so wanted to ask how you guys are doing the floaties. Do you have one large "network" in openstack, which is external. And then project routers and floating ip's are all assigned from there? | 10:52 |
| Tony31 | The "provider" networks has a use case, but the floating ip's allow you to move ip's to other VMs which also can be good | 10:53 |
| Tony31 | I am probably thinking too deep on this one | 10:54 |
| *** mixor has joined #openstack-kolla | 10:55 | |
| Tony31 | Another option I could do, which I dont like - add the one flat network in openstack. Then each project, have the project router attach to this network. Then on the network side, I would need to forward host routes for each floating IP, to each project router. This means first checking which IP each router has, then going to the core routers to | 10:56 |
| Tony31 | create the host routes for each of the projects floating IPs | 10:56 |
| *** vmixor has quit IRC | 10:56 | |
| Tony31 | I guess the simplest way Ive thought of is, simply have the one flat external network `/24`. Add a router for each project and attach to the external network. Then assign floating ip's from the same external network. | 10:58 |
| Tony31 | The network will request ARP for the floating IP and the project routers will respond | 10:58 |
| Tony31 | this is a layer 2 design rather than a layer 3 routed design | 10:58 |
| Tony31 | I just googled layer 3 floating ip and have some things to read. Thanks for this discussion :) | 10:59 |
| *** mixor has quit IRC | 11:06 | |
| Wellie | can you check this ? https://review.opendev.org/#/c/707379/ | 11:10 |
| patchbot | patch 707379 - kolla-ansible - Cloudkitty cant not conncet to Auth - 2 patch sets | 11:10 |
| *** rlljorge has joined #openstack-kolla | 11:11 | |
| rlljorge | Hello there, Someone can help-me with cloudkitty ... I deployed using kolla-ansible kolla-ansible 8.1.0 and I don't receive nenhum data on reports. I created the service/hasmaps. | 11:15 |
| rlljorge | I am stopped a couple of days | 11:16 |
| rlljorge | I cannot get any erros on logs | 11:17 |
| Wellie | rlljorge: can you show on your node what the cloudkitty-processor say | 11:18 |
| Wellie | rlljorge: docker logs cloudkitty_processor | 11:19 |
| rlljorge | Wellie sure give a second | 11:19 |
| Wellie | rlljorge: no problem | 11:20 |
| rlljorge | Wellie http://paste.openstack.org/show/789566/ | 11:23 |
| Wellie | rlljorge what say docker logs? | 11:25 |
| Wellie | rlljorge not file logs | 11:25 |
| rlljorge | Sorry this is the correct log ? http://paste.openstack.org/show/789567/ | 11:26 |
| Wellie | rlljorge yes what is your version? | 11:27 |
| rlljorge | kolla-ansible 8.1.0 | 11:28 |
| Wellie | rlljorge openstack version? | 11:28 |
| rlljorge | * stable/stein | 11:29 |
| Wellie | rlljorge I suspect it has to do with the deprecated you see in the log | 11:30 |
| Wellie | rlljorge I'm not sure | 11:30 |
| rlljorge | UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>. | 11:31 |
| Wellie | no | 11:32 |
| Wellie | 2020-02-13 19:28:12.368 6 WARNING oslo_config.cfg [-] Deprecated: Option "auth_section" from group "keystone_fetcher" is deprecated. Use option "auth_section" from group "fetcher_keystone". | 11:32 |
| Wellie | 2020-02-13 19:28:12.445 6 WARNING oslo_config.cfg [-] Deprecated: Option "keystone_version" from group "keystone_fetcher" is deprecated. Use option "keystone_version" from group "fetcher_keystone". | 11:32 |
| Wellie | 2020-02-13 19:28:13.176 6 WARNING oslo_config.cfg [-] Deprecated: Option "auth_section" from group "gnocchi_collector" is deprecated. Use option "auth_section" from group "collector_gnocchi". | 11:32 |
| Wellie | 2020-02-13 19:28:13.180 6 WARNING oslo_config.cfg [-] Deprecated: Option "region_name" from group "gnocchi_collector" is deprecated. Use option "region_name" from group "collector_gnocchi". | 11:32 |
| rlljorge | Wellie this is a kolla issue ? | 11:35 |
| Wellie | rlljorge It probably can't tell you exactly, just wait. What others say | 11:36 |
| yoctozepto | deprecated usually still work | 11:39 |
| openstackgerrit | Yongjun Bai proposed openstack/kolla master: Add Apache packages to glance containers https://review.opendev.org/707306 | 11:40 |
| openstackgerrit | Merged openstack/kolla-ansible master: CI: Add CentOS 8 ceph-ansible job https://review.opendev.org/706886 | 11:49 |
| *** pbing19 has quit IRC | 11:57 | |
| *** shyamb has quit IRC | 11:59 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Revert "Use OpenStack Train release" https://review.opendev.org/701747 | 12:01 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: Use python3 for local kolla-ansible execution https://review.opendev.org/705000 | 12:01 |
| *** shyamb has joined #openstack-kolla | 12:01 | |
| *** kplant has joined #openstack-kolla | 12:04 | |
| rlljorge | openstack rating report tenant list, not return any values ... maybe this is a problem | 12:06 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make local kolla-ansible Python executable configurable https://review.opendev.org/705000 | 12:07 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Revert "Use OpenStack Train release" https://review.opendev.org/701747 | 12:08 |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: Allow setting gmcast.peer_timeout value https://review.opendev.org/707817 | 12:09 |
| kplant | mgoddard: https://bugs.launchpad.net/kolla-ansible/+bug/1863107 -- are you asking if something changed in my configuration? | 12:20 |
| openstack | Launchpad bug 1863107 in kolla-ansible ussuri "Stein->Train Upgrade : Old nova services not cleaned" [Medium,Triaged] | 12:20 |
| mgoddard | kplant: yes | 12:24 |
| mgoddard | or possibly in the kolla-ansible config | 12:24 |
| *** shyamb has quit IRC | 12:25 | |
| kplant | got it, will reply on lp | 12:26 |
| Wellie | mgoddard: haproxy.pem dir? | 12:32 |
| yoctozepto | rlljorge, Wellie: if you were so kind as to produce a report similar to this: https://bugs.launchpad.net/kolla-ansible/+bug/1863094 (obviously without analysis :-) ) | 12:35 |
| openstack | Launchpad bug 1863094 in kolla-ansible ussuri "Creating volume backup fails when iscsi_tcp module is not inserted when using LVM backend" [Medium,In progress] - Assigned to Radosław Piliszek (yoctozepto) | 12:35 |
| yoctozepto | this level of details allows us to reproduce and debug | 12:35 |
| Wellie | yoctozepto i have no bug im search the dir to deploy a tls certificate | 12:36 |
| yoctozepto | Wellie: sorry then, I saw you two talking about cloudkitty with rlljorge | 12:37 |
| Wellie | okay :-) | 12:37 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 12:38 |
| Wellie | i report bugs and fix the bug :D | 12:38 |
| mgoddard | Wellie: https://docs.openstack.org/kolla-ansible/latest/admin/advanced-configuration.html#tls-configuration | 12:39 |
| Wellie | mgoddard: what is node_config ? | 12:40 |
| mgoddard | Wellie: default is /etc/kolla/ | 12:41 |
| Wellie | mgoddard: thx | 12:41 |
| yoctozepto | 13:38:23 <Wellie> i report bugs and fix the bug :D | 12:41 |
| yoctozepto | that is praised! ;D | 12:41 |
| kplant | "i don't always report bugs... but when i do, i fix them" | 12:42 |
| yoctozepto | kplant: +3 | 12:48 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Haproxy: fix haproxy_cmd for Debian https://review.opendev.org/707790 | 12:53 |
| *** dave-mccowan has joined #openstack-kolla | 13:02 | |
| *** cah_link has quit IRC | 13:04 | |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: Ensure iscsi_tcp module is loaded for cinder-backup https://review.opendev.org/707838 | 13:05 |
| *** skramaja has quit IRC | 13:16 | |
| rlljorge | yoctozepto I open a bug about Cloudkitty | 13:21 |
| *** ivve has quit IRC | 13:28 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707776 | 13:28 |
| *** rgogunskiy has quit IRC | 13:52 | |
| *** rgogunskiy has joined #openstack-kolla | 13:53 | |
| *** rgogunskiy has quit IRC | 14:05 | |
| openstackgerrit | Michal Nasiadka proposed openstack/kolla-ansible master: CI: Fine tune Galera gmcast.peer_timeout to 15 seconds https://review.opendev.org/707818 | 14:13 |
| *** TrevorV has joined #openstack-kolla | 14:18 | |
| Tony31 | hi :) In kolla.yml for Kayobe, there are options like `kolla_enable_neutron_fwaas:` - are these functional? And is there documentation for how to configure? | 14:26 |
| mnasiadka | Tony31: have you read the limitations of neutron fwaas v2? https://docs.openstack.org/neutron/latest/admin/fwaas.html | 14:28 |
| openstackgerrit | Merged openstack/kolla master: Remove kolla Ceph container images https://review.opendev.org/707603 | 14:28 |
| mnasiadka | and I don't think it has any people actively developing it and making it better | 14:28 |
| Tony31 | That was the first one my mouse pointer highlighted, I didn't mean to pick that one intentionally. Are there any docs for `kolla_enable_neutron_vpnaas:` ? | 14:29 |
| Tony31 | So I should just stick to whats here on this page? https://docs.openstack.org/kolla-ansible/train/reference/index.html | 14:34 |
| *** pbing19 has joined #openstack-kolla | 14:39 | |
| mnasiadka | Tony31: that one is in similar state I think - I mean it might not survive a lot of cycles | 14:52 |
| mnasiadka | (development cycles = OpenStack releases) | 14:52 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Make local kolla-ansible Python executable configurable https://review.opendev.org/705000 | 14:54 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Revert "Use OpenStack Train release" https://review.opendev.org/701747 | 14:54 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Avoid writing out requirements.txt in kolla-ansible role https://review.opendev.org/707853 | 14:59 |
| mgoddard | Tony31: those flags generally just map to kolla flags | 15:00 |
| mgoddard | kolla_enable_neutron_fwaas -> enable_neutron_fwaas | 15:00 |
| *** rlljorge has quit IRC | 15:12 | |
| mnasiadka | mgoddard: should we make ceph-ansible jobs voting, or not? | 15:13 |
| mgoddard | mnasiadka: old ceph jobs were non-voting right/ | 15:14 |
| mnasiadka | mgoddard: might be, don't remember | 15:14 |
| mgoddard | maybe keep it the same | 15:14 |
| mnasiadka | ok | 15:14 |
| mgoddard | multinode jobs aren't always reliable | 15:15 |
| Tony31 | Hi mgoddard thanks for the info. How about `kolla_enable_freezer:` ? I dont see this listed in https://docs.openstack.org/kolla-ansible/latest/reference/index.html Would anything occur if I set this variable ? | 15:23 |
| mgoddard | Tony31: freezer is one of the less commonly used services. Setting the flag should enable the service, but there may be other dependencies to get it working | 15:24 |
| Tony31 | How would I need to go about that? Is it dependencies such as installing containers? Or configuration? | 15:25 |
| mgoddard | configuration | 15:25 |
| mgoddard | I'd suggest reading the freezer docs so you know how to use it, then just try enabling it and see what happens | 15:26 |
| Tony31 | thanks you :) | 15:27 |
| yoctozepto | mgoddard, mnasiadka: better yet, we don't know how relable c-a is ;-) | 15:29 |
| mnasiadka | yoctozepto: better tell me why such thing shows up in singlenode - https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_bd8/707818/2/check/kolla-ansible-ubuntu-source-upgrade/bd8c3d3/primary/logs/kolla/all-CRITICAL.txt | 15:30 |
| yoctozepto | mnasiadka: let's correlate time with the other events | 15:31 |
| yoctozepto | 2020-02-14 15:08:21.663 | 15:31 |
| *** vedup19 has joined #openstack-kolla | 15:31 | |
| yoctozepto | nothing in mariadb logs | 15:32 |
| yoctozepto | it was already past upgrade of mariadb | 15:32 |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: Avoid writing out requirements.txt in kolla-ansible role https://review.opendev.org/707853 | 15:32 |
| yoctozepto | mnasiadka: must have been late to log | 15:33 |
| *** pbing19 has quit IRC | 15:33 | |
| yoctozepto | it could be that mariadb was still restarting when it tried to access it | 15:33 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Introduce influxdb_datadir_volume https://review.opendev.org/707861 | 15:33 |
| yoctozepto | because wsrep is not synced immediately even with single node | 15:33 |
| openstackgerrit | Will Szumski proposed openstack/kolla-ansible master: Introduce influxdb_datadir_volume https://review.opendev.org/707861 | 15:36 |
| mnasiadka | yoctozepto: well, docker logs for mariadb say it was starter 15:01 - want to tell me it takes more than 7 minutes to launch it? :) | 15:37 |
| mnasiadka | *started 15:01 | 15:37 |
| yoctozepto | 2020-02-14 15:01:15 0 [Note] WSREP: Read nil XID from storage engines, skipping position init | 15:38 |
| yoctozepto | 2020-02-14 15:01:16 0 [Note] InnoDB: Buffer pool(s) load completed at 200214 15:01:16 | 15:38 |
| yoctozepto | theoretically it started up in one second | 15:38 |
| yoctozepto | 2020-02-14 15:02:26 88 [Warning] Aborted connection 88 to db: 'keystone' user: 'keystone' host: 'primary' (Got an error reading communication packets) | 15:38 |
| yoctozepto | first timeout hit | 15:38 |
| yoctozepto | 1:10 | 15:38 |
| mnasiadka | yeah, question is why :) | 15:38 |
| yoctozepto | so it started accepting | 15:39 |
| yoctozepto | but why such a late message from wsrep | 15:39 |
| yoctozepto | 2020-02-14 15:01:10 0 [Note] /usr/sbin/mysqld: Shutdown complete | 15:39 |
| yoctozepto | it had clean shutdown | 15:39 |
| yoctozepto | so it's like nothing mariadb gone wrong | 15:40 |
| yoctozepto | very weird | 15:40 |
| mnasiadka | maybe we need some more debug, bump log_warnings to 3/4/.../9? :D | 15:40 |
| yoctozepto | mnasiadka: whose param is that? | 15:40 |
| mnasiadka | yoctozepto: mariadb's | 15:41 |
| yoctozepto | mnasiadka: and currently we have..? | 15:41 |
| mnasiadka | default, which is 2 or 1 depending on version | 15:41 |
| mnasiadka | we don't have a config for mariadb? geez | 15:41 |
| yoctozepto | https://mariadb.com/resources/blog/what-exactly-does-log_warnings2-log/ | 15:41 |
| yoctozepto | we don't? | 15:42 |
| yoctozepto | you mean we don't have it overridable? | 15:42 |
| yoctozepto | mnasiadka | 15:42 |
| yoctozepto | neutron being lame about reporting things | 15:42 |
| yoctozepto | 2020-02-14 15:01:16.055 29 ERROR oslo_db.sqlalchemy.exc_filters [req-da3b8910-eb3b-4445-96cd-8a4f53e5aecd - - - - -] DBAPIError exception wrapped from (pymysql.err.InternalError) (1047, 'WSREP has not yet prepared node for application use') | 15:42 |
| mnasiadka | yoctozepto: well it's rather oslo that claims it's unrecoverable error | 15:43 |
| mnasiadka | question if it did retry the connection a couple of times | 15:43 |
| yoctozepto | 2020-02-14 14:45:22.591 6 DEBUG neutron.plugins.ml2.plugin [req-b7615e16-e470-4444-8342-86fca030273c - - - - -] neutron.plugins.ml2.plugin.Ml2Plugin method _start_rpc_notifiers called with arguments () {} wrapper /var/lib/kolla/venv/lib/python3.6/site-packages/oslo_log/helpers.py:66 | 15:43 |
| yoctozepto | mnasiadka: it's probably correct | 15:43 |
| yoctozepto | check this one out | 15:43 |
| yoctozepto | it's a very old "req" | 15:43 |
| yoctozepto | so it's a permanent one | 15:43 |
| yoctozepto | and it most likely got stuck | 15:44 |
| yoctozepto | due to that error | 15:44 |
| yoctozepto | maybe let's catch your friend for a quick chat some time today or next week | 15:44 |
| yoctozepto | so he could explain how it works (or should) | 15:44 |
| yoctozepto | 2020-02-14 15:08:21.663 29 CRITICAL neutron [req-b7615e16-e470-4444-8342-86fca030273c - - - - -] Unhandled error: oslo_db.exception.DBError: (pymysql.err.InternalError) (1047, 'WSREP has not yet prepared node for application use') | 15:46 |
| yoctozepto | this one is probably due to timeout and it happened at that 15:01:16 as mariadb was not exactly ready then yet | 15:46 |
| yoctozepto | it's a very small window that we hit from time to time | 15:47 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/train: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707863 | 15:47 |
| mnasiadka | yoctozepto: but those aborted connections are also interesting | 15:47 |
| mnasiadka | 'error reading communication packets' | 15:47 |
| yoctozepto | mnasiadka: increase haproxy timeout and they are gone | 15:47 |
| yoctozepto | yeah, haproxy kills them, hence why | 15:48 |
| mnasiadka | mhm | 15:48 |
| yoctozepto | these are always recoverable it seems | 15:48 |
| yoctozepto | so not a big deal | 15:48 |
| yoctozepto | just logs cruft | 15:48 |
| mnasiadka | ok, I'll look into bumping up some debug for mariadb next week | 15:51 |
| mnasiadka | this is starting to be... weird | 15:51 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/stein: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707866 | 15:51 |
| mnasiadka | wonder it's only neutron :) | 15:51 |
| yoctozepto | mnasiadka: it's not | 15:52 |
| yoctozepto | mnasiadka: but you have a friend that we can ask about it specifically | 15:52 |
| yoctozepto | and well, neutron is one of the most talkative | 15:52 |
| yoctozepto | the best is placement and it was silenced | 15:52 |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible stable/rocky: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707867 | 15:53 |
| *** bengates has quit IRC | 16:06 | |
| *** TrevorV has quit IRC | 16:09 | |
| hrw | https://review.opendev.org/707787 - who will vote? zuul gave +1 | 16:13 |
| patchbot | patch 707787 - kolla - kibana: enable for non-x86 on Debian/Ubuntu - 1 patch set | 16:13 |
| dking_desktop | This is probably more of an ironic question, but does anybody here know, when creating a baremetal server (openstack server create ...), where should the DHCP response come from to initiate the PXE boot to deploy_ramdisk, etc.? | 16:28 |
| mgoddard | dking_desktop: neutron dhcp agent on the provisioning network | 16:34 |
| dking_desktop | mgoddard: Could you help suggest how I could troubleshoot that? I'm in the network namespace for the qdhcp-<UUID of the provisioning_network>, but it ignores the requests. I'm not sure where to look for its configuration files yet. | 16:36 |
| mgoddard | dking_desktop: so you can see dhcp requests arriving? | 16:36 |
| mgoddard | possibly the port did not bind? check neutron logs | 16:37 |
| mgoddard | have you set a physical_network on the baremetal port? For flat networks this is necessary to enable binding | 16:37 |
| dking_desktop | Yes, I can, and it even logs them: DHCPDISCOVER(tapa3b6faa2-e0) <MAC> no address available | 16:37 |
| dking_desktop | Would the port issue be ruled out if introspect worked? AS for the network, I created the provisioning network as a flat network for physnet1. For the baremetal port create, I only set the MAC address and node ID. | 16:40 |
| dking_desktop | So, I should have used "--physical-network physnet1" when creating the port? | 16:40 |
| mgoddard | dking_desktop: yes | 16:44 |
| mgoddard | I recently updated our ironic docs to include it | 16:44 |
| mgoddard | as someone else had the same issue | 16:44 |
| dking_desktop | Okay. That shouldn't be hard to try. Thanks, I'll try that now. | 16:46 |
| mgoddard | also check neutron-server logs for errors | 16:48 |
| *** noxoid has joined #openstack-kolla | 16:52 | |
| dking_desktop | Oh, there's something: Failed to bind port ... on host ... for vnic_type baremetal using segments [{'network_id': '...', 'segmentation_id': 74, 'physical_network': None, 'id': '...', 'network_type': u'vxlan'}] | 16:56 |
| dking_desktop | I'm also adding "--pxe-enabled" just in case. | 17:02 |
| yoctozepto | mgoddard: precheck in a followup or squasg? | 17:24 |
| mgoddard | yoctozepto: it's a separate issue | 17:24 |
| yoctozepto | mgoddard: ack, thanks, | 17:24 |
| mgoddard | dking_desktop: looks like you're using a vxlan network | 17:25 |
| mgoddard | better use the flat network | 17:25 |
| dking_desktop | mgoddard: Where would I specify that? | 17:26 |
| mgoddard | when you create the instance, use the flat network | 17:26 |
| mgoddard | --network <net> | 17:26 |
| noxoid | dking_desktop, its what i was telling you in slack | 17:28 |
| *** chrizl has quit IRC | 17:33 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #openstack-kolla | 17:34 | |
| dking_desktop | That certainly did something! | 17:41 |
| *** igordc has joined #openstack-kolla | 17:43 | |
| openstackgerrit | Mark Goddard proposed openstack/kayobe master: WIP: CentOS 8 https://review.opendev.org/707690 | 17:54 |
| *** gfidente has quit IRC | 18:04 | |
| *** iniazi has joined #openstack-kolla | 18:13 | |
| dking_desktop | mgoddard: Thank you for your help! That did it. I finally have a baremetal node provisioned. It's not connected to any other networks, but at least it's up. | 18:18 |
| *** vedup19 has quit IRC | 18:30 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix RabbitMQ hostname address resolution precheck https://review.opendev.org/707892 | 18:31 |
| yoctozepto | hrw, mnasiadka, osmanlicilegi: https://review.opendev.org/#/q/topic:bug/1863094+(status:open) a quickie for cinder-backup | 18:41 |
| openstackgerrit | Chason Chan proposed openstack/kolla-ansible master: [Docs] Pin kolla-anisble to the same version of quickstart guide https://review.opendev.org/707896 | 18:45 |
| *** tonythomas has quit IRC | 18:45 | |
| *** negronjl has quit IRC | 19:06 | |
| *** negronjl has joined #openstack-kolla | 19:10 | |
| *** riuzen has joined #openstack-kolla | 19:15 | |
| riuzen | TASK [baremetal : Generate /etc/hosts for all of the nodes] ************************************************************************************ | 19:16 |
| riuzen | fatal: [node0]: FAILED! => {"msg": "The task includes an option with an undefined variable. The error was: 'ansible.vars.hostvars.HostVars object' has no attribute u'node0'\n\nThe error appears to have been in '/home/kolla/.local/ansible/roles/baremetal/tasks/pre-install.yml': line 31, column 3, but may\nbe elsewhere in the file depending on the exact syntax problem. | 19:16 |
| yoctozepto | riuzen: seems node0 failed to gather facts | 19:17 |
| riuzen | encounter that error when bootstrap-server. Has been deploy before with no error but today I trying to deploy again and that error appear. Anyone know how to resolve that? | 19:17 |
| riuzen | yeah, im wondering why that happende. Kolla-ansible 9.0 | 19:18 |
| riuzen | sorry, i found the problem.still using old ansible. | 19:23 |
| *** vedup19 has joined #openstack-kolla | 19:25 | |
| openstackgerrit | Radosław Piliszek proposed openstack/kolla-ansible master: Fix RabbitMQ hostname address resolution precheck https://review.opendev.org/707892 | 19:26 |
| openstackgerrit | Merged openstack/kayobe master: Make local kolla-ansible Python executable configurable https://review.opendev.org/705000 | 19:36 |
| openstackgerrit | Merged openstack/kayobe master: Revert "Use OpenStack Train release" https://review.opendev.org/701747 | 19:49 |
| *** riuzen has quit IRC | 19:53 | |
| *** igordc has quit IRC | 20:15 | |
| *** kplant has quit IRC | 20:36 | |
| hrw | yoctozepto: done | 20:55 |
| *** dave-mccowan has quit IRC | 21:23 | |
| openstackgerrit | James Kirsch proposed openstack/kolla-ansible master: Add support for encrypting backend HAProxy traffic https://review.opendev.org/664516 | 21:30 |
| openstackgerrit | Merged openstack/kolla-ansible stable/stein: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707866 | 22:01 |
| openstackgerrit | Merged openstack/kolla-ansible stable/train: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707863 | 22:17 |
| openstackgerrit | Merged openstack/kolla-ansible stable/rocky: Fix Cinder Backup access to kernel modules (iscsi_tcp issue) https://review.opendev.org/707867 | 22:17 |
| *** Tony31 has quit IRC | 23:03 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!