| spsurya | morning | 00:01 |
|---|---|---|
| *** rwsu has quit IRC | 00:01 | |
| kfox1111 | ok... another step closer to figuring out why the 4 gates fail. | 00:05 |
| kfox1111 | I got the 3 gate to fail the same way. :) | 00:05 |
| kfox1111 | seems like a credential mismatch. | 00:05 |
| kfox1111 | so 99% there. :) | 00:06 |
| *** eanylin has quit IRC | 00:09 | |
| *** zul has quit IRC | 00:11 | |
| kollian | fatal: [localhost]: FAILED! => {"attempts": 6, "changed": false, "cmd": ["docker", "exec", "mariadb", "mysql", "-h", "192.168.121.106", "-P", "3306", "-u", "haproxy", "-e", "show databases;"], "delta": "0:00:00.291034", "end": "2017-05-18 00:06:22.376496", "failed": true, "rc": 1, "start": "2017-05-18 00:06:22.085462", "stderr": "ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.121.106' (113 \"No route to host\" | 00:12 |
| kollian | anyone familiar this error | 00:13 |
| kollian | ? | 00:13 |
| masuberu | good morning all, is it possible to deploy magnum + ironic for bare metal deployments of kubernetes from a kolla-ansible openstack installation? | 00:13 |
| kollian | got during kolla-ansible deployment | 00:13 |
| masuberu | kollian, the error message is clear, it can't connect to 192.168.121.106 (mysql) | 00:14 |
| masuberu | check ip routing and port is open from "localhost" | 00:14 |
| kollian | masuberu i canping 192.168.121.106 | 00:15 |
| masuberu | is the mysql container up and running? | 00:15 |
| masuberu | can you access mysql on port 3306? | 00:16 |
| sbezverk | kfox1111: interesting way of troubleshooting ;) by breaking you should patent it ;) | 00:17 |
| * kfox1111 chuckles. | 00:18 | |
| masuberu | kollian, telnet 192.168.121.106 3306 from localhost an see if you can connect | 00:18 |
| kfox1111 | "sometimes things gotta get worse before they can get better" ;) | 00:18 |
| sbezverk | kfox1111: :) | 00:18 |
| Reepicheep | kollian: also check the mariadb logs | 00:19 |
| kollian | masuberu: No route to host | 00:19 |
| masuberu | kollian, that is your issue | 00:19 |
| Reepicheep | kollian: did it pass the prechecks? | 00:20 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 00:20 |
| kollian | masuberu: Reepicheep yes precheck passed | 00:20 |
| Reepicheep | masuberu: yes you should be able to deploy magnum + ironic | 00:20 |
| kollian | Reepicheep: masuberu do i need to forward the port ? | 00:21 |
| kfox1111 | sbezverk: so, in 3, it was honoriing the rbd_user=nova flag in the config. in 4, it ignores it. | 00:21 |
| Reepicheep | kollian: when you say you can ping the node is that from the nodes that are running haproxy? | 00:21 |
| kfox1111 | doing some diffs showed 3 using 'nova' and 4 using 'cinder'. | 00:22 |
| kfox1111 | so I changed the config to use 'cinder' and 3 broke the same as 4. | 00:22 |
| kfox1111 | so we just gotta figure out why the cinder user creds are wrong in nova. | 00:22 |
| kollian | Reepicheep: no proxy running | 00:22 |
| kollian | Reepicheep: i am using it on Vm | 00:22 |
| kollian | as free ip | 00:22 |
| kollian | the entry we provide in global.yml | 00:23 |
| Reepicheep | oh. yeah i see the localhost now | 00:23 |
| *** kbaegis has quit IRC | 00:25 | |
| sbezverk | kfox1111: hm interesting | 00:25 |
| kollian | Reepicheep: masuberu what should i do ? | 00:25 |
| *** zul has joined #openstack-kolla | 00:25 | |
| *** kbaegis has joined #openstack-kolla | 00:26 | |
| sbezverk | kfox1111: some config changes in newton? | 00:27 |
| Reepicheep | let me see if I got your layout .. you have a machine "localhost" that your are running kolla-ansible from and 192.168.121.106 is a vm on that host? | 00:27 |
| kfox1111 | I dug through. in ocata, they change who they listen too first. they now listen to what cinder tells them rather then whats in the nova config. | 00:27 |
| kollian | Reepicheep: I am doing AIO | 00:28 |
| kollian | with a vm | 00:28 |
| kollian | Reepicheep: 192.168.121.106 is the free ip | 00:29 |
| kollian | while actual ip is 192.168.121.221 | 00:29 |
| kollian | i.e of eth0 | 00:30 |
| kfox1111 | hmmm. that might be the issue.... | 00:30 |
| kollian | using 92.168.121.106 for the public ip which we provide as kolla_internal_vip_address | 00:31 |
| Reepicheep | kollian: the network_interface is set to the correct interface? | 00:32 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 00:33 |
| kollian | yes it is i guess | 00:33 |
| kollian | Reepicheep: | 00:33 |
| *** duonghq_ is now known as duonghq | 00:33 | |
| Reepicheep | and it's different then the neutron_external_interface? | 00:33 |
| kollian | Reepicheep: yes network interface is eth0 and neutron_external_interface is eth1 | 00:34 |
| kollian | Reepicheep: I provided the kolla_internal_vip_address=192.168.121.106 by running below command | 00:36 |
| Reepicheep | kollian: and the network_interface should be the interface that you have already setup with an IP within the subnet that your kolla_internal_vip_address in | 00:36 |
| kollian | sudo arp-scan -I eth0 192.168.121.221/24 | 00:36 |
| *** zul has quit IRC | 00:36 | |
| kollian | Reepicheep: any lead | 00:41 |
| kollian | ? | 00:41 |
| Reepicheep | what vm platform is it on? Is it possible that the network port is doing something cute with the interface that is preventing a subinterface from working correctly? | 00:42 |
| kollian | Reepicheep: hostos is ubuntu on whcih i am running vagrant and with vagrant launched vm with libvirt provider and into that vm i am deploying containerized openstack | 00:45 |
| *** jrist has joined #openstack-kolla | 00:45 | |
| *** jrist has quit IRC | 00:45 | |
| *** jrist has joined #openstack-kolla | 00:45 | |
| kollian | i would like to share my global.yml with with network interface | 00:45 |
| Reepicheep | hmm. AFAIK that should work, I do believe the vagrant stuff has not been worked on in quite a while though | 00:46 |
| Reepicheep | I actually only did an AIO once it worked and I immediately moved on to figuring out the multihost | 00:47 |
| kollian | Reepicheep: http://paste.openstack.org/show/609857/ | 00:48 |
| Reepicheep | what is the ip address on eth1? | 00:49 |
| *** jamesPR has joined #openstack-kolla | 00:50 | |
| kollian | 192.168.33.10 | 00:51 |
| kollian | Reepicheep: | 00:51 |
| Reepicheep | yeah.. I see that .. is it important? because it will quit working when kolla-ansible runs | 00:51 |
| kollian | Reepicheep: couldn'r parsed correctly | 00:52 |
| Reepicheep | what does your routing table look like? "ip route show" | 00:52 |
| kollian | [root@vmachine1 vagrant]# ip route show default via 192.168.121.1 dev eth0 proto static metric 100 default via 192.168.33.1 dev eth2 proto static metric 101 169.254.0.0/16 dev eth1 scope link metric 1003 172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 192.168.33.0/24 dev eth1 proto kernel scope link src 192.168.33.10 192.168.33.0/24 dev eth2 proto kernel scope link src 192.168.33.34 metr | 00:52 |
| kollian | Reepicheep: | 00:52 |
| *** unicell1 has quit IRC | 00:54 | |
| kollian | Reepicheep: I followed https://docs.openstack.org/project-deploy-guide/kolla-ansible/ocata/quickstart.html | 00:55 |
| Reepicheep | it looks like you have multiple default routes .. it may not be the issue but rarely is a desired thing | 00:55 |
| *** cuongnv has joined #openstack-kolla | 00:56 | |
| Reepicheep | I would try removing the ip address from eth1 and possible from eth2 depending on if it is needed or not | 00:58 |
| Reepicheep | then make sure you only have one default route | 00:59 |
| kollian | Reepicheep: I multiple route should not be the problem with kolla deployment | 01:00 |
| kollian | if it is | 01:00 |
| kollian | then it is bug i thing | 01:00 |
| kollian | think* | 01:00 |
| Reepicheep | then I would try adding a subinterface to eth0 with "ip addr add 192.168.121.106/24 dev eth0" and see if you can ping it (you should not be able to before) | 01:01 |
| Reepicheep | if that all works then remove that sub interface and make sure your prechecks pass and try again | 01:02 |
| Reepicheep | I agree the multiple default routes should not be an issue because that ip should match a direct attached route, it just tends to cause issues when traffic needs to use a router | 01:03 |
| kollian | able to ping | 01:05 |
| kollian | 192.168.121.106 | 01:05 |
| Reepicheep | and if you remove the interface you should not be able to ping | 01:05 |
| kollian | Reepicheep: we should PC | 01:05 |
| Reepicheep | the subinterface that is | 01:05 |
| kollian | i think toomuch flooding here | 01:05 |
| kollian | of text | 01:05 |
| *** stefan_bo has joined #openstack-kolla | 01:08 | |
| *** zhubingbing has joined #openstack-kolla | 01:10 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add Glance Swift backend support https://review.openstack.org/452059 | 01:11 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 01:14 |
| jamesPR | hi all | 01:17 |
| *** Pavo has joined #openstack-kolla | 01:28 | |
| *** hrw has quit IRC | 01:32 | |
| *** hrw has joined #openstack-kolla | 01:33 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 01:34 |
| *** zhubingbing has quit IRC | 01:36 | |
| *** rhallisey has quit IRC | 01:38 | |
| *** rhallisey has joined #openstack-kolla | 01:39 | |
| *** eanylin has joined #openstack-kolla | 01:51 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 01:56 |
| *** YuYangWang has joined #openstack-kolla | 02:00 | |
| *** qiliang28 has joined #openstack-kolla | 02:01 | |
| *** rwellum has joined #openstack-kolla | 02:05 | |
| *** qiliang27 has quit IRC | 02:05 | |
| *** qiliang28 is now known as qiliang27 | 02:05 | |
| *** kbaegis has quit IRC | 02:28 | |
| duonghq | any cr around? can somebody please review this ps: https://review.openstack.org/#/c/462025/ | 02:31 |
| *** common_ has quit IRC | 02:31 | |
| *** common has joined #openstack-kolla | 02:32 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 02:33 |
| *** zhubingbing_ has joined #openstack-kolla | 02:35 | |
| *** feisky has joined #openstack-kolla | 02:40 | |
| *** kbaegis has joined #openstack-kolla | 02:41 | |
| duonghq | kbaegis, how are you | 02:41 |
| *** kbaegis has quit IRC | 02:44 | |
| *** salv-orlando has joined #openstack-kolla | 02:48 | |
| *** zul has joined #openstack-kolla | 02:56 | |
| *** aagate has quit IRC | 03:01 | |
| *** rwsu has joined #openstack-kolla | 03:03 | |
| *** jamesPR has quit IRC | 03:12 | |
| *** zul has quit IRC | 03:13 | |
| *** zul has joined #openstack-kolla | 03:14 | |
| *** kbaegis has joined #openstack-kolla | 03:15 | |
| *** YuYangWang has quit IRC | 03:23 | |
| *** YuYangWang has joined #openstack-kolla | 03:23 | |
| *** feisky has quit IRC | 03:24 | |
| *** feisky has joined #openstack-kolla | 03:25 | |
| *** feisky has quit IRC | 03:27 | |
| *** feisky has joined #openstack-kolla | 03:27 | |
| kbaegis | duonghq: Hey! Doing alright, you? | 03:30 |
| duonghq | ya, as usual, except something wrong with my testing cluster | 03:33 |
| *** stefan_bo has quit IRC | 03:33 | |
| *** salv-orlando has quit IRC | 03:34 | |
| *** caowei has joined #openstack-kolla | 03:37 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 03:47 |
| *** rwsu has quit IRC | 03:59 | |
| *** stefan_bo has joined #openstack-kolla | 04:00 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix invalid literal for int in swift_proxy_server https://review.openstack.org/465799 | 04:05 |
| *** eaguilar has quit IRC | 04:05 | |
| openstackgerrit | pangliye proposed openstack/kolla master: Delete heka document https://review.openstack.org/465801 | 04:09 |
| *** daidv has joined #openstack-kolla | 04:11 | |
| *** kbaegis has quit IRC | 04:12 | |
| *** kbaegis has joined #openstack-kolla | 04:14 | |
| *** unicell has joined #openstack-kolla | 04:19 | |
| *** stefan_bo has quit IRC | 04:27 | |
| *** unicell1 has joined #openstack-kolla | 04:35 | |
| *** caowei has quit IRC | 04:35 | |
| *** unicell has quit IRC | 04:37 | |
| *** dave-mccowan has quit IRC | 04:43 | |
| *** kbaegis has quit IRC | 04:47 | |
| *** aagate has joined #openstack-kolla | 04:50 | |
| openstackgerrit | Merged openstack/kolla master: Add dragonflow container https://review.openstack.org/418193 | 04:51 |
| *** jaosorior_away is now known as jaosorior | 05:00 | |
| *** lpetrut has joined #openstack-kolla | 05:06 | |
| *** caowei has joined #openstack-kolla | 05:06 | |
| *** pc_m has quit IRC | 05:11 | |
| masuberu | good afternoon, which options do I need to enable on globals.yml file in order to have magnum and ironic, I mean I can see there is 1 options for each of them, but is there any other dependency I need to install? ? | 05:20 |
| *** zhubingbing_ has quit IRC | 05:25 | |
| *** zhubingbing has joined #openstack-kolla | 05:26 | |
| *** salv-orlando has joined #openstack-kolla | 05:27 | |
| openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 05:27 |
| *** stefan_bo has joined #openstack-kolla | 05:28 | |
| *** chkumar|off is now known as chandankumar | 05:31 | |
| *** zhubingbing has quit IRC | 05:36 | |
| *** skramaja has joined #openstack-kolla | 05:37 | |
| *** salv-orlando has quit IRC | 05:44 | |
| *** lpetrut has quit IRC | 05:47 | |
| masuberu | according to this doc kolla ironic is on tech preview state https://docs.openstack.org/developer/kolla/newton/ironic-guide.html | 05:50 |
| masuberu | does it means that is not production ready yet? | 05:50 |
| masuberu | I am actually running into problems trying to install it | 05:51 |
| masuberu | https://pastebin.com/raw/aM6zyZWs | 05:51 |
| openstackgerrit | Dennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support https://review.openstack.org/445858 | 05:53 |
| *** janki has joined #openstack-kolla | 05:57 | |
| masuberu | anyone? | 06:07 |
| *** zhubingbing has joined #openstack-kolla | 06:13 | |
| *** magicboiz has joined #openstack-kolla | 06:18 | |
| *** pc_m has joined #openstack-kolla | 06:19 | |
| *** pcaruana has joined #openstack-kolla | 06:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/ocata: Bump OpenStack service version https://review.openstack.org/465830 | 06:31 |
| *** satyar has joined #openstack-kolla | 06:34 | |
| *** mannamne has joined #openstack-kolla | 06:34 | |
| *** rwellum has quit IRC | 06:35 | |
| *** magicboiz has quit IRC | 06:39 | |
| *** salv-orlando has joined #openstack-kolla | 06:45 | |
| *** mannamne has quit IRC | 06:46 | |
| *** salv-orlando has quit IRC | 06:50 | |
| *** matrohon has joined #openstack-kolla | 06:52 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla master: Add ceilometermiddleware in swift-proxy-server https://review.openstack.org/465841 | 06:54 |
| manheim | masuberu: enabling the services in globals should be sufficient | 06:55 |
| manheim | What distro are you using eith what images | 06:55 |
| *** zhubingbing has quit IRC | 06:58 | |
| *** zhubingbing has joined #openstack-kolla | 06:59 | |
| *** unicell1 has quit IRC | 07:00 | |
| *** unicell has joined #openstack-kolla | 07:00 | |
| *** duonghq has quit IRC | 07:03 | |
| *** unicell1 has joined #openstack-kolla | 07:03 | |
| *** unicell has quit IRC | 07:04 | |
| *** manheim_ has joined #openstack-kolla | 07:06 | |
| *** Serlex has joined #openstack-kolla | 07:10 | |
| *** zhubingbing has quit IRC | 07:19 | |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla master: Add skydive service https://review.openstack.org/459995 | 07:20 |
| openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Fix panko source and dest file path in ceilometer config https://review.openstack.org/463663 | 07:23 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Support OSprofile usage https://review.openstack.org/455628 | 07:24 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: Bump OpenStack service https://review.openstack.org/465849 | 07:25 |
| *** mgoddard__ has joined #openstack-kolla | 07:28 | |
| bjolo__ | morning | 07:32 |
| bjolo__ | can someone please confirm something for me | 07:32 |
| bjolo__ | i have a compute node with a couple of VM's on it | 07:33 |
| bjolo__ | docker restart nova_libvirt | 07:33 |
| bjolo__ | should work right? | 07:33 |
| bjolo__ | pbourke egonzales Jeffrey4l_ ? | 07:34 |
| openstackgerrit | pangliye proposed openstack/kolla-ansible master: Modify the configuration file of telegraf https://review.openstack.org/465446 | 07:37 |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla master: Add skydive service https://review.openstack.org/459995 | 07:41 |
| *** lpetrut has joined #openstack-kolla | 07:41 | |
| *** duritong has quit IRC | 07:41 | |
| *** duritong has joined #openstack-kolla | 07:42 | |
| *** stefan_bo has quit IRC | 07:43 | |
| *** stefan_bo has joined #openstack-kolla | 07:44 | |
| *** jmccarthy has joined #openstack-kolla | 07:44 | |
| Jeffrey4l_ | bjolo__, yes. | 07:46 |
| Jeffrey4l_ | when u using using qemu+kvm. | 07:46 |
| *** tovin07_ has joined #openstack-kolla | 07:46 | |
| Jeffrey4l_ | qemu without kvm should be not. | 07:46 |
| *** salv-orlando has joined #openstack-kolla | 07:46 | |
| pomac | Jeffrey4l_: libvirt uses socket files to communicate to qemu - so the nspawn is not complete and it doesn't quite work with, atleast, newer docker systems | 07:48 |
| pomac | Using local sockets might have been a better and more disconnected approach | 07:48 |
| bjolo__ | still trying to pinpoint down the issue, but what we see is this | 07:49 |
| bjolo__ | kernel 4.11.1 and docker 17.05 and 1.12.6 | 07:49 |
| Jeffrey4l_ | what issue are we talking? is there any bug to track this? | 07:49 |
| bjolo__ | if you restart nova_libvirt container, the compute node stops working. you cannot launch a new vm or attach a volume etc | 07:50 |
| Jeffrey4l_ | which kolla version are u using? | 07:50 |
| *** duritong has quit IRC | 07:50 | |
| bjolo__ | will do some more TS and then open bug, just wanted to confirm that docker restart nova_libvirt *should* work | 07:50 |
| bjolo__ | ocata | 07:50 |
| bjolo__ | stable | 07:50 |
| *** duritong has joined #openstack-kolla | 07:51 | |
| Jeffrey4l_ | i tried upgrade the whole openstack stack from newton to ocata and between ocata minor version. | 07:51 |
| Jeffrey4l_ | never saw such issue. | 07:51 |
| Jeffrey4l_ | after upgrade ( nova_libvirt is restarted or re-created ), anything is find. | 07:51 |
| *** jmccarthy has quit IRC | 07:51 | |
| Jeffrey4l_ | fine* | 07:51 |
| bjolo__ | whats your OS, kernel version? | 07:52 |
| *** powerd has joined #openstack-kolla | 07:52 | |
| *** salv-orlando has quit IRC | 07:52 | |
| Jeffrey4l_ | cenos 7 with default kernel ( 3.10.0-327.22.2.el7.x86_64 ) | 07:52 |
| bjolo__ | and what type of docker filesystem are you using? we use btrfs | 07:52 |
| Jeffrey4l_ | i tested again direct-lvm , btrfs is tested but i am not 100% sure. | 07:53 |
| Jeffrey4l_ | pomac, libvirt connect qemu with socket? where the socket file is located? | 07:54 |
| pomac | Jeffrey4l_: the problem shows as all qemu communication, including starting new machines and attaching volumes doesn't work until all vm:s are restarted | 07:55 |
| *** gfidente has joined #openstack-kolla | 07:55 | |
| Jeffrey4l_ | even virsh list do not work? | 07:55 |
| pomac | example: "/var/lib/libvirt/qemu/domain-53-instance-00000379/monitor.sock" | 07:55 |
| *** jmccarthy has joined #openstack-kolla | 07:55 | |
| *** egonzalez has joined #openstack-kolla | 07:56 | |
| pomac | Jeffrey4l_: i think it sees them but can't communicate - not certain of this though | 07:56 |
| Jeffrey4l_ | this folder is mount on docker libvirt volume. | 07:56 |
| bjolo__ | as pomac says... issue seems to have something todo with /var/lib/libvirt/qemu. which is a layer inside the docker container | 07:56 |
| *** duritong has quit IRC | 07:56 | |
| bjolo__ | no not mounted in our case | 07:56 |
| Jeffrey4l_ | after nova_libvirt restarted, these files is not changed. | 07:56 |
| *** shardy has joined #openstack-kolla | 07:57 | |
| pomac | Jeffrey4l_: yep, and when nova_libvirt is restarted something goes wrong | 07:57 |
| Jeffrey4l_ | oops. | 07:57 |
| bjolo__ | on the host, there is no /var/lib/libvirt | 07:57 |
| Jeffrey4l_ | do u mean this folder it mounted *before* restart nova_libvirt, but disappear *after* restart nova_libvirt? | 07:57 |
| bjolo__ | it only exist inside the container nova_libvirt | 07:57 |
| Jeffrey4l_ | it is a docker libvirt volume. check "docker volume ls" | 07:58 |
| pomac | Jeffrey4l_: something like that yes, it definetly feels a bit like schrodingers cat | 07:58 |
| Jeffrey4l_ | and "docker inspect nova_libvirt" to check whether "/var/lib/libvirt" in container is mounted from host OS | 07:58 |
| bjolo__ | ah ok | 07:58 |
| bjolo__ | libvirtd | 07:58 |
| bjolo__ | there it is | 07:58 |
| Jeffrey4l_ | yes. libvirtd. | 07:59 |
| bjolo__ | nova_libvirt_qemu what is that one used for? | 07:59 |
| Jeffrey4l_ | then the socket file is never changed, no matter what docker storage driver are u using. | 07:59 |
| Jeffrey4l_ | it is instance-xx.xml file | 07:59 |
| bjolo__ | http://paste.openstack.org/show/609876/ | 07:59 |
| Jeffrey4l_ | it is mounted on /etc/libvirtd/qemu | 07:59 |
| bjolo__ | hmmm | 08:00 |
| bjolo__ | ok | 08:00 |
| Jeffrey4l_ | look good | 08:00 |
| *** zhubingbing has joined #openstack-kolla | 08:04 | |
| pomac | Jeffrey4l_: when you started saying this i thought that the sockets should remain in the host system but according to this it's a subvolume that might be unmounted and mounted again - no unix socket will survive this | 08:04 |
| Jeffrey4l_ | it is file mount not device mount. file survived. | 08:05 |
| Jeffrey4l_ | anyway, please try to prove it ;) | 08:05 |
| *** powerd has quit IRC | 08:06 | |
| bjolo__ | agree that it should work | 08:06 |
| bjolo__ | we just have an issue that it does not work for us right now | 08:06 |
| *** athomas has joined #openstack-kolla | 08:07 | |
| bjolo__ | my hunch is that docker has an issue with btrfs | 08:07 |
| Jeffrey4l_ | bjolo__, what kinda issue? | 08:07 |
| *** duritong has joined #openstack-kolla | 08:08 | |
| pomac | Jeffrey4l_: but if the file is unmounted, especially lazily, and then remounted, then there is no connection between the 'socket' and the process | 08:09 |
| bjolo__ | as i said. if we do a docker restart nova_libvirt, the compute node stops working | 08:10 |
| *** mgoddard__ has quit IRC | 08:10 | |
| bjolo__ | we cannot launch new vms, attach volumes, etc | 08:10 |
| bjolo__ | the way to clear it up is to stop/start all vms or migrate them out | 08:11 |
| bjolo__ | then the compute node "works" again | 08:11 |
| bjolo__ | the logs indicate that once we do a restart nova_libvirt, it looses all communication to the VMs | 08:12 |
| bjolo__ | as pomac states... the unix socket does not seem to survive this operation | 08:12 |
| Jeffrey4l_ | hrm.. | 08:12 |
| bjolo__ | from a ps tree perspective. the qemu process changes parent from nova_libvirt to systemd | 08:13 |
| Jeffrey4l_ | bjolo__, this change is expected, imo. | 08:14 |
| bjolo__ | i know | 08:14 |
| bjolo__ | for cleanlisness i would like to see all qemu processes under systemd | 08:14 |
| Jeffrey4l_ | i still curiosity about the loss of socket. could u check the filesystem status when the issue happen? for example /var/lib/libvirt files | 08:15 |
| *** caowei has quit IRC | 08:16 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role https://review.openstack.org/465862 | 08:18 |
| *** mannamne has joined #openstack-kolla | 08:20 | |
| *** zhubingbing has quit IRC | 08:20 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role https://review.openstack.org/465862 | 08:22 |
| bjolo__ | Jeffrey4l_, you have a compute node with some vms on it? ocata | 08:23 |
| bjolo__ | the issue does not show if you dont have running VM's | 08:24 |
| bjolo__ | can you do a docker restart nova_libvirt in that case | 08:24 |
| Jeffrey4l_ | i tested upgrade with vm running. anyway is file. | 08:24 |
| Jeffrey4l_ | anyway, let me launch a env to test this. | 08:24 |
| bjolo__ | ok | 08:24 |
| bjolo__ | what filesystem/volume driver do you have for docker? btrfs? overlayfs? | 08:25 |
| *** zhubingbing_ has joined #openstack-kolla | 08:25 | |
| *** kbaegis has joined #openstack-kolla | 08:26 | |
| Jeffrey4l_ | bjolo__, what filesystem you want me to use? | 08:26 |
| bjolo__ | we use btrfs | 08:26 |
| bjolo__ | kernel-ml from elrepo | 08:26 |
| *** Shunli has joined #openstack-kolla | 08:27 | |
| Jeffrey4l_ | nice. my env is using btrfs + kernel-ml . | 08:28 |
| Shunli | Hi all, could anyone help review this patch https://review.openstack.org/#/c/458791/? | 08:28 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Don't configure external bridge on computes if DVR is disabled https://review.openstack.org/465865 | 08:29 |
| Jeffrey4l_ | Shunli, done. | 08:31 |
| Shunli | @Jeffrey41, thx | 08:32 |
| Jeffrey4l_ | np. | 08:32 |
| Jeffrey4l_ | bjolo__, tested. | 08:32 |
| Jeffrey4l_ | vm is still pingable. | 08:32 |
| bjolo__ | wtf | 08:32 |
| *** chuck__ has joined #openstack-kolla | 08:32 | |
| bjolo__ | hmmm | 08:32 |
| bjolo__ | well our VMs that are running still runs | 08:33 |
| bjolo__ | try launch a new vm | 08:33 |
| Jeffrey4l_ | what i testedis: 1. boot a vm 2. ping is OK 3. restart nova_libvirt 4. ping is OK 5 , start nova_libvirt, ping is still OK. | 08:33 |
| Jeffrey4l_ | let me try | 08:33 |
| bjolo__ | testvector 1. boot vm 2. docker restart nova_libvirt 3. boot another vm | 08:34 |
| *** zul has quit IRC | 08:34 | |
| * Jeffrey4l_ is trying to boot a new vm. | 08:34 | |
| bjolo__ | 4. try attach volume to runing VM. 5. try migrate running vm | 08:34 |
| bjolo__ | step 3 4 5 all fails for us after we have done a restart of nova_libvirt | 08:35 |
| Jeffrey4l_ | hrm.. this is AIO env. | 08:35 |
| bjolo__ | we have multinode. 8 compute nodes | 08:35 |
| bjolo__ | skip the migrate | 08:35 |
| Jeffrey4l_ | boot a new vm works. | 08:35 |
| bjolo__ | hmmm | 08:36 |
| *** kbaegis has quit IRC | 08:36 | |
| bjolo__ | lets compare setups | 08:36 |
| bjolo__ | host OS centos7, kernel-ml 4.11.1, btrfs, docker 1.12.6 or 17.05 | 08:37 |
| bjolo__ | kolla is ocata stable. our images where built about a week ago. | 08:38 |
| Jeffrey4l_ | centos / kernel 4.9 / docker 1.12.6 / btrfs / | 08:39 |
| openstackgerrit | Merged openstack/kolla stable/newton: Add ssh-keyscan to unblock kolla gate https://review.openstack.org/462025 | 08:39 |
| bjolo__ | centos and source. build.conf update to use the ocata stable tarballs for each openstack project | 08:39 |
| Jeffrey4l_ | kolla is ocata stable. but image is built 2 month ago ;( | 08:39 |
| bjolo__ | ok | 08:39 |
| Jeffrey4l_ | i will built lastest ocata image tonight on physical machine and test this. | 08:39 |
| *** powerd has joined #openstack-kolla | 08:40 | |
| Jeffrey4l_ | but i think it should work. | 08:40 |
| bjolo__ | tnx Jeffrey4l_ we will do some more TS on our side | 08:40 |
| Jeffrey4l_ | np. | 08:40 |
| *** zhurong has joined #openstack-kolla | 08:40 | |
| bjolo__ | one more thing. what boot flags do you have for your kernel? | 08:41 |
| Jeffrey4l_ | BOOT_IMAGE=/vmlinuz-4.9.0-1.el7.elrepo.x86_64 root=/dev/mapper/centos-root ro rd.lvm.lv=centos/root rd.lvm.lv=centos/swap crashkernel=auto rhgb quiet | 08:41 |
| bjolo__ | args="ro rootflags=subvol=root crashkernel=auto rhgb nosplash intel_iommu=on hpsa.hpsa_allow_any=1 nofb video=vesafb:off nomodeset vga=0 hpsa.hpsa_simple_mode=0 scsi_mod.use_blk_mq=Y radeon.modeset=0" | 08:41 |
| Jeffrey4l_ | it is default. | 08:41 |
| Jeffrey4l_ | i never changed this. | 08:41 |
| bjolo__ | ok | 08:41 |
| pomac | Jeffrey4l_: which partitions do you use btrfs on? | 08:43 |
| Jeffrey4l_ | / is xfs and /var/lib/docker is btrfs. | 08:44 |
| *** mgoddard__ has joined #openstack-kolla | 08:46 | |
| openstackgerrit | Dan Ardelean proposed openstack/kolla-ansible master: Add Hyper-V role https://review.openstack.org/455684 | 08:47 |
| egonzalez | zhubingbing_, have a second to check https://review.openstack.org/#/c/458791/ | 08:47 |
| zhubingbing_ | ok | 08:48 |
| *** salv-orlando has joined #openstack-kolla | 08:48 | |
| pomac | Jeffrey4l_: can we invite you in upgrading the kernel and see if you feel the same punishment? ;) | 08:49 |
| Jeffrey4l_ | i recommend you fix the issue manually when it happen to dig what's going on. for example remove nova_libvirt and create again. "virst list" in nova_libvirt etc. | 08:51 |
| openstackgerrit | pangliye proposed openstack/kolla master: Install docker_client and groupadd https://review.openstack.org/465880 | 08:51 |
| pomac | Jeffrey4l_: currently waiting for testnode to reboot - bjolo__ will test with 4.9 to see if it's differnet | 08:52 |
| *** salv-orlando has quit IRC | 08:53 | |
| *** jaosorior is now known as jaosorior_lunch | 08:57 | |
| *** salv-orlando has joined #openstack-kolla | 09:00 | |
| *** kbaegis has joined #openstack-kolla | 09:01 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Do Not Merge, kolla-kubernetes gate tests https://review.openstack.org/461063 | 09:03 |
| *** palexster has quit IRC | 09:04 | |
| *** zhurong has quit IRC | 09:05 | |
| *** salv-orlando has quit IRC | 09:05 | |
| *** palexster has joined #openstack-kolla | 09:10 | |
| *** satyar has quit IRC | 09:10 | |
| egonzalez | manheim, was thinking about the hyperv passwd, please add in the doc to move the passwd into /etc/kolla/passwords.yml, so users can encrypt the file with all passwd | 09:13 |
| egonzalez | s/to move/suggest to move/ | 09:14 |
| *** dasTor_ has joined #openstack-kolla | 09:16 | |
| *** dasTor has quit IRC | 09:18 | |
| *** palexster has quit IRC | 09:19 | |
| hrw | morning | 09:22 |
| *** jascott1 has quit IRC | 09:27 | |
| *** zhubingbing_ has quit IRC | 09:27 | |
| *** jascott1 has joined #openstack-kolla | 09:28 | |
| manheim_ | hello | 09:31 |
| *** jascott1 has quit IRC | 09:32 | |
| manheim_ | egonzalez ok, i will move the password in the passwords.yml file | 09:33 |
| manheim_ | and edit the docs | 09:33 |
| manheim_ | how does it look other than this? | 09:33 |
| egonzalez | manheim, im adding some other comments, wait a sec | 09:34 |
| manheim_ | ok | 09:34 |
| *** zhubingbing_ has joined #openstack-kolla | 09:35 | |
| *** Shunli has quit IRC | 09:36 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix doc warning https://review.openstack.org/465900 | 09:38 |
| *** sambetts|afk is now known as sambetts | 09:44 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Remove kolla-ansible dependencies https://review.openstack.org/465089 | 09:51 |
| *** tovin07_ has quit IRC | 09:53 | |
| *** zhubingbing_ has quit IRC | 09:58 | |
| *** stefan_bo has quit IRC | 10:06 | |
| pester | hi all, I was running Kolla-Centos on ubuntu, and just now switched to Kolla-Centos on Centos and got error - "libvirtError: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken" | 10:07 |
| egonzalez | pester, is selinux enabled? are you using kvm or qemu? | 10:08 |
| *** cuongnv has quit IRC | 10:08 | |
| pester | selinux enabled, using KVM | 10:09 |
| pester | I just followed quick start at - "https://docs.openstack.org/developer/kolla-ansible/quickstart.html" | 10:10 |
| pester | no mentions about selinux, it should be disabled ? | 10:10 |
| *** bjolo has joined #openstack-kolla | 10:13 | |
| *** bjolo_ has joined #openstack-kolla | 10:13 | |
| egonzalez | pester, with some docker version selinux worked but is not officially supported by kolla https://docs.openstack.org/developer/kolla-ansible/security.html#selinux | 10:14 |
| egonzalez | iirc 1.12.6 worked | 10:14 |
| *** zhubingbing_ has joined #openstack-kolla | 10:15 | |
| *** zhubingbing_ has quit IRC | 10:15 | |
| *** jrich523_ has joined #openstack-kolla | 10:16 | |
| egonzalez | something else we have to add at quickstart guide ;/ | 10:16 |
| *** zhubingbing has joined #openstack-kolla | 10:16 | |
| pester | yes, it was selinux | 10:17 |
| *** jrich523 has quit IRC | 10:18 | |
| *** zhubingbing has quit IRC | 10:19 | |
| pester | egonzalez: on centos it is same as on ubuntu, this is the result of ./init-runonce with cirros boot - https://ibb.co/c1E4i5 | 10:20 |
| egonzalez | i've seen that recently, are you using vms inside vmware? | 10:21 |
| egonzalez | yeah, missed nick ;) | 10:22 |
| pester | it was me, we disscused this some days ago) I just changed host OS to be sure... | 10:22 |
| egonzalez | have the bug at hand? cannot find it atm | 10:24 |
| pester | https://bugs.launchpad.net/bugs/1688496 | 10:24 |
| openstack | Launchpad bug 1688496 in kolla "instance stuck at "booting from hard disk"" [Undecided,Invalid] | 10:24 |
| pester | I changed bug due to it is qemu bug, not kolla | 10:24 |
| egonzalez | so finally is some issue with esxi | 10:25 |
| egonzalez | have you tried with older esxi version than 6? | 10:26 |
| pester | oh) not ofc, it would be a pain for several days) | 10:26 |
| *** salv-orlando has joined #openstack-kolla | 10:28 | |
| egonzalez | already a pain using vmware ;) | 10:28 |
| *** daidv has quit IRC | 10:28 | |
| kollian | egonzalez: after deploying to openstack | 10:29 |
| kollian | how will i access the horizon | 10:29 |
| kollian | ? | 10:30 |
| kollian | not able to find the port | 10:30 |
| spsurya | kollian: it runs on 80 | 10:31 |
| spsurya | access via http://localhost:80/dashboard | 10:32 |
| *** salv-orlando has quit IRC | 10:33 | |
| spsurya | kollian: in kolla services use the same port as they use devstack installation | 10:34 |
| kollian | spsurya: thanks | 10:34 |
| *** kollian has quit IRC | 10:36 | |
| pester | if using centos - fierst open port in iptables | 10:40 |
| pester | using vmware for 10 years almost without issues) btw got VM on esx 6.5 will try it now | 10:42 |
| egonzalez | pester, if run `/usr/libexec/qemu-kvm -M ?` (? included) in the host whats the output and the same in libvirt container | 10:45 |
| pester | -bash: /usr/libexec/qemu-kvm: No such file or directory | 10:46 |
| egonzalez | install qemu-kvm | 10:46 |
| pester | no qemu-kvm on hostmachine | 10:46 |
| pester | I traced those -machine setting to be hardcoded inside package... | 10:47 |
| *** openstackgerrit has quit IRC | 10:48 | |
| pester | hm, they differ very much - http://paste.openstack.org/show/609897/ | 10:49 |
| egonzalez | hrm, so host is centos7 and centos images 7.3? | 10:50 |
| pester | no, it is fresh 7.3 CentOS Linux release 7.3.1611 (Core) | 10:51 |
| pester | in container I got package qemu-kvm-ev-2.6.0-28.el7_3.6.1.x86_64 | 10:52 |
| pester | in host - qemu-kvm-1.5.3-126.el7_3.6.x86_64 | 10:53 |
| egonzalez | may try settign hw_machine_type to qemu-kvm-1.5.3-126.el7_3.6.x86_64 and check if works with 7.0.0 type? | 10:55 |
| egonzalez | *pc-i440fx-rhel7.0.0 | 10:55 |
| *** openstackgerrit has joined #openstack-kolla | 10:56 | |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Delete unnecessary code and insert a blank lines. https://review.openstack.org/465919 | 10:56 |
| *** YuYangWang has quit IRC | 10:56 | |
| openstackgerrit | Dan Prince proposed openstack/kolla master: Switch setup_RedHat.sh to overlay2 https://review.openstack.org/465920 | 10:58 |
| pester | I tried, but looks like this setting is broken, I just add metadata hw_machie_type=x86_64 and even if you dont edit config it fails | 10:58 |
| openstackgerrit | Chen proposed openstack/kolla-ansible master: Fix several grammer issues on comments https://review.openstack.org/465921 | 10:58 |
| pester | error is libvirt-PCI-something | 10:59 |
| pester | found this - https://lists.centos.org/pipermail/centos-announce/2015-October/021445.html | 10:59 |
| egonzalez | settings are | 11:01 |
| egonzalez | [libvirt] | 11:02 |
| egonzalez | hw_machine_type=x86_64=pc-i440fx-rhel7.0.0 | 11:02 |
| pester | I know, tried it - didn`t help, got libvirt error even more untraceable | 11:03 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 11:03 |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Remove kolla-ansible dependencies https://review.openstack.org/465089 | 11:03 |
| egonzalez | weird, I have a working env with those versions | 11:04 |
| egonzalez | well, i dont even have any qemu package in the compute nodes, only in containers | 11:05 |
| *** kbaegis has quit IRC | 11:19 | |
| pester | egonzalez: I installed qemu-kvm-ev on host machine, so -M listings are same now, problem still persist | 11:21 |
| *** mgoddard__ has quit IRC | 11:22 | |
| *** salv-orlando has joined #openstack-kolla | 11:28 | |
| *** rwellum has joined #openstack-kolla | 11:29 | |
| *** jaosorior_lunch is now known as jaosorior | 11:29 | |
| *** sklgromek has joined #openstack-kolla | 11:29 | |
| *** chuck__ is now known as zul | 11:32 | |
| sklgromek | hi Guys, | 11:33 |
| sklgromek | i have some problem with override cinder.conf configuration. | 11:33 |
| sklgromek | I wanna add new cinder backend. | 11:33 |
| sklgromek | I put my configuration file in | 11:33 |
| sklgromek | /etc/kolla/config/cinder/cinder.conf, and nothing was override;/ | 11:33 |
| *** skramaja has quit IRC | 11:33 | |
| *** salv-orlando has quit IRC | 11:33 | |
| sklgromek | what can be wrong? | 11:33 |
| egonzalez | the path ;) /etc/kolla/config/cinder.conf | 11:34 |
| egonzalez | sklgromek, ^^ | 11:34 |
| egonzalez | sklgromek, you can override: <config>/cinder.conf , <config>/cinder/<service-name>.conf (cinder-api.conf) or <config>/cinder/<inventory_hostname>/cinder.conf | 11:36 |
| sklgromek | ohh , ok thank you, i will try. | 11:37 |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla master: Add skydive service https://review.openstack.org/459995 | 11:51 |
| matrohon | egonzalez, hi | 11:51 |
| egonzalez | matrohon, sup | 11:51 |
| matrohon | egonzalez, concerning your comment about the skydive dockerfile : https://review.openstack.org/#/c/459995/8/docker/skydive/Dockerfile.j2@14 | 11:52 |
| manheim_ | Dockerfiles are used only when building the images and never touched on pull->deploy? | 11:52 |
| egonzalez | manheim, yep, pull/deploy uses already existing images | 11:53 |
| matrohon | egonzalez, in the current version both the analyzer and the agent are run as root user | 11:53 |
| matrohon | this is mandatory for the agent | 11:53 |
| matrohon | but not for the analyzer | 11:54 |
| matrohon | I saw you set +2 for this version, so it's ok as is? | 11:54 |
| egonzalez | matrohon, then i'd prefer having skydive-base, skydive-analyzer and skydive-agent | 11:55 |
| egonzalez | if we avoid exposing root to an image is better imo | 11:55 |
| matrohon | egonzalez, I was afraid you answered that :) | 11:55 |
| matrohon | egonzalez, but I agree | 11:55 |
| matrohon | I'll try to revisit my patch asap | 11:56 |
| sklgromek | egonzalez: thank you, now it's working. | 11:57 |
| matrohon | egonzalez, thanks | 11:57 |
| egonzalez | sklgromek, what backend have implemented? something we can have upstream? | 11:57 |
| *** salv-orlando has joined #openstack-kolla | 11:57 | |
| *** matrohon has quit IRC | 12:01 | |
| manheim_ | egonzalez i put [hyperv] in inventory, should i put [hyperv:vars] also? | 12:13 |
| egonzalez | manheim, think no, just add on docs | 12:15 |
| manheim_ | AHA | 12:18 |
| manheim_ | sorry for caps | 12:19 |
| manheim_ | so about the automated networking-hyperv driver in neutron Dockerfile | 12:19 |
| egonzalez | add the binary/source in neutron images | 12:23 |
| egonzalez | dont make people install stuff on already running containers(anti CI friendly) | 12:23 |
| manheim_ | yea | 12:26 |
| manheim_ | but this implies that new images will be generated | 12:26 |
| manheim_ | and this in kolla project | 12:27 |
| egonzalez | manheim, yep, open a new change and add the packages in the image, for example for neutron-server https://github.com/openstack/kolla/blob/master/docker/neutron/neutron-server/Dockerfile.j2#L14 | 12:28 |
| manheim_ | this for all base distros and both binary and source | 12:29 |
| egonzalez | if binary have packages build by distros, add binary into that install type, then add pip package for source | 12:29 |
| manheim_ | only pip packages | 12:30 |
| *** schwicht has joined #openstack-kolla | 12:31 | |
| manheim_ | issue is that the the driver is per openstack version | 12:31 |
| manheim_ | how can I deal with this | 12:32 |
| manheim_ | e.g "networking-hyperv>=3.0.0,<4.0.0" is for ocata | 12:32 |
| *** schwicht has quit IRC | 12:36 | |
| *** schwicht has joined #openstack-kolla | 12:39 | |
| *** salv-orlando has quit IRC | 12:39 | |
| *** salv-orlando has joined #openstack-kolla | 12:40 | |
| *** goldyfruit has joined #openstack-kolla | 12:42 | |
| jaosorior | Hey guys, what's the process of adding a new image to kolla? We've been thinking about adding an image for a service called novajoin ( https://github.com/openstack/novajoin ) however, there are no ubuntu packages for this (only CentOS), will this be a problem? | 12:43 |
| *** salv-orlando has quit IRC | 12:45 | |
| *** jbadiapa has quit IRC | 12:45 | |
| *** jrist has quit IRC | 12:45 | |
| *** jbadiapa has joined #openstack-kolla | 12:46 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/465257 | 12:46 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/464393 | 12:46 |
| *** bjolo_ has quit IRC | 12:47 | |
| *** bjolo has quit IRC | 12:47 | |
| *** bjolo_ has joined #openstack-kolla | 12:47 | |
| egonzalez | jaosorior, if there is no ubuntu packages just add an exception on building for base_distro https://github.com/openstack/kolla/blob/master/docker/dragonflow/dragonflow-base/Dockerfile.j2#L12 | 12:50 |
| jaosorior | egonzalez: awesome, thanks! | 12:51 |
| egonzalez | also here https://github.com/openstack/kolla/blob/master/tests/test_build.py#L114 to not make gates cry ;) | 12:51 |
| *** rwallner has joined #openstack-kolla | 12:53 | |
| *** schwicht has quit IRC | 12:54 | |
| jaosorior | egonzalez: noted, thanks | 12:56 |
| *** rwallner has quit IRC | 13:01 | |
| *** rwallner has joined #openstack-kolla | 13:01 | |
| *** fguillot has joined #openstack-kolla | 13:02 | |
| openstackgerrit | Brad P. Crochet proposed openstack/kolla master: Add mistral-event-engine image https://review.openstack.org/462972 | 13:02 |
| openstackgerrit | Dan Prince proposed openstack/kolla master: multipathd: modprobe dm-multipath on startup https://review.openstack.org/465952 | 13:03 |
| openstackgerrit | Brent Eagles proposed openstack/kolla master: Add Neutron SR-IOV agent image https://review.openstack.org/465953 | 13:04 |
| *** mgoddard__ has joined #openstack-kolla | 13:05 | |
| *** schwicht has joined #openstack-kolla | 13:09 | |
| *** dave-mccowan has joined #openstack-kolla | 13:13 | |
| *** jrist has joined #openstack-kolla | 13:14 | |
| *** bjolo_ has quit IRC | 13:18 | |
| *** bmaltais_ has joined #openstack-kolla | 13:19 | |
| rwellum | Simple basic question: I have a kolla-k8s cluster up and running. Now I want to generate a vm with Nova. Not sure where to do it from and how in this environment. Pointers are fine... | 13:21 |
| sdake | rwellum have you generated teh openrc file? | 13:21 |
| rwellum | yes. I can do this in horizon. Although it's slow because I think my VM is under-powered. | 13:22 |
| *** sgrasley has joined #openstack-kolla | 13:22 | |
| sdake | rwellum cool - typically what I do is source that file and then run a bootstrap script | 13:23 |
| sdake | e..g: https://github.com/openstack/kolla-ansible/blob/master/tools/init-runonce | 13:23 |
| sdake | rwellum after running that command, a "openstack server boot xyz" operation will be printed out that is appropriate for the network env | 13:24 |
| sdake | you may need to customize that script to your network setup | 13:24 |
| rwellum | Thanks sdake - looking at it now. | 13:25 |
| *** janki has quit IRC | 13:27 | |
| *** eaguilar has joined #openstack-kolla | 13:28 | |
| *** eanylin has quit IRC | 13:30 | |
| *** ducttape_ has joined #openstack-kolla | 13:30 | |
| *** ducttape_ has quit IRC | 13:31 | |
| openstackgerrit | Andy Smith proposed openstack/kolla master: Add qdrouterd container https://review.openstack.org/465145 | 13:39 |
| *** salv-orlando has joined #openstack-kolla | 13:41 | |
| *** ducttape_ has joined #openstack-kolla | 13:41 | |
| *** eanylin has joined #openstack-kolla | 13:44 | |
| *** salv-orlando has quit IRC | 13:46 | |
| inc0 | good morning | 13:47 |
| *** ducttape_ has quit IRC | 13:47 | |
| *** ducttape_ has joined #openstack-kolla | 13:48 | |
| sdake | anyone have any idea if we need: res create bootstrap openvswitch-set-external-ip | 13:52 |
| rwellum | Morning | 13:52 |
| sdake | and if it is needed, would a microchart work for that? | 13:52 |
| sdake | sup inc0 | 13:52 |
| inc0 | sdake: what this chart would do? | 13:53 |
| sbezverk | sdake: I think we got rid of it in the gate after started using keepalived | 13:53 |
| inc0 | not to mention openvswitch doesn't have IPs;) | 13:53 |
| inc0 | ahh | 13:53 |
| sbezverk | inc0: it was done to be able to test external connectivity | 13:53 |
| inc0 | yeah, we need keepalived for that | 13:54 |
| sbezverk | at the gate | 13:54 |
| rwellum | sbezverk: I have recreated the issue where I seem to run out of ip addresses. Just seems to require me to bring up my environment enough times. | 13:54 |
| sdake | sbezverk godo news thanks :) | 13:54 |
| sbezverk | rwellum: ok let's dig a bit deeper if you have time now | 13:54 |
| openstackgerrit | Dan Ardelean proposed openstack/kolla master: Add networking-hyperv driver when building neutron-server image https://review.openstack.org/465981 | 13:55 |
| sbezverk | rwellum: when you bring up new env, it is clean VM, no any leftovers is possible? | 13:56 |
| rwellum | sbezverk: typically I have a meeting now but will be free in about an hour. | 13:56 |
| sbezverk | rwellum: ok sounds good | 13:56 |
| rwellum | Yes it's fine from a clean vm so assuming something goes bad over time. | 13:57 |
| sbezverk | rwellum: I think then it might be related to etcd left overs. It keep previously alocated ranges and with enough attempts it runs out of available subnets for pod. It is just a theory, but we could check that by looking into etcd raw data | 13:59 |
| rwellum | sbezverk: cool | 14:00 |
| manheim_ | egonzalez you on? | 14:01 |
| egonzalez | manheim, yep | 14:07 |
| manheim_ | about rdp_port | 14:08 |
| manheim_ | so this is used only on hyperv nodes | 14:08 |
| manheim_ | but from the comment i think you got that | 14:08 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Remove kolla-ansible dependencies https://review.openstack.org/465089 | 14:09 |
| egonzalez | manheim, is also used by heat_cfn, but i'd prefer not share same port for other services, even not being in the same node | 14:10 |
| egonzalez | just for consistency | 14:10 |
| *** krtaylor has quit IRC | 14:13 | |
| *** kbaegis has joined #openstack-kolla | 14:18 | |
| manheim_ | i see | 14:18 |
| *** srwilkers has joined #openstack-kolla | 14:19 | |
| *** kbaegis has quit IRC | 14:19 | |
| *** mkoderer has joined #openstack-kolla | 14:19 | |
| *** shardy has quit IRC | 14:31 | |
| openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 14:31 |
| *** shardy has joined #openstack-kolla | 14:32 | |
| openstackgerrit | Merged openstack/kolla master: Updated from global requirements https://review.openstack.org/465257 | 14:33 |
| *** palexster has joined #openstack-kolla | 14:34 | |
| manheim_ | hmm, any port preference for rdp_port? | 14:36 |
| *** krtaylor has joined #openstack-kolla | 14:42 | |
| openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 14:42 |
| inc0 | kfox1111: up and about? | 14:44 |
| *** krtaylor has quit IRC | 14:44 | |
| egonzalez | anyone have a hint why ubuntu binary build gates are crying with python deps? | 14:45 |
| *** imcsk8 has quit IRC | 14:46 | |
| *** lpetrut has quit IRC | 14:48 | |
| *** krtaylor has joined #openstack-kolla | 14:48 | |
| hawi | hi. i have kind a silly question: how to upgrade from 4.0.0 to 4.0.x | 14:49 |
| inc0 | hawi: once you have 4.0.1 (for example available) | 14:50 |
| inc0 | just change /etc/kolla/globals.yml openstack_release="4.0.1" and run kolla-ansible upgrade | 14:50 |
| inc0 | or better, pip install -U kolla-ansible | 14:50 |
| *** shardy has quit IRC | 14:50 | |
| inc0 | pip freeze | grep kolla-ansible => 4.0.1 | 14:51 |
| inc0 | ensure openstack_release in globals is "auto" | 14:51 |
| inc0 | and run kolla-ansible upgrade | 14:51 |
| *** shardy has joined #openstack-kolla | 14:53 | |
| *** imcsk8 has joined #openstack-kolla | 14:54 | |
| hawi | inc0: thanks, will try out. i do have some broken things i think are fixed in 4.0.x, so will try out | 14:54 |
| inc0 | sure, it's good to have your containers up-to-date | 14:54 |
| hawi | inc0: is kolla_install_type also important? i think i installed last one as source | 14:55 |
| inc0 | well, if you change install type it's going to be upgraded to new install type | 14:55 |
| inc0 | so if you have source, you need to have source images available and source is going to be upgraded to | 14:56 |
| manheim_ | 4.0.1 is out? :> | 14:57 |
| openstackgerrit | Brad P. Crochet proposed openstack/kolla master: Add mistral-event-engine image https://review.openstack.org/462972 | 14:59 |
| sdake | inc0 can you help me with some ansibulls | 14:59 |
| sdake | or alternately egonzalez :) | 14:59 |
| inc0 | whats up? | 14:59 |
| sdake | i want to unlabel nodes in ansibulls | 14:59 |
| inc0 | as in ansible calling k8s? | 14:59 |
| sdake | https://review.openstack.org/#/c/464031/ | 14:59 |
| inc0 | I'd just use command for that | 14:59 |
| sdake | ya i wrote one in bash already | 15:00 |
| sdake | it was pretty terribad :) | 15:00 |
| *** jaosorior is now known as jaosorior_away | 15:00 | |
| *** yingjun has joined #openstack-kolla | 15:01 | |
| sdake | # Unlabel the nodes | 15:01 |
| sdake | kubectl get nodes -o name | xargs -i kubectl label {} kolla_compute- kolla_controller- kolla_conductor- | 15:01 |
| sdake | the problem is sometimes nodes are not labeled | 15:01 |
| sdake | and ansibulls needs to detect that scenario and proceed | 15:01 |
| *** dasTor_ has quit IRC | 15:01 | |
| *** dasTor has joined #openstack-kolla | 15:02 | |
| inc0 | hmm, can you paste worst case scenerio somewherE? | 15:02 |
| inc0 | I don't have k8s cluster at hand | 15:02 |
| pester | how do I debug network without brctl ? >_< can`t if my interface plugged into br-ex or not.... | 15:02 |
| sdake | i dont have any ansible implemented | 15:02 |
| sdake | it just spits out a buch of errors aboutn o label present | 15:02 |
| sdake | (it is untidy) | 15:02 |
| inc0 | pester: if you're using ovs, that's ovs-vsctl | 15:02 |
| inc0 | and you can have it by doing | 15:03 |
| inc0 | docker exec -it openvswitch_vswitchd bash | 15:03 |
| sdake | inc0 that review contains the rest of the code i've wr itten for destroy | 15:03 |
| Reepicheep | manheim_: I think the release schedule is every 45 days or so but I'm not sure that it is hard and fast and 4.0.1 came out 28 days ago | 15:03 |
| inc0 | and there you have ovs-vsctl | 15:03 |
| sdake | Reepicheep it is a consideration not a guideline or policy | 15:03 |
| inc0 | sdake: what I'm asking is kubectl get nodes -o name | 15:03 |
| Reepicheep | now the stable/ocata branch continues to move forward between releases | 15:03 |
| inc0 | with mix of labeled and non-labeled | 15:04 |
| pester | inc0: thx | 15:04 |
| sdake | inc0 oh yo uwant to see the output? | 15:04 |
| inc0 | the input:) | 15:04 |
| Reepicheep | sdake: that is kinda what I was thinking | 15:04 |
| *** jascott1 has joined #openstack-kolla | 15:04 | |
| inc0 | also you can add grep kolla;) | 15:04 |
| inc0 | before xargs | 15:04 |
| sdake | sometimes nodes have only 2 labels sometimes 1 sometimes 3 | 15:05 |
| sdake | inc0 the input to what? to the xargs? | 15:05 |
| manheim_ | :q | 15:05 |
| manheim_ | sorry | 15:05 |
| inc0 | ok, so what we can do is kubectl get nodes | grep kolla | awk ..(get node name) | xargs.. | 15:06 |
| sdake | that selects only the nodes labeled with kolla | 15:07 |
| sdake | sometimes no compute or conductor label is applied (which spits out an error) | 15:07 |
| sdake | let me generate the output - moment deploying :) | 15:07 |
| sdake | rather the kubectl get nodes output | 15:07 |
| sdake | bah its just a list of nodes | 15:08 |
| *** jascott1 has quit IRC | 15:08 | |
| sdake | inc0 $ kubectl get nodes -o name | 15:09 |
| sdake | node/falkor-node0 | 15:09 |
| sdake | node/falkor-node1 | 15:09 |
| sdake | node/falkor-node2 | 15:09 |
| inc0 | remove -o name? | 15:09 |
| *** jrich523_ is now known as jrich523 | 15:12 | |
| sdake | this works: [admin@falkor-node0 ~]$ kubectl get nodes --selector=kolla_conductor | 15:14 |
| sdake | -o name prints out output that that other kubectl command can work with :) | 15:14 |
| sdake | [admin@falkor-node0 ~]$ kubectl get nodes --selector=kolla_conductor -o name | 15:15 |
| sdake | node/falkor-node0 | 15:15 |
| sdake | inc0 thanks for the brainstorm session :) | 15:15 |
| sdake | I can implement those as 3 separate operations | 15:15 |
| *** goldyfruit has quit IRC | 15:15 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 15:16 |
| *** goldyfruit has joined #openstack-kolla | 15:17 | |
| pester | hm, looks like ovs inserted my tap0 into br-ex incorrectly :( can`t see any traffic on tap0 | 15:17 |
| *** gomarivera_ has joined #openstack-kolla | 15:18 | |
| openstackgerrit | Merged openstack/kolla master: multipathd: modprobe dm-multipath on startup https://review.openstack.org/465952 | 15:18 |
| pester | I love IRC so much, just write question and google it youself and get the solution) | 15:19 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:20 |
| *** manheim_ has quit IRC | 15:21 | |
| inc0 | yeah, it's handy:) but we need better docs for troubleshooting in container world | 15:21 |
| egonzalez | inc0, multinode gates yay! | 15:23 |
| *** hongbin has joined #openstack-kolla | 15:24 | |
| Serlex | anyone tested bare-metal guide on Ubuntu? | 15:25 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 15:25 |
| sdake | inc0 can you ahve a review of the above plz | 15:25 |
| inc0 | looks good | 15:27 |
| inc0 | you want to make it an "operator"? | 15:27 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 15:28 |
| sdake | inc0 nope - want to make it a controller | 15:28 |
| sdake | an operator would in theory action on that controller | 15:28 |
| *** rwsu has joined #openstack-kolla | 15:28 | |
| sdake | i think for now human operators are fine :) | 15:28 |
| inc0 | ok...controller? | 15:28 |
| inc0 | what's a controller? | 15:28 |
| inc0 | new name for what we used to call operator? | 15:29 |
| *** shardy has quit IRC | 15:29 | |
| sdake | a controller are scripts/ansible/wahtever that execute inside a container to do something specific on kubernetes | 15:29 |
| sdake | inc0 nah its an old term | 15:29 |
| sdake | its a do one job doe it will kind of hting | 15:29 |
| *** gomarivera_ has quit IRC | 15:29 | |
| sdake | operator is by contrast a jack of all trades _master_ of many :) | 15:29 |
| inc0 | lol | 15:29 |
| inc0 | gotta love nomenclature creep in new technologies | 15:30 |
| sdake | its like a 18 month old term but ya I hear ya :) | 15:30 |
| inc0 | "this thing is little diffirent than that thing, let's invent a term!" | 15:30 |
| inc0 | I don't know which was first | 15:30 |
| sdake | controller was first | 15:30 |
| inc0 | yeah, whatever, you know what I mean;) | 15:31 |
| *** shardy has joined #openstack-kolla | 15:31 | |
| sdake | the concepts of a controller have a couple different intrepetations in the operator setting, that is 1) the operator can call the controoller ,or 2) the operator can implement all controller logic | 15:31 |
| sdake | in kfox1111 's case, I think he is doing #2 (as he is a human operator) | 15:31 |
| sdake | nah I meant kevin is doing the workflow stuff manually :) | 15:32 |
| sdake | BACK TO THE SHOWERS FOR SDAKE | 15:32 |
| *** Serlex has quit IRC | 15:33 | |
| *** salv-orlando has joined #openstack-kolla | 15:33 | |
| rwellum | sbezverk: I have some time now if you do? | 15:33 |
| *** fooliouno has joined #openstack-kolla | 15:34 | |
| sbezverk | rwellum: ok, please paste me /etc/systemd/system/kubelet.service.d/10-kubeadm.conf | 15:36 |
| rwellum | inc0: a 'computer' was a human operator once. | 15:36 |
| sbezverk | rwellum: then cat /etc/kubernetes/manifests/kube-controller-manager.yaml | 15:37 |
| inc0 | rwellum: so now we're going back to roots?;) meat automation?;) | 15:37 |
| sbezverk | rwellum: then kubectl describe pod which is crashing | 15:37 |
| rwellum | I'm just saying we could technically call kfox1111 a 'computer' :) | 15:38 |
| sbezverk | rwellum: and then grep Network in your canal.yaml | 15:38 |
| *** fooliouno has quit IRC | 15:39 | |
| *** eaguilar has quit IRC | 15:39 | |
| *** matrohon has joined #openstack-kolla | 15:40 | |
| rwellum | sbezverk: http://paste.openstack.org/show/609928/ | 15:41 |
| rwellum | Also sbezverk : http://paste.openstack.org/show/609929/ | 15:43 |
| rwellum | The second one is the tiller pod crashing with the ip issue - I think that's more important that the other one | 15:44 |
| sbezverk | rwellum: you have canal and kube-controller-manager configured for different ip ranges | 15:45 |
| *** fooliouno has joined #openstack-kolla | 15:46 | |
| sbezverk | check carefully the manifest and canal.yaml | 15:46 |
| rwellum | which manifest? | 15:47 |
| sbezverk | rwellum: cat /etc/kubernetes/manifests/kube-controller-manager.yaml | 15:49 |
| sbezverk | there is a key there - --cluster-cidr=X.X.X.X/Y | 15:51 |
| sbezverk | it must match to what you use in canal.yaml | 15:51 |
| inc0 | sbezverk: can you please point me to place wehre you set up disks for ceph? | 15:52 |
| inc0 | in gates | 15:52 |
| sbezverk | if you do not see that key in command line then it means you did not specify it in your kubeadm init command line and you should, as now it is requirement | 15:52 |
| sbezverk | inco0: one sec, let me find it, I nromally use iscsi ;) | 15:52 |
| sbezverk | inc0: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_rbd_volumes.sh | 15:53 |
| inc0 | I meant where do we deploy ceph itself? | 15:54 |
| openstackgerrit | Merged openstack/kolla stable/ocata: Bump OpenStack service version https://review.openstack.org/465830 | 15:55 |
| inc0 | https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/setup_gate_loopback.sh that what I need;) | 15:55 |
| openstackgerrit | Merged openstack/kolla master: Move shebang customization into jinja2 block https://review.openstack.org/454637 | 15:55 |
| sbezverk | inc0: cool, sorry did not get what you meant.. | 15:56 |
| inc0 | sure, let's see if I can get to deploy ceph in ansible gates | 15:56 |
| sdake | inc0 what is wrong with this: | 16:00 |
| sdake | TASK [destroy : Delete kolla_controller labels] ********************************************************************************************* | 16:00 |
| sdake | fatal: [localhost]: FAILED! => {"failed": true, "msg": "template error while templating string: expected token ':', got '}'. String: kubectl label {{{ item }} kolla_compute-"} | 16:00 |
| kfox1111 | morning. | 16:00 |
| sdake | sup kfox1111 | 16:00 |
| *** kolla-slack1 has quit IRC | 16:00 | |
| *** kolla-slack has joined #openstack-kolla | 16:00 | |
| sbezverk | rwellum: did you see what I meant? | 16:00 |
| *** kbaegis has joined #openstack-kolla | 16:01 | |
| sbezverk | kfox1111: morning, in v4 gate jobs by some reason horizon's svc does not have external ip, I guess it is missing it yaml thingy.. | 16:01 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 16:02 |
| kfox1111 | sbezverk: weird. | 16:02 |
| kfox1111 | I wouldn't have thought that one would be any different. | 16:03 |
| *** gomarivera_ has joined #openstack-kolla | 16:03 | |
| sbezverk | kfox1111: well if horizon does not listen on external ip then testing will fail exactly the way we can see now | 16:03 |
| *** stefan_bo has joined #openstack-kolla | 16:04 | |
| kfox1111 | yeah, but in the v4 gate ps I didn't touch the horizon svc. so it shouldn't be any different. | 16:04 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 16:04 |
| rwellum | sbezverk: I think so but I am confused... because the deployment guide says: | 16:04 |
| rwellum | https://www.irccloud.com/pastebin/0nhBpLUn/ | 16:04 |
| sbezverk | kfox1111: not sure what to say, maybe we merged something and did not test it by running experimental gate | 16:06 |
| rwellum | sbezverk: but canal.yaml only has one ip addr: "Network": "10.244.0.0/16", | 16:06 |
| sbezverk | because standard gate jobs are all green, checked this morning | 16:06 |
| kfox1111 | yeah. though still, I'd think it would affect the v2 and 3 gates equally. | 16:06 |
| kfox1111 | but.... YAY! the ceph cinder volume issue was resolved. :) | 16:06 |
| kfox1111 | very close to v4 gate now. :) | 16:07 |
| sbezverk | rwellum: the ip range in canal.yaml and controller-manager manifest must match, if DG says otherwise, it is wrong.. | 16:07 |
| *** yingjun has quit IRC | 16:07 | |
| *** stefan_bo has quit IRC | 16:08 | |
| sbezverk | rwellum: that applies ONLY to NEW canal.yaml | 16:09 |
| fooliouno | egonzalez: need a little help with opendaylight on kolla-k8s. have a kolla controller on one node and kolla compute on another. VMs get DHCP addresses, but unable to ping router interface. | 16:10 |
| fooliouno | I see the ARP requests going from compute to controller over vxlan tunnel | 16:11 |
| rwellum | sbezverk: sorry what 'that'? | 16:11 |
| fooliouno | egonzalez: but nothing received at qrouter | 16:11 |
| sbezverk | rwellum: ignore that, but the rest is true ;) | 16:11 |
| *** Serlex has joined #openstack-kolla | 16:13 | |
| *** unicell1 has quit IRC | 16:14 | |
| fooliouno | sbezverk: Did this issue cause any ping issues between VM and qrouter for you. I see the same ERROR in my l3-agent logs (ip6tables-restore failure) https://bugs.launchpad.net/neutron/+bug/1615715 | 16:14 |
| openstack | Launchpad bug 1615715 in neutron "ip6tables-restore fails " [Undecided,Invalid] | 16:14 |
| rwellum | sbezverk: deploymeny guide is wrong then :) Better to just call: kolla-kubernetes/tests/bin/setup_canal.sh | 16:16 |
| sbezverk | fooliouno: do not recall what issue I was chasing at that time, sorry, but I doubt it was ping.. | 16:16 |
| sbezverk | probably container or process crash | 16:16 |
| fooliouno | sbezverk: ok .. thank you | 16:16 |
| *** yingjun has joined #openstack-kolla | 16:17 | |
| sbezverk | rwellum: good idea ;) | 16:17 |
| *** matrohon has quit IRC | 16:21 | |
| *** manheim_ has joined #openstack-kolla | 16:24 | |
| *** salv-orlando has quit IRC | 16:25 | |
| *** salv-orlando has joined #openstack-kolla | 16:26 | |
| rwellum | sbezverk: now they match up but same issue seen. https://www.irccloud.com/pastebin/6TgcVqsA/ | 16:27 |
| *** salv-orlando has quit IRC | 16:27 | |
| *** Mannheim has joined #openstack-kolla | 16:27 | |
| *** salv-orl_ has joined #openstack-kolla | 16:27 | |
| sbezverk | rwellum: have you killed tiller pod? | 16:28 |
| sbezverk | if not, please do it because ip address allocation will still point to old wrong ip | 16:28 |
| *** manheim_ has quit IRC | 16:29 | |
| rwellum | I thought so. Let me re-clean up. | 16:30 |
| sdake | cool the destroy playbook seems to work now with unlabeling | 16:30 |
| *** yingjun has quit IRC | 16:30 | |
| *** jemcevoy has quit IRC | 16:31 | |
| sbezverk | sdake: how do you know it?? | 16:32 |
| sdake | sbezverk run test manually - although i am going to integrate with the gate as you requested | 16:32 |
| sbezverk | sdake: thanks, would be great and keep everything in healthy state.. | 16:33 |
| sdake | agreed | 16:33 |
| sdake | i was thinking that wouldn't work originally (as there is no deploy) however there is a deploy, jsut not a owrkflow based one :) | 16:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 16:36 |
| *** pcaruana has quit IRC | 16:40 | |
| sdake | sbezverk what kind of verification do you want on the cleanup phase? | 16:41 |
| sdake | rather destroy phase | 16:41 |
| sbezverk | sdake: I think if you can add this script at the end of one of gate jobs, then at the end you should be able to check | 16:44 |
| sbezverk | no pods, no namespace, no labels etc | 16:44 |
| sbezverk | if all checks out, the script is done exit 0 ;) | 16:45 |
| *** mgoddard__ has quit IRC | 16:48 | |
| kfox1111 | sbezverk: btw, some folks here are facinated by your demo. :) | 16:52 |
| sbezverk | kfox1111: are they going to use kolla-k8s in production then ;)?? | 16:53 |
| kfox1111 | it turned out really well. | 16:53 |
| kfox1111 | thats till the plan. | 16:54 |
| kfox1111 | wheter we use ironic or not was always more iffy. | 16:54 |
| openstackgerrit | Dan Ardelean proposed openstack/kolla master: Add networking-hyperv driver when building neutron-server image https://review.openstack.org/465981 | 16:54 |
| kfox1111 | but the demo may sway them there. :) | 16:54 |
| sbezverk | kfox1111: nice :) I was just lucky to get support from ironic folks... | 16:55 |
| *** eaguilar has joined #openstack-kolla | 16:55 | |
| *** salv-orl_ has quit IRC | 16:56 | |
| openstackgerrit | Merged openstack/kolla master: Add mistral-event-engine image https://review.openstack.org/462972 | 16:56 |
| *** salv-orlando has joined #openstack-kolla | 16:56 | |
| *** kbaegis has quit IRC | 16:59 | |
| *** salv-orlando has quit IRC | 17:00 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 17:01 |
| *** sambetts is now known as sambetts|afk | 17:03 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:04 |
| *** gomarivera_ has quit IRC | 17:05 | |
| sdake | sbezverk does this meet your criteria or want something more? https://review.openstack.org/#/c/464031/8/tools/setup_gate.sh | 17:06 |
| sdake | kfox1111 nobody requires an ironic deployment - its more for completeness to provide VMs, BM, and containers | 17:07 |
| Serlex | ping blallau | 17:07 |
| Serlex | not here -_- | 17:07 |
| sdake | kfox1111 can you ahve a look at https://review.openstack.org/#/c/464031 | 17:10 |
| sbezverk | sdake: almost, just add some checks | 17:10 |
| sdake | sbezverk check for all kolla namespaced objects? | 17:11 |
| sbezverk | sdake: ideally for all things you remove in your script, just to confirm they are really gone.. | 17:11 |
| sdake | the ansible code removes all kolla namespaced objects | 17:12 |
| sdake | so should be relatively easy | 17:12 |
| sbezverk | sdake: exactly it is one liner check for each thing you remove | 17:12 |
| sdake | could probably just be a one liner in general - did all the namespace objects get deleted | 17:13 |
| *** gfidente is now known as gfidente|afk | 17:13 | |
| sdake | [admin@falkor-node0 oscore]$ kubectl get all -n kolla -o name | 17:13 |
| sdake | No resources found. | 17:13 |
| sbezverk | sdake: not sure if it can be done in one line, example pv are not namespace specific, same as labels | 17:14 |
| sdake | right forgot about the pvs | 17:14 |
| sdake | are pvcs namespace specific? | 17:14 |
| sdake | atleast after 6-9 months i finally understand how kubernetes works :) | 17:15 |
| *** egonzalez has quit IRC | 17:15 | |
| rwellum | sbezverk: everything lines up cluster ip wise but still have the same issue. I cleaned up the cluster completely before starting. Thinking a new VM... | 17:21 |
| rwellum | Maybe doing canal correctly will prevent this issue in the future. | 17:22 |
| *** pbourke has quit IRC | 17:25 | |
| kfox1111 | sdake: we've wanted to use ironic for a while, but the effort involved in standing it up was more then we could afford time wise. | 17:26 |
| kfox1111 | now that its in kolla-kubernetes, the effort is much less. so we may reevaluate it. | 17:26 |
| sdake | kfox1111 it is fairly easy to setup if you have the proper switch config | 17:26 |
| *** mannamne has quit IRC | 17:26 | |
| sdake | (if you desire multitenancy:) | 17:26 |
| sbezverk | sdake: wow, not too many people can say they understand how k8s works, I envy you :) | 17:26 |
| *** pbourke has joined #openstack-kolla | 17:27 | |
| sdake | sbezverk well i am not to your level of knowledge, but gaining fast :) | 17:27 |
| sbezverk | sdake: pvc's are namespace specific, pv's are noty | 17:27 |
| sdake | thanks | 17:27 |
| kfox1111 | sdake: last we looked it didn't support multitenancy at all either. hoping thats stable now. :) | 17:28 |
| sdake | kfox1111 works in my lab | 17:28 |
| sbezverk | sdake: it is wrong perseption, when rwellum comes up with an issue I feel that I know nothing ;) | 17:28 |
| sdake | kfox1111 although I have a nexus 9k switch to make it so | 17:28 |
| sdake | sbezverk ya - i struggle with the weeds as well | 17:28 |
| rwellum | I am not a weed! | 17:29 |
| *** salv-orlando has joined #openstack-kolla | 17:29 | |
| kfox1111 | sdake: we have some 7k's I think, so I think we might be good to go. :) | 17:29 |
| sdake | kfox1111 i think that should work but can't guarantee it | 17:29 |
| sdake | kfox1111 also there is a generic driver that should work on many switches | 17:29 |
| sdake | kfox1111 YMMV on its quality unfortunately | 17:29 |
| sdake | time for some ---- 3000 ---- | 17:30 |
| sdake | minus 1k :) | 17:30 |
| *** harlowja has joined #openstack-kolla | 17:30 | |
| kfox1111 | I've heard a lot of switches fall over if yo ussh into them too often. | 17:32 |
| kfox1111 | they never were designed for it and are buggy. | 17:32 |
| kfox1111 | wouldn't surprise me. | 17:32 |
| sdake | is there any objection to having the PVCs do GC | 17:33 |
| sdake | (configurable of course) | 17:33 |
| kfox1111 | not sure I quite understand the question. | 17:33 |
| *** salv-orlando has quit IRC | 17:34 | |
| kfox1111 | are you talking about dynamic provisioning support in the pvc's? | 17:34 |
| sdake | kfox1111 https://da.gd/4S69 | 17:34 |
| sdake | specifically line 36 | 17:35 |
| sdake | kfox1111 that deletes al lthe crap in the shared volumes on a destroy operation | 17:36 |
| kfox1111 | oh. whether releaseing the pvc automatically reclaims the pv. | 17:36 |
| *** manheim_ has joined #openstack-kolla | 17:36 | |
| kfox1111 | we could just make it configurable. | 17:37 |
| kfox1111 | I kind of like it not reclaiming them for helping dealing with desaster. | 17:37 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:37 |
| kfox1111 | but I can see those with transient clouds prefering to be able to just delete/relaunch them. | 17:37 |
| sdake | kfox1111 ya - config option - not hardcode | 17:38 |
| kfox1111 | k. works for me. | 17:39 |
| sdake | i'd also like to make some of the other stuff in the pvs configurable (or taek them out of the helm chart) - such as storage driver, mode, etc | 17:39 |
| sdake | i'm not really convinced pvs belong in the helm charts - or if they do they beong in their own i think | 17:39 |
| kfox1111 | I'm with you on all of that. | 17:39 |
| sdake | since admins theoretically setup the pvs on their own | 17:39 |
| kfox1111 | pv's are an admin thing. pvc's are a user thing. | 17:39 |
| sdake | right | 17:40 |
| *** mgoddard__ has joined #openstack-kolla | 17:40 | |
| kfox1111 | adding better support for dynamic pv creation in the pvc's is also something we need to do. | 17:40 |
| sdake | could you expand on that, I don't follow :) | 17:40 |
| kfox1111 | its possible now that rather hten create a bunch of pv's up front, | 17:40 |
| kfox1111 | they can setup a driver for a storage class that lets the pvc automatically create a pv on request. | 17:41 |
| sdake | sbezverk kfox1111 my shell-fu is pretty terribad, is there a better way to do this: https://review.openstack.org/#/c/464031/9/tools/setup_gate.sh | 17:41 |
| kfox1111 | sdake: I'd add a ' | wc -l' to the kubectl's, | 17:41 |
| kfox1111 | then you can test $foo -eq 0 | 17:42 |
| sdake | cool | 17:42 |
| sdake | thanks | 17:42 |
| kfox1111 | np. | 17:42 |
| kfox1111 | also, might move that test to its own shell script. would make it more readable in the main gate script. | 17:42 |
| kfox1111 | not the ansible part. just the tests. | 17:42 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:43 |
| sdake | kfox1111 k | 17:43 |
| *** jmccarthy has quit IRC | 17:47 | |
| *** jtriley has joined #openstack-kolla | 17:48 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:48 |
| *** vhosakot has joined #openstack-kolla | 17:48 | |
| sbezverk | kfox1111: if we create PV in charts, it would make sense to remove them as well in clean up, no? | 17:48 |
| sdake | sbezverk pvs are deleted by helm delete operation | 17:49 |
| sdake | erg, only with certain types of charts tho | 17:50 |
| sbezverk | sdake: do you call for these charts from your script? | 17:50 |
| sdake | sbezverk the ansible code lsits all helm charts, then deletes all of them | 17:50 |
| sdake | sbezverk so yes :) | 17:51 |
| sbezverk | sdake: nvm, you hack through with helm delete | 17:51 |
| *** eaguilar has quit IRC | 17:51 | |
| *** fooliouno has quit IRC | 17:55 | |
| sdake | kfox1111 see PM | 17:57 |
| kfox1111 | sbezverk: depends on what the operator wants to do. :/ | 17:57 |
| rwellum | Guys - does anyone know why the compute kit has a funny timestamp - causes a lot of noise when I install? | 17:58 |
| rwellum | https://www.irccloud.com/pastebin/AHBb0lLe/ | 17:58 |
| rwellum | It's older than me! | 17:58 |
| *** jtriley has quit IRC | 17:59 | |
| *** harlowja has quit IRC | 18:01 | |
| sbezverk | rwellum: if I am not mistaken compute kit gets packaged on a local host | 18:01 |
| *** srwilkers has quit IRC | 18:01 | |
| sbezverk | before you deploy it with tools/helm_prebuild_compute_kits.py | 18:02 |
| sbezverk | maybe your clock on local VM is not set? | 18:02 |
| rwellum | I checked with 'date' and it's fine | 18:03 |
| sbezverk | rwellum: have you run helm_prebuild_compute_kits.py | 18:04 |
| sbezverk | ? | 18:04 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:04 |
| rwellum | No - not that I'm aware - unless it's buried in a shell script somewhere | 18:05 |
| *** ducttap__ has joined #openstack-kolla | 18:07 | |
| *** ducttape_ has quit IRC | 18:10 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:12 |
| kfox1111 | rwellum: because go's tar library does weird things. :/ | 18:12 |
| *** manheim_ has quit IRC | 18:12 | |
| kfox1111 | I don't think helm sets a date on the files it adds to the package. and go by default sets an extreamly old time. | 18:13 |
| kfox1111 | I'd call it a bug, but its unclear if its in helm or go | 18:13 |
| rwellum | kfox1111: why just this chart? | 18:14 |
| *** harlowja has joined #openstack-kolla | 18:15 | |
| sbezverk | rwellum: try to regenerate it by running that command | 18:15 |
| sbezverk | curious if new tar will still have pre-historic date ;) | 18:15 |
| rwellum | The helm_prebuild one? | 18:15 |
| sbezverk | rwellum: yes | 18:16 |
| rwellum | Will do once my cluster is up. | 18:16 |
| openstackgerrit | Martin Mágr proposed openstack/kolla master: Sensu images https://review.openstack.org/463593 | 18:17 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:18 |
| *** zul has quit IRC | 18:22 | |
| *** jascott1 has joined #openstack-kolla | 18:24 | |
| sdake | is there any way to determine when helm has completed all of its operations outstanding ? | 18:24 |
| sdake | i.e. tiller is not doing anything and has no work queued | 18:25 |
| rwellum | sbezverk: that command just spat out " Saving 13 charts" | 18:25 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:25 |
| rwellum | sdake: when | grep -v Running | wc -l == 0 ? | 18:26 |
| sbezverk | rwellum: can you check if it has created tar file? | 18:26 |
| rwellum | yes it's there: compute-kit-0.7.0-1.tgz | 18:27 |
| *** jascott1 has quit IRC | 18:29 | |
| *** ducttap__ has quit IRC | 18:29 | |
| *** ducttape_ has joined #openstack-kolla | 18:30 | |
| *** rwsu has quit IRC | 18:32 | |
| *** cali_box has joined #openstack-kolla | 18:35 | |
| cali_box | hello | 18:35 |
| sdake | kfox1111 quick Q - http://logs.openstack.org/31/464031/11/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/f0bc6ed/console.html#_2017-05-18_18_19_56_384561 | 18:35 |
| cali_box | i am having an issue in openstack-kolla deployment | 18:36 |
| cali_box | i am having an issue in openstack-kolla deployment | 18:36 |
| sdake | kfox1111 i notice ansible-playbook is not installed on the system - probably a virtualenv thing | 18:36 |
| sdake | kfox1111 any suggestions to get access to the ansible bits? | 18:36 |
| kfox1111 | hmm.. | 18:36 |
| cali_box | i am getting an error when trying to do kolla-ansible pull at the stage kolla-ansible-toolbox | 18:36 |
| sdake | kfox1111 a different ansible-playbook call (genconfig) works properly | 18:37 |
| cali_box | ansible is pip right | 18:37 |
| cali_box | ? | 18:37 |
| sdake | cali_box yes | 18:37 |
| cali_box | so pip reinstall ansible | 18:37 |
| kfox1111 | sdake: yeah, trying to figure out why the one is working. | 18:37 |
| kfox1111 | tools/setup_gate_common.sh:. .venv/bin/activate | 18:38 |
| kfox1111 | yeah. its in a venv. just do that before. | 18:38 |
| sdake | kfox1111 another problem - in that setup_gate there is a bunch of exit 0 operations | 18:38 |
| sdake | for each gate | 18:38 |
| sdake | not sure how to handle that without duplicating code everywhere or making a function | 18:38 |
| *** tonanhngo has joined #openstack-kolla | 18:40 | |
| sbezverk | sdake: I do not think we need to run it for all flavors | 18:41 |
| sdake | sbezverk you mean all jobs | 18:42 |
| sbezverk | I think selecting the one you hate the most and add it there | 18:42 |
| sbezverk | sdake: right | 18:42 |
| cali_box | fatal: [localhost]: FAILED! => {"changed": true, "failed": true, "msg": "'Traceback (most recent call last):\ | 18:42 |
| cali_box | File \"/tmp/ansible_GZ9qtM/ansible_module_kolla_docker.py\", line 781, in main\ | 18:42 |
| cali_box | result = bool(getattr(dw, module.params.get(\\'action\\'))())\ | 18:42 |
| cali_box | File \"/tmp/ansible_GZ9qtM/ansible_module_kolla_docker.py\", line 445, in pull_image\ | 18:42 |
| cali_box | repository=image, tag=tag, stream=True\ | 18:42 |
| cali_box | File \"/usr/lib/python2.7/site-packages/docker/api/image.py\", line 175, in pull\ | 18:42 |
| cali_box | self._raise_for_status(response)\ | 18:43 |
| sbezverk | sdake: this script, imho, does not require to be tested by all flavors of gate jobs | 18:43 |
| cali_box | File \"/usr/lib/python2.7/site-packages/docker/client.py\", line 173, in _raise_for_status\ | 18:43 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 18:43 |
| cali_box | raise errors.NotFound(e, response, explanation=explanation)\ | 18:43 |
| cali_box | NotFound: 404 Client Error: Not Found (\"{\"message\":\"manifest for kolla/centos-binary-kolla-toolbox:4.0.1 not found\"}\")\ | 18:43 |
| cali_box | '"} | 18:43 |
| sdake | sbezverk yup i had wanted to add a specific gate for it once the deploy workflow is done | 18:43 |
| *** zul has joined #openstack-kolla | 18:43 | |
| sdake | then it would be tested in the confines of the deployment workflow we use | 18:43 |
| cali_box | thats the error i get on kolla-ansible pull | 18:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 18:47 |
| *** eaguilar has joined #openstack-kolla | 18:51 | |
| *** rwsu has joined #openstack-kolla | 18:51 | |
| sdake | berendt around? | 18:52 |
| kfox1111 | sdake: yeah, I think we only need to test it in one gate. | 18:55 |
| sdake | kfox1111 yup - the plan is as follows I'd like to follow: | 18:55 |
| sdake | 1. build destroy chart | 18:55 |
| sdake | rather build destroy ansible playbook | 18:55 |
| sdake | 2. build destroy manifest | 18:55 |
| sdake | 3. build destroy helm chart | 18:55 |
| sdake | 4. build deploy playbook | 18:56 |
| sdake | build deploy container | 18:56 |
| sdake | build deploy manifest | 18:56 |
| sdake | build deploy helm chart | 18:56 |
| sdake | build a bootstrap of the same | 18:56 |
| sdake | and integrate it into one workflow specific gate | 18:56 |
| berendt | sdake: jo | 18:56 |
| vhosakot | cali_box: pastebin is also useful to paste error instead of IRC :) yes, ansible is pip, cal you pull anything (non-kolla) images from your registry? | 18:56 |
| kfox1111 | sdake: sounds good. | 18:56 |
| *** harlowja has quit IRC | 18:56 | |
| sdake | berendt at one point you had offered to build images as needed by the community - does that offer still stand? | 18:56 |
| vhosakot | cali_box: can* you pull anything..... | 18:56 |
| cali_box | i can pull he;llo-world on docker | 18:56 |
| berendt | sdake: yes | 18:57 |
| sdake | berendt cool what do you need to make it happen | 18:57 |
| vhosakot | cali_box: from which registry? dockerhub? | 18:57 |
| sdake | inc0 ^^ | 18:57 |
| cali_box | the default repo (i am fairly new to to docker) | 18:58 |
| berendt | sdake: time.. can we discuss at the beginning of the next week? | 18:58 |
| kfox1111 | sdake: we need to finish up the fingerprinting stuff in the periodic gate. | 18:58 |
| sdake | berendt sounds good :) | 18:58 |
| *** rhallisey has quit IRC | 18:58 | |
| sdake | berendt thanks | 18:58 |
| kfox1111 | then prepair a cronjob that pulls from tarballs and pushes to the hub. | 18:58 |
| kfox1111 | incn0's been working on the latter. | 18:58 |
| sdake | kfox1111 it looks like some of the tc pushing back on the idea of having docker pushed images | 18:58 |
| kfox1111 | I have a prototype of a fingerprinting script up for review. | 18:58 |
| vhosakot | cali_box: you can try the same command (that failed in the kolla_docker ansible module ) manually inside the toolbox container using "docker exec" CLI. | 19:00 |
| cali_box | sure | 19:01 |
| cali_box | but there's no docker image yet | 19:01 |
| cali_box | it doesn't complete pulling the image | 19:01 |
| vhosakot | let me check in my setup | 19:03 |
| *** salv-orlando has joined #openstack-kolla | 19:04 | |
| *** salv-orlando has quit IRC | 19:05 | |
| inc0 | kfox1111: my mini tool does both | 19:06 |
| *** harlowja has joined #openstack-kolla | 19:07 | |
| vhosakot | inc0: finished reading https://review.openstack.org/#/c/447524/ the past 2 days. so, will there be next steps to put all of that into code/cron? I'd like to contribute to it. | 19:10 |
| vhosakot | mnaser: good spec ^^ :) | 19:10 |
| inc0 | vhosakot: issue is, http://lists.openstack.org/pipermail/openstack-dev/2017-May/116677.html | 19:11 |
| vhosakot | yes, I tried my best to understand those emails :) | 19:12 |
| mnaser | yep i think that has to be taken care of first | 19:12 |
| vhosakot | right | 19:12 |
| vhosakot | mnaser: cool meeting you at the summit :) | 19:12 |
| mnaser | vhosakot like wise :) | 19:12 |
| vhosakot | the subject did not fit it in 13-inch laptop ;) [openstack-dev][tc][infra][release][security][stable][kolla][loci][tripleo][docker][kubernetes] do we want to be publishing binary container images? | 19:13 |
| vhosakot | s/laptop/laptop's screen I meant :) | 19:14 |
| cali_box | vhosakot: were you able to find anything on this issue? | 19:16 |
| vhosakot | cali_box: no sorry, my centos setup is down.. will re-deploy it today. | 19:16 |
| sdake | whatever resolution the tc comes up with can be worked around | 19:17 |
| sdake | i wouldn't really be concerned about it | 19:17 |
| *** schwicht has quit IRC | 19:17 | |
| sdake | if the tc doesn't want oversight of the container images -that is their perogative | 19:17 |
| sdake | pretty sure someone else will take on oversight :) | 19:17 |
| *** eaguilar has quit IRC | 19:18 | |
| *** manheim_ has joined #openstack-kolla | 19:19 | |
| *** schwicht has joined #openstack-kolla | 19:20 | |
| vhosakot | sdake: right, I agree. | 19:23 |
| *** mgoddard_ has joined #openstack-kolla | 19:23 | |
| kfox1111 | inc0: you got fingerprinting working too? | 19:24 |
| inc0 | kfox1111: negative, I still don't really have idea how to do it | 19:24 |
| *** gfidente|afk is now known as gfidente | 19:24 | |
| inc0 | other than doing docker run of each image | 19:24 |
| kfox1111 | inc0: the ps I posted does it. | 19:24 |
| *** zul has quit IRC | 19:24 | |
| *** srwilkers has joined #openstack-kolla | 19:24 | |
| kfox1111 | it generates a manifest with the fingerprints. | 19:25 |
| inc0 | remind me link plz? | 19:25 |
| kfox1111 | if we upload them to tarballs.o.o with the containers, the we can compare more easily. | 19:25 |
| inc0 | well, we can add fingerprinting to build | 19:25 |
| kfox1111 | https://review.openstack.org/#/c/458174/ | 19:25 |
| *** mgoddard__ has quit IRC | 19:26 | |
| *** eaguilar has joined #openstack-kolla | 19:26 | |
| kfox1111 | it looks like this: http://logs.openstack.org/74/458174/5/check/gate-kolla-dsvm-deploy-centos-binary-centos-7-nv/68c4d3c/logs/fingerprints.txt.gz | 19:26 |
| inc0 | so ideally I'd love for it to be bound to container itself as for example piece of metadata | 19:27 |
| kfox1111 | we can do that too, | 19:27 |
| kfox1111 | but its not remotely querable as far as I can tell. | 19:27 |
| kfox1111 | we need a way to downlolad the fingerprint without pulling all the containers. | 19:27 |
| inc0 | and I don't have good idea how to pull that stuff from dockerhub without pulling actual images | 19:27 |
| inc0 | to see what needs uploading | 19:27 |
| inc0 | that ^:) | 19:28 |
| kfox1111 | if we cant keep the state in the hub, we can on tarballs.o.o at least. | 19:28 |
| inc0 | 2 sources of truth is asking for trouble | 19:28 |
| kfox1111 | less then ideal for sure. | 19:29 |
| kfox1111 | still. whatever lets it work at all. | 19:29 |
| inc0 | I was also thinking of having container with all the fingerprints | 19:29 |
| *** egonzalez has joined #openstack-kolla | 19:29 | |
| inc0 | then we pull one | 19:29 |
| kfox1111 | that may work... but has similar out of sync issues. | 19:29 |
| inc0 | and regardless we'll have that info bound to container itself | 19:29 |
| inc0 | yeah | 19:30 |
| inc0 | little better but similar | 19:30 |
| kfox1111 | yeah. | 19:30 |
| inc0 | anyhow, I'd start by daily just push | 19:30 |
| inc0 | and go from there | 19:30 |
| inc0 | it's not gonna be easy anyway | 19:30 |
| kfox1111 | won't work. | 19:30 |
| kfox1111 | we need rev numbers. | 19:30 |
| kfox1111 | don't want to be incrementing them daily I think. | 19:30 |
| inc0 | rev numbers are easy | 19:31 |
| *** dave-mccowan has quit IRC | 19:31 | |
| kfox1111 | but I guess worst worst case, we can pull, fingerprint both, compare, upload only the different. | 19:31 |
| inc0 | well, what I'm saying is let's handle one issue at the time | 19:31 |
| kfox1111 | and optomize away the download later. | 19:31 |
| kfox1111 | download is usually cheeper in the cloud then uploading. | 19:32 |
| *** krtaylor has quit IRC | 19:33 | |
| *** eaguilar_ has joined #openstack-kolla | 19:33 | |
| *** eaguilar has quit IRC | 19:34 | |
| sdake | waiting for gates = my idea of fun :) | 19:34 |
| kfox1111 | sdake: need more ps's. :) | 19:35 |
| sdake | kfox1111 indeed | 19:35 |
| *** mannamne has joined #openstack-kolla | 19:36 | |
| inc0 | btf kfox1111 I'll be nearby your area next week;) | 19:36 |
| sdake | waiting for jascott's patch to merge and then we are off to the races :) | 19:36 |
| *** mannamne has quit IRC | 19:36 | |
| sdake | how is this for oxymoronic - my ISP provides IPv6 to 150mbit and slower customers | 19:36 |
| sdake | my ISP only provides IPv4 to gige customers... | 19:37 |
| inc0 | sdake: jascott is on vacation | 19:37 |
| inc0 | so you might want to take it over | 19:37 |
| sdake | inc0 how long? | 19:37 |
| inc0 | week | 19:37 |
| *** mgoddard__ has joined #openstack-kolla | 19:37 | |
| sdake | ok i amy do so then | 19:37 |
| sdake | or break parts of it out | 19:37 |
| inc0 | cool | 19:37 |
| *** krtaylor has joined #openstack-kolla | 19:37 | |
| sdake | (like the deploy of kolla-kubernetes) | 19:37 |
| *** mgoddard_ has quit IRC | 19:39 | |
| *** krtaylor has quit IRC | 19:40 | |
| kfox1111 | inc0: how close by? :) | 19:40 |
| inc0 | just noticed, not that close after all | 19:41 |
| *** krtaylor has joined #openstack-kolla | 19:41 | |
| inc0 | you're in Kennwick right? | 19:41 |
| kfox1111 | pasco/richland. but yeah. | 19:41 |
| kfox1111 | where you going? | 19:41 |
| inc0 | I have corporate event near Dalles | 19:42 |
| inc0 | but it's still pretty far | 19:42 |
| kfox1111 | yeah. thats a few hours. | 19:42 |
| hawi | hmmm, upgrade ended with repairing mariadb. | 19:42 |
| inc0 | hawi: that's not good, what happened? | 19:43 |
| sdake | inc0 re https://review.openstack.org/#/c/464031/ | 19:43 |
| sdake | inc0 shall we put the container that runs that code in kolla repo or kolla-kubernetes | 19:43 |
| sdake | hawi zug | 19:44 |
| sdake | 7 more minutes to timeout | 19:45 |
| * sdake groans | 19:45 | |
| hawi | hmm, it seems that this recovery also doesnt help it up any more | 19:47 |
| inc0 | hawi: what exactly happened? | 19:47 |
| hawi | first kolla-toolbox containers gave fatal, and then when it reached mariadb, Waiting for MariaDB service to be ready, and said Connection reset by peer and that's it | 19:49 |
| sdake | hawl what version are you upgrading to? | 19:49 |
| hawi | 4.0.0 -> 4.0.1 | 19:49 |
| sdake | centos binary? | 19:50 |
| sdake | https://hub.docker.com/r/kolla/centos-binary-kolla-toolbox/tags/ | 19:50 |
| sdake | no toolbox tag for 4.0.1 | 19:50 |
| sdake | this is going to be a problem for upgrades | 19:51 |
| hawi | ubuntu | 19:51 |
| hawi | source | 19:51 |
| sdake | same story: https://hub.docker.com/r/kolla/ubuntu-source-kolla-toolbox/tags/ | 19:51 |
| *** itlinux has joined #openstack-kolla | 19:51 | |
| sdake | there are no 4.0.1 images in the dockerhub at all | 19:51 |
| hawi | i built them locally | 19:52 |
| sdake | roger | 19:52 |
| hawi | what choices i have if any | 19:53 |
| *** bmaltais_ has quit IRC | 19:54 | |
| inc0 | hawi: so first of all, what was the fatal? | 19:56 |
| inc0 | and what versions are you running? | 19:56 |
| hawi | 4.0.0 and | 19:57 |
| hawi | https://pastebin.com/mjk6PY0P | 19:58 |
| *** zul has joined #openstack-kolla | 19:58 | |
| hawi | kolla-toolbox gave it for all storage and compute nodes | 19:58 |
| sdake | kfox1111 this almost works: http://logs.openstack.org/31/464031/13/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/ce27bdf/console.html#_2017-05-18_19_26_06_570606 | 19:59 |
| sdake | kfox1111 what is the 4th PV? | 19:59 |
| hawi | and then next thing was FAILED - RETRYING: TASK: mariadb : Waiting for MariaDB service to be ready, 10x and controllers got socket.error: [Errno 104] Connection reset by peer\ | 20:00 |
| *** mgoddard__ has quit IRC | 20:01 | |
| *** manheim_ has quit IRC | 20:02 | |
| *** harlowja has quit IRC | 20:03 | |
| hawi | ends up again with https://pastebin.com/VWf03EDt | 20:06 |
| hawi | when tried again | 20:06 |
| hawi | no toolbox errors any more | 20:06 |
| hawi | now mariadb containers flap up and down | 20:07 |
| *** eaguilar_ has quit IRC | 20:10 | |
| *** zul has quit IRC | 20:10 | |
| *** rhallisey has joined #openstack-kolla | 20:11 | |
| *** eaguilar has joined #openstack-kolla | 20:11 | |
| hawi | i guess there is now "downgrade" options any more | 20:11 |
| hawi | no | 20:12 |
| *** itlinux has quit IRC | 20:12 | |
| kfox1111 | sdake: not sure without looking. whats it labeled? | 20:18 |
| *** rwallner has quit IRC | 20:18 | |
| hawi | docker versions for stor/compute and controller are different | 20:24 |
| hawi | controllers and machine what i used to build containers (if it matters) has 17.05.0-ce (api 1.29, git commit 89658be) and stor/comp has 17.03.0-ce (api 1.26, git commit 60ccb22) | 20:27 |
| hawi | and running on 16.04.2 lts | 20:27 |
| *** ducttap__ has joined #openstack-kolla | 20:28 | |
| *** ducttape_ has quit IRC | 20:28 | |
| *** zul has joined #openstack-kolla | 20:28 | |
| *** harlowja has joined #openstack-kolla | 20:30 | |
| *** Serlex has quit IRC | 20:31 | |
| *** fguillot has quit IRC | 20:32 | |
| *** krtaylor has quit IRC | 20:34 | |
| *** yee379 has quit IRC | 20:35 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 20:35 |
| *** yee379 has joined #openstack-kolla | 20:35 | |
| sbezverk | sdake: glance/mariadb/rabbitmq/helm-repo pvs | 20:35 |
| *** schwicht has quit IRC | 20:36 | |
| *** schwicht has joined #openstack-kolla | 20:37 | |
| openstackgerrit | serlex proposed openstack/kolla-ansible master: Run adding kibana index pattern once to avoid failures in multi-node setup. HTTP return from consequent hosts will not be 201 as index pattern is PUT already https://review.openstack.org/466101 | 20:38 |
| *** itlinux has joined #openstack-kolla | 20:39 | |
| *** harlowja has quit IRC | 20:39 | |
| *** zul has quit IRC | 20:41 | |
| openstackgerrit | serlex proposed openstack/kolla-ansible master: kibana index pattern only needs to be created once. Consequent requests from other hosts will fail as HTTP status will not be 201 as index pattern is already created. https://review.openstack.org/466101 | 20:41 |
| *** jrist has quit IRC | 20:42 | |
| *** ducttap__ has quit IRC | 20:44 | |
| *** salv-orlando has joined #openstack-kolla | 20:45 | |
| *** Serlex has joined #openstack-kolla | 20:46 | |
| *** Serlex has quit IRC | 20:46 | |
| *** schwicht has quit IRC | 20:46 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 20:50 |
| *** ducttape_ has joined #openstack-kolla | 20:52 | |
| *** jrist has joined #openstack-kolla | 20:56 | |
| *** tonanhngo has quit IRC | 20:56 | |
| *** dave-mccowan has joined #openstack-kolla | 20:57 | |
| *** schwicht has joined #openstack-kolla | 20:57 | |
| *** fooliouno has joined #openstack-kolla | 20:58 | |
| *** tonanhngo has joined #openstack-kolla | 20:59 | |
| *** bmace__ has quit IRC | 21:03 | |
| *** bmace__ has joined #openstack-kolla | 21:03 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 21:05 |
| *** athomas has quit IRC | 21:10 | |
| hawi | so, there is no so straight forward path from 4.0.0 -> 4.0.1 on ubuntu source? | 21:11 |
| inc0 | hawi: how did you install Kolla | 21:11 |
| inc0 | ? | 21:11 |
| inc0 | because if you install from source it'll still mark it as 4.0 even tho it's master | 21:12 |
| *** jrist has quit IRC | 21:15 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 21:20 |
| hawi | inc0: built containers, pushed to local registry, source ubuntu 4.0.0 and did install | 21:23 |
| inc0 | installed with git or pip install? | 21:23 |
| hawi | pip | 21:23 |
| hawi | definitely pip | 21:23 |
| inc0 | so, when you do pip install -U kolla | 21:24 |
| inc0 | that bumped your version to 4.0.1? | 21:24 |
| hawi | yes | 21:24 |
| inc0 | then you should do pip install -U kolla-ansible and run kolla-ansible upgrade | 21:25 |
| inc0 | if it's not working, we have bug or misconfiguration or sth | 21:25 |
| inc0 | so if that failed, tell me what is exact error you're getting | 21:25 |
| inc0 | with toolbox | 21:25 |
| hawi | this is what i actually did | 21:28 |
| hawi | pip -U and kolla-ansible upgrade | 21:28 |
| inc0 | then let's see what error did you get | 21:30 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements https://review.openstack.org/464393 | 21:31 |
| egonzalez | hawi what docker storage backend you use? had that error with devicemapper before | 21:32 |
| hawi | ceph | 21:33 |
| egonzalez | storage for docker service, docker info will say | 21:33 |
| hawi | a, i see, sorry | 21:34 |
| hawi | https://pastebin.com/2SCty2EX | 21:37 |
| hawi | tbh, didn't expect that | 21:37 |
| Reepicheep | I'm not sure on Ubuntu but I know I had issues with the newer versions of Docker on CentOS | 21:38 |
| Reepicheep | I currently have mine pinned to 1.12.6 | 21:38 |
| inc0 | ocata is pinned to 1.12.* too | 21:40 |
| *** schwicht has quit IRC | 21:41 | |
| Reepicheep | could that be part of hawi's issues? | 21:42 |
| hawi | i'm slightly in trouble understanding what version do i have | 21:42 |
| Reepicheep | hawi: looks like 17.03.0-ce | 21:43 |
| egonzalez | hawi 17.03.0-ce new docker's versioning | 21:43 |
| hawi | but it seems to be newer or ... | 21:43 |
| Reepicheep | they changed version format | 21:43 |
| hawi | aaaa, i seee | 21:43 |
| egonzalez | hawi is your data important or is just a demo/test environment so you can destroy and try upgrading with docker 1.12.*? | 21:44 |
| hawi | it definitely worked for 17.03.0-ce version, except one fluentd container flaping up-down | 21:44 |
| inc0 | oh don't use 17.03 and ocata | 21:45 |
| hawi | no data (except some vm-s with prepared configs), i do have configuration that is pretty much the thing i wanted to have as prod. | 21:45 |
| inc0 | api changed in docker (fun right?) | 21:45 |
| inc0 | downgrade it to 1.12.* | 21:46 |
| hawi | yes, i hope i can have it running again | 21:46 |
| Reepicheep | lol, there seems to be two very import to kolla upstream projects that have a bad habbit of breaking things docker & k8s :| | 21:46 |
| Reepicheep | both for very different reasons | 21:47 |
| *** ducttape_ has quit IRC | 21:48 | |
| *** ducttape_ has joined #openstack-kolla | 21:49 | |
| hawi | guys gonna kill me tomorrow | 21:49 |
| hawi | that's bad. good is that i can still go to sleep :) | 21:49 |
| sdake | sbezverk thanks | 21:52 |
| *** haplo37 has quit IRC | 21:53 | |
| *** manheim_ has joined #openstack-kolla | 21:54 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 21:57 |
| *** manheim_ has quit IRC | 21:58 | |
| hawi | i think this is the main reason i have had quite weird things | 22:00 |
| *** yee379 has quit IRC | 22:01 | |
| *** yee379 has joined #openstack-kolla | 22:01 | |
| *** AzazeL_ has quit IRC | 22:02 | |
| *** haplo37 has joined #openstack-kolla | 22:02 | |
| *** itlinux has quit IRC | 22:03 | |
| *** itlinux has joined #openstack-kolla | 22:04 | |
| *** Jeffrey4l_ has quit IRC | 22:08 | |
| *** schwicht has joined #openstack-kolla | 22:10 | |
| *** yuanying has quit IRC | 22:12 | |
| *** yuanying has joined #openstack-kolla | 22:12 | |
| *** Jeffrey4l has joined #openstack-kolla | 22:13 | |
| *** fooliouno has quit IRC | 22:15 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 22:19 |
| *** salv-orlando has quit IRC | 22:29 | |
| *** salv-orlando has joined #openstack-kolla | 22:34 | |
| *** schwicht has quit IRC | 22:35 | |
| *** schwicht has joined #openstack-kolla | 22:36 | |
| *** salv-orlando has quit IRC | 22:38 | |
| *** ducttape_ has quit IRC | 22:40 | |
| *** ducttape_ has joined #openstack-kolla | 22:41 | |
| *** schwicht has quit IRC | 22:42 | |
| *** powerd has quit IRC | 22:43 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 22:44 |
| *** unicell has joined #openstack-kolla | 22:44 | |
| *** itlinux has quit IRC | 22:44 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 22:49 |
| *** srwilkers has quit IRC | 22:54 | |
| openstackgerrit | Dan Ardelean proposed openstack/kolla-ansible master: Add Hyper-V role https://review.openstack.org/455684 | 23:00 |
| *** srwilkers has joined #openstack-kolla | 23:06 | |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 23:08 |
| *** Mannheim has quit IRC | 23:08 | |
| sdake | servers are here | 23:09 |
| *** powerd has joined #openstack-kolla | 23:09 | |
| *** goldyfruit has quit IRC | 23:12 | |
| *** schwicht has joined #openstack-kolla | 23:17 | |
| *** manheim_ has joined #openstack-kolla | 23:21 | |
| *** salv-orlando has joined #openstack-kolla | 23:22 | |
| *** eaguilar has joined #openstack-kolla | 23:23 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 23:24 |
| *** egonzalez has quit IRC | 23:25 | |
| *** manheim_ has quit IRC | 23:25 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 23:26 |
| *** salv-orlando has quit IRC | 23:26 | |
| *** ducttape_ has quit IRC | 23:29 | |
| *** salv-orlando has joined #openstack-kolla | 23:31 | |
| *** salv-orlando has quit IRC | 23:31 | |
| *** schwicht has quit IRC | 23:35 | |
| *** powerd has quit IRC | 23:38 | |
| *** manheim_ has joined #openstack-kolla | 23:39 | |
| *** vhosakot has quit IRC | 23:39 | |
| *** mattmceuen has joined #openstack-kolla | 23:42 | |
| *** eaguilar has quit IRC | 23:44 | |
| *** kbaegis has joined #openstack-kolla | 23:47 | |
| *** schwicht has joined #openstack-kolla | 23:49 | |
| *** schwicht has quit IRC | 23:49 | |
| sdake | Jeffrey4l about? | 23:54 |
| Jeffrey4l | sdake, yep. sup | 23:55 |
| sdake | hey dude just haven't seen you for while | 23:56 |
| sdake | wanted to say hi:) | 23:56 |
| Jeffrey4l | aha, ;) | 23:56 |
| sdake | i have 2 m0ar servers | 23:57 |
| Jeffrey4l | cool. | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!