| *** JamesBenson has joined #openstack-kolla | 00:04 | |
| *** dardelea_ has joined #openstack-kolla | 00:06 | |
| *** dardelean has quit IRC | 00:07 | |
| *** mnasiadka has joined #openstack-kolla | 00:08 | |
| *** JamesBenson has quit IRC | 00:08 | |
| *** bmace_ has joined #openstack-kolla | 00:09 | |
| *** mnasiadka has quit IRC | 00:12 | |
| *** goldyfruit has quit IRC | 00:20 | |
| *** mnasiadka has joined #openstack-kolla | 00:23 | |
| *** pradk has joined #openstack-kolla | 00:23 | |
| *** mrunge_ has quit IRC | 00:25 | |
| *** hw_wutianwei has quit IRC | 00:26 | |
| *** mrunge has joined #openstack-kolla | 00:26 | |
| *** hw_wutianwei has joined #openstack-kolla | 00:26 | |
| *** k_mouza has quit IRC | 00:26 | |
| *** mnasiadka has quit IRC | 00:27 | |
| *** k_mouza has joined #openstack-kolla | 00:35 | |
| *** pramodrj07 has quit IRC | 00:35 | |
| *** mnasiadka has joined #openstack-kolla | 00:38 | |
| *** k_mouza has quit IRC | 00:40 | |
| *** mnasiadka has quit IRC | 00:43 | |
| *** tovin07_ has joined #openstack-kolla | 00:51 | |
| *** mnasiadka has joined #openstack-kolla | 00:53 | |
| *** dardelea_ has quit IRC | 00:54 | |
| kfox1111 | rwellum: not there yet, but: http://logs.openstack.org/17/543617/1/check/legacy-kolla-kubernetes-deploy-centos-binary-2-external-ovs/e8b541f/logs/openstack_server_list.txt | 00:56 |
|---|---|---|
| *** mnasiadka has quit IRC | 00:58 | |
| kfox1111 | http://logs.openstack.org/17/543617/1/check/legacy-kolla-kubernetes-deploy-centos-binary-2-external-ovs/e8b541f/logs/openstack_port_list.txt looks odd. | 00:58 |
| kfox1111 | so, it looks like the lack of client only effected the debug loggin. | 00:59 |
| kfox1111 | logging. | 00:59 |
| *** zhaochao has joined #openstack-kolla | 01:00 | |
| *** openstackgerrit has quit IRC | 01:03 | |
| *** pengdake_ has joined #openstack-kolla | 01:04 | |
| *** openstackgerrit has joined #openstack-kolla | 01:07 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fix the gate. https://review.openstack.org/543617 | 01:07 |
| *** mnasiadka has joined #openstack-kolla | 01:08 | |
| *** yangyapeng has quit IRC | 01:12 | |
| *** mnasiadka has quit IRC | 01:12 | |
| *** yangyapeng has joined #openstack-kolla | 01:12 | |
| *** jtriley has joined #openstack-kolla | 01:14 | |
| *** itlinux has joined #openstack-kolla | 01:15 | |
| kfox1111 | rwellum: if you get a minute to play... I suspect openstackclient/neutron/nova clients got updated and behave maybe differently. | 01:23 |
| *** mdnadeem has joined #openstack-kolla | 01:23 | |
| *** mnasiadka has joined #openstack-kolla | 01:23 | |
| *** signed8bit_Zzz is now known as signed8b_ | 01:24 | |
| *** signed8b_ is now known as signed8bit_Zzz | 01:25 | |
| *** mnasiadka has quit IRC | 01:28 | |
| *** mnasiadka has joined #openstack-kolla | 01:39 | |
| *** jtriley has quit IRC | 01:41 | |
| *** itlinux has quit IRC | 01:43 | |
| *** mnasiadka has quit IRC | 01:43 | |
| SamYaple | kfox1111: i think neutron and nova stopped getting distributed with openstack client | 01:43 |
| *** yangyapeng has quit IRC | 01:53 | |
| *** mnasiadka has joined #openstack-kolla | 01:54 | |
| *** yangyapeng has joined #openstack-kolla | 01:54 | |
| *** itlinux has joined #openstack-kolla | 01:56 | |
| *** kiennt26 has joined #openstack-kolla | 01:57 | |
| *** mnasiadka has quit IRC | 01:59 | |
| *** goldyfruit has joined #openstack-kolla | 02:02 | |
| openstackgerrit | Qi Peng proposed openstack/kolla-ansible master: Update task about selinux set. https://review.openstack.org/539835 | 02:03 |
| openstackgerrit | Qi Peng proposed openstack/kolla-ansible master: Update task about selinux set. https://review.openstack.org/539835 | 02:05 |
| *** signed8bit_Zzz has quit IRC | 02:06 | |
| openstackgerrit | Qi Peng proposed openstack/kolla-ansible master: Update task about selinux set. https://review.openstack.org/539835 | 02:07 |
| *** mnasiadka has joined #openstack-kolla | 02:09 | |
| *** mnasiadka has quit IRC | 02:13 | |
| *** salv-orl_ has joined #openstack-kolla | 02:16 | |
| *** salv-orlando has quit IRC | 02:19 | |
| *** k_mouza has joined #openstack-kolla | 02:21 | |
| *** itlinux has quit IRC | 02:22 | |
| *** mnasiadka has joined #openstack-kolla | 02:23 | |
| *** kevzha01_ has joined #openstack-kolla | 02:24 | |
| *** kevzha01 has joined #openstack-kolla | 02:24 | |
| *** kevzha01_ is now known as kevinz | 02:24 | |
| *** kevzha01 has quit IRC | 02:25 | |
| *** k_mouza has quit IRC | 02:26 | |
| *** itlinux has joined #openstack-kolla | 02:26 | |
| *** mnasiadka has quit IRC | 02:28 | |
| *** mnasiadka has joined #openstack-kolla | 02:38 | |
| *** hongbin has joined #openstack-kolla | 02:38 | |
| *** itlinux has quit IRC | 02:40 | |
| *** mnasiadka has quit IRC | 02:43 | |
| *** mnasiadka has joined #openstack-kolla | 02:53 | |
| *** dardelean has joined #openstack-kolla | 02:55 | |
| *** tovin07_ has quit IRC | 02:56 | |
| *** mnasiadka has quit IRC | 02:58 | |
| *** dardelean has quit IRC | 02:59 | |
| *** chmarkus has quit IRC | 03:00 | |
| *** chmarkus has joined #openstack-kolla | 03:02 | |
| *** harlowja has quit IRC | 03:04 | |
| *** mnasiadka has joined #openstack-kolla | 03:08 | |
| *** hnl has joined #openstack-kolla | 03:11 | |
| *** mnasiadka has quit IRC | 03:13 | |
| *** unicell has quit IRC | 03:16 | |
| *** mnasiadka has joined #openstack-kolla | 03:21 | |
| openstackgerrit | Trinh Nguyen proposed openstack/kolla-ansible master: Use bash instead of sh when calling deploy-tacker-demo-sfc https://review.openstack.org/512025 | 03:26 |
| *** mnasiadka has quit IRC | 03:27 | |
| *** yangyapeng has quit IRC | 03:28 | |
| *** dangtrinhnt has joined #openstack-kolla | 03:29 | |
| *** yangyapeng has joined #openstack-kolla | 03:29 | |
| *** yangyapeng has quit IRC | 03:33 | |
| *** mnasiadka has joined #openstack-kolla | 03:38 | |
| *** mnasiadka has quit IRC | 03:43 | |
| *** mnasiadka has joined #openstack-kolla | 03:53 | |
| *** dave-mccowan has quit IRC | 03:56 | |
| *** mnasiadka has quit IRC | 03:57 | |
| *** gkadam has joined #openstack-kolla | 04:04 | |
| *** liuzz has quit IRC | 04:06 | |
| *** mnasiadka has joined #openstack-kolla | 04:08 | |
| *** pengdake_ has quit IRC | 04:11 | |
| *** goldyfruit has quit IRC | 04:13 | |
| *** mnasiadka has quit IRC | 04:13 | |
| *** mnasiadka has joined #openstack-kolla | 04:23 | |
| *** mnasiadka has quit IRC | 04:27 | |
| *** kiennt26 has quit IRC | 04:32 | |
| *** lpetrut has joined #openstack-kolla | 04:37 | |
| *** mnasiadka has joined #openstack-kolla | 04:38 | |
| *** pengdake_ has joined #openstack-kolla | 04:40 | |
| *** mnasiadka has quit IRC | 04:43 | |
| *** skramaja has joined #openstack-kolla | 04:47 | |
| *** Teck7 has quit IRC | 04:49 | |
| *** rfxn has joined #openstack-kolla | 04:50 | |
| *** threestrands has quit IRC | 04:50 | |
| *** mnasiadka has joined #openstack-kolla | 04:53 | |
| *** kevinz has quit IRC | 04:57 | |
| *** mnasiadka has quit IRC | 04:58 | |
| *** Teck7 has joined #openstack-kolla | 05:02 | |
| *** caoyuan has joined #openstack-kolla | 05:02 | |
| *** rfxn has quit IRC | 05:05 | |
| *** lpetrut has quit IRC | 05:05 | |
| *** mnasiadka has joined #openstack-kolla | 05:08 | |
| *** lpetrut has joined #openstack-kolla | 05:11 | |
| *** rfxn has joined #openstack-kolla | 05:12 | |
| *** mnasiadka has quit IRC | 05:13 | |
| *** pengdake_ has quit IRC | 05:15 | |
| *** Teck7 has quit IRC | 05:16 | |
| *** mdnadeem has quit IRC | 05:18 | |
| openstackgerrit | Merged openstack/kolla master: Seem there are some wrong in infra ubuntu repo https://review.openstack.org/543387 | 05:22 |
| *** Teck7 has joined #openstack-kolla | 05:22 | |
| *** mnasiadka has joined #openstack-kolla | 05:23 | |
| *** rfxn has quit IRC | 05:25 | |
| *** mnasiadka has quit IRC | 05:28 | |
| *** lpetrut has quit IRC | 05:29 | |
| *** hongbin has quit IRC | 05:37 | |
| *** mnasiadka has joined #openstack-kolla | 05:38 | |
| *** janki has joined #openstack-kolla | 05:43 | |
| *** mnasiadka has quit IRC | 05:44 | |
| *** harlowja has joined #openstack-kolla | 05:52 | |
| *** mnasiadka has joined #openstack-kolla | 05:53 | |
| *** pengdake_ has joined #openstack-kolla | 05:54 | |
| *** mdnadeem has joined #openstack-kolla | 05:55 | |
| *** mnasiadka has quit IRC | 05:58 | |
| *** itlinux has joined #openstack-kolla | 05:59 | |
| *** mnasiadka has joined #openstack-kolla | 06:06 | |
| *** itlinux has quit IRC | 06:08 | |
| *** mnasiadka has quit IRC | 06:12 | |
| *** kiennt26 has joined #openstack-kolla | 06:13 | |
| *** hieulq has joined #openstack-kolla | 06:17 | |
| *** mnasiadka has joined #openstack-kolla | 06:23 | |
| duonghq | Jeffrey4l, how long is the new year vacation at China? | 06:26 |
| *** mnasiadka has quit IRC | 06:30 | |
| *** caoyuan_ has joined #openstack-kolla | 06:33 | |
| *** harlowja has quit IRC | 06:33 | |
| *** caoyuan_ has quit IRC | 06:37 | |
| openstackgerrit | Saravanan KR proposed openstack/kolla master: Add DPDK support for centos and rhel family in ovsdpdk container https://review.openstack.org/512247 | 06:50 |
| *** unicell has joined #openstack-kolla | 07:04 | |
| *** caoyuan_ has joined #openstack-kolla | 07:04 | |
| *** caoyuan has quit IRC | 07:04 | |
| *** olivierbourdon38 has quit IRC | 07:07 | |
| *** k_mouza has joined #openstack-kolla | 07:08 | |
| *** k_mouza has quit IRC | 07:12 | |
| *** egonzalez has joined #openstack-kolla | 07:13 | |
| *** unicell1 has joined #openstack-kolla | 07:14 | |
| *** unicell has quit IRC | 07:15 | |
| *** lpetrut has joined #openstack-kolla | 07:15 | |
| *** mnasiadka has joined #openstack-kolla | 07:16 | |
| *** hieulq has quit IRC | 07:17 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Use apt Ansible module instead of apt-get to update cache https://review.openstack.org/540872 | 07:20 |
| *** b_bezak has joined #openstack-kolla | 07:21 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Test fernet deploy https://review.openstack.org/484309 | 07:24 |
| *** numans_ has joined #openstack-kolla | 07:24 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Update task about selinux set. https://review.openstack.org/539835 | 07:24 |
| *** b_bezak has quit IRC | 07:26 | |
| openstackgerrit | Trinh Nguyen proposed openstack/kolla-ansible master: Use bash instead of sh when calling deploy-tacker-demo-sfc https://review.openstack.org/512025 | 07:27 |
| *** numans has quit IRC | 07:28 | |
| *** AJaeger has quit IRC | 07:29 | |
| *** zhangfei has joined #openstack-kolla | 07:31 | |
| Jeffrey4l | duonghq, 7 days | 07:31 |
| *** hnl has quit IRC | 07:31 | |
| Jeffrey4l | rwellum, there are 3 days now. wednesday to friday | 07:31 |
| Jeffrey4l | how about Vietnam? | 07:32 |
| Jeffrey4l | duonghq, ^^ | 07:32 |
| *** k_mouza has joined #openstack-kolla | 07:35 | |
| *** numans_ has quit IRC | 07:35 | |
| *** olivierbourdon38 has joined #openstack-kolla | 07:37 | |
| *** k_mouza has quit IRC | 07:39 | |
| *** numans has joined #openstack-kolla | 07:40 | |
| *** pcaruana has joined #openstack-kolla | 07:41 | |
| *** lvdombrkr has joined #openstack-kolla | 07:44 | |
| *** dciabrin_ has joined #openstack-kolla | 07:45 | |
| *** dciabrin_ has quit IRC | 07:45 | |
| *** dciabrin_ has joined #openstack-kolla | 07:46 | |
| openstackgerrit | Merged openstack/kolla master: Use absolute path in install_pip macro https://review.openstack.org/539859 | 07:46 |
| openstackgerrit | Merged openstack/kolla master: Fix ceilometer deployment https://review.openstack.org/542481 | 07:48 |
| *** dciabrin has quit IRC | 07:48 | |
| *** dciabrin_ has quit IRC | 07:48 | |
| duonghq | Jeffrey4l, the same here, 7 days included weekends | 07:48 |
| *** dciabrin_ has joined #openstack-kolla | 07:48 | |
| *** duonghq has quit IRC | 08:01 | |
| *** dardelean has joined #openstack-kolla | 08:02 | |
| *** duonghq has joined #openstack-kolla | 08:02 | |
| *** b_bezak has joined #openstack-kolla | 08:03 | |
| *** olivierbourdon38 has quit IRC | 08:06 | |
| *** numans_ has joined #openstack-kolla | 08:10 | |
| *** numans has quit IRC | 08:13 | |
| *** numans_ has quit IRC | 08:15 | |
| *** hieulq has joined #openstack-kolla | 08:18 | |
| *** xinliang has quit IRC | 08:23 | |
| *** dardelean has quit IRC | 08:29 | |
| *** hamza21 has joined #openstack-kolla | 08:29 | |
| *** dardelean has joined #openstack-kolla | 08:29 | |
| openstackgerrit | Merged openstack/kolla master: Remove heat-api-cloudwatch image https://review.openstack.org/540714 | 08:30 |
| *** dardelean has quit IRC | 08:34 | |
| *** hw_wutianwei_ has joined #openstack-kolla | 08:36 | |
| *** robbbe has joined #openstack-kolla | 08:37 | |
| *** gfidente has joined #openstack-kolla | 08:38 | |
| *** gfidente has quit IRC | 08:38 | |
| *** gfidente has joined #openstack-kolla | 08:38 | |
| *** dardelean has joined #openstack-kolla | 08:40 | |
| *** jmccarthy has joined #openstack-kolla | 08:42 | |
| *** ktibi has joined #openstack-kolla | 08:54 | |
| *** k_mouza has joined #openstack-kolla | 08:58 | |
| *** skramaja_ has joined #openstack-kolla | 09:06 | |
| *** kong has quit IRC | 09:07 | |
| lvdombrkr | hello folks, after i run kolla-ansible reconfigure my deployment failed on TASK [mariadb : Waiting for MariaDB service to be ready through VIP] and mariadb container loops restarting.. this is logs i collected, http://paste.openstack.org/raw/670973/ | 09:07 |
| lvdombrkr | maybe any quick ideas? | 09:07 |
| *** skramaja has quit IRC | 09:09 | |
| *** k_mouza has quit IRC | 09:11 | |
| hrw | morning | 09:12 |
| *** dardelea_ has joined #openstack-kolla | 09:12 | |
| *** dardelea_ has quit IRC | 09:17 | |
| dangtrinhnt | Hi lvdombrkr, looks like there's something wrong with your VIP interface `Waiting for MariaDB service to be ready through VIP` | 09:20 |
| *** dardelea_ has joined #openstack-kolla | 09:20 | |
| lvdombrkr | dangtrinhnt: but why mariadb container not started? is restarting | 09:23 |
| dangtrinhnt | It guess It's failed to start the container so it trying to restart it. | 09:25 |
| *** k_mouza has joined #openstack-kolla | 09:26 | |
| lvdombrkr | dangtrinhnt: but if i tried start container manually docker start mariadb is continue restarting not starting | 09:26 |
| *** k_mouza has quit IRC | 09:28 | |
| *** k_mouza has joined #openstack-kolla | 09:29 | |
| dangtrinhnt | try this command to see the logs: docker logs --tail 50 --follow --timestamps <container name> | 09:29 |
| lvdombrkr | dangtrinhnt: http://paste.openstack.org/raw/670978/ | 09:31 |
| *** tvignaud has quit IRC | 09:33 | |
| *** gkadam_ has joined #openstack-kolla | 09:34 | |
| *** gkadam has quit IRC | 09:36 | |
| *** n0isyn0ise has quit IRC | 09:38 | |
| *** tvignaud has joined #openstack-kolla | 09:38 | |
| *** skramaja_ is now known as skramaja | 09:41 | |
| dangtrinhnt | Are you sure that 94.140.122.148 is working? | 09:41 |
| *** ktibi has quit IRC | 09:43 | |
| *** ktibi has joined #openstack-kolla | 09:43 | |
| ktibi | Hi kolla, Anyone have already this kernel error : < kernel:unregister_netdevice: waiting for qr-80dff4f8-bc to become free. Usage count = 1 > ? | 09:43 |
| *** dciabrin has joined #openstack-kolla | 09:45 | |
| lvdombrkr | dangtrinhnt: how can i check it? there is no problem to ping it | 09:46 |
| *** dciabrin_ has quit IRC | 09:47 | |
| *** k_mouza has quit IRC | 09:47 | |
| ktibi | I think I have a clean pb with neutron. neutron can't clean all old namespace on a specific controler :/ | 09:47 |
| dangtrinhnt | lvdombrkr, Any port conflict or firewall? Or, maybe the image itself has been broken. | 09:47 |
| *** b_bezak has quit IRC | 09:50 | |
| lvdombrkr | dangtrinhnt: it worked well before i run reconfigure today, so its not image | 09:50 |
| *** numans has joined #openstack-kolla | 09:50 | |
| *** b_bezak has joined #openstack-kolla | 09:50 | |
| dangtrinhnt | Hi ktibi, maybe this relevant to your bug? https://github.com/moby/moby/issues/5618 | 09:52 |
| dangtrinhnt | lvdombrkr, what did you changed? | 09:52 |
| *** b_bezak_ has joined #openstack-kolla | 09:54 | |
| lvdombrkr | dangtrinhnt: some neutron conf, change api_workers, and rpc_workers, but i think it is not issue | 09:54 |
| *** b_bezak has quit IRC | 09:55 | |
| ktibi | dangtrinhnt, yes maybe, but I use last docker version | 09:55 |
| *** duonghq has quit IRC | 09:58 | |
| *** kiennt26 has quit IRC | 09:59 | |
| *** hieulq has quit IRC | 10:02 | |
| *** pengdake_ has quit IRC | 10:05 | |
| dangtrinhnt | lvdombrkr, must be something wrong with your neutron conf | 10:07 |
| dangtrinhnt | ktibi, maybe it's your docker images. | 10:07 |
| ktibi | dangtrinhnt, I use official dockerfile from kolla, I have this bug since I enable ha on neutron. So when I create a dhcp, I have 1 dhcp on each ctrl. | 10:08 |
| ktibi | And I don't know why, but one ctrl on three, fail to delete own dhcp | 10:09 |
| ktibi | how can I start neutron-netns-cleanup with kolla ? | 10:11 |
| ktibi | directly in the container ? | 10:11 |
| openstackgerrit | wangwei proposed openstack/kolla-ansible master: Fix ipc_mode comparison in the latest docker https://review.openstack.org/541160 | 10:13 |
| lvdombrkr | dangtrinhnt: i returned config as before but no success same failed state | 10:18 |
| *** shardy has quit IRC | 10:19 | |
| *** shardy has joined #openstack-kolla | 10:21 | |
| lvdombrkr | egonzalez: hi. are you somewhere around? | 10:24 |
| egonzalez | lvdombrkr, yep | 10:38 |
| *** dardelea_ has quit IRC | 10:39 | |
| *** dardelea_ has joined #openstack-kolla | 10:40 | |
| lvdombrkr | egonzalez: if you have minute can you look into i write erlier : after i run kolla-ansible reconfigure my deployment failed on TASK [mariadb : Waiting for MariaDB service to be ready through VIP] and mariadb container loops restarting.. this is logs i collected, http://paste.openstack.org/raw/670973/ | 10:42 |
| lvdombrkr | egonzalez: maybe some quick ideas or clues | 10:42 |
| *** skramaja has quit IRC | 10:44 | |
| egonzalez | lvdombrkr, the image changed? | 10:51 |
| egonzalez | what versions? | 10:51 |
| lvdombrkr | egonzalez: i changed only some neutron.conf before run reconfigure, no manipulations with images or other changes | 10:54 |
| lvdombrkr | tag 4.0.0 | 10:54 |
| *** n0isyn0ise has joined #openstack-kolla | 10:55 | |
| egonzalez | lvdombrkr, reconfigure shouldnt be starting mariadb containers if they were already up | 10:55 |
| egonzalez | did something changed in any mariadb task | 10:55 |
| egonzalez | or maybe the containers were restarting before | 10:55 |
| *** dardelea_ has quit IRC | 10:56 | |
| openstackgerrit | Kevin Tibi proposed openstack/kolla-ansible master: Fix SSL api for multiple services https://review.openstack.org/543488 | 10:56 |
| *** owalsh_ has joined #openstack-kolla | 10:57 | |
| lvdombrkr | egonzalez: maybe it was restarting before i didnt check to tell the truth.. but i can confirm that yesterday all containers was up and running | 10:57 |
| lvdombrkr | before run reconfigure today i didnt check | 10:57 |
| *** owalsh has quit IRC | 10:59 | |
| *** owalsh has joined #openstack-kolla | 11:01 | |
| *** dardelea_ has joined #openstack-kolla | 11:02 | |
| *** owalsh_ has quit IRC | 11:03 | |
| egonzalez | ktibi, why need https in the request for services if services does not have https | 11:06 |
| egonzalez | haproxy is an ssl termination proxy, so everything behind it is in plain http | 11:07 |
| ktibi | egonzalez, because some client use url return by API for make the next call | 11:07 |
| ktibi | api return an URL which doesn't exist :/ | 11:07 |
| egonzalez | will fail because the https in the uri dont exists | 11:08 |
| ktibi | nop | 11:08 |
| ktibi | it's like endpoint | 11:08 |
| ktibi | kolla add https in public endpoint for each service | 11:08 |
| ktibi | so some clients use endpoint for construct the API call, but sometimes (like tempest) clients use the URI return by api like /v1 or /v2, and when you call the public API on haproxy (with SSL) api return http://public instead of https://public | 11:10 |
| ktibi | egonzalez, so the client fail to contact http://public because haproxy expose just SSL ! | 11:10 |
| egonzalez | oo, right, gotcha | 11:10 |
| egonzalez | only those services are affected by the issue? i guess probably most of services are | 11:10 |
| *** k_mouza has joined #openstack-kolla | 11:14 | |
| *** owalsh_ has joined #openstack-kolla | 11:14 | |
| lvdombrkr | egonzalez: any ideas about troubleshooting? | 11:17 |
| *** owalsh has quit IRC | 11:18 | |
| *** goldyfruit has joined #openstack-kolla | 11:19 | |
| *** owalsh_ is now known as owalsh | 11:20 | |
| *** shardy has quit IRC | 11:20 | |
| *** shardy has joined #openstack-kolla | 11:21 | |
| *** mnasiadka has quit IRC | 11:27 | |
| *** pbourke has quit IRC | 11:27 | |
| *** pbourke has joined #openstack-kolla | 11:29 | |
| *** goldyfruit has quit IRC | 11:31 | |
| *** goldyfruit has joined #openstack-kolla | 11:31 | |
| *** shardy has quit IRC | 11:35 | |
| *** dasTor has joined #openstack-kolla | 11:41 | |
| *** olivierbourdon38 has joined #openstack-kolla | 11:47 | |
| *** b_bezak_ has quit IRC | 11:48 | |
| *** b_bezak has joined #openstack-kolla | 11:48 | |
| *** mwhahaha has quit IRC | 11:51 | |
| dangtrinhnt | lvdombrbk, I think last time I got the same problem with 5.0.0 and I have to build my own images (5.0.1): https://hub.docker.com/u/dangtrinhnt/ | 11:51 |
| *** mwhahaha has joined #openstack-kolla | 11:52 | |
| *** shardy has joined #openstack-kolla | 11:52 | |
| *** b_bezak has quit IRC | 11:53 | |
| lvdombrkr | dangtrinhnt: maybe, but i need to understood what exactly happened and why... because yesterday it worked well | 11:57 |
| lvdombrkr | it not fresh deployment etc... it was working well for 8-10 months | 11:58 |
| lvdombrkr | till today | 11:58 |
| *** aikema has joined #openstack-kolla | 11:58 | |
| dangtrinhnt | Is there any thing changed on your servers? upgrades? new firewall rules? | 11:59 |
| egonzalez | lvdombrkr, if the image changed, changes o ansible are required | 11:59 |
| egonzalez | ahve you built/pull newer images | 11:59 |
| lvdombrkr | egonzalez: i have not change anything ( not pull new images etc) and will not do that until will understand what caused my issue | 12:01 |
| *** mnasiadka has joined #openstack-kolla | 12:01 | |
| egonzalez | lvdombrkr, do you have the reconfigure logs? | 12:02 |
| *** portdirect has quit IRC | 12:03 | |
| *** portdirect has joined #openstack-kolla | 12:03 | |
| lvdombrkr | egonzalez: it should be under /var/lib/docker/volumes/kolla_logs/_data/ ? | 12:03 |
| egonzalez | nope, the execution output | 12:04 |
| lvdombrkr | dangtrinhnt: not change anything | 12:04 |
| egonzalez | there are only the service log | 12:04 |
| egonzalez | want to see if something in mariadb changed or is because container was restarting before | 12:04 |
| egonzalez | because the role is idempotent and shouldnt change anything if nothing changes | 12:05 |
| lvdombrkr | egonzalez: i can run reconfigure right now again and show you output if it will matters | 12:06 |
| egonzalez | nope, the service is already broken | 12:06 |
| egonzalez | need the why failed the first time | 12:06 |
| egonzalez | in mariadb logs there is something about "gcomm cannot connect to" error ? | 12:07 |
| *** jbryce has quit IRC | 12:11 | |
| *** jbryce has joined #openstack-kolla | 12:11 | |
| *** hamza21 has quit IRC | 12:15 | |
| lvdombrkr | egonzalez: thsis is output from reconfigure execution log, when i run reconfigure first time, i see something very similar : http://paste.openstack.org/raw/670994/ | 12:16 |
| egonzalez | lvdombrkr, was something before? | 12:17 |
| lvdombrkr | egonzalez: http://paste.openstack.org/raw/670996/ | 12:18 |
| egonzalez | lvdombrkr, the container was restarting before deploy | 12:18 |
| egonzalez | nothing in config changed | 12:19 |
| *** gkadam_ has quit IRC | 12:19 | |
| egonzalez | when docker module is set to state: started, but was restarting so the module changed and tried to start | 12:19 |
| lvdombrkr | egonzalez: this is what i greped from mariadb.log about gcomm http://paste.openstack.org/raw/670998/ | 12:21 |
| egonzalez | lvdombrkr, need to investigate what happened before reconfigure | 12:21 |
| dangtrinhnt | lvdombrkr, correct me if I'm wrong, in your very first logs, 94.140.122.148 is your VIP that MariaDB is listening on. And according to your last logs, 94.140.122.146 is one of the cluster members? | 12:30 |
| *** hw_wutianwei has quit IRC | 12:30 | |
| lvdombrkr | dangtrinhnt: 94.140.122.148 is VIP, 94.140.122.146 is controller host ip | 12:33 |
| egonzalez | didnt know with zuulv3 can see live execution logs | 12:36 |
| *** gfidente has quit IRC | 12:37 | |
| *** gfidente has joined #openstack-kolla | 12:38 | |
| *** gfidente has quit IRC | 12:38 | |
| *** gfidente has joined #openstack-kolla | 12:38 | |
| *** b_bezak has joined #openstack-kolla | 12:41 | |
| lvdombrkr | egonzalez: ^ its for me? | 12:42 |
| egonzalez | lvdombrkr, nope, for everyone in here ;) | 12:42 |
| egonzalez | ex http://zuul.openstack.org/stream.html?uuid=58900cb5ba8645d3ba7ce4d33dce39d5&logfile=console.log | 12:43 |
| *** rhallisey has joined #openstack-kolla | 12:52 | |
| *** rhallisey has quit IRC | 12:52 | |
| *** rhallisey has joined #openstack-kolla | 12:52 | |
| *** tovin07 has joined #openstack-kolla | 12:57 | |
| *** sambetts|afk is now known as sambetts | 12:57 | |
| *** hamza21 has joined #openstack-kolla | 12:59 | |
| *** ktibi has quit IRC | 13:11 | |
| *** ktibi has joined #openstack-kolla | 13:15 | |
| ktibi | egonzalez, for the moment, I tested only service in my review, maybe need to deploy and test other services. | 13:15 |
| *** pcaruana has quit IRC | 13:16 | |
| *** signed8bit has joined #openstack-kolla | 13:20 | |
| *** olivierbourdon38 has quit IRC | 13:22 | |
| *** tovin07 has quit IRC | 13:25 | |
| jmccarthy | Is this the latest upgrade guide ? https://docs.openstack.org/kolla-ansible/latest/user/operating-kolla.html | 13:27 |
| openstackgerrit | Kevin Tibi proposed openstack/kolla-ansible master: Fix SSL api for multiple services https://review.openstack.org/543488 | 13:28 |
| *** janki has quit IRC | 13:32 | |
| *** oanson has quit IRC | 13:33 | |
| dangtrinhnt | Hi jmccarthy, 'UPDATED: 2018-02-13 07:33' => I think so :) | 13:33 |
| jmccarthy | :) Ok ! | 13:34 |
| *** oanson has joined #openstack-kolla | 13:35 | |
| *** dave-mccowan has joined #openstack-kolla | 13:36 | |
| *** caoyuan_ has quit IRC | 13:42 | |
| *** rwellum has quit IRC | 13:46 | |
| *** rwellum has joined #openstack-kolla | 13:46 | |
| jmccarthy | I think what is there in the docs really only describes a minor upgrade, not a major one | 13:47 |
| sdake | morning peeps | 13:53 |
| lvdombrkr | egonzalez: in launchpad i found something like my issue and resolution was "This problem is resolved by removing volume for mariadb,docker volume rm mariadb" . as i understood i loose all my database data if i do that, right? | 13:54 |
| *** signed8bit is now known as signed8bit_Zzz | 13:55 | |
| *** zul has quit IRC | 13:56 | |
| jmccarthy | morning ! | 13:57 |
| *** zul has joined #openstack-kolla | 13:57 | |
| *** hachi has joined #openstack-kolla | 14:05 | |
| *** signed8bit_Zzz is now known as signed8bit | 14:07 | |
| *** zhangfei has quit IRC | 14:07 | |
| *** hamza21 has quit IRC | 14:08 | |
| *** zhaochao has quit IRC | 14:09 | |
| *** signed8bit has quit IRC | 14:09 | |
| *** signed8b_ has joined #openstack-kolla | 14:10 | |
| *** pradk has quit IRC | 14:10 | |
| *** coolsvap has joined #openstack-kolla | 14:15 | |
| *** pradk has joined #openstack-kolla | 14:17 | |
| *** zhangfei has joined #openstack-kolla | 14:20 | |
| *** shardy has quit IRC | 14:24 | |
| *** shardy has joined #openstack-kolla | 14:25 | |
| *** rhallisey has quit IRC | 14:26 | |
| *** rhallisey has joined #openstack-kolla | 14:26 | |
| *** liangqi has quit IRC | 14:27 | |
| *** liangqi has joined #openstack-kolla | 14:27 | |
| dasTor | lvdombrkr, it depends, when it's one server of a galera cluster, it will get the data back | 14:28 |
| lvdombrkr | dasTor: thanks, but what you mean by "one server of galera cluster" i have only one controller already | 14:31 |
| dasTor | do you have a link, where that is suggested? | 14:33 |
| *** david-lyle has quit IRC | 14:33 | |
| *** openstackgerrit has quit IRC | 14:33 | |
| dasTor | if you docker rm volume mariadb, your data is gone | 14:33 |
| *** jtriley has joined #openstack-kolla | 14:33 | |
| lvdombrkr | dasTor: one more question under : /var/lib/docker/volumes/mariadb/_data there is galera.cache , can i clear this cache? | 14:34 |
| lvdombrkr | dasTor: this is link https://bugs.launchpad.net/kolla/+bug/1543906 | 14:39 |
| openstack | Launchpad bug 1543906 in kolla "mariadb cannot start in master" [Undecided,Invalid] | 14:39 |
| dasTor | yes, but the bug says it can't start, so there no data to loose | 14:40 |
| dasTor | because it never was able to start | 14:41 |
| dasTor | then it's really ok | 14:41 |
| lvdombrkr | dasTor: thanks! what about galera.cache ? can i clear it? | 14:43 |
| dasTor | yes, otherwise it will never come up | 14:44 |
| *** signed8b_ is now known as signed8bit_Zzz | 14:44 | |
| dasTor | it's a binary cluster state, which in your state is broken | 14:45 |
| lvdombrkr | dastor: how can i clear it? | 14:45 |
| dasTor | remove the file | 14:45 |
| *** hachi has quit IRC | 14:48 | |
| lvdombrkr | dasTor: deleted but no success mariadb container is restarting anyway | 14:49 |
| dasTor | you removed the volume also? | 14:50 |
| dasTor | it shoud start clear after that | 14:50 |
| *** openstackgerrit has joined #openstack-kolla | 14:51 | |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Don't check api_interface for 'deployment' group https://review.openstack.org/543959 | 14:51 |
| lvdombrkr | dasTor: i removed just galera.cache | 14:52 |
| *** mgoddard_ has joined #openstack-kolla | 14:53 | |
| dasTor | if you have no data there, please remove the volume also | 14:56 |
| lvdombrkr | dasTor: but i have data.. lot of data | 14:56 |
| *** pradk has quit IRC | 14:59 | |
| *** zhangfei has quit IRC | 15:00 | |
| dasTor | lvdombrkr, you have only one mariadb container? can you look that up? | 15:04 |
| lvdombrkr | dasTor: yes just one | 15:06 |
| openstackgerrit | Merged openstack/kolla-ansible master: Use bash instead of sh when calling deploy-tacker-demo-sfc https://review.openstack.org/512025 | 15:10 |
| aikema | anyone know why I can't get initial setup of mariadb to work? Had found a bug that appears to show the same problem - https://bugs.launchpad.net/kolla/+bug/1732314 - but still haven't figured out how to resolve it | 15:11 |
| openstack | Launchpad bug 1732314 in kolla "mariadb Connection refused" [Undecided,New] | 15:11 |
| openstackgerrit | Merged openstack/kolla master: Add one more step to generate kolla-build.conf file https://review.openstack.org/538804 | 15:13 |
| dasTor | lvdombrkr, make a backup of your volume (just in case), then edit grastate.dat and set safe_to_bootstrap: 1 | 15:14 |
| lvdombrkr | dasTor: is already set to 1 | 15:15 |
| dasTor | this makes less and less sense | 15:16 |
| dasTor | why would it fail then? | 15:16 |
| lvdombrkr | i have lot of wsrep_recovery.M4iKBG files with this into: http://paste.openstack.org/raw/671020/ | 15:17 |
| *** hw_wutianwei_ has quit IRC | 15:18 | |
| *** hw_wutianwei_ has joined #openstack-kolla | 15:18 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Catch an error when pool creation failed for ceph https://review.openstack.org/540845 | 15:20 |
| openstackgerrit | Merged openstack/kolla-ansible master: Modify grammatical error https://review.openstack.org/541974 | 15:20 |
| dasTor | lvdombrkr, this looks bad | 15:20 |
| dasTor | you already did kolla-ansible -i your_inventory mariadb_recovery i guess? | 15:20 |
| lvdombrkr | dasTor: not yet, i must do that? | 15:21 |
| dasTor | yes | 15:21 |
| *** rmart04 has joined #openstack-kolla | 15:21 | |
| dasTor | but your wsrep_recovery looks don't look very good | 15:22 |
| *** lpetrut has quit IRC | 15:24 | |
| lvdombrkr | dasTor: http://paste.openstack.org/raw/671021/ | 15:25 |
| *** pradk has joined #openstack-kolla | 15:25 | |
| *** tovin07 has joined #openstack-kolla | 15:30 | |
| dasTor | this is, where it gets ugly, you can do: docker restart mariadb && docker exec -it -u root mariadb mysqld_safe --tc-heuristic-recover=ROLLBACK | 15:33 |
| dasTor | and then: docker restart mariadb && docker exec -it -u root mariadb mysqld_safe --tc-heuristic-recover=COMMIT | 15:34 |
| dasTor | this gets you rid of all recovery errors, and then it shurely should start | 15:34 |
| *** Teck7 has quit IRC | 15:35 | |
| lvdombrkr | dasTor: after restart mariadb && docker exec -it -u root mariadb mysqld_safe --tc-heuristic-recover=ROLLBACK container starts normaly | 15:38 |
| lvdombrkr | dasTor: so right now is rolled back to last working configuration? | 15:38 |
| dasTor | lvdombrkr, yes, some data _might_ be lost | 15:39 |
| dasTor | if you had an open transaction, which was not already on your disk, then it is surely gone | 15:39 |
| lvdombrkr | dasTor: it :must bet latest data, yes? | 15:40 |
| dasTor | yes | 15:40 |
| dasTor | and in production please use at least 3 mariadb containers | 15:40 |
| dasTor | otherwise you might run into real problems, when your single db container crashes | 15:40 |
| *** rhallisey has quit IRC | 15:42 | |
| *** aikema has quit IRC | 15:45 | |
| *** signed8bit_Zzz is now known as signed8b_ | 15:45 | |
| lvdombrkr | dasTor: thanks, i owe you some pints of beer )) | 15:46 |
| dangtrinhnt | wow, awesome. lvdombrkr, could you please make a writeup about this? I guess there will be someone get this error too. Thanks. | 15:48 |
| *** rmart04 has quit IRC | 15:50 | |
| *** david-lyle has joined #openstack-kolla | 15:54 | |
| *** shardy has quit IRC | 15:54 | |
| dasTor | lvdombrkr, i'm glad it now works for you, please don't forget: multiple nodes in producion saves you from this hazard :) | 15:59 |
| *** itlinux has joined #openstack-kolla | 16:01 | |
| *** egonzalez has quit IRC | 16:02 | |
| openstackgerrit | Martin Mágr proposed openstack/kolla master: Add socat to iscsid image https://review.openstack.org/543624 | 16:03 |
| inc0 | good mornign | 16:03 |
| *** openstackgerrit has quit IRC | 16:04 | |
| *** rhallisey has joined #openstack-kolla | 16:05 | |
| *** b_bezak has quit IRC | 16:09 | |
| *** mgoddard_ has quit IRC | 16:22 | |
| *** tovin07 has quit IRC | 16:24 | |
| *** k_mouza has quit IRC | 16:26 | |
| *** sambetts is now known as sambetts|afk | 16:29 | |
| lvdombrkr | dasTor: yes, and you save my day today ) | 16:31 |
| *** k_mouza has joined #openstack-kolla | 16:32 | |
| *** b_bezak has joined #openstack-kolla | 16:34 | |
| *** JamesBenson has joined #openstack-kolla | 16:41 | |
| *** mnasiadka has quit IRC | 16:42 | |
| *** itlinux has quit IRC | 16:46 | |
| *** pcaruana has joined #openstack-kolla | 16:47 | |
| *** olivierbourdon38 has joined #openstack-kolla | 16:49 | |
| *** mdnadeem has quit IRC | 16:49 | |
| *** itlinux has joined #openstack-kolla | 16:50 | |
| *** rhallisey has quit IRC | 16:51 | |
| *** rhallisey has joined #openstack-kolla | 16:51 | |
| *** cliles has joined #openstack-kolla | 16:51 | |
| *** dciabrin has quit IRC | 16:52 | |
| *** mnasiadka has joined #openstack-kolla | 16:52 | |
| *** lpetrut has joined #openstack-kolla | 16:53 | |
| *** mnasiadka has quit IRC | 16:57 | |
| *** openstackgerrit has joined #openstack-kolla | 16:58 | |
| openstackgerrit | Martin Mágr proposed openstack/kolla master: Add socat to base image https://review.openstack.org/543624 | 16:58 |
| jmccarthy | morning ! | 17:01 |
| *** mnasiadka has joined #openstack-kolla | 17:07 | |
| *** mnasiadka has quit IRC | 17:12 | |
| *** mnasiadka has joined #openstack-kolla | 17:22 | |
| *** mdnadeem has joined #openstack-kolla | 17:24 | |
| *** signed8b_ is now known as signed8bit_Zzz | 17:24 | |
| *** itlinux has quit IRC | 17:26 | |
| *** mnasiadka has quit IRC | 17:27 | |
| *** itlinux has joined #openstack-kolla | 17:31 | |
| *** k_mouza has quit IRC | 17:32 | |
| *** dciabrin has joined #openstack-kolla | 17:33 | |
| *** mnasiadka has joined #openstack-kolla | 17:37 | |
| *** lvdombrkr has quit IRC | 17:40 | |
| *** mnasiadka has quit IRC | 17:42 | |
| *** devananda has joined #openstack-kolla | 17:45 | |
| *** mnasiadka has joined #openstack-kolla | 17:52 | |
| *** pcaruana has quit IRC | 17:55 | |
| *** mnasiadka has quit IRC | 17:57 | |
| *** mgoddard_ has joined #openstack-kolla | 17:58 | |
| *** mnasiadka has joined #openstack-kolla | 18:05 | |
| *** pcaruana has joined #openstack-kolla | 18:06 | |
| *** david-lyle has quit IRC | 18:11 | |
| *** mnasiadka has quit IRC | 18:12 | |
| *** hamza21 has joined #openstack-kolla | 18:12 | |
| *** signed8bit_Zzz is now known as signed8b_ | 18:12 | |
| *** jmccarthy has left #openstack-kolla | 18:16 | |
| *** gfidente is now known as gfidente|afk | 18:16 | |
| *** unicell1 has quit IRC | 18:20 | |
| *** mnasiadka has joined #openstack-kolla | 18:22 | |
| *** mnasiadka has quit IRC | 18:27 | |
| hogepodge | hey, I was wondering how kolla handles log rotation. Is there a way for me to clear things out as disk space fills up? | 18:29 |
| hogepodge | (kolla-ansible in particular) | 18:29 |
| *** itlinux has quit IRC | 18:30 | |
| *** dardele__ has joined #openstack-kolla | 18:31 | |
| *** spiette_ has quit IRC | 18:33 | |
| *** mdnadeem has quit IRC | 18:33 | |
| *** dardelea_ has quit IRC | 18:33 | |
| *** dardele__ has quit IRC | 18:35 | |
| *** dardelea_ has joined #openstack-kolla | 18:36 | |
| *** pradk has quit IRC | 18:37 | |
| *** mnasiadka has joined #openstack-kolla | 18:37 | |
| *** dardelea_ has quit IRC | 18:41 | |
| *** mnasiadka has quit IRC | 18:42 | |
| *** itlinux has joined #openstack-kolla | 18:44 | |
| inc0 | hogepodge: we have cron container | 18:45 |
| *** pcaruana has quit IRC | 18:45 | |
| inc0 | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/common/handlers/main.yml#L53 | 18:46 |
| inc0 | these are all logrotate confs https://github.com/openstack/kolla-ansible/tree/master/ansible/roles/common/templates | 18:46 |
| *** mnasiadka has joined #openstack-kolla | 18:52 | |
| *** lpetrut has quit IRC | 18:53 | |
| *** pradk has joined #openstack-kolla | 18:56 | |
| *** unicell has joined #openstack-kolla | 18:56 | |
| *** mnasiadka has quit IRC | 18:57 | |
| *** itlinux has quit IRC | 18:57 | |
| *** itlinux has joined #openstack-kolla | 18:59 | |
| *** david-lyle has joined #openstack-kolla | 19:02 | |
| *** harlowja has joined #openstack-kolla | 19:03 | |
| *** dardelea_ has joined #openstack-kolla | 19:05 | |
| *** mnasiadka has joined #openstack-kolla | 19:07 | |
| *** dardelea_ has quit IRC | 19:09 | |
| *** mnasiadka has quit IRC | 19:12 | |
| *** awiddersheim has joined #openstack-kolla | 19:15 | |
| *** mnasiadka has joined #openstack-kolla | 19:22 | |
| *** mnasiadka has quit IRC | 19:27 | |
| *** david-lyle has quit IRC | 19:29 | |
| *** david-lyle has joined #openstack-kolla | 19:32 | |
| *** mnasiadka has joined #openstack-kolla | 19:37 | |
| *** itlinux has quit IRC | 19:39 | |
| *** dciabrin has quit IRC | 19:40 | |
| *** dciabrin has joined #openstack-kolla | 19:41 | |
| *** mnasiadka has quit IRC | 19:42 | |
| *** itlinux has joined #openstack-kolla | 19:43 | |
| *** itlinux has quit IRC | 19:46 | |
| *** Angeh has joined #openstack-kolla | 19:49 | |
| *** b_bezak has quit IRC | 19:49 | |
| *** mnasiadka has joined #openstack-kolla | 19:52 | |
| *** mnasiadka has quit IRC | 19:57 | |
| *** salv-orlando has joined #openstack-kolla | 19:58 | |
| *** salv-orl_ has quit IRC | 20:01 | |
| *** mnasiadka has joined #openstack-kolla | 20:07 | |
| *** mnasiadka has quit IRC | 20:12 | |
| *** hamza21 has quit IRC | 20:14 | |
| *** coolsvap has quit IRC | 20:14 | |
| *** hamza21 has joined #openstack-kolla | 20:14 | |
| *** goldyfruit has quit IRC | 20:17 | |
| *** mnasiadka has joined #openstack-kolla | 20:23 | |
| *** mnasiadka has quit IRC | 20:28 | |
| *** dardelea_ has joined #openstack-kolla | 20:29 | |
| *** mgoddard_ has quit IRC | 20:30 | |
| *** itlinux has joined #openstack-kolla | 20:36 | |
| *** pramodrj07 has joined #openstack-kolla | 20:37 | |
| *** dardelea_ has quit IRC | 20:37 | |
| *** dardelea_ has joined #openstack-kolla | 20:38 | |
| *** pramodrj07 has quit IRC | 20:38 | |
| *** mnasiadka has joined #openstack-kolla | 20:38 | |
| *** awiddersheim has quit IRC | 20:39 | |
| *** dardelea_ has quit IRC | 20:43 | |
| *** mnasiadka has quit IRC | 20:44 | |
| *** goldyfruit has joined #openstack-kolla | 20:51 | |
| *** mnasiadka has joined #openstack-kolla | 20:53 | |
| *** mnasiadka has quit IRC | 20:59 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible stable/pike: Increase vm.max_map_count on elasticsearch nodes https://review.openstack.org/544079 | 21:00 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible stable/pike: Non-lvm iscsi backends should run iscsid on compute and storage nodes https://review.openstack.org/544080 | 21:01 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible stable/pike: Fixing missing copy of iscsi container config files https://review.openstack.org/544083 | 21:02 |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible stable/pike: Add /lib/modules as volume to openvswitch_db container https://review.openstack.org/544085 | 21:05 |
| *** mnasiadka has joined #openstack-kolla | 21:08 | |
| *** mnasiadka has quit IRC | 21:13 | |
| *** spiette has joined #openstack-kolla | 21:14 | |
| *** gfidente|afk has quit IRC | 21:19 | |
| *** rhallisey has quit IRC | 21:20 | |
| *** olivierbourdon38 has quit IRC | 21:22 | |
| *** mnasiadka has joined #openstack-kolla | 21:23 | |
| *** mnasiadka has quit IRC | 21:28 | |
| *** mnasiadka has joined #openstack-kolla | 21:31 | |
| *** awiddersheim has joined #openstack-kolla | 21:32 | |
| *** dciabrin has quit IRC | 21:33 | |
| *** mnasiadka has quit IRC | 21:36 | |
| *** mnasiadka has joined #openstack-kolla | 21:38 | |
| *** hamza21 has quit IRC | 21:42 | |
| *** openstackstatus has quit IRC | 21:43 | |
| *** mnasiadka has quit IRC | 21:43 | |
| *** openstackstatus has joined #openstack-kolla | 21:44 | |
| *** ChanServ sets mode: +v openstackstatus | 21:44 | |
| *** mnasiadka has joined #openstack-kolla | 21:53 | |
| *** mnasiadka has quit IRC | 21:58 | |
| *** jtriley has quit IRC | 21:58 | |
| *** Angeh has quit IRC | 21:59 | |
| bkero- | Hey all. I'm attempting to upgrade the nova_compute tag on a kolla deployment, and I noticed that the nova/tasks/update.yml playbook seems a bit sparse there | 22:06 |
| bkero- | Sorry, upgrade. https://git.openstack.org/cgit/openstack/kolla-ansible/tree/ansible/roles/nova/tasks/upgrade.yml?h=stable/pike | 22:06 |
| bkero- | That seems to just handle the conductor. I need to upgrade the compute containers, and I'm wondering how much of that is going to be manual. | 22:07 |
| *** mnasiadka has joined #openstack-kolla | 22:08 | |
| *** wwriverrat has joined #openstack-kolla | 22:10 | |
| *** mnasiadka has quit IRC | 22:13 | |
| *** awidders_ has joined #openstack-kolla | 22:18 | |
| *** awiddersheim has quit IRC | 22:18 | |
| *** itlinux has quit IRC | 22:23 | |
| *** mnasiadka has joined #openstack-kolla | 22:23 | |
| *** itlinux has joined #openstack-kolla | 22:27 | |
| *** mnasiadka has quit IRC | 22:28 | |
| wwriverrat | kolla-ansible question: We have a custom “internal” neutron service plugin, call it “special-route-thingy”. I’d like this appended to the final neutron.conf’s service_plugins. I see all of the “enable_*” flags (like enable_neutron_fwaas), I just dont see how I add a new “internal” one. Would having a line in my global neutron.conf something like service_plugins=special-route-thingy get picked up and merge_config’d if I ha | 22:28 |
| wwriverrat | some others enabled (like enable_neutron_fwaas) | 22:28 |
| *** emccormick has joined #openstack-kolla | 22:33 | |
| emccormick | hey guys, any idea what causes "container init caused \\\\\"open /dev/console: input/output error" trying to start kolla-toolbox during a deploy? | 22:34 |
| emccormick | I know I"ve seen it before, but can't for the life of me remember where. Somehow only 2 of 8 nodes in the deployment are doing that | 22:34 |
| SamYaple | emccormick: oh man. that goes back | 22:37 |
| SamYaple | ill grep my logs | 22:37 |
| emccormick | thanks | 22:37 |
| emccormick | trying to diff the hosts. One of these things is not like the others. | 22:37 |
| SamYaple | i wanna say exhausted resources for some reason | 22:38 |
| *** mnasiadka has joined #openstack-kolla | 22:38 | |
| emccormick | plenty of memory / disk | 22:40 |
| SamYaple | emccormick: what are the permissiosn on /dev/console? | 22:40 |
| emccormick | crw--w---- 0 root tty 136, 3 Feb 6 22:26 /dev/console | 22:41 |
| SamYaple | yea thats not it then. i do remember this on centos boxes only | 22:42 |
| SamYaple | i cant seem to find the conversation through | 22:42 |
| emccormick | yep centos indeed | 22:42 |
| *** mnasiadka has quit IRC | 22:43 | |
| SamYaple | version of docker? and kernel? | 22:43 |
| emccormick | oddly the nodes that work don't have the group write bit set, but that may just be from me running into this once before | 22:43 |
| emccormick | docker-ce-17.12.0.ce-1 | 22:43 |
| emccormick | kernel-3.10.0-514 | 22:43 |
| SamYaple | let me look at the mounts for kolla-toolbo | 22:44 |
| SamYaple | emccormick: oh yea. its binding /dev:/dev | 22:45 |
| SamYaple | that might be the issue | 22:46 |
| SamYaple | emccormick: try reseting it to a working nodespermissions | 22:46 |
| *** itlinux has quit IRC | 22:46 | |
| SamYaple | i forget why we arebinding in /dev/ read-write | 22:46 |
| SamYaple | im pretty sure i added that too, but im blanking | 22:47 |
| emccormick | just did that on one and it borked on it still :/ | 22:47 |
| SamYaple | probably ceph related | 22:47 |
| SamYaple | yea so i think you are leaking resources, arethese dev nodeswhere containers come up anddown alot? | 22:47 |
| SamYaple | or is it pretty static | 22:47 |
| emccormick | hmm well they are both computes so fairly static, but I jsut restarted docker also | 22:48 |
| emccormick | might try windows solution on one | 22:48 |
| emccormick | just in case relic is sitting around. | 22:48 |
| SamYaple | restarting docker might have done it lol yea | 22:48 |
| SamYaple | restarting the container might do it too if its not a gracefulshutdown | 22:49 |
| *** mnasiadka has joined #openstack-kolla | 22:53 | |
| *** kong has joined #openstack-kolla | 22:55 | |
| *** pradk has quit IRC | 22:55 | |
| emccormick | #@$%%@! windows solution worked | 22:57 |
| emccormick | That is *not* supposed to happen | 22:57 |
| *** mnasiadka has quit IRC | 22:57 | |
| *** dardelean has quit IRC | 23:01 | |
| *** JamesBenson has quit IRC | 23:01 | |
| *** dardelean has joined #openstack-kolla | 23:01 | |
| *** JamesBenson has joined #openstack-kolla | 23:03 | |
| *** JamesBenson has quit IRC | 23:07 | |
| *** mnasiadka has joined #openstack-kolla | 23:08 | |
| sdake | kfox1111 around? | 23:10 |
| kfox1111 | sdake: yeah. whats up? | 23:10 |
| sdake | seepms :) | 23:10 |
| *** awidders_ has quit IRC | 23:11 | |
| *** awiddersheim has joined #openstack-kolla | 23:11 | |
| *** mnasiadka has quit IRC | 23:12 | |
| *** Zara has quit IRC | 23:18 | |
| *** Zara has joined #openstack-kolla | 23:18 | |
| *** dardelean has quit IRC | 23:19 | |
| *** dardelean has joined #openstack-kolla | 23:21 | |
| emccormick | Is this worth filing a bug? During upgrade from Ocata -> Pike, if you have to do a --limit to run through a node that blew up, it will hang on "Creating the Nova service and endpoint" | 23:21 |
| emccormick | it's kind of an edge case, but it's a thing | 23:22 |
| emccormick | basically means you can't --limit upgrades | 23:22 |
| *** mnasiadka has joined #openstack-kolla | 23:22 | |
| emccormick | and now get the /dev/console junk again part way through the deploy. One dies on nova_libvirt and the other on nova_compute | 23:25 |
| *** k_mouza has joined #openstack-kolla | 23:26 | |
| *** mnasiadka has quit IRC | 23:27 | |
| *** zshi has quit IRC | 23:30 | |
| SamYaple | upgrade as in containers are recreated? | 23:34 |
| SamYaple | if so, youll stillneed to fix the issue | 23:34 |
| *** kolla-slack has quit IRC | 23:35 | |
| emccormick | yes | 23:35 |
| emccormick | upgrade as in kolla-ansible upgrade | 23:35 |
| emccormick | not sure which issue I'm fixing | 23:36 |
| *** kolla-slack has joined #openstack-kolla | 23:36 | |
| SamYaple | emccormick: the issue is the /dev/:/dev binding. its probably anissue with the container distro and the host distro | 23:37 |
| *** kolla-slack has quit IRC | 23:38 | |
| *** mnasiadka has joined #openstack-kolla | 23:38 | |
| *** kolla-slack has joined #openstack-kolla | 23:38 | |
| emccormick | it's super random | 23:38 |
| SamYaple | but consistentafter it happens? | 23:39 |
| *** kolla-slack has quit IRC | 23:40 | |
| emccormick | well once it happens then it you can't do anything with the container it blows up on | 23:41 |
| emccormick | seems maybe it's happening during the replacement of the container | 23:41 |
| *** kolla-slack has joined #openstack-kolla | 23:42 | |
| *** mnasiadka has quit IRC | 23:43 | |
| emccormick | restarting dockerd doesn't clean it up. Trying a reboot again, but that's not really figuring out the problem :/ | 23:43 |
| *** kolla-slack has quit IRC | 23:43 | |
| *** klindgren has quit IRC | 23:43 | |
| *** kolla-slack has joined #openstack-kolla | 23:44 | |
| emccormick | preceding the console failure are two other errors form dockerd | 23:44 |
| emccormick | stream copy error: reading from a closed fifo | 23:44 |
| emccormick | ignoring event" module=libcontainerd namespace=moby topic=/containers/delete type="*events.ContainerDelete" | 23:45 |
| emccormick | cleanup: failed to delete container from containerd: no such container" | 23:45 |
| emccormick | and then can't open /dev/console | 23:45 |
| *** kolla-slack has quit IRC | 23:46 | |
| *** dardelean_ has joined #openstack-kolla | 23:47 | |
| *** kolla-slack has joined #openstack-kolla | 23:47 | |
| *** kolla-slack has joined #openstack-kolla | 23:48 | |
| emccormick | and yet a reboot makes it start the new version of the container just fine | 23:48 |
| emccormick | not being able to --limit to those two nodes is making new runs take forever though *heh* | 23:49 |
| *** kolla-slack has quit IRC | 23:50 | |
| *** kolla-slack has joined #openstack-kolla | 23:51 | |
| *** dardelean has quit IRC | 23:52 | |
| *** kolla-slack has quit IRC | 23:53 | |
| *** mnasiadka has joined #openstack-kolla | 23:53 | |
| *** dardelean_ has quit IRC | 23:53 | |
| *** kolla-slack has joined #openstack-kolla | 23:54 | |
| *** signed8b_ is now known as signed8bit_Zzz | 23:54 | |
| *** kolla-slack has quit IRC | 23:55 | |
| *** kolla-slack has joined #openstack-kolla | 23:57 | |
| *** mnasiadka has quit IRC | 23:57 | |
| *** kolla-slack has quit IRC | 23:58 | |
| *** kolla-slack has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!