| *** yangyape_ has quit IRC | 00:00 | |
| sbezverk | esmute: how do you deploy mariadb? what command you use? | 00:01 |
|---|---|---|
| inc0 | sbezverk: cool you're here, what's best way to access /var/log/?:) | 00:02 |
| inc0 | do we mount volume to /var/log somewhere? | 00:02 |
| sbezverk | inc0: on crashing pod it is a bit tricky, timing is everything | 00:03 |
| sbezverk | you do docker inspect on crashing log | 00:03 |
| *** eanylin has quit IRC | 00:03 | |
| sbezverk | catch where kubernetes mount /var/log/kolla | 00:03 |
| sbezverk | and then access logs right on the kube host | 00:03 |
| *** eaguilar has quit IRC | 00:03 | |
| inc0 | ah yeah, k8s won't let you just stop container and examine it right? | 00:04 |
| *** ssurana has joined #openstack-kolla | 00:04 | |
| sbezverk | docker inspect f9cd5dce622b | grep /var/log/kolla | 00:04 |
| sbezverk | "/var/lib/kubelet/pods/1aa516f6-2932-11e7-a732-525400a3621f/volumes/kubernetes.io~empty-dir/kolla-logs:/var/log/kolla/:Z" | 00:04 |
| esmute | sbezverk: im just following the deployment-guide doc. I used helm install --debug kolla-kubernetes/helm/service/mariadb --namespace kolla --name mariadb --values ./cloud.yaml | 00:05 |
| sbezverk | inc0: right | 00:05 |
| sbezverk | esmute: kubectl get pv | 00:05 |
| Jeffrey4l | pbourke, re keyfile. checked in my env. it is not generated. | 00:05 |
| sbezverk | esmute: kubectl get pvc -n kolla | 00:05 |
| esmute | inc0: im not using ceph nor any storageclass.. Should i? | 00:05 |
| sbezverk | esmute: it should be created for you | 00:05 |
| esmute | $ kubectl get pvc -n kolla NAME STATUS VOLUME CAPACITY ACCESSMODES STORAGECLASS AGE mariadb Bound mariadb 10Gi RWO 1h | 00:05 |
| esmute | it is bounded | 00:05 |
| sbezverk | as long as you have correct settings for storage | 00:06 |
| sbezverk | esmute: kubectl get job -n kolla | 00:06 |
| sbezverk | esmute: before mariadb can start you needs to be bootstraped | 00:07 |
| esmute | $ kubectl -n kolla get job NAME DESIRED SUCCESSFUL AGE mariadb-init-element 1 1 1h | 00:07 |
| *** yingjun has joined #openstack-kolla | 00:08 | |
| sbezverk | esmute: hm, it looks like bootstrap went well. check /var/lib/kolla/volumes | 00:08 |
| esmute | sbezverk: https://gist.github.com/kokhang/cf6ccde75374947d5a878b05faa20d7f | 00:10 |
| esmute | the volume seems to have been mounted and data written to it | 00:10 |
| sbezverk | esmute: right. oh well easy way did not help, we need to access the logs to try to figure out | 00:12 |
| *** imcsk8 has quit IRC | 00:13 | |
| jascott1_ | anyone know where this gets created? /var/lib/kolla/config_files | 00:14 |
| sbezverk | esmute: docker ispect $(docker ps -a | grep mariadb@ | awk '{print $1}') | grep /var/log/kolla | 00:15 |
| sbezverk | then quickly cat mariadb log from that folder | 00:16 |
| *** georgem1 has joined #openstack-kolla | 00:16 | |
| *** georgem1 has quit IRC | 00:17 | |
| *** georgem1 has joined #openstack-kolla | 00:17 | |
| jascott1_ | my neutron containers are looking for that path but its not there | 00:18 |
| esmute | sbezverk: cool.. here is what i have | 00:19 |
| esmute | 170504 0:16:37 [Note] InnoDB: Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1600965 170504 0:16:37 [Note] Plugin 'FEEDBACK' is disabled. 170504 0:16:37 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. 170504 0:16:37 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 170504 | 00:19 |
| esmute | https://gist.github.com/kokhang/9e6965061934f6476ee6864139c670cb | 00:19 |
| sbezverk | esmute: it looks like init has not been completed properly | 00:24 |
| sbezverk | I suggest helm delete mariadb --purge | 00:24 |
| sbezverk | then rm -rf /var/lib/kolla/volumes/* | 00:24 |
| *** reddydodda has joined #openstack-kolla | 00:24 | |
| sbezverk | and rerun mariadb deploy | 00:24 |
| *** manheim has joined #openstack-kolla | 00:25 | |
| jascott1_ | sbezverk: do you know where /var/lib/kolla/config_files get created? | 00:25 |
| jascott1_ | mine is missing | 00:25 |
| sbezverk | jascott1_: a little bit more of a context please? these are files from the configmap | 00:26 |
| sbezverk | if they are missing then probably configmap is not correct | 00:26 |
| esmute | sbezverk: ok this time it is running and not crashing | 00:26 |
| esmute | do you have an idea why the init failed? | 00:27 |
| jascott1_ | installing from guide, everything comes up but neutron containers. they cant mount config and thats the path volume references | 00:27 |
| sbezverk | esmute: nice!! man, I am not oracle ;) need logs | 00:27 |
| sbezverk | jascott1_: you are second one who hit this issue | 00:27 |
| jascott1_ | yay! | 00:28 |
| jascott1_ | that means there is easy answer right? ;) | 00:28 |
| sbezverk | jascott1_: DO NOT USE kolla-ansible to generate configs :) | 00:28 |
| jascott1_ | ah | 00:28 |
| sbezverk | jascott1_: they removed ml2_conf.ini | 00:28 |
| sbezverk | from neutron configuration | 00:28 |
| sbezverk | so kube is not happy as it is missing a key for ml2 | 00:29 |
| jascott1_ | hmm | 00:29 |
| sbezverk | jascott1_: inc0 created a playbook to generate configuration independently | 00:29 |
| esmute | sbezverk: i used `sudo kolla-ansible genconfig` following the guide | 00:29 |
| esmute | would i run into issues? | 00:29 |
| *** manheim has quit IRC | 00:29 | |
| sbezverk | esmute: you will be third one then ;) | 00:29 |
| jascott1_ | haha | 00:29 |
| jascott1_ | playbook is merged? | 00:30 |
| sbezverk | jascott1_: Yep, you can get command to use from tests/bin/ iscsi workflow | 00:30 |
| sbezverk | look for ansible | 00:30 |
| jascott1_ | sbezverk thanks! | 00:30 |
| sbezverk | command, you can use that command as is without any modifications | 00:31 |
| sbezverk | jascott1_: any time ;) | 00:31 |
| *** zhurong has joined #openstack-kolla | 00:31 | |
| *** eaguilar has joined #openstack-kolla | 00:32 | |
| jascott1_ | esmute: the command is | 00:35 |
| jascott1_ | ansible-playbook -e ansible_python_interpreter=/usr/bin/python -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla ansible/site.yml | 00:35 |
| jascott1_ | from kolla-kubernetes dir | 00:35 |
| sbezverk | jascott1_: yep, that is the one | 00:35 |
| jascott1_ | works for me (TM) | 00:36 |
| sbezverk | jascott1_: somebody need to update DG asap | 00:36 |
| jascott1_ | yeah also canal.yml url has changed | 00:36 |
| jascott1_ | i was planning on making PS | 00:36 |
| esmute | jascott1_: thank you | 00:36 |
| jascott1_ | also they have added an RBAC.yml with flannel stuff in it | 00:37 |
| esmute | do i have to delete all helm deployments ? | 00:37 |
| sbezverk | jascott1_: check setup_canal.sh script | 00:37 |
| sbezverk | it is workaround we use for the gate | 00:37 |
| jascott1_ | the way is the gate, the gate is the way ;) | 00:37 |
| sbezverk | jascott1_: why not ;)?? | 00:38 |
| jascott1_ | no doubt | 00:38 |
| jascott1_ | yeah just now getting back to playbook, its funny how much things break in a week or so | 00:38 |
| sbezverk | jascott1_: always something happens, if not kubernetes screw up, then canal, if not that then something else. fun all the time ;) | 00:39 |
| jascott1_ | job security | 00:40 |
| *** eanylin has joined #openstack-kolla | 00:43 | |
| *** tovin07_ has joined #openstack-kolla | 00:46 | |
| *** cuongnv has joined #openstack-kolla | 00:46 | |
| *** reddydodda has quit IRC | 01:02 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Add timeout parameter for precheck tasks https://review.openstack.org/462332 | 01:02 |
| *** dave-mccowan has joined #openstack-kolla | 01:03 | |
| *** Pavo has quit IRC | 01:03 | |
| *** gianpietro has joined #openstack-kolla | 01:08 | |
| *** yangyapepeng has joined #openstack-kolla | 01:11 | |
| *** reddydodda has joined #openstack-kolla | 01:11 | |
| *** yangyapepeng has quit IRC | 01:11 | |
| *** david-lyle has joined #openstack-kolla | 01:16 | |
| *** zhurong has quit IRC | 01:16 | |
| *** goldyfruit has joined #openstack-kolla | 01:20 | |
| *** yangyapeng has joined #openstack-kolla | 01:21 | |
| *** srwilkers_ has quit IRC | 01:25 | |
| *** srwilkers has quit IRC | 01:26 | |
| *** zhurong has joined #openstack-kolla | 01:26 | |
| *** hrw has quit IRC | 01:31 | |
| *** hrw has joined #openstack-kolla | 01:33 | |
| *** dixiaoli has joined #openstack-kolla | 01:33 | |
| *** david-lyle has quit IRC | 01:34 | |
| *** esmute has quit IRC | 01:37 | |
| *** ducttape_ has joined #openstack-kolla | 01:51 | |
| *** masber has quit IRC | 01:56 | |
| *** unicell has quit IRC | 02:01 | |
| *** jtriley has joined #openstack-kolla | 02:01 | |
| *** salv-orlando has joined #openstack-kolla | 02:05 | |
| *** salv-orlando has quit IRC | 02:09 | |
| *** dave-mccowan has quit IRC | 02:10 | |
| *** daidv has joined #openstack-kolla | 02:11 | |
| *** caowei has joined #openstack-kolla | 02:18 | |
| *** dave-mccowan has joined #openstack-kolla | 02:20 | |
| sdake | jascott1_ you assusme cisco pays him for working on kolla-kubenretes :) | 02:22 |
| sdake | jascott1_ bad assumption imo :) | 02:22 |
| *** zhurong has quit IRC | 02:34 | |
| *** eaguilar has quit IRC | 02:34 | |
| *** ducttape_ has quit IRC | 02:36 | |
| *** zhurong has joined #openstack-kolla | 02:37 | |
| *** caowei has quit IRC | 02:39 | |
| *** ssurana has quit IRC | 02:40 | |
| *** ducttape_ has joined #openstack-kolla | 02:45 | |
| *** caowei has joined #openstack-kolla | 02:45 | |
| *** dave-mccowan has quit IRC | 02:51 | |
| *** masber has joined #openstack-kolla | 02:52 | |
| *** david-lyle has joined #openstack-kolla | 02:53 | |
| *** zhubingbing_ has joined #openstack-kolla | 02:55 | |
| openstackgerrit | Merged openstack/kolla-ansible stable/ocata: Add ssh-keyscan to unblock kolla-ansible gate https://review.openstack.org/462052 | 02:58 |
| *** lucasxu has joined #openstack-kolla | 02:58 | |
| *** lucasxu has quit IRC | 03:00 | |
| *** britthouser10 has joined #openstack-kolla | 03:01 | |
| *** shash____ has joined #openstack-kolla | 03:02 | |
| *** unicell has joined #openstack-kolla | 03:04 | |
| *** sbezverk_ has joined #openstack-kolla | 03:04 | |
| *** gianpietro has quit IRC | 03:05 | |
| *** gianpietro has joined #openstack-kolla | 03:06 | |
| *** shasha_t- has joined #openstack-kolla | 03:06 | |
| *** salv-orlando has joined #openstack-kolla | 03:06 | |
| *** slagle__ has joined #openstack-kolla | 03:07 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix a typo and complement doc in ceph-guide.rst https://review.openstack.org/460431 | 03:09 |
| *** kbyrne_ has joined #openstack-kolla | 03:10 | |
| *** salv-orlando has quit IRC | 03:11 | |
| *** slagle has quit IRC | 03:11 | |
| *** pron_ has quit IRC | 03:11 | |
| *** harbie has quit IRC | 03:11 | |
| *** shasha___ has quit IRC | 03:11 | |
| *** britthouser has quit IRC | 03:11 | |
| *** sbezverk has quit IRC | 03:11 | |
| *** portdirect has quit IRC | 03:11 | |
| *** kbyrne has quit IRC | 03:11 | |
| *** kbyrne_ is now known as kbyrne | 03:11 | |
| *** harbie has joined #openstack-kolla | 03:12 | |
| *** tovin07_ has quit IRC | 03:16 | |
| *** tovin07_ has joined #openstack-kolla | 03:17 | |
| *** pron_ has joined #openstack-kolla | 03:18 | |
| *** masber has quit IRC | 03:24 | |
| *** goldyfruit has quit IRC | 03:26 | |
| *** dixiaoli has quit IRC | 03:30 | |
| *** ducttape_ has quit IRC | 03:33 | |
| *** georgem1 has quit IRC | 03:39 | |
| *** reddydodda has quit IRC | 03:54 | |
| *** salv-orlando has joined #openstack-kolla | 04:07 | |
| *** shash____ has quit IRC | 04:08 | |
| *** shasha_t_ has joined #openstack-kolla | 04:09 | |
| *** masber has joined #openstack-kolla | 04:11 | |
| *** salv-orlando has quit IRC | 04:11 | |
| *** zhurong has quit IRC | 04:12 | |
| *** ssurana has joined #openstack-kolla | 04:12 | |
| openstackgerrit | Merged openstack/kolla stable/ocata: Add missing cleanup step for new apache images https://review.openstack.org/462088 | 04:18 |
| *** xinliang has quit IRC | 04:20 | |
| *** gianpietro has quit IRC | 04:22 | |
| *** gianpietro has joined #openstack-kolla | 04:28 | |
| *** xinliang has joined #openstack-kolla | 04:33 | |
| *** zhurong has joined #openstack-kolla | 04:35 | |
| spsurya | sup all | 04:39 |
| *** salv-orlando has joined #openstack-kolla | 04:52 | |
| *** jtriley has quit IRC | 04:54 | |
| *** caowei has quit IRC | 05:02 | |
| *** unicell1 has joined #openstack-kolla | 05:02 | |
| *** unicell has quit IRC | 05:05 | |
| *** portdirect has joined #openstack-kolla | 05:13 | |
| *** jascott1_ has quit IRC | 05:15 | |
| *** gkadam has quit IRC | 05:31 | |
| *** Serlex has joined #openstack-kolla | 05:33 | |
| *** gkadam has joined #openstack-kolla | 05:34 | |
| *** japestinho has quit IRC | 05:40 | |
| *** janki has joined #openstack-kolla | 05:40 | |
| *** shasha_t_ has quit IRC | 05:44 | |
| *** gkadam has quit IRC | 06:00 | |
| *** caowei has joined #openstack-kolla | 06:00 | |
| *** gkadam has joined #openstack-kolla | 06:01 | |
| *** salv-orlando has quit IRC | 06:04 | |
| *** dciabrin has joined #openstack-kolla | 06:06 | |
| *** esmute1 has joined #openstack-kolla | 06:08 | |
| *** jascott1 has joined #openstack-kolla | 06:15 | |
| *** jascott1 has quit IRC | 06:20 | |
| *** sbezverk_ has quit IRC | 06:23 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix saraha-engine authencation issue https://review.openstack.org/461814 | 06:38 |
| openstackgerrit | Merged openstack/kolla-ansible master: mount /run in sahara-engine container https://review.openstack.org/461784 | 06:40 |
| *** rmart04_ has joined #openstack-kolla | 06:42 | |
| *** pcaruana has joined #openstack-kolla | 06:45 | |
| *** matrohon has joined #openstack-kolla | 06:51 | |
| *** shardy has joined #openstack-kolla | 07:08 | |
| openstackgerrit | junbo proposed openstack/kolla-ansible stable/ocata: Fix saraha-engine authencation issue https://review.openstack.org/462411 | 07:12 |
| openstackgerrit | junbo proposed openstack/kolla-ansible stable/ocata: mount /run in sahara-engine container https://review.openstack.org/462412 | 07:13 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Remove "project" empty variable from group_vars https://review.openstack.org/462413 | 07:16 |
| *** gianpietro has quit IRC | 07:17 | |
| *** manheim has joined #openstack-kolla | 07:19 | |
| *** salv-orlando has joined #openstack-kolla | 07:24 | |
| hrw | morning | 07:32 |
| *** nathharp has joined #openstack-kolla | 07:35 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 07:36 |
| *** egonzalez has joined #openstack-kolla | 07:43 | |
| *** shasha_t_ has joined #openstack-kolla | 07:44 | |
| *** ssurana has quit IRC | 07:48 | |
| *** shasha_t_ has quit IRC | 07:49 | |
| *** rmart04_ has quit IRC | 07:49 | |
| *** lpetrut has joined #openstack-kolla | 07:51 | |
| *** esmute1 has quit IRC | 08:00 | |
| *** blallau has joined #openstack-kolla | 08:08 | |
| *** jaosorior_away is now known as jaosorior | 08:14 | |
| *** rmart04 has joined #openstack-kolla | 08:15 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Add dots at first line in docstring https://review.openstack.org/462433 | 08:26 |
| apuimedo | egonzalez: do you use kolla-kubernetes? | 08:27 |
| egonzalez | apuimedo, nope for now | 08:28 |
| apuimedo | pity, I wanted to ask you about a couple of things off https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html | 08:28 |
| apuimedo | I finally managed to get kubernetes installed from scratch and helm on the archlinux server | 08:28 |
| apuimedo | :-) | 08:29 |
| apuimedo | so next step is kolla | 08:29 |
| *** esmute1 has joined #openstack-kolla | 08:31 | |
| *** ducttape_ has joined #openstack-kolla | 08:34 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/ocata: Fix cinder-api ubuntu binary https://review.openstack.org/458152 | 08:35 |
| *** athomas has joined #openstack-kolla | 08:38 | |
| *** ducttape_ has quit IRC | 08:39 | |
| pester | installing kolla 4.0.0, with pip install and default `all-in-one` inventory - at 'RUNNING HANDLER [neutron : Restart neutron-openvswitch-agent container] ********' - lost all connectivity to node | 08:40 |
| *** mgoddard has joined #openstack-kolla | 08:41 | |
| pester | can`t even imagine how it is possible :D route looks ok, iptables also... | 08:43 |
| pester | funny part, I can see bridges in host system but no brctl installed to inspect them - and network is down so I can`t install it)) | 08:46 |
| egonzalez | pester, whats the content of globals.yml and output of ip a? | 08:50 |
| egonzalez | apuimedo, most k8s folks are in US timezones, for very k8s specific questions I suggest you to wait for them ;) | 08:52 |
| pester | can`t see any errors, kolla-ansible was run in putty and it just disconnected. | 08:52 |
| apuimedo | egonzalez: I will. Thanks | 08:54 |
| pester | as I can see my external interface was added to br-ex, but address is left on inteface, but it should migrate to br-ex... I did it by hand and ot connectivity back to work | 08:54 |
| apuimedo | egonzalez: how long should "kolla-ansible genconfig" last | 08:54 |
| pester | maybe it is Xenial new behavior | 08:54 |
| apuimedo | it is taking alreadya a few minutes... | 08:55 |
| openstackgerrit | junbo proposed openstack/kolla stable/ocata: Add netcat package in sahara-engine image https://review.openstack.org/462444 | 08:55 |
| *** salv-orl_ has joined #openstack-kolla | 08:56 | |
| egonzalez | apuimedo, gets stuck in some task? or is generating a lot of config files? | 08:57 |
| pester | will try to reset my VM and run this in screen... may be kolla will set it right by itself =) | 08:58 |
| apuimedo | egonzalez: http://paste.openstack.org/show/608819/ | 08:59 |
| egonzalez | pester, i mean globals.yml is config file with kolla settings and "ip a" to match if there is some error between setting and network interfaces | 08:59 |
| openstackgerrit | Merged openstack/kolla stable/ocata: Add libosinfo package for source installation https://review.openstack.org/461940 | 08:59 |
| openstackgerrit | Merged openstack/kolla master: Update oraclelinux tag to new slim base image https://review.openstack.org/437329 | 08:59 |
| *** salv-orlando has quit IRC | 08:59 | |
| pester | egonzalez: can`t actually copy-paste cause no network, doing reset now | 09:00 |
| hrw | ;D | 09:00 |
| hrw | I just rebased patch and it need rebasing ;D | 09:01 |
| egonzalez | apuimedo, hrm maybe ssh issue, can you run ansible -m setup -i <all-in-one-inventory> localhost? | 09:02 |
| apuimedo | egonzalez: http://paste.openstack.org/show/608820/ | 09:02 |
| *** yangyapeng has quit IRC | 09:03 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 09:03 |
| * hrw off | 09:03 | |
| *** zhurong has quit IRC | 09:06 | |
| esmute1 | Hi guys, i got a question, anyway to find out why my neutron-openvswitch is stuck in init? | 09:06 |
| esmute1 | neutron-openvswitch-agent-network-9408j 0/1 Init:2/3 0 | 09:06 |
| esmute1 | i didnt see anything obvious when i did `kubectl describe pod neutron-openvswitch-agent-network-9408j` | 09:07 |
| zhubingbing_ | hi guys | 09:12 |
| *** pbourke has quit IRC | 09:13 | |
| *** pbourke has joined #openstack-kolla | 09:15 | |
| *** zhurong has joined #openstack-kolla | 09:15 | |
| *** japestinho has joined #openstack-kolla | 09:16 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Update docker_py_version_min version https://review.openstack.org/454561 | 09:17 |
| *** tovin07 has quit IRC | 09:18 | |
| *** jascott1 has joined #openstack-kolla | 09:18 | |
| *** esmute1 has quit IRC | 09:19 | |
| pester | ok I set neutron_external_interface to my real public interface and kolla just turned it off... | 09:20 |
| *** jascott1 has quit IRC | 09:23 | |
| pester | egonzalez: so I need to set 2x interfaces, network_interface: - my real public interface (where VIP will be allocated) and neutron_external_interface: - can be dummy interface for VMs outgoing connections ? | 09:24 |
| pester | it is correct ? | 09:24 |
| egonzalez | sup zhubingbing_ | 09:24 |
| zhubingbing_ | hi | 09:24 |
| egonzalez | pester, yep, neutron_external_interface cannot be network_interface since it will be managed by ovs and lost connection once br-ex is created | 09:25 |
| *** zhouya has joined #openstack-kolla | 09:25 | |
| egonzalez | you can use a dummy interfaces or whatever virt interface ansible recognizes | 09:26 |
| zhouya | I have met some problem when I deploy the kolla newton release,wan't to ask kolla team for help. | 09:27 |
| zhouya | When the kolla-ansible deployed to ceph | 09:27 |
| *** athomas has quit IRC | 09:27 | |
| zhouya | https://thepasteb.in/p/Anhr4zv9pV0cv | 09:28 |
| zhouya | failed: [192.168.71.6] (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdb', u'journal': u'/dev/sdb2', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': False})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "external_journal": false, | 09:28 |
| zhouya | "fs_label": "", "fs_uuid": "", "journal": "/dev/sdb2", "journal_device": "/dev/sdb", "journal_num": 2, "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "Container exited with non-zero return code"} | 09:28 |
| zhouya | TASK [ceph : Bootstrapping Ceph OSDs] ****************************************** | 09:28 |
| zhouya | The docker I see in my target node is : | 09:29 |
| zhouya | 3141141eb5c6 10.20.7.3:4000/kolla/centos-binary-ceph-mon:3.0.3 "kolla_start" 25 minutes ago Up 25 minutes ceph_mon | 09:29 |
| zhouya | 1f7a0ff1ad60 10.20.7.3:4000/kolla/centos-binary-cron:3.0.3 "kolla_start" 26 minutes ago Up 25 minutes cron | 09:29 |
| zhouya | 8ade2db9ceda 10.20.7.3:4000/kolla/centos-binary-kolla-toolbox:3.0.3 "kolla_start" 26 minutes ago Up 26 minutes kolla_toolbox | 09:29 |
| zhouya | 830071e08577 10.20.7.3:4000/kolla/centos-binary-heka:3.0.3 "kolla_start" 26 minutes ago Up 26 minutes heka | 09:29 |
| *** rmart04 has quit IRC | 09:29 | |
| zhouya | I don't know how to deal with this error,could someone help me to take a look as this? | 09:30 |
| *** yingjun has quit IRC | 09:31 | |
| zhouya | My email is zhou.ya@zte.com.cn.I am glad if anyone could help me,thank you very much. | 09:32 |
| egonzalez | zhouya, is there any failed container in target nodes? docker ps -a ? | 09:36 |
| *** cuongnv_ has joined #openstack-kolla | 09:39 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance https://review.openstack.org/453738 | 09:39 |
| pbourke | egonzalez: would 'outward' suit better than 'external' for our new rabbitmq instance? | 09:39 |
| *** zhouya has left #openstack-kolla | 09:39 | |
| *** zhuzeyu has joined #openstack-kolla | 09:40 | |
| *** cuongnv has quit IRC | 09:40 | |
| *** _zhouya has joined #openstack-kolla | 09:41 | |
| _zhouya | yes | 09:41 |
| *** lpetrut_ has joined #openstack-kolla | 09:41 | |
| _zhouya | the container:10.20.7.3:4000/kolla/centos-binary-ceph-osd:3.0.3 | 09:41 |
| _zhouya | failed | 09:41 |
| *** lpetrut has quit IRC | 09:41 | |
| _zhouya | Is this affected by ceph-mon? | 09:42 |
| *** lpetrut_ has quit IRC | 09:42 | |
| *** lpetrut_ has joined #openstack-kolla | 09:42 | |
| egonzalez | _zhouya, check docker logs <failed_osd_container> | 09:43 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Fix permission on chrony.conf https://review.openstack.org/462462 | 09:44 |
| *** zhouya has joined #openstack-kolla | 09:45 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Fix logging for chrony https://review.openstack.org/462463 | 09:46 |
| *** zhouya has quit IRC | 09:47 | |
| bjolo | morning | 09:47 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Fix logging for chrony https://review.openstack.org/462463 | 09:47 |
| egonzalez | pbourke, i had to look at google translator for that word hehe, but yes wfm | 09:48 |
| pbourke | egonzalez: :) | 09:48 |
| pbourke | maybe a little more obscure, but I agree we already use the term 'external' for non kolla deployed services | 09:48 |
| *** zhouya has joined #openstack-kolla | 09:48 | |
| egonzalez | btw, toolbox re-parent is not needed anymore? | 09:49 |
| bjolo | morning | 09:49 |
| egonzalez | sup bjolo | 09:49 |
| bjolo | frustration | 09:49 |
| bjolo | are there any knowns issues with kolla stable/ocata and docker 1.12.6 (i know i just dig through docs and bugs but... ) | 09:50 |
| bjolo | i can do a deploy | 09:51 |
| bjolo | but if i then do a destroy and deploy again, docker barfs | 09:51 |
| *** zhouya has quit IRC | 09:51 | |
| bjolo | i then have to remove all images, clean out docker cache and btrfs subvolumes manually and restart docker before i can deploy again | 09:54 |
| egonzalez | bjolo, i have 1.12.6 on my dev env and deploy/destroy few times per day and still working | 09:54 |
| bjolo | hmmm | 09:54 |
| bjolo | wtf | 09:54 |
| *** zhouya has joined #openstack-kolla | 09:54 | |
| egonzalez | hrm, i dont use brtfs | 09:54 |
| bjolo | what kernel? | 09:54 |
| egonzalez | 3.14.32-xxxx-grs-ipv6-64 | 09:55 |
| bjolo | i use the 4.11.0-1 from elrepo | 09:55 |
| bjolo | latest kernel-ml package | 09:55 |
| bjolo | centos 7 of course as host os | 09:55 |
| *** zhouya has quit IRC | 09:55 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance https://review.openstack.org/453738 | 09:56 |
| egonzalez | bjolo, sorry, this is the kernel in compute nodes 3.10.0-514.16.1.el7.x86_64 | 09:56 |
| egonzalez | other is from deployer | 09:56 |
| *** magicboiz has joined #openstack-kolla | 09:57 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance https://review.openstack.org/453738 | 09:58 |
| bjolo | but not using btrfs? | 09:58 |
| egonzalez | nope | 10:00 |
| *** eaguilar has joined #openstack-kolla | 10:00 | |
| *** zhouya has joined #openstack-kolla | 10:01 | |
| *** magicboiz has quit IRC | 10:01 | |
| *** zhouya has quit IRC | 10:02 | |
| *** tovin07_ has quit IRC | 10:04 | |
| *** zhouya has joined #openstack-kolla | 10:04 | |
| *** eaguilar has quit IRC | 10:05 | |
| *** zhouya has quit IRC | 10:06 | |
| *** zhouya has joined #openstack-kolla | 10:06 | |
| egonzalez | bjolo, have you tried with other kernel / docker versions/ storage driver? | 10:06 |
| bjolo | not yet | 10:07 |
| *** zhurong has quit IRC | 10:07 | |
| *** zhouya has quit IRC | 10:08 | |
| bjolo | we have had a ton of infra issues, so we just did the first kolla eploy this morning | 10:08 |
| bjolo | some more TS and googling before trying things out :) | 10:09 |
| *** magicboiz has joined #openstack-kolla | 10:14 | |
| *** guoshan has joined #openstack-kolla | 10:20 | |
| *** daidv has quit IRC | 10:21 | |
| *** cuongnv has joined #openstack-kolla | 10:22 | |
| *** cuongnv has quit IRC | 10:23 | |
| *** athomas has joined #openstack-kolla | 10:23 | |
| *** cuongnv_ has quit IRC | 10:24 | |
| *** zhurong has joined #openstack-kolla | 10:24 | |
| *** ever has joined #openstack-kolla | 10:24 | |
| *** guoshan has quit IRC | 10:26 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance https://review.openstack.org/453738 | 10:29 |
| *** zhuzeyu has quit IRC | 10:30 | |
| *** _zhouya has quit IRC | 10:30 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano https://review.openstack.org/454206 | 10:30 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance https://review.openstack.org/453738 | 10:30 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 10:32 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano https://review.openstack.org/454206 | 10:32 |
| *** apuimedo has quit IRC | 10:33 | |
| *** gfidente has joined #openstack-kolla | 10:39 | |
| *** gfidente has quit IRC | 10:39 | |
| *** gfidente has joined #openstack-kolla | 10:39 | |
| hrw | egonzalez: can you +2 https://review.openstack.org/430940 patch? | 10:44 |
| manheim | Hi, post deployment, can i run kolla-ansible to only restart a certaint existing service? like cinder | 10:53 |
| *** zhouya has joined #openstack-kolla | 10:54 | |
| zhouya | egonzalez | 10:54 |
| zhouya | sorry，the network is not stable | 10:54 |
| zhouya | seems the keyring error？ | 10:55 |
| *** salv-orl_ has quit IRC | 10:55 | |
| egonzalez | manheim, kolla-ansible only restart the container if image changed or config files changed | 10:55 |
| egonzalez | manheim, if something changed, use kolla-ansible deploy --tags cinder to only apply common and cinder role | 10:56 |
| manheim | hmm, i need this for nova-hyperv stuff | 10:56 |
| manheim | to control the service via ansible | 10:56 |
| egonzalez | manheim, do register on config files tasks and notify handlers to restart the service (similar to what is doing with containers but with services) | 10:57 |
| egonzalez | zhouya, what error is giving? | 10:58 |
| manheim | i already do that, but only on deploy | 10:58 |
| zhouya | [root@host-192-168-71-5 ~]# docker logs 263244459b40 | 10:58 |
| zhouya | INFO:__main__:Kolla config strategy set to: COPY_ALWAYS | 10:58 |
| zhouya | INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json | 10:58 |
| zhouya | INFO:__main__:Validating config file | 10:58 |
| zhouya | INFO:__main__:Copying service configuration files | 10:58 |
| zhouya | INFO:__main__:Copying /var/lib/kolla/config_files/ceph.conf to /etc/ceph/ceph.conf | 10:58 |
| zhouya | INFO:__main__:Setting permissions for /etc/ceph/ceph.conf | 10:58 |
| zhouya | INFO:__main__:Copying /var/lib/kolla/config_files/ceph.client.admin.keyring to /etc/ceph/ceph.client.admin.keyring | 10:58 |
| zhouya | INFO:__main__:Setting permissions for /etc/ceph/ceph.client.admin.keyring | 10:58 |
| zhouya | INFO:__main__:Writing out command to execute | 10:58 |
| zhouya | 2017-04-28 16:32:51.854980 7f6e3795b700 0 monclient(hunting): authenticate timed out after 300 | 10:58 |
| zhouya | 2017-04-28 16:32:51.855032 7f6e3795b700 0 librados: client.admin authentication error (110) Connection timed out | 10:58 |
| zhouya | Error connecting to cluster: TimedOut | 10:58 |
| zhouya | 6 | 10:58 |
| zhouya | egonzalez | 11:02 |
| zhouya | have you met this problem？ | 11:02 |
| egonzalez | zhouya, is monitor running fine? do docker exec ceph_mon ceph -s | 11:03 |
| egonzalez | manheim, to restart a service with no other change, we dont actually have nothing similar | 11:04 |
| manheim | i see | 11:05 |
| manheim | coz to do that you simply restart the container | 11:05 |
| manheim | i guessi can add an aditional playbook to do that on hyperv | 11:05 |
| egonzalez | yes, but only if restart is needed(changes) if nothing changes cannot see a reason to restart a service | 11:06 |
| egonzalez | container task ensure the container is running, if is down starts if is up do nothing | 11:06 |
| manheim | i see | 11:07 |
| manheim | one other question | 11:07 |
| manheim | so I have the ml2.conf file for both neutron-server and neutron-ovs-agent | 11:07 |
| manheim | if i want to make a change in ml2, i have to edit both files in the locations, right? | 11:07 |
| manheim | like /etc/kolla/neutron-server and /etc/kolla/neutron-ovs... | 11:08 |
| manheim | asking if there is a simpler solution | 11:08 |
| *** dave-mccowan has joined #openstack-kolla | 11:08 | |
| zhouya | the env is redeployed，if I met this situation time，Iwill contact you，thankyou for your help egonzalez | 11:08 |
| zhouya | if I met this situation next time，sorry | 11:09 |
| egonzalez | manheim, with custom config you can modify only one, both, or all ml2 files. | 11:09 |
| egonzalez | manheim, etc/kolla/config/ml2.conf | 11:10 |
| *** rwellum has joined #openstack-kolla | 11:10 | |
| egonzalez | etc/kolla/config/<service_name>/ml2.conf | 11:10 |
| manheim | etc/kolla/config/ml2.conf is only pre-deployment, right? | 11:10 |
| egonzalez | etc/kolla/config/neutron/<inventory_hostname>/ml2.conf | 11:11 |
| egonzalez | https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/config.yml#L117 | 11:11 |
| egonzalez | manheim, can be made after deployment with a reconfigure or deploy again | 11:12 |
| *** zhouya has quit IRC | 11:12 | |
| manheim | what does the reconfigure do exactly? i never fully understood | 11:12 |
| egonzalez | manheim, from ocata reconfigure do the same as deploy. changes config files, start containers if needed, etc | 11:14 |
| *** ever has quit IRC | 11:14 | |
| manheim | like a factory reset | 11:15 |
| *** ever has joined #openstack-kolla | 11:15 | |
| egonzalez | not really, lets say you want to change debug from false to true, you run reconfigure to apply that change into config files and restart containers to refresh service with new config | 11:17 |
| *** jascott1 has joined #openstack-kolla | 11:20 | |
| manheim | ah, so in this case you set debug=true, in let's say /etc/kolla/config/nova.conf and run reconfigure. this sets nova.conf for all services using it? | 11:21 |
| egonzalez | yep | 11:22 |
| manheim | ah, I see, thanks | 11:24 |
| *** jascott1 has quit IRC | 11:25 | |
| egonzalez | hrw, added comments on arch change | 11:28 |
| hrw | egonzalez: ok | 11:28 |
| hrw | thx | 11:29 |
| egonzalez | base image is getting quite complex and hard to follow, especially repo config stuff | 11:30 |
| *** rwallner has joined #openstack-kolla | 11:31 | |
| hrw | egonzalez: yes, it is | 11:31 |
| hrw | egonzalez: especially with that delorean stuff moved to code | 11:31 |
| pester | finally deployed kolla 4.0.0, with KVM, but forgot "expose virtualization bit", I get error from nova - "Error: invalid argument: could not find capabilities for domaintype=kvm" | 11:32 |
| egonzalez | pester, mkdir -p /etc/kolla/config/nova | 11:32 |
| egonzalez | cat << EOF > /etc/kolla/config/nova/nova-compute.conf | 11:32 |
| egonzalez | [libvirt] | 11:32 |
| egonzalez | virt_type = qemu | 11:32 |
| egonzalez | cpu_mode = none | 11:32 |
| egonzalez | EOF | 11:32 |
| pester | so, now I got it, how to tell kolla to use it ? | 11:32 |
| egonzalez | then, kolla-ansible deploy --tags nova | 11:33 |
| *** sbezverk has joined #openstack-kolla | 11:40 | |
| *** shardy is now known as shardy_lunch | 11:43 | |
| *** srwilkers has joined #openstack-kolla | 11:50 | |
| *** athomas has quit IRC | 11:52 | |
| *** salv-orlando has joined #openstack-kolla | 11:54 | |
| *** rwallner has quit IRC | 11:55 | |
| hrw | egonzalez: will https://pastebin.com/qLrHGqep be enough? for delorean todo | 11:58 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute https://review.openstack.org/449529 | 11:59 |
| egonzalez | hrw, wfm | 11:59 |
| pester | finally deployed kolla with kvm, but when I boot image it stuck at start with "GRUB _" on VNC | 12:03 |
| hrw | egonzalez: thx for hint about whitespace stuff | 12:03 |
| *** srwilkers has quit IRC | 12:03 | |
| hrw | egonzalez: kolla is the only project I played with jinja2 | 12:03 |
| egonzalez | hrw, will do with ansible ;) | 12:04 |
| hrw | ;d | 12:04 |
| egonzalez | pester, that issue is likely an image issue, are you using cirros image or other? | 12:04 |
| pester | cirros, tried 2 different versions | 12:04 |
| pester | downloading trusty now) | 12:05 |
| egonzalez | pester, can you share console logs? nova console-log <instance_name> | 12:05 |
| *** rwallner has joined #openstack-kolla | 12:06 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 12:08 |
| hrw | egonzalez: 47th version is a bit shorter thanks to you ;) | 12:09 |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable. https://review.openstack.org/452094 | 12:10 |
| *** tvignaud_ has quit IRC | 12:11 | |
| *** srwilkers has joined #openstack-kolla | 12:14 | |
| *** shardy_lunch is now known as shardy | 12:16 | |
| *** schwicht has quit IRC | 12:23 | |
| *** tvignaud_ has joined #openstack-kolla | 12:26 | |
| *** zhurong has quit IRC | 12:30 | |
| pester | egonzalez: no output from console logs and ubuntu also can`t load... I think it cause I run it inside ESX with "expose virtualization bit" option | 12:32 |
| egonzalez | pester, some years since the last time i touched vmware stuff, guess if virtualization is exposed to the VM, can use KVM instead of qemu | 12:34 |
| egonzalez | pester, have you checked if virtualization is enabled in compute nodes? egrep -c '(vmx|svm)' /proc/cpuinfo | 12:34 |
| pester | yes, and I use it, and devstack ocata is ok, and many mitaka installations | 12:34 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Remove keystone-paste.ini file in kolla https://review.openstack.org/462535 | 12:34 |
| pester | yes, returns '2' | 12:34 |
| pester | I can see kvm loaded | 12:35 |
| *** athomas has joined #openstack-kolla | 12:36 | |
| *** tvignaud_ has quit IRC | 12:36 | |
| *** slagle__ is now known as slagle | 12:40 | |
| *** gianpietro has joined #openstack-kolla | 12:41 | |
| egonzalez | pester, dunno then, can you share full trace of nova compute and libvirt logs with kvm enabled as virt_type? | 12:44 |
| *** blallau has quit IRC | 12:45 | |
| *** rwallner has quit IRC | 12:47 | |
| pester | egonzalez: http://paste.openstack.org/show/608859/ - as you can see some errors in libvirt when "HW virtualization" was not enabled - it is old logs | 12:47 |
| *** tvignaud_ has joined #openstack-kolla | 12:48 | |
| *** goldyfruit has joined #openstack-kolla | 12:50 | |
| *** schwicht has joined #openstack-kolla | 12:53 | |
| *** rwallner has joined #openstack-kolla | 12:53 | |
| srwilkers | morning | 12:54 |
| *** eaguilar has joined #openstack-kolla | 12:58 | |
| *** guoshan has joined #openstack-kolla | 13:00 | |
| Serlex | Morning | 13:00 |
| *** schwicht has quit IRC | 13:03 | |
| *** eaguilar has quit IRC | 13:03 | |
| *** eaguilar has joined #openstack-kolla | 13:03 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano https://review.openstack.org/454206 | 13:04 |
| *** schwicht has joined #openstack-kolla | 13:06 | |
| *** georgem1 has joined #openstack-kolla | 13:11 | |
| *** janem has joined #openstack-kolla | 13:14 | |
| janem | Hey everyone, I am having issues with configuring "kolla_external_vip_address" and "kolla_external_fqdn" | 13:15 |
| *** eanylin has quit IRC | 13:16 | |
| janem | my "kolla_external_vip_address" is a "local" address, and we have a external firewall that NAT's a public IP to that address. | 13:16 |
| janem | but we cant get it to work for some reasin, if we go to our "kolla_external_vip_address" behind the firewall, it works. | 13:17 |
| janem | we opened port 80 and 443 from the public IP to the internal one. So we should se the horizon login page, right? | 13:18 |
| janem | probably out of the scope of this channel, but I thought I'd try here. | 13:20 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/462547 | 13:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/462548 | 13:23 |
| *** janki has quit IRC | 13:24 | |
| *** imcsk8 has joined #openstack-kolla | 13:29 | |
| *** rwallner has quit IRC | 13:29 | |
| janem | it says in the global.yml config file, that the "kolla_external_fqdn" must point to the "kolla_external_vip_address", wondering if that must be direct, and hence not work trough NAT. | 13:29 |
| *** schwicht has quit IRC | 13:29 | |
| *** srwilkers has quit IRC | 13:32 | |
| *** eanylin_ has joined #openstack-kolla | 13:34 | |
| *** eanylin_ has quit IRC | 13:36 | |
| *** jtriley has joined #openstack-kolla | 13:36 | |
| *** eanylin__ has joined #openstack-kolla | 13:38 | |
| *** zhurong has joined #openstack-kolla | 13:41 | |
| *** skramaja has joined #openstack-kolla | 13:42 | |
| *** skramaja has quit IRC | 13:45 | |
| *** Pavo has joined #openstack-kolla | 13:48 | |
| jemcevoy | janem: To get horizon to work... You'd need to do port forwarding on the firewall to the kolla_external_vip_address ans actually connect to the firewall's Public IP... | 13:48 |
| hrw | question to those who build debian/ubuntu images: do you use some kind of web proxy while building? | 13:49 |
| *** zhurong_ has joined #openstack-kolla | 13:50 | |
| janem | jemcevoy: yeah, we have done that, and if we change the NAT rule for the public IP to a functioning webserver on the same network, it works. So the port mapping should be ok. | 13:50 |
| hrw | I have a change to merge which will add some fetching | 13:50 |
| *** zhurong has quit IRC | 13:51 | |
| *** lucasxu has joined #openstack-kolla | 13:52 | |
| *** gkadam has quit IRC | 13:53 | |
| jemcevoy | jamem: You really need to use a VPN to access the network behind the firewall. Port forwarding may work for Horizon but you won;t have access to the VMs if you are using non-routing rfc 1918 IP address inside the firewall. | 13:54 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: always update APT lists when install packages https://review.openstack.org/462567 | 13:56 |
| janem | jemcevoy: yes, we are using vpn to access the network. Thats how I can confirm that the internal "kolla_external_vip_address" is working | 13:56 |
| janem | jemcevoy: we have a separate "neutron_external_interface" with a different network setup. | 13:56 |
| hrw | egonzalez, sdake, akwasnie, Jeffrey4l: https://review.openstack.org/462567 is imho important for those people who do not build all images at once. | 13:57 |
| Jeffrey4l | hrw, yes. apt-get update do not run as "yum" which could update local cache automatically. | 13:58 |
| egonzalez | hrw, have to remove this apt-get update too https://github.com/openstack/kolla/blob/master/docker/opendaylight/Dockerfile.j2#L28 | 13:58 |
| openstackgerrit | Merged openstack/kolla master: Add netcat package in sahara-engine image https://review.openstack.org/461706 | 13:58 |
| hrw | egonzalez: ok | 13:58 |
| *** jemcevoy has quit IRC | 13:59 | |
| hrw | egonzalez: that opendaylight/Dockerfile.j2 should be rewritten first | 13:59 |
| hrw | it is abomination | 13:59 |
| hrw | adding repos is job of base image | 14:01 |
| egonzalez | ya know! will be reworking that image soon, at least to enable odl binary in source images(same as ceph,mariadb and rabbit) | 14:01 |
| hrw | egonzalez: I prefer to not touch it then | 14:01 |
| egonzalez | in kolla we do not support source packages for non openstack images, odl is the same case, install binary on both source and binary images | 14:02 |
| *** srwilkers has joined #openstack-kolla | 14:03 | |
| hrw | egonzalez: I am perfectly fine with such ones. but handling repos is base image work | 14:04 |
| egonzalez | hrw, can remove in your change, won't harm at all, just will avoid running clean up twice | 14:04 |
| hrw | egonzalez: my brain cells die when I look at that dockerfile | 14:04 |
| hrw | will rewrite it probably instead | 14:04 |
| eanylin__ | egonzalez: Can we use centos source for rabbitmq for the 4.0.0 kolla-ansible multi-node deployment or we need to use binary? I see the rabbitmq going into crash loop with centos source 4.0.0 image... | 14:05 |
| egonzalez | hrw, the issue with odl repos is that are incredible slow, at least when i tested increased a lot build times | 14:05 |
| hrw | ;( | 14:06 |
| egonzalez | eanylin__, rabbitmq binary and source are almost the same, just a few differences from base image, but rabbit packages are the same | 14:07 |
| egonzalez | eanylin__, what issue you having? | 14:07 |
| eanylin__ | egonzalez: Ok, I am seeing rabbitmq crashing and hence cell0 cannot be set up | 14:08 |
| egonzalez | eanylin__, common rabbitmq issue is hostname resolution | 14:08 |
| eanylin__ | I did cleanup of hosts and containers a few times but didnt really get past that, kind of strange | 14:08 |
| eanylin__ | oh, ok | 14:08 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: always update APT lists when install packages https://review.openstack.org/462567 | 14:08 |
| eanylin__ | I ran the pre-checks and have put the hostname in /etc/hosts | 14:09 |
| hrw | opendaylight added | 14:09 |
| eanylin__ | maybe I need to check again, will that be sufficient or? | 14:09 |
| egonzalez | eanylin__, if fail again, share rabbitmq logs | 14:10 |
| eanylin__ | ok | 14:10 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: handle mariadb for aarch64/ubuntu/source https://review.openstack.org/461072 | 14:11 |
| *** zhurong_ has quit IRC | 14:12 | |
| eanylin__ | egonzalez: http://paste.openstack.org/show/608866/ | 14:12 |
| eanylin__ | seems like its trying to start the rabbitmq-server and wasnt able to | 14:13 |
| *** janem has left #openstack-kolla | 14:19 | |
| *** janem_ has joined #openstack-kolla | 14:19 | |
| *** jascott1 has joined #openstack-kolla | 14:23 | |
| egonzalez | eanylin__, can you share rabbitmq logs /var/lib/docker/volumes/kolla_logs/_data/rabbitmq/ | 14:25 |
| *** schwicht has joined #openstack-kolla | 14:27 | |
| *** jascott1 has quit IRC | 14:28 | |
| eanylin__ | egonzalez, I think I found the problem. I did something dumb, my hostname and hostname -f are different | 14:29 |
| eanylin__ | egonzalez: I changed it and the rabbitmq seems to be stable now | 14:29 |
| eanylin__ | egonzalez: Thanks for the tip :) | 14:29 |
| egonzalez | eanylin__, np | 14:29 |
| eanylin__ | egonzalez: Will continue to deploy the cluster. Thanks! | 14:30 |
| *** lrensing has joined #openstack-kolla | 14:31 | |
| *** manheim has quit IRC | 14:33 | |
| *** guoshan has quit IRC | 14:33 | |
| *** jtriley has quit IRC | 14:34 | |
| openstackgerrit | Sam Betts proposed openstack/kolla-kubernetes master: Ensure that ironic-dnsmasq can find its IP address https://review.openstack.org/462310 | 14:35 |
| *** ducttape_ has joined #openstack-kolla | 14:37 | |
| *** jtriley has joined #openstack-kolla | 14:37 | |
| pester | egonzalez: where I should place bug ? github or launchpad ? | 14:40 |
| inc0 | good morning | 14:40 |
| eanylin__ | egonzalez: Seems like I may not have found the real root cause for the nova-compute service not coming up issue. So seems like rabbitmq is up now but nova-compute didnt come up. Any other tips for that? Current situation looks like this, http://paste.openstack.org/show/608872/ | 14:41 |
| egonzalez | pester, launchpad, provide complete information: logs, environment, versions, config, custom_configs, etc | 14:41 |
| *** srwilkers has quit IRC | 14:42 | |
| egonzalez | eanylin__, is the compute node container up? | 14:42 |
| *** ducttape_ has quit IRC | 14:42 | |
| *** ducttape_ has joined #openstack-kolla | 14:43 | |
| *** lpetrut_ has quit IRC | 14:44 | |
| eanylin__ | egonzalez: Yeah, the nova-compute container is up. As I can see, http://paste.openstack.org/show/608873/ | 14:44 |
| egonzalez | eanylin__, what kolla and kolla-ansible version are you using? check nova api version also. docker exec nova_api pip freeze | grep nova | 14:45 |
| eanylin__ | egonzalez, was trying master branch and later just went back to "pip install kolla-ansible" | 14:46 |
| egonzalez | images still master or rebuild with stable? | 14:46 |
| eanylin__ | egonzalez: Image is Centos Source 4.0.0 | 14:47 |
| egonzalez | ansible stable/ocata nova code is not compatible with master images | 14:47 |
| eanylin__ | I just did kolla-build with push to the local registry | 14:47 |
| eanylin__ | egonzalez: http://paste.openstack.org/show/608876/ | 14:48 |
| eanylin__ | egonzalez: Ok | 14:48 |
| eanylin__ | egonzalez: I did this, kolla-build --push --type source --registry 192.168.122.21:4000 | 14:49 |
| egonzalez | the issue is with cell creation code in ansible ocata does not work with nova multi cells new code | 14:49 |
| janem_ | when I change settings in my globals.yml file, I should run "kolla-ansible reconfigure" right? | 14:50 |
| egonzalez | not sure if we should backport the change or not :/ | 14:50 |
| eanylin__ | egonzalez: I see. What should be approach in this case? | 14:50 |
| egonzalez | janem_, yep | 14:50 |
| egonzalez | janem_, well, depends of what changed, if added a new service need deploy | 14:50 |
| egonzalez | eanylin__, use same version of images and ansible, bump both to master or downgrade both to ocata | 14:51 |
| janem_ | egonzalez: I changed "kolla_external_vip_address" and "kolla_external_fqdn" | 14:51 |
| eanylin__ | egonzalez: Ok. By master image, we are referring to 4.0.1 right? | 14:52 |
| janem_ | changes doesnt seem to happen though, still se "kolla_internal_vip_address" in my openrc files .. shouldnt that have changed to the new ips/fqdn? | 14:52 |
| janem_ | egonzalez: no errors when I run reconfigure, and do report changes. | 14:54 |
| egonzalez | janem_, the issue with those settings is that old fqdn is associated with keystone public endpoints | 14:54 |
| *** eaguilar has quit IRC | 14:55 | |
| *** srwilkers has joined #openstack-kolla | 14:55 | |
| egonzalez | janem_, nope, adminrc file always is created with internal url | 14:56 |
| *** salv-orl_ has joined #openstack-kolla | 14:56 | |
| janem_ | egonzalez: ok. Hm.. It's strange. The new vip is working for the "external_vip_address" but cant seem to get the fqdn working. | 14:57 |
| *** salv-orlando has quit IRC | 14:59 | |
| janem_ | egonzalez: my external_vip is just a different internal vip address, and we have an external firewall that NAT's a public IP to that one. Then external DNS setup to point to the public IP. | 14:59 |
| janem_ | done this a million times with other web servers, never been an issue. So I dont think it's the firewall/nat thats the problem. | 15:00 |
| *** yingjun has joined #openstack-kolla | 15:00 | |
| *** gfidente has quit IRC | 15:01 | |
| *** eaguilar has joined #openstack-kolla | 15:02 | |
| *** unicell1 has quit IRC | 15:03 | |
| *** unicell has joined #openstack-kolla | 15:06 | |
| egonzalez | janem_, have a similar env with external on same network than internal but without all the NAT/firewall stuff, this is my config http://paste.openstack.org/show/608879/ | 15:07 |
| *** shasha_t_ has joined #openstack-kolla | 15:07 | |
| egonzalez | had some issues when reconfigured, just restarted all services and went fine iirc | 15:07 |
| *** Manheim has joined #openstack-kolla | 15:09 | |
| janem_ | egonzalez: thanks! It looks similar, but I have not defined the "kolla_external_vip_interface" , but since the external_vip address is working, I guess I dont need that configured (?) | 15:09 |
| janem_ | egonzalez: I'll try to restart the services | 15:11 |
| *** shasha_t_ has quit IRC | 15:11 | |
| *** jtriley has quit IRC | 15:19 | |
| janem_ | egonzalez: didn't help. So strange that the vip is working, but not the NAT'ed ip.. I'll meditate on it ;) | 15:20 |
| janem_ | Thanks for the help though! | 15:20 |
| *** jascott1 has joined #openstack-kolla | 15:23 | |
| *** mgoddard has quit IRC | 15:24 | |
| janem_ | is it enough to open port 80 and 443 to the vip to get horizon loaded? Maybe I'm missing some ports? | 15:25 |
| *** jascott1 has quit IRC | 15:29 | |
| *** eaguilar has quit IRC | 15:30 | |
| *** jrich523 has joined #openstack-kolla | 15:30 | |
| *** lrensing has quit IRC | 15:34 | |
| *** ducttap__ has joined #openstack-kolla | 15:38 | |
| *** ducttape_ has quit IRC | 15:38 | |
| *** lrensing has joined #openstack-kolla | 15:39 | |
| *** mgoddard has joined #openstack-kolla | 15:39 | |
| *** magicboiz has quit IRC | 15:41 | |
| *** lrensing has quit IRC | 15:47 | |
| *** lrensing has joined #openstack-kolla | 15:49 | |
| *** lucasxu has quit IRC | 15:53 | |
| *** jaosorior has quit IRC | 15:54 | |
| georgem1 | I'm a first time kolla user and I just ran tag 4.0.0 but the login page for the dashboard has no logo and I see a lot of 404s for "/static/dashboard/js/" stuff | 15:54 |
| georgem1 | any idea what I did wrong? | 15:54 |
| *** janem__ has joined #openstack-kolla | 15:54 | |
| *** srwilkers_ has joined #openstack-kolla | 15:55 | |
| *** lucasxu has joined #openstack-kolla | 15:56 | |
| *** lucasxu has quit IRC | 15:56 | |
| *** lucasxu has joined #openstack-kolla | 15:56 | |
| *** janem_ has quit IRC | 15:57 | |
| *** srwilkers_ has quit IRC | 15:57 | |
| *** vhosakot has joined #openstack-kolla | 15:59 | |
| *** jtriley has joined #openstack-kolla | 15:59 | |
| *** lucasxu has quit IRC | 15:59 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Fix Incorrect timezone in ubuntu horizon images https://review.openstack.org/460770 | 16:01 |
| *** lucasxu has joined #openstack-kolla | 16:02 | |
| *** egonzalez has quit IRC | 16:02 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Advertise public heat endpoint for wait conditions https://review.openstack.org/462655 | 16:04 |
| *** caowei has quit IRC | 16:05 | |
| *** caowei has joined #openstack-kolla | 16:05 | |
| *** janem has joined #openstack-kolla | 16:06 | |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla-ansible master: Add skydive service https://review.openstack.org/460922 | 16:07 |
| *** janem__ has quit IRC | 16:09 | |
| *** eaguilar has joined #openstack-kolla | 16:10 | |
| *** athomas has quit IRC | 16:14 | |
| *** eaguilar has quit IRC | 16:15 | |
| *** lucasxu has quit IRC | 16:15 | |
| *** unicell has quit IRC | 16:16 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix heat ec2 keystone auth https://review.openstack.org/444953 | 16:17 |
| *** eaguilar has joined #openstack-kolla | 16:18 | |
| *** eaguilar has quit IRC | 16:18 | |
| *** vhosakot_ has joined #openstack-kolla | 16:21 | |
| *** vhosakot has quit IRC | 16:23 | |
| *** jascott1 has joined #openstack-kolla | 16:24 | |
| *** salv-orl_ has quit IRC | 16:26 | |
| openstackgerrit | David Moreau Simard proposed openstack/kolla-ansible master: Do not merge: Test ARA 0.13.0rc2 https://review.openstack.org/439963 | 16:27 |
| *** eaguilar has joined #openstack-kolla | 16:28 | |
| *** jascott1 has quit IRC | 16:29 | |
| *** nathharp has quit IRC | 16:33 | |
| *** eaguilar has quit IRC | 16:34 | |
| *** tumble has joined #openstack-kolla | 16:38 | |
| *** mgoddard has quit IRC | 16:39 | |
| *** pcaruana has quit IRC | 16:40 | |
| *** athomas has joined #openstack-kolla | 16:40 | |
| *** jascott1 has joined #openstack-kolla | 16:43 | |
| *** shasha_t_ has joined #openstack-kolla | 16:44 | |
| *** erlon has joined #openstack-kolla | 16:48 | |
| *** lucasxu has joined #openstack-kolla | 16:49 | |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: handle mariadb for aarch64/ubuntu/source https://review.openstack.org/461072 | 16:51 |
| *** shardy has quit IRC | 16:57 | |
| *** athomas has quit IRC | 16:59 | |
| *** srwilkers has quit IRC | 17:00 | |
| *** rwallner has joined #openstack-kolla | 17:01 | |
| *** unicell has joined #openstack-kolla | 17:01 | |
| *** athomas has joined #openstack-kolla | 17:01 | |
| *** rwallner_ has joined #openstack-kolla | 17:02 | |
| *** rwallner has quit IRC | 17:05 | |
| *** hieulq_ has joined #openstack-kolla | 17:05 | |
| *** rwallner_ has quit IRC | 17:07 | |
| *** jascott1 has quit IRC | 17:07 | |
| *** rwallner has joined #openstack-kolla | 17:08 | |
| *** vhosakot_ has quit IRC | 17:09 | |
| *** rwallner has quit IRC | 17:09 | |
| *** rwallner has joined #openstack-kolla | 17:09 | |
| *** vhosakot has joined #openstack-kolla | 17:10 | |
| *** jascott1 has joined #openstack-kolla | 17:16 | |
| *** rwallner has quit IRC | 17:21 | |
| *** lpetrut has joined #openstack-kolla | 17:24 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/462547 | 17:25 |
| *** igordcar1 has quit IRC | 17:27 | |
| *** igordcard has joined #openstack-kolla | 17:27 | |
| *** hieulq_ has quit IRC | 17:37 | |
| *** hieulq_ has joined #openstack-kolla | 17:39 | |
| *** egonzalez has joined #openstack-kolla | 17:42 | |
| *** unicell has quit IRC | 17:47 | |
| *** yingjun has quit IRC | 17:48 | |
| *** unicell has joined #openstack-kolla | 17:48 | |
| *** ducttap__ has quit IRC | 17:51 | |
| *** mgoddard has joined #openstack-kolla | 17:51 | |
| *** ducttape_ has joined #openstack-kolla | 17:52 | |
| inc0 | kfox1111: it was ovs 2.6 when ovs got this massive performance boost right? | 17:53 |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Advertise public heat endpoint for wait conditions https://review.openstack.org/462655 | 17:54 |
| *** ducttape_ has quit IRC | 17:56 | |
| *** lpetrut has quit IRC | 18:03 | |
| *** srwilkers has joined #openstack-kolla | 18:13 | |
| *** dciabrin has quit IRC | 18:14 | |
| *** dciabrin has joined #openstack-kolla | 18:18 | |
| *** harlowja has quit IRC | 18:19 | |
| *** rwallner has joined #openstack-kolla | 18:20 | |
| sbezverk | inc0: you will see performance boost when we add networking-vpp ;) | 18:22 |
| inc0 | lol | 18:23 |
| inc0 | or when we add dpdk | 18:23 |
| inc0 | but yeah, I googled, it was 2.6 | 18:23 |
| inc0 | 160+% perf increase | 18:23 |
| *** tonanhngo has joined #openstack-kolla | 18:28 | |
| *** ducttape_ has joined #openstack-kolla | 18:29 | |
| *** rwallner has quit IRC | 18:33 | |
| *** Pavo has quit IRC | 18:34 | |
| sbezverk | inc0: there were some studies and trials, it seems ovs has performance issue with some type of packets I do not recall exactly the details, but similar tests with networking-vpp showed more consistent performance results :) | 18:47 |
| sbezverk | cannot wait to get it in kk | 18:47 |
| jascott1 | anyone used this? https://github.com/trailofbits/algo | 18:48 |
| jascott1 | my security minded friend swears by it | 18:48 |
| *** srwilkers has quit IRC | 18:48 | |
| *** jascott1 has quit IRC | 18:51 | |
| *** rwallner has joined #openstack-kolla | 18:51 | |
| *** mgoddard has quit IRC | 18:52 | |
| *** manheim_ has joined #openstack-kolla | 18:54 | |
| *** jascott1 has joined #openstack-kolla | 18:55 | |
| *** ducttape_ has quit IRC | 18:57 | |
| *** ducttap__ has joined #openstack-kolla | 18:58 | |
| *** zhubingbing_ has quit IRC | 19:02 | |
| *** zhubingbing_ has joined #openstack-kolla | 19:02 | |
| *** esmute has joined #openstack-kolla | 19:02 | |
| *** srwilkers has joined #openstack-kolla | 19:06 | |
| *** rwallner has quit IRC | 19:09 | |
| *** manheim_ has quit IRC | 19:10 | |
| *** ducttap__ has quit IRC | 19:13 | |
| *** ducttape_ has joined #openstack-kolla | 19:13 | |
| *** rhallisey has quit IRC | 19:13 | |
| *** srwilkers has quit IRC | 19:17 | |
| *** lrensing has quit IRC | 19:19 | |
| *** hieulq_ has quit IRC | 19:28 | |
| *** dave-mccowan has quit IRC | 19:30 | |
| *** mgoddard has joined #openstack-kolla | 19:32 | |
| *** esmute has quit IRC | 19:33 | |
| *** shasha_t_ has quit IRC | 19:39 | |
| *** hieulq_ has joined #openstack-kolla | 19:41 | |
| *** dave-mccowan has joined #openstack-kolla | 19:43 | |
| *** dave-mcc_ has joined #openstack-kolla | 19:45 | |
| *** harlowja has joined #openstack-kolla | 19:48 | |
| *** dave-mccowan has quit IRC | 19:48 | |
| *** salv-orlando has joined #openstack-kolla | 19:52 | |
| *** Pavo has joined #openstack-kolla | 19:53 | |
| *** hieulq_ has quit IRC | 19:55 | |
| *** rwallner has joined #openstack-kolla | 19:59 | |
| *** ipsecguy_ has joined #openstack-kolla | 20:05 | |
| *** ipsecguy has quit IRC | 20:07 | |
| *** Pavo has quit IRC | 20:09 | |
| *** rwallner has quit IRC | 20:13 | |
| *** Pavo has joined #openstack-kolla | 20:13 | |
| *** mgoddard has quit IRC | 20:14 | |
| *** georgem1 has quit IRC | 20:17 | |
| *** skramaja has joined #openstack-kolla | 20:25 | |
| *** dave-mcc_ has quit IRC | 20:34 | |
| *** shasha_t_ has joined #openstack-kolla | 20:35 | |
| *** rwallner has joined #openstack-kolla | 20:39 | |
| *** shasha___ has joined #openstack-kolla | 20:41 | |
| *** shasha_t_ has quit IRC | 20:41 | |
| *** rwallner has quit IRC | 20:44 | |
| *** Serlex has quit IRC | 20:46 | |
| *** athomas has quit IRC | 20:52 | |
| *** salv-orl_ has joined #openstack-kolla | 20:56 | |
| *** salv-orlando has quit IRC | 20:59 | |
| *** lucasxu has quit IRC | 21:01 | |
| *** jtriley has quit IRC | 21:06 | |
| *** eaguilar has joined #openstack-kolla | 21:06 | |
| *** bmace has quit IRC | 21:09 | |
| *** bmace has joined #openstack-kolla | 21:09 | |
| *** goldyfruit has quit IRC | 21:14 | |
| *** mgoddard has joined #openstack-kolla | 21:14 | |
| *** esmute_ has joined #openstack-kolla | 21:25 | |
| esmute_ | Hey guys, looks like i got kolla running on kubernetes. but my nova vm are in error. Nova shows 'Binding failed for port 03eadd66-f818-4453-a5d7-746630239bd5, please check neutron logs for more information.' | 21:26 |
| esmute_ | i also notice that the neutron-openvswitch pod is stuck in init. Not sure if that is the cause | 21:26 |
| esmute_ | neutron-openvswitch-agent-network-hzvf7 0/1 Init:2/3 0 39m | 21:26 |
| *** eaguilar has quit IRC | 21:26 | |
| *** skramaja has quit IRC | 21:31 | |
| *** Manheim has quit IRC | 21:33 | |
| sdake | esmute_ a few peopel have reported that although it odes't happen to me nor in the gate | 21:33 |
| sdake | esmute_ which guide did you followt o install? | 21:33 |
| esmute_ | I followed the baremetal guide | 21:33 |
| esmute_ | https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html | 21:34 |
| sdake | cool | 21:34 |
| sdake | that was working a few days ago - prior to the change in canal | 21:34 |
| sdake | since then - been a little problematic :( | 21:34 |
| esmute_ | i had an issue when i generated the config using sudo kolla-ansible genconfig... I was told that this didnt work and to use the ansible playbook | 21:34 |
| *** schwicht has quit IRC | 21:36 | |
| *** mgoddard has quit IRC | 21:40 | |
| sdake | esmute_ indeed that may be the problem | 21:42 |
| *** manheim has joined #openstack-kolla | 21:43 | |
| *** Pavo has quit IRC | 21:43 | |
| *** imcsk8 has quit IRC | 21:46 | |
| *** imcsk8 has joined #openstack-kolla | 21:46 | |
| *** Pavo has joined #openstack-kolla | 21:46 | |
| *** eaguilar has joined #openstack-kolla | 21:48 | |
| *** salv-orl_ has quit IRC | 21:51 | |
| *** esmute_ has quit IRC | 21:51 | |
| *** eaguilar has quit IRC | 21:52 | |
| *** gianpietro has quit IRC | 21:53 | |
| *** gianpietro has joined #openstack-kolla | 21:53 | |
| *** eaguilar has joined #openstack-kolla | 21:54 | |
| jascott1 | esmute_ thats because of using genconfig | 21:54 |
| *** esmute_ has joined #openstack-kolla | 21:54 | |
| esmute_ | sdake: anything i can do to figure out what is causing this problem? | 21:54 |
| jascott1 | when you describe neutron pods do you see volume mount error? | 21:55 |
| *** jbadiapa has quit IRC | 22:01 | |
| sbezverk | esmute_: kubectl describe pod -n kolla neutron-openvswitch-agent-network-hzvf7 | 22:02 |
| inc0 | hmm | 22:04 |
| *** schwicht has joined #openstack-kolla | 22:05 | |
| esmute_ | sbezverk: https://gist.github.com/kokhang/581f9daa3d4ea115da394fc3637bcb66 | 22:06 |
| esmute_ | thanks for the help | 22:06 |
| esmute_ | no mounting error that i can see | 22:07 |
| esmute_ | i deployed kolla multiple times and the neutron-openvswitch pod is always stuck in init | 22:07 |
| sbezverk | esmute_: it stuck on init container, it is waiting for vswitchd to be up and running. on the host where this pod is running and run this commands: | 22:10 |
| sbezverk | lsmod | grep openvswitch | 22:10 |
| sbezverk | ovs-dpctl show | 22:11 |
| sbezverk | ovs-appctl version | 22:11 |
| sbezverk | if any of these do not produce expected result, then this pod will be in stuck mode forever | 22:12 |
| sbezverk | but the issue is not with it, but with openvswitch | 22:12 |
| esmute_ | sbezverk: https://gist.github.com/kokhang/6628ffc3ac9c5da0c3bde327f9c5ad71 | 22:12 |
| esmute_ | it is coreos. So it doesnt have neither ovs-dpctl nor ovs-appctl | 22:13 |
| sbezverk | esmute_: here is your answre :) | 22:13 |
| sbezverk | we run run these commands and EXPECT them to be available | 22:13 |
| sbezverk | one sec, you need to run them from that container | 22:14 |
| sbezverk | they are packaged with the container | 22:14 |
| sbezverk | docker exec -tu root blah ovs-dpctl show | 22:14 |
| sbezverk | docker exec -tu root blah ovs-appctl version | 22:15 |
| *** jbadiapa has joined #openstack-kolla | 22:15 | |
| *** rwallner has joined #openstack-kolla | 22:16 | |
| esmute_ | sbezverk: https://gist.github.com/kokhang/6628ffc3ac9c5da0c3bde327f9c5ad71 | 22:17 |
| sbezverk | esmute_: looks good so far | 22:19 |
| sbezverk | esmute_: docker logs blah | 22:19 |
| esmute_ | + modprobe openvswitch + /usr/sbin/ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid | 22:20 |
| *** rwallner has quit IRC | 22:21 | |
| esmute_ | sbezverk: https://gist.github.com/kokhang/f492cf9068991dd817fd8193352b1d67 | 22:22 |
| esmute_ | not sure if that is useful | 22:22 |
| sbezverk | esmute_: no, I need logs from neutron-openvswitch-agent-network-hzvf7 container | 22:23 |
| esmute_ | how do you get that if the container is in init? | 22:24 |
| sbezverk | docker logs $(docker ps -a | grep neutron-openvswitch-agent-network-hzvf7@ | awk '{print $1}') | 22:24 |
| sbezverk | try to find docker container id for this ID 4317b6f7f8787f6e3f1e6bd8925b4851f167b390492c520a5049dcbd92f6bc7e | 22:25 |
| *** goldyfruit has joined #openstack-kolla | 22:27 | |
| esmute_ | the log shows a bunch of: | 22:27 |
| esmute_ | 2017-05-04T21:56:43Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: stale pidfile for pid 8 being deleted by pid 0 ovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such process) Trying again. | 22:27 |
| *** vhosakot has quit IRC | 22:30 | |
| sbezverk | esmute_: when have you updated kolla-kubernetes? check on the host if that file really exist. | 22:32 |
| esmute_ | yes it exists on the host | 22:33 |
| esmute_ | pid 8 | 22:33 |
| *** Pavo has quit IRC | 22:34 | |
| esmute_ | kolla-kubernetes is at cf085cce0f14ed968d435959ad7ebae6e9542daa Merge "Add local registry option in deployment-guide" | 22:34 |
| sbezverk | esmute_: ah ok you are still running kubernetes 1.6.1 and you forgot to add one parameters which is required | 22:35 |
| sbezverk | esmute_: you need to add "--enable-cri=false" | 22:37 |
| sbezverk | to your kubelet service command line | 22:37 |
| esmute_ | yes im running 1.6.1 | 22:38 |
| esmute_ | ok let me try that | 22:38 |
| sbezverk | yeah, there is a bug which gets workedaround by using that keywork I gave you above | 22:38 |
| *** schwicht has quit IRC | 22:38 | |
| sbezverk | esmute_: I am positive it will work :) I need to step out, will be back later | 22:42 |
| *** eaguilar has quit IRC | 22:42 | |
| esmute_ | sbezverk: thank you for your help. | 22:43 |
| *** manheim has quit IRC | 22:51 | |
| *** salv-orlando has joined #openstack-kolla | 22:51 | |
| *** egonzalez has quit IRC | 22:53 | |
| *** salv-orlando has quit IRC | 22:58 | |
| *** goldyfruit has quit IRC | 23:04 | |
| *** kolla-slack3 has quit IRC | 23:06 | |
| *** kolla-slack has joined #openstack-kolla | 23:06 | |
| *** esmute_ has quit IRC | 23:13 | |
| *** esmute has joined #openstack-kolla | 23:13 | |
| *** esmute has quit IRC | 23:15 | |
| *** esmute has joined #openstack-kolla | 23:15 | |
| *** erlon has quit IRC | 23:24 | |
| *** rwellum has quit IRC | 23:38 | |
| *** guoshan has joined #openstack-kolla | 23:40 | |
| *** ducttape_ has quit IRC | 23:44 | |
| *** eanylin__ has quit IRC | 23:45 | |
| *** Pavo has joined #openstack-kolla | 23:48 | |
| *** salv-orlando has joined #openstack-kolla | 23:54 | |
| *** salv-orlando has quit IRC | 23:58 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!