Thursday, 2017-05-04

*** yangyape_ has quit IRC00:00
sbezverkesmute: how do you deploy mariadb? what command you use?00:01
inc0sbezverk: cool you're here, what's best way to access /var/log/?:)00:02
inc0do we mount volume to /var/log somewhere?00:02
sbezverkinc0: on crashing pod it is a bit tricky, timing is everything00:03
sbezverkyou do docker inspect on crashing log00:03
*** eanylin has quit IRC00:03
sbezverkcatch where kubernetes mount /var/log/kolla00:03
sbezverkand then access logs right on the kube host00:03
*** eaguilar has quit IRC00:03
inc0ah yeah, k8s won't let you just stop container and examine it right?00:04
*** ssurana has joined #openstack-kolla00:04
sbezverk docker inspect f9cd5dce622b | grep /var/log/kolla00:04
sbezverk                "/var/lib/kubelet/pods/1aa516f6-2932-11e7-a732-525400a3621f/volumes/kubernetes.io~empty-dir/kolla-logs:/var/log/kolla/:Z"00:04
esmutesbezverk: im just following the deployment-guide doc. I used helm install --debug kolla-kubernetes/helm/service/mariadb --namespace kolla --name mariadb --values ./cloud.yaml00:05
sbezverkinc0: right00:05
sbezverkesmute: kubectl get pv00:05
Jeffrey4lpbourke, re keyfile. checked in my env. it is not generated.00:05
sbezverkesmute: kubectl get pvc -n kolla00:05
esmuteinc0: im not using ceph nor any storageclass.. Should i?00:05
sbezverkesmute: it should be created for you00:05
esmute$ kubectl get pvc -n kolla NAME      STATUS    VOLUME    CAPACITY   ACCESSMODES   STORAGECLASS   AGE mariadb   Bound     mariadb   10Gi       RWO                          1h00:05
esmuteit is bounded00:05
sbezverkas long as you have correct settings for storage00:06
sbezverkesmute: kubectl get job -n kolla00:06
sbezverkesmute: before mariadb can start you needs to be bootstraped00:07
esmute$ kubectl -n kolla get job NAME                   DESIRED   SUCCESSFUL   AGE mariadb-init-element   1         1            1h00:07
*** yingjun has joined #openstack-kolla00:08
sbezverkesmute: hm, it looks like bootstrap went well. check /var/lib/kolla/volumes00:08
esmutesbezverk: https://gist.github.com/kokhang/cf6ccde75374947d5a878b05faa20d7f00:10
esmutethe volume seems to have been mounted and data written to it00:10
sbezverkesmute: right. oh well easy way did not help, we need to access the logs to try to figure out00:12
*** imcsk8 has quit IRC00:13
jascott1_anyone know where this gets created? /var/lib/kolla/config_files00:14
sbezverkesmute: docker ispect $(docker ps -a | grep mariadb@ | awk '{print $1}') | grep /var/log/kolla00:15
sbezverkthen quickly cat mariadb log from that folder00:16
*** georgem1 has joined #openstack-kolla00:16
*** georgem1 has quit IRC00:17
*** georgem1 has joined #openstack-kolla00:17
jascott1_my neutron containers are looking for that path but its not there00:18
esmutesbezverk: cool.. here is what i have00:19
esmute170504  0:16:37 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.34-79.1 started; log sequence number 1600965 170504  0:16:37 [Note] Plugin 'FEEDBACK' is disabled. 170504  0:16:37 [ERROR] Can't open the mysql.plugin table. Please run mysql_upgrade to create it. 170504  0:16:37 [Warning] Failed to load slave replication state from table mysql.gtid_slave_pos: 1146: Table 'mysql.gtid_slave_pos' doesn't exist 17050400:19
esmutehttps://gist.github.com/kokhang/9e6965061934f6476ee6864139c670cb00:19
sbezverkesmute: it looks like init has not been completed properly00:24
sbezverkI suggest helm delete mariadb --purge00:24
sbezverkthen rm -rf /var/lib/kolla/volumes/*00:24
*** reddydodda has joined #openstack-kolla00:24
sbezverkand rerun mariadb deploy00:24
*** manheim has joined #openstack-kolla00:25
jascott1_sbezverk: do you know where /var/lib/kolla/config_files get created?00:25
jascott1_mine is missing00:25
sbezverkjascott1_: a little bit more of a context please? these are files from the configmap00:26
sbezverkif they are missing then probably configmap is not correct00:26
esmutesbezverk: ok this time it is running and not crashing00:26
esmutedo you have an idea why the init failed?00:27
jascott1_installing from guide, everything comes up but neutron containers. they cant mount config and thats the path volume references00:27
sbezverkesmute: nice!! man, I am not oracle ;) need logs00:27
sbezverkjascott1_: you are second one who hit this issue00:27
jascott1_yay!00:28
jascott1_that means there is easy answer right? ;)00:28
sbezverkjascott1_: DO NOT USE kolla-ansible to generate configs :)00:28
jascott1_ah00:28
sbezverkjascott1_: they removed ml2_conf.ini00:28
sbezverkfrom neutron configuration00:28
sbezverkso kube is not happy as it is missing a key for ml200:29
jascott1_hmm00:29
sbezverkjascott1_: inc0 created a playbook to generate configuration independently00:29
esmutesbezverk: i used `sudo kolla-ansible genconfig` following the guide00:29
esmutewould i run into issues?00:29
*** manheim has quit IRC00:29
sbezverkesmute: you will be third one then ;)00:29
jascott1_haha00:29
jascott1_playbook is merged?00:30
sbezverkjascott1_: Yep, you can get command to use from tests/bin/ iscsi workflow00:30
sbezverklook for ansible00:30
jascott1_sbezverk thanks!00:30
sbezverkcommand, you can use that command as is without any modifications00:31
sbezverkjascott1_: any time ;)00:31
*** zhurong has joined #openstack-kolla00:31
*** eaguilar has joined #openstack-kolla00:32
jascott1_esmute: the command is00:35
jascott1_ansible-playbook -e ansible_python_interpreter=/usr/bin/python -e @/etc/kolla/globals.yml -e @/etc/kolla/passwords.yml -e CONFIG_DIR=/etc/kolla ansible/site.yml00:35
jascott1_from kolla-kubernetes dir00:35
sbezverkjascott1_: yep, that is the one00:35
jascott1_works for me (TM)00:36
sbezverkjascott1_: somebody need to update DG asap00:36
jascott1_yeah also canal.yml url has changed00:36
jascott1_i was planning on making PS00:36
esmutejascott1_: thank you00:36
jascott1_also they have added an RBAC.yml with flannel stuff in it00:37
esmutedo i have to delete all helm deployments ?00:37
sbezverkjascott1_: check setup_canal.sh script00:37
sbezverkit is workaround we use for the gate00:37
jascott1_the way is the gate, the gate is the way ;)00:37
sbezverkjascott1_: why not ;)??00:38
jascott1_no doubt00:38
jascott1_yeah just now getting back to playbook, its funny how much things break in a week or so00:38
sbezverkjascott1_: always something happens, if not kubernetes screw up, then canal, if not that then something else. fun all the time ;)00:39
jascott1_job security00:40
*** eanylin has joined #openstack-kolla00:43
*** tovin07_ has joined #openstack-kolla00:46
*** cuongnv has joined #openstack-kolla00:46
*** reddydodda has quit IRC01:02
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Add timeout parameter for precheck tasks  https://review.openstack.org/46233201:02
*** dave-mccowan has joined #openstack-kolla01:03
*** Pavo has quit IRC01:03
*** gianpietro has joined #openstack-kolla01:08
*** yangyapepeng has joined #openstack-kolla01:11
*** reddydodda has joined #openstack-kolla01:11
*** yangyapepeng has quit IRC01:11
*** david-lyle has joined #openstack-kolla01:16
*** zhurong has quit IRC01:16
*** goldyfruit has joined #openstack-kolla01:20
*** yangyapeng has joined #openstack-kolla01:21
*** srwilkers_ has quit IRC01:25
*** srwilkers has quit IRC01:26
*** zhurong has joined #openstack-kolla01:26
*** hrw has quit IRC01:31
*** hrw has joined #openstack-kolla01:33
*** dixiaoli has joined #openstack-kolla01:33
*** david-lyle has quit IRC01:34
*** esmute has quit IRC01:37
*** ducttape_ has joined #openstack-kolla01:51
*** masber has quit IRC01:56
*** unicell has quit IRC02:01
*** jtriley has joined #openstack-kolla02:01
*** salv-orlando has joined #openstack-kolla02:05
*** salv-orlando has quit IRC02:09
*** dave-mccowan has quit IRC02:10
*** daidv has joined #openstack-kolla02:11
*** caowei has joined #openstack-kolla02:18
*** dave-mccowan has joined #openstack-kolla02:20
sdakejascott1_ you assusme cisco pays him for working on kolla-kubenretes :)02:22
sdakejascott1_ bad assumption imo :)02:22
*** zhurong has quit IRC02:34
*** eaguilar has quit IRC02:34
*** ducttape_ has quit IRC02:36
*** zhurong has joined #openstack-kolla02:37
*** caowei has quit IRC02:39
*** ssurana has quit IRC02:40
*** ducttape_ has joined #openstack-kolla02:45
*** caowei has joined #openstack-kolla02:45
*** dave-mccowan has quit IRC02:51
*** masber has joined #openstack-kolla02:52
*** david-lyle has joined #openstack-kolla02:53
*** zhubingbing_ has joined #openstack-kolla02:55
openstackgerritMerged openstack/kolla-ansible stable/ocata: Add ssh-keyscan to unblock kolla-ansible gate  https://review.openstack.org/46205202:58
*** lucasxu has joined #openstack-kolla02:58
*** lucasxu has quit IRC03:00
*** britthouser10 has joined #openstack-kolla03:01
*** shash____ has joined #openstack-kolla03:02
*** unicell has joined #openstack-kolla03:04
*** sbezverk_ has joined #openstack-kolla03:04
*** gianpietro has quit IRC03:05
*** gianpietro has joined #openstack-kolla03:06
*** shasha_t- has joined #openstack-kolla03:06
*** salv-orlando has joined #openstack-kolla03:06
*** slagle__ has joined #openstack-kolla03:07
openstackgerritMerged openstack/kolla-ansible master: Fix a typo and complement doc in ceph-guide.rst  https://review.openstack.org/46043103:09
*** kbyrne_ has joined #openstack-kolla03:10
*** salv-orlando has quit IRC03:11
*** slagle has quit IRC03:11
*** pron_ has quit IRC03:11
*** harbie has quit IRC03:11
*** shasha___ has quit IRC03:11
*** britthouser has quit IRC03:11
*** sbezverk has quit IRC03:11
*** portdirect has quit IRC03:11
*** kbyrne has quit IRC03:11
*** kbyrne_ is now known as kbyrne03:11
*** harbie has joined #openstack-kolla03:12
*** tovin07_ has quit IRC03:16
*** tovin07_ has joined #openstack-kolla03:17
*** pron_ has joined #openstack-kolla03:18
*** masber has quit IRC03:24
*** goldyfruit has quit IRC03:26
*** dixiaoli has quit IRC03:30
*** ducttape_ has quit IRC03:33
*** georgem1 has quit IRC03:39
*** reddydodda has quit IRC03:54
*** salv-orlando has joined #openstack-kolla04:07
*** shash____ has quit IRC04:08
*** shasha_t_ has joined #openstack-kolla04:09
*** masber has joined #openstack-kolla04:11
*** salv-orlando has quit IRC04:11
*** zhurong has quit IRC04:12
*** ssurana has joined #openstack-kolla04:12
openstackgerritMerged openstack/kolla stable/ocata: Add missing cleanup step for new apache images  https://review.openstack.org/46208804:18
*** xinliang has quit IRC04:20
*** gianpietro has quit IRC04:22
*** gianpietro has joined #openstack-kolla04:28
*** xinliang has joined #openstack-kolla04:33
*** zhurong has joined #openstack-kolla04:35
spsuryasup all04:39
*** salv-orlando has joined #openstack-kolla04:52
*** jtriley has quit IRC04:54
*** caowei has quit IRC05:02
*** unicell1 has joined #openstack-kolla05:02
*** unicell has quit IRC05:05
*** portdirect has joined #openstack-kolla05:13
*** jascott1_ has quit IRC05:15
*** gkadam has quit IRC05:31
*** Serlex has joined #openstack-kolla05:33
*** gkadam has joined #openstack-kolla05:34
*** japestinho has quit IRC05:40
*** janki has joined #openstack-kolla05:40
*** shasha_t_ has quit IRC05:44
*** gkadam has quit IRC06:00
*** caowei has joined #openstack-kolla06:00
*** gkadam has joined #openstack-kolla06:01
*** salv-orlando has quit IRC06:04
*** dciabrin has joined #openstack-kolla06:06
*** esmute1 has joined #openstack-kolla06:08
*** jascott1 has joined #openstack-kolla06:15
*** jascott1 has quit IRC06:20
*** sbezverk_ has quit IRC06:23
openstackgerritMerged openstack/kolla-ansible master: Fix saraha-engine authencation issue  https://review.openstack.org/46181406:38
openstackgerritMerged openstack/kolla-ansible master: mount /run in sahara-engine container  https://review.openstack.org/46178406:40
*** rmart04_ has joined #openstack-kolla06:42
*** pcaruana has joined #openstack-kolla06:45
*** matrohon has joined #openstack-kolla06:51
*** shardy has joined #openstack-kolla07:08
openstackgerritjunbo proposed openstack/kolla-ansible stable/ocata: Fix saraha-engine authencation issue  https://review.openstack.org/46241107:12
openstackgerritjunbo proposed openstack/kolla-ansible stable/ocata: mount /run in sahara-engine container  https://review.openstack.org/46241207:13
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Remove "project" empty variable from group_vars  https://review.openstack.org/46241307:16
*** gianpietro has quit IRC07:17
*** manheim has joined #openstack-kolla07:19
*** salv-orlando has joined #openstack-kolla07:24
hrwmorning07:32
*** nathharp has joined #openstack-kolla07:35
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094007:36
*** egonzalez has joined #openstack-kolla07:43
*** shasha_t_ has joined #openstack-kolla07:44
*** ssurana has quit IRC07:48
*** shasha_t_ has quit IRC07:49
*** rmart04_ has quit IRC07:49
*** lpetrut has joined #openstack-kolla07:51
*** esmute1 has quit IRC08:00
*** blallau has joined #openstack-kolla08:08
*** jaosorior_away is now known as jaosorior08:14
*** rmart04 has joined #openstack-kolla08:15
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Add dots at first line in docstring  https://review.openstack.org/46243308:26
apuimedoegonzalez: do you use kolla-kubernetes?08:27
egonzalezapuimedo, nope for now08:28
apuimedopity, I wanted to ask you about a couple of things off https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html08:28
apuimedoI finally managed to get kubernetes installed from scratch and helm on the archlinux server08:28
apuimedo:-)08:29
apuimedoso next step is kolla08:29
*** esmute1 has joined #openstack-kolla08:31
*** ducttape_ has joined #openstack-kolla08:34
openstackgerritEduardo Gonzalez proposed openstack/kolla stable/ocata: Fix cinder-api ubuntu binary  https://review.openstack.org/45815208:35
*** athomas has joined #openstack-kolla08:38
*** ducttape_ has quit IRC08:39
pesterinstalling kolla 4.0.0, with pip install and default `all-in-one` inventory - at 'RUNNING HANDLER [neutron : Restart neutron-openvswitch-agent container] ********' - lost all connectivity to node08:40
*** mgoddard has joined #openstack-kolla08:41
pestercan`t even imagine how it is possible :D route looks ok, iptables also...08:43
pesterfunny part, I can see bridges in host system but no brctl installed to inspect them - and network is down so I can`t install it))08:46
egonzalezpester, whats the content of globals.yml and output of ip a?08:50
egonzalezapuimedo, most k8s folks are in US timezones, for very k8s specific questions I suggest you to wait for them ;)08:52
pestercan`t see any errors, kolla-ansible was run in putty and it just disconnected.08:52
apuimedoegonzalez: I will. Thanks08:54
pesteras I can see my external interface was added to br-ex, but address is left on inteface, but it should migrate to br-ex... I did it by hand and ot connectivity back to work08:54
apuimedoegonzalez: how long should "kolla-ansible genconfig" last08:54
pestermaybe it is Xenial new behavior08:54
apuimedoit is taking alreadya a few minutes...08:55
openstackgerritjunbo proposed openstack/kolla stable/ocata: Add netcat package in sahara-engine image  https://review.openstack.org/46244408:55
*** salv-orl_ has joined #openstack-kolla08:56
egonzalezapuimedo, gets stuck in some task? or is generating a lot of config files?08:57
pesterwill try to reset my VM and run this in screen... may be kolla will set it right by itself =)08:58
apuimedoegonzalez: http://paste.openstack.org/show/608819/08:59
egonzalezpester, i mean globals.yml is config file with kolla settings and "ip a" to match if there is some error between setting and network interfaces08:59
openstackgerritMerged openstack/kolla stable/ocata: Add libosinfo package for source installation  https://review.openstack.org/46194008:59
openstackgerritMerged openstack/kolla master: Update oraclelinux tag to new slim base image  https://review.openstack.org/43732908:59
*** salv-orlando has quit IRC08:59
pesteregonzalez: can`t actually copy-paste cause no network, doing reset now09:00
hrw;D09:00
hrwI just rebased patch and it need rebasing ;D09:01
egonzalezapuimedo, hrm maybe ssh issue, can you run ansible -m setup -i <all-in-one-inventory> localhost?09:02
apuimedoegonzalez: http://paste.openstack.org/show/608820/09:02
*** yangyapeng has quit IRC09:03
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094009:03
* hrw off09:03
*** zhurong has quit IRC09:06
esmute1Hi guys, i got a question, anyway to find out why my neutron-openvswitch is stuck in init?09:06
esmute1neutron-openvswitch-agent-network-9408j   0/1       Init:2/3   009:06
esmute1i didnt see anything obvious when i did `kubectl describe pod neutron-openvswitch-agent-network-9408j`09:07
zhubingbing_hi guys09:12
*** pbourke has quit IRC09:13
*** pbourke has joined #openstack-kolla09:15
*** zhurong has joined #openstack-kolla09:15
*** japestinho has joined #openstack-kolla09:16
openstackgerritshaofeng cheng proposed openstack/kolla-ansible master: Update docker_py_version_min version  https://review.openstack.org/45456109:17
*** tovin07 has quit IRC09:18
*** jascott1 has joined #openstack-kolla09:18
*** esmute1 has quit IRC09:19
pesterok I set neutron_external_interface to my real public interface and kolla just turned it off...09:20
*** jascott1 has quit IRC09:23
pesteregonzalez: so I need to set 2x interfaces, network_interface: - my real public interface (where VIP will be allocated) and  neutron_external_interface: - can be dummy interface for VMs outgoing connections ?09:24
pesterit is correct ?09:24
egonzalezsup zhubingbing_09:24
zhubingbing_hi09:24
egonzalezpester, yep, neutron_external_interface cannot be network_interface since it will be managed by ovs and lost connection once br-ex is created09:25
*** zhouya has joined #openstack-kolla09:25
egonzalezyou can use a dummy interfaces or whatever virt interface ansible recognizes09:26
zhouyaI have met some problem when I deploy the kolla newton release,wan't to ask kolla team for help.09:27
zhouyaWhen the kolla-ansible deployed to ceph09:27
*** athomas has quit IRC09:27
zhouyahttps://thepasteb.in/p/Anhr4zv9pV0cv09:28
zhouyafailed: [192.168.71.6] (item=(0, {u'fs_uuid': u'', u'journal_device': u'/dev/sdb', u'journal': u'/dev/sdb2', u'partition': u'/dev/sdb1', u'partition_num': u'1', u'journal_num': 2, u'fs_label': u'', u'device': u'/dev/sdb', u'external_journal': False})) => {"changed": true, "failed": true, "item": [0, {"device": "/dev/sdb", "external_journal": false,09:28
zhouya "fs_label": "", "fs_uuid": "", "journal": "/dev/sdb2", "journal_device": "/dev/sdb", "journal_num": 2, "partition": "/dev/sdb1", "partition_num": "1"}], "msg": "Container exited with non-zero return code"}09:28
zhouyaTASK [ceph : Bootstrapping Ceph OSDs] ******************************************09:28
zhouyaThe docker I see in my target node is :09:29
zhouya3141141eb5c6        10.20.7.3:4000/kolla/centos-binary-ceph-mon:3.0.3        "kolla_start"       25 minutes ago      Up 25 minutes                           ceph_mon09:29
zhouya1f7a0ff1ad60        10.20.7.3:4000/kolla/centos-binary-cron:3.0.3            "kolla_start"       26 minutes ago      Up 25 minutes                           cron09:29
zhouya8ade2db9ceda        10.20.7.3:4000/kolla/centos-binary-kolla-toolbox:3.0.3   "kolla_start"       26 minutes ago      Up 26 minutes                           kolla_toolbox09:29
zhouya830071e08577        10.20.7.3:4000/kolla/centos-binary-heka:3.0.3            "kolla_start"       26 minutes ago      Up 26 minutes                           heka09:29
*** rmart04 has quit IRC09:29
zhouyaI don't know how to deal with this error,could someone help me to take a look as this?09:30
*** yingjun has quit IRC09:31
zhouyaMy email is zhou.ya@zte.com.cn.I am glad if anyone could help me,thank you very much.09:32
egonzalezzhouya, is there any failed container in target nodes? docker ps -a ?09:36
*** cuongnv_ has joined #openstack-kolla09:39
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373809:39
pbourkeegonzalez: would 'outward' suit better than 'external' for our new rabbitmq instance?09:39
*** zhouya has left #openstack-kolla09:39
*** zhuzeyu has joined #openstack-kolla09:40
*** cuongnv has quit IRC09:40
*** _zhouya has joined #openstack-kolla09:41
_zhouyayes09:41
*** lpetrut_ has joined #openstack-kolla09:41
_zhouyathe container:10.20.7.3:4000/kolla/centos-binary-ceph-osd:3.0.309:41
_zhouyafailed09:41
*** lpetrut has quit IRC09:41
_zhouyaIs this affected by ceph-mon?09:42
*** lpetrut_ has quit IRC09:42
*** lpetrut_ has joined #openstack-kolla09:42
egonzalez_zhouya, check docker logs <failed_osd_container>09:43
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Fix permission on chrony.conf  https://review.openstack.org/46246209:44
*** zhouya has joined #openstack-kolla09:45
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Fix logging for chrony  https://review.openstack.org/46246309:46
*** zhouya has quit IRC09:47
bjolomorning09:47
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla master: Fix logging for chrony  https://review.openstack.org/46246309:47
egonzalezpbourke, i had to look at google translator for that word hehe, but yes wfm09:48
pbourkeegonzalez: :)09:48
pbourkemaybe a little more obscure, but I agree we already use the term 'external' for non kolla deployed services09:48
*** zhouya has joined #openstack-kolla09:48
egonzalezbtw, toolbox re-parent is not needed anymore?09:49
bjolomorning09:49
egonzalezsup bjolo09:49
bjolofrustration09:49
bjoloare there any knowns issues with kolla stable/ocata and docker 1.12.6 (i know i just dig through docs and bugs but... )09:50
bjoloi can do a deploy09:51
bjolobut if i then do a destroy and deploy again, docker barfs09:51
*** zhouya has quit IRC09:51
bjoloi then have to remove all images, clean out docker cache and btrfs subvolumes manually and restart docker before i can deploy again09:54
egonzalezbjolo, i have 1.12.6 on  my dev env and deploy/destroy few times per day and still working09:54
bjolohmmm09:54
bjolowtf09:54
*** zhouya has joined #openstack-kolla09:54
egonzalezhrm, i dont use brtfs09:54
bjolowhat kernel?09:54
egonzalez3.14.32-xxxx-grs-ipv6-6409:55
bjoloi use the 4.11.0-1 from elrepo09:55
bjololatest kernel-ml package09:55
bjolocentos 7 of course as host os09:55
*** zhouya has quit IRC09:55
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373809:56
egonzalezbjolo, sorry, this is the kernel in compute nodes 3.10.0-514.16.1.el7.x86_6409:56
egonzalezother is from deployer09:56
*** magicboiz has joined #openstack-kolla09:57
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373809:58
bjolobut not using btrfs?09:58
egonzaleznope10:00
*** eaguilar has joined #openstack-kolla10:00
*** zhouya has joined #openstack-kolla10:01
*** magicboiz has quit IRC10:01
*** zhouya has quit IRC10:02
*** tovin07_ has quit IRC10:04
*** zhouya has joined #openstack-kolla10:04
*** eaguilar has quit IRC10:05
*** zhouya has quit IRC10:06
*** zhouya has joined #openstack-kolla10:06
egonzalezbjolo, have you tried with other kernel / docker versions/ storage driver?10:06
bjolonot yet10:07
*** zhurong has quit IRC10:07
*** zhouya has quit IRC10:08
bjolowe have had a ton of infra issues, so we just did the first kolla eploy this morning10:08
bjolosome more TS and googling before trying things out :)10:09
*** magicboiz has joined #openstack-kolla10:14
*** guoshan has joined #openstack-kolla10:20
*** daidv has quit IRC10:21
*** cuongnv has joined #openstack-kolla10:22
*** cuongnv has quit IRC10:23
*** athomas has joined #openstack-kolla10:23
*** cuongnv_ has quit IRC10:24
*** zhurong has joined #openstack-kolla10:24
*** ever has joined #openstack-kolla10:24
*** guoshan has quit IRC10:26
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373810:29
*** zhuzeyu has quit IRC10:30
*** _zhouya has quit IRC10:30
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano  https://review.openstack.org/45420610:30
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'external' rabbitmq instance  https://review.openstack.org/45373810:30
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance  https://review.openstack.org/45373810:32
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano  https://review.openstack.org/45420610:32
*** apuimedo has quit IRC10:33
*** gfidente has joined #openstack-kolla10:39
*** gfidente has quit IRC10:39
*** gfidente has joined #openstack-kolla10:39
hrwegonzalez: can you +2 https://review.openstack.org/430940 patch?10:44
manheimHi, post deployment, can i run kolla-ansible to only restart a certaint existing service? like cinder10:53
*** zhouya has joined #openstack-kolla10:54
zhouyaegonzalez10:54
zhouyasorry,the network is not stable10:54
zhouyaseems the keyring error?10:55
*** salv-orl_ has quit IRC10:55
egonzalezmanheim, kolla-ansible only restart the container if image changed or config files changed10:55
egonzalezmanheim, if something changed, use kolla-ansible deploy --tags cinder to only apply common and cinder role10:56
manheimhmm, i need this for nova-hyperv stuff10:56
manheimto control the service via ansible10:56
egonzalezmanheim, do register on config files tasks and notify handlers to restart the service (similar to what is doing with containers but with services)10:57
egonzalezzhouya, what error is giving?10:58
manheimi already do that, but only on deploy10:58
zhouya[root@host-192-168-71-5 ~]# docker logs 263244459b4010:58
zhouyaINFO:__main__:Kolla config strategy set to: COPY_ALWAYS10:58
zhouyaINFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json10:58
zhouyaINFO:__main__:Validating config file10:58
zhouyaINFO:__main__:Copying service configuration files10:58
zhouyaINFO:__main__:Copying /var/lib/kolla/config_files/ceph.conf to /etc/ceph/ceph.conf10:58
zhouyaINFO:__main__:Setting permissions for /etc/ceph/ceph.conf10:58
zhouyaINFO:__main__:Copying /var/lib/kolla/config_files/ceph.client.admin.keyring to /etc/ceph/ceph.client.admin.keyring10:58
zhouyaINFO:__main__:Setting permissions for /etc/ceph/ceph.client.admin.keyring10:58
zhouyaINFO:__main__:Writing out command to execute10:58
zhouya2017-04-28 16:32:51.854980 7f6e3795b700  0 monclient(hunting): authenticate timed out after 30010:58
zhouya2017-04-28 16:32:51.855032 7f6e3795b700  0 librados: client.admin authentication error (110) Connection timed out10:58
zhouyaError connecting to cluster: TimedOut10:58
zhouya610:58
zhouyaegonzalez11:02
zhouyahave you met this problem?11:02
egonzalezzhouya, is monitor running fine? do docker exec ceph_mon ceph -s11:03
egonzalezmanheim, to restart a service with no other change, we dont actually have nothing similar11:04
manheimi see11:05
manheimcoz to do that you simply restart the container11:05
manheimi guessi can add an aditional playbook to do that on hyperv11:05
egonzalezyes, but only if restart is needed(changes) if nothing changes cannot see a reason to restart a service11:06
egonzalezcontainer task ensure the container is running, if is down starts if is up do nothing11:06
manheimi see11:07
manheimone other question11:07
manheimso I have the ml2.conf file for both neutron-server and neutron-ovs-agent11:07
manheimif i want to make a change in ml2, i have to edit both files in the locations, right?11:07
manheimlike /etc/kolla/neutron-server and /etc/kolla/neutron-ovs...11:08
manheimasking if there is a simpler solution11:08
*** dave-mccowan has joined #openstack-kolla11:08
zhouyathe env is redeployed,if I met this situation time,Iwill contact you,thankyou for your help egonzalez11:08
zhouyaif I met this situation next time,sorry11:09
egonzalezmanheim, with custom config you can modify only one, both, or all ml2 files.11:09
egonzalezmanheim, etc/kolla/config/ml2.conf11:10
*** rwellum has joined #openstack-kolla11:10
egonzalez etc/kolla/config/<service_name>/ml2.conf11:10
manheimetc/kolla/config/ml2.conf is only pre-deployment, right?11:10
egonzalez etc/kolla/config/neutron/<inventory_hostname>/ml2.conf11:11
egonzalezhttps://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/config.yml#L11711:11
egonzalezmanheim, can be made after deployment with a reconfigure or deploy again11:12
*** zhouya has quit IRC11:12
manheimwhat does the reconfigure do exactly? i never fully understood11:12
egonzalezmanheim, from ocata reconfigure do the same as deploy. changes config files, start containers if needed, etc11:14
*** ever has quit IRC11:14
manheimlike a factory reset11:15
*** ever has joined #openstack-kolla11:15
egonzaleznot really, lets say you want to change debug from false to true, you run reconfigure to apply that change into config files and restart containers to refresh service with new config11:17
*** jascott1 has joined #openstack-kolla11:20
manheimah, so in this case you set debug=true, in let's say /etc/kolla/config/nova.conf and run reconfigure. this sets nova.conf for all services using it?11:21
egonzalezyep11:22
manheimah, I see, thanks11:24
*** jascott1 has quit IRC11:25
egonzalezhrw, added comments on arch change11:28
hrwegonzalez: ok11:28
hrwthx11:29
egonzalezbase image is getting quite complex and hard to follow, especially repo config stuff11:30
*** rwallner has joined #openstack-kolla11:31
hrwegonzalez: yes, it is11:31
hrwegonzalez: especially with that delorean stuff moved to code11:31
pesterfinally deployed kolla 4.0.0, with KVM, but forgot "expose virtualization bit", I get error from nova - "Error: invalid argument: could not find capabilities for domaintype=kvm"11:32
egonzalezpester, mkdir -p /etc/kolla/config/nova11:32
egonzalezcat << EOF > /etc/kolla/config/nova/nova-compute.conf11:32
egonzalez[libvirt]11:32
egonzalezvirt_type = qemu11:32
egonzalezcpu_mode = none11:32
egonzalezEOF11:32
pesterso, now I got it, how to tell kolla to use it ?11:32
egonzalezthen, kolla-ansible deploy --tags nova11:33
*** sbezverk has joined #openstack-kolla11:40
*** shardy is now known as shardy_lunch11:43
*** srwilkers has joined #openstack-kolla11:50
*** athomas has quit IRC11:52
*** salv-orlando has joined #openstack-kolla11:54
*** rwallner has quit IRC11:55
hrwegonzalez: will https://pastebin.com/qLrHGqep be enough? for delorean todo11:58
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute  https://review.openstack.org/44952911:59
egonzalezhrw, wfm11:59
pesterfinally deployed kolla with kvm, but when I boot image it stuck at start with "GRUB _" on VNC12:03
hrwegonzalez: thx for hint about whitespace stuff12:03
*** srwilkers has quit IRC12:03
hrwegonzalez: kolla is the only project I played with jinja212:03
egonzalezhrw, will do with ansible ;)12:04
hrw;d12:04
egonzalezpester, that issue is likely an image issue, are you using cirros image or other?12:04
pestercirros, tried 2 different versions12:04
pesterdownloading trusty now)12:05
egonzalezpester, can you share console logs? nova console-log <instance_name>12:05
*** rwallner has joined #openstack-kolla12:06
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: Support non-x86 architectures (aarch64, ppc64le)  https://review.openstack.org/43094012:08
hrwegonzalez: 47th version is a bit shorter thanks to you ;)12:09
openstackgerritjimmygc proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable.  https://review.openstack.org/45209412:10
*** tvignaud_ has quit IRC12:11
*** srwilkers has joined #openstack-kolla12:14
*** shardy_lunch is now known as shardy12:16
*** schwicht has quit IRC12:23
*** tvignaud_ has joined #openstack-kolla12:26
*** zhurong has quit IRC12:30
pesteregonzalez: no output from console logs and ubuntu also can`t load... I think it cause I run it inside ESX with "expose virtualization bit" option12:32
egonzalezpester, some years since the last time i touched vmware stuff, guess if virtualization is exposed to the VM, can use KVM  instead of qemu12:34
egonzalezpester, have you checked if virtualization is enabled in compute nodes? egrep -c '(vmx|svm)' /proc/cpuinfo12:34
pesteryes, and I use it, and devstack ocata is ok, and many mitaka installations12:34
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Remove keystone-paste.ini file in kolla  https://review.openstack.org/46253512:34
pesteryes, returns '2'12:34
pesterI can see kvm loaded12:35
*** athomas has joined #openstack-kolla12:36
*** tvignaud_ has quit IRC12:36
*** slagle__ is now known as slagle12:40
*** gianpietro has joined #openstack-kolla12:41
egonzalezpester, dunno then, can you share full trace of nova compute and libvirt logs with kvm enabled as virt_type?12:44
*** blallau has quit IRC12:45
*** rwallner has quit IRC12:47
pesteregonzalez: http://paste.openstack.org/show/608859/ - as you can see some errors in libvirt when "HW virtualization" was not enabled - it is old logs12:47
*** tvignaud_ has joined #openstack-kolla12:48
*** goldyfruit has joined #openstack-kolla12:50
*** schwicht has joined #openstack-kolla12:53
*** rwallner has joined #openstack-kolla12:53
srwilkersmorning12:54
*** eaguilar has joined #openstack-kolla12:58
*** guoshan has joined #openstack-kolla13:00
SerlexMorning13:00
*** schwicht has quit IRC13:03
*** eaguilar has quit IRC13:03
*** eaguilar has joined #openstack-kolla13:03
openstackgerritPaul Bourke (pbourke) proposed openstack/kolla-ansible master: Add vhost to outward rabbitmq for Murano  https://review.openstack.org/45420613:04
*** schwicht has joined #openstack-kolla13:06
*** georgem1 has joined #openstack-kolla13:11
*** janem has joined #openstack-kolla13:14
janemHey everyone, I am having issues with configuring "kolla_external_vip_address" and "kolla_external_fqdn"13:15
*** eanylin has quit IRC13:16
janemmy "kolla_external_vip_address" is a "local" address, and we have a external firewall that NAT's a public IP to that address.13:16
janembut we cant get it to work for some reasin, if we go to our "kolla_external_vip_address" behind the firewall, it works.13:17
janemwe opened port 80 and 443 from the public IP to the internal one. So we should se the horizon login page, right?13:18
janemprobably out of the scope of this channel, but I thought I'd try here.13:20
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/46254713:23
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements  https://review.openstack.org/46254813:23
*** janki has quit IRC13:24
*** imcsk8 has joined #openstack-kolla13:29
*** rwallner has quit IRC13:29
janemit says in the global.yml config file, that the "kolla_external_fqdn" must point to the "kolla_external_vip_address", wondering if that must be direct, and hence not work trough NAT.13:29
*** schwicht has quit IRC13:29
*** srwilkers has quit IRC13:32
*** eanylin_ has joined #openstack-kolla13:34
*** eanylin_ has quit IRC13:36
*** jtriley has joined #openstack-kolla13:36
*** eanylin__ has joined #openstack-kolla13:38
*** zhurong has joined #openstack-kolla13:41
*** skramaja has joined #openstack-kolla13:42
*** skramaja has quit IRC13:45
*** Pavo has joined #openstack-kolla13:48
jemcevoyjanem: To get horizon to work... You'd need to do port forwarding on the firewall to the kolla_external_vip_address ans actually connect to the firewall's Public IP...13:48
hrwquestion to those who build debian/ubuntu images: do you use some kind of web proxy while building?13:49
*** zhurong_ has joined #openstack-kolla13:50
janemjemcevoy: yeah, we have done that, and if we change the NAT rule for the public IP to a functioning webserver on the same network, it works. So the port mapping should be ok.13:50
hrwI have a change to merge which will add some fetching13:50
*** zhurong has quit IRC13:51
*** lucasxu has joined #openstack-kolla13:52
*** gkadam has quit IRC13:53
jemcevoyjamem: You really need to use a VPN to access the network behind the firewall.   Port forwarding may work for Horizon but you won;t have access to the VMs if you are using non-routing rfc 1918 IP address inside the firewall.13:54
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: always update APT lists when install packages  https://review.openstack.org/46256713:56
janemjemcevoy: yes, we are using vpn to access the network. Thats how I can confirm that the internal "kolla_external_vip_address" is working13:56
janemjemcevoy: we have a separate "neutron_external_interface" with a different network setup.13:56
hrwegonzalez, sdake, akwasnie, Jeffrey4l: https://review.openstack.org/462567 is imho important for those people who do not build all images at once.13:57
Jeffrey4lhrw, yes. apt-get update do not run as "yum" which could update local cache automatically.13:58
egonzalezhrw, have to remove this apt-get update too https://github.com/openstack/kolla/blob/master/docker/opendaylight/Dockerfile.j2#L2813:58
openstackgerritMerged openstack/kolla master: Add netcat package in sahara-engine image  https://review.openstack.org/46170613:58
hrwegonzalez: ok13:58
*** jemcevoy has quit IRC13:59
hrwegonzalez: that opendaylight/Dockerfile.j2 should be rewritten first13:59
hrwit is abomination13:59
hrwadding repos is job of base image14:01
egonzalezya know! will be reworking that image soon, at least to enable odl binary in source images(same as ceph,mariadb and rabbit)14:01
hrwegonzalez: I prefer to not touch it then14:01
egonzalezin kolla we do not support source packages for non openstack images, odl is the same case, install binary on both source and binary images14:02
*** srwilkers has joined #openstack-kolla14:03
hrwegonzalez: I am perfectly fine with such ones. but handling repos is base image work14:04
egonzalezhrw, can remove in your change, won't harm at all, just will avoid running clean up twice14:04
hrwegonzalez: my brain cells die when I look at that dockerfile14:04
hrwwill rewrite it probably instead14:04
eanylin__egonzalez: Can we use centos source for rabbitmq for the 4.0.0 kolla-ansible multi-node deployment or we need to use binary? I see the rabbitmq going into crash loop with centos source 4.0.0 image...14:05
egonzalezhrw, the issue with odl repos is that are incredible slow, at least when i tested  increased a lot build times14:05
hrw;(14:06
egonzalezeanylin__, rabbitmq binary and source are almost the same, just a few differences from base image, but rabbit packages are the same14:07
egonzalezeanylin__, what issue you having?14:07
eanylin__egonzalez: Ok, I am seeing rabbitmq crashing and hence cell0 cannot be set up14:08
egonzalezeanylin__, common rabbitmq issue is hostname resolution14:08
eanylin__I did cleanup of hosts and containers a few times but didnt really get past that, kind of strange14:08
eanylin__oh, ok14:08
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: always update APT lists when install packages  https://review.openstack.org/46256714:08
eanylin__I ran the pre-checks and have put the hostname in /etc/hosts14:09
hrwopendaylight added14:09
eanylin__maybe I need to check again, will that be sufficient or?14:09
egonzalezeanylin__, if fail again, share rabbitmq logs14:10
eanylin__ok14:10
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle mariadb for aarch64/ubuntu/source  https://review.openstack.org/46107214:11
*** zhurong_ has quit IRC14:12
eanylin__egonzalez: http://paste.openstack.org/show/608866/14:12
eanylin__seems like its trying to start the rabbitmq-server and wasnt able to14:13
*** janem has left #openstack-kolla14:19
*** janem_ has joined #openstack-kolla14:19
*** jascott1 has joined #openstack-kolla14:23
egonzalezeanylin__, can you share rabbitmq logs /var/lib/docker/volumes/kolla_logs/_data/rabbitmq/14:25
*** schwicht has joined #openstack-kolla14:27
*** jascott1 has quit IRC14:28
eanylin__egonzalez, I think I found the problem. I did something dumb, my hostname and hostname -f are different14:29
eanylin__egonzalez: I changed it and the rabbitmq seems to be stable now14:29
eanylin__egonzalez: Thanks for the tip :)14:29
egonzalezeanylin__, np14:29
eanylin__egonzalez: Will continue to deploy the cluster. Thanks!14:30
*** lrensing has joined #openstack-kolla14:31
*** manheim has quit IRC14:33
*** guoshan has quit IRC14:33
*** jtriley has quit IRC14:34
openstackgerritSam Betts proposed openstack/kolla-kubernetes master: Ensure that ironic-dnsmasq can find its IP address  https://review.openstack.org/46231014:35
*** ducttape_ has joined #openstack-kolla14:37
*** jtriley has joined #openstack-kolla14:37
pesteregonzalez: where I should place bug ? github or launchpad ?14:40
inc0good morning14:40
eanylin__egonzalez: Seems like I may not have found the real root cause for the nova-compute service not coming up issue.  So seems like rabbitmq is up now but nova-compute didnt come up. Any other tips for that? Current situation looks like this, http://paste.openstack.org/show/608872/14:41
egonzalezpester, launchpad, provide complete information: logs, environment, versions, config, custom_configs, etc14:41
*** srwilkers has quit IRC14:42
egonzalezeanylin__, is the compute node container up?14:42
*** ducttape_ has quit IRC14:42
*** ducttape_ has joined #openstack-kolla14:43
*** lpetrut_ has quit IRC14:44
eanylin__egonzalez: Yeah, the nova-compute container is up. As I can see, http://paste.openstack.org/show/608873/14:44
egonzalezeanylin__, what kolla and kolla-ansible version are you using? check nova api version also. docker exec nova_api pip freeze |  grep nova14:45
eanylin__egonzalez, was trying master branch and later just went back to "pip install kolla-ansible"14:46
egonzalezimages still master or rebuild with stable?14:46
eanylin__egonzalez: Image is Centos Source 4.0.014:47
egonzalezansible stable/ocata nova code is not compatible with master images14:47
eanylin__I just did kolla-build with push to the local registry14:47
eanylin__egonzalez: http://paste.openstack.org/show/608876/14:48
eanylin__egonzalez: Ok14:48
eanylin__egonzalez: I did this, kolla-build --push --type source --registry 192.168.122.21:400014:49
egonzalezthe issue is with cell creation code in ansible ocata does not work with nova multi cells new code14:49
janem_when I change settings in my globals.yml file, I should run "kolla-ansible reconfigure" right?14:50
egonzaleznot sure if we should backport the change or not :/14:50
eanylin__egonzalez: I see. What should be approach in this case?14:50
egonzalezjanem_, yep14:50
egonzalezjanem_, well, depends of what changed, if added a new service need deploy14:50
egonzalezeanylin__, use same version of images and ansible, bump both to master or downgrade both to ocata14:51
janem_egonzalez: I changed "kolla_external_vip_address" and "kolla_external_fqdn"14:51
eanylin__egonzalez: Ok.  By master image, we are referring to 4.0.1 right?14:52
janem_changes doesnt seem to happen though, still se "kolla_internal_vip_address" in my openrc files .. shouldnt that have changed to the new ips/fqdn?14:52
janem_egonzalez: no errors when I run reconfigure, and do report changes.14:54
egonzalezjanem_, the issue with those settings is that old fqdn is associated with keystone public endpoints14:54
*** eaguilar has quit IRC14:55
*** srwilkers has joined #openstack-kolla14:55
egonzalezjanem_, nope, adminrc file always is created with internal url14:56
*** salv-orl_ has joined #openstack-kolla14:56
janem_egonzalez: ok. Hm.. It's strange. The new vip is working for the "external_vip_address" but cant seem to get the fqdn working.14:57
*** salv-orlando has quit IRC14:59
janem_egonzalez: my external_vip is just a different internal vip address, and we have an external firewall that NAT's a public IP to that one. Then external DNS setup to point to the public IP.14:59
janem_done this a million times with other web servers, never been an issue. So I dont think it's the firewall/nat thats the problem.15:00
*** yingjun has joined #openstack-kolla15:00
*** gfidente has quit IRC15:01
*** eaguilar has joined #openstack-kolla15:02
*** unicell1 has quit IRC15:03
*** unicell has joined #openstack-kolla15:06
egonzalezjanem_, have a similar env with external on same network than internal but without all the NAT/firewall stuff, this is my config http://paste.openstack.org/show/608879/15:07
*** shasha_t_ has joined #openstack-kolla15:07
egonzalezhad some issues when reconfigured, just restarted all services and went fine iirc15:07
*** Manheim has joined #openstack-kolla15:09
janem_egonzalez: thanks! It looks similar, but I have not defined the "kolla_external_vip_interface" , but since the external_vip address is working, I guess I dont need that configured (?)15:09
janem_egonzalez: I'll try to restart the services15:11
*** shasha_t_ has quit IRC15:11
*** jtriley has quit IRC15:19
janem_egonzalez: didn't help. So strange that the vip is working, but not the NAT'ed ip.. I'll meditate on it ;)15:20
janem_Thanks for the help though!15:20
*** jascott1 has joined #openstack-kolla15:23
*** mgoddard has quit IRC15:24
janem_is it enough to open port 80 and 443 to the vip to get horizon loaded? Maybe I'm missing some ports?15:25
*** jascott1 has quit IRC15:29
*** eaguilar has quit IRC15:30
*** jrich523 has joined #openstack-kolla15:30
*** lrensing has quit IRC15:34
*** ducttap__ has joined #openstack-kolla15:38
*** ducttape_ has quit IRC15:38
*** lrensing has joined #openstack-kolla15:39
*** mgoddard has joined #openstack-kolla15:39
*** magicboiz has quit IRC15:41
*** lrensing has quit IRC15:47
*** lrensing has joined #openstack-kolla15:49
*** lucasxu has quit IRC15:53
*** jaosorior has quit IRC15:54
georgem1I'm a first time kolla user and I just ran tag 4.0.0 but the login page for the dashboard has no logo and I see a lot of 404s for  "/static/dashboard/js/" stuff15:54
georgem1any idea what I did wrong?15:54
*** janem__ has joined #openstack-kolla15:54
*** srwilkers_ has joined #openstack-kolla15:55
*** lucasxu has joined #openstack-kolla15:56
*** lucasxu has quit IRC15:56
*** lucasxu has joined #openstack-kolla15:56
*** janem_ has quit IRC15:57
*** srwilkers_ has quit IRC15:57
*** vhosakot has joined #openstack-kolla15:59
*** jtriley has joined #openstack-kolla15:59
*** lucasxu has quit IRC15:59
openstackgerritEduardo Gonzalez proposed openstack/kolla master: Fix Incorrect timezone in ubuntu horizon images  https://review.openstack.org/46077016:01
*** lucasxu has joined #openstack-kolla16:02
*** egonzalez has quit IRC16:02
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Advertise public heat endpoint for wait conditions  https://review.openstack.org/46265516:04
*** caowei has quit IRC16:05
*** caowei has joined #openstack-kolla16:05
*** janem has joined #openstack-kolla16:06
openstackgerritMathieu Rohon proposed openstack/kolla-ansible master: Add skydive service  https://review.openstack.org/46092216:07
*** janem__ has quit IRC16:09
*** eaguilar has joined #openstack-kolla16:10
*** athomas has quit IRC16:14
*** eaguilar has quit IRC16:15
*** lucasxu has quit IRC16:15
*** unicell has quit IRC16:16
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Fix heat ec2 keystone auth  https://review.openstack.org/44495316:17
*** eaguilar has joined #openstack-kolla16:18
*** eaguilar has quit IRC16:18
*** vhosakot_ has joined #openstack-kolla16:21
*** vhosakot has quit IRC16:23
*** jascott1 has joined #openstack-kolla16:24
*** salv-orl_ has quit IRC16:26
openstackgerritDavid Moreau Simard proposed openstack/kolla-ansible master: Do not merge: Test ARA 0.13.0rc2  https://review.openstack.org/43996316:27
*** eaguilar has joined #openstack-kolla16:28
*** jascott1 has quit IRC16:29
*** nathharp has quit IRC16:33
*** eaguilar has quit IRC16:34
*** tumble has joined #openstack-kolla16:38
*** mgoddard has quit IRC16:39
*** pcaruana has quit IRC16:40
*** athomas has joined #openstack-kolla16:40
*** jascott1 has joined #openstack-kolla16:43
*** shasha_t_ has joined #openstack-kolla16:44
*** erlon has joined #openstack-kolla16:48
*** lucasxu has joined #openstack-kolla16:49
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: handle mariadb for aarch64/ubuntu/source  https://review.openstack.org/46107216:51
*** shardy has quit IRC16:57
*** athomas has quit IRC16:59
*** srwilkers has quit IRC17:00
*** rwallner has joined #openstack-kolla17:01
*** unicell has joined #openstack-kolla17:01
*** athomas has joined #openstack-kolla17:01
*** rwallner_ has joined #openstack-kolla17:02
*** rwallner has quit IRC17:05
*** hieulq_ has joined #openstack-kolla17:05
*** rwallner_ has quit IRC17:07
*** jascott1 has quit IRC17:07
*** rwallner has joined #openstack-kolla17:08
*** vhosakot_ has quit IRC17:09
*** rwallner has quit IRC17:09
*** rwallner has joined #openstack-kolla17:09
*** vhosakot has joined #openstack-kolla17:10
*** jascott1 has joined #openstack-kolla17:16
*** rwallner has quit IRC17:21
*** lpetrut has joined #openstack-kolla17:24
openstackgerritOpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements  https://review.openstack.org/46254717:25
*** igordcar1 has quit IRC17:27
*** igordcard has joined #openstack-kolla17:27
*** hieulq_ has quit IRC17:37
*** hieulq_ has joined #openstack-kolla17:39
*** egonzalez has joined #openstack-kolla17:42
*** unicell has quit IRC17:47
*** yingjun has quit IRC17:48
*** unicell has joined #openstack-kolla17:48
*** ducttap__ has quit IRC17:51
*** mgoddard has joined #openstack-kolla17:51
*** ducttape_ has joined #openstack-kolla17:52
inc0kfox1111: it was ovs 2.6 when ovs got this massive performance boost right?17:53
openstackgerritMark Goddard proposed openstack/kolla-ansible master: Advertise public heat endpoint for wait conditions  https://review.openstack.org/46265517:54
*** ducttape_ has quit IRC17:56
*** lpetrut has quit IRC18:03
*** srwilkers has joined #openstack-kolla18:13
*** dciabrin has quit IRC18:14
*** dciabrin has joined #openstack-kolla18:18
*** harlowja has quit IRC18:19
*** rwallner has joined #openstack-kolla18:20
sbezverkinc0: you will see performance boost when we add networking-vpp ;)18:22
inc0lol18:23
inc0or when we add dpdk18:23
inc0but yeah, I googled, it was 2.618:23
inc0160+% perf increase18:23
*** tonanhngo has joined #openstack-kolla18:28
*** ducttape_ has joined #openstack-kolla18:29
*** rwallner has quit IRC18:33
*** Pavo has quit IRC18:34
sbezverkinc0: there were some studies and trials, it seems ovs has performance issue with some type of packets I do not recall exactly the details, but similar tests with networking-vpp showed more consistent performance results :)18:47
sbezverkcannot wait to get it in kk18:47
jascott1anyone used this? https://github.com/trailofbits/algo18:48
jascott1my security minded friend swears by it18:48
*** srwilkers has quit IRC18:48
*** jascott1 has quit IRC18:51
*** rwallner has joined #openstack-kolla18:51
*** mgoddard has quit IRC18:52
*** manheim_ has joined #openstack-kolla18:54
*** jascott1 has joined #openstack-kolla18:55
*** ducttape_ has quit IRC18:57
*** ducttap__ has joined #openstack-kolla18:58
*** zhubingbing_ has quit IRC19:02
*** zhubingbing_ has joined #openstack-kolla19:02
*** esmute has joined #openstack-kolla19:02
*** srwilkers has joined #openstack-kolla19:06
*** rwallner has quit IRC19:09
*** manheim_ has quit IRC19:10
*** ducttap__ has quit IRC19:13
*** ducttape_ has joined #openstack-kolla19:13
*** rhallisey has quit IRC19:13
*** srwilkers has quit IRC19:17
*** lrensing has quit IRC19:19
*** hieulq_ has quit IRC19:28
*** dave-mccowan has quit IRC19:30
*** mgoddard has joined #openstack-kolla19:32
*** esmute has quit IRC19:33
*** shasha_t_ has quit IRC19:39
*** hieulq_ has joined #openstack-kolla19:41
*** dave-mccowan has joined #openstack-kolla19:43
*** dave-mcc_ has joined #openstack-kolla19:45
*** harlowja has joined #openstack-kolla19:48
*** dave-mccowan has quit IRC19:48
*** salv-orlando has joined #openstack-kolla19:52
*** Pavo has joined #openstack-kolla19:53
*** hieulq_ has quit IRC19:55
*** rwallner has joined #openstack-kolla19:59
*** ipsecguy_ has joined #openstack-kolla20:05
*** ipsecguy has quit IRC20:07
*** Pavo has quit IRC20:09
*** rwallner has quit IRC20:13
*** Pavo has joined #openstack-kolla20:13
*** mgoddard has quit IRC20:14
*** georgem1 has quit IRC20:17
*** skramaja has joined #openstack-kolla20:25
*** dave-mcc_ has quit IRC20:34
*** shasha_t_ has joined #openstack-kolla20:35
*** rwallner has joined #openstack-kolla20:39
*** shasha___ has joined #openstack-kolla20:41
*** shasha_t_ has quit IRC20:41
*** rwallner has quit IRC20:44
*** Serlex has quit IRC20:46
*** athomas has quit IRC20:52
*** salv-orl_ has joined #openstack-kolla20:56
*** salv-orlando has quit IRC20:59
*** lucasxu has quit IRC21:01
*** jtriley has quit IRC21:06
*** eaguilar has joined #openstack-kolla21:06
*** bmace has quit IRC21:09
*** bmace has joined #openstack-kolla21:09
*** goldyfruit has quit IRC21:14
*** mgoddard has joined #openstack-kolla21:14
*** esmute_ has joined #openstack-kolla21:25
esmute_Hey guys, looks like i got kolla running on kubernetes. but my nova vm are in error. Nova shows 'Binding failed for port 03eadd66-f818-4453-a5d7-746630239bd5, please check neutron logs for more information.'21:26
esmute_i also notice that the neutron-openvswitch pod is stuck in init. Not sure if that is the cause21:26
esmute_neutron-openvswitch-agent-network-hzvf7   0/1       Init:2/3   0          39m21:26
*** eaguilar has quit IRC21:26
*** skramaja has quit IRC21:31
*** Manheim has quit IRC21:33
sdakeesmute_ a few peopel have reported that although it odes't happen to me nor in the gate21:33
sdakeesmute_ which guide did you followt o install?21:33
esmute_I followed the baremetal guide21:33
esmute_https://docs.openstack.org/developer/kolla-kubernetes/deployment-guide.html21:34
sdakecool21:34
sdakethat was working a few days ago - prior to the change in canal21:34
sdakesince then - been a little problematic :(21:34
esmute_i had an issue when i generated the config using sudo kolla-ansible genconfig... I was told that this didnt work and to use the ansible playbook21:34
*** schwicht has quit IRC21:36
*** mgoddard has quit IRC21:40
sdakeesmute_ indeed that may be the problem21:42
*** manheim has joined #openstack-kolla21:43
*** Pavo has quit IRC21:43
*** imcsk8 has quit IRC21:46
*** imcsk8 has joined #openstack-kolla21:46
*** Pavo has joined #openstack-kolla21:46
*** eaguilar has joined #openstack-kolla21:48
*** salv-orl_ has quit IRC21:51
*** esmute_ has quit IRC21:51
*** eaguilar has quit IRC21:52
*** gianpietro has quit IRC21:53
*** gianpietro has joined #openstack-kolla21:53
*** eaguilar has joined #openstack-kolla21:54
jascott1esmute_ thats because of using genconfig21:54
*** esmute_ has joined #openstack-kolla21:54
esmute_sdake: anything i can do to figure out what is causing this problem?21:54
jascott1when you describe neutron pods do you see volume mount error?21:55
*** jbadiapa has quit IRC22:01
sbezverkesmute_: kubectl describe pod -n kolla neutron-openvswitch-agent-network-hzvf722:02
inc0hmm22:04
*** schwicht has joined #openstack-kolla22:05
esmute_sbezverk: https://gist.github.com/kokhang/581f9daa3d4ea115da394fc3637bcb6622:06
esmute_thanks for the help22:06
esmute_no mounting error that i can see22:07
esmute_i deployed kolla multiple times and the neutron-openvswitch pod is always stuck in init22:07
sbezverkesmute_: it stuck on init container, it is waiting for vswitchd to be up and running. on the host where this pod is running and run this commands:22:10
sbezverklsmod | grep openvswitch22:10
sbezverkovs-dpctl show22:11
sbezverkovs-appctl version22:11
sbezverkif any of these do not produce expected result, then this pod will be in stuck mode forever22:12
sbezverkbut the issue is not with it, but with openvswitch22:12
esmute_sbezverk: https://gist.github.com/kokhang/6628ffc3ac9c5da0c3bde327f9c5ad7122:12
esmute_it is coreos. So it doesnt have neither ovs-dpctl nor ovs-appctl22:13
sbezverkesmute_: here is your answre :)22:13
sbezverkwe run run these commands and EXPECT them to be available22:13
sbezverkone sec, you need to run them from that container22:14
sbezverkthey are packaged with the container22:14
sbezverkdocker exec -tu root blah ovs-dpctl show22:14
sbezverkdocker exec -tu root blah ovs-appctl version22:15
*** jbadiapa has joined #openstack-kolla22:15
*** rwallner has joined #openstack-kolla22:16
esmute_sbezverk: https://gist.github.com/kokhang/6628ffc3ac9c5da0c3bde327f9c5ad7122:17
sbezverkesmute_: looks good so far22:19
sbezverkesmute_: docker logs blah22:19
esmute_+ modprobe openvswitch + /usr/sbin/ovs-vswitchd unix:/var/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/kolla/openvswitch/ovs-vswitchd.log --pidfile=/var/run/openvswitch/ovs-vswitchd.pid22:20
*** rwallner has quit IRC22:21
esmute_sbezverk: https://gist.github.com/kokhang/f492cf9068991dd817fd8193352b1d6722:22
esmute_not sure if that is useful22:22
sbezverk esmute_: no, I need logs from neutron-openvswitch-agent-network-hzvf7 container22:23
esmute_how do you get that if the container is in init?22:24
sbezverkdocker logs $(docker ps -a | grep neutron-openvswitch-agent-network-hzvf7@ | awk '{print $1}')22:24
sbezverktry to find docker container id for this ID 4317b6f7f8787f6e3f1e6bd8925b4851f167b390492c520a5049dcbd92f6bc7e22:25
*** goldyfruit has joined #openstack-kolla22:27
esmute_the log shows a bunch of:22:27
esmute_2017-05-04T21:56:43Z|00001|daemon_unix|WARN|/var/run/openvswitch/ovs-vswitchd.pid: stale pidfile for pid 8  being deleted by pid 0 ovs-appctl: cannot read pidfile "/var/run/openvswitch/ovs-vswitchd.pid" (No such process) Trying again.22:27
*** vhosakot has quit IRC22:30
sbezverkesmute_: when have you updated kolla-kubernetes? check on the host if that file really exist.22:32
esmute_yes it exists on the host22:33
esmute_pid 822:33
*** Pavo has quit IRC22:34
esmute_kolla-kubernetes is at cf085cce0f14ed968d435959ad7ebae6e9542daa Merge "Add local registry option in deployment-guide"22:34
sbezverkesmute_: ah ok you are still running kubernetes 1.6.1 and you forgot to add one parameters which is required22:35
sbezverkesmute_: you need to add "--enable-cri=false"22:37
sbezverkto your kubelet service command line22:37
esmute_yes im running 1.6.122:38
esmute_ok let me try that22:38
sbezverkyeah, there is a bug which gets workedaround by using that keywork I gave you above22:38
*** schwicht has quit IRC22:38
sbezverkesmute_: I am positive it will work :) I need to step out, will be back later22:42
*** eaguilar has quit IRC22:42
esmute_sbezverk: thank you for your help.22:43
*** manheim has quit IRC22:51
*** salv-orlando has joined #openstack-kolla22:51
*** egonzalez has quit IRC22:53
*** salv-orlando has quit IRC22:58
*** goldyfruit has quit IRC23:04
*** kolla-slack3 has quit IRC23:06
*** kolla-slack has joined #openstack-kolla23:06
*** esmute_ has quit IRC23:13
*** esmute has joined #openstack-kolla23:13
*** esmute has quit IRC23:15
*** esmute has joined #openstack-kolla23:15
*** erlon has quit IRC23:24
*** rwellum has quit IRC23:38
*** guoshan has joined #openstack-kolla23:40
*** ducttape_ has quit IRC23:44
*** eanylin__ has quit IRC23:45
*** Pavo has joined #openstack-kolla23:48
*** salv-orlando has joined #openstack-kolla23:54
*** salv-orlando has quit IRC23:58

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!