Wednesday, 2017-11-15

*** salv-orlando has quit IRC00:01
*** salv-orlando has joined #openstack-kolla00:02
*** yangyapeng has quit IRC00:05
*** salv-orlando has quit IRC00:06
*** hongbin has quit IRC00:10
*** yangyapeng has joined #openstack-kolla00:19
*** yangyapeng has quit IRC00:23
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/50944300:24
*** yangyapeng has joined #openstack-kolla00:38
*** zshi has joined #openstack-kolla00:45
*** yangyapeng has quit IRC00:47
kfox1111ok. lets see how badly this breaks things. :)00:48
*** chas has joined #openstack-kolla00:49
*** zshi has quit IRC00:51
*** Pavo has quit IRC00:53
*** chas has quit IRC00:53
*** Pavo has joined #openstack-kolla00:53
*** Pavo has quit IRC00:54
*** zhangfei has joined #openstack-kolla00:57
*** kiennt26 has joined #openstack-kolla01:01
*** Pavo has joined #openstack-kolla01:01
lxdongkfox1111 jamesbenson I came across this [root@compute01 ceph]# docker logs bootstrap_mariadb INFO:__main__:Loading config file at /var/lib/kolla/config_files/config.json INFO:__main__:Validating config file INFO:__main__:Kolla config strategy set to: COPY_ALWAYS INFO:__main__:Copying service configuration files INFO:__main__:Deleting /etc//my.cnf INFO:__main__:Copying /var/lib/kolla/config_files/galera.cnf to /etc//my.cnf INF01:04
*** zshi_ has joined #openstack-kolla01:04
lxdongat TASK [mariadb : Running MariaDB bootstrap container]01:04
*** yangyapeng has joined #openstack-kolla01:05
lxdongCopying /var/lib/kolla/config_files/galera.cnf to /etc//my.cnf01:05
*** yangyapeng has quit IRC01:06
lxdongI think it's a broken path01:06
*** yangyapeng has joined #openstack-kolla01:06
Jeffrey4lkfox1111, re ceph works, are you meant kolla external ceph support?01:06
kfox1111Jeffrey4l: more ceph in general.01:06
*** duonghq has joined #openstack-kolla01:06
Jeffrey4lgot. thanks.01:06
kfox1111lxdong: not sure on that one.... mysql looks for configs in weird places.01:06
Jeffrey4llxdong, it is a bad filename, galera.cnf. but this is how it works now. the optimal filename is change galera.cnf to my.cnf01:07
Jeffrey4linc0,  around?01:09
lxdongWhat should I do , tried /etc//my.cnf it can work.01:09
Jeffrey4llxdong, what's you issue?01:09
lxdongTASK [mariadb : Running MariaDB bootstrap container] fatal: [compute01]: FAILED! => {"changed": true, "failed": true, "msg": "Container exited with non-zero return code"}01:10
lxdongdocker logs show above01:10
*** gfhellma_ has quit IRC01:11
openstackgerritJeffrey Zhang proposed openstack/kolla master: [wip]Publish docker images to tarballs.openstack.org site  https://review.openstack.org/51966801:14
inc0hey now I am01:14
*** jtriley has joined #openstack-kolla01:14
inc0Jeffrey4l: how was summit? I saw your presentations, you did good!01:14
*** lxdong has quit IRC01:14
Jeffrey4linc0, thank. it is great.01:15
Jeffrey4linc0, i want to talk this patch https://review.openstack.org/51966801:15
Jeffrey4linc0, will we drop the support for tarballs.o.o site images? or support it and dockerhub on the same time?01:16
inc0Jeffrey4l: infra guys said that they don't really like tarballs01:16
*** tovin07_ has joined #openstack-kolla01:16
Jeffrey4laha, so we will remove all the files in tarballs?01:16
inc0well first we need to push daily for stable branches and change deploy gates to use dockerhub - and be very careful to use cache there01:17
*** lxdong has joined #openstack-kolla01:17
Jeffrey4lcache means?01:17
inc0http cache, we used it before - via apache01:18
lxdonglost connection. What's the reply?01:18
inc0every nodepool has local https cache, so if we pull through it, next gate will get cached images01:19
inc0that will minimize bandwidth requirement for our gates01:19
lxdongdocker logs bootstrap_mariadb http://paste.openstack.org/show/626284/01:19
Jeffrey4linc0, if we use dockerhub, we will not use the cache, right?01:20
inc0lxdong: also check /var/lib/docker/volumes/kolla_logs01:20
inc0Jeffrey4l: why?01:20
Jeffrey4lhrm. about the http cache, is it a http proxy cache?01:20
inc0docker pull is just http, so if we pull through http cache, it should just work01:20
Jeffrey4linc0, docker pull is https. and https is never be cache, i think.01:20
inc0I think docker can use https cache, it's a config option, let me check01:21
Jeffrey4linc0, do you have any link or doc for the http cache?01:21
inc0https://docs.docker.com/registry/recipes/mirror/01:21
Jeffrey4lthis is registry mirror. not http proxy cache.01:22
*** rhallisey has quit IRC01:22
inc0right, hold on I remember we were setting up cache for image pulls01:23
Jeffrey4lif the infra guys could set up a docker mirror service, that will be helpful.01:23
lxdonghttp://paste.openstack.org/show/626307/ ERROR 2003 (HY000): Can't connect to MySQL server on '192.168.4.150' (111 "Connection refused")01:24
*** tovin07__ has joined #openstack-kolla01:24
Jeffrey4llxdong, what are you doing? first deploy?01:25
inc0I'll experiment with cache tomorrow, I can't remember what we planned with infra01:25
inc0anyway, we want to get rid of tarballs asap01:26
Jeffrey4lgot. thanks.01:26
Jeffrey4lthen i will fix the kolla-ansible to use dockerhub today.01:26
lxdongwith the latest kolla version ,yes. deployed before01:26
Jeffrey4llxdong, the first deploy is successful? then what are you doing now?01:27
inc0first let's backport all gate work to stable branches01:27
*** tovin07_ has quit IRC01:28
*** tovin07__ has quit IRC01:28
Jeffrey4linc0, there is one issue now. tarballs.o.o image is not updated for one month. the old image affect the jobs now.01:29
inc0yeah01:30
Jeffrey4linc0, could you review https://review.openstack.org/51905201:30
inc0look at infra01:30
*** duonghq has quit IRC01:30
*** hieulq has quit IRC01:30
*** kiennt26 has quit IRC01:30
Jeffrey4linc0, and https://review.openstack.org/515579 if you have time ;)01:30
*** tovin07_ has joined #openstack-kolla01:30
lxdongupdate kolla to latest version ,rebuilt images and redeploy with external ceph.01:31
inc0done01:31
*** duonghq has joined #openstack-kolla01:31
*** kiennt26 has joined #openstack-kolla01:31
*** hieulq has joined #openstack-kolla01:31
inc0I'll rebase rest of our gate changes including pusher tomorrow01:31
inc0Jeffrey4l: can you experiment with this docker cache in meantime on master?01:31
Jeffrey4linc0, how? where is the cache?01:32
inc0look at infra chat01:32
Jeffrey4lroger. will try it today.01:34
inc0thanks01:35
inc0drop a comment how it went so I can pick it up tomorrow01:35
Jeffrey4lnp01:36
lxdongwith external ceph didn't succeed last time because of the same issue.01:37
*** absubram has quit IRC01:37
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/50944301:38
Jeffrey4llxdong, could you file a bug for this?01:40
lxdongsure01:43
*** chas has joined #openstack-kolla01:43
lxdongkfox1111 jamesbenson used external ceph too ,they work well.01:45
*** chas has quit IRC01:47
*** caowei has joined #openstack-kolla01:50
*** chialiang has joined #openstack-kolla01:51
*** salv-orlando has joined #openstack-kolla02:03
*** threestrands has joined #openstack-kolla02:07
*** salv-orlando has quit IRC02:08
*** itlinux has joined #openstack-kolla02:12
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993902:13
lxdonghttp://paste.openstack.org/show/626311/ /usr/local/bin/wsrep-notify.sh --status Joined --uuid 5c2b0a40-c9a8-11e7-8068-8a9646dd46c7 --primary yes --index 0 --members 5c2a55c6-c9a8-11e7-be2e-07b00d412d97/compute01/192.168.4.150:0: 1 (Operation not permitted)02:13
SamYapleJeffrey4l: inc0 its a pretty simple setup to use the cache02:15
Jeffrey4lyes. Got the answer from infra channel. and thanks.02:15
SamYapleJeffrey4l: https://github.com/openstack/loci/blob/master/playbooks/vars.yaml#L1-L802:15
SamYapleif you do the daemon.json for config it gets easier02:16
SamYaplebecause you can declare it all in vars yaml02:16
Jeffrey4lyes.02:16
SamYaplehttps://github.com/openstack/loci/blob/master/playbooks/setup-gate.yaml#L32-L3302:16
SamYaplesomething like that02:16
SamYaplewe should probably make a shared docker install playbook02:16
*** itlinux has quit IRC02:17
Jeffrey4lSamYaple, shared docker install is a good idea.02:18
Jeffrey4lor implement a common role to install docker.02:18
SamYapleyea we can do that easy like02:20
*** danardelean has joined #openstack-kolla02:20
SamYapleespecially with the daemon.json in the vars like that02:20
SamYaplethen each project can set thatup the way the wist02:21
SamYaplewish*02:21
*** danardelean has quit IRC02:25
korean101something wrong...02:34
korean101i try to kolla-ansible upgrade (from ocata to pike)02:34
korean1011. i deploy ocata release via kolla02:34
korean101but can't reach http://169.254.169.254/2009-04-04/instance-id02:35
korean1012. i deploy pike release via kolla02:35
korean101nothing's wrong02:35
korean101some bugs in ocata release? via kolla02:36
*** chas has joined #openstack-kolla02:37
*** chas has quit IRC02:42
*** tovin07__ has joined #openstack-kolla02:42
*** tovin07_ has quit IRC02:44
*** tovin07 has joined #openstack-kolla02:53
*** tovin07__ has quit IRC02:56
-openstackstatus- NOTICE: Due to an unexpected outage with Zuul (1 hour), you'll need to recheck any jobs that were in progress. Sorry for the inconvenience.02:56
*** tovin07 has quit IRC02:58
*** Pavo has quit IRC02:59
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993903:00
*** unicell has quit IRC03:00
*** danardelean has joined #openstack-kolla03:02
*** Pavo has joined #openstack-kolla03:02
*** salv-orlando has joined #openstack-kolla03:04
*** danardelean has quit IRC03:06
*** salv-orlando has quit IRC03:09
*** dave-mccowan has quit IRC03:09
lxdongThere must be something wrong, I didn't do anything and  TASK [mariadb : Running MariaDB bootstrap container] passed03:11
*** japestinho has joined #openstack-kolla03:13
*** itlinux has joined #openstack-kolla03:15
lxdongIs this a dim bug of galera cluster sync ? keep watching03:18
openstackgerritMerged openstack/kolla-ansible stable/ocata: Move to zuul v3 in project jobs  https://review.openstack.org/51557903:28
*** chas has joined #openstack-kolla03:31
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993903:33
*** tovin07_ has joined #openstack-kolla03:35
*** zhangfei has quit IRC03:35
*** chas has quit IRC03:35
*** jamesbenson has joined #openstack-kolla03:36
*** mdnadeem has joined #openstack-kolla03:39
*** jamesbenson has quit IRC03:40
*** zhangfei has joined #openstack-kolla03:47
lxdongtimeout at TASK [nova : Waiting for nova-compute service up] http://paste.openstack.org/show/626315/03:54
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Add note about overwriting policy.json file  https://review.openstack.org/51118603:56
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Add support policy.yaml file - Aodh  https://review.openstack.org/51456503:56
*** kiennt26 has quit IRC03:57
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Add note about overwriting policy.json file  https://review.openstack.org/51118603:59
openstackgerritDai Dang Van proposed openstack/kolla-ansible master: Add note about overwriting policy.json file  https://review.openstack.org/51118603:59
*** kiennt26 has joined #openstack-kolla04:01
lxdongno error in nova logs .nova-compute.conf [libvirt] images_rbd_pool=vms images_type=rbd images_rbd_ceph_conf=/etc/ceph/ceph.conf rbd_user=nova04:02
*** salv-orlando has joined #openstack-kolla04:05
*** salv-orlando has quit IRC04:11
*** kiennt26 has quit IRC04:16
lxdongFound the same issue of my own in 2017-10-13 logs  and didn't happen during the next deployment04:16
*** jamesbenson has joined #openstack-kolla04:17
*** jamesbenson has quit IRC04:22
*** chas has joined #openstack-kolla04:25
*** chas has quit IRC04:29
*** ntpttr_laptop has joined #openstack-kolla04:30
*** skramaja has joined #openstack-kolla04:43
*** magicboiz has quit IRC04:46
kolla-slack<inc0> SamYaple we generally use bootstrap servers for k-ans which will need it04:50
kolla-slack<inc0> But daemon.json is simple template task04:51
*** jamesbenson has joined #openstack-kolla04:59
*** Pavo has quit IRC04:59
*** Pavo has joined #openstack-kolla05:03
*** jamesbenson has quit IRC05:04
*** salv-orlando has joined #openstack-kolla05:07
lxdongissue remains unfortunately this time.05:09
openstackgerritMerged openstack/kolla-ansible master: Revert "Omit outward_rabbitmq from check for upgrade"  https://review.openstack.org/51559805:11
openstackgerritMerged openstack/kolla-ansible master: Clean update endpoints task in keystone upgrade  https://review.openstack.org/51560005:11
*** salv-orlando has quit IRC05:12
Jeffrey4lpbourke, berendt please review https://review.openstack.org/51959205:14
*** caowei has quit IRC05:16
*** chas has joined #openstack-kolla05:19
openstackgerritJeffrey Zhang proposed openstack/kolla master: Improve gate post jobs  https://review.openstack.org/51996905:20
*** chas has quit IRC05:23
*** jaosorior has quit IRC05:24
*** itlinux has quit IRC05:27
*** chialiang has quit IRC05:30
*** chialiang_ has joined #openstack-kolla05:30
*** chialiang_ has quit IRC05:32
*** chialiang_ has joined #openstack-kolla05:32
*** janki has joined #openstack-kolla05:33
openstackgerritJeffrey Zhang proposed openstack/kolla stable/pike: Install python-openstackclient in kolla-toolbox container  https://review.openstack.org/51997105:37
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Install python-openstackclient in kolla-toolbox container  https://review.openstack.org/51997205:38
*** janki has quit IRC05:40
*** caowei has joined #openstack-kolla05:57
korean101i try to deploy Ocata releases via kolla06:00
korean101some ERRORs ("The conditional check ''169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none' failed. The error was: The ipaddr filter requires python-netaddr be installed on the ansible controller")06:00
korean101anyone knows?06:00
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/51849006:02
openstackgerritSai Ram Peesapati proposed openstack/kolla master: Typo in neutron-server/extend_start.sh  https://review.openstack.org/51997606:05
*** ntpttr_laptop has quit IRC06:05
*** salv-orlando has joined #openstack-kolla06:08
*** janki has joined #openstack-kolla06:12
*** salv-orlando has quit IRC06:12
*** chas has joined #openstack-kolla06:13
*** lpetrut_ has joined #openstack-kolla06:15
*** chas has quit IRC06:18
*** yingjun has joined #openstack-kolla06:20
*** kevzha01 is now known as kevinz06:24
*** daidv has joined #openstack-kolla06:26
ashuhi alll06:27
ashui am having issue with ceph06:27
ashuthere is no ceph-osd container in storage node using kolla-ansible pike06:28
*** danardelean has joined #openstack-kolla06:31
*** unicell has joined #openstack-kolla06:33
*** danardelean has quit IRC06:34
*** threestrands has quit IRC06:34
*** unicell1 has joined #openstack-kolla06:35
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/51849006:37
*** unicell has quit IRC06:37
*** salv-orlando has joined #openstack-kolla06:38
*** lpetrut_ has quit IRC06:40
*** lpetrut_ has joined #openstack-kolla06:41
lxdongThere is \"exec: \\\"openstack\\\": executable file not found in $PATH\"" in kolla-toolbox based centos as well.06:42
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/51849006:46
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993906:50
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/51849006:55
*** lpetrut_ has quit IRC06:56
*** jamesbenson has joined #openstack-kolla06:56
*** jaosorior has joined #openstack-kolla06:57
*** egonzalez has joined #openstack-kolla06:58
*** Pavo has quit IRC06:59
korean101ashu: parted /dev/sdb -s -- mklabel gpt mkpart KOLLA_CEPH_OSD_BOOTSTRAP 1 -107:01
*** jamesbenson has quit IRC07:01
*** cah_link has joined #openstack-kolla07:02
Jeffrey4legonzalez, could u review https://review.openstack.org/519592 first?07:02
egonzalezjust made it :)07:02
*** Pavo has joined #openstack-kolla07:04
Jeffrey4ltanks.07:06
*** chas has joined #openstack-kolla07:07
korean101egonzalez: Hi. i try to kolla-ansible upgrade07:10
korean101egonzalez: can i ask something?07:11
*** danardelean has joined #openstack-kolla07:11
korean101egonzalez: i try to upgrade from ocata to pike07:11
korean101i install kolla-ansible 4.0.007:11
korean101and prechecks fails07:11
korean101"The conditional check ''169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none' failed. The error was: The ipaddr filter requires python-netaddr be installed on the ansible controller"07:11
*** chas has quit IRC07:12
korean101i already succeed to deploy pike release via kolla-ansible07:12
korean101same globals.yaml07:12
korean101but can't deploy pike release07:12
korean101am i missing something?07:12
*** kiennt26 has joined #openstack-kolla07:14
kolla-slack<egonzalez> korean101 error says a missing library in the deploy host07:15
*** danardelean has quit IRC07:15
korean101egonzalez: i already bootstrap-servers07:16
korean101egonzalez: and manually install ( yum install python-netaddr)07:16
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Install python-openstackclient in kolla-toolbox container  https://review.openstack.org/51997207:17
korean101egonzalez: Moreover, there was no problem in pike release.07:17
*** kelv_ has quit IRC07:18
korean101egonzalez: nothings about upgrade documents07:18
kolla-slack<egonzalez> korean101 ensure your ansible version meets pike requirements07:20
kolla-slack<egonzalez> Min version changed from ocata to pike07:20
korean101egonzalez: oh...07:21
*** zhangfei has quit IRC07:21
korean101egonzalez: and i already deploy pike release07:21
*** mdnadeem has quit IRC07:23
*** lpetrut_ has joined #openstack-kolla07:24
lxdongWhat do you mean already deploy pike release yet try to upgrade from ocata to pike?07:26
lvdombrkrmorning folks someone have use ipv6 prefix_delegation in kolla ?07:26
lxdongWhy do you use ipv6 prefix_delegation ?07:27
*** zhubingbing__ has quit IRC07:27
korean101lxdong: i wanna rolling upgrade new openstack releases07:29
korean101lxdong: use kolla-ansible07:29
egonzalezlvdombrkr, what driver are you using for ipv6 dhcp?07:29
korean101lxdong: 1. deploy ocata release and create 10 VMs07:29
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993907:29
korean101lxdong: 2. upgrade to pike release and keep 10 VMs07:30
korean101lxdong: i wanna live upgrade use kolla-ansible07:30
korean101https://www.openstack.org/assets/presentation-media/20170505-ZeroDowntimeUpgradeWithKolla.2.16.9.pdf07:30
*** danardelean has joined #openstack-kolla07:30
*** danardel_ has joined #openstack-kolla07:32
egonzalezkorean101, what steps are you doing? when is failing?07:32
*** zhubingbing__ has joined #openstack-kolla07:33
*** mdnadeem has joined #openstack-kolla07:33
egonzalezkorean101, what versions? 4.0.?, 5.0.? or stable/<branch> git?07:34
*** danardelean has quit IRC07:35
*** danardel_ has quit IRC07:37
*** zhangfei has joined #openstack-kolla07:37
*** salv-orlando has quit IRC07:39
*** salv-orlando has joined #openstack-kolla07:39
korean101egonzalez: git checkout stable/ocata07:40
korean101egonzalez: pip install kolla-ansible/07:40
korean101egonzalez: kolla-ansible (4.0.3.dev62)07:40
korean101and bootstrap-servers [OK]07:41
korean101prechecks [FAIL]07:41
korean101egonzalez: but stable/pike --> all is [OK]07:41
korean101egonzalez: same globals.yaml07:41
egonzalezkorean101, ansible and jinja version? is installed python-netaddr in deploy host?07:42
korean101egonzalez:  1 minutes07:42
egonzalezerrors?07:42
*** chas has joined #openstack-kolla07:42
egonzalezimage version? own builds?07:43
*** salv-orlando has quit IRC07:43
korean101egonzalez: huh!07:43
*** lpetrut_ has quit IRC07:44
korean101egonzalez: must install python-netaddr in deployment hosts?07:44
korean101egonzalez: not only target hosts?07:44
egonzalezin deployment says the error07:44
korean101fatal: [172.18.0.11]: FAILED! => {"failed": true, "msg": "The conditional check ''169.254.' not in kolla_internal_vip_address and kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) is none' failed. The error was: The ipaddr filter requires python-netaddr be installed on the ansible controller"}07:44
korean101Jinja2 (2.8.1) ansible (2.3.2.0)07:44
korean101i use hub.docker.com (4.0.0) images07:45
korean101egonzalez: python-netaddr-0.7.18-1.el7.noarch (deployment hosts)07:45
korean101egonzalez: python-netaddr-0.7.5-7.el7.noarch (target hosts)07:45
lvdombrkregonzalez: mhmmm, what you mean by driver?07:46
egonzalezkorean101, did you installed requirements from git repo? pip install -r requirements.txt?07:47
egonzaleznetaddr is in requirements file07:47
egonzalezlvdombrkr, https://docs.openstack.org/ocata/networking-guide/config-ipv6.html#configuring-the-dibbler-server07:48
korean101egonzalez: yes07:48
lxdongAfter adding python-openstackclient to Dockerfile.j2 of kolla-toolbox it failed with empty result [] of docker exec kolla_toolbox openstack ... compute service list -f json --service nova-compute07:48
lxdongat TASK [nova : Waiting for nova-compute service up]07:48
*** pcaruana has joined #openstack-kolla07:51
*** mgoddard has joined #openstack-kolla07:53
korean101egonzalez: and some another bugs (https://bugs.launchpad.net/kolla-ansible/+bug/1665364)07:53
openstackLaunchpad bug 1665364 in kolla-ansible "kolla-ansible bootstrap-servers fails when api_interface is invalid" [Medium,Fix released] - Assigned to Mark Goddard (mgoddard)07:53
egonzalezkorean101, was fixed in pike07:54
korean101egonzalez: but i try ocata release07:54
korean101egonzalez: in ocata (kolla-ansible) same ERRORs07:55
*** lpetrut_ has joined #openstack-kolla07:55
korean101egonzalez: in pike, no bugs07:55
korean101egonzalez: in ocata, some bugs...07:56
korean101egonzalez: so hard to me07:56
*** zhangfei has quit IRC07:57
*** mandre has quit IRC07:59
*** hemna has quit IRC07:59
egonzalezkorean101, all your hosts have same interfaces name?08:00
korean101egonzalez: yes08:01
korean101egonzalez: OMG something wierd08:01
korean101egonzalez: pip uninstall netaddr and pip install -r requirements.txt08:01
korean101egonzalez: netaddr-0.7.18 --> netaddr-0.7.1908:01
korean101egonzalez: and that ERRORs dissappears08:02
korean101egonzalez: sorry my faultsw08:02
egonzalezkorean101, you can workaround the missing interface in deploy host by changing ``all`` to ``baremetal`` in https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/baremetal/tasks/pre-install.yml#L2408:02
korean101egonzalez: i patch this bugs (https://bugs.launchpad.net/kolla-ansible/+bug/1665364)08:03
openstackLaunchpad bug 1665364 in kolla-ansible "kolla-ansible bootstrap-servers fails when api_interface is invalid" [Medium,Fix released] - Assigned to Mark Goddard (mgoddard)08:03
egonzalezlxdong, check nova compute logs, rabbitmq running?08:03
*** hemna has joined #openstack-kolla08:04
*** mandre has joined #openstack-kolla08:05
korean101egonzalez: thanks for answer (and sorry about waste time because of my faults)08:05
lxdonghttp://paste.openstack.org/show/626331/08:05
korean101hmmmmmmmmm08:07
egonzalezcannot cherry-pick https://review.openstack.org/#/c/434940/ to ocata atm, if someone can do it :)08:07
korean101why ceilometer still use mongo backend in ocata release?08:07
egonzalezlxdong, what storage backend used for nova?08:07
korean101https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/ceilometer/tasks/precheck.yml#L2508:08
lxdongno error in nova-compute log ,rabbit running with ceph backend08:08
*** inc0 has quit IRC08:09
lxdongnova_backend_ceph: "yes"08:10
egonzalezkorean101, have diff database types for mongo, that task checks if using mongo as backend but is not enabled08:10
*** zhangfei has joined #openstack-kolla08:10
egonzalezlxdong, ceph is running properly? ceph -s , ceph health?08:11
korean101egonzalez: but enable_mongodb: "no" --> same prechecks errors08:11
lxdongHEALTH_WARN08:11
egonzalezkorean101, https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/group_vars/all.yml#L43508:11
egonzalezbut by default is mongo the databas type08:12
egonzalezerror is OK in this case08:12
lxdong health HEALTH_WARN             too many PGs per OSD (410 > max 300)08:12
korean101egonzalez: hmmmmmmm still default backend is mongodb......08:13
korean101egonzalez: sounds so difficult upgrade from mongodb to gnocchi08:13
korean101egonzalez: thanks08:13
egonzalezkorean101, mongo was still supported by ocata, we cannot change defaults from release to release as will break existing environments08:13
korean101egonzalez: yes! thanks!08:14
lxdongcame across this situation too when use external ceph.08:14
lxdongthough containerized ceph will work fine.08:15
lxdongthought08:16
*** jiriprox has joined #openstack-kolla08:16
lxdongIs adding  'python-openstackclient==3.12.0' to Dockerfile.j2 enough?08:17
korean101prechecks is [OK]08:17
egonzalezlxdong, yep08:19
*** lpetrut_ has quit IRC08:20
lxdongso what's the matter?08:20
lxdongIs there any possbility that it's a  sequensial bug of lack of python-openstackclient?08:22
*** magicboiz has joined #openstack-kolla08:22
*** zhurong has joined #openstack-kolla08:23
lxdongor version 3.12.0 matter08:23
*** tovin07_ has quit IRC08:25
egonzalezversion is to avoid running from latest version in stable branch08:26
lxdongI mean version 3.12.0 client may has some integrated issue with current release.08:28
lxdongIs 3.12.0 the version before?08:29
chasonhttps://review.openstack.org/#/c/432608/08:32
chasonegonzalez What can I do to push it forward? :) ^^^^^^08:33
lxdongwith own built images based ubuntu I came across this situable as well in the morning, by adding 'python-openstackclient==3.12.0' and failed at TASK [nova : Waiting for nova-compute service up]08:33
*** mgoddard has quit IRC08:33
egonzalezchason, will update the patch today08:34
*** zhangfei has quit IRC08:34
chasonegonzalez OK, thx!08:34
*** amoralej|off is now known as amoralej08:40
*** salv-orlando has joined #openstack-kolla08:40
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993908:42
*** danardelean has joined #openstack-kolla08:43
*** jbadiapa_ is now known as jbadiapa08:43
*** salv-orlando has quit IRC08:44
*** zhangfei has joined #openstack-kolla08:47
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993908:48
*** salv-orlando has joined #openstack-kolla08:55
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993908:57
*** Pavo has quit IRC08:59
*** jascott1 has quit IRC09:00
*** jascott1 has joined #openstack-kolla09:00
lxdongWhat's the difference between source and binary ,will binary be more stable?09:02
*** zhongshengping has joined #openstack-kolla09:02
*** gkadam has joined #openstack-kolla09:02
*** Pavo has joined #openstack-kolla09:03
*** jascott1 has quit IRC09:05
*** gfidente has joined #openstack-kolla09:06
*** shardy has joined #openstack-kolla09:07
*** dougsz has joined #openstack-kolla09:10
*** pratapagoutham has joined #openstack-kolla09:11
*** mgoddard has joined #openstack-kolla09:12
masbergood afternoon, I would like to setup external ceph on kolla, I guess i don't need to specify the storage nodes in the inventory file, is that right?09:12
*** chialiang_ has quit IRC09:13
*** lpetrut_ has joined #openstack-kolla09:20
*** chialiang_ has joined #openstack-kolla09:24
*** zhongshengping has quit IRC09:28
*** serlex has joined #openstack-kolla09:31
*** magicboiz has quit IRC09:39
*** mdnadeem has quit IRC09:49
*** yingjun has quit IRC09:55
*** duonghq has quit IRC09:58
lxdongright09:59
lxdongwhat version are you using masber10:00
*** mdnadeem has joined #openstack-kolla10:02
lxdongIs there anyone using kolla-kubernetes came across unable to start kubelet at the very beginning?10:04
*** strigazi has quit IRC10:05
*** strigazi has joined #openstack-kolla10:06
*** zhangfei has quit IRC10:18
*** daidv has quit IRC10:23
*** hieulq has quit IRC10:24
*** kiennt26 has quit IRC10:25
*** magicboiz has joined #openstack-kolla10:30
*** zhangfei has joined #openstack-kolla10:31
*** rmart04 has joined #openstack-kolla10:40
*** kiennt26 has joined #openstack-kolla10:41
*** daidv has joined #openstack-kolla10:41
*** hieulq has joined #openstack-kolla10:41
*** caowei has quit IRC10:42
*** kiennt26 has quit IRC10:51
*** daidv has quit IRC10:52
*** hieulq has quit IRC10:53
*** ktibi has joined #openstack-kolla10:57
*** magicboiz has quit IRC10:57
*** Pavo has quit IRC10:59
*** yingjun has joined #openstack-kolla11:01
*** yangyapeng has quit IRC11:01
ktibihi egonzalez11:01
ktibihttp://paste.openstack.org/show/626360/ You can check11:01
*** yangyapeng has joined #openstack-kolla11:01
*** zhangfei has quit IRC11:02
*** Pavo has joined #openstack-kolla11:03
egonzalezktibi, not related to your issue but will be a future issue11:03
egonzalezneutron_external_interface: "p-brbond0-ovs"11:03
egonzalezneutron_bridge_name: "brbond0-ovs"11:03
egonzalezinterfaces cannot have - dashes, ansible will evaluate them as underscores and fail11:04
*** magicboiz has joined #openstack-kolla11:04
*** yee37929 has quit IRC11:05
*** yee379 has joined #openstack-kolla11:05
egonzalezktibi, what IP address resolvs at cloud-interne.local domain?11:07
*** kiennt26 has joined #openstack-kolla11:09
*** ArminderSingh has quit IRC11:09
*** yangyapeng has quit IRC11:09
egonzalezat first glance, only thing i can think is domain resolvs to public VIP rather than internal11:09
*** daidv has joined #openstack-kolla11:09
*** hieulq has joined #openstack-kolla11:09
*** hieulq has quit IRC11:12
*** kiennt26 has quit IRC11:12
*** kiennt26 has joined #openstack-kolla11:12
*** nguyentrihai has joined #openstack-kolla11:12
*** daidv has quit IRC11:12
*** mrunge has quit IRC11:15
*** mgoddard has quit IRC11:15
*** ArminderSingh has joined #openstack-kolla11:16
*** kiennt26 has quit IRC11:20
*** yangyapeng has joined #openstack-kolla11:22
*** cah_link has quit IRC11:24
*** chialiang_ has quit IRC11:26
*** daidv has joined #openstack-kolla11:27
*** hieulq has joined #openstack-kolla11:27
*** hieulq has quit IRC11:30
*** daidv has quit IRC11:31
hrwmorning11:31
hrwcan someone look at https://review.openstack.org/#/c/516883/ patch to kolla-ansible? it fixes use with Debian11:31
*** magicboiz has quit IRC11:33
*** rmart04 has quit IRC11:36
hrwegonzalez: thanks11:36
*** mrunge has joined #openstack-kolla11:37
*** rmart04 has joined #openstack-kolla11:41
openstackgerritMerged openstack/kolla-ansible master: Fix ansible running issue on Debian  https://review.openstack.org/51688311:44
*** daidv has joined #openstack-kolla11:45
*** hieulq has joined #openstack-kolla11:45
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/50944311:46
lvdombrkrhello folks, after i run kolla-ansible prechecks i got this error: http://paste.openstack.org/raw/626367/11:51
lvdombrkrmaybe quick ideas?11:51
*** mgoddard has joined #openstack-kolla11:52
openstackgerritMarcin Juszkiewicz proposed openstack/kolla master: mongodb: handle "user already exists" error on Debian  https://review.openstack.org/51937111:56
egonzalezlvdombrkr, ansible version11:56
*** cah_link has joined #openstack-kolla11:56
hrwegonzalez: updated to your comments11:56
egonzalezlvdombrkr, that syntax is not working with ansible 2.3 or newer11:57
lvdombrkregonzalez: ansible 2.4.1.011:57
lvdombrkregonzalez: thanks i will downgrade11:57
egonzalezlvdombrkr, is ocata or newton?11:58
lvdombrkregonzalez: 4.0.0. tag seems Newton11:58
egonzalezlvdombrkr, is ocata, that was fixed in 4.0.211:59
*** coolsvap has joined #openstack-kolla12:05
lvdombrkregonzalez: thanks for assist12:05
*** pbourke has quit IRC12:07
*** magicboiz has joined #openstack-kolla12:08
*** pbourke has joined #openstack-kolla12:08
hrwnew macro created.12:09
*** janki is now known as janki|afk12:10
hrwdebian_fix_existing_user(package)12:12
hrwas we had same code twice and I added third one12:12
lxdongegonzalez I saw a bug reply from you ,it's a similar situation of mine . I used control, network, compute, storage in the same nodes  which used to work before.12:21
openstackgerritMarcin Juszkiewicz proposed openstack/kolla stable/pike: Add ansible sudo running for Debian family  https://review.openstack.org/52005812:21
lxdonghttps://bugs.launchpad.net/kolla-ansible/+bug/172139012:21
openstackLaunchpad bug 1721390 in kolla-ansible "nova-compute service don't go up" [Undecided,New]12:21
hrwthis is backport of one sudoers change12:21
openstackgerritMarcin Juszkiewicz proposed openstack/kolla-ansible stable/pike: Fix ansible running issue on Debian  https://review.openstack.org/52005912:22
*** yingjun has quit IRC12:27
*** janki|afk is now known as janki12:27
lxdongthe only difference is compute01 compute02 and compute[01:02]12:29
*** ansmith has quit IRC12:33
*** rhallisey has joined #openstack-kolla12:40
*** rhallisey_ has joined #openstack-kolla12:40
*** yingjun has joined #openstack-kolla12:42
*** inc0 has joined #openstack-kolla12:42
*** rhallisey has quit IRC12:44
*** jascott1 has joined #openstack-kolla12:53
*** jascott1 has quit IRC12:57
*** Pavo has quit IRC12:59
*** Pavo has joined #openstack-kolla13:03
*** athomas has quit IRC13:07
*** absubram has joined #openstack-kolla13:07
*** zshi_ has quit IRC13:08
*** absubram_ has joined #openstack-kolla13:09
*** absubram has quit IRC13:11
*** absubram_ is now known as absubram13:11
*** nguyentrihai has quit IRC13:16
*** pratapagoutham has quit IRC13:20
*** amoralej is now known as amoralej|lunch13:23
*** rfxn has joined #openstack-kolla13:25
ktibiegonzalez re, sorry for my late answer13:27
ktibiso for brbond0-ovs, no pb with 50 deployments13:28
*** Teck7 has quit IRC13:28
ktibiFOR "what IP address resolvs at cloud-interne.local domain? " resolve 172.16.7.113:28
ktibihi inc0, can you help me with your comment on https://review.openstack.org/#/c/519302/13:30
*** athomas has joined #openstack-kolla13:31
*** skramaja has quit IRC13:33
*** Teck7 has joined #openstack-kolla13:48
*** ansmith has joined #openstack-kolla13:48
lvdombrkrfolks, maybe some quick ideas why after deployment my dashboard looks like https://pasteboard.co/GTLwp1X.png13:50
rwellumlxdong: yes there is a known issue with kubectl on ubuntu I've seen lately and had to work around. Do you have any logs?13:51
*** rfxn has quit IRC13:52
*** mdnadeem has quit IRC13:52
hrwlvdombrkr: check are all css files loaded13:53
lvdombrkrhrw: thanks, how to do that?13:56
hrwlvdombrkr: in chrome I use developer tools13:59
*** yingjun has quit IRC13:59
*** dave-mccowan has joined #openstack-kolla14:01
*** chas_ has joined #openstack-kolla14:02
lvdombrkrhrw: there is no css loaded at all14:04
hrwlvdombrkr: then check was it served at all, is it in image14:04
hrwlvdombrkr: you may need rebuild of horizon image as it got fix for similar issue14:05
*** chas has quit IRC14:05
openstackgerritJeffrey Zhang proposed openstack/kolla stable/ocata: Moving jobs to kolla repo  https://review.openstack.org/51905214:09
openstackgerritJeffrey Zhang proposed openstack/kolla master: [DNM]test large timeout  https://review.openstack.org/52008314:10
ktibiIf someone have time to check that ==> http://paste.openstack.org/show/626360/   kolla use external IP I don't know why :/14:12
*** dave-mccowan has quit IRC14:13
*** jtriley has quit IRC14:13
*** goldyfruit has joined #openstack-kolla14:14
*** krtaylor has joined #openstack-kolla14:19
*** amoralej|lunch is now known as amoralej14:20
*** dave-mccowan has joined #openstack-kolla14:20
*** yingjun has joined #openstack-kolla14:20
*** dave-mcc_ has joined #openstack-kolla14:25
*** dave-mccowan has quit IRC14:25
*** cah_link has quit IRC14:31
*** goutham has joined #openstack-kolla14:37
ktibiegonzalez I found, this is because I try to change http ==> https, and kolla no update old endpoint, so openstack client use old endpoints14:38
ktibiand I found that : https://github.com/openstack/kolla-ansible/commit/953861bde1118a300b5e608442f07e9fb70dabe914:39
egonzalezktibi, different task, that one is not being executed yet when is failing for you14:40
*** jtriley has joined #openstack-kolla14:40
egonzalezkeystone bootstrap is using http (auth_url is ok) + internal_fqdn (this point to public IP which is wrong)14:41
egonzaleztry to ping -a cloud-interne.local and dig cloud-interne.local14:42
ktibihttp://paste.openstack.org/show/626389/14:42
egonzalezmaybe somewhere is pointing to wrong IP14:42
ktibino ping is good14:42
egonzalezetc/hosts, dns server14:42
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Remove the unnecessary "-"  https://review.openstack.org/52009214:42
pbourkeegonzalez: ever get the bottom of this error? “libvirtError: Activation of org.freedesktop.machine1 timed out”14:43
ktibiegonzalez os_keystone_role use url for AUTH but use url return by endpoint for tasks14:43
ktibilike all python client, I think14:44
egonzalezpbourke, yep, no idea what's the root cause, restart libvirt fix the issue sometimes :/14:44
ktibiegonzalez http://paste.openstack.org/show/626390/14:45
*** tvignaud has quit IRC14:45
*** tvignaud has joined #openstack-kolla14:46
egonzalezktibi, https://github.com/openstack/kolla-ansible/commit/953861bde1118a300b5e608442f07e9fb70dabe9#diff-396a88e08b0a80fde05a596751a7d37eL14 says the module to use admin endpoint14:47
ktibiyep but for my task https://github.com/openstack/kolla-ansible/blob/f977ee2970fcad7133678d8736710cc563bc7279/ansible/roles/keystone/tasks/register.yml#L1514:48
ktibihttps://github.com/openstack/kolla-ansible/blob/f977ee2970fcad7133678d8736710cc563bc7279/ansible/roles/keystone/defaults/main.yml#L7614:48
ktibihttps://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L32614:49
ktibimissing endpoint_type14:49
egonzalezktibi, found14:50
egonzalezhttps://review.openstack.org/#/c/509186/14:50
egonzalezyep14:50
egonzalezthats the fix14:50
egonzalezforgot to backport t opike14:50
ktibi:D14:50
*** yee379 has quit IRC14:51
egonzalezktibi, will do later this evening, in the mean time can cherry-pick to your local14:51
*** yee379 has joined #openstack-kolla14:51
*** serlex has left #openstack-kolla14:53
*** serlex has quit IRC14:53
*** danardelean has quit IRC14:53
lxdongrwellum ,logs are normal14:55
rwellumlxdong: have a meeting now but I'll check back in 1hr. We can look at it.14:56
*** danardelean has joined #openstack-kolla14:58
pbourkeegonzalez: rebooting compute node worked! :/14:58
egonzalezhavent found a root cause, randomly appears with no reason14:59
*** Pavo has quit IRC14:59
egonzalezhave to say it was aroung 6months not seeing such issue14:59
egonzalezis like libvirt cannot connect with dbus/machined services14:59
*** coolsvap has quit IRC15:00
*** rhallisey_ is now known as rhallisey15:02
*** Pavo has joined #openstack-kolla15:03
lxdongrwellum thanks for you reply, I would be in dream by that time :)15:06
*** yingjun has quit IRC15:07
*** Teck7 has quit IRC15:11
*** Teck7 has joined #openstack-kolla15:11
*** jamesbenson has joined #openstack-kolla15:13
*** goldyfruit has quit IRC15:13
*** janki has quit IRC15:13
*** hongbin has joined #openstack-kolla15:16
*** danardelean has quit IRC15:17
*** goldyfruit has joined #openstack-kolla15:18
*** danardelean has joined #openstack-kolla15:21
*** zhubingbing_ has joined #openstack-kolla15:25
*** dave-mcc_ has quit IRC15:38
*** Teck7 is now known as rfxn15:38
lxdongyes, rebooting compute node which is all-in-one node in my case seems solved the issue15:38
rfxnbeen running into issues with neutron_dhcp_agent on a dvr setup where neutron-ns-metadata-proxy in the container keeps reporting no space left on device and causing the proxy to die15:39
rfxnCRITICAL neutron [-] OSError: [Errno 28] No space left on device: '/var/lib/neutron/external/pids/ffc3d413-2951-4424-9c71-381623aef45e.pid'15:39
rfxnbut there is plenty of space15:39
rfxn:\15:39
rfxnany thoughts?15:39
rfxnive set a neutron.conf child process check to exit cleanly so that the container restarts itself as a dirty fix15:39
openstackgerritOliver Walsh proposed openstack/kolla master: Include required packages for libvirt SASL auth on CentOs/RHEL  https://review.openstack.org/52011415:45
*** duonghq has joined #openstack-kolla15:45
*** ntpttr_laptop has joined #openstack-kolla15:46
*** goldyfruit has quit IRC15:48
*** goutham has quit IRC15:51
*** danardelean has quit IRC15:52
*** magicboiz has quit IRC15:53
duonghqevening gúy15:53
duonghqguys15:53
ktibiegonzalez with cherrypick commit, deploy work but endpoint for keystone is not update15:54
inc0o/15:54
egonzalezktibi, that was only a workaround to remove v3 from keystone endpoints (only ocata->pike upgrade)15:55
egonzalezif your endpoints doesnt have v3, you are good :)15:55
*** danardelean has joined #openstack-kolla15:55
ktibiall public endpoint are  https://cloud.local:SERVICE_PORT15:55
*** jascott1 has joined #openstack-kolla15:56
egonzalezthats fine then15:56
ktibiexcept keystone http://172.16.5.1:500015:56
egonzaleznot have fqdn on it?15:56
ktibinop15:56
ktibiand http :/15:56
inc0meeting:)15:58
ktibiI can find $SERVICE_public_endpoint for all services15:58
ktibibut no keystone_public_endpoint15:58
ktibihi inc015:58
*** serlex has joined #openstack-kolla15:58
egonzalezktibi, is wrong, will push a fix later, update endpoints use wrong variable15:59
egonzalez:/15:59
*** egonzalez has quit IRC15:59
*** dave-mccowan has joined #openstack-kolla15:59
*** danardelean has quit IRC15:59
hrwmeeting now or next hour?16:00
*** jascott1 has quit IRC16:00
*** itlinux has joined #openstack-kolla16:00
*** egonzalez has joined #openstack-kolla16:01
*** coolsvap has joined #openstack-kolla16:01
*** magicboiz has joined #openstack-kolla16:01
*** dave-mccowan has quit IRC16:02
inc0hrw: now16:04
*** jdandrea has joined #openstack-kolla16:04
*** magicboiz has quit IRC16:06
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993916:06
*** rmart04 has quit IRC16:07
*** goldyfruit has joined #openstack-kolla16:08
*** egonzalez has quit IRC16:09
*** magicboiz has joined #openstack-kolla16:13
*** egonzalez has joined #openstack-kolla16:13
*** goutham has joined #openstack-kolla16:16
gouthamHi all is 5.0.0 a stable kolla version16:17
goutham??16:18
*** zhurong has quit IRC16:18
ktibigoutham yep16:19
gouthamBut when i try to deploy the fluentd container is always on restarting mode ??16:19
ktibibut use stable/pike if you want official branch16:19
ktibican you copy "docker logs fluentd16:20
gouthamhttps://hastebin.com/uririzinun.vbs16:20
gouthamyes16:20
egonzalezyep, is broken atm due fluentd version upgrade16:20
gouthamktibi: https://hastebin.com/ixutosexih.sql docker logs16:21
gouthamegonzalez: yes but it worked perfectly fine may be 2 weeks back now it is causing issue16:21
egonzalezSince last week is the issue16:22
egonzalezIs being worked now16:22
gouthamegonzalez: ok :) and one more thing no matter how much hard i debug if i use the stable branch i couldnt ping my vm's16:22
gouthamdid u ever encounter that issue ?16:23
egonzalezcan be for many causes, logs from neutron, ovs and console log will help16:24
gouthamegonzalez: one more thing in 5.0.0 the task TASK [nova : Waiting for nova-compute service up] is failing16:24
gouthamegonzalez: https://hastebin.com/qapuracuxa.vbs16:25
jdandreaNaive question: Are there production-ready containers for Pike? (New to kolla, pardon naivete, reading docs now...)16:26
jdandreaLooking at https://docs.openstack.org/kolla/pike/image-building.html16:26
jdandreaBut are there Docker images out there, or are folks expected to build their own using those instructions.16:27
ktibijdandrea I build my own images, centos binary pike https://hub.docker.com/u/ktibi/16:27
jdandreaktibi thx, excellent! (I'm ISO Ubuntu 14/16-friendly stuffs.)16:28
egonzalezjdandrea, today build your own, next weeks will be in DH16:28
jdandreaegonzalez That's where I was looking. I didn't see anything in DH. I could also just be lousy at searching. :)16:29
egonzalezThere are for newton and older, an automatic publisher will do the jobs16:30
jdandreaegonzalez Right, ok, that makes sense. I'm also looking at OpenStack-Helm (which may or may not support Ocata/Pike; checking on that).16:30
egonzalezNo idea about o-helm, we have a shiny k8s based deployer :)16:32
ktibiegonzalez Issue with skydive which use public URL and no trust CA selft signe https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/skydive/templates/skydive-agent.conf.j2#L1716:33
ktibiI think we need to use admin or private endpoint for that (no SSL)16:33
ktibido you agree ?16:33
*** zhubingbing_ has quit IRC16:34
*** unicell has joined #openstack-kolla16:35
*** danardelean has joined #openstack-kolla16:36
*** unicell1 has quit IRC16:37
jdandreaegonzalez Point taken. Hooray for shiny k8s deployers!16:38
*** mrunge_ has joined #openstack-kolla16:39
*** mrunge has quit IRC16:39
*** danardelean has quit IRC16:44
openstackgerritChason Chan proposed openstack/kolla-ansible master: Fix the deprecated syntax in 01-rewrite.conf  https://review.openstack.org/51849016:52
openstackgerritMerged openstack/kolla master: Install python-openstackclient in kolla-toolbox container  https://review.openstack.org/51959216:54
Jeffrey4linc0, https://review.openstack.org/519939  pull docker hub image patch. it works as expect. except there are some issue in the current kolla-ansible code.16:58
*** Pavo has quit IRC16:59
*** danardelean has joined #openstack-kolla16:59
*** jascott1 has joined #openstack-kolla16:59
*** egonzalez has quit IRC16:59
Jeffrey4lthere are several patch trying to fix the gate.16:59
inc0cool!16:59
Jeffrey4land i am not sure whether Depends-On works, now.16:59
inc0I'll check it out in a few - need to eat something and move to office17:00
inc0but I'll pick up this work17:00
inc0and review/fix17:00
Jeffrey4lok, cool17:00
inc0thanks Jeffrey4l and good night17:00
Jeffrey4linc0, could you give a approve for https://review.openstack.org/51953817:00
*** dave-mccowan has joined #openstack-kolla17:00
rwellumgema: well drat - I guess my time changed and kolla meeting was one hour earlier....17:02
rwelluminc0: ^^ apologies for missing17:02
*** absubram has quit IRC17:03
inc0rwellum: daylight savings claimed another victim...17:03
Jeffrey4lrwellum, lol it is painful to have different timezone in a year ;(17:03
*** Pavo has joined #openstack-kolla17:03
*** krtaylor_ has joined #openstack-kolla17:04
* rwellum slinks off to the idiots lounge...17:04
Jeffrey4lany core member, please review https://review.openstack.org/51905217:04
Jeffrey4land https://review.openstack.org/51997117:05
duonghqJeffrey4l,  ocata doesn't need periodic job?17:05
Jeffrey4lduonghq, it need. but it is in another patch.17:06
duonghqokay17:06
gouthamHi all is the fluentd container problem there in stable pike also r8?17:06
Jeffrey4lwe have several gate related patch merged in master. and it is painful to track the patches and backport them. ;(17:06
Jeffrey4lgoutham, i am afraid yes.17:07
gouthamI see that the fluentd is still restarting17:07
Jeffrey4lit installed the wrong packages. we are fixing it.17:08
gouthamHi Jeffrey4l if i can checkout to ocata and try deploying will i get the same error even on that?17:08
Jeffrey4lnot sure. maybe..17:08
gouthamor is it with pike only i dont know the code base so asking please dont mind :)17:08
Jeffrey4lhttps://review.openstack.org/518490   https://review.openstack.org/518820  these two patches are trying fix the issue.17:08
*** gfhellma has joined #openstack-kolla17:09
Jeffrey4lgoutham, are you using ubuntu?17:09
gouthamYes i use ubuntu and one more thing how does this fluentd issue effect my setup ???17:09
chasonhttps://review.openstack.org/#/q/topic:bp/optimize-the-documentation-format+(status:open+OR+status:merged)17:09
gouthamjust asking because i dont know :)17:10
gouthamJeffrey4l: ^17:10
chasonCore guys, really need you review these two patch. If you approve, I will start the kolla-ansible and kolla-k8s doc works. There is much more doc work needs to do and many new stuff can be imported, thanks! ^^^^^^^ :)17:10
Jeffrey4laha, cherry-pick these two patch, it should fix your issue.17:10
gouthamJeffrey4l: how does this fluentd issue effect my setup ???17:11
gouthamjust asking17:11
Jeffrey4lgoutham, check the commit message.17:11
gouthamok  :')17:11
Jeffrey4lgonna to sleep, here is the 1am.17:11
duonghqbye Jeffrey4l17:11
chasonJeffrey4l Good night! :p17:12
Jeffrey4lbye everyone ;)17:12
*** duonghq has quit IRC17:13
*** dardelean_ has quit IRC17:13
*** zhubingbing__ has quit IRC17:14
rwellumchason: one has unanswered comments, the other a merge conflict?17:15
*** masber has quit IRC17:15
*** goldyfruit has quit IRC17:15
chasonrwellum Both comments' answer is nope. :)17:16
rwellumchason: then you have to submit a new PS with the changes?17:17
*** ntpttr_laptop has quit IRC17:18
chasonrwellum I am solving the other one.17:18
*** zhubingbing__ has joined #openstack-kolla17:18
rwellumchason: just pointing out I typically won't review when I'm expecting another PS based on comments or issues already there.17:20
*** danardelean has quit IRC17:23
gouthamHi all nova doesnt seem working in kolla 5.0.1 because when i do nova console-log <instance-id> it gives response once17:27
gouthamand if i try immediately17:27
gouthamit say instance not found any idea?17:27
*** ktibi has quit IRC17:28
*** serlex has quit IRC17:28
jamesbensonmasber: ping17:29
jamesbensonjdandrea: we run 4.0.2 for production here... had issues with 5.0.0, so rolled back until 5.0.1 releases17:32
*** rmart04 has joined #openstack-kolla17:34
*** gfhellma_ has joined #openstack-kolla17:34
*** rmart04 has quit IRC17:37
*** gkadam has quit IRC17:37
*** gfhellma has quit IRC17:37
openstackgerritChason Chan proposed openstack/kolla master: Upgrade the rst convention of the Admin Guide  https://review.openstack.org/51165917:39
*** goutham has quit IRC17:39
*** jascott1 has quit IRC17:40
*** jascott1 has joined #openstack-kolla17:40
*** jascott1 has quit IRC17:45
*** unicell has quit IRC17:47
*** shardy has quit IRC17:48
*** krtaylor_ has quit IRC17:48
*** lpetrut_ has quit IRC17:51
rwelluminc0: you around?17:52
*** rhallisey has quit IRC17:58
*** magicboiz has quit IRC18:00
*** dougsz has quit IRC18:01
*** mgoddard has quit IRC18:09
inc0rwellum: now I am18:10
rwellumgot an interesting error with kolla build - wondering if you've seen it before?18:11
rwellumhttps://www.irccloud.com/pastebin/4g0R287u/18:11
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/50944318:11
*** pcaruana has quit IRC18:11
inc0hmm18:12
inc0no, but do pip freeze18:12
rwellumhttps://www.irccloud.com/pastebin/cst7aclA/18:12
inc0which version of kolla you're building?18:12
rwellumPike18:13
rwellumstable18:13
inc0ok so afair it's already docker not docker-py18:13
inc0(they changed libs, it's both import docker and they're not compatible...)18:14
*** jascott1 has joined #openstack-kolla18:15
*** danardelean has joined #openstack-kolla18:19
*** gfhellma__ has joined #openstack-kolla18:20
*** chas_ has quit IRC18:20
rwelluminc0: do you saying my setup is correct? It should be as I did a virtual enc, and then pip install -r requirements.txt18:21
rwellumso not do18:21
inc0yeah, I'm comparing versions now18:21
inc0you have newer docker package18:21
inc0try runniung pip install docker==2.5.118:23
inc0maybe docker broke their lib yet again18:23
rwellumAck will do.18:23
*** rhallisey has joined #openstack-kolla18:23
*** unicell has joined #openstack-kolla18:24
*** gfhellma_ has quit IRC18:24
*** mrunge_ is now known as mrunge18:34
*** mgoddard has joined #openstack-kolla18:36
mgoddardNova bug related to ironic: https://bugs.launchpad.net/nova/+bug/1732506. Should I mark it as 'also affects ironic'? Or is there another way to ensure the ironic team remains aware of it?18:37
openstackLaunchpad bug 1732506 in OpenStack Compute (nova) "Baremetal instance stuck in BUILD state following ironic node tear down or delete" [Undecided,New]18:37
*** ntpttr_laptop has joined #openstack-kolla18:38
rwelluminc0: I think pip is not cooperating18:38
rwellumhttps://www.irccloud.com/pastebin/elm7ElbC/18:38
*** lpetrut_ has joined #openstack-kolla18:39
*** coolsvap has quit IRC18:40
*** iniazi_ has quit IRC18:40
mgoddardSorry folks, wrong channel...18:40
*** goldyfruit has joined #openstack-kolla18:41
*** ntpttr_laptop has quit IRC18:45
*** chas_ has joined #openstack-kolla18:46
*** jascott1 has quit IRC18:47
*** jascott1 has joined #openstack-kolla18:48
inc0rwellum: hmm18:48
inc0try pip uninstall docker18:49
inc0ahh18:49
inc0I know18:49
inc0pip install -U docker==2...18:49
inc02.5.118:49
inc0add -U flag18:49
rwellumhttps://www.irccloud.com/pastebin/XTSbyATy/18:51
rwellumIt's weird - it installed18:51
rwellumAh...18:52
rwellumhttps://www.irccloud.com/pastebin/N52jfrny/18:52
rwellumsudo...18:52
*** itlinux has quit IRC18:54
inc0:D18:54
*** TxGirlGeek has joined #openstack-kolla18:55
*** TxGirlGeek has quit IRC18:55
*** TxGirlGeek has joined #openstack-kolla18:56
rwelluminc0: new failure - maybe my env is now in bad state due to previous error? https://www.irccloud.com/pastebin/AC1s7Kmd/18:56
inc0hmf, check syslog18:57
*** TxGirlGeek has quit IRC18:58
*** Pavo has quit IRC18:59
*** TxGirlGeek has joined #openstack-kolla18:59
*** TxGirlGeek has quit IRC18:59
*** TxGirlGeek has joined #openstack-kolla19:02
*** TxGirlGeek has quit IRC19:02
*** TxGirlGeek has joined #openstack-kolla19:02
inc0rwellum: progress?19:02
rwelluminc0: just trawling through the log - I see several issues but not sure root cause yet19:03
inc0ehh19:03
inc0which version of docker you're using19:03
*** Pavo has joined #openstack-kolla19:03
inc0?19:03
rwellumhttps://www.irccloud.com/pastebin/YhlEA6Cq/19:05
rwellumSeems too high...19:05
inc0no, that's fine19:06
inc01.13 should work well19:06
rwellumoh ok19:07
inc0try kolla-build --no-cache19:07
rwellumyeah tried - same19:07
inc0damn19:07
*** goldyfruit has quit IRC19:17
*** goldyfruit has joined #openstack-kolla19:18
*** rwsu has quit IRC19:24
rwelluminc0: systemctl stop docker.service && systemctl start docker.service - seems to have unblocked me :S19:25
*** TxGirlGeek has quit IRC19:26
inc0there ya go;)19:26
inc0so we need to pin version of docker in requirements it seems19:26
*** itlinux has joined #openstack-kolla19:30
openstackgerritMerged openstack/kolla stable/ocata: Moving jobs to kolla repo  https://review.openstack.org/51905219:32
openstackgerritMerged openstack/kolla stable/pike: Install python-openstackclient in kolla-toolbox container  https://review.openstack.org/51997119:32
*** TxGirlGeek has joined #openstack-kolla19:34
rwelluminc0: certainly looks like it - I can add a PS if you want.19:35
rwelluminc0: now I am getting push permission errors - assuming I am not authenticating to dockerhub19:35
*** TxGirlGeek has quit IRC19:43
*** TxGirlGeek has joined #openstack-kolla19:44
*** mgoddard has quit IRC19:50
*** TxGirlGeek has quit IRC19:53
*** salv-orl_ has joined #openstack-kolla19:55
*** salv-orlando has quit IRC19:55
*** TxGirlGeek has joined #openstack-kolla19:59
*** amoralej is now known as amoralej|off19:59
*** TxGirlGeek has quit IRC20:00
*** gfidente has quit IRC20:02
openstackgerritBertrand Lallau proposed openstack/kolla-ansible master: Implement Almanach ansible role  https://review.openstack.org/49894420:02
*** itlinux has quit IRC20:19
*** jiriprox has quit IRC20:20
*** hachi_ has joined #openstack-kolla20:30
*** jbadiapa_ has joined #openstack-kolla20:33
*** goldyfruit has quit IRC20:34
*** rwsu has joined #openstack-kolla20:35
*** jbadiapa has quit IRC20:36
*** itlinux has joined #openstack-kolla20:40
*** Pavo has quit IRC20:47
*** hachi_ has quit IRC21:00
*** itlinux has quit IRC21:00
*** TxGirlGeek has joined #openstack-kolla21:00
*** TxGirlGeek has quit IRC21:04
*** livelace has quit IRC21:08
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993921:13
*** Pavo has joined #openstack-kolla21:17
*** itlinux has joined #openstack-kolla21:19
*** livelace has joined #openstack-kolla21:21
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993921:26
*** threestrands has joined #openstack-kolla21:32
*** threestrands has quit IRC21:32
*** threestrands has joined #openstack-kolla21:32
*** ansmith has quit IRC21:35
*** jtriley has quit IRC21:35
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277921:41
*** itlinux has quit IRC21:42
*** rwsu has quit IRC21:47
*** lpetrut_ has quit IRC21:53
*** jascott1 has quit IRC21:58
*** jascott1 has joined #openstack-kolla21:59
*** jascott1 has quit IRC21:59
*** jascott1 has joined #openstack-kolla22:02
*** jascott1 has quit IRC22:02
*** jascott1 has joined #openstack-kolla22:03
*** rwsu has joined #openstack-kolla22:08
*** itlinux has joined #openstack-kolla22:11
*** rwsu has quit IRC22:14
*** rhallisey has quit IRC22:14
*** llua has joined #openstack-kolla22:15
*** danardelean has quit IRC22:22
*** itlinux has quit IRC22:23
*** danardelean has joined #openstack-kolla22:23
*** rwsu has joined #openstack-kolla22:25
*** danardelean has quit IRC22:28
*** jascott1 has quit IRC22:36
*** jascott1 has joined #openstack-kolla22:37
*** jascott1 has quit IRC22:40
*** jascott1 has joined #openstack-kolla22:43
*** itlinux has joined #openstack-kolla22:44
*** ssurana has joined #openstack-kolla22:45
*** ssurana has quit IRC22:45
*** jascott1 has quit IRC22:45
*** jascott1 has joined #openstack-kolla22:48
*** jascott1 has quit IRC22:53
*** jascott1 has joined #openstack-kolla22:55
*** jascott1_ has joined #openstack-kolla22:58
*** jamesbenson has quit IRC22:59
*** jascott1 has quit IRC22:59
*** jascott1_ has quit IRC23:00
*** jascott1 has joined #openstack-kolla23:00
*** rwsu has quit IRC23:01
*** chas_ has quit IRC23:01
*** chas_ has joined #openstack-kolla23:03
*** jascott1 has quit IRC23:05
*** chas_ has quit IRC23:08
*** itlinux has quit IRC23:15
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: [WIP] pull image from dockerhub  https://review.openstack.org/51993923:16
*** Pavo has quit IRC23:17
*** dave-mccowan has quit IRC23:19
*** Pavo has joined #openstack-kolla23:19
*** masber has joined #openstack-kolla23:30
masbergood morning, I am following the documentation in regards how to deploy openstack using kolla-ansible and external ceph cluster23:32
masberhttps://docs.openstack.org/kolla-ansible/latest/reference/external-ceph-guide.html23:33
masberdo I need to specify the storage nodes in globals.yml file?23:33
inc0yes23:33
inc0but at least one23:33
inc0doesn't have to be osd ones23:33
inc0thing is, storage ndoes is where your cinder-volume lands23:33
inc0and if you specify none, you won't have cinder-volume23:34
inc0and you want cinder-volume;)23:34
inc0you can do sth like [storage:children] control23:34
inc0that'll be fine23:34
*** jtriley has joined #openstack-kolla23:34
inc0masber: mind creating doc patch with this? jamesbenson hit that issue as well23:34
inc0which means it's not clear23:34
masberyes, I can do that, happy to help23:36
inc0thank you23:36
masbernp23:36
masberI got a lot of help from this team so this is the least I can do23:36
inc0haha, it's great feeling to give some back, I always like when I land a patch23:37
inc0let me know if you need any help with gerrit or anything, if you're unfamiliar with it23:38
masberyes, how should I start? do I need to read the contribute/developers section23:38
inc0https://docs.openstack.org/infra/manual/developers.html <- this doc describes process well23:39
inc0feel free to hit me if you have any problems in it23:39
masberwill do23:40
masberanother question, is kolla-ansible going to create the volumes (images, backup, volume, etc) on ceph? or do I need to create them manually before deployment?23:46
SamYaplemasber: you mean pools?23:47
masberyes pools sorry23:48
masberSamYaple, ^^^^23:53
SamYapleif you arent doing external ceph, yes we create the pools23:55
masberSamYaple, ok, I am doing external ceph, shall I create the pools myself then?23:56

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!