Monday, 2017-10-23

*** salv-orlando has quit IRC00:07
*** salv-orlando has joined #openstack-kolla00:08
*** yangyape_ has quit IRC00:17
*** yangyapeng has joined #openstack-kolla00:18
*** lukepatrick has joined #openstack-kolla00:18
*** yangyapeng has quit IRC00:22
*** Pavo has quit IRC00:31
*** hieulq has joined #openstack-kolla00:43
*** huanxie has joined #openstack-kolla00:54
*** nguyentrihai has joined #openstack-kolla01:04
*** lukepatrick has quit IRC01:06
*** salv-orl_ has joined #openstack-kolla01:07
*** salv-orlando has quit IRC01:10
*** kevzha01 has joined #openstack-kolla01:14
*** kevzha01 is now known as kevinz01:14
*** jinke_ has joined #openstack-kolla01:14
*** genek has quit IRC01:16
*** yangyapeng has joined #openstack-kolla01:20
*** tovin07_ has joined #openstack-kolla01:27
*** zhubingbing_ has joined #openstack-kolla01:44
*** caowei has joined #openstack-kolla02:02
*** zhangfei has joined #openstack-kolla02:07
*** kiennt26 has joined #openstack-kolla02:08
*** xinliang has quit IRC02:23
*** salv-orl_ has quit IRC02:25
*** salv-orlando has joined #openstack-kolla02:25
*** xinliang has joined #openstack-kolla02:35
*** xinliang has joined #openstack-kolla02:35
*** salv-orlando has quit IRC02:43
*** ArminderSingh- has quit IRC02:44
*** salv-orlando has joined #openstack-kolla02:44
*** masber has quit IRC02:47
*** ArminderSingh has joined #openstack-kolla02:48
*** duonghq has joined #openstack-kolla02:52
*** dave-mccowan has quit IRC03:05
*** hieulq has quit IRC03:23
*** hieulq has joined #openstack-kolla03:23
*** masber has joined #openstack-kolla03:33
masbergood afternoon, I am having this error while deploying kolla-ansible. fatal: [test-openstack-controller]: FAILED! => {"changed": false, "elapsed": 301, "failed": true, "msg": "Timeout when waiting for 192.168.20.5:3306"}03:34
masber any thoughts?03:34
*** gkadam has joined #openstack-kolla03:35
*** gkadam_ has joined #openstack-kolla03:40
*** gkadam has quit IRC03:41
*** lukepatrick has joined #openstack-kolla03:56
*** mdnadeem has joined #openstack-kolla03:59
*** janki has joined #openstack-kolla04:22
kolla-slack<lxdong.007> How to find macros.install_packages() , we need to customize Horizon.04:28
*** skramaja has joined #openstack-kolla04:39
openstackgerritHeetae Ahn proposed openstack/kolla master: Add RHEL subscription registration  https://review.openstack.org/49514804:41
openstackgerritJeffrey Zhang proposed openstack/kolla master: implement build jobs in kolla  https://review.openstack.org/51397104:58
*** jaosorior has joined #openstack-kolla05:02
*** lukepatrick has quit IRC05:03
*** pratapagoutham has quit IRC05:07
*** pratapagoutham has joined #openstack-kolla05:24
pratapagouthamHi all this is with reference to this link to deploy openstack-kolla multinode "https://docs.openstack.org/project-deploy-guide/kolla-ansible/ocata/multinode.html#deploy-a-registry"05:26
openstackgerritJeffrey Zhang proposed openstack/kolla master: enable bifrost in oraclelinux+source  https://review.openstack.org/50694505:28
pratapagouthamshould we also do pip install kolla after doing pip install kolla-ansible ??05:29
*** pratapagoutham has left #openstack-kolla05:33
*** pratapagoutham has joined #openstack-kolla05:33
kolla-slack<lxdong.007> How to customize source of horizon image building?Like we have a customized directory override the original.05:36
*** yangyapeng has quit IRC05:36
*** yangyapeng has joined #openstack-kolla05:36
pratapagouthamhi all when i execute tools/start-registry why am i getting this error "Could not reach any registry endpoint"  "?? any help ?05:40
*** yangyapeng has quit IRC05:41
*** danardelean has joined #openstack-kolla05:57
*** danardelean has quit IRC05:57
*** dardelean has joined #openstack-kolla06:16
*** danardelean has joined #openstack-kolla06:16
*** danardelean has quit IRC06:21
pratapagouthamHi all i have a doubt is there way where we can specify the services required to be installed?06:21
*** magicboiz has joined #openstack-kolla06:21
pratapagouthamfor eg: say i dont want all the services i just want only  nova, neutron, glance  and ceilometer to be installed can i do it if yes how to do that???06:22
pratapagouthamany help ?06:22
pratapagouthamthanks in advance :)06:23
*** cah_link has joined #openstack-kolla06:25
*** magicboiz has quit IRC06:26
*** cah_link has quit IRC06:29
*** magicboiz has joined #openstack-kolla06:35
*** athomas has joined #openstack-kolla06:41
*** hrw has quit IRC06:44
*** tvignaud has joined #openstack-kolla06:45
*** cah_link has joined #openstack-kolla06:47
openstackgerritJeffrey Zhang proposed openstack/kolla master: implement build jobs in kolla  https://review.openstack.org/51397106:47
*** danardelean has joined #openstack-kolla06:51
*** hrw has joined #openstack-kolla06:59
*** pcaruana has joined #openstack-kolla07:00
*** serlex has joined #openstack-kolla07:02
*** jmccarthy has joined #openstack-kolla07:18
*** zhangfei has quit IRC07:18
masberpratapagoutham, yes you have to specify the services you want to install in the globals.yml file07:26
*** jistr is now known as jistr|mtgs07:26
bjolomorning07:28
masberpratapagoutham, use this document for kolla-ansible deployment on multiple nodes https://docs.openstack.org/kolla-ansible/latest/user/multinode.html07:28
masberpratapagoutham, you can also use this document as a reference https://docs.openstack.org/kolla-ansible/latest/reference/ceph-guide.html#simple-3-node-example07:29
*** zhangfei has joined #openstack-kolla07:32
openstackgerrityanpeifei proposed openstack/kolla master: update "mitaka" to "ocata"  https://review.openstack.org/51167007:35
openstackgerrityanpeifei proposed openstack/kolla master: update "opendaylight" tar "distribution-karaf-0.6.2-Carbon.tar.gz" has the latest features.  https://review.openstack.org/51417507:35
*** salv-orlando has quit IRC07:50
*** salv-orlando has joined #openstack-kolla07:50
openstackgerrityanpeifei proposed openstack/kolla master: fix "openvswitch-base-plugin-dpdk" url.  https://review.openstack.org/51417807:51
hrwmorning07:53
*** dougsz has joined #openstack-kolla07:54
*** salv-orl_ has joined #openstack-kolla07:55
*** salv-orlando has quit IRC07:58
kolla-slack<aslancimbom100> Morning08:02
spsuryasup all08:04
spsuryaduonghq: sup08:04
spsuryamorning hrw08:04
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Optimize zuulv2 jobs  https://review.openstack.org/51361208:11
*** Zophar has joined #openstack-kolla08:14
ZopharGood morning08:15
ZopharCould we go ahead with this one: https://review.openstack.org/#/c/509792/2 ?08:15
ZopharI want to push a new feature on Horizon for image sources... but i need this one to be push in master and stable first08:16
hrwZophar: make a patch and use "Depends-on: CHANGEID" in commit08:18
ZopharHum.... the point is that my actual patch https://review.openstack.org/#/c/509784/ will need a rebase after this one....08:19
ZopharI'll will update it once 509792 is merged08:22
duonghqspsurya,  hi08:22
pratapagouthamthanq masber :)08:27
magicboizmorning08:42
magicboizanyone can check this? https://bugs.launchpad.net/kolla-ansible/+bug/172315508:42
openstackLaunchpad bug 1723155 in kolla-ansible "No valid host was found error" [Undecided,Incomplete]08:42
openstackgerritMerged openstack/kolla master: Update Image Builder Links in Readme  https://review.openstack.org/51270208:47
openstackgerritMerged openstack/kolla master: Add EC2-API to Kolla providing images list  https://review.openstack.org/51046608:47
magicboizis there any difference on Ceph version installed with kolla-ansible/Pike from Centos to Ubuntu/debian??09:01
*** sambetts|afk is now known as sambetts09:15
*** jiriprox has joined #openstack-kolla09:17
flaper87hey guys, have you started migrating kolla CI to zuul? I checked last week and it didn't seem like it but I wanted to double check if you had.09:18
flaper87I'd like to take a look and see if there are some things we can share :)09:19
flaper87kfox1111: inc0 ^09:19
openstackgerritzhongshengping proposed openstack/kolla-ansible stable/pike: Add ipc_mode=host for nova-compute conainer  https://review.openstack.org/50702809:21
*** janki has quit IRC09:21
openstackgerritzhongshengping proposed openstack/kolla-ansible stable/ocata: Add ipc_mode=host for nova-compute conainer  https://review.openstack.org/50702909:21
*** zhangfei has quit IRC09:25
pratapagouthamHi while running the prechecks im getting this error https://hastebin.com/owaxorucir.tex can anyone guide me ?? thanks in advance09:35
*** salv-orl_ has quit IRC09:36
*** salv-orlando has joined #openstack-kolla09:37
pbourkeflaper87: pretty sure its done, change merged late last week09:37
openstackgerritJeffrey Zhang proposed openstack/kolla master: implement build jobs in kolla  https://review.openstack.org/51397109:37
*** zhangfei has joined #openstack-kolla09:38
pbourkepratapagoutham: please cross check your ansible version with https://docs.openstack.org/kolla-ansible/latest/user/quickstart.html#install-dependencies09:38
pratapagouthammy version is 2.0.0 but still it is throwing up this error09:41
*** salv-orlando has quit IRC09:41
flaper87kolla-ansible or kolla-kubernetes?09:46
flaper87pbourke: ^09:46
pbourkeflaper87: kolla-ansible, unsure about kolla-k8s09:46
flaper87gotcha, thanks09:49
*** zhangfei has quit IRC09:51
openstackgerritzenghui.shi proposed openstack/kolla master: Add ptp service docker image  https://review.openstack.org/51370309:52
openstackgerritDuong Ha-Quang proposed openstack/kolla master: Add lbaas dashboard in horizon ubuntu binary  https://review.openstack.org/50819109:58
*** ioni has joined #openstack-kolla09:59
openstackgerrityanpeifei proposed openstack/kolla master: update "copy_links:true" ARG.   synchronize options of the "copy_link" required "yes".  https://review.openstack.org/51420210:00
ionihello guys10:00
ionii want to install pike and i have an error10:00
ionihttps://paste.xinu.at/s2p/10:00
ionimanifest for kolla/ubuntu-binary-fluentd:5.0.0 not found10:00
ionii do see that the tag for fluentd doesn't exists upstream10:00
ionii'm not sure how to resolve this problem10:01
ionihttps://hub.docker.com/r/kolla/ubuntu-binary-fluentd/tags/10:01
openstackgerritDuong Ha-Quang proposed openstack/kolla master: nova-libvirt: fix kvm permission issue  https://review.openstack.org/50165310:01
*** paramite has joined #openstack-kolla10:04
*** kiennt26 has quit IRC10:05
*** jinke_ has quit IRC10:08
*** zhangfei has joined #openstack-kolla10:08
*** jinke_ has joined #openstack-kolla10:08
*** magicboiz has quit IRC10:10
*** calbers has joined #openstack-kolla10:11
hrwioni: I do not know did they got uploaded at all (:5.0.0 images)10:12
*** tovin07_ has quit IRC10:12
ionihrw, right. what's the best way to deploy 5.0 using kolla?10:13
hrwioni: spend 2-3h and build them locally?10:13
hrwioni: with local registry deploy goes quick10:14
*** magicboiz has joined #openstack-kolla10:15
openstackgerritJames McCarthy proposed openstack/kolla master: Create murano applications package for import  https://review.openstack.org/51363210:17
*** magicboiz has quit IRC10:19
*** caowei has quit IRC10:20
*** magicboiz has joined #openstack-kolla10:26
ionihrw, ok, thanks10:29
*** pbourke has quit IRC10:37
*** pbourke has joined #openstack-kolla10:38
*** yangyapeng has joined #openstack-kolla10:38
*** janki has joined #openstack-kolla10:40
*** yangyapeng has quit IRC10:43
*** yangyapeng has joined #openstack-kolla10:44
*** zhubingbing has joined #openstack-kolla10:51
*** zhubingbing_ has quit IRC10:51
*** zhubingbing has quit IRC10:58
openstackgerritOpenStack Proposal Bot proposed openstack/kolla-kubernetes master: Updated from global requirements  https://review.openstack.org/50944311:01
*** sean-k-mooney has joined #openstack-kolla11:03
*** zhangfei has quit IRC11:03
*** dave-mccowan has joined #openstack-kolla11:04
pratapagouthamHi all can anyone help me with this please??11:34
pratapagouthamhttps://askubuntu.com/questions/967990/upgrade-docker-installed-via-pip11:34
pratapagouthamplease11:34
pratapagouthamonlything left out in my installation11:34
*** jiriprox has quit IRC11:38
*** jiriprox has joined #openstack-kolla11:38
*** lpetrut has joined #openstack-kolla11:39
*** erlon has joined #openstack-kolla11:42
hrwpratapagoutham: check (with strace) where from it loaded docker11:43
hrwyou may have python-docker installed systemwide while pip installed into your home directory.11:44
*** janki has quit IRC11:44
hrwpython checks system dirs and then user one11:44
hrwimport sys11:45
hrwprint sys.path11:45
hrwand then check11:45
pratapagouthamyes strangly why is it taking this ??? " /home/kolla/.local/lib/python2.7/site-packages/docker/__init__.py"11:46
pratapagouthamshould i delete that docker directory in that location hrw ??11:47
hrwpratapagoutham: if you do not understand how it happened then I suggest you to use python-virtualenv instead of randomly playing with pip11:47
pratapagouthamok but can i delete that directory in that location??11:47
hrwpratapagoutham: you used pip as a user. so it stored in user homedir11:47
pratapagouthamoh ok11:47
hrwremove /home/kolla/.local/lib/python2.7/ and use virtualenv11:48
hrwthat way you will have separate python installation in which you have permissions11:48
pratapagouthamhrw:  thanks it worked :) thankyou11:48
hrwyw11:49
pratapagouthamyes yes i will do it from now on :)11:49
hrwI was in your situation few months ago ;D11:49
pratapagouthamhaha :D11:53
pratapagouthami now get it11:53
openstackgerritChen proposed openstack/kolla master: Update links in README.rst  https://review.openstack.org/51085412:01
*** jmccarthy has left #openstack-kolla12:03
*** salv-orlando has joined #openstack-kolla12:04
*** huanxie has quit IRC12:16
rwellumhrw: ioni we have the tarballs correct? That's what I've been using for Pike/Master12:17
ionirwellum, i didn't use source, just binary12:18
ionii kinda failed to use local registry, i have them all built but cannot be used by ansible12:18
rwellumI think these are binary - just tarballed up.12:19
ionirwellum, i don't know then, all that failed(binary) don't have tags12:19
rwellumhttp://tarballs.openstack.org/kolla/images/12:19
ionion hub docker12:19
ionimaybe i failed to read the documentation somehow12:20
rwellumRight that's correct, images post ocata are not on docker hub until infra steps are done12:20
rwellumtarballs an alternative - not sure if kolla-ansible folks are using12:21
ionirwellum, ok, until know i used openstack ansible12:21
ionibut i wanted something fast and tried kolla12:21
*** gnupyx has joined #openstack-kolla12:32
hrwrwellum: do not know. I usually use aarch64 images so have to build them anyway12:36
*** pcaruana has quit IRC12:41
pratapagouthamHi all i have been trying to deploy multi-node openstack-kolla in the precheck it shows this error https://hastebin.com/yuhimudapo.php any help??12:43
*** gkadam_ has quit IRC12:44
pratapagouthamhrw: any idea on this??12:44
openstackgerritMerged openstack/kolla master: Add lbaas dashboard in horizon ubuntu binary  https://review.openstack.org/50819112:46
openstackgerritMerged openstack/kolla stable/pike: Bump OpenStack service version  https://review.openstack.org/51360512:47
*** huanxie has joined #openstack-kolla12:56
*** pcaruana has joined #openstack-kolla12:57
*** ktibi has joined #openstack-kolla12:58
*** pcaruana has quit IRC13:01
*** jtriley has quit IRC13:07
kolla-slack<aslancimbom100> pratapagoutham what is running on the IP/port which is currently inuse?13:12
kolla-slack<aslancimbom100> if you are running local registry, then re-create it with another port, ie 544313:13
pratapagouthamkolla-slack:   https://hastebin.com/ocanucezav.vbs13:13
*** salv-orl_ has joined #openstack-kolla13:15
pratapagouthamkolla-slack:  i changed that to 400013:15
pratapagouthamhow to check on what port my regsitry is running??13:16
*** salv-orlando has quit IRC13:16
*** salv-orl_ has quit IRC13:17
*** salv-orlando has joined #openstack-kolla13:18
pratapagouthamkolla-slack: https://hastebin.com/jadebizipu.nginx13:19
pratapagouthami think this is a bit clear .13:19
*** jtriley has joined #openstack-kolla13:21
*** xinliang has quit IRC13:23
*** jtriley has quit IRC13:26
*** jinke_ has quit IRC13:27
*** huanxie has quit IRC13:27
*** goldyfruit has joined #openstack-kolla13:32
kolla-slack<aslancimbom100> You need to change the registry container to run on different port, do you not have the command in your history?13:33
kolla-slack<aslancimbom100> You have changed the client to accept registry running on port 400013:34
*** cah_link has quit IRC13:37
pratapagouthamkolla-slack: thanks it worked13:37
pratapagouthamwhile deploying im getting this error " https://hastebin.com/vefijivahi.tex "13:37
*** mdnadeem has quit IRC13:39
kolla-slack<aslancimbom100> ok so that fails because you haven't changed the registry container port13:41
kolla-slack<aslancimbom100> do "docker ps -a" is there a registry container?13:41
pratapagouthamhttps://hastebin.com/ehevuyocos.rb13:42
pratapagouthamyes i redirected it to 544313:42
pratapagouthamdonno why 5000,tcp is still there .. :(13:43
*** salv-orl_ has joined #openstack-kolla13:44
*** salv-orlando has quit IRC13:45
*** jtriley has joined #openstack-kolla13:46
*** jiriprox has quit IRC13:47
*** jiriprox has joined #openstack-kolla13:47
*** yangyapeng has quit IRC13:50
pratapagouthamhttps://hastebin.com/uvoqoseloq.pas any help why im getting this error ???13:50
*** yangyapeng has joined #openstack-kolla13:51
kolla-slack<aslancimbom100> what is the output of docker ps -a13:51
pratapagouthamhttps://hastebin.com/exuyazowem.nginx13:52
pratapagouthamkolla-slack: the above one is the output13:52
pratapagouthamkolla-build --base ubuntu --type source --registry 172.16.69.71:5443 --push I tried this and i got error like this "https://hastebin.com/uvoqoseloq.pas"13:55
pratapagouthamany help on how to do ??? :'( please13:55
*** yangyapeng has quit IRC13:55
*** Zophar has quit IRC13:55
*** skramaja has quit IRC13:56
*** hongbin has joined #openstack-kolla13:56
*** hrw has quit IRC13:59
*** hrw has joined #openstack-kolla14:01
*** Zophar has joined #openstack-kolla14:04
*** jaosorior has quit IRC14:08
*** hachi_ has joined #openstack-kolla14:18
*** salv-orl_ has quit IRC14:20
*** salv-orlando has joined #openstack-kolla14:20
*** david-lyle has joined #openstack-kolla14:21
kolla-slack<aslancimbom100> hey, can you show me the command you use to create the registry14:31
*** zhangfei has joined #openstack-kolla14:34
pratapagouthamhttps://hastebin.com/nifezekizu.bash14:36
pratapagouthamkolla-slack: https://hastebin.com/nifezekizu.bash this is the command14:36
openstackgerritcaoyuan proposed openstack/kolla-ansible master: Add the api works num for watcher  https://review.openstack.org/51434014:45
*** jiriprox has quit IRC14:50
pratapagouthamkolla-slack: ping14:53
*** zhangfei has quit IRC14:53
*** jaosorior has joined #openstack-kolla14:57
*** hachi_ has quit IRC14:58
*** hachi_ has joined #openstack-kolla14:58
*** yangyapeng has joined #openstack-kolla15:03
*** jtriley_ has joined #openstack-kolla15:03
*** yangyapeng has quit IRC15:04
*** yangyapeng has joined #openstack-kolla15:04
*** jtriley has quit IRC15:06
*** lukepatrick has joined #openstack-kolla15:10
*** hachi_ has quit IRC15:19
*** salv-orlando has quit IRC15:19
*** salv-orlando has joined #openstack-kolla15:20
kfox1111morning.15:33
ktibiHi all, I try to deploy infra with kolla. For now, all works except the vxlan network. I can create VM, console VNC works, all services works15:45
ktibisomeone have same bug I guess https://ask.openstack.org/en/question/110570/vxlan-peers-not-being-created-on-compute-node-vm-not-getting-dhcp/15:45
ktibiI use stable/pike15:45
ktibiI don't know why but the creation of vxlan port on br-tun is not called :/15:46
*** rmart04 has joined #openstack-kolla15:49
kolla-slack<aslancimbom100> @pratapagoutham try something like this "docker run -d -p 5443:5000..."16:06
*** erlon has left #openstack-kolla16:09
*** lebauce has quit IRC16:11
inc0good morning16:16
inc0ktibi: ansible?16:16
ktibiinc0 yep16:16
inc0let me read through this bug, is it accurate description of what you're seeing?16:17
ktibihum, in my link, user can create vxlan port with restart container. for me no works :/16:17
inc0you mean network itself isn't created?16:18
ktibiyou can see my conf here : http://paste.openstack.org/show/624372/16:18
*** itlinux has joined #openstack-kolla16:18
inc0this is all in one?16:18
ktibiI can create network, I can create VM, I can console in my VM, dhcp and VM port is up in br-int16:18
ktibibut vxlan port in br-tun are not created16:19
ktibinop 1 ctrl 1 compute16:19
inc0what's your globals.yml? and check /var/lib/docker/volumes/kolla_logs on hosts for logs16:19
ktibiall service with debug=true but no error16:19
ktibiIt looks like some openvswitch flows are not being applied16:20
inc0to me worrisome is that you have 127.0.0.1 as tunnel endpoints..16:21
ktibiinc0 globals.yml http://paste.openstack.org/show/624390/16:23
inc0right, so tunnels should default to brbond0.128716:25
ktibiinc0 neutron try to insert endpoint vxlan16:25
ktibiEndpoint with ip 172.16.7.3 already exists: DBDuplicateEntry: (pymysql.err.IntegrityError) (1062, u"Duplicate entry '172.16.7.3' for key 'PRIMARY'")16:26
ktibiyes brbond0.128716:26
inc0hmm interesting16:26
inc0seems like bloated db?16:26
ktibithey can ping themselves16:26
ktibithe end : [SQL: u'INSERT INTO ml2_vxlan_endpoints (ip_address, udp_port, host) VALUES (%(ip_address)s, %(udp_port)s, %(host)s)'] [parameters: {'host': u'compute01', 'ip_address': u'172.16.7.3', 'udp_port': 4789}]16:27
ktibiit's just a warning16:27
inc0right, that's what is issue16:27
inc0still seems to be problematic16:28
ktibihttp://paste.openstack.org/show/624391/16:28
inc0can you clean up your env and try to redeploy?16:29
ktibialready try16:29
inc0did you run kolla-ansible destroy?16:30
ktibidestroy all, container, volumes16:30
ktibiyep16:30
ktibidestroy and deploy16:30
*** lpetrut has quit IRC16:30
inc0ehh dunno, but this endpoirt in db seems to me what's broken16:32
ktibierror come when I restart neutron agent on compute16:32
ktibiI'll try to delete line in DB and restart agent16:32
ktibierror is in neutron-server.log on ctrl16:33
ktibiinc0 no error but vxlan port not crreated16:35
*** jaosorior has quit IRC16:35
*** jaosorior has joined #openstack-kolla16:36
*** tvignaud has quit IRC16:39
*** jmccarthy has joined #openstack-kolla16:39
inc0ktibi: uhh I don't know:/16:44
ktibiyeah me too ^^16:44
ktibiBut with the bug on ask, i'am not alone :D16:44
inc0right, it might be good to ask neutron people too16:45
ktibiyeah I tried16:45
inc0hmm, try to do kolla-ansible destroy, reboot all machines16:47
inc0and deploy fresh16:47
*** rmart04 has quit IRC16:49
ktibiok16:50
*** serlex has quit IRC16:58
*** salv-orlando has quit IRC17:09
*** salv-orlando has joined #openstack-kolla17:10
*** salv-orlando has quit IRC17:14
*** lpetrut has joined #openstack-kolla17:18
*** sambetts is now known as sambetts|afk17:23
*** sambetts has joined #openstack-kolla17:24
*** lpetrut has quit IRC17:29
*** danardelean has quit IRC17:33
*** danardelean has joined #openstack-kolla17:34
*** danardelean has quit IRC17:38
*** jmccarthy has left #openstack-kolla17:42
*** vhosakot has joined #openstack-kolla17:47
*** danardelean has joined #openstack-kolla18:08
*** danardelean has quit IRC18:12
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277918:16
*** aagate has joined #openstack-kolla18:18
iniazimaybe a stupid question, but what network (kolla-ansible config wise) would live migration use?  api_network?18:23
inc0not a stupid question and yes, api_iface18:25
inc0you'd need to hack your way around nova configuration out of this18:26
*** rwsu has quit IRC18:29
iniazino thats fine.  i'm trying to figure out (and add additional) 1gbe and 10gbe interfaces.  i'm kind of limited to 4 interfaces per node (trying to be also cheap, using off-lease equipemnt etc.)18:31
iniazi2x 1gbe, and 2x10gbe is easy.  but i'm trying to figure out how I'd configure kolla-ansible so it effectively uses those ports.18:32
inc0one way would be to use bonding across 10gbe18:33
inc0and vlan + quos18:33
inc0qos18:33
*** lukepatrick has quit IRC18:33
iniazion the bond, send ceph + api_iface?18:33
inc0well if you configure qos for each, you'd be safe18:34
inc0and you want bonding anyway for HA18:34
iniazii'll have 2 network nodes, 2 controller, and 4 computes.  network nodes will actually have 6 interfaces, 4x1gbe, and 2x10gbe.  but the other nodes I'm kind of limited to 4 total ports18:35
inc0but also depends on your planned usecase - if it's all pet vms, you want good lm network18:35
inc0if it's cattle, you might not need lm at all18:35
iniaziits mostly web servers, and then some internal db/elasticsearch and light hadoop type of clusters18:36
inc0well it's always a judgement call18:37
iniaziyeah.  i think i have plenty of bandwidth.18:37
iniaziso besides ceph & api subnet, there would be external/public ip network per node, and then the last 1gbe... for tunnel? or stick that on 10gbe too?18:40
inc0tunnel usually is heaviest one tho18:40
iniazioh need one for internal non-routable subnet18:40
inc0it's vm-to-vm communication18:40
inc0non routable is for external network in neutron18:41
inc0technically it's only required in network node unless you use DVR18:41
iniazii was thinking of using dvr.  going to configure it and see how it works.18:42
inc0then you need neutron_external_interface for every node18:42
iniaziso fi i have dvr, i'd need 1 interface for public and 1 for internal subnet (non-ha)18:43
inc0aka compute nodes gonna external network access18:43
iniaziyeah18:44
*** dardelean_ has joined #openstack-kolla18:45
iniazion some nodes I can have 4x1gbe (i.e. most of the compute nodes) so I can put bonded 1gbe on those for neutron net and then 1 bond for internal public18:45
iniaziwell its really these two nodes i have that i'm running into 4 interface limitation (2x1gbe and 2x10gbe nic on pcie)18:45
iniaziI can just use those for controller only.18:46
iniaziinc0: thanks a bunch.  I think I know what to buy/upgrade18:47
*** dardelean has quit IRC18:48
inc0any time, designing cables is always fun ;)18:48
*** lukepatrick has joined #openstack-kolla18:48
*** jgriffith has quit IRC18:51
*** jgriffith_ has joined #openstack-kolla18:52
*** itlinux has quit IRC18:52
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job  https://review.openstack.org/51385518:52
*** harlowja has joined #openstack-kolla18:53
*** jamesbenson has joined #openstack-kolla18:54
*** itlinux has joined #openstack-kolla18:55
*** jascott1 has joined #openstack-kolla18:58
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job  https://review.openstack.org/51385519:02
jascott1inc0 did you stash any Ubuntu specific notes from yesterday?19:02
inc0jascott1: nothing much to it tbh19:03
SamYapleo/ jascott119:03
jascott1whatup SamYaple!19:03
inc0only difference was that I didn't add any ceph repos19:03
inc0just ran apt-get install ceph-deploy19:03
jascott1ok im working by that etherpad19:03
jascott1want to make it centos and ubuntu19:04
inc0I'm juggling few things, but let me know if you need any help19:04
jascott1k thanks19:04
*** jgriffith_ has quit IRC19:06
*** jgriffith has joined #openstack-kolla19:10
*** jgriffith has quit IRC19:11
*** jgriffith has joined #openstack-kolla19:11
*** jgriffith has quit IRC19:11
*** jgriffith has joined #openstack-kolla19:15
kfox1111jascott1: cool. let me know if you run into any issues.19:18
kfox1111I threw it together pretty quickly, so there is probably a bump or two in the road somewhere.19:18
*** jinke_ has joined #openstack-kolla19:23
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job  https://review.openstack.org/51385519:26
jamesbensonjascott1 Can I ask what's going on with ceph/working on?19:27
*** jinke_ has quit IRC19:27
*** lukepatrick has quit IRC19:28
*** hachi_ has joined #openstack-kolla19:33
inc0jamesbenson: yesterday we made little hackathon for ceph+k8s19:36
*** jgriffith has quit IRC19:37
jamesbensonoh nice!19:37
*** newmember has quit IRC19:38
*** jgriffith has joined #openstack-kolla19:38
*** itlinux has quit IRC19:39
jamesbensonI have 2 questions... if anyone is free to help.  Networking/cinder :-)19:39
inc0ask away19:41
jamesbensonI have DVR setup and I can ping my router, but can't access/ping my instances.  I usually don't do DVR, so I'm not sure if this is a config I messed up or something else...19:43
inc0tcpdump your way through network stack to see where ping gets lost?19:44
jamesbenson(with DVR you should have L3 agents,  metadata-agents, and openvswitch-agents on all of your compute, correct?)19:45
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277919:47
jamesbensoncinder: booting with volume always fails, is there a way to either get cinder working properly so this works, or disable it by default so it wont always have that option enabled?19:48
kfox1111jascott1: ping19:53
jascott1kfox1111 pong19:53
kfox1111jascott1: quick go question.19:54
kfox1111and my go's rusty.19:54
jascott1"rusty" ha19:54
jascott1go on :)19:54
kfox1111say I wanted to have a list which contains hashes. whats the prototype for that?19:54
kfox1111ie:19:54
kfox1111foo:19:54
kfox1111  - key: value19:54
jascott1the values are hashes or?19:55
kfox1111key and value are both strings.19:55
kfox1111but each list entry could have abritrary numbers of unique key/values.19:55
kfox1111so is that []map[string]string ?19:56
jascott1map[string][]string19:57
kfox1111so its inner type -> outer type order?19:57
jascott1map["foo"] = []string{key:value}19:57
jascott1^ just for illustration19:57
jascott1map["foo"] = []string{"key:value"}19:58
jascott1so would really be19:58
kfox1111oh.I think thats:19:58
kfox1111foo:19:58
jascott1map[string][]string{"mykey1:myval1","mykey2:myval2"}19:58
kfox1111  - "key: value"19:59
jascott1if you want the key and value not to be a string you have to split19:59
kfox1111which isn't the same as - key: value  (at least in yaml)19:59
jascott1then you would have to use something there19:59
kfox1111lets switch to json as it may be more clear...19:59
jascott1yeah if it wasnt a list it then you would use nested ma20:00
jascott1map20:00
kfox1111foo: [{"key":"value"}]20:00
kfox1111and I wanted to parse foo's data type.20:00
jascott1thats a map not a string20:00
kfox1111yes.20:00
jascott1then its a map[string]interface20:00
kfox1111so, foo = []map[string]string20:01
jascott1map[string][]map[string]interface20:01
kfox1111k. let me try that. thanks.20:01
jascott1a map with key strings, that containts a slice of maps taking string key and having interface value20:01
kfox1111I think we're up a level. I just need the decoder for foo itself, not the thing containing foo.20:02
jascott1ah20:02
kfox1111like in the strict definition:  Foo map[string]interface `json:"foo" yaml:"foo"`20:03
jascott1value of foo is still a slice of map[string]interface20:03
jascott1then you have to check the type for interface :( but thats the way its done20:04
*** bmace has joined #openstack-kolla20:06
kfox1111hmm. ok.20:06
jascott1its all over helm, im trying to find one20:06
sbezverkkfox1111: as soon as api changed gets merged, I will push changes for rbd external provisioner to support namespace..20:08
kfox1111sbezverk: which api change?20:08
sbezverkkfox1111: rbdvolumesource which was used for persistent volume, did not support user's secret name space20:09
sbezverkit is using LocalObject reference20:10
kfox1111hmm....20:10
kfox1111can a pod be launched by a user with a reference to a secret in another namespace anyway?20:10
sbezverknew API rbdpersistentvolumesource will use secret reference where you can specify secret name and namespace20:10
kfox1111I didn't think rbac would allow that.20:10
kfox1111hmm...20:10
sbezverkkfox1111: that part I am not sure20:10
kfox1111have a link for that?20:10
sbezverkneed to check the code20:11
sbezverksure20:11
kfox1111thx.20:11
sbezverkkfox1111: https://github.com/kubernetes/kubernetes/pull/54302/20:11
kfox1111cool. thanks.20:11
sbezverkso we do in tree first because of api change20:12
sbezverknext step out of tree provisioner20:12
kfox1111I'd definatly double check this against rbac enabled k8s. as that is increasingly common.20:13
kfox1111there may need to be an exception that allows a dynamically created pv's rbd secret to come from a different namespace whether the user has access to it or not.20:13
kfox1111I'm pretty sure it will be broken out of the box right now. :(20:13
kfox1111otherwise, I think a user could just put in any volum request for an rbd volume specifying any users creds.20:20
sbezverkrootfs just tested it in his test bed and it works20:21
*** hachi_ has quit IRC20:22
kfox1111with a non admin user?20:22
sbezverkyep20:22
kfox1111have him try just a bare pod with rbd volume directly specified.20:22
kfox1111that one shouldn't work.20:23
sbezverkthat will not work20:23
kfox1111are you sure?20:23
sbezverkdirect reference from pod uses different spec20:23
kfox1111ah.20:23
kfox1111so the ps allows it as its a different route and its under the control of the creater of the pv.20:24
kfox1111should be ok then.20:24
sbezverkright20:24
jamesbensoninc0: my instances can't ping the external gateway, but can ping google.com20:26
jamesbensonand outside of the cloud, I can ping the external gateway IP20:27
*** salv-orlando has joined #openstack-kolla20:33
*** lukepatrick has joined #openstack-kolla20:34
*** salv-orlando has quit IRC20:37
*** salv-orlando has joined #openstack-kolla20:37
*** lpetrut has joined #openstack-kolla20:45
*** jascott1_ has joined #openstack-kolla20:49
*** jascott1 has quit IRC20:49
*** ktibi has quit IRC20:52
*** aagate has quit IRC20:59
*** jgriffith has quit IRC20:59
*** jgriffith has joined #openstack-kolla21:00
*** danardelean has joined #openstack-kolla21:08
*** dardelean_ has quit IRC21:08
*** jtriley_ has quit IRC21:10
*** spiette has joined #openstack-kolla21:14
*** Pavo has joined #openstack-kolla21:15
*** lukepatrick has quit IRC21:17
*** lukepatrick has joined #openstack-kolla21:19
*** lukepatrick has quit IRC21:20
jamesbensoninc0: is it possible that this could be soemthing with iptables:21:22
jamesbenson?21:22
*** lukepatrick has joined #openstack-kolla21:22
*** dave-mccowan has quit IRC21:28
*** dougsz has quit IRC21:28
*** dougsz has joined #openstack-kolla21:29
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277921:31
*** dougsz has quit IRC21:33
*** MasterOfBugs has joined #openstack-kolla21:34
*** pramodrj07 has joined #openstack-kolla21:34
*** Pavo has quit IRC21:35
*** Pavo has joined #openstack-kolla21:37
*** jgriffith has quit IRC21:50
SamYapleinc0: did you figure out the mariadb issue?21:51
*** sambetts has quit IRC21:53
kolla-slack<inc0> Negative, trying few  things to confirm that net works21:54
*** lpetrut has quit IRC21:55
*** jgriffith has joined #openstack-kolla21:58
*** jamesbenson has quit IRC22:03
danardeleanhi guys22:07
danardeleanthere is a slack channel for kolla?22:07
masberdanardelean, yes it is a channel within the kubernetes.slack.com22:09
danardeleanty22:10
masbergood morning, I am having an issue deploying kolla, where the kolla_internal_vip_address is not created on the control/network host, as a consequence I am getting this error during deployment --> fatal: [test-openstack-controller]: FAILED! => {"changed": false, "elapsed": 301, "failed": true, "msg": "Timeout when waiting for 192.168.20.5:3306"}22:11
masbermy question is, on which interface should kolla_internal_vip_address live?22:12
*** Pavo has quit IRC22:13
masberthese are the details http://paste.openstack.org/raw/624412/22:19
*** salv-orlando has quit IRC22:34
*** salv-orlando has joined #openstack-kolla22:34
*** hrw has quit IRC22:38
*** salv-orlando has quit IRC22:39
*** vhosakot has quit IRC22:44
*** dave-mccowan has joined #openstack-kolla22:49
inc0hmm masber interesting, that might be same issue I'm experiencing in gates22:53
SamYapleinc0: im pretty sure it is. its the issue i was saying keeps popping up in the channel22:54
inc0masber: but you're saying IP doesn't appear on controller?22:55
inc0it should live in api interface22:55
inc0that would mean there is issue with keepalived22:55
masberinc0, correct, if you see the paste log, kolla_internal_vip_address: "192.168.20.5" is missing on the controller/network node22:56
masberinc0, yes I assumed same thing, the 192.168.20.5 belongs to the management segment assigned to the api_interface22:57
inc0masber: just to confirm it's keepalived issue, try putting kolla_internal_vip_address : << actual address of one of controllers >>22:57
inc0and enable_haproxy: false22:57
*** janki has joined #openstack-kolla22:57
inc0that would bypass all keepalived mechanisms and deployment should be successful22:57
masberinc0, ok, let me check22:57
inc0that would also explain why our previous gates works22:58
inc0or why my local environment worked22:59
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277923:01
*** jgriffith has quit IRC23:01
*** janki has quit IRC23:02
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode scenarios in zuulv3  https://review.openstack.org/51277923:03
*** jgriffith has joined #openstack-kolla23:08
*** danardelean has quit IRC23:14
*** danardelean has joined #openstack-kolla23:15
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: WIP Dockerhub publisher job  https://review.openstack.org/51385523:15
inc0masber: any progress?23:16
masberinc0, yes, it works23:16
inc0so keepalived is broken23:16
masberinc0, deployment is successful if haproxy is disabled23:16
inc0tomorrow I'll prep env for myself to enable keepalived and try to reproduce23:17
inc0this is critical23:17
inc0SamYaple: we both were right, it's not networking but it's not mariadb as well23:17
SamYapleheh23:17
SamYaplemariadb is just the first time the vip is used eh23:18
*** duonghq has quit IRC23:18
inc0right23:18
inc0masber: you're using ubuntu?23:18
masberin my case issue was that kolla_internal_vip_address was missing, the creating of that subinterface was part of the hapoxy?23:18
*** duonghq has joined #openstack-kolla23:18
masberinc0, I am using centos binary and host is centos 723:19
SamYapleinc0: you said you deployed multinode successfully, no?23:19
SamYapleby hand23:19
inc0masber: it's not about interface, keepalived adds new IP address to existing iface23:19
inc0SamYaple: yes, but without haproxy23:19
inc0I have it disabled locally because neutron address pairs23:19
SamYapleoh... well thats no multinode then23:20
*** danardelean has quit IRC23:20
inc0not full ha, but good for most of clustering tests23:20
inc0but yeah, I'll fix that up tomorrow23:20
SamYapleclearly not lol23:20
SamYaplebut yes this clears up the confusion23:20
inc0I'd wish you could just tell neutron to not care about address pairs -.-23:21
SamYapleinc0: you can23:21
inc0in configs?23:21
SamYapleya23:21
SamYapleor as admin on the cluster per port23:21
inc0I don't have acccess to this cloud:/23:21
SamYaplewell yea if you are just a user then that would be really bad to expose23:22
inc0I know23:22
SamYapleas a user, no. you get what youre given23:22
inc0even on your tenant network you created?23:22
SamYapleyes because you could still do evil things on the hypervisor otherwise23:23
inc0yeah I guess23:23
inc0well, I screwed:/23:23
SamYaplethis is why i vxlan in my cluster of hosts on clouds23:23
SamYaplefull l2 capabilities23:23
inc0yeah I might add this to my cluster too23:24
inc0Jeffrey4l: when you're awake, can you try to deploy multinode with master? Keepalived seems to be broken23:24
inc0I'll deal with this tomorrow morning when ops for my cloud comes to work23:25
SamYapleaccording to my logs, im pretty sure its been broken for the life of 5.0.0 too, just fyi23:25
SamYaplegive you a place to git blame23:25
inc0unlikely since people did deploy 5.0 already and with success23:25
SamYaplewith haproxy :)23:26
SamYaplecould just be a subset of distros that are borked23:26
*** dave-mccowan has quit IRC23:27
*** goldyfruit has quit IRC23:31
masberI would like to ask if I can use this guide https://docs.openstack.org/kolla-ansible/latest/reference/external-ceph-guide.html with ceph luminus?23:32
SamYaplemasber: kolla code doesnt have support for lumonous yet23:33
SamYapleso you would have to use external, but youll have to modify the sources to pull in loumous in the build23:34
masberSamYaple, yes, I mean deploy external ceph using ceph-ansible and connect it to openstack23:35
kolla-slack<britthouser> @masber you'll still need to pull luminus client into the kolla containers.23:37
SamYaplemasber: right im saying youll have to modify the sources.list or the centos stuff when you build the containers23:37
SamYaplefor the lumionous clients23:37
masberSamYaple, ok, yes the clients!23:38
SamYaplesorry, ill attempt to be more clear :)23:38
masbernp, thank you for your help!, I still remember you were the one who helped me deploying my first kolla environment23:39
*** hongbin has quit IRC23:39
SamYaplei remember that too23:40
*** jascott1_ has quit IRC23:44
*** jascott1 has joined #openstack-kolla23:44
*** lukepatrick has quit IRC23:44
SamYapleinc0: btw, i have a significantly better way (in my opinoin) t odeploy ceph now23:45
inc0which is?23:45
*** jascott1 has quit IRC23:45
SamYapleit involves creating the ceph keys and saving them as passwords. they are generic not-linked-to-the-cluster base64 encoded 16 byte strings23:46
SamYaplethat way we dont have to do alot of the fetch distribute dances23:46
SamYapleand they really do act like password23:46
SamYapleto change kolla to it though would require some hands on migration23:46
SamYaplebecause you would need ot update teh secrets23:47
SamYaplewith teh existing secrets from teh existing clusters23:47
SamYaplebut basically it lets you bring up new nodes or new clusters at the same time23:47
SamYapleceph can work itself out23:47
SamYapleno more monmap copying or anything like that23:47
SamYapleim doing it in salt now, but i basically start all the tasks at once. osds. mons. mgrs.23:48
SamYapledoesnt matter if its a new cluster or not23:48
SamYaplei suppose you could build that into the kolla-ansible command23:49
SamYaplelike a fetch-ceph-secrets thing23:49
inc0probably, yes23:49
SamYaplethen use that output to update secrets.yml23:49
SamYaplebut anyway. its way better and easier to work with23:49
SamYaplealot less ordering (read: none)23:50
inc0although we're thinking of dropping ceph in general23:50
SamYapledo you recall me saying "im not sure if ceph should like in kolla since its basically like another project"23:50
SamYaplei stand by that :P23:50
SamYaples/like/live/23:51
inc0we'll upgrade to L now and probably deprecate it alltogether or at least externalise ceph deployment23:51
SamYapleif you provide a migration path to ceph-docker that would be best23:51
SamYaplenot the best images... but at least you could galaxy in the ceph-ansible playbooks23:51
inc0it's really cool to have simple auto-orchestration of cinder and such23:52
inc0so it's going to be fun, but we'll have whole next release for that23:52
inc0I'm worried about upgrade tho23:53
SamYaplei was happy with my ceph work in ansible there, but its not standard, ceph-ansible is standard23:53
SamYaplemoving to that would be best23:54
inc0yeah, I agree, ideally we'd have migration path from one to another23:54
inc0only reason I'm hesitant is that people already have ceph done by us and I don't want them to be screwed23:54
SamYaplea bit of parted magic to recombobulate the drives and youll be fine23:55
SamYaplei brought my home cluster from bare-metal -> kolla-ceph -> baremetal -> ceph-salt23:55
SamYapleso its doable for sure23:55
inc0maybe it's documentation issue23:57
inc0we will upgrade to L this release, but maybe deprecate it too23:58
SamYapledocumentation and a playbook to assist and itll be easy23:58
SamYapleyour bigger problem will be doing the ceph-ansible integration23:58
SamYaplebecause you arent using any outside galaxy roles now23:59
SamYapleso thats a whole other piece oyu gotta work out23:59
inc0yup23:59

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!