Thursday, 2017-07-20

*** schwicht has quit IRC00:05
*** tonanhngo has joined #openstack-kolla00:06
*** yingjun has joined #openstack-kolla00:08
*** tonanhngo has quit IRC00:10
*** jamesbenson has joined #openstack-kolla00:15
*** tonanhngo has joined #openstack-kolla00:15
*** rhallisey has quit IRC00:15
*** jamesbenson has quit IRC00:19
*** tonanhngo has quit IRC00:20
*** masber has joined #openstack-kolla00:23
*** lucasxu has joined #openstack-kolla00:24
*** awiddersheim has joined #openstack-kolla00:30
*** awidders_ has quit IRC00:30
masbergood morning, I am trying to configure ironic using kolla-ansible, according to ironic's docs the property scheduler_use_baremetal_filters=True, scheduler_tracks_instance_changes=False and scheduler_host_subset_size=9999999 needs to be on nova.conf, however they are missing from /etc/kolla/nova-compute-ironic/nova.conf, is that normal?00:31
*** zhurong has joined #openstack-kolla00:32
*** jamesbenson has joined #openstack-kolla00:36
*** vhosakot has quit IRC00:37
*** lucasxu has quit IRC00:39
*** jamesbenson has quit IRC00:40
*** vhosakot has joined #openstack-kolla00:41
*** mdnadeem has joined #openstack-kolla00:41
*** lucasxu has joined #openstack-kolla00:43
*** sbezverk has joined #openstack-kolla00:49
*** lucasxu has quit IRC00:53
*** lucasxu has joined #openstack-kolla00:54
*** ansmith has quit IRC00:58
emccormickThanks sdake. I will take a look at them. If RDO would simply release packages in a timely manner I wouldn't even have to worry abou thtis ;)01:03
sdakemasber ironic has lacked maintenance for a cuple cycles (in kolla-ansible) - kolla-kubernetes is known to work01:04
sdakemasber for your specific problem, you might ask sambetts|afk when he isnt |AFK01:04
sdakehe is in UK tz01:05
masbersdake, thanks! I have a lot of questions for him :)01:05
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add Swift backend support to Glance  https://review.openstack.org/45205901:07
*** pramodrj07 has quit IRC01:08
*** MasterOfBugs has quit IRC01:08
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add vmware support to ceilometer  https://review.openstack.org/46284201:12
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add vSphere support to nova_compute  https://review.openstack.org/44952901:16
*** kevzha01 has joined #openstack-kolla01:17
*** jtriley has joined #openstack-kolla01:21
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add vmware DVS support to kolla-ansible  https://review.openstack.org/45927001:21
*** manheim has joined #openstack-kolla01:28
*** lucasxu has quit IRC01:31
*** harlowja has quit IRC01:32
*** manheim has quit IRC01:32
*** mdnadeem has quit IRC01:41
*** mdnadeem has joined #openstack-kolla01:41
*** awiddersheim has quit IRC01:41
*** mdnadeem is now known as mdnadeem|afk01:41
*** awiddersheim has joined #openstack-kolla01:42
*** myoung has quit IRC01:42
*** caowei has joined #openstack-kolla01:44
*** mdnadeem|afk has quit IRC01:46
openstackgerritChason Chan proposed openstack/kolla-ansible master: Update the documentation link for doc migration  https://review.openstack.org/48516101:49
*** jamesbenson has joined #openstack-kolla01:59
*** mdnadeem has joined #openstack-kolla01:59
*** ducttap__ has quit IRC02:02
*** jamesbenson has quit IRC02:03
*** vhosakot has quit IRC02:09
*** jtriley has quit IRC02:17
*** ducttape_ has joined #openstack-kolla02:25
*** ducttape_ has quit IRC02:30
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426202:32
*** ddyer has quit IRC02:32
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426202:34
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426202:36
*** ddyer has joined #openstack-kolla02:38
openstackgerritChason Chan proposed openstack/kolla-kubernetes master: Update the documentation link for doc migration  https://review.openstack.org/48541102:39
*** jamesbenson has joined #openstack-kolla02:40
*** jamesbenson has quit IRC02:44
*** harlowja has joined #openstack-kolla02:52
openstackgerritSerguei Bezverkhi proposed openstack/kolla master: Fix incorrect tarballs naming  https://review.openstack.org/48535702:54
*** schwicht has joined #openstack-kolla03:05
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426203:12
*** caowei has quit IRC03:16
*** awidders_ has joined #openstack-kolla03:18
*** awiddersheim has quit IRC03:18
*** jamesbenson has joined #openstack-kolla03:22
*** schwicht has quit IRC03:23
*** jamesbenson has quit IRC03:26
*** schwicht has joined #openstack-kolla03:42
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426203:45
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426203:46
*** schwicht has quit IRC04:10
*** harlowja has quit IRC04:18
*** dave-mccowan has quit IRC04:23
*** ducttape_ has joined #openstack-kolla04:27
*** harlowja has joined #openstack-kolla04:30
*** ducttape_ has quit IRC04:31
*** awiddersheim has joined #openstack-kolla04:46
*** awidders_ has quit IRC04:48
*** jamesbenson has joined #openstack-kolla05:00
*** kristian__ has joined #openstack-kolla05:00
*** jamesbenson has quit IRC05:04
*** kristian__ has quit IRC05:05
*** jbadiapa has quit IRC05:21
*** harlowja has quit IRC05:33
spsuryamorning05:41
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585805:45
*** jamesbenson has joined #openstack-kolla06:02
*** omenvlad has joined #openstack-kolla06:06
*** jamesbenson has quit IRC06:07
*** mewald has joined #openstack-kolla06:08
openstackgerritzhubingbing proposed openstack/kolla-ansible stable/ocata: Add pipeline file into ceilometer-central and ceilomter-compute node  https://review.openstack.org/48501706:08
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585806:09
*** lpetrut_ has joined #openstack-kolla06:18
*** janki has joined #openstack-kolla06:20
*** jamesbenson has joined #openstack-kolla06:23
*** jamesbenson has quit IRC06:27
*** ducttape_ has joined #openstack-kolla06:28
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585806:29
*** manheim has joined #openstack-kolla06:30
*** ducttape_ has quit IRC06:33
*** manheim has quit IRC06:34
*** skramaja has joined #openstack-kolla06:37
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585806:50
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Add cinder client keyring requirements for extnerla ceph in nova  https://review.openstack.org/48546406:53
*** skramaja_ has joined #openstack-kolla06:58
*** skramaja has quit IRC06:58
*** jbadiapa has joined #openstack-kolla06:59
*** jmccarthy has quit IRC07:03
*** shardy has joined #openstack-kolla07:04
*** itlinux has quit IRC07:04
*** jamesbenson has joined #openstack-kolla07:05
openstackgerritDennis Mueller proposed openstack/kolla-ansible master: Added neutron metering-agent support  https://review.openstack.org/44585807:08
*** jamesbenson has quit IRC07:09
*** mewald has quit IRC07:10
*** mewald has joined #openstack-kolla07:11
*** jascott1 has quit IRC07:12
*** jascott1 has joined #openstack-kolla07:13
*** serlex has joined #openstack-kolla07:13
*** nrado has joined #openstack-kolla07:16
*** jascott1 has quit IRC07:17
*** gfidente has joined #openstack-kolla07:18
*** jamesbenson has joined #openstack-kolla07:25
*** mewald1 has joined #openstack-kolla07:27
*** nrado1 has joined #openstack-kolla07:27
nrado1Hi folks, does anyone have experience with the update procedure in Kolla? Thinking about updating from Mitaka to Ocata. How does a typical update look like and what could possibly go wrong (database, …) in a production environment?07:27
*** mewald has quit IRC07:30
*** jamesbenson has quit IRC07:30
*** nrado has quit IRC07:30
*** reidrac has joined #openstack-kolla07:33
SeyongHello guys, Does any one try to run Swift service in container? like kolla-kubernetes? To do that where can I start from?07:34
*** manheim has joined #openstack-kolla07:35
*** lily has joined #openstack-kolla07:36
*** lpetrut_ has quit IRC07:37
spsuryaSeyong: have you installed OpenStack on kubernetes  ?07:37
*** manheim has quit IRC07:37
*** manheim has joined #openstack-kolla07:37
*** cliles has quit IRC07:37
spsuryausing deployment guide of kolla-k8s07:37
Seyongspsurya: Yes I did. I have running environment of kolla-kubernetes07:38
Seyongand want to add Swift on that..07:39
*** kristian__ has joined #openstack-kolla07:39
*** kristian__ has quit IRC07:40
*** kristian__ has joined #openstack-kolla07:40
lilyhi~ i'm working with seyong. we are interested in Swift...07:41
*** reidrac has quit IRC07:43
lilyI saw the work item in blueprint on kolla-k8s there are work items for more microservices, right?07:43
lilyand swift would be the one of them...07:43
lilyI'm wondering when the microservices of Swift will be merged on the code...07:44
*** kristian__ has quit IRC07:44
spsuryahi....07:45
spsuryalily: Seyong swift work in progress07:45
lilyand If I have a chance, I wanna contribute on it... however i'm still a beginner for helm...07:45
spsuryalily: you are welcome07:46
spsuryalily: Seyong we are still in heavy development phase07:46
spsuryahop soon we will be providing every services07:47
*** egonzalez has joined #openstack-kolla07:49
Seyongspsurya: yea the working is very interesting.. Can I get a set of code by email or other, even it is in heavy devel. phare and not working. mine is mail@syum.net07:50
SeyongIt will be very thanks :)07:51
*** jmccarthy has joined #openstack-kolla07:53
spsuryajust git clone the kolla-kuberentes repo07:54
spsuryaall the current code will be there07:55
openstackgerritMerged openstack/kolla-ansible stable/ocata: Fix the incompatible of docker-py 2.0.0  https://review.openstack.org/48153707:59
Seyongspsurya: ok thanks very mush :D08:00
spsuryaSeyong: you  can follow the code structure already implemented for other services ;)08:01
*** magicboiz has quit IRC08:02
spsuryanrado1:   you can follow the https://github.com/openstack/kolla-ansible/blob/master/doc/operating-kolla.rst08:02
spsuryaand i think you are talking about the upgrade not update;)08:03
nrado1thanks spsurya, yes, I was talking about upgrade ;)08:04
nrado1But what about services like heka/fluentd when I want to upgrade? Do I have to delete the old service and deploy the new one manually?08:06
*** rmart04 has joined #openstack-kolla08:06
spsuryanrado1: yes i guess so :)08:07
*** omenvlad has quit IRC08:07
*** kevzha01 is now known as kevinz08:08
spsuryaheka is deprecated in Ocata08:08
spsuryayou have to pull ocata images for fluentd08:08
*** omenv has joined #openstack-kolla08:09
*** cliles has joined #openstack-kolla08:19
*** mgoddard_ has joined #openstack-kolla08:24
*** lpetrut_ has joined #openstack-kolla08:25
*** lily has quit IRC08:27
*** jamesbenson has joined #openstack-kolla08:28
*** magicboiz has joined #openstack-kolla08:28
*** jamesbenson has quit IRC08:32
*** skramaja_ is now known as skramaja08:36
*** lpetrut_ has quit IRC08:40
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426208:44
*** tvignaud has quit IRC08:59
mewald1Master branch on multinode deployment fails to start keepalived. the service keeps respawning: https://gist.github.com/mewald1/8733ad10e09b8cec8264113ed2c87c77 Is anyone familiar with the issue?09:05
*** coolsvap has joined #openstack-kolla09:05
*** jamesbenson has joined #openstack-kolla09:09
*** tvignaud has joined #openstack-kolla09:13
*** jamesbenson has quit IRC09:13
*** sambetts|afk is now known as sambetts09:16
*** imcsk8_ has joined #openstack-kolla09:23
*** imcsk8 has quit IRC09:23
*** denaitre has quit IRC09:28
*** yingjun has quit IRC09:31
*** srnbckr has joined #openstack-kolla09:36
*** mewald1 has quit IRC09:38
*** jiriprox has joined #openstack-kolla09:45
*** magicboiz has quit IRC09:48
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Add cinder client keyring requirements for extnerla ceph in nova  https://review.openstack.org/48546409:52
*** magicboiz has joined #openstack-kolla09:54
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Add cinder client keyring requirements for external ceph in nova  https://review.openstack.org/48546409:54
*** mewald has joined #openstack-kolla09:55
*** shardy is now known as shardy_mtg09:57
*** magicboiz has quit IRC09:59
egonzalezmewald, are the interfaces on which VIPs should be bind UP?10:01
*** kevinz has quit IRC10:02
openstackgerritMerged openstack/kolla master: implement panko api for ubuntu binary  https://review.openstack.org/48464910:03
*** magicboiz has joined #openstack-kolla10:05
*** jamesbenson has joined #openstack-kolla10:05
*** caowei has joined #openstack-kolla10:05
*** jamesbenson has quit IRC10:10
*** ramishra has joined #openstack-kolla10:14
jbadiapaegonzalez, have you got a minute?10:19
*** mewald has quit IRC10:21
egonzalezjbadiapa, hey10:21
jbadiapaegonzalez, hi10:22
jbadiapaegonzalez, were you able to take a look at your patch with on the koll-ansible https://review.openstack.org/#/c/482528/?10:23
jbadiapaegonzalez, I meant does it work for you to have this https://review.openstack.org/#/c/467219/ ok?10:24
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Test fluentd kolla change  https://review.openstack.org/48252810:24
egonzalezjbadiapa, what I'm concern migrating fluentd to opstools repo is that maybe, centos update their fluentd version to a different one in fluent repos10:25
egonzalezthat will cause compatibility issues with .deb distros10:25
jbadiapa.deb distros are not affected by this patch10:26
egonzalezyeah, i mean, centos update fluentd to 1.14, while fluendt repos still 1.12 (for .deb)10:27
egonzalez1.14 is not compatible with 1.12 afik10:27
egonzalez*configuration/plugins are not compatible10:28
*** ansiwen has quit IRC10:28
jbadiapaegonzalez, afaik, there is a note on the fluentd/Dockerfile.j2 that specific says that the fluentd 0.12 version is used10:28
*** xinliang has quit IRC10:29
jbadiapaegonzalez, https://github.com/openstack/kolla/blob/master/docker/fluentd/Dockerfile.j2#L2910:29
*** pc_m has quit IRC10:32
*** ansiwen has joined #openstack-kolla10:32
*** pc_m has joined #openstack-kolla10:34
*** ansiwen has quit IRC10:34
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional  https://review.openstack.org/46462010:35
*** ramishra has quit IRC10:38
*** ramishra has joined #openstack-kolla10:40
*** xinliang has joined #openstack-kolla10:41
*** xinliang has joined #openstack-kolla10:41
openstackgerritMerged openstack/kolla-ansible master: Allow for running standalone Ironic w/o keystone  https://review.openstack.org/47692710:46
*** mdnadeem has quit IRC10:47
*** ramishra has quit IRC10:47
*** athomas has quit IRC10:48
*** ramishra has joined #openstack-kolla10:49
*** athomas has joined #openstack-kolla10:49
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible master: Optimize reconfiguration for mariadb  https://review.openstack.org/43348010:51
Jeffrey4legonzalez, around?10:51
egonzalezJeffrey4l, yep10:51
Jeffrey4legonzalez, could u check above patch idea?10:51
Jeffrey4lit is still wip. but it works.10:51
Jeffrey4lwanna to know how to you think about this idea.10:51
egonzalezJeffrey4l, thanks!  will take a look and test10:53
Jeffrey4lok. thanks. just remember that, it is still wip. just check the idea.  there are some optimize need be done.10:54
*** mewald has joined #openstack-kolla10:57
*** jbadiapa is now known as jbadiapa|lunch11:01
*** mewald has quit IRC11:09
*** pc_m has quit IRC11:16
*** mewald has joined #openstack-kolla11:17
*** shardy_mtg has quit IRC11:18
mewaldegonzalez: It was a stupid mistake: passwords.yaml not generated -> authpass option empty -> keepalived goes mad11:19
*** shardy_mtg has joined #openstack-kolla11:19
*** shardy_mtg is now known as shardy11:19
*** pc_m has joined #openstack-kolla11:20
*** dave-mccowan has joined #openstack-kolla11:21
pbourkeegonzalez: Jeffrey4l: someone want to ack this? https://review.openstack.org/#/c/467608/11:23
pbourkesame with https://review.openstack.org/#/c/474598/11:23
pbourkezhubingbing_: ^11:23
zhubingbing_Hi11:24
*** shardy has quit IRC11:24
*** shardy has joined #openstack-kolla11:25
zhubingbing_pbourke11:25
zhubingbing_done11:25
pbourkezhubingbing_: thanks very much11:25
openstackgerritMerged openstack/kolla-ansible master: Add dev mode for Murano  https://review.openstack.org/46760811:27
zhubingbing_Np ;)11:28
*** rwallner has joined #openstack-kolla11:43
mewaldHorizon is broken in upstream kolla. But I am quite sure it's not a kolla bug11:45
sbezverkkolla cores inc0's tarball fix needs 1 more +2 https://review.openstack.org/48535711:45
openstackgerritjimmygc proposed openstack/kolla-ansible master: Add VMware Documentation  https://review.openstack.org/48426211:51
sean-k-mooneyegonzalez: o/11:54
sean-k-mooneyegonzalez: qq im going to address the comments in https://review.openstack.org/#/c/408872/ and https://review.openstack.org/#/c/408876/ today. assumeing there are no other objections or they are adressed quickly do we still have time to include these in the pike release?11:55
*** kristian__ has joined #openstack-kolla11:55
*** skramaja has quit IRC11:57
*** dennismueller has joined #openstack-kolla11:58
*** kristian__ has quit IRC11:59
manheimon the 5.0.0 images, I think there is a bug in the horizon image, if I edit the default security group, when I add a rule the ADD button is no clickable12:01
*** k_mouza has joined #openstack-kolla12:01
*** mattmceuen has joined #openstack-kolla12:02
dennismuellerHi, someone to review https://review.openstack.org/#/c/445858/ ?12:03
dennismuelleregonzalez: ... maybe? ;)12:03
*** magicboiz has quit IRC12:05
*** magicboiz has joined #openstack-kolla12:06
*** k_mouza has quit IRC12:07
*** awidders_ has joined #openstack-kolla12:17
egonzalezsean-k-mooney, hey!  I would like it in Pike :) as well as ODL12:17
*** rhallisey has joined #openstack-kolla12:17
egonzalezmgkwill, added a comment in ODL change with an error I faced a week ago. Going try know and see if was a bug in neutron `` 'No module named tests.unit.plugins.ml2.drivers.mechanism_logger'``12:18
*** athomas has quit IRC12:19
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add prometheus and node_exporter service  https://review.openstack.org/48515012:19
*** k_mouza has joined #openstack-kolla12:20
*** awiddersheim has quit IRC12:20
hawihi there. is it possible to add external members without adding network in newer versions (pike) for lbaas? it still seems that subnet is required for ocata. or is it possible by using octavia? i have no idea how to make octavia working for 4.0.0.12:20
*** iniazi has joined #openstack-kolla12:21
egonzalezmgkwill, those are my odl related settings in globals.yml http://paste.openstack.org/show/616005/12:22
sdakemorning12:23
*** iniazi_ has quit IRC12:24
*** lpetrut has joined #openstack-kolla12:25
*** JoseMello has joined #openstack-kolla12:26
*** hrw has quit IRC12:28
*** hrw has joined #openstack-kolla12:28
*** wlfightup has joined #openstack-kolla12:29
wlfightupsos12:30
wlfightup {"failed": true, "item": "01-syslog", "msg": "AnsibleUndefinedVariable: 'dict object' has no attribute 'ipv4'"}12:30
wlfightuphow to solve the problem12:31
wlfightuphow to debug?12:31
mewaldsdake: morning12:31
wlfightupmoring12:31
wlfightupmewald sdake moring12:31
egonzalezwlfightup, check api_interface in the node is failing, likely does not have IP, non existent interface o invalid ansible interface name(no underscores allowed iirc)12:33
*** ducttape_ has joined #openstack-kolla12:33
wlfightup2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100012:34
wlfightup    link/ether fa:16:3e:6b:14:e1 brd ff:ff:ff:ff:ff:ff12:34
wlfightup    inet 172.16.32.7/23 brd 172.16.33.255 scope global dynamic eth012:34
wlfightup       valid_lft 83595sec preferred_lft 83595sec12:34
wlfightup    inet6 fe80::f816:3eff:fe6b:14e1/64 scope link12:34
wlfightup       valid_lft forever preferred_lft forever12:34
wlfightup3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100012:34
wlfightup    link/ether fa:16:3e:41:d1:c0 brd ff:ff:ff:ff:ff:ff12:34
wlfightup    inet6 fe80::f816:3eff:fe41:d1c0/64 scope link12:34
wlfightup       valid_lft forever preferred_lft forever12:34
wlfightup4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP qlen 100012:34
wlfightup    link/ether fa:16:3e:65:de:cd brd ff:ff:ff:ff:ff:ff12:34
wlfightup    inet6 fe80::f816:3eff:fe65:decd/64 scope link12:34
wlfightup       valid_lft forever preferred_lft forever12:34
wlfightupthis is my ip info12:34
openstackgerritMathias Ewald proposed openstack/kolla master: Add container for prometheus mysqld exporter  https://review.openstack.org/48559612:35
k_mouzawlfightup: what's the value of "network_interface" in your globals.yaml file ?12:35
wlfightupnetwork_interface: "eth1"12:36
wlfightupneutron_external_interface: "eth2"12:36
mewaldwlfightup: eth1 has to IP address as egonzalez said :)12:37
serlex:)12:37
openstackgerritJuan Badia Payno proposed openstack/kolla master: Changed fluentd repository for CentOS/RHEL/Oraclelinux  https://review.openstack.org/46721912:37
sean-k-mooneyls12:37
sean-k-mooney.. wrong window12:37
serlexalso wlfighup, try to use a paste website when sharing errors or config ie https://paste.ubuntu.com/12:37
egonzalezdennismueller, looks like neutron not recognize metering entrypoints, use full path as it was12:38
*** ducttape_ has quit IRC12:38
wlfightupmewald: my eth1 has no ip address.  ip a don't see any ip of eth112:39
egonzalezwlfightup, network_interface need an IP address12:39
egonzalezis where services communicate with each other12:40
*** mattmceuen has quit IRC12:40
dimsflaper87 : nice write up on the kolla/tripleo. ++ to move things forward together12:41
k_mouzahey guys! I have an issue with Cinder+lvm which I haven't seen before and I was wondering if you can help. Basically when I create a new volume it gets to an error state. The log from the cinder_volume container shows this error:12:42
k_mouza2017-07-19 18:31:30.877 36 ERROR oslo_messaging.rpc.server Stderr: u'File descriptor 20 (/dev/urandom) leaked on lvcreate invocation. Parent PID 9590: /usr/bin/python2\n  /dev/cinder-volumes/volume-3668d9bb-9c6f-463a-927a-81cf41733d78: not found: device not cleared\n  Aborting. Failed to wipe start of new LV.\n  semid 21889024: semop failed for cookie 0xd4d1ec3: incorrect semaphore state\n  Failed to set a proper state for no12:42
k_mouzatification semaphore identified by cookie value 223157955 (0xd4d1ec3) to initialize waiting for incoming notifications.\n'12:42
k_mouzaany idea on this ?12:43
wlfightupegonzalez: oh, thank you. but i have a question that vm communication between the network which card ?12:44
*** ducttape_ has joined #openstack-kolla12:44
flaper87dims: thanks12:44
*** ansmith has joined #openstack-kolla12:44
wlfightupnetwork_interface: manager network , neutron_external_interface: external network,   vm network which inc?12:45
sdakeflaper87 morning-  ijust read your summary - looks  fairly accurate12:45
sbezverkk_mouza : cinder service-list see if volume service is up12:45
egonzalezdennismueller, added a few more comments, looks good12:45
flaper87sdake: phew, thanks! Was worried about missing some important points12:46
sdakeflaper87 it seems liek your moving orward wit hthat 4 pull request - one sugestion I'd make is to really take a look at how the fundamental building block templates operate12:46
sdakeI personally posit without such information no real decision can be made12:46
k_mouzasbezverk: yup, cinder backup, scheduler and volume are all enabled and up12:47
sdakeand i dont mean how he tempaltes rae implemented - I mean answering the question "do they work and do they make sense for this use case"12:47
sdakeI believe the answer is yes, however, my beliefs and yours may night be consistent :)12:47
sdakeyours may not be consistent that is12:48
sdakesorry 5am here12:48
sbezverkk_mouza I remember seeing this error in initial stages of developing cinder/lvm support12:48
sbezverkk_mouza which version you deploy?12:49
k_mouzathat's the stable/ocata one12:49
wlfightupnetwork_interface: manager network , neutron_external_interface: external network,   vm network which inc?12:49
sdakeflaper87 ^^12:49
k_mouzasbezverk: I wan't seeing this when I was deploying with the newton one12:49
wlfightupegonzalez12:49
wlfightupk_mouza12:50
wlfightupsbezverk12:50
wlfightupnetwork_interface: manager network , neutron_external_interface: external network,   vm network which inc?12:50
egonzalezwlfightup, quick resume about interfaces http://paste.openstack.org/show/616009/12:50
wlfightupvm communication between the network which network card ?12:50
sbezverkk_mouza but which version you deploy now?12:50
wlfightupmewald12:51
k_mouzasbezverk: stable/ocata12:51
sbezverkk_mouza hm, that is odd, we have gates testing ocata images with lvm, works ok12:52
sdakeflaper87 i'd suggest such a technical determination could be made in 4-8 hours - its not a trivial thing to do - and I am willing to help you do such a thing12:52
sdake(the eval not the decisi9no)12:52
*** jbadiapa|lunch is now known as jbadiapa12:54
openstackgerritMerged openstack/kolla master: Fix incorrect tarballs naming  https://review.openstack.org/48535712:54
wlfightupegonzalez: the interface in the page, don't exist globals.yml12:55
sdakeflaper87 the worst outcome that could come of such effort is you woudl learn how kolla-kubernetes decomposed openstack - and decide kolla-kubernetes was not suitable for tripleo's use case - i think on balance the learning > the time cost12:55
egonzalezwlfightup, by default all are under network_interface, you can override the value, those interface names are from group_vars/all12:56
k_mouzasbezverk yeah, I don't know what's wrong. The only difference between this and my newton-testing system is that in the ocata one I've created the lvm group directly on the hard drive device (ie /dev/sdb), while in the newton one I've first created a partition on the device and then created the lvm group on the partition (ie /dev/sdb1). Would that make any difference?12:56
wlfightupegonzalez, thank you , i understand now12:58
sbezverkk_mouza it looks more lvm issue to me, please check pvs/vgs/lvs see that all these produce sensible output and then try manually create a volume..12:58
*** pbourke has quit IRC12:58
*** pbourke has joined #openstack-kolla13:00
*** rmart04_ has joined #openstack-kolla13:01
*** ducttape_ has quit IRC13:02
mewaldwould someone be opposed to adding "-b" to kolla-ansible? All tasks work without being root, at least the stuff I've been working with13:03
*** ducttape_ has joined #openstack-kolla13:03
flaper87sdake: hey, sorry, kinda swamped today. I've actually done such a technical research already but I'm sure my knowledge of gotpl and kolla are not as good as yours. I'd like to take some time to go through some of your thoughts on this. Unfortunately, I'll be traveling for the next two weeks and sync convos are not going to work great for me during this time.13:04
flaper87sdake: feel free to drop me some emails if you feel like it13:04
*** rmart04 has quit IRC13:04
sdakeflaper87 roger on the 2 weeks of travel being incompatible with this objective13:04
flaper87I'm curious to see your points but I don't feel like dragging the entire community into this discussion (re m-l discussions on this topic)13:05
sdakeflaper87 i think email is not sufficient - the only real sufficient solution is experience13:05
flaper87note that I'm not suggesting having closed conversations but rather valuing the sync'ed conversations on this specific argument13:05
sdakei have no points to make - only for the desire for you to experience what the building blocks offer13:05
sdakei am one of those people that believes without experience no true decision can be made - data leaves out all kinds of valuable human traits - e.g. intuition13:07
*** ducttape_ has quit IRC13:07
wlfightupThere was a problem: :4000/v1/_ping: http: server gave HTTP13:08
wlfightuphttp://paste.openstack.org/show/616010/13:08
sdakei think its possible you have some data points - but hae you deployed via the deployment guide kolla-kubernetes?13:08
*** rmart04_ has quit IRC13:08
*** goldyfruit has joined #openstack-kolla13:08
sdakeflaper87 ^^13:09
*** kristian__ has joined #openstack-kolla13:10
egonzalezwlfightup, registry IP does not match with the insecure-registry configured in docker systemd file 192.168.32.7 / 172.16.32.713:11
wlfightupyes, I'm careless, sorry13:12
sdakewlfightup its not like thi stuff is super trivial - your deploying a cloud - no reason to be sorry13:15
*** eaguilar_ has joined #openstack-kolla13:15
sdakewe have given every thought to simplification of the deployment process and still there are many things that could go wrong as a result of invalid human input :)13:15
*** eaguilar has quit IRC13:16
wlfightupMainly I do not understand the container13:17
sdakewlfightup the best analogy I can think of is chroot on steroids (if your familiar with chroot)13:17
wlfightup"The requested image does not exist: 172.16.32.7:4000/lokolla/centos-source-fluentd:4.0.013:18
sdakelokolla13:18
sdaketht a problem13:18
sdakethat a problem13:18
sdakeit should b e"kolla"13:18
sdakeit looks like your providing your own registry13:18
*** zhurong has quit IRC13:19
sdakethat message says "you didnt build a container in the lokolla namespace on your local registry server"13:19
*** lucasxu has joined #openstack-kolla13:19
*** kristian__ has quit IRC13:20
wlfightupwget http://tarballs.openstack.org/kolla/images/centos-source-registry-ocata.tar.gz13:21
wlfightuptar zxvf centos-source-registry-ocata.tar.gz -C /opt/registry/13:21
wlfightupi use this13:21
*** schwicht has joined #openstack-kolla13:23
egonzalezwlfightup, can you do ``docker pull 172.16.32.7:4000/lokolla/centos-source-fluentd:4.0.3``13:29
egonzalezwlfightup, replace 4.0.0 to 4.0.3 which is latest tag13:29
*** mewald has left #openstack-kolla13:33
*** athomas has joined #openstack-kolla13:38
wlfightup4.0.3 is version of ocata ?13:39
egonzalezwlfightup, yep, is ocata with latest bug fixes13:40
rwellumsdake: I saw sbezverk use 'lokolla' in common_iscsi_config_v5.sh13:41
rwellumecho "      docker_namespace: lokolla"13:42
sbezverkrwellum it is because I use registry at the gate13:42
egonzalezJeffrey4l, mind re-check this sfc PS? https://review.openstack.org/#/c/461748/ sfc image is being deprecated in https://review.openstack.org/#/c/463700/13:42
sbezverkand namespace is lokolla13:42
Jeffrey4lsure13:42
rwellumah ok sbezverk - just a coincidence then13:42
*** goldyfruit has quit IRC13:43
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add mysqld_exporter to prometheus role  https://review.openstack.org/48563613:43
Jeffrey4legonzalez, done.13:44
*** mewald has joined #openstack-kolla13:44
egonzalezJeffrey4l, many thanks!13:44
*** goldyfruit has joined #openstack-kolla13:44
openstackgerritMerged openstack/kolla-ansible master: Add vlan tenant network support  https://review.openstack.org/47441613:44
openstackgerritMerged openstack/kolla-ansible master: Make Ironic inspector honour inventory  https://review.openstack.org/43692913:44
egonzalezJeffrey4l, btw, forgot +2 in https://review.openstack.org/#/c/463700/ only added +w13:45
Jeffrey4lah, thanks.13:45
*** mattmceuen has joined #openstack-kolla13:48
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Disable trace for all containers running httpd  https://review.openstack.org/48564113:48
*** yingjun_1 has joined #openstack-kolla13:51
*** k_mouza has quit IRC13:51
openstackgerritPavel Gluschak (scsnow) proposed openstack/kolla-ansible master: Make cinder-backup service optional  https://review.openstack.org/46462013:52
*** jtriley has joined #openstack-kolla13:54
*** k_mouza has joined #openstack-kolla13:54
wlfightuphttp://paste.openstack.org/show/616014/13:55
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add mysqld_exporter to prometheus role  https://review.openstack.org/48563613:55
egonzalezwlfightup, is there any nova-compute container running? any error in rabbitmq? output of nova service-list13:56
*** ddyer has quit IRC13:57
wlfightuphow to debug? where is log ?13:57
openstackgerritAndy Smith proposed openstack/kolla-ansible master: Add support for hybrid messaging backends  https://review.openstack.org/46896613:57
*** ducttape_ has joined #openstack-kolla13:57
wlfightuphttp://paste.openstack.org/show/616016/13:57
wlfightupnova-compute container running13:58
wlfightupnova-compute process alos running13:58
wlfightupalso13:58
egonzalezwlfightup, logs are at /var/lib/kolla/volumes/kolla_logs/_data/nova13:59
egonzalezshow ``nova service-list`` output command14:00
wlfightupls: cannot access /var/lib/kolla: No such file or directory14:00
wlfightup-bash: nova: command not found14:00
*** ddyer has joined #openstack-kolla14:00
mewaldit is /var/lib/docker/volumes/kolla_logs/_data/nova14:00
mewaldnova command not found => install nova client14:01
wlfightupkolla dont' install novaclient  ?14:01
mewaldnope14:01
mewaldnot to my knowledge14:02
wlfightupAMQP server on 172.16.32.7:5672 is unreachable14:04
wlfightupconnect rabbitmq error14:04
*** prateek has quit IRC14:05
egonzalezcheck hostname, network_interface IP should resolve it hostname or rabbit wont work14:05
wlfightupwhere is the nova.conf ?14:06
mewaldwlfightup: in the container and in /etc/kolla/nova-api/nova.conf14:07
wlfightuptransport_url = rabbit://openstack:3yvhjOArRGuoCHj6j9UzCt12S227atGZbBZFaYyW@172.16.32.7:567214:08
wlfightupthe ip and port is correctly14:08
wlfightuprabbitmq don't start service14:08
wlfightupkolla don't deploy rabbitmq ?14:09
*** ducttap__ has joined #openstack-kolla14:15
egonzalezwlfightup, kolla deploy rabbitmq by default, check hostname and resolution, rabbitmq containers will fail to start if cannot resolve their name14:15
wlfightup4d62e94e707b        172.16.32.7:4000/lokolla/centos-source-rabbitmq:4.0.3             "kolla_start"            33 minutes ago      Up 33 minu14:15
wlfightupi look rabbit docker14:15
wlfightuphow to debug?14:16
mewaldcheck stdout when starting the container14:17
*** yingjun_1 has quit IRC14:17
mewaldcheck log files14:17
mewaldare you running a single node setup?14:18
*** ducttape_ has quit IRC14:19
*** vhosakot has joined #openstack-kolla14:19
*** jamesbenson has joined #openstack-kolla14:23
wlfightuprunning a single node14:23
wlfightupyes14:23
*** jtriley has quit IRC14:25
*** rwsu has joined #openstack-kolla14:25
jamesbensonmorning all :-D14:30
jamesbensonvhosakot & inc0: Thanks, I guess a destroy/deploy fixed the weird ceph/glance/cinder/keystone issue.  Now another issue to work on :-/14:30
*** zhangfei has joined #openstack-kolla14:31
vhosakotjamesbenson: yeah, destroy is very constructive :)14:31
jamesbensonlol14:32
jamesbensonI thought at one point I heard a deploy would override all settings.... guess not always the case14:32
jamesbensonand typically I do a destroy/deploy but being lazy I suppose14:32
mewaldHow do we handle port conflicts between services? I created a prometheus server role and had to move it's port from 9090 to 9091 since it conflicted with freezer. Now I had to find out prometheus's haproxy_exporter wants to listen on port 9091 … Any suggestions how to deal with this best?14:32
vhosakotyeah, I reboot too after destroy as I seem some used resources in /var/lib/docker after destroy.14:33
jamesbensongah, reboot eats up so much time!  But I understand...14:34
vhosakotmewald: all the default service ports can be overridden in https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L128.14:35
egonzalezmewald, Usually when ports are not their default, use a higher number like 3909014:35
sbezverkmewald: it is relatively easy to deal with in kolla-k8s by using nodeport14:36
vhosakotyes, k8s handles ports conflicts os apps really well.14:37
vhosakotof*14:37
*** zhangfei has quit IRC14:37
vhosakotkeystone's 8080 conflicted with switft_proxy port for a long time... I hit https://bugs.launchpad.net/keystone/+bug/843159 many times.14:38
openstackLaunchpad bug 843159 in OpenStack Identity (keystone) "Default port of 8080 conflicts with Swfit" [High,Fix released]14:38
openstackgerritMerged openstack/kolla master: Fix Horizon container starting with Watcher enabled  https://review.openstack.org/48354214:39
*** dennismueller has quit IRC14:39
*** lucasxu has quit IRC14:39
*** ggillies has quit IRC14:40
*** ggillies has joined #openstack-kolla14:42
*** janki has quit IRC14:45
*** janki has joined #openstack-kolla14:45
*** yingjun has joined #openstack-kolla14:46
*** janki has quit IRC14:47
rwellumsbezverk: odd openvswitch issue in 5.0.0 - have you seen this?14:49
*** jtriley has joined #openstack-kolla14:49
rwellumhttps://www.irccloud.com/pastebin/7i5Uj636/14:49
rwellumI think I am not understanding your registry approach14:49
*** zhangfei has joined #openstack-kolla14:50
sbezverkrwellum: happy to clarify14:52
sbezverkplease try to pull the image manually14:52
wlfightuphttp://paste.openstack.org/show/616029/14:52
wlfightupegonzalez, this is my rabbitmq error14:52
sbezverkfrom the kubelet host try to do docker pull 127.0.0.1:30401/lokolla/{image name}:5.0.014:53
*** jbadiapa has quit IRC14:55
egonzalezwlfightup, is there something else before the error report?14:55
rwellumsbezverk: same issue -14:55
wlfightupyes14:56
sbezverkrwellum kubectl get pod -n kolla | grep reg14:56
*** zhangfei has quit IRC14:56
wlfightupWhat to do next14:56
sbezverkkubectl get svc -n kolla14:56
jamesbensonokay, stupid question, having issues with git and uploading bug fixes, yelling at me about merge conflicts?14:57
jamesbensoncan someone pm me?14:57
rwellum1st one nothing, second 'no resources found' - sbezverk14:58
egonzalezwlfightup, please, share ``getent ahostsv4 <hostname>`` and check that your api_interface IP appears there14:58
sbezverkrwellum : then you do not have registry running14:58
wlfightup[root@kolla kolla-ansible]# getent ahostsv4 kolla.com14:59
wlfightup172.16.32.7     STREAM kolla.com14:59
wlfightup172.16.32.7     DGRAM14:59
wlfightup172.16.32.7     RAW14:59
rwellumsbezverk: yeah that's what I meant by I'm not understanding the registry approach.14:59
rwellumI'm just looking at your cloud.yaml changes15:00
rwellumFeels like steps missing15:00
sbezverkrwellum it should be started 1st before any other charts15:00
sbezverkhelm install helm/microservice/registry-deployment blah blah15:00
rwellumok that's what I am missing.15:00
sbezverkman you missed the elephant ;)15:01
egonzalezwlfightup, your network_interface is eth0 or eth1?15:01
rwellumsbezverk: I'm good at that15:01
sbezverkthere is setup_registry.sh call from15:01
wlfightupeth015:01
sbezverksetup_gate_iscsi15:01
wlfightupegonzalez15:01
wlfightupnetwork_interface: "eth0"15:01
openstackgerritJames Benson proposed openstack/kolla-ansible master: adds keyring to doc  https://review.openstack.org/48570415:02
openstackgerritJames Benson proposed openstack/kolla-ansible master: adds keyring to doc  https://review.openstack.org/48570415:04
rwellumsbezverk: is there a log of the actual helm install command? That'll help me understand all the actual params.15:05
sbezverkrwellum I pushed this chart upstream as well, see a short readme here (it is still WIP though) http://github.com/sbezverk/charts15:06
sbezverkrwellum https://github.com/sbezverk/charts/tree/docker_registry_chart/incubator/docker-registry15:07
sbezverkrwellum here is example how I ran it on my local setup: helm install helm/microservice/registry-deployment --namespace kolla --name registry-ubuntu --set distro=ubuntu --set node_port=30402 --set initial_load=true --set svc_name=registry-ubuntu15:09
rwellumBut first you have to do: helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator ?15:10
sbezverkrwellum no15:10
sbezverkthat was only for upstream15:11
rwellumoh ok15:11
sbezverkfor kolla-k8s it is already included in the kolla-k8s repo15:11
sbezverkif you updated it yesterday15:11
*** eaguilar has joined #openstack-kolla15:13
*** dmsimard is now known as dmsimard|cave15:14
rwellumtrying it now15:16
*** eaguilar_ has quit IRC15:16
rwellumso far it's 4m in and still in init state. But I wonder if openvswitch is blocking it.15:17
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add mysqld_exporter to prometheus role  https://review.openstack.org/48563615:18
sbezverkrwellum it takes about 10 minutes to load 4g tarball15:20
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add haproxy_exporter to prometheus role  https://review.openstack.org/48571515:21
rwellumsbezverk: adding 10 minutes to deployment?15:21
*** lucasxu has joined #openstack-kolla15:22
openstackgerritMathias Ewald proposed openstack/kolla master: Add haproxy_exporter image for prometheus  https://review.openstack.org/48571615:22
sbezverkrwellum yes and no, pulling all images locally is much faster and more reliable15:23
jamesbensonegonzalez: bug 1705116 has been fixed, please review :-)15:24
openstackbug 1705116 in kolla-ansible "External Ceph in kolla-ansible missing keyring" [Medium,In progress] https://launchpad.net/bugs/1705116 - Assigned to James (james-o-benson)15:24
rwellumsbezverk: much faster than what? Currently it takes me about 30m to go from nothing to full OpenStack. If I add another 10m..15:24
sbezverkrwellum you pull from docker hub each image individually15:25
sbezverksee how much traffic goes and then compare with 4G total15:26
rwellumYou're saying this one hit then all the other images are loaded much faster15:27
sbezverkyes15:29
sbezverkonly nova images would take about 4g15:29
openstackgerritJames Benson proposed openstack/kolla-ansible master: Added comment about the passwords.yml to help clearify that it doesn't get pulled in directly.  https://review.openstack.org/48572015:31
rwellumsbezverk: well 20m in and not past init15:31
sbezverkthen something is wrong15:32
sbezverkat the gate it takes about 10 minutes, in my local setup it takes about 8 minutes15:32
sbezverkare you suing dial up to the internet ;) ??15:32
rwellumDSL not good sbezverk ?15:33
sbezverktry curl http://tarballs.openstack.org/kolla/images/centos-source-registry-master.tar.gz15:33
sbezverksee how long it takes to download it15:34
*** eaguilar has quit IRC15:34
rwellumlooking about 20m15:35
rwellumfrom the progress meter15:35
*** serlex has quit IRC15:37
sbezverkit will take the same time for registry to get that file..15:38
sbezverkrwellum another big reason to use registry is the fact that there is no master images on docker hub :) so it is this way or nothing ..15:40
rwellumOk sbezverk - cleaning up and retrying will wait long enough and see,15:40
rwellumsbezverk: ack15:40
*** eaguilar has joined #openstack-kolla15:40
*** yingjun has quit IRC15:43
*** nrado1 has quit IRC15:45
sdakerwellum DSL - is there no other option? :)15:46
mewaldegonzalez: Would you prefer it all merged in a single commit? I was thinking the smaller the commits the more likely are reviews :D15:46
rwellumsdake: that was a joke :)15:46
*** jbrooks has quit IRC15:47
openstackgerritEduardo Gonzalez proposed openstack/kolla-ansible master: Add 777 permission to kolla gate logs  https://review.openstack.org/48572315:47
egonzalezmewald, yeah, but now that we're close to feature freeze, not complete changes likely wont be merged15:48
*** goldyfruit has quit IRC15:48
mewaldegonzalez: Ok I will merge it quickly and abandon the other commits15:49
inc0good morning15:50
inc0god I couldn't wake up today15:50
*** jbrooks has joined #openstack-kolla15:51
*** goldyfruit has joined #openstack-kolla15:51
openstackgerritMathias Ewald proposed openstack/kolla master: Add prometheus and related containers  https://review.openstack.org/48488215:51
rwellumToo much partying late into the night inc0 ?15:52
inc0I'd wish15:52
inc0there is no logical reason for me to be this sleepy15:53
inc0I think there was shift in weather15:53
mewaldegonzalez: part 1 https://review.openstack.org/#/c/484882/15:53
jamesbensonlol, kids?  for me, it's the kids...15:53
rwellum+1 jamesbenson15:53
inc0nope15:53
jamesbensongetting up a 3am to bring the little one into our bed because of non-stop screaming... every. night.15:53
inc0no kids, just can and it's not her15:53
egonzalezmewald, I prefer new containers from the same service in the same changes, because gives me context of the change. will review tomorrow15:54
*** coolsvap has quit IRC15:54
rwellumjamesbenson: by the third kid I just slept through that. People would ask 'how's the new baby sleeping' - and I'd say 'no idea'.15:55
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add role to deploy prometheus  https://review.openstack.org/48515015:55
mewald<<= part 215:55
mewaldthx15:55
*** rwallner has quit IRC15:59
*** rwallner has joined #openstack-kolla16:01
*** egonzalez has quit IRC16:01
*** rwallner has quit IRC16:02
*** rwallner_ has joined #openstack-kolla16:02
openstackgerritJames Benson proposed openstack/kolla-ansible master: Added note  https://review.openstack.org/48572716:02
k_mouzasbezverk: just FYI, I solved the LVM problem. I needed to echo "" > /sys/kernel/uevent_helper16:02
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: DONOTMERGE testing gate  https://review.openstack.org/48572816:02
k_mouzasbezverk: which was weird, because in the kernel config file it's set to be empty...16:02
openstackgerritMathias Ewald proposed openstack/kolla master: Add prometheus and related containers  https://review.openstack.org/48488216:02
sbezverkk_mouza nice catch!16:03
jamesbensonrwellum: I do, my wife kids by butt.... although to be fair 2 nights ago she was in my bed and I have no idea how, wifey claims it was me, but no idea.  we have 2, and 2 is the limit for us... but back on topic here ;-)16:04
openstackgerritMathias Ewald proposed openstack/kolla-ansible master: Add role to deploy prometheus  https://review.openstack.org/48515016:04
rwellumjamesbenson: :)16:04
vhosakotI fear leaving the third kid on the bus :)16:11
vhosakotwe have one kid that keeps us too busy already16:11
vhosakotsomebody is college forgot their 3rd kid on the bus and realized after half a day or soemthing.16:12
*** mewald has quit IRC16:12
jamesbensonoh my....16:15
*** shardy has quit IRC16:16
rwellumvhosakot: couple of years ago, the AC broke down at our kids school. They texted out that they were all being sent home, many parents didn't get the texts or were miles away. Buses just dropped all these kids off with no warning in the neighborhood... I went to pick mine up and ended up with about 11 kids in my living room. Ordered pizza and then called all their parents one-be-one...16:17
vhosakotrwellum: lol, 11 kids and pizza... haha16:18
*** ducttape_ has joined #openstack-kolla16:18
*** ducttap__ has quit IRC16:19
*** ducttap__ has joined #openstack-kolla16:19
manheimjamesbenson nice :D16:19
*** itlinux has joined #openstack-kolla16:20
openstackgerritJames Benson proposed openstack/kolla-ansible master: Added note  https://review.openstack.org/48572716:21
*** itlinux has quit IRC16:21
*** itlinux_ has joined #openstack-kolla16:22
*** iniazi_ has joined #openstack-kolla16:22
*** itlinux has joined #openstack-kolla16:22
*** ducttape_ has quit IRC16:23
*** iniazi has quit IRC16:25
openstackgerritChason Chan proposed openstack/kolla master: Update the documentation link for doc migration  https://review.openstack.org/48515116:26
openstackgerritJames Benson proposed openstack/kolla-ansible master: adds keyring to doc  https://review.openstack.org/48570416:31
sbezverkinc0: ping16:34
inc0I'm here16:34
inc0trying to debug why it's not fixed16:34
sbezverk:)16:34
sbezverkthanks a lot16:35
inc0aand I know16:37
inc0I hate bash16:37
jamesbensonbash?  maybe I can help?16:37
*** jbrooks has quit IRC16:38
sbezverkinc0: bash is great !!16:39
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla master: Fix tarball name again  https://review.openstack.org/48573916:39
*** jbrooks has joined #openstack-kolla16:40
sbezverkinc0: would it not be safer to compare if [ "x$ZUUL_BRANCH" == "x" ]; then16:44
inc0I hate bash:P16:45
inc0-n seems to be working tho16:45
kfox1111morning.16:45
sbezverkif -n gets changed in future, very unlikely though, it will still work..16:45
inc0if -n gets changed in future, bad names for tarballs will be least of our problems;)16:45
sbezverkkfox1111 good morning, good timing ;) inc0 expresses his displease towards bash ;)16:46
kfox1111hehe.16:46
kfox1111its not the best language. but its not the worst either.16:46
*** k_mouza has quit IRC16:47
* inc0 is greatly displeased with behavior of one named bash. Of course reson is that inc0 is also really bad at it, but that doesn't stop him to bash on one named bash16:47
kfox1111indeed.16:47
*** jmccarthy has left #openstack-kolla16:47
kfox1111launguages you don't know are always worse then the ones you do know. :)16:47
inc0that.16:48
kfox1111but, if you want to get into ops's heads to better understand them, knowing the language they usually use has some benifits. :)16:48
inc0well, clue is where do we draw a line and write it in something that was meant to be full fledged programming language;)16:49
inc0question*16:49
kfox1111I don't disagree.16:49
kfox1111shell isn't the best tool for all tasks.16:49
inc0it's like PHP16:49
kfox1111yeah.16:49
inc0it was meant to be templating language! and it's not horrible at that16:49
inc0problems started to arise when people started to build complex systems using templating language16:50
rwelluminc0: even using [ instead of [[ in pure bash throws up all sorts of issues.16:51
kfox1111inc0: yeah.16:51
kfox1111rwellum: yeah, those are two completely different animals.16:51
rwellumI think these scripts are shell called by a bash interpreter with some smatterings of bash around.16:51
*** lucasxu has quit IRC16:51
kfox1111thats like saying, when I swithc a { to a [ in c, they don't behave the same. :)16:52
kfox1111[ is an alais to the 'test' command.16:52
rwellumAt least in C you can compile :)16:52
kfox1111 [[ is a built in function.16:52
rwellumYup16:52
*** david-lyle has quit IRC16:52
kfox1111rwellum: compiling is a feature or an antifeature depending on who you ask. :)16:52
*** david-lyle has joined #openstack-kolla16:52
kfox1111its a good feature of a kernel.16:52
kfox1111its a bad feature of a shell. :)16:53
*** lpetrut has quit IRC16:53
*** prateek has joined #openstack-kolla16:53
kfox1111bash is in a pretty sweet spot with me in a shell that can do some programming tasks with ease, but not a full blown language.16:54
kfox1111its much more functonal the csh/command.com.16:54
kfox1111but doesn't try to be awk/python/etc.16:55
rwellumJust don't ever work in a team who has an ex professor of coding kfox1111. He destroyed my bash.16:55
rwellumFortunately he left.16:56
kfox1111hehe16:56
rwellumsbezverk: so I got the registry deployed and openvswitch is still in imagepullbackoff :(16:56
*** tonanhngo has joined #openstack-kolla16:57
rwellumsbezverk https://www.irccloud.com/pastebin/lIJ0EZZQ/16:58
*** lucasxu has joined #openstack-kolla16:58
rwellumhttps://www.irccloud.com/pastebin/5jnFuIbm/16:59
*** magicboiz has quit IRC17:00
sbezverkrwellum: please try to pull image manually17:01
*** rwallner_ has quit IRC17:01
sbezverkrwellum: million dollar question, why you have registry configured for 30402 port but you try to pull images from 30401??17:02
rwellummillion dollar answer - because I'm an idiot.17:03
rwellumI copied both from your files and then the helm install from what you pasted before17:03
*** sambetts is now known as sambetts|afk17:03
*** rwallner has joined #openstack-kolla17:03
rwellumchanged to correct port and image pulled :)17:04
*** mgoddard_ has quit IRC17:06
openstackgerritChason Chan proposed openstack/kolla-ansible master: Update the documentation link for doc migration  https://review.openstack.org/48516117:07
sbezverkrwellum nice!!17:07
*** vhosakot has quit IRC17:12
*** vhosakot has joined #openstack-kolla17:12
sbezverkrwellum please let me know if you see increase/decrease deployment time with registry17:15
rwellumsbezverk: total time yeah.  I am re-deploying with the port change. Expecting to see other issue because of the differences in our cloud.yaml files.17:19
sbezverkrwellum cool17:19
*** goldyfruit has quit IRC17:20
*** gfidente is now known as gfidente|afk17:22
*** prateek has quit IRC17:22
*** harlowja has joined #openstack-kolla17:23
rwellumsbezverk: in your helm install command, you are using ubuntu - stupid question - will that only work on a ubuntu VM?17:24
*** manheim has quit IRC17:25
sbezverkrwellum : no I was running two instance of registry one for centos and one for ubuntu, just for fun/testing purposes and unfortunately I pasted you ubuntu command17:25
*** david-lyle has quit IRC17:26
sbezverkthey will run distro independent17:26
rwellumok17:26
jamesbensonegonzalez see any other issues with bug: https://review.openstack.org/#/c/485704/17:26
rwellumIs there a default - i.e do I need to pass --set distro=centos etc17:27
*** harlowja has quit IRC17:27
sbezverkrwellum yes, please check values.yaml that comes with registry17:28
sbezverkyou will see all defaults17:28
*** vincent_1dk has joined #openstack-kolla17:32
*** vincent_1dk has quit IRC17:32
*** vincent_1dk has joined #openstack-kolla17:32
*** goldyfruit has joined #openstack-kolla17:34
*** vincent_1dk has quit IRC17:35
*** vincent_vdk has joined #openstack-kolla17:35
*** vincent_vdk has quit IRC17:38
*** vincent_vdk has joined #openstack-kolla17:39
*** mgoddard_ has joined #openstack-kolla17:42
*** vincent_vdk has left #openstack-kolla17:44
*** athomas has quit IRC17:51
rwellumsbezverk: I don't see svc_name=registry-centos though17:51
sbezverkrwellum : I do not follow, do you have a question?17:52
rwellumsbezverk: my bad I read values.yaml as cloud.yaml17:54
sbezverkrwellum got it, values.yaml gets automatically used by helm when you deploy a chart17:56
*** krtaylor has quit IRC17:56
sbezverkjust by virtue of being in the same folder as Chart.yaml17:56
*** leni1 has joined #openstack-kolla17:57
*** jascott1 has joined #openstack-kolla17:58
rwellumnice - thanks sbezverk17:58
*** david-lyle has joined #openstack-kolla18:03
*** harlowja has joined #openstack-kolla18:06
rwellumsbezverk: so right now, nova-api-create-cell is in int 2/3 and has been there for 14m total... That's the last pod.18:12
sbezverkrwellum : if you check v5 config, you will see that I disable cell to wait for compute to be up18:12
*** ducttap__ has quit IRC18:13
sbezverkit is the change I was talking about the other day18:13
*** manheim has joined #openstack-kolla18:13
sbezverkwith master nova service-list does not return anything until cell is created18:14
sbezverkbut we used in v4 this cli to detect is compute is completely up before we start cell18:14
rwellumsbezverk: I knew that and somehow missed it18:14
sbezverkrwellum no worries you are very close now ;)18:15
rwellumthe bad part is that it took nearly 1h to get here - so about 2x the time for v4...18:15
*** manheim has quit IRC18:18
*** ducttape_ has joined #openstack-kolla18:23
sbezverkrwellum hmm, I guess we will need to investigate. I do not see any time increase for two master/registry jobs running at the gate..18:24
sbezverkif you can collect kebectl describe pod for some pods like neutron or nova18:25
sbezverkwe should see the life cycle with timestamps18:26
rwellumsbezverk:  https://www.irccloud.com/pastebin/HtuVaUsB/18:27
*** krtaylor has joined #openstack-kolla18:28
sbezverkrwellum that one is not representative, as all images were already on the local host18:28
rwellumWhy'd it take so long then?18:29
openstackgerritMerged openstack/kolla master: Deprecate neutron-sfc-agent image  https://review.openstack.org/46370018:29
sbezverkrwellum not sure, it would require build complete time line form the moment you start chart deployment, then go through kubectl describe pod and then check for example nova-api log and see where it takes longest18:31
sbezverkmaybe it is new "feature" of master ;)18:31
rwellumso sbezverk the image are already on my localhost because I ran on this VM before?18:34
rwellumimages18:34
sbezverkrwellum do not think so, master is not on docker, so you could not get it18:35
*** vhosakot has quit IRC18:35
sbezverkI think previous nova related activity like nova manage db, pulled the image18:35
*** vhosakot has joined #openstack-kolla18:36
*** ducttape_ has quit IRC18:46
*** ducttape_ has joined #openstack-kolla19:02
sean-k-mooneyinc0: as an fyi https://review.openstack.org/#/c/477605/6 breaks the kolla bootstrap playbook behind proxies as there is noway to turn it off and it does not provide a way to spefiy the ntp servers to use19:09
inc0jamesbenson: ^19:09
inc0sean-k-mooney: let me fix that19:10
sean-k-mooneyinc0: it shoudl probaly template the ntp config based on https://github.com/openstack/kolla-ansible/blob/66ed21b991d7d8f68771e20b1d2ea5cde34085a8/ansible/group_vars/all.yml#L59-L6319:10
inc0yeah19:10
*** bmace has quit IRC19:10
inc0mind filing a bug?19:11
*** bmace has joined #openstack-kolla19:11
sean-k-mooneysure im just heading home but ill log one before i go19:11
jamesbensonsorry about that sean-k-mooney19:11
jamesbensoninc0: should we push the related bug too, still think this is valid ;-)19:12
sean-k-mooneyjamesbenson: no worries there are a few other things that should be added to properly support configuring proxies else were19:12
inc0well fix is easy to simply make a hook that allow ppl to disable it19:12
sean-k-mooneyjamesbenson: the quick fix is just add a flag to turn it off19:12
jamesbensonin the globals?19:13
sean-k-mooneyin the role defaults19:13
sean-k-mooneyjamesbenson: basically here https://github.com/openstack/kolla-ansible/blob/66ed21b991d7d8f68771e20b1d2ea5cde34085a8/ansible/roles/baremetal/defaults/main.yml#L919:14
sean-k-mooneyjamesbenson: just add enable_host_ntp: True and add when conndtions to the ntpd lines in post_install19:15
inc0fwiw this is one of last things happening, only reboot left, so even if it fails you should be good to deploy anyway19:16
inc0still needs fixing19:16
sean-k-mooneyinc0: yep speaking of which im going to refactor the reboot into a handeler when i respin https://review.openstack.org/#/c/408876/ tomorow19:16
inc0cool thanks Sean19:17
sean-k-mooneyjamesbenson: inc0 ill be touching the barmetal role anyway tomorow so if you want i can fix the ntp issue then too19:17
sean-k-mooneyin a speerate patch that is19:18
sean-k-mooneyi see you are plannign to back port it19:18
jamesbensonsure sean-k-mooney... can you send me the bug when you do, I'd like to see how it's done19:18
inc0jamesbenson: do you want to fix it? or I can do it19:18
jamesbensoninc0 I can do it if someone doesn't mind helping out19:19
inc0I do not19:19
jamesbensonI might need a little guidance as I've been doing only simple bug fixes19:19
jamesbensonotherwise sean-k-mooney can do it19:19
inc0idea is to basically create new var here: https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/baremetal/defaults/main.yml19:20
inc0that defaults to true19:20
inc0and then add when: statement to all ntp tasks "when: my_new_var | bool"19:20
inc0that's it19:20
jamesbensonah okay, shouldn't be too bad then...19:20
jamesbensonyeah I'll do it19:20
jamesbensonlove to get more bug fixes :-)19:20
inc0it's really simple change19:20
inc0cool, give it a shot and let me know if you run into any hurdles19:21
jamesbensonwill do19:21
jamesbensonbrb19:21
*** nrado has joined #openstack-kolla19:22
*** nrado has left #openstack-kolla19:22
jamesbensonback19:24
*** ducttape_ has quit IRC19:24
*** ducttape_ has joined #openstack-kolla19:25
sean-k-mooneyjamesbenson: bug open https://bugs.launchpad.net/kolla/+bug/170555819:28
*** dmsimard|cave is now known as dmsimard19:28
openstackLaunchpad bug 1705558 in kolla "the synchroizing time one-time task fails behind proxies." [Undecided,New]19:28
sean-k-mooneyinc0: once i finish up with the dpdk role i notice a few other issue with the barmatel role. 1 the hosts file update is broken. and second there were several steps i had to do by hand to confiugre the hosts and docker to use proxies.19:32
inc0cool, we are pretty close with odl role too19:33
inc0hopefully we can have ovs+dpdk+odl deployable by Pike19:33
sean-k-mooneyinc0: yes hopefully. ill have the outstanding comments in the ovs-dpdk role adressed tomroww or early next week. currently seting up some centos nodes to test dpdk. it works fine on ubuntu19:34
jamesbensonokay, sean-k-mooney, I assigned myself to it... I'll try to get that out today19:35
jamesbensonI'm going to open a ticket about some prechecks with external ceph...found some issues19:35
sean-k-mooneyi might open some bugs for the proxy changes for the barmetal role. it would be nice if the bootstrap playbook was able to fully configure the host and docker to work with our proxies.19:35
jamesbenson+119:36
sean-k-mooneyanyway im getting peckish so going to head home. catch ye tomorow19:37
*** itlinux has quit IRC19:38
*** itlinux_ has quit IRC19:38
jamesbensonlater19:39
*** rwsu has quit IRC19:52
openstackgerritJames Benson proposed openstack/kolla-ansible master: ntpd behind proxy fails  https://review.openstack.org/48570419:54
openstackgerritJames Benson proposed openstack/kolla-ansible master: ntpd behind proxy fails  https://review.openstack.org/48570419:55
*** openstack has joined #openstack-kolla20:00
openstackgerritJames Benson proposed openstack/kolla-ansible master: ntpd behind proxy fails  https://review.openstack.org/48570420:11
*** openstackgerrit has quit IRC20:17
*** rwsu has joined #openstack-kolla20:25
*** ansmith has quit IRC20:43
*** mgoddard_ has quit IRC20:49
*** jascott1 has quit IRC21:01
*** jascott1 has joined #openstack-kolla21:02
*** lucasxu has quit IRC21:02
rwellumsbezverk: no luck nova-api-create-cell not coming up. I don't think the wait for compute to be up logic is working for me.21:03
sbezverkrwellum have you added disable waiting into the cloud?21:04
*** schwicht has quit IRC21:05
rwellumsbezverk:  https://www.irccloud.com/pastebin/xLHDUJoe/21:05
sbezverkrwellum see this paste http://paste.openstack.org/show/616087/21:05
sbezverkit is working in my local test bed21:05
*** rhallisey has quit IRC21:05
*** jascott1_ has joined #openstack-kolla21:06
*** jascott1 has quit IRC21:06
*** krtaylor has quit IRC21:07
rwellumOne difference I see is your nova api is the same level as all:21:07
rwellumI also have "cell_enabled: true" - you do not21:09
*** eaguilar has quit IRC21:16
kfox1111arg.... half the groovy examples I can find for jenkins leave out vital information like what includes are needed. :/21:17
*** itlinux has joined #openstack-kolla21:19
*** itlinux_ has joined #openstack-kolla21:19
*** schwicht has joined #openstack-kolla21:23
sbezverkrwellum when in doubt about at which level one or another parameter should be, always check the corresponding template and its search path21:23
sbezverkusually it is first line, so you can see how templating will try to find a value of a variable21:24
sbezverkkfox1111: btw I submitter PR for registry chart :) waiting for a second round of critic21:25
kfox1111sbezverk: nice. thanks. :)21:25
*** gfidente|afk is now known as gfidente21:26
*** vhosakot has quit IRC21:28
*** vhosakot has joined #openstack-kolla21:28
*** omenv has quit IRC21:30
*** dave-mccowan has quit IRC21:31
*** ansmith has joined #openstack-kolla21:32
rwellumsbezverk: helm/microservice/registry-deployment/templates/registry.yaml ?21:32
sbezverkrwellum yep21:34
*** JoseMello has quit IRC21:36
*** manheim has joined #openstack-kolla21:36
*** goldyfruit has quit IRC21:46
*** rwallner has quit IRC21:48
*** rwallner has joined #openstack-kolla21:49
*** gfidente has quit IRC21:52
*** jascott1_ has quit IRC21:54
*** jascott1 has joined #openstack-kolla21:54
*** jascott1 has quit IRC21:56
*** jascott1 has joined #openstack-kolla21:56
*** rwallner has quit IRC21:59
*** jamesbenson has quit IRC22:00
*** jascott1 has quit IRC22:01
*** rwallner has joined #openstack-kolla22:03
*** rwallner_ has joined #openstack-kolla22:05
*** rwsu has quit IRC22:05
*** dave-mccowan has joined #openstack-kolla22:06
*** rwallner has quit IRC22:08
*** rwallner_ has quit IRC22:09
*** ducttape_ has quit IRC22:10
*** ducttape_ has joined #openstack-kolla22:11
*** vhosakot has quit IRC22:17
*** schwicht has quit IRC22:18
*** tonanhngo has quit IRC22:19
*** jascott1 has joined #openstack-kolla22:19
*** tonanhngo has joined #openstack-kolla22:19
*** tonanhngo has quit IRC22:19
*** manheim has quit IRC22:20
*** manheim has joined #openstack-kolla22:20
*** jascott1 has quit IRC22:24
*** dave-mccowan has quit IRC22:24
*** manheim has quit IRC22:25
*** tonanhngo has joined #openstack-kolla22:26
*** tonanhngo has quit IRC22:30
*** manheim has joined #openstack-kolla22:32
hawihmmm. fatal: [dc2-comp-001]: FAILED! => {"changed": false, "failed": true, "msg": "Please check the api_interface settings - interface eno1 is not active"}22:38
hawii defined it already for this host22:38
hawidoing prechecks and i'm having some boxes with different interface names - normal boxes have eno1 and eno2, compute ones have enp22s0f4 enp22s0f522:40
hawidc2-comp-001 api_interface=enp22s0f4 storage_interface=enp22s0f4 cluster_interface=enp22s0f4 tunnel_interface=enp22s0f4 dns_interface=enp22s0f4 neutron_external_interface=enp22s0f522:41
*** ipsecguy has quit IRC22:46
*** jascott1 has joined #openstack-kolla22:48
*** ipsecguy has joined #openstack-kolla22:52
*** manheim has quit IRC23:05
*** leni1 has quit IRC23:10
*** jamesbenson has joined #openstack-kolla23:32
*** jamesbenson has quit IRC23:36
*** ducttape_ has quit IRC23:47
*** eaguilar has joined #openstack-kolla23:52
*** zhangfei has joined #openstack-kolla23:53
*** david-lyle has quit IRC23:54

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!