Thursday, 2016-12-29

*** david-lyle has joined #openstack-kolla00:02
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move neutron precheck into its own role  https://review.openstack.org/41555200:04
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move neutron precheck into its own role  https://review.openstack.org/41555200:07
*** david-lyle has quit IRC00:07
*** jtriley has joined #openstack-kolla00:37
*** huzhengchuan has joined #openstack-kolla00:44
*** duonghq has joined #openstack-kolla00:45
*** jmccarthy has quit IRC00:47
*** jmccarthy has joined #openstack-kolla00:48
*** Pavo has joined #openstack-kolla00:49
duonghqmorning guys00:53
*** sayantan_ has quit IRC00:59
srwilkersmorning duonghq o.01:01
srwilkerso/01:01
openstackgerritjimmygc proposed openstack/kolla-ansible: Fix wrong register in ceph reconfigure task  https://review.openstack.org/41515001:06
duonghqhi srwilkers01:09
srwilkershows it going man?01:10
*** hfu has joined #openstack-kolla01:11
duonghqstill try to make ceph work. This new year eve is not important in Vietnam,01:11
duonghqhow about you?01:11
*** Pavo has quit IRC01:14
srwilkersnot bad. poking around with some ceph stuff too coincidentally01:16
duonghqsrwilkers, which ceph part are you working on?01:18
*** zhubingbing has joined #openstack-kolla01:19
duonghqdo you have vacation, srwilkers ?01:19
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move nova precheck into its own role  https://review.openstack.org/41556401:34
srwilkersduonghq, nope. no vacation for me. currently working on a side project with ceph01:36
duonghqsrwilkers, ya01:36
*** hfu has quit IRC01:37
*** pramodrj07 has quit IRC01:39
*** MasterOfBugs has quit IRC01:40
*** hfu has joined #openstack-kolla01:40
SamYapleinc0: oh man talk about your backwardness01:43
SamYapleinc0: so galera has automatic recovery unless....... all nodes gracefully stopped01:43
SamYapleas in, if all node gracefully stopped, it can't autorecover lol01:43
duonghqsrwilkers, where is source code of kollakube?01:45
Jeffrey4lSamYaple, could u review this? https://review.openstack.org/410057  iirc, we disabled horizon in gate long time ago. But i do not know why.01:52
srwilkersduonghq: https://github.com/openstack/kolla-kubernetes01:53
*** sacharya has joined #openstack-kolla01:53
duonghqsrwilkers, which dir in this repo?01:54
duonghqthe kollakube tool?01:54
SamYapleJeffrey4l: ran out of ram01:54
Jeffrey4lbut in my test, it only use 3G ram ( with horizon )01:54
Jeffrey4lwithout*01:54
SamYapleJeffrey4l: i left a review. im just telling you why it was originally disabled01:56
Jeffrey4lcool. thanks.01:57
*** sacharya has quit IRC01:58
*** jtriley has quit IRC02:01
*** david-lyle has joined #openstack-kolla02:04
*** fguillot has joined #openstack-kolla02:08
*** david-lyle has quit IRC02:08
*** gomarivera has joined #openstack-kolla02:20
openstackgerritMerged openstack/kolla: Download kafka binaries from within the container  https://review.openstack.org/41245202:26
v1k0d3nduonghq: what srwilkers is trying to say is that he's addicted to this stuff.02:34
v1k0d3nhe could walk away...but NNOOoooOO...just has to keep learning stuff.02:34
v1k0d3n:)02:34
inc0SamYaple, lol02:37
inc0so best way to stop galera is to just kill -9 it?02:37
SamYapleinc0: actually, yea02:38
SamYapleinc0: if you want it to auto recover02:38
SamYapleits so funny. but thats like official advice02:38
inc02016 - a year to remember02:39
*** dave-mccowan has joined #openstack-kolla02:41
SamYapleinc0: but in all seriousness, the autorecovery scenarios are perfectly predicatable now02:41
SamYapleif data exists, but gvwstate.dat doesnt exist on any node, it can't autorecover02:42
duonghqv1k0d3n, thanks02:52
*** fguillot has quit IRC02:54
*** gomarivera has quit IRC02:56
v1k0d3nduonghq: haha for what?03:07
v1k0d3ni was just joking :) srwilkers is a machine...good dude.03:07
*** caowei has joined #openstack-kolla03:07
*** sacharya has joined #openstack-kolla03:19
duonghqv1k0d3n, just understand something, you broke my mind loop03:20
duonghqit's so good03:20
*** huzhengchuan has quit IRC03:26
*** huzhengchuan has joined #openstack-kolla03:26
*** dave-mccowan has quit IRC03:28
srwilkersoh hey v1k0d3n03:32
srwilkersduonghq, mindloops are terrible. need to break: every so often03:32
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move mariadb precheck into its own role  https://review.openstack.org/41558703:33
srwilkersv1k0d3n, machines dont have legs, so i literally cant walk away :(03:34
v1k0d3ni'm sure you could find a desk that walks away from you now-a-days.03:35
v1k0d3ni want that.03:35
v1k0d3njust pull out the office...pull in the living room with couch, tv and wife!03:35
v1k0d3ngood to go. never have to leave03:35
srwilkershaha. solves all our problems03:35
v1k0d3nwell, and the full room VR03:36
v1k0d3ni need that automatically set up please lol03:36
v1k0d3ntripods and all03:36
v1k0d3n:)03:36
*** srwilkers has quit IRC03:41
*** prameswar has joined #openstack-kolla04:04
*** david-lyle has joined #openstack-kolla04:05
*** david-lyle has quit IRC04:10
*** hfu has quit IRC04:10
*** bmace has joined #openstack-kolla04:13
*** mdnadeem has joined #openstack-kolla04:18
*** sayantan_ has joined #openstack-kolla04:54
*** fragatina has quit IRC04:56
*** fragatina has joined #openstack-kolla04:56
openstackgerritsayantani proposed openstack/kolla: Clean up kolla-ansible related files from Kolla  https://review.openstack.org/40642904:58
*** fragatina has quit IRC05:01
*** sayantan_ has quit IRC05:03
*** sayantan_ has joined #openstack-kolla05:04
*** Pavo has joined #openstack-kolla05:08
*** sayantan_ has quit IRC05:15
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move cinder precheck into its own role  https://review.openstack.org/41549205:21
*** Pavo has quit IRC05:25
*** maniram477 has joined #openstack-kolla05:36
maniram477Hi all..05:36
*** hfu has joined #openstack-kolla05:42
*** gomarivera has joined #openstack-kolla05:48
*** maniram477 has quit IRC05:50
*** l4yerffeJ has quit IRC06:02
*** l4yerffeJ has joined #openstack-kolla06:03
openstackgerritzhongshengping proposed openstack/kolla-ansible: Remove signing_dir option  https://review.openstack.org/41560706:03
*** britthouser5 has quit IRC06:06
*** britthouser has joined #openstack-kolla06:06
*** david-lyle has joined #openstack-kolla06:07
*** sayantan_ has joined #openstack-kolla06:07
*** l4yerffeJ has quit IRC06:08
*** l4yerffeJ has joined #openstack-kolla06:08
*** saneax-_-|AFK is now known as saneax06:09
*** david-lyle has quit IRC06:12
*** caowei has quit IRC06:13
*** prameswar has quit IRC06:23
*** satyar has joined #openstack-kolla06:26
satyarHi jeffrey4l06:27
Jeffrey4lsup satyar06:28
*** sacharya has quit IRC06:28
satyari was getting a syntax error with the code "openstack compute service list -f json --service nova-compute"06:28
satyartemplate error06:29
Jeffrey4lsatyar, logs?06:29
satyarwill be posting you in a bit06:31
*** prameswar has joined #openstack-kolla06:35
*** gomarivera has quit IRC06:36
*** inc0 has quit IRC06:44
portdirecto/ morning06:51
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: optimize bootstrap.yml file  https://review.openstack.org/41550406:52
*** sayantan_ has quit IRC07:01
openstackgerritKuo-tung Kao proposed openstack/kolla-ansible: provide a universal collectd template  https://review.openstack.org/41449907:13
satyarjeffrey4l http://paste.openstack.org/show/593568/07:15
*** sacharya has joined #openstack-kolla07:29
*** sacharya has quit IRC07:34
satyarjeffrey4l there?07:36
openstackgerritzhubingbing proposed openstack/kolla: Add zun support in README.rst  https://review.openstack.org/41562807:46
openstackgerritcaoyuan proposed openstack/kolla-ansible: Move ceilometer precheck into its own role  https://review.openstack.org/41563508:04
*** david-lyle has joined #openstack-kolla08:09
openstackgerritzhubingbing proposed openstack/kolla: Add zun container  https://review.openstack.org/41061608:09
*** salv-orlando has joined #openstack-kolla08:09
*** david-lyle has quit IRC08:13
*** awidders_ has joined #openstack-kolla08:16
-openstackstatus- NOTICE: All CI tests are currently broken since logs.openstack.org is down. Refrain from recheck or approval until this is fixed.08:17
*** ChanServ changes topic to "All CI tests are currently broken since logs.openstack.org is down. Refrain from recheck or approval until this is fixed."08:17
*** awiddersheim has quit IRC08:19
*** yatin is now known as yatin_on_leave08:19
Jeffrey4lsatyar, still around?08:19
*** matrohon has quit IRC08:21
*** yatin_on_leave has quit IRC08:24
openstackgerritcaoyuan proposed openstack/kolla-ansible: Update the horizon port into all.yml  https://review.openstack.org/41549508:30
openstackgerritSurya Prakash Singh proposed openstack/kolla: Adding missing package in kolla  https://review.openstack.org/41564208:36
*** msimonin has joined #openstack-kolla08:51
*** ocaner has joined #openstack-kolla08:51
*** hfu has quit IRC08:52
*** prameswar has quit IRC08:52
satyarjeffrey4l yes08:52
Jeffrey4lsatyar, are u deploying or upgrading in the logs?08:53
*** huzhengchuan has quit IRC08:53
satyarupgrade08:53
Jeffrey4lhave u try run openstack compute service manually?08:54
satyaryes08:54
satyarthat works08:54
Jeffrey4lwhat's the output?08:54
Jeffrey4lok.08:54
Jeffrey4lweird08:55
satyaryes08:56
satyari removed that code08:56
satyarand upgrade works08:56
satyarfrom l3 to l708:56
ocanerHi guys!08:57
satyarjeffrey4l service list result "http://paste.openstack.org/show/593580/"08:58
satyarhi ocaner08:58
openstackgerritSurya Prakash Singh proposed openstack/kolla-ansible: Adding "coverage" missing package in kolla-ansible  https://review.openstack.org/41564709:00
Jeffrey4lcan u try split the task into two? the first just run the command and register the variable, second  print the variable. and check what we got.09:00
satyarsure09:00
satyari am not sure but seems like something wrong with "until: "{{ nova_compute_services.stdout|from_json|length }} != 0""09:01
Jeffrey4lyep. nova_compute_services.stdout get some wrong.09:03
Jeffrey4lansible raise badly error message.09:03
satyaryes09:03
satyarfor transport url i added " --transport-url rabbit://{% for host in groups['rabbitmq'] %}{{ rabbitmq_user }}:{{ rabbitmq_password }}@{% if orchestration_engine == 'KUBERNETES' %}rabbitmq{% else %}{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}{% endif %}:{{ rabbitmq_port }}{% if not loop.last %},{% endif %}{% endfor %}"09:04
ocanerI've a question regarding the change https://review.openstack.org/#/c/411808 and the bug report https://bugs.launchpad.net/kolla/+bug/1650534, the bug is assigned to me, but I don't know what the expected action is?09:05
openstackLaunchpad bug 1650534 in kolla newton "HA router fails using neutron-vpn-agent" [High,Triaged]09:05
satyaras it was giving error transport url is missing09:05
Jeffrey4lhey ocaner09:05
Jeffrey4lsatyar, seems good09:05
satyar:)09:06
Jeffrey4locaner, the fix LGTM.09:10
ocaneranything that needs to be done from my side?09:11
ocaners/anything/anything left/09:11
satyarjeffrey4l please check "http://paste.openstack.org/show/593581/"09:16
Jeffrey4locaner, i think no . i will +2 for you patch.09:18
* sp_ is away: leave a message if you have something to say09:18
Jeffrey4lneed another core member +2+w to merge. ;) ocaner09:19
Jeffrey4lsatyar, the openstack compute xxx need be moved to kolla_toolbox. so i guess some library is miss when running it in the host.09:23
*** sp_ has quit IRC09:24
ocanerJeffrey4l, thanks for the +2, I'm on vacation next week and wanted to complete the work on the change09:25
Jeffrey4locaner, i pinged another core to review. it should be merged soon ;)09:26
ocanerJeffrey4l, thanks09:27
satyar(kolla-toolbox)[ansible@server1 /]$ openstack compute service list -f json --service nova-compute ->09:29
satyarargument of type 'OSC_Config' is not iterable09:29
satyari guess thats the issue09:30
*** sacharya has joined #openstack-kolla09:30
satyarjeffery4l09:30
Jeffrey4lroger.09:33
satyarbut it seems i have the latest one09:34
satyarhttp://paste.openstack.org/show/593582/09:34
*** sacharya has quit IRC09:35
satyarjeffrey4l09:37
Jeffrey4lbusy. moment.09:37
satyar:)09:38
*** duonghq has quit IRC09:56
*** tovin07 has quit IRC09:58
*** prameswar has joined #openstack-kolla10:07
*** satyar has quit IRC10:10
*** hfu has joined #openstack-kolla10:12
*** kong__ has joined #openstack-kolla10:24
*** turvey_ has joined #openstack-kolla10:24
*** smekel__ has joined #openstack-kolla10:24
*** cliles_ has joined #openstack-kolla10:24
*** serverascode_ has joined #openstack-kolla10:25
*** tonyb has quit IRC10:30
*** cliles has quit IRC10:30
*** apuimedo has quit IRC10:30
*** classicsnail has quit IRC10:30
*** jascott1 has quit IRC10:30
*** otavio_ has quit IRC10:30
*** n0isyn0ise has quit IRC10:30
*** timss has quit IRC10:30
*** Kdecherf has quit IRC10:30
*** smekel_ has quit IRC10:30
*** srwilker- has quit IRC10:30
*** jgriffith has quit IRC10:30
*** turvey has quit IRC10:30
*** kong_ has quit IRC10:30
*** jroll has quit IRC10:30
*** serverascode has quit IRC10:30
*** cloudnull has quit IRC10:30
*** turvey_ is now known as turvey10:31
*** pcaruana has joined #openstack-kolla10:31
*** smekel__ is now known as smekel_10:31
*** kong__ is now known as kong_10:31
*** serverascode_ is now known as serverascode10:35
*** tonyb has joined #openstack-kolla10:36
*** apuimedo has joined #openstack-kolla10:36
*** jascott1 has joined #openstack-kolla10:36
*** otavio_ has joined #openstack-kolla10:36
*** n0isyn0ise has joined #openstack-kolla10:36
*** timss has joined #openstack-kolla10:36
*** srwilker- has joined #openstack-kolla10:36
*** jgriffith has joined #openstack-kolla10:36
*** cloudnull has joined #openstack-kolla10:36
*** Kdecherf has joined #openstack-kolla10:38
*** jroll has joined #openstack-kolla10:39
*** classicsnail has joined #openstack-kolla10:40
*** msimonin has quit IRC10:42
*** msimonin has joined #openstack-kolla10:42
*** msimonin has quit IRC10:43
*** mdnadeem has quit IRC10:43
*** zhubingbing has quit IRC10:48
*** mdnadeem has joined #openstack-kolla10:56
*** klindgren has quit IRC11:00
*** saneax is now known as saneax-_-|AFK11:03
openstackgerritjimmygc proposed openstack/kolla: Add sanity check in macros.j2  https://review.openstack.org/41484811:04
*** ChanServ changes topic to "Have support questions? Please give ask.openstack.org a try. New to Kolla? Please read the documentation here: http://docs.openstack.org/developer/kolla/; Kolla IRC meetngs on Wednesdays @ 16:00 UTC - see agenda @ https://goo.gl/OXB0DL - IRC channel is *LOGGED* @ http://goo.gl/3mzZ7b"11:08
-openstackstatus- NOTICE: logs.openstack.org is up again. Feel free to recheck any failures.11:08
*** david-lyle has joined #openstack-kolla11:11
*** klindgren has joined #openstack-kolla11:14
*** haplo37_ has quit IRC11:14
*** david-lyle has quit IRC11:15
*** haplo37_ has joined #openstack-kolla11:23
*** pcaruana has quit IRC11:28
*** sacharya has joined #openstack-kolla11:31
*** sacharya has quit IRC11:35
*** brad[] has quit IRC11:44
openstackgerritYumengBao proposed openstack/kolla: Fix Docker registry port error in image-building.rst  https://review.openstack.org/41567811:47
*** brad[] has joined #openstack-kolla11:52
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: optimize bootstrap.yml file  https://review.openstack.org/41550411:56
openstackgerritYumengBao proposed openstack/kolla: Fix Docker registry port error in image-building.rst  https://review.openstack.org/41567812:02
*** skramaja has quit IRC12:10
openstackgerritJeffrey Zhang proposed openstack/kolla-ansible: optimize bootstrap.yml file  https://review.openstack.org/41550412:12
*** hfu has quit IRC12:28
*** mdnadeem has quit IRC12:40
*** mdnadeem has joined #openstack-kolla12:54
*** mdnadeem has quit IRC12:57
*** mdnadeem has joined #openstack-kolla12:57
*** jheroux has joined #openstack-kolla13:08
*** mdnadeem has quit IRC13:09
*** david-lyle has joined #openstack-kolla13:12
*** salv-orl_ has joined #openstack-kolla13:13
*** salv-orlando has quit IRC13:16
*** david-lyle has quit IRC13:17
*** eaguilar has joined #openstack-kolla13:18
*** hfu has joined #openstack-kolla13:22
*** lamt has quit IRC13:24
*** zhubingbing has joined #openstack-kolla13:24
*** hfu has quit IRC13:25
*** hfu has joined #openstack-kolla13:27
*** sacharya has joined #openstack-kolla13:31
*** hfu has quit IRC13:33
*** eaguilar has quit IRC13:35
*** sacharya has quit IRC13:36
*** fguillot has joined #openstack-kolla13:41
*** lamt has joined #openstack-kolla13:53
*** goldyfruit has joined #openstack-kolla13:57
*** eaguilar has joined #openstack-kolla13:58
*** Bico_Fino has joined #openstack-kolla13:59
ocanerJeffrey4l, are you there?14:01
*** Satyap has joined #openstack-kolla14:02
*** salv-orl_ has quit IRC14:04
*** Bico_Fino has quit IRC14:08
*** britthouser6 has joined #openstack-kolla14:13
*** britthouser has quit IRC14:13
*** salv-orlando has joined #openstack-kolla14:17
*** Bico_Fino has joined #openstack-kolla14:17
*** Bico_Fino_ has joined #openstack-kolla14:22
*** Bico_Fino has quit IRC14:22
*** Bico_Fino_ is now known as Bico_Fino14:22
*** satyar has joined #openstack-kolla14:22
*** TxGirlGeek has joined #openstack-kolla14:36
*** britthouser6 has quit IRC14:43
*** britthouser has joined #openstack-kolla14:44
*** Bico_Fino has quit IRC14:45
*** TxGirlGeek has quit IRC14:47
satyarjeffrey4l i tried running nova service-list from kolla_toolbox asking for openrc14:48
*** salv-orl_ has joined #openstack-kolla14:49
*** salv-orlando has quit IRC14:49
*** TxGirlGeek has joined #openstack-kolla14:50
*** david-lyle has joined #openstack-kolla15:14
*** Pavo has joined #openstack-kolla15:16
*** dave-mccowan has joined #openstack-kolla15:16
*** david-lyle has quit IRC15:18
Satyapi am deploying openstack services using latest trunk code. after 20 mins the cinder volume container all tcp connections ( connection with rabbit mq and mysql) are lost. after that once  in every 2 to 3 mins connections are getting reestablished and terminated.15:19
SatyapDoes anyone observe this issue with latest trunk code15:20
*** sacharya has joined #openstack-kolla15:32
*** sacharya has quit IRC15:37
*** jtriley has joined #openstack-kolla15:41
Jeffrey4locaner, yep.15:42
ocanerJeffrey4l, jenkins added a -2, because gate-kolla-dsvm-build-ubuntu-source-ubuntu-xenial failed, from the logs it seems that tarballs.openstack.org couldn't be reached, could somebody retrigger the jenkins build?15:44
Jeffrey4llet me check.15:45
ocanerJeffrey4l, I think SamYaple retriggered it15:45
Jeffrey4lroger. let us wait for the gate result.15:46
ocaneryeah :)15:48
*** Bico_Fino has joined #openstack-kolla16:11
*** eaguilar has quit IRC16:11
*** brad[] has quit IRC16:11
*** dave-mcc_ has joined #openstack-kolla16:17
*** haplo37_ has quit IRC16:18
*** srwilkers has joined #openstack-kolla16:18
*** dave-mccowan has quit IRC16:19
*** brad[] has joined #openstack-kolla16:22
*** dave-mcc_ has quit IRC16:24
*** haplo37_ has joined #openstack-kolla16:26
*** sayantani has joined #openstack-kolla16:27
*** jemcevoy has joined #openstack-kolla16:29
openstackgerritMerged openstack/kolla-kubernetes: Fix python tox -e cover failure  https://review.openstack.org/41527316:33
*** zhubingbing has quit IRC16:34
srwilkersdupengfei, ping16:38
jemcevoyPavo: I am trying to change neutron to use vlan provider networking... Where do I make the config changes?  I ran kolla-ansible genconfig then edited multiple neutron.conf files in /etc/kolla then ran kolla-ansible reconfigure...  No configs in the cluster changed.  What am I doing wrong?16:39
Pavocreate a folder on the deployer named /etc/kolla/config/neutron16:40
Pavoone sec16:40
Pavothen create a file in that directory named ml2_conf.ini16:41
Pavothen edit that file and add this16:41
Pavohttp://paste.openstack.org/show/593613/16:42
Pavothen do a kolla-ansible reconfigure -i /usr/share/kolla/ansible/inventory/multinode16:42
Pavothat will allow you to use neutron with vlans as either tenant or provider16:43
jemcevoyI was following these directions: http://docs.openstack.org/liberty/networking-guide/scenario-provider-ovs.html#controller-node16:43
Pavoyou do understand there difference between tenant and provider networks correct?16:43
jemcevoyWith provider I need to setup the network manually to my switch16:44
Pavook thats fine but you need to add those settings in the ml2_conf.ini file on the deployer16:44
jemcevoySo the file only need the values I want to change from what is in the ml2_conf.ini that is deployed into the docker image...16:46
jemcevoycorrect?16:46
Pavoyeap16:46
Pavothats the beauty of reconfigure16:47
jemcevoyOk here goes again....  Thanks16:47
Pavoit keeps default template values and merges the changes that you need16:47
Pavobut only in the sections of the conf file that you specify16:47
Pavoif you want to make changes to a conf that do not have sections as in [blah] you have to have every setting in the conf file that you want changes in, example would be horizon/local_settings.ini file, it doesn't have sections16:48
Pavoso you have to basically copy the entire conf file and make what ever changes you want in and then do a reconfigure16:49
Pavobut remember to always make a folder in the /etc/kolla/config file for what ever service you are changing16:49
Pavotook me a few days to figure that out16:50
*** fguillot has quit IRC16:52
*** Bico_Fino_ has joined #openstack-kolla16:52
*** Bico_Fino has quit IRC16:55
*** Bico_Fino_ is now known as Bico_Fino16:55
*** eaguilar has joined #openstack-kolla17:11
*** david-lyle has joined #openstack-kolla17:15
openstackgerritMerged openstack/kolla: HA router fails using neutron-vpn-agent  https://review.openstack.org/41180817:17
ocanerYippie yeah :)17:18
jemcevoyPavo:  One more question: Does the file /etc/neutron/plugins/ml2/openvswitch_agent.ini go into the directory /etc/kolla/config/neutron/plugins/ml2 ?17:19
*** david-lyle has quit IRC17:20
Pavono goes into /etc/kolla/config/neutron17:20
jemcevoyThanks17:23
*** msimonin has joined #openstack-kolla17:27
*** prameswar has quit IRC17:29
*** sacharya has joined #openstack-kolla17:33
*** harlowja has joined #openstack-kolla17:36
*** msimonin has quit IRC17:36
*** sacharya has quit IRC17:38
*** prameswar has joined #openstack-kolla17:41
jrich523hey guys, what CLI command do i use to show the ports used on a network/subnet? in Horizon its under the network, but i asssume it would be from the subnet?17:41
ocanerjrich523, neutron port-list for all and neutron port-list | grep SUBNET_UUID for a specific subnet17:43
jrich523thanks!17:43
jrich523ugh i hate when you've entered the shell version of it that you dont get grep :(17:44
jrich523i should just create one letter aliases for each cli :)17:44
ocanerjrich523, yeah or the neutron port-list should support some kind of query parameters17:47
jrich523there seems to be an "ip floating list" which is really (i think) what i want17:47
jrich523i actualy want to test what happens with the script when a float IP is set, that hasnt been created17:48
jrich523so i need to find all floats in use or not, so i can find one that hasnt been created, to make sure the script will create it if need be17:48
jrich523oh that shows, because the fixed ip for that output would be none if not in use, so they are all here... nice17:49
*** jtriley has quit IRC17:54
ocanerjrich523, yes this should be the case, also the port id should be none, too17:54
jrich523yeah, perfect, thanks :)17:54
jrich523so that jsut failed... guess it doesnt catch that fact and just dies lol17:55
*** inc0 has joined #openstack-kolla17:56
inc0good day to the best team in whole of openstack17:57
jrich523well hello to you too inc0 :)17:57
ocanerhi inc017:57
jrich523i think this script is actually working, its amazing!17:57
inc0which script? (curiosity)17:58
jrich523oh, its an ansible playbook to create instances17:58
jrich523sadly it works via the cli since the module that is provided requires libraries that my broken server cant update :(17:58
ocanerjrich523, are you using the os_server module with the shade library?18:00
jrich523nope18:00
jrich523two issues, 1, shade was being a whore (i told the engineer to add the install to the playbook, not sure he tried it)18:00
ocanerwhy not?18:00
jrich523but the real issue was that the RHEL server (Tower/Ansible) isnt correctly registered (RHELs fault) and i cant update the repo's18:01
jrich523so i cant update pyOpenSSL (its at like 0.13, current is 16.x)18:01
jrich523i get some 509 error, which is apparently related to that18:01
SamYaplewhats wrong with shade?18:01
jrich523nothing18:01
SamYapledamn right18:01
jrich523its a smaller, probably more solvable problem18:02
jrich523so, jsut as an FYI, if you have RHEL, and you buy the hyper visor based lic (unlimited guests) you MUST run RHEL as the hypervisor18:02
*** sayantan_ has joined #openstack-kolla18:02
jrich523they have no method for handling it if the hypervisor isnt RHEL18:02
inc0jrich523, ever tried virtualenv?18:03
jrich523i think that might be part of my problem18:03
jrich523i updated the pyOpenSSL via pip, but its still choking, the system version is the 0.1318:04
inc0virtualenv is safely installed and separated python libs18:04
jrich523yeah, i jsut dont know how to tell Tower which to use18:04
inc0ahh...it might not help with pyopenssl18:04
jrich523which is the real problem18:04
jrich523which is why i assumed, maybe adding it to the playbook might fix it?18:04
inc0ok never used tower18:04
jrich523me either :)18:04
inc0might want to ask on #ansible18:04
jrich523i tried18:04
jrich523they are "open source only"18:04
jrich523i quickly got shot down as soon as i said "tower"18:05
inc0you're saying that they don't support for things you pay for18:05
inc0isn't it like...I pay for it?18:05
jrich523lol yes18:05
inc0so you should help me?18:05
jrich523they expect me to contact support18:05
inc0my god18:05
jrich523i mean... i get it... but usually there is a 'best effort' approach18:05
jrich523who wants to call support? so slow and painful18:05
inc0and expensive for company18:06
inc0really18:06
jrich523this client pays for it (you might have to for tower?)18:06
SamYaplejrich523: they used to best-effort tower in #ansible18:06
ocanerRH should open source tower this would be the best18:06
SamYaplebut that got shutdown18:06
jrich523no, someone should make tower like platform and opensource it :)18:06
jrich523force RHEL to eat it18:06
jrich523its not even that impressive really18:07
jrich523im going to build a jenkins/ansible server to play around with, see if i can get it close to what tower offers18:07
inc0out of curiosity...tower is gplv3?18:07
ocanerI never used tower, but it has some features which are nice18:07
jrich523yeah, i dont dislike it... just not all that impressive... figured a community member would have made something ages ago18:08
inc0or if it isn't could you do grep -r "import ansible" . on tower sources?18:08
jrich523i will say, compared to other options, i do prefer ansible18:08
inc0would be fun if they did import gplv3 code in tower18:08
jrich523interesting... hmm18:08
*** MasterOfBugs has joined #openstack-kolla18:14
*** pramodrj07 has joined #openstack-kolla18:14
SatyapHi inc0 , I am facing below issue with latest trunk code18:19
Satyapi am deploying openstack services using latest trunk code. after 20 mins the cinder volume container all tcp connections ( connection with rabbit mq and mysql) are lost. after that once  in every 2 to 3 mins connections are getting reestablished and terminated.18:19
SatyapAny pointers or suggestions will help me18:20
inc0so, you deploy with success, all is working and after 20min it dies?18:20
inc0is rabbitmq alive?18:20
Satyapdeployment is successfull, first 20 mins the rabbitmq and mysql tcp connections are stable18:21
Satyapafter 20 mins the connections keeps getting closed reconnected and closed again18:21
inc0rabbitmq is alive after 20min18:21
inc0?18:21
Satyaprabbitmq and mysql are fine18:21
Satyapyes its alive18:21
inc0can you ping them from cinder nodes?18:22
Satyapand cinder api and cinder schedulet mysql and rebbitmq connections are also stable18:22
Satyaponly problem with cinder volume container18:22
Satyapyes i can ping18:22
inc0do you have STP enabled in this network?18:23
Satyapthe between connect and close is around 3 mins18:23
Satyapyou mean ntp18:23
inc0no...I mean stp18:23
inc0spanning tree protocol18:23
Satyapi have not checked that18:23
inc0my first guess would be loop in L2 networking18:24
inc0but that's a long shot18:24
Satyapi am wondering why it is happening only to cinder volume18:25
Satyapcinder api and cinder scheduler container tcp conections with mysql and rabbitmq are stable18:25
inc0because I bet they're on same host as rabbitmq and mysql18:25
Satyapeven my cinder volume is also on controller node18:26
Satyapfrom there it connects to ceph cluster18:26
inc0can you run top?18:27
inc0to see if you're not running out of memory?18:27
Satyapyes I RAN18:27
Satyapit was below 10%18:27
Satyapusage18:27
inc0uhh18:27
inc0can you do docker ps18:28
inc0and see if one of containers has low uptime18:28
Satyapyes docker ps show all services are up18:28
Satyapuptime is proper18:28
Satyapit seems in cinder-volume container  with in initial 20 mins resources are over utilized18:29
Satyapafter that it is becoming unstable18:29
Satyapbut cinder container uses host networking, if any networking resources are over utilized then it should have effect on other containers but that is not happening18:30
inc0we also don't use cgroups so any other resource utilized should affect others18:31
srwilkershello o/18:31
inc0hmm18:31
inc0you're running ceph there too right?18:31
Satyapyes ceph cluster is running on other servers18:32
inc0hey srwilker18:32
inc0ok so it's not affecting anything18:32
Satyapon controller node we are running cinder-volume that talks to ceph cluster18:32
inc0yeah that makes sense, but have you tried to move cinder volume to different node?18:32
Satyaponly issue i can see is cinder process wise consumption of resources18:33
Satyapi mean cinder volume container process wise consumption of resources18:33
Satyapbut not able to figure out how to check that18:33
Satyapi did not move cinder volume to other node18:34
Satyapwill try that18:34
SatyapDo you have any other suggestions18:35
inc0not for nw18:35
inc0we need to narrow the issue18:35
Satyaphow can we do that18:35
SamYapleSatyap: check to see how many open file handles you have18:36
SamYapleincreasing the open file handles is rabbitmq performance fixing 10118:36
Satyaphere rabbitmq is not terminating the connection because it has not heared hearbeat messages from cinder volume18:37
Satyapin rabbitmq logs i can see this18:39
Satyap=ERROR REPORT==== 27-Dec-2016::19:36:05 === closing AMQP connection <0.11343.0> (x.x.x.x:39353 -> x.x.x.x:5672 - cinder-volume:63:db4b0016-15e2-486d-831f-2bce7e396a1b): missed heartbeats from client, timeout: 60s18:39
Satyapbased on the above log culprit is cinder-volume i guess18:39
inc0yup18:39
Satyapinitial 20 mins everything is working fine18:41
Satyapmy guess is cinder volume process which is running container is over utilizing something because of this it happened18:42
inc0again, no cgroups so it can take all of resources of host18:43
inc0hence top would show it18:43
Satyapif it is taking all resources of host then it should effect other containers also because we are using host networking for all containers18:44
inc0yup18:44
Satyapbut other containers connections are stable18:44
inc0thats why I doubt it's basic resource utilization18:44
inc0maybe something less obvious18:44
inc0so let's move cinder vol to different host to limit noise18:45
Satyapwill try that and get back to you by tomorrow18:45
inc0yeah, Jeffrey should be on during your day too18:45
Satyapdo you think any issue with oslo.db package or any oslo packages18:46
inc0seriously doubt that18:46
inc0but I have no idea what it can be now18:46
Satyaphow to check that18:46
inc0unless you know whats wrong, it's going to be hard18:46
Satyapwhile searching in google there was some connection issue which got resolved after changing oslo db package version18:47
inc0all the oslo packages will be newest18:47
inc0in container18:47
Satyapok18:47
SatyapThanks18:48
*** ocaner has quit IRC18:50
SatyapOne clue I found, while checking netstat -s on controller i found below values for tcp connections18:52
SatyapTcp:     16782371 active connections openings     239386 passive connection openings     2768005 failed connection attempts     2887493 connection resets received18:52
Satyap16782371 active connections openings is very huge value18:52
Satyapdo you think it is effecting the connections?18:52
*** harlowja has quit IRC18:53
inc0hard to tell18:59
inc0but if we separate cinder-volume it's going to be easier to say18:59
*** ssurana has joined #openstack-kolla18:59
Satyapok I will change that19:02
Satyapthanks for your time19:02
awidders_I noticed that in the example multinode inventory19:03
awidders_nova targets the control group... shouldn't it use the compute group instead?19:03
awidders_or am I just crazy19:04
awidders_https://github.com/openstack/kolla-ansible/blob/master/ansible/inventory/multinode#L82-L8319:05
*** prameswar has quit IRC19:07
awidders_this is all a little awkward actually beacuse the nova tasks target both compute and nova19:09
awidders_and nova targets control19:09
inc0awidders_, well most of nova services are in control19:10
inc0all besides nova-compute19:10
awidders_ah ok19:11
awidders_maybe that is my disconnect19:11
inc0https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/start_compute.yml#L3119:12
awidders_ah ok19:12
awidders_make sense now thanks19:12
inc0np19:13
*** dave-mccowan has joined #openstack-kolla19:13
*** salv-orlando has joined #openstack-kolla19:13
*** Satyap has quit IRC19:13
*** sacharya has joined #openstack-kolla19:14
*** salv-orl_ has quit IRC19:16
*** Satyap has joined #openstack-kolla19:25
*** dave-mccowan has quit IRC19:32
openstackgerritAndrew Widdersheim proposed openstack/kolla-ansible: Fix typo in multinode inventory  https://review.openstack.org/41574819:35
*** hogepodge has quit IRC19:39
*** Bico_Fino has quit IRC19:40
openstackgerritAndrew Widdersheim proposed openstack/kolla: Add footer to Ceilometer base image  https://review.openstack.org/41574919:43
SatyapHi inc0 , Just now I verified my issue after moving cinder volume to network node but again same issue after 20 mins the connections became unstable19:43
inc0interesting19:44
Satyapany other thoughts19:44
inc0did you check amount of connections on new node?19:44
Satyapnetstat -s output for tcp is given below19:45
SatyapTcp:     162800 active connections openings     25 passive connection openings     2 failed connection attempts     3 connection resets received     56 connections established     14739103 segments received     13934904 segments send out     3899 segments retransmited     51 bad segments received.     14 resets sent19:45
SatyapTCP active connections openings are keep on increasing with time19:47
inc0check if it's growing19:47
inc0kk19:47
Satyapyes it is growing rapidly19:47
inc0so it seems cinder-volume is running out of connections after 20min19:47
Satyap168011 active connections openings19:47
Satyapjust now19:47
Satyapincreased more than 5000 in just mins19:48
inc0seems like cinder issue to me19:48
Satyap3 mins19:48
inc0cinder logs doesn't show anything special?19:48
Satyapin cinder logs when the connection become unstable it shows disconnected message for both rabbit and mysql19:49
inc0and nothing before that? are you using debug logging?19:49
Satyapand also i can see debug log19:49
Satyapopening connection to ceph cluster (timeout=-1). _connect_to_rados /var/lib/kolla/venv/lib/python2.7/site-packages/cinder/volume/drivers/rbd.py:22019:50
Satyaponly above logs for every few seconds19:50
inc0check netstat -natp19:51
Satyapi can see few entries , most of TIME_WAIT is with ceph cluster19:53
inc0soo, is ceph cluster healthy?19:54
Satyapyes it is heathy19:54
Satyapno issue from ceph side , i tried to copy volumes from one pool to other pool that is also working19:55
inc0I ran out of ideas...might want to check it with cinder guys19:55
inc0it's seems to be issue of cinder-volumes spawning tons of connections19:55
*** sacharya has quit IRC19:56
Satyapfor me also it seems like that19:56
Satyapwill check with cinder team thanks19:56
*** sacharya has joined #openstack-kolla19:56
*** Pavo has quit IRC19:57
*** sacharya has quit IRC19:58
*** sacharya has joined #openstack-kolla19:59
*** TxGirlGeek has quit IRC20:01
*** TxGirlGeek has joined #openstack-kolla20:01
*** sacharya has quit IRC20:03
*** sayantan_ has quit IRC20:04
*** sayantani has quit IRC20:08
*** msimonin has joined #openstack-kolla20:08
satyarHi inc020:09
satyarHow are you?20:09
satyarthis is Satya_ :)20:09
inc0it's Satya P or R?;)20:11
satyarR :)20:11
inc0ok20:12
satyarsatyar wass Satya_20:12
inc0ok, got it20:12
satyarregarding the above conversation20:12
satyari saw there are 2 process cinder volume is running at the container20:13
satyaris it expected?20:13
inc0both of them being cinder-volume?20:13
inc0brb, going to eat something20:13
satyarPID USER      PR  NI    VIRT    RES    SHR S  %CPU %MEM     TIME+ COMMAND20:13
satyar   48 cinder    20   0 6640884 158804  18584 R 231.0  0.1 104:04.05 cinder-volume20:13
satyar   23 cinder    20   0  636016 102516  16452 S   0.7  0.0   0:39.43 cinder-volume20:13
satyaryes20:13
*** david-lyle has joined #openstack-kolla20:17
Satyapinc0 , what you said is coorect %CPU for cinder-volume process is 231.0%20:17
Satyapit means it is doing something wrong20:17
*** david-lyle has quit IRC20:21
*** Satyap has quit IRC20:36
*** srwilkers has quit IRC20:37
*** harlowja has joined #openstack-kolla20:49
*** eaguilar has quit IRC20:51
*** sacharya has joined #openstack-kolla20:59
*** satyar has quit IRC20:59
*** sacharya has quit IRC21:04
*** l4yerffeJ has quit IRC21:08
*** l4yerffeJ has joined #openstack-kolla21:08
*** msimonin has quit IRC21:23
*** salv-orlando has quit IRC21:23
*** l4yerffeJ_ has joined #openstack-kolla21:30
*** l4yerffeJ__ has joined #openstack-kolla21:33
*** salv-orlando has joined #openstack-kolla21:34
*** Jeffrey4l has quit IRC21:34
*** l4yerffeJ has quit IRC21:34
*** l4yerffeJ_ has quit IRC21:35
*** kbyrne has joined #openstack-kolla21:36
*** TxGirlGeek has quit IRC21:59
*** TxGirlGeek has joined #openstack-kolla22:05
*** TxGirlGeek has quit IRC22:14
*** msimonin has joined #openstack-kolla22:17
*** goldyfruit has quit IRC22:19
*** msimonin1 has joined #openstack-kolla22:19
inc0anyone from k8s around? kfox1111 srwilker-22:20
inc0others22:20
*** TxGirlGeek has joined #openstack-kolla22:20
*** msimonin has quit IRC22:22
*** rbergero1 is now known as rbergeron22:27
*** rbergeron has joined #openstack-kolla22:27
*** TxGirlGeek has quit IRC22:27
*** TxGirlGeek has joined #openstack-kolla22:30
*** jheroux has quit IRC22:31
*** sacharya has joined #openstack-kolla22:32
*** haplo37_ has quit IRC22:52
*** TxGirlGeek has quit IRC22:55
*** msimonin1 has quit IRC22:55
*** inc0 has quit IRC22:56
*** haplo37_ has joined #openstack-kolla23:01
*** zhangshuai has quit IRC23:08
*** zhangshuai has joined #openstack-kolla23:08
*** lamt has quit IRC23:10
*** TxGirlGeek has joined #openstack-kolla23:14
*** TxGirlGeek has quit IRC23:17
*** mattmceuen is now known as mattmceuen_away23:20
*** TxGirlGeek has joined #openstack-kolla23:21
*** TxGirlGeek has quit IRC23:27
*** salv-orlando has quit IRC23:29
*** Jeffrey4l has joined #openstack-kolla23:31
*** TxGirlGeek has joined #openstack-kolla23:38
*** awidders_ has quit IRC23:42
*** TxGirlGeek has quit IRC23:43
*** awiddersheim has joined #openstack-kolla23:47

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!