Sunday, 2016-04-03

*** salv-orlando has joined #openstack-kolla00:01
*** salv-orlando has quit IRC00:06
*** iceyao has joined #openstack-kolla00:08
sdakewell my nova vm booting gets stuck in spawning state00:20
sdakei find that highly uninspiring00:20
sdakeanyone else run int othis problem?00:21
sdake2016-04-03 00:22:24.469 1 ERROR nova.compute.manager [instance: c758ed8c-5ae0-4bd1-9f40-7721045557d8] BuildAbortException: Build of instance c758ed8c-5ae0-4bd1-9f40-7721045557d8 aborted: Failed to allocate the network(s), not rescheduling.00:27
sdakewell i'm gong to trust the community on this one and go a head and tag00:31
sdakeand sume something is busted in my environment00:31
*** achanda has quit IRC00:31
sdakerhallisey ping00:33
sdakeneed this merged: https://review.openstack.org/#/c/300829/00:33
patchbotsdake: patch 300829 - kolla (stable/mitaka) - Restoring openstack_release option00:33
ccesario__sdake: keystone logs still remain in /var/log/kolla/apache2/00:34
rhalliseysdake, k00:34
sdakeccesario__ is there a bug for that00:34
ccesario__sdake: yes https://review.openstack.org/#/c/297967/00:35
patchbotccesario__: patch 297967 - kolla - Change keystone log dir00:35
ccesario__not merged yet00:35
sdakerhallisey please ack https://review.openstack.org/#/c/297967/00:36
patchbotsdake: patch 297967 - kolla - Change keystone log dir00:36
sdakethat tempet problem is known and ifxed - i think a rebae of your patch wuld fix it but i'm not certain00:39
openstackgerritMerged openstack/kolla: Restoring openstack_release option  https://review.openstack.org/30082900:40
sdakerhallisey merge https://review.openstack.org/#/c/297967/ pleae00:40
patchbotsdake: patch 297967 - kolla - Change keystone log dir00:40
sdakevikram has tested it00:40
sdakerhallisey PING00:42
sdakenihilifer ?00:42
sdakemandre ?00:42
sdakecomone folks throw me a bone00:42
rhalliseyk all set00:44
*** weiyu has joined #openstack-kolla00:44
sdakewe need a backpor t on it00:44
*** iceyao has quit IRC00:44
openstackgerritSteven Dake proposed openstack/kolla: Change keystone log dir  https://review.openstack.org/30083900:44
sdakerhallisey ack pz :000:44
*** weiyu has quit IRC00:45
*** iceyao has joined #openstack-kolla00:45
sdakerhallisey ack that backport plz00:45
sdakeand then ill wait for those patche sto merge00:46
sdakeand tag00:46
sdakesquirly launchpad is00:46
ccesario__bbiab ...00:47
openstackgerritMerged openstack/kolla: Change keystone log dir  https://review.openstack.org/29796700:48
*** iceyao has quit IRC00:50
*** sdake_ has joined #openstack-kolla00:52
*** sdake has quit IRC00:54
*** sdake_ is now known as sdake01:00
*** weiyu has joined #openstack-kolla01:00
rhalliseyI'm off for the night01:05
rhalliseysee ya guys tmr01:05
*** rhallisey has quit IRC01:06
openstackgerritMerged openstack/kolla: Change keystone log dir  https://review.openstack.org/30083901:13
*** SiRiuS has quit IRC01:14
*** Jeffrey4l_ has joined #openstack-kolla01:15
*** salv-orlando has joined #openstack-kolla01:16
*** ccesario_ has joined #openstack-kolla01:21
*** weiyu has quit IRC01:22
*** salv-orlando has quit IRC01:23
*** vhosakot has joined #openstack-kolla01:25
*** dwalsh has quit IRC01:25
*** iceyao has joined #openstack-kolla01:25
*** achanda has joined #openstack-kolla01:32
*** achanda has quit IRC01:37
ccesario_vhosakot hi ... Any tip about ?  https://review.openstack.org/#/c/297984/01:39
patchbotccesario_: patch 297984 - kolla - Added hostname param into kolla_docker01:39
vhosakothey ccesario_: I am seeing error when I test that PS.. but, the error I see is different that what you've posted in gerrit review01:41
sdakehttps://github.com/openstack/kolla/releases/tag/2.0.0.0rc301:43
ccesario_What error? Because the patch only it is executed when hostname param is set...01:43
ccesario_sdake .. cool!01:44
vhosakotccesario_: the mariadb container is not starting in my env.... let me pull the latest master01:45
ccesario_right01:45
vhosakotsdake: nice! 25 bugs fixed per week is awesome :)01:45
*** jasonsb has joined #openstack-kolla01:45
*** jasonsb_ has joined #openstack-kolla01:48
*** sdake has quit IRC01:48
*** achanda has joined #openstack-kolla01:49
*** sdake has joined #openstack-kolla01:49
*** jasonsb has quit IRC01:49
vhosakotsdake: saw the mails about release model... so, is kolla is tagged as release:cycle-with-milestones, does it show up in http://releases.openstack.org/independent.html ?01:51
sdakevhosakot we are tagged as independent atm01:54
sdakewhich is wrong01:54
sdakei will get it fixed in newton01:54
sdakechanging your release model is frozen for 2 months01:55
sdakeso we will show up at that link once the patches merge01:55
vhosakotah ok.. cool..so, what should will the new tag be ?  cycle-with-milestones ?01:55
sdakewhat i want to happen is for our wor for liberty and mitaka to show up in the integrated release01:55
sdakeyup01:55
sdakeratherthen rleleae:indepdentn01:55
vhosakotah ok01:55
sdakewe were not told of this plan to creat ethe releaes website01:56
sdakeand i didn't filter it from my openstack-dev ml appropriately01:56
sdakewhen we joined the big tent there was no releaes.oepnstack.org01:56
sdakewhich really blows, becaue now it looks like we have a bunch of independent releases01:56
sdakewhen one is for liberty and another is for mitaka01:56
sdakeworst case, we will tag a z stream and add that to lierty and mitaka01:57
sdakeand histroy will be lost01:57
vhosakotkolla is not in  http://releases.openstack.org/mitaka/index.html  due to same reason (independant tag)01:57
vhosakot?01:57
sdakeit will not be in there01:57
sdakei have a  patch here:01:57
sdakehttps://review.openstack.org/#/c/297976/01:58
patchbotsdake: patch 297976 - releases - Add kolla releases to _independent01:58
vhosakotah cool01:59
vhosakotccesario_: can you pastebin your new /usr/local/share/kolla/ansible/roles/mariadb/tasks/start.yml in which you have added hostname: "hostdatabase" ?01:59
sdakesorry guys i drpped the ball on this one01:59
sdakethere is a bunch of fstuff that needs to happen to project config and other repos to make all this happen02:00
sdakeand it didn't start getting done until the last month02:00
*** jasonsb_ has quit IRC02:00
sdakeall completely new to me02:01
sdakehad to learn the hard way02:01
vhosakotcool... thanks for those reviews... hopefully, the release team will correct it.. plus, kolla needs some time (2-3 weeks) to validate other OpenStack projects (like nova, glance, neutron, keystone, etc) after they release and kolla must build them and verify/fix breakages02:02
ccesario__vhosakot: http://paste.openstack.org/show/492825/02:03
sdakeNo, I am your father.02:04
sdakevhosakot that is what z streams are for02:04
vhosakotah ok cool02:05
*** vhosakot has quit IRC02:07
*** vhosakot has joined #openstack-kolla02:08
vhosakotcool02:08
vhosakotccesario: thanks for the link02:08
ccesario_Your welcome!02:10
*** sdake has quit IRC02:12
*** sdake has joined #openstack-kolla02:13
sdakevhosakot fwiw i tried for 10 days and got 1 week instead https://review.openstack.org/#/c/298278/02:14
patchbotsdake: patch 298278 - governance - Add type:deployment to release:cycle_with_miestones02:14
vhosakotah ok cool02:16
sdakewe can get our act together in 7 days02:20
sdake2 weeks wuld be better02:20
sdakebut it is what it is atm02:20
sdakemore red tape ftl02:20
*** ccesario_ has quit IRC02:25
*** ccesario_ has joined #openstack-kolla02:25
*** SiRiuS has joined #openstack-kolla02:41
*** weiyu_ has joined #openstack-kolla02:43
*** dave-mccowan has joined #openstack-kolla02:48
*** weiyu_ has quit IRC02:50
*** salv-orlando has joined #openstack-kolla02:53
*** klint has joined #openstack-kolla02:55
*** salv-orlando has quit IRC03:10
*** weiyu has joined #openstack-kolla03:13
*** SiRiuS has quit IRC03:15
*** weiyu has quit IRC03:26
*** achanda has quit IRC03:26
*** achanda has joined #openstack-kolla03:33
*** achanda has quit IRC03:48
*** vhosakot has quit IRC04:07
*** sdake_ has joined #openstack-kolla04:24
*** sdake has quit IRC04:27
*** salv-orlando has joined #openstack-kolla04:32
*** salv-orlando has quit IRC04:45
*** achanda has joined #openstack-kolla04:50
*** sdake_ has quit IRC05:35
*** salv-orlando has joined #openstack-kolla06:03
*** salv-orlando has quit IRC06:10
*** iceyao has quit IRC06:15
*** ccesario_ has quit IRC07:12
*** Serlex has joined #openstack-kolla07:15
openstackgerritJeffrey Zhang proposed openstack/kolla: Implement nova-ssh container  https://review.openstack.org/29796307:21
openstackgerritJeffrey Zhang proposed openstack/kolla: Track the api_interface in the keepalived  https://review.openstack.org/30085207:38
*** achanda has quit IRC08:03
*** Jeffrey4l__ has joined #openstack-kolla08:11
*** achanda has joined #openstack-kolla08:12
*** Jeffrey4l_ has quit IRC08:14
*** iceyao has joined #openstack-kolla08:45
*** ccesario_ has joined #openstack-kolla08:57
*** mbound has joined #openstack-kolla08:59
*** salv-orlando has joined #openstack-kolla09:07
*** salv-orlando has quit IRC09:16
*** mbound has quit IRC09:21
*** dwalsh has joined #openstack-kolla09:35
*** achanda has quit IRC09:42
*** allen_gao has quit IRC10:02
*** iceyao has quit IRC10:04
*** allen_gao has joined #openstack-kolla10:07
*** mbound has joined #openstack-kolla10:21
*** mbound has quit IRC10:26
*** salv-orlando has joined #openstack-kolla10:27
*** salv-orlando has quit IRC10:30
*** iceyao has joined #openstack-kolla10:51
*** Serlex has quit IRC11:04
*** ccesario_ has quit IRC11:10
*** salv-orlando has joined #openstack-kolla12:09
*** salv-orlando has quit IRC12:11
*** rhallisey has joined #openstack-kolla12:12
*** rhallisey has quit IRC12:17
*** rhallisey has joined #openstack-kolla12:29
*** rhallisey has quit IRC12:35
*** rhallisey has joined #openstack-kolla12:38
*** rhallisey has quit IRC12:43
*** rhallisey has joined #openstack-kolla12:44
ccesario__morning!12:45
*** rhallisey has quit IRC12:49
*** klint has quit IRC12:58
*** rhallisey has joined #openstack-kolla13:17
*** stvnoyes has quit IRC13:21
*** stvnoyes has joined #openstack-kolla13:21
*** rhallisey has quit IRC13:22
*** SiRiuS has joined #openstack-kolla13:23
*** rhallisey has joined #openstack-kolla13:27
*** rhallisey has quit IRC13:34
*** salv-orlando has joined #openstack-kolla13:45
*** macsz_ has joined #openstack-kolla13:45
*** macsz_ has quit IRC13:46
*** macsz_ has joined #openstack-kolla13:46
*** macsz_ has quit IRC13:46
*** dwalsh has quit IRC13:50
*** macsz_ has joined #openstack-kolla13:52
*** macsz_ is now known as macsz13:52
*** salv-orlando has quit IRC13:52
*** sdake has joined #openstack-kolla14:00
*** salv-orlando has joined #openstack-kolla14:07
*** salv-orlando has quit IRC14:11
*** unicell1 has joined #openstack-kolla14:40
*** unicell has quit IRC14:40
*** ajafo has quit IRC14:55
*** ajafo has joined #openstack-kolla14:55
ccesario__sdake: ping?15:05
ccesario__sdake: http://paste.openstack.org/show/492840/15:05
sdakeccesario__ sup15:06
sdakeyou got haproxy logging working?15:07
sdakenice job!15:07
sdakegot a patch in the queue to review?15:07
*** salv-orlando has joined #openstack-kolla15:08
ccesario__sdake: not yet .. let me provide15:09
sdakeccesario__ the centralized logging per node is pretty well done, no?15:14
sdakeI realize it may have some gaps15:14
sdakelike keepalived and haproxy :)15:14
*** mgoddard has quit IRC15:17
*** SiRiuS has quit IRC15:18
ccesario__sdake: look this.... http://pastebin.com/2GQVyged15:19
ccesario__for haproxy it works :P15:19
sdakeya i saw your work15:19
sdakei mean i saw the output of your work15:19
sdakepretty sure it wasn't working previosuly15:19
sdakebut what do yo think about how logging works?15:20
sdakegood/bad/indifferent?15:20
ccesario__sdake: very very nice.... excellent feature15:20
sdakeelemoine_ ftw :)15:20
sdakealthough alot of other people have helped fill in the gaps15:21
ccesario__it help a lot debbuging ...  very very easy15:21
sdakehis initial implementaiton was incomplete15:21
sdakeplus with alicja's work, we centralize logging into elasticsearch and kibana15:21
sdakeopen a web page to your local network on port 560115:21
ccesario__sdake: my currently deploy does not contain kibana :P15:22
ccesario__I need check it.. I 'm curious about kibana15:22
*** salv-orlando has quit IRC15:26
openstackgerritCarlos Cesario proposed openstack/kolla: Added missing haproxy log dir.  https://review.openstack.org/30088815:27
sbezverkgood morning, anyone has latest kolla deployed openstack up and running? I saw some wierdness and wanted to check if it is a local or global issue.15:28
sbezverkwhen I create new flavor, I get this error: nova.objects.flavor Main database contains 5 unmigrated flavors, never seen that before..15:29
ccesario__sdake: https://review.openstack.org/30088815:30
sdakeccesario__ what is your launchpad id15:32
ccesario__sdake: ccesario15:34
sdakeok i assigned that bug to you15:34
ccesario__k15:35
sdakesurprised that is all ther ewa to fixing it15:35
sdakenothing needed in the haproxy config?15:35
sdakesbezverk i'll look after i finish up with this work with ccesario__ \15:36
ccesario__sdake: nothing ... therefore I fixed it :P15:36
sdakei htink i have it deployed15:36
*** mbound has joined #openstack-kolla15:36
sdakeccesario__ your patch doesn't have any changes to haproxy.cfg.j215:36
sdakeare any needed?15:36
sdakeor does haproxy log to /var/log/kolla/haproxy.log by default?15:36
ccesario__sdake: no changes into haproxy.cfg.j215:39
sdakecoool15:39
sdakecan you look at solving same problem for keepalived?15:39
ccesario__ haproxy.cfg.j2 logs point to log /var/lib/kolla/heka/log local015:40
sdakeor are there keepalived logs already15:40
sdakeso your patch mounts the named volume for keepalived and haproxy15:41
sdakehowever it only creates the haproxy directory15:41
sdakeshould it also create the keepalived directory?15:41
ccesario__sdake: http://paste.openstack.org/show/492841/15:42
sdakeok so i think there is a followup bug here to solve15:43
sdakeand that is gettign keepalived logs in /var/log/kolla/keepalived15:44
sdakebut that can be solved as a different patch15:44
*** dwalsh has joined #openstack-kolla15:45
sdakeok sbezverk15:46
sdakemoment to switch networks15:46
ccesario__sdake: ok15:46
*** sdake_ has joined #openstack-kolla15:47
sdake_sbezverk which test do you wnat me to run?15:47
sdake_create a flavor?15:48
sdake_my system was built and deployed laslt night from the tip of stable/mitaka15:48
sdake_sbezverk do you use init-runonce?15:48
sdake_ccesario mind I ask where you are employed?15:50
*** sdake has quit IRC15:50
sbezverksdake please just create a new flavor15:50
sdake_give me exact command you want me to type15:50
sbezverknova flavor-create xrv9k-32G_14xvCPU auto 32768 55 1415:51
sdake_ERROR (Conflict): Not all flavors have been migrated to the API database (HTTP 409) (Request-ID: req-843c7d2b-a80e-4421-a3b9-b7140b361c22)15:52
sbezverkyeeessss it is not just me then15:53
sdake_please file a bug15:53
sbezverksdake sure will do15:53
sdake_perhaps we need to do some api database migration in our bootstrap15:53
sdake_sbezverk any chance you cn track this bug down15:53
sdake_i'd say not being ale to create a flavor is pretty serious15:53
sdake_some places to look might be devstack codebase15:54
sdake_for what may be missing in our bootstrapping15:55
sbezverksdake yep but it will take bit of time as I will need to find nova folks to help me.. and today is sunday15:55
sdake_this also affects upgrade i suspect15:55
sdake_nova people wont help you15:55
sdake_they are not a support channel15:55
*** SiRiuS has joined #openstack-kolla15:55
sbezverksdake got it, will try to deal with it15:55
sdake_try/do ;)15:56
sdake_when you get teh bug filed link it plz15:56
sdake_and i'll triage it appropriately15:56
sdake_and confirm it15:56
sdake_which fails complaining about the flavor not been migrated and so that I should run nova-manage db migrate_flavor_data before.15:57
sdake_https://bugs.launchpad.net/nova/+bug/151146615:58
openstackLaunchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed]15:58
sdake_i think we are missing bootstrap steps15:58
sbezverksdake here you go: https://bugs.launchpad.net/kolla/+bug/156549715:59
openstackLaunchpad bug 1565497 in kolla "unable to create new flavors" [Undecided,New]15:59
ccesario__sdake_: I 'm from Brazil .... I work at small company www.tecnomega.com.br :)16:02
sbezverkccesario__ I hope you got already some tickets for Olympics ;-)16:02
sdake_ccesario__ cool - your company doing anything with kolla or just intereted in it personally16:03
ccesario__sbezverk: ahahaha not yet man :(16:04
sdake_ccesario__ well in either case, welcome aboard the development team ;)16:05
ccesario__sdake_: the both ... the company has a openstack project  to be deployed/created ... and I 'm in group that will do it..... and after a lots of tests, tools... I proposed to team usage it :)16:06
sdake_ccesario__ sweet16:07
ccesario__since then... I 'm personally testing and helpin with minimal changes :)16:07
sdake_creating a deployment tool on your own is daunting16:07
sdake_i htink yoru changes are starting to add up:)16:07
sbezverksdake in normal opensatck installation nova DB gets prepped by thids command: su -s /bin/sh -c "nova-manage db sync" nova16:07
sdake_can you file a bug for the keepalived in the correct directory thing16:07
sdake_and link it so i cn triage it16:08
sdake_sbezverk yes but there is also another migration step required16:08
sdake_if you RTFM :)16:08
sdake_read the bug i linked in the bug in our launchpad16:08
sdake_we are missing the api migration step16:08
sdake_sbezverk do you use init-runonce or create your networks your own way?16:09
sbezverksdake this one: nova-manage api_db sync?16:09
sbezverksdake I built my own networks16:10
sdake_yes api_db sync16:10
sdake_do we hve that?16:11
sdake_there is also a flavor migrate16:11
ccesario__sdake_: I though the kolla concepts vry cool.... and simple.... no needed a lot of changes ... like TripleO (my personally opinion)16:11
sdake_which migrates the llalvors to the api database16:11
ccesario__sdake_: sure... let create a file bug16:11
sdake_read the bug i linked sbezverk16:11
sdake_it has some pointers16:11
sdake_so I get this when i try to start a nova vm:16:12
*** Jeffrey4l__ has quit IRC16:12
SiRiuShas anyone ever experienced this error when trying to create an instance? -> http://paste.openstack.org/show/492842/16:12
sdake_Build of instance c758ed8c-5ae0-4bd1-9f40-7721045557d8 aborted: Failed to allocate the network(s), not rescheduling.16:13
SiRiuSlibvirtError: Start job for unit machine-qemu\x2d3\x2dinstance\x2d00000002.scope failed with 'failed'16:13
sdake_sbezverk i assume that doesnt happen to you?16:13
SiRiuSvery useful error message : failed with 'failed'16:13
sdake_api_server 'http://glance-api-glance-api-openstack-vagrant.marathon.mesos:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)16:14
sdake_i have a fix for that somewhere on my harddisk16:15
SiRiuSsdake_, wow, awesome, please give :)16:15
sbezverksdake_ nope, flavor is the only issue I see so far.. evcerything else worked fine..16:15
sdake_let me find it16:15
sdake_ok well I need a working environment16:16
sdake_sbezverk any chance ou can try with init-runonce before edigging into that nova db flavor bug16:16
ccesario__sdake_: https://bugs.launchpad.net/kolla/+bug/156549916:16
openstackLaunchpad bug 1565499 in kolla "Keepalived logs needs to be writed on correct directory" [Undecided,New]16:16
sdake_and see if its a problem with the init-runonce bug?16:16
sdake_ccesario__ are you taking that keepalived bug or should I leae it unassigned16:18
*** dwalsh has quit IRC16:18
sdake_sbezverk it means doing a fresh deploy16:18
sbezverksdake_ can I change ip addresses used in init-runonce?16:20
sdake_sbezverk i guess if you need to16:20
sdake_i'd like to try it unmoidifed tho16:20
sdake_to see if the problem is with my environment or with init-runonce16:20
sdake_the vms dont have to actually work they just have ot enter ACTiVe state16:21
ccesario__sdake_: please leave it unassigned... I have no idea how change it.... because the keepalived does not have logoutput dir config .... It seems that heka make the magic ....  I need understand better it16:21
sdake_i dont care if you can network from them or not ;)16:21
sdake_cool re kolla being simple, i agree we have optimized kolla in such a way as to be fast for developers and operators to pick up and understand the design16:22
sbezverksdake ok let me get on it16:22
sdake_rather then requiriing a rocket scientist to understand16:22
sdake_sbezverk thanks a illion16:22
sdake_btw i'm probably going to be kicking everyone off the ddrivers team except the core reviewers16:23
sdake_because peopel are not doing bug  tirage properly16:23
sdake_let me train the core reviewers on how to do bug triage first16:23
sdake_;-)16:23
sdake_dont  take offense at this if it happens to any of you ;)16:24
*** iceyao has quit IRC16:26
sbezverksdake_ I ran init-runonce do not see issues16:32
elemoine_ccesario__: keepalived does not write its logs to a file16:33
sbezverksdake_ it did not launch any VM but it correctly ctreated cirros image, sec group, and quotas..16:33
elemoine_keepalived writes its logs to the /dev/log Unix socket16:35
elemoine_heka reads keepalived logs from that socket and writes them to /var/log/kolla/haproxy/keepalived.log16:35
elemoine_ccesario__: see https://github.com/openstack/kolla/blob/master/ansible/roles/common/templates/heka-haproxy.toml.j2#L1416:36
elemoine_that's the path where heka writes the keepalived logs on the filesystem16:36
sdake_sbezverk please launch a vm - that is what doens't work for me16:38
sdake_elemoine_ thanks for the assistance ;-)16:38
elemoine_I am currently commenting in the launchpad bug16:39
sdake_elemoine_ thanks ;)16:39
sdake_ccesario__ do you see any other heka gaps in your deployment16:40
sbezverk3f4e5688-050f-4325-b4b5-074705f1cb3e | cirros-1 | ACTIVE  | -          | Running     | demo-net=10.0.0.316:41
sbezverksdake I need to leave for about 3 hours, when I am back I will digg into flavor bug, unless you decide to re-assign it to core reviewrs ;-)16:43
*** salv-orlando has joined #openstack-kolla16:44
*** salv-orlando has quit IRC16:46
sdake_hey folks i just wanted to let you all know i will be afk for 2-3 hours16:47
sdake_sbezverk you own it now ;)16:47
sdake_sbezverk thanks - not sure what the deal is with the network assignment thign16:47
sdake_my tile jsut finished laying yesterday and i am goign to put my office back together so i can have my multinode lab operational16:48
SiRiuSsdake_, did you find the fix?16:50
ccesario__elemoine_: yes yes.. I  have changed it and testing in my env16:53
ccesario__sdake_: finishing th deploy16:54
ccesario__elemoine_: thanks by tip!16:54
*** sdake_ has quit IRC16:58
ccesario__elemoine_: http://paste.openstack.org/show/492844/17:05
ccesario__:D17:05
ccesario__elemoine_: I will make the changes that you suggest..... because I have changed directly into heka-haproxy.tom.j217:09
*** sdake has joined #openstack-kolla17:10
sdakeccesario what was that?17:10
ccesario__sdake: http://paste.openstack.org/show/492844/17:11
ccesario__sdake: for while this.... http://pastebin.com/wfkyiajw17:13
ccesario__but I will split the configs as elemoine_ suggest17:13
ccesario__sdake: question.... is it possible send these using too this ps https://review.openstack.org/#/c/300888/    and add two Closes-Bug entry ?17:19
patchbotccesario__: patch 300888 - kolla - Added missing haproxy log dir.17:19
*** sdake has quit IRC17:22
*** mbound has quit IRC17:45
*** salv-orlando has joined #openstack-kolla18:12
*** salv-orlando has quit IRC18:19
*** mdnadeem has quit IRC18:25
*** achanda has joined #openstack-kolla18:30
openstackgerritCarlos Cesario proposed openstack/kolla: Make keepalived logs write in correct directory.  https://review.openstack.org/30089618:33
ccesario__elemoine_: ^18:36
*** mbound has joined #openstack-kolla18:46
*** mbound has quit IRC18:50
*** ccesario_ has joined #openstack-kolla18:52
*** daneyon has joined #openstack-kolla19:00
openstackgerritCarlos Cesario proposed openstack/kolla: Make Heka write Keepalived logs to dedicated directory  https://review.openstack.org/30089619:04
*** achanda has quit IRC19:22
*** salv-orlando has joined #openstack-kolla19:37
*** sdake has joined #openstack-kolla19:47
sdakehey folks19:48
sdakeccesario what is your question19:48
sdakei'm back from office reconstruction while taking a break19:48
sdakenot quite done yet19:48
*** salv-orlando has quit IRC19:50
*** sdake_ has joined #openstack-kolla19:51
*** SiRiuS has quit IRC19:52
*** sdake has quit IRC19:53
*** SiRiuS has joined #openstack-kolla19:57
SiRiuSFile "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 417, in build_instances context, request_spec, filter_properties) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 463, in _schedule_instances hosts = self.scheduler_client.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/utils.py", line 370, in wrapped return func(*20:19
SiRiuSargs, **kwargs) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations return self.queryclient.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/quer20:19
SiRiuSy.py", line 32, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 126, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call retry=self.retry) File "/var/lib/kolla/venv/lib/python2.7/site-packa20:19
SiRiuSges/oslo_messaging/transport.py", line 90, in _send timeout=timeout, retry=retry) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send retry=retry) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send raise result20:20
SiRiuSsdake_, any ideas on why I can't create an instance ?20:20
sdake_oh dont paste in channel unless its a 1 liner plz20:21
sdake_put in a paste tool so i can read it properly20:21
sdake_thanks :)20:21
*** daneyon_ has joined #openstack-kolla20:21
SiRiuSsdake_, http://paste.openstack.org/show/492846/20:22
SiRiuShow could I debug this?20:23
sdake_need full log not the assertion20:24
sdake_vms not starting is hard to debug20:24
*** daneyon has quit IRC20:24
sdake_grep for ERROR20:24
sdake_usually there is an error before the assertio nthat throws you a bone20:24
SiRiuSsdake_, http://paste.openstack.org/show/492842/20:24
SiRiuSfrom nova compute20:24
sdake_fwiw I can't boot a vvm because no networks ar efound20:24
sdake_look in nova-api log file too20:25
sdake_nova-compute isn't very helpful for debugging20:25
sdake_2016-04-03 16:08:49.537 43 WARNING nova.image.glance [req-57352596-c2fe-4a73-84f8-301299460fc9 7031b527e44d43ce9cf5feb2b79a40e4 d374cd289132475f86c4258b0f882ced - - -] No protocol specified in for api_server 'http://glance-api-glance-api-openstack-vagrant.marathon.mesos:9292', please update [glance] api_servers with fully qualified url including scheme (http / https)20:26
sdake_file a bug for that please20:26
sdake_and link it in irc20:26
sdake_and i'll post a fix when I have an opportunity20:27
SiRiuSsdake_, ok20:27
*** daneyon_ has quit IRC20:27
SiRiuSsdake_, also, that is not the issue here :)20:27
sdake_possibly not20:27
sdake_but it could be ;)20:27
SiRiuSI looked at the code and it adds http:// automatically20:28
SiRiuSin the config file it's glance-api-glance-api-openstack-vagrant.marathon.mesos:929220:28
sdake_i see20:28
sdake_this is a mesos bug20:28
SiRiuSand the code will add a http in front and give a friendly warning20:28
sdake_this is also an ansible bug20:28
sdake_turn on debug in nova.conf20:29
sdake_reconfigure20:29
sdake_and paste logs20:29
sdake_wit reconfgiure not implemented for kolla-mesos20:29
sdake_probably need to redeploy with debug enabled20:29
sdake_instead20:29
SiRiuSsdake_, https://github.com/openstack/nova/blob/master/nova/image/glance.py#L12720:31
SiRiuSit's not really a big issue20:31
sdake_its an issue because we can end up with different protocols for internal and external networks20:37
sdake_currently internal is http20:37
sdake_but we may at ome point want to add internal ssl20:37
sdake_some people want it for whateve reason20:37
SiRiuSsdake_, aha, I see20:38
sdake_but thanks for chasing that code down20:38
sdake_i was worried it was a real problem20:38
sdake_but it is just a warning20:38
*** ccesario_ has quit IRC20:38
sdake_still we should ix warnings20:38
sdake_look at nova api log20:38
sdake_it may have something useful20:38
SiRiuSsdake_, just Info level stuff, no error or warnings20:41
SiRiuSredeploying with debug enabled20:41
sbezverksdake_ found another potential issue with sriov, something is wrong with sriov mech driver..20:46
sdake_i am not a networkign dude20:47
sdake_can you explain what sriov does20:47
sdake_and how are you enabling the sriov mechanism driver?20:48
sdake_currently we only have ovs and linuxbridge iirc20:48
sbezverksdake sriov allows pass pci device to VM directly, the only difference between sriov and pci passthrough in case sriov you pass pci of virtual function VF and not PF physical function.20:49
sbezverksriov is enable when you add sriovnicswitch mech driver in ml2_conf.ini20:49
sbezverkand it was working before SSamYaple also used this20:50
sbezverkanyway it has lower priority than flavor20:51
sdake_i see20:55
sdake_well im sure you can sort it out20:55
sdake_it may just be an upstream bug20:55
sdake_recommend trying stable/mitaka rather then master20:55
sdake_upstream bug in nuetorn or nova20:55
sdake_the db migrtaion is mor elikely a kolla bug where the implementation of the last minute api database broke our code20:56
*** andrewvwebber has joined #openstack-kolla21:01
sbezverksdake_ I do not get where this migration is comming from when  it is a brand new installation, all it has to do is run nova-manage db sync and this command in in bootstrap step for nova..21:02
*** salv-orlando has joined #openstack-kolla21:03
sdake_sbezverk there is a different migration comand21:03
sdake_read the bug i linked in the bug you filed21:03
sbezverksdake_ I read it multiple times21:03
sbezverkthe command they mention there is not even available in the code we have21:04
sdake_fantastic21:04
sbezverkI tried to run it manually21:04
*** dwalsh has joined #openstack-kolla21:04
sbezverkto see if it resolves the issue21:04
sdake_what happened wit hthat21:05
*** salv-orlando has quit IRC21:05
sbezverksdake it just gave me an error saying that "migrate" keyword does not exist..21:08
andrewvwebberis there a way to search the irc for previous questions - like botbot.me21:08
sbezverkalso I suspect that it was applicable only to kilo21:08
sdake_i see, well clearly there must be a wayt o migrate teh db in modern ways21:09
sdake_sbezverk did you try searching for other bugs or ask on nova stating your a kolla develoepr fixing a breakage in the api database change21:10
sbezverksdake_ have you seen new nova_api database before? I have not seen it, maybe that is where the issue is coming from.21:10
sdake_there is one now21:11
*** ccesario_ has joined #openstack-kolla21:11
sdake_it needs to be bootstrapped21:11
sdake_i though that code hit the repo already21:11
sbezverksdake I do see it in the list in my test bed, it looks like something has to be migrated from nova to nova _api and we do not do it.21:13
sdake_next step  - look at devstack21:13
*** salv-orlando has joined #openstack-kolla21:14
sdake_https://github.com/openstack-dev/devstack/blob/master/lib/nova#L71621:15
sdake_bingo :)21:15
sdake_dont forget the nova-mange api_db sync21:16
sdake_all this needs to happen in nova bootstrpaping21:16
sbezverksdake trying to run it21:16
*** dwalsh has quit IRC21:17
*** salv-orl_ has joined #openstack-kolla21:17
sdake_api_db sync may already be in th code21:18
*** dwalsh has joined #openstack-kolla21:18
sbezverkyep21:18
sdake_we create a db api in nova already21:18
sbezverkit migrated 5 flavors21:18
*** salv-orlando has quit IRC21:18
sdake_but the online migration may not be21:18
*** salv-orl_ has quit IRC21:19
sbezverk+--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+21:19
sbezverk| ID                                   | Name              | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public |21:19
sbezverk+--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+21:19
sbezverk| 6bb641cd-55fb-40eb-b5e4-fe63d1c46da2 | xrv9k-32G_14xvCPU | 32768     | 55   | 0         |      | 14    | 1.0         | True      |21:19
sbezverk+--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+21:19
sbezverksdake man you are good!!!21:19
sdake_just experienced21:19
sdake_i know quite a bit about devstack21:19
sdake_i used to futilly make it try to work on fedora :)21:20
sdake_but thanks fo rthe compliment :)21:20
sdake_if i wa sgood i'd be able to figure out how to get my vms booting on my gear21:20
sbezverksdake so we then just add this extra migration step to bootstrap process right?21:21
sdake_thats correct21:21
sdake_dont forget upgrade21:21
sdake_its needed there in some fashion as well21:21
sdake_not sure how21:21
sdake_it may just work for upgrade as is21:21
*** salv-orlando has joined #openstack-kolla21:21
sbezverksdake ok I will make a patch and send for revfiew soon.21:22
sdake_sounds good21:24
*** sdake_ is now known as sdake21:24
*** salv-orlando has quit IRC21:26
*** andrewvwebber has quit IRC21:33
openstackgerritSerguei Bezverkhi proposed openstack/kolla: Adds migration step from nova to nova_api database  https://review.openstack.org/30090421:42
sdakeneeed sleep21:45
sdaketired21:45
ccesario__sdake: https://review.openstack.org/#/c/300896/21:45
patchbotccesario__: patch 300896 - kolla - Make Heka write Keepalived logs to dedicated direc...21:45
*** ccesario_ has quit IRC21:50
ccesario__well.. let me fight a bit more with https://review.openstack.org/#/c/297984/ .... it seems hostname param broke something in pull when using kolla_docker.py...  :/21:51
patchbotccesario__: patch 297984 - kolla - Added hostname param into kolla_docker21:51
ccesario__if someone guru has some idea ... I appreciate :)21:52
*** sdake has quit IRC21:55
openstackgerritCarlos Cesario proposed openstack/kolla: Make Heka write Keepalived logs to dedicated directory  https://review.openstack.org/30089621:56
*** dave-mccowan has quit IRC22:16
*** achanda has joined #openstack-kolla22:21
*** dwalsh has quit IRC22:31
*** achanda has quit IRC22:55
*** ccesario__ has quit IRC23:10
*** dave-mccowan has joined #openstack-kolla23:15
*** achanda has joined #openstack-kolla23:17
*** dave-mcc_ has joined #openstack-kolla23:17
*** sdake has joined #openstack-kolla23:19
*** dave-mccowan has quit IRC23:20
*** sdake has quit IRC23:25
*** Jeffrey4l__ has joined #openstack-kolla23:33
*** sdake has joined #openstack-kolla23:37
sdakeyo23:37
*** sdake_ has joined #openstack-kolla23:43
*** sdake has quit IRC23:46
SiRiuSsdake_, http://paste.openstack.org/show/492851/23:55
SiRiuSthere is something about filtering all hosts23:55
SiRiuSwhy would it do that?23:56
sdake_SiRiuS no clue - maybe cnnot ommunitcte with teh  compute nodes23:57
sdake_what does nova hypervisor-list show?23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!