| *** salv-orlando has joined #openstack-kolla | 00:01 | |
| *** salv-orlando has quit IRC | 00:06 | |
| *** iceyao has joined #openstack-kolla | 00:08 | |
| sdake | well my nova vm booting gets stuck in spawning state | 00:20 |
|---|---|---|
| sdake | i find that highly uninspiring | 00:20 |
| sdake | anyone else run int othis problem? | 00:21 |
| sdake | 2016-04-03 00:22:24.469 1 ERROR nova.compute.manager [instance: c758ed8c-5ae0-4bd1-9f40-7721045557d8] BuildAbortException: Build of instance c758ed8c-5ae0-4bd1-9f40-7721045557d8 aborted: Failed to allocate the network(s), not rescheduling. | 00:27 |
| sdake | well i'm gong to trust the community on this one and go a head and tag | 00:31 |
| sdake | and sume something is busted in my environment | 00:31 |
| *** achanda has quit IRC | 00:31 | |
| sdake | rhallisey ping | 00:33 |
| sdake | need this merged: https://review.openstack.org/#/c/300829/ | 00:33 |
| patchbot | sdake: patch 300829 - kolla (stable/mitaka) - Restoring openstack_release option | 00:33 |
| ccesario__ | sdake: keystone logs still remain in /var/log/kolla/apache2/ | 00:34 |
| rhallisey | sdake, k | 00:34 |
| sdake | ccesario__ is there a bug for that | 00:34 |
| ccesario__ | sdake: yes https://review.openstack.org/#/c/297967/ | 00:35 |
| patchbot | ccesario__: patch 297967 - kolla - Change keystone log dir | 00:35 |
| ccesario__ | not merged yet | 00:35 |
| sdake | rhallisey please ack https://review.openstack.org/#/c/297967/ | 00:36 |
| patchbot | sdake: patch 297967 - kolla - Change keystone log dir | 00:36 |
| sdake | that tempet problem is known and ifxed - i think a rebae of your patch wuld fix it but i'm not certain | 00:39 |
| openstackgerrit | Merged openstack/kolla: Restoring openstack_release option https://review.openstack.org/300829 | 00:40 |
| sdake | rhallisey merge https://review.openstack.org/#/c/297967/ pleae | 00:40 |
| patchbot | sdake: patch 297967 - kolla - Change keystone log dir | 00:40 |
| sdake | vikram has tested it | 00:40 |
| sdake | rhallisey PING | 00:42 |
| sdake | nihilifer ? | 00:42 |
| sdake | mandre ? | 00:42 |
| sdake | comone folks throw me a bone | 00:42 |
| rhallisey | k all set | 00:44 |
| *** weiyu has joined #openstack-kolla | 00:44 | |
| sdake | we need a backpor t on it | 00:44 |
| *** iceyao has quit IRC | 00:44 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Change keystone log dir https://review.openstack.org/300839 | 00:44 |
| sdake | rhallisey ack pz :0 | 00:44 |
| *** weiyu has quit IRC | 00:45 | |
| *** iceyao has joined #openstack-kolla | 00:45 | |
| sdake | rhallisey ack that backport plz | 00:45 |
| sdake | and then ill wait for those patche sto merge | 00:46 |
| sdake | and tag | 00:46 |
| sdake | squirly launchpad is | 00:46 |
| ccesario__ | bbiab ... | 00:47 |
| openstackgerrit | Merged openstack/kolla: Change keystone log dir https://review.openstack.org/297967 | 00:48 |
| *** iceyao has quit IRC | 00:50 | |
| *** sdake_ has joined #openstack-kolla | 00:52 | |
| *** sdake has quit IRC | 00:54 | |
| *** sdake_ is now known as sdake | 01:00 | |
| *** weiyu has joined #openstack-kolla | 01:00 | |
| rhallisey | I'm off for the night | 01:05 |
| rhallisey | see ya guys tmr | 01:05 |
| *** rhallisey has quit IRC | 01:06 | |
| openstackgerrit | Merged openstack/kolla: Change keystone log dir https://review.openstack.org/300839 | 01:13 |
| *** SiRiuS has quit IRC | 01:14 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 01:15 | |
| *** salv-orlando has joined #openstack-kolla | 01:16 | |
| *** ccesario_ has joined #openstack-kolla | 01:21 | |
| *** weiyu has quit IRC | 01:22 | |
| *** salv-orlando has quit IRC | 01:23 | |
| *** vhosakot has joined #openstack-kolla | 01:25 | |
| *** dwalsh has quit IRC | 01:25 | |
| *** iceyao has joined #openstack-kolla | 01:25 | |
| *** achanda has joined #openstack-kolla | 01:32 | |
| *** achanda has quit IRC | 01:37 | |
| ccesario_ | vhosakot hi ... Any tip about ? https://review.openstack.org/#/c/297984/ | 01:39 |
| patchbot | ccesario_: patch 297984 - kolla - Added hostname param into kolla_docker | 01:39 |
| vhosakot | hey ccesario_: I am seeing error when I test that PS.. but, the error I see is different that what you've posted in gerrit review | 01:41 |
| sdake | https://github.com/openstack/kolla/releases/tag/2.0.0.0rc3 | 01:43 |
| ccesario_ | What error? Because the patch only it is executed when hostname param is set... | 01:43 |
| ccesario_ | sdake .. cool! | 01:44 |
| vhosakot | ccesario_: the mariadb container is not starting in my env.... let me pull the latest master | 01:45 |
| ccesario_ | right | 01:45 |
| vhosakot | sdake: nice! 25 bugs fixed per week is awesome :) | 01:45 |
| *** jasonsb has joined #openstack-kolla | 01:45 | |
| *** jasonsb_ has joined #openstack-kolla | 01:48 | |
| *** sdake has quit IRC | 01:48 | |
| *** achanda has joined #openstack-kolla | 01:49 | |
| *** sdake has joined #openstack-kolla | 01:49 | |
| *** jasonsb has quit IRC | 01:49 | |
| vhosakot | sdake: saw the mails about release model... so, is kolla is tagged as release:cycle-with-milestones, does it show up in http://releases.openstack.org/independent.html ? | 01:51 |
| sdake | vhosakot we are tagged as independent atm | 01:54 |
| sdake | which is wrong | 01:54 |
| sdake | i will get it fixed in newton | 01:54 |
| sdake | changing your release model is frozen for 2 months | 01:55 |
| sdake | so we will show up at that link once the patches merge | 01:55 |
| vhosakot | ah ok.. cool..so, what should will the new tag be ? cycle-with-milestones ? | 01:55 |
| sdake | what i want to happen is for our wor for liberty and mitaka to show up in the integrated release | 01:55 |
| sdake | yup | 01:55 |
| sdake | ratherthen rleleae:indepdentn | 01:55 |
| vhosakot | ah ok | 01:55 |
| sdake | we were not told of this plan to creat ethe releaes website | 01:56 |
| sdake | and i didn't filter it from my openstack-dev ml appropriately | 01:56 |
| sdake | when we joined the big tent there was no releaes.oepnstack.org | 01:56 |
| sdake | which really blows, becaue now it looks like we have a bunch of independent releases | 01:56 |
| sdake | when one is for liberty and another is for mitaka | 01:56 |
| sdake | worst case, we will tag a z stream and add that to lierty and mitaka | 01:57 |
| sdake | and histroy will be lost | 01:57 |
| vhosakot | kolla is not in http://releases.openstack.org/mitaka/index.html due to same reason (independant tag) | 01:57 |
| vhosakot | ? | 01:57 |
| sdake | it will not be in there | 01:57 |
| sdake | i have a patch here: | 01:57 |
| sdake | https://review.openstack.org/#/c/297976/ | 01:58 |
| patchbot | sdake: patch 297976 - releases - Add kolla releases to _independent | 01:58 |
| vhosakot | ah cool | 01:59 |
| vhosakot | ccesario_: can you pastebin your new /usr/local/share/kolla/ansible/roles/mariadb/tasks/start.yml in which you have added hostname: "hostdatabase" ? | 01:59 |
| sdake | sorry guys i drpped the ball on this one | 01:59 |
| sdake | there is a bunch of fstuff that needs to happen to project config and other repos to make all this happen | 02:00 |
| sdake | and it didn't start getting done until the last month | 02:00 |
| *** jasonsb_ has quit IRC | 02:00 | |
| sdake | all completely new to me | 02:01 |
| sdake | had to learn the hard way | 02:01 |
| vhosakot | cool... thanks for those reviews... hopefully, the release team will correct it.. plus, kolla needs some time (2-3 weeks) to validate other OpenStack projects (like nova, glance, neutron, keystone, etc) after they release and kolla must build them and verify/fix breakages | 02:02 |
| ccesario__ | vhosakot: http://paste.openstack.org/show/492825/ | 02:03 |
| sdake | No, I am your father. | 02:04 |
| sdake | vhosakot that is what z streams are for | 02:04 |
| vhosakot | ah ok cool | 02:05 |
| *** vhosakot has quit IRC | 02:07 | |
| *** vhosakot has joined #openstack-kolla | 02:08 | |
| vhosakot | cool | 02:08 |
| vhosakot | ccesario: thanks for the link | 02:08 |
| ccesario_ | Your welcome! | 02:10 |
| *** sdake has quit IRC | 02:12 | |
| *** sdake has joined #openstack-kolla | 02:13 | |
| sdake | vhosakot fwiw i tried for 10 days and got 1 week instead https://review.openstack.org/#/c/298278/ | 02:14 |
| patchbot | sdake: patch 298278 - governance - Add type:deployment to release:cycle_with_miestones | 02:14 |
| vhosakot | ah ok cool | 02:16 |
| sdake | we can get our act together in 7 days | 02:20 |
| sdake | 2 weeks wuld be better | 02:20 |
| sdake | but it is what it is atm | 02:20 |
| sdake | more red tape ftl | 02:20 |
| *** ccesario_ has quit IRC | 02:25 | |
| *** ccesario_ has joined #openstack-kolla | 02:25 | |
| *** SiRiuS has joined #openstack-kolla | 02:41 | |
| *** weiyu_ has joined #openstack-kolla | 02:43 | |
| *** dave-mccowan has joined #openstack-kolla | 02:48 | |
| *** weiyu_ has quit IRC | 02:50 | |
| *** salv-orlando has joined #openstack-kolla | 02:53 | |
| *** klint has joined #openstack-kolla | 02:55 | |
| *** salv-orlando has quit IRC | 03:10 | |
| *** weiyu has joined #openstack-kolla | 03:13 | |
| *** SiRiuS has quit IRC | 03:15 | |
| *** weiyu has quit IRC | 03:26 | |
| *** achanda has quit IRC | 03:26 | |
| *** achanda has joined #openstack-kolla | 03:33 | |
| *** achanda has quit IRC | 03:48 | |
| *** vhosakot has quit IRC | 04:07 | |
| *** sdake_ has joined #openstack-kolla | 04:24 | |
| *** sdake has quit IRC | 04:27 | |
| *** salv-orlando has joined #openstack-kolla | 04:32 | |
| *** salv-orlando has quit IRC | 04:45 | |
| *** achanda has joined #openstack-kolla | 04:50 | |
| *** sdake_ has quit IRC | 05:35 | |
| *** salv-orlando has joined #openstack-kolla | 06:03 | |
| *** salv-orlando has quit IRC | 06:10 | |
| *** iceyao has quit IRC | 06:15 | |
| *** ccesario_ has quit IRC | 07:12 | |
| *** Serlex has joined #openstack-kolla | 07:15 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Implement nova-ssh container https://review.openstack.org/297963 | 07:21 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Track the api_interface in the keepalived https://review.openstack.org/300852 | 07:38 |
| *** achanda has quit IRC | 08:03 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 08:11 | |
| *** achanda has joined #openstack-kolla | 08:12 | |
| *** Jeffrey4l_ has quit IRC | 08:14 | |
| *** iceyao has joined #openstack-kolla | 08:45 | |
| *** ccesario_ has joined #openstack-kolla | 08:57 | |
| *** mbound has joined #openstack-kolla | 08:59 | |
| *** salv-orlando has joined #openstack-kolla | 09:07 | |
| *** salv-orlando has quit IRC | 09:16 | |
| *** mbound has quit IRC | 09:21 | |
| *** dwalsh has joined #openstack-kolla | 09:35 | |
| *** achanda has quit IRC | 09:42 | |
| *** allen_gao has quit IRC | 10:02 | |
| *** iceyao has quit IRC | 10:04 | |
| *** allen_gao has joined #openstack-kolla | 10:07 | |
| *** mbound has joined #openstack-kolla | 10:21 | |
| *** mbound has quit IRC | 10:26 | |
| *** salv-orlando has joined #openstack-kolla | 10:27 | |
| *** salv-orlando has quit IRC | 10:30 | |
| *** iceyao has joined #openstack-kolla | 10:51 | |
| *** Serlex has quit IRC | 11:04 | |
| *** ccesario_ has quit IRC | 11:10 | |
| *** salv-orlando has joined #openstack-kolla | 12:09 | |
| *** salv-orlando has quit IRC | 12:11 | |
| *** rhallisey has joined #openstack-kolla | 12:12 | |
| *** rhallisey has quit IRC | 12:17 | |
| *** rhallisey has joined #openstack-kolla | 12:29 | |
| *** rhallisey has quit IRC | 12:35 | |
| *** rhallisey has joined #openstack-kolla | 12:38 | |
| *** rhallisey has quit IRC | 12:43 | |
| *** rhallisey has joined #openstack-kolla | 12:44 | |
| ccesario__ | morning! | 12:45 |
| *** rhallisey has quit IRC | 12:49 | |
| *** klint has quit IRC | 12:58 | |
| *** rhallisey has joined #openstack-kolla | 13:17 | |
| *** stvnoyes has quit IRC | 13:21 | |
| *** stvnoyes has joined #openstack-kolla | 13:21 | |
| *** rhallisey has quit IRC | 13:22 | |
| *** SiRiuS has joined #openstack-kolla | 13:23 | |
| *** rhallisey has joined #openstack-kolla | 13:27 | |
| *** rhallisey has quit IRC | 13:34 | |
| *** salv-orlando has joined #openstack-kolla | 13:45 | |
| *** macsz_ has joined #openstack-kolla | 13:45 | |
| *** macsz_ has quit IRC | 13:46 | |
| *** macsz_ has joined #openstack-kolla | 13:46 | |
| *** macsz_ has quit IRC | 13:46 | |
| *** dwalsh has quit IRC | 13:50 | |
| *** macsz_ has joined #openstack-kolla | 13:52 | |
| *** macsz_ is now known as macsz | 13:52 | |
| *** salv-orlando has quit IRC | 13:52 | |
| *** sdake has joined #openstack-kolla | 14:00 | |
| *** salv-orlando has joined #openstack-kolla | 14:07 | |
| *** salv-orlando has quit IRC | 14:11 | |
| *** unicell1 has joined #openstack-kolla | 14:40 | |
| *** unicell has quit IRC | 14:40 | |
| *** ajafo has quit IRC | 14:55 | |
| *** ajafo has joined #openstack-kolla | 14:55 | |
| ccesario__ | sdake: ping? | 15:05 |
| ccesario__ | sdake: http://paste.openstack.org/show/492840/ | 15:05 |
| sdake | ccesario__ sup | 15:06 |
| sdake | you got haproxy logging working? | 15:07 |
| sdake | nice job! | 15:07 |
| sdake | got a patch in the queue to review? | 15:07 |
| *** salv-orlando has joined #openstack-kolla | 15:08 | |
| ccesario__ | sdake: not yet .. let me provide | 15:09 |
| sdake | ccesario__ the centralized logging per node is pretty well done, no? | 15:14 |
| sdake | I realize it may have some gaps | 15:14 |
| sdake | like keepalived and haproxy :) | 15:14 |
| *** mgoddard has quit IRC | 15:17 | |
| *** SiRiuS has quit IRC | 15:18 | |
| ccesario__ | sdake: look this.... http://pastebin.com/2GQVyged | 15:19 |
| ccesario__ | for haproxy it works :P | 15:19 |
| sdake | ya i saw your work | 15:19 |
| sdake | i mean i saw the output of your work | 15:19 |
| sdake | pretty sure it wasn't working previosuly | 15:19 |
| sdake | but what do yo think about how logging works? | 15:20 |
| sdake | good/bad/indifferent? | 15:20 |
| ccesario__ | sdake: very very nice.... excellent feature | 15:20 |
| sdake | elemoine_ ftw :) | 15:20 |
| sdake | although alot of other people have helped fill in the gaps | 15:21 |
| ccesario__ | it help a lot debbuging ... very very easy | 15:21 |
| sdake | his initial implementaiton was incomplete | 15:21 |
| sdake | plus with alicja's work, we centralize logging into elasticsearch and kibana | 15:21 |
| sdake | open a web page to your local network on port 5601 | 15:21 |
| ccesario__ | sdake: my currently deploy does not contain kibana :P | 15:22 |
| ccesario__ | I need check it.. I 'm curious about kibana | 15:22 |
| *** salv-orlando has quit IRC | 15:26 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Added missing haproxy log dir. https://review.openstack.org/300888 | 15:27 |
| sbezverk | good morning, anyone has latest kolla deployed openstack up and running? I saw some wierdness and wanted to check if it is a local or global issue. | 15:28 |
| sbezverk | when I create new flavor, I get this error: nova.objects.flavor Main database contains 5 unmigrated flavors, never seen that before.. | 15:29 |
| ccesario__ | sdake: https://review.openstack.org/300888 | 15:30 |
| sdake | ccesario__ what is your launchpad id | 15:32 |
| ccesario__ | sdake: ccesario | 15:34 |
| sdake | ok i assigned that bug to you | 15:34 |
| ccesario__ | k | 15:35 |
| sdake | surprised that is all ther ewa to fixing it | 15:35 |
| sdake | nothing needed in the haproxy config? | 15:35 |
| sdake | sbezverk i'll look after i finish up with this work with ccesario__ \ | 15:36 |
| ccesario__ | sdake: nothing ... therefore I fixed it :P | 15:36 |
| sdake | i htink i have it deployed | 15:36 |
| *** mbound has joined #openstack-kolla | 15:36 | |
| sdake | ccesario__ your patch doesn't have any changes to haproxy.cfg.j2 | 15:36 |
| sdake | are any needed? | 15:36 |
| sdake | or does haproxy log to /var/log/kolla/haproxy.log by default? | 15:36 |
| ccesario__ | sdake: no changes into haproxy.cfg.j2 | 15:39 |
| sdake | coool | 15:39 |
| sdake | can you look at solving same problem for keepalived? | 15:39 |
| ccesario__ | haproxy.cfg.j2 logs point to log /var/lib/kolla/heka/log local0 | 15:40 |
| sdake | or are there keepalived logs already | 15:40 |
| sdake | so your patch mounts the named volume for keepalived and haproxy | 15:41 |
| sdake | however it only creates the haproxy directory | 15:41 |
| sdake | should it also create the keepalived directory? | 15:41 |
| ccesario__ | sdake: http://paste.openstack.org/show/492841/ | 15:42 |
| sdake | ok so i think there is a followup bug here to solve | 15:43 |
| sdake | and that is gettign keepalived logs in /var/log/kolla/keepalived | 15:44 |
| sdake | but that can be solved as a different patch | 15:44 |
| *** dwalsh has joined #openstack-kolla | 15:45 | |
| sdake | ok sbezverk | 15:46 |
| sdake | moment to switch networks | 15:46 |
| ccesario__ | sdake: ok | 15:46 |
| *** sdake_ has joined #openstack-kolla | 15:47 | |
| sdake_ | sbezverk which test do you wnat me to run? | 15:47 |
| sdake_ | create a flavor? | 15:48 |
| sdake_ | my system was built and deployed laslt night from the tip of stable/mitaka | 15:48 |
| sdake_ | sbezverk do you use init-runonce? | 15:48 |
| sdake_ | ccesario mind I ask where you are employed? | 15:50 |
| *** sdake has quit IRC | 15:50 | |
| sbezverk | sdake please just create a new flavor | 15:50 |
| sdake_ | give me exact command you want me to type | 15:50 |
| sbezverk | nova flavor-create xrv9k-32G_14xvCPU auto 32768 55 14 | 15:51 |
| sdake_ | ERROR (Conflict): Not all flavors have been migrated to the API database (HTTP 409) (Request-ID: req-843c7d2b-a80e-4421-a3b9-b7140b361c22) | 15:52 |
| sbezverk | yeeessss it is not just me then | 15:53 |
| sdake_ | please file a bug | 15:53 |
| sbezverk | sdake sure will do | 15:53 |
| sdake_ | perhaps we need to do some api database migration in our bootstrap | 15:53 |
| sdake_ | sbezverk any chance you cn track this bug down | 15:53 |
| sdake_ | i'd say not being ale to create a flavor is pretty serious | 15:53 |
| sdake_ | some places to look might be devstack codebase | 15:54 |
| sdake_ | for what may be missing in our bootstrapping | 15:55 |
| sbezverk | sdake yep but it will take bit of time as I will need to find nova folks to help me.. and today is sunday | 15:55 |
| sdake_ | this also affects upgrade i suspect | 15:55 |
| sdake_ | nova people wont help you | 15:55 |
| sdake_ | they are not a support channel | 15:55 |
| *** SiRiuS has joined #openstack-kolla | 15:55 | |
| sbezverk | sdake got it, will try to deal with it | 15:55 |
| sdake_ | try/do ;) | 15:56 |
| sdake_ | when you get teh bug filed link it plz | 15:56 |
| sdake_ | and i'll triage it appropriately | 15:56 |
| sdake_ | and confirm it | 15:56 |
| sdake_ | which fails complaining about the flavor not been migrated and so that I should run nova-manage db migrate_flavor_data before. | 15:57 |
| sdake_ | https://bugs.launchpad.net/nova/+bug/1511466 | 15:58 |
| openstack | Launchpad bug 1511466 in OpenStack Compute (nova) "migrate flavor data impossible" [Medium,Confirmed] | 15:58 |
| sdake_ | i think we are missing bootstrap steps | 15:58 |
| sbezverk | sdake here you go: https://bugs.launchpad.net/kolla/+bug/1565497 | 15:59 |
| openstack | Launchpad bug 1565497 in kolla "unable to create new flavors" [Undecided,New] | 15:59 |
| ccesario__ | sdake_: I 'm from Brazil .... I work at small company www.tecnomega.com.br :) | 16:02 |
| sbezverk | ccesario__ I hope you got already some tickets for Olympics ;-) | 16:02 |
| sdake_ | ccesario__ cool - your company doing anything with kolla or just intereted in it personally | 16:03 |
| ccesario__ | sbezverk: ahahaha not yet man :( | 16:04 |
| sdake_ | ccesario__ well in either case, welcome aboard the development team ;) | 16:05 |
| ccesario__ | sdake_: the both ... the company has a openstack project to be deployed/created ... and I 'm in group that will do it..... and after a lots of tests, tools... I proposed to team usage it :) | 16:06 |
| sdake_ | ccesario__ sweet | 16:07 |
| ccesario__ | since then... I 'm personally testing and helpin with minimal changes :) | 16:07 |
| sdake_ | creating a deployment tool on your own is daunting | 16:07 |
| sdake_ | i htink yoru changes are starting to add up:) | 16:07 |
| sbezverk | sdake in normal opensatck installation nova DB gets prepped by thids command: su -s /bin/sh -c "nova-manage db sync" nova | 16:07 |
| sdake_ | can you file a bug for the keepalived in the correct directory thing | 16:07 |
| sdake_ | and link it so i cn triage it | 16:08 |
| sdake_ | sbezverk yes but there is also another migration step required | 16:08 |
| sdake_ | if you RTFM :) | 16:08 |
| sdake_ | read the bug i linked in the bug in our launchpad | 16:08 |
| sdake_ | we are missing the api migration step | 16:08 |
| sdake_ | sbezverk do you use init-runonce or create your networks your own way? | 16:09 |
| sbezverk | sdake this one: nova-manage api_db sync? | 16:09 |
| sbezverk | sdake I built my own networks | 16:10 |
| sdake_ | yes api_db sync | 16:10 |
| sdake_ | do we hve that? | 16:11 |
| sdake_ | there is also a flavor migrate | 16:11 |
| ccesario__ | sdake_: I though the kolla concepts vry cool.... and simple.... no needed a lot of changes ... like TripleO (my personally opinion) | 16:11 |
| sdake_ | which migrates the llalvors to the api database | 16:11 |
| ccesario__ | sdake_: sure... let create a file bug | 16:11 |
| sdake_ | read the bug i linked sbezverk | 16:11 |
| sdake_ | it has some pointers | 16:11 |
| sdake_ | so I get this when i try to start a nova vm: | 16:12 |
| *** Jeffrey4l__ has quit IRC | 16:12 | |
| SiRiuS | has anyone ever experienced this error when trying to create an instance? -> http://paste.openstack.org/show/492842/ | 16:12 |
| sdake_ | Build of instance c758ed8c-5ae0-4bd1-9f40-7721045557d8 aborted: Failed to allocate the network(s), not rescheduling. | 16:13 |
| SiRiuS | libvirtError: Start job for unit machine-qemu\x2d3\x2dinstance\x2d00000002.scope failed with 'failed' | 16:13 |
| sdake_ | sbezverk i assume that doesnt happen to you? | 16:13 |
| SiRiuS | very useful error message : failed with 'failed' | 16:13 |
| sdake_ | api_server 'http://glance-api-glance-api-openstack-vagrant.marathon.mesos:9292', please update [glance] api_servers with fully qualified url including scheme (http / https) | 16:14 |
| sdake_ | i have a fix for that somewhere on my harddisk | 16:15 |
| SiRiuS | sdake_, wow, awesome, please give :) | 16:15 |
| sbezverk | sdake_ nope, flavor is the only issue I see so far.. evcerything else worked fine.. | 16:15 |
| sdake_ | let me find it | 16:15 |
| sdake_ | ok well I need a working environment | 16:16 |
| sdake_ | sbezverk any chance ou can try with init-runonce before edigging into that nova db flavor bug | 16:16 |
| ccesario__ | sdake_: https://bugs.launchpad.net/kolla/+bug/1565499 | 16:16 |
| openstack | Launchpad bug 1565499 in kolla "Keepalived logs needs to be writed on correct directory" [Undecided,New] | 16:16 |
| sdake_ | and see if its a problem with the init-runonce bug? | 16:16 |
| sdake_ | ccesario__ are you taking that keepalived bug or should I leae it unassigned | 16:18 |
| *** dwalsh has quit IRC | 16:18 | |
| sdake_ | sbezverk it means doing a fresh deploy | 16:18 |
| sbezverk | sdake_ can I change ip addresses used in init-runonce? | 16:20 |
| sdake_ | sbezverk i guess if you need to | 16:20 |
| sdake_ | i'd like to try it unmoidifed tho | 16:20 |
| sdake_ | to see if the problem is with my environment or with init-runonce | 16:20 |
| sdake_ | the vms dont have to actually work they just have ot enter ACTiVe state | 16:21 |
| ccesario__ | sdake_: please leave it unassigned... I have no idea how change it.... because the keepalived does not have logoutput dir config .... It seems that heka make the magic .... I need understand better it | 16:21 |
| sdake_ | i dont care if you can network from them or not ;) | 16:21 |
| sdake_ | cool re kolla being simple, i agree we have optimized kolla in such a way as to be fast for developers and operators to pick up and understand the design | 16:22 |
| sbezverk | sdake ok let me get on it | 16:22 |
| sdake_ | rather then requiriing a rocket scientist to understand | 16:22 |
| sdake_ | sbezverk thanks a illion | 16:22 |
| sdake_ | btw i'm probably going to be kicking everyone off the ddrivers team except the core reviewers | 16:23 |
| sdake_ | because peopel are not doing bug tirage properly | 16:23 |
| sdake_ | let me train the core reviewers on how to do bug triage first | 16:23 |
| sdake_ | ;-) | 16:23 |
| sdake_ | dont take offense at this if it happens to any of you ;) | 16:24 |
| *** iceyao has quit IRC | 16:26 | |
| sbezverk | sdake_ I ran init-runonce do not see issues | 16:32 |
| elemoine_ | ccesario__: keepalived does not write its logs to a file | 16:33 |
| sbezverk | sdake_ it did not launch any VM but it correctly ctreated cirros image, sec group, and quotas.. | 16:33 |
| elemoine_ | keepalived writes its logs to the /dev/log Unix socket | 16:35 |
| elemoine_ | heka reads keepalived logs from that socket and writes them to /var/log/kolla/haproxy/keepalived.log | 16:35 |
| elemoine_ | ccesario__: see https://github.com/openstack/kolla/blob/master/ansible/roles/common/templates/heka-haproxy.toml.j2#L14 | 16:36 |
| elemoine_ | that's the path where heka writes the keepalived logs on the filesystem | 16:36 |
| sdake_ | sbezverk please launch a vm - that is what doens't work for me | 16:38 |
| sdake_ | elemoine_ thanks for the assistance ;-) | 16:38 |
| elemoine_ | I am currently commenting in the launchpad bug | 16:39 |
| sdake_ | elemoine_ thanks ;) | 16:39 |
| sdake_ | ccesario__ do you see any other heka gaps in your deployment | 16:40 |
| sbezverk | 3f4e5688-050f-4325-b4b5-074705f1cb3e | cirros-1 | ACTIVE | - | Running | demo-net=10.0.0.3 | 16:41 |
| sbezverk | sdake I need to leave for about 3 hours, when I am back I will digg into flavor bug, unless you decide to re-assign it to core reviewrs ;-) | 16:43 |
| *** salv-orlando has joined #openstack-kolla | 16:44 | |
| *** salv-orlando has quit IRC | 16:46 | |
| sdake_ | hey folks i just wanted to let you all know i will be afk for 2-3 hours | 16:47 |
| sdake_ | sbezverk you own it now ;) | 16:47 |
| sdake_ | sbezverk thanks - not sure what the deal is with the network assignment thign | 16:47 |
| sdake_ | my tile jsut finished laying yesterday and i am goign to put my office back together so i can have my multinode lab operational | 16:48 |
| SiRiuS | sdake_, did you find the fix? | 16:50 |
| ccesario__ | elemoine_: yes yes.. I have changed it and testing in my env | 16:53 |
| ccesario__ | sdake_: finishing th deploy | 16:54 |
| ccesario__ | elemoine_: thanks by tip! | 16:54 |
| *** sdake_ has quit IRC | 16:58 | |
| ccesario__ | elemoine_: http://paste.openstack.org/show/492844/ | 17:05 |
| ccesario__ | :D | 17:05 |
| ccesario__ | elemoine_: I will make the changes that you suggest..... because I have changed directly into heka-haproxy.tom.j2 | 17:09 |
| *** sdake has joined #openstack-kolla | 17:10 | |
| sdake | ccesario what was that? | 17:10 |
| ccesario__ | sdake: http://paste.openstack.org/show/492844/ | 17:11 |
| ccesario__ | sdake: for while this.... http://pastebin.com/wfkyiajw | 17:13 |
| ccesario__ | but I will split the configs as elemoine_ suggest | 17:13 |
| ccesario__ | sdake: question.... is it possible send these using too this ps https://review.openstack.org/#/c/300888/ and add two Closes-Bug entry ? | 17:19 |
| patchbot | ccesario__: patch 300888 - kolla - Added missing haproxy log dir. | 17:19 |
| *** sdake has quit IRC | 17:22 | |
| *** mbound has quit IRC | 17:45 | |
| *** salv-orlando has joined #openstack-kolla | 18:12 | |
| *** salv-orlando has quit IRC | 18:19 | |
| *** mdnadeem has quit IRC | 18:25 | |
| *** achanda has joined #openstack-kolla | 18:30 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Make keepalived logs write in correct directory. https://review.openstack.org/300896 | 18:33 |
| ccesario__ | elemoine_: ^ | 18:36 |
| *** mbound has joined #openstack-kolla | 18:46 | |
| *** mbound has quit IRC | 18:50 | |
| *** ccesario_ has joined #openstack-kolla | 18:52 | |
| *** daneyon has joined #openstack-kolla | 19:00 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Make Heka write Keepalived logs to dedicated directory https://review.openstack.org/300896 | 19:04 |
| *** achanda has quit IRC | 19:22 | |
| *** salv-orlando has joined #openstack-kolla | 19:37 | |
| *** sdake has joined #openstack-kolla | 19:47 | |
| sdake | hey folks | 19:48 |
| sdake | ccesario what is your question | 19:48 |
| sdake | i'm back from office reconstruction while taking a break | 19:48 |
| sdake | not quite done yet | 19:48 |
| *** salv-orlando has quit IRC | 19:50 | |
| *** sdake_ has joined #openstack-kolla | 19:51 | |
| *** SiRiuS has quit IRC | 19:52 | |
| *** sdake has quit IRC | 19:53 | |
| *** SiRiuS has joined #openstack-kolla | 19:57 | |
| SiRiuS | File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 417, in build_instances context, request_spec, filter_properties) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/conductor/manager.py", line 463, in _schedule_instances hosts = self.scheduler_client.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/utils.py", line 370, in wrapped return func(* | 20:19 |
| SiRiuS | args, **kwargs) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 51, in select_destinations return self.queryclient.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/__init__.py", line 37, in __run_method return getattr(self.instance, __name)(*args, **kwargs) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/client/quer | 20:19 |
| SiRiuS | y.py", line 32, in select_destinations return self.scheduler_rpcapi.select_destinations(context, spec_obj) File "/var/lib/kolla/venv/lib/python2.7/site-packages/nova/scheduler/rpcapi.py", line 126, in select_destinations return cctxt.call(ctxt, 'select_destinations', **msg_args) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/rpc/client.py", line 158, in call retry=self.retry) File "/var/lib/kolla/venv/lib/python2.7/site-packa | 20:19 |
| SiRiuS | ges/oslo_messaging/transport.py", line 90, in _send timeout=timeout, retry=retry) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 470, in send retry=retry) File "/var/lib/kolla/venv/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py", line 461, in _send raise result | 20:20 |
| SiRiuS | sdake_, any ideas on why I can't create an instance ? | 20:20 |
| sdake_ | oh dont paste in channel unless its a 1 liner plz | 20:21 |
| sdake_ | put in a paste tool so i can read it properly | 20:21 |
| sdake_ | thanks :) | 20:21 |
| *** daneyon_ has joined #openstack-kolla | 20:21 | |
| SiRiuS | sdake_, http://paste.openstack.org/show/492846/ | 20:22 |
| SiRiuS | how could I debug this? | 20:23 |
| sdake_ | need full log not the assertion | 20:24 |
| sdake_ | vms not starting is hard to debug | 20:24 |
| *** daneyon has quit IRC | 20:24 | |
| sdake_ | grep for ERROR | 20:24 |
| sdake_ | usually there is an error before the assertio nthat throws you a bone | 20:24 |
| SiRiuS | sdake_, http://paste.openstack.org/show/492842/ | 20:24 |
| SiRiuS | from nova compute | 20:24 |
| sdake_ | fwiw I can't boot a vvm because no networks ar efound | 20:24 |
| sdake_ | look in nova-api log file too | 20:25 |
| sdake_ | nova-compute isn't very helpful for debugging | 20:25 |
| sdake_ | 2016-04-03 16:08:49.537 43 WARNING nova.image.glance [req-57352596-c2fe-4a73-84f8-301299460fc9 7031b527e44d43ce9cf5feb2b79a40e4 d374cd289132475f86c4258b0f882ced - - -] No protocol specified in for api_server 'http://glance-api-glance-api-openstack-vagrant.marathon.mesos:9292', please update [glance] api_servers with fully qualified url including scheme (http / https) | 20:26 |
| sdake_ | file a bug for that please | 20:26 |
| sdake_ | and link it in irc | 20:26 |
| sdake_ | and i'll post a fix when I have an opportunity | 20:27 |
| SiRiuS | sdake_, ok | 20:27 |
| *** daneyon_ has quit IRC | 20:27 | |
| SiRiuS | sdake_, also, that is not the issue here :) | 20:27 |
| sdake_ | possibly not | 20:27 |
| sdake_ | but it could be ;) | 20:27 |
| SiRiuS | I looked at the code and it adds http:// automatically | 20:28 |
| SiRiuS | in the config file it's glance-api-glance-api-openstack-vagrant.marathon.mesos:9292 | 20:28 |
| sdake_ | i see | 20:28 |
| sdake_ | this is a mesos bug | 20:28 |
| SiRiuS | and the code will add a http in front and give a friendly warning | 20:28 |
| sdake_ | this is also an ansible bug | 20:28 |
| sdake_ | turn on debug in nova.conf | 20:29 |
| sdake_ | reconfigure | 20:29 |
| sdake_ | and paste logs | 20:29 |
| sdake_ | wit reconfgiure not implemented for kolla-mesos | 20:29 |
| sdake_ | probably need to redeploy with debug enabled | 20:29 |
| sdake_ | instead | 20:29 |
| SiRiuS | sdake_, https://github.com/openstack/nova/blob/master/nova/image/glance.py#L127 | 20:31 |
| SiRiuS | it's not really a big issue | 20:31 |
| sdake_ | its an issue because we can end up with different protocols for internal and external networks | 20:37 |
| sdake_ | currently internal is http | 20:37 |
| sdake_ | but we may at ome point want to add internal ssl | 20:37 |
| sdake_ | some people want it for whateve reason | 20:37 |
| SiRiuS | sdake_, aha, I see | 20:38 |
| sdake_ | but thanks for chasing that code down | 20:38 |
| sdake_ | i was worried it was a real problem | 20:38 |
| sdake_ | but it is just a warning | 20:38 |
| *** ccesario_ has quit IRC | 20:38 | |
| sdake_ | still we should ix warnings | 20:38 |
| sdake_ | look at nova api log | 20:38 |
| sdake_ | it may have something useful | 20:38 |
| SiRiuS | sdake_, just Info level stuff, no error or warnings | 20:41 |
| SiRiuS | redeploying with debug enabled | 20:41 |
| sbezverk | sdake_ found another potential issue with sriov, something is wrong with sriov mech driver.. | 20:46 |
| sdake_ | i am not a networkign dude | 20:47 |
| sdake_ | can you explain what sriov does | 20:47 |
| sdake_ | and how are you enabling the sriov mechanism driver? | 20:48 |
| sdake_ | currently we only have ovs and linuxbridge iirc | 20:48 |
| sbezverk | sdake sriov allows pass pci device to VM directly, the only difference between sriov and pci passthrough in case sriov you pass pci of virtual function VF and not PF physical function. | 20:49 |
| sbezverk | sriov is enable when you add sriovnicswitch mech driver in ml2_conf.ini | 20:49 |
| sbezverk | and it was working before SSamYaple also used this | 20:50 |
| sbezverk | anyway it has lower priority than flavor | 20:51 |
| sdake_ | i see | 20:55 |
| sdake_ | well im sure you can sort it out | 20:55 |
| sdake_ | it may just be an upstream bug | 20:55 |
| sdake_ | recommend trying stable/mitaka rather then master | 20:55 |
| sdake_ | upstream bug in nuetorn or nova | 20:55 |
| sdake_ | the db migrtaion is mor elikely a kolla bug where the implementation of the last minute api database broke our code | 20:56 |
| *** andrewvwebber has joined #openstack-kolla | 21:01 | |
| sbezverk | sdake_ I do not get where this migration is comming from when it is a brand new installation, all it has to do is run nova-manage db sync and this command in in bootstrap step for nova.. | 21:02 |
| *** salv-orlando has joined #openstack-kolla | 21:03 | |
| sdake_ | sbezverk there is a different migration comand | 21:03 |
| sdake_ | read the bug i linked in the bug you filed | 21:03 |
| sbezverk | sdake_ I read it multiple times | 21:03 |
| sbezverk | the command they mention there is not even available in the code we have | 21:04 |
| sdake_ | fantastic | 21:04 |
| sbezverk | I tried to run it manually | 21:04 |
| *** dwalsh has joined #openstack-kolla | 21:04 | |
| sbezverk | to see if it resolves the issue | 21:04 |
| sdake_ | what happened wit hthat | 21:05 |
| *** salv-orlando has quit IRC | 21:05 | |
| sbezverk | sdake it just gave me an error saying that "migrate" keyword does not exist.. | 21:08 |
| andrewvwebber | is there a way to search the irc for previous questions - like botbot.me | 21:08 |
| sbezverk | also I suspect that it was applicable only to kilo | 21:08 |
| sdake_ | i see, well clearly there must be a wayt o migrate teh db in modern ways | 21:09 |
| sdake_ | sbezverk did you try searching for other bugs or ask on nova stating your a kolla develoepr fixing a breakage in the api database change | 21:10 |
| sbezverk | sdake_ have you seen new nova_api database before? I have not seen it, maybe that is where the issue is coming from. | 21:10 |
| sdake_ | there is one now | 21:11 |
| *** ccesario_ has joined #openstack-kolla | 21:11 | |
| sdake_ | it needs to be bootstrapped | 21:11 |
| sdake_ | i though that code hit the repo already | 21:11 |
| sbezverk | sdake I do see it in the list in my test bed, it looks like something has to be migrated from nova to nova _api and we do not do it. | 21:13 |
| sdake_ | next step - look at devstack | 21:13 |
| *** salv-orlando has joined #openstack-kolla | 21:14 | |
| sdake_ | https://github.com/openstack-dev/devstack/blob/master/lib/nova#L716 | 21:15 |
| sdake_ | bingo :) | 21:15 |
| sdake_ | dont forget the nova-mange api_db sync | 21:16 |
| sdake_ | all this needs to happen in nova bootstrpaping | 21:16 |
| sbezverk | sdake trying to run it | 21:16 |
| *** dwalsh has quit IRC | 21:17 | |
| *** salv-orl_ has joined #openstack-kolla | 21:17 | |
| sdake_ | api_db sync may already be in th code | 21:18 |
| *** dwalsh has joined #openstack-kolla | 21:18 | |
| sbezverk | yep | 21:18 |
| sdake_ | we create a db api in nova already | 21:18 |
| sbezverk | it migrated 5 flavors | 21:18 |
| *** salv-orlando has quit IRC | 21:18 | |
| sdake_ | but the online migration may not be | 21:18 |
| *** salv-orl_ has quit IRC | 21:19 | |
| sbezverk | +--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+ | 21:19 |
| sbezverk | | ID | Name | Memory_MB | Disk | Ephemeral | Swap | VCPUs | RXTX_Factor | Is_Public | | 21:19 |
| sbezverk | +--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+ | 21:19 |
| sbezverk | | 6bb641cd-55fb-40eb-b5e4-fe63d1c46da2 | xrv9k-32G_14xvCPU | 32768 | 55 | 0 | | 14 | 1.0 | True | | 21:19 |
| sbezverk | +--------------------------------------+-------------------+-----------+------+-----------+------+-------+-------------+-----------+ | 21:19 |
| sbezverk | sdake man you are good!!! | 21:19 |
| sdake_ | just experienced | 21:19 |
| sdake_ | i know quite a bit about devstack | 21:19 |
| sdake_ | i used to futilly make it try to work on fedora :) | 21:20 |
| sdake_ | but thanks fo rthe compliment :) | 21:20 |
| sdake_ | if i wa sgood i'd be able to figure out how to get my vms booting on my gear | 21:20 |
| sbezverk | sdake so we then just add this extra migration step to bootstrap process right? | 21:21 |
| sdake_ | thats correct | 21:21 |
| sdake_ | dont forget upgrade | 21:21 |
| sdake_ | its needed there in some fashion as well | 21:21 |
| sdake_ | not sure how | 21:21 |
| sdake_ | it may just work for upgrade as is | 21:21 |
| *** salv-orlando has joined #openstack-kolla | 21:21 | |
| sbezverk | sdake ok I will make a patch and send for revfiew soon. | 21:22 |
| sdake_ | sounds good | 21:24 |
| *** sdake_ is now known as sdake | 21:24 | |
| *** salv-orlando has quit IRC | 21:26 | |
| *** andrewvwebber has quit IRC | 21:33 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Adds migration step from nova to nova_api database https://review.openstack.org/300904 | 21:42 |
| sdake | neeed sleep | 21:45 |
| sdake | tired | 21:45 |
| ccesario__ | sdake: https://review.openstack.org/#/c/300896/ | 21:45 |
| patchbot | ccesario__: patch 300896 - kolla - Make Heka write Keepalived logs to dedicated direc... | 21:45 |
| *** ccesario_ has quit IRC | 21:50 | |
| ccesario__ | well.. let me fight a bit more with https://review.openstack.org/#/c/297984/ .... it seems hostname param broke something in pull when using kolla_docker.py... :/ | 21:51 |
| patchbot | ccesario__: patch 297984 - kolla - Added hostname param into kolla_docker | 21:51 |
| ccesario__ | if someone guru has some idea ... I appreciate :) | 21:52 |
| *** sdake has quit IRC | 21:55 | |
| openstackgerrit | Carlos Cesario proposed openstack/kolla: Make Heka write Keepalived logs to dedicated directory https://review.openstack.org/300896 | 21:56 |
| *** dave-mccowan has quit IRC | 22:16 | |
| *** achanda has joined #openstack-kolla | 22:21 | |
| *** dwalsh has quit IRC | 22:31 | |
| *** achanda has quit IRC | 22:55 | |
| *** ccesario__ has quit IRC | 23:10 | |
| *** dave-mccowan has joined #openstack-kolla | 23:15 | |
| *** achanda has joined #openstack-kolla | 23:17 | |
| *** dave-mcc_ has joined #openstack-kolla | 23:17 | |
| *** sdake has joined #openstack-kolla | 23:19 | |
| *** dave-mccowan has quit IRC | 23:20 | |
| *** sdake has quit IRC | 23:25 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 23:33 | |
| *** sdake has joined #openstack-kolla | 23:37 | |
| sdake | yo | 23:37 |
| *** sdake_ has joined #openstack-kolla | 23:43 | |
| *** sdake has quit IRC | 23:46 | |
| SiRiuS | sdake_, http://paste.openstack.org/show/492851/ | 23:55 |
| SiRiuS | there is something about filtering all hosts | 23:55 |
| SiRiuS | why would it do that? | 23:56 |
| sdake_ | SiRiuS no clue - maybe cnnot ommunitcte with teh compute nodes | 23:57 |
| sdake_ | what does nova hypervisor-list show? | 23:57 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!