| *** sdake has joined #openstack-kolla | 00:00 | |
| *** sdake_ has quit IRC | 00:00 | |
| *** sdake has quit IRC | 00:13 | |
| *** salv-orl_ has joined #openstack-kolla | 00:17 | |
| *** salv-orlando has quit IRC | 00:20 | |
| *** oanson has quit IRC | 00:21 | |
| *** oanson has joined #openstack-kolla | 00:30 | |
| *** Margin has joined #openstack-kolla | 00:45 | |
| *** qiliang27 has joined #openstack-kolla | 01:02 | |
| *** tovin07 has joined #openstack-kolla | 01:08 | |
| *** manheim has quit IRC | 01:10 | |
| *** qiliang28 has joined #openstack-kolla | 01:10 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/437728 | 01:11 |
|---|---|---|
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/438306 | 01:11 |
| *** qiliang27 has quit IRC | 01:11 | |
| *** qiliang28 is now known as qiliang27 | 01:11 | |
| *** Jeffrey4l has joined #openstack-kolla | 01:14 | |
| *** sp__ has joined #openstack-kolla | 01:22 | |
| sbezverk_ | Jeffrey4l: ping | 01:23 |
| sp__ | Jeffrey4l: ping.... | 01:26 |
| *** cuongnv has joined #openstack-kolla | 01:27 | |
| sbezverk_ | sp__: I was first ;) | 01:28 |
| sp__ | sbezverk_: :) sure | 01:28 |
| *** caowei has joined #openstack-kolla | 01:28 | |
| sp__ | sbezverk_: you are at the stage | 01:28 |
| sp__ | sbezverk_: :) | 01:29 |
| Jeffrey4l | sbezverk_, pong sp__ pong ;) | 01:29 |
| Jeffrey4l | sup sbezverk_ sp__ | 01:30 |
| sbezverk_ | Jeffrey4l: about the horizon bug I hit with 3.0.3-beta.1 images | 01:30 |
| Jeffrey4l | sbezverk_, 3.0.3? sorry what's the detail.. | 01:30 |
| sbezverk_ | Jeffrey4l: could you check proposed fix, I have already tested in our gates and it fixed the ossue | 01:30 |
| sbezverk_ | https://review.openstack.org/#/c/438268/3 | 01:31 |
| sbezverk_ | Jeffrey4l: ^^ | 01:31 |
| Jeffrey4l | sbezverk_, yep. there is a bug for this. but i still curiosity why? | 01:32 |
| Jeffrey4l | who depends on /usr/share/openstack-dashboard? and why master is OK( right? )? | 01:33 |
| Jeffrey4l | ok. i got it. | 01:34 |
| Jeffrey4l | 1 sec | 01:34 |
| Jeffrey4l | sbezverk_, i think this is more than a bug in kolla-ansible. | 01:35 |
| sbezverk_ | Jeffrey4l: :) | 01:35 |
| Jeffrey4l | but i am not sure how and why it happen. | 01:35 |
| Jeffrey4l | based on the bug description: WSGIScriptAlias / /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi | 01:35 |
| Jeffrey4l | WSGIScriptAlias use that folder in source install type. | 01:35 |
| sbezverk_ | Jeffrey4l: right, it hits only with source install | 01:36 |
| sbezverk_ | but with both ubuntu and centos | 01:36 |
| Jeffrey4l | but from your provided logs, you are testing ubuntu+binary? http://logs.openstack.org/04/437504/20/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-3-ironic-nv/b8092e7/logs/openstack/kolla-horizon-1768984591-7xz23.txt | 01:36 |
| Jeffrey4l | and base on the horizon.conf.j2 file, | 01:37 |
| Jeffrey4l | https://github.com/openstack/kolla-ansible/blob/242c559a044ede8c20134ba40f6643719498c6a2/ansible/roles/horizon/templates/horizon.conf.j2#L12 | 01:37 |
| Jeffrey4l | python_path should point to /var/lib/kolla/venv/lib/python2.7/site-packages which is expected. | 01:38 |
| Jeffrey4l | sbezverk_, ^^ | 01:38 |
| sbezverk_ | Jeffrey4l: I might have got the file name from the wrong place | 01:39 |
| sbezverk_ | but it was definetly only with source | 01:39 |
| Jeffrey4l | hrm. | 01:39 |
| Jeffrey4l | could u find the logs and check what the horizon.conf looks like? | 01:39 |
| sbezverk_ | basically I added this fix to horizon image with 3.0.3-beta.2 tag and after that out gate job can verify horizon properly | 01:40 |
| Jeffrey4l | if it is really issue, i think we need fix in kolla-ansible site. | 01:40 |
| Jeffrey4l | side* | 01:40 |
| Jeffrey4l | yep. this should fix the issue. | 01:40 |
| sbezverk_ | Jeffrey: sure, let me fig up old log | 01:40 |
| Jeffrey4l | but it is meaningless to create /usr/share/openstack-dashboard for source install. ( this is actually only used by binary ) | 01:41 |
| sbezverk_ | Jeffrey4l: ok I got it, binary it was a title for the job | 01:42 |
| Jeffrey4l | so it is testing ubuntu+source actually? | 01:42 |
| sbezverk_ | but in reality it uses source images I just did not rename the gate job to refelct it | 01:42 |
| Jeffrey4l | got. | 01:42 |
| Jeffrey4l | does the horizon.conf file is collected? | 01:42 |
| sbezverk_ | Jeffrey4l: both centos source and ubuntu source showed the same issue | 01:42 |
| Jeffrey4l | WSGIScriptAlias / /usr/share/openstack-dashboard/openstack_dashboard/wsgi/django.wsgi | 01:43 |
| Jeffrey4l | got it. it really uses /usr/s/o path. | 01:43 |
| Jeffrey4l | weird. | 01:43 |
| Jeffrey4l | sbezverk_, which kolla-ansible version are u using in gate? | 01:44 |
| sbezverk_ | Jeffrey4l: here you go http://paste.openstack.org/show/600522/ | 01:44 |
| sbezverk_ | Jeffrey4l: should be master | 01:45 |
| Jeffrey4l | it make no sense. check https://github.com/openstack/kolla-ansible/blob/242c559a044ede8c20134ba40f6643719498c6a2/ansible/roles/horizon/templates/horizon.conf.j2#L12 | 01:46 |
| Jeffrey4l | python_path should be /var/lib/kolla/venv/lib/python2.7/site-packages when using source. | 01:46 |
| Jeffrey4l | sbezverk_, are u sure kolla_install_type is source in your installation? | 01:46 |
| sbezverk_ | Jeffrey4l: hmm, I do use install_type source, but maybe it defaults to binary when we run genconfig | 01:48 |
| Jeffrey4l | sbezverk_, this is the horizon.conf in kolla-ansible with ubuntu+source installation. | 01:49 |
| Jeffrey4l | http://logs.openstack.org/37/438237/2/check/gate-kolla-ansible-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/e69a09f/logs/kolla_configs/horizon/horizon.conf | 01:49 |
| Jeffrey4l | it looks as expected. | 01:49 |
| Jeffrey4l | WSGIScriptAlias point to the virtualenv rather than /usr/s/o path. | 01:49 |
| sbezverk_ | Jeffrey4l: yeah :( it looks like it defaults to binary | 01:50 |
| Jeffrey4l | could u fix this in kolla-k8s? mix binary and source may cause other issue ;) | 01:50 |
| Jeffrey4l | there are other place, where depends in kolla_install_type variable. | 01:50 |
| sbezverk_ | Jeffrey4l: sure in this case it is not kolla or kolla-ansible problem | 01:51 |
| Jeffrey4l | thanks ;) | 01:51 |
| sbezverk_ | Jeffrey4l: thanks for pointing it out | 01:51 |
| Jeffrey4l | so how can we handle the bug https://bugs.launchpad.net/kolla/+bug/1662972 .. | 01:51 |
| openstack | Launchpad bug 1662972 in kolla "horizon source build missing symlink" [High,In progress] - Assigned to Serguei Bezverkhi (sbezverk) | 01:51 |
| Jeffrey4l | in fact i still can not found what is happening there. | 01:51 |
| sbezverk_ | Jeffrey4l: probably that guy hit the same issue | 01:51 |
| sbezverk_ | generated config for binary but used with source images | 01:52 |
| Jeffrey4l | could u leave some message there? | 01:52 |
| sbezverk_ | sure thing, let me address it first so I can be 100% sure | 01:52 |
| *** dave-mccowan has joined #openstack-kolla | 01:52 | |
| Jeffrey4l | yes. maybe he forget to change the variable in kolla-ansible ;) | 01:53 |
| sbezverk_ | Jeffrey4l: yep there is good chance ;) | 01:56 |
| *** jrobinson is now known as jrobinson-afk | 01:56 | |
| *** japestin_ has joined #openstack-kolla | 02:01 | |
| sp__ | Jeffrey4l: please follow up the https://review.openstack.org/#/c/422444/16 need to complete it soon i.e before pk1 | 02:02 |
| Jeffrey4l | yep. i will review it later ;) | 02:02 |
| sp__ | Jeffrey4l: always thankful of your continuous review | 02:03 |
| Jeffrey4l | np. kolla lack lots of UT, thanks for improving it ;) | 02:03 |
| *** manjeets has joined #openstack-kolla | 02:08 | |
| *** jrobinson-afk is now known as jrobinson | 02:19 | |
| openstackgerrit | Surya Prakash Singh proposed openstack/kolla-ansible master: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 02:21 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Configuring ironic gates (basic tests) https://review.openstack.org/437504 | 02:22 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Configuring ironic gates (basic tests) https://review.openstack.org/437504 | 02:23 |
| *** duonghq has joined #openstack-kolla | 02:24 | |
| duonghq | morning guys | 02:24 |
| cliles | Jeffrey4l: I couldn't find a "good" way to only deploy horizon as source and everything else as binary | 02:25 |
| Jeffrey4l | cliles, what's your current deployment type? | 02:25 |
| cliles | binary | 02:26 |
| Jeffrey4l | ubuntu or centos? | 02:27 |
| cliles | centos | 02:27 |
| cliles | but https://bugs.launchpad.net/kolla/+bug/1654133 and https://bugs.launchpad.net/kolla/+bug/1647837 make centos binary horizon practically worthless for me | 02:27 |
| openstack | Launchpad bug 1654133 in kolla "horizon dashboard broken in master with centos binaries" [Undecided,New] | 02:27 |
| Jeffrey4l | whihch branch? master | 02:27 |
| openstack | Launchpad bug 1647837 in kolla "trove dashboard does not deploy with centos packages" [High,Triaged] | 02:27 |
| cliles | master | 02:27 |
| cliles | well | 02:27 |
| cliles | master as of 2.6.2017 | 02:27 |
| Jeffrey4l | wait, which issue are we talking about? | 02:28 |
| cliles | haha | 02:28 |
| cliles | I am trying to explain why I an doing the source + binary deployment | 02:28 |
| Jeffrey4l | that one, or this https://bugs.launchpad.net/kolla/+bug/1662972 | 02:28 |
| openstack | Launchpad bug 1662972 in kolla "horizon source build missing symlink" [High,In progress] - Assigned to Serguei Bezverkhi (sbezverk) | 02:28 |
| cliles | 1662972 | 02:28 |
| Jeffrey4l | Seems we need to drop a symlink (or do some other magic) on a source build of horizon on CentOS. | 02:29 |
| cliles | yes | 02:29 |
| Jeffrey4l | but in description, you said you are : centos+source | 02:29 |
| Jeffrey4l | could we make clear of this? | 02:29 |
| cliles | horizon is bulit from source | 02:29 |
| cliles | ..... | 02:30 |
| Jeffrey4l | but others are build from binary? | 02:30 |
| cliles | I guess it's an invalid bug | 02:30 |
| cliles | yes | 02:30 |
| Jeffrey4l | please do not do this ... | 02:30 |
| Jeffrey4l | we do not support mix source + binary. | 02:30 |
| Jeffrey4l | if binary won't work, please fix it in binary. rather than mix source + binary. ;) | 02:30 |
| cliles | my binary bugs are older than this one :) | 02:31 |
| Jeffrey4l | let's solve the bug one by one ;) | 02:32 |
| cliles | if it's not a supported deployment, then my apologies for taking away critical dev cycles | 02:32 |
| Jeffrey4l | yes. we won't support this. | 02:32 |
| Jeffrey4l | so could u leave some message in the bug? | 02:33 |
| sbezverk_ | Jeffrey4l: I sent a gate job with installation type set to source.. will see in a bit if it works.. it should.. | 02:33 |
| Jeffrey4l | great. | 02:33 |
| cliles | I've updated 1662972 | 02:35 |
| Margin | for several components like kafka , manila-data , there isn't binary package in ubuntu/centos repository. | 02:35 |
| Jeffrey4l | thanks. | 02:35 |
| sp__ | duonghq: hi... | 02:36 |
| duonghq | hi sp__ | 02:36 |
| Margin | If kolla don't support mixed build , how to solve the scene ? | 02:37 |
| duonghq | we need more custom repo | 02:37 |
| duonghq | and some more bugs id | 02:38 |
| sp__ | duonghq: thanks for working on this, I am also following up with thish ttps://review.openstack.org/#/c/425446/ | 02:38 |
| Margin | yes, how to supply the binary of software ? | 02:38 |
| sp__ | duonghq: just wanted to know what is our plan for its completion | 02:39 |
| sp__ | duonghq: https://blueprints.launchpad.net/kolla/+spec/ks-rolling-upgrade-role | 02:39 |
| duonghq | sp__, we need test | 02:39 |
| duonghq | more test | 02:39 |
| duonghq | to confirm it works properly | 02:39 |
| Jeffrey4l | Margin, cliles how did u try to mix source and binary? | 02:42 |
| sbezverk_ | Jeffrey4l: yep it worked, I will abandon the fix.. | 02:42 |
| *** rwallner has joined #openstack-kolla | 02:42 | |
| Jeffrey4l | sbezverk_, thanks. | 02:42 |
| cliles | I hacked the deploy.yml task for the horizon build to always deploy a source build | 02:42 |
| Jeffrey4l | cliles, how? change which varaible? | 02:43 |
| Margin | I'm just thinking about it . | 02:43 |
| Jeffrey4l | i think it should works if you change the variable "kolla_install_type" from binary to source. | 02:44 |
| Jeffrey4l | in horizon roles. | 02:44 |
| *** rwallner has quit IRC | 02:44 | |
| cliles | jeffrey4l: ok, I was wrong, it was horizon_image in the role's default.yml | 02:44 |
| Jeffrey4l | cliles, tell me the details. what did u try? | 02:45 |
| Jeffrey4l | diff is better. | 02:45 |
| cliles | horizon_image: "{{ docker_registry ~ '/' if docker_registry else '' }}{{ docker_namespace }}/{{ kolla_base_distro }}-source-horizon" | 02:45 |
| cliles | it was just a 1 liner | 02:45 |
| Jeffrey4l | ok i got one idea. could u try it? | 02:46 |
| cliles | ok | 02:46 |
| * Jeffrey4l is preparing a diff. | 02:46 | |
| Jeffrey4l | cliles, try http://paste.openstack.org/show/600525/ | 02:48 |
| Jeffrey4l | two line ;) | 02:48 |
| cliles | If I remember why I ended up there, something about the fluentd container trying to deploy from source if I set kolla_install_type as a group var | 02:51 |
| sp__ | duonghq: just let me how can help to test this | 02:55 |
| duonghq | sp__, at least, you can test manually: deploy an cluster, try to upgrade why it still running, | 02:57 |
| duonghq | better: write some test case for make sure the upgrade progress doesn't break anything | 02:57 |
| duonghq | but it's quite hard | 02:57 |
| sp__ | duonghq: I will | 02:58 |
| duonghq | sp__, thanks | 02:59 |
| sp__ | duonghq: where can i share test cases with you , just to take a look on that of you | 02:59 |
| duonghq | sp__, you can put in an etherpad and write down the link to ps comment | 03:00 |
| *** sdake has joined #openstack-kolla | 03:01 | |
| duonghq | sup sdake | 03:01 |
| sdake | sup duonghq | 03:01 |
| sdake | just got back from dinner | 03:01 |
| sdake | Jeffrey4l ping | 03:01 |
| Jeffrey4l | sdake, pong | 03:01 |
| sp__ | duonghq: ok, I will | 03:01 |
| duonghq | sp__, nice | 03:02 |
| sp__ | duonghq: np | 03:03 |
| *** sudipto has joined #openstack-kolla | 03:05 | |
| *** sudipto_ has joined #openstack-kolla | 03:05 | |
| *** sudipto has quit IRC | 03:12 | |
| *** sudipto_ has quit IRC | 03:12 | |
| cliles | jeffrey4l: same thing, the fluentd container tries to deploy from source | 03:14 |
| sdake | Jeffrey4l sory for lag, there is a review that sbezverk_ put up for horizon being busted | 03:21 |
| Jeffrey4l | oh right, depends.. | 03:21 |
| sdake | i'm not sure if its correct | 03:21 |
| sdake | but it does fi the problem | 03:21 |
| Jeffrey4l | sdake, sbezverk_ has confirmed it is a issue in kolla-k8s's gate. | 03:22 |
| sdake | so not a container problem then? | 03:22 |
| sdake | i knew i shouldn't have +2'ed that patch | 03:22 |
| sdake | I hope someone -2ed it | 03:22 |
| sdake | so it doesn't merge :) | 03:22 |
| Jeffrey4l | i -1ed for that. | 03:22 |
| Jeffrey4l | in k8s, he is trying mix use source and binary deployments | 03:23 |
| sdake | i see | 03:23 |
| sdake | thats a huge problem ;) | 03:23 |
| sdake | sometimes that works other times not | 03:23 |
| Jeffrey4l | yes. | 03:23 |
| sdake | duonghq are you ready set go with kollla-kbuernetes or sitll blocked? | 03:27 |
| sdake | and if blocked, what on | 03:27 |
| sdake | we decided at ptg we need cross repo gating with kolla and kolla-kubernetes | 03:28 |
| sdake | it is a real shame more cores wer enot at the ptg | 03:29 |
| sdake | the ptg > the summit for devs | 03:29 |
| *** sp__ has quit IRC | 03:29 | |
| sdake | brb | 03:29 |
| *** sdake_ has joined #openstack-kolla | 03:30 | |
| duonghq | sdake_, sure, I am ready to go | 03:30 |
| sdake_ | duonghq cool i see you commented on the 1.0.0 requirements | 03:31 |
| sdake_ | duonghq how terrible wa the remote audio feed? | 03:31 |
| sdake_ | plz be honest ;) | 03:31 |
| duonghq | sdake_, 1st morning, really terrible, late in 1st afternoon, much better | 03:32 |
| sdake_ | ya i fixed the gain on the mic | 03:32 |
| sdake_ | how about after that? | 03:32 |
| duonghq | 2nd day, it's really cool but I think we should have more than 1 mic and some kind of software mixer, | 03:32 |
| sdake_ | software mixwer does what? | 03:32 |
| duonghq | combine input from multiple mic | 03:33 |
| sdake_ | keep in mind its a volunteer thing - i brought a mic at the last minute that i scavanaged from my daughter's christmas stash | 03:33 |
| sdake_ | was the adio volume too low? | 03:33 |
| *** sdake has quit IRC | 03:34 | |
| duonghq | sure, I do not mean one person needs bring everything, | 03:34 |
| duonghq | just that if somebody can donate, it's better | 03:34 |
| duonghq | your mic is really awesome but it cannot capture far people voice | 03:35 |
| sdake_ | btw I highly recommend disposing of your passwords for any critical services bekuz of cloudbleed | 03:36 |
| sdake_ | also I recommend going directly to the website of your relatinoships to change your passwords | 03:36 |
| sdake_ | i am getting tons of phishing requests as a result of clodubleed | 03:37 |
| sdake_ | launchpad for example | 03:37 |
| sdake_ | plz plz change your password | 03:37 |
| duonghq | maybe somebody use same password for many site | 03:38 |
| duonghq | iirc launchpad do not affect directly from cloudbleed event? | 03:39 |
| sdake_ | duonghq not sure | 03:42 |
| sdake_ | however if you reuse passwords its sitll good to change em :) | 03:42 |
| duonghq | it's lucky that I barely reuse passwords | 03:43 |
| duonghq | phew | 03:43 |
| sdake_ | interestingly cloudflare offers a tshirt for a security bug bounty | 03:43 |
| duonghq | really fun | 03:44 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Add newui into kolla-ansible https://review.openstack.org/438323 | 03:44 |
| sdake_ | i have dumped lastpass as well | 03:44 |
| duonghq | what do you mean "the ptg > the summit for devs"? | 03:44 |
| sdake_ | and moved to passwords.openstack.org | 03:44 |
| sdake_ | passwords.google.com that is | 03:44 |
| duonghq | you do not trust lastpass anymore? | 03:45 |
| sdake_ | ptg is more effective for dev comms then openstack summit | 03:45 |
| sdake_ | although summit is important | 03:45 |
| sdake_ | i never trusted lastpass | 03:45 |
| *** sayantan_ has joined #openstack-kolla | 03:45 | |
| sdake_ | although i couldn't handle resetting 20 passwords a day | 03:45 |
| sdake_ | necessary evil | 03:45 |
| sdake_ | breaking into google is a serious challenge | 03:46 |
| sdake_ | not that the govt couldn't do it | 03:46 |
| sdake_ | or the mossad | 03:46 |
| sdake_ | but script kiddies there were mining cloudflare for passswords can't really get into google | 03:46 |
| sdake_ | they might be able to crack lastpass | 03:46 |
| sdake_ | and lastpass from what i hear has bad security praticies - although I dont know first hand | 03:46 |
| duonghq | I lastpass do what they say, I think it's quite safe mechanism | 03:47 |
| duonghq | *if | 03:47 |
| sdake_ | the IF is in question | 03:47 |
| duonghq | sure | 03:47 |
| sdake_ | i knwo a ton about security i find it hard to believe they encrypt passwords on disk | 03:48 |
| sdake_ | or ssd or whever they store them | 03:48 |
| sdake_ | (as in they are really palintext) | 03:48 |
| sdake_ | i dont know if passwords.google.com is any better | 03:49 |
| sdake_ | but i'm moving on to that :) | 03:49 |
| duonghq | ack | 03:49 |
| sdake_ | i've almost got lastpass ejected from my systems :) | 03:50 |
| sdake_ | although passwords.google.com doesn't work for some countries | 03:50 |
| duonghq | hmm, does it work for iDevices? | 03:50 |
| duonghq | sdake_, can you review it https://review.openstack.org/#/c/437819/ ? | 03:59 |
| sdake_ | duonghq not passing gate, i tickled it with a recheck | 04:00 |
| sdake_ | not sure on the idivices q | 04:00 |
| duonghq | about lastpass, what do you mean "they are really plaintext"? | 04:02 |
| duonghq | it's hard to find really "anonymous" password manager service | 04:02 |
| SamYaple | duonghq: lastpass is not plaintext. it is fully encrypted | 04:03 |
| SamYaple | but trusting any external password manager is putting your faith in other people | 04:03 |
| duonghq | got it | 04:04 |
| SamYaple | all of them have had leaks and XSS vulnerabilities or other means of getting passwords out | 04:04 |
| SamYaple | personally, i have lastpass. realistically, i should be using keepass | 04:04 |
| duonghq | so, it's better to keep some password in local password manager, and some in our physical safe | 04:04 |
| SamYaple | but i happen to like the app injection | 04:04 |
| Margin | maybe we can keep account and password seperately | 04:06 |
| SamYaple | i dont know the context. i just jumped in like i do | 04:06 |
| duonghq | it's better that solution like u2f key is widely adopt | 04:08 |
| duonghq | but I smell some vendor lock-ins here | 04:08 |
| SamYaple | i use lastpass with a yubikey | 04:11 |
| duonghq | (y) | 04:16 |
| *** lamt has joined #openstack-kolla | 04:18 | |
| *** saneax-_-|AFK is now known as saneax | 04:22 | |
| duonghq | sdake_, should we pump default tag of kolla-build? | 04:26 |
| duonghq | due to we are in Pike | 04:26 |
| *** mdnadeem has joined #openstack-kolla | 04:29 | |
| *** dave-mccowan has quit IRC | 04:36 | |
| *** sayantan_ has quit IRC | 04:38 | |
| *** sdake_ has quit IRC | 04:42 | |
| *** lamt has quit IRC | 04:45 | |
| *** trinaths has joined #openstack-kolla | 04:56 | |
| *** sudipto_ has joined #openstack-kolla | 04:57 | |
| *** sudipto has joined #openstack-kolla | 04:57 | |
| *** jrobinson is now known as jrobinson-afk | 05:51 | |
| *** unicell has joined #openstack-kolla | 05:51 | |
| *** skramaja has joined #openstack-kolla | 05:53 | |
| openstackgerrit | iswarya vakati proposed openstack/kolla master: Removed the lines which states to gothrough the quickstart guide as the link was removed from kolla https://review.openstack.org/437255 | 05:56 |
| *** lamt has joined #openstack-kolla | 06:11 | |
| *** salv-orlando has joined #openstack-kolla | 06:17 | |
| *** salv-orlando has quit IRC | 06:20 | |
| *** salv-orlando has joined #openstack-kolla | 06:20 | |
| *** salv-orl_ has quit IRC | 06:21 | |
| *** lamt has quit IRC | 06:21 | |
| *** Administrator_ has joined #openstack-kolla | 06:22 | |
| *** jrobinson-afk is now known as jrobinson | 06:30 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: DEBUGING Ubuntu owner issue https://review.openstack.org/438341 | 06:35 |
| Margin | https://bugs.launchpad.net/kolla/+bug/1668059 , who can help me ? | 06:44 |
| openstack | Launchpad bug 1668059 in kolla "Failed to Starting kolla-toolbox container" [Critical,Triaged] | 06:44 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: DEBUGING Ubuntu owner issue https://review.openstack.org/438341 | 06:54 |
| *** devnull has quit IRC | 06:55 | |
| *** Margin is now known as MarginHu | 06:59 | |
| *** sp__ has joined #openstack-kolla | 07:05 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove Heka docs and yaml file https://review.openstack.org/438346 | 07:06 |
| sp__ | duonghq: ping..... | 07:06 |
| Jeffrey4l | MarginHu, try to install docker from docker's repo rather than centos repo. | 07:07 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove Heka docs and yaml file https://review.openstack.org/438346 | 07:08 |
| MarginHu | Jeffrey41, maybe it is caused by docker version. | 07:08 |
| MarginHu | which version is your recommand ? | 07:08 |
| Jeffrey4l | MarginHu, install docker base on this https://docs.docker.com/engine/installation/linux/centos/ | 07:09 |
| Jeffrey4l | and install 1.12.x | 07:09 |
| MarginHu | OK, I will try. | 07:10 |
| *** bmace has quit IRC | 07:15 | |
| *** bmace has joined #openstack-kolla | 07:16 | |
| *** pcaruana has joined #openstack-kolla | 07:20 | |
| openstackgerrit | caoyuan proposed openstack/kolla master: Remove log file about heka https://review.openstack.org/438351 | 07:23 |
| *** japestin_ has quit IRC | 07:23 | |
| openstackgerrit | caoyuan proposed openstack/kolla master: Remove log file about heka https://review.openstack.org/438351 | 07:25 |
| *** daidv has joined #openstack-kolla | 07:26 | |
| daidv | Hi! | 07:26 |
| daidv | Anyone here? I'm deploy ceilometer via kolla (Newton release). But I'm facing with exception: Api failed to start. Failed to connect to databases, purpose: metering, event | 07:30 |
| daidv | ceilometer-api.log: http://paste.ubuntu.com/24076920/ | 07:30 |
| daidv | Can anyone help me? | 07:31 |
| *** magicboiz has joined #openstack-kolla | 07:32 | |
| *** sdake_ has joined #openstack-kolla | 07:40 | |
| sdake_ | duonghq for master yes | 07:42 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update the chrony to yes https://review.openstack.org/438355 | 07:44 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Add the missing "ironic_inspector" into precheks https://review.openstack.org/438233 | 07:57 |
| *** matrohon has joined #openstack-kolla | 07:59 | |
| *** aolwas74 has joined #openstack-kolla | 08:01 | |
| *** jrobinson has quit IRC | 08:06 | |
| *** sayantan_ has joined #openstack-kolla | 08:11 | |
| sp__ | duonghq: hi... | 08:11 |
| duonghq | sp__, pong | 08:11 |
| sp__ | duonghq: whr should i add etherpad link of Test case written for "ks-rolling-upgrade" | 08:12 |
| duonghq | sp__, just leave a comment with etherpad link to the ps | 08:12 |
| sp__ | duonghq: ok | 08:13 |
| sp__ | duonghq: I will be adding more and more, just keep your eye on that | 08:13 |
| duonghq | sp__, roger | 08:13 |
| *** shardy has joined #openstack-kolla | 08:17 | |
| *** salv-orlando has quit IRC | 08:22 | |
| hrw | elo after pto | 08:23 |
| *** sdake has joined #openstack-kolla | 08:25 | |
| hrw | debian-binary: 140 debian-source: 142 ubuntu-binary: 159 ubuntu-source: 205 | 08:25 |
| hrw | x86-64 with git HEAD + my patches | 08:26 |
| hrw | will have to check what failed | 08:26 |
| *** manheim has joined #openstack-kolla | 08:26 | |
| *** sdake_ has quit IRC | 08:27 | |
| duonghq | sdake, for dev discussion, you prefer PTG or summit? | 08:31 |
| sdake | ptg is superior although really 4 times a year in person travel is the most beneficial | 08:33 |
| sdake | so its not either or, its both | 08:33 |
| duonghq | ya | 08:34 |
| duonghq | hope that I can join with Kolla team next time | 08:35 |
| duonghq | do you have any picture of our team? | 08:35 |
| sdake | duonghq we didn't taek a pciture that I nkow of | 08:35 |
| *** ccesario has quit IRC | 08:35 | |
| sdake | although there was a dinner event pic that was taken | 08:35 |
| duonghq | oops, I remember that somebody called for team picture | 08:36 |
| openstackgerrit | Merged openstack/kolla stable/newton: fix magnum [keystone_authtoken] configuration https://review.openstack.org/435778 | 08:37 |
| *** egonzalez has joined #openstack-kolla | 08:39 | |
| *** nathharp has joined #openstack-kolla | 08:41 | |
| hrw | INFO:kolla.image.build.horizon:adduser: The user `horizon' already exists. Exiting. | 08:43 |
| hrw | ;( | 08:43 |
| hrw | binary builds sucks | 08:44 |
| *** ccesario has joined #openstack-kolla | 08:46 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Remove serial in playbooks https://review.openstack.org/438374 | 08:47 |
| duonghq | Jeffrey4l, we do not use serial at site.yml level anymore? | 08:49 |
| Jeffrey4l | duonghq, i hope. serial will cause big issue. | 08:50 |
| Jeffrey4l | any idea? | 08:50 |
| *** gfidente has joined #openstack-kolla | 08:51 | |
| duonghq | I think move it to task or role level is better | 08:51 |
| duonghq | agree with out | 08:51 |
| duonghq | you | 08:51 |
| *** shardy has quit IRC | 08:53 | |
| Jeffrey4l | duonghq, ansible do not support that. duonghq | 08:53 |
| *** shardy has joined #openstack-kolla | 08:53 | |
| Jeffrey4l | serial only work in playbook level. not role and include level. | 08:54 |
| duonghq | ah, sorry, I mis recall that | 08:54 |
| *** matrohon has quit IRC | 08:56 | |
| daidv | Jeffrey4l, did you deploy ceilometer via kolla ever? | 08:57 |
| Jeffrey4l | daidv, yep. sup? | 08:57 |
| egonzalez | daidv, there is an issue with dependencies in rdo packages | 08:58 |
| daidv | Jeffrey4l, I'm trying to do that but get a bug like "Api failed to start, Failed to connect to database" | 08:59 |
| *** sdake has quit IRC | 08:59 | |
| Jeffrey4l | daidv, could u provide the logs? | 08:59 |
| Jeffrey4l | egonzalez, we need install openstack-celometer-api in ceilometer-base, i think. | 08:59 |
| daidv | Jeffrey4l, ceilometer-api.log : http://paste.ubuntu.com/24076920/ | 08:59 |
| Jeffrey4l | hrm. it is ubuntu + binary? | 09:00 |
| *** matrohon has joined #openstack-kolla | 09:01 | |
| daidv | Jeffrey4l, centos 7 and binary. | 09:01 |
| Jeffrey4l | ok. master? or ocata? | 09:01 |
| daidv | Jeffrey4l, I deploy Newton release. | 09:02 |
| Jeffrey4l | daidv, could u provide more logs? i guess there should be a info like: failed to connect to db blablabla | 09:02 |
| Jeffrey4l | egonzalez, could u review this https://review.openstack.org/438228 ? | 09:03 |
| daidv | sure, wait a minute | 09:03 |
| egonzalez | Jeffrey4l, added a comment, can we add --filter status=restarting in the command also | 09:05 |
| Jeffrey4l | yes. that's a issue too. | 09:06 |
| egonzalez | we do in the same PS or in a new one? | 09:06 |
| Jeffrey4l | the same is OK. because when container is restarting, gate lie too. | 09:07 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Change the panko location https://review.openstack.org/438380 | 09:07 |
| daidv | Jeffrey4l, I don't get any more ERROR from ceilometer api or gnocchi api. What log file should I show u? | 09:10 |
| Jeffrey4l | daidv, all of them | 09:10 |
| Jeffrey4l | egonzalez, " docker ps -a --filter status=exited --filter status=restarting --filter status=created --filter status=dead" | 09:11 |
| Jeffrey4l | is that OK? | 09:11 |
| Jeffrey4l | here is the all status, created, restarting, running, removing, paused, exited and dead | 09:12 |
| Jeffrey4l | i thin removing and paused is not matter. | 09:12 |
| Jeffrey4l | i think* | 09:12 |
| egonzalez | removing maybe when we have upgrade gates | 09:12 |
| Jeffrey4l | ok. all of them except running ;) | 09:13 |
| egonzalez | otherwise all OK | 09:13 |
| egonzalez | btw, never seen created status, you know when that happen? | 09:14 |
| Jeffrey4l | egonzalez, yep. i saw this. | 09:14 |
| *** nhlfr has quit IRC | 09:14 | |
| Jeffrey4l | guess docker created the container, but failed before running process in it. | 09:14 |
| egonzalez | hrm, thanks, all status codes that are not running are issues, is OK add 'em all | 09:16 |
| Jeffrey4l | agree. i am adding them. | 09:16 |
| *** rmart04 has joined #openstack-kolla | 09:17 | |
| *** zioproto has joined #openstack-kolla | 09:17 | |
| daidv | Jeffrey4l, http://paste.ubuntu.com/24077226/ | 09:17 |
| daidv | I just found a problem with Mongodb? | 09:17 |
| Jeffrey4l | ceilometer try to connect localhost mongodb? | 09:18 |
| *** matrohon has quit IRC | 09:18 | |
| Jeffrey4l | what's you globals.yml , which datbase/storage you want ceilometer to write to? | 09:18 |
| daidv | Jeffrey4l, btw, I dont' see any ERROR in mongdo container logs | 09:19 |
| Jeffrey4l | daidv, Unable to reconnect to the primary mongodb: localhost:27017: it trying to conect localhsot, this is wrong. | 09:19 |
| daidv | Mongodb log: http://paste.ubuntu.com/24077239/ | 09:20 |
| Jeffrey4l | mongodb isn't listen on localhost | 09:20 |
| Jeffrey4l | check the ceilometer.conf file, | 09:21 |
| Jeffrey4l | you want the ceilometer write data into mongodb? or gnocchi? | 09:21 |
| daidv | This is celometer config file: http://paste.ubuntu.com/24077245/ | 09:23 |
| daidv | Jeffrey4l, I want to write data in o gnnochi. | 09:23 |
| daidv | Can you give me an idea if I want to do that? | 09:24 |
| Jeffrey4l | ok. seems there is bug/issue there. ceilometer try to connect mongodb even though you do not configure it. | 09:24 |
| Jeffrey4l | i need check the ceilometer's source code. | 09:24 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Exit with error code 1 when there is failure container in gate https://review.openstack.org/438228 | 09:25 |
| *** nhlfr has joined #openstack-kolla | 09:28 | |
| *** manheim has quit IRC | 09:33 | |
| hrw | Jeffrey4l: hi. I saw in logs that you worked on debian jobs. | 09:33 |
| *** Manheim has joined #openstack-kolla | 09:35 | |
| *** matrohon has joined #openstack-kolla | 09:35 | |
| *** athomas has joined #openstack-kolla | 09:36 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Update the panko location of site.yml https://review.openstack.org/438380 | 09:41 |
| Jeffrey4l | hrw, aha yep. | 09:45 |
| hrw | Jeffrey4l: with current state of kolla they may fail all over so do not be surprised | 09:46 |
| Jeffrey4l | hrw, at least, one job is OK now https://review.openstack.org/437761 , even though i add lots of image into exclude image list. | 09:47 |
| Jeffrey4l | hrw, you can use such method to test your change. just change ubuntu to debian at the beginning to the jobs. | 09:48 |
| hrw | Jeffrey4l: https://review.openstack.org/#/c/434431/ handles gnupg stuff for you | 09:48 |
| *** Administrator_ has quit IRC | 09:50 | |
| *** Administrator_ has joined #openstack-kolla | 09:50 | |
| *** sp____ has joined #openstack-kolla | 09:52 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Remove Heka yaml file https://review.openstack.org/438346 | 09:53 |
| Jeffrey4l | hrw i will split that patch into two. one just adding debian jobs related to code: tox.ini and test_build.py , then we can merge it. | 09:55 |
| hrw | Jeffrey4l: ok | 09:55 |
| Jeffrey4l | then you can simple change ubuntu to debian, it will run debian rather than ubuntu before we have really debian jobs in openstack infra. | 09:56 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Fix Creating Panko MongoDB database failed https://review.openstack.org/438408 | 10:02 |
| *** MarginHu has quit IRC | 10:03 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add debian related test env https://review.openstack.org/437761 | 10:04 |
| daidv | Jeffrey4l, any idea? can we bump it with some ways? | 10:04 |
| * hrw out | 10:04 | |
| *** pbourke has joined #openstack-kolla | 10:05 | |
| Jeffrey4l | daidv, found nothing. but i guess there is something wrong in ceilometer.conf. still diging. | 10:05 |
| *** mgoddard has joined #openstack-kolla | 10:10 | |
| *** trinaths1 has joined #openstack-kolla | 10:12 | |
| *** trinaths has quit IRC | 10:14 | |
| *** cuongnv has quit IRC | 10:16 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Fix keystone idempotency https://review.openstack.org/437900 | 10:24 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate deploy/genconfig https://review.openstack.org/438420 | 10:32 |
| *** magicboiz has quit IRC | 10:35 | |
| *** strigazi_ is now known as strigazi | 10:35 | |
| *** manheim_ has joined #openstack-kolla | 10:42 | |
| *** manheim_ has quit IRC | 10:44 | |
| egonzalez | Jeffrey4l, when this PS is merged by releases team, we should update nova tag in newton. Then we have to create another release candidate for newton? | 10:44 |
| egonzalez | https://review.openstack.org/#/c/434928/ | 10:44 |
| egonzalez | In that tag is the backport for simple_cell_setup | 10:45 |
| *** sayantan_ has quit IRC | 10:46 | |
| egonzalez | I'm trying to understand how all releases stuff works | 10:46 |
| *** Serlex has joined #openstack-kolla | 10:53 | |
| *** wznoinsk has quit IRC | 10:55 | |
| *** wznoinsk has joined #openstack-kolla | 10:55 | |
| *** strigazi has quit IRC | 10:56 | |
| *** strigazi has joined #openstack-kolla | 10:56 | |
| *** wznoinsk has left #openstack-kolla | 10:56 | |
| *** sambetts|afk is now known as sambetts | 11:05 | |
| *** manheim_ has joined #openstack-kolla | 11:10 | |
| *** nathharp has quit IRC | 11:13 | |
| Jeffrey4l | egonzalez, yep it is. we are waiting for that tag. after it is merged. we will bump the nova tag in our newton branch. | 11:23 |
| *** caowei has quit IRC | 11:23 | |
| egonzalez | Jeffrey4l, thanks | 11:23 |
| *** mgoddard has quit IRC | 11:25 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add debian related test env https://review.openstack.org/437761 | 11:25 |
| *** dciabrin has quit IRC | 11:27 | |
| *** magicboiz has joined #openstack-kolla | 11:27 | |
| *** dciabrin has joined #openstack-kolla | 11:27 | |
| Jeffrey4l | np | 11:28 |
| *** magicboiz has quit IRC | 11:32 | |
| *** caowei has joined #openstack-kolla | 11:32 | |
| *** daidv has quit IRC | 11:35 | |
| *** caowei has quit IRC | 11:38 | |
| *** magicboiz has joined #openstack-kolla | 11:44 | |
| *** Margin has joined #openstack-kolla | 11:57 | |
| Margin | the process of building image is inturrupted by elasticsearch repository. | 11:58 |
| *** sudipto_ has quit IRC | 11:58 | |
| *** sudipto has quit IRC | 11:58 | |
| des_consolado | ah I was highlighted but now it's gone out of my buffer D: | 11:59 |
| *** sp____ has quit IRC | 12:00 | |
| Margin | why fluentd is necessary ? I see there is "fluentd, kolla-toolbox,cron" in kolla-ansible-4.0.0.0rc1/ansible/roles/common/tasks/start.yml | 12:05 |
| pbourke | Margin: its for logging | 12:11 |
| Margin | yes, why it is in roles/common/tasks/start.yml ? I think it is option. | 12:11 |
| Margin | not necessary. | 12:12 |
| pbourke | Margin: i agree, I think the common role currently does too much | 12:12 |
| *** mgoddard has joined #openstack-kolla | 12:14 | |
| *** rwallner has joined #openstack-kolla | 12:21 | |
| *** rwallner has quit IRC | 12:21 | |
| *** dave-mccowan has joined #openstack-kolla | 12:21 | |
| *** rwallner has joined #openstack-kolla | 12:22 | |
| *** saneax is now known as saneax-_-|AFK | 12:27 | |
| *** aolwas74 is now known as aolwas | 12:28 | |
| *** rhallisey has joined #openstack-kolla | 12:30 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Exit with error code 1 when there is failure container in gate https://review.openstack.org/438228 | 12:33 |
| *** sbezverk_ has quit IRC | 12:40 | |
| *** gema has quit IRC | 12:40 | |
| *** gema has joined #openstack-kolla | 12:41 | |
| *** gema has quit IRC | 12:41 | |
| *** gema has joined #openstack-kolla | 12:41 | |
| *** sbezverk has joined #openstack-kolla | 12:44 | |
| *** trinaths1 has quit IRC | 12:50 | |
| *** sudipto has joined #openstack-kolla | 12:50 | |
| *** sudipto_ has joined #openstack-kolla | 12:50 | |
| *** trinaths has joined #openstack-kolla | 12:50 | |
| *** bachp has quit IRC | 12:51 | |
| *** bachp has joined #openstack-kolla | 12:51 | |
| *** mgiles has joined #openstack-kolla | 12:56 | |
| *** mgiles has quit IRC | 12:58 | |
| *** rwallner has quit IRC | 12:59 | |
| *** rmart04 has quit IRC | 13:04 | |
| *** rmart04 has joined #openstack-kolla | 13:04 | |
| *** satyar has joined #openstack-kolla | 13:08 | |
| berendt | is anybody using the netapp driver in cinder volume at the moment? | 13:09 |
| satyar | Hi Jeffrey4l: egonzalez | 13:10 |
| egonzalez | berendt, not now but used in my previous company | 13:11 |
| *** lamt has joined #openstack-kolla | 13:11 | |
| berendt | egonzalez: in kolla? | 13:12 |
| berendt | i just hit https://bugs.launchpad.net/cinder/+bug/1668261 on a customer site | 13:12 |
| openstack | Launchpad bug 1668261 in Cinder "NetApp driver requires rpm/dpkg-query binaries" [Undecided,New] | 13:12 |
| *** lamt has quit IRC | 13:12 | |
| berendt | the utils package in netapp driver requires dpkg-query/rpm to check for an installed python-cinder package | 13:12 |
| berendt | the driver fails when python-cinder is not installed | 13:13 |
| egonzalez | berendt, nope, custom deployment with Kilo | 13:13 |
| satyar | have you guys seen any issue with nova quota? | 13:18 |
| *** rwallner has joined #openstack-kolla | 13:30 | |
| *** manheim_ has quit IRC | 13:35 | |
| *** manheim_ has joined #openstack-kolla | 13:50 | |
| *** goldyfruit has joined #openstack-kolla | 13:59 | |
| *** erhudy has joined #openstack-kolla | 14:05 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate deploy/genconfig https://review.openstack.org/438420 | 14:07 |
| *** Serlex has quit IRC | 14:08 | |
| *** manheim_ has quit IRC | 14:09 | |
| *** n0isyn0i1e is now known as n0isyn0ise | 14:09 | |
| *** crushil has joined #openstack-kolla | 14:13 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Set default value for osds/osd_ids facts when facts not set https://review.openstack.org/438503 | 14:13 |
| berendt | This issue bites me while running a reconfiguration of a customer deployment, please review, I need to backport this to newton | 14:14 |
| *** salv-orlando has joined #openstack-kolla | 14:15 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla-ansible master: Set default value for osds/osd_ids facts when facts not set https://review.openstack.org/438503 | 14:16 |
| *** manheim_ has joined #openstack-kolla | 14:16 | |
| *** sdake has joined #openstack-kolla | 14:21 | |
| *** jtriley has joined #openstack-kolla | 14:21 | |
| sdake | morning | 14:24 |
| *** sdake has quit IRC | 14:31 | |
| *** sdake has joined #openstack-kolla | 14:36 | |
| *** mgiles has joined #openstack-kolla | 14:37 | |
| *** mgoddard has quit IRC | 14:38 | |
| *** lamt has joined #openstack-kolla | 14:39 | |
| *** mgoddard has joined #openstack-kolla | 14:45 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla stable/newton: Updated from global requirements https://review.openstack.org/438523 | 14:48 |
| *** Serlex has joined #openstack-kolla | 14:49 | |
| sbezverk | sdake: good morning | 14:51 |
| sbezverk | sdake: need your review on this again, I had to do a couple minor adjustments: https://review.openstack.org/#/c/437504/26 | 14:52 |
| gema | is development open for business in pike yet? | 14:52 |
| *** unicell has quit IRC | 14:54 | |
| *** sps_ has joined #openstack-kolla | 14:55 | |
| magicboiz | Hi there, I'm testing newton -> ocata upgrade, and besides the nova/cellsv2 issue, I'm having problems with mongodb: "Unable to create/open lock file: /var/lib/mongodb/mongod.lock errno:13 Permission denied Is a mongod instance already running?, terminating" | 14:56 |
| magicboiz | any hints/ideas? thanks | 14:56 |
| *** unicell has joined #openstack-kolla | 14:56 | |
| pbourke | gema: yes | 14:57 |
| gema | pbourke: thanks | 14:57 |
| *** trinaths has quit IRC | 14:58 | |
| gema | inc0: trying to figure out how to land jeffrey's patch to add debian jobs together with hrw's patches so that it all makes sense | 14:58 |
| gema | inc0: https://review.openstack.org/#/c/434946/ | 14:58 |
| sdake | gema yes | 14:59 |
| gema | sdake: what's your advice, then? we land debian kolla changes first and then the infra jobs? | 15:00 |
| sdake | duno just woke up | 15:00 |
| gema | sdake: good morning :D | 15:00 |
| gema | sdake: I thought you were replying to my second question, not the first one :D | 15:00 |
| *** trinaths has joined #openstack-kolla | 15:03 | |
| *** lamt has quit IRC | 15:03 | |
| *** trinaths has quit IRC | 15:05 | |
| *** Margin has quit IRC | 15:06 | |
| *** Margin has joined #openstack-kolla | 15:06 | |
| *** lucasxu has joined #openstack-kolla | 15:11 | |
| *** skramaja has quit IRC | 15:13 | |
| *** lamt has joined #openstack-kolla | 15:16 | |
| *** imcsk8_ has quit IRC | 15:17 | |
| *** imcsk8 has joined #openstack-kolla | 15:17 | |
| *** kong has quit IRC | 15:19 | |
| *** athomas has quit IRC | 15:20 | |
| *** sdake_ has joined #openstack-kolla | 15:20 | |
| *** Margin has quit IRC | 15:22 | |
| *** sdake has quit IRC | 15:23 | |
| *** satyar has quit IRC | 15:25 | |
| *** athomas has joined #openstack-kolla | 15:25 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding ironic inspector microservices https://review.openstack.org/438556 | 15:28 |
| *** TxGirlGeek has joined #openstack-kolla | 15:31 | |
| *** TxGirlGeek has quit IRC | 15:32 | |
| *** satyar has joined #openstack-kolla | 15:32 | |
| *** TxGirlGeek has joined #openstack-kolla | 15:33 | |
| sbezverk | sdake: to rename get job, I guess I need to push infra ps right? | 15:43 |
| sbezverk | I meant gate job | 15:43 |
| *** manheim_ has quit IRC | 15:46 | |
| *** mdnadeem has quit IRC | 15:46 | |
| sdake_ | sbezverk you need to submit it for review yes | 15:48 |
| *** lamt has quit IRC | 15:49 | |
| sbezverk | sdake_: ok thanks, I want to change ironic jobs from binary to source | 15:51 |
| *** manheim_ has joined #openstack-kolla | 15:52 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix designate deploy/genconfig https://review.openstack.org/438420 | 15:57 |
| *** lucasxu has quit IRC | 15:58 | |
| *** sdake_ is now known as sdake | 16:01 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding ironic inspector microservices https://review.openstack.org/438556 | 16:04 |
| *** sayantan_ has joined #openstack-kolla | 16:05 | |
| *** unicell has quit IRC | 16:06 | |
| *** sayantan_ has quit IRC | 16:07 | |
| *** kbaegis has joined #openstack-kolla | 16:12 | |
| *** lamt has joined #openstack-kolla | 16:14 | |
| *** lucasxu has joined #openstack-kolla | 16:17 | |
| *** wirehead_ has quit IRC | 16:19 | |
| *** manheim_ has quit IRC | 16:25 | |
| sdake | shardy you got the tag wrong for openstack-ansible | 16:26 |
| sdake | I am pretty sure its [openstack-ansible] | 16:26 |
| sdake | shardy rest of email looks good tho :) | 16:26 |
| kbaegis | Just asked a question on ask.openstack: https://ask.openstack.org/en/question/103378/is-this-a-kolla-or-docker-py-issue/ | 16:26 |
| kbaegis | docker-py 2.1.0 seems broken on my system, and I can't figure out why. I'm running docker 1.13.1, but rolling back didn't fix | 16:27 |
| shardy | sdake: Ok thanks, I wasn't sure if it was osad, openstack-ansible or just ansible ;) | 16:27 |
| kbaegis | It's not populating the docker.Client | 16:27 |
| shardy | sdake: please add it to any reply, thanks! | 16:27 |
| sdake | shardy I can't figure out how to chang ethe subject in gmail | 16:28 |
| sdake | which is the only place I can reply inline | 16:28 |
| sdake | I will follow up your original thread from my cisco address I guess | 16:28 |
| shardy | sdake: mutt ftw ;) | 16:28 |
| shardy | sdake: I'll reply and add the tag, thanks for pointing it out | 16:28 |
| sdake | shardy unfortunately outlook is mandated at cisco, so I'm stuck with it | 16:28 |
| sdake | shardy sounds good | 16:28 |
| sdake | shardy also point them at the rest of the thread :) | 16:28 |
| kbaegis | Can anyone explain what's happening there? | 16:30 |
| *** financeMann has joined #openstack-kolla | 16:30 | |
| financeMann | Any idea why my deploy is failing -> http://pastebin.com/ASTE7zX9 | 16:31 |
| *** manheim_ has joined #openstack-kolla | 16:32 | |
| kbaegis | financeMann: lol Same problem as mine exactly | 16:32 |
| kbaegis | financeMann: I think docker-py is broken | 16:32 |
| financeMann | for being simple kolla openstack is by far the most complicated tool i’ve ever used | 16:32 |
| financeMann | it never works | 16:33 |
| kbaegis | financeMann: It's still in development. I'm on the stable/ocata branch though | 16:33 |
| financeMann | i’m thinking of just trying openstack dev edition, maybe that’s easier to do | 16:33 |
| kbaegis | financeMann: And I'm trying to deploy on gentoo instead of a crappy ubuntu vm | 16:33 |
| financeMann | to test it out | 16:33 |
| financeMann | my background is in high frequency trading systems so i am not particularly used to software that can break | 16:33 |
| kbaegis | financeMann: So what's happening there, both with kolla-ansible prechecks and kolla-ansible deploy is that docker-py isn't able to figure out how to talk to docker | 16:33 |
| kbaegis | I don't know more than that | 16:34 |
| kbaegis | https://ask.openstack.org/en/question/103378/is-this-a-kolla-or-docker-py-issue/ | 16:34 |
| sdake | sbezverk did you ask a q earlier, I saw it fly by but was writing an email | 16:34 |
| kbaegis | financeMann: ooc, what are your docker-py and docker versions? | 16:35 |
| kbaegis | Also ansible? | 16:35 |
| financeMann | my deploy literally worked the other day | 16:35 |
| financeMann | i changed nothing | 16:35 |
| financeMann | and now it dont work | 16:35 |
| financeMann | i just restarted the computer | 16:35 |
| kbaegis | financeMann: It's possible that docker hadn't been restarted | 16:36 |
| *** sps_ has quit IRC | 16:36 | |
| kbaegis | idk | 16:36 |
| financeMann | you mean i should start-up docker? | 16:36 |
| kbaegis | sdake: Any word on this issue? | 16:36 |
| sdake | kbaegis no idea what issue your talkingabout | 16:37 |
| kbaegis | financeMann: No, I'm saying that it could've been why restarting broke it | 16:37 |
| kbaegis | sdake: Well, in less than 5 minutes 2 people confirmed it | 16:37 |
| kbaegis | docker-py seems broken | 16:37 |
| sdake | not surprising | 16:37 |
| kbaegis | sdake: client = docker.Client(version=module.params.get('api_version'))\nAttributeError: 'module' object has no attribute 'Client'\n", | 16:37 |
| sdake | which version of ocker-py arey ou using? | 16:37 |
| sdake | pip show docker-py | 16:37 |
| financeMann | Docker version 1.13.1, build 092cba3 | 16:37 |
| financeMann | but I literally did not do any updates/pulls etc | 16:37 |
| sbezverk | sdake: you answered my question, thanks.. I just need you to check infra PS I psuhed.. | 16:37 |
| financeMann | how could this just magically break | 16:37 |
| sbezverk | sdake: | 16:38 |
| sbezverk | https://review.openstack.org/438581 | 16:38 |
| kbaegis | sdake: This is the broken code: "client = docker.Client(version=module.params.get('api_version'))" | 16:38 |
| *** dciabrin has quit IRC | 16:39 | |
| sdake | sbezverk see coments | 16:39 |
| financeMann | kbaegis is this broken because of some remote server change or something external to my hsot | 16:39 |
| financeMann | host | 16:39 |
| kbaegis | one copy of it can be found in ansible/library/kolla-container_facts | 16:39 |
| *** dciabrin has joined #openstack-kolla | 16:39 | |
| financeMann | new error kbaegis http://pastebin.com/cTAQnCb7 | 16:39 |
| kbaegis | financeMann: I don't fully understand the problem. It could be inside the container/docker images. | 16:40 |
| sdake | financeMann nothing magically breaks - usually someone does a yum update or apt update or whatever | 16:40 |
| kbaegis | financeMann: Try docker logs against any hung containers | 16:40 |
| financeMann | is there a way to force docker to just redeploy everything | 16:40 |
| sdake | docker and docker-py versioning is a pile o garb imo :) | 16:40 |
| kbaegis | sdake: top line of my ask.openstack.org question: docker-py 2.1.0 looks broken with docker 1.13.1 which breaks Kolla stable/ocata | 16:41 |
| sbezverk | sdake: got it thanks.. I guess at one point we will need to build a setup of jobs for consistent testing, but without going overboard ;) | 16:41 |
| sdake | kbaegis i see, we need a top pin then pronto | 16:41 |
| kbaegis | sdake: Well, the critical issue then is that kolla-ansible is critically dependent on itt | 16:41 |
| sdake | kbaegis let me get my morning rush out of the way and i'll validate that | 16:41 |
| sdake | could someone plz file a bug | 16:41 |
| kbaegis | sdake: I'll do it I guess. I need to create an account | 16:41 |
| sdake | kbaegis you ahv a luanchpad account already right? | 16:42 |
| sdake | kbaegis if not, then yup will have to make an account - I need logs and whatnot and a bug to move around to the requirements team if necessary to top pin docker-py | 16:42 |
| kbaegis | sdake: Looks liek I do | 16:42 |
| sdake | kbaegis cool - just 1 password reset away :) | 16:42 |
| kbaegis | sdake: Going to do a quick search for duplicates | 16:42 |
| * sdake knows about that since i reset about 50 passwords yesterday | 16:43 | |
| sdake | groan | 16:43 |
| egonzalez | financeMann, your issue sounds like docker service is not running, can you simply execute docker ps or docker images? | 16:43 |
| financeMann | sounds good im rebooting first thenw ill | 16:44 |
| egonzalez | check if the service is enabled at boot, maybe at restarting just keep disabled | 16:44 |
| egonzalez | kbaegis, answer not found in ask.o.o | 16:45 |
| sdake | egonzalez can you duplicate the problem reported - my env is all setup for kolla-kubernetes atm and would have to tear it all down | 16:45 |
| sdake | egonzalez if its a real problem we have to top pin in the stable ocata requirements which will take some time | 16:46 |
| sdake | egonzalez current theory is a new version of docker-py is incompatible in some way with the version of docker-engine being used | 16:46 |
| sdake | if docker-py has to be top pinned it takes like a week to get that job done | 16:47 |
| sdake | which puts us very close to our march 10th deadline | 16:47 |
| *** kbaegis has quit IRC | 16:48 | |
| sdake | egonzalez your theory is also good, although two people reported theproblem aroudn the ame time (is my understanding) | 16:48 |
| egonzalez | sdake, going to deploy, where is failing exacly, cannot find the question in ask(maybe still under moderation) | 16:48 |
| sdake | egonzalez although I didn't personally witness that, so YMMV :) | 16:48 |
| sdake | egonzalez probably under moderation, SamYaple I think can get it moderated | 16:49 |
| sdake | egonzalez i did ask kbaegis to file a bug - he just logged off, but maybe he will file it shortly | 16:49 |
| sdake | moderated/unmoderated or whatever the correct term is ;) | 16:49 |
| *** skramaja has joined #openstack-kolla | 16:49 | |
| *** gfhellma has joined #openstack-kolla | 16:50 | |
| inc0 | good morning | 16:55 |
| mgoddard | morning inc0 | 16:56 |
| inc0 | mgoddard: so, actually we can get kfox1111 to the discussion too;) | 16:56 |
| inc0 | correct me if I'm wrong, but you scientific guys runs batch jobs, am I right? | 16:56 |
| mgoddard | #openstack-kolla-4-science? | 16:56 |
| inc0 | we can do it here too, maybe someone else would like to join | 16:57 |
| mgoddard | frequently yes, it depends on the application | 16:57 |
| inc0 | what I'm thinking, is kubernetes is also pretty damn good for this use case | 16:57 |
| inc0 | but not everything will be k8s-native | 16:57 |
| mgoddard | slurm is a popular open source batch scheduler | 16:57 |
| egonzalez | sdake, btw, there is not any docker-py 2.1.0 in pip nor in rpm | 16:58 |
| inc0 | so let say slurm support k8s for certain jobs | 16:58 |
| sdake | sup mgoddard | 16:58 |
| inc0 | I'd think that running kolla-k8s with same network overlay as k8s underneath | 16:58 |
| mgoddard | hi sdake | 16:58 |
| sdake | nice to meet you at the ptg and have a drink with ya :) | 16:58 |
| inc0 | will allow you to have both and use whichever is best for the case | 16:59 |
| mgoddard | indeed, nice to meet you too | 16:59 |
| inc0 | wile have same way to access your runners | 16:59 |
| inc0 | (over calico) | 16:59 |
| inc0 | am I totally in the woods or this makes some sense? | 16:59 |
| sdake | mgoddard to create irc channels is not an easy or trivial task and we want to reduce channel creation as much as possible to reduce siloing | 16:59 |
| inc0 | sdake: it's not that hard;) /join #new-channel | 17:00 |
| mgoddard | sdake I wasn't being entirely serious :) | 17:00 |
| *** adrian_otto has joined #openstack-kolla | 17:01 | |
| inc0 | mgoddard: toughts? Also keep in mind that I'm talking abot not too far future as kolla-k8s is not prod ready yet | 17:01 |
| mgoddard | inc0, not quite sure I see how it fits together currently. are you suggesting to use the same instance of k8s used by kolla-k8s to run some k8s native batch jobs, then openstack-managed slurm for everything else? | 17:03 |
| sdake | what is slurm :) | 17:03 |
| *** rmart04 has quit IRC | 17:03 | |
| sdake | oh open source batch scheduler | 17:03 |
| sdake | inc0 we may have an emergency related to docker-py | 17:04 |
| *** manheim_ has quit IRC | 17:04 | |
| sdake | inc0 and the need to top pin it asap | 17:04 |
| sdake | inc0 my env is not setup atm for testing kolla-ansible | 17:04 |
| egonzalez | financeMann, were you able to issue regular docker commands? | 17:04 |
| inc0 | mgoddard: pretty much | 17:04 |
| inc0 | if we could make slurm understand nodes in k8s (maybe have one huge k8s node?) | 17:05 |
| inc0 | that could work | 17:05 |
| inc0 | but I don't know first thing about slurm | 17:05 |
| inc0 | sdake: whats up? | 17:05 |
| sdake | inc0 you will have ot read the scrollback unforutnately - there is not yet a record of the problem in a bug - the only record exists in the scrollback and an unmoderated askopenstack.org Q | 17:06 |
| *** lamt has quit IRC | 17:07 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding ironic inspector microservices https://review.openstack.org/438556 | 17:08 |
| inc0 | sdake: do you have link to this question? | 17:11 |
| *** raddaoui has joined #openstack-kolla | 17:11 | |
| sbezverk | inc0: if you have time, please review https://review.openstack.org/#/c/437504/26 | 17:11 |
| sdake | inc0 it hasn't passed through moderation, so I don't - SamYaple can get it through moderation but he isn't around atm | 17:13 |
| *** matrohon has quit IRC | 17:14 | |
| *** nhlfr has quit IRC | 17:15 | |
| mgoddard | inc0 I think there's definitely a need to support mixed environments with traditional batch schedulers and more modern container environments such as k8s | 17:15 |
| *** sudipto_ has quit IRC | 17:15 | |
| *** sudipto has quit IRC | 17:15 | |
| mgoddard | our current plan uses kolla-ansible to deploy openstack, then we will deploy various workloads onto bare metal using ironic | 17:15 |
| inc0 | mgoddard: kolla-k8s would ease the transfer or keep cross-compatibility between both for as long as needed | 17:16 |
| mgoddard | workloads including slurm, hadoop (sahara), k8s (magnum) | 17:16 |
| inc0 | I'd say for today this is probably best | 17:16 |
| inc0 | for future having k8s across all the bare metal and kolla on top of it for stuff that need a vm can be interesting | 17:17 |
| *** david-lyle has joined #openstack-kolla | 17:17 | |
| *** jrich523 has joined #openstack-kolla | 17:17 | |
| mgoddard | our current project is quite sexy. we're building a performance prototype for this: http://www.ska-sdp.org/ | 17:17 |
| inc0 | especially given that you can setup k8s and openstack to use same network overlay | 17:17 |
| inc0 | mgoddard: yeah, makes my nerd soul soar;) | 17:18 |
| *** rwallner has quit IRC | 17:18 | |
| inc0 | if Kolla will run stuff for biggest radiotelescope in the world, I'll be much happier person | 17:19 |
| mgoddard | inc0: we want to avoid putting containers everywhere. some of our use cases need bare metal for performance | 17:19 |
| *** rwallner has joined #openstack-kolla | 17:19 | |
| inc0 | yeah makes sense | 17:19 |
| mgoddard | as will I :) | 17:19 |
| inc0 | while containers doesn't seem to have impact on performance whatsoever | 17:19 |
| inc0 | vm's do | 17:19 |
| inc0 | (that was one of reasons why k8s might be compelling - it shouldn't affect performance in any way) | 17:20 |
| mgoddard | there is a small overhead with containers | 17:20 |
| inc0 | really? | 17:20 |
| inc0 | on which part? | 17:20 |
| mgoddard | networking performance can take a hit if you're not using net=host | 17:20 |
| inc0 | networking maybe | 17:20 |
| inc0 | yeah | 17:20 |
| mgoddard | not sure on the effect of filesystem overlay but I suspect it's not zero | 17:20 |
| inc0 | but as you said net=host works, and k8s understands it, although then calico will not add much value | 17:21 |
| inc0 | mgoddard: it should be zero, there is no magic involved | 17:21 |
| *** david-lyle has quit IRC | 17:21 | |
| *** vhosakot has joined #openstack-kolla | 17:21 | |
| *** kbaegis has joined #openstack-kolla | 17:21 | |
| inc0 | containers can separate stuff if you need stuff separated | 17:21 |
| kbaegis | back- sorry | 17:21 |
| mgoddard | I think it depends on the graph backend | 17:22 |
| inc0 | but in reality it's glorified chroot | 17:22 |
| kbaegis | Did anyone report the bug or anything while I was gone? | 17:22 |
| inc0 | mgoddard: only if you want to save stuff outside of volume | 17:22 |
| inc0 | volume/bindmount is just a directory | 17:22 |
| mgoddard | that's true | 17:22 |
| *** lucasxu has quit IRC | 17:22 | |
| sdake | inc0 chroot on steriods = those steroids being the ability to share some specific kernel namespaces or not share them at your desire :) | 17:22 |
| sdake | kbaegis no we are waiting on a bug :) | 17:22 |
| kfox1111 | morning. | 17:23 |
| jascott1 | people of earf, | 17:23 |
| jascott1 | good morning! | 17:23 |
| sdake | jascott1 sup dude | 17:23 |
| inc0 | people of urf | 17:23 |
| jascott1 | people of disc world :) | 17:23 |
| sdake | i'm surprised both of you are back in the office so quickly :) | 17:23 |
| sdake | inc0 we have a gate job that needs a review | 17:23 |
| jascott1 | well I see one of us ;) | 17:23 |
| kbaegis | https://bugs.launchpad.net/kolla/+bug/1668346 | 17:23 |
| openstack | Launchpad bug 1668346 in kolla "docker-py 2.1.0 looks broken with docker 1.13.1 which breaks Kolla stable/ocata" [Undecided,New] | 17:23 |
| inc0 | hey jascott1, I'll be working from home today as I have rendez vous with dentist at 2 | 17:23 |
| mgoddard | inc0: I'm currently building a tool to automate zero to bare metal cloud using kolla, bifrost and ironic | 17:23 |
| sdake | inc0 and your the only oen thta can review it that is around | 17:23 |
| inc0 | doing 2 root canals today ;) | 17:24 |
| sdake | sbezverk ^^ | 17:24 |
| inc0 | ;( | 17:24 |
| kbaegis | sdake: That's my very first bug report | 17:24 |
| sdake | kbaegis grats ! you ahve karma now on launnchpad :) | 17:24 |
| kbaegis | inc0: Good luck w/ the root canals! | 17:24 |
| jascott1 | inc0 oh man | 17:24 |
| sdake | wow 2 in 1 day | 17:24 |
| sdake | make sure to get plenty o drugs | 17:24 |
| kbaegis | sdake: I've been trying to reduce my karma footprint | 17:24 |
| jascott1 | no doubt | 17:24 |
| sdake | kbaegis ya I went through that stage when i was 30, now expanding my karma footprint | 17:24 |
| sdake | kbaegis hopefully not overexpanding however :) | 17:25 |
| inc0 | oh joy, docker-py broke docker between versions yet again | 17:25 |
| inc0 | how fun | 17:25 |
| sdake | inc0 we don't have verification | 17:25 |
| sdake | it is speculation | 17:25 |
| sdake | inc0 plz beforey our root anals ack sbezverk 'z patch | 17:25 |
| sdake | canals that is | 17:25 |
| jascott1 | lol | 17:26 |
| sdake | inc0 its blocking his work - we can deal with the docker-py thing whiel your out for however long it takes to recover | 17:26 |
| sdake | its not that bad if it doesn't fail | 17:26 |
| sdake | when i was out for 6-8 weeks, mine failed | 17:26 |
| sdake | thats a clusterfoxtrot | 17:26 |
| sbezverk | inc0: sorry to hear that, may the force be with you!! | 17:27 |
| sdake | inc0 this is blocking further progress - we got your back while your out: https://review.openstack.org/#/c/437504/26 | 17:27 |
| inc0 | may the force be with this dentist | 17:27 |
| inc0 | and guide his hand so it won't kill me with pain | 17:28 |
| *** nhlfr has joined #openstack-kolla | 17:28 | |
| sdake | inc0 are you seeing a dentist or a specialist? | 17:28 |
| sbezverk | inc0: just do not bring your lightsaber with you ;) | 17:28 |
| egonzalez | kbaegis, how you installed docker-py, cannot find it in pip or yum (only source from github) | 17:28 |
| sdake | inc0 if its not an endodontist, I'd recommend canceling your appointment and scheduling with an endodontist stat | 17:29 |
| kfox1111 | I'm reading through the gate job | 17:29 |
| sdake | oh kfox1111 is here yay :) | 17:29 |
| sbezverk | kfox1111: morning | 17:29 |
| inc0 | sooo...guys we need to get rid of this mix and match of image tags | 17:29 |
| sdake | inc0 yes we know this | 17:29 |
| inc0 | I lost track which job/container uses which image | 17:29 |
| sdake | inc0 we are waiting on 3.0.3 to be cut | 17:29 |
| financeMann | fyi kbaegis sdake egonzalez i fixed my problem by running the following command - > “sudo systemctl start docker” LOL | 17:29 |
| sbezverk | kfox1111: I was told that after wild parties at PTG you were going to take at least 1 week off ;) | 17:29 |
| kbaegis | egonzalez: It's gentoo. There are two options: emerge and pip | 17:29 |
| kbaegis | docker is definitely running on my env | 17:29 |
| egonzalez | financeMann, don't forget to systemctl enable docker ;) | 17:30 |
| sdake | sbezverk that was speculation :) | 17:30 |
| inc0 | financeMann: having service running does help | 17:30 |
| kfox1111 | sbezverk: heh. I don't get time off. ;) | 17:30 |
| financeMann | inc0 I already ran that before dunno why it’s not enabled by default. | 17:30 |
| kbaegis | egonzalez tried again after emerging docker-py | 17:30 |
| egonzalez | financeMann, when install a new service have to manually enable on boot, never is made by default | 17:30 |
| kbaegis | egonzalez still broken | 17:31 |
| kbaegis | egonzalez: I can run python2/3 and do import docker without error | 17:31 |
| egonzalez | financeMann, the same for disable libvirt on boot | 17:31 |
| egonzalez | kbaegis, docker images work? | 17:32 |
| kbaegis | yes | 17:33 |
| kbaegis | definitely running | 17:33 |
| *** zioproto has quit IRC | 17:33 | |
| kbaegis | egonzalez: I built them on an ubuntu vm, ported the images over to my gentoo docker setup | 17:33 |
| kbaegis | egonzalez: Where is it looking for docker-py? Again, docker can find it, but is it looking for a specific path? | 17:34 |
| egonzalez | kbaegis, please add in the bug report that is running on gentoo, with ubuntu is working | 17:34 |
| inc0 | mgoddard talk to sayantani01 about bifrost, she also worked on this piece | 17:34 |
| kbaegis | egonzalez: Okay, I'll update it | 17:34 |
| mgoddard | will do, thanks inc0 | 17:35 |
| financeMann | I used virtualbox to create a cowq image and such, i am uploading it to openstack kolla interface under the images, i selected cowq2 image and it’s uploading, is this going to be ok? | 17:35 |
| financeMann | I am not sure is virtualboxs cowq image is the same thing as what openstack says cowq2 | 17:36 |
| egonzalez | financeMann, once kolla is deployed inside a virtualbox VM, forget about virtualbox. But if you cannot virtualize inside vms deployed have to configure libvirt to use qemu instead of kvm | 17:37 |
| egonzalez | financeMann, check https://docs.openstack.org/developer/kolla-ansible/quickstart.html#deploy-kolla | 17:39 |
| egonzalez | financeMann, lines from "In case of deployment using the nested environment..." | 17:39 |
| financeMann | I am not using virtualbox, i just used virtualbox on my mac to create the qcow image | 17:40 |
| financeMann | and im uploading it to my openstack computer | 17:40 |
| *** sacharya has joined #openstack-kolla | 17:40 | |
| financeMann | but virtualbox calls it qcow and openstack calls it qcow2 | 17:40 |
| financeMann | is that comaptaible | 17:40 |
| kbaegis | So for AttributeError: 'module' object has no attribute 'Client', is this something I can code directly in? | 17:41 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Reparent kolla-toolbox from openstack-base https://review.openstack.org/435023 | 17:42 |
| *** manheim_ has joined #openstack-kolla | 17:43 | |
| *** david-lyle has joined #openstack-kolla | 17:44 | |
| egonzalez | kbaegis, https://github.com/docker/docker-py/releases/tag/2.0.0 | 17:45 |
| egonzalez | docker.Client has been renamed to docker.APIClient since 2.0.0 | 17:45 |
| *** david-lyle has quit IRC | 17:45 | |
| egonzalez | is s change that will have to do, but not now IMO since will break all operating systems (none have docker-py bigger that 2) | 17:46 |
| *** david-lyle has joined #openstack-kolla | 17:46 | |
| kbaegis | egonzalez: Oh, okay | 17:49 |
| kbaegis | So I can just change docker.Client to docker.APIClient? | 17:49 |
| kbaegis | Ncie | 17:50 |
| kbaegis | *nice | 17:50 |
| sdake | egonzalez in that case we need a top pin | 17:50 |
| sdake | egonzalez if your willing to do the work i'll make an introduction | 17:50 |
| inc0 | or fix it and add bottom pin | 17:50 |
| kbaegis | sdake: Hey! I'm helping you guys find issues before they become issues :) Finally gentoo is useful for something ;) | 17:50 |
| sdake | inc0 the problem with that is all distors have old versions of docker-py | 17:51 |
| sdake | kbaegis winning ;-) | 17:51 |
| inc0 | sdake: but we install it with pip | 17:51 |
| sdake | inc0 some people do - our docs recommend that - however distros are starting to adopt kolla and we don't want to make it harder on them | 17:51 |
| inc0 | I don't think there is signle repo-based kolla install | 17:52 |
| *** mgoddard has quit IRC | 17:56 | |
| *** david-lyle has quit IRC | 17:58 | |
| sdake | inc0 rdo has packaged kolla containers atleast | 17:59 |
| sdake | not sure about kolla-ansible | 17:59 |
| *** crushil has quit IRC | 17:59 | |
| sdake | so there is a distro packaged version of the container build at minimum :) | 17:59 |
| *** david-lyle has joined #openstack-kolla | 18:01 | |
| *** egonzalez has quit IRC | 18:02 | |
| manheim_ | hi, where/which is the data source for the variables inside j2 Dockerfiles? | 18:07 |
| *** athomas has quit IRC | 18:09 | |
| *** egonzalez has joined #openstack-kolla | 18:11 | |
| *** manheim_ has quit IRC | 18:12 | |
| egonzalez | sdake, if you don't mind tomorrow show me how to top pin docker-py version | 18:14 |
| *** crushil has joined #openstack-kolla | 18:14 | |
| sdake | egonzalez i can introduce you to someone that can show you how | 18:15 |
| sdake | or to the channel that is | 18:15 |
| egonzalez | Cool, something new to learn | 18:16 |
| sdake | egonzalez if it is a legitimate problem - i'm not sure if it is or isn't given that financeMann was able to systemctl start docker and get to a workingstate | 18:16 |
| *** salv-orl_ has joined #openstack-kolla | 18:18 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Configuring ironic gates (basic tests) https://review.openstack.org/437504 | 18:18 |
| *** egonzalez has quit IRC | 18:19 | |
| *** salv-orlando has quit IRC | 18:20 | |
| *** dciabrin has quit IRC | 18:20 | |
| *** dciabrin has joined #openstack-kolla | 18:21 | |
| *** lamt has joined #openstack-kolla | 18:24 | |
| *** kbaegis has quit IRC | 18:31 | |
| *** mgoddard has joined #openstack-kolla | 18:40 | |
| *** sudipto_ has joined #openstack-kolla | 18:40 | |
| *** sudipto has joined #openstack-kolla | 18:40 | |
| *** lamt has quit IRC | 18:41 | |
| *** shamail has joined #openstack-kolla | 18:45 | |
| *** lamt has joined #openstack-kolla | 18:47 | |
| *** kbaegis has joined #openstack-kolla | 18:49 | |
| *** gfidente is now known as gfidente|afk | 18:52 | |
| *** shamail has quit IRC | 18:58 | |
| *** sambetts is now known as sambetts|afk | 19:01 | |
| kbaegis | Has anyone seen fluentd throw "no command specified"? | 19:01 |
| *** mgoddard has quit IRC | 19:04 | |
| financeMann | sdake egonzalez I just threw in a 1TB SSD, how do I tell kolla to use that drive for everything open-stack related | 19:04 |
| kbaegis | financeMann: lvm is used by default | 19:04 |
| financeMann | images, vm, etc | 19:04 |
| financeMann | oh kvm | 19:05 |
| financeMann | ? | 19:05 |
| kbaegis | you should see vgscan return cinder-volumes as a | 19:05 |
| inc0 | https://thehftguy.com/2017/02/23/docker-in-production-an-update/ fun read | 19:06 |
| kbaegis | Any pvs under the cinder-volume vg will be allocated to cinder | 19:06 |
| *** shamail has joined #openstack-kolla | 19:06 | |
| *** shardy is now known as shardy_afk | 19:07 | |
| financeMann | kbaegis can I do that in the GUI from nova? | 19:07 |
| financeMann | configure the new disk | 19:07 |
| *** m1dev has joined #openstack-kolla | 19:08 | |
| kbaegis | financeMann: This is an all-in-one installation? | 19:09 |
| financeMann | ya | 19:09 |
| financeMann | I’m new to this all so it’s on a single host | 19:09 |
| financeMann | trying out stuff | 19:09 |
| financeMann | I just slid in a new SSD and now I want to use since since my main HD is full and it cant handle anymore images | 19:10 |
| kbaegis | https://docs.openstack.org/developer/kolla-ansible/advanced-configuration.html#openstack-service-configuration-in-kolla | 19:10 |
| kbaegis | So you'll ideally want to do mkdir -p /etc/kolla/config/cinder/ | 19:11 |
| financeMann | kbaegis where does kolla store it’s image files, i tried uploading a 50GB one and it failed half way through | 19:12 |
| financeMann | need to delet eit | 19:12 |
| kbaegis | https://docs.openstack.org/ocata/install-guide-ubuntu/cinder-storage-install.html | 19:13 |
| kbaegis | So that's cinder | 19:13 |
| kbaegis | Glance manages images | 19:13 |
| *** crushil has quit IRC | 19:13 | |
| kbaegis | Typically from the local storage | 19:13 |
| kbaegis | Your drive might be too small | 19:13 |
| financeMann | any idea where on the file system | 19:14 |
| kbaegis | Your AIO is on a VM? | 19:14 |
| financeMann | it stores them | 19:14 |
| financeMann | no, it’s one a metal box | 19:14 |
| *** ntpttr_ has joined #openstack-kolla | 19:14 | |
| financeMann | straight host | 19:14 |
| financeMann | no vm | 19:14 |
| *** unicell has joined #openstack-kolla | 19:14 | |
| *** MasterOfBugs has joined #openstack-kolla | 19:14 | |
| *** pramodrj07 has joined #openstack-kolla | 19:14 | |
| kbaegis | Anyone else know where the image store is pulling from? inspect glance container? | 19:14 |
| kbaegis | try /var/lib/docker/volumes/glance/_data | 19:15 |
| inc0 | kbaegis: are you asking where glance images are stored? | 19:15 |
| kbaegis | inc0: Asking anyone- financeMann needs to know | 19:16 |
| *** unicell has quit IRC | 19:16 | |
| inc0 | so default is volume, which is exactly dir you mentioned | 19:16 |
| inc0 | if you're using ceph, then well...ceph | 19:16 |
| kbaegis | He's doing an AIO. Wouldn't make sense to use ceph :) | 19:17 |
| kbaegis | Not that it wouldn't be possible | 19:17 |
| *** sudipto_ has quit IRC | 19:17 | |
| *** sudipto has quit IRC | 19:17 | |
| inc0 | interesting use case tho - what if we create docker volume on ceph?:) | 19:17 |
| *** skramaja has quit IRC | 19:17 | |
| inc0 | no, ceph for aio is pointless | 19:17 |
| inc0 | unless to play around | 19:17 |
| inc0 | docker supports rbd volume driver | 19:18 |
| kbaegis | Docker volume on ceph should work | 19:18 |
| kbaegis | The block device would layer btrfs ontop of rbd | 19:18 |
| inc0 | might be neat idea to extend kolla-ansible external ceph to make use of it | 19:18 |
| kbaegis | inc0: ooc, prechecks is hitting an issue on octavia-certs. Is there a handy way to generate those? | 19:19 |
| *** matrohon has joined #openstack-kolla | 19:19 | |
| *** unicell has joined #openstack-kolla | 19:20 | |
| inc0 | well we do have playbook to generate self signed certs | 19:21 |
| inc0 | but I'd bet octavia doesn't use it as it was merged after play | 19:21 |
| kbaegis | inc0: Gotcha. Yeah, generate-certs or whatever wasn't generating ones for octavia | 19:21 |
| inc0 | might be a bug or me missing few changes to implement it | 19:22 |
| kbaegis | food-for-though for stable | 19:22 |
| inc0 | yeah definetly a bug | 19:22 |
| kbaegis | I'll file that one away as well | 19:23 |
| rwallner | is there a way for kolla to override the local_settings for horizon for such things as OPENSTACK_KEYSTONE_BACKEND and OPENSTACK_KEYSTONE_MULTIDOMAIN_SUPPORT? | 19:23 |
| rwallner | AFAICT, this would have to be edited post deploy | 19:23 |
| financeMann | Can I easily add my new SSD through the website? | 19:23 |
| *** crushil has joined #openstack-kolla | 19:23 | |
| kbaegis | rwallner: advanced config in the docs might be able to do that alongside custom globals.yml config. | 19:24 |
| kbaegis | rwallner: Hopefully nothing crazy with customizing the containers is necessary | 19:24 |
| rwallner | the advanced config seems to only take overrides for ini | 19:24 |
| rwallner | this file is not ini based, and doesnt have jinja template to override | 19:25 |
| inc0 | rwallner: you're right, horizon doesn't use ini so that won't work very well | 19:25 |
| rwallner | it seems | 19:25 |
| inc0 | let me check one thing tho | 19:25 |
| *** bpattewar has joined #openstack-kolla | 19:26 | |
| kbaegis | You might have to do something with a custom .j2 file and kolla-build then. It's a pain | 19:26 |
| *** alop has joined #openstack-kolla | 19:26 | |
| rwallner | yeah, thats my second option, to fork and do something like that | 19:27 |
| *** sudipto has joined #openstack-kolla | 19:27 | |
| *** sudipto_ has joined #openstack-kolla | 19:27 | |
| rwallner | just seeing it i can avoid it atm | 19:27 |
| inc0 | rwallner: so what you can do is to place your customized loca_conf in same dir as you would ini override | 19:27 |
| *** Serlex has quit IRC | 19:27 | |
| inc0 | with caveat of it needing to be full config, not just stuff you've changed | 19:27 |
| inc0 | it will consume template tho, so worst case scenerio you can just cp our own template, place it in /etc/kolla/... | 19:28 |
| inc0 | and add your stuff | 19:28 |
| inc0 | you want this file https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/horizon/templates/local_settings.j2 | 19:28 |
| rwallner | ok, just making sure I understand, so if I place a full local_settings.py in custome config dir, kolla would pick it up | 19:28 |
| rwallner | yup | 19:28 |
| inc0 | yeah, just name it local_settings - without file extension | 19:29 |
| *** pcaruana has quit IRC | 19:29 | |
| rwallner | do i need the .j2 extension? | 19:29 |
| inc0 | looking at this task https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/horizon/tasks/config.yml#L42-L57 | 19:29 |
| inc0 | no, you want just name local_settings | 19:29 |
| rwallner | ah ok i see | 19:29 |
| inc0 | with no extension whatsoevere | 19:29 |
| rwallner | cool thanks for the pointer | 19:29 |
| kbaegis | https://bugs.launchpad.net/kolla/+bug/1668377 | 19:30 |
| openstack | Launchpad bug 1668377 in kolla "kolla-ansible certificates not generating certs for Octavia" [Undecided,New] | 19:30 |
| inc0 | thanks kbaegis | 19:30 |
| inc0 | rwallner: np. Not ideal but we can't really support overrides for all the config formats | 19:30 |
| rwallner | yeah, completely understand the reasoning | 19:31 |
| inc0 | our normal way to handle it is just that - you put whole templatable file in your /etc | 19:31 |
| *** sudipto_ has quit IRC | 19:32 | |
| *** sudipto has quit IRC | 19:32 | |
| *** manheim_ has joined #openstack-kolla | 19:32 | |
| kbaegis | Here's a fun one: https://hastebin.com/idohoronaj.vbs | 19:34 |
| *** sdake has quit IRC | 19:36 | |
| *** gfhellma has quit IRC | 19:36 | |
| *** vhosakot has quit IRC | 19:36 | |
| *** manheim_ has quit IRC | 19:37 | |
| kbaegis | Granted, that's after I upgraded the code to read docker.APIClient on docker-py 2.1.0 | 19:38 |
| *** sudipto has joined #openstack-kolla | 19:39 | |
| *** sudipto_ has joined #openstack-kolla | 19:39 | |
| financeMann | So do I need to add the new harddrive to just KVM or to glance and cinder? | 19:44 |
| *** kbaegis has quit IRC | 19:45 | |
| *** kbaegis has joined #openstack-kolla | 19:46 | |
| *** sudipto has quit IRC | 19:46 | |
| *** sudipto_ has quit IRC | 19:46 | |
| kbaegis | financeMann: Here ceph is actually great. You can use unified storage | 19:47 |
| kbaegis | but otherwise yes, you can append the added disk to the root LV if it's set up that way to expand glance storage, or add it to cinder-volumes | 19:48 |
| kbaegis | Technically there are dozens of ways to do both, but if that didn't already give you some ideas then we're teaching you how lvm works | 19:49 |
| financeMann | kbaegis, what’s the easiest not nessasarily cleanest? Can I just set it in some config file and specify the HD to also be picked up and restart kolla-openstack? | 19:51 |
| kbaegis | financeMann: If your system was installed with lvm, just add it to the root disk | 19:52 |
| *** thumpba has joined #openstack-kolla | 19:54 | |
| kbaegis | financeMann: Commands to learn- pvadd, vgextend, and whatever filesystem utilities are in use | 19:54 |
| *** shamail has quit IRC | 20:00 | |
| *** adrian_otto has quit IRC | 20:00 | |
| kbaegis | This part is confusing the hell out of me | 20:07 |
| kbaegis | "No command specified" | 20:07 |
| kbaegis | Switched to python 3.4, same problem | 20:08 |
| kbaegis | https://hastebin.com/uwuqigumuc.vbs | 20:08 |
| kbaegis | api version 1.26 is definitely supported by docker- | 20:08 |
| financeMann | kbaegis, when you say add it to the root disk, do you mean at the OS level or does KVM have a virtual root disk? | 20:10 |
| *** salv-orl_ has quit IRC | 20:12 | |
| kbaegis | I mean at the os/hypervisor level | 20:13 |
| kbaegis | You just need expanded drive space for glance, correct? | 20:13 |
| kbaegis | For a 50GB image? | 20:14 |
| kbaegis | Is there a way to get kolla to pipe out the command you can use to run a container manually from cli- as opposed to the api call directly? | 20:15 |
| kbaegis | inc0: ? | 20:15 |
| kbaegis | Huh. Well maybe this is a better explanation: https://hastebin.com/aborekugey.vbs | 20:20 |
| inc0 | kbaegis: hmm | 20:20 |
| kbaegis | inc0: Trying to run manually left that in the logs | 20:20 |
| inc0 | that's strange, we specify command in every container | 20:20 |
| inc0 | kbaegis: maybe it's docker 1.13 quirk? | 20:21 |
| inc0 | have you tried to run 1.12 and see if it works? | 20:21 |
| inc0 | frankly I haven't tried 1.13 myself yet | 20:21 |
| kbaegis | let me mask it. I'll get back to you | 20:21 |
| inc0 | one of excersises on my todo list | 20:21 |
| *** ccesario has quit IRC | 20:22 | |
| sbezverk | kfox1111: is there anything I need to clean up in the ironic gate ps? I need these checks in place to start testing inspector portion.. | 20:22 |
| *** gfhellma has joined #openstack-kolla | 20:23 | |
| kbaegis | inc0: Same for me with docker 1.12 | 20:26 |
| inc0 | hmm | 20:26 |
| inc0 | can you break down all the versions and stuff you're using? | 20:26 |
| kbaegis | 1.12.6 specifically | 20:26 |
| inc0 | and publish it to patch? | 20:26 |
| inc0 | bug* | 20:26 |
| kbaegis | okay, I can | 20:26 |
| inc0 | I need to go in a moment and I'm probably not be here till tomorrow | 20:27 |
| kbaegis | Oh, okay! | 20:27 |
| kbaegis | I'll write up the bug then and ttyl | 20:27 |
| kbaegis | Thanks for the help | 20:27 |
| inc0 | depends on how traumatized I'll be in dentists | 20:27 |
| inc0 | any time | 20:27 |
| kbaegis | My wife gets a root canal tomorrow herself | 20:27 |
| kbaegis | gl | 20:27 |
| *** jtriley_ has joined #openstack-kolla | 20:35 | |
| *** jtriley has quit IRC | 20:37 | |
| *** salv-orlando has joined #openstack-kolla | 20:39 | |
| *** jtriley_ has quit IRC | 20:39 | |
| hogepodge | I'm setting up an all-in-one node to use ironic instead of nova as the compute driver. Is there much I need to do to make that switch-over? Has anyone else run that configuration with success from kolla-ansible? | 20:45 |
| *** crushil has quit IRC | 20:45 | |
| inc0 | I'm off, I hope I'll be here tomorrow | 20:46 |
| hogepodge | Also, bit of random feedback, having kolla-build default the number of threads to 2xcores seems a bit aggressive. I wasn't having any luck doing a source build yesterday with the standard settings. Even backing off to just 1 thread I still had to rebuild one container. This was on dedicated hardware. Granted, I think part of the problem under load came from | 20:47 |
| hogepodge | what appears to be a flaky implementation of xfs on top of centos. | 20:47 |
| *** dave-mccowan has quit IRC | 20:47 | |
| kbaegis | ansible/roles/prechecks/tasks/package_checks.yml needs brackets around print (docker.__version__) | 20:51 |
| kbaegis | Can someone claim this free commit? | 20:51 |
| kbaegis | currently just docker.__version__ which should fail | 20:51 |
| financeMann | I am going to run this command “lvresize -l 100%VG /dev/cl_supercomputer/vvolume” but I don’t know what to put for vvolume, I see home root or swap | 20:51 |
| financeMann | I already expanded the group | 20:51 |
| kbaegis | you did lvextend? | 20:51 |
| kbaegis | Because it's just btrfs.resize or whatever after that | 20:52 |
| financeMann | vgextend cl_supercomputer /dev/sde1 | 20:52 |
| financeMann | after pvcreate /dev/sde1 | 20:52 |
| kbaegis | sweet | 20:52 |
| financeMann | so now I need to run the above command but dunno what to put as vvolume | 20:53 |
| kbaegis | root | 20:53 |
| financeMann | vgdisplay is showing the 800 extra gbs now | 20:53 |
| financeMann | gotcha | 20:53 |
| kbaegis | You want the root disk | 20:53 |
| financeMann | alright i went from 50gb to 800gb | 20:54 |
| financeMann | whoohoo | 20:54 |
| kbaegis | nice | 20:54 |
| kbaegis | Now just add them to your filesystem and you're good to go | 20:55 |
| financeMann | “e2fsck -f /dev/cl_supercomputer/root” says “Cannot continue, aborted, dev/cl_supercomputer/root is mounted. | 20:57 |
| kbaegis | You're using ext2? | 20:57 |
| kbaegis | Eww | 20:57 |
| kbaegis | Yeah, so you might need to run that from a livecd | 20:57 |
| financeMann | can i just format the drive? | 20:58 |
| kbaegis | Use an ubuntu recovery cd or something. | 20:58 |
| financeMann | it’s new | 20:58 |
| kbaegis | Umm | 20:58 |
| financeMann | to make it’s patition scheme | 20:58 |
| kbaegis | Well I mean everything is installed on top of that drive | 20:58 |
| kbaegis | So you're talking about a full reinstall | 20:58 |
| kbaegis | Because you're using a fs from 1990 | 20:58 |
| *** crushil has joined #openstack-kolla | 20:58 | |
| financeMann | The patition type on the drive is LINUX LVM | 20:59 |
| kbaegis | Just drop a ubuntu install cd or something in there and run the same command when the drive isn't in use | 20:59 |
| financeMann | nothing is installed on this drive, i literally just connected it in | 20:59 |
| financeMann | it’s empty | 20:59 |
| financeMann | I just formated it to LVM like 2 hours ago | 21:00 |
| kbaegis | So let's explain what you just did | 21:00 |
| kbaegis | You appended that drive to your root filesystem | 21:00 |
| financeMann | 2 hours ago, inserted new SSD, formated it to Linux LVM, then ran the pvcreate, then vgextend | 21:01 |
| kbaegis | So they're logically the same hard drive- the one you installed everything on, and the one that you just added | 21:01 |
| financeMann | lvresize also worked | 21:01 |
| financeMann | oh gotcha | 21:01 |
| financeMann | why isn’t e2fsck working | 21:01 |
| kbaegis | Because ext2 is a PoS | 21:01 |
| kbaegis | And it won't work on "live filesystems" | 21:01 |
| financeMann | But am I using ext2? | 21:02 |
| financeMann | I thought LVM wasn’t ext2 | 21:02 |
| kbaegis | ext sits on top of lvm | 21:02 |
| kbaegis | lvm is a block layer abstraction | 21:02 |
| kbaegis | filesystems sit on top | 21:02 |
| financeMann | But i didn’t personally configure anything ext2 as far as i recall | 21:03 |
| *** TxGirlGeek has quit IRC | 21:03 | |
| kbaegis | ext4 supports this | 21:04 |
| kbaegis | http://askubuntu.com/questions/24027/how-can-i-resize-an-ext-root-partition-at-runtime | 21:04 |
| kbaegis | Same utilities. No idea if ext3 does. Been a while | 21:04 |
| financeMann | i’m centos | 21:05 |
| financeMann | all the way | 21:05 |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:07 | |
| *** Jeffrey4l has quit IRC | 21:07 | |
| *** jemcevoy has joined #openstack-kolla | 21:09 | |
| kbaegis | https://bugs.launchpad.net/kolla/+bug/1668402 | 21:10 |
| openstack | Launchpad bug 1668402 in kolla "Kolla-ansible complains, "No command specified"" [Undecided,New] | 21:10 |
| kbaegis | financeMann: Doesn't matter here | 21:10 |
| kbaegis | You're using a 20-30 yo filesystem | 21:10 |
| *** adrian_otto has joined #openstack-kolla | 21:17 | |
| financeMann | how do ya know that | 21:20 |
| SamYaple | financeMann: (almost) never use ext2. ext2 was not a journaling filesystem. that means it has to do a full fsck everytime it crashes uncleanly | 21:20 |
| financeMann | because i cant even confirm it | 21:20 |
| SamYaple | use ext3 or, preferably, ext4 | 21:21 |
| financeMann | i’m not convinced i’m using ext2, how do i confirm | 21:21 |
| financeMann | everything I am seeing is showing LVM | 21:22 |
| *** bpattewar has quit IRC | 21:22 | |
| SamYaple | im honestly not sure what started this | 21:22 |
| SamYaple | but blkid is going to probe and tell you the FS | 21:22 |
| *** kbaegis has quit IRC | 21:23 | |
| financeMann | SamYaple kbaegis pastebin.com/dmwTwcSk | 21:24 |
| financeMann | I dont see ext | 21:24 |
| financeMann | LVM2 | 21:24 |
| SamYaple | again, im not sure what started this at all | 21:24 |
| SamYaple | you dont format lvm stuff with filesystems at the host level. thats inside the instance | 21:25 |
| financeMann | SamYaple e2fsck -f /dev/vgroup/vvolume | 21:25 |
| financeMann | is failing | 21:25 |
| financeMann | saying aborted, /dev/cl_supercomputer/root is already mounted | 21:25 |
| financeMann | and all I got was dont use ext2 as an answer lol | 21:26 |
| SamYaple | a few questions, where is that failing? on the host? in an instance? | 21:26 |
| financeMann | host | 21:27 |
| financeMann | tryin to give more room to openstack | 21:27 |
| financeMann | for deploys and images and such | 21:27 |
| financeMann | i threw in a new HD | 21:27 |
| financeMann | Im following this guide and am on the last two steps -> http://rgheck.blogspot.com/2012/07/how-to-add-to-lvm-drive.html | 21:27 |
| SamYaple | you shouldnt blindly follow internet guides! | 21:28 |
| SamYaple | if youve done all the steps up until that point, you are done | 21:28 |
| SamYaple | you can double check with `pvs` and `vgs` | 21:28 |
| *** jtriley has joined #openstack-kolla | 21:28 | |
| SamYaple | you should see your new disk and it should be in the old volume group | 21:28 |
| financeMann | it’s all good it’s just when I try to add my new image my computer stops workin cause it says theres only 23k left on the HD | 21:29 |
| financeMann | but the other Drive shows up just fine | 21:29 |
| financeMann | maybe i need to redeploy kolla-openstack | 21:29 |
| financeMann | maybe just aint seein it | 21:30 |
| financeMann | or something | 21:30 |
| SamYaple | financeMann: pastebin the output of `pvs` and `vgs` and `parted -l` please | 21:30 |
| financeMann | SamYaple http://pastebin.com/AJpZ8rEc | 21:32 |
| *** sdake_ has joined #openstack-kolla | 21:33 | |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** Jeffrey4l__ has joined #openstack-kolla | 21:35 | |
| SamYaple | financeMann: run `xfs_growfs /` | 21:35 |
| *** Margin has joined #openstack-kolla | 21:41 | |
| *** Margin has quit IRC | 21:41 | |
| financeMann | SamYaple http://pastebin.com/YgxPcshQ | 21:41 |
| SamYaple | yep. youre done | 21:42 |
| SamYaple | now stop running random commands from the internet and irc! | 21:42 |
| financeMann | just re-deploy? | 21:44 |
| SamYaple | you have additional space now. whatever didn't have enough space before should have enough space now | 21:44 |
| *** dave-mccowan has joined #openstack-kolla | 21:51 | |
| *** satyar has quit IRC | 21:51 | |
| *** jtriley has quit IRC | 21:57 | |
| *** crushil has quit IRC | 21:58 | |
| *** sdake_ has quit IRC | 22:05 | |
| *** mgoddard has joined #openstack-kolla | 22:05 | |
| *** matrohon has quit IRC | 22:06 | |
| *** jtriley has joined #openstack-kolla | 22:10 | |
| sbezverk | jascott1: ping | 22:12 |
| jascott1 | sbezverk pong! | 22:13 |
| sbezverk | jascott1: do you know how to get the location of helm client configuration folder in go? | 22:13 |
| *** Mannheom has joined #openstack-kolla | 22:14 | |
| jascott1 | hmmm not off hand but can look into it | 22:14 |
| Mannheom | hi, how am i supposed to edit manually ovs? let's say i want to run ovs-vsctl add-br, stuff like this | 22:14 |
| SamYaple | Mannheom: you have to do it from inside the container. personally, i always setup an alias | 22:15 |
| Mannheom | i tried to attach to the ovs-vswitch d container | 22:15 |
| sbezverk | jascott1: nvm technosophos replied ;) | 22:15 |
| Mannheom | docker attach <container name> | 22:15 |
| SamYaple | alias ovs-vsctl='docker exec -it openvswitchd ovs-vsctl' | 22:15 |
| Mannheom | but i don;t get outpt | 22:15 |
| jascott1 | sbezverk cool | 22:15 |
| SamYaple | something like that | 22:15 |
| Mannheom | but i don;t get output | 22:15 |
| Mannheom | aaa | 22:16 |
| SamYaple | Mannheom: `docker exec -it` | 22:16 |
| jascott1 | SamYaple just expensed the "Peacock" experience | 22:16 |
| Mannheom | i cannot attach to the container? | 22:16 |
| SamYaple | attach is different than exec | 22:16 |
| jascott1 | :) | 22:16 |
| SamYaple | exec says execute this other process, attach is "attach to running container" which is not a shell | 22:16 |
| Mannheom | i see | 22:16 |
| jascott1 | docker exec -ti <container> /bin/bash | 22:16 |
| jascott1 | forget about attach (usually) | 22:17 |
| Mannheom | there is no bash running in the openstack contaier | 22:17 |
| jascott1 | ah well then | 22:17 |
| SamYaple | Mannheom: well, "bash" isn't a service | 22:17 |
| jascott1 | so its not a nix based container? | 22:17 |
| SamYaple | so no, there isn't a bash _running_ but the executable still exists | 22:17 |
| jascott1 | imean thats true if you say build a scratch ontainer with a go binary | 22:17 |
| jascott1 | but as sam says, bash is there is you are based on a linux distro | 22:18 |
| Mannheom | so, basically when i want to interact with a cotnainer i have to use exec in my case | 22:18 |
| jascott1 | attach is just attaching you to the main pid so yeah you need exec | 22:19 |
| jascott1 | (usually) | 22:19 |
| *** TxGirlGeek has joined #openstack-kolla | 22:19 | |
| Mannheom | but i guess i can fire a bash process along lets say ovs-vswitch in the existing container right? sorry, i know this is not kolla related but docker | 22:20 |
| *** jtriley has quit IRC | 22:20 | |
| *** shardy_afk has quit IRC | 22:23 | |
| jascott1 | Mannheom so yeah exec gets you shell and then you can look at your logs and what not | 22:24 |
| jascott1 | Mannheom also if you need vim and its not there you can mount vim-tiny as a volume when you launch container (something like "-v /bin/vim-tiny:/bin/vim-tiny") | 22:26 |
| Mannheom | yeah, mount a local host folder, nice | 22:26 |
| jascott1 | has to be self contained or will freak without dynamic libs | 22:27 |
| *** mgoddard has quit IRC | 22:27 | |
| Mannheom | so that's why it's tiny and not the standart one | 22:27 |
| jascott1 | ja | 22:27 |
| SamYaple | Mannheom: if you run `docker exec -it <container> ovs-vsctl` it will launch ovs-vsctl directly, no bash invovled | 22:28 |
| Mannheom | so i have a question | 22:28 |
| SamYaple | if you want a shell inside of a container `docker exec -it <container> bash` gives you a shell | 22:28 |
| Mannheom | i've noticed br-tun is not hooked to any interface i would use as rata network | 22:28 |
| Mannheom | where i can tune that before deploy | 22:28 |
| Mannheom | same as br-ex | 22:29 |
| SamYaple | leave br-tun alone. it doesnt get and interface | 22:31 |
| SamYaple | neutron_external_interface is the interface that gets added into br-ex | 22:31 |
| Mannheom | yes, but i am interested in the data network, used to compute node traffice, vms | 22:32 |
| Mannheom | br-data let's say | 22:32 |
| jemcevoy | I am stuck trying to get jumbo MTUs to a VM instance. Has anyone gotten an MTU of 9000 to a VM? | 22:32 |
| *** goldyfruit has quit IRC | 22:34 | |
| *** sdake has joined #openstack-kolla | 22:34 | |
| Mannheom | i guess i can create another bridge for that manually, but what can I do about this to automate at deploy time | 22:34 |
| SamYaple | jemcevoy: yes | 22:34 |
| Mannheom | something in globals.yml i assume | 22:34 |
| SamYaple | Mannheom: kolla-ansible only sets up the br-ex interface. anything beyond that youll have to do by hand | 22:35 |
| SamYaple | or, preferably, extend kolla-ansible to do | 22:35 |
| sdake | SamYaple running random commands is so much fun! | 22:35 |
| Mannheom | hmm, i see, but why? | 22:35 |
| Mannheom | i assume there is a reason for this, would be easy to set a br-data bridge from ansible | 22:35 |
| SamYaple | Mannheom: a script runs to setup the bridge. but only for br-ex. it just needs more logic in the playbooks to handle multiple interfaces | 22:36 |
| Mannheom | i see | 22:36 |
| *** rwallner has quit IRC | 22:40 | |
| *** dave-mccowan has quit IRC | 22:48 | |
| financeMann | SamYaple I made a cowq image and i’m tryin to upload it and i selected cowq2 and when the upload is done it says “TypeError: Cannot read property 'data' of undefined" | 22:51 |
| financeMann | and the image isnt there | 22:51 |
| Mannheom | oh, so logs can be checked in the heka container, nice | 22:59 |
| *** gfidente|afk has quit IRC | 22:59 | |
| *** mgiles has quit IRC | 23:00 | |
| *** rhallisey has quit IRC | 23:01 | |
| jemcevoy | SamYaple: Is my MTU issue caused by the br-int, br-tun and br-ex all have MTU's of 1500? How do I change the MTU's? My ip link show and ovs-vsctl show are here: http://paste.openstack.org/show/600683/ | 23:05 |
| jemcevoy | We planned to go live at pod.penguincomputing.com yesterday but this is one of the things hanging us up... | 23:06 |
| financeMann | where are the logs stored for image uploads? I am trying to upload a new image and it’s failing for some reason | 23:11 |
| *** saneax-_-|AFK is now known as saneax | 23:16 | |
| *** sp__ has quit IRC | 23:21 | |
| *** gfhellma has quit IRC | 23:22 | |
| *** gfhellma has joined #openstack-kolla | 23:23 | |
| *** lamt has quit IRC | 23:23 | |
| Mannheom | hmm, should be there any problem if i attached br-tun to an physical network which i use as private-net in neutron | 23:26 |
| Mannheom | i get a no host found and "Binding failed for port" when i try to nova boot | 23:26 |
| *** lamt has joined #openstack-kolla | 23:27 | |
| *** sp_ has joined #openstack-kolla | 23:29 | |
| *** jrobinson has joined #openstack-kolla | 23:32 | |
| SamYaple | jemcevoy: none of the ovs interfaces MTU matters | 23:33 |
| *** goldyfruit has joined #openstack-kolla | 23:34 | |
| SamYaple | jemcevoy: are you sure the vxlan is not going over the 1gig interfaces? | 23:35 |
| SamYaple | jemcevoy: quick test is to launch and instance on a vlan network | 23:35 |
| jemcevoy | I am sure that the vxlan is on the 10Gig bond... | 23:36 |
| SamYaple | can you post your globals.yaml | 23:36 |
| jemcevoy | I can ping 8972 sized packets over it... | 23:36 |
| jemcevoy | Sure | 23:36 |
| Mannheom | http://paste.openstack.org/show/600685/ | 23:39 |
| hrw | hm. | 23:39 |
| Mannheom | this is what i get when i nova boot, no errors in neutron or ovs logs | 23:39 |
| hrw | ERROR:kolla.image.build.base:Error'd with the following message | 23:39 |
| hrw | ERROR:kolla.image.build.base:mkdir /var/lib/docker/overlay/20419a67c41e532d53038a4e6deafb19c915e3512de4b7b26782c8e5a82b8ed9/tmproot634680353/usr/share/man/man1x: no space left on device | 23:39 |
| *** crushil has joined #openstack-kolla | 23:40 | |
| hrw | with 46GB free on / (no partition for /var /var/lib /var/lib/docker) | 23:40 |
| hrw | any ideas? would love to leave buildrun going through rest of night | 23:40 |
| SamYaple | hrw: running out of inodes? | 23:41 |
| SamYaple | hrw: if you are using overlay (not overlay2) then you can easily run out of inodes | 23:41 |
| hrw | SamYaple: how to switch to overlay2? | 23:42 |
| SamYaple | hrw: whats your distro | 23:42 |
| hrw | debian/jessie + docker.io 1.12 | 23:42 |
| SamYaple | hrw: kernel? | 23:42 |
| sp_ | duonghq: any comment on https://etherpad.openstack.org/p/ks-rolling-upgrade-test ? | 23:42 |
| hrw | SamYaple: 4.4 | 23:43 |
| jemcevoy | my globals.yml http://paste.openstack.org/show/600686/ | 23:43 |
| SamYaple | hrw: should be good. youll just need to set it in your /etc/system/docker.service | 23:44 |
| SamYaple | hrw: where you set the other overrides | 23:44 |
| sp_ | duonghq: your comments on this would be highly helpful to make "ks-rolling-upgrade" success release | 23:44 |
| SamYaple | but youll need to purge all the images | 23:44 |
| hrw | SamYaple: purging all 1515 is fine | 23:44 |
| SamYaple | hrw: reboot is quickest, but youll need to unmount all the overlay mounted crap and then purge /var/lib/docker/* overlay backend stuff | 23:46 |
| hrw | SamYaple: I usually stop docker and rm -rf /var/lib/docker/* | 23:47 |
| SamYaple | hrw: might be ok? might have stuff mounted. but you know what youre doing :) | 23:47 |
| *** salv-orlando has quit IRC | 23:48 | |
| SamYaple | jemcevoy: so. all that looks good. just a note you can use br-10gig as br-ex, you dont need that patch interface (also your config says br0?) | 23:48 |
| *** kong has joined #openstack-kolla | 23:49 | |
| SamYaple | jemcevoy: set 'global_physnet_mtu = 9000' in an override in you neutron.conf as well? | 23:50 |
| sbezverk | jascott1: still around? | 23:51 |
| *** gfhellma_ has joined #openstack-kolla | 23:51 | |
| hrw | SamYaple: thanks for help | 23:52 |
| jemcevoy | br0 is a dummy that I delete .... | 23:52 |
| *** gfhellma has quit IRC | 23:53 | |
| jemcevoy | SamYaple: This is on my ansible vm: [jmcevoy@kolla4echo kolla]$ cat /etc/kolla/config/neutron/neutron.conf | 23:53 |
| jemcevoy | [DEFAULT] | 23:53 |
| jemcevoy | verbose = False | 23:53 |
| jemcevoy | global_physnet_mtu = 9000 | 23:53 |
| jemcevoy | What docker containers should have the setting? | 23:54 |
| SamYaple | jemcevoy: at least the server. possibly all the agents | 23:55 |
| SamYaple | just do it everywhere if in doubt | 23:55 |
| SamYaple | jemcevoy: oh. wait. your doing things funky. can you paste your ml2_conf.ini | 23:55 |
| jemcevoy | how do I get kolla-ansible reconfigure to do it? | 23:55 |
| SamYaple | lets hold off on that for a minute. | 23:56 |
| SamYaple | i think i know your problem, just post your ml2_conf.ini so i can double check | 23:56 |
| jemcevoy | I am doing something funky for sure... It was a long painful weekend... | 23:56 |
| SamYaple | jemcevoy: we can discuss how you make it less silly and more standard without changing your config | 23:57 |
| jemcevoy | SamYaple: my ml2_conf.ini http://paste.openstack.org/show/600687/ | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!