| *** haplo37_ has joined #openstack-kolla | 00:00 | |
| sdake_ | kfox1111 sam tried ceph in the gatte and couldn't get it to work | 00:03 |
|---|---|---|
| sdake_ | however i dont recall if he was done trying or not | 00:03 |
| sdake_ | kfox1111 may ask him for adice | 00:03 |
| sdake_ | kfox1111 he seems open to answering questions atleast on irc :) | 00:04 |
| kfox1111 | ok. cool. | 00:09 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:11 |
| *** diogogmt has joined #openstack-kolla | 00:12 | |
| *** sdake_ has quit IRC | 00:16 | |
| kfox1111 | arg... need both ps's combined now... | 00:20 |
| kfox1111 | I think the remaining issues in the Start testing may be ceph related, which the other ps tests. | 00:20 |
| kfox1111 | may be time to merge the Start testing review. | 00:21 |
| *** schwicht has joined #openstack-kolla | 00:22 | |
| kfox1111 | or, I guess I can pull off the wip to get it ready, and then rease the ceph one on top as a follow on. will have to do that anyway. | 00:22 |
| kfox1111 | heh... this one worked sort of: http://logs.openstack.org/41/381041/31/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/5f94252/console.html | 00:28 |
| kfox1111 | I can't really see much between the working and not working ones. :/ | 00:28 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Split endpoint jobs and start testing the deployments https://review.openstack.org/380868 | 00:30 |
| kfox1111 | sbezverk_: https://review.openstack.org/#/c/380868 Is ready I think. it does create all the endpoints fine. | 00:31 |
| kfox1111 | arg... | 00:34 |
| kfox1111 | ceph-mon image has xfs mkfs but not ext4... weird. | 00:34 |
| *** huikang has joined #openstack-kolla | 00:36 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 00:38 |
| *** schwicht has quit IRC | 00:39 | |
| *** tonanhngo_ has quit IRC | 00:40 | |
| *** HyperJohnGraham has joined #openstack-kolla | 00:42 | |
| *** tonanhngo has joined #openstack-kolla | 00:42 | |
| *** inc0 has quit IRC | 00:45 | |
| *** schwicht has joined #openstack-kolla | 00:46 | |
| *** tonanhngo has quit IRC | 00:47 | |
| *** senk__ has joined #openstack-kolla | 00:49 | |
| *** senk_ has quit IRC | 00:49 | |
| *** tonanhngo has joined #openstack-kolla | 00:49 | |
| *** v1k0d3n has quit IRC | 00:51 | |
| *** v1k0d3n has joined #openstack-kolla | 00:52 | |
| *** tonanhngo_ has joined #openstack-kolla | 00:52 | |
| *** david-lyle has quit IRC | 00:52 | |
| *** sdake has joined #openstack-kolla | 00:53 | |
| *** daneyon has joined #openstack-kolla | 00:54 | |
| *** tonanhngo has quit IRC | 00:54 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Improve VIP existence check https://review.openstack.org/381589 | 00:56 |
| *** tonanhngo_ has quit IRC | 00:56 | |
| *** daneyon has quit IRC | 00:58 | |
| *** g3ek has quit IRC | 00:59 | |
| *** phuongnh has joined #openstack-kolla | 01:00 | |
| *** haplo37 has quit IRC | 01:02 | |
| *** schwicht has quit IRC | 01:03 | |
| *** karlamrhein has quit IRC | 01:03 | |
| *** haplo37 has joined #openstack-kolla | 01:05 | |
| *** duonghq has joined #openstack-kolla | 01:07 | |
| duonghq | morning | 01:07 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (general roles) https://review.openstack.org/358539 | 01:08 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (default roles) https://review.openstack.org/359031 | 01:08 |
| *** g3ek has joined #openstack-kolla | 01:09 | |
| HyperJohnGraham | hi all | 01:09 |
| *** eaguilar has joined #openstack-kolla | 01:11 | |
| *** karlamrhein has joined #openstack-kolla | 01:13 | |
| *** otavio has joined #openstack-kolla | 01:14 | |
| MarMat | sdake fyi, magnum is broken for me https://bugs.launchpad.net/magnum/+bug/1630418 | 01:16 |
| openstack | Launchpad bug 1630418 in Magnum "Minions are not registering because of hostname translation failure" [Undecided,New] | 01:16 |
| sdake | MarMat what happened to "looksgood" :) | 01:16 |
| sdake | MarMat are ou talking about the kube case or the ansible case? | 01:17 |
| sdake | that bug above looks like it has nothign to do with magnum? | 01:17 |
| MarMat | sdake well, ya know, that's the way forward to brigher tomorrows | 01:17 |
| MarMat | sdake it's a magnum thing | 01:17 |
| sdake | oh, you mean inside magnum the minions can't register | 01:18 |
| MarMat | sdake yes, they made a change recently and it broke things, at least for me | 01:18 |
| sdake | MarMat follow me to #openstack-containers please | 01:19 |
| sdake | MarMat what we have done in the past is apply a revert patch on top | 01:24 |
| sdake | as a short term workaround | 01:24 |
| sdake | marmat this is slightly harder then it sounds unforutnately :) | 01:24 |
| sdake | marmat I have faith in you :) | 01:24 |
| *** v1k0d3n has quit IRC | 01:25 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (all other roles) https://review.openstack.org/359096 | 01:25 |
| MarMat | sdake luckily I have some guys behind me who are helping me | 01:26 |
| *** duonghq has quit IRC | 01:26 | |
| sdake | MarMat sweet - its not super hard | 01:26 |
| sdake | just annoying to make the patch ;) | 01:26 |
| sdake | i'd point you at an example that exists today but we have none | 01:26 |
| sdake | if you check the history of the horizon j2 file, yo uwill find an example | 01:27 |
| *** v1k0d3n has joined #openstack-kolla | 01:27 | |
| sdake | i know for certain i added work there to hack around this problem | 01:27 |
| MarMat | sdake well first let's see what they reply to the report, right? | 01:27 |
| sdake | there are also other files that have reverts in them | 01:27 |
| sdake | marmat we work faster then upstream in some cases, my recommendation is to put in the revert now, and revert it later once magnum fixes it (if they think its a bug) | 01:28 |
| MarMat | sdake good, i have to teleport home now and will take a look on it later in the evening | 01:29 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 01:30 |
| sdake | MarMat ship me a teleporter pls :) | 01:30 |
| sdake | i'll even pay for shipping :) | 01:30 |
| sdake | if you want to get fancy teleport it :) | 01:30 |
| *** schwicht has joined #openstack-kolla | 01:32 | |
| *** MarMat has quit IRC | 01:37 | |
| *** duonghq has joined #openstack-kolla | 01:45 | |
| *** huikang has quit IRC | 01:48 | |
| *** huikang has joined #openstack-kolla | 01:48 | |
| *** v1k0d3n has quit IRC | 01:49 | |
| *** huikang has quit IRC | 01:52 | |
| *** v1k0d3n has joined #openstack-kolla | 01:55 | |
| *** v1k0d3n has quit IRC | 02:02 | |
| *** v1k0d3n has joined #openstack-kolla | 02:05 | |
| *** severion has joined #openstack-kolla | 02:07 | |
| *** v1k0d3n has quit IRC | 02:07 | |
| *** MarMat has joined #openstack-kolla | 02:10 | |
| *** unicell1 has quit IRC | 02:17 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 02:18 |
| *** sdake has quit IRC | 02:18 | |
| *** sdake has joined #openstack-kolla | 02:20 | |
| *** schwicht_at_work has joined #openstack-kolla | 02:25 | |
| MarMat | sdake not sure what you mean by history of horizon j2 file, I cannot see anything what would look like a revert patch... anyway now we ate talking about application of a revert patch on magnum, right? | 02:25 |
| *** schwicht has quit IRC | 02:25 | |
| sdake | MarMat right | 02:28 |
| sdake | make the revert patch | 02:28 |
| sdake | then apply it to the dockerile in kolla | 02:28 |
| sdake | this is normal operational behavior for us to unblock others | 02:29 |
| sdake | the reality is this revert patch will be reverted once magnum fixes the issue upstream ;) | 02:29 |
| sdake | MarMat can you ping the mailing list with your problem as well and tag it [kolla][magnum] Patch causes regression in magnum | 02:29 |
| sdake | and then link the patch and give a brief explination along with a link to the bug and ask for bug triage on it. | 02:30 |
| sdake | point out the bug triage is needed from a magnum-driver not a kolla-driver member | 02:30 |
| sdake | otherwise people reading it might think its SEP's ;) | 02:30 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 02:31 |
| *** HyperJohnGraham has quit IRC | 02:32 | |
| *** HyperJohnGraham has joined #openstack-kolla | 02:34 | |
| MarMat | sdake looks like a plan for a nice evening :-) | 02:38 |
| *** yuanying_ has quit IRC | 02:39 | |
| *** haplo37 has quit IRC | 02:39 | |
| *** haplo37 has joined #openstack-kolla | 02:44 | |
| *** eaguilar has quit IRC | 02:46 | |
| *** diogogmt has quit IRC | 02:54 | |
| sdake | rbergeron ping | 03:00 |
| otavio | Is someone working on magnum-ui integration? | 03:12 |
| duonghq | anybody got MariaDB error like this: http://pastebin.com/0cxkUzGe | 03:22 |
| duonghq | after 3 controller form a HA cluster of mariadb is restarted | 03:22 |
| sdake | duonghq master? | 03:30 |
| duonghq | sdake: mitaka | 03:31 |
| sdake | otavio that has not been integrated and noboddy to my knowledge is owrking on it | 03:31 |
| sdake | duonghq how did you install mitaka? | 03:31 |
| sdake | via git or pip | 03:31 |
| sdake | run pip show kolla -> this will give me answer | 03:31 |
| sdake | otavio if you wnat to take a crack - feel free - also note our third party plugins work should enable a customization for htis quite easily | 03:32 |
| sdake | duonghq mitaka mariadb worked from registry when pushed | 03:33 |
| sdake | (and then pulled on a fresh 3 node setup) | 03:33 |
| duonghq | sdake: I'm supporting a operator, he is using pip | 03:34 |
| sdake | so he did pip install kolla? | 03:34 |
| sdake | or pip install . | 03:34 |
| duonghq | pip install kolla | 03:34 |
| sdake | need output of pip show kolla | 03:34 |
| sdake | ok - do you ahve any other data? | 03:34 |
| sdake | such as globals.yml | 03:34 |
| sdake | or are we playing super telephone here :) | 03:34 |
| sdake | prechecks worked? | 03:34 |
| duonghq | It failed after system reboot | 03:35 |
| duonghq | Worked previously | 03:35 |
| sdake | ok so it worked prior | 03:35 |
| sdake | there is this little playbook called "maraidb_recovery" | 03:35 |
| sdake | run that pls :) | 03:35 |
| sdake | mariadb | 03:35 |
| sdake | its an action in kolla-ansible | 03:36 |
| duonghq | yup, seen that before but I'm not sure about this, | 03:36 |
| sdake | lights out requires special recovery circumstances | 03:36 |
| sdake | a reboot = lights out | 03:36 |
| *** daneyon has joined #openstack-kolla | 03:37 | |
| sdake | were all 3 nodes rebooted at same time? | 03:37 |
| duonghq | ya, same time | 03:37 |
| sdake | ya you need mariadb_recovery | 03:37 |
| sdake | it exists for that exact situation | 03:37 |
| sdake | and its in mitaka | 03:37 |
| *** severion has quit IRC | 03:38 | |
| sdake | kolla-ansible -i /path/to/inventory mariadb_recovery should get it g oing | 03:38 |
| duonghq | understood | 03:38 |
| sdake | if there is something else i'm missing let me know ;) | 03:38 |
| sdake | the info on the reboot was ninja slipped in there - that should have been first ;) | 03:38 |
| duonghq | roger | 03:39 |
| sdake | if that doesn't fix it | 03:39 |
| sdake | iptables could be busted in some way | 03:39 |
| sdake | lets try mariadb_recovery and get back to us | 03:39 |
| sdake | you shouldn't need mariadb recovery on a single node reboot | 03:40 |
| sdake | just on a full lights out | 03:40 |
| sdake | the odds of 3 servers failing at the same time are astronomically low | 03:40 |
| sdake | usually this failure is triggered by power loss in the data center | 03:41 |
| *** daneyon has quit IRC | 03:41 | |
| duonghq | sdake: he is testing Kolla and he restarted all servers 'cause of some mystic reasons | 03:41 |
| *** sdake has quit IRC | 03:43 | |
| *** sdake has joined #openstack-kolla | 03:43 | |
| duonghq | sdake: everything is fine now, thanks | 03:44 |
| sdake | duonghq mariadb_recovery worked? | 03:45 |
| duonghq | yup, the cluster is up and running | 03:45 |
| duonghq | but why we need the specific task? | 03:45 |
| duonghq | *action | 03:45 |
| sdake | mariadb has specific recovery mechanism for lights out | 03:46 |
| sdake | no other service does | 03:46 |
| sdake | i debated a long time for a general "kolla-ansible lights-out-recover" action | 03:46 |
| duonghq | and we must trigger the maechanism by hand? | 03:46 |
| sdake | but lost the debate once it became clear only mariaadb was needed | 03:46 |
| sdake | ya that part - the by hand part - is annoying | 03:46 |
| sdake | a proper cluster infrasturcture would just work without a special recovery mechanism | 03:47 |
| sdake | i dont know why mariadb is designed the way it is | 03:47 |
| sdake | this is not a kolla bug tho :) | 03:47 |
| duonghq | sorry but it's mariadb's fault or Galera one? | 03:47 |
| duonghq | just for clarify | 03:47 |
| sdake | galera | 03:47 |
| duonghq | roger | 03:47 |
| sdake | google galera power outage recovery or something like that | 03:48 |
| sdake | you will probably find the docs that were used to construct that playbook :) | 03:48 |
| duonghq | seen | 03:48 |
| sdake | duonghq we can't *make* upstreams of ours do anything | 03:49 |
| sdake | they have to want to do it, and i'm not sure if they care to fix the lights out recovery problem or not | 03:49 |
| *** mdnadeem_ has joined #openstack-kolla | 03:49 | |
| sdake | until that time, the best we can do that I know of atm is make a playbook to handle it - which is what happend 9+ months ago :) | 03:49 |
| duonghq | hmm, it's a bug or a feature? | 03:50 |
| sdake | lgihts out recovery of mariadb is a feature for us | 03:52 |
| sdake | i think its a general design defect of mariadb+galera | 03:52 |
| sdake | answer to your q is - it depends on your pov :) | 03:52 |
| duonghq | I interested in Galera POV, | 03:52 |
| sdake | no idea- ask them :) | 03:53 |
| duonghq | ya | 03:53 |
| duonghq | why we need xtradb backup from percona? | 03:53 |
| sdake | i would find it hard to make an argument for rationalizing it as a feature | 03:53 |
| sdake | duonghq no idea why that is needed | 03:53 |
| sdake | duonghq if you can figure out how to get rid of it, more power to ya | 03:54 |
| sdake | i dont want it in there | 03:54 |
| sdake | thats the only thing we use from percona | 03:54 |
| sdake | the last i heard from sean I htink is that is enables replication in some special way | 03:54 |
| sdake | i dont recall the details | 03:54 |
| sdake | it sounded good | 03:54 |
| sdake | and it is mandatory at present | 03:54 |
| sdake | i think it doesn't need to be | 03:55 |
| duonghq | okay | 03:55 |
| sdake | to be not mandatory requires R&D time tho | 04:02 |
| sdake | its not like "remove it and thingss still work" | 04:02 |
| sdake | (I've tried that..:) | 04:02 |
| duonghq | ya | 04:02 |
| *** yuanying has joined #openstack-kolla | 04:02 | |
| *** jmccarthy has quit IRC | 04:04 | |
| *** jmccarthy has joined #openstack-kolla | 04:05 | |
| duonghq | sdake: do you deploy latest master code? | 04:08 |
| sdake | daily | 04:08 |
| sdake | but not today - internet was out | 04:08 |
| duonghq | I just stuck at "Waiting for virtual IP to appear" | 04:08 |
| duonghq | timetou | 04:08 |
| sdake | still is -at my parents leaching off their internet atm | 04:08 |
| duonghq | I'm not sure why it's use DB port as a sign | 04:10 |
| duonghq | and the DB task hasn't run yet | 04:10 |
| *** yuanying has quit IRC | 04:12 | |
| *** yuanying has joined #openstack-kolla | 04:13 | |
| sdake | duonghq not sure - can't tell from here if master has problem or not | 04:16 |
| sdake | duonghq as i am not able to access my gear | 04:16 |
| duonghq | ya, | 04:16 |
| sdake | duonghq my network at home is down | 04:16 |
| sdake | so can't just ssh into my boxes | 04:17 |
| sdake | and its still down - just tried again | 04:17 |
| duonghq | ya, I think that in US, Internet is very good | 04:17 |
| sdake | it is pretty good in my neighboorhood | 04:19 |
| sdake | gige | 04:19 |
| sdake | but its broken atm | 04:19 |
| sdake | i am going to jet home and work on getting a tech out to fix it or repair it myself | 04:20 |
| sdake | ttyl :) wish me well | 04:20 |
| duonghq | ya | 04:21 |
| duonghq | blame the ISP is good enough? | 04:22 |
| duonghq | how long is it fixed last time? | 04:22 |
| *** sdake has quit IRC | 04:25 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 04:28 |
| *** g3ek has quit IRC | 04:34 | |
| *** haplo37 has quit IRC | 04:35 | |
| *** yuanying has quit IRC | 04:36 | |
| *** g3ek has joined #openstack-kolla | 04:40 | |
| *** haplo37 has joined #openstack-kolla | 04:40 | |
| *** unicell has joined #openstack-kolla | 04:40 | |
| *** salv-orlando has joined #openstack-kolla | 04:41 | |
| *** yuanying has joined #openstack-kolla | 04:42 | |
| *** coolsvap has joined #openstack-kolla | 04:45 | |
| *** unicell has quit IRC | 04:45 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 04:46 |
| *** unicell has joined #openstack-kolla | 04:49 | |
| *** sdake has joined #openstack-kolla | 04:51 | |
| *** unicell1 has joined #openstack-kolla | 04:53 | |
| *** senk__ has quit IRC | 04:53 | |
| *** unicell has quit IRC | 04:54 | |
| sdake | sweet internet working :) | 04:55 |
| sdake | ping rbergeron | 04:55 |
| *** bjolo_ has joined #openstack-kolla | 04:58 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 04:58 |
| *** skramaja has joined #openstack-kolla | 05:02 | |
| bjolo_ | sdake, right channel this time | 05:15 |
| bjolo_ | triage please :) | 05:15 |
| bjolo_ | https://bugs.launchpad.net/kolla/+bug/1626958 | 05:15 |
| openstack | Launchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Undecided,New] | 05:15 |
| bjolo_ | https://bugs.launchpad.net/kolla/+bug/1629246 | 05:15 |
| openstack | Launchpad bug 1629246 in kolla "keystone fernet deploy fail" [Undecided,New] | 05:15 |
| sdake | bjolo_ the second one is marked critical although it feels more like high | 05:16 |
| sdake | the second one (fernet fails to deploy) super critical | 05:16 |
| sdake | bjolo_ that said, we are tagging on the 12th come hell or highwater ;) | 05:17 |
| sdake | and that is a short 8 days away | 05:17 |
| sdake | Jeffrey4l_ if you could turn your considerable talent on this fernet issue | 05:17 |
| sdake | Jeffrey4l_ it would be appreciated | 05:17 |
| bjolo_ | i know, tag is soon | 05:17 |
| sdake | i hear rumblings fernet is not ready to go | 05:17 |
| sdake | Jeffrey4l_ it absolutely needs to be ready to go :) | 05:17 |
| bjolo_ | will do what i can to help out | 05:17 |
| sdake | bjolo_ not sure unless you want to pick up a keyboard - moment wife pingin gme | 05:18 |
| Jeffrey4l_ | will check it . | 05:18 |
| sdake | Jeffrey4l_ cool - the other bug - with the vpn and lbaas not building - we need someone to take a look | 05:26 |
| sdake | Jeffrey4l_ if someone else (other then you) could look that would be great | 05:26 |
| sdake | we got a whole slew of high/critical bugs and I don't think you can solve them all alone :) | 05:26 |
| sdake | i was mia today jerking around with getting my internet working | 05:27 |
| bjolo_ | sdake, i have picked up the keyboard and submitted a few PS (simple ones, but still) :) | 05:27 |
| coolsvap | sdake: i have ubuntu-source build in progress i can triage https://bugs.launchpad.net/kolla/+bug/1626958 | 05:28 |
| openstack | Launchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Critical,Triaged] | 05:28 |
| sdake | well its been triaged | 05:28 |
| sdake | a confirmation is next ;) | 05:28 |
| sdake | if you want to tackle that wfm - although i htink bjolo has pretty much provided evidence it is confirmed | 05:29 |
| Jeffrey4l_ | coolsvap, cool. we need add neutron-service-plugin section in kolla-build.conf file. | 05:29 |
| sdake | coolsvap if by triageyou mean fix, that would rock :) | 05:29 |
| coolsvap | Jeffrey4l_: alright i will pick that up | 05:30 |
| Jeffrey4l_ | hmm. i know the root cause and solution. i will leave some comments in the bug page. | 05:30 |
| sdake | Jeffrey4l_ root cause of which? | 05:30 |
| Jeffrey4l_ | neutron-server + lbaas | 05:30 |
| sdake | fernet? | 05:30 |
| Jeffrey4l_ | https://bugs.launchpad.net/kolla/+bug/1626958 this one. | 05:30 |
| openstack | Launchpad bug 1626958 in kolla "neutron_server not working when lbaas enabled" [Critical,Triaged] - Assigned to Swapnil Kulkarni (coolsvap) | 05:30 |
| sdake | oh - ya that looks pretty straightforward | 05:31 |
| sdake | the fernet bug is the one I'd ask you to look at - since its complex :) | 05:31 |
| Jeffrey4l_ | yep. | 05:31 |
| sdake | and fernet is a highlight feature | 05:31 |
| Jeffrey4l_ | checking it. | 05:32 |
| bjolo_ | Jeffrey4l_, please let me know if i should test anything or provide more logs | 05:33 |
| Jeffrey4l_ | bjolo_, ok. | 05:33 |
| *** msimonin has joined #openstack-kolla | 05:35 | |
| Jeffrey4l_ | bjolo_, could u run the playbook again with `-vvv` parameter. | 05:38 |
| *** v1k0d3n has joined #openstack-kolla | 05:38 | |
| bjolo_ | sure can | 05:38 |
| bjolo_ | just give me a few, my daily build is not completed yet | 05:38 |
| Jeffrey4l_ | OK. np | 05:38 |
| bjolo_ | ubuntu or centos? | 05:38 |
| bjolo_ | the bug was reported on ubuntu source | 05:39 |
| Jeffrey4l_ | either | 05:40 |
| Jeffrey4l_ | so ubuntu + source. | 05:40 |
| bjolo_ | ok | 05:40 |
| sdake | elrog 845 tubes = winning | 05:40 |
| Jeffrey4l_ | sdake, what's that means? elrog? | 05:41 |
| sdake | elrog is a german brand | 05:41 |
| sdake | they make vacuum tubes | 05:41 |
| sdake | for my hobby :) | 05:41 |
| bjolo_ | sdake, if i want to submit some BP, is there deadline for ocata cycle or how does it work? | 05:42 |
| sdake | bjolo_ any deadlines for ocata are at this point undefined | 05:42 |
| Jeffrey4l_ | cool. got. | 05:42 |
| sdake | bjolo_ what Id recommend is to submit the bp | 05:42 |
| *** unicell1 has quit IRC | 05:42 | |
| sdake | we are going to move to a specs model this cycle I suspect | 05:42 |
| *** unicell has joined #openstack-kolla | 05:43 | |
| sdake | really up to what the core team wants to do | 05:43 |
| bjolo_ | i will, but it is all about priorities | 05:43 |
| sdake | the goal of that is to flatten out workload over the cycle | 05:43 |
| sdake | atm, our workload is very spikey around milestones | 05:43 |
| *** v1k0d3n has quit IRC | 05:43 | |
| sdake | and everyone is in chill mode the rest of the time | 05:43 |
| bjolo_ | specs modell as in specs.openstack.org? | 05:43 |
| sdake | i'd like to see our community hae a flat (high) workload | 05:43 |
| sdake | bjolo_ yes like that, but lighter weight | 05:44 |
| bjolo_ | hehe | 05:44 |
| sdake | as LIGHT WEIGHT as possible | 05:44 |
| sdake | the purpose of specs isn't to slow down dev (as used in other projects) | 05:44 |
| *** senk_ has joined #openstack-kolla | 05:44 | |
| sdake | its to help keep priorities straight | 05:44 |
| sdake | and even out workload as well | 05:44 |
| sdake | those two merge together in my mind | 05:45 |
| wznoinsk | sdake: +1 | 05:45 |
| sdake | most projects use specs to slow work down | 05:45 |
| sdake | roadblocks = bad | 05:45 |
| sdake | roadblocks = hassle or everyone | 05:45 |
| sdake | so lets not do that ;) | 05:45 |
| sdake | i suspect a whole bunch of ocata will be spent jerking around with the repo split | 05:46 |
| wznoinsk | sdake: what's the view on new features in ocata in kolla? | 05:46 |
| wznoinsk | (given the shorter release timeframe and "less focus" on features this cycle) | 05:47 |
| sdake | wznoinsk who said less focus on features? | 05:52 |
| sdake | you mean inc0's statements? | 05:53 |
| sdake | I think our focus on features or lack thereof will come out of the specs process we use | 05:53 |
| wznoinsk | nope, I can try to dig out that I think it was on ML, Ocata suppose to be a less/non-feature release | 05:53 |
| sdake | not of any dictate anyone makes | 05:53 |
| sdake | wznoinsk openstack wide? | 05:53 |
| wznoinsk | I better find out the message I'm reffering to first | 05:54 |
| sdake | wznoinsk please dig for subject | 05:54 |
| wznoinsk | while I'm trying to find it, lack of spec/feature freeze dates and other struck me the other day: https://releases.openstack.org/ocata/schedule.html | 05:55 |
| sdake | wznoinsk i dont think its mandatory for us to set that right now | 05:55 |
| sdake | wznoinsk lets tackle specs first | 05:55 |
| sdake | then when to freeze them if/when the core team gets comfortable with specs in the first place ;) | 05:55 |
| *** msimonin has quit IRC | 05:58 | |
| *** salv-orlando has quit IRC | 06:01 | |
| *** egonzalez90 has joined #openstack-kolla | 06:04 | |
| *** tonanhngo has joined #openstack-kolla | 06:06 | |
| wznoinsk | when I think about it now it might have been my wrong interpretation of the shortest release cycle and Ocata being the transision release while we introduce PTGs | 06:07 |
| *** tonanhngo has quit IRC | 06:07 | |
| *** david-lyle has joined #openstack-kolla | 06:08 | |
| bjolo_ | Jeffrey4l_, http://paste.openstack.org/show/584367 | 06:14 |
| bjolo_ | ticket updated as well | 06:14 |
| *** yuanying has quit IRC | 06:16 | |
| bjolo_ | wznoinsk, PTGs? | 06:18 |
| *** bjolo_ has quit IRC | 06:24 | |
| duonghq | anybody stuck at waiting for virtual IP appear? | 06:33 |
| *** shardy has joined #openstack-kolla | 06:33 | |
| *** unicell has quit IRC | 06:33 | |
| *** unicell has joined #openstack-kolla | 06:34 | |
| *** tonanhngo has joined #openstack-kolla | 06:34 | |
| Jeffrey4l_ | roger | 06:35 |
| *** tonanhngo has quit IRC | 06:35 | |
| Jeffrey4l_ | bjolo, it is not full of log. | 06:35 |
| *** salv-orlando has joined #openstack-kolla | 06:36 | |
| *** Serlex has joined #openstack-kolla | 06:39 | |
| *** salv-orlando has quit IRC | 06:40 | |
| *** mnasiadka has joined #openstack-kolla | 06:45 | |
| *** tonanhngo has joined #openstack-kolla | 06:54 | |
| *** MarMat has quit IRC | 06:55 | |
| *** hieulq has quit IRC | 06:55 | |
| *** tonanhngo has quit IRC | 06:56 | |
| *** salv-orlando has joined #openstack-kolla | 06:56 | |
| *** hieulq has joined #openstack-kolla | 06:58 | |
| *** hieulq has quit IRC | 06:59 | |
| *** gfidente has joined #openstack-kolla | 06:59 | |
| *** david-lyle has quit IRC | 07:02 | |
| openstackgerrit | Merged openstack/kolla: Create /var/log/kolla/rally before running rally-manage db create/upgrade https://review.openstack.org/382074 | 07:03 |
| *** hieulq has joined #openstack-kolla | 07:05 | |
| *** hieulq has quit IRC | 07:06 | |
| *** tonanhngo has joined #openstack-kolla | 07:08 | |
| *** haplo37 has quit IRC | 07:09 | |
| duonghq | I got dead keepalived container, anybody see that? | 07:09 |
| *** g3ek has quit IRC | 07:09 | |
| *** tonanhngo has quit IRC | 07:09 | |
| *** matrohon has joined #openstack-kolla | 07:09 | |
| *** hieulq has joined #openstack-kolla | 07:11 | |
| *** daneyon has joined #openstack-kolla | 07:13 | |
| *** athomas has joined #openstack-kolla | 07:15 | |
| bjolo | Jeffrey4l_, no i cut away first 3000 lines that had nothing todo with keystone | 07:15 |
| *** g3ek has joined #openstack-kolla | 07:15 | |
| bjolo | Jeffrey4l_, http://paste.openstack.org/show/584374 | 07:16 |
| *** haplo37 has joined #openstack-kolla | 07:16 | |
| bjolo | ouch | 07:17 |
| *** daneyon has quit IRC | 07:18 | |
| *** b_bezak has joined #openstack-kolla | 07:20 | |
| *** hogepodge has quit IRC | 07:21 | |
| bjolo | whats the char limit on paste.openstack? | 07:23 |
| bjolo | Jeffrey4l_, here is the grep fernet version of the log http://paste.openstack.org/show/584376/ | 07:24 |
| *** msimonin has joined #openstack-kolla | 07:30 | |
| *** msimonin has quit IRC | 07:34 | |
| *** shardy_ has joined #openstack-kolla | 07:34 | |
| *** shardy has quit IRC | 07:36 | |
| *** tonanhngo has joined #openstack-kolla | 07:44 | |
| *** tonanhngo has quit IRC | 07:46 | |
| *** salv-orl_ has joined #openstack-kolla | 07:54 | |
| *** salv-orlando has quit IRC | 07:57 | |
| *** shardy_ is now known as shardy | 07:57 | |
| *** egonzalez90 has quit IRC | 07:59 | |
| *** msimonin has joined #openstack-kolla | 08:00 | |
| *** yuanying has joined #openstack-kolla | 08:01 | |
| *** tonanhngo has joined #openstack-kolla | 08:02 | |
| *** tonanhngo has quit IRC | 08:04 | |
| duonghq | sometime I got kolla_start not found when deploy, after some restart, destroy... it's run again, no clue. | 08:08 |
| *** bmace has joined #openstack-kolla | 08:09 | |
| duonghq | same images, same revision | 08:09 |
| *** hogepodge has joined #openstack-kolla | 08:11 | |
| *** mgoddard has joined #openstack-kolla | 08:16 | |
| *** HyperJohnGraham has quit IRC | 08:18 | |
| wznoinsk | bjolo: http://lists.openstack.org/pipermail/openstack-dev/2016-September/102981.html | 08:24 |
| bjolo | wznoinsk, tnx | 08:26 |
| *** egonzalez90 has joined #openstack-kolla | 08:29 | |
| sdake | Jeffrey4l_ did you see that rtnetlink bug is not a regression | 08:35 |
| sdake | it has always been around | 08:35 |
| *** vincent_vdk has left #openstack-kolla | 08:39 | |
| *** tonanhngo has joined #openstack-kolla | 08:43 | |
| *** strigazi_AFK is now known as strigazi | 08:44 | |
| *** tonanhngo has quit IRC | 08:44 | |
| *** salv-orl_ has quit IRC | 08:48 | |
| *** mkoderer has joined #openstack-kolla | 08:54 | |
| *** sdake has quit IRC | 08:59 | |
| *** tonanhngo has joined #openstack-kolla | 09:03 | |
| *** awiddersheim has quit IRC | 09:06 | |
| *** tonanhngo has quit IRC | 09:08 | |
| *** egonzalez90 has quit IRC | 09:11 | |
| *** awiddersheim has joined #openstack-kolla | 09:11 | |
| denaitre | hello | 09:11 |
| denaitre | I would like to work on kolla based on the newton version of OS | 09:11 |
| denaitre | are the docker images available somewhere? | 09:12 |
| pbourke | denaitre: it's recommend you build them manually | 09:15 |
| denaitre | pbourke: thanks, from master would be fine? | 09:17 |
| pbourke | denaitre: yes | 09:17 |
| denaitre | ok thanks | 09:17 |
| *** berendt has joined #openstack-kolla | 09:20 | |
| *** berendt has quit IRC | 09:20 | |
| *** berendt has joined #openstack-kolla | 09:20 | |
| *** huikang has joined #openstack-kolla | 09:27 | |
| *** tonanhngo has joined #openstack-kolla | 09:32 | |
| *** huikang has quit IRC | 09:33 | |
| *** tonanhngo has quit IRC | 09:33 | |
| *** v1k0d3n has joined #openstack-kolla | 09:40 | |
| *** v1k0d3n has quit IRC | 09:45 | |
| *** haplo37 has quit IRC | 09:50 | |
| *** egonzalez90 has joined #openstack-kolla | 09:51 | |
| *** g3ek has quit IRC | 09:52 | |
| *** tonanhngo has joined #openstack-kolla | 09:53 | |
| *** tonanhngo has quit IRC | 09:54 | |
| *** daneyon has joined #openstack-kolla | 09:55 | |
| *** haplo37 has joined #openstack-kolla | 09:56 | |
| *** g3ek has joined #openstack-kolla | 09:57 | |
| *** daneyon has quit IRC | 10:00 | |
| *** egonzalez90 has quit IRC | 10:02 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix horizon to use cache https://review.openstack.org/382244 | 10:10 |
| *** hieulq has quit IRC | 10:13 | |
| *** rstarmer has joined #openstack-kolla | 10:17 | |
| rstarmer | morning. | 10:17 |
| rstarmer | is it expected that the 2.0 stable/mitaka branch support config merge? Or is that a 3.0 capability? | 10:18 |
| *** salv-orlando has joined #openstack-kolla | 10:19 | |
| *** mgoddard_ has joined #openstack-kolla | 10:22 | |
| pbourke | rstarmer: config merge is in 2.0 | 10:23 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 10:23 |
| rstarmer | pbourke: ok, then I must be doing somethign wrong. I've added an ini file in /etc/kolla/config/cinder-api/cinder.conf on the system running kolla-ansible, and none of the parameters are showing up on my target system. thoughts on where I can look to see what might be breaking? | 10:25 |
| *** mgoddard has quit IRC | 10:25 | |
| *** salv-orlando has quit IRC | 10:25 | |
| pbourke | rstarmer: try putting it in /etc/kolla/config/cinder.conf | 10:26 |
| rstarmer | ok, will give that a run. thanks! | 10:26 |
| pbourke | rstarmer: I'm going to file a bug to make this more obvious as you're not the first to trip over this ;) | 10:27 |
| rstarmer | ah, ok, thanks! | 10:27 |
| rstarmer | alll good now... | 10:31 |
| *** egonzalez90 has joined #openstack-kolla | 10:32 | |
| *** duonghq has quit IRC | 10:38 | |
| *** athomas has quit IRC | 10:40 | |
| *** eaguilar has joined #openstack-kolla | 10:42 | |
| *** zhurong has joined #openstack-kolla | 10:56 | |
| rstarmer | pbourke: if you let me know the bug number, perhaps I can fix the docs | 10:57 |
| *** awiddersheim has quit IRC | 10:57 | |
| *** rstarmer has quit IRC | 11:01 | |
| *** rstarmer has joined #openstack-kolla | 11:02 | |
| *** athomas has joined #openstack-kolla | 11:03 | |
| berendt | when trying to bootstrap neutron with current rc packages i got the error "ImportError: No module named setup" inside the bootstrap container | 11:06 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Fix horizon to use cache https://review.openstack.org/382244 | 11:13 |
| *** DanyC has joined #openstack-kolla | 11:14 | |
| *** DanyC has left #openstack-kolla | 11:15 | |
| pbourke | rstarmer: Im not sure if its so much a doc thing or we need to improve the mechanism. Here's the bug anyway feel free to pitch in https://bugs.launchpad.net/kolla/+bug/1630519 | 11:18 |
| openstack | Launchpad bug 1630519 in kolla "Improvements required for config merging" [Wishlist,New] | 11:19 |
| rstarmer | thanks, I'll see if I can contribute meaningfully :) | 11:19 |
| pbourke | thanks! | 11:19 |
| *** salv-orlando has joined #openstack-kolla | 11:21 | |
| *** salv-orlando has quit IRC | 11:26 | |
| *** eaguilar has quit IRC | 11:26 | |
| *** ccesario has joined #openstack-kolla | 11:28 | |
| *** hrito has joined #openstack-kolla | 11:29 | |
| *** salv-orlando has joined #openstack-kolla | 11:30 | |
| *** coolsvap is now known as coolsvap_ | 11:30 | |
| *** hrito has quit IRC | 11:32 | |
| *** jtriley has joined #openstack-kolla | 11:32 | |
| mliima | morning guys | 11:34 |
| *** b_bezak_ has joined #openstack-kolla | 11:35 | |
| *** shardy is now known as shardy_lunch | 11:35 | |
| *** b_bezak has quit IRC | 11:37 | |
| *** mgoddard_ has quit IRC | 11:44 | |
| *** v1k0d3n has joined #openstack-kolla | 11:45 | |
| *** jtriley has quit IRC | 11:48 | |
| *** v1k0d3n has quit IRC | 11:51 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use keystone_internal_url to access keystone from horizon https://review.openstack.org/382356 | 12:00 |
| *** b_bezak has joined #openstack-kolla | 12:02 | |
| *** msimonin has quit IRC | 12:04 | |
| *** b_bezak_ has quit IRC | 12:04 | |
| openstackgerrit | Merged openstack/kolla: Fix horizon to use cache https://review.openstack.org/382244 | 12:05 |
| *** dwalsh has joined #openstack-kolla | 12:06 | |
| openstackgerrit | Merged openstack/kolla: fixed kestone fernet prechecks for multinode deployments https://review.openstack.org/380014 | 12:09 |
| *** msimonin has joined #openstack-kolla | 12:11 | |
| *** msimonin has quit IRC | 12:11 | |
| *** coolsvap_ is now known as coolsvap | 12:13 | |
| *** jtriley has joined #openstack-kolla | 12:17 | |
| *** phuongnh has quit IRC | 12:17 | |
| *** sdake has joined #openstack-kolla | 12:19 | |
| *** eaguilar has joined #openstack-kolla | 12:20 | |
| *** fguillot has joined #openstack-kolla | 12:22 | |
| *** sean-k-mooneyAFK has quit IRC | 12:22 | |
| *** jtriley has quit IRC | 12:23 | |
| *** tonanhngo has joined #openstack-kolla | 12:26 | |
| kfox1111 | darn... this just missed 1.4: https://github.com/kubernetes/kubernetes/pull/31251 | 12:27 |
| kfox1111 | that woudl have been awesome to use. | 12:27 |
| *** tonanhngo has quit IRC | 12:27 | |
| *** shardy_lunch is now known as shardy | 12:28 | |
| *** haplo37 has quit IRC | 12:28 | |
| sdake | morning peeps | 12:28 |
| sdake | kfox1111 do you ever sleep ;) | 12:28 |
| kfox1111 | my body doesn't believe in it. :/ | 12:28 |
| kfox1111 | morning. :) | 12:29 |
| kfox1111 | seems like you don't sleep either. | 12:29 |
| *** g3ek has quit IRC | 12:29 | |
| *** haplo37 has joined #openstack-kolla | 12:30 | |
| *** schwicht_at_work has quit IRC | 12:30 | |
| *** mgoddard has joined #openstack-kolla | 12:30 | |
| *** g3ek has joined #openstack-kolla | 12:30 | |
| sdake | kfox1111 ptl's job is to be sleep deprived :) | 12:39 |
| kfox1111 | :) | 12:39 |
| sdake | mexican coke coming in today | 12:40 |
| sdake | 24$ a box | 12:40 |
| * sdake liking amazon primenow | 12:40 | |
| kfox1111 | man, this ceph thing is so weird... everything loks to be setup right, no errors in the logs, it works in minikube fine, and it works randomly in the gate occationally. | 12:40 |
| sdake | they also had a 10$ first timer coupuon | 12:40 |
| kfox1111 | :) | 12:41 |
| sdake | 24$ a box is a good deal | 12:41 |
| sdake | and i don't have to lug it around | 12:41 |
| sdake | right to my doorstep | 12:41 |
| sdake | $1 a coke | 12:41 |
| sdake | the lugging part is what stops me from getting the magic of hecho en mexico coca-cola rolling in our house :) | 12:42 |
| kfox1111 | hehe | 12:42 |
| sdake | so lets talk about failures of ceph | 12:43 |
| sdake | why is it failing - and why is it succeeding | 12:43 |
| sdake | understanding this will solve your problem for you | 12:43 |
| sdake | you may just be looking now at "why is it failing" | 12:43 |
| kfox1111 | yup. | 12:44 |
| sdake | looking at when it goes right is also important | 12:44 |
| kfox1111 | I've so far not been able to spot any differences. | 12:44 |
| sdake | the fact that it works tells me your running into some environmental issue of some sort | 12:44 |
| kfox1111 | doesn't seem to make a difference which node types. | 12:44 |
| sdake | how are you doing disk labeling? | 12:44 |
| kfox1111 | seen failures on each. | 12:44 |
| sdake | or using some other ceph | 12:44 |
| kfox1111 | loopback device and just telling it to bootstrap it. | 12:45 |
| sdake | telling which to bootstrap what in which way? | 12:45 |
| kfox1111 | the block device is zero'ed. | 12:45 |
| sdake | the block device comes from a losetup? | 12:46 |
| kfox1111 | yeah. | 12:47 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh is the code | 12:47 |
| kfox1111 | losetup bits here: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L159 | 12:48 |
| otavio | sdake: 3rd-party plugins? where do I think information about it? | 12:48 |
| *** jtriley has joined #openstack-kolla | 12:49 | |
| sdake | kfox1111 in that example your using a bootstrap method which is documented but I don't test | 12:49 |
| sdake | kfox1111 havey ou tried simplifying by using the other parted method? | 12:50 |
| sdake | otavio undocumented atm I htink | 12:50 |
| sdake | kfox1111 re your diskspace, are you sure you are not running out of it | 12:50 |
| otavio | sdake: but in the source, where I can find it? | 12:50 |
| sdake | otavio the docker directory contains j2 files | 12:51 |
| sdake | if you want to customize them, you can read the customization commands by looking at the j2 files | 12:51 |
| sdake | they are in a file called "macros.j2" | 12:51 |
| sdake | otavio this may be documented at htis point - I know pbourke did some work to write some docs in this area recently (iirc) | 12:51 |
| kfox1111 | sdake: the other method requires it to do parted bits itself. I ran into issues with that a while back when I tried it. Cant remember the details though. | 12:52 |
| sdake | kfox1111 "requires it" requires who? | 12:52 |
| sdake | kfox1111 are you saying tha tthe bootstrap 0 -1 method with parted ends up looking like lines 162-164? | 12:52 |
| *** jtriley has quit IRC | 12:53 | |
| sdake | kfox1111 specifically on line 159, bs=1, seek=3g | 12:53 |
| *** rstarmer has quit IRC | 12:54 | |
| sdake | this is wierd usage of dd from my pov | 12:54 |
| sdake | I'd go with something like bs=1M count=3000 | 12:54 |
| *** schwicht has joined #openstack-kolla | 12:54 | |
| sdake | or bs=1m | 12:54 |
| sdake | i don't know what that seek thing does | 12:54 |
| sdake | if it creates a sparse file, that may nto work well with your use case | 12:54 |
| *** tonanhngo has joined #openstack-kolla | 12:54 | |
| kfox1111 | when letting the bootstrap container do the partioning, it wanted to add partitions and labels and things I think that caused issues when loopback. cant remember the exact details. | 12:55 |
| kfox1111 | sdake: the seek dd does a sparse file. | 12:55 |
| kfox1111 | garanteed to be all 0's. | 12:56 |
| *** tonanhngo has quit IRC | 12:56 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Install MySQL-python with pip in horizon container (type source) https://review.openstack.org/382398 | 12:56 |
| sdake | sparse files aren't guaranteed to be anything :) | 12:56 |
| sdake | try taking out the sparse file part | 12:56 |
| sdake | see if that fixes it | 12:56 |
| sdake | while your about it, put a df in there to check disk space | 12:56 |
| sdake | you could be running out of disk space later as that sparse file fills up | 12:57 |
| sdake | this would cause all kinds of wierd crater behavior | 12:57 |
| sdake | or sparse files on their own could be the cause | 12:57 |
| kfox1111 | kk | 12:58 |
| sdake | when debugging software, when we find something busted, we fix that - then on to the next possible problem ;) | 12:58 |
| sdake | not that seek=3g is busted | 12:58 |
| sdake | it seems elegant | 12:58 |
| sdake | but to me, it also seems fragile | 12:58 |
| kfox1111 | certainly couildn't hurt to try. | 12:58 |
| sdake | right | 12:58 |
| sdake | do you have a gate log of where it fails? | 12:58 |
| sdake | vs where it succeeds? | 12:59 |
| sdake | if so, does it always fail in the same way ? | 12:59 |
| *** jheroux has joined #openstack-kolla | 12:59 | |
| kfox1111 | I've got logs up the wazoo. :) | 12:59 |
| kfox1111 | sec | 12:59 |
| kfox1111 | http://logs.openstack.org/41/381041/31/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/5f94252/console.html ceph comes up ok | 13:00 |
| sdake | osic provider | 13:00 |
| sdake | what about fails? | 13:00 |
| kfox1111 | http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/1b20e62/console.html | 13:00 |
| kfox1111 | http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/console.html | 13:00 |
| sdake | so works and fails on same provider? | 13:00 |
| kfox1111 | http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/42ec450/console.html | 13:01 |
| sdake | ok another problem: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/console.html#_2016-10-05_04_57_58_665669 | 13:01 |
| sdake | openstackclient is required on gates | 13:01 |
| *** huikang has joined #openstack-kolla | 13:01 | |
| kfox1111 | thats fine. I just added a huge amount of logging and some of the commands are not there yet when the log dumpper runs on error. | 13:02 |
| sdake | kfox1111 what do you make of this line: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/console.html#_2016-10-05_04_57_54_471518 | 13:02 |
| kfox1111 | same f or that one. | 13:02 |
| kfox1111 | let me show you... | 13:02 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L80 and | 13:03 |
| *** schwicht has quit IRC | 13:03 | |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L39 | 13:03 |
| kfox1111 | so if anything goes wrong, it graps a metric crap ton of logs. :) | 13:04 |
| sdake | right - what i'm getting at is what could possibly cause cat /var/log/messages to fail | 13:04 |
| kfox1111 | ubuntu. | 13:04 |
| kfox1111 | they call it /var/log/syslog. | 13:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 13:06 |
| kfox1111 | so in the logs, the real error happens right before trap_error gets called. | 13:08 |
| kfox1111 | the rest is log collection. | 13:09 |
| kfox1111 | if you slice off the console.html off the url, you c an see all the rest of the logs collected. | 13:09 |
| *** schwicht has joined #openstack-kolla | 13:14 | |
| *** tonanhngo has joined #openstack-kolla | 13:14 | |
| *** jtriley has joined #openstack-kolla | 13:15 | |
| *** jtriley has quit IRC | 13:23 | |
| kfox1111 | sdake: http://logs.openstack.org/41/381041/37/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/946d70e/console.html | 13:23 |
| kfox1111 | unsparse | 13:23 |
| *** rhallisey has joined #openstack-kolla | 13:23 | |
| kfox1111 | sdake: http://logs.openstack.org/41/381041/37/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/946d70e/console.html#_2016-10-05_13_12_58_983862 | 13:25 |
| kfox1111 | looks like there is pleanty of space. | 13:25 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 13:30 |
| *** HyperJohnGraham has joined #openstack-kolla | 13:30 | |
| sdake | kfox1111 https://www.youtube.com/watch?v=4gyeixJLabo | 13:31 |
| sdake | kfox1111 when you have time - might check this video | 13:31 |
| kfox1111 | cool. thanks. | 13:31 |
| *** schwicht has quit IRC | 13:32 | |
| sdake | kfox1111 re http://logs.openstack.org/41/381041/37/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/946d70e/console.html#_2016-10-05_13_12_58_983862 | 13:32 |
| *** daneyon has joined #openstack-kolla | 13:32 | |
| sdake | run a recheck on that | 13:32 |
| *** zhenguo has joined #openstack-kolla | 13:32 | |
| sdake | kfox1111 we want to eliminate variance in the cloud environment | 13:32 |
| sdake | we are most likely to be scheduled on osic | 13:32 |
| sdake | so get the job scheduled to osic (to find the df) | 13:33 |
| sdake | that df you gave was for internap | 13:33 |
| *** rstarmer has joined #openstack-kolla | 13:34 | |
| kfox1111 | http://logs.openstack.org/41/381041/37/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/8c0caa9/console.html | 13:35 |
| *** rstarmer has quit IRC | 13:36 | |
| *** daneyon has quit IRC | 13:37 | |
| *** rstarmer has joined #openstack-kolla | 13:37 | |
| sdake | kfox1111 are you setting /etc/kolla/config/ceph.conf? | 13:38 |
| kfox1111 | should look like: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/console.html#_2016-10-05_04_53_54_257860 | 13:39 |
| *** rstarmer has quit IRC | 13:39 | |
| *** eaguilar has quit IRC | 13:44 | |
| *** mgoddard_ has joined #openstack-kolla | 13:45 | |
| *** schwicht has joined #openstack-kolla | 13:46 | |
| *** ayoung has quit IRC | 13:47 | |
| *** mgoddard has quit IRC | 13:48 | |
| kfox1111 | sdake: cool video. | 13:50 |
| *** jtriley has joined #openstack-kolla | 13:50 | |
| *** dave-mccowan has joined #openstack-kolla | 13:50 | |
| *** huikang has quit IRC | 13:50 | |
| *** MarMat has joined #openstack-kolla | 13:51 | |
| *** huikang has joined #openstack-kolla | 13:51 | |
| *** mgoddard has joined #openstack-kolla | 13:52 | |
| *** caowei has joined #openstack-kolla | 13:53 | |
| *** mgoddard_ has quit IRC | 13:53 | |
| *** salv-orl_ has joined #openstack-kolla | 13:54 | |
| *** huikang has quit IRC | 13:55 | |
| *** salv-orlando has quit IRC | 13:57 | |
| *** mnasiadka has quit IRC | 13:57 | |
| sdake | kfox1111 where is the df in that log | 13:58 |
| sdake | kfox1111 i have looked for it for about 15 mins, and still dont see it | 13:58 |
| *** inc0 has joined #openstack-kolla | 13:59 | |
| sdake | kfox1111 ok, so your working directly with /etc/ceph/ceph.conf then? | 13:59 |
| inc0 | good morning | 13:59 |
| sdake | or generating that from genconfig | 13:59 |
| sdake | sup inc0 | 13:59 |
| sdake | if your generating that from genconfig, it is unclear to me where exactly you specify that you are using only one disk | 13:59 |
| sdake | the merge configs stuff is an ansible work, not part of genconfig | 14:00 |
| *** pbourke has quit IRC | 14:03 | |
| *** pbourke has joined #openstack-kolla | 14:03 | |
| *** lrensing has joined #openstack-kolla | 14:05 | |
| sdake | kfox1111 - can you link me the df line from tha tlast log - i dont see it | 14:05 |
| Jeffrey4l_ | sdake, could u review this https://review.openstack.org/372737 | 14:05 |
| kfox1111 | sdake: both. genconfig, then tweaking it a bit. | 14:05 |
| sdake | Jeffrey4l_ yes | 14:06 |
| Jeffrey4l_ | thanks. | 14:06 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L126 | 14:06 |
| kfox1111 | I tweak it a bit more in the PS, but didn't seem to help. | 14:06 |
| *** huikang has joined #openstack-kolla | 14:08 | |
| *** absubram has quit IRC | 14:08 | |
| *** LamT__ has joined #openstack-kolla | 14:10 | |
| *** huikang has quit IRC | 14:10 | |
| *** huikang has joined #openstack-kolla | 14:11 | |
| *** dims has quit IRC | 14:11 | |
| *** dwalsh has quit IRC | 14:14 | |
| sdake | kfox1111 this magic number: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/console.html#_2016-10-05_04_53_54_354144 | 14:15 |
| sdake | kfox1111 you said infra told you you could use that address | 14:15 |
| sdake | kfox1111 who specifically? | 14:15 |
| sdake | kfox1111 that magic number looks totally suspect to me | 14:15 |
| sdake | and I still dont see df after looking at the logs for 30 minutes | 14:15 |
| *** huikang has quit IRC | 14:15 | |
| *** jtriley has quit IRC | 14:16 | |
| *** dims has joined #openstack-kolla | 14:17 | |
| kfox1111 | that one's dockers default address. | 14:19 |
| kfox1111 | you can see ip stuff here: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/logs/ip.txt | 14:19 |
| kfox1111 | and | 14:19 |
| kfox1111 | here: http://logs.openstack.org/41/381041/36/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/52c3137/logs/routes.txt | 14:20 |
| *** lamt has joined #openstack-kolla | 14:20 | |
| kfox1111 | looks like there shouldn't be any conflict there. | 14:20 |
| sdake | kfox1111 what I want to see is df at the same point as the mons report the 64 stuck states | 14:20 |
| kfox1111 | k. I'll add one to the trace_error hook | 14:23 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 14:24 |
| Jeffrey4l_ | sdake, what's wrong here? https://review.openstack.org/#/c/372737/9/docker/ceilometer/ceilometer-base/Dockerfile.j2 | 14:24 |
| sdake | Jeffrey4l_ nothing | 14:24 |
| Jeffrey4l_ | /var/log/ceilometer is useless and /var/lib/ceilometer is needed | 14:24 |
| sdake | Jeffrey4l_ just commenting on the fact that /var/log/ceilometer was an error | 14:25 |
| Jeffrey4l_ | got. | 14:25 |
| sdake | it should be ignored - nothing for you to be concerned with | 14:25 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: integrate gnocchi with ceilometer https://review.openstack.org/372737 | 14:26 |
| *** dwalsh has joined #openstack-kolla | 14:27 | |
| sdake | pbourke whats the story on the -2 on https://review.openstack.org/#/c/375989/ | 14:32 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla: Heka template missing optional params https://review.openstack.org/382463 | 14:32 |
| pbourke | sdake: its not critical to newton being released | 14:33 |
| sdake | pbourke it is indeed critical | 14:33 |
| sdake | all openstack should have same requirements | 14:33 |
| sdake | otherwise deps dont install properly | 14:33 |
| pbourke | sdake: hmm ok fair point | 14:33 |
| kfox1111 | sdake: konw when 3.x containers will land in the hub? it would be good to test with jewel too.. maybe this is a hammer related bug. | 14:33 |
| pbourke | -2 removed | 14:33 |
| pbourke | sdake: thanks | 14:34 |
| sdake | kfox1111 after the 12th | 14:34 |
| sdake | pbourke roger | 14:34 |
| Jeffrey4l_ | sdake, this one https://review.openstack.org/380824 | 14:34 |
| sdake | Jeffrey4l_ thanks for fixin that | 14:36 |
| sdake | i noticed that the other day and was like "GROAN" | 14:36 |
| Jeffrey4l_ | np ;) | 14:36 |
| openstackgerrit | Merged openstack/kolla: Updated from global requirements https://review.openstack.org/375989 | 14:37 |
| *** senk_ has quit IRC | 14:38 | |
| Jeffrey4l_ | sdake, pbourke shouldn't we stop merge requirements PS ^^ the requirements is now O branch, and kolla is still on N branch. ^^ | 14:38 |
| *** LamT__ has quit IRC | 14:38 | |
| *** mkoderer has quit IRC | 14:38 | |
| *** hogepodge has quit IRC | 14:38 | |
| *** bmace has quit IRC | 14:38 | |
| *** imcsk8 has quit IRC | 14:38 | |
| *** imcsk8 has joined #openstack-kolla | 14:38 | |
| pbourke | :/ | 14:39 |
| *** bmace has joined #openstack-kolla | 14:39 | |
| sdake | oh christ | 14:39 |
| *** hogepodge has joined #openstack-kolla | 14:39 | |
| pbourke | revert? | 14:39 |
| openstackgerrit | Merged openstack/kolla: Handle the KeyboardInterrunpt properly for build.py script https://review.openstack.org/380824 | 14:39 |
| sdake | i thought that was the fix for docker-py | 14:39 |
| sdake | not all requirements | 14:39 |
| sdake | yes revert | 14:39 |
| sdake | and then cherry-pick the docker-py requirements change line | 14:40 |
| sdake | i can do if you like | 14:40 |
| *** mkoderer has joined #openstack-kolla | 14:40 | |
| sdake | pbourke the basic deal was a week or so ago, magnum needed a requirements change for docker-py | 14:40 |
| sdake | I thought that change was this change | 14:40 |
| sdake | without looking at the actual change seeing it changed all kinds of deps | 14:40 |
| sdake | we don't want that obviously :) | 14:41 |
| *** DuncanT has quit IRC | 14:41 | |
| *** LamT__ has joined #openstack-kolla | 14:43 | |
| openstackgerrit | Steven Dake proposed openstack/kolla: Revert "Updated from global requirements" https://review.openstack.org/382470 | 14:43 |
| sdake | pbourke Jeffrey4l_ can you ack that real quick plz | 14:43 |
| Jeffrey4l_ | np | 14:43 |
| *** MagnumBonum has joined #openstack-kolla | 14:43 | |
| MagnumBonum | Hi! I need some help. I have set up a two-node | 14:44 |
| MagnumBonum | Hi! I need some help. I have set up a two-node OpenStack system using Kolla. However, I cannot launch any instances on the second node. When the first node is full of instances, additional launches fail with Host Not Found. | 14:45 |
| *** david-lyle has joined #openstack-kolla | 14:45 | |
| MagnumBonum | It passes all filters except for the last one, "Filter ComputeFilter returned 0 hosts" | 14:45 |
| MagnumBonum | My hunch is that it is related to networking... | 14:46 |
| *** zhurong has quit IRC | 14:46 | |
| *** DuncanT has joined #openstack-kolla | 14:47 | |
| *** caowei has quit IRC | 14:47 | |
| sdake | MagnumBonum interesting hunch | 14:47 |
| sdake | MagnumBonum run nova hypervisor-list | 14:48 |
| MagnumBonum | sdake: yup | 14:48 |
| sdake | paste to a paste service | 14:48 |
| sdake | MagnumBonum can you ssh from one host to another? | 14:49 |
| sdake | and visa-versa | 14:49 |
| MagnumBonum | ➜ ~ openstack hypervisor list -f value 3 control01.beans.local 5 control02.beans.local 7 host04 9 host05 | 14:49 |
| MagnumBonum | sdake, yes | 14:49 |
| MagnumBonum | sdake, kolla-ansible deploy went well | 14:50 |
| sdake | ok | 14:50 |
| sdake | so just run openstack hypervisor list without any options and paste to a paste service | 14:50 |
| MagnumBonum | OK. | 14:50 |
| MagnumBonum | http://pastebin.com/kx6wa5jT | 14:51 |
| sdake | you ahve a total of 4 nodes there | 14:51 |
| MagnumBonum | I have now disabled control01 to force instances to control02 | 14:52 |
| sdake | is it that you have 2 control 2 compute? | 14:52 |
| sdake | ok well dont do that :) | 14:52 |
| sdake | we are debugging your environment | 14:52 |
| MagnumBonum | sdake, yes there are Windows Hyper-V hosts (host4 and host5) | 14:52 |
| MagnumBonum | OK I will enable control02 | 14:52 |
| MagnumBonum | sry control01 | 14:52 |
| sdake | well I know nothing of hyperv :) | 14:53 |
| sdake | so run nova hypervisor-show 3 and 5 in pastes | 14:53 |
| *** afranc has quit IRC | 14:53 | |
| *** afranc has joined #openstack-kolla | 14:53 | |
| *** david-lyle has quit IRC | 14:54 | |
| MagnumBonum | http://pastebin.com/UNzhZa9V | 14:54 |
| sdake | t-7 days :) | 14:54 |
| MagnumBonum | sdake: me neither :-D | 14:54 |
| MagnumBonum | They may be a culprit... | 14:55 |
| sdake | MagnumBonum in this case, you have no vms running | 14:55 |
| sdake | with the hypervisor show you showed me | 14:55 |
| sdake | vcpu=0 on both nodes | 14:55 |
| MagnumBonum | yup. should I deploy? | 14:55 |
| sdake | can you load it up to where yo uthink it breaks | 14:55 |
| sdake | deploy which? | 14:56 |
| *** Serlex has quit IRC | 14:56 | |
| sdake | your application? or openstack | 14:56 |
| MagnumBonum | the app. | 14:56 |
| sdake | yup deploy the app | 14:56 |
| MagnumBonum | I am deploying a stack to see that control01 fills up. | 14:56 |
| sdake | your assertion that the problem is networking related does not seem correct - nova hypervisor shows both hypervisors as active and ready to rock | 14:57 |
| MagnumBonum | http://pastebin.com/qZsvrt1T | 14:59 |
| *** david-lyle has joined #openstack-kolla | 15:00 | |
| MagnumBonum | sry pasted hypervisor 5 twice, hypervisor 3 http://pastebin.com/mVLpjsHg | 15:00 |
| sdake | MagnumBonum hypervisor show 5 and 3 pls, you did 5 twice | 15:01 |
| MagnumBonum | sdake, yes sry check second paste: sry pasted hypervisor 5 twice, hypervisor 3 http://pastebin.com/mVLpjsHg | 15:01 |
| *** eaguilar has joined #openstack-kolla | 15:01 | |
| MagnumBonum | as can be seen, control01 does not have sufficient disk, the flavor requires 20 GB. But control02 does not seem to be valid... | 15:02 |
| *** HyperJohnGraham has quit IRC | 15:02 | |
| sdake | line 35 | 15:02 |
| sdake | your running out of disk space | 15:02 |
| sdake | not vcpus | 15:02 |
| sdake | MagnumBonum ^ | 15:02 |
| sdake | that "host can't schedule" thing means some kind of capacity constraint has been hit in nova | 15:02 |
| sdake | yes - the error isn't helpful at all | 15:03 |
| sdake | I think its a huge problem for nova | 15:03 |
| sdake | everyone complains nobody fixes | 15:03 |
| MagnumBonum | error message: http://pastebin.com/7wAgFmGF | 15:03 |
| MagnumBonum | yup, sdake. there is not enough disk. so it should launch on control02, no? | 15:03 |
| sdake | MagnumBonum how about a simple test before deploying your application | 15:04 |
| *** afranc has quit IRC | 15:04 | |
| MagnumBonum | yes, please! | 15:04 |
| sdake | MagnumBonum possibly it should - but it isn't | 15:04 |
| sdake | MagnumBonum that really isn't kolla's fault, its novas :) | 15:04 |
| MagnumBonum | :) | 15:04 |
| *** dwalsh has quit IRC | 15:04 | |
| sdake | the scheduler in nova - if it can't find resources sometimes it sort of just "gives up" | 15:04 |
| *** g3ek has quit IRC | 15:04 | |
| MagnumBonum | so what test should we do | 15:04 |
| sdake | create a flavor with small disk space requirements | 15:04 |
| sdake | 1 vcpu | 15:05 |
| sdake | 256mb of ram | 15:05 |
| *** schwicht has quit IRC | 15:05 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix the fail when using keystone fernet https://review.openstack.org/382492 | 15:05 |
| sdake | then launch 10 vms one at a time | 15:05 |
| sdake | with this new flavor | 15:05 |
| MagnumBonum | what about m1.tiny? | 15:06 |
| sdake | MagnumBonum run flavor-show m1.tiny | 15:06 |
| sdake | the m1.* was removed from nova | 15:06 |
| sdake | so unless you added it, not sure where thats coming from | 15:06 |
| sdake | nova flavor-show m1.tiny | 15:06 |
| sdake | I'm not sure what m1.tiny is in your environment | 15:06 |
| *** afranc has joined #openstack-kolla | 15:07 | |
| MagnumBonum | I am on mitaka :-/ | 15:07 |
| MagnumBonum | downloading cirros... | 15:08 |
| sdake | right - it was removed in mitaka | 15:08 |
| sdake | MagnumBonum can you do the nova flavor-show m1.tiny | 15:08 |
| sdake | i want t o see what m1.tiny is deifned as | 15:08 |
| *** senk_ has joined #openstack-kolla | 15:08 | |
| sdake | we need something with small ram and small disk and 1 vcpu | 15:08 |
| sdake | enough to fit 10 of em on that 50gb hard disk you have :) | 15:09 |
| sdake | DiskFilter: (start: 4, end: 2) | 15:11 |
| MagnumBonum | http://pastebin.com/3L32cAPt | 15:11 |
| sdake | what this tells me is "hey I tried to schedule 4, but could only end up getting 2 going" | 15:11 |
| Jeffrey4l_ | bjolo, around? | 15:11 |
| Jeffrey4l_ | bjolo, could u try this fix PS for keystone fernet https://review.openstack.org/382492 | 15:11 |
| MagnumBonum | sdake so m1.tiny is 1 GB each which should comfortably fit | 15:12 |
| sdake | MagnumBonum yup m1.tiny is good | 15:12 |
| sdake | how did you get kolla to deploy hyperv? | 15:12 |
| *** schwicht has joined #openstack-kolla | 15:13 | |
| MagnumBonum | sdake, they are installed via Cobbler + the CloudBase stuff. May use Ironic in future | 15:13 |
| sdake | MagnumBonum they are speaking to kolla's control nodes, which means they are authenticating with rabbitmq | 15:14 |
| sdake | how did you get that part to work? | 15:14 |
| *** salv-orl_ has quit IRC | 15:15 | |
| MagnumBonum | just entered the rabbitmq user "openstack" + password from /etc/kolla/passwords.yml | 15:15 |
| MagnumBonum | had to specify a raw IP, not VIP for connection though | 15:15 |
| MagnumBonum | http://pastebin.com/AfXbMLZY | 15:15 |
| *** huikang has joined #openstack-kolla | 15:15 | |
| MagnumBonum | 10 instances of cirros running! | 15:15 |
| *** g3ek has joined #openstack-kolla | 15:16 | |
| MagnumBonum | and they are spread between control01 and control02 ---- weirdness | 15:16 |
| sdake | there ya go | 15:16 |
| sdake | root caused :) | 15:16 |
| sdake | MagnumBonum we recommend ceph for storage for your nodes | 15:17 |
| sdake | or atleast I do :) | 15:17 |
| MagnumBonum | me too, we haven't gotten around to it yet. | 15:17 |
| sdake | you can use external ceph (your own deploy) or we have a ceph that is containerized | 15:17 |
| sdake | both work well | 15:17 |
| *** dwalsh has joined #openstack-kolla | 15:18 | |
| sdake | would this scenario happen in the real world (the nova problem)? not sure - i think quota management comes into play here | 15:18 |
| sdake | recommend filing a bug against nova | 15:18 |
| sdake | saying that when nova runs out of disk space, other machines are not used to schedule | 15:18 |
| MagnumBonum | ok so there is one weirdness, instance 1,2,3 and 6 have two Floating IP. | 15:18 |
| sdake | and say you are using local disk storage | 15:18 |
| MagnumBonum | OK. so we may be better off using Ceph, this problem would not occur? | 15:19 |
| sdake | if you dont use ceph your in for a world of pain :) | 15:19 |
| MagnumBonum | OK. sounds like a plan for tomorrow then... | 15:19 |
| sdake | ceph centralizes storage | 15:19 |
| MagnumBonum | yup we want to go there. | 15:20 |
| sdake | rather then decentralizing it as is done with the default config | 15:20 |
| sdake | MagnumBonum file a nova bug - and then link to me when done and then we can tackle your next issue | 15:20 |
| sdake | MagnumBonum give them enough logs to work with :) | 15:21 |
| MagnumBonum | I will. thank you for your help. | 15:21 |
| sdake | i'd file bug myself but you have all the data (and the particular problem) | 15:21 |
| sdake | MagnumBonum roger - thats what we are here for (besides implementing all this stuff in addition:) | 15:22 |
| MagnumBonum | sdake: should I follow this: https://wiki.openstack.org/wiki/Bugs ? | 15:22 |
| sdake | MagnumBonum here is the link you use: | 15:22 |
| *** diogogmt has joined #openstack-kolla | 15:22 | |
| sdake | https://bugs.launchpad.net/nova/+filebug | 15:22 |
| sdake | just roll with what yo uthink is right - try to give them more then enough information to prove the case | 15:23 |
| sdake | then once done, I'll confirm that I see same behavior in your environment | 15:23 |
| *** david-lyle has quit IRC | 15:26 | |
| *** david-lyle has joined #openstack-kolla | 15:32 | |
| kfox1111 | back | 15:32 |
| kfox1111 | sdake: http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/dd875d4/console.html | 15:33 |
| kfox1111 | df at time of error http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/dd875d4/logs/df.txt | 15:33 |
| kfox1111 | looks fine | 15:33 |
| *** duonghq has joined #openstack-kolla | 15:34 | |
| *** vhosakot has joined #openstack-kolla | 15:37 | |
| sbezverk_ | sdake: any links for flannel troubleshooting guides? | 15:38 |
| sdake | sbezverk_ none that i know of | 15:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 15:39 |
| sdake | kfox1111 that is 90gb, the other time you ran df, it was 150gb | 15:39 |
| sdake | did this run in osic? | 15:39 |
| sdake | or somewhere else | 15:39 |
| kfox1111 | sbezverk_: what issue are you having with flannel? | 15:39 |
| kfox1111 | oh. here's an osc one: http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/6c04d05/console.html | 15:40 |
| kfox1111 | http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/6c04d05/logs/df.txt | 15:40 |
| kfox1111 | each check experimental is producing 3 runs right now. | 15:41 |
| *** zhenguo has quit IRC | 15:43 | |
| sbezverk_ | kfox1111: no connectivity between remove nodes, no packets going through flannel interface | 15:43 |
| sbezverk_ | kfox1111: remote nodes | 15:43 |
| kfox1111 | iptables in the way? (sorry, have to ask) | 15:43 |
| kfox1111 | flannel.1 ip addresses non overlapping and in the right space? | 15:44 |
| kfox1111 | are there multiple nics in the machines? | 15:44 |
| sbezverk_ | kfox1111: nope, nothing no firewalld I have | 15:44 |
| sbezverk_ | the same setup and it is working perfectly | 15:45 |
| sbezverk_ | but on John's it does not | 15:45 |
| kfox1111 | (did you stand it up or did you use a deamonset from the new repo's?) | 15:45 |
| *** salv-orlando has joined #openstack-kolla | 15:45 | |
| sbezverk_ | kofox1111: it is the one that came with kubeadm installation | 15:46 |
| kfox1111 | etcdctl get /coreos.com/network/config (or whichever prefix you used | 15:46 |
| kfox1111 | ah. | 15:46 |
| kfox1111 | there isn't a pure flannel one last I looked. which version did you use? | 15:46 |
| kfox1111 | cannal? | 15:46 |
| inc0 | kfox1111, why do we need storage_ceph.key if we already have secret defined? | 15:46 |
| sbezverk_ | kfox1111: actually they call it canal, combination of flannel with calico | 15:47 |
| sdake | pbourke re https://review.openstack.org/#/c/382463/1 | 15:47 |
| sbezverk_ | and it is version 0.61 | 15:47 |
| sdake | pbourke is that needed for everything else too? | 15:47 |
| kfox1111 | sbezverk_: ok. was just curious. | 15:47 |
| inc0 | https://github.com/openstack/kolla-kubernetes/blob/master/services/common/common-pv.yml.j2#L35 | 15:47 |
| kfox1111 | it has a hard coded addres space. does that conflict with anything he has? | 15:47 |
| sbezverk_ | kfox1111: it does not have hardcoded space, I figured out how to customize it | 15:48 |
| kfox1111 | inc0: that stuff is a bit in flux. I never really liked that way of doing things and am working on gutting that code. | 15:48 |
| sbezverk_ | and I made sure it is not overlapping anywhere | 15:48 |
| openstackgerrit | Waldemar Znoinski (wznoinsk) proposed openstack/kolla: use ironic_conductor volume for conductor's /var/lib/ironic https://review.openstack.org/372118 | 15:48 |
| kfox1111 | sbezverk_: yeah, you can easily sed it to something else before importing it. was just double checking you saw that. | 15:48 |
| inc0 | kfox1111, so we should be good to just remove this conditional right? | 15:48 |
| kfox1111 | inc0: I'm leaving it for now, as if you just delete the entry in the config, it ignores it. I've configuring the gate job to do it differently, using the secret path: https://review.openstack.org/#/c/381041/ | 15:49 |
| sdake | MagnumBonum did you wrap up that nova bug - i was dc'ed briefly | 15:49 |
| kfox1111 | inc0: once I get that worklfow working relyiably, then I'll push to have the torage_ceph.key thing totaly removed. its pretty dangerious I think. | 15:50 |
| kfox1111 | too corse grained. | 15:50 |
| MagnumBonum | sdake: still working on it... | 15:50 |
| sdake | MagnumBonum roger welcome to my world :) | 15:50 |
| kfox1111 | sbezverk_: can you find the service ip for the cannal etcd? | 15:50 |
| kfox1111 | then do something like: etcdctl --endpoint http://... get /coreos.com/network/config | 15:51 |
| inc0 | also..fsType does anything in tpls? | 15:52 |
| kfox1111 | lets see if averything in there looks ok and the nodes are all registering themselves. | 15:52 |
| kfox1111 | tpls? | 15:52 |
| rhallisey | inc0, what's on the schedule for the meeting? | 15:52 |
| inc0 | we hardcode ext4 in templates | 15:52 |
| kfox1111 | inc0: the ps fixes that. | 15:53 |
| *** schwicht has quit IRC | 15:53 | |
| inc0 | rhallisey, rc2 and summit schedule are my topics | 15:53 |
| *** david-lyle has quit IRC | 15:53 | |
| rhallisey | inc0, ok | 15:53 |
| *** schwicht has joined #openstack-kolla | 15:54 | |
| inc0 | wanna wanna add k8s state-of-union to the bunch? | 15:54 |
| inc0 | ;) | 15:54 |
| *** matrohon has quit IRC | 15:54 | |
| rhallisey | the 2 kolla-kubernetes sessions are 1) kolla-kubernetes architecture 2) kolla-kubernetes road map | 15:54 |
| rhallisey | road map will be the state-of-the-union | 15:55 |
| sdake | kfox1111 my speculation that out of disk space was the problem is wrong: http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/6c04d05/console.html#_2016-10-05_14_30_31_741797 | 15:55 |
| sdake | kfox1111 add dmesg to the end of the setup_gate.sh | 15:55 |
| inc0 | I was thinking of todays meeting | 15:55 |
| rhallisey | inc0, have a meeting conflict | 15:55 |
| rhallisey | oh for today | 15:55 |
| kfox1111 | sdake: think that will have anything syslog doesnt have? | 15:55 |
| sdake | kfox1111 also recommend checking your not ooming | 15:55 |
| sdake | kfox1111 YES | 15:55 |
| rhallisey | inc0, if we do it first I can make it | 15:55 |
| sdake | kfox1111 because your not actually saving syslog :) | 15:55 |
| *** hrito has joined #openstack-kolla | 15:55 | |
| kfox1111 | sdake: yeah I am.: http://logs.openstack.org/41/381041/38/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/6c04d05/logs/messages | 15:56 |
| kfox1111 | its syslog on ubuntu machines, and messages on redhat machines. | 15:56 |
| sbezverk_ | kfox1111: http://paste.openstack.org/show/584495/ | 15:57 |
| sdake | kfox1111 i see | 15:57 |
| kfox1111 | sbezverk_: that looks ok... | 15:58 |
| sdake | kfox1111 well what we need is just dmesg without all the other stuff :) | 15:58 |
| sdake | kfox1111 and a memory check | 15:58 |
| kfox1111 | sbezverk_: are there multiple nics on the boxes, or some of the boxes? | 15:58 |
| sbezverk_ | kfox1111: yes | 15:58 |
| kfox1111 | sdake: k. I'll gather dmesg. what do you want for the memory check? | 15:58 |
| sdake | kfox1111 not quite sure | 15:59 |
| sdake | kfox1111 if there is a way to get top to print out memory without updating - that would be ideal | 15:59 |
| sdake | there is probably a one-shot cli operation | 15:59 |
| sdake | but i am connected to vpn atm | 15:59 |
| sdake | and can't actually login to my machines to tell you and we have team meeting now ;) | 15:59 |
| kfox1111 | probably a way to get ps to do that... | 15:59 |
| kfox1111 | sbezverk_: one issue I had with flannel on multinic machines was which ip it used for tunneling. | 16:00 |
| inc0 | meeting time folks | 16:00 |
| kfox1111 | if the nodes have different nics and each one can't talk to all the others, it might cause issues. | 16:00 |
| kfox1111 | I've seen it by picking two hosts, A and B, | 16:00 |
| kfox1111 | running ping on A to the ip of flannel.1 on B | 16:01 |
| kfox1111 | then doing a tcpdump on B for the vlxan traffic. | 16:01 |
| kfox1111 | I noticed the ip on the response from B was going to a not desired ip. | 16:01 |
| MagnumBonum | sdame | 16:02 |
| sbezverk_ | kfox1111: well it is not the case here, stats on flannel interface are all 0 | 16:02 |
| MagnumBonum | sdake so how do I tag you | 16:02 |
| sbezverk_ | so the traffic from pod is not even making to the interface to be encapsulated | 16:02 |
| MagnumBonum | sdake here is the bug https://bugs.launchpad.net/nova/+bug/1630658 | 16:03 |
| openstack | Launchpad bug 1630658 in OpenStack Compute (nova) "nova-scheduler fails when running out of disk space" [Undecided,New] | 16:03 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 16:05 |
| sdake | MagnumBonum ok we have team meeting now but i did leave a comment in your support :) | 16:07 |
| *** huikang has quit IRC | 16:11 | |
| *** huikang has joined #openstack-kolla | 16:11 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Split endpoint jobs and start testing the deployments https://review.openstack.org/380868 | 16:14 |
| *** eaguilar has quit IRC | 16:14 | |
| *** huikang has quit IRC | 16:16 | |
| sbezverk_ | kfox1111: please check my comments in PS 30 380868 | 16:17 |
| *** haplo37_ has quit IRC | 16:18 | |
| openstackgerrit | Merged openstack/kolla: Heka template missing optional params https://review.openstack.org/382463 | 16:18 |
| *** haplo37_ has joined #openstack-kolla | 16:21 | |
| *** DanyC has joined #openstack-kolla | 16:22 | |
| sdake | short meeting | 16:25 |
| duonghq | very short indeed, huh? | 16:25 |
| inc0 | yeah, everyone knows what to do:) | 16:25 |
| duonghq | today, I hit my keyboard many times because keepalived container failed to grap the VIP | 16:26 |
| duonghq | anybody got this kind of error? | 16:26 |
| kfox1111 | rhallisey: thanks for the note about the blueprints... I havent looked at them in a while. I just closed a bunch of done stuff and left some status comments. | 16:26 |
| duonghq | the keepalived doesn't get MASTER role, indeed | 16:26 |
| *** DanyC has quit IRC | 16:26 | |
| *** egonzalez90 has quit IRC | 16:27 | |
| kfox1111 | sbezverk_: you see no tcpdump traffic at all over the vxlan ports? | 16:28 |
| kfox1111 | sbezverk_: k | 16:28 |
| *** ayoung has joined #openstack-kolla | 16:28 | |
| kfox1111 | sbezverk_: we need to go through the code base and just do a ws cleanup. theres a lot of that in there. :/ | 16:29 |
| *** DanyC has joined #openstack-kolla | 16:29 | |
| sdake | MagnumBonum ok meeting over- you have a second problem ? | 16:29 |
| berendt | sorry, i missed the meeting :/ | 16:31 |
| sdake | berendt there are logs | 16:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Split endpoint jobs and start testing the deployments https://review.openstack.org/380868 | 16:31 |
| sdake | berendt along with a general finish the job 4 step process | 16:32 |
| sdake | berendt worth reading i think - atleast the steps ;) | 16:32 |
| *** DanyC has left #openstack-kolla | 16:32 | |
| *** unicell1 has joined #openstack-kolla | 16:32 | |
| duonghq | sdake: did you got above keepalived problem? | 16:33 |
| *** unicell has quit IRC | 16:33 | |
| kfox1111 | sbezverk_: I fixed the ws issue. | 16:33 |
| sdake | duonghq i haven't - can you file a bug | 16:33 |
| sbezverk_ | kfox1111: ok | 16:34 |
| duonghq | sure, but I'm not sure it's really a bug, today it hit me many time, tomorrow maybe it run away | 16:35 |
| hrito | sdake: hi, i wrote a bp and its status is discussion now. what is the next step? | 16:36 |
| sdake | hrito in flux - inc0 has talked about going to a specs process and I think we think its a good idea | 16:36 |
| sdake | duonghq i see - so is inconsistent from day to day, but comes and goes? | 16:37 |
| inc0 | hrito, basically our code is frozen now, for week more | 16:37 |
| inc0 | but please link the bp so we can discuss | 16:37 |
| inc0 | until we agree on new spec process old process keeps working | 16:38 |
| sdake | inc0 wfm | 16:38 |
| hrito | https://blueprints.launchpad.net/kolla/+spec/graceful-shutdown | 16:38 |
| duonghq | sdake: it comes today, for about 80% | 16:38 |
| *** huikang has joined #openstack-kolla | 16:38 | |
| inc0 | hrito, it's accepted:) | 16:38 |
| inc0 | and imho very important, so as soon as we branch newton (next week approx) you're free to push code | 16:39 |
| hrito | :) | 16:39 |
| kfox1111 | hrito: awesome. :) do you know which processes support it? I implemented it a different way for kolla-k8s and would like to use the native solution if possible. | 16:39 |
| inc0 | you still can publish reviews now, just will not be reviewed with high priority (bugs comes first) or merged (feature freeze) | 16:40 |
| sdake | hrito ya - just lookingat it already it was marked for ocata | 16:40 |
| inc0 | for next week | 16:40 |
| sdake | hrito you can work on it now, but wont be able to commit it to repo for about a week | 16:40 |
| sdake | its priority was also essential | 16:40 |
| hrito | graceful shutdown is implemented in oslo.service | 16:40 |
| kfox1111 | hrito: kubernetes has a preStop hook that we can use to ask it to gracefully shutdown. | 16:41 |
| hrito | so if process uses it, we can use graceful shutdown | 16:41 |
| kfox1111 | just need to know what the command is, and which services support it. | 16:41 |
| duonghq | sdake: ah, remembered, if it's failed and I leave it alone for awhile and deploy again, it's ok until I destroy, reboot the node and deploy | 16:41 |
| *** dwalsh has quit IRC | 16:44 | |
| hrito | send SIGTERM to processes hooks graceful shutdown, and its implemented nova, cinder and newtron processes as far as i know | 16:45 |
| *** unicell1 has quit IRC | 16:46 | |
| qwang | rhallisey: Hi Ryan. I have some questions about the workflows in kolla-k8s, do you have a moment now? | 16:47 |
| inc0 | qwang, he mentioned that he had another meeting around now, so you might want to ask agian later:) | 16:48 |
| qwang | inc0: sure. thanks | 16:49 |
| *** david-lyle has joined #openstack-kolla | 16:52 | |
| *** huikang has quit IRC | 16:56 | |
| *** huikang has joined #openstack-kolla | 16:56 | |
| sdake | duonghq wierd | 16:58 |
| kfox1111 | hrito: was it added in neuton or is it supported in mitaka too? | 16:58 |
| sdake | duonghq is it reproducibel? | 16:58 |
| sdake | duonghq sounds like a bug, but I don't see it | 16:58 |
| sdake | duonghq but I don't reboot running nodes often | 16:58 |
| duonghq | sdake: tommorow I think I can reproduce it quite easy | 16:59 |
| *** senk_ has quit IRC | 16:59 | |
| duonghq | sdake it fails even without reboot | 16:59 |
| duonghq | just after the cluster is destroy | 16:59 |
| sdake | definately neverseen that | 16:59 |
| sdake | and I do that all the time | 16:59 |
| duonghq | the keepalived container take very long time to got MASTER state | 16:59 |
| duonghq | sdake: yup, today is the 1st time I got it | 17:00 |
| sdake | duonghq please file a bug :) | 17:00 |
| sdake | duonghq atleast we can get a place to track this kind of conversation | 17:00 |
| duonghq | on both 14.04 and 16.04 host (VM in VirtualBox) | 17:01 |
| sbezverk_ | kfox1111: I kind of pinpoint the failure point, but not sure where to go after. have second to discuss? | 17:01 |
| kfox1111 | yeah | 17:01 |
| *** huikang has quit IRC | 17:01 | |
| sbezverk_ | kfox1111: the issue is veth is not plugged or plugged incorrectly into docker | 17:01 |
| kfox1111 | I don't think they plug into docker0 with canal | 17:02 |
| sbezverk_ | I tested connectivity and between flannels node I can ping their interfaces | 17:02 |
| hrito | kfox1111: i think it is supported in mitaka, but i have not confirmed it | 17:02 |
| kfox1111 | from the flannel interfaces? | 17:02 |
| sbezverk_ | right | 17:02 |
| kfox1111 | k. | 17:03 |
| kfox1111 | hrito: k. thanks. | 17:03 |
| sbezverk_ | when I ping from container I do see ping traffic on veth cali414aa30554e | 17:03 |
| sbezverk_ | but then nothing gets bridged to flannel | 17:03 |
| sbezverk_ | so whatever ties these to together is either missing | 17:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 17:04 |
| sbezverk_ | or misconfigured | 17:04 |
| kfox1111 | hmm... | 17:04 |
| kfox1111 | can you ping the host its on from the container? | 17:04 |
| sbezverk_ | kfox1111: if it is not docker then what bridge canal would use to plug veth interfaces and flannel? | 17:04 |
| kfox1111 | this may be a calico issue instead of a flannel one. unfortunately I've never debugged calico yet. but do want to learn... :) | 17:05 |
| sbezverk_ | kfox1111: nope when I try to ping from container local's host flannel interface it fails | 17:05 |
| kfox1111 | what does a 'brctl show' show? | 17:05 |
| sbezverk_ | but in my setup it works | 17:05 |
| *** dwalsh has joined #openstack-kolla | 17:05 | |
| sbezverk_ | there is no linux bridge installed | 17:06 |
| kfox1111 | no linux bridges or no tools? | 17:06 |
| kfox1111 | cause it can be setting up the bridges without tools. | 17:06 |
| sbezverk_ | neither I think | 17:06 |
| sbezverk_ | will install | 17:07 |
| kfox1111 | lets double check. on both the working and non working systems. | 17:07 |
| kfox1111 | k. | 17:07 |
| sbezverk_ | kfox1111: on both working and not working see the same thing, one liner for docker0 | 17:08 |
| kfox1111 | and no members? | 17:08 |
| *** daneyon has joined #openstack-kolla | 17:08 | |
| sbezverk_ | it seems calico is not using bridging I remember it is pure L3 solutiob | 17:09 |
| sbezverk_ | nada | 17:09 |
| kfox1111 | thats what I thought. | 17:09 |
| kfox1111 | so the calico interfaces get traffic to the flannel.1 interface somehow else. | 17:09 |
| *** mkoderer has quit IRC | 17:09 | |
| kfox1111 | I dont think the calico interrfaces have ip's... | 17:09 |
| kfox1111 | http://logs.openstack.org/41/381041/40/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/22c3b34/logs/ip.txt | 17:10 |
| kfox1111 | hmm.. not ipv4 addressses anyway. | 17:10 |
| sbezverk_ | kfox1111: nope but I do see traffic coming from container | 17:10 |
| *** strigazi is now known as strigazi_AFK | 17:10 | |
| kfox1111 | http://docs.projectcalico.org/en/1.3.0/architecture.html | 17:11 |
| kfox1111 | it might mess with iptables a bunch? | 17:12 |
| kfox1111 | ah, here we go: http://logs.openstack.org/41/381041/40/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/22c3b34/logs/routes.txt | 17:12 |
| kfox1111 | the calico interfaces get routes on the host. | 17:13 |
| kfox1111 | do you see that? | 17:13 |
| *** daneyon has quit IRC | 17:13 | |
| sbezverk_ | default 169.254.1.1 0.0.0.0 UG 0 0 0 eth0 | 17:14 |
| sbezverk_ | 169.254.1.1 * 255.255.255.255 UH 0 0 0 eth0 | 17:14 |
| sbezverk_ | same thing on both sides, working and not working | 17:15 |
| kfox1111 | thats it? | 17:15 |
| kfox1111 | and canal on both? | 17:15 |
| sbezverk_ | this is from the container perspective | 17:16 |
| kfox1111 | oh. sry. do the route on the host. | 17:16 |
| sbezverk_ | yes they are identical | 17:16 |
| kfox1111 | should see calico interfaces and flannel | 17:17 |
| sbezverk_ | http://paste.openstack.org/show/584512/ | 17:17 |
| kfox1111 | looks like enp2s0f1 already has a 10 space address. | 17:19 |
| kfox1111 | looks non overlapping.... | 17:19 |
| sbezverk_ | right I calculated all subnet masks :-) | 17:20 |
| rhallisey | qwang, hey | 17:20 |
| kfox1111 | and on the host you can ping flannel.1 but not the ip of the container? | 17:20 |
| qwang | rhallisey: hi Ryan | 17:21 |
| *** hrito has quit IRC | 17:21 | |
| kfox1111 | hmm.... is that eth0 route from above from the container on the non working host? it seems like it only has a link local address? | 17:21 |
| rhallisey | qwang, what questions did you have? | 17:21 |
| kfox1111 | that seems maybe wrong. | 17:21 |
| *** unicell has joined #openstack-kolla | 17:21 | |
| sbezverk_ | eth0 side is working | 17:22 |
| qwang | rhallisey: I am reading your patch set about ansible support for a few openstack services: mariadb/rabbimq/glance/keystone | 17:23 |
| *** mgoddard has quit IRC | 17:23 | |
| qwang | rhallisey: and i am trying to help with it. | 17:23 |
| kfox1111 | sbezverk_: but it doesn't have the ip the route lists? | 17:23 |
| rhallisey | qwang, excellent | 17:23 |
| rhallisey | we can divide up some of the other services | 17:23 |
| qwang | rhallisey: I don't know how you do the tests, tho | 17:24 |
| openstackgerrit | Martin Matyas proposed openstack/kolla: Patch for magnum minion register issue https://review.openstack.org/382579 | 17:24 |
| rhallisey | qwang, ok. I'll write a doc as part of that patch chain that will describe it | 17:24 |
| sbezverk_ | kfox1111: are you refering to paste here above or paste.openstack.org | 17:24 |
| sbezverk_ | ? | 17:24 |
| kfox1111 | sbezverk_: the one in the irc logs here. | 17:25 |
| *** mark-casey1 has joined #openstack-kolla | 17:25 | |
| qwang | rhallisey: great. thank you so much | 17:25 |
| MarMat | sdake something like this? https://review.openstack.org/#/c/382579/ | 17:25 |
| sbezverk_ | kfox1111: there are identical on both sides working and not working | 17:25 |
| *** senk_ has joined #openstack-kolla | 17:25 | |
| kfox1111 | rhallisey: have a look at: https://review.openstack.org/#/c/380868/ it will cause the ansible workflow to need to change a little big | 17:25 |
| kfox1111 | bit | 17:25 |
| kfox1111 | sbezverk_: yeah, but I'd expect different ip addresses, so near identicle? I mean 'ip a', not 'route' | 17:26 |
| sbezverk_ | k | 17:26 |
| rhallisey | kfox1111, so difference names for the bootstrap jobs | 17:27 |
| rhallisey | gotcha | 17:27 |
| kfox1111 | rhallisey: yeah. its split out, and there's a new ordering constraint to work around ansible doing too much in one call. | 17:27 |
| rhallisey | and ordering constraint | 17:28 |
| sdake | marmat what about binary distros? | 17:28 |
| sbezverk_ | kfox1111: http://paste.openstack.org/show/584518/ | 17:28 |
| MarMat | sdake hm... | 17:28 |
| kfox1111 | there's a race in the ansible jobs. it does 2 things in one cli call. 'a, look up service_id. if it doesn't exist, create service_d. b, create endpoint associated with service_id' | 17:29 |
| sdake | MarMat the work looks good - but arey ou saying we should say binary distros dont work for magnum? | 17:29 |
| *** mark-casey1 has quit IRC | 17:29 | |
| *** DanyC has joined #openstack-kolla | 17:29 | |
| kfox1111 | if you launch multiple of them at the same time, the a part can race, and cuase wo different service_id's to be created for the same service. causing later issues. | 17:29 |
| sbezverk_ | kfox1111: btw that ps 380868, deployment logs even though shows success, does not really deploy anyhitng | 17:29 |
| sdake | MarMat lets go to #openstack-containers again plz | 17:30 |
| sbezverk_ | kfox1111: so before I workflow it I will have to test it, unless Ryan can test it earlier | 17:30 |
| MarMat | sdake any example how are the binary packages patched usually? | 17:30 |
| MarMat | sdake I'm there already, no reaction on my question | 17:30 |
| kfox1111 | sbezverk_: that looks good. | 17:30 |
| rhallisey | kfox1111, ok cool. Ya we need everything to be fully granular | 17:30 |
| kfox1111 | sbezverk_: it tests things only in the ceph case. the iscsi stuff doesn't work until there's a release of kolla. | 17:31 |
| kfox1111 | sbezverk_: so long as it gets past the part where it deployes the endoints, and checks to see if they are all properly created, then it works. | 17:31 |
| *** shardy is now known as shardy_afk | 17:32 | |
| kfox1111 | rhallisey: longer term, I think we want to break that up into the two different pieces, and at that point, we can just use the openstack cli rather then ansible. will make people hapier anyway. | 17:32 |
| sbezverk_ | kfox1111: totaly agree but I could not find any occurences of success, please point me out | 17:32 |
| kfox1111 | sbezverk_: sec | 17:32 |
| kfox1111 | http://logs.openstack.org/68/380868/32/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/67e577b/console.html | 17:33 |
| kfox1111 | ceph is still not working in the gate, so it got all the way up to where it tried to upload the glance image to ceph and hang. | 17:33 |
| *** DanyC has quit IRC | 17:34 | |
| kfox1111 | but it got to the point where all services were running, and passed the endpoint test here: | 17:34 |
| kfox1111 | http://logs.openstack.org/68/380868/32/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/67e577b/console.html#_2016-10-05_16_58_56_280024 | 17:34 |
| kfox1111 | all endpoints have 3 entries. | 17:34 |
| kfox1111 | it was failing that check before the gate test. | 17:35 |
| kfox1111 | it was failing that check before the job split/order. | 17:35 |
| rhallisey | qwang, to test the existing ansible patches use: ansible-playbook -i workflows/ansible/inventory/all-in-one workflows/ansible/site.yml -e @/etc/kolla-kubernetes/kolla-kubernetes.yml | 17:35 |
| rhallisey | qwang, that should be able to get you started | 17:35 |
| qwang | rhallisey: thank you. will test with it. I'll start with the horizon role | 17:36 |
| *** huikang has joined #openstack-kolla | 17:36 | |
| kfox1111 | sdake: http://logs.openstack.org/41/381041/41/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/4ef0df5/logs/dmesg not much interesting. :/ | 17:37 |
| rhallisey | qwang, awesome :) | 17:37 |
| sbezverk_ | kfox1111: ok | 17:38 |
| sdake | kfox1111 try without selinux | 17:38 |
| sdake | kfox1111 and try with btrfs | 17:39 |
| kfox1111 | sdake: selinux is disabled. | 17:39 |
| kfox1111 | the setup script disables it. | 17:39 |
| kfox1111 | so its enabled for a bit, then turned off. | 17:39 |
| sdake | kfox1111 permissive or disabled? | 17:39 |
| kfox1111 | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L134 | 17:40 |
| sdake | kfox1111 what about btrfs? | 17:40 |
| kfox1111 | not sure why it would matter? ceph is using the loopback device. | 17:41 |
| kfox1111 | hmm... | 17:41 |
| sdake | kfox1111 i mean for the graph driver | 17:41 |
| sdake | there is an ioctl error in there | 17:41 |
| sdake | in devicemapper | 17:41 |
| kfox1111 | well, I ugess its using aufs on ubuntu and loopback on centos. | 17:41 |
| *** bjolo_ has joined #openstack-kolla | 17:41 | |
| kfox1111 | and both fail the same way | 17:41 |
| sdake | dm is a pile of groan | 17:41 |
| *** MagnumBonum has quit IRC | 17:42 | |
| *** haplo37 has quit IRC | 17:43 | |
| kfox1111 | hre's one for an ubuntu box: http://logs.openstack.org/41/381041/41/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/997569e/logs/dmesg | 17:44 |
| *** haplo37 has joined #openstack-kolla | 17:44 | |
| *** athomas has quit IRC | 17:45 | |
| kfox1111 | sdake: here's a slightly different one, where ceph seems to be quite happy: http://logs.openstack.org/68/380868/32/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/a2bcfb0/logs/ | 17:47 |
| kfox1111 | don't see these too often. | 17:48 |
| kfox1111 | unfortunatly, its not in the ceph branch, so not much ceph logging. :/ | 17:48 |
| kfox1111 | but you might be able to spot some node differences. | 17:48 |
| *** dwalsh has quit IRC | 17:54 | |
| *** tonanhngo has quit IRC | 17:57 | |
| *** tonanhngo has joined #openstack-kolla | 18:01 | |
| *** SamYaple_ has joined #openstack-kolla | 18:04 | |
| *** SamYaple_ has quit IRC | 18:04 | |
| *** SamYaple has quit IRC | 18:04 | |
| *** SamYaple has joined #openstack-kolla | 18:04 | |
| kfox1111 | SamYaple: alive? | 18:05 |
| SamYaple | kfox1111: always | 18:05 |
| SamYaple | unless this has to do with ceph | 18:05 |
| kfox1111 | rumber has it you .... | 18:05 |
| kfox1111 | hehe :) | 18:05 |
| kfox1111 | saw me coming. :) | 18:05 |
| *** berendt has quit IRC | 18:05 | |
| SamYaple | heh. is this ceph in the gate question? | 18:05 |
| kfox1111 | ceph/gate=bad, ceph/anywhere else, good. | 18:06 |
| kfox1111 | totally weirded out by this. | 18:06 |
| *** mark-casey1 has joined #openstack-kolla | 18:06 | |
| SamYaple | nah its not that. its a space issue *mostly* | 18:06 |
| SamYaple | so there is limited and varying amount of space throughout all the gates | 18:07 |
| kfox1111 | runs with the same params in minikube. :/ | 18:07 |
| kfox1111 | I near doubled the space in the gate just to make sure. and validated there is pleanty of space on the gate vm's. | 18:07 |
| SamYaple | kfox1111: oh i doubt it! | 18:07 |
| SamYaple | you forget, there are like 8 different cloud providers | 18:08 |
| *** huikang has quit IRC | 18:08 | |
| SamYaple | 1 of them has like 20GB root disk or something | 18:08 |
| SamYaple | i forget the exact details to be honest | 18:08 |
| SamYaple | and its possible it changed | 18:08 |
| *** huikang has joined #openstack-kolla | 18:08 | |
| kfox1111 | I've set up logging up the wazoo and verify on failure that df shows pleanty of free space. | 18:08 |
| SamYaple | but it was not possible to run on all the gates with enough space for ceph | 18:08 |
| kfox1111 | its not out of free space. | 18:08 |
| SamYaple | oh wait, lets take a step back | 18:08 |
| kfox1111 | I'm doing it on minikube with 3gigs of loopback for ceph. | 18:09 |
| SamYaple | you are saying ceph doesnt _work_ at all? | 18:09 |
| SamYaple | i could have implemetned ceph, but i didnt have space too | 18:09 |
| kfox1111 | The issue I see is, | 18:09 |
| SamYaple | are you saying you did implemnet and its not working? | 18:09 |
| kfox1111 | most of the time, ceph gets stuck allocating the pg's, for example: | 18:09 |
| kfox1111 | http://logs.openstack.org/41/381041/41/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/ddb4997/console.html | 18:09 |
| kfox1111 | after 4 min there, it still hadn't allocated any pg's. | 18:10 |
| *** duonghq has quit IRC | 18:10 | |
| kfox1111 | but the same test occationally works, and the pg's schedule. | 18:10 |
| kfox1111 | and then all is well with ceph from then on out. | 18:10 |
| SamYaple | im sure you just didnt update the crush map ruleset, right? | 18:10 |
| kfox1111 | I didn't. but did tweak the config file a bit. right before the rbd create there, you can see the ceph.conf used. | 18:11 |
| *** Satya_ has joined #openstack-kolla | 18:11 | |
| Satya_ | Hi Everyone | 18:11 |
| kfox1111 | i did tweak the pool sizes and crush leaf type. | 18:11 |
| *** LamT__ has quit IRC | 18:11 | |
| Satya_ | Want to check if anyone configured the designate with kolla | 18:11 |
| Satya_ | i saw the config and templates are there already with kolla | 18:12 |
| kfox1111 | with minikube with the same setup, I can laucnh vm's with glance images in ceph, and cinder volumes out of ceph attached. | 18:12 |
| *** huikang has quit IRC | 18:13 | |
| kfox1111 | somethings weird about the gate vm's though that differ from minikube in some way. (a lot of ways, but I've thrown out a lot of them. not iptables, not multiple nics, not out of space, not distro) | 18:13 |
| kfox1111 | the logs from ceph look basically identicle between working and nonworking. | 18:13 |
| kfox1111 | the pg's just don't schedule. | 18:13 |
| SamYaple | kfox1111: just looked at the logs, so based on the proceedure of what you did, you need to restart your ceph-osd container after you create the kollavolumes pool and deleted the rbd pool | 18:15 |
| SamYaple | sometimes teh pgs get stuck when dropping to 1 replica | 18:16 |
| SamYaple | *technically* you can do all this stuff before creating the first ceph-osd in teh ceph-mon container and then you wont have this issue | 18:16 |
| kfox1111 | oh, really? interesting... | 18:16 |
| kfox1111 | ok. let me reorder the script. :) | 18:16 |
| SamYaple | this is really only a problem because of the single osd | 18:17 |
| SamYaple | you might be better off keeping the default of 3 (or dropping it to 2) and changing the rush map so its ok with single host | 18:17 |
| SamYaple | so two osds on a single host satifies | 18:17 |
| kfox1111 | yeah. it was just more work doing 2 then 1. (I thought) | 18:18 |
| SamYaple | you arent wrong. but ceph does get stuck like this when you drop below 2 replicas | 18:19 |
| SamYaple | restarting the osd is normally enough to fix it | 18:19 |
| SamYaple | the behaviour is pretty undefined though, consider it a Wishlist bug for ceph | 18:19 |
| kfox1111 | yeah. probably not well tested... | 18:20 |
| *** salv-orlando has quit IRC | 18:22 | |
| kfox1111 | hmm.... gota redo a little bit of logic to get ceph-admin going... if osd is later... | 18:22 |
| SamYaple | i would personally jsut restart the ceph-osd | 18:23 |
| SamYaple | seems easiest | 18:23 |
| SamYaple | you guys need to have a ceph health check parser anyway | 18:23 |
| SamYaple | for upgrades | 18:23 |
| kfox1111 | k. will try that. | 18:23 |
| kfox1111 | could just kubectl exec -it ceph-osd 'killall ceph-osd' i guess. | 18:24 |
| SamYaple | ok well im going back to Rust! let me know if you have other questions i might be able to help with. because that kube thing is beyond me | 18:24 |
| kfox1111 | thanks for the help. :) | 18:24 |
| *** david-lyle has quit IRC | 18:25 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 18:25 |
| *** tonanhngo has quit IRC | 18:27 | |
| *** DanyC has joined #openstack-kolla | 18:28 | |
| Satya_ | @SamYaple just want to check the deployment of designate with kolla | 18:31 |
| *** senk_ has quit IRC | 18:32 | |
| rhallisey | SamYaple, rust :) | 18:32 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 18:32 |
| *** DanyC has quit IRC | 18:33 | |
| SamYaple | Satya_: i dont think its up to date for mitaka | 18:38 |
| Satya_ | i am looking at the master branch | 18:39 |
| SamYaple | Satya_: but someone will have to answer that for sure. there was major changes in Mitaka that im 95% sure never made it (or were worked on since) | 18:39 |
| SamYaple | there are even more changes in Newton | 18:39 |
| SamYaple | i would be shocked if designate is up to speed in Kolla | 18:39 |
| SamYaple | it has radically changed in the way it is setup | 18:40 |
| SamYaple | rhallisey: ive been using rust! i like it | 18:40 |
| Satya_ | i want to deploy that but not sure what changes need to go to globals and multinode | 18:40 |
| SamYaple | Satya_: designate has no ansible playbooks in the master branch of the kolla repo | 18:42 |
| SamYaple | Satya_: so its certianly not there | 18:42 |
| Satya_ | i saw something here "https://github.com/openstack/kolla/tree/master/docker/designate" | 18:43 |
| SamYaple | Satya_: thats an image, not the deployment of said image | 18:43 |
| Satya_ | ok | 18:43 |
| SamYaple | its confusing because currently Kolla has images in the same repo as deployment, but my understanding is that is changing? | 18:43 |
| Satya_ | ansible playbook is not yet created? | 18:43 |
| SamYaple | ansible playbook was removed because it wasnt maintained IIRC | 18:44 |
| rhallisey | SamYaple, neat :). I'll have to explore :) | 18:44 |
| kfox1111 | SamYaple: yeah. going to break out the ansible stuff to its own repo. | 18:44 |
| rhallisey | SamYaple, yes that is changing | 18:44 |
| kfox1111 | a lot of folks are using the kolla containers. | 18:44 |
| kfox1111 | some folks are scared off by ansible. | 18:44 |
| SamYaple | rhallisey: im using it to interface with my waterrower! ive built a small app that I will eventually be using to in-real-time inform me if my launch-to-recovery time ratio is good | 18:45 |
| SamYaple | rhallisey: https://libraries.io/cargo/waterrower | 18:45 |
| SamYaple | i dont know which dev community is worse, Ansible or Docker | 18:45 |
| SamYaple | probably Docker, but Ansible gives them a run for thier money! | 18:46 |
| *** HyperJohnGraham has joined #openstack-kolla | 18:47 | |
| kfox1111 | arg.. the kill didn't seem to work... | 18:47 |
| kfox1111 | big guns time... | 18:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 18:49 |
| rhallisey | SamYaple, oh nice! | 18:49 |
| rhallisey | SamYaple, on the rower there is a display that graphs you drive | 18:50 |
| rhallisey | could also be a useful indicator | 18:50 |
| SamYaple | the rower displays distance, stroke rate, (heartrate if attached), and time i believe | 18:50 |
| SamYaple | no graph | 18:50 |
| SamYaple | at least on mine | 18:50 |
| rhallisey | SamYaple, that's right you don't have a concept2 | 18:51 |
| rhallisey | generally those have it | 18:51 |
| SamYaple | i have a waterrower, with s4 monitor | 18:51 |
| SamYaple | but thats why im writting this! its been a fun project for the past few days | 18:51 |
| rhallisey | nice :) | 18:51 |
| rhallisey | will you be in Barcelona? | 18:52 |
| SamYaple | i will not be. im not sure how many summits I will be going to in the future. ive really settled into this working form home thing | 18:52 |
| rhallisey | gotcha | 18:52 |
| *** DanyC has joined #openstack-kolla | 18:53 | |
| SamYaple | who am i kidding? im just going to rewrite openstack in rust | 18:53 |
| SamYaple | without all the openstack pieces | 18:53 |
| rhallisey | :) | 18:53 |
| *** haplo37_ has quit IRC | 18:53 | |
| *** DanyC has quit IRC | 18:53 | |
| rhallisey | perfect | 18:53 |
| *** DanyC has joined #openstack-kolla | 18:54 | |
| kfox1111 | I'd settle for just nova. ;) | 18:54 |
| SamYaple | kfox1111: yesh. i know right? | 18:55 |
| kfox1111 | I want someone to take a stab at implementing a nova api wrapper that just launches k8s pods with qemu in them to launch the vm's. | 18:55 |
| *** haplo37_ has joined #openstack-kolla | 18:56 | |
| kfox1111 | I think all the api I can think of has coresponding stuff in k8s that it can be mapped to. | 18:57 |
| kfox1111 | flavors/az/hostaggregates = nodelabels/podlables. it already has a scheduler and restapi. | 18:58 |
| kfox1111 | there would be a 1/1 mapping between vm and k8s pod/container. | 18:58 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 18:58 |
| openstackgerrit | Qin Wang (qwang) proposed openstack/kolla-kubernetes: Add ansible workflow for Horizon https://review.openstack.org/382620 | 18:58 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 19:00 |
| kfox1111 | SamYaple: same error. | 19:02 |
| *** matrohon has joined #openstack-kolla | 19:02 | |
| kfox1111 | I flat out killed the osd container and rebuilt it. no joy. :/ | 19:02 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 19:05 |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Ansible workflow for Glance https://review.openstack.org/380801 | 19:05 |
| *** lamt has quit IRC | 19:08 | |
| *** tonanhngo has joined #openstack-kolla | 19:09 | |
| SamYaple | kfox1111: im not sure what rebuilt means here, it should just be restarted | 19:12 |
| kfox1111 | I deleted the container and k8s rebuilt it. all hte data's kept though. | 19:13 |
| kfox1111 | I'm redoing everything to launch a second osd now. :/ | 19:14 |
| SamYaple | you could also try not deleting rbd and simply changing the size of it to 1 copy | 19:15 |
| kfox1111 | didn't help. tried that already. :/ | 19:15 |
| SamYaple | there is a combination of little things like that which will work for sure | 19:15 |
| SamYaple | its a bit annoying, but ceph is designed to eb distributed and redundant | 19:15 |
| kfox1111 | yeah. thats why I just gave up and am adding a second. | 19:15 |
| SamYaple | removing both of those like you are doing is bad :P | 19:15 |
| *** HyperJohnGraham has quit IRC | 19:16 | |
| *** DanyC_ has joined #openstack-kolla | 19:22 | |
| *** david-lyle has joined #openstack-kolla | 19:23 | |
| *** DanyC has quit IRC | 19:23 | |
| *** lrensing has quit IRC | 19:24 | |
| kfox1111 | oh... I see one thing off in the template... hostname is set to minikube... | 19:24 |
| kfox1111 | I wonder if that will make a difference... shouldn't... but still... | 19:24 |
| *** DanyC_ has quit IRC | 19:24 | |
| sdake | MarMat for some reason i thought we were in this channel | 19:25 |
| MarMat | sdake back from other room, it's https://bugs.launchpad.net/kolla/+bug/1630248 I need to double check, the fact is that I did not test it in reconfiigure scenario | 19:25 |
| openstack | Launchpad bug 1630248 in kolla "magnum genconfig fails" [Undecided,New] | 19:25 |
| kfox1111 | oh. nm... just on my machine.... | 19:26 |
| kfox1111 | darn. | 19:26 |
| sdake | MarMat been triaged | 19:28 |
| *** DanyC has joined #openstack-kolla | 19:29 | |
| *** dwalsh has joined #openstack-kolla | 19:31 | |
| *** bjolo_ has quit IRC | 19:33 | |
| *** DanyC has quit IRC | 19:34 | |
| *** HyperJohnGraham has joined #openstack-kolla | 19:34 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 19:35 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:35 |
| openstackgerrit | Mauricio Lima proposed openstack/kolla: [WIP] Add HNAS as Manila backend https://review.openstack.org/382054 | 19:42 |
| *** sdake has quit IRC | 19:45 | |
| *** sdake has joined #openstack-kolla | 19:46 | |
| sdake | rbergeron hecho en mexico coca cola has arrived - 4 24 packs straight to my door at store pricing | 19:48 |
| kfox1111 | I wonder if this is the orcestrator deleting the bootstrap pods before they are done... | 19:49 |
| kfox1111 | no other service does it that way, as the kolla bootstrap scripts are a little sketchy in the ceph containers... | 19:50 |
| *** daneyon has joined #openstack-kolla | 19:51 | |
| kfox1111 | hmm... first try worked... so either lucky, 2 osd's helped, or it was the extra sleep time on bootstrap... | 19:51 |
| *** HyperJohnGraham has quit IRC | 19:52 | |
| kfox1111 | 2 in a row... | 19:52 |
| kfox1111 | so either extremely lucky, 2 osd's helped or it was the extra sleep time on bootstrap. :) | 19:52 |
| kfox1111 | 3 in a row! :) | 19:53 |
| *** salv-orlando has joined #openstack-kolla | 19:53 | |
| kfox1111 | I don't buy I'm that lucky. not with the week I've had so far. :) | 19:53 |
| kfox1111 | this is a good sign. :) | 19:54 |
| *** salv-orl_ has joined #openstack-kolla | 19:54 | |
| *** daneyon has quit IRC | 19:55 | |
| *** HyperJohnGraham has joined #openstack-kolla | 19:57 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 19:58 |
| *** salv-orlando has quit IRC | 19:58 | |
| *** dwalsh has quit IRC | 19:59 | |
| *** jtriley has joined #openstack-kolla | 20:06 | |
| *** schwicht has quit IRC | 20:07 | |
| *** matrohon has quit IRC | 20:11 | |
| kfox1111 | consistently healty ceph now. :) | 20:14 |
| *** mark-casey1 has left #openstack-kolla | 20:17 | |
| *** Pavo has joined #openstack-kolla | 20:17 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 20:21 |
| *** tonanhngo has quit IRC | 20:24 | |
| *** schwicht has joined #openstack-kolla | 20:24 | |
| openstackgerrit | Ryan Hallisey proposed openstack/kolla-kubernetes: Workflow Gate Testing https://review.openstack.org/382021 | 20:24 |
| sdake | kfox1111 what fixed teh gatefor you? | 20:24 |
| *** Pavo has quit IRC | 20:27 | |
| *** Pavo has joined #openstack-kolla | 20:27 | |
| kfox1111 | sdake: one of two things... I added a second osd, and I found a potential case where the bootstrap doesn't finish before the container is killed. I added a 10 second sleep. | 20:29 |
| kfox1111 | haven't narrowed it down to which of the two it is. | 20:29 |
| kfox1111 | I suspect the second case is the real cause. | 20:29 |
| sdake | why wuld the bootstrap container get killed? | 20:29 |
| kfox1111 | as it works with one osd with minikube, but its manually orchestrated, so slower to delete the bootstrap. | 20:30 |
| kfox1111 | so... | 20:30 |
| kfox1111 | heres the deal..... :) | 20:30 |
| sdake | bootstrap doesn't exit automatically? | 20:30 |
| kfox1111 | the bootstrap script in the docker container does some... interesting things. | 20:31 |
| sdake | i know i wrote alot of it | 20:31 |
| *** HyperJohnGraham has quit IRC | 20:31 | |
| sdake | i understand it may not be ottally compatibl with k8s atm | 20:31 |
| kfox1111 | it allocates a ceph id, then formats the disks, dadada da da, then registers it all in the ceph and alls well. | 20:31 |
| kfox1111 | the problem was, when I mapped it to a kubernetes job, if it failed after the first step, | 20:32 |
| sdake | oh that part | 20:32 |
| kfox1111 | kubernetes restarts the job, and it spins creating random ceph id's. | 20:32 |
| kfox1111 | :/ | 20:32 |
| kfox1111 | so I didn't do it as a job, but a pod with restart policy none. | 20:32 |
| *** schwicht has quit IRC | 20:32 | |
| kfox1111 | but my orchestration script doesn't really handle a pod entering ready state, but isn't done running. as its kind of half way between a job and a pod. | 20:33 |
| MarMat | Jeffrey4l_ ping | 20:34 |
| kfox1111 | I just need to make a different check for those pods. | 20:34 |
| sdake | kfox1111 cool sounds good | 20:35 |
| sdake | kfox1111 sleep 10s not ideal :) | 20:35 |
| kfox1111 | +1. | 20:35 |
| kfox1111 | horibly hackish. but identified a potential problem. | 20:36 |
| kfox1111 | if I pull the multiost thing and it still works, then its the sleep for sure. | 20:36 |
| *** schwicht has joined #openstack-kolla | 20:36 | |
| kfox1111 | and that should be easy to get rid of. | 20:36 |
| kfox1111 | so weird... | 20:40 |
| kfox1111 | somehow docker exec and kubectl exec are very different when it comes to /sys read/write... | 20:40 |
| *** jtriley has quit IRC | 20:40 | |
| sdake | just guessing, kubectl exec would be slower | 20:41 |
| sdake | perhaps far far slower | 20:41 |
| kfox1111 | rbd map works when running via kubectl exec but not via docker exec into the same type of container. | 20:42 |
| sdake | hmm | 20:43 |
| sdake | makes sense if the speed thing is accurate | 20:43 |
| sdake | docker exec is faster | 20:43 |
| sdake | docker exec may not be synchronized in our scripts where it is needed | 20:43 |
| sdake | typically we rely on ansible for that as well | 20:44 |
| kfox1111 | thits a | 20:48 |
| kfox1111 | the weird thing is, | 20:49 |
| kfox1111 | the error is "/sys is read only" on the docker exec case, but read write on the kubectl exec case. | 20:49 |
| kfox1111 | I think thats the only thing stopping this idea from working. | 20:49 |
| sdake | why not retry the docker exec casse | 20:50 |
| kfox1111 | what do you mean? | 20:50 |
| sdake | i think what is happening is your docker execing before the container is setup | 20:50 |
| sdake | so delay the docker exec | 20:50 |
| kfox1111 | it can.t let me show you... | 20:50 |
| kfox1111 | https://review.openstack.org/#/c/381041/46/services/ceph/ceph-rbd-pod.yml.j2 | 20:50 |
| sdake | how about a line # :) | 20:51 |
| kfox1111 | the container comes up, writes out a /usr/bin/rbd shell script on the host, then sleeps. | 20:51 |
| kfox1111 | kubelet on the host then calls /usr/bin/rbd whenever it needs to mount a ceph rbd volume. | 20:51 |
| kfox1111 | the script should docker exec back into the same container and run the rbd command there. | 20:52 |
| kfox1111 | which it does, but its erroring because the /sys volume is read only. | 20:52 |
| sdake | in the docker case? | 20:52 |
| kfox1111 | yeah. | 20:52 |
| *** schwicht has quit IRC | 20:52 | |
| kfox1111 | but, in the setup_gate.sh script, | 20:52 |
| sdake | how fast after container startup until the ceph rbd volume is created? | 20:53 |
| kfox1111 | I kubectl exec into the ceph-admin script, rbd map volume, format it, and unmount it just fine. | 20:53 |
| sdake | or mounted | 20:53 |
| kfox1111 | minutes. and kubelet retries a few times with a minute or two between. | 20:53 |
| kfox1111 | so the container has to be there and ready even to write out the script, and its calling the script ok, so its got to be running. | 20:54 |
| sdake | how is /sys accessed? | 20:54 |
| kfox1111 | by rbd. | 20:54 |
| sdake | which file in partcilar | 20:55 |
| *** lamt has joined #openstack-kolla | 20:55 | |
| sdake | did you try docker exec --privileged? | 20:55 |
| kfox1111 | that and -u 0 just to be extra sure. | 20:56 |
| sdake | ok well permissions may be not working sa expected | 20:56 |
| kfox1111 | I tried with a docker run -i --rm -v /sys:/sys -v /dev:/dev <image> rbd ... too. same issue. | 20:56 |
| sdake | can you do that with a df? | 20:57 |
| sdake | would love to see how the filesystems are mounted | 20:57 |
| kfox1111 | yeah. let me add that. | 20:57 |
| *** huikang has joined #openstack-kolla | 20:57 | |
| sdake | kfox1111 and mnttab\ | 20:58 |
| sdake | iirc its in /etc dir | 20:58 |
| sdake | just cat it :) | 20:59 |
| kfox1111 | k | 20:59 |
| sdake | i dont have accesss to my env atm or i'd tell yu for sure wehre it is | 20:59 |
| kfox1111 | no mnttab... | 20:59 |
| kfox1111 | /proc/mounts? | 21:00 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:01 |
| sdake | kfox1111 that file would rock thanks | 21:02 |
| sdake | kfox1111 sorry was stuffing some food in face | 21:02 |
| sdake | need to eat more often | 21:03 |
| sdake | i've lost 15 pounds in the last 6 months | 21:03 |
| sdake | happens a bit in the summer here | 21:03 |
| sdake | 5-8 pounds = ok | 21:03 |
| sdake | more then that not good - need more minerals :) | 21:03 |
| kfox1111 | np. | 21:10 |
| kfox1111 | heh. I must be a geek when starcraft comes to mind when you say that. :) | 21:11 |
| *** ayoung has quit IRC | 21:12 | |
| rhallisey | kfox1111, http://logs.openstack.org/21/382021/10/experimental/gate-kolla-kubernetes-ansible-workflow-ceph-nv/6f541a9/console.html | 21:12 |
| rhallisey | progressing a bit | 21:12 |
| rhallisey | wow | 21:12 |
| rhallisey | mariadb made it through on it's last try | 21:13 |
| rhallisey | O.o | 21:13 |
| rhallisey | bbiab | 21:13 |
| *** rhallisey has quit IRC | 21:13 | |
| *** fguillot has quit IRC | 21:15 | |
| kfox1111 | sdake: http://logs.openstack.org/41/381041/47/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/7a4e15a/logs/pods/kolla-ceph-rbd-qf7b6-main.txt | 21:16 |
| sdake | kfox1111 sysfs is read/write | 21:17 |
| sdake | kfox1111 was that via docker exec? | 21:17 |
| kfox1111 | http://logs.openstack.org/41/381041/47/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/7a4e15a/logs/pods/kolla-mariadb-bootstrap-zntqq.txt | 21:17 |
| kfox1111 | maybe its a false error... there are a bit of other error messages there... maybe I should just dump the whole thing to a log file... | 21:18 |
| *** schwicht has joined #openstack-kolla | 21:18 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:20 |
| inc0 | ehh k8s doesn't have superb logs :/ | 21:22 |
| *** haplo37_ has quit IRC | 21:24 | |
| *** tonanhngo has joined #openstack-kolla | 21:24 | |
| kfox1111 | they kind of assume you buidl out a logging infrastructure of your choice. | 21:25 |
| kfox1111 | as that seems to be a deeply personal thing. :/ | 21:26 |
| *** haplo37_ has joined #openstack-kolla | 21:26 | |
| *** tonanhngo has quit IRC | 21:26 | |
| *** tonanhngo has joined #openstack-kolla | 21:26 | |
| kfox1111 | it is one of their unwritten rules, but seems to be followed, that they don't want to get into the middle of religious wars. | 21:26 |
| kfox1111 | they do one thing, container orchestration and leave the rest to be built on top. | 21:27 |
| kfox1111 | sdake: thats kind of weird.... | 21:35 |
| kfox1111 | http://logs.openstack.org/41/381041/48/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/7995d17/logs/rbd.log | 21:35 |
| kfox1111 | the commands kubelets issueing. | 21:35 |
| kfox1111 | I wouldn't have expected it to write the temp keyring out to etc... | 21:35 |
| kfox1111 | I'll have to mount that in. maybe thats the problem. | 21:35 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:36 |
| kfox1111 | sdake: know why this wont merge? https://review.openstack.org/#/c/380868/ | 21:43 |
| kfox1111 | sbezverk_: alive? | 21:43 |
| sdake | kfox1111 lacking a workflow? | 21:44 |
| kfox1111 | sdake: I'm thinking maybe it unworkflowed it when the experimental results came back? | 21:44 |
| kfox1111 | the last action on it was a workflow though. | 21:44 |
| kfox1111 | haven't seen a ps in this state before. | 21:45 |
| kfox1111 | should we juts reworkflow it? | 21:45 |
| sdake | kfox1111 i can merge it without reviewing it if you like | 21:46 |
| sdake | since its already been erviewed by others | 21:46 |
| *** b_bezak has quit IRC | 21:47 | |
| sdake | dont have time right at this moment to review it - but trust your udgement and if its broken we can fix it later ;) | 21:47 |
| kfox1111 | k. I think it just needs the workflow put back, as the last action in the list was a workflow. | 21:47 |
| kfox1111 | thanks. | 21:47 |
| kfox1111 | once that patch, and the ceph one I've got going, we should have a pretty close to solid gate I think. | 21:47 |
| kfox1111 | (don't want to jinks it by saying solid. :) | 21:48 |
| *** b_bezak_ has joined #openstack-kolla | 21:48 | |
| openstackgerrit | Merged openstack/kolla-kubernetes: Split endpoint jobs and start testing the deployments https://review.openstack.org/380868 | 21:51 |
| kfox1111 | OH... net=host.... arg | 21:51 |
| *** jheroux has quit IRC | 21:52 | |
| kfox1111 | shouldn't matter, but maybe... | 21:52 |
| *** b_bezak_ has quit IRC | 21:53 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 21:59 |
| *** lamt has quit IRC | 21:59 | |
| *** huikang has quit IRC | 22:00 | |
| *** huikang has joined #openstack-kolla | 22:01 | |
| SamYaple | kfox1111: ceph is going to require net=host btw | 22:04 |
| kfox1111 | yup. | 22:04 |
| *** lamt has joined #openstack-kolla | 22:04 | |
| SamYaple | its a hard requirement if you have multiple osds on the same host | 22:04 |
| kfox1111 | and pid namespace too. | 22:04 |
| kfox1111 | that one's a really subtile bug we ran into here. | 22:04 |
| kfox1111 | otherwise containers kill each others connections. | 22:04 |
| kfox1111 | if yo udon't have much traffic you might not notice it. | 22:04 |
| SamYaple | well they mark each other down | 22:04 |
| kfox1111 | yeah. | 22:05 |
| SamYaple | afaik i was the first one to get ceph running in docker with ipv4 where each osd was there own container | 22:05 |
| SamYaple | i filed a bug about this, but ceph closed it as a docker bug | 22:05 |
| SamYaple | its not | 22:05 |
| kfox1111 | nice. :) | 22:05 |
| *** HyperJohnGraham has joined #openstack-kolla | 22:05 | |
| kfox1111 | yeah. I'd say not too. | 22:05 |
| SamYaple | the ceph osds know they are on the same host and do healthchecks against pids | 22:05 |
| kfox1111 | right. | 22:06 |
| *** huikang has quit IRC | 22:06 | |
| kfox1111 | the clients have the same issue we've found. | 22:06 |
| SamYaple | if you did ipv6 this wouldn't be an issue fyi | 22:06 |
| kfox1111 | the client connects, has the same pid as in another container, and the osd's kill all the connectsions, as a new client came in. | 22:06 |
| kfox1111 | then the two containers fight with each other over. | 22:06 |
| kfox1111 | :) | 22:06 |
| kfox1111 | ipv6 is the answer to a lot of problems. | 22:07 |
| SamYaple | kfox1111: related http://tracker.ceph.com/issues/10763 https://github.com/ceph/ceph-docker/issues/19 | 22:07 |
| kfox1111 | its also still way to far out. :/ | 22:07 |
| * kfox1111 nods | 22:07 | |
| *** HyperJohnGraham has quit IRC | 22:11 | |
| *** HyperJohnGraham has joined #openstack-kolla | 22:14 | |
| kfox1111 | wha... | 22:16 |
| kfox1111 | oh... $@ includes $0? | 22:16 |
| kfox1111 | that may be the problem.... | 22:16 |
| bmace | for bug https://bugs.launchpad.net/kolla/+bug/1629024 feel free to nuke it. i'm going to abandon my change in this area and handle things in a different way. | 22:21 |
| openstack | Launchpad bug 1629024 in kolla "Destroy needs to have the option to be slightly less destructive" [Undecided,In progress] - Assigned to Borne Mace (borne-mace) | 22:21 |
| kfox1111 | hmm... no. seems to not. everything's passed through docker exec proper. | 22:22 |
| kfox1111 | bash test.sh foo bar baz "ark barm" baz | 22:23 |
| kfox1111 | foo|bar|baz|ark barm|baz | 22:23 |
| sdake | bmace ok let me have a look | 22:24 |
| sdake | bmace oh right this work | 22:24 |
| sdake | bmace so - curious how you are going to address it | 22:25 |
| sdake | bmace but i guess we can find that out later | 22:25 |
| bmace | some of the changes i am just going to make internally. especially the nuke of the files under /etc/kolla. for some i am also going to probably add a simple stop playbook. | 22:25 |
| sdake | bmace yup makes sense | 22:26 |
| bmace | i appreciate the feeling that the containers are just cattle so the systems can be wiped aggressively, but for our environment we have the need at times to not nuke everything under /etc/kolla for example, since our kollacli config is under there. | 22:26 |
| sdake | bmace oh ya that makes sense | 22:26 |
| sdake | bmace our destroy atm is not ideal | 22:27 |
| sdake | i wish it destroyed only things relevant to kolla | 22:27 |
| sdake | not other htings the operator adds | 22:27 |
| bmace | well, in most cases /etc/kolla should be pretty safe.. we just happen to put some of our own stuff in there :( | 22:27 |
| sdake | an implementation of such a thing is difficult | 22:27 |
| sdake | bmace your not alone | 22:27 |
| *** Pavo has quit IRC | 22:27 | |
| sdake | we have reverse filters for certain items | 22:27 |
| sdake | may be able to make use of that | 22:27 |
| *** Pavo has joined #openstack-kolla | 22:28 | |
| sdake | that would probably work with less work then maintaining some variance internally :) | 22:28 |
| sdake | e.g. globals.yml and passwords.yml and config are filtered from removal | 22:29 |
| sdake | could probably be configurable | 22:29 |
| bmace | yup. for our internal stuff i can probably just add kollacli into that list. | 22:29 |
| sdake | yup | 22:29 |
| bmace | true, but configurable how? feels odd to pass in as an extra arg or env_var | 22:29 |
| sdake | the stop playbook | 22:29 |
| kfox1111 | oh... keyring's the default.... maybe its not fetching the secret right.... | 22:29 |
| sdake | what is that for | 22:29 |
| sdake | bmace ya - i know configuration how here is hard | 22:30 |
| sdake | bmace there is a kolla-build.conf | 22:30 |
| sdake | or globals.yml would work | 22:30 |
| sdake | but its kind of clunky | 22:30 |
| bmace | sdake mostly i think small environment / AIO type stuff on stop.. you may want to bring down your services but not actually need to nuke all your containers. also if you want to mess with some container contents, whatever.. | 22:31 |
| sdake | bmace that sounds good for master to me | 22:31 |
| bmace | also, internally i think we will still support not deleting the images as an option to the destroy playbooks, because not everyone is on a great internet connection or has local caches registries | 22:32 |
| sdake | (the part that sounds good for master is a stop action) | 22:32 |
| bmace | sdale just sort of simple stuff that maybe not everyone wants but some people do | 22:32 |
| bmace | sdake got it | 22:32 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:34 |
| sdake | bmace i thin kthe not deleting containers as an option would make a ifne config value | 22:36 |
| *** Satya_ has quit IRC | 22:38 | |
| *** awiddersheim has joined #openstack-kolla | 22:39 | |
| kfox1111 | yeah... the code path changes if its a secret or a keyfile. and the command line indicates its using a keyfile. | 22:39 |
| kfox1111 | thats the issue. | 22:39 |
| sdake | bmace no real reason to do these thigns internally unless externally just doesn't want it :) | 22:41 |
| sdake | bmace i like the stop idea - i have wanted this for a long time | 22:41 |
| sdake | bmace but been too busy to do it on my own :) | 22:41 |
| bmace | sdake happy to toss it upstream when it is ready, which shouldn't take very long and agreed that we like to keep as little internal as possible. better for everyone. | 22:42 |
| sdake | bmace sweet - now with that said master is blocked for about 1 more week :) | 22:43 |
| sdake | inc0 ping | 22:43 |
| bmace | sdake yeah, no sweat, i don't mind it lingering up there and feedback is great and if there is some stuff you guys don't want that is fine too :) | 22:44 |
| *** lamt has quit IRC | 22:44 | |
| kfox1111 | oh... wait.. wat is storage_ceph.key for.... hmm... | 22:45 |
| sdake | bmace you guys = us guys :and gals ;) | 22:45 |
| bmace | sdake sure.. you folks :) | 22:45 |
| sdake | bmace just because you left the drivers team doesns't mean your not one of us :) | 22:46 |
| sdake | there is no escape ;-) | 22:46 |
| bmace | sdake lol, fair enough. now i'm a back seat driver ;) | 22:46 |
| kfox1111 | there's the bug I think.... | 22:46 |
| sdake | bmace me too, me too | 22:46 |
| sdake | bmace hope to get out of that shortly :) | 22:46 |
| openstackgerrit | Qin Wang (qwang) proposed openstack/kolla-kubernetes: Add ansible workflow for Horizon https://review.openstack.org/382620 | 22:47 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 22:47 |
| sdake | kfox1111 sbezverk_ can you have a look at that workflow patch above | 22:48 |
| *** Pavo has quit IRC | 22:49 | |
| sdake | kfox1111 sbezverk_ is that the direction we are headed re workflows? | 22:49 |
| *** vhosakot has quit IRC | 22:50 | |
| kfox1111 | I think so, at least until kolla-ansible is moved out. then I think the plan is to store the kolla-kubernetes ansible workflow with the other ansible stuff in kolla-ansible. | 22:50 |
| sdake | intereseting | 22:51 |
| sdake | cool thanks | 22:51 |
| sdake | i'd review that patch but don't want to break anything :) | 22:51 |
| *** Pavo has joined #openstack-kolla | 22:51 | |
| kfox1111 | there isn't anything to break yet. all the ansible stuff is under review right now. nothing has merged yet. | 22:52 |
| sdake | got it | 22:52 |
| sdake | i'd rather you and sbezverk_ review it | 22:52 |
| kfox1111 | the ansible stuff is like, a couple days old so far. :) | 22:52 |
| sdake | make sure it suits your tastes | 22:52 |
| kfox1111 | k. ryan's doing all the work so far, so he's probably most suited to it. | 22:52 |
| kfox1111 | it looks inline with what he's been doing though. | 22:53 |
| sdake | i think as far as where hte workflow goes, probably not kolla-ansible | 22:53 |
| sdake | probably a new repo | 22:53 |
| kfox1111 | it doens't really belong in kolla-kubernetes either. | 22:53 |
| kfox1111 | yeah. I suggested a kolla-kubernetes-ansible but inc0's didn't like the idea. | 22:54 |
| sdake | we are going to be spending a lot of time on this repo split in ocata :( | 22:54 |
| kfox1111 | yeah. | 22:54 |
| kfox1111 | but Ithink it will be very good for kolla's future. | 22:54 |
| sdake | agree | 22:54 |
| sdake | now is the time to do it | 22:54 |
| kfox1111 | I'd really rather start the kolla-kubernetes ansible stuff out of repo. | 22:55 |
| kfox1111 | for similar reasons. | 22:55 |
| kfox1111 | keeping it in, then moving it out has all the same issues keeping ansible in the main kolla repo has had I think. | 22:55 |
| kfox1111 | I'd rather just start seperate. | 22:55 |
| sdake | come up with a name people can agree on | 22:55 |
| sdake | and i'll get you a fresh repo to start with | 22:56 |
| kfox1111 | so, there's two ways to go I think. | 22:56 |
| kfox1111 | kolla-kubernetes is the repo name, and kollakube is the cli name. | 22:56 |
| kfox1111 | I'd kind of like for the ansible workflwo cli name to be kollakube-ansible to match up with kolla-ansible if they are kept seperate. | 22:56 |
| kfox1111 | so the repo name could be ither kolla-kubernetes-ansible or if thats too long, kollakube-ansible | 22:57 |
| sdake | instead of calling it kolla-kubernetes should have called it kollakube | 22:58 |
| sdake | that would have made that choice alot easier | 22:58 |
| sdake | what you just proposed in a is too long and in b is inconsistent :( | 22:59 |
| kfox1111 | yeah. hindsite and all. :) | 22:59 |
| sdake | right | 22:59 |
| sdake | if i could predict the future i'd be a billionaire ;) | 22:59 |
| kfox1111 | or c, kolla-kube-ansible | 22:59 |
| sdake | thats good | 22:59 |
| sdake | bounce it off the ml for a vote | 22:59 |
| sdake | any core can request a vote | 23:00 |
| kfox1111 | inc0 already said he prefers sticking it in kolla-ansible... I don't want to step on the ptl's shoes. | 23:00 |
| sdake | actuallly your not corei n kolla itself so i think yo ucan't propose a vote | 23:00 |
| sbezverk_ | kfox1111: why do we need ansible in kube repo name? or I missed something? | 23:00 |
| sdake | well i dont really want it in kolla-ansible ;) | 23:01 |
| kfox1111 | should we discuss it more first to see how strongly he feels? | 23:01 |
| *** ccesario has quit IRC | 23:01 | |
| sdake | yes needs more discussion over some indian food in barcelona | 23:01 |
| kfox1111 | sbezverk_: the topic is, seperating the ansible code from the kolla-kubernetes repo. what name would the ansible repo have. | 23:01 |
| sbezverk_ | kfox1111: frankly I do not see reason to mention ansible in the name of kubernetes repo, it will send wrong message | 23:01 |
| sdake | but that means it blocks workflow dev | 23:01 |
| *** eaguilar has joined #openstack-kolla | 23:01 | |
| kfox1111 | sdake: the issue is, the code's starting to land soon. its all in review now, so could be pushed off to another repo. but how long to we keep the stuff in review? | 23:02 |
| kfox1111 | yeah. | 23:02 |
| sdake | kfox1111 right - either need decision now | 23:02 |
| sbezverk_ | kfox1111: got it, it is not about naming the main kube repo | 23:02 |
| sdake | or punt and make a new repo later | 23:02 |
| sdake | sbezverk_ we should have named that differently I think ;-) | 23:02 |
| sbezverk_ | sdake: I really hoe we will.. | 23:03 |
| kfox1111 | sbezverk_: yeah. | 23:03 |
| sdake | anyway i guess we need to circle aorund with inc0 | 23:03 |
| kfox1111 | +1 | 23:03 |
| kfox1111 | oh... goodie... | 23:03 |
| sdake | sbezverk_ hope we will what | 23:03 |
| kfox1111 | it looks like the ceph backed mariadb finally cleared! :) | 23:03 |
| sdake | sweet | 23:03 |
| sbezverk_ | sdake: name that differently | 23:03 |
| sdake | sbezverk_ you mean kolla-kubernetes? | 23:03 |
| sdake | sbezverk_ it is basically impossible to do a rename i think | 23:04 |
| sbezverk_ | sdake: no I am all for kolla-kubernetes name :-) by mistake I thought you want to rename it into kolla-kube-ansible | 23:04 |
| *** diogogmt has quit IRC | 23:05 | |
| sdake | no kollakube woudl have been better | 23:05 |
| sdake | so we could call the ansible workflow kollakube-ansible | 23:05 |
| kfox1111 | repos and commands are two different things. the cli's much more imporatnt I think. | 23:06 |
| kfox1111 | as users type it in every day. | 23:06 |
| kfox1111 | devs don't clone as often. | 23:07 |
| kfox1111 | (usually) | 23:07 |
| sdake | what i mean about the repo rename | 23:07 |
| sdake | is infra i dont think does that anymore | 23:07 |
| sdake | we could create a new mirror repo with "upstream" | 23:07 |
| *** salv-orl_ has quit IRC | 23:07 | |
| sdake | but woudl have to retire the old name | 23:07 |
| kfox1111 | soo close now: http://logs.openstack.org/41/381041/52/experimental/gate-kolla-kubernetes-deploy-ubuntu-binary-ceph-nv/62580c4/console.html | 23:07 |
| kfox1111 | I think that was because I didn't create the glance volume in rbd... now to add that back.... | 23:08 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:08 |
| kfox1111 | sdake: finally got that rbd trick to work. so now you don't have to install any ceph stuff on the hosts. :) | 23:10 |
| kfox1111 | it uses a daemonset for the rbd binary. :) | 23:10 |
| *** ccesario has joined #openstack-kolla | 23:13 | |
| kfox1111 | sdake: I guess it wouldn't hurt to ask infra if it was still possible. | 23:20 |
| kfox1111 | worst they could say is no. | 23:21 |
| kfox1111 | I guess the ohter thing we could do.... | 23:21 |
| kfox1111 | is make a new repo, import the existing one into it, | 23:21 |
| sdake | well if we decide on that option - lets decide and then ask :) | 23:21 |
| sdake | kfox1111 we can do that one easily | 23:21 |
| kfox1111 | and add a final commit to the old one delting all the stuff and leaving a pointer. | 23:21 |
| sdake | but it results in a repo that has to be deprecated | 23:21 |
| kfox1111 | yeah. | 23:22 |
| kfox1111 | so whats infra want more in that case, to manage a depericated repo, or to rename one. | 23:22 |
| *** lamt has joined #openstack-kolla | 23:25 | |
| sdake | kfox1111 let me ask | 23:25 |
| kfox1111 | kk | 23:25 |
| sdake | kfox1111 join #openstack-infra | 23:26 |
| sdake | just asked | 23:26 |
| kfox1111 | k | 23:26 |
| *** logan- has quit IRC | 23:28 | |
| sdake | kfox1111 - clarkb said renames are possible | 23:28 |
| kfox1111 | :) | 23:28 |
| sdake | clarkb further indicated they do them about once a month | 23:28 |
| sdake | it has to be scheduled | 23:28 |
| sdake | they prefer that to what you proposed (a new proejct and deprecate old repo) | 23:29 |
| kfox1111 | k. so lets decide on if we're doing anything, and what, and then get it scheduled if we are. | 23:29 |
| sdake | would probbly make agood mailing list discussion ;) | 23:29 |
| sdake | vs 2 guys kibitzing in an irc channel :) | 23:29 |
| sdake | kolla | 23:30 |
| sdake | kollakube | 23:30 |
| sdake | kolla-ansible | 23:30 |
| sdake | kollakube-ansible | 23:30 |
| sdake | seems reasonble to me | 23:30 |
| kfox1111 | +1 | 23:30 |
| sdake | consistent and short | 23:31 |
| kfox1111 | nice.... vm lauched! :) | 23:31 |
| sbezverk_ | sdake kfox1111: one thing that kills me in all virtualization networking is difficulty and sometimes impossibility to test the complete path packet takes:-( | 23:31 |
| kfox1111 | sbezverk_: yeah. it can be quite twisty. :) | 23:32 |
| sbezverk_ | like the problem I am facing now, it freaking crazy | 23:32 |
| kfox1111 | debugging neutron dvr was a lot of fun until I figured it out. :) | 23:32 |
| *** HyperJohnGraham has quit IRC | 23:32 | |
| kfox1111 | sbezverk_: still fighting the canal thing? | 23:32 |
| sbezverk_ | kfox1111: yep and it is getting more wierd | 23:33 |
| sdake | kfox1111 that aha moment | 23:33 |
| sdake | kfox1111 then it passes | 23:33 |
| sdake | kfox1111 i like that too :) | 23:33 |
| kfox1111 | sdake: have a look: http://logs.openstack.org/41/381041/53/experimental/gate-kolla-kubernetes-deploy-centos-binary-ceph-nv/3a1637c/console.html | 23:33 |
| kfox1111 | all thats left is testing cinder volume attachment and floating ips... just a few more bugs to work out. :) | 23:34 |
| sdake | kfox1111 only 17 min to run | 23:34 |
| kfox1111 | yup. :) | 23:35 |
| sdake | kfox1111 looking good :) | 23:35 |
| kfox1111 | I might be able to shave a bit more back off. | 23:35 |
| kfox1111 | sbezverk_: whats it doing? | 23:35 |
| sdake | kfox1111 bindeps.txt will shave 1-2 minutes | 23:35 |
| kfox1111 | already did bindeps. :/ | 23:36 |
| sbezverk_ | kfox1111: from host where pod is running I can ping container by ip addressed provided to it by flannel, but from the container to the local host or anything beyon does not work.. | 23:36 |
| kfox1111 | sbezverk_: hmm... that really feels like a firewall issue... | 23:37 |
| *** logan- has joined #openstack-kolla | 23:37 | |
| sbezverk_ | kfox1111: or calico policy | 23:37 |
| kfox1111 | yeah. but in theory, there shouldn't be any of those. | 23:37 |
| sbezverk_ | also firewall usually protect inbound traffic so I would expect in reverese behavior | 23:38 |
| kfox1111 | I'm deploying canal in the gate, and haven't run into that yet. | 23:38 |
| kfox1111 | sbezverk_: yeah, but, indbound and outbound get reversed at odd types in virtualization. | 23:38 |
| sbezverk_ | kfox1111: well as I said it is working perfectly fine in my cluster | 23:38 |
| kfox1111 | the host firewall might consider the container poking twards it, incoming. | 23:38 |
| kfox1111 | have you tried two containers on the same host pinging eachother? | 23:39 |
| sbezverk_ | so it is somehitng special to John setup and it escapes me :-( | 23:39 |
| kfox1111 | as that would be a forward rather then an input. | 23:39 |
| kfox1111 | that might rule out a calico policy. | 23:39 |
| kfox1111 | OH.. I know why cinder isn't working... | 23:40 |
| kfox1111 | there are 2 sets of endpoints for it... | 23:41 |
| kfox1111 | one for v1 and one for v2... | 23:41 |
| sdake | kfox1111 ca nyou start a discussion on the ml re the irc discussion we just had | 23:41 |
| kfox1111 | I can probably start it tomrorow. if you can't get to it sooner. I have to go in a couple minutes. | 23:41 |
| sbezverk_ | kfox1111: yeah, end points reg job was registering both v1 and v2 | 23:43 |
| sdake | kfox1111 its your idea not mine | 23:44 |
| sdake | kfox1111 it is viable and i'd support it if it makes sense | 23:44 |
| sdake | kfox1111 rely on the cores to make that decision - rather then operate in a vacuum | 23:44 |
| sdake | rather I rely | 23:44 |
| sdake | rename before summit not possible | 23:45 |
| sdake | rename after possible | 23:45 |
| sdake | might as well start with correct name for workflow engine | 23:45 |
| sdake | or rather the workflow bits | 23:45 |
| kfox1111 | hmm... yup. the cinder entries have 2 versions... | 23:45 |
| sdake | kfox1111 that is normal - v2 and v3 | 23:45 |
| sdake | iirc | 23:45 |
| sdake | or v1/v2 | 23:45 |
| kfox1111 | k. I'll add them back. | 23:46 |
| sdake | dc with sbezverk_ | 23:46 |
| sbezverk_ | yeah v1/v2 | 23:46 |
| sdake | i'm pretty sure they are there for a reason :) | 23:46 |
| sdake | ok gotta jet to folks house | 23:46 |
| sdake | bbl | 23:47 |
| kfox1111 | yup. the opentsack cinder client fails. its not autofailing back to v1 | 23:47 |
| *** david-lyle has quit IRC | 23:47 | |
| kfox1111 | k. have a good one. | 23:47 |
| sbezverk_ | take care | 23:47 |
| kfox1111 | looks like its the same with just v2 suffixed. | 23:47 |
| *** sdake has quit IRC | 23:48 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: WIP: External Ceph Tools and Docs https://review.openstack.org/381041 | 23:49 |
| kfox1111 | heading out. bbl | 23:50 |
| sbezverk_ | kfox1111: I probably mentioned it to you, if not sorry I have not done it earlier, please check this bug I filed for cinder I hit on kube. https://bugs.launchpad.net/cinder/+bug/1619701 | 23:51 |
| openstack | Launchpad bug 1619701 in Cinder newton "in k8s environment vgs return extra line in output" [Medium,Fix released] - Assigned to Gorka Eguileor (gorka) | 23:51 |
| sbezverk_ | kfox1111: the fix they released does not fix it :-( | 23:52 |
| sbezverk_ | kfox1111: hopefully it does not impact ceph.. | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!