*** thumpba_ has joined #openstack-kolla | 00:03 | |
*** bmace has quit IRC | 00:03 | |
*** zhurong has quit IRC | 00:04 | |
*** thumpba has quit IRC | 00:05 | |
*** jtriley has joined #openstack-kolla | 00:09 | |
*** severion has quit IRC | 00:13 | |
*** v1k0d3n has joined #openstack-kolla | 00:13 | |
*** Jeffrey4l has joined #openstack-kolla | 00:16 | |
*** bmace has joined #openstack-kolla | 00:20 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Bump the ansible version to 2.1.1.0 in kolla-toolbox https://review.openstack.org/356311 | 00:28 |
---|---|---|
*** duonghq has joined #openstack-kolla | 00:34 | |
*** sdake has joined #openstack-kolla | 00:35 | |
sdake | pavo back alive | 00:36 |
duonghq | morning sdake | 00:37 |
sdake | duonghq afternon actuallyy | 00:38 |
sdake | powernap + powerdinner | 00:38 |
sdake | ready to rck | 00:38 |
sdake | any9one need any help | 00:38 |
duonghq | what is current temperature sdake? | 00:39 |
sdake | hot | 00:39 |
duonghq | dose the winter is better? | 00:39 |
sdake | 5pm | 00:39 |
sdake | winter is much better | 00:39 |
sdake | prorbably 105-1110 atm | 00:39 |
sdake | 110 that is | 00:40 |
duonghq | good weather, huh? does it rain? | 00:41 |
duonghq | I mean winter | 00:41 |
sdake | during summer yes | 00:42 |
sdake | during winter no | 00:42 |
sdake | during summer we have monsoons | 00:42 |
duonghq | monsoons festival? | 00:43 |
sdake | monsoons are like a big rain storm | 00:43 |
duonghq | ah | 00:43 |
sdake | which cools off the heat | 00:43 |
sdake | peopel in arizona are angry during summer time | 00:43 |
sdake | and happy during winter | 00:43 |
sdake | its kind of wierd | 00:43 |
sdake | i think its opposite to the rest of the us ;) | 00:44 |
duonghq | there is storm come here in about 1-2 days :(, long time no storm go here | 00:44 |
sdake | angry might be the wrong word | 00:44 |
sdake | displeased might be better | 00:44 |
duonghq | I like the winter, too | 00:44 |
sdake | i've lost 10 punds in the lat 1 month because of the heat affecting my appetite | 00:44 |
sdake | continuous problem in ariona | 00:44 |
sdake | arizona isn't problem free | 00:45 |
sdake | but all the datacentres in the us are here | 00:45 |
sdake | becuase nothing ever goes wrong | 00:45 |
sdake | except its hotter then hell ;) | 00:45 |
duonghq | yay | 00:45 |
sdake | no tornadoes no earthquakes no nothing | 00:46 |
duonghq | just hot? | 00:46 |
sdake | yup just hot | 00:46 |
sdake | and 280% murder rate above avereage | 00:46 |
sdake | although i don't live in that part of town | 00:46 |
sdake | town/city | 00:47 |
sdake | murder rate increases in summer time - presumably because of resource constraints | 00:47 |
duonghq | bad "habbit" | 00:47 |
*** jtriley has quit IRC | 00:48 | |
sdake | i feel safe all the time | 00:48 |
sdake | i dont even lock my doors | 00:48 |
sdake | others - in other parts of town - have 5 deadbolts on their door | 00:49 |
duonghq | so good for you | 00:49 |
sdake | location location location | 00:49 |
wirehead_ | Most of the year, it’s the valley of the sun. Part of the year, it’s the surface of the sun. | 00:49 |
sdake | wirehead_ wow - never picked up on that | 00:50 |
duonghq | the chant spirit from list meeting still around, | 00:50 |
sdake | wirehead_ surpried- and i'm 42 years old :) | 00:50 |
wirehead_ | One of my grand-bosses at Rackspace lived in Phoenix and said that. | 00:50 |
sdake | he must have been grand | 00:50 |
sdake | and more experienced then I | 00:51 |
sdake | wirehead_ do you use jabber | 00:51 |
wirehead_ | I think I’m supposed to, but I don't. | 00:51 |
sdake | i see your not on | 00:51 |
sdake | would you mind indulging me for a moment | 00:51 |
sdake | battery at 2% brb if power dies out | 00:54 |
wirehead_ | K, well the Jabber client is forcing me to download a 65 meg update. But I think I’m on. | 00:55 |
sbezverk | wirehead_ I see some issue with neutron boostrap, wondering if you see something similar | 00:56 |
sbezverk | wirehead_ when it tries to run db upgrade on neutron db and is it was already upgraded, I see continuous restart of bootstrap process | 00:57 |
wirehead_ | hmmmm. | 00:58 |
sbezverk | wirehead_ wondering if there is way to check if db was already upgraded and then ignore upgrade | 00:58 |
*** sdake has quit IRC | 00:59 | |
Pavo | sdake cool, I am setting up nodes agan | 00:59 |
wirehead_ | That seems like a bug in the update process. | 00:59 |
Pavo | lol like for the 6th time | 00:59 |
duonghq | sdake: how about the SAP case? | 01:00 |
sbezverk | wirehead_ most likely, they should handle it more gracefully | 01:00 |
*** jtriley has joined #openstack-kolla | 01:00 | |
sbezverk | will open a bug tomorrow against neutron | 01:01 |
Pavo | so whats the most stable release 2.0.3? | 01:01 |
*** sdake has joined #openstack-kolla | 01:02 | |
duonghq | sdake: how about the SAP case? | 01:03 |
sdake | wirehead_ around on jabber? | 01:03 |
sdake | i lost power | 01:03 |
sdake | duonghq huh? | 01:03 |
duonghq | SAP use our work without license include | 01:03 |
sdake | duonghq someone told me that had been fixed | 01:04 |
wirehead_ | sbezverk: When I test, I usually nuke everything and start from scratch. | 01:04 |
sdake | have a link to the offending repo? | 01:04 |
*** zhurong has joined #openstack-kolla | 01:10 | |
*** jtriley has quit IRC | 01:11 | |
*** v1k0d3n has quit IRC | 01:11 | |
*** v1k0d3n has joined #openstack-kolla | 01:11 | |
*** sdake has quit IRC | 01:11 | |
*** v1k0d3n has quit IRC | 01:14 | |
*** v1k0d3n has joined #openstack-kolla | 01:15 | |
duonghq | sdake: still there? | 01:15 |
kbaegis | sdake I still need help :) | 01:17 |
*** chrichip has joined #openstack-kolla | 01:19 | |
*** v1k0d3n has quit IRC | 01:23 | |
*** dave-mccowan has quit IRC | 01:25 | |
kbaegis | I guess another way to go would be to build my own l2 agent | 01:26 |
duonghq | kbaegis: from scratch? | 01:26 |
kbaegis | No alternative :) | 01:29 |
kbaegis | duonghq so I guess so. Centos works for EVERYTHING but ovs or linux bridging | 01:29 |
kbaegis | there's vikraman/gentoo | 01:29 |
duonghq | how about the ubuntu ones? | 01:29 |
*** huikang has joined #openstack-kolla | 01:30 | |
kbaegis | Might do. I didn't test that. | 01:35 |
kbaegis | I'll hardcode and reset. I'm pretty sure it's a kernel version thing though | 01:38 |
kbaegis | who's responsible for the centos one ooc? Could give me pointers? | 01:38 |
*** unicell has quit IRC | 01:39 | |
sbezverk | sdake ping | 01:42 |
duonghq | sbezverk: he is go out | 01:43 |
sbezverk | duonghq thank you will try to catch him later | 01:43 |
duonghq | (try stop openstack meeting bot from logging from last meeting, insane bug) | 01:45 |
duonghq | #endmeeting | 01:45 |
openstack | Meeting ended Thu Aug 18 01:45:50 2016 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 01:45 |
openstack | Minutes: http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.html | 01:45 |
openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.txt | 01:45 |
openstack | Log: http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.log.html | 01:45 |
kbaegis | lol | 01:46 |
duonghq | insane bot | 01:46 |
duonghq | http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.log.txt | 01:46 |
kbaegis | I'm the most talkative! | 01:46 |
*** daneyon has joined #openstack-kolla | 01:47 | |
duonghq | Jeffrey4l: can you review my patchset? https://review.openstack.org/#/c/356183/ | 01:48 |
patchbot | duonghq: patch 356183 - kolla - Port with_ loop variable syntax to Ansbile 2 (finish) | 01:48 |
duonghq | kbaegis: really bad case, seem that you lost our meeting log | 01:49 |
duonghq | *you -> we | 01:49 |
*** haplo37__ has joined #openstack-kolla | 01:50 | |
*** daneyon has quit IRC | 01:52 | |
*** thumpba_ has quit IRC | 01:52 | |
kbaegis | Yeah, hope I didn't do anything :) | 01:56 |
duonghq | kbaegis: somebody call startmeeting on this channel, | 01:59 |
Pavo | does this look normal during image build process | 02:02 |
Pavo | http://pastebin.com/S5v35xZ6 | 02:02 |
duonghq | Pavo: centos-based? | 02:02 |
Pavo | yes | 02:03 |
duonghq | I saw this sometime when yum try to find best mirror | 02:03 |
Pavo | so... its normal then | 02:04 |
duonghq | guess that, as long as this can install needed package | 02:04 |
Pavo | because looks like I am getting ti alot | 02:04 |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: In kubernetes neutron-ovs-agent needs to run as root https://review.openstack.org/356827 | 02:05 |
duonghq | does it get package eventually | 02:05 |
Pavo | looks like it | 02:05 |
duonghq | so, I'll ok, just wait for summary report, but it seem that the mirror is not in sync | 02:06 |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: In kubernetes neutron-ovs-agent needs to run as root https://review.openstack.org/356827 | 02:07 |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: In kubernetes neutron-ovs-agent needs to run as root https://review.openstack.org/356827 | 02:12 |
*** neophy has joined #openstack-kolla | 02:13 | |
*** chrichip has quit IRC | 02:13 | |
*** chrichip has joined #openstack-kolla | 02:15 | |
*** diogogmt has joined #openstack-kolla | 02:17 | |
*** diogogmt has quit IRC | 02:18 | |
openstackgerrit | Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 | 02:19 |
*** haplo37__ has quit IRC | 02:20 | |
*** v1k0d3n has joined #openstack-kolla | 02:22 | |
*** thumpba has joined #openstack-kolla | 02:23 | |
*** mewald has joined #openstack-kolla | 02:24 | |
*** mewald1 has quit IRC | 02:26 | |
*** v1k0d3n has quit IRC | 02:28 | |
*** caowei has joined #openstack-kolla | 02:32 | |
*** unicell has joined #openstack-kolla | 02:37 | |
*** inc0 has quit IRC | 02:38 | |
*** coolsvap has joined #openstack-kolla | 02:45 | |
*** chrichip has quit IRC | 02:45 | |
*** yuanying_ has quit IRC | 02:46 | |
*** chrichip has joined #openstack-kolla | 02:47 | |
*** huikang has quit IRC | 02:47 | |
*** huikang has joined #openstack-kolla | 02:48 | |
*** bjolo has quit IRC | 02:51 | |
*** huikang has quit IRC | 02:52 | |
*** yuanying has joined #openstack-kolla | 02:52 | |
openstackgerrit | Josh Lothian proposed openstack/kolla: Customizations for heka https://review.openstack.org/346861 | 03:00 |
openstackgerrit | Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 | 03:03 |
openstackgerrit | Merged openstack/kolla: Fix tar command path for magnum conductor using ubuntu https://review.openstack.org/356008 | 03:07 |
openstackgerrit | Merged openstack/kolla: Skips a debug log during image build https://review.openstack.org/351108 | 03:08 |
openstackgerrit | Lu lei proposed openstack/kolla: Fix Kolla home-page URL https://review.openstack.org/355275 | 03:10 |
*** huikang has joined #openstack-kolla | 03:12 | |
openstackgerrit | Merged openstack/kolla: Minor doc fix https://review.openstack.org/356068 | 03:12 |
duonghq | coolsvap: are you there? | 03:13 |
coolsvap | duonghq: yup | 03:13 |
duonghq | as described in http://docs.openstack.org/developer/manila/devref/commit_message_tags.html | 03:14 |
duonghq | the TrivialFix tag is when the commit does not require the creation of a separate bug or blueprint in Launchpad | 03:14 |
openstackgerrit | Merged openstack/kolla: Move the keystone_*_url variable from common role to group vars https://review.openstack.org/356319 | 03:14 |
duonghq | but my Port .... commit is belong to bp ansbile2 | 03:14 |
duonghq | so, it is still should have TrivialFix tag? | 03:15 |
coolsvap | duonghq: http://docs.openstack.org/developer/kolla/CONTRIBUTING.html #4 | 03:15 |
*** thumpba has quit IRC | 03:15 | |
duonghq | ah, thanks | 03:15 |
openstackgerrit | Merged openstack/kolla: Add a note about the sandbox environment to the CONTRIBUTING doc https://review.openstack.org/356017 | 03:15 |
coolsvap | duonghq: wc :) | 03:16 |
coolsvap | sorry i should have added it in review | 03:16 |
kbaegis | So since I'm trying to reverse engineer a specific container (openvswitchd), is there a doc/description on how kolla is actually using kolla_start? | 03:17 |
kbaegis | It would be really handy to know the actual command that's getting run on a functional container | 03:17 |
kbaegis | I'm assuming that I can't simply run openvswitch on the host | 03:18 |
*** yuanying has quit IRC | 03:18 | |
*** yuanying has joined #openstack-kolla | 03:19 | |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Port with_ loop variable syntax to Ansbile 2 (finish) https://review.openstack.org/356183 | 03:19 |
Pavo | does ansible have to be installed on all nodes, including target nodes? | 03:19 |
duonghq | Pavo: no | 03:19 |
duonghq | ansible using push model | 03:19 |
Pavo | hmmm prechecks are failing because no version is being reported from target nodes | 03:20 |
duonghq | weird | 03:20 |
Pavo | yeah | 03:20 |
coolsvap | duonghq: we dont need both bp and TrivialFix | 03:20 |
Pavo | also I should only use ansible 1.9.4 right | 03:20 |
Pavo | not 2.1.1 | 03:20 |
duonghq | Pavo: 2.1.x has trouble with OpenSSH 6.6, | 03:21 |
duonghq | sad | 03:21 |
openstackgerrit | Merged openstack/kolla: Fix the Ubuntu Xenial build gate https://review.openstack.org/355274 | 03:21 |
Pavo | so should use 1.9.4 | 03:21 |
*** coolsvap is now known as _coolsvap_ | 03:21 | |
duonghq | _coolsvap_: thanks, next time I'll pay more attention | 03:22 |
kbaegis | Oh, I see what's going on here :) | 03:22 |
duonghq | Pavo: I'm looking for a way to upgrade may OpenSSH | 03:22 |
*** v1k0d3n has joined #openstack-kolla | 03:25 | |
openstackgerrit | Md Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy https://review.openstack.org/350629 | 03:31 |
*** daneyon has joined #openstack-kolla | 03:36 | |
duonghq | Jeffrey4l: so in this case, I should use TrivialFix or bp link? | 03:40 |
duonghq | in Port ... case | 03:40 |
*** daneyon has quit IRC | 03:40 | |
Jeffrey4l | duonghq, bp is OK. | 03:40 |
Jeffrey4l | Port? | 03:40 |
Jeffrey4l | duonghq, you mean backport? | 03:40 |
duonghq | ah, the patchset Port with_ loop | 03:40 |
duonghq | I don't know with tag is better | 03:41 |
duonghq | for this case | 03:41 |
Jeffrey4l | duonghq, one of them both OK. choose one. | 03:42 |
*** yuanying has quit IRC | 03:42 | |
*** inc0 has joined #openstack-kolla | 03:42 | |
duonghq | ah, ok | 03:42 |
*** yuanying has joined #openstack-kolla | 03:42 | |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Port with_ loop variable syntax to Ansbile 2 (finish) https://review.openstack.org/356183 | 03:43 |
kbaegis | got it! :) | 03:43 |
duonghq | Jeffrey4l: fixed, thank you | 03:43 |
*** yuanying has quit IRC | 03:43 | |
duonghq | kbaegis: anything news? | 03:43 |
Jeffrey4l | thanks. | 03:43 |
kbaegis | I hacked a container | 03:44 |
duonghq | which ones? | 03:45 |
kbaegis | Pretty simple actually. Built ovs-vswitchd from source mapping it to my current kernel on a stage3, mapped through run, and found the command actually run by kolla | 03:45 |
*** huikang has quit IRC | 03:45 | |
kbaegis | Agent was working fine | 03:45 |
kbaegis | So I just had to hack the ovs | 03:45 |
*** huikang has joined #openstack-kolla | 03:46 | |
duonghq | ah, nice | 03:46 |
*** ad_rien_ has joined #openstack-kolla | 03:47 | |
*** ad_rien_ has quit IRC | 03:47 | |
*** yuanying has joined #openstack-kolla | 03:48 | |
openstackgerrit | Merged openstack/kolla: Fix neutron-server section in AIO sample inventory https://review.openstack.org/356056 | 03:50 |
*** huikang_ has joined #openstack-kolla | 03:50 | |
*** huikang has quit IRC | 03:50 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Remove duplicate items https://review.openstack.org/355377 | 03:56 |
*** v1k0d3n has quit IRC | 04:01 | |
openstackgerrit | Hui Kang proposed openstack/kolla: Add performance monitoring container to build config file https://review.openstack.org/356854 | 04:09 |
*** neophy has quit IRC | 04:19 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Update instructions for ubuntu with systemd https://review.openstack.org/346900 | 04:22 |
*** neophy has joined #openstack-kolla | 04:26 | |
*** inc0 has quit IRC | 04:30 | |
*** v1k0d3n has joined #openstack-kolla | 04:31 | |
*** sdake___ has joined #openstack-kolla | 04:38 | |
sdake___ | hey folks | 04:39 |
sdake___ | my laptop is broken | 04:39 |
sdake___ | so I will be impaired a bit for 1-2 weeks | 04:39 |
sdake___ | but i have full access to my entire lab | 04:39 |
sdake___ | which i got operational today | 04:39 |
sdake___ | just an fyi | 04:40 |
duonghq | sdake___, why? | 04:40 |
Pavo | that sucks sdake___ | 04:42 |
sdake___ | shit happens | 04:42 |
Pavo | thats why I have a mac they never break | 04:43 |
Pavo | lol | 04:43 |
sdake___ | ya thats what i got too | 04:43 |
Pavo | ouch how did you break it | 04:44 |
sdake___ | the power popped off as I was closing it, and it jammed between the screen and the keyboard as i was closing it | 04:44 |
Pavo | ah | 04:44 |
sdake___ | this busted the keyboard | 04:44 |
duonghq | sdake___: do we have cleanup-container for multinode deploy? | 04:44 |
duonghq | sdake___: unimageable case | 04:44 |
sdake___ | duonghq yes - you do | 04:44 |
Pavo | duonghq kolla-ansible destroy -i ansible/inventory/multinode | 04:44 |
duonghq | thank Pavo, sdake___ | 04:45 |
sdake___ | duonghq i know pretty odd huh | 04:45 |
sdake___ | sorry this busted the screen not keyboard | 04:45 |
sdake___ | so ive been running it like that for about 3 weeks | 04:45 |
Pavo | and you can scp the tool folder in kolla over to nodes and run the script separate | 04:45 |
sdake___ | and now it wont turn on | 04:45 |
sdake___ | i think the bad display probably burned out the power supply but who knows | 04:45 |
Pavo | yeah those screen are expensive | 04:46 |
sdake___ | ya - may be chepaer t obuy a new laptop then get mine repaired | 04:47 |
sdake___ | screens are 800 bucks | 04:47 |
sdake___ | the power supply rebuild is about 1600 total | 04:47 |
sdake___ | i dont nkow what the total bil lwould be | 04:47 |
sdake___ | my mac is top of the line tho - so not sure | 04:47 |
Pavo | yeah just buy a new one, they have skylake cpus now | 04:48 |
Pavo | so on board graphics are pretty good | 04:48 |
Pavo | grrr getting stuck at TASK: [neutron | Starting neutron-server container] during deployment | 04:52 |
duonghq | for how many min? | 04:52 |
Pavo | for about 3 minutes now | 04:52 |
Pavo | seems like its not finishing the bridging on the node | 04:53 |
*** chrichip has quit IRC | 04:53 | |
*** chrichip has joined #openstack-kolla | 04:53 | |
*** ajpower has joined #openstack-kolla | 04:53 | |
Pavo | br-ex and ovs-system interfaces are down | 04:55 |
Pavo | could this be a ansible version issue? | 04:56 |
Pavo | now docker0 interface is down also | 04:57 |
*** sdake___ has quit IRC | 05:00 | |
*** fragatina has quit IRC | 05:01 | |
duonghq | no way | 05:01 |
Pavo | how about kolla version | 05:02 |
Pavo | was using stable/mitaka version | 05:02 |
Pavo | gonna try with latest now | 05:02 |
*** v1k0d3n has quit IRC | 05:08 | |
Pavo | well now a different issue, /kolla/centos-binary-openvswitch-db-server not starting | 05:13 |
Pavo | logs show /usr/local/bin/kolla_start: line 24: exec: start-ovsdb-server: not found | 05:13 |
duonghq | ah, | 05:20 |
duonghq | misalign docker and ansible version | 05:20 |
duonghq | :) | 05:20 |
duonghq | Pavo, | 05:20 |
Pavo | running latest docker and latest ansible | 05:21 |
duonghq | do you clean image and rebuild? | 05:22 |
duonghq | uninstall kolla and install | 05:22 |
Pavo | just rebuild that image | 05:22 |
Pavo | now pulling it to nodes | 05:22 |
*** daneyon has joined #openstack-kolla | 05:24 | |
duonghq | Pavo: did you get 'module' object has no attribute 'EEXISTS'", "path": "/etc/kolla//memcached", "state": "absent"} error | 05:27 |
Pavo | don'tt think so | 05:27 |
Pavo | damnit stuck back at the starting neutron server part again | 05:28 |
duonghq | ansible cannot create /etc/kolla/memcached folder | 05:28 |
duonghq | Pavo: what is the logs tell? | 05:28 |
*** daneyon has quit IRC | 05:28 | |
Pavo | its not generating logs in /var/lib/docker/volumes/kolla_logs/neutron | 05:29 |
Pavo | and the container isn't running | 05:30 |
duonghq | sound familiar | 05:31 |
Pavo | guess I'll rebuild images to latest version | 05:34 |
Pavo | that was using 2.0.3 images build | 05:34 |
duonghq | 2.0.3 image with latest ansible playbook? | 05:34 |
Pavo | yeah | 05:34 |
Pavo | guess thats bad huh lol | 05:35 |
duonghq | the ovsdb start command just change recently | 05:35 |
Pavo | well it was getting stuck at the same spot using 1.9.4 ansible | 05:35 |
Pavo | and stable kolla | 05:35 |
Pavo | well its 1:30am here gonna start this build images cmd and head to bed | 05:37 |
Pavo | will try again tomorrow | 05:37 |
duonghq | bye now Pavo | 05:37 |
Pavo | see ya laters | 05:37 |
*** mntdevops has quit IRC | 05:42 | |
*** bootsha has joined #openstack-kolla | 05:42 | |
*** bootsha has quit IRC | 05:47 | |
*** bootsha has joined #openstack-kolla | 05:48 | |
*** bootsha has quit IRC | 05:53 | |
*** rstarmer has joined #openstack-kolla | 05:59 | |
*** huikang_ has quit IRC | 06:06 | |
duonghq | do I need root access for deploying Kolla multinode? | 06:06 |
*** neophy has quit IRC | 06:08 | |
*** Mr_Broken has quit IRC | 06:09 | |
*** Mr_Broken has joined #openstack-kolla | 06:09 | |
*** rstarmer has quit IRC | 06:10 | |
duonghq | kbaegis: do you try multinode deploy? | 06:12 |
*** Mr_Broken has quit IRC | 06:14 | |
duonghq | mewald: do you deploy mulitnode with Kolla? | 06:27 |
duonghq | mdnadeem: I see you join in osic testing? so you already do multinode deploy? | 06:29 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add warning to kolla-ansible destroy action https://review.openstack.org/356957 | 06:30 |
mdnadeem | duonghq, yes, i have try it once | 06:30 |
duonghq | Do you need any special settings for deploy node? | 06:31 |
mdnadeem | duonghq, I dnt think so | 06:32 |
duonghq | just normal user? | 06:32 |
mdnadeem | duonghq, running kolla-ansible with sudo | 06:32 |
duonghq | but what for other nodes? | 06:33 |
duonghq | (not deploy node) | 06:33 |
duonghq | which ansible_user do you use? | 06:33 |
mdnadeem | duonghq, What issue you gets | 06:33 |
duonghq | ah, what does control node meaning? | 06:33 |
duonghq | I got error the ansible cannot create /etc/kolla/memcached | 06:34 |
mdnadeem | duonghq, can you paste full log | 06:34 |
duonghq | I'll try again with sudo and copy log | 06:35 |
mdnadeem | duonghq, it seems permission issue, sudo will resolve it | 06:35 |
duonghq | but sudo just for deploy node | 06:35 |
mdnadeem | duonghq, In case of multinode, you need to execute sudo kolla-ansible deploy from operator node only | 06:36 |
duonghq | okay | 06:36 |
duonghq | I don't know why I already ssh-copy-id to target node but still cannot access this node use public key | 06:37 |
duonghq | trying | 06:38 |
*** ad_rien_ has joined #openstack-kolla | 06:44 | |
duonghq | mdnadeem: I got this http://paste.openstack.org/show/560725/ | 06:46 |
*** ad_rien_ has joined #openstack-kolla | 06:46 | |
mdnadeem | duonghq, what command you run? | 06:47 |
duonghq | ssudo -E kolla-ansible -i multinode deploy | 06:47 |
*** unicell1 has joined #openstack-kolla | 06:48 | |
*** unicell has quit IRC | 06:48 | |
*** Mr_Broken has joined #openstack-kolla | 06:48 | |
duonghq | the inventory file: http://paste.openstack.org/show/560727/ | 06:48 |
duonghq | (just head of inventory) | 06:49 |
duonghq | mdnadeem | 06:50 |
mdnadeem | duonghq, it may be due to ansible_ssh_user=stack | 06:50 |
duonghq | if I removed it, the user to ssh is None | 06:51 |
mdnadeem | duonghq, remove it and try with sudo | 06:51 |
duonghq | mdnadeem: It cannot ssh to target node, of cousre | 06:52 |
duonghq | fatal: [k8s]: UNREACHABLE! => {"changed": false, "msg": "Failed to connect to the host via ssh.", "unreachable": true} | 06:52 |
mdnadeem | have u resolve the ip in etc/host? | 06:53 |
duonghq | yes | 06:53 |
duonghq | I can ping using hostname | 06:53 |
duonghq | with ansible_ssh_user, the prechecks command run success | 06:54 |
mdnadeem | duonghq, i can deploy using http://paste.openstack.org/show/560729/ | 06:56 |
duonghq | from normal user? | 06:56 |
duonghq | mdnadeem: do you have any ssh setting? | 06:57 |
duonghq | like ansible_ssh_key_file | 06:57 |
duonghq | the target nodes has any special settingS/ | 06:57 |
mdnadeem | duonghq, no | 06:57 |
duonghq | mdnadeem: you set just control node? | 06:58 |
duonghq | only control node? | 06:58 |
duonghq | for test | 06:58 |
*** b_bezak has joined #openstack-kolla | 06:58 | |
mdnadeem | duonghq, just resolve host name in /etc/hosts | 06:59 |
duonghq | mdnadeem: the ip and hostname is already in place | 06:59 |
mdnadeem | duonghq, let me try once, will let u know if i missed something | 07:00 |
duonghq | thanks mdnadeem | 07:00 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Create the _member_ role in the horizon role https://review.openstack.org/356082 | 07:04 |
*** imcsk8 has quit IRC | 07:05 | |
*** imcsk8 has joined #openstack-kolla | 07:06 | |
*** dmueller-evoila has joined #openstack-kolla | 07:07 | |
*** imcsk8_ has joined #openstack-kolla | 07:11 | |
*** imcsk8 has quit IRC | 07:11 | |
*** daneyon has joined #openstack-kolla | 07:12 | |
*** dmueller-evoila has quit IRC | 07:15 | |
*** daneyon has quit IRC | 07:17 | |
*** dmueller-evoila has joined #openstack-kolla | 07:21 | |
*** matrohon has joined #openstack-kolla | 07:26 | |
*** dmueller-evoila has quit IRC | 07:31 | |
duonghq | mdnadeem: do user need nopasswd sudoer | 07:33 |
mdnadeem | duonghq, you can do password less ssh login by adding id_rsa.pub entry of operator node to authorized key of deployed node | 07:35 |
duonghq | already do this | 07:35 |
duonghq | mdnadeem: I mean do you make users in sudoers groups with nopasswd option? | 07:36 |
mdnadeem | duonghq, can you post output of ansible control -i mutinode -m ping -vvv | 07:37 |
*** fragatina has joined #openstack-kolla | 07:37 | |
duonghq | mdnadeem: without ansible_ssh_user option? | 07:38 |
mdnadeem | duonghq, with user | 07:38 |
mdnadeem | with ansible_ssh_user | 07:39 |
duonghq | with ansible_ssh_user option, the ping is successful | 07:39 |
mdnadeem | hmm | 07:39 |
mdnadeem | can you change ansible_ssh_user=root and try | 07:40 |
duonghq | ubuntu don't allow remote root login | 07:40 |
duonghq | the root is disable indeed | 07:40 |
*** fragatina has quit IRC | 07:42 | |
*** matrohon has quit IRC | 07:43 | |
mdnadeem | duonghq, i will try to deploy and check it. | 07:47 |
duonghq | thanks | 07:47 |
duonghq | mdnadeem I don't understand why the config playbook can run without any escalate permission | 07:52 |
duonghq | it copy file to system dir | 07:53 |
duonghq | wirehead_, are you free for some minutes? | 07:55 |
*** berendt has joined #openstack-kolla | 08:17 | |
berendt | Can sombody please have a look at https://review.openstack.org/#/c/346444/ | 08:26 |
patchbot | berendt: patch 346444 - kolla - Optimize horizon cache and compress for http response | 08:26 |
*** mnasiadka has joined #openstack-kolla | 08:36 | |
*** athomas has joined #openstack-kolla | 08:41 | |
*** papacz has joined #openstack-kolla | 08:41 | |
duonghq | berendt: do you have any experience in multinode deployment? | 08:43 |
berendt | duonghq i think so | 08:43 |
duonghq | do you need root user for deploy? | 08:43 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use dumb-init to manage the pid 1 process https://review.openstack.org/357015 | 08:44 |
duonghq | I don't understand why ansible can copy file to /etc/kolla... without escalate priviledge | 08:44 |
berendt | duonghq at the moment i use root for deployment, not using root is on my todo list, i think setting become in the ansible configuration file will work for kolla-ansible | 08:45 |
duonghq | hmm, you don't use Debian-based as host? | 08:45 |
duonghq | I grep in ansible dir and cannot find any become | 08:46 |
*** bootsha has joined #openstack-kolla | 08:50 | |
*** Serlex has joined #openstack-kolla | 08:52 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Sync the local_settings.j2 with horizon https://review.openstack.org/356471 | 08:53 |
*** daneyon has joined #openstack-kolla | 09:00 | |
berendt | duonghq i use ubuntu hosts | 09:04 |
duonghq | berendt: So you enable root and deploy using this? | 09:05 |
berendt | duonghq but i have not yet played with non-root user deployment | 09:05 |
berendt | duonghq for the moment, yes | 09:05 |
*** daneyon has quit IRC | 09:05 | |
duonghq | berendt: okay, thanks :D | 09:05 |
berendt | duonghq but this should be tested, i think it makes sense to work on this during the next cycle | 09:06 |
duonghq | agree | 09:06 |
*** _coolsvap_ is now known as coolsvap | 09:06 | |
duonghq | but I wonder if nobody test multinode deploy with non-root yet? | 09:07 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Vagrant doc updates https://review.openstack.org/353240 | 09:09 |
*** bootsha has quit IRC | 09:11 | |
openstackgerrit | Merged openstack/kolla: Fix Kolla home-page URL https://review.openstack.org/355275 | 09:15 |
*** papacz has quit IRC | 09:18 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for aodh https://review.openstack.org/346299 | 09:20 |
duonghq | Jeffrey41: it seems that my patchset at https://review.openstack.org/#/c/356183/ pass almost test, expect ubuntu-related ones | 09:29 |
patchbot | duonghq: patch 356183 - kolla - Port with_ loop variable syntax to Ansbile 2 (finish) | 09:29 |
duonghq | but afaik the ubuntu gate has trouble? | 09:29 |
duonghq | Jeffrey4l: it seems that my patchset at https://review.openstack.org/#/c/356183/ pass almost test, expect ubuntu-related ones | 09:29 |
patchbot | duonghq: patch 356183 - kolla - Port with_ loop variable syntax to Ansbile 2 (finish) | 09:30 |
duonghq | but afaik the ubuntu gate has trouble? | 09:30 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for murano https://review.openstack.org/346340 | 09:31 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for zaqar https://review.openstack.org/346357 | 09:38 |
*** berendt has quit IRC | 09:40 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for sahara https://review.openstack.org/346590 | 09:41 |
duonghq | coolsvap: https://review.openstack.org/#/c/356183/ I fixed my commit message, can you review my work? | 09:41 |
patchbot | duonghq: patch 356183 - kolla - Port with_ loop variable syntax to Ansbile 2 (finish) | 09:41 |
coolsvap | duonghq: i think i already +2ed it | 09:41 |
duonghq | coolsvap: but Jeffrey4l want me remove bp or trivialfix | 09:42 |
coolsvap | alright | 09:42 |
coolsvap | checking | 09:42 |
*** zhubingbing has joined #openstack-kolla | 09:42 | |
duonghq | Seems that we have not fixed ubuntu gates | 09:43 |
*** papacz has joined #openstack-kolla | 09:44 | |
*** berendt has joined #openstack-kolla | 09:44 | |
duonghq | and the commit: https://review.openstack.org/#/c/356235/ unblock doc generation at my pc | 09:46 |
patchbot | duonghq: patch 356235 - kolla - Fix guide formating errors | 09:46 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for trove https://review.openstack.org/346342 | 09:48 |
openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla: Updated from global requirements https://review.openstack.org/356930 | 09:51 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for trove https://review.openstack.org/346342 | 09:52 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for sahara https://review.openstack.org/346590 | 09:52 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for murano https://review.openstack.org/346340 | 09:53 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for aodh https://review.openstack.org/346299 | 09:53 |
coolsvap | Jeffrey4l: ping | 09:56 |
*** zhurong has quit IRC | 09:58 | |
openstackgerrit | Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 | 09:58 |
mewald | any ideas what the comment means? https://review.openstack.org/#/c/346450/20 | 09:59 |
patchbot | mewald: patch 346450 - kolla - Added grafana role | 09:59 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update cirros image visibility in init-runonce https://review.openstack.org/353933 | 10:03 |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Fix rabbitmq-clusterer template format https://review.openstack.org/357065 | 10:07 |
duonghq | coolsvap: can you review this commit, too https://review.openstack.org/#/c/356235/ | 10:10 |
patchbot | duonghq: patch 356235 - kolla - Fix guide formating errors | 10:10 |
*** bootsha has joined #openstack-kolla | 10:11 | |
duonghq | this unblock doc generation at my machine | 10:12 |
*** ad_rien_ has joined #openstack-kolla | 10:13 | |
openstackgerrit | Wei Cao proposed openstack/kolla: Update instructions for ubuntu with systemd https://review.openstack.org/346900 | 10:14 |
*** berendt has quit IRC | 10:14 | |
*** jrich523 has quit IRC | 10:15 | |
*** berendt has joined #openstack-kolla | 10:15 | |
*** egonzalez90 has joined #openstack-kolla | 10:15 | |
coolsvap | duonghq: I would recommend you sync with rhallisey regarding doc updates | 10:26 |
*** bootsha has quit IRC | 10:26 | |
duonghq | thank coolsvap for the contact point | 10:27 |
coolsvap | since he's reorganizing the docs | 10:27 |
duonghq | thanks | 10:27 |
*** berendt has quit IRC | 10:31 | |
openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Improve rabbitmq-clusterer template code format https://review.openstack.org/357065 | 10:31 |
*** duonghq has quit IRC | 10:34 | |
Jeffrey4l | coolsvap, pong | 10:34 |
coolsvap | Jeffrey4l: I am currently confused with the footer part in the base images with customization work | 10:34 |
coolsvap | i feel we dont have it and we dont need it | 10:35 |
coolsvap | but I see mixed reviews open and merged | 10:35 |
coolsvap | what do you think about it? | 10:36 |
Jeffrey4l | coolsvap, customization base can help us like: add custom repo? | 10:36 |
Jeffrey4l | or change the default repo. | 10:36 |
Jeffrey4l | coolsvap, is there any difficulty to implement the customization base? | 10:37 |
coolsvap | Jeffrey4l: no | 10:37 |
coolsvap | but just a thought | 10:37 |
Jeffrey4l | coolsvap, like how about i want to use a higher ceph repo. | 10:37 |
Jeffrey4l | i mush change something in the base image. | 10:38 |
Jeffrey4l | must* | 10:38 |
coolsvap | Jeffrey4l: i get what you are trying to do here, but ceph is not the correct example for this imho | 10:40 |
coolsvap | adding repo to the footer will make no sense in base images | 10:40 |
coolsvap | Jeffrey4l: to be clear i am talking about <<service>>-base images here | 10:41 |
coolsvap | not the base/openstack-base image | 10:41 |
Jeffrey4l | oh. I misunderstand. | 10:42 |
*** mnasiadka has quit IRC | 10:42 | |
Jeffrey4l | it is really useless to add the footer in the <<service>>-base image, i think. | 10:43 |
Jeffrey4l | coolsvap, ^ | 10:43 |
Jeffrey4l | i can not image a use case for that. | 10:43 |
*** berendt has joined #openstack-kolla | 10:44 | |
coolsvap | Jeffrey4l: yes | 10:44 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Update kubectl version at magnum-conductor container https://review.openstack.org/352986 | 10:44 |
Jeffrey4l | But 1. it is easy to add. 2. maybe some one need this: add something in all the nova-* images | 10:46 |
Jeffrey4l | so i am OK to add this. coolsvap | 10:46 |
coolsvap | fine with me as well | 10:46 |
coolsvap | but I was not too keen on adding this and having -1 comments for it | 10:46 |
coolsvap | thats why I asked | 10:46 |
Jeffrey4l | we should have a agreement, whether we should add this feature. i -1 for this, just because other image do not have this. | 10:47 |
egonzalez90 | agree with Jeffrey4l, but it need to be documented so all people know that footers in base images. Actually there are a lot with footers and others without them | 10:47 |
Jeffrey4l | we have a agreement now. So let's add this in all base images. | 10:48 |
coolsvap | yes | 10:48 |
coolsvap | i already updated all my change sets for adding in base images | 10:49 |
Jeffrey4l | cool | 10:49 |
*** daneyon has joined #openstack-kolla | 10:49 | |
pbourke | coolsvap: Jeffrey4l: we're making use of the template customisations heavily in our internal fork | 10:49 |
pbourke | coolsvap: Jeffrey4l: and the base footers are needed imo | 10:50 |
coolsvap | pbourke: ack | 10:50 |
pbourke | we also potentially need headers | 10:50 |
Jeffrey4l | yes. headers is also need. | 10:50 |
kbaegis | Morning everyone | 10:52 |
kbaegis | Jeffrey4l: I hear that you're the one to thank for all the timely patches :) ty | 10:52 |
*** daneyon has quit IRC | 10:53 | |
Jeffrey4l | kbaegis, :) | 10:55 |
Jeffrey4l | brb | 10:55 |
*** coolsvap is now known as _coolsvap_ | 10:55 | |
*** egonzalez90 has quit IRC | 10:55 | |
kbaegis | Alright, testing my proposed hack | 11:00 |
*** egonzalez90 has joined #openstack-kolla | 11:01 | |
openstackgerrit | Merged openstack/kolla: Updated from global requirements https://review.openstack.org/356930 | 11:04 |
openstackgerrit | Wei Cao proposed openstack/kolla: Update instructions for ubuntu with systemd https://review.openstack.org/346900 | 11:06 |
*** caowei has quit IRC | 11:07 | |
openstackgerrit | Merged openstack/kolla: Customizations for heka https://review.openstack.org/346861 | 11:08 |
*** neophy has joined #openstack-kolla | 11:08 | |
*** JoseMello has joined #openstack-kolla | 11:09 | |
*** neophy has quit IRC | 11:09 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Customizations for zaqar https://review.openstack.org/346357 | 11:11 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Create the _member_ role in the horizon role https://review.openstack.org/356082 | 11:15 |
openstackgerrit | Merged openstack/kolla: Port with_ loop variable syntax to Ansbile 2 (finish) https://review.openstack.org/356183 | 11:16 |
openstackgerrit | Merged openstack/kolla: Customizations for Cinder https://review.openstack.org/350587 | 11:23 |
openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Fix murano notification driver https://review.openstack.org/355979 | 11:23 |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: In kubernetes ovs and l3 agents need to run as root https://review.openstack.org/356827 | 11:24 |
openstackgerrit | Merged openstack/kolla: Remove duplicate items https://review.openstack.org/355377 | 11:26 |
*** bootsha has joined #openstack-kolla | 11:27 | |
openstackgerrit | Merged openstack/kolla: Build a more organized doc structure https://review.openstack.org/348029 | 11:27 |
sbezverk | Jeffrey4l: ping | 11:28 |
*** berendt has quit IRC | 11:29 | |
*** berendt has joined #openstack-kolla | 11:30 | |
*** papacz has quit IRC | 11:31 | |
*** bootsha has quit IRC | 11:32 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 11:32 |
Jeffrey4l | kbaegis, yo, what's it? | 11:33 |
Jeffrey4l | sbezverk, pong | 11:33 |
kbaegis | Jeffrey4l: I built a gentoo based container with openvswitch and mapped the kernel source into it | 11:34 |
kbaegis | That was having permission issues though. I'm verifying this right now: ovs-vswitchd unix:/run/openvswitch/db.sock -vconsole:emer -vsyslog:err -vfile:info --mlockall --log-file=/var/log/openstack/openvswitch/ovs-vswitchd.log | 11:34 |
Jeffrey4l | kbaegis, do u want the kolla to support gentoo? | 11:34 |
kbaegis | Jeffrey4l: I'd like you guys to be distro agnostic, yes :) | 11:35 |
kbaegis | That may or may not be feasible, and I understand either way. | 11:35 |
Jeffrey4l | kbaegis, hmm that will be hard for the current kolla. | 11:35 |
sbezverk | Jeffrey4l: there was a concern about ovs. I am now testing it on kube | 11:36 |
sbezverk | would like to understand better | 11:36 |
*** Mr_Broken has quit IRC | 11:36 | |
Jeffrey4l | now we have centos/ubuntu/debian/oraclelinux, we almost can not maintain them. | 11:36 |
kbaegis | Jeffrey4l: I've already been using ovs on my standard systems. Trivial modification for me :) | 11:36 |
*** Mr_Broken has joined #openstack-kolla | 11:36 | |
Jeffrey4l | kbaegis, cool | 11:36 |
Jeffrey4l | sbezverk, OK | 11:37 |
kbaegis | Jeffrey4l: Agreed. You need either direct support from the distros or to abstract away and become agnostic | 11:37 |
kbaegis | Either change is terribly difficult actually | 11:37 |
openstackgerrit | Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 | 11:37 |
Jeffrey4l | kbaegis, if we can reach that, that will be ideal | 11:38 |
kbaegis | sbezverk: There's fuel-ccp too | 11:38 |
kbaegis | sbezverk: Fuel microservices :) I'm avoiding using that for my lower layer | 11:38 |
kbaegis | I really like the design behind kolla, plus gentoo is not an easy host for k8s | 11:39 |
*** Mr_Broken has quit IRC | 11:40 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Sync the local_settings.j2 with horizon https://review.openstack.org/356471 | 11:42 |
*** haplo37__ has joined #openstack-kolla | 11:42 | |
*** papacz has joined #openstack-kolla | 11:48 | |
*** dwalsh has joined #openstack-kolla | 11:49 | |
*** ayoung has quit IRC | 11:50 | |
kbaegis | Dang | 11:56 |
kbaegis | iptables are still breaking my deployment | 11:56 |
*** papacz has quit IRC | 11:56 | |
kbaegis | traceback, for any and all interested http://hastebin.com/qikeqiboza.py | 11:57 |
*** Mr_Broken has joined #openstack-kolla | 11:59 | |
*** dave-mccowan has joined #openstack-kolla | 11:59 | |
kbaegis | Well, this looks like it's entirely neutron's fault | 12:05 |
kbaegis | and the iptables_manager code | 12:05 |
*** kbaegis has quit IRC | 12:16 | |
*** kbaegis has joined #openstack-kolla | 12:16 | |
*** haplo37__ has quit IRC | 12:21 | |
*** berendt has quit IRC | 12:22 | |
*** papacz has joined #openstack-kolla | 12:24 | |
*** bootsha has joined #openstack-kolla | 12:28 | |
*** papacz has quit IRC | 12:29 | |
*** bootsha has quit IRC | 12:34 | |
*** athomas has quit IRC | 12:34 | |
*** Mr_Broken has quit IRC | 12:34 | |
*** Mr_Broken has joined #openstack-kolla | 12:35 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Use dumb-init to manage the pid 1 process https://review.openstack.org/357015 | 12:39 |
*** athomas has joined #openstack-kolla | 12:39 | |
*** egonzalez90 has quit IRC | 12:40 | |
*** berendt has joined #openstack-kolla | 12:41 | |
*** zhurong has joined #openstack-kolla | 12:45 | |
*** imcsk8_ is now known as imcsk8 | 12:52 | |
*** mnasiadka has joined #openstack-kolla | 12:54 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Add pull.yml file for tempest https://review.openstack.org/357185 | 12:54 |
*** b_bezak has quit IRC | 13:00 | |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO_NOT_MERGE: Use better URL for China https://review.openstack.org/329060 | 13:01 |
*** b_bezak has joined #openstack-kolla | 13:01 | |
*** v1k0d3n has joined #openstack-kolla | 13:01 | |
*** papacz has joined #openstack-kolla | 13:03 | |
*** papacz has quit IRC | 13:06 | |
kbaegis | Where is kolla pulling the /etc/ directory from? I don't see it mapped as a volume | 13:10 |
kbaegis | But kolla start is taking those from a file, correct? | 13:11 |
kbaegis | copying into /etc? | 13:11 |
kbaegis | nvm. also in /etc :) | 13:11 |
*** inc0 has joined #openstack-kolla | 13:12 | |
inc0 | good morning | 13:13 |
kbaegis | HAHA! | 13:13 |
kbaegis | Got it. :) | 13:13 |
kbaegis | inc0: Good morning | 13:13 |
kbaegis | Okay, so I need to write a post of my adventures getting kolla-on-gentoo. Created my first instance, and required me to run the 'noop' firewall driver | 13:14 |
kbaegis | But it's up and running with ovs. Created instances, volumes, networks, etc | 13:14 |
openstackgerrit | Ross Krumbeck proposed openstack/kolla: Ansible-ize OpenStack Designate https://review.openstack.org/353261 | 13:15 |
*** thumpba has joined #openstack-kolla | 13:17 | |
*** berendt has quit IRC | 13:18 | |
*** sdake has joined #openstack-kolla | 13:21 | |
sdake | pbourke around? | 13:21 |
sdake | re https://review.openstack.org/#/c/356181/7 | 13:21 |
patchbot | sdake: patch 356181 - kolla - Simplify install process for faster quickstart exe... | 13:21 |
sdake | i really want to get this merged | 13:21 |
pbourke | sdake: hey | 13:21 |
kbaegis | sdake: Sorry to hear about your laptop. Also, works! Just spun up 32 networked instances on kolla/gentoo | 13:21 |
sdake | you said "Clean up the first note about branches" | 13:23 |
sdake | do you mean the pip pin? | 13:23 |
pbourke | sdake: i meant the initial note of master vs stable | 13:23 |
pbourke | sdake: but yeah the pin needs to be removed also | 13:23 |
sdake | i think that should say | 13:23 |
sdake | stay | 13:23 |
*** mntdevops has joined #openstack-kolla | 13:23 | |
sdake | the first part | 13:23 |
sdake | we can remove the pin tho - i just tested that works correctly | 13:23 |
sdake | i had thought we were pubhlsing milestone releases to pypyi | 13:24 |
pbourke | ok | 13:24 |
sdake | you can see my rationale in the review | 13:24 |
sdake | please can you respond to it | 13:25 |
sdake | so we can ge tthtis merged this morning | 13:25 |
sdake | if you disagree strongly and it will block the commit then i'll just accept what you want :) | 13:25 |
sdake | this causes me serious time sink | 13:26 |
sdake | and leon hs done a fantastic job writing up the solution | 13:26 |
sdake | although i sort of f igured out all the stuff along with your help and dhellman and jrolls' help :) | 13:26 |
sdake | the part that cuses e time sink is the support on this point | 13:26 |
sdake | not reviewing... | 13:27 |
sdake | jeffrey4l need your email address | 13:28 |
sdake | _coolsvap_ need your email address | 13:28 |
pbourke | sdake: as soon as the pin is removed +2 | 13:28 |
sdake | my outlook hasn't cached it yet | 13:28 |
Jeffrey4l | sdake, zhang.lei.fly@gmail.com | 13:28 |
sdake | pbourke great thanks | 13:28 |
*** egonzalez90 has joined #openstack-kolla | 13:28 | |
*** ad_rien_ has quit IRC | 13:30 | |
*** bootsha has joined #openstack-kolla | 13:31 | |
*** daneyon has joined #openstack-kolla | 13:31 | |
sdake | btw foolks my laptop is busted | 13:31 |
sdake | i am working on a 3rd laptop i have available to me | 13:31 |
sdake | i am a little bit impaired as a result | 13:32 |
sdake | no idea how long my other 2 laptops will take to get repaired | 13:32 |
sdake | probably after 8/31 | 13:32 |
*** ad_rien_ has joined #openstack-kolla | 13:33 | |
*** ad_rien_ has quit IRC | 13:34 | |
*** jtriley has joined #openstack-kolla | 13:34 | |
*** bootsha has quit IRC | 13:35 | |
sdake | damn missed chuck on cnbc | 13:35 |
*** daneyon has quit IRC | 13:36 | |
_coolsvap_ | sdake: me@coolsvap.net | 13:36 |
openstackgerrit | Merged openstack/kolla: Customizations for trove https://review.openstack.org/346342 | 13:38 |
sdake | brb | 13:39 |
openstackgerrit | Merged openstack/kolla: Update kubectl version at magnum-conductor container https://review.openstack.org/352986 | 13:39 |
*** caowei has joined #openstack-kolla | 13:40 | |
*** _coolsvap_ is now known as coolsvap | 13:42 | |
*** sdake has quit IRC | 13:43 | |
openstackgerrit | Merged openstack/kolla: Customizations for aodh https://review.openstack.org/346299 | 13:47 |
sbezverk | Jeffrey4l: ping | 13:48 |
Jeffrey4l | sbezverk, pong | 13:48 |
sbezverk | Jeffrey4l: in kube there is no way to use socket | 13:49 |
Jeffrey4l | sbezverk, socket? or ip:port? | 13:49 |
sbezverk | ip:port | 13:49 |
sbezverk | sorry | 13:49 |
sbezverk | because there is no way to know in advance which ip will be used | 13:49 |
Jeffrey4l | sbezverk, why? just treat it as a normal api service? | 13:50 |
sbezverk | and ovs does not support 0.0.0.0:port | 13:50 |
Jeffrey4l | sbezverk, hmm. isn't it? | 13:50 |
*** kbaegis has quit IRC | 13:50 | |
Jeffrey4l | let me test the 0.0.0.0:port format. | 13:50 |
Jeffrey4l | or do you have any confirm link? | 13:50 |
sbezverk | also this option is supported: ovsdb_connection = unix:/var/run/openvswitch/db.sock | 13:51 |
*** sdake has joined #openstack-kolla | 13:51 | |
sbezverk | at least some folks from neutron team think so | 13:51 |
*** magnumbonum has joined #openstack-kolla | 13:51 | |
Jeffrey4l | sbezverk, socket is slow and the ip:port format is fast. | 13:51 |
openstackgerrit | Merged openstack/kolla: Fix murano notification driver https://review.openstack.org/355979 | 13:51 |
inc0 | coolsvap, pbourke any progress on rally front? | 13:51 |
Jeffrey4l | so the ip:port became the default configuration in neutron. | 13:52 |
pbourke | inc0: none from me, did you get anywhere yesterday? | 13:52 |
Jeffrey4l | sbezverk, yep. it support the both format: socket and ip:port | 13:52 |
Jeffrey4l | two different drivers. | 13:52 |
sbezverk | Jeffrey4l: well, but for kube environment we need to use socket | 13:52 |
inc0 | coolsvap was playing with it, there were some errors | 13:52 |
sdake | i hyink we were waiting on some undefined misconfigurations that coolsvap was going to fix | 13:52 |
coolsvap | inc0: not much was bit occupied today we need to split and work on it individually | 13:52 |
sbezverk | so I am confused by your -2 when it has no impact on kolla?? | 13:52 |
inc0 | ok | 13:52 |
coolsvap | sdake: yes | 13:52 |
Jeffrey4l | sbezverk, my question is: why the ip:port not work? | 13:53 |
coolsvap | i did the validation yesterday | 13:53 |
sdake | coolsvap write down a list ofissues | 13:53 |
sdake | so others know whats wrong | 13:53 |
coolsvap | many default scenarios cannot be validated | 13:53 |
inc0 | Jeffrey4l, sbezverk can you plz link review in question? | 13:53 |
magnumbonum | I am trying out creating nested templates with python-heatclient. When I try to reference a local file, it fails with ERROR: Resource could not be loaded | 13:53 |
*** kbaegis has joined #openstack-kolla | 13:53 | |
magnumbonum | I am not running on the control host, but from an admin workstation... should this work? | 13:54 |
sdake | magnumbonum did you try #openstack-heat? | 13:54 |
sbezverk | Jeffrey4l: because when I tried to use it it was crashing generating tracebacks for invalid ip address | 13:54 |
Jeffrey4l | sbezverk, there is no orchestration_engine variable in the build stage. | 13:54 |
sdake | or rather #heat | 13:54 |
Jeffrey4l | inc0, https://review.openstack.org/356827 | 13:54 |
magnumbonum | sdake yup no answer | 13:54 |
*** papacz has joined #openstack-kolla | 13:54 | |
Jeffrey4l | so yes. it break the ansible now. | 13:54 |
sbezverk | ovsdb server fails to start when configured to listen on a wildcard ip | 13:54 |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Missing hosts at haproxy in site.yml https://review.openstack.org/355861 | 13:54 |
sdake | magnumbonum well we dont support heat here ;) | 13:54 |
inc0 | sbezverk, Jeffrey is correct, you can't put this if into build | 13:54 |
magnumbonum | I am thinking along the lines that it is related to kolla / docker etc | 13:55 |
Jeffrey4l | sbezverk, OK. let me check this first. `ovsdb server fails to start when configured to listen on a wildcard ip` | 13:55 |
sdake | magnumbonum but yes, I would expect that to work - perhaps you have an environmental problem | 13:55 |
sdake | magnumbonum i use heat in kolla all the time | 13:55 |
sdake | works for me | 13:55 |
sdake | i use resource groups though, not nested stacks | 13:55 |
sbezverk | inc0: why? | 13:55 |
Jeffrey4l | sbezverk, this variable is defined in ansible-playbooks | 13:55 |
inc0 | sbezverk, build should be totally indifferent to orchiestration | 13:55 |
Jeffrey4l | no docker build stage. | 13:56 |
inc0 | if you do this, you're effectively make impossible for us to push images to dockerhub | 13:56 |
Jeffrey4l | these two are totally difference. | 13:56 |
sdake | magnumbonum try using the heat example templates | 13:56 |
*** Mr_Broken has quit IRC | 13:56 | |
sdake | magnumbonum if those dont work - we can talk more :) | 13:56 |
inc0 | as images in dockerhub have no idea what orchiestration tools will be used, nor they should have | 13:56 |
magnumbonum | sdake, the referenced template works if created directly. but not when nesting... | 13:56 |
magnumbonum | sdake: yeah I am typing from examples. :-o | 13:57 |
*** Mr_Broken has joined #openstack-kolla | 13:57 | |
sdake | magnumbonum not sure - we need ot wait for the heat experts to wake up | 13:57 |
inc0 | can you please explain how this is relevant only to k8s? | 13:57 |
inc0 | maybe we can work around it | 13:57 |
sbezverk | inc0: then selection of a user under which this process run must be done by orchestration and NOT hardcoded in docker image | 13:57 |
*** ayoung has joined #openstack-kolla | 13:57 | |
inc0 | sbezverk, that is different discussion, having underprivileged user is one of biggest values in kolla security model | 13:58 |
coolsvap | Now we have the list of scenarios in a file and shell script to iterate through it. | 13:58 |
coolsvap | I have made the image public and changed its name to match in the default scenarios. | 13:58 |
coolsvap | Now we have two options | 13:58 |
coolsvap | 1] Update the shell script to modify values in the scenarios dynamically | 13:58 |
coolsvap | 2] Duplicate the scenario files for different values to benchmark against | 13:58 |
coolsvap | I personally feel #2 is fast and everyone can validate the scenarios as soon as they update it | 13:58 |
coolsvap | Once we create this list, its just running the shell script ove rand over again | 13:58 |
sdake | magnumbonum please join #heat so we can see if i can get anyone's attention | 13:58 |
magnumbonum | sdake: i am in there... | 13:58 |
inc0 | if we make k8s much less secure than ansible, we need to know why | 13:58 |
sbezverk | inc0: have you read PS notes? I think it explain pretty clear why | 13:58 |
*** ad_rien_ has joined #openstack-kolla | 13:58 | |
sbezverk | I mean commit message | 13:59 |
inc0 | yes, I've read this, but why is this happening on k8s and not ansible? | 13:59 |
coolsvap | inc0: sdake pbourke ^^ | 13:59 |
sbezverk | inc0: because with ansible you can use ip:port to start ovsdb process | 14:00 |
coolsvap | if we take 1-2 services each its a 1-2 hr job to get all scenarios upto date and validated | 14:00 |
sbezverk | inc0: in kube I have to use named socket for ovsdb as there is no way to know in advance when config gets built which ip address will be used by the compute node | 14:00 |
sdake | dave-mccowan do you have any tricks for getting irc working insdide the vpn? | 14:00 |
inc0 | sbezverk, and if you run ovsdb with net=host, will that help? | 14:00 |
inc0 | I see | 14:01 |
sbezverk | inc0: we already do it | 14:01 |
inc0 | hmm, let me think about this for a moment | 14:01 |
*** Mr_Broken has quit IRC | 14:01 | |
sbezverk | but since we want to leverage kube dynamic nature and scale of/off | 14:01 |
*** papacz has quit IRC | 14:01 | |
sbezverk | without any intervention from orcestration | 14:02 |
inc0 | so Jeffrey4l is totally right, build phase shouldn't have any notion of orchiestration engine | 14:02 |
Jeffrey4l | sbezverk, is it possible to regenerate the configure file before the openvswitch is deployed or moved? | 14:02 |
*** b_bezak has quit IRC | 14:02 | |
sbezverk | guys, we want to use kube because it is DYNAMIC!!! | 14:02 |
sbezverk | we do not want to do anything when new compute node gets added/removed | 14:03 |
inc0 | sbezverk, computes can't be that dynamic anyway | 14:03 |
sbezverk | yes they can | 14:03 |
sdake | sbezverk got it -but docker file is not place to put orchestration variable imo | 14:03 |
sbezverk | we use daemon sets | 14:03 |
kbaegis | Thanks kolla team! :) http://pasteboard.co/a96c8uI6N.png | 14:03 |
inc0 | if k8s removes a compute but VMs are still there, you're in a world of pain | 14:03 |
Jeffrey4l | sbezverk, i means when the openvswitch container get deployed or moved. not the nodes. | 14:03 |
sdake | kbaegis thats gentoo? | 14:04 |
sbezverk | ok for removal you are right, but for adding it can totally work | 14:04 |
sdake | kbaegis what was the problem | 14:04 |
*** duonghq has joined #openstack-kolla | 14:04 | |
*** caowei has quit IRC | 14:04 | |
inc0 | sbezverk, adding will be no different than deployment | 14:05 |
inc0 | in any case, you need ovs on control nodes too | 14:05 |
inc0 | network nodes rather | 14:05 |
sbezverk | inc0: you think in terms of ansible | 14:06 |
kbaegis | sdake Yes :D | 14:06 |
inc0 | so let me ask you this sbezverk, ovsdb starts to listen on ip:port, but vswitchd seems to use sock | 14:06 |
*** huikang has joined #openstack-kolla | 14:06 | |
kbaegis | Well, I had to set firewall_driver (ml2) to noop | 14:06 |
sbezverk | but in kube all compute nodes are netowrk nodes and compute nodes | 14:06 |
kbaegis | Huge issue. You guys already fixed the mount --make-shared /run | 14:06 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update heat base Dockerfile to include footers https://review.openstack.org/357240 | 14:06 |
inc0 | sbezverk, that won't work like you think | 14:06 |
*** caowei has joined #openstack-kolla | 14:06 | |
kbaegis | And make sure the correct packages/use-flags and kernel parameters were configured | 14:06 |
inc0 | router still has one IP | 14:07 |
sbezverk | inc0: no ovsdb start with named socket because ip:port cannot be used | 14:07 |
inc0 | routes will still be A-P | 14:07 |
duonghq | kbaegis: did you try deploy multinode with non-root user? | 14:07 |
kbaegis | Also, openvswitch has to be run on the host, not in a container. | 14:07 |
kbaegis | duonghq: Not yet | 14:07 |
mntdevops | duonghq: is it possible to deploy certain containers as non-root? | 14:08 |
*** Mr_Broken has joined #openstack-kolla | 14:08 | |
inc0 | sbezverk, it cannot be used because of net=host? | 14:08 |
mntdevops | like | 14:08 |
Jeffrey4l | sbezverk, why u get the permission issue when using socket? | 14:08 |
sbezverk | inc0: no because we do not know which ip address to use at the time config gets generated | 14:08 |
mntdevops | can you run nova-compute as a regular user? | 14:08 |
*** ad_rien_ has quit IRC | 14:08 | |
Jeffrey4l | sbezverk, ansible never hit such issue. | 14:08 |
duonghq | mntdevops: ansible has become option, so the ansible_ssh_user do not need to be root | 14:08 |
duonghq | I mean the user use to connect to nodes | 14:09 |
mntdevops | ah, right | 14:09 |
sbezverk | Jeffrey4l: why you keep refering to ansible?!?! we are talking about different operating environment. | 14:09 |
inc0 | sbezverk, and shared socker has perms issues? can you change perms of this socket? | 14:09 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update heat base Dockerfile to include footers https://review.openstack.org/357240 | 14:09 |
Jeffrey4l | sbezverk, ya. So what permission issue you get? | 14:09 |
inc0 | sbezverk, ansible is relevant | 14:09 |
duonghq | mntdevops: did you try this? | 14:10 |
mntdevops | duonghq: are you having an issue with the playbooks running on a non-local system during deploy? | 14:10 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update openvswitch base Dockerfile to include footers https://review.openstack.org/357243 | 14:10 |
Jeffrey4l | before one month ago, ansible use the socket for communication only. I just want to the details of the issue. | 14:10 |
Jeffrey4l | i admit i am not familiar with the kube. | 14:10 |
mntdevops | duonghq: I look forward to my current work project being done this week to have time to go multi-node next week | 14:11 |
sbezverk | inc0: I though about changing permissions for the socket, but that would require extra step in bootstrap process or modification of extended_start | 14:11 |
Jeffrey4l | But any change in the image may break the ansible deploy. So we should be careful with that. | 14:11 |
inc0 | sbezverk, so modification of extended start is totally ok | 14:11 |
mntdevops | kolla is a hobby project of mine right now | 14:11 |
inc0 | you can inject orchiestration engine info there | 14:11 |
inc0 | via ENV | 14:11 |
duonghq | mntdevops: yeah, it cannot create /etc/kolla/... without enough priviledge, but allow root remote access is not so wise | 14:11 |
inc0 | on runtime | 14:11 |
duonghq | ansible has become for these purpose | 14:12 |
duonghq | *become options | 14:12 |
inc0 | so just put this perm change in if $K8S | 14:12 |
duonghq | but in current playbook, there is no become yet | 14:12 |
sbezverk | inc0: ok I thought you guys will not accept it | 14:12 |
pbourke | coolsvap: I can try updating the scripts now | 14:12 |
duonghq | mntdevops: am I wrong? | 14:12 |
inc0 | nah, I think it's cool | 14:12 |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update ceilometer base Dockerfile to include footers https://review.openstack.org/357245 | 14:13 |
pbourke | coolsvap: i just had one question though, how do we decide the values to update | 14:13 |
*** ad_rien_ has joined #openstack-kolla | 14:13 | |
Jeffrey4l | sbezverk, i really want to make clear what the issue is. could u explain it more? | 14:13 |
inc0 | just make sure to get privileges right, not 777 unless it's totally required | 14:13 |
mntdevops | duonghq: I'm very familiar with ansible, so I know what those options do. You're probably not wrong. I just haven't run into those issues yet | 14:13 |
coolsvap | pbourke: have you seen the scenario files? | 14:13 |
pbourke | coolsvap: ya | 14:13 |
Jeffrey4l | why u get permission issue when using socket? | 14:13 |
coolsvap | lets jump to tmux | 14:13 |
coolsvap | pbourke: do you have time? | 14:13 |
pbourke | coolsvap: yes | 14:14 |
sbezverk | inc0: it should be 755 | 14:14 |
pbourke | just getting connected | 14:14 |
sbezverk | as process need to read/write | 14:14 |
duonghq | mntdevops: yeah, I need some more people confirm my observe, | 14:14 |
mntdevops | duonghq: in my experience with nested roles and includes, sometimes handlers and includes don't automatically "become: yes" "become_user: root" even though the parent does | 14:14 |
inc0 | sbezverk, but also...I vaguely remember that ovs did indeed use socket | 14:14 |
inc0 | as Jeffrey4l pointed ot | 14:14 |
inc0 | so what is different? | 14:15 |
pbourke | coolsvap: why are the default values not ok | 14:15 |
duonghq | mntdevops: can you grep through the kolla source, I cannot find any "become" | 14:15 |
coolsvap | pbourke: it will not test the cluster we got | 14:15 |
mntdevops | duonghq: I've had plays to hit ec2 resources that call roles. Inside those roles, the tasks run as root if needed but the handlers needed "help" | 14:15 |
sbezverk | inc0: when I start ovsdb with punix:/blah/blah socket it becomes compute node config agnostic | 14:15 |
pbourke | coolsvap: why not | 14:15 |
mntdevops | duonghq: stand by | 14:15 |
Jeffrey4l | sorry need to out, brb. | 14:16 |
duonghq | mntdevops: ok, thanks | 14:16 |
sbezverk | inc0 you follow me? | 14:16 |
inc0 | sbezverk, wich is cool | 14:16 |
openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 14:16 |
coolsvap | it benchmarks on the values 10-20 instances/volumes and concurrency level of single digit | 14:16 |
openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 14:16 |
coolsvap | we need it to be in 100s/10000s imo | 14:16 |
inc0 | https://github.com/openstack/kolla/blob/stable/mitaka/ansible/roles/neutron/templates/openvswitch-db-server.json.j2 so look, there is no notion of IP | 14:16 |
sbezverk | inc0: for kube env this level of lfexibility is requirement, otherwise you just kill it | 14:16 |
inc0 | no, you make a pod with ovsdb and vswitchd | 14:17 |
coolsvap | so my idea was to create 2-3 set of template copies with these different values | 14:17 |
inc0 | they will have to be on same host but trust me, that's how you want it | 14:17 |
inc0 | you don't want ovsdb to be somewhere else | 14:17 |
inc0 | it's only relevant to local ovs | 14:17 |
coolsvap | and just run those everytime so that we are testing on a constant values and see the results | 14:18 |
sbezverk | inc0: correct each node will be running local copy of ovsdb | 14:18 |
sdake | kbaegis sounds like a whole lot of caveats | 14:18 |
inc0 | ok, so what's the issue with sock? | 14:18 |
inc0 | it used to work | 14:18 |
pbourke | coolsvap: ok lets do that | 14:18 |
kbaegis | sdake Yeah, big ones too | 14:19 |
sbezverk | inc0: when it gets created by ovsdb process sock has permissions 700 | 14:19 |
mntdevops | duonghq: you're right: I don't see it in the source. However, you can edit your inventory file to add per-host ansible options: http://docs.ansible.com/ansible/become.html | 14:19 |
sbezverk | inc0 ovsdb process runs as root | 14:19 |
mntdevops | that *should* work | 14:20 |
sdake | sbezverk extensions to run scripts are fine | 14:20 |
coolsvap | pbourke: okay I will start with nova then | 14:20 |
inc0 | ok, in stable/mitaka we didn't drop-root ovs | 14:20 |
duonghq | mntdevops: I'm not really comfort with this workaround | 14:20 |
mntdevops | duonghq: I'd add ansible_become=yes ansible_become_user=root ansible_ssh_user=mysshuser | 14:21 |
duonghq | mntdevops: because many groups need write to /etc/kolla, it require root | 14:21 |
mntdevops | what makes you uncomfortable with editing your inventory file? | 14:21 |
sbezverk | inc0: neutron-openvswitch-agent runs as neutron hence fails accessing this socket | 14:21 |
mntdevops | so that when ansible runs on that host, it knows how to sudo when needed? | 14:22 |
duonghq | mntdevops: it will give the become for whole host, but I prefer just give to neccesary task | 14:22 |
inc0 | sbezverk, ...no...stable/mitaka works... | 14:22 |
inc0 | https://github.com/openstack/kolla/blob/stable/mitaka/docker/neutron/neutron-l3-agent/Dockerfile.j2 | 14:22 |
inc0 | look | 14:22 |
inc0 | in stable/mitaka there is nothing about IP | 14:23 |
inc0 | it's all over sockets | 14:23 |
mntdevops | duonghq: makes sense. I thought it was just a matter of getting it to work first. then getting it locked down | 14:23 |
duonghq | mntdevops: agree with you | 14:23 |
mntdevops | duonghq: I figured it was a roadblock for you | 14:23 |
duonghq | something like that | 14:24 |
mntdevops | duonghq: it would mean editing each role that needs it | 14:24 |
mntdevops | at the start of each role | 14:24 |
sbezverk | inc0: I am not here to explain why stable/mitaka works, I here to show that permissions for db.sock do not allow neutron-openvswitch to access it | 14:24 |
mntdevops | it gets messy really quickly adding it to specific tasks only | 14:24 |
inc0 | sbezverk, yet mitaka works...so we need to figure out what changed | 14:24 |
inc0 | because if you can get it to work same way as mitaka | 14:24 |
inc0 | you're good | 14:24 |
duonghq | thank you mntdevops | 14:24 |
sbezverk | inc0: have you tried to run stable/mitaka on kube cluster? | 14:25 |
inc0 | sbezverk, kube doesn't matter - this is low level container runtime | 14:25 |
duonghq | sdake, can you review this bp: https://blueprints.launchpad.net/kolla/+spec/ansible-become-support I think it is quite importance bp | 14:25 |
inc0 | and it has nothing to do with IP kube will give it | 14:25 |
mntdevops | duonghq: I hope I've helped | 14:26 |
mntdevops | if not, no worries | 14:26 |
*** kangh_ has joined #openstack-kolla | 14:26 | |
duonghq | mntdevops: your help is valuable to me | 14:27 |
*** haplo37__ has joined #openstack-kolla | 14:28 | |
mntdevops | duonghq: well, I hope it helps me next week, too :-) | 14:28 |
duonghq | :) | 14:28 |
sbezverk | inc0: btw here is the line used to start ovsdb in master: /usr/sbin/ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err -vfile:info --remote=punix:/run/openvswitch/db.sock --remote=ptcp:6640:$ovsdb_ip --log-file=/var/log/kolla/openvswitch/ovsdb-server.log | 14:28 |
openstackgerrit | Leon Zachery proposed openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 | 14:28 |
inc0 | ok, I know that it uses remote, you can use different line in k8s | 14:29 |
inc0 | this is config of runtime | 14:29 |
*** ad_rien_ has quit IRC | 14:29 | |
inc0 | what I mean is: -remote=punix:/run/openvswitch/db.sock does work in stable/mitaka yet it doesn't work in your case | 14:30 |
*** ad_rien_ has joined #openstack-kolla | 14:30 | |
inc0 | if you can get it to work, k8s will be fine | 14:30 |
*** huikang has quit IRC | 14:30 | |
sbezverk | inc0: if you have kolla running handly could you check the actual permission | 14:30 |
sbezverk | on the socket file? | 14:30 |
inc0 | yeah, let me log in to OSIC | 14:31 |
pbourke | inc0: is the osic deploy broken | 14:31 |
pbourke | inc0: to your knowledge | 14:31 |
coolsvap | it was working yesterday when I configured rally and validated scenarios | 14:32 |
inc0 | sbezverk, root@729579-comp-s3700-027:~# docker exec -it openvswitch_db ls -la /run/openvswitch/db.sock | 14:34 |
inc0 | srwxr-x--- 1 root root 0 Aug 16 10:06 /run/openvswitch/db.sock | 14:34 |
*** sdake has quit IRC | 14:34 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update openvswitch base Dockerfile to include footers https://review.openstack.org/357243 | 14:36 |
*** ad_rien_ has quit IRC | 14:36 | |
openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update ceilometer base Dockerfile to include footers https://review.openstack.org/357245 | 14:36 |
pbourke | coolsvap: disk seems full | 14:36 |
sbezverk | inc0: same thing I have | 14:37 |
sbezverk | inc0: docker exec neutron-openvswitch-agent whoami | 14:37 |
inc0 | sbezverk, but we need to check stable mitaka for that | 14:37 |
*** ad_rien_ has joined #openstack-kolla | 14:38 | |
egonzalez90 | coolsvap: i'll prefer to push all base footers in a single commit, that will be easier to review | 14:38 |
sbezverk | inc0: would you agree that under normal circumstances neutron use will not have access to this sock file? | 14:38 |
sbezverk | I meant neutron user. | 14:39 |
openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 14:39 |
*** kbaegis has quit IRC | 14:39 | |
inc0 | so not exactly, this is how ovs-agent talks to ovs | 14:39 |
inc0 | via db | 14:39 |
inc0 | and socket is exactly right way to do it | 14:39 |
inc0 | what I would agree is that normally neutron and ovs are run by same used | 14:40 |
inc0 | user | 14:40 |
sbezverk | inc0: but you need permissions to open it and neutron user has none | 14:40 |
inc0 | sbezverk, yeah...yet it used to work right? | 14:40 |
*** kbaegis has joined #openstack-kolla | 14:42 | |
sbezverk | inc0: I guess but it does not help much.. | 14:44 |
inc0 | it should, because all we need to do in k8s is to mirror this exact behaviour | 14:45 |
kbaegis | Is there a convenient way to change the hostname for novncproxy? | 14:45 |
Jeffrey4l | sbezverk, i tried: `/usr/sbin/ovsdb-server /etc/openvswitch/conf.db --remote=ptcp:6641:0.0.0.0` it works | 14:46 |
kbaegis | I can just change novncproxy_base_url = right? | 14:47 |
Jeffrey4l | the 0.0.0.0 can be omitted. | 14:47 |
*** ajpower has quit IRC | 14:49 | |
sbezverk | Jeffrey4l: I will re-try but still neutron-openvswitch-agent needs to communicate with ovsdb via named socket | 14:50 |
kbaegis | I think it's vncserver_proxyclient_address, novncproxy_base_url if I want to hide the vncproxy behind an apache proxy, right? | 14:50 |
sbezverk | as ml2_conf.ini will not have actual kube compute node ip address in advance | 14:50 |
Jeffrey4l | sbezverk, if you change the ovsdb_connection and ovsdb_interface=native, it will communicate though tcp port. | 14:51 |
Jeffrey4l | and i still confused the permission issue. it shouldn't be happen. :( | 14:51 |
Jeffrey4l | sbezverk, here is the related option in neutron. https://github.com/openstack/neutron/blob/master/neutron/agent/ovsdb/api.py#L35 | 14:52 |
coolsvap | pbourke: rally task list has the option to filter by deployment uuid | 14:52 |
pbourke | coolsvap: im thinking seen as there's one deployment all these tasks should go into it by default | 14:52 |
pbourke | coolsvap: maybe not though | 14:52 |
coolsvap | currently they will | 14:53 |
coolsvap | but if we have multiple deployments then | 14:53 |
sbezverk | Jeffrey4l: neutron-openvswitch-agent at least at this point cannot communicate via tcp port because see my explanation above | 14:53 |
coolsvap | pbourke: http://rally.readthedocs.io/en/latest/cli/cli_reference.html#rally-task-list | 14:53 |
openstackgerrit | Paul Bourke proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 | 14:54 |
inc0 | pbourke, I'll put bug about this elastic | 14:55 |
Jeffrey4l | sbezverk, sorry i can not find the point. would u mind to copy them again? | 14:55 |
pbourke | inc0: thanks | 14:55 |
inc0 | we need to document that /var/lib/docker needs a lot of space | 14:55 |
*** mnasiadka has quit IRC | 14:57 | |
sbezverk | Jeffrey4l: ml2_conf.ini gets generated offline and it points to vip address. we do not know in advance which ip actually will be used on kube compute node | 14:57 |
sbezverk | so we need to use something generic which will work in every situation, which is named socket | 14:58 |
egonzalez90 | kbaegis: proxyclient_address is the api_interface IP address, base_url is the URL nova will give you to access from a browser, in kolla points to kolla_external_fqdn | 14:58 |
pbourke | coolsvap: we should generate json + html incase html can't be generated from the json after the fact | 14:58 |
inc0 | https://bugs.launchpad.net/kolla/+bug/1614572 | 14:58 |
openstack | Launchpad bug 1614572 in kolla "Elastic search eats up space quickly" [Medium,Triaged] | 14:58 |
egonzalez90 | to hide the address you only need to change base_url at compute nodes | 14:59 |
*** michauds has joined #openstack-kolla | 15:00 | |
Jeffrey4l | sbezverk, ok. | 15:00 |
*** mewald has quit IRC | 15:00 | |
*** salv-orlando has joined #openstack-kolla | 15:00 | |
openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible role https://review.openstack.org/351294 | 15:01 |
coolsvap | pbourke: yes I always do both | 15:03 |
inc0 | pbourke, coolsvap lots of "unauthorized" Traces | 15:03 |
inc0 | seems like rally issue? | 15:03 |
coolsvap | inc0: default scenarios have hardcoded values | 15:04 |
pbourke | can't see whats wrong with that neutron scenario | 15:07 |
*** Mr_Broken has quit IRC | 15:08 | |
*** Guest52040 has joined #openstack-kolla | 15:08 | |
*** Mr_Broken has joined #openstack-kolla | 15:08 | |
*** sdake has joined #openstack-kolla | 15:11 | |
sdake | yo | 15:11 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Customizations for Multipathd https://review.openstack.org/354421 | 15:12 |
duonghq | sdake: can you review my bp: https://blueprints.launchpad.net/kolla/+spec/ansible-specific-task-become | 15:12 |
*** Mr_Broken has quit IRC | 15:13 | |
*** magnumbonum has quit IRC | 15:13 | |
duonghq | still do not get used to different timezone at IRC :( | 15:14 |
duonghq | sdake: how about your mac? | 15:15 |
* coolsvap needs to step out for dinner | 15:16 | |
*** coolsvap is now known as _coolsvap_ | 15:16 | |
duonghq | _coolsvap_ quite late for dinner? | 15:16 |
*** zhubingbing has quit IRC | 15:17 | |
*** ajpower has joined #openstack-kolla | 15:17 | |
*** sdake has quit IRC | 15:17 | |
*** michauds has quit IRC | 15:19 | |
*** sdake has joined #openstack-kolla | 15:19 | |
duonghq | sdake: how about your mac? | 15:19 |
*** vbel has quit IRC | 15:20 | |
*** vbel has joined #openstack-kolla | 15:20 | |
sdake | what about it | 15:21 |
sdake | i have like 6 macs at home | 15:21 |
sdake | i keep spar egear around in case of emergency | 15:21 |
duonghq | really? | 15:22 |
sdake | yup - wife in high tech too | 15:23 |
sdake | we are pretty tortourous on our gear | 15:23 |
duonghq | oh | 15:23 |
*** dave-mccowan has quit IRC | 15:24 | |
*** dave-mccowan has joined #openstack-kolla | 15:24 | |
duonghq | your amp arrived? | 15:24 |
*** sdake has quit IRC | 15:24 | |
*** sdake has joined #openstack-kolla | 15:25 | |
duonghq | sdake: you're blinking? | 15:25 |
sdake | i'm going back to bed | 15:25 |
duonghq | huh? | 15:26 |
sdake | duonghq busy atm trying to setup my new laptop | 15:26 |
duonghq | ah, sorry | 15:26 |
egonzalez90 | were you able to vagrant up kolla in mac with virtualbox? | 15:27 |
egonzalez90 | always getting error with ssh key | 15:28 |
*** vhosakot has joined #openstack-kolla | 15:28 | |
Pavo | egonzalez90 is this your site http://egonzalez.org/openstack-kolla-deployment-from-rdo-packages/ | 15:28 |
*** sdake_ has joined #openstack-kolla | 15:29 | |
egonzalez90 | pavo: yes, probably there are a lot of errors in there | 15:29 |
Pavo | yeah I found that out | 15:30 |
*** hieulq_ has joined #openstack-kolla | 15:31 | |
sdake_ | wirehead_ about? | 15:33 |
*** zhurong has quit IRC | 15:34 | |
*** papacz has joined #openstack-kolla | 15:37 | |
*** sdake has quit IRC | 15:41 | |
*** HyperJohnGraham_ has joined #openstack-kolla | 15:42 | |
*** duonghq has left #openstack-kolla | 15:43 | |
*** HyperJohnGraham has quit IRC | 15:44 | |
*** HyperJohnGraham_ is now known as HyperJohnGraham | 15:44 | |
*** jrich523 has joined #openstack-kolla | 15:44 | |
openstackgerrit | Merged openstack/kolla: Simplify install process for faster quickstart execution https://review.openstack.org/356181 | 15:50 |
openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Implement mongodb replication set cluster https://review.openstack.org/345751 | 15:51 |
*** michauds has joined #openstack-kolla | 15:52 | |
openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Remove unnecessary host at haproxy in site.yml https://review.openstack.org/355861 | 15:52 |
Pavo | keep getting ERROR:kolla.image.build:neutron-networking-sfc-agent Failed with status: error | 15:55 |
Pavo | guessing its because of ERROR:kolla.image.build.neutron-networking-sfc-agent:The command '/bin/sh -c echo 'binary not yet available for centos' && /bin/false' returned a non-zero code: 1 | 15:56 |
egonzalez90 | Pavo: review link? | 15:58 |
*** dwalsh has quit IRC | 15:58 | |
Pavo | think thats normal error since I am building centos images | 15:59 |
sdake_ | that is normal pavo | 15:59 |
Pavo | just kind of weird that it would even try to build images that are not centos based | 15:59 |
sdake_ | that is not an error its a warning | 15:59 |
sdake_ | pavo we don't have a mapping of those things | 15:59 |
sdake_ | since we dont have a mapping, it builds it all | 16:00 |
Pavo | roger that | 16:00 |
sdake_ | we could easily create a mapping using the profiles feature | 16:00 |
Pavo | think I am gonna try and build a repo container and have all nodes poit to it to get their packages | 16:01 |
wirehead_ | sdake_: now I am | 16:01 |
sdake_ | wirehead_ up for jabber | 16:01 |
openstackgerrit | Merged openstack/kolla: Add pull.yml file for tempest https://review.openstack.org/357185 | 16:01 |
Pavo | instead of the internet | 16:01 |
sdake_ | my laptop cratered last night | 16:01 |
jrich523 | i need to drop my laptop | 16:02 |
jrich523 | a few times... | 16:02 |
jrich523 | work wont give me a new one until it breaks | 16:03 |
sdake_ | jrich523 once is all it takes :) | 16:07 |
jrich523 | lol.. im getting pretty close to doing it | 16:07 |
jrich523 | figure a few smashes on the couch should ruin the drive but not show any external damage | 16:08 |
jrich523 | "i dont know, just started to throw errors" | 16:08 |
*** Serlex has quit IRC | 16:08 | |
*** daneyon has joined #openstack-kolla | 16:14 | |
*** dwalsh has joined #openstack-kolla | 16:14 | |
jrich523 | not that you guys probably care, but you can do this now: yum install powershell :) | 16:15 |
jrich523 | or apt-get, whatever | 16:15 |
sdake_ | off to breafast brb | 16:16 |
*** sdake_ has quit IRC | 16:16 | |
*** egonzalez90 has quit IRC | 16:16 | |
*** daneyon has quit IRC | 16:18 | |
sbezverk | inc0: ping | 16:19 |
inc0 | I'm here | 16:19 |
sbezverk | inc0: I want to use the same approach I used in ovsdb, replacing command in ovsdb template by the name of the script, in this case I should be able to get IP address I need and not use socket | 16:21 |
sbezverk | would it be acceptable solution from kolla perspective? | 16:21 |
inc0 | you mean to change command for running ovsdb and vswitchd depending on orch engine? | 16:22 |
inc0 | yeah, thats acceptable | 16:23 |
sbezverk | inc0: ok I will test it and if it resolves socket problem I will abandon previous PS | 16:24 |
*** michauds has quit IRC | 16:25 | |
sbezverk | inc0: this time I will use script for starting neutron-openvswitch-agent and neutron-l3-agent, both need access to ovsdb | 16:25 |
inc0 | ok, that's fair | 16:25 |
openstackgerrit | Merged openstack/kolla: Add support for building vmtp container https://review.openstack.org/354362 | 16:26 |
*** mark-casey has joined #openstack-kolla | 16:28 | |
*** caowei has quit IRC | 16:32 | |
*** hieulq_ has quit IRC | 16:35 | |
kbaegis | So I'm trying to get novnc behind a reverse proxy for ease-of-use | 16:36 |
kbaegis | Anyone have an apache vhost for this already? | 16:36 |
kbaegis | Ideally I'd like it to bind to something on https://<frontend.domain>:443/vnc/<openstackurl> with the actual novnc configuration just being <ip>:6080 | 16:37 |
*** thumpba_ has joined #openstack-kolla | 16:37 | |
kbaegis | I keep my lab on a different network, and it's handy to have a web proxy :) | 16:37 |
kbaegis | I'm editing cd /etc/kolla/nova-novncproxy/nova.conf directly w/ COPY_ALWAYS | 16:38 |
kbaegis | Recommendations? I asked on #openstack-nova and #openstack already too | 16:39 |
*** thumpba has quit IRC | 16:39 | |
*** dwalsh has quit IRC | 16:40 | |
*** unicell has joined #openstack-kolla | 16:41 | |
*** unicell1 has quit IRC | 16:43 | |
*** sdake has joined #openstack-kolla | 16:43 | |
sdake | inc0 ping - can you forward that email to ryan | 16:45 |
sdake | it bounced on my end | 16:45 |
sdake | he isn't on | 16:45 |
sdake | and i dont know what his email address it | 16:45 |
sdake | is | 16:45 |
pbourke | britthou_: inc0: _coolsvap_: im headed off soon, I think I fixed up the auth error in rally so all scenarios are being run through again. Next steps: 1) fix up any remaining failed scenarios 2) tweak the values in each scenario to better excercise the cluster 3) re-do the first deploy scenario on the etherpad and obtain full set of rally results (json+html) | 16:48 |
sdake | i thin kwe need to not burn up alot of time on part #2 | 16:49 |
sdake | i could see that taking weeks | 16:50 |
sdake | _coolsvap_ or if you could forward the mail that would work too | 16:50 |
sdake | pretty epic: http://www.opensym.org/os2016/proceedings-files/p101-schneider.pdf | 16:54 |
*** stingray_ has joined #openstack-kolla | 16:57 | |
*** fragatina has joined #openstack-kolla | 16:59 | |
*** fragatina has quit IRC | 17:00 | |
*** fragatina has joined #openstack-kolla | 17:00 | |
*** kangh_ has quit IRC | 17:02 | |
*** rstarmer has joined #openstack-kolla | 17:10 | |
*** rstarmer has quit IRC | 17:11 | |
*** fragatina has quit IRC | 17:16 | |
*** athomas has quit IRC | 17:25 | |
mark-casey | sdake: that pdf is amazing | 17:26 |
*** unicell has quit IRC | 17:33 | |
*** rstarmer has joined #openstack-kolla | 17:35 | |
*** Guest52040 has quit IRC | 17:40 | |
*** gfidente has quit IRC | 17:43 | |
*** dwalsh has joined #openstack-kolla | 17:46 | |
*** HyperJohnGraham has quit IRC | 17:46 | |
*** dmueller-evoila has joined #openstack-kolla | 17:53 | |
*** unicell has joined #openstack-kolla | 17:55 | |
*** dmueller-evoila has quit IRC | 17:58 | |
*** rstarmer has quit IRC | 18:01 | |
*** daneyon has joined #openstack-kolla | 18:02 | |
*** inc0 has quit IRC | 18:02 | |
*** daneyon has quit IRC | 18:06 | |
*** dmueller-evoila has joined #openstack-kolla | 18:08 | |
*** fragatina has joined #openstack-kolla | 18:11 | |
*** HyperJohnGraham has joined #openstack-kolla | 18:13 | |
*** dwalsh has quit IRC | 18:18 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 | 18:21 |
*** inc0 has joined #openstack-kolla | 18:22 | |
*** rstarmer has joined #openstack-kolla | 18:22 | |
*** rstarmer has quit IRC | 18:24 | |
wirehead_ | Hm. The thanks vs. actually table. | 18:26 |
*** ayoung has quit IRC | 18:37 | |
*** michauds has joined #openstack-kolla | 18:38 | |
*** HyperJohnGraham has quit IRC | 18:43 | |
kbaegis | lol | 18:46 |
kbaegis | I always remember the f*#! you NVIDIA speech | 18:48 |
kfox1111 | sbezverk: have you gotten a chance to test https://review.openstack.org/#/c/356114 ? | 18:57 |
patchbot | kfox1111: patch 356114 - kolla-kubernetes - WIP- openvswitch ip | 18:57 |
kfox1111 | I think its probably ready to take off the WIP. | 18:57 |
kfox1111 | or it could wait until crudini merges. | 18:58 |
*** HyperJohnGraham has joined #openstack-kolla | 19:02 | |
*** stingray_ has quit IRC | 19:02 | |
*** mewald has joined #openstack-kolla | 19:03 | |
*** mewald has left #openstack-kolla | 19:07 | |
*** HyperJohnGraham has quit IRC | 19:08 | |
*** HyperJohnGraham has joined #openstack-kolla | 19:09 | |
*** salv-orl_ has joined #openstack-kolla | 19:15 | |
*** salv-orlando has quit IRC | 19:18 | |
*** vhosakot has quit IRC | 19:18 | |
*** vhosakot has joined #openstack-kolla | 19:18 | |
Daviey | sdake: hey... sorry for the slow response... but RE: Master images. They will probably never be correct on docker's hub, due to each commit rightly needing a new image. Any developer would surely be building their own imaages locally? | 19:20 |
kfox1111 | sbezverk: Just looked at ingress a bit to get a handle on what its doing. | 19:21 |
Daviey | sdake: If we are saying that Master doesn't need a new image per-commit, i strongly disagree. | 19:21 |
kfox1111 | it looks like its mostly just a l7 lb. | 19:21 |
kfox1111 | I'm not sure it buys us anything outside of when running within GCE/GKE. | 19:22 |
kfox1111 | we still need the plubming to get network access into the ingress pod. | 19:22 |
*** dwalsh has joined #openstack-kolla | 19:23 | |
*** vhosakot has quit IRC | 19:23 | |
*** vhosakot has joined #openstack-kolla | 19:24 | |
*** thumpba_ has quit IRC | 19:29 | |
sdake | Daviey - right master will be rebuilt each time | 19:30 |
sdake | by the devs | 19:30 |
sdake | Daviey - perhaps i don't totally understand your idea, but it applies to stable branches? | 19:31 |
sdake | i didn't think it worked well with stable branches when i looked at it - but willing to look again :) | 19:31 |
*** egonzalez90 has joined #openstack-kolla | 19:32 | |
*** dgonzalez has quit IRC | 19:34 | |
openstackgerrit | Merged openstack/kolla: Add crudini https://review.openstack.org/356155 | 19:38 |
*** thumpba has joined #openstack-kolla | 19:39 | |
*** dgonzalez has joined #openstack-kolla | 19:45 | |
inc0 | sdake, crudini...?:P | 19:50 |
inc0 | deja vu? | 19:50 |
sdake | whatever moves things along | 19:50 |
sdake | can clean up the mess later | 19:50 |
sdake | k8s wil take along time to mature just as kolla-ansible did | 19:50 |
*** daneyon has joined #openstack-kolla | 19:50 | |
*** HyperJohnGraham has quit IRC | 19:51 | |
inc0 | yeah, I know | 19:51 |
inc0 | just I've seen this before | 19:51 |
inc0 | all I'm saying;) | 19:51 |
sdake | i've seen the whole patterns couple times | 19:51 |
sdake | but things are slightly different this time environmentally | 19:51 |
*** egonzalez90 has quit IRC | 19:53 | |
*** huikang has joined #openstack-kolla | 19:55 | |
*** daneyon has quit IRC | 19:55 | |
*** egonzalez90 has joined #openstack-kolla | 19:56 | |
*** HyperJohnGraham has joined #openstack-kolla | 19:58 | |
*** dwalsh has quit IRC | 19:59 | |
egonzalez90 | ntpttr: ping | 20:01 |
*** thumpba has quit IRC | 20:04 | |
Pavo | so.... why would my networks go down and not come back up when it gets to the starting Neutron server during deployment? | 20:05 |
*** ayoung has joined #openstack-kolla | 20:09 | |
*** JoseMello has quit IRC | 20:10 | |
ntpttr___ | egonzalez90: hi | 20:11 |
ntpttr___ | egonzalez90: sorry my IRC bouncer is experiencing some technical difficulties at the moment so I missed your ping | 20:12 |
ntpttr___ | hence the double underscore there | 20:12 |
egonzalez90 | can you try changing privsep_entrypoint section to privsep_osbrick at nova/cinder.conf in compute and backup containers? | 20:14 |
ntpttr___ | egonzalez90: sure, can I just manually change those and restart the services in those containers for the change to take effect? | 20:15 |
egonzalez90 | I tried to find the root cause of your issue, I'm thinking on oslo.privsep issue | 20:16 |
ntpttr___ | egonzalez90: I don't see privsep_osbrick in the cinder configuration reference, is that something that gets used? http://docs.openstack.org/mitaka/config-reference/block-storage/block-storage-sample-configuration-files.html#cinder-conf | 20:20 |
ntpttr___ | I don't see privsep_entrypoint in there either actually | 20:20 |
egonzalez90 | new configuration at master, is not documented yet. devstack people use privsep_osbrick section | 20:21 |
*** _coolsvap_ has quit IRC | 20:21 | |
Daviey | sdake: Why wouldn't it work well with stable? | 20:25 |
Daviey | sdake: If they are building from git, they need to build their local images. If they are installing via "pip install kolla" then they have the option of using docker hob images | 20:26 |
sdake | Daviey i don't recall specifics now, perhaps we can start over ;) | 20:26 |
Daviey | This seems reasonable to me :) | 20:26 |
sdake | Daviey our docs now recommend not to use git for stable eval/deployments | 20:26 |
sdake | this was after about 20+ hours of research on how to best handle the versioining problems pbr creates on tagged images | 20:26 |
Daviey | sdake: yeah, but people have the option to install from git - but they need to build their own images | 20:26 |
Pavo | doc right now say just git https://git.openstack.org/openstack/kolla | 20:27 |
sdake | if they do that, they are on their own | 20:27 |
Daviey | sdake: 20 hours?! | 20:27 |
sdake | Daviey estimate on how much time i spent jering around on the problem | 20:27 |
Daviey | sdake: Geez. You must be burned out. | 20:27 |
*** diogogmt has joined #openstack-kolla | 20:27 | |
sdake | reference: https://review.openstack.org/356181 | 20:28 |
sdake | Daviey far from it | 20:28 |
sdake | Daviey but on this particular problem yes :) | 20:28 |
Pavo | shouldn't there really only 2 taggings | 20:28 |
Pavo | stable and latest? | 20:28 |
Pavo | and docs should say what versions of kolla and ansible for each | 20:29 |
sdake | pavo we support liberty and mitaka - so we have two stables | 20:29 |
sdake | pavo look at the above review | 20:29 |
Pavo | ok for each openstack version then | 20:29 |
sdake | Daviey read that review, see if it makes sense to you | 20:29 |
sdake | it started off wrong, but i think we have the right model going forward baed upon conversations with dhellmann and jroll | 20:30 |
*** HyperJohnGraham_ has joined #openstack-kolla | 20:30 | |
Daviey | sdake: sorry, i meant this problem.. not generally | 20:30 |
*** HyperJohnGraham has quit IRC | 20:30 | |
*** HyperJohnGraham_ is now known as HyperJohnGraham | 20:31 | |
sdake | Daviey yes - honetly I don't recall exactly what your proposal was | 20:31 |
sdake | I remember reading it | 20:31 |
sdake | and thinking 3 things | 20:31 |
sdake | 1. does it have impact on upgrades | 20:31 |
sdake | 2. does it have impact on reocnfigure | 20:31 |
sdake | 3. does it work with two stable branches | 20:31 |
sdake | i dug into 3 a bit, so we can get the correct install model going forward | 20:31 |
sdake | (thats the 20 hours part) | 20:31 |
sdake | so given the above docs, any chancey ou could rfresh me on your solution | 20:32 |
sdake | (the ones in the review) | 20:32 |
Pavo | ok sounds like an issue with he two different versions ie pip installed or git installed | 20:33 |
sdake | pavo - right there is no precedence in pbr | 20:34 |
sdake | its one or the other | 20:34 |
Pavo | yeah I can see how thats confusing | 20:34 |
Daviey | sdake: wait, has someone landed the pbr version tagging in the building process now??? | 20:34 |
sdake | if pip installed - pip version is used - if pip not installed, git is used | 20:34 |
sdake | Daviey who knows - this project movesfaster then i can keep u p:) | 20:34 |
Pavo | so wait guess I have been doing wrong the entire time then | 20:34 |
*** Jeffrey4l has quit IRC | 20:35 | |
Daviey | sdake: that review suggests it is.. but i thought i was on the hook to do that | 20:35 |
Pavo | I git clone kolla from git and then use pip install /kolla | 20:35 |
*** Jeffrey4l has joined #openstack-kolla | 20:35 | |
sdake | Daviey that doc review was a short term put out the fire house problem | 20:35 |
Pavo | thats the wrong way? | 20:35 |
sdake | Daviey i expected you would come up with something more long term | 20:35 |
sdake | pavo yup your doing it wrong | 20:35 |
sdake | because our docs were wrong | 20:35 |
ntpttr___ | egonzalez90: huh not sure what's wrong but now creating a volume doesn't even work - it's hanging on 'creating' | 20:35 |
Pavo | well no wonder I am not getting anywhere lol | 20:36 |
Pavo | can the docs alos be changed to have separation for different distros also, ie Centos only shows Centos cmds and Ubuntu only shows Ubuntu cmds? | 20:37 |
*** jtriley has quit IRC | 20:37 | |
Pavo | sdake so if I git clone kolla I should use what to install its dependiecies? | 20:39 |
Daviey | sdake: Odd, but i don't seem to have voting rights for that review | 20:41 |
*** vhosakot has quit IRC | 20:41 | |
sdake | Daviey the review is merged | 20:41 |
Daviey | Oh ffs | 20:41 |
sdake | hence no voting rights | 20:41 |
egonzalez90 | ntpttr: ups, it should work. in theory just need add the section and the command to make it work https://openstack.nimeyo.com/75137/openstack-brick-cinder-brick-privsep-change-awaiting-review | 20:41 |
Daviey | sdake: well, i just added a bunch of comments | 20:41 |
Pavo | so how do you install kolla if you git clone it? | 20:43 |
Daviey | Pavo... git clone http://etc/kolla ; cd kolla ; sudo pip install . | 20:45 |
Pavo | I just asked if that way was wrong and sdake said yup | 20:45 |
Pavo | I have been doing this: git clone https://git.openstack.org/openstack/kolla then pip install kolla/ | 20:46 |
Pavo | I'm so.... confused lol | 20:47 |
sdake | Daviey i responded on your review | 20:48 |
sdake | pavo - read the docs in that review | 20:48 |
sdake | pavo you dont install kolla as a developer | 20:48 |
sdake | you run from git | 20:49 |
Pavo | I am and they say to git clone the pip install kolla/ | 20:49 |
Pavo | oh ok | 20:49 |
sdake | pavo read this review: https://review.openstack.org/356181 | 20:49 |
sdake | pavo - docs fix | 20:49 |
sdake | the docs job hasn't finished publishing the update yet so docs.oo doesn't have the latest stuff | 20:50 |
Pavo | so last commit has latest corrected docs? | 20:50 |
sdake | pavo that patch is mergedd | 20:51 |
Pavo | ok | 20:51 |
Pavo | sorry if I am sounding newbish | 20:51 |
sdake | read the changes by clicking the quickstart.rst button | 20:51 |
sdake | it will show you a "diff' of the changess | 20:51 |
sdake | which should be easy enough to understand what we haned in the docs | 20:51 |
sdake | vs you determining them rom a visual inspection :) | 20:52 |
*** vhosakot has joined #openstack-kolla | 20:52 | |
Pavo | ok gotta redo my deployer completely then | 20:53 |
sdake | shouldn't need to | 20:53 |
sdake | but how to get rom old docs to new docs is a manual process | 20:54 |
Pavo | well atleast delete everything from pip and re clone master | 20:54 |
sdake | i can walk you throug hit | 20:54 |
kfox1111 | inc0: have you seen it cause problems with single entry changes? | 20:54 |
sdake | pavo you want to work from git trees as a developer then? | 20:54 |
kfox1111 | crudini that is | 20:54 |
sdake | pavo if I repeat myself with questions its only because my brian has limited capacity for remembering everything ;) | 20:54 |
Pavo | right now I just want to get up a working system to see it work | 20:54 |
sdake | follow the "Installing Kolla for eval/deployment" | 20:55 |
kfox1111 | I've used it to make single entry changes in my cobbler kickstarts to good effect. | 20:55 |
sdake | and not the other branch which is for development | 20:55 |
inc0 | kfox1111, not really, just super early kolla used crudini to configure *EVERYTHING* | 20:55 |
kfox1111 | inc0: ah. yeah. I would believe that would cause problems. :) | 20:56 |
inc0 | and when I saw this review I felt nostalgic | 20:56 |
Pavo | ok so I yum -y remove ansible and then pip install -U ansible==1.9.4 | 20:57 |
*** rstarmer has joined #openstack-kolla | 20:58 | |
kfox1111 | sbezverk: alive? | 20:59 |
sdake | yup you want that | 20:59 |
Pavo | I alos done a pip uninstall kolla and reinstalled it using pip install kolla | 21:00 |
Pavo | which installed kolla 2.0.2 | 21:00 |
Pavo | sdake ok in the review it says install using pip install kolla then it says Kolla holds configurations files in ``etc/kolla``. Copy the configuration files to ``/etc``: | 21:01 |
Pavo | there is no kolla folder now in my root directory | 21:02 |
Pavo | so can not cd kolla | 21:02 |
sdake | sweet | 21:02 |
sdake | you found a bug in the docs ;) | 21:02 |
Pavo | nor can I cp -r etc/kolla /etc/ | 21:02 |
sdake | how about in /usr/share/kolla/etc? | 21:02 |
Daviey | Pavo: "pip install kolla" will install the latest release, which is Mitaka. git clone .. ; cd kolla ; pip install , will install the dev release | 21:02 |
sdake | pavo we sort of made up the docs in the review via iteration | 21:03 |
Pavo | sdake /usr/share/kolla/etc_examples/kolla/ | 21:03 |
sdake | pavo i didn't specifically tet tthem | 21:03 |
sdake | Pavo cool - will get that into the docs right now | 21:03 |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance-api https://review.openstack.org/354895 | 21:03 |
Pavo | so... cp /usr/share/kolla/etc_examples/kolla/ /etc/ | 21:04 |
sdake | pavo cp -aR I think | 21:04 |
Pavo | sdake cp -r /usr/share/kolla/etc_examples/kolla/ /etc/ | 21:05 |
Pavo | worked | 21:05 |
*** HyperJohnGraham has quit IRC | 21:06 | |
wznoinsk | Pavo: tox -e genconfig should work too | 21:09 |
sdake | tox is not available in a pip install | 21:09 |
sdake | wznoinsk - docs have changed | 21:09 |
wznoinsk | ah, ok | 21:09 |
sdake | I'd encourage everyone to read this review: https://review.openstack.org/356181 | 21:09 |
sdake | its a pretty significant change in thinking of how things should be done | 21:10 |
sdake | but its the only sanitary path forward i know of atm | 21:10 |
sdake | maybe Daviey has better ideas on how to handle it going forward | 21:10 |
Pavo | is it possible to add the tools directory from git into the pip install? | 21:10 |
*** dmueller-evoila has quit IRC | 21:11 | |
sdake | pavo if that needs to be done i think we should do so selectively | 21:11 |
sdake | pavo which tools do you need specifically? | 21:11 |
sdake | i thought we already did that though, in /usr/share/kolla | 21:11 |
Pavo | I deffenantly think the docs should be sectioned out for different distros, ie.. CentOS cmds only shows for CentOS and Unbuntu cmds only shows Ubuntu | 21:12 |
sdake | pavo the answer for current stable releaes is no | 21:12 |
Pavo | ok sdake no problem | 21:12 |
sdake | pavo yup we got that on our plates | 21:12 |
Pavo | the clean scripts are a god sin to me | 21:12 |
sdake | pavo we plan to do that is what i mean | 21:12 |
wznoinsk | sdake: thanks, was thinking of ansible way of verifying deps, +1 | 21:12 |
sdake | validating deps would be great | 21:13 |
wznoinsk | (and was hit badly by not having everything on the right version a couple of times too) | 21:13 |
sdake | I think we need to also tell what deps should be there up front | 21:13 |
Pavo | agreed | 21:14 |
*** jtriley has joined #openstack-kolla | 21:16 | |
*** egonzalez90 has quit IRC | 21:16 | |
sbezverk | inc0: ping | 21:23 |
inc0 | here | 21:24 |
kfox1111 | sbezverk: ingress has some interesting limitations. | 21:25 |
sbezverk | inc0: so the workaround with script worked | 21:25 |
inc0 | cool, just put it in if and we're good | 21:25 |
sbezverk | kfox1111: I am not fun of it either, but other folks suggested it is a way to go | 21:26 |
kfox1111 | we still need to do a vip somehow. it doesn't really cover how to do that. | 21:26 |
sbezverk | inc0: yep it is already with if, I will push it for review soon | 21:26 |
kfox1111 | it also seems to want to bind to port 80/443 and own it. | 21:26 |
inc0 | cool, thanks, I'm glad we got this figured out | 21:27 |
kfox1111 | so everything shows up there, instead of the regular ports. which means we'd need dns entries for everything and need to redo all the public url's in the keystone bootstraps. | 21:27 |
kfox1111 | I'm guessing thats a show stopper? | 21:28 |
sbezverk | inc0: there is still mistery, how it could work in stable mitaka ;-) | 21:28 |
sbezverk | kfox1111: we already introduced kube public kind of vip which is used for public urls | 21:29 |
kfox1111 | where's that? | 21:29 |
kfox1111 | kolla_kubernetes_external_vip | 21:30 |
sbezverk | kfox1111, please check kubernetes.yml | 21:30 |
sbezverk | yes | 21:30 |
kfox1111 | yeah. I see it. | 21:30 |
sbezverk | so it is just a hook, you can then plumb it to whatever you want | 21:30 |
kfox1111 | but with ingress, you'd use it as glance-api.cloud.pnnl.gov:80 that resolves to kolla_kubernetes_external_vip | 21:30 |
sbezverk | I use it with external haproxy | 21:30 |
kfox1111 | we can't register kolla_kubernetes_external_vip as the address for public as the port has to be port 80 or 443. | 21:31 |
kfox1111 | ingress wants to either use host based naming, or path based naming. | 21:32 |
kfox1111 | so kolla_kubernetes_external_vip:443/glance-api/... might work. | 21:32 |
sbezverk | kfox1111: as I said I am not ingress advocate, but I am not in a position to decide anything, hence my review has 0 | 21:32 |
kfox1111 | who's deciding that one? | 21:33 |
sbezverk | I would defer it to folks who suggested ingress | 21:33 |
kfox1111 | who's that? | 21:33 |
sbezverk | wirehead and David were suggesting to use it | 21:33 |
kfox1111 | k. I'll try and talk with them. thx. | 21:34 |
wirehead_ | I don’t think I have a super-strong opinion about ingress versus otherwise. | 21:34 |
sbezverk | inc0: one more question.. do we still have requirement for rabbitmq to have host name running rabbitmq resolvable by dns and pingable by its name? | 21:34 |
wirehead_ | Ingress is probably the right long-term direction, but that also complicates install. | 21:35 |
wirehead_ | Ergo, I’m fairly OK with doing something and then ripping it out and replacing it with something else later. | 21:38 |
*** dave-mccowan has quit IRC | 21:38 | |
wirehead_ | (Actually, I’m OK with that even in shipping production stable code, as long as the users are prevented from unpleasant consequences) | 21:39 |
kfox1111 | k. | 21:39 |
*** daneyon has joined #openstack-kolla | 21:39 | |
*** rstarmer has quit IRC | 21:42 | |
*** daneyon has quit IRC | 21:43 | |
sdake | for those looking to buy gold - this is a monster good price: http://www.ebay.com/itm/121556311471 | 21:44 |
inc0 | sbezverk, not for single node | 21:45 |
inc0 | it's requirement for clusters | 21:46 |
sbezverk | inc0: thank you that is whatI thought too | 21:46 |
*** jtriley has quit IRC | 21:47 | |
kfox1111 | I'm going to try the ingress thing real quick with path based prefixes. that would be a simple solution if it works. | 21:51 |
kfox1111 | and see if multiple ingress rules in different yamls will merge. if not, then I think its pretty much a show stopper. | 21:53 |
sbezverk | wirehead_: ping | 21:53 |
wirehead_ | k | 21:54 |
*** vhosakot has quit IRC | 21:54 | |
sbezverk | wirehead_: I see some instability in bootstraping, it seems there is some sort of racing between differnet container in a job. I suspect some containers try to access DB before it got created and they fail and then retry | 21:55 |
sbezverk | eventually it goes through but sometimes I failures especially in db upgrade process | 21:56 |
sbezverk | I think it will be much safer to split db create and db upgrade into two different job and start them after 1st succeed | 21:57 |
sbezverk | what do you think? | 21:57 |
kfox1111 | sbezverk: +1 | 21:58 |
wirehead_ | ok | 21:58 |
sbezverk | I have been fighting whole day today trying to see the pattern of this failures | 21:58 |
wirehead_ | Failing for poorly-explained reasons on a regular basis is for other OpenStack installation and orchestrating tooling sets. :D | 21:59 |
kfox1111 | wirehead_: also +1. :) | 21:59 |
wirehead_ | Well, you might try merging all of the steps into one bash script. | 21:59 |
wirehead_ | Potentially with a sleep, instead of 2 jobs. | 21:59 |
sbezverk | wirehead_: cool then I document this behavior as a sophisticated feature | 21:59 |
kfox1111 | if split, then you can use the db upgrade job seperate anyway for major version upgrades. | 22:00 |
wirehead_ | I did that on some of my later bootstrap jobs because david pointed out how many discrete container jobs there were. | 22:00 |
wirehead_ | Hm. Also true. | 22:00 |
kfox1111 | the tasks really shoudl be seperate. then ansible workflow can order them. | 22:00 |
sbezverk | wirehead_ they use differnet images, not sure if putting everything in one script is good solution | 22:00 |
*** dave-mccowan has joined #openstack-kolla | 22:01 | |
wirehead_ | yeah, kfox1111 is right about that | 22:01 |
sbezverk | I can repro this instability in my cluster | 22:02 |
sbezverk | so I will break neutron bootstrap into small chunks and test | 22:03 |
*** huikang has quit IRC | 22:06 | |
*** huikang has joined #openstack-kolla | 22:06 | |
sbezverk | the only problem I see, is kolla-kubernetes will not render these chunks | 22:08 |
kfox1111 | what do you mean? | 22:08 |
sbezverk | kfox1111: I mean there is bootstrap resource and it is associated with 1 template | 22:09 |
kfox1111 | and we can't have more then one currently? | 22:10 |
sbezverk | if I have 4 templates for bootstrap now, I am not sure if it can be associated with multiple templates | 22:10 |
kfox1111 | we need a way to be able to name each one individually and launch/status it. | 22:10 |
*** egonzalez has joined #openstack-kolla | 22:11 | |
*** huikang has quit IRC | 22:11 | |
kfox1111 | hmm... made a second ingress type object, loaded it, and its not working.... | 22:12 |
*** huikang has joined #openstack-kolla | 22:13 | |
kfox1111 | gona nuke the ingress and see if it comes back healthier. | 22:13 |
kfox1111 | maybe it is not auto refreshing config right. | 22:14 |
kfox1111 | yup. it came back better. it merged both ingress rules... | 22:14 |
kfox1111 | gotta try path based next. | 22:14 |
*** dave-mccowan has quit IRC | 22:18 | |
sdake | sup egonzalez | 22:23 |
sdake | sup huikang | 22:23 |
*** dave-mccowan has joined #openstack-kolla | 22:24 | |
egonzalez | Hi sdake | 22:24 |
huikang | Hi Sdake | 22:25 |
kfox1111 | ingress just crashed.. :/ | 22:26 |
openstackgerrit | Abishek Subramanian proposed openstack/kolla: Fix the location of Kolla configurations in the evaluation workflow https://review.openstack.org/357520 | 22:30 |
*** ntpttr has quit IRC | 22:31 | |
kfox1111 | arg... | 22:33 |
kfox1111 | unless we do an ingress reverse proxy of some kind, path based will be expensive. | 22:33 |
kfox1111 | no, let me reword that. if we do it path based, we must use a reverse proxy. | 22:34 |
kfox1111 | which is expensive. | 22:34 |
kfox1111 | k. my vote is -1 on ingress for now. | 22:34 |
*** ntpttr has joined #openstack-kolla | 22:36 | |
sbezverk | kfox1111: maybe we should put aside all fancy stuff until we get basics? like running vm and pinging from that VM to upstream router :-) | 22:37 |
kfox1111 | I'm tangengially working on kolla-kubernetes right now. was looking at ingress for a different project and finding out limitations that affect kolla-kubernetes. so relaying that info as I find things. | 22:38 |
Pavo | ok I have to have something wrong either in globals.yml or something because its getting stuck at Starting Neutron-server even on stable/mitaka deployment | 22:39 |
*** inc0 has quit IRC | 22:42 | |
Pavo | someone mind taking at look at this and tell wtf I am doing wrong | 22:42 |
*** inc0 has joined #openstack-kolla | 22:42 | |
Pavo | https://www.dropbox.com/s/2694rq4mkojt9ef/globals.yml?dl=0 | 22:42 |
Pavo | this is my network diagram https://www.lucidchart.com/invitations/accept/edf94421-2e30-4f04-a035-0c02676cf400 | 22:43 |
Pavo | I have no clue why it stops networking on target nodes | 22:44 |
*** daneyon has joined #openstack-kolla | 22:45 | |
*** huikang has quit IRC | 22:48 | |
*** huikang has joined #openstack-kolla | 22:48 | |
*** daneyon has quit IRC | 22:50 | |
*** huikang has quit IRC | 22:53 | |
openstackgerrit | Serguei Bezverkhi proposed openstack/kolla: Kube OVS DB to listen on tcp:port and neutron-ovs-agent to use script https://review.openstack.org/357533 | 22:59 |
*** Pavo has quit IRC | 23:00 | |
*** haplo37__ has quit IRC | 23:03 | |
*** Pavo has joined #openstack-kolla | 23:06 | |
kfox1111 | wirehead_: are you different then wirehead:? | 23:07 |
*** michauds has quit IRC | 23:07 | |
wirehead_ | yah | 23:08 |
kfox1111 | ah. ok. | 23:08 |
*** severion has joined #openstack-kolla | 23:11 | |
*** sdake_ has joined #openstack-kolla | 23:12 | |
*** v1k0d3n has quit IRC | 23:13 | |
*** sdake has quit IRC | 23:13 | |
*** egonzalez has quit IRC | 23:14 | |
openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: openvswitch ip https://review.openstack.org/356114 | 23:14 |
Mech422 | wirehead_: hey - do you happen to know if us randoms are eligble to vote for maintainers? | 23:20 |
Mech422 | wirehead_: I just saw sdace's nomination come thru | 23:21 |
wirehead_ | i forget | 23:21 |
Mech422 | wirehead_: thanks - I'll ask sdace later | 23:23 |
Mech422 | err..sdake :-P | 23:23 |
*** inc0 has quit IRC | 23:41 | |
*** ajpower has quit IRC | 23:41 | |
*** eaguilar has joined #openstack-kolla | 23:50 | |
*** zhurong has joined #openstack-kolla | 23:53 | |
*** ajpower has joined #openstack-kolla | 23:54 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!