Friday, 2016-08-19

*** zhurong has quit IRC00:01
*** zhurong has joined #openstack-kolla00:02
sdake_Mech422 first off your not a random00:02
sdake_Mech422 second off, only core reviewers may vote00:02
*** zhurong has quit IRC00:04
*** bmace has quit IRC00:20
*** bmace has joined #openstack-kolla00:20
sdake_sup bmace00:31
sbezverkwirehead_ kfox1111: I tested splitted neutron bootstrap and it wokred without hicups.. I will run more tests, I think it should be the way to go until kube introduce a mechanism to run containers serially in the job..00:32
*** daneyon has joined #openstack-kolla00:33
*** daneyon has quit IRC00:38
*** duonghq has joined #openstack-kolla00:42
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - l3 DaemonSets + DVR  https://review.openstack.org/35755700:48
*** dave-mccowan has quit IRC00:58
bmacehi sdake_ :)01:02
*** caowei has joined #openstack-kolla01:04
*** eaguilar has quit IRC01:05
*** HyperJohnGraham has joined #openstack-kolla01:07
*** zhurong has joined #openstack-kolla01:09
*** kbaegis has quit IRC01:10
*** salv-orlando has joined #openstack-kolla01:15
*** salv-orlando has quit IRC01:15
*** salv-orlando has joined #openstack-kolla01:16
*** salv-orl_ has quit IRC01:17
*** salv-orlando has quit IRC01:21
*** unicell has quit IRC01:21
Pavoanyone have an idea on how to fix this issue, this fix in this review does not work01:39
Pavohttps://bugs.launchpad.net/kolla/+bug/157161201:39
openstackLaunchpad bug 1571612 in kolla "Prechecks fails when checking if kolla_internal_vip_address is in the same network as network_interface on all nodes" [High,Triaged] - Assigned to Hiroki Ito (hrito)01:39
openstackgerritMerged openstack/kolla: Customizations for Senlin  https://review.openstack.org/35201301:40
duonghqPavo: which solution did you try?01:43
Pavoboth01:43
Pavoin that review01:43
Pavodoes not work01:43
duonghqDo you mean the comment?01:43
Pavonow trying this https://review.openstack.org/gitweb?p=openstack/kolla.git;a=commitdiff;h=7e00288ff09072eda41cc8a7daa53ac8bb94323f01:43
duonghqah, ok, cannot see this link in bug page01:44
Pavoyeah tried changing /usr/share/kolla/ansible/roles/prechecks/task/ports_check.yml at the end of the file01:45
*** rstarmer has joined #openstack-kolla01:45
Pavowell second fix didn't work either01:46
duonghqare you deploying multinode?01:46
Pavoits the TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] part of the ports_check.yml file at the end01:46
Pavoyes01:46
duonghq(just for more information): which distro do you use and which user do you use for ssh?01:47
Pavocentos and root01:47
duonghqah, ok01:47
PavoI have ssh keys from deployer to all nodes01:48
duonghqsure :)01:48
*** eaguilar has joined #openstack-kolla01:48
duonghqI'm trying deploy multinode without remote root login01:49
openstackgerritMerged openstack/kolla: Customizations for sahara  https://review.openstack.org/34659001:50
openstackgerritMerged openstack/kolla: Update cirros image visibility in init-runonce  https://review.openstack.org/35393301:50
duonghqPavo: can you try to debug with Ansible?01:50
openstackgerritMerged openstack/kolla: Improve rabbitmq-clusterer template code format  https://review.openstack.org/35706501:50
Pavohow would I do that?01:51
duonghqwe have friend here http://docs.ansible.com/ansible/debug_module.html01:52
duonghqand here01:52
duonghq#link https://docs.ansible.com/ansible/playbooks_debugger.html01:52
Pavoyeah I can look into that01:52
PavoI am also having an issue when deploying it gets stuck at Starting Neutron Server01:56
duonghqhmm, mariadb container failed (again)02:00
*** diogogmt has quit IRC02:00
Pavothis is exactly whats happening to me https://ask.openstack.org/en/question/93376/during-kolla-deploy-when-neutron-comes-up-networking-goes-down/02:00
Pavono answers :(02:01
duonghqhow many NIC do you have?02:01
openstackgerritMerged openstack/kolla: Customizations for murano  https://review.openstack.org/34634002:01
duonghqPavo: Do your mariadb keep restarting?02:02
Pavo2 per node02:02
Pavothis is a network digram02:02
Pavo*diagram02:02
openstackgerritMerged openstack/kolla: Create the _member_ role in the horizon role  https://review.openstack.org/35608202:03
Pavohttps://www.lucidchart.com/invitations/accept/ddb42620-afc3-4489-aa63-a024034a190f02:03
Pavoeth0 is what neutron will use02:03
duonghqI don't have lucidchart acc02:04
Pavothis is my globals02:04
Pavohttps://www.dropbox.com/s/2694rq4mkojt9ef/globals.yml?dl=002:04
Pavooh ok hang on02:04
sdake_pavo if stuck at starting neutron server, your interfaces are backwards02:04
sdake_pavo though twe have been down this road already...02:05
Pavoyeah I thought so also02:05
Pavodoing exactly how you told me to02:05
duonghqanybody touch mariadb container recently?02:05
Pavohttps://www.dropbox.com/s/n76ds55mzyyfeyt/Screen%20Shot%202016-08-18%20at%2010.05.31%20PM.png?dl=002:05
PavoI don't understand why its failing02:06
*** eaguilar has quit IRC02:06
openstackgerritMerged openstack/kolla: Customizations for zaqar  https://review.openstack.org/34635702:06
Pavowhen it gets stuck on node1 which is control and network node I check the interfaces and eth0 and eth1 are up but docker0, ovs-system and br-ex are down02:08
Pavonothing in logs and neutron-server container never gets started02:08
openstackgerritMerged openstack/kolla: Fix publicURL endpoint not found issue in CI  https://review.openstack.org/35651302:08
duonghqPavo: your config seems ok02:10
Pavoonly thing I can think of is the I am using the wrong kolla_internal_vip_address02:10
Pavoif that IP is being used on the api interface like the docs say then that might be the issue02:11
Pavodocs say this02:11
Pavohttp://pastebin.com/grgntY3U02:11
duonghqiirc, the api_interface is network_interface by default02:12
duonghqso, nothing wrong02:13
*** huikang has joined #openstack-kolla02:13
Pavobut the IP I am using is on a different network than the network_interface02:13
Pavothat IP in globals in on the eth0 network with is being used for neutron_external_interface02:14
*** rstarmer has quit IRC02:15
duonghqah, miss reading, sorry02:16
Pavogonna try changing that IP02:17
Pavoand use a different IP for kolla_external_vip_address02:18
*** zhurong_ has joined #openstack-kolla02:19
*** dave-mccowan has joined #openstack-kolla02:19
*** zhurong has quit IRC02:20
duonghqPavo: do you ever get mariadb container keep restating?02:20
Pavonot yet02:20
openstackgerritMerged openstack/kolla: Document the Dockerfile customisation mechanism  https://review.openstack.org/35112302:20
Pavobut I am also using stable02:20
duonghqah, ok02:21
duonghqPavo: mitaka?02:22
Pavoyeah02:22
Pavo2.0.202:22
duonghqokay02:22
*** rstarmer has joined #openstack-kolla02:24
openstackgerritMerged openstack/kolla: Update openvswitch base Dockerfile to include footers  https://review.openstack.org/35724302:24
openstackgerritMerged openstack/kolla: Fix guide formating errors  https://review.openstack.org/35623502:24
duonghqsdake_: heavy lifting works :) tyvm02:25
openstackgerritMerged openstack/kolla: Remove heat dev environment  https://review.openstack.org/35211902:25
Pavoguess I should make myself an account on launchpad so I can start submitting bugs I encounter02:30
duonghqPavo: clearly02:30
Pavolol funny02:30
duonghqwelcome to the bug reporter team, lol02:31
Pavolol02:31
PavoI truly truly enjoy kolla's way of deploying openstack, coming from packstack thats saying alot02:31
Pavowent from Mirantis fuel to packstack now to kolla02:32
Pavofuel is by far the worse in my book02:32
duonghqreally?02:33
*** huikang has quit IRC02:33
Pavoyeah very proprietary doesn't fit our use case at all02:34
duonghqah,02:35
duonghqyour seems very interesting?02:35
sdake_fuel is proprietary in what way?02:35
Pavorequires a minimum of 3 target nodes, best suited in enterprise environment02:35
sdake_packstack had alot of potential shame red hat never turned it into their deployment system.  They had a winner on their hands..02:37
Pavoyeah our use case is VERY unique02:37
Pavoyeah I agree but packstack is very easy to deploy02:37
Pavothey switched focus to TripleO02:39
Pavowhich is weird to me, reminds me of inception02:39
mark-caseyPavo you having maria issues?02:40
mark-caseyjust reading scrollback02:40
Pavonot yet02:40
Pavoduonghq was though I think02:41
duonghqPavo: mark-casey02:41
duonghqyes, mariadb keep restarting02:42
duonghqthank Pavo for mentioned02:42
Pavonp02:42
duonghqtrying to rebuild image02:44
mark-caseyduonghq: do you have an error message/logs for the maria failure?02:45
duonghqI'm pull and reploying02:45
duonghqbut with config do you use mark-casey02:45
duonghq*which02:46
mark-caseyI've been working on a 3 node ubuntu source install02:48
duonghqI'm working on AIO, but get (probably) same issue02:48
Pavook here goes *crosses fingers*02:48
mark-caseybut I use galera elsewhere so I thought I'd try to take a look if there was a maria issue02:48
sdake_mark-casey - we had a report earlier thi sweek on osic not forming properly with galera02:49
duonghqin Kolla, Galera is always deployed, isn't it?02:49
sdake_mark-casey don't know if its related - but it may be - did you try twice?02:49
sdake_duonghq it can be turned off if you like02:49
sdake_not sure how that would work in practice02:49
*** rstarmer has quit IRC02:49
duonghqsdake_: are you done with the commit review?02:50
sdake_which one02:50
sdake_all of them?02:50
duonghqyeah, I see that you review very much commit till now02:50
sdake_ya taking a break02:50
sdake_reviewing is really difficult work02:50
sdake_requires very good attention to detail02:51
sdake_which is not my strength :)02:51
duonghqtyvm sdake_02:51
mark-caseysdake_: I'm not having a galera issue, just have some familiarity with it's sometimes-rough edges02:51
*** huikang has joined #openstack-kolla02:52
duonghqstill do not understand why need twice deploy before bootstrap maria success02:52
duonghqbut mariadb container keep failing and restarting02:53
Mech422sdake_: thanks for the reply :-)  Sounds like a good addition to the team...02:53
duonghqmark-casey: do you stuck here?02:53
mark-caseyduonghq: no. I was asking about what you're seeing02:55
duonghqhttp://pastebin.com/1jLmn8w302:55
duonghqhere02:55
duonghqmark-casey: please tell me if you have any idea02:56
duonghqthanks02:56
*** yuanying has quit IRC02:56
duonghqah, Galera-related ones02:57
*** yuanying has joined #openstack-kolla02:58
*** rstarmer has joined #openstack-kolla02:58
*** huikang has quit IRC03:02
mark-caseyduonghq: that is the container log, right?03:02
duonghqright03:02
mark-caseyis this happening during deploy when bootstrapping from scratch? or upgrade/reconfigure?03:03
duonghqfrom scatch03:03
mark-caseydoes the kolla-ansible deploy fail the first time?03:03
duonghqyes03:03
mark-caseyon which task?03:04
duonghqRunning MariaDB bootstrap container03:04
duonghqafter 2nd run without clean it pass this task03:04
duonghqbut the mariadb cannot start03:05
duonghqit worked before03:06
mark-caseyduonghq: distro? source/binary?03:06
duonghqhost is trusty, image is ubuntu/source03:07
duonghqlike you?03:07
duonghqmark-casey03:07
mark-caseyduonghq: in the past yes. testing others now03:08
mark-caseyduonghq: and you said this is aio?03:08
duonghqyup03:08
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936603:12
mark-caseyduonghq: can you kolla-ansible deploy with '-vvvv' and paste the first "Running MariaDB bootstrap container"?03:12
duonghqokay, hang on03:13
*** coolsvap has joined #openstack-kolla03:13
duonghqtrying mariadb_recovery03:13
duonghqmark-casey: my log: http://pastebin.com/SzpwVwK603:18
coolsvapsdake_: around?03:18
sdake_unfortuantely03:19
sdake_what can i do for ya03:19
*** salv-orlando has joined #openstack-kolla03:19
duonghqmark-casey http://pastebin.com/RVTA73en03:20
duonghqboostrap log03:20
coolsvapsdake_: i didnt get the last message, should I expect a mail or something?03:20
coolsvaprelated to the status?03:20
coolsvapor I need to send?03:20
*** ayoung has quit IRC03:22
*** yuanying has quit IRC03:22
*** huikang has joined #openstack-kolla03:22
sdake_coolsvap that was for informational purposes03:22
duonghqmark-casey: cannot jump in dead container to get more log03:23
sdake_ttx said to forward on to other ptl candidates03:23
sdake_hence why i ssent to you03:23
mark-caseyduonghq: do you have deploy log like:03:24
mark-caseyTASK [mariadb : Running MariaDB bootstrap container] ***************************03:24
mark-caseychanged: [localhost] => {"changed": true, "invocation": {"module_args": {"action": "start_container", ...03:24
mark-caseyprobably just that task03:24
coolsvapsdake_: I think we are talking about two different things03:24
duonghqmark-casey: 1st or 2nd time run deploy?03:24
coolsvapI did not receive the mail you are talkin gabout03:25
mark-caseyboth if possible03:25
coolsvapI wwill start with osic based on pbourke last message03:25
duonghqmark-casey: 1st time is like I sent you03:25
duonghqin the end of log03:26
mark-caseyduonghq: I see container logs but not kolla-ansible log03:27
*** salv-orlando has quit IRC03:27
duonghqhttp://pastebin.com/SzpwVwK603:27
duonghqmark-casey: http://pastebin.com/SzpwVwK6 I sent you both03:27
duonghqPavo: any good news?03:28
Pavonot yet waiting for mariadb from VIP errored out03:29
Pavochanged IP in globals03:30
Pavoseems like it taking WAY long then it use to03:30
duonghqso you solved the VIP error?03:30
Pavoredeploying now, still waiting to get back to that spot03:30
*** rstarmer has quit IRC03:30
duonghqYou don't have MariaDB error?03:30
Pavothis is the error I got03:31
Pavohttp://pastebin.com/Li4MQYHK03:31
duonghqseems that haproxy-related03:32
duonghq?03:32
Pavomaybe03:32
duonghqsdake_: are you there03:33
*** huikang has quit IRC03:33
duonghqPavo: seem that MariaDB cause many troubles03:33
sdake_coolsvap are you talking abou tthat mail from thierry i forwarded to you?03:33
*** huikang has joined #openstack-kolla03:33
sdake_duonghq yes here03:33
coolsvapsdake_: yes I did not receive the mail you're talking about03:34
duonghqsdake_: can you take a look at my bp: https://blueprints.launchpad.net/kolla/+spec/ansible-specific-task-become03:34
sdake_be with yo ugys in a bit03:35
*** huikang has quit IRC03:38
openstackgerritMd Nadeem proposed openstack/kolla: Add Dockerfile for congress  https://review.openstack.org/30562603:39
*** huikang has joined #openstack-kolla03:40
Pavodocker0 is DOWN, is that normal?03:44
sdake_snafu03:44
sdake_we use host networking03:45
sdake_docker0 does nothing03:45
Pavook03:45
Pavobeen TASK: [mariadb | Waiting for MariaDB service to be ready through VIP] ********* for about 10 minutes now03:46
Pavocontainer log shows 160818 23:36:37 [Note] WSREP: Service disconnected.03:47
duonghqhow can I disable galera?03:48
*** severion has quit IRC03:48
*** v1k0d3n has joined #openstack-kolla03:49
mark-caseyPavo: could you paste docker logs for mariadb container for what it does when kolla-ansible is running "Running MariaDB bootstrap container"?03:50
*** huikang has quit IRC03:50
Pavohttp://pastebin.com/D56M6jLA03:51
*** v1k0d3n has quit IRC03:51
duonghqPavo: better than me03:52
duonghqno idea03:52
Pavothats the entire mariadb log and still setting at Waiting for MariaDB service to be ready through VIP03:53
Pavonow it finally errored out03:53
PavoCan't connect to MySQL server on '192.168.188.25403:53
*** yuanying has joined #openstack-kolla03:54
*** fragatina has quit IRC03:56
mark-caseyPavo: that was from 'docker logs' for the mariadb container? any lines like "INFO:__main__:Setting permissions for /etc/mysql/my.cnf"?03:58
duonghqPavo: reach same error as your03:58
Pavonot seeing any no03:59
duonghqPavo: waiting for MariaDB... through VIP03:59
openstackgerritMd Nadeem proposed openstack/kolla: Add Dockerfile for congress  https://review.openstack.org/30562603:59
Pavowelcome to the waiting club lol04:00
duonghqit failed me quite fast :))04:00
Pavomine took forever04:01
duonghqah, just my fault,04:01
duonghqlol04:01
openstackgerritMerged openstack/kolla: Add force flag to cleanup-images  https://review.openstack.org/35636404:02
Pavodamn you04:02
Pavolol04:02
mdnadeemsdake_, coolsvap , Jeffrey4l , can you please review patch set: https://review.openstack.org/#/c/349808/        https://review.openstack.org/#/c/349771/04:02
patchbotmdnadeem: patch 349808 - kolla - Added Test case for env and volume comp04:02
patchbotmdnadeem: patch 349771 - kolla - Added Test case for attribute comp04:02
openstackgerritMerged openstack/kolla: Fix the variable name mismatch  https://review.openstack.org/34986504:03
openstackgerritMd Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy  https://review.openstack.org/35062904:03
sdake_sorry folks busy04:09
Pavosdake_ have you tried out 10.12 yet?04:10
Pavoas in macOS 10.1204:10
PavoI mean04:10
*** huikang has joined #openstack-kolla04:11
mark-caseyPavo: duonghq: what was the docker logs command you used?04:14
mark-caseyI think one must be mariadb and one bootstrap-mariadb04:14
*** zhubingbing has joined #openstack-kolla04:14
PavoI did a less /var/lib/docker/volumes/kolla_logs/_data/mariadb/mairadb.log04:15
duonghqdocker logs mariadb04:15
mark-caseyoh cool.04:15
Pavofixed my waiting for MariaDB service through VIP04:15
mark-caseywhat was it?04:15
Pavoenabled haproxy in globals lol04:15
mark-caseyhehe04:15
Pavoforgot I disabled it04:16
duonghqlol04:16
Pavofell like such an idoit04:16
duonghqbut iirc it's enable by default04:16
Pavofeel04:17
Pavonot if you uncomment it lol04:17
mark-caseycould you docker logs mariadb on the VM that ran "Running MariaDB bootstrap container" (if more than one node is running db)04:17
Pavosure once this deployment either finishes or errors out04:17
mark-caseycool. thx04:17
mark-caseyso duonghq... this is the task failing https://github.com/openstack/kolla/blob/master/ansible/roles/mariadb/tasks/bootstrap_cluster.yml#L2 on the first run right?04:18
Pavojust a fyi I am running stable version04:18
Pavoas in 2.0.204:18
duonghqmark-casey: yup, everytime after clean04:19
mark-caseyit sets KOLLA_BOOTSTRAP: on line 8 to try and trigger this https://github.com/openstack/kolla/blob/master/docker/mariadb/extend_start.sh#L4604:19
mark-caseywhich in turn calls "mysqld_safe --wsrep-new-cluster"04:20
mark-caseyinside bootstrap_db04:20
mark-caseyhowever...04:20
*** rstarmer has joined #openstack-kolla04:20
mark-caseythe paste you sent (http://pastebin.com/1jLmn8w3) shows "Running command: '/usr/bin/mysqld_safe'" and "WSREP: Running position recovery"04:21
mark-caseywhere as Pavo's log shows "WSREP: 'wsrep-new-cluster' option used, bootstrapping the cluster"04:22
mark-caseyI don't know why, but I think when your mariadb container bootstraps it is trying to restart a cluster, not start a new cluster04:22
*** rstarmer has quit IRC04:22
Pavook almost at the neutron server part, lets see if it gets stuck again04:22
Pavo*crosses fingers*04:23
Pavowoot it didn't get stuck04:24
*** rstarmer has joined #openstack-kolla04:26
*** rstarmer has quit IRC04:27
mark-caseyhmm. duonghq can you try 'docker logs bootstrap-mariadb' during "Running MariaDB bootstrap container" task after clean?04:28
*** rstarmer has joined #openstack-kolla04:29
*** rstarmer has quit IRC04:31
duonghqmark-casey: I'll re deploy after check horizon if working04:31
*** salv-orlando has joined #openstack-kolla04:32
mark-caseycool. remember it may not work after task fails. I think the container will be removed04:32
Pavowhy dos Gathering Facts take so long on stable04:33
*** rstarmer has joined #openstack-kolla04:33
*** rstarmer has joined #openstack-kolla04:36
duonghqhorizon is working now04:37
duonghqcool04:37
*** dims has quit IRC04:38
Pavonice04:38
*** salv-orlando has quit IRC04:39
duonghqmark-casey: the error disappear even after i clean container and host04:43
duonghqno idea04:43
*** dims has joined #openstack-kolla04:43
mark-caseynot happening now/04:44
mark-casey*?04:44
duonghqyup04:44
mark-caseyvery interesting. ok good :)04:44
duonghqI will record and pass you the log when it happen04:44
duonghqthank mark-casey04:45
mark-caseysure, happy to try04:45
PavoI thought the kolla_external_vip_address was suppose to be added to the neutron_external_interface?04:48
duonghqit's reasonable to guess that04:50
Pavoyeah well I am done deploying lol but can't get to anything because all the IPs were added to the internal network lol04:50
Pavonot accessible from outside04:50
Pavook need some help here04:52
Pavohttps://www.dropbox.com/s/n76ds55mzyyfeyt/Screen%20Shot%202016-08-18%20at%2010.05.31%20PM.png?dl=004:52
Pavowhat interface would you use for network_interface and what interface would you use for neutron_external_interface04:53
Pavoeth1 and eth0 right04:53
Pavo?04:53
Pavothe reason I ask is because this is what happened doing it that way04:56
Pavohttp://pastebin.com/Awy4DLkH04:56
Pavoboth the kolla_internal_vip_address and kolla_external_vip_address got set on eth104:57
Pavowhich I thought was only for api interfaces04:57
Pavowhich I thought was separated from all other traffic oh well, guess I am deplying again04:58
Pavogonna do that tomorrow, cya05:05
*** unicell has joined #openstack-kolla05:13
*** mark-casey has quit IRC05:13
*** dave-mccowan has quit IRC05:23
sdake_i'm off to bed05:23
duonghqbye sdake_05:23
sdake_night05:23
sdake_1 more day to go05:24
duonghqstorm is coming05:24
sdake_no doubt05:24
*** sdake_ has quit IRC05:24
*** rstarmer has quit IRC05:30
*** rstarmer has joined #openstack-kolla05:31
openstackgerritHui Kang proposed openstack/kolla: Add missing container image names to build config file  https://review.openstack.org/35685405:31
*** rstarmer has quit IRC05:34
*** rstarmer has joined #openstack-kolla05:35
*** rstarmer has quit IRC05:37
*** salv-orlando has joined #openstack-kolla05:38
*** rstarmer has joined #openstack-kolla05:40
openstackgerritHui Kang proposed openstack/kolla: Add missing container image names to build config file  https://review.openstack.org/35685405:41
*** rstarmer has quit IRC05:43
*** huikang has quit IRC05:43
*** rstarmer has joined #openstack-kolla05:44
*** dmueller-evoila has joined #openstack-kolla05:48
*** rstarmer has quit IRC05:56
*** rstarmer has joined #openstack-kolla05:56
*** dmueller-evoila has quit IRC05:56
*** unicell has quit IRC05:57
*** unicell has joined #openstack-kolla05:57
*** daneyon has joined #openstack-kolla05:58
*** sdake has joined #openstack-kolla05:59
*** daneyon has quit IRC06:02
*** sdake has quit IRC06:08
*** zhubingbing_ has joined #openstack-kolla06:13
duonghqhi zhubingbing_06:14
zhubingbing_hi06:14
zhubingbing_duonghq heloo06:14
zhubingbing_hello06:14
duonghq:D last working day of the week :D06:15
*** fragatina has joined #openstack-kolla06:15
*** zhubingbing has quit IRC06:16
*** fragatina has quit IRC06:16
*** caowei has quit IRC06:17
*** fragatina has joined #openstack-kolla06:17
zhubingbing_yes,today is Friday06:18
zhubingbing_ Hui Kang06:19
*** dmueller-evoila has joined #openstack-kolla06:23
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935106:28
*** hieulq has quit IRC06:30
*** mdnadeem_ has joined #openstack-kolla06:30
*** dmueller-evoila has quit IRC06:32
*** hieulq has joined #openstack-kolla06:33
*** mdnadeem has quit IRC06:33
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838806:33
*** rstarmer has quit IRC06:34
*** rstarmer has joined #openstack-kolla06:35
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071506:38
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071506:44
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102706:48
*** Mr_Broken has joined #openstack-kolla06:51
zhubingbing_hi, coolsvap06:54
coolsvapzhubingbing_: hello06:54
zhubingbing_i remove gnoochi/aodh user(in api),there changes are all in order to be able to run as httpd06:56
zhubingbing_i  refer to horizon and keystone.06:56
coolsvapzhubingbing_: sure thanks I have removed my -107:02
*** cu5 has quit IRC07:09
zhubingbing_thanks07:13
zhubingbing_thank you very much07:13
*** salv-orl_ has joined #openstack-kolla07:16
*** salv-orlando has quit IRC07:19
*** salv-orl_ has quit IRC07:20
*** mnasiadka has joined #openstack-kolla07:21
openstackgerritzhubingbing proposed openstack/kolla: fix sahara dockerfile  https://review.openstack.org/35132007:21
*** cu5 has joined #openstack-kolla07:24
*** Serlex has joined #openstack-kolla07:31
*** rstarmer has quit IRC07:32
*** rstarmer has joined #openstack-kolla07:33
*** ad_rien_ has joined #openstack-kolla07:38
*** dmueller-evoila has joined #openstack-kolla07:41
*** daneyon has joined #openstack-kolla07:46
*** Mr_Broken has quit IRC07:47
openstackgerritzhubingbing proposed openstack/kolla: fix sahara dockerfile  https://review.openstack.org/35132007:47
*** Mr_Broken has joined #openstack-kolla07:47
*** daneyon has quit IRC07:51
*** Mr_Broken has quit IRC07:52
*** dmueller-evoila has quit IRC07:55
*** dmueller-evoila has joined #openstack-kolla07:55
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936607:56
*** dmueller-evoila has quit IRC07:58
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113907:59
*** Mr_Broken has joined #openstack-kolla08:00
*** matrohon has joined #openstack-kolla08:01
openstackgerritJeffrey Zhang proposed openstack/kolla: Sync the local_settings.j2 with horizon  https://review.openstack.org/35647108:01
*** openstackgerrit has quit IRC08:03
*** openstackgerrit has joined #openstack-kolla08:03
*** joyee has joined #openstack-kolla08:09
*** duonghq has left #openstack-kolla08:14
*** gfidente has joined #openstack-kolla08:15
*** joyee has quit IRC08:22
*** Jeffrey4l has quit IRC08:27
*** athomas has joined #openstack-kolla08:29
*** Jeffrey4l has joined #openstack-kolla08:32
*** zhubingbing_ has quit IRC08:40
*** HyperJohnGraham has quit IRC08:49
Mr_BrokenHey Guys09:05
*** salv-orlando has joined #openstack-kolla09:06
Mr_Brokendeploying master/newton branch for the first time (ubuntu / binary )09:06
Mr_Brokenis this normal that some dockers dont build ?09:06
Mr_Brokenhttp://paste.openstack.org/show/561159/09:06
Mr_BrokenI wanted to test out telemetry stuff (ceilometer + gnocchi)09:06
Mr_Brokenbut as you can see gnochi is building09:07
pbourkeMr_Broken: I think some of them are available for source builds only09:12
pbourkedouble check you should see something to that effect in the build output09:12
Mr_Brokenyeah, noticed when tried to build only gnocchi from binary09:12
Mr_Brokenthere are no binaries yet09:13
Mr_Brokenits building from sources now09:13
Mr_Brokenthanks09:13
*** egonzalez90 has joined #openstack-kolla09:16
pbourkecoolsvap: all rally tasks passing :)09:18
pbourkecoolsvap: only ones failed seem to be cinder09:18
pbourkewhich is not deployed09:18
coolsvappbourke: cool09:18
pbourkeso lets sed in decent values now09:19
pbourkecoolsvap: does concurrency of 2 and say times value of 5 mean it will do 10 tasks overall?09:19
pbourkecoolsvap: or 5 tasks 2+2+109:19
*** kbaegis has joined #openstack-kolla09:20
pbourkeactually it must be the second as otherwise you would have a lot more tasks showing in the list09:21
pbourkecoolsvap: should we set the 'times' value to 100 or so?09:21
coolsvappbourke: sorry was connecting to vpn, i think yes it would be 2nd09:22
egonzalez90Jeffrey4l: ping09:25
*** daneyon has joined #openstack-kolla09:35
Jeffrey4legonzalez90, pong09:35
egonzalez90I've been thinking about removing the host list at https://review.openstack.org/#/c/355861/09:36
patchbotegonzalez90: patch 355861 - kolla - Remove unnecessary host at haproxy in site.yml09:36
egonzalez90gates are passing, but imho is because is an all in one, it will break in multimode deploys09:36
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071509:36
*** zhubingbing_ has joined #openstack-kolla09:36
zhubingbing_hello,coolsvap09:37
egonzalez90because haproxy template need fact about other the other host09:37
zhubingbing_I've already dealt with that problem.09:37
zhubingbing_https://review.openstack.org/#/c/350715/09:37
patchbotzhubingbing_: patch 350715 - kolla - repair aodh image dockerfile09:37
coolsvapzhubingbing_: alright I will wait for the the gate results09:37
zhubingbing_ok,thanks09:38
*** daneyon has quit IRC09:39
Jeffrey4legonzalez90, no check the haproxy/tasks/main.yml file09:39
openstackgerritPaul Bourke proposed openstack/kolla: Add basic checking for network_interface to prechecks  https://review.openstack.org/35551509:39
egonzalez90I checked, but that defines where the role is applied. But https://github.com/openstack/kolla/blob/master/ansible/roles/haproxy/templates/haproxy.cfg.j2 need gather fact about the other hosts t get IP info09:41
egonzalez90maybe I'm wrong, is just for ensure it will not break multinode09:41
Jeffrey4legonzalez90, the facter is gather before the role is run.09:42
Jeffrey4legonzalez90, it is not related.09:42
egonzalez90ok, thanks09:42
Jeffrey4legonzalez90, so i think leave only the haproxy is enough.09:43
Jeffrey4lbut need eyes from other guys.09:44
egonzalez90Jeffrey4l: thanks for clarify me on that09:45
pbourkecoolsvap: can you let us know what values you think we need09:49
coolsvappbourke: I think we can start with 100/500/1000 and concurrency level of 25/50/10009:50
pbourkeok thanks09:50
pbourkecoolsvap: replaced all values, can you have a quick look?10:02
coolsvappbourke: just a min10:09
coolsvappbourke: looks good10:10
pbourkecoolsvap: ok think will go back and do the first deploy scenario10:11
coolsvappbourke: yeah10:11
Mr_BrokenI will never have a normal deploy:10:14
Mr_Brokenhttp://paste.openstack.org/show/561232/10:14
Mr_Brokenanyone seen such a thing ?10:14
openstackgerritChristian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce kolla_enable_cadf_notifications  https://review.openstack.org/34900610:14
Mr_Brokenlooks like someone misplaced a quotation somewhere ?10:14
pbourkeMr_Broken: check your globals.yml10:15
pbourkefor a backtick10:15
Mr_Brokenwhy didnt i think of that one ? :]10:17
Mr_Broken<--- feels like a moron10:17
Mr_Brokenthanks10:17
*** iamkube has joined #openstack-kolla10:20
*** gbialas has joined #openstack-kolla10:21
openstackgerritChristian Berendt proposed openstack/kolla: Add cloudkitty containers for type source  https://review.openstack.org/34929510:26
*** gbialas has quit IRC10:29
*** psanchez has quit IRC10:29
openstackgerritChristian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container  https://review.openstack.org/33549510:30
*** psanchez has joined #openstack-kolla10:31
*** iamkube has quit IRC10:33
openstackgerritSwapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Dockerfile to include footers  https://review.openstack.org/35774610:33
*** athomas has quit IRC10:33
*** zhurong_ has quit IRC10:38
*** athomas has joined #openstack-kolla10:38
openstackgerritMerged openstack/kolla: Updated from global requirements  https://review.openstack.org/35769910:48
openstackgerritPaul Bourke proposed openstack/kolla: Handle empty package list for install_packages  https://review.openstack.org/35442910:49
openstackgerritMerged openstack/kolla: Add Dockerfile for congress  https://review.openstack.org/30562610:51
openstackgerritChristian Berendt proposed openstack/kolla: Add cloudkitty containers for type source  https://review.openstack.org/34929510:56
openstackgerritChristian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce kolla_enable_cadf_notifications  https://review.openstack.org/34900611:01
*** Mr_Broken has quit IRC11:16
*** Mr_Broken has joined #openstack-kolla11:16
-openstackstatus- NOTICE: Precise tests on OSIC provider are currently failing, please stop your checks until the issue is resolved.11:16
openstackgerritChristian Berendt proposed openstack/kolla: Customizations for Neutron  https://review.openstack.org/35077711:18
openstackgerritChristian Berendt proposed openstack/kolla: Add cloudkitty containers for type source  https://review.openstack.org/34929511:20
*** Mr_Broken has quit IRC11:20
-openstackstatus- NOTICE: DSVM jobs on OSIC currently failing because of IP collisions, fix is in the gate - https://review.openstack.org/#/c/357764/ - please hold rechecks until merged11:21
*** psanchez has quit IRC11:26
*** dmueller-evoila has joined #openstack-kolla11:26
*** psanchez has joined #openstack-kolla11:27
openstackgerritMd Nadeem proposed openstack/kolla: Added Ansible playbook for congress deploy  https://review.openstack.org/35062911:28
*** dwalsh has joined #openstack-kolla11:30
openstackgerritChristian Berendt proposed openstack/kolla: Add script to validate syntax of Jinja2 template files  https://review.openstack.org/35237811:31
*** dmueller-evoila has quit IRC11:35
openstackgerritChristian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container  https://review.openstack.org/33549511:35
openstackgerritChristian Berendt proposed openstack/kolla: Integrate neutron-vpnaas-agent into the neutron Ansible role  https://review.openstack.org/33550211:35
Davieypbourke: Hey.  What is the current state of OSIC?11:37
openstackgerritChristian Berendt proposed openstack/kolla: Add neutron-vpnaas-agent container  https://review.openstack.org/33549511:38
openstackgerritChristian Berendt proposed openstack/kolla: Integrate neutron-vpnaas-agent into the neutron Ansible role  https://review.openstack.org/33550211:38
*** ad_rien_ has quit IRC11:41
-openstackstatus- NOTICE: OSIC has burned through the problematic IP range with failures, things should be back to normal now.11:49
pbourkeDaviey: hey, rally scripts are ready now, so we're going back to the first deploy scenario and redoing11:54
pbourkeDaviey: right now the destroy op seems to be hung though11:54
pbourkeDaviey: if you have time could you take a look?11:54
*** mnasiadka has quit IRC11:54
*** dave-mccowan has joined #openstack-kolla12:08
*** zhurong has joined #openstack-kolla12:16
Davieypbourke: sure12:22
pbourkeDaviey: prechecks just ran through there12:22
pbourkeDaviey: not sure whats wrong with the destroy12:22
Davieypbourke: I bet my marbles it is ceph still mounted?12:23
pbourkeDaviey: hmm12:24
pbourkeDaviey: if prechecks pass we should be ok to deploy12:24
Davieypbourke: wedged again?12:25
Davieyah no12:25
pbourkeDaviey: :/12:28
*** rhallisey has joined #openstack-kolla12:32
*** dmueller-evoila has joined #openstack-kolla12:34
*** Pavo has quit IRC12:34
*** sdake has joined #openstack-kolla12:35
*** rhallisey_ has joined #openstack-kolla12:35
sdakemorning12:35
*** egonzalez90 has quit IRC12:36
*** zhubingbing_ has quit IRC12:36
*** zhubingbing_ has joined #openstack-kolla12:37
*** Pavo has joined #openstack-kolla12:38
Davieypbourke: Have we seen it wedge like that before?12:39
pbourkeDaviey: no12:39
Davieyoh hm12:39
*** rhallisey has quit IRC12:39
pbourkeDaviey: also dont know why passwords check is failing12:39
pbourkesigh12:39
*** dmueller-evoila has quit IRC12:40
Davieypbourke: somehow nova key was "null"12:41
Davieypbourke: regenen'd the passwords12:42
*** rhallisey_ is now known as rhallisey12:47
sdakeheads up re summit:12:50
sdakeExcellent question. The maximum number of Fishbowl+Workroom slots that12:50
sdakeyou can ask for in Barcelona is *16* (3 on Wednesday, 9 on Thursday, 412:50
sdakeon Friday morning).12:50
sdakefrom ttx12:50
sdakeanother gem is that there are more teams to accomadate, less time in the summit schedule, and less overall capacity12:51
*** dave-mcc_ has joined #openstack-kolla12:51
sdakemorning Daviey12:51
sdakemoring dave-mcc_12:51
*** diogogmt has joined #openstack-kolla12:51
sdakewhich means less slots12:51
dave-mcc_good morning steve12:51
sdakewe had the amximum allotment yesterday12:51
sdakesorry last summit12:51
sdakewe may get trimmed considerably :)12:52
sdakeespecially since one of our slots went sort of wasted from wht I ouldd tell12:52
*** mnasiadka has joined #openstack-kolla12:52
*** v1k0d3n has joined #openstack-kolla12:52
*** dave-mccowan has quit IRC12:53
Pavomorning sdake12:53
*** dave-mcc_ has quit IRC12:55
*** salv-orlando has quit IRC12:57
*** duonghq has joined #openstack-kolla13:03
pbourkesdake: coolsvap: Jeffrey4l: can we merge this please https://review.openstack.org/#/c/354429/13:03
patchbotpbourke: patch 354429 - kolla - Handle empty package list for install_packages13:03
pbourkeotherwise need to keep updating as other customisation patches merge13:04
sdakei thought i did13:04
sdakemaybe I just +2'ed it13:04
*** Serlex has quit IRC13:04
sdakeoh i ran a recheck on it13:04
sdakebecause it was failing the gate13:04
sdakeJeffrey4l coolsvap can one of you ack that patch if it lgtu13:05
sdakerhallisey ping13:06
pbourkesdake: yeah it was failing as other patches merged13:06
rhalliseysdake, hey13:06
coolsvapsdake: I'm afk13:07
pbourkeguys does anyone have the details of the networks we need to create post deploy13:08
*** coolsvap is now known as _coolsvap_13:08
pbourkei thought we'd updated init-runonce but it seems no13:08
pbourke*not13:08
*** Mr_Broken has joined #openstack-kolla13:08
sdake_coolsvap_ roger - got your msg last night - ping me if when your ready13:08
sdakepbourke yes just cut and paste the init-runonce script13:09
sdakeit creates a couple of things13:09
pbourkesdake: yeah but im not sure what needs to go in there13:09
sdakedaneyon really worked most of that script out13:09
pbourkenetwork ranges and the like13:09
pbourkeare custom for osic13:09
sdakethat is environmental13:09
sdakei hae no idea on that13:09
sdakethere is a tenant network and a public floating ip network13:10
sdakeboth are needed13:10
sdaketenant network can be anything iirc13:10
sdakefloating ip network must be something you can ssh into13:10
sdakebe careful with tenant networking be anything part though - not sure if that would break others vlan's in osic13:10
sdakebest to ping #osic13:11
sdakeif you need immediate answer13:11
*** Pavo has quit IRC13:12
duonghqwould you mind review my bp: https://blueprints.launchpad.net/kolla/+spec/ansible-specific-task-become13:12
sdakeduonghq too late for this cycle unless you make it finished and merged prior to 3113:13
duonghqsdake: of course not for this cycle13:14
duonghqso I should wait until next cycle and propose again?13:14
sdakeduonghq if youcan get it done for this cycle i'd love to see that implemented13:14
sdakeits a solid security feature13:14
sdakeits marked as essential and approved13:14
sdakeyou pick which cycle yo uwant it to go into based upon that13:14
sdakebut if not done by the 31st, it wont merge13:15
*** jtriley has joined #openstack-kolla13:15
pbourkeduonghq: you should ping sean-k-mooney before starting on that also, as I believe he did some work there as part of the baremetal stuff13:15
duonghqsdake: thank you very much, IMO it's quite heavy work but will try my best13:15
duonghqpbourke: glad to here that13:16
duonghqthank you13:16
sdakenote as long as the gates are green - even if it breaks something we can fix in the rc cycles13:16
sdakeduonghq roger - wasn't aware it was alot of work13:16
sdakepleae fill in the work items field din the blueprint13:16
sdakei'll leave it untargeted to a specific release for now13:16
duonghqsdake: I'll start on basic block module than build up13:17
sdakerhallisey can you review pbourke 's patch aove13:17
sdakehttps://review.openstack.org/#/c/354429/13:17
patchbotsdake: patch 354429 - kolla - Handle empty package list for install_packages13:17
duonghqsdake: iirc the Ubuntu gate is not stable yet?13:17
sdakeduonghq right - its on my plate as soon as I have 5 minutes to think about it13:18
pbourkeduonghq: it is not13:18
openstackgerritSzymon Konefal proposed openstack/kolla: Remove Kazoo dependency and Kolla Zookeeper helper  https://review.openstack.org/35786613:19
sdakerhallisey it looks long and ominous but its 99% sed and 1 file of importance that needs reviewing13:20
sdakepbourke would you mind expanding on what you know of this ubuntu gate problem13:20
sdakeyou said keys are wrong?13:21
sdakedo you have evidence of that ?13:21
sdakeif you know where to look in the log file - it would help me out considierably :)13:21
pbourkesdake: the problems seem to be changing each time i look at it13:21
sdakeprobably infra working on it my guess13:22
sdakeneed to sync up with them on the mirroring if its a mirroring problem13:22
pbourkethere was a key problem but im not seeing that on my latest patch13:22
pbourkeso that may be fixed13:22
sdakeif you ahve a link to a gate fialure recent one13:22
pbourkehttp://logs.openstack.org/29/354429/7/check/gate-kolla-dsvm-deploy-ubuntu-source-ubuntu-xenial-nv/ac3ca74/console.html#_2016-08-19_11_52_16_71289213:22
sdaketo the specific place it fails..13:22
sdakeya infra may have fixed hte keys automagically13:23
sdakelots of people rely on the uca and ubuntu mirrors in openstack13:23
sdakeso lots of folks pitch in to fix those and keep them operational13:23
pbourkewe'll maybe hold off and see how persistent that mariadb issue is, those errors tend to fix after a while also13:26
sdakeoh mysqldb module13:27
* sdake groans13:27
sdakeok well we can do an emergency fix there13:27
sdakefollowed by a proper fix in the rcs13:27
sdaketo unblock the gate13:27
sdakethat mysqldb module problem isn't going to fix itself magically13:27
sdakeand i don't have ubuntu - but know 1 (incorrect) way to resolve it13:28
pbourkeit might do, there was a similar neutron one other other day and by the time I had pushed a patch it had disappeared13:28
sdakeand I really dont want to reload my boxes with ubuntu at this time13:28
*** ayoung has joined #openstack-kolla13:28
sdakewas it mysqldb?13:28
sdakekolla-toolbox is a source-only container13:28
pbourkeno it was a fixtures module13:28
openstackgerritMerged openstack/kolla: Handle empty package list for install_packages  https://review.openstack.org/35442913:28
pbourkethe packages churn on their dependencies13:28
sdakeoh fixtures - ya dependencies like fixtures in neutron and friends always fix themselves over time13:29
pbourkethanks rhallisey13:29
sdakemariadb, not so much13:29
pbourkehmm13:29
rhalliseyno problem13:29
sdakeas you say they churn their deps13:29
sdakemaraidb has much less dep churn13:29
sdakemoment let me run a patch against the gate13:29
sdakebbiaf13:29
pbourkelooking a bit closer this is ansible13:30
pbourkethat should not have changed13:30
sdakeits complaining about a missing dependnecy13:30
sdakemysqldb module not present13:30
sdakei'll sort it out today - focused effort13:31
*** egonzalez90 has joined #openstack-kolla13:35
sdakemorning egonzalez9013:35
sdakeor evening as it may be ;)13:35
egonzalez90evening :)13:35
*** egonzalez90 has left #openstack-kolla13:35
*** egonzalez90 has joined #openstack-kolla13:36
sdakeman realestate in my area is off the hook pricewise13:36
duonghqevening13:36
sdakeglad i bought when i did13:36
openstackgerritChristian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce kolla_enable_cadf_notifications  https://review.openstack.org/34900613:39
openstackgerritChristian Berendt proposed openstack/kolla: Add cloudkitty containers for type source  https://review.openstack.org/34929513:42
*** ad_rien_ has joined #openstack-kolla13:43
openstackgerritChristian Berendt proposed openstack/kolla: Add script to validate syntax of Jinja2 template files  https://review.openstack.org/35237813:44
sdakeakwasnie akwasnie_ ping13:47
*** kolladude has joined #openstack-kolla13:51
sdakehey look we have a kolladude ;)13:51
Davieymorning sdake o/13:55
duonghqmany kollish around13:56
openstackgerritJeffrey Zhang proposed openstack/kolla: Create the nova compute instances when need during starting  https://review.openstack.org/35789813:57
sdakesup Daviey13:58
*** eaguilar has joined #openstack-kolla14:00
*** eaguilar has quit IRC14:01
*** diogogmt has quit IRC14:03
*** diogogmt has joined #openstack-kolla14:04
openstackgerritChristian Berendt proposed openstack/kolla: Enable CADF events in Keystone and introduce enable_cadf_notifications  https://review.openstack.org/34900614:05
*** daneyon has joined #openstack-kolla14:05
*** mnasiadka has quit IRC14:05
*** eaguilar has joined #openstack-kolla14:07
sdakewtb sean-k-mooney14:09
sdakeanyone seen him as of late on irc?14:09
*** huikang has joined #openstack-kolla14:10
wznoinsksdake: I'll ping him to drop in14:10
*** daneyon has quit IRC14:10
sdakewznoinsk thanks!14:10
openstackgerritJeffrey Zhang proposed openstack/kolla: Implement mongodb replication set cluster  https://review.openstack.org/34575114:10
openstackgerritJeffrey Zhang proposed openstack/kolla: Use dumb-init to manage the pid 1 process  https://review.openstack.org/35701514:14
openstackgerritMerged openstack/kolla: Update heat base Dockerfile to include footers  https://review.openstack.org/35724014:14
Jeffrey4lDaviey, ping14:17
Jeffrey4lDaviey, need your feedback https://review.openstack.org/34976414:17
openstackgerritPaul Bourke proposed openstack/kolla: Customizations for Nova  https://review.openstack.org/35167614:17
*** ad_rien_ has quit IRC14:18
*** wznoinsk has quit IRC14:19
*** Pavo has joined #openstack-kolla14:21
*** sean-k-mooney has joined #openstack-kolla14:23
sean-k-mooneyo/14:23
sean-k-mooneysdake where you looking for me a few minutes ago?14:24
sdakesean-k-mooney - sbezverk has a problem14:24
sdakesean-k-mooney I saw you sbumitted a review for neutron14:25
*** JoseMello has joined #openstack-kolla14:25
sdakesean-k-mooney any chance you can look at sbezverk's backtrace and provide guidance on how to resolve?14:25
sdakesean-k-mooney or find somone that can?14:25
sean-k-mooneyi have one open for vhost-user reconnect.14:25
sdakeI hae little contacts in neutron community14:25
sean-k-mooneysdake: sure i can take a look14:25
sdakeand don't understand its implementation one bit14:25
sbezverksean-k-mooney: I see some tracebacks generated by l3 agent14:27
sdakeone traceback I thught14:27
sbezverksdake: we started with one :-)14:27
*** Mr_Broken has quit IRC14:27
sean-k-mooneysbezverk: can you point me to them. im looking back at the irc logs now but i was not logged in when you strated talking about the issue14:27
sdakeok well sean-k-mooney can help you debug in his own style14:27
*** Mr_Broken has joined #openstack-kolla14:28
sbezverksean-k-mooney: https://paste.fedoraproject.org/410484/47155530/14:28
sean-k-mooneysbezverk: thanks ill take a look now14:29
*** wznoinsk has joined #openstack-kolla14:30
sean-k-mooneysbezverk: so first off i see an RTNETLINK answers: Invalid argument message. are you using the same host os as the container os? i havent read the full error yet but just taught i would ask14:30
sdakesean-k-mooney i picked upon that as well14:31
sbezverksean-k-mooney: yes, host is cenots and container is centos14:31
sdakesean-k-mooney he is using k8s as an underlay however14:31
sdakesbezverk ar eyou running with host networking mode?14:31
*** inc0 has joined #openstack-kolla14:32
*** Mr_Broken has quit IRC14:32
sdakemorning inc014:32
sdakehow was diablo grinding :)14:32
inc0I was drinking instead of diablo14:33
inc0but today I may play a bit14:33
sean-k-mooneyok so if they are both centos it should not be a netlink format issue. not sure k8s should change the networking behavior in this way but perhaps it would.14:33
sdakeso my wife showed me a picture of an alcohol isle in poland somewheerre14:33
sdakethere were two full isles of vodka ;)14:33
inc0isles of vodka?14:33
sdakeisle = isle in grocery store14:33
inc0ahh14:33
inc0yeah, sounds about right14:34
sdakeyou guys mut really like your vodka :)14:34
sean-k-mooneysbezverk: what mount points are shared between the l3 agent continer and the host14:34
sdakesean-k-mooney hadn't thought of that....14:34
inc0I'm personally not a huge fan, but we have few good ones, yeah14:34
*** eaguilar has quit IRC14:34
inc0Belvedere is best one you can buy around here14:34
sdakei like that vodka14:34
inc0yeah, it's Polish14:35
sdakeyou can buy all kinds of fantastic vodka in the us14:35
sean-k-mooneysbezverk: sdake: if /sys is not shared it may not be able to read/edit the network namespaces14:35
sbezverksean-k-mooney: /run and /run/netns14:35
inc0this is mainstream I know is here14:35
inc0havent seen much more14:35
inc0also, I'm not exactly a specialist in this field14:35
duonghqsdake: I draft some work items, but need to breakdown some item in times https://blueprints.launchpad.net/kolla/+spec/ansible-specific-task-become14:35
sean-k-mooneysbezverk: ok let me check if we share anyting else in the ansible deployment14:35
sdakebut you have to go to speciality store14:35
*** michauds has joined #openstack-kolla14:36
sdakeduonghq thanks :)14:36
NachoDuckHi all, I've been trying to get our Kolla eval to connect to our external network on a vlan provider but it doesn't seem to work. I've added the custom config in ml2_config.ini from the Red Hat docs (seems to be the most complete docs I've found) but I still can't ping an instance that has been assigned an IP from our local range, is there anything Kolla14:36
NachoDuckspecific I might have missed for this?14:36
duonghqsdake: I think n-3 is achivable, will try my best14:36
sbezverksean-k-mooney: but we do not have /sys in kolla14:36
sdakeduonghq - just so i know, are you working on any other current kolla related work?14:36
sdakeduonghq want to understand if that other work is higher prioirty14:36
sbezverksean-k-mooney: basically we mimic all shares from kolla in kube14:36
sdakesbezverk docker reporting the shares via docker inspect?14:37
sdakemaybe your specifying shares, but the shares raen't happening14:37
duonghqsdake: I also work on swift-kubernetes, but I think this bp is more important, so I will balance my time,14:37
sdakeduonghq definatley more important (new bp)14:37
inc0NachoDuck, soooo...we debugged tons of it in our osic cluster, but at the end it worked without any issue with kolla, so I can confirm it does work14:37
duonghqsdake: (the ansible is more important, I mean)14:37
duonghqsdake: agree14:37
sdakeduonghq agree 100%14:37
inc0so, what's your configs?14:37
sdakei'd say not even balance at this point14:37
*** zhubingbing_ has quit IRC14:38
duonghqsdake: much more effort on ansbile, of course14:38
sdakefocus on become localization14:38
sdakeanyway up to you14:38
sdakedo what you like :)14:38
sdakejust wanted to make sure something critical wasn't being dropped in favor14:38
sdake(critical in the next 2 weeks)14:38
sdakekubernetes-swift can wait awhile14:39
duonghqsdake: thanks, I will do ansible first, absolutely14:39
*** zhubingbing_ has joined #openstack-kolla14:39
sdakeduonghq we need compute kit oeprational first on k8s14:39
NachoDuckinc0: Alright well that's encouraging at least cheers. I didn't have to change much, here's what I added for the ml2_config - https://gist.github.com/anonymous/4d34064fcc832ebdc215258d69211d4314:39
sdakeduonghq if you prefer to work on k8s, may i suggest focusing on getting nova going there14:39
sdakeduonghq get a vm to boot is the objective14:39
sean-k-mooneysbezverk: with the k8s deployment can you still docker exec into the continer?14:39
inc0NachoDuck, just one tag? ok, how about your globals.yml?14:39
sdakeNachoDuck kolla community provides basic support on getting up and running14:40
sean-k-mooneysbezverk: im wondering if you can run the failing command youreslf and see if there is any other output?14:40
duonghqsdake: due to nova and neutron is on good proress, it is take time to catch up with them, so I will do the ansible become first14:40
duonghqsdake: do you agree with me?14:40
sdakebut we don't have a ton of operator experience on our team, so figuring out each environment for yu is not really our cup of tea14:40
sdakeit is hard for us to help there is what I mean to say :)14:40
sdakeduonghq agree14:40
sdakeneutron is covered14:40
sdakenova is not14:40
sdakethat may change by time become is reoslved14:41
NachoDuckinc0: Yep at the moment that's it. globals - https://gist.github.com/anonymous/8363ebfecbcf0b5bdcb6a899bc9493dc14:41
duonghqsdake: hope that, thank you14:41
sdakeNachoDuck but the architecture and implementation have been proved at 100 node scale14:41
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - l3/metadata DaemonSets + DVR  https://review.openstack.org/35755714:41
NachoDucksdake: Yeah I realise that's the case, I don't expect you to debug the entire thing but if its just a case of "yep what you've got should work" then maybe I can did a big deeper14:41
sdakewith ssh into vms and all the other things14:42
sbezverksean-k-mooney: I will give it ia try, for now I am back to fight with mutual hateness between kube and ovs14:42
sdakeNachoDuck one thing you might have setup wrong is your neutron netowrking after deployment14:42
sdakeNachoDuck - there is a developer script in the repo called init-runonce14:42
sdakeNachoDuck worth referencing - but recommend not using unless you tailor to your environment14:42
sean-k-mooneysbezverk: well the error look like the network namespace is not visable from within the l3 agent contaienr14:43
sdakesbezverk i'd double check that docker inspect on the nodes shows the shares you think it should be showing14:43
sdakesean-k-mooney thanks for the help :)14:43
NachoDucksdake: Alright noted, I have run that after my deployments just to get the images setup etc but it doesn't include an example of a vlan provider iirc?14:43
sdakeNachoDuck no idea what it does - daneyon wrote it14:44
sbezverksean-k-mooney: thank you for your help, so far your are the only one from neutron who even tried to help us :-)14:44
inc0NachoDuck, and em2 switch is configured for trunk right?14:44
sdakeNachoDuck all I know is it works in my environment ;)14:44
sdakeso i checked it in to the repo14:44
sdakeNachoDuck i am not a networking expert14:44
sdakeand as my wife says "Your company didn't  build their empire because networking is easy."14:44
NachoDuckinc0: Well the vlan I'm trying is the native vlan for that em2 port, so while I realise that might be a bit weird, it should work right?14:45
sean-k-mooneysbezverk: no worries. i have worked on nova and neutron for almost the last  3 years so if i can help im glad too14:45
NachoDucksdake: Yeah, no worries, I know some of the basics but by no means and expert either14:46
inc0NachoDuck, if it's native, try using flat network rather than vlan14:47
sean-k-mooneysbezverk: if ip netns shows the router namespace can you do a ip netns qrouter-cd1706eb-47a3-4496-83bc-8b5bef26a533 bash and let me know if it works14:48
NachoDuckinc0: Gotcha I'll give it a go14:48
inc0I'm not sure if tagged traffic can even work on non-trunked switchport14:49
egonzalez90+1 to inc0 if is native you have to use flat networks as the interface doesn't know that a vlan exists14:49
NachoDuckRight makes sense, well TIL. Cheers guys14:49
inc0no problem NachoDuck let us know if it worked14:50
duonghqsdake: can you set the milestones of ansbile bp to n-3, quite tight schedule but achivable14:51
sdakeduonghq you got it14:52
sdakeduonghq if it gets into danger territory let me know14:53
sdakeink to bp?14:53
duonghqsdake: sure14:54
sdakeok sean-k-mooney if you need to depart irc to focus on something else - understand - thanks for dropping in quickly :)14:55
sbezverksean-k-mooney: I tried that and got something like invalid namespace14:55
sbezverksean-k-mooney: ip netns exec qrouter-120a14d1-c943-4a4e-ab2b-a045d6386665 ifconfig14:56
sbezverksetting the network namespace "qrouter-120a14d1-c943-4a4e-ab2b-a045d6386665" failed: Invalid argument14:56
*** HyperJohnGraham has joined #openstack-kolla14:58
NachoDuckYep that works, cheers again :)15:00
Pavocan I specify what interface the kolla_external_vip_address goes on?15:00
*** salv-orlando has joined #openstack-kolla15:01
*** lrensing_ has joined #openstack-kolla15:01
sdakePavo - kolla_external_vip_address is for tls - are you using tls?15:01
sdakeif not, leave it commented out15:01
Pavoroger15:01
inc0NachoDuck, glad I could help15:01
Pavoso... kolla_internal_vip_address gets set on the network_interface which by default is the api_interface correct?15:02
openstackgerritHui Kang proposed openstack/kolla: Add missing container image names to build config file  https://review.openstack.org/35685415:03
Pavocan I set the api_interface to be different from the network_interface?15:03
*** zhurong has quit IRC15:03
inc0Pavo, it should, but never tried that;)15:03
inc0it has to be in same network tho15:03
Pavook15:04
inc0well...hmm...maybe even not15:06
openstackgerritMerged openstack/kolla: Start using orchestration_engine variable  https://review.openstack.org/35653815:06
inc0as long as it's routable15:06
openstackgerritAbishek Subramanian proposed openstack/kolla: Fix the location of Kolla configurations in the evaluation workflow  https://review.openstack.org/35752015:06
lrensing_hey guys, how would i go about fixing an error during kolla-ansible deploy:neutron : Waiting the openvswitch_db service to be ready15:07
sean-k-mooneysbezverk: interesting  so did qrouter-120a14d1-c943-4a4e-ab2b-a045d6386665 was visable when you did ip netns but ip netns exec failed15:07
*** salv-orlando has quit IRC15:08
inc0lrensing_, docker logs openvswitch_db15:08
openstackgerritSzymon Konefal proposed openstack/kolla: Remove Kazoo dependency and Kolla Zookeeper helper  https://review.openstack.org/35786615:09
sean-k-mooneysbezverk: one think i noticed on my ubuntu 16.04 system is that network nampseaces now show up in mount. http://paste.openstack.org/show/561354/15:10
sean-k-mooneysbezverk: do you see the namespaces mounted in /run/netns/15:11
sbezverksean-k-mooney: My setup is not running atm, since I started chasing another issue. I will double check when I recover it..15:17
kfox1111in a jinja2 template, whats the easiest way to get the first defined var?15:17
sean-k-mooneysbezverk: ok no worries. the command that the agent is running look correct so my assumtion is that for some reason the info cannont be read withing the continer.15:18
*** Serlex has joined #openstack-kolla15:18
sean-k-mooneysbezverk: either /run is not mapped shared, or something else in the container enviroment is preventing it from working15:19
sean-k-mooneysbezverk: from the error though im not sure what it is currently15:19
*** vbel has quit IRC15:20
*** vbel has joined #openstack-kolla15:21
inc0pbourke, damn15:29
inc0you guys merged too fast coolswaps reviews;)15:30
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838815:30
kfox1111ah. got it. |default()15:33
*** haplo37__ has joined #openstack-kolla15:34
*** mark-casey has joined #openstack-kolla15:36
Pavook here are all the cmds I ran to get the target nodes ready for deployment, someone mind seeing if I am missing anything http://pastebin.com/fZeGub9e15:36
*** daneyon_ has quit IRC15:36
Pavofor CentOS btw15:37
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935115:38
openstackgerritzhubingbing proposed openstack/kolla: repair aodh image dockerfile  https://review.openstack.org/35071515:38
*** ayoung has quit IRC15:39
openstackgerritMerged openstack/kolla: Customizations for Elasticsearch  https://review.openstack.org/35450815:41
*** _coolsvap_ is now known as coolsvap15:42
coolsvapinc0: any issues15:43
inc0coolsvap, sooo15:44
inc0yeah15:44
zhubingbing_hello15:44
inc0we can't ahve footers in base images15:44
inc0that will make build under proxy impossible15:44
coolsvapinc0: i was under the same impression15:44
sdakesup folks15:44
coolsvapand i -1ed 3-4 reviews15:44
coolsvapbut Jeffrey4l and pbourke convinced me that we need it15:45
inc0ehh15:45
zhubingbing_I want to know how this string can be obtained, implement-mongo-replicate-set-cluster-0d3f140f7116c3ba.yaml15:45
zhubingbing_0d3f140f7116c3ba15:45
sdakeinc0 - we have 1 week left of access to osic, is that correct?15:45
*** salv-orlando has joined #openstack-kolla15:45
Pavosdake does this look right for getting target host ready for deployment for CentOS http://pastebin.com/fZeGub9e15:45
Jeffrey4lcoolsvap, what?15:45
*** duonghq has quit IRC15:46
coolsvapJeffrey4l: the footer part in customization15:46
irtermitesdake: til 8/3015:46
inc0initial reason we have header and footer was to enable people to put ENV http_proxy=whatevere and in footer ENV no_proxy=my-networks15:46
sdakeirtermite thanks!15:46
Jeffrey4lcoolsvap, i think it is find. when need some thing like: i want to add a package in all nova related image.15:46
irtermiteand don't go over or I'll brake your knee caps for it back   ;)15:46
Jeffrey4ls/find/fine15:46
kfox11111;3D15:47
inc0Jeffrey4l, that's different thing15:47
irtermitejust don't tell anyone I gave you 4 instead of 3 weeks, sdake    ;)15:47
inc0and will be done with customizations15:47
mark-caseyrofl... that escalated quickly15:47
inc0and I'm ok with having named blocks in dockerfiles15:47
inc0but not anonymus footer15:47
Davieyinc0: TBH, i didn't bother adding proxy to the header.. I just use http_proxy in the environment before executing kolla-build15:48
Jeffrey4linc0, OK. but something like that, I want to add one file to the all nova images?15:49
Jeffrey4lDaviey, need your feedback https://review.openstack.org/34976415:49
DavieyLooking15:49
sdakeirtermite - i tell noone - unless its already in our docs :(15:49
DavieyAnyone on OSIC atm?15:49
irtermitehahaha is ok... not a big deal sdake15:50
inc0Daviey, that didn't work before, maybe docker fixed this15:50
coolsvapDaviey: I am not15:50
DavieyJeffrey4l: Sorry for the delay in responding to that.... I'll see if we can get any traction in #openstack-nova now?15:50
inc0Jeffrey4l, so with new customizations, you need to use customization file with nova_base_packages_append=[my_packages]15:51
Jeffrey4lDaviey, np. I am in now.15:51
sdakemark-casey - which escalated quickly15:51
inc0also if you'd just use --append-footer it would add it to every containers15:51
Jeffrey4linc0, my bad example.15:51
sdakemark-casey you mean the osic discussion above, or the dockerfile customizations?15:52
inc0Jeffrey4l, https://github.com/openstack/kolla/blob/master/doc/image-building.rst#package-customisation15:52
inc0also we can add named blocks, as I said15:52
inc0but not block footer15:52
inc0we can add block nova-base-footer15:53
openstackgerritMerged openstack/kolla: Add basic checking for network_interface to prechecks  https://review.openstack.org/35551515:53
mark-caseysdake: just irtermite's knee cap joke15:53
*** daneyon has joined #openstack-kolla15:54
openstackgerritMerged openstack/kolla: Optimize horizon cache and compress for http response  https://review.openstack.org/34644415:54
Jeffrey4linc0, hmm. wait. what we are talk?   nova-base-footer block i think...15:54
inc0in coolsvap15:55
inc0coolsvap's review, it's also block footer15:55
inc0so maybe I panicked;)15:55
irtermitehahha mark-casey15:55
inc0I'll make -1 out of -2, just removing block footer should be ok15:55
Jeffrey4linc0, coolsvap could we talk base on a code example?15:55
inc0https://review.openstack.org/#/c/357245/215:56
patchbotinc0: patch 357245 - kolla - Update ceilometer base Dockerfile to include footers15:56
coolsvapinc0: i will take care of it15:57
coolsvapwill update the patch15:57
coolsvapand rest of things which are merged15:57
inc0kk I'll scan our code to catch footers in base15:57
inc0ok, you can do it if you want15:57
inc0I'll remove -215:57
Jeffrey4lmy bad. the ceilomter_base_footer is find. but the footer block is not.15:58
*** daneyon has quit IRC15:58
inc0yeah, ceilometer_base_footer is ok15:58
*** matrohon has quit IRC16:02
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935116:03
openstackgerritzhubingbing proposed openstack/kolla: repair gnocchi  dockerfile  https://review.openstack.org/34838816:03
egonzalez90coolsvap: include senlin-base at https://review.openstack.org/#/c/357746/, block footers were merged today16:04
patchbotegonzalez90: patch 357746 - kolla - Update Dockerfile to include footers16:04
coolsvapegonzalez90: sure16:05
coolsvapI willcreate single patch16:05
egonzalez90cool16:05
coolsvapor probably update the last one16:05
DavieyJeffrey4l: FWIW, Kolla exposed a race condition in live migrations a few weeks ago, based on all the services starting concurrently16:06
DavieyJeffrey4l: they fixed it pretty quickly16:06
openstackgerritzhubingbing proposed openstack/kolla: Add gnocchi ansible role  https://review.openstack.org/34935116:07
Jeffrey4lDaviey, so?16:07
DavieyJeffrey4l: Just saying that #openstack-nova were pretty receptive to helping solve issues previously16:08
DavieyJeffrey4l: But not very responsive on a Friday evening i guess16:08
Daviey(no response yet)16:08
Jeffrey4lDaviey, yep they are.16:08
openstackgerritzhubingbing proposed openstack/kolla: Add aodh role  https://review.openstack.org/35102716:09
*** egonzalez90 has quit IRC16:14
DavieyJeffrey4l: If we get no response, i'll kick off a thread on the ML on Monday16:16
*** Serlex has quit IRC16:17
Jeffrey4lDaviey, OK. cool.16:17
openstackgerritKevin Fox proposed openstack/kolla-kubernetes: WIP - l3/metadata DaemonSets + DVR, & labels  https://review.openstack.org/35755716:18
openstackgerritJeffrey Zhang proposed openstack/kolla: Optimize horizon cache and compress for http response  https://review.openstack.org/35799616:19
*** coolsvap is now known as _coolsvap_16:20
openstackgerritJeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived  https://review.openstack.org/35799716:20
openstackgerritJeffrey Zhang proposed openstack/kolla: Use a random virtual_router_id in the keepalived  https://review.openstack.org/35799816:21
*** fragatina has quit IRC16:22
*** zhubingbing_ has quit IRC16:25
openstackgerritJeffrey Zhang proposed openstack/kolla: Fix the kolla_docker issue with docker 1.12  https://review.openstack.org/35800216:26
kfox1111does neutron-server actually talk to ovs? I didn't think it did.16:28
kfox1111I thought it talked to neutron-openvswitch-agent which did.16:28
inc0kfox1111, or lbaas right?16:42
openstackgerritChristian Berendt proposed openstack/kolla: Improve script to check used source versions  https://review.openstack.org/35264016:42
kfox1111yeah.16:42
kfox1111I'm wondering why there is an effort to expose ovs via tcp.16:43
kfox1111if it works with containers now, why does it need to be opened up to work with kolla-kubernetes? it shoudl work the same I think.16:43
inc0that would remove need of having ovs agent on cnode16:43
kfox1111ovs is needed on the network nodes.16:44
kfox1111but neutron-server can run without it I think.16:44
*** ayoung has joined #openstack-kolla16:44
kfox1111neutron-server shouldn't ever be talking directly to ovs.16:44
*** fragatina has joined #openstack-kolla16:45
sbezverkkfox1111: it does not16:47
sbezverkkfox1111: all goes via neutron-openvswitch-agent16:47
sbezverkkfox1111: according to Jeffrey4l accessing ovs via tcp is default way and it is more efficient than via socket16:48
kfox1111sbezverk: can you give this a try please: http://efox.cc/temp/hello-initcontainer.yaml16:48
*** athomas has quit IRC16:49
sbezverkalso the issue with socket is it gets created by root, but neutron-openvswitch-agent runs as neutron, so it fails accessing this socket16:49
kfox1111sbezverk: default for what?16:49
sbezverkfor openstack16:49
kfox1111...16:49
* kfox1111 goes and checks his systems...16:49
kfox1111its not default, at least in rdo.16:51
kfox1111socket permissions, I could see being an issue.16:51
kfox1111somehow rdo solved it though.16:51
*** fragatina has quit IRC16:51
*** fragatina has joined #openstack-kolla16:52
sbezverkkfox1111: that is why I proposed first to run as root, but kolla cores refused, second option was to switch to tcp port16:52
kfox1111my neutron-openvswitch-agent is also running as neutron.16:52
sbezverkbtw your example with init container worked16:53
sbezverkkfox1111: can you check permissions on ovs dn socket?16:54
kfox1111know off hand where it is?16:54
kfox1111sbezverk: awesome. :)16:54
*** srwilkers_ has joined #openstack-kolla16:54
kfox1111got a 10:00. so gota leave in a sec.16:55
kfox1111but somehow rdo's working with a very similar ovs setup.16:55
kfox1111ls -l /var/run/openvswitch/*sock /var/run/openvswitch/*.ctl16:56
kfox1111srwx------ 1 root root 0 Apr 19 18:48 /var/run/openvswitch/db.sock16:56
kfox1111srwx------ 1 root root 0 Apr 19 18:48 /var/run/openvswitch/ovsdb-server.1220.ctl16:56
kfox1111srwx------ 1 root root 0 Apr 19 18:48 /var/run/openvswitch/ovs-vswitchd.1270.ctl16:56
kfox1111neutron has User=neutron in systemd's unit.16:56
sbezverkkfox1111: can you explain how neutron-openvswitch can access this socket without any permissions ;-)16:56
kfox1111so its never starting as root.16:56
sbezverkkfox1111: I bet it goes over tcp port16:57
*** sould has joined #openstack-kolla16:57
kfox1111don't know. no ovs stuff is showing up in netstat as listening on tcp.16:58
kfox1111I'll look some more after the meeting.16:58
sdakeuid of ovsdb is what?16:58
kfox1111root.16:59
sdakewhy not change permission on the socket to group kolla?16:59
sdakesurely that is configurable16:59
sdakesbezverk ^^16:59
*** srwilkers_ has quit IRC17:00
sdakerather owner of socket17:00
sdakeand permission to 77017:00
sdakeeven if it is not configurable it can be changed after startup with a sudo operation17:00
sbezverksdake socket gets created when we start ovsdb process, I do not see where we can change17:01
*** dmueller-evoila has joined #openstack-kolla17:01
sbezverkthis permissions, I tried to change it manually whiule the process was running ir did not help17:01
sdakeafter ovsdb socket is created17:01
sdakewhich permissions did yo uuse?17:02
sdakesbezverk is this the root cause of your segfault?17:02
kfox1111sdake: for our system, we use openvswitch for the root networking too.17:02
PavoOMG finally have a complete deployment17:02
kfox1111we have 1 40gig connection coming in for the system, then break it out with vlans.17:02
sdakepavo grats!17:02
kfox1111so kolla can't own openvswitch on our system.17:03
sdakekfox1111 inereting17:03
sdakekfox1111 not sure how to add that in at the moment17:03
kfox1111el7 supports ovs in its native networkstack.17:03
kfox1111we can just skip the launching of the ovs containers I think.17:03
sdakethat may do it17:03
kfox1111sbezverk: is it a namespace issue of some kind maybe?17:05
sdake_coolsvap_ i responded on that vote thread so you could see the email now17:05
kfox1111oh....17:06
kfox1111maybe it is a namespace thing.17:06
kfox1111sbezverk: have you tried: hostIPC=true >17:06
kfox1111?17:06
sbezverksdake: no, as I mentioned there are multiple issue with ovs in kube it is just one of them..17:06
kfox1111otherwise the container has its own socket namespace.17:07
sdakeoh ya17:07
sbezverkkfox1111: usually with ipc I would see process gets stuck, but I can try17:07
sdakewonder if thats a bug in ansible code as well17:07
sdakesbezverk kfox1111 suggested using host ipc rather then container namespaced ipc17:08
sdakelthough I have only found this to be a problem on abstract sockets17:08
sbezverkkfox1111: IMHO the issue is kube starting by some unknown to me reason two identical pod on the same compute node, 1 pod stays 2nd goes away after a bit of time, but since they start at the same time they cause conflicts to each other17:09
sdakeas long as /run is host mounted17:09
Pavosurprising running actually well in VMs17:09
sdakepavo as it should :)17:09
sdakePavo once you sort out your vm networking config17:09
Pavofinally, after 4 days17:09
sbezverkkfox1111: they try to access the same socket, open the same port17:10
Pavoso... I had to put both interfaces on the same network17:10
sdakepavo fwiw - if you used bare metal would likely take 1-2 hours ;)17:10
Pavoyeah gonna try it out on 3 HP servers with 96 CPUS, 512Gb RAM and 24Tb of storage17:11
Pavowanna try out all services on allnodes17:11
sdakepavo some services probably work better then others17:11
*** lrensing_ has quit IRC17:11
Pavoin the inventory file there is nothing gonna go wrong if I add all nodes to all roles right?17:12
Pavotesting out reconfigure now17:14
sdakereconfigure and adding nodes = not tested17:14
sdakebut i think it shold work17:14
inc0512 Gig of RAM...you sir have a monster.17:15
Pavoyeah have 3 of them identical17:15
Pavoall of them have max cpus and RAM and 24Tb storage17:15
Pavoso reconfigure it used for adding nodes correct?17:16
sdakepavo deploy is used to add nodes17:16
sdakedeploy is idempotent17:16
Pavoah17:16
inc0it's like Daenerys and her 3 dragons:P (btw I just compared server hardware to fantasy dragons...you don't get more nerdy than that)17:16
sdakereconfigure might get you there17:16
sdakenot sure17:16
inc0anyway, that's an idea for hostnames isn't it?17:17
Pavoinc0 you know this is the last season of GoT coming correct?17:17
Pavoand its only gonna have 7 episodes :(17:17
inc0yeah17:17
Pavovery sad17:18
souldvery sad indeed17:18
inc0well, I've been through many endings like that that left me half-empty17:18
inc0Witcher 3 was the last one17:18
souldthe reimagined battlestar series left me feeling that way17:18
inc0battlestar was another.17:19
Pavoyeah that ending was weird17:19
*** dwalsh has quit IRC17:19
inc0it was kinda far fetched17:19
inc0but it was cool17:20
Pavoyeah, seemed like it rushed17:21
*** gfhellma has joined #openstack-kolla17:22
inc0gfhellma, Gustavo?:) welcome17:22
sdakebattlestar galactica season 3 could have been skipped and right on to seson 417:22
souldagreed17:23
sdakethe reason it seemed rushed is the writers didn't know what to do17:23
inc0sdake, not really, whole thing about Baltars trial17:23
sdakebecaue it was hard to top season 1 and 217:23
gfhellmainc0, good to be here! Thanks for the pointer! :)17:23
sdakethe right thing to do there would be to finish on a high note with season 4 :)17:23
souldthat was around the time of the writers strike right?17:23
souldi think thats why it went on hiatus for a bit with everything else17:23
sdakedont recall details17:24
sdakethat was years ago17:24
sdakethat was just the ipression it left me with :)17:24
inc0I kinda liked season 3 with all the post-new-caprica aftermath17:24
inc0anyway, all the good things has to end before they become bad things17:25
Pavotrue17:26
inc0that being said, I sometimes come back to OST from battlestar17:27
inc0I thing it is brilliant17:27
PavoOST?17:27
inc0soundtrack17:27
Pavooh17:27
souldthe rendition of all along the watchtower gives me chills -- i like it17:28
inc0like their version of All Along The Watchtower17:28
sbezverkkfox1111: ping17:28
inc0sould, exactly.17:28
sould:D17:28
inc0listening to it right now btw17:28
souldi just queued it up on spotify17:28
inc0the scene it was soundtrack was brilliant too17:29
inc0I think this was conclusion of season 4, when Kara went to the vortex17:29
kbaegisThey should've ended it a few episodes early17:30
kbaegisimho17:30
sdake30 rock was good all the wy to the end17:30
sdakefor those into more lighthearted idiot tube watching:)17:30
Pavowaiting for new Walking Dead17:31
sdakei tried to watch breakingbad17:31
Pavobreaking bad was awesome17:31
sdakebut after the 4th episode, too intense for me ;)17:31
kbaegissdake That's a great show. It keeps jumping the shark a little though17:32
sdakekbaegis a bit like the US election :)17:33
Pavowow lol17:33
kbaegissdake I'm trying to figure out if Trump or Clinton is the shark in this analogy :)17:33
inc0speaking of which, house of cards17:33
sdakehouse of cards rocks17:34
sdakebest scenee is the typewriter scene17:34
Pavothey both are bad and to think one of them is gonna be my commander in cheif17:34
Pavoso so sad17:34
inc0I think both of them...and keep in mind that fish aren't actually famous by their intellectual brilliance17:34
inc0for*17:34
sdakeinc0++17:35
*** HyperJohnGraham has quit IRC17:35
kbaegisinc0 lol17:35
sdakeinc0 learning to mindread ;)17:35
kbaegisSo I finally broke down and installed CentOS7 sdake17:37
kbaegisSince evidently no one likes supporting linux 4.8-rc2 :)17:37
Pavonothing wrong with CentOS17:37
Pavolol17:37
kbaegisYeah, I'm the crazy guy who actually got kolla running w/ gentoo, remember17:38
kbaegisIt works, but there are tons of workarounds17:38
PavoI would imagine sp17:38
Pavoso*17:38
kbaegisGreat learning opportunity :)17:39
sdakekbaegis glad you appreciated it;)17:39
Pavois sad that were I work we using only Windows machines but the actual job requires me to know everything about Linux17:40
Pavohow messed up is that17:40
souldi feel your pain17:41
inc0Pavo, such is life in big corpo....17:42
*** dwalsh has joined #openstack-kolla17:42
Pavoyeah17:42
Pavocan't wait to retire, so I can actually get a job that pays more for my skillset17:43
Pavo3 more years ugg17:43
souldalmost there :)17:46
kbaegisWindows, those things you stare at instead of getting work done :)17:46
Pavo^ +100017:46
sdakeinc0 feels your pain too :)17:46
sdakeat cisco we get a choice between mac or thinkpad17:46
kbaegissdake You're at Cisco?17:47
sdakeyup17:47
Pavonow that would awesome17:47
kbaegissdake Cool.  I was in AS for 5 years17:47
sdakemac gets  no support17:47
Pavonice I am military17:47
sdakekbaegis nice17:47
PavoI love macs17:47
sdakepavo thank you for your service :)17:47
PavoTy17:47
kbaegissdake All the hackers at Cisco get macs :)17:47
Pavosdake CCIE?17:48
kbaegisViso was the only reason I considered having a win machine17:48
sdakepavo not a big fan of certifications17:48
Pavono reason for Visio anymore, Lucid or Giffy to the rescue17:48
sdakegliffy rocks17:49
sdakehave to be a seroius rocket scientisst to get a ccie17:49
sdakesbezverk and daneyon have one17:49
Pavolol yeah heard that exam is a nightmare17:49
inc0Pavo, you'll gonna run kolla on tanks? damn our job just became even more awesome17:49
sdakepavo one piece of software i was responsible for birthing was corosync - used all over the military17:50
Pavonah no longer combat arms, was Inantry for 12yrs then got my right knee blown off in Iraq back in 2007 then went 25B basically IT and now Cyber Security17:50
Pavoinfantry*17:50
Pavodamn I can type17:51
Pavonice sdake17:51
*** huikang has quit IRC17:54
*** lyncos has quit IRC17:55
*** HyperJohnGraham has joined #openstack-kolla17:58
*** inc0 has quit IRC17:58
*** ad_rien_ has joined #openstack-kolla17:59
*** inc0 has joined #openstack-kolla17:59
*** lyncos has joined #openstack-kolla18:07
kfox1111sbezverk: back.18:14
kfox1111looking at your email.18:14
kfox1111sbezverk: ah. I think i see the issue.18:19
kfox1111the annotation is in the wrong place.18:19
kfox1111it needs to be in the spec->template->metadata18:19
kfox1111here's a little snippet I've been using for validation tests: http://pastebin.com/zahuBbTd18:22
kfox1111as its json encoded in yaml, sometimes you can mess up the format of the json and not know it. This helps debug it a little.18:22
kfox1111hopefully they make it proper yaml soon.18:23
Pavoso.. what are the neutron mapping kolla uses? ie physnet0:br-ex?18:24
Pavotrying to add a network and need it to be flat for external network18:25
inc0Pavo, default is physnet:<<neutron external interface >>18:25
Pavook18:25
inc0https://github.com/openstack/kolla/blob/master/ansible/roles/neutron/templates/ml2_conf.ini.j2#L5418:26
Pavogetting Failed to create nework18:26
inc0actually it's physnet118:26
inc0https://github.com/openstack/kolla/blob/ba30579b3b3e69536f5bd59e57d39ed4b63a2dd6/ansible/roles/neutron/defaults/main.yml#L6118:26
inc0so br-ex18:26
inc0and br-ex is set up on neutron external interface18:27
*** britthou_ is now known as britthouser18:28
Pavothe /var/lib/docker/volumes/kolla_logs/_data/neutron/neutron-openvswitch-agent.log shows this http://pastebin.com/69r8dbNF18:30
Pavolooks like it messed up18:30
*** dwalsh has quit IRC18:31
Pavonvm fixed it18:32
*** HyperJohnGraham has quit IRC18:35
*** HyperJohnGraham_ has joined #openstack-kolla18:35
*** HyperJohnGraham_ is now known as HyperJohnGraham18:35
*** daneyon has joined #openstack-kolla18:36
openstackgerritMichal Jastrzebski (inc0) proposed openstack/kolla: Customization for base - centos  https://review.openstack.org/35804218:38
*** daneyon has quit IRC18:41
*** lyncos has quit IRC18:48
*** egonzalez90 has joined #openstack-kolla18:59
pbourkeinc0: can you help us update init-runonce to create the correct networks for osic?19:00
*** HyperJohnGraham has quit IRC19:02
inc0yeah19:02
inc0let me get to tmux19:02
*** ad_rien_ has quit IRC19:08
inc0pbourke, I think we need to restart .2619:09
inc0oh...19:10
inc0pbourke, lets mount /dev/mapper/lxc-nova00 to /var/lib/docker19:10
*** salv-orl_ has joined #openstack-kolla19:15
pbourkeinc0: could do, i really want to get going on these tests though19:15
pbourkeinc0: seeing this hanging issue today19:15
pbourkeinc0: ansible just sits there19:15
*** salv-orlando has quit IRC19:17
inc0seems like this node is out of order on it's own19:17
*** gfidente has quit IRC19:18
pbourkewhich one19:18
*** egonzalez90 has quit IRC19:18
inc0pbourke, ok, we should be good now19:22
kfox1111sbezverk: looks like rdo does it with rootwrap.19:22
*** salv-orl_ has quit IRC19:26
*** salv-orlando has joined #openstack-kolla19:27
sdakegotta go - dentist appointment19:28
inc0sdake, dont tell it's root canal again19:28
sdakecleaning i hope19:28
sdakedefinately not a root canal19:28
sdakeunless they find something ominous19:29
sdakethen that will be a different day19:29
sdakehey folks - welcome egonzalez90 to the CR team!19:29
sdakeeven though he just logged off for the day ;-)19:29
sdakeif he needs any help getting started, please provide him a primer19:30
sbezverkkfox1111: strange we use the same image, it should aloso use rootwrap, all rootwrap configs are there?!?!19:31
pbourkeinc0: why are we building everything again?19:32
inc0i did git reset master19:33
pbourkedamn someones been ignoring the gates19:35
kfox1111sbezverk: guessing its just the ipc namespace thing then.19:36
*** dwalsh has joined #openstack-kolla19:38
*** sdake has quit IRC19:41
Mech422So how's OSIC going ?19:51
inc0going as we speak:)19:54
inc0neutron is deploying now19:54
Mech422inc0: oh cool - that should be a fun cluster to play with :-)19:54
pbourkeinc0: fyi the rally stuff is ready to go19:54
inc0cool!19:54
pbourkeso the current deploy is scenario 1 again19:54
Mech422I've got another week or two on the CloudStack stuff... then I should be back to Kolla19:54
pbourkeshould be quick enough to jump through them19:55
inc0Mech422, I think it's fair to say that we all learned a lot during this excersises19:55
Mech422inc0: oh?  Any big take-away's ?19:55
inc0besides that kolla is awesome?19:55
Mech422LOL19:55
inc0it took us a while to brush up our configs so we can run rally19:59
inc0but we managed to deploy openstack probably over a dozen times in last 2 weeks19:59
inc0big takeaway - elastic will eat through your storage space like there is no tomorrow20:00
inc0I wonder if we could back it up on ceph20:00
pbourkeinc0: I would say we need to limit it somehow20:01
Mech422inc0: I don't see why not?  You just just attach and RBD device to the host, mount/format it, and hand it to docker ?20:02
inc0yeah, but as part of kolla deployment20:02
pbourkeinc0: speaking of storage, we have users who dont like the data volumes thing as it lumps all data under /var/lib/kolla. Any appetite for making those configurable?20:02
pbourkes/kolla/docker20:02
inc0well, that depends on docker really20:03
inc0if docker can configure it, we do20:03
rhalliseypbourke, that's a docker change20:03
inc0you can configure /var/lib/docker location tho20:03
rhalliseyyou can specifcy where you want your data20:03
inc0which wiould be volumes + images20:03
pbourkeits more the abstraction of the data20:03
pbourkepeople want to get logs under /var/log/kolla, not /var/lib/docker/volumes/_data/blah/blah20:04
pbourkesame with /var/lib/nova and the like20:04
pbourkeso I was thinking of making the list of volumes for the main images variables20:05
pbourkeso they're configurable20:06
inc0well named volumes are something bound to docker20:07
inc0we would need to have different logic20:07
inc0that being said ln -s /var/lib/docker/volumes/kolla_logs/_data /var/log/kolla works20:08
pbourkehmm20:12
pbourkei like it!20:12
pbourkewill see if that does the trick, thanks inc020:12
pbourkerally tests are off now20:12
pbourkeseem to take a while20:12
inc0yeah20:13
pbourkeif they all work, then dump json+html, then deploy next scenario20:13
pbourkebritthouser: ^20:13
*** salv-orlando has quit IRC20:16
*** rhallisey has quit IRC20:23
*** daneyon has joined #openstack-kolla20:25
*** daneyon has quit IRC20:29
*** Jeffrey4l_ has joined #openstack-kolla20:34
*** Jeffrey4l has quit IRC20:35
*** dave-mccowan has joined #openstack-kolla20:51
*** ad_rien_ has joined #openstack-kolla20:55
*** sould has quit IRC20:55
*** kolladude has quit IRC20:57
*** fragatin_ has joined #openstack-kolla21:00
*** _coolsvap_ has quit IRC21:01
inc0it's done, next one with ceph21:02
*** mark-casey has quit IRC21:03
*** fragatina has quit IRC21:04
*** jtriley has quit IRC21:16
*** tummy has joined #openstack-kolla21:18
*** haplo37__ has quit IRC21:19
*** ntpttr___ has quit IRC21:20
*** ntpttr___ has joined #openstack-kolla21:24
*** pbourke has quit IRC21:25
*** unicell has quit IRC21:28
*** unicell has joined #openstack-kolla21:28
*** michauds has quit IRC21:30
*** dave-mccowan has quit IRC21:33
*** sdake has joined #openstack-kolla21:35
*** sdake has quit IRC21:37
*** sdake has joined #openstack-kolla21:38
*** fragatin_ has quit IRC21:39
sdakeyo21:39
*** fragatina has joined #openstack-kolla21:39
sdakerecruiters in a feeding frenzy on csco21:41
inc0cisco has mass layoff or mass recriutment?>21:41
sdakei've got 8 recruitment spams today21:43
sdaketypically on avg 1-2 a day21:43
inc0I had something like that when intel announced big layoff21:43
*** diogogmt has quit IRC21:48
Mech422inc0: cisco appears to be letting go a bunch of hardware types and hiring a bunch of software types21:53
sdakeI currently have 4 Senior Software Engineer positions with my client eBay based in San Jose. These are full-time positions and they will accept visa holders. The salary is in the 150-180K range with some flexibility.21:55
*** aNupoisc has joined #openstack-kolla21:55
sdakeeeeeee-bay21:55
sdakeyou know they are desparate when they include salary ranges in a recruiting email :)21:56
aNupoiscHi guys I am trying to install ceph with kolla on a single node, so my OSD, monitor both will sit in the same machine21:56
*** dmueller-evoila has quit IRC21:56
aNupoiscI am using all-in-one inventory file with hostname as localhost but it fails saying no hosts found21:56
aNupoiscis that not the correct way to do21:56
inc0aNupoisc, https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph make sure to read this21:56
aNupoisc?21:56
aNupoiscinc0: so that means it is a requirement to have 3 hosts atleast and one block device21:57
aNupoisc?21:57
inc0for ceph to work properly, yes, but there is hack to make it work aio21:58
inc0however that's not problem you're having21:58
inc0did you use default inventory or made any changes to it?21:58
aNupoiscinc0: i did followed this document and followed each step21:58
sdakekfox1111 about21:59
aNupoiscI use default inventory file21:59
aNupoiscinc0: i use default inventory file as there is nothing else than ansible_local and hostname as local in all blocks21:59
aNupoisc*localhost21:59
inc0paste plz your ansible output21:59
aNupoiscinc0: give me just few minute, it is setup on my other machine22:00
*** salv-orlando has joined #openstack-kolla22:00
sdakei'm not getting paid enough ;)22:00
inc0sdake, lol22:01
inc0I wish I'd be paid as much as you are22:01
sdakeIf you are not interested, I am almost positive you know of someone who might be interested and a good fit. For that, I will pay out a $1200 bonus if we can close this position as a result of your referral. Please introduce me to the referred person by giving me their name & email or phone.22:04
sdakeanyone wnat a job at ebay?  1200 bucks would be fun ;)22:04
sdakewhere did the kevin fox go22:07
aNupoiscinc0: here is my globals.yml http://pastebin.com/w74wLDg6 and here is my ansible output http://pastebin.com/XhQsHdT122:07
aNupoisci am now getting host unreachable22:07
aNupoiscas i changed the kolla_internal_ip_address to my current machine22:08
sdakeaNupoisc you shoudln't do that imo22:08
sdakethat will crater your network22:08
aNupoiscis it? I thought as kolla always uses current machine's address for internal_vip_address22:09
inc0aNupoisc, btw Anup?;) btw Michal here22:09
aNupoiscYup its Anup Michal22:09
sdakeJoin us at the Microsoft22:09
aNupoiscoh22:09
inc0sdake, really?22:09
sdakeboy you got to be kiddding22:09
sdakedo these poeple even read my linked in profile22:09
inc0well, they did recruit kubernetes lead engineer like a month ago22:10
sdakegood point22:10
inc0so there is something interesting out there22:10
aNupoiscsdake: what is there at Microsoft?22:10
aNupoiscsome conference?22:10
sdakea job apparently22:10
aNupoiscsdake: haha22:11
sdakemicrosoft turne dme down out of college22:11
inc0pain never healed fully?22:11
sdakenope22:11
inc0Mirantis turned me down, same thing22:11
sdakei had ot suck it up at intel instead out of university22:12
sdakea hardware company22:12
sdakefor a software nerd22:12
sdakeint he 90s22:12
sdakenot ideal22:12
sdakeduring the dot com crash22:12
aNupoiscsdake: inc0: there is no pain at least for now22:12
aNupoischaha22:12
ntpttr___inc0: this conversation led me to your linkedin22:13
ntpttr___inc0: you look a lot like eligah wood without your beard22:13
ntpttr___elijah*22:13
sdakeinc0 has a pic without a beard?22:13
ntpttr___https://www.linkedin.com/in/inc007?authType=name&authToken=_KpT&trk=prof-connections-name22:13
inc0it's an lod photo22:13
inc0old22:13
sdakeya i remember hwen michal linked in with me22:13
sdakeand i had no idea who he was22:13
sdakebut whatever ;)22:14
ntpttr___that's a young frodo baggins if I've ever seen one22:14
aNupoiscntpttr___ sdake: inc): curious to see everyone's profile now22:14
aNupoischehe22:14
sdakemichl we have known each other for about a year :)22:14
aNupoiscntpttr___ why have you put a looong underscore after your name. ;)22:14
ntpttr___aNupoisc: My IRC bouncer is having technical difficulties and I can't access it atm...22:15
ntpttr___I need to fix it at home22:15
ntpttr___so it's logged in but I can't reach it basically22:15
aNupoisclol22:15
aNupoiscokay22:15
inc0sdake, close to 1.5yrs now22:16
aNupoiscsdake: inc0: so does kolla_internal_vip_address you are assuming might be wrong?22:16
inc0but this photo is like 6yrs old22:16
sdakeaNupoisc interna lvip address should be unused address on your network22:16
inc0aNupoisc, something much more wrong is happening;) node unreachable == try to ssh to it manually22:16
inc0sdake, but tbh, your photo is not exactly actual as well now isn't it?;)22:17
aNupoiscinc0: sdake: ah okay, I am actually not using --private-key while running deploy command22:17
aNupoiscso ssh login is expected to fail22:18
sdakeinc0 its a head shot22:18
sdakeinc0 ever done those before?22:18
inc0only in computer games22:18
aNupoisclet me try to use private key and then ssh login22:18
*** tummy has quit IRC22:18
sdakesome day you will need  one ;)22:18
*** dmueller-evoila has joined #openstack-kolla22:18
sdakeprobably 33-36 range22:18
sdakeor you could get one now u pto you22:18
inc0yeah, when Intel will feel compelled to show me around, but for now I'm just lowly footslogger, no need for fancy photos22:19
sdakeinc0 bro ure almost on a world stage already - you just don't know it ;)22:19
openstackgerritMerged openstack/kolla-kubernetes: Adding NOTE for Kubernetes Secret Generator  https://review.openstack.org/35671922:20
inc0sshhhh not so loud or recruiters from microsoft will hear22:20
*** JoseMello has quit IRC22:20
inc0anyway I'm going for corporate dinner (yay for free food), cya around22:21
inc0sdake, if you have a while, there is deployment going on in osic cluster22:21
sdakeinc0 i'e had a rough week22:21
inc0when it finishes, would you plz run rally and dump results to ~/rally-scenerios/<< the later dir >>22:21
sdakefollowed by dentist22:21
inc0fair enough22:21
sdakeso prettym much no22:21
inc0ok, I'm off22:23
inc0have good weekend folks22:23
*** dmueller-evoila has quit IRC22:25
*** tummy has joined #openstack-kolla22:29
Pavoso... after finally getting running setup with 2 nodes, separate controller, compute. Tried adding all the services to both nodes for more of a HA fashion and gets stuck at starting neutron_server container on second node lol22:33
sdakepavo this is bare metal ?22:33
Pavonah, same VMs22:34
sdakeyou sure your interfaces are the same ordering on each vm?22:34
sdakethat sounds alot like you just cratered your network because you hve your interfaces backwards22:34
Pavoyes, used same globals just changed inventory22:35
sdakebut what about the vms?22:35
Pavoall the same as before22:35
aNupoiscinc0: hey so i was able to ssh to the machine using the key22:36
Pavoits all good, gonna clean the nodes and try again22:36
Pavothen gonna test adding a 3rd node22:36
aNupoiscinc0: and now I have again changed the kolla_internal_vip_address to something unused22:36
inc0yeah22:37
aNupoiscbut still the ansible fails22:37
aNupoiscthrowing unreachable22:37
inc0ansible can't ssh for some reason22:37
inc0you need to debug this22:38
inc0unfortunately I gtg22:38
aNupoiscactually this is strange I was first getting "no hosts found error" and now unreachable22:38
aNupoiscbut then do i need to change all-in-one?22:38
inc0talk to you guys later22:38
inc0!22:38
aNupoiscOkay. Happy friday22:38
inc0aNupoisc, you shouldn't22:38
aNupoiscokay cool22:38
inc0something is wrong in your env, that's my guess22:38
inc0anyway, see you all later22:38
Mech422aNupoisc: are you using IP address or hostname22:39
Mech422aNupoisc: if hostname, does it resolv properly22:39
*** inc0 has quit IRC22:39
aNupoiscMech422: IP address or hostname in what?22:39
Mech422aNupoisc: inventory file22:39
Mech422ssh can't find/contact your host22:39
aNupoiscMech422: oh in inventory I am using IP address22:40
Mech422ok22:40
aNupoisclet me paste that22:40
aNupoiscI have actually altered it22:40
Mech422aNupoisc: and you setup ansible to use the correct user for ssh ?22:41
aNupoiscMech422: here is my inventory http://pastebin.com/38DJy4T022:41
sdakeaNupoisc there is only one host in that inventory file22:42
Mech422seems like your trying to do an all-in-one install to a remote host ?22:42
aNupoiscsdake: yes I am doing it on that machine22:42
aNupoiscsdake: MEch422: let me change it tp localhost again22:43
aNupoiscsdake: Mech422: http://pastebin.com/UUrzfx2G22:46
aNupoiscthis is the new file22:46
sdakeaNupoisc localhost=10.3.66.64?22:46
aNupoiscYes.22:46
sdakehve you palced that in your /etc/host file?22:46
aNupoiscsdake: it is my current machine where I want to setup ceph22:46
sdakebefore setting u pceph22:47
sdakeget a working multinode deployment imo22:47
*** ad_rien_ has quit IRC22:47
Pavosdake here is my globals, http://pastebin.com/VekDu45P and here is my multinode inventory http://pastebin.com/01PbdfxD. Anything look wrong?22:48
sdakeline 26 is wrong22:48
Pavoin?22:48
sdakeor altnerivately your host ip is wrong22:48
sdakein that goba.syml22:48
aNupoiscsdake: in my /etc/hosts i have 127.0.0.1 as localhost 10.3.66.64 as otc-ironic(myhostname)22:49
Pavokolla_internal_vip_address: "192.168.1.254"22:49
sdakeya - that is not on the same network as 10.3.66.6422:49
PavoI don't have a network of 10.3.66.6422:49
*** tummy has quit IRC22:49
sdakeanything in inventory file should be a mangement network ip22:49
aNupoiscMech422: 127.0.0.1  localhost and 10.3.66.64  otc_ironic(hostname)22:49
sdakeyou do in your previous inventory file22:50
aNupoiscsdake: Mech422: should the /etc/hosts/ should have localhost 10.3.66.6422:50
sdakenope22:50
*** dwalsh has quit IRC22:50
Pavothink you might be thinking of someone else lol, I have always had only 2 networks, 192.168.1.0/24 and 192.168.188.0/2422:50
sdakeit should have a machine name insted of localhost22:50
sdakepavo sorry got mixed up between you and aNupoisc22:51
Pavoyeah figured that, your brain is getting fried brother22:51
sdakeaNupoisc paste your globals.yml please22:51
aNupoiscsdake: haha yeah i was reading how is his ip address same as mine22:51
aNupoisc:)22:51
sdakepavo tell me about it22:51
sdakei hate the dentist22:51
sdakewith a pasion22:51
Pavome too22:51
PavoI have to have oral surgery next month :(22:52
sdakemy condolences22:52
Pavoso.... sdake does anyting in my globals and inventory look wrong for having all services running on both nodes in a HA fashion?22:53
sdakepavo what is the result of doing a deploy?22:54
sdakecrater on openvswitch start?22:54
sdakeor neutron start?22:54
Pavostuck on neutron_server container on second node, first node gets done fine22:54
sdakepavo your interfaces are probably mixed around on the second host22:55
aNupoiscsdake: Mech422: here is my globals.yml http://pastebin.com/WjrPrvcY and my inventory file http://pastebin.com/UUrzfx2G and my /etc/hosts http://pastebin.com/8x9AXhjz22:55
sdaketry switching the network cables on the back of the gear22:55
Pavoroger22:55
sdakeon the 2nd node pavo not on the first22:55
Pavoroger22:55
aNupoiscsdake: are you having some dental surgery?22:55
sdakeaNupoisc no22:55
aNupoiscsdake: Pavo: I am scared of dentist22:56
sdakeaNupoisc just a cleaning where the flossing broke TWO of my fillings22:56
* sdake growls22:56
sdakebad bad mood22:56
PavoaNupoisc me too, everytime I got something bad or worse happens22:56
aNupoiscsdake: ahh and that too on Fri-Yay ;)22:56
sdakeya dentist is like "nah there wont be any sensitivity"22:57
sdakebullshit22:57
sdakehurts like a mofo22:57
aNupoiscsdake: seriously they always say such crap22:57
sbezverksdake: you need to ask extra shot of something.. ;-)22:57
sdakeshot of morphine in the ass maybe22:58
Mech422sdake: I had like 12 teeth pulled in the last couple of months22:58
sdakeok sbezverk on to your issue22:58
sdakepaste pl22:58
sdakeMech422 yieks!22:58
Mech422sdake: finally getting my partial lower done22:58
sdakei hae all original teeth22:58
sdakebut lots o crowns22:58
sdakeif I lose a tooth permanently -> implants22:58
Pavoone dentist once told me that they actually use the metal stuff on purpose because it causes pain, so people will learn to have better hygiene, I was thats so..... messed up22:58
Mech422sdake: I got kicked in the mouth as a kid - my front lower and jaw are all messed up22:59
sdakeMech422 sorry to hear it22:59
Mech422eh - it was nothing compared to all my wisdom teeth coming in wrong22:59
sdakeya i had those pulled at 1622:59
Pavoyeah I had all the enamel broken off on mine of being blown up 4 times in Iraq, so they are just dying22:59
Mech422I'm actually glad to be rid of them - nothing hurts for the first time in like 10-15 years23:00
*** salv-orlando has quit IRC23:00
sdakepavo war is hell :(23:00
*** v1k0d3n has quit IRC23:01
Pavomeh23:01
Pavobeen deployed 6 times, 4 times to Iraq, 2 times to Afghan23:01
*** v1k0d3n has joined #openstack-kolla23:01
Pavoit is what it is23:01
Mech422Pavo: wow - I didn't even know that was possible23:01
sdakewow - pretty hard core23:01
aNupoiscMech422: sdake: wisdom teeth surgery is baaaad23:01
Pavo67 months total deployment time23:01
*** v1k0d3n has quit IRC23:01
sdakethey wouldn't let me in the military23:01
sdakeeven if i wanted in23:02
aNupoiscPavo: you were in military?23:02
Pavoyeah Mech422 the force of an explosion is pretty bad on your teeth23:02
Mech422aNupoisc: I didn't have the surgery ... But when they come in and break/crack your other teeth, its a bitch23:02
*** v1k0d3n has joined #openstack-kolla23:02
PavoI am in the military23:02
aNupoiscPavo: Woww23:02
sdakei have a good riend who was in the rangers23:02
Mech422Pavo: of all the things I would think it would be bad for....I'd have never guessed that23:03
sdakehe said "bigger the gob better the job" is the dentist motto in the military ;)23:03
Pavoit wasn't so bad, only main one that really sucked was when I got my right knee blown off with a EFP23:03
aNupoiscMech422: Oh thank god still my wisdom teeth are giving that pain23:03
Pavolol sdake23:03
sdakehis kneeds afu because of jumping out of helicopters and airlaines and whatnot23:03
sdakepavo ya that sounds rough23:04
aNupoischaha sdake23:04
sdakesbezverk can you pasate here plz23:04
aNupoiscsdake: did you got chance to see my globals.yml23:04
aNupoiscand other files23:04
sdakesbezverk my next suggestion was going to be -> look in v/ar/run/openvswitchd in /run23:04
aNupoiscMech422:23:05
sdakeaNupoisc one person at a tiem guys- i'm not feeling super23:05
aNupoiscsdake: yes actually23:05
aNupoisci will wait23:05
*** v1k0d3n has quit IRC23:05
*** v1k0d3n has joined #openstack-kolla23:05
sdakespeaking of v1k0d3n23:05
Mech422aNupoisc: what happens when you try with the current setup ?23:05
sdakewtb23:05
*** daneyon has joined #openstack-kolla23:07
aNupoiscMech422: sdake: oh wow i am getting some new error http://pastebin.com/1ZKbRkLb23:07
sdakeaNupoisc run kolla-deploy with sudo pls23:07
Mech422aNupoisc: your ansible user doesn't have passwordless sudo setup ?23:08
* jrich523 takes a number23:08
Mech422aNupoisc: its a perms error...23:08
sbezverksdake: what do you want me to paste, lots track, sorry23:08
sdakeok folks - debugging 101 -> go from simple to complex23:08
sdakesbezverk the last docker inspect23:08
Pavoso... I think I might have figured what the issue is with the neutron server getting stuck during deployment23:09
sdakepavo cables swapped?23:09
Pavodamn vmware fusion keeps asking for my password when the interfaces go into promisc mode23:09
Pavobah23:10
sdakepavo wierd23:10
sbezverksdake: http://paste.openstack.org/show/561475/23:10
Pavoyeah23:10
aNupoiscMech422: sdake: yes there was no permission apparently and I forgot to run using sudo23:11
sdakeaNupoisc kolla can be run without sudo but its harder to setup and undocumented23:11
*** daneyon has quit IRC23:12
aNupoiscsdake: yeah I have documented it though on my side coz sometimes the errors that come are pretty weird to get to know the actual reason of failure23:12
*** peterjenkins has quit IRC23:13
*** smekel_ has quit IRC23:13
aNupoiscsdake: Mech422: now it is looking for Docker module, it is unable to find that.23:14
aNupoiscmay be i will install docker23:14
Mech422aNupoisc: yeah - the hosts your deploying to need docker23:15
sdakeaNupoisc precheck ftw :)23:15
aNupoiscsdake: Mech422: yeah i did not run prechecks23:15
*** peterjenkins has joined #openstack-kolla23:15
aNupoiscit is not mentioned in https://github.com/openstack/kolla/blob/master/doc/ceph-guide.rst#managing-ceph so missed23:16
*** smekel_ has joined #openstack-kolla23:18
*** zhurong has joined #openstack-kolla23:18
aNupoiscsdake: Mech422: let me run run prechecks.yml first that would be better23:19
aNupoiscsdake: Mech422: Is there any yml file that installs all the packages,services required for kolla setup?23:20
aNupoiscsdake: Mech422: i think the machine should have many packages like python-pip,python-dev,libssl,docker right?23:21
Mech422aNupoisc: I don't think so - I have my own ansible plays that set everything up23:21
Mech422aNupoisc: yeah - lots of stuff to install23:22
sdakethere are playbooks in the review queue that setup the hosts23:22
sdakebut its master only23:22
sdakeand wont be backported23:22
sdakeso that wont reallly help you23:22
aNupoiscsdake: Mech422: okay then let me do that otherwise the deploy ansible might fail at various sections23:23
aNupoiscMech422: that's good man23:23
aNupoiscMech422: even I will prepare something like taht23:23
Pavowould be nice to have a playbook to setup the target host23:24
aNupoiscMech422: sdake: my very first concern was it was not recognizing the hosts and that got resolved.23:24
Mech422aNupoisc, Pavo: here's a partial dump of my setup plays: http://paste.openstack.org/show/561487/23:25
Mech422it shows the kolla specific stuff I install23:26
Mech422I also install other crap, so the list might not be exhaustive23:26
Pavonice Mech422 but is that for latest Kolla or stable?23:26
Mech422(like I install openvswitch before I install kolla, so its not listed )23:26
Mech422Pavo: master23:27
Mech422Pavo: ansible 2.123:27
Pavoah23:27
Pavoyeah I'm running stable23:27
Mech422Pavo: just change the ansible=2.1.0.0 to ansible=1.9.x.y ?23:27
*** dmueller-evoila has joined #openstack-kolla23:28
sean-k-mooneythe kolla host playbook i have in review is for master but it might work with stable. i could probaly make the ansible version setable23:28
Pavohere is all the cmds I used on my deployer, ie.. master23:28
Pavohttp://pastebin.com/NSbEcZ1C23:28
aNupoiscMech422: much appreciate man. But do we need to create virtual env? Or is that for your setup?23:28
Pavoand for all target nodes http://pastebin.com/q0bNJ5e123:29
Mech422aNupoisc: thats just me23:29
Mech422aNupoisc: ignore everything except the package names23:29
aNupoiscMech422: okay and also we need to "mount --make-shared /run"23:29
aNupoiscis that right?23:30
aNupoiscMech422: okay23:30
Mech422Pavo: very nice :-)23:31
sean-k-mooneyaNupoisc: if you are on ubuntu 14.04 yes23:31
Mech422Pavo: looks like you have to install a lot less then me :-P23:31
* Mech422 mutters about ubuntu23:31
aNupoiscsean-k-mooney: yes i am using that23:31
PavoI have never really liked ubuntu myself23:31
Mech422I liked it a lot more before I started playing with Kolla23:32
Mech422:-P23:32
Mech422Not sure I'm gonna like the systemd crap in 16.0423:32
sean-k-mooneyafter years of devstack where i had to support ubuntu centos and fedora ubuntu has less issues but theyare all a pain23:33
Mech422it was a real pain to find docs on convertying my udev rules for renaming interfaces to systemd .link files23:33
aNupoiscMech422: Pavo: sean-k-mooney: you can also add --insecure-registry option in /etc/default/docker in this setup file23:33
Pavoyou can yes23:33
PavoI use a secure registry though23:33
sean-k-mooneyaNupoisc: yes my kolla host playbook will do that for you23:33
Mech422aNupoisc: yeah - Pavo does it right, I'm lazy and use insecure23:33
*** zhurong has quit IRC23:34
aNupoischere is my setup ansible http://pastebin.com/fUhusWqi23:34
PavoI need to start getting into ansible23:35
Mech422aNupoisc: looks good :-)23:35
Pavohas alot of power that I could leverage to automated slot of stuff23:35
*** dmueller-evoila has quit IRC23:35
Mech422Pavo: oh hey - do you happen to know anything about generating certs for openvpn with easy-rsa?23:35
aNupoiscsean-k-mooney Mech422:I also use insecure ;)23:35
sdakesbezverk - check out this path on the HOST :                 "/var/run/openvswitch:/var/run/openvswitch",23:35
sdakesee if it contains a socket file23:36
aNupoiscsean-k-mooney: can you paste your host playbook may be i missed it23:36
sbezverksdake: it does23:37
sean-k-mooneyaNupoisc: sure its availble for review here https://review.openstack.org/#/c/325631/823:37
patchbotsean-k-mooney: patch 325631 - kolla - add baremetal role to install kolla deps.23:37
sbezverksdake: it seems it is in "good: phase now. I restarted multiple times and it holds..23:37
sean-k-mooneyaNupoisc: i will be adding the bootstrap-build-host part later next week23:38
sdakesbezverk so - your all set then?23:38
sdakesbezverk mind sharing with the channel what you learned about yaml files23:38
sbezverksdake: for these two process at this point of time yes, but I have no confidence it will not break in next 10 minutes23:39
Pavosdake does kolla supprt trunking network_interface and neutron_external_interface?23:39
sdakepavo if by trunking you mean bonding23:39
sdakepavo yes we have it setup that way in osic23:39
Pavotrunking I mean by 802.1q23:40
Pavoallowing multiple vlans23:40
sean-k-mooneyPavo: as in vlan trunking23:40
sdakepavo i am not a networking expert, but I believe we setup 802.1q mode in bonding driver23:40
Pavoroger23:40
Pavothats good23:40
sdakeno idea what vlan trunking is sorry :)23:40
Pavoah23:40
sdakewe setup some 80xxxx mode in bonding ddriver23:41
sbezverkfor folks intersted in kube. if you use construct: command/args to execute a command in a pod, it is safer to provide arguments on each line instead of on one line separated by comma23:41
sdakei could be off on the actual mode23:41
PavoI am a networking geek myself23:41
sean-k-mooneypavo you can check out the osic network config in the ether pad23:41
aNupoiscsean-k-mooney: great!!23:41
sdakesbezverk mind pasting a before and after for kfox111123:41
Pavosean-k-mooney doesn't that require kubernates?23:41
sean-k-mooneyPavo: we were testing with both bonds and vlan subports23:41
sbezverksdake: kfox1111 always using right way :-)23:41
aNupoiscalso do we need configuring virtualization? I mean kvm everytime23:42
sean-k-mooneyPavo: doent what require kuberneties?23:42
sdakeconfigure how?23:42
sdakeinsmod?23:42
sdakeaNupoisc if you mean insmod, no kolla does that for you23:42
Pavoosic sean-k-mooney ?23:42
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113923:42
sean-k-mooneyosic no23:43
sdakepavo open stack innovation center is providing us with 131 nodes for kolla scale testing23:43
Pavoah lol23:43
PavoI'm a newb23:43
Pavomybad23:43
sdakeinitial results look good23:43
sdakewe have documented the setup in an etherpad as well as a gerrit review23:43
Pavonice23:43
aNupoiscsdake: i mean just enable virtualization feature on that machine23:43
sdakeand eventually it will be published on docs.oo23:43
sbezverksdake: sean-k-mooney: I have concern about location of ovs db. Do you mind share your thoughts about it.23:43
*** dmueller-evoila has joined #openstack-kolla23:43
sdakeaNupoisc - you have to do that in bios23:43
aNupoiscfor me its intel kvm module23:43
Pavowell I would really clean the docs up before adding something that complex, lol just saying23:44
sean-k-mooneytesting kolla-k8s is out of scope currently but it would be interesting to do maybe next cycle when it gets closer to a complete solutio n23:44
openstackgerritShaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet  https://review.openstack.org/35113923:44
sdakepavo - its a new doc - like to see?23:44
Pavosure why not23:44
Pavolol23:44
sean-k-mooneysbezverk: sure23:44
aNupoiscsdake: but no need to change anything in system files right?23:44
sdakepavo: https://review.openstack.org/#/c/352101/23:45
patchbotsdake: patch 352101 - kolla - Add OSIC Scale Testing Documentation23:45
sean-k-mooneysbezverk: what are you worried about in terms of the ovsdb?23:45
Mech422aNupoisc: actually, I think you need to enable tcp listener and stuff for live migration23:45
sbezverksdake: sean-k-mooney: Originally ovs db was located on a host mount and I noticed that this approach is not ideal as from time to time I see some leftovers from previous runs..23:45
sdakejrich523 your up23:45
aNupoiscsdake: i just remember sometime in between using "modprobe kvm-intel" so thought to ask23:45
jrich523lol awesome thanks :)23:45
aNupoiscMech422: okay then I should not worry23:45
sdakeaNupoisc - we do that automatically23:45
jrich523im trying to get the ansible compute deployment working23:45
sbezverkthen I decided to try to use it on local to the pod volume, since no need other processes accessing directly23:46
jrich523so i took the all-in-one and basically added my host in the [compute] section at the top like so: 1.2.3.4 ansible-connection=ssh23:46
sbezverkthis approach also suffers from some issue. example if pod gets restarted that volume is gone and previous config needs to be replayed23:46
sdakejrich523 i'd start with the multinode example23:46
jrich523which seemed cool? so i then attempted to import it like you said, kolla-ansible -i filename23:47
jrich523ok23:47
jrich523but, the real problem (i think) is the syntax for adding it23:47
sdakei think you dont want ansible-connection=ssh23:47
sdakebut i am not sure23:47
jrich523it basically just spews help23:47
Pavoouch latency seems pretty high23:47
sdakekolla-ansible -i /etc/kolla/multinode-inventory deploy23:47
jrich523ahh missed deploy23:48
jrich523i'll take a look at multinode before i do this tho23:48
jrich523thanks23:48
sean-k-mooneysbezverk: well when a new ovsdb container is spawned the ovsdb should be iniilised from a clean copy of the schema and bootstraped correct.23:48
sean-k-mooneysbezverk: is that the case with the k8s deployment23:48
sdakepavo ya those are old results23:48
Pavoah23:48
sdakepavo we had a kernel problem in the scale testing23:48
sbezverkso my question is should stick with host location and try to deal with leftovers or keep it on local container space and see how old config can be replayed.23:48
sdakewe are in process of rerunning the tests23:48
Pavoonly 4 control nodes for 101 computes?23:48
sdakewe have about 10-12 scenarios23:48
Pavoseems not enough23:48
sdakepavo our model is 3 control + 90 compute + 8 storage23:49
sdakeor a ratio thereof23:49
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936623:49
sbezverksean-k-mooney: yes it is bot kube23:49
Pavolooks very promising and interesting23:49
sbezverkI mean for kube case23:49
Pavowould love to see a networking diagram23:49
sdakepavo however you could run 9 control node 90 compute if you like23:49
sdake3 control nodes is plenty for 100 nodes23:49
openstackgerritShaun Smekel proposed openstack/kolla: Add full support for fernet  https://review.openstack.org/34936623:50
sean-k-mooneysdake 1 contoler should be plenty for 100 compute nodes but 3 for ha23:50
sdakesean-k-mooney agree23:50
Pavoseriously?23:51
sean-k-mooneysbezverk: i would lean towords keeping the ovsdb local to the container23:51
Pavoseems like it would be api overload lol23:51
sean-k-mooneyPavo: it depends on how beefy the contoler is23:51
Pavotrue23:52
Pavowe plan on using a quad 40Gb DAC for our apis23:52
*** dmueller-evoila has quit IRC23:52
sean-k-mooneysbezverk: the ovs agent will recreate the flows and interconnect the bridges as needed when it connects to the ovsdb23:52
sbezverksean-k-mooney: thanks I will see how replay might be done23:52
sbezverksean-k-mooney: not all23:52
sean-k-mooneysbezverk: what will need to be done by k8 is adding the phyical interfaces23:53
sbezverkit will not created initial br-ex23:53
sbezverkwhich in kolla is done during bootstrap23:53
sean-k-mooneysbezverk: it will if you put it in the OVS_BRIDGE_MAPPINGS23:53
sean-k-mooneysbezverk: at least i think it does23:54
sbezverksean-k-mmoney: hm it is not what I observed. if I do not have br-ex added to ovs neutron-opnevswitch-agent crashes23:54
sbezverkor exists with error code23:55
sean-k-mooneyi will have to check but i know it will create the br-int if it does not exist23:55
aNupoiscwhile i was trying to install docker using above playbook I am hitting this error http://pastebin.com/m2Buvzw223:55
sean-k-mooneysbezverk: i taught it had the same logic to create br-ex23:55
sbezverksean-k-mooney: yep br-int for sure23:55
aNupoiscit is unable to authenticate docker-engine ;)23:56
sbezverkunfortunately no, I hope it is a bug becuase if fixed it will simplify my life23:56
jrich523ls23:57
jrich523opps23:57

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!