| *** hieulq has quit IRC | 00:01 | |
| *** sdake has quit IRC | 00:10 | |
| *** hieulq has joined #openstack-kolla | 00:16 | |
| *** daneyon has joined #openstack-kolla | 00:18 | |
| *** banix has quit IRC | 00:33 | |
| *** haplo37__ has quit IRC | 00:40 | |
| *** zhiwei has joined #openstack-kolla | 00:46 | |
| *** ssurana has quit IRC | 00:47 | |
| *** phuongnh has joined #openstack-kolla | 00:58 | |
| *** zhurong has joined #openstack-kolla | 01:06 | |
| *** yuanying has quit IRC | 01:14 | |
| *** yuanying has joined #openstack-kolla | 01:14 | |
| *** yuanying has quit IRC | 01:15 | |
| *** zhurong has quit IRC | 01:16 | |
| *** zhurong has joined #openstack-kolla | 01:17 | |
| *** yuanying has joined #openstack-kolla | 01:18 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 01:22 |
|---|---|---|
| *** unicell1 has quit IRC | 01:37 | |
| *** banix has joined #openstack-kolla | 01:56 | |
| openstackgerrit | Lu lei proposed openstack/kolla: Modify some spelling mistakes https://review.openstack.org/350544 | 02:06 |
| *** dave-mccowan has quit IRC | 02:19 | |
| *** lulei has joined #openstack-kolla | 02:25 | |
| *** daneyon_ has joined #openstack-kolla | 02:29 | |
| *** daneyon_ has quit IRC | 02:33 | |
| *** zhurong has quit IRC | 02:36 | |
| *** zhurong has joined #openstack-kolla | 02:39 | |
| *** yuanying has quit IRC | 02:51 | |
| openstackgerrit | sean mooney proposed openstack/kolla: add baremetal role to install kolla deps. https://review.openstack.org/325631 | 02:53 |
| *** hieulq_ has joined #openstack-kolla | 02:53 | |
| *** yuanying has joined #openstack-kolla | 02:54 | |
| *** yuanying has quit IRC | 02:56 | |
| *** sdake has joined #openstack-kolla | 03:04 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 03:05 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 03:05 |
| *** banix has quit IRC | 03:14 | |
| *** senk has joined #openstack-kolla | 03:17 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi image dockerfile https://review.openstack.org/348388 | 03:20 |
| *** coolsvap has joined #openstack-kolla | 03:22 | |
| *** sdake has quit IRC | 03:23 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 03:33 |
| *** jrist has joined #openstack-kolla | 03:34 | |
| *** senk has quit IRC | 03:39 | |
| *** godleon has joined #openstack-kolla | 03:49 | |
| godleon | hi all, one question about the 'openstack_release: "3.0.0"' in /etc/kolla/globals.yml | 03:49 |
| *** sdake has joined #openstack-kolla | 03:50 | |
| godleon | where can I get informations of openstack_releases ? | 03:50 |
| *** yuanying has joined #openstack-kolla | 03:51 | |
| *** yuanying has quit IRC | 03:51 | |
| *** yuanying has joined #openstack-kolla | 03:52 | |
| openstackgerrit | Md Nadeem proposed openstack/kolla: Correct mistral-executor image name https://review.openstack.org/350638 | 03:55 |
| sdake | godleon could you expand on your question? Do you mean where can you determine what changed in 2.0.2 vs 2.0.1? | 03:57 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi image dockerfile https://review.openstack.org/348388 | 03:57 |
| *** daneyon has quit IRC | 04:01 | |
| coolsvap | morning :) | 04:08 |
| *** ayoung has quit IRC | 04:08 | |
| sdake | yo coolsvap | 04:11 |
| coolsvap | sdake: hehy | 04:12 |
| coolsvap | hey* | 04:12 |
| coolsvap | did we come to conclusion how we are tracking osic cluster updates | 04:12 |
| coolsvap | etherpad? | 04:12 |
| sdake | good question | 04:12 |
| sdake | i've been looking for inc0 to answer that question today | 04:13 |
| sdake | i think that was the idea behind the "geo leaders" thread he was going to kick off | 04:13 |
| coolsvap | hmm | 04:13 |
| *** senk has joined #openstack-kolla | 04:29 | |
| *** sdake has quit IRC | 04:37 | |
| *** zhiwei has quit IRC | 04:38 | |
| *** zhiwei has joined #openstack-kolla | 04:39 | |
| *** senk has quit IRC | 04:43 | |
| *** unicell has joined #openstack-kolla | 04:58 | |
| *** unicell has quit IRC | 05:05 | |
| godleon | skake: yap, something like that. Now 3.0.0 seems is the default setting of openstack_releases. | 05:06 |
| *** unicell has joined #openstack-kolla | 05:08 | |
| godleon | sdake: yap, something like that. Now 3.0.0 seems is the default setting of openstack_releases. | 05:08 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 05:08 |
| *** daneyon has joined #openstack-kolla | 05:11 | |
| *** daneyon has quit IRC | 05:16 | |
| *** yee_ has joined #openstack-kolla | 05:29 | |
| *** senk has joined #openstack-kolla | 05:33 | |
| *** haplo37__ has joined #openstack-kolla | 05:34 | |
| *** senk has quit IRC | 05:35 | |
| *** dwalsh has joined #openstack-kolla | 05:36 | |
| *** haplo37__ has quit IRC | 05:42 | |
| *** dwalsh has quit IRC | 05:47 | |
| *** dwalsh has joined #openstack-kolla | 05:49 | |
| *** dwalsh has quit IRC | 05:54 | |
| *** unicell has quit IRC | 05:58 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 06:14 |
| *** yee_ has quit IRC | 06:15 | |
| *** mewald has joined #openstack-kolla | 06:15 | |
| *** yee_ has joined #openstack-kolla | 06:18 | |
| openstackgerrit | zhongshengping proposed openstack/kolla: Add upgrade tasks for Ceilometer https://review.openstack.org/351007 | 06:19 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi image dockerfile https://review.openstack.org/348388 | 06:24 |
| *** zhubingbing has joined #openstack-kolla | 06:27 | |
| zhubingbing | hi | 06:27 |
| *** belmoreira has joined #openstack-kolla | 06:30 | |
| *** yee_ has quit IRC | 06:34 | |
| *** unicell has joined #openstack-kolla | 06:35 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 06:39 |
| *** unicell1 has joined #openstack-kolla | 06:39 | |
| *** unicell has quit IRC | 06:39 | |
| mewald | morning | 06:43 |
| *** imcsk8 has quit IRC | 06:49 | |
| *** imcsk8 has joined #openstack-kolla | 06:49 | |
| *** Mr_Broken has joined #openstack-kolla | 06:50 | |
| *** kklimonda_ has quit IRC | 06:50 | |
| *** kklimonda_ has joined #openstack-kolla | 06:50 | |
| *** bootsha has joined #openstack-kolla | 06:52 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi aodh role https://review.openstack.org/351027 | 06:54 |
| *** imcsk8 has quit IRC | 06:54 | |
| *** nradojevic has joined #openstack-kolla | 06:55 | |
| *** imcsk8 has joined #openstack-kolla | 06:55 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 06:56 |
| *** dwalsh has joined #openstack-kolla | 06:57 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 06:58 |
| nradojevic | Hi, I'm trying to build tempest (with image base tag: 14.04 | docker tag: 3.0.0) but it fails with the error:Depends: python-tempest (= 2013.2.a1291.g23a1b4f-0ubuntu2) but it is not going to be installed. E: Unable to correct problems, you have held broken packages. Did someone faced similar issues? | 07:05 |
| *** b_bezak has joined #openstack-kolla | 07:07 | |
| zhubingbing | hi sadel | 07:19 |
| *** anbehl has quit IRC | 07:21 | |
| *** Serlex has joined #openstack-kolla | 07:26 | |
| *** berendt has quit IRC | 07:30 | |
| *** openstackgerrit has quit IRC | 07:33 | |
| *** openstackgerrit has joined #openstack-kolla | 07:33 | |
| *** dwalsh has quit IRC | 07:35 | |
| *** Mr_Broken has quit IRC | 07:43 | |
| openstackgerrit | Jack Ning proposed openstack/kolla: Replace keystone's wsgi config with custom config https://review.openstack.org/351045 | 07:44 |
| *** Mr_Broken has joined #openstack-kolla | 07:44 | |
| *** shardy has joined #openstack-kolla | 07:48 | |
| *** Mr_Broken has quit IRC | 07:48 | |
| *** daneyon has joined #openstack-kolla | 07:53 | |
| *** daneyon has quit IRC | 07:58 | |
| *** Mr_Broken has joined #openstack-kolla | 07:58 | |
| *** Serlex has quit IRC | 07:59 | |
| *** dwalsh has joined #openstack-kolla | 08:02 | |
| *** bmace has quit IRC | 08:07 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron-metering-agent https://review.openstack.org/346631 | 08:08 |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron_metering_agent to ansible tasks for neutron https://review.openstack.org/346630 | 08:08 |
| *** Mr_Broken has quit IRC | 08:08 | |
| *** dmueller-evoila has joined #openstack-kolla | 08:09 | |
| *** timonwong has joined #openstack-kolla | 08:14 | |
| *** Mr_Broken has joined #openstack-kolla | 08:15 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 08:18 |
| *** bootsha has quit IRC | 08:19 | |
| *** Serlex has joined #openstack-kolla | 08:37 | |
| *** zhubingbing_ has joined #openstack-kolla | 08:38 | |
| *** zhubingbing has quit IRC | 08:41 | |
| *** bootsha has joined #openstack-kolla | 08:46 | |
| *** zhubingbing_ has quit IRC | 08:51 | |
| *** dwalsh has quit IRC | 08:59 | |
| *** dwalsh has joined #openstack-kolla | 08:59 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 09:10 |
| *** berendt has joined #openstack-kolla | 09:17 | |
| *** mnasiadka has joined #openstack-kolla | 09:39 | |
| *** mnasiadka has quit IRC | 09:39 | |
| *** mnasiadka has joined #openstack-kolla | 09:39 | |
| *** daneyon has joined #openstack-kolla | 09:42 | |
| *** daneyon has quit IRC | 09:46 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 09:48 |
| *** mnasiadka has quit IRC | 09:52 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 09:53 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Install neutron-lbaas-dashboard from source in horizon container https://review.openstack.org/336943 | 09:54 |
| *** zhubingbing_ has joined #openstack-kolla | 09:57 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Install neutron-lbaas-dashboard from source in horizon container https://review.openstack.org/336943 | 10:01 |
| *** zhurong has quit IRC | 10:04 | |
| *** dwalsh has quit IRC | 10:11 | |
| *** zhiwei has quit IRC | 10:15 | |
| *** hieulq_ has quit IRC | 10:18 | |
| openstackgerrit | Sajauddin proposed openstack/kolla: Fixes Bug: 1513904. Added a condition to skip debug log for those services which are build from other base service images https://review.openstack.org/351108 | 10:21 |
| openstack | bug 1513904 in kolla "Clean up debug logs when building images" [Wishlist,In progress] https://launchpad.net/bugs/1513904 - Assigned to Sajauddin (sajauddin-mohammad) | 10:21 |
| *** timonwong has quit IRC | 10:33 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 10:52 |
| *** mnasiadka has joined #openstack-kolla | 10:55 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Document the Dockerfile customisation mechanism https://review.openstack.org/351123 | 10:57 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Document the Dockerfile customisation mechanism https://review.openstack.org/351123 | 11:00 |
| *** bootsha has quit IRC | 11:01 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 11:03 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 11:03 |
| *** zhubingbing_ has quit IRC | 11:05 | |
| mewald | hey guys, question on architecture: We currently have elasticsearch and kibana on the control nodes. Personally, I think monitoring should go away from the controllers. I dislike the idea of having the monitoring stack on those nodes with the highest density of services to be monitored … I would also like to see that load go away from controllers. What do you think? | 11:07 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Document the Dockerfile customisation mechanism https://review.openstack.org/351123 | 11:07 |
| *** Mr_Broken has quit IRC | 11:07 | |
| *** Mr_Broken has joined #openstack-kolla | 11:08 | |
| *** Mr_Broken has joined #openstack-kolla | 11:08 | |
| *** godleon has quit IRC | 11:09 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron-metering-agent https://review.openstack.org/346629 | 11:13 |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron_metering_agent to ansible tasks for neutron https://review.openstack.org/346630 | 11:13 |
| *** bootsha has joined #openstack-kolla | 11:14 | |
| pbourke | mewald: it probably makes sense in a prod environment, not sure we need to splitting out dedicated logging nodes in the sample inventories though | 11:20 |
| mewald | pbourke: yeah makes sense maybe to keep the footprint small for small installations | 11:24 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 11:24 |
| berendt | mewald I think it makes sense to introduce a new type for monitoring, but to install on the control nodes by default | 11:26 |
| mewald | right that would be another way, ok! | 11:30 |
| *** daneyon has joined #openstack-kolla | 11:30 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 11:33 |
| *** dwalsh has joined #openstack-kolla | 11:33 | |
| *** rhallisey has joined #openstack-kolla | 11:34 | |
| *** daneyon has quit IRC | 11:34 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add aodh role https://review.openstack.org/351027 | 11:37 |
| *** mliima has joined #openstack-kolla | 11:42 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 11:43 |
| *** srwilkers-mac has joined #openstack-kolla | 11:49 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes: Glance bootstrap to create service/project/role https://review.openstack.org/346215 | 11:51 |
| *** srwilkers-mac has quit IRC | 11:51 | |
| *** srwilkers-mac has joined #openstack-kolla | 11:52 | |
| sbezverk | rhallisey: last night I got an experimental version of hyperkube supporting net=host and custom dns | 11:53 |
| *** srwilkers-mac has quit IRC | 11:53 | |
| sbezverk | rhallisey: it is working now, querying for internal dns names from net=host containers | 11:54 |
| *** dave-mccowan has joined #openstack-kolla | 11:55 | |
| *** nradojevic has quit IRC | 11:56 | |
| rhallisey | sbezverk, what was the issue before | 11:56 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 11:56 |
| rhallisey | brb | 11:57 |
| *** dwalsh has quit IRC | 11:58 | |
| *** phuongnh has quit IRC | 11:58 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 12:00 |
| *** srwilkers-mac has joined #openstack-kolla | 12:02 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 12:04 |
| *** srwilkers-mac has quit IRC | 12:04 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 12:05 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 12:07 |
| *** lulei has quit IRC | 12:09 | |
| *** ppowell has joined #openstack-kolla | 12:09 | |
| *** zhurong has joined #openstack-kolla | 12:13 | |
| *** timonwong has joined #openstack-kolla | 12:13 | |
| *** timonwong has quit IRC | 12:13 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 12:15 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 12:15 |
| *** diogogmt has joined #openstack-kolla | 12:17 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 12:23 |
| *** zhiwei has joined #openstack-kolla | 12:23 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Use yaml.safe_load instead of yaml.load https://review.openstack.org/351147 | 12:24 |
| *** diogogmt has quit IRC | 12:24 | |
| *** diogogmt has joined #openstack-kolla | 12:25 | |
| *** zhiwei has quit IRC | 12:28 | |
| *** diogogmt has quit IRC | 12:35 | |
| pbourke | berendt: ping | 12:38 |
| berendt | pbourke pong | 12:39 |
| pbourke | berendt: hey, Im just provisioning the deploy node in osic | 12:39 |
| pbourke | berendt: I'm booting a Ubuntu ISO they reference, but its dropping me right into a shell rather than giving the option to install | 12:39 |
| pbourke | berendt: any ideas how to kick off the install by any chance? | 12:40 |
| *** ankush has quit IRC | 12:41 | |
| *** anbehl has joined #openstack-kolla | 12:42 | |
| *** anbehl is now known as ankush | 12:42 | |
| *** jtriley has joined #openstack-kolla | 13:04 | |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Add doc8 test and improve rst syntax https://review.openstack.org/351173 | 13:11 |
| *** sdake has joined #openstack-kolla | 13:11 | |
| *** zhiwei has joined #openstack-kolla | 13:11 | |
| *** dmueller-evoila has quit IRC | 13:15 | |
| *** dmueller-evoila has joined #openstack-kolla | 13:15 | |
| *** zhiwei has quit IRC | 13:16 | |
| *** Liuqing has joined #openstack-kolla | 13:17 | |
| *** daneyon has joined #openstack-kolla | 13:18 | |
| sdake | pbourke how goes the osic gear | 13:19 |
| sdake | or britthouser ? | 13:19 |
| pbourke | sdake: britthouser will be online in a few mins | 13:19 |
| sdake | btw, 1% power | 13:19 |
| pbourke | sdake: good so far, querying cloudnull atm about an issue | 13:19 |
| sdake | so if i dc, i'll be back in a bit | 13:19 |
| *** daneyon has quit IRC | 13:23 | |
| *** lulei has joined #openstack-kolla | 13:23 | |
| *** sdake has quit IRC | 13:24 | |
| *** srwilkers-mac has joined #openstack-kolla | 13:24 | |
| *** srwilkers-mac has left #openstack-kolla | 13:25 | |
| *** srwilkers-mac has joined #openstack-kolla | 13:25 | |
| cloudnull | o/ :) | 13:26 |
| pbourke | cloudnull: I was chatting with asettle who pointed me at you :) | 13:27 |
| pbourke | cloudnull: I said I'd feed back doc changes | 13:27 |
| * cloudnull goes to yell at asettle :) | 13:27 | |
| pbourke | ha | 13:27 |
| cloudnull | that's great. | 13:27 |
| cloudnull | the folks managing that doc, and the ones that will be updating it will greatly appreciate it | 13:28 |
| *** godleon has joined #openstack-kolla | 13:29 | |
| *** srwilkers-mac has quit IRC | 13:29 | |
| *** gfidente has quit IRC | 13:29 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 13:30 |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 | 13:30 |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb container https://review.openstack.org/346446 | 13:30 |
| *** srwilkers_ has joined #openstack-kolla | 13:30 | |
| *** srwilkers_ has left #openstack-kolla | 13:30 | |
| *** matrohon has joined #openstack-kolla | 13:30 | |
| *** srwilkers_ has joined #openstack-kolla | 13:30 | |
| *** srwilkers_ has left #openstack-kolla | 13:31 | |
| *** inc0 has joined #openstack-kolla | 13:32 | |
| inc0 | good morining | 13:33 |
| *** Jeffrey4l_ has quit IRC | 13:34 | |
| pbourke | hey inc0 | 13:35 |
| inc0 | hey Paul, did you try OSIC cluster already? | 13:35 |
| pbourke | inc0: yeah been working on it | 13:35 |
| *** srwilkers_ has joined #openstack-kolla | 13:35 | |
| inc0 | hows it going? | 13:35 |
| *** gfidente has joined #openstack-kolla | 13:36 | |
| *** gfidente has joined #openstack-kolla | 13:36 | |
| *** srwilkers_ has quit IRC | 13:36 | |
| pbourke | inc0: got a little stuck installing the deploy node, the image linked in the docs is bust. Your colleages pointed me in the right direction :) | 13:36 |
| *** srwilkers_ has joined #openstack-kolla | 13:36 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 13:36 |
| *** srwilkers_ has quit IRC | 13:36 | |
| *** srwilkers_ has joined #openstack-kolla | 13:37 | |
| *** srwilkers_ has left #openstack-kolla | 13:39 | |
| *** srwilkers has joined #openstack-kolla | 13:39 | |
| *** zhubingbing_ has joined #openstack-kolla | 13:43 | |
| zhubingbing_ | hi | 13:43 |
| inc0 | pbourke, cool, I'll help you | 13:43 |
| inc0 | let me get first coffee today | 13:43 |
| pbourke | inc0: im confused | 13:43 |
| inc0 | what confuses you? | 13:43 |
| pbourke | inc0: these servers should have *no* OS installed right | 13:43 |
| pbourke | currently | 13:44 |
| inc0 | is there OS installed? | 13:44 |
| inc0 | I thought they'll be scrubbed clean | 13:44 |
| pbourke | well that's whats confusing me, I thought the ISO was booting me to a shell but now I think it may be booting from the drive | 13:44 |
| inc0 | there might be OS tho | 13:44 |
| cloudnull | pbourke: that might be an issue. | 13:44 |
| inc0 | as they were cleaning old OS and there is no better way than to simply reinstall | 13:45 |
| * cloudnull assumed they we're scrubbed too | 13:45 | |
| britthouser | The email says, "We have finished deploying your bare metal server allocation" | 13:45 |
| *** mgoddard_ has joined #openstack-kolla | 13:45 | |
| pbourke | ok, I guess I just need to find why its not booting from the ISO/cdrom then | 13:45 |
| britthouser | which I took to mean "OS install completed" | 13:45 |
| inc0 | pbourke, try root:cobbler pass | 13:45 |
| pbourke | inc0: yeah that works | 13:45 |
| inc0 | maybe we can just use it then | 13:46 |
| inc0 | but raid configuration might be wrong for us | 13:46 |
| zhubingbing_ | cobbler？ | 13:46 |
| pbourke | britthouser: let me check some of the other nodes | 13:46 |
| pbourke | inc0: ^ | 13:46 |
| zhubingbing_ | boys，help | 13:47 |
| zhubingbing_ | review my code | 13:47 |
| zhubingbing_ | https://review.openstack.org/#/c/351027/ | 13:47 |
| patchbot | zhubingbing_: patch 351027 - kolla - Add aodh role | 13:47 |
| *** mgoddard has quit IRC | 13:48 | |
| zhubingbing_ | https://review.openstack.org/#/c/350715/ | 13:48 |
| patchbot | zhubingbing_: patch 350715 - kolla - repair aodh image dockerfile | 13:48 |
| *** matrohon has quit IRC | 13:48 | |
| zhubingbing_ | https://review.openstack.org/#/c/348388/ | 13:48 |
| patchbot | zhubingbing_: patch 348388 - kolla - repair gnocchi image dockerfile | 13:48 |
| zhubingbing_ | https://review.openstack.org/#/c/349351/ | 13:49 |
| patchbot | zhubingbing_: patch 349351 - kolla - Add gnocchi ansible role | 13:49 |
| *** v1k0d3n has joined #openstack-kolla | 13:52 | |
| *** sdake has joined #openstack-kolla | 13:54 | |
| *** wrouesnel1 has joined #openstack-kolla | 13:55 | |
| *** srwilkers has quit IRC | 13:57 | |
| *** jemcevoy has joined #openstack-kolla | 13:57 | |
| pbourke | ok second host also has an OS | 13:58 |
| *** mgoddard_ has quit IRC | 13:58 | |
| pbourke | so these nodes are provisioned | 13:58 |
| inc0 | ok, cool | 13:58 |
| inc0 | we can use it | 13:58 |
| *** mgoddard has joined #openstack-kolla | 13:58 | |
| inc0 | let's try to log into the | 13:58 |
| inc0 | m | 13:58 |
| pbourke | will push on in the guide and see what other set up is needed | 13:58 |
| pbourke | I dont think they have IPs | 13:58 |
| inc0 | nothing useful for us | 13:58 |
| pbourke | ssh to the IP in the email brings you into the ILO | 13:59 |
| inc0 | let me log in | 13:59 |
| openstackgerrit | Will Rouesnel proposed openstack/kolla: Volume specification for ironic-api was invalid. https://review.openstack.org/344775 | 13:59 |
| inc0 | yeah, we need to login via console | 13:59 |
| inc0 | and see ip a | 13:59 |
| pbourke | it still should have booted off the ISO though | 13:59 |
| pbourke | dont know whats wrong there | 13:59 |
| inc0 | did you change boot order? | 13:59 |
| pbourke | yeah | 14:00 |
| sdake | yo cats | 14:00 |
| pbourke | by default it tries to boot via dhcp | 14:00 |
| pbourke | but fails | 14:00 |
| sdake | inc0 - i haven't RTFMed yet on the osic cluster - is there a vpn client for macos? | 14:00 |
| inc0 | pbourke, sorry what I'm about to say...but I hate Java. | 14:00 |
| pbourke | sdake: no mention of one in the email | 14:01 |
| pbourke | sdake: unless you can find a generic OSX F5 client | 14:01 |
| inc0 | sdake, yeah, it's just web browser plugin | 14:01 |
| pbourke | inc0: everyone does | 14:01 |
| sdake | inc0 ok thanks | 14:01 |
| pbourke | inc0: there's worse though | 14:01 |
| *** Mr_Broken has quit IRC | 14:01 | |
| pbourke | sdake: I couldnt get the browser plugin working fyi | 14:01 |
| pbourke | sdake: prob be ok on OSX though | 14:02 |
| *** belmoreira has quit IRC | 14:02 | |
| *** Mr_Broken has joined #openstack-kolla | 14:02 | |
| pbourke | inc0: https://imgur.com/a/RsPOV | 14:02 |
| openstackgerrit | Will Rouesnel proposed openstack/kolla: Volume specification for ironic-api was invalid. https://review.openstack.org/344775 | 14:02 |
| *** dwalsh has joined #openstack-kolla | 14:02 | |
| pbourke | dhclient hangs | 14:03 |
| inc0 | hmmm | 14:03 |
| inc0 | ipv6 | 14:03 |
| inc0 | let's try this one | 14:03 |
| inc0 | nah | 14:03 |
| inc0 | my bad | 14:03 |
| inc0 | no ipv6 | 14:03 |
| inc0 | pbourke, so in the pdf attachment you have networks | 14:04 |
| inc0 | pxe network will be native vlan | 14:04 |
| inc0 | so let's try to manually add ip from network 172.21.0.0/22 | 14:04 |
| inc0 | first 20 IPs are reserved, so let's make it .21:) | 14:05 |
| *** matrohon has joined #openstack-kolla | 14:05 | |
| pbourke | let me double check there's nothing in the docs about this | 14:05 |
| inc0 | I don't think so | 14:05 |
| *** matrohon has quit IRC | 14:05 | |
| *** Mr_Broken has quit IRC | 14:07 | |
| jemcevoy | I am installing kolla for the first time on four hosts... Got as far as installing MariaDB when it failed trying to set the password for root in the database. I did not specify any versions when I ran kolla-build to populate my local registry... | 14:07 |
| jemcevoy | This is the command I ran to load the registry... /home/jmcevoy/Kolla/kolla/.tox/genconfig/bin/kolla-build --base centos --type binary --registry ops1.penguincomputing.com:4000 --push | 14:07 |
| inc0 | jemcevoy, did you prepare password.yml? | 14:07 |
| pbourke | cobbler is meant to configure networking | 14:07 |
| wrouesnel1 | urgh, to save me spamming a whole bunch of change proposals - what's the process to propose a change against multiple branches with gerrit ? | 14:08 |
| jemcevoy | Yes... I ran the passwd generator | 14:08 |
| inc0 | so you do have /etc/kolla/password.yml right? all filled? | 14:08 |
| inc0 | ahh | 14:09 |
| inc0 | so build looks good | 14:09 |
| jemcevoy | Yes I have /etc/kolla/passwords.yml filled with password | 14:09 |
| inc0 | ok, that's strange, can you show mi log from your ansible run | 14:09 |
| inc0 | ? | 14:09 |
| jemcevoy | It built like 100 docker images and took hours | 14:10 |
| inc0 | also, make sure that there is no mysql or otherwise already running | 14:10 |
| jemcevoy | Paste the log here? | 14:10 |
| inc0 | ok jemcevoy soo...did you use default driver? | 14:10 |
| inc0 | no, use paste.openstack.org | 14:10 |
| inc0 | for docker? | 14:10 |
| inc0 | change it to aufs/btrfs/overlayfs | 14:11 |
| inc0 | default storage driver for docker is devicemapper and it's horrible horrible thing | 14:11 |
| inc0 | build shouldn't take more than 30min unless your network packets are carried by pidgeons | 14:11 |
| *** coolsvap has quit IRC | 14:11 | |
| pbourke | inc0: could you find out why these nodes are provisioned and how? | 14:13 |
| openstackgerrit | Will Rouesnel proposed openstack/kolla: Volume specification for ironic-api was invalid. https://review.openstack.org/351206 | 14:14 |
| sean-k-mooney | inc0: IPoAC will out last us all :) | 14:14 |
| pbourke | inc0: the way I see it in the current state we may be as well to reinstall them according to the guide, then again they may be 95% of the way there | 14:14 |
| inc0 | pbourke, I bet they just reinstalled OS from previous users | 14:15 |
| jemcevoy | The log of MariaDB is here Paste #549233 | 14:15 |
| openstackgerrit | Will Rouesnel proposed openstack/kolla: Volume specification for ironic-api was invalid. https://review.openstack.org/351208 | 14:15 |
| pbourke | inc0: if you have a moment then see if you can get the ISO to boot? | 14:16 |
| pbourke | inc0: actually its fine you're busy, me and britthouser will figure it out | 14:16 |
| inc0 | nah, don't worry | 14:16 |
| inc0 | but since we already have OS we can log to | 14:16 |
| sean-k-mooney | wrouesnel1: generally you propose against master and then it gets backported via a cherry pick | 14:16 |
| inc0 | let's just use it as deployment node at least | 14:17 |
| inc0 | so we won't need ISO at all | 14:17 |
| britthouser | did you use java console pbourke? or install the .NET shtuff? | 14:17 |
| inc0 | britthouser, Paul is from Oracle...I think not usig java can get him fired;) | 14:18 |
| inc0 | :P | 14:18 |
| britthouser | oh right....lol | 14:18 |
| pbourke | (cartman voice) i use what I want | 14:18 |
| britthouser | I'm about to fire up my third browser | 14:18 |
| pbourke | except .NET I'm alergic to windows | 14:18 |
| pbourke | the java console works well though | 14:18 |
| sdake | is will rousnel in channel? | 14:19 |
| sdake | oh i see sean-k-mooney is already on it | 14:19 |
| pbourke | britthouser: next step is to configure networking on the deploy node then | 14:19 |
| sdake | wrouesnel1 - re your patch - looks good, but the stable:follows-policy mandates a master first approach | 14:19 |
| sdake | wrouesnel1 if you submit your patch to master, the stable team will take care to backport it to the various branches (or you can if you like) | 14:20 |
| britthouser | Are we deciding .226 is the deploy node? | 14:20 |
| pbourke | yes | 14:21 |
| sdake | wrouesnel1 nm, I see you have submitted a patch for master | 14:21 |
| sean-k-mooney | pbourke: :( i always liked .net c# was a nice improvemtet on java imho. | 14:21 |
| britthouser | :burn: | 14:21 |
| pbourke | sean-k-mooney: decent language but windows only | 14:21 |
| *** matrohon has joined #openstack-kolla | 14:22 | |
| pbourke | sure they open sourced it eventually but too little too late | 14:22 |
| sean-k-mooney | pbourke: well there is mono and the .net foundation is bringin it to linux with full support from microst redhat+others | 14:22 |
| pbourke | java is the best: https://deadhomersociety.files.wordpress.com/2011/06/margegetsajob4_thumb.png?w=512&h=384 | 14:23 |
| sdake | pbourke ;) | 14:23 |
| sdake | i think a knife is missing from that picture | 14:23 |
| *** signed8bit has joined #openstack-kolla | 14:24 | |
| sdake | i guess smithers doesn't bring a knife to a gunfight tho :) | 14:24 |
| *** mnasiadka has quit IRC | 14:25 | |
| sean-k-mooney | pbourke: java is fine too ill happily code in both but i miss things like delegate/function pointer and some of the other features that c# has. | 14:26 |
| sean-k-mooney | so how is the osic cluster setup going | 14:27 |
| sdake | i am teaching my children K&R C ATM | 14:27 |
| pbourke | sean-k-mooney: getting there | 14:28 |
| inc0 | hmm so I created iface tagged with 702 | 14:28 |
| inc0 | added an IP there | 14:28 |
| inc0 | but doesn't seem to be working | 14:28 |
| *** srwilkers has joined #openstack-kolla | 14:29 | |
| jemcevoy | inc0 what is the default driver? | 14:29 |
| sean-k-mooney | sdake: wow nerver hered of it is there a reason for not going with modern c11 or c14? | 14:29 |
| sdake | sean-k-mooney they are 11 and 13 - starting with the basics | 14:30 |
| sdake | k&r c is c prior to ansi C | 14:30 |
| sdake | I want them to understand how languages evolved | 14:30 |
| sean-k-mooney | sdake: just reading the wiki article. yes its good to have that persective | 14:30 |
| sdake | otherwise i'd probably start with somethign simpler like python | 14:31 |
| inc0 | jemcevoy, devicemapper | 14:31 |
| inc0 | for centos and ubuntu | 14:31 |
| *** b_bezak has quit IRC | 14:31 | |
| inc0 | pbourke, hmmm | 14:31 |
| sdake | plus I am a C master, so its easy for me to teach them that :) | 14:31 |
| sean-k-mooney | sdake: preferably python 3 at this point. someday openstack will get there .... | 14:31 |
| inc0 | sooo root@172.21.0.21 throws connection refused | 14:31 |
| jemcevoy | inc0 I tried to use ceph for cinder... | 14:31 |
| *** b_bezak has joined #openstack-kolla | 14:31 | |
| sdake | python i am more intermediate level | 14:32 |
| sdake | I have 25k+ hours experience writing c code | 14:32 |
| sdake | its a shame openstack wasn't written in C - or I'd really tear things up :) | 14:32 |
| *** matrohon has quit IRC | 14:32 | |
| jemcevoy | inc0, The log is at http://paste.openstack.org/show/549233/ | 14:33 |
| sean-k-mooney | sdake: i learned c++ first so i nerver really felt comfrotable writing c. i keep going wait this is not a ting in c | 14:33 |
| *** diogogmt has joined #openstack-kolla | 14:33 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 14:33 |
| sdake | sean-k-mooney precisely - c++ is a bad language to learn first IMO | 14:33 |
| sdake | but you are doing great - so what do I know :) | 14:33 |
| inc0 | and I can access the node...I mean my network traffic reaches it | 14:33 |
| sdake | i am not super pro with C++ | 14:34 |
| sean-k-mooney | sdake: thanks but if you every want to go back c or a non object orient langue i would agree that you should not learn c++ first | 14:34 |
| sdake | my first language was x86 assembly | 14:35 |
| sdake | my parents gave me 10 bucks to buy a book | 14:35 |
| sdake | the only one i could afford with that was a quick reference | 14:35 |
| sdake | i magically figured out how assembly worked from that - not sure how I did it :) | 14:36 |
| jemcevoy | inc0, I am using centos 7.2 and the network is 1Gig public, 10Gig cluster | 14:36 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Create the heat_stack_user role in the heat role https://review.openstack.org/351220 | 14:36 |
| *** b_bezak has quit IRC | 14:36 | |
| *** srwilkers has quit IRC | 14:37 | |
| inc0 | jemcevoy, did you try re-running kolla-ansible deploy? | 14:37 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Create the heat_stack_user role in the heat role https://review.openstack.org/351220 | 14:37 |
| inc0 | after cleanup? | 14:37 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Create the heat_stack_user role in the heat role https://review.openstack.org/351220 | 14:38 |
| jemcevoy | inc0, I ran it twice but kickstarted the hosts between the runs... | 14:39 |
| inc0 | did you run tools/cleanup-containers between runs? | 14:39 |
| *** banix has joined #openstack-kolla | 14:40 | |
| jemcevoy | I did not do any cleanup on the ansible host... | 14:40 |
| inc0 | try to run this scipt on every host | 14:40 |
| inc0 | and rerun deployment | 14:40 |
| inc0 | also | 14:40 |
| sean-k-mooney | sdake: nice. ok i better get back to the kolla-host playbook. but ill leave this open on my other screen. ill be about for the next 6-8 hours if anyone wants me just ping | 14:40 |
| inc0 | make sure you don't have any mysql running | 14:40 |
| sean-k-mooney | jemcevoy: it might be a good idea to run the precheks if you have not already done so | 14:41 |
| sdake | jemcevoy - if your using ceph - you need to rm -rf /etc/kolla/*ceph* | 14:42 |
| sdake | jemcevoy to clean up effectively | 14:42 |
| sdake | and also remove the mount point from /etc/fstab | 14:42 |
| sdake | jemcevoy another option is to just use "cleanup-hosts" script | 14:42 |
| sdake | or cleanup-host | 14:42 |
| sean-k-mooney | sdake: i was just going to ask do we have that in one of the tools scripts | 14:42 |
| sdake | sean-k-mooney pretty sure it was added | 14:43 |
| britthouser | inc0 - are you able to get onto the virtual console? You had a mac right (trying to remember from midcycle)? | 14:44 |
| inc0 | britthouser, yeah, using java | 14:44 |
| jemcevoy | The scripts you mention are only on the ansible host. should I copy them over to the target hosts? | 14:45 |
| sdake | spot price of gold up to 1361 - yay :) | 14:45 |
| britthouser | what browser inc0? | 14:45 |
| *** Jeffrey4l has joined #openstack-kolla | 14:45 | |
| sdake | jemcevoy there is an ansible playbook that will cleanup all hosts for you | 14:45 |
| sdake | jemcevoy i htink its called kolla-ansible cleanup | 14:45 |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 14:46 |
| inc0 | britthouser, safari | 14:46 |
| inc0 | but you had to do some security setting | 14:46 |
| inc0 | give me a second | 14:46 |
| sean-k-mooney | sdake: yep you can run it multinode by passing -i e.g. kolla-ansible cleanup. -i <path to multinode inventory> | 14:47 |
| *** Liuqing has quit IRC | 14:47 | |
| inc0 | britthouser, Preferences->security->plug-in settings | 14:47 |
| inc0 | you need to set up java to alow | 14:48 |
| inc0 | allow | 14:48 |
| jemcevoy | thanks , sean-k-mooney | 14:48 |
| sean-k-mooney | jemcevoy: be careful runing that if you added any addtional files to /etc/kolla as they will be removed | 14:48 |
| *** Liuqing has joined #openstack-kolla | 14:48 | |
| britthouser | Aha! Thanks inc0. Safari thinks my java is so old it wasn't even prompting me. | 14:48 |
| *** matrohon has joined #openstack-kolla | 14:49 | |
| jemcevoy | oh... That is where I put my docker registry cert... thanks for the heads up | 14:49 |
| sean-k-mooney | jemcevoy: it will leave the password.yml globals.yml kolla-build.yml and config dir but all other files and directorys will be removed | 14:49 |
| *** mewald has quit IRC | 14:49 | |
| sean-k-mooney | jemcevoy: i lost my first multinode inventory file that way... | 14:50 |
| pbourke | inc0: fyi the template customisations are working fantastically for our fork | 14:50 |
| *** berendt has quit IRC | 14:50 | |
| pbourke | inc0: will make rebasing alot easier in the future | 14:51 |
| jemcevoy | sean-k-mooney, is my cert file safe in /etc/docker? | 14:51 |
| inc0 | pbourke, britthouser ssh root@172.21.0.250 | 14:51 |
| inc0 | pbourke, I'm glad I could help | 14:51 |
| sdake | sean-k-mooney right - left that detail out | 14:51 |
| pbourke | inc0: that ip works | 14:52 |
| pbourke | inc0: nice job. what did you do? | 14:52 |
| sean-k-mooney | jemcevoy: yep it should be. this is the bit that cleans up /etc/kolla https://github.com/openstack/kolla/blob/master/tools/cleanup-host#L36-L47 | 14:52 |
| inc0 | well manually added IP and fixed routes | 14:52 |
| pbourke | next step cobbler I guess | 14:52 |
| inc0 | I also added mgmt network iface in p1p1.702 | 14:52 |
| inc0 | do we keep this IP or making something cleaner? | 14:53 |
| inc0 | like 172.21.0.21? | 14:53 |
| sdake | can folks write down the ips somewhere | 14:53 |
| sdake | probably the etherpad is not the safest place | 14:53 |
| inc0 | sdake, these are only available via VPN | 14:53 |
| sdake | (when the bootstrapping gets done) | 14:54 |
| sdake | oh ok then | 14:54 |
| sdake | etherpad should be fine | 14:54 |
| sdake | I just dont want to do wrong by osic | 14:54 |
| inc0 | nah, these are our private IPs | 14:54 |
| inc0 | I don't think we have any public IPs there | 14:54 |
| inc0 | anyway pbourke go ahead and setup cobbler | 14:55 |
| inc0 | use tmux please | 14:55 |
| inc0 | from now on | 14:55 |
| inc0 | so we won't break each others stuff | 14:55 |
| britthouser | woot...new firefox luanches console | 14:55 |
| britthouser | sheesh | 14:55 |
| inc0 | britthouser, just about time I enabled us civilized ssh access;) | 14:55 |
| pbourke | boo seems we can't get out | 14:56 |
| pbourke | its DNS | 14:56 |
| inc0 | yeah | 14:56 |
| *** papacz1 has quit IRC | 14:56 | |
| britthouser | Before we get to far along, I think we need to discuss host setup. RAID? bonding? etc? What do we want all these hosts to look like so we can set them up correctly in cobbler. | 14:57 |
| sean-k-mooney | is the osic cluster behind a proxy or will it have direct internet access when its running. | 14:57 |
| inc0 | britthouser, I don't care about eirher really | 14:57 |
| sean-k-mooney | proxy config can be a pain | 14:57 |
| inc0 | I mean ideally we don't use raid at all | 14:57 |
| inc0 | so swift image from guide has raid 1 on first 2 disks and raid 0 with 1 disk on rest | 14:58 |
| inc0 | which is fine by me | 14:58 |
| britthouser | I understand no RAID for ceph, but compute/control surely we want disk redundancy? | 14:58 |
| pbourke | inc0: can we just whack google DNS into resolv.conf ? | 14:59 |
| sean-k-mooney | britthouser: are you expecting disk failure during the test? | 14:59 |
| inc0 | pbourke, I just did | 14:59 |
| pbourke | good man | 14:59 |
| inc0 | tmux session up | 14:59 |
| inc0 | jump in lads | 14:59 |
| sean-k-mooney | britthouser: if its not complicated to set up then i dont see a down side but kolla should not be effect by raid config | 14:59 |
| *** banix has quit IRC | 15:00 | |
| pbourke | britthouser: tmux att | 15:00 |
| inc0 | sean-k-mooney, so raid config will be relevant when we go cephy | 15:01 |
| sean-k-mooney | pbourke: its worth a try 8.8.8.8 and 8.8.4.4 but you probably know that | 15:01 |
| pbourke | sean-k-mooney: yeah it works | 15:01 |
| pbourke | sean-k-mooney: just it has that 'this will be overwritten by resolvconf' at the top | 15:01 |
| pbourke | sean-k-mooney: I looked into what that means at one stage but cant remember the details | 15:01 |
| inc0 | since we don't have dhcp server here | 15:01 |
| britthouser | sean-k-mooney : I'm thinking more we're doing scale testing. I don't want disk failure to cuase tests to reset nor do I want disk to be bottleneck | 15:01 |
| inc0 | yet | 15:01 |
| inc0 | britthouser, so if I'd deploy prod on these machines | 15:02 |
| inc0 | I'd go for bond on p1p4 and p2p4 | 15:02 |
| inc0 | raid 1 on 2 first disks and 10 other disks without raid | 15:03 |
| inc0 | and deploy ceph on top of it | 15:03 |
| britthouser | I'd propose RAID10 all the drives for compute/control. Ceph install OS on one drive and leave rest for OSD | 15:03 |
| sean-k-mooney | pbourke: yes if the ip was assigned by dhcp it would be overrien by the nameserver in the dhcp reply | 15:03 |
| *** banix has joined #openstack-kolla | 15:03 | |
| sean-k-mooney | but without a dhcp server you are safe. | 15:03 |
| britthouser | inc0 are these cisco switches? are we allowed to configure them for bonding? | 15:03 |
| inc0 | britthouser, afair that's exactly how images for cobbler are setup | 15:03 |
| inc0 | britthouser, I think they already are | 15:03 |
| inc0 | look at pdf | 15:04 |
| pbourke | sean-k-mooney: i see | 15:04 |
| inc0 | britthouser, pbourke are we keeping ip .250? | 15:04 |
| *** Liuqing has quit IRC | 15:05 | |
| inc0 | or we want to change it to something that makes sense? | 15:05 |
| britthouser | .250 is easy to remember. | 15:05 |
| inc0 | although its easy number to remember | 15:05 |
| inc0 | kk let's keep it | 15:05 |
| pbourke | dont care really | 15:05 |
| *** Liuqing has joined #openstack-kolla | 15:05 | |
| *** ppowell has quit IRC | 15:06 | |
| *** daneyon has joined #openstack-kolla | 15:07 | |
| *** vhosakot has joined #openstack-kolla | 15:07 | |
| inc0 | ok so that's done | 15:08 |
| *** zhiwei has joined #openstack-kolla | 15:08 | |
| inc0 | pbourke, you up to deploy cobbler? | 15:08 |
| pbourke | inc0: sure lets go for it | 15:08 |
| inc0 | you start, I need to eat breakfast;) | 15:10 |
| britthouser | I did tmux att, but only see my typing | 15:11 |
| britthouser | I'm more used to screen | 15:11 |
| *** daneyon_ has joined #openstack-kolla | 15:11 | |
| pbourke | im typing atm | 15:11 |
| *** daneyon has quit IRC | 15:11 | |
| pbourke | I saw a broadcast | 15:11 |
| britthouser | yeah I wall'd | 15:11 |
| inc0 | yeah I see multiple sessions open | 15:11 |
| pbourke | sheesh | 15:11 |
| inc0 | pbourke, session 1 | 15:12 |
| britthouser | does tmux not allow you to name sessions like screen? | 15:12 |
| pbourke | im closing 0 | 15:12 |
| pbourke | it does | 15:12 |
| inc0 | yeah | 15:12 |
| *** daneyon has joined #openstack-kolla | 15:12 | |
| *** zhiwei has quit IRC | 15:12 | |
| pbourke | unsure if that ip is ok for the lxc bridge | 15:14 |
| pbourke | britthouser: inc0: you guys have the guide open in front of you? | 15:15 |
| britthouser | I have the guide open | 15:16 |
| *** daneyon_ has quit IRC | 15:16 | |
| pbourke | see it says to ifdown p1p1 | 15:16 |
| pbourke | if I do that we'll get disconnected | 15:16 |
| pbourke | dont see why that's needed to bring up the bridge | 15:16 |
| britthouser | DHCP detection will fail. You will need to manually select the proper network interface - typically p1p1 - and manually configure networking on the PXE network (refer to your onboarding email to find the PXE network information). When asked for name servers, type 8.8.8.8 8.8.4.4. | 15:17 |
| britthouser | oh wait...that is from iso install | 15:17 |
| *** ayoung has joined #openstack-kolla | 15:17 | |
| *** dwalsh has quit IRC | 15:17 | |
| pbourke | oh it wants to add p1p1 to the bridge | 15:17 |
| inc0 | pbourke, so lxc container will connect to it | 15:17 |
| inc0 | as cobbler is in lxc | 15:17 |
| pbourke | so the IP should match p1p1 | 15:17 |
| pbourke | lets try it | 15:18 |
| * britthouser crosses fingers | 15:18 | |
| britthouser | don't forget to make p1p1 inet manual | 15:18 |
| britthouser | instead of static | 15:18 |
| pbourke | lol | 15:19 |
| britthouser | doh | 15:19 |
| inc0 | :D | 15:19 |
| inc0 | let me fix it | 15:19 |
| britthouser | p1p1 needs to be inet manual | 15:19 |
| inc0 | or britt you do it | 15:19 |
| inc0 | as I disconected this vpn | 15:19 |
| britthouser | I can | 15:20 |
| inc0 | thanks | 15:20 |
| britthouser | I attached console to tmux session 1 | 15:21 |
| pbourke | the screen is tiny | 15:21 |
| pbourke | dont think its me | 15:21 |
| britthouser | I disconnected the console | 15:22 |
| britthouser | is that bigger? | 15:22 |
| pbourke | a little | 15:22 |
| *** ppowell has joined #openstack-kolla | 15:22 | |
| britthouser | I think tmux reduces the size to match smaller terminal attached | 15:22 |
| *** matrohon has quit IRC | 15:24 | |
| *** Serlex has quit IRC | 15:25 | |
| *** lrensing_ has joined #openstack-kolla | 15:26 | |
| pbourke | gave the lxc container 251 hope that's ok | 15:27 |
| britthouser | sure | 15:27 |
| britthouser | I'm following you in the guide | 15:27 |
| inc0 | so lxc are "lighweight vms" | 15:27 |
| inc0 | so normally it's even running sshd | 15:28 |
| *** zhiwei has joined #openstack-kolla | 15:28 | |
| britthouser | I can't ssh into .251 | 15:29 |
| britthouser | no biggee | 15:29 |
| pbourke | britthouser: doesn't seem to be running ssh | 15:30 |
| inc0 | nvm | 15:30 |
| pbourke | sooo | 15:30 |
| inc0 | lxc-attach is ok | 15:30 |
| pbourke | what is the networking here | 15:30 |
| *** zhurong has quit IRC | 15:30 | |
| britthouser | This is pxe network | 15:30 |
| *** mgoddard_ has joined #openstack-kolla | 15:30 | |
| pbourke | we have only 20 IPs on the pxe VLAN? | 15:30 |
| jemcevoy | sdake,sean-k-mmoney,inc0 I found a misconfig in my globals.yml... I had to specify the kolla_external_vip_interface: 'enp1s0f0.10' since its IP was getting assigned to the wrong interface... rerunning precheck found the problem... It passed the first time 'cuz the IP was not assigned then... hmmm and cleanup did not remove the kolla_external_vip_address from the interface... | 15:31 |
| britthouser | pbourke I think we have all BUT the first 20 | 15:31 |
| inc0 | jemcevoy, it might not do this | 15:31 |
| pbourke | k | 15:32 |
| pbourke | *ok | 15:32 |
| inc0 | but if you restart iface it won't be there | 15:32 |
| inc0 | pbourke, exactly that, we have almost all of /22 | 15:32 |
| *** zhiwei has quit IRC | 15:32 | |
| *** dwalsh has joined #openstack-kolla | 15:32 | |
| pbourke | just shout if Im doing this wrong | 15:32 |
| sdake | jem the ultimate cleanup is a reboot ;) | 15:32 |
| sdake | jemcevoy ^ | 15:33 |
| pbourke | dynamic-bootp ? | 15:33 |
| britthouser | that is where you start at .21->.249 | 15:33 |
| pbourke | thanks | 15:33 |
| britthouser | Actually.... | 15:33 |
| britthouser | Yeah that's right. | 15:34 |
| *** mgoddard has quit IRC | 15:34 | |
| jemcevoy | sdake, I got it with... "ip addr del 10.103.17.253/32 dev enp6s0" | 15:34 |
| inc0 | we need only 130 sth addresses there | 15:34 |
| inc0 | jemcevoy, that works;) | 15:34 |
| pbourke | britthouser: does that look ok? | 15:34 |
| britthouser | yes | 15:34 |
| sdake | jemcevoy the cleanup scripts are imperfect | 15:35 |
| sdake | jemcevoy they work most of the time - but special cases (like misconfiguration) they dont work as you might want :( | 15:35 |
| sdake | anyone screen capturing the deployment? | 15:36 |
| jemcevoy | guess it is a reboot then... never got the propt back from the ip addr command ;-) my cleanup was a kickstart.... A bit drastic sdake | 15:36 |
| zhubingbing_ | sdake，I have finished (gnocchi, aodh, Sahara)bp， I saw Barbican on BP plan, I don't know if you're doing it or not, I can finish it.。 | 15:36 |
| sdake | jemcevoy i hear ya | 15:36 |
| sdake | zhubingbing_ all yours | 15:36 |
| zhubingbing_ | Barbican contaier and roles | 15:37 |
| britthouser | our subnet is wrong pbourke | 15:37 |
| *** dwalsh has quit IRC | 15:37 | |
| britthouser | =) | 15:37 |
| pbourke | cool, terminal got bigger too ;) | 15:38 |
| *** mewald has joined #openstack-kolla | 15:40 | |
| britthouser | So it looks like we have to pxe over p1p1, since the native VLAN for those interfaces is Vlan700 | 15:40 |
| britthouser | which is pxe network. | 15:40 |
| inc0 | britthouser, it might be native vlan on all of them | 15:40 |
| *** matrohon has joined #openstack-kolla | 15:40 | |
| inc0 | however we already put p1p1 on bridge | 15:40 |
| britthouser | I mean on all the other servers | 15:41 |
| britthouser | we need the MAC of p1p1 | 15:41 |
| inc0 | yeah | 15:41 |
| britthouser | p1p2 + p4p2 can be bonded | 15:41 |
| inc0 | yup | 15:41 |
| britthouser | but p1p1 is what we'll pxe over. | 15:41 |
| *** v1k0d3n has quit IRC | 15:41 | |
| inc0 | and we'll have p4p1 for whatever we might need | 15:41 |
| britthouser | What are you naming these suckers pbourke? | 15:42 |
| britthouser | ceph replication. =) | 15:42 |
| *** diogogmt has quit IRC | 15:42 | |
| inc0 | glorious.omnipotent.kolla-1? | 15:42 |
| britthouser | I htink same hostnames as in email. | 15:42 |
| inc0 | boring...:P | 15:42 |
| *** diogogmt has joined #openstack-kolla | 15:43 | |
| pbourke | gonna be cool to have an inventory with 138 nodes :) | 15:44 |
| britthouser | +1 | 15:45 |
| britthouser | I'm putting some notes into the etherpad under "Working Notes" | 15:45 |
| britthouser | like which node is deployment | 15:45 |
| britthouser | b/c I'm gonna foget | 15:46 |
| pbourke | good idea | 15:46 |
| britthouser | I'm reading through the "Assigning a Cobbler Profile" section | 15:47 |
| britthouser | It sounds like cobbler...or the preseed reconfigures teh h/w RAID for us? | 15:47 |
| inc0 | yeah | 15:47 |
| britthouser | Am I reading that right? | 15:47 |
| britthouser | cool! | 15:47 |
| inc0 | yup | 15:47 |
| inc0 | swift image will be perfect for ceph | 15:48 |
| britthouser | I was expecting to have to do that ahead of time | 15:48 |
| britthouser | yeah your comments make more sense now inc0 =) | 15:48 |
| *** banix has quit IRC | 15:49 | |
| zhubingbing_ | Sorry, just network delay. | 15:50 |
| britthouser | looks good pbourke! | 15:50 |
| pbourke | 8 missing? | 15:50 |
| pbourke | well, 7 minus deploy | 15:50 |
| britthouser | there are 131 nodes | 15:50 |
| pbourke | ah | 15:51 |
| pbourke | perfect | 15:51 |
| britthouser | and that is the p1p1, right? | 15:51 |
| pbourke | so for this input.csv, why do I have to give IP | 15:52 |
| pbourke | I thought cobbler assigned that | 15:52 |
| inc0 | pbourke, ipmi IP | 15:52 |
| inc0 | it uses ipmi to get mac address | 15:52 |
| britthouser | We did that inc0 | 15:52 |
| zhubingbing_ | sdake，Trouble you to Barbican containers and role assigned to me | 15:52 |
| britthouser | This is next step | 15:52 |
| britthouser | You can do it either way pbourke | 15:53 |
| britthouser | you can tell cobbler "give it this IP" | 15:53 |
| britthouser | or you can let cobbler pick an IP | 15:53 |
| pbourke | so which ip do we use for the input.csv | 15:53 |
| sdake | zhubingbing_ will do - launchpad id pls? | 15:53 |
| britthouser | I'm guessing their script assumes prescriptive | 15:53 |
| zhubingbing_ | ok，my launchpad id 1392607554-o | 15:54 |
| inc0 | pbourke, there is script for that in docs | 15:54 |
| pbourke | oh? | 15:54 |
| inc0 | look slightly below | 15:55 |
| pbourke | where | 15:55 |
| inc0 | or maybe not.. | 15:55 |
| zhubingbing_ | ok thinks | 15:55 |
| sdake | zhubingbing_ did I get that correctly in launchpad? | 15:55 |
| zhubingbing_ | yes right | 15:56 |
| zhubingbing_ | i have received an e mail message. | 15:56 |
| pbourke | arg | 15:57 |
| pbourke | feel like im in an interview or something :p | 15:57 |
| * britthouser judges harshly | 15:57 | |
| britthouser | =P | 15:57 |
| britthouser | honestly, I'm a little lost | 15:58 |
| britthouser | you're trying to generate input.csv? | 15:58 |
| britthouser | So if we're gonna assign each server a specific IP, w can change that DHCP command | 15:59 |
| britthouser | instead of bootp range | 15:59 |
| britthouser | deny-unknown-....something | 15:59 |
| britthouser | lemme look it up | 15:59 |
| britthouser | deny unknown-clients; | 16:00 |
| *** matrohon has quit IRC | 16:00 | |
| *** banix has joined #openstack-kolla | 16:01 | |
| pbourke | thats the plan yeah | 16:01 |
| sdake | rhallisey around? | 16:01 |
| britthouser | I changed the dhcp.tempalte | 16:02 |
| inc0 | soo...how do we do it? all cinder? | 16:02 |
| inc0 | cinder image | 16:02 |
| britthouser | it=? | 16:02 |
| inc0 | so raid 1 for OS and raid 10 on the rest | 16:02 |
| inc0 | or swift? raid 1 on OS and 10* raid0 with 1 disk | 16:03 |
| sdake | inc0 the raid choice has performance implications | 16:03 |
| sdake | inc0 can you bring me up to speed on what is being decided here? | 16:03 |
| britthouser | to me Cinder is good for compute roles | 16:03 |
| britthouser | How to partition disks | 16:03 |
| inc0 | tbh I'd run all on swift;) | 16:03 |
| britthouser | we have soem pre-baked configs | 16:03 |
| britthouser | https://github.com/osic/ref-impl/blob/master/documents/bare_metal_provisioning.md | 16:03 |
| inc0 | sdake, we're deciding how to raid our nodes | 16:03 |
| britthouser | scrolls down to Assigning a cobbler profile | 16:04 |
| sbezverk | sdake: could you refer me somebody from the community with deep openvswitch knowldege? | 16:04 |
| *** mgoddard_ has quit IRC | 16:04 | |
| inc0 | sbezverk, sean-k-mooney has some | 16:04 |
| *** mgoddard has joined #openstack-kolla | 16:04 | |
| sbezverk | inc0: thank you | 16:04 |
| sbezverk | sean-k-mooney: ping | 16:04 |
| sean-k-mooney | sbezverk: hi | 16:04 |
| *** Liuqing has quit IRC | 16:04 | |
| sdake | sbezverk there are many in channel - but not sure who is deeply embedded in that project | 16:04 |
| britthouser | cinder means we could do RAID1 for OS and RAID10 for /var | 16:04 |
| *** jtriley has quit IRC | 16:04 | |
| sdake | sbezverk one person to speak with is russell bryant | 16:04 |
| britthouser | which seems natural for compute node | 16:05 |
| sdake | sbezverk he knows all about ovs/ovn/linuxbridge | 16:05 |
| sean-k-mooney | sbezverk: haveing ovs issues? how can i help? | 16:05 |
| sbezverk | sean-k-mooney: I am trying to bring up openvswitch db in kubernetes, I see the process up but it does not create socket | 16:05 |
| britthouser | swift definitely matches up to Ceph role | 16:05 |
| britthouser | For control nodes, we coudl go either generic (single RAID10) or cinder | 16:05 |
| pbourke | we can mod the script up for different roles | 16:05 |
| sbezverk | sean-k-mooney: logs does not show anby problems, any troubleshooting pointers would be great help | 16:05 |
| britthouser | Just my thoughts on it. | 16:05 |
| sean-k-mooney | ok what os are you using? | 16:05 |
| britthouser | thx pbourke for persisting | 16:06 |
| sbezverk | sean-k-mooney: it is kolla image, based on centos source with ovs 2.5.0 | 16:06 |
| britthouser | oh hmm.... | 16:06 |
| britthouser | Someone tell me about how btrfs and docker work together. | 16:07 |
| sdake | my take is for ceph storage nodes, dont use any raid | 16:07 |
| inc0 | britthouser, works well | 16:07 |
| sdake | for the host os, use raid 10 | 16:07 |
| inc0 | however this is ubuntu 14.04 | 16:07 |
| inc0 | we can go aufs | 16:07 |
| sean-k-mooney | sbezverk: oh ok the socket should be created in /usr/var/run/openvswtich by default. is there any error in the openvswitch log | 16:07 |
| britthouser | yeah what are storage requirements for that? | 16:07 |
| britthouser | does it need its own drive? | 16:07 |
| pbourke | britthouser: can you drive now for a bit? 5pm here so have some stuff to take care of before home (or we can pick up tomorrow) | 16:07 |
| pbourke | I think we 3/4 the way there | 16:07 |
| inc0 | well we can use loopback for btrfs | 16:07 |
| * britthouser gets in the driver sat | 16:07 | |
| inc0 | but aufs doesn't need anythning | 16:07 |
| inc0 | just -s aufs and that's it | 16:08 |
| inc0 | well, we might have to install few packages | 16:08 |
| sean-k-mooney | sbezverk: are you deploying with kolla kubernetees by the way or manualy? | 16:08 |
| britthouser | yeah that's fine inc0....just trying to discern if aufs requires special disk setup or not | 16:08 |
| inc0 | britthouser, nope, aufs doesn't need anything | 16:09 |
| *** jtriley has joined #openstack-kolla | 16:09 | |
| inc0 | it's slightly less performant | 16:09 |
| *** dmueller-evoila has quit IRC | 16:09 | |
| inc0 | than btrfs | 16:09 |
| inc0 | but nothing too horrible | 16:09 |
| sbezverk | sean-k-mooney: at this point deploy manually :-) | 16:09 |
| *** godleon has quit IRC | 16:09 | |
| sean-k-mooney | sbezverk: ok my first taught would be a permssions issue preventing the ovsdb server createing the socket. what is the command you are using to lauch the ovsdb? | 16:10 |
| inc0 | my question: is selinux a whiny b*tch? | 16:11 |
| britthouser | only in permissive mode. =P | 16:11 |
| inc0 | :D | 16:12 |
| vhosakot | hahaha | 16:13 |
| sbezverk | sean-k-mooney: root 1 0 0 11:55 ? 00:00:00 /usr/sbin/ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err --remote=punix:/run/openvswitch/db.sock -vfile:info --log-file=/var/log/kolla/openvswitch/ovsdb-server.log | 16:13 |
| sbezverk | sean-k-mooney: how about nice webex, I will walk you through my kubernetes cluster | 16:14 |
| britthouser | My recommendation for profiles is this: control=generic=RAID10 of all drives, compute=cinder=RAID1+RAID10, ceph=swift=RAID1+JBOD | 16:14 |
| sean-k-mooney | sbezverk: if ovs was installed from yum /run/openvswitch/ will not exist unless you create it | 16:15 |
| sean-k-mooney | centos expects to use /usr/var/run/openvswitch | 16:15 |
| sbezverk | sean-k-mooney: the folder is there, also we run as root so there should not be permission issue | 16:15 |
| *** ppowell has quit IRC | 16:16 | |
| sean-k-mooney | sbezverk: and there are no errors in /var/log/kolla/openvswitch/ovsdb-server.log | 16:16 |
| *** unicell1 has quit IRC | 16:17 | |
| *** matrohon has joined #openstack-kolla | 16:17 | |
| sbezverk | sean-k-mooney: http://paste.openstack.org/show/549261/ | 16:17 |
| sdake | britthouser does ceph really need raid? It has replication built in | 16:18 |
| sdake | britthouser by adding raid, extra memcpy() operations are needed which will slow down perf | 16:18 |
| inc0 | sdake, base OS needs a raid | 16:18 |
| inc0 | all other disks are unraided | 16:18 |
| sdake | inc0 clearly | 16:18 |
| inc0 | which is what Britt meant | 16:18 |
| sbezverk | sean-k-mooney: http://paste.openstack.org/show/549263/ | 16:19 |
| *** jtriley_ has joined #openstack-kolla | 16:19 | |
| sbezverk | sean-k-mooney: any debuging I can enable in the command line to see why it does not create socket? | 16:19 |
| sean-k-mooney | if it was the ovs-vswitchd you would just add -v | 16:20 |
| sean-k-mooney | sbezverk: not sure if that will help in this case. | 16:20 |
| inc0 | 90 compute nodes? | 16:21 |
| *** jtriley has quit IRC | 16:22 | |
| sean-k-mooney | sbezverk: looking at the command you are running and the pastes there is nothing obviously wrong | 16:23 |
| *** matrohon has quit IRC | 16:23 | |
| *** jtriley_ has quit IRC | 16:24 | |
| inc0 | pbourke, britthouser ready to roll? | 16:24 |
| pbourke | inc0: I think so | 16:24 |
| sdake | rhallisey happen to be around - re your question about summit space | 16:25 |
| sbezverk | sean-k-mooney: found the issue damn pod difinition lost security context priviliged during translating it to daemonset | 16:27 |
| britthouser | I have a couple meetings right now pbourke | 16:27 |
| britthouser | I'll catch up to you in a bit | 16:27 |
| pbourke | britthouser: np looks like inc0 is on it | 16:27 |
| sean-k-mooney | sbezverk: ah so it did not have cap_sys_admin to be able to create the socket | 16:27 |
| britthouser | or pick up if you have to drop off | 16:28 |
| pbourke | britthouser: catch you tomorrow! | 16:28 |
| inc0 | hmm images doesn't seem right | 16:28 |
| pbourke | inc0: check input.csv | 16:31 |
| pbourke | did I see you use >> instead of > | 16:31 |
| inc0 | ahh | 16:32 |
| inc0 | yeah | 16:32 |
| inc0 | good catch | 16:33 |
| sbezverk | sean-k-mooney: :( it did not fix it.. still no socket | 16:33 |
| sdake | sbezverk strace | 16:34 |
| sdake | sbezverk that will tell you why the socket isn't being created | 16:34 |
| sdake | may be an openvswitch lack of error detection | 16:34 |
| inc0 | so far so good | 16:34 |
| sdake | but strace output will tell you if there is a permissiosn problem or provide some hints as to the root cause | 16:34 |
| inc0 | ok, here goes noting | 16:35 |
| *** jtriley has joined #openstack-kolla | 16:35 | |
| inc0 | restarting all servers | 16:36 |
| sean-k-mooney | bootstrapping hosts is fun ... had to figure out how to install python with ansible to that ansible modules would work... | 16:36 |
| inc0 | sean-k-mooney, -m raw | 16:36 |
| inc0 | ansible -m raw "apt-get install python" :) | 16:36 |
| sean-k-mooney | yep thats basically what i did | 16:37 |
| inc0 | fun fact, ubuntu xenial doesn't have python preinstalled | 16:37 |
| inc0 | pbourke, and now we waid | 16:37 |
| inc0 | wait | 16:37 |
| sean-k-mooney | you also have to disable gatering facts then re enable them when python is installed | 16:37 |
| pbourke | nice | 16:37 |
| inc0 | ahh, for playbook | 16:38 |
| inc0 | yeah | 16:38 |
| inc0 | btw we might use your kolla-host play;) | 16:38 |
| inc0 | to bootstrap hosts we have now | 16:38 |
| pbourke | was gonna say | 16:38 |
| sean-k-mooney | inc0: yep i just discovered that when working on the kolla-host playbook | 16:38 |
| inc0 | pbourke, sean-k-mooney you had beer together right? | 16:39 |
| sdake | who would win a drinking contest - pbourke or sean-k-mooney | 16:39 |
| britthouser | inc0 pbourke - back from meetings | 16:39 |
| inc0 | correct me if I'm wrong but both of you are from Dublin? | 16:39 |
| *** wrouesnel1 has quit IRC | 16:39 | |
| sdake | from my recolleciton pbourke has hallow legs :) | 16:39 |
| sean-k-mooney | inc0: didnt have an opertunity yet but maybe at the summit | 16:39 |
| pbourke | only one way to find out | 16:39 |
| inc0 | britthouser, hosts are installing | 16:39 |
| sdake | hollow | 16:39 |
| pbourke | sdake is buying | 16:39 |
| inc0 | I'll play | 16:39 |
| inc0 | ;) | 16:39 |
| britthouser | I'm gonna bring up a console inc0 and watch | 16:40 |
| britthouser | Are you in 225? | 16:40 |
| *** matrohon has joined #openstack-kolla | 16:40 | |
| sdake | i sometimes wonder if that night we got rocked playing fooseball pbourke was drinking odouls | 16:40 |
| inc0 | britthouser, yeah | 16:40 |
| inc0 | none of servers seems to be up yet | 16:41 |
| inc0 | but servers take ages to boot up | 16:41 |
| sean-k-mooney | :) life would be less interesting without alcohol. less painful too ... | 16:41 |
| pbourke | inc0: .225 is not | 16:41 |
| pbourke | inc0: that's just one I've checked | 16:41 |
| inc0 | kk | 16:41 |
| pbourke | checking another random one | 16:42 |
| inc0 | are they even booting up? | 16:42 |
| britthouser | I'm seeing dhcp requests | 16:43 |
| sean-k-mooney | inc0: by the way what os are ye deploying first | 16:43 |
| inc0 | ubu 14.04 | 16:43 |
| britthouser | did we run cobble sync after loading hosts? | 16:43 |
| inc0 | yeah | 16:44 |
| britthouser | Aug 4 16:43:43 osic-prep dhcpd: DHCPDISCOVER from 68:05:ca:32:e0:b8 via eth1: network 172.21.0.0/22: no free leases | 16:44 |
| pbourke | .221 is black screen | 16:44 |
| openstackgerrit | Merged openstack/kolla: Volume specification for ironic-api was invalid. https://review.openstack.org/344775 | 16:44 |
| inc0 | duh | 16:44 |
| rhallisey | sdake, ya | 16:44 |
| britthouser | maybe those are hosts in a different cluster | 16:44 |
| sdake | hey rhallisey | 16:44 |
| britthouser | 68:05:ca:32:e0:3c and 68:05:ca:32:e0:b8 | 16:45 |
| sean-k-mooney | inc0: by the way what os are ye deploying first | 16:45 |
| rhallisey | wow long backlog | 16:45 |
| britthouser | ubuntu | 16:45 |
| britthouser | sean-k-mooney ^^ | 16:45 |
| sean-k-mooney | opps didnt mean to double post | 16:45 |
| sdake | there was a message from ttx on teh mailing list indicating the space in barcelona will not hae has much capacity as austin | 16:45 |
| sdake | has/as | 16:45 |
| rhallisey | gotcha | 16:46 |
| sean-k-mooney | britthouser: 16.04 or 14.04? | 16:46 |
| inc0 | 14 | 16:46 |
| britthouser | 14.04 | 16:46 |
| sdake | rhallisey my take on that is kolla will definately not get as many slots sa we got last time | 16:46 |
| britthouser | I apparently don't type as fast as inc0 | 16:46 |
| sean-k-mooney | ok good to know | 16:46 |
| inc0 | because you type full name') | 16:46 |
| rhallisey | sdake, ok we'll see what we get | 16:46 |
| rhallisey | good to know | 16:46 |
| inc0 | on that note, we might need to figure out deployment of centos on these nodes | 16:46 |
| sdake | rhallisey still - it is on us to produce a list of sessions and a request for session counts (fishbowls/design sessions) | 16:47 |
| sean-k-mooney | im currently testing my kolla-host playbook with 16.04 but its easy to clean down the vms with a 14.04 image and test with that | 16:47 |
| inc0 | but raid should stay regardless so it should be ok | 16:47 |
| sdake | rhallisey and we will see what we end up with | 16:47 |
| rhallisey | sdake, for sure | 16:47 |
| sdake | rhallisey we typically brainstorm the sessions at the end of m3 | 16:47 |
| sdake | for 2-3 meetings 15 minutes a crack | 16:47 |
| sdake | then we vote on em and negotiate the schedule | 16:48 |
| sdake | ultimately its up to the ptl to put the schedule together | 16:48 |
| rhallisey | yup. We can address it then | 16:48 |
| sdake | cool | 16:48 |
| rhallisey | sdake, thanks | 16:49 |
| sdake | but i like our process for selecting sessions - and would hate to see that change :) | 16:49 |
| britthouser | I logged into .227 inc0 | 16:49 |
| britthouser | its been up for 7min | 16:49 |
| britthouser | but does not look like it kickstarted | 16:50 |
| britthouser | cobbler status reports nobody | 16:50 |
| * rhallisey brb | 16:50 | |
| inc0 | yeah seems wrong | 16:50 |
| *** unicell has joined #openstack-kolla | 16:50 | |
| britthouser | so do we need to go through and change boot seetings? | 16:52 |
| sdake | inc0 did you get Jeffrey4l creds to the osic cluster? | 16:52 |
| inc0 | sdake, everyone from core team should get mail | 16:52 |
| sdake | nice | 16:53 |
| inc0 | britthouser, nah, cobbler should do it | 16:53 |
| sdake | thanks - i had said i would give them to him but then promptly forgot :( | 16:53 |
| *** david-lyle has quit IRC | 16:53 | |
| britthouser | on .227, the boot order had HDD above network | 16:53 |
| *** david-lyle has joined #openstack-kolla | 16:53 | |
| britthouser | I changed it | 16:53 |
| britthouser | rebooted | 16:53 |
| britthouser | crossed fingers | 16:53 |
| inc0 | britthouser, look at tmux | 16:53 |
| inc0 | ok so dhcp goes through, I thought that's the issue | 16:54 |
| inc0 | but no response | 16:54 |
| britthouser | "dhcp goes through" | 16:55 |
| britthouser | means? | 16:55 |
| inc0 | no dhcp response tho | 16:55 |
| *** berendt has joined #openstack-kolla | 16:55 | |
| inc0 | I'm looking at traffic to see if dhcp traffic goes through to container | 16:55 |
| inc0 | it sees the requests but it doesn't respond | 16:55 |
| inc0 | no free leases | 16:56 |
| britthouser | my host gets a DHCP lease | 16:56 |
| britthouser | its trying tftp | 16:56 |
| britthouser | tftp timeout | 16:56 |
| britthouser | is tftp running? | 16:56 |
| inc0 | seems like it | 16:56 |
| pbourke | britthouser: when you updated the ip range in cobbler.conf | 16:56 |
| pbourke | britthouser: was there some sync or something needed | 16:57 |
| sdake | there is a sync button | 16:57 |
| britthouser | the next cobbler sync would have taken care of it | 16:57 |
| sdake | hidden in the ui | 16:57 |
| pbourke | did we do another sync though yeah | 16:57 |
| inc0 | sdake, our ui is white-on-black | 16:57 |
| britthouser | yeah it made it | 16:57 |
| inc0 | with some green at the bottom | 16:57 |
| sdake | inc0 what i mean is the ui has a sync button which doens't pop out to people as "this must be pressed" :) | 16:58 |
| inc0 | sdake, and I mean we're over ssh, no ui present;) | 16:58 |
| britthouser | we did the sync via CLI | 16:58 |
| sdake | oh duh | 16:58 |
| sdake | i'll stu then nm :) | 16:58 |
| pbourke | whats all this eth1 business | 16:59 |
| britthouser | thats in the container pbourke | 16:59 |
| *** belmoreira has joined #openstack-kolla | 17:00 | |
| inc0 | lxc doesn't have notion of net=host | 17:00 |
| britthouser | this is perplexing | 17:00 |
| britthouser | the host I'm on is getting DHCP lease | 17:00 |
| britthouser | but I'm not seeig any leases in /var/lib/dhcp | 17:01 |
| inc0 | why are ips commented? | 17:01 |
| britthouser | where inc0? | 17:01 |
| britthouser | oh. | 17:01 |
| britthouser | b/c cobbler puts host entries in | 17:01 |
| *** ppowell has joined #openstack-kolla | 17:01 | |
| inc0 | well, let's try that | 17:02 |
| *** yee_ has joined #openstack-kolla | 17:02 | |
| britthouser | sure | 17:02 |
| inc0 | maybe it doesn't work like we think it does | 17:02 |
| inc0 | reset all the machines! | 17:02 |
| britthouser | no..lets test one at a time | 17:03 |
| britthouser | or not.... | 17:03 |
| inc0 | too late | 17:03 |
| pbourke | ha | 17:03 |
| *** flaper87 has quit IRC | 17:03 | |
| * britthouser makes mental note not to let inc0 near the EPO button | 17:03 | |
| britthouser | =P | 17:03 |
| inc0 | I'm devops | 17:03 |
| inc0 | I have worst qualities of both devs and ops | 17:04 |
| britthouser | ok well, before my host reset, I had an IP addres in 172.21.0 network | 17:04 |
| sean-k-mooney | lol well i guess go big or go home | 17:04 |
| britthouser | but could not ping 251 | 17:04 |
| britthouser | so I suspect our lxc bridge isn't workign like we think it is | 17:04 |
| sean-k-mooney | why are ye using lxc out of interest? | 17:05 |
| britthouser | I can't ping the lxc from my dkestop either | 17:05 |
| pbourke | docker isnt cool anymore | 17:05 |
| britthouser | its what the guide uses sean-k-mooney | 17:05 |
| inc0 | hmm | 17:05 |
| britthouser | trying not to blaze any more trails that we have too | 17:05 |
| inc0 | britthouser, soo...it seems we messed up our ports | 17:06 |
| inc0 | bridges | 17:06 |
| pbourke | my node is at tftp now | 17:06 |
| pbourke | timedout | 17:06 |
| britthouser | same for me pbourke | 17:06 |
| inc0 | but hhold on | 17:06 |
| *** ppowell has quit IRC | 17:06 | |
| inc0 | if it's on tftp | 17:06 |
| britthouser | but what makes me scratch my head is where is it getting DHCP | 17:06 |
| britthouser | b/c its not from our container | 17:06 |
| inc0 | it went through dhcp | 17:06 |
| inc0 | and it doesn't seem like it | 17:06 |
| britthouser | i.e. there appears to be a rogue DHCP out there | 17:06 |
| inc0 | from dhcp server standpoint | 17:07 |
| sean-k-mooney | you wouldnt hapen to have 2 dhcp servers? | 17:07 |
| pbourke | dang, I have to go :( will try check in later depending how busy my evening is | 17:07 |
| pbourke | chat later! | 17:07 |
| inc0 | np | 17:07 |
| inc0 | cya Paul! | 17:07 |
| sean-k-mooney | so i had the same issue at home. | 17:08 |
| sean-k-mooney | or close | 17:08 |
| sean-k-mooney | i had to put the pxe network on a seperate vlan | 17:08 |
| britthouser | cya pbourke | 17:09 |
| inc0 | I'll ask about this | 17:09 |
| britthouser | I htink we're facing three problems at the moment: 1) rogue dhcp server 2) lxc container not reachable from the network 3) nodes might not be set to boot from network | 17:10 |
| inc0 | quick question | 17:10 |
| sbezverk | sdake: could you please take a look at strace here: http://paste.openstack.org/show/549274/ | 17:10 |
| *** matrohon has quit IRC | 17:11 | |
| inc0 | why eth0 has this strange address? | 17:11 |
| *** unicell has quit IRC | 17:12 | |
| Mech422 | morning | 17:12 |
| *** belmoreira has quit IRC | 17:13 | |
| britthouser | I htink that's how it came from wget inc0 | 17:13 |
| sdake | sbezverk looking | 17:13 |
| britthouser | i proute | 17:13 |
| *** zhubingbing_ has quit IRC | 17:13 | |
| britthouser | ip route | 17:13 |
| britthouser | =/ | 17:14 |
| *** zhubingbing_ has joined #openstack-kolla | 17:14 | |
| britthouser | that isn't our issue though | 17:14 |
| britthouser | b/c all these hosts are l2 adjacent | 17:14 |
| britthouser | route wont' come into play | 17:14 |
| britthouser | its AN issue, but not why pings are failing | 17:14 |
| sdake | sbezverk is db.sock the socket name your looking for? | 17:15 |
| inc0 | hmm thats strange | 17:15 |
| inc0 | nvm | 17:15 |
| inc0 | arp | 17:15 |
| inc0 | so we have connection to lxc | 17:16 |
| britthouser | Can you hit the LXC from off host? | 17:16 |
| *** signed8bit is now known as signed8bit_Zzz | 17:16 | |
| sbezverk | sdake correct | 17:16 |
| inc0 | from host yeah | 17:16 |
| inc0 | from external no, but that's because of ip r | 17:17 |
| britthouser | sure | 17:17 |
| inc0 | and we don't want to mess ip r because of ipmi networking | 17:17 |
| britthouser | but what about another host in Vlan700? | 17:17 |
| sdake | sbezverk it definately is there - return error 0 = success on line 277 | 17:17 |
| britthouser | ? | 17:17 |
| inc0 | lxc has to acces ipmi network | 17:17 |
| inc0 | and I think that's baked into container | 17:17 |
| sdake | /run/openvswitch/db.sock | 17:17 |
| sdake | sbezverk are you trying to get multiple containers to work together with this socket? | 17:18 |
| britthouser | I thought the ILO was 10.15.240? | 17:18 |
| sbezverk | sdake: how about line 262 | 17:18 |
| Mech422 | inc0: if your doing vlan for ipmi - the kolla ovs setup isn't eating it is it.. | 17:18 |
| inc0 | default route is via 10.0.3.1 | 17:18 |
| sdake | sbezverk right - unlink first to remove the socket, then bind to create the socket | 17:19 |
| sdake | sbezverk that is what it hsould be doing | 17:19 |
| *** unicell has joined #openstack-kolla | 17:19 | |
| britthouser | but what does eth0 attach to inc0? | 17:19 |
| britthouser | I don't see 10.0.3 anywhere in teh docs they gave us | 17:19 |
| sdake | sbezverk if you could give more explination of about why you think the socket is the problem that would help | 17:19 |
| *** unicell has quit IRC | 17:19 | |
| *** unicell has joined #openstack-kolla | 17:19 | |
| inc0 | p1p1 as it seems | 17:19 |
| inc0 | aaand it's routable | 17:20 |
| sdake | sbezverk up for quick webex? | 17:20 |
| britthouser | ...... | 17:20 |
| sbezverk | sdake: each compute node will run 3 containers db vswitchd and neutron-openvswitch-agent | 17:20 |
| inc0 | btw let's run hangouts | 17:21 |
| inc0 | typing in irc is hard | 17:21 |
| sbezverk | sdake: last 2 communicate with DB via that db.socket | 17:21 |
| sdake | ok, well is /run being shared on the node? | 17:21 |
| britthouser | 10.0.3.1 is the host itself | 17:21 |
| sbezverk | sdake yep | 17:21 |
| britthouser | I gues this is the lxc version of docker0 ? | 17:21 |
| Mech422 | inc0: mind if I join? I'm curious as I do a lot of funky networking here...I might learn something :-) | 17:21 |
| sdake | sbezverk lets validate that assumption | 17:22 |
| sdake | webex? :) | 17:22 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add sahara ansible roles https://review.openstack.org/351294 | 17:22 |
| sdake | Mech422 anyone welcome to join as long as they are not disruptive ;) | 17:22 |
| sbezverk | sdake: yes, sending | 17:22 |
| sbezverk | sdake: https://cisco.webex.com/join/sbezverk | 17:22 |
| inc0 | Mech422, we're deploying baremetal in osic cluster | 17:23 |
| inc0 | and fighting with cobbler inside lxc container | 17:23 |
| *** ppowell has joined #openstack-kolla | 17:23 | |
| Mech422 | inc0: cool - haven't done cobbler in about a year ( I do foreman here..) but I'd love to follow along | 17:24 |
| sean-k-mooney | inc0: do you still think its related to the lxc bridge? | 17:24 |
| sean-k-mooney | inc0: if so maybe move the phyical interface into the lxc containers network namespace and drop the bridge entirely | 17:24 |
| britthouser | yes | 17:24 |
| Mech422 | inc0: btw - the big issue I have with bare metal is my nics don't pxe boot off VLANs - so I have to have an untagged vlan for eth0 pxe, and tagged vlans for everything else | 17:24 |
| britthouser | So from a host in Vlan700 | 17:25 |
| britthouser | I can ping .250 | 17:25 |
| britthouser | but not ping .251 | 17:25 |
| *** vhosakot has quit IRC | 17:25 | |
| *** vhosakot has joined #openstack-kolla | 17:25 | |
| britthouser | something if funky | 17:26 |
| inc0 | Mech422, yeah, pxe has to be native | 17:26 |
| britthouser | inc0 - so my ping from a differnet host got one reply to .251 | 17:27 |
| britthouser | but nothing else | 17:27 |
| britthouser | 97% packet loss | 17:27 |
| inc0 | lol | 17:27 |
| inc0 | not 100% | 17:27 |
| britthouser | new ping: now 100% | 17:27 |
| sean-k-mooney | if you run tcpdump on .251 do you recive the pings? | 17:28 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 17:28 |
| *** irtermite has joined #openstack-kolla | 17:29 | |
| irtermite | /wave | 17:29 |
| inc0 | irtermite, britthouser and me are trying to repave these machines | 17:29 |
| Mech422 | Does a ping on vlan700 work host-to-host? (ie is it just the containers that are hosed?) | 17:29 |
| inc0 | fun fact, machines gets dhcp but timeouts on tftp | 17:30 |
| sean-k-mooney | Mech422: do you have a managed switch? is so you can have the switch push/pop the vlans | 17:30 |
| inc0 | and no dhpc response goes out of lxc which runs dhcpd | 17:30 |
| britthouser | funner fact machines get dhcp, but not from our server | 17:30 |
| Mech422 | sean-k-mooney: yeah - I got it all working - it was just a PITA | 17:30 |
| Mech422 | inc0: so you have a rogue dhcp server? | 17:30 |
| inc0 | britthouser, irtermite is a generous guy who helped us with this cluster, he's operator out there | 17:30 |
| inc0 | Mech422, seems like it | 17:31 |
| britthouser | w00t irtermite | 17:31 |
| Mech422 | sean-k-mooney: btw - I've been very impressed with the lil desktop tp-link switch I got - most of the features of my 'real' switches (lacp, vlan,etc) for like $50 8-port | 17:31 |
| sean-k-mooney | Mech422: ya i picked up a cheap 24 port dlink for use at home for about 100 | 17:32 |
| Mech422 | sean-k-mooney: my 'real' switches are Dell powerconnects 24 port of ebay for $50-75 | 17:33 |
| sean-k-mooney | Mech422: at least at the 1G level you can get a pretty nice switch for not much | 17:33 |
| Mech422 | sean-k-mooney: yep | 17:33 |
| Mech422 | sean-k-mooney: 10G switches are still too expensive though | 17:33 |
| britthouser | it looks like host can ping the lxc container | 17:33 |
| Mech422 | sean-k-mooney: I was really hesitant about the tp-link, as it seemed 'no-name' - very pleasantly surprised | 17:33 |
| britthouser | and vice versa | 17:33 |
| sean-k-mooney | yep at least for home but sdake might have one | 17:33 |
| britthouser | but no packets from beyond the host | 17:34 |
| Mech422 | sean-k-mooney: I'm looking for 16 10G ports - preferably arrista as thats what we use for TOR | 17:34 |
| inc0 | britthouser, do you have external host in this L2? | 17:34 |
| sean-k-mooney | Mech422: tp-link are pretty well known for there consumer gateways/routers | 17:34 |
| britthouser | yes 0.63 | 17:34 |
| *** unicell has quit IRC | 17:34 | |
| britthouser | from 0.63 I can ping .250 | 17:34 |
| britthouser | and very very very rarely .251 | 17:35 |
| Mech422 | sean-k-mooney: oh? I've never used them before .. just netgear mostly | 17:35 |
| inc0 | soo..pings seems to go one way | 17:35 |
| inc0 | but not back? | 17:35 |
| *** mark-casey has joined #openstack-kolla | 17:35 | |
| inc0 | arp is fucked up? | 17:35 |
| britthouser | So I'm sending pings from .251 | 17:36 |
| sean-k-mooney | Mech422: nice sdake i think runs 10G or cat7/rj45 at home . i think he picked up a a cheap 8-16 port switch but not sure which one | 17:36 |
| irtermite | inc0: hrm... I am just going to guess cobbler was left running on the deploy host after the repave | 17:36 |
| britthouser | but i ond't see them on the host at all | 17:36 |
| irtermite | what are you getting dhcp from inc0? | 17:36 |
| inc0 | irtermite, which one was deployment | 17:36 |
| inc0 | ? | 17:36 |
| irtermite | not one you have access to | 17:36 |
| britthouser | lol | 17:36 |
| irtermite | becauseReasons | 17:36 |
| sean-k-mooney | inc0: so they are being recived on .251 but no reply sent or repley is sent but not recived | 17:36 |
| inc0 | britthouser, we got it from vlan700 right? | 17:37 |
| inc0 | which is well..pxe network so all is good | 17:37 |
| inc0 | besides that we really wanted to use this network for our cobbler;) | 17:37 |
| Mech422 | sean-k-mooney: yeah - he has a netgear 8 port I think, he said he likes it a lot | 17:37 |
| rhallisey | inc0, do you guys have tmux + hangout session going? | 17:37 |
| inc0 | just tmux | 17:37 |
| irtermite | 700 is pxe, correct. but if our deploy hose still have dhcp active, it will conflict with yours | 17:37 |
| inc0 | which seems to be the case | 17:37 |
| britthouser | I think that is one of our problems itermite | 17:38 |
| inc0 | and if it by any chance have IP .251, it'll explain other stuff too;) | 17:38 |
| britthouser | but another problem is our lxc is not able to send packets out on the wire | 17:38 |
| irtermite | to confirm, are you able to give me where you are getting the dhcp addresses from? | 17:38 |
| irtermite | .251? ok one min | 17:38 |
| openstackgerrit | Rohit Agarwalla proposed openstack/kolla: Customizations for iscsid https://review.openstack.org/351319 | 17:38 |
| *** shardy has quit IRC | 17:39 | |
| inc0 | irtermite, not sure if this is good addr | 17:39 |
| Mech422 | sean-k-mooney: I might ditch my 10G cards, and get myranet cards/switch - the cards are only like $20, and the switches are much cheaper | 17:39 |
| inc0 | but we have funky experience with this one | 17:39 |
| irtermite | ahhhh | 17:39 |
| irtermite | hrm | 17:39 |
| Mech422 | sean-k-mooney: but then I lose the 'make-it-like-production' part | 17:39 |
| irtermite | nope, that is not our cobble IP | 17:39 |
| irtermite | I wonder... | 17:39 |
| irtermite | give me a couple more minutes to go look at something. I have a curious thought. | 17:40 |
| sean-k-mooney | Mech422: myranet cards? | 17:40 |
| britthouser | sure itermite. Appreciate the help | 17:40 |
| britthouser | sure irtermite. Appreciate the help | 17:40 |
| inc0 | lets add more whitespaces for bigger impact;) | 17:41 |
| sean-k-mooney | Mech422: this https://en.wikipedia.org/wiki/Myrinet ? | 17:41 |
| Mech422 | sean-k-mooney: bad spelling... yeah | 17:41 |
| britthouser | it is SO hard to make my brain switch from ctl-a to ctl-b | 17:41 |
| Mech422 | sean-k-mooney: you can get 40G myrinetcheap now | 17:41 |
| sean-k-mooney | Mech422: i have never heard of it with either spelling so it would not have helped | 17:42 |
| Mech422 | sean-k-mooney: oh..heh..I don't feel bad then :-P it can carry ip or 'remote dma' (rdmi?) | 17:42 |
| Mech422 | sean-k-mooney: low latency, high speed, funky cables | 17:42 |
| *** vhosakot has quit IRC | 17:42 | |
| sean-k-mooney | Mech422: ok so its kinda like infiniband | 17:43 |
| wirehead_ | Go on your myri little way with cheaper cards? | 17:43 |
| Mech422 | sean-k-mooney: guess so - never used infiniband (or myrinet for that matter) | 17:44 |
| irtermite | inc0: BRB, mind hitting this and dropping me the IP ??? grep dhcp-server-identifier /var/lib/dhcp3/dhclient.leases | 17:44 |
| Mech422 | wirehead_: I'm considering it - getting tired of waiting for 10G ethernet switch prices to drop :-/ | 17:44 |
| britthouser | inc0 | 17:44 |
| britthouser | you put mgmt IP in pxe vlan? | 17:44 |
| sdake | sbezverk lets try a new webex | 17:45 |
| britthouser | oh wait....nvm | 17:45 |
| irtermite | inc0: just need the option dhcp-server-identifier | 17:45 |
| *** blahrus has quit IRC | 17:45 | |
| sbezverk | sdake sure | 17:45 |
| wirehead_ | Peer pressure. | 17:45 |
| sbezverk | do you want start yours, hopefully it will be better | 17:45 |
| britthouser | irtermite - we need to add to our cobbler template? | 17:46 |
| irtermite | britthouser: ?? | 17:46 |
| inc0 | irtermite, 172.21.3.254 | 17:46 |
| britthouser | the dhcp-server-identifier? | 17:46 |
| Mech422 | wirehead_: I _really_ want an arrista 24 port for home, as thats what we use for TOR...but $$$ | 17:46 |
| irtermite | I am trying to determine where you are getting dhcp from, so I can see if it is our cobbler or something else | 17:46 |
| britthouser | oh..nvm | 17:46 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair sahara dockerfile https://review.openstack.org/351320 | 17:47 |
| irtermite | inc0: hrmmm, still not it. weird. ummmm | 17:47 |
| irtermite | have you tried connecting to that IP and seeing what it has installed? | 17:47 |
| inc0 | irtermite, btw, feel free to join us on 172.21.0.250 | 17:48 |
| inc0 | tmux session standing | 17:48 |
| sean-k-mooney | Mech422: they are cheaper then i remember http://www.ebay.ie/itm/Arista-Switch-DCS-7148S-R-48x-10GbE-SFP-7100-Series-Full-Rail-Acc-Kit-Included-/152102169839?hash=item2369feecef:g:gz4AAOSwTdJXRIaz | 17:48 |
| irtermite | ack | 17:48 |
| irtermite | inc0: ack | 17:48 |
| inc0 | you know password | 17:48 |
| inc0 | this is our deployment node | 17:48 |
| inc0 | deployed with your cobbler | 17:48 |
| irtermite | neat | 17:48 |
| inc0 | we want to repave all other nodes | 17:48 |
| Mech422 | sean-k-mooney: oh yeah - I haven't seen them below $1500 before... | 17:49 |
| *** signed8bit_Zzz is now known as signed8bit | 17:49 | |
| sbezverk | sdake: https://cisco.webex.com/join/sbezverk | 17:49 |
| Mech422 | sean-k-mooney: and thats 48 ports, not 24...though I only need 16-24 so far | 17:50 |
| *** imcsk8 has quit IRC | 17:50 | |
| *** imcsk8 has joined #openstack-kolla | 17:50 | |
| *** vhosakot has joined #openstack-kolla | 17:51 | |
| wirehead_ | Well, lessee, my favorite operational counterpart ever in my career is married to a very important arista engineer. The correct strategy for me is to see if she can talk to him and see if Arista finds it so amusing that a Cisco employee runs an Arista switch at home that they’ll give me one for free. | 17:51 |
| sean-k-mooney | Mech422: not arista but http://www.ebay.ie/itm/Quanta-LB6M-10GB-24-Port-SFP-Switch-QY139A-Dual-Power-Supply-with-Rack-Ears-/152020333438?hash=item23651e337e:g:vP4AAOSwgY9XfCli | 17:51 |
| wirehead_ | And/or just check the damn Cisco employee store. | 17:51 |
| Mech422 | sean-k-mooney: I keep hearing about the Quanta - but don't know anyone that runs one...good price points though | 17:52 |
| Mech422 | wirehead_: get 2 and send me 1 :-) | 17:52 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi image dockerfile https://review.openstack.org/348388 | 17:52 |
| inc0 | Mech422, I had some fun with quanta storage servers | 17:52 |
| britthouser | I think we might have some kind of L2 loop inc0 | 17:52 |
| inc0 | pretty cool | 17:52 |
| Mech422 | inc0: were they any good ? | 17:52 |
| Mech422 | inc0: oh nice | 17:53 |
| inc0 | hmm | 17:53 |
| britthouser | we shoudl not be getting that many BPDUs | 17:53 |
| britthouser | would explain the intermittent pings | 17:53 |
| inc0 | soo...we never touched p1p2 and p4p2 right? | 17:54 |
| britthouser | not that I'm aware of | 17:54 |
| inc0 | I might have...I see an IP at p1p2 | 17:54 |
| britthouser | sorry | 17:54 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair aodh image dockerfile https://review.openstack.org/350715 | 17:55 |
| britthouser | gotta block your own ssh | 17:55 |
| britthouser | just do stp | 17:55 |
| inc0 | hmm | 17:56 |
| britthouser | the container is seeing TONS! | 17:56 |
| inc0 | a bridge then? | 17:56 |
| britthouser | hmm..stp is not right | 17:56 |
| britthouser | leave off stp in the container | 17:56 |
| britthouser | you won't se your own shs | 17:56 |
| zhubingbing_ | hi | 17:57 |
| *** zhubingbing_ has quit IRC | 17:57 | |
| *** unicell has joined #openstack-kolla | 17:57 | |
| inc0 | also both linus bridges has stp disabled | 17:57 |
| britthouser | I bounced eth1 in the container | 17:57 |
| *** berendt has quit IRC | 18:00 | |
| *** mgoddard_ has joined #openstack-kolla | 18:01 | |
| inc0 | lots of stp | 18:03 |
| inc0 | just our filter didt show it;) | 18:03 |
| *** mgoddard has quit IRC | 18:04 | |
| britthouser | I'm not seeing tons of STP on this other host | 18:05 |
| britthouser | so i'm wondering if the loop is on jut this host | 18:05 |
| britthouser | want to try to reboot just this host inc0? | 18:05 |
| inc0 | yeah let's try | 18:05 |
| irtermite | inc0: .254 not found... | 18:05 |
| openstackgerrit | sean mooney proposed openstack/kolla: add baremetal role to install kolla deps. https://review.openstack.org/325631 | 18:06 |
| *** mgoddard_ has quit IRC | 18:06 | |
| britthouser | I'll open console and reboot inc0 | 18:07 |
| inc0 | I just rebooted | 18:07 |
| * britthouser watches console | 18:08 | |
| *** lrensing_ has quit IRC | 18:08 | |
| britthouser | did we set those addresses in /etc/ files? | 18:08 |
| inc0 | I think so | 18:08 |
| irtermite | ^^ hahaha | 18:08 |
| inc0 | well I did | 18:09 |
| irtermite | no static... bye bye | 18:09 |
| irtermite | ;) | 18:09 |
| britthouser | =) | 18:09 |
| irtermite | Guess you are about to find out... | 18:09 |
| inc0 | but I'm thinking whether we removed it when we setup these bridges | 18:09 |
| inc0 | britthouser, has console open so we're good | 18:09 |
| inc0 | ;) | 18:09 |
| irtermite | damn it... where is the [disable-michal-accent] button? | 18:10 |
| britthouser | lol...I know pbourke edited the br-pxe stuff | 18:10 |
| inc0 | hahaha | 18:10 |
| inc0 | it is quite an achievement to have slavic accen over irc | 18:10 |
| *** zhiwei has joined #openstack-kolla | 18:11 | |
| *** gfidente has quit IRC | 18:11 | |
| irtermite | inc0: 172.21.3.254 Destination Host Unreachable... you sure that is what is serving dhcp to you? | 18:11 |
| inc0 | irtermite, so that dhcp config | 18:11 |
| inc0 | however when we logged to node ip wasn't there | 18:11 |
| irtermite | possibly you were serving back at yourself? | 18:12 |
| irtermite | hrm | 18:12 |
| sean-k-mooney | inc0: as an fyi the latest version of the kolla-host playbook brings clean ubuntu 16.04 could images to the point that they pass the precheck. | 18:12 |
| *** tpeoples has quit IRC | 18:12 | |
| inc0 | yay | 18:12 |
| sean-k-mooney | inc0: im going to go back through the quickstart and see if anything is missing then ill check on centos and ubuntu 14.04 | 18:12 |
| inc0 | sean-k-mooney, you're setting up storage driver? | 18:13 |
| inc0 | I think it's overlay for ubuntu | 18:13 |
| sean-k-mooney | no but ill check what it is by default. | 18:13 |
| sean-k-mooney | inc0: its easy to add | 18:13 |
| inc0 | default is devicemapper afaik | 18:14 |
| inc0 | which is an abomination | 18:14 |
| sean-k-mooney | inc0: yep its devicemapper... so what would you prefer instead? | 18:14 |
| inc0 | britthouser, I guess no static ip set;) | 18:14 |
| inc0 | sean-k-mooney, anything;) | 18:15 |
| *** zhiwei has quit IRC | 18:15 | |
| inc0 | aufs or overlay are easiest to setup | 18:15 |
| inc0 | I'm not sure if there is aufs in xenial | 18:15 |
| sean-k-mooney | ill make it a variable and default to overlay unless anyone objects | 18:15 |
| inc0 | I run my labs on overlay happily | 18:15 |
| sean-k-mooney | inc0: i usally use btrfs but i dont want to mess with haveing to convert the filesystem so overlay it is | 18:16 |
| inc0 | britthouser, mind adding an ip if you're on console? | 18:16 |
| inc0 | yeah btrfs is harder to setu | 18:16 |
| britthouser | its still BIOSing | 18:17 |
| inc0 | p | 18:17 |
| britthouser | that blue bar at the top seems to have stopped | 18:17 |
| * britthouser gets out a bigger hammer | 18:18 | |
| inc0 | britthouser, can you throw it all the way to Dallas? | 18:18 |
| britthouser | usually threatening gestures get the job done | 18:18 |
| britthouser | blue bar about about 10% | 18:21 |
| britthouser | but I think if i look really hard its moving... | 18:21 |
| * britthouser holds ruler up to monitor | 18:21 | |
| mark-casey | <lotr meme> one does not simply reboot the deployment host | 18:26 |
| britthouser | lol | 18:27 |
| britthouser | ok its up | 18:28 |
| inc0 | o, even IP is up | 18:28 |
| britthouser | \o/ | 18:28 |
| * britthouser pats pbourke on the back | 18:28 | |
| inc0 | tmux up | 18:29 |
| rhallisey | you guys should have a hangout session | 18:29 |
| inc0 | aaaand lxc is broken | 18:29 |
| rhallisey | so I can listen | 18:29 |
| inc0 | nah | 18:29 |
| britthouser | might learn some polish swear words | 18:29 |
| rhallisey | :) | 18:30 |
| britthouser | lxc-attach: attach.c: lxc_attach: 710 failed to get the init pid | 18:30 |
| britthouser | still seeing BOAT loads of spanning tree packts | 18:31 |
| inc0 | ok lxc is up | 18:32 |
| jemcevoy | How is the cobbler install going? On my network I have cobbler PXE booting hosts using VLAN 99... I had to change the switch to set the default VLAN on the port to be VLAN 99 'cuz the PXE firmware did not have the feature to set the VLAN for PXE booting. | 18:32 |
| britthouser | same behavior as before inc0 | 18:33 |
| inc0 | surprise surprise | 18:33 |
| *** vhosakot has quit IRC | 18:33 | |
| *** vhosakot has joined #openstack-kolla | 18:33 | |
| britthouser | can we turn the lxc off | 18:34 |
| britthouser | and see if that goes away? | 18:34 |
| britthouser | nope..still hamered. | 18:34 |
| sean-k-mooney | im assuming the lxc vm is connected to a linux bridge? | 18:35 |
| britthouser | irtermite you still around? | 18:35 |
| sean-k-mooney | maybe remove the phyical nic from the linux bridge | 18:35 |
| britthouser | that's how we're ssh'd in sean-k-mooney | 18:36 |
| sean-k-mooney | ah that would be a problem | 18:36 |
| inc0 | well we can create new iface for pxe | 18:36 |
| inc0 | ssh* | 18:36 |
| inc0 | which actually might be good idea | 18:36 |
| inc0 | ip a | 18:36 |
| britthouser | yeah on vlan702 | 18:36 |
| britthouser | so p4p1.700 | 18:37 |
| *** unicell has quit IRC | 18:37 | |
| sean-k-mooney | ya it would be good to have a seperate management interface | 18:37 |
| britthouser | .702 | 18:37 |
| britthouser | thx sean-k-mooney | 18:37 |
| *** ayoung has quit IRC | 18:38 | |
| britthouser | don't forget to change the default route | 18:40 |
| jemcevoy | britthouser, sean-k-mooney and inc0, I installed cobber in a VM and used a macvtap interface to a vlan tagged interface. macvtap1@enp1s0f0.99 | 18:40 |
| britthouser | should I do that from console inc0? | 18:41 |
| inc0 | yeah | 18:41 |
| inc0 | ip r del default | 18:41 |
| inc0 | and then add | 18:41 |
| irtermite | britthouser: nope... | 18:41 |
| irtermite | sup? | 18:42 |
| *** diogogmt has quit IRC | 18:42 | |
| inc0 | irtermite, do you have STP enabled on these switches? | 18:42 |
| *** diogogmt has joined #openstack-kolla | 18:42 | |
| *** unicell has joined #openstack-kolla | 18:43 | |
| britthouser | bringing br-pxe down, still seeing TONS of STP | 18:43 |
| irtermite | inc0: why, did you cause a broadcast storm? | 18:44 |
| irtermite | you are still up, aren't you? | 18:45 |
| inc0 | we see a lot...a lot of STP | 18:45 |
| inc0 | yeah, it's not growing so we can actually work | 18:45 |
| inc0 | I don't think it's an loop | 18:45 |
| sean-k-mooney | britthouser: whats the output of brctl show? | 18:45 |
| inc0 | but I wonder where these stp comes from | 18:45 |
| britthouser | # brctl show | 18:46 |
| britthouser | bridge name bridge id STP enabled interfaces | 18:46 |
| britthouser | br-pxe 8000.3cfdfe9c6e24 no p1p1 | 18:46 |
| britthouser | lxcbr0 8000.000000000000 no | 18:46 |
| sean-k-mooney | well there is nothing there that would cause a loop atleast not on that host | 18:47 |
| britthouser | boatloads like this irtermite: | 18:47 |
| britthouser | 13:47:07.310685 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 62e1.e8:65:49:e1:8a:41.80ab, length 42 | 18:47 |
| britthouser | 13:47:07.312425 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 62e2.e8:65:49:e1:8a:41.80ab, length 42 | 18:47 |
| britthouser | 13:47:07.314196 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 62e5.e8:65:49:e1:8a:41.80ab, length 42 | 18:47 |
| britthouser | 13:47:07.315973 STP 802.1w, Rapid STP, Flags [Learn, Forward], bridge-id 62e9.e8:65:49:e1:8a:41.80ab, length 42 | 18:47 |
| britthouser | doesn't seem to be due to our bridging | 18:47 |
| britthouser | But I don't se it on a server in m9-08 | 18:48 |
| britthouser | lemme try a different server in m9-06 | 18:48 |
| sdake | sbezverk RDO_MIRROR = "http://trunk.rdoproject.org/centos7" | 18:50 |
| sdake | DELOREAN = "{}/current-passed-ci/delorean.repo".format(RDO_MIRROR) | 18:50 |
| *** bootsha has quit IRC | 18:51 | |
| britthouser | not seeing this on a different server in m9-06 | 18:52 |
| sdake | DELOREAN_DEPS = "{}/delorean-deps.repo".format(RDO_MIRROR) | 18:53 |
| irtermite | inc0: britthouser: talking with our networking team | 18:53 |
| britthouser | thx irtermite | 18:53 |
| inc0 | appreciate that irtermite | 18:53 |
| inc0 | but I don't think this is issue | 18:53 |
| inc0 | I mean, lxc is acting up | 18:53 |
| inc0 | for whatever reason | 18:53 |
| britthouser | its weird to me irtermite b/c if it was a true broadcast storm, all nodes in the vlan should see it. | 18:54 |
| britthouser | its like this port is a span destination or somthing | 18:54 |
| britthouser | or lxc has really hosed us up | 18:54 |
| inc0 | lxc is down now | 18:55 |
| inc0 | so unless it's a loop that just keep looping | 18:55 |
| irtermite | britthouser: correct | 18:55 |
| britthouser | I guess lets bring the lxc back up inc0? | 18:56 |
| inc0 | yeah | 18:57 |
| inc0 | also let's ssh over 8.250 | 18:57 |
| britthouser | ok I think i prerserved 8.250 config | 19:00 |
| sbezverk | sdake: I got and install debug info | 19:06 |
| sdake | cool | 19:06 |
| sdake | now run gdb openvswitchbinary coefilename | 19:07 |
| inc0 | hmm...seems working? | 19:09 |
| inc0 | britthouser? | 19:09 |
| britthouser | dhcp is handing stuff out | 19:09 |
| britthouser | but TFTP still timing out | 19:09 |
| britthouser | tftp only listening on v6? | 19:10 |
| britthouser | or that includes v4 too? | 19:10 |
| inc0 | hmm | 19:11 |
| sdake | sbezverk lets get on webex again | 19:11 |
| inc0 | no idea | 19:11 |
| sdake | sbezverk gie me 5 mins to check email | 19:11 |
| sbezverk | sdake: ok | 19:12 |
| *** sdake_ has joined #openstack-kolla | 19:13 | |
| sdake_ | sbezverk i'm in | 19:16 |
| *** sdake has quit IRC | 19:16 | |
| irtermite | sdake_: ohai | 19:16 |
| sdake_ | hi irtermite | 19:16 |
| irtermite | hah, just doing my job, fellas. ;) | 19:17 |
| inc0 | still, that doesn't mean we can't appreciate it right? | 19:17 |
| irtermite | inc0: appreciate away! | 19:18 |
| irtermite | and yes, STP enabled | 19:18 |
| *** haplo37__ has joined #openstack-kolla | 19:20 | |
| *** yee_ has quit IRC | 19:22 | |
| irtermite | inc0: do you have the device number of the node(s) that you were seeing the STP issue on | 19:24 |
| britthouser | h729495 | 19:25 |
| britthouser | drop the h | 19:25 |
| britthouser | is that the device number? | 19:25 |
| britthouser | ILO is 10.15.243.226 | 19:25 |
| britthouser | hostname is 729495-comp-s3500-001 | 19:26 |
| inc0 | britthouser, so off to tftp we go | 19:29 |
| britthouser | I can't tftp from the host to the LXC | 19:29 |
| britthouser | I get timeout | 19:29 |
| britthouser | hmm..now it works | 19:30 |
| inc0 | there ya go | 19:30 |
| britthouser | oh..that's in the container | 19:30 |
| inc0 | all I had to do is look at it;) | 19:30 |
| inc0 | nah, that's a host | 19:30 |
| britthouser | yeah it knows you're watching | 19:30 |
| inc0 | lets try on other host | 19:30 |
| inc0 | so it times out somewhere else | 19:32 |
| britthouser | it wasn't working from teh host either | 19:32 |
| britthouser | earlier | 19:32 |
| britthouser | pings are mostly dropped | 19:32 |
| britthouser | 64 bytes from 172.21.0.251: icmp_seq=9 ttl=64 time=0.112 ms | 19:33 |
| britthouser | 64 bytes from 172.21.0.251: icmp_seq=62 ttl=64 time=0.101 ms | 19:33 |
| britthouser | 64 bytes from 172.21.0.251: icmp_seq=115 ttl=64 time=0.103 ms | 19:33 |
| britthouser | 1/min | 19:33 |
| britthouser | I thinking maybe we jsut power this host for now | 19:33 |
| britthouser | move to .225 | 19:34 |
| inc0 | nah it's something with lxc I think | 19:34 |
| *** berendt has joined #openstack-kolla | 19:35 | |
| *** daneyon_ has joined #openstack-kolla | 19:37 | |
| jemcevoy | I have seen strange network problems like this when the MTU's are too large for the VLAN... | 19:37 |
| inc0 | hmm | 19:37 |
| inc0 | might be good | 19:37 |
| britthouser | looks like we're all set to 1500 jemcevoy | 19:38 |
| jemcevoy | any overlays creeping into the 1500? | 19:38 |
| jemcevoy | but your ping is only 64 bytes... not that | 19:39 |
| *** haplo37__ has quit IRC | 19:41 | |
| jemcevoy | I have gotten bitten by ebtables... | 19:41 |
| inc0 | sooo....I hate this. | 19:41 |
| britthouser | I'm tempted to configure ebtables to block all this STP | 19:41 |
| britthouser | this=? | 19:42 |
| inc0 | br-pxe sees pings but veth in the same br doesnt | 19:42 |
| *** daneyon_ has quit IRC | 19:42 | |
| britthouser | can we compare MAC addresses of br-pxe and eth1 ? | 19:42 |
| britthouser | ok they are very different | 19:43 |
| *** jeh has quit IRC | 19:43 | |
| *** jrist has quit IRC | 19:43 | |
| inc0 | soo...these arps doesn't seem right | 19:43 |
| inc0 | nope it's all good | 19:44 |
| britthouser | eth1 Link encap:Ethernet HWaddr 00:16:3e:60:fa:6d | 19:44 |
| britthouser | yeah | 19:44 |
| britthouser | RX bytes:4106136 (4.1 MB) TX bytes:63702 (63.7 KB) | 19:44 |
| britthouser | RX packets:50045 errors:0 dropped:0 overruns:0 frame:0 | 19:45 |
| inc0 | I blame linux bridge being broken | 19:45 |
| inc0 | no clue how, but something is just wrong | 19:45 |
| britthouser | just for grins, I'm gonna start walking through steps on .225 | 19:46 |
| inc0 | ok | 19:46 |
| inc0 | you can copy all the files we prepared | 19:46 |
| *** jeh has joined #openstack-kolla | 19:46 | |
| *** jeh has quit IRC | 19:46 | |
| *** jeh has joined #openstack-kolla | 19:46 | |
| inc0 | remember to remove .225 and this one from the list tho | 19:47 |
| inc0 | I'm off for 30min, gonna check new doom, my brain needs some mayhem about now | 19:48 |
| openstackgerrit | sean mooney proposed openstack/kolla: add baremetal role to install kolla deps. https://review.openstack.org/325631 | 19:49 |
| *** vhosakot has quit IRC | 19:55 | |
| *** ayoung has joined #openstack-kolla | 19:56 | |
| britthouser | looks like we might have skipped the kernel update on .250 | 19:56 |
| britthouser | gonna do that and reboot | 19:56 |
| *** jrist has joined #openstack-kolla | 19:57 | |
| * britthouser is looking for more darts to throw at the wall | 19:57 | |
| *** matrohon has joined #openstack-kolla | 19:58 | |
| sdake_ | anyone got any tricks up their sleeves to execute a command inside a container as pid 2 instead of pid 1 | 20:03 |
| inc0 | sdake_, docker exec -it my_container whatever | 20:04 |
| inc0 | when container is runnning | 20:04 |
| sdake_ | we ar running it from kubernetes | 20:04 |
| sdake_ | so that wont work | 20:04 |
| sdake_ | weneed it to come in a deamon set | 20:04 |
| sdake_ | we have tried bash -c "sleep 1;operatoin" | 20:05 |
| sdake_ | but tahat doesn't work | 20:05 |
| mark-casey | bash -c 'bash -c'? | 20:05 |
| britthouser | what about sleep 1 & operation | 20:07 |
| britthouser | to bg the sleep | 20:07 |
| *** vhosakot has joined #openstack-kolla | 20:08 | |
| sdake_ | we tried bash -c 'sleep 1;command' | 20:08 |
| sdake_ | the code is executed by kolla's python scripts,s so we cna't really use shell comands directly | 20:08 |
| sbezverk | mark-case: here is what I have | 20:08 |
| sbezverk | http://paste.openstack.org/show/549299/ | 20:08 |
| sdake_ | putting thatin quotes doesn't work | 20:09 |
| britthouser | what about & instead of ; | 20:09 |
| sean-k-mooney | sdake_: if its fro debug why not run the command in screen | 20:09 |
| sdake_ | sean-k-mooney syntax? | 20:10 |
| sdake_ | britthouser also fails | 20:10 |
| *** vhosakot has quit IRC | 20:10 | |
| mark-casey | so... do you need a second process or you have a single process that refuses to be pid1 | 20:11 |
| sean-k-mooney | http://paste.openstack.org/show/549300/ | 20:11 |
| mark-casey | nice | 20:11 |
| mark-casey | hehe... screen: command not found | 20:12 |
| sdake_ | pid 1 is unkillable | 20:12 |
| sean-k-mooney | hum was worth a shot the double bash? | 20:13 |
| mark-casey | sean-k-mooney: no no that was just me guessing | 20:14 |
| mark-casey | don't know that screen didn't work | 20:14 |
| sean-k-mooney | oh ok | 20:14 |
| sean-k-mooney | we run the ovs-vswithd in screen like that in our devstack plugin so it should work if its available in the container https://github.com/openstack/networking-ovs-dpdk/blob/master/devstack/ovs-dpdk/ovs-dpdk-init#L542 | 20:15 |
| britthouser | The variance on server reboot times is quite large | 20:18 |
| sdake_ | thanks mark-casey - sbezverk said he will gie that a try ilater | 20:19 |
| *** vhosakot has joined #openstack-kolla | 20:19 | |
| mark-casey | sdake_: screen wasn't me. unless he mean the double bash :D | 20:19 |
| sdake_ | rather sean-k-mooney :) | 20:19 |
| mark-casey | ah, cool | 20:20 |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb container https://review.openstack.org/346446 | 20:20 |
| sean-k-mooney | no worries. is he still have the ovsdb issue? | 20:20 |
| sdake_ | yup | 20:22 |
| openstackgerrit | Mathias proposed openstack/kolla: Added influxdb role https://review.openstack.org/346449 | 20:22 |
| sdake_ | was attempting to get a core dump | 20:22 |
| sdake_ | to understand what the problem is | 20:22 |
| sdake_ | but it sppears PID 1 does not dump a core | 20:22 |
| *** banix has quit IRC | 20:22 | |
| britthouser | ssh ssh root@172.21.8.253 if you want to play along inc0 | 20:23 |
| inc0 | kk | 20:23 |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 20:23 |
| britthouser | I'm using .253/.254 for host/container | 20:23 |
| sean-k-mooney | sdake_: its really is strange. the strace he sent before looked fine and so does the argument its starting with | 20:23 |
| sdake_ | sean-k-mooney the strace is fine because the process crashes and restarts | 20:24 |
| openstackgerrit | Mathias proposed openstack/kolla: Added grafana container https://review.openstack.org/346447 | 20:24 |
| sdake_ | if you use strace, it doesn't crash at all | 20:24 |
| sdake_ | rather | 20:24 |
| openstackgerrit | Mathias proposed openstack/kolla: Added grafana role https://review.openstack.org/346450 | 20:24 |
| sdake_ | using strace = no crash | 20:24 |
| sean-k-mooney | sdake_: oh strange | 20:24 |
| sdake_ | ya - looks like timing problem to me | 20:24 |
| sdake_ | but i'd like a backtrace to back it up | 20:25 |
| sdake_ | easy solution is to put some sleep nonsense in the ovs container | 20:25 |
| sean-k-mooney | this is only happening with k8s though correct? | 20:25 |
| sdake_ | yup only k8s | 20:25 |
| sdake_ | no idea why - without a backtrace it is undebuggable | 20:26 |
| sdake_ | but sbezverk had to roll out for awhile | 20:26 |
| *** ppowell has quit IRC | 20:27 | |
| sdake_ | if you delay the start of the ovs db there is no crash | 20:28 |
| sdake_ | libc is generating hte segfault | 20:28 |
| *** jtriley has quit IRC | 20:28 | |
| *** Jeffrey4l has quit IRC | 20:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 20:35 | |
| openstackgerrit | Mathias proposed openstack/kolla: Added telegraf role https://review.openstack.org/346448 | 20:35 |
| Mech422 | sdake_: ubuntu can have wierd timing issues with ovs - sometimes the partition isn't mounted for ovs | 20:35 |
| Mech422 | sdake_: a boot ordering issue | 20:35 |
| Mech422 | sdake_: I haven't seen it in a while though, can't remember if its something I patched, or if it was fixed upstream | 20:36 |
| inc0 | britthouser, sooo...that empty tcpdump doesn't seem good | 20:36 |
| britthouser | I removed the interface from teh bridge | 20:37 |
| inc0 | ok..why? | 20:37 |
| britthouser | see if those pkts are coming from network or local | 20:37 |
| britthouser | better? | 20:38 |
| mewald | I just rebased my monitoring stack patches into a different order. I noticed something was wrong. We can now get the influxdb container and role patches ready and after that the telegraf role which depends on influxdb role. Would be nice to get a few reviews, I'll go to bed now. gn8 | 20:38 |
| *** jrist has quit IRC | 20:38 | |
| britthouser | network has tons or ARPs now for .251 b/c its *still* booting | 20:38 |
| inc0 | thanks mewald ! | 20:38 |
| inc0 | oh really? | 20:39 |
| inc0 | after kernel update right? | 20:39 |
| inc0 | maybe kernel was an issue there | 20:39 |
| britthouser | its still BIOSing | 20:39 |
| *** mewald has quit IRC | 20:40 | |
| *** vhosakot has quit IRC | 20:43 | |
| *** vhosakot has joined #openstack-kolla | 20:44 | |
| britthouser | I'm rebooting another node, gonna see if it gets a lease | 20:45 |
| britthouser | then manually test tftp | 20:45 |
| britthouser | before going down route populating cobbler again | 20:45 |
| britthouser | doh..tx | 20:47 |
| inc0 | :D | 20:47 |
| britthouser | I'm starting to think .226 is never coming back | 20:48 |
| sean-k-mooney | 1 down 134 to go :) | 20:49 |
| britthouser | I'll break all nodes before we're done! =P | 20:50 |
| *** jrist has joined #openstack-kolla | 20:50 | |
| *** vhosakot has quit IRC | 20:51 | |
| *** mliima has quit IRC | 20:53 | |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Add Ceph support for Mariadb https://review.openstack.org/350312 | 20:53 |
| britthouser | ok server gets IP | 20:53 |
| britthouser | everyone cross your fingers | 20:53 |
| britthouser | gonna try a remote tftp | 20:53 |
| inc0 | britthouser, just shoot it and restart | 20:53 |
| inc0 | 226 I mean | 20:54 |
| jemcevoy | I am still hitting the mariadb bootstrap failure... I am using version 3.0.0 should I try 2.0.2? | 20:54 |
| inc0 | jemcevoy, you're doing stable/mitaka? | 20:54 |
| inc0 | or master? | 20:54 |
| jemcevoy | First install... I do not know.... stable sounds good to me... | 20:55 |
| britthouser | Yeah tftp worked | 20:55 |
| britthouser | thx for all the finger crossings | 20:55 |
| *** zhiwei has joined #openstack-kolla | 20:55 | |
| inc0 | jemcevoy, how did you get your kolla installed?;) | 20:56 |
| inc0 | github? | 20:56 |
| jemcevoy | yes | 20:56 |
| sdake_ | jemcevoy 3.0.0 is master | 20:56 |
| sdake_ | you want stable/mitaka or stable/liberty | 20:56 |
| sdake_ | unfortunately this is not documented | 20:56 |
| inc0 | yeah, git checkout stable/mitaka | 20:56 |
| inc0 | that's what prod should run today | 20:56 |
| jemcevoy | mitaka... need to support UEFI for a customer. Moving from juno | 20:56 |
| inc0 | although this will require ansible 1.9.4 | 20:57 |
| sdake_ | paying 1 USD to anyone to fix our docs ;) | 20:57 |
| *** matrohon has quit IRC | 20:57 | |
| inc0 | juno->mitaka? that's gonna be...interesting | 20:57 |
| sdake_ | jemcevoy in that case i'd go from juno to liberty ot mitaka | 20:57 |
| jemcevoy | pave over mostly... | 20:57 |
| sdake_ | openstack is not designed to upgrade more then 1 release | 20:58 |
| sdake_ | if its a datacenter pave, mitaka is what yu want | 20:58 |
| inc0 | mitaka is latest stable jemcevoy | 20:58 |
| inc0 | you don't want to deploy master on prod, trust me;) | 20:58 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Add doc for configuring private docker registry https://review.openstack.org/350785 | 20:58 |
| inc0 | and master kolla deploys master openstack | 20:58 |
| *** zhiwei has quit IRC | 21:00 | |
| *** rhallisey has quit IRC | 21:01 | |
| inc0 | britthouser, I'm shooting down .226 | 21:01 |
| britthouser | I powered it off aleady | 21:01 |
| britthouser | I'm to the "gather mac addresses" | 21:03 |
| jemcevoy | inc0, so I just need to checkout remotes/origin/stable/mitaka, downgrade ansible to 1.9.4 and do the kolla-build again... Will the /etc/kolla/globals.yml and passwords.yml need to be changed? | 21:04 |
| inc0 | you need to change openstack_release in globals.yml to 2.0.2 | 21:05 |
| *** godleon has joined #openstack-kolla | 21:09 | |
| jemcevoy | cool | 21:10 |
| britthouser | inc0 and pbourke - I'm about done for today. Gonna test one server install and then let you due your full scale assault inc0. That will tell us how many nodes of the cluster are workable. Provided those all come online, then tomorrow we need to rework the cobbler configs to: 1) use bonding for p1p2+p4p2 2) assign servers specific role and assign profile accordingly 3) I forgot. | 21:16 |
| inc0 | kk thanks britthouser ! | 21:16 |
| *** signed8bit is now known as signed8bit_Zzz | 21:17 | |
| *** signed8bit_Zzz is now known as signed8bit | 21:21 | |
| sdake_ | inc0 heidi has answered your quesstion re custom feedback into the design process of the mascots | 21:25 |
| sdake_ | inc0 http://lists.openstack.org/pipermail/openstack-dev/2016-August/100915.html | 21:27 |
| britthouser | cobbler can be programmed in python via xmlrpc | 21:27 |
| britthouser | but osic has a python script to generate cobbler CLi commands | 21:27 |
| britthouser | inc0 booting that first server.... | 21:28 |
| * britthouser crosses fingers | 21:28 | |
| irtermite | britthouser: you like it? | 21:29 |
| britthouser | It works, so yeah I like it. Just makes me scratch my head a bit. | 21:29 |
| britthouser | python->bash->cobbler | 21:29 |
| britthouser | when it could be | 21:29 |
| irtermite | heh | 21:30 |
| britthouser | python->xmlrpc->cobbler | 21:30 |
| irtermite | do what you know? | 21:30 |
| irtermite | I don't know. I wasn't part of that solution | 21:30 |
| britthouser | =) I used it, I worked, I can't really complain. =) | 21:30 |
| irtermite | heh | 21:30 |
| britthouser | server is booting!!!!! | 21:30 |
| britthouser | well its kickstarting!!!! | 21:30 |
| britthouser | ok inc0 I'm going to dinner. Feel free to mass-reboot all the nodes. | 21:32 |
| irtermite | britthouser: https://cdn.meme.am/instances/500x/56438302.jpg | 21:32 |
| britthouser | ROFL | 21:32 |
| britthouser | We'll probably hack the script up tomorrow to bond the p1p2 and p4p2 interfaces together. Would that be something we coudl contribute back to OSIC, irtermite ? | 21:33 |
| inc0 | ok | 21:33 |
| inc0 | britthouser, I'll answer that: yeah | 21:34 |
| irtermite | of course. maybe may it an option and submit a pull? | 21:34 |
| inc0 | we can prep an ansible playbook for that | 21:34 |
| britthouser | you mean post-kickstart? | 21:34 |
| britthouser | I was thinking let cobbler do it in kickstart. | 21:34 |
| britthouser | anyway, we can discuss more tamale | 21:35 |
| britthouser | ciao! | 21:35 |
| irtermite | hahhaa @ tamale | 21:35 |
| irtermite | ok | 21:35 |
| inc0 | cyas | 21:35 |
| irtermite | inc0: we don't have an osic-cluser org, do we? | 21:35 |
| irtermite | on it | 21:36 |
| inc0 | well we have osic on github | 21:36 |
| irtermite | I guess that works... | 21:37 |
| irtermite | is melvin owner? | 21:37 |
| inc0 | irtermite, https://github.com/osic/ref-impl/blob/master/documents/bare_metal_provisioning.md | 21:37 |
| *** yee_ has joined #openstack-kolla | 21:37 | |
| inc0 | yeah, I think I have admin powers as well | 21:37 |
| irtermite | curious if we should leave this forked or pull it over completely... | 21:38 |
| inc0 | why would you fork it? | 21:38 |
| inc0 | I mean, this stuff was made for this hardware | 21:38 |
| irtermite | no no | 21:38 |
| irtermite | https://github.com/osic/osic-bare-metal-deployment-process | 21:39 |
| irtermite | it is forked from our rpc org | 21:39 |
| inc0 | the thing I linked is same think pretty much | 21:39 |
| inc0 | with few changes as jthorne moved around his stuff | 21:39 |
| inc0 | so we re-hosted it | 21:39 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Added support for template variables https://review.openstack.org/351408 | 21:40 |
| irtermite | I know... I'm just saying it is forked from rsoprivatecloud into osic... So, should we just own it in osic or leave it owned by rsoprivatecloud | 21:40 |
| irtermite | I guess we should just leave it the way it is | 21:40 |
| openstackgerrit | David Wang proposed openstack/kolla-kubernetes: Added support for template variables https://review.openstack.org/351408 | 21:41 |
| *** ayoung has quit IRC | 21:43 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 21:45 |
| inc0 | britthouser, seems to be ok | 21:46 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 21:47 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 21:48 |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add dockerfiles for keystone fernet https://review.openstack.org/351139 | 21:50 |
| *** mark-casey has quit IRC | 21:57 | |
| openstackgerrit | Shaun Smekel proposed openstack/kolla: Add full support for fernet [WIP] https://review.openstack.org/349366 | 22:01 |
| *** banix has joined #openstack-kolla | 22:01 | |
| *** jemcevoy has quit IRC | 22:04 | |
| *** jemcevoy has joined #openstack-kolla | 22:07 | |
| *** inc0 has quit IRC | 22:08 | |
| *** signed8bit has quit IRC | 22:11 | |
| openstackgerrit | sean mooney proposed openstack/kolla: add baremetal role to install kolla deps. https://review.openstack.org/325631 | 22:13 |
| openstackgerrit | sean mooney proposed openstack/kolla: This change allow setting kernel commandline args https://review.openstack.org/351412 | 22:13 |
| sbezverk | sdake: ping | 22:16 |
| *** daneyon_ has joined #openstack-kolla | 22:20 | |
| *** daneyon__ has joined #openstack-kolla | 22:24 | |
| *** daneyon_ has quit IRC | 22:24 | |
| *** ayoung has joined #openstack-kolla | 22:26 | |
| *** daneyon has quit IRC | 22:28 | |
| *** daneyon__ has quit IRC | 22:28 | |
| sean-k-mooney | ok im going to head off before midnight today so ill catch people tomorow | 22:30 |
| *** ntpttr has quit IRC | 22:31 | |
| *** ntpttr has joined #openstack-kolla | 22:31 | |
| sean-k-mooney | ill test the kolla-host playbook with ubuntu 14.04 and centos but it seams to be working with 16.04 | 22:32 |
| sdake_ | sean-k-mooney nice work! | 22:58 |
| kfox1111 | https://github.com/kubernetes/kubernetes/issues/29607 | 23:03 |
| kfox1111 | ^----- kolla-kubernetes folks, we should all follow/weigh in on this. | 23:03 |
| kfox1111 | there is a lot of discussion about doing templating in k8s for configy bits right now. | 23:04 |
| sdake_ | i dont get why people say "for the record" | 23:04 |
| sdake_ | pointless speech | 23:05 |
| wirehead_ | Same reason they say “Upright and locked position” in the safety briefing. | 23:05 |
| kfox1111 | hehe. | 23:05 |
| *** ayoung has quit IRC | 23:06 | |
| sdake_ | about as pointless as "food for thought" | 23:06 |
| sbezverk | sdake_: it turned out to be issue with daemonset | 23:07 |
| sdake_ | sbezverk out of curiosity ind expanding | 23:07 |
| sdake_ | openvswitch shouldn't segfault regardless of how daemonset rolls | 23:07 |
| sbezverk | sdake_ I changed kind of the pod to replication controller and it is working 100% | 23:08 |
| sbezverk | same image, same config same everything else | 23:08 |
| kfox1111 | daemonsets not working? | 23:08 |
| sdake_ | does a rc meet the requirements? | 23:09 |
| sdake_ | sbezverk i'd still like a backtrce on openvswitch crash | 23:09 |
| sdake_ | so we can make all 3 peices of software better | 23:10 |
| kfox1111 | what works with an rc and not a daemonset? I've never heard of that before. | 23:10 |
| wirehead_ | Yeah, that seems like a deep dark nasty buggy issue. | 23:10 |
| sdake_ | a backtrace would open it up | 23:10 |
| wirehead_ | A portent, signifying doom, destruction, and really bad coffee for the rest of your life. | 23:10 |
| kfox1111 | yeah. I'm worried some of my daemonsets for other projects will fail. what is the unexpected behavior? | 23:11 |
| sdake_ | libc segfaults in openvswitch on startup | 23:11 |
| sbezverk | sdake_ I sent email to ovs folks with that error for their input, maybe somebody saw something similar | 23:11 |
| sdake_ | sbezverk can you do the screen thing | 23:12 |
| sdake_ | your 90% of the way there | 23:12 |
| wirehead_ | Yeah, I kinda get what they are going on about with the downward API and stuff. | 23:12 |
| wirehead_ | The more I think about it, the more we might have to contend with that as part of our startup to fix issues I haven’t encountered yet. | 23:12 |
| kfox1111 | and it segfaults when started with a daemonset but not with a rc? | 23:13 |
| wirehead_ | (I switched subject to the patch kfox mentioned) | 23:13 |
| sbezverk | sdake_ as soon as I get it working with replaciation controller.. I already wasted whole day on this PoC I mean Daemonset | 23:13 |
| kfox1111 | sbezverk: I'm still concerned its something else... daemonsets and rc's really arn't different enough that one shoudl segfault and the other not. It could be something we see crash on RC's sometimes too. | 23:15 |
| kfox1111 | wirehead_: yeah. I figured. | 23:15 |
| kfox1111 | wirehead_: I really want to see secrets spilt out of the configmaps. this templating thing may be a good way to do that. | 23:15 |
| wirehead_ | Yeah. | 23:16 |
| wirehead_ | Frankly though, I think the way that there’s no way to config Nova such that the secrets are intermingled with the rest of the config is a bug. | 23:16 |
| kfox1111 | Agreed. | 23:16 |
| wirehead_ | I just don’t want to undergo the challenge of doom and submit a patch for that. :P | 23:16 |
| kfox1111 | at very least, it shoudl be 2 ini files. | 23:16 |
| kfox1111 | but they mix config and secret today. | 23:17 |
| kfox1111 | ip address for mysql, and creds. so it can't easily be seperated. | 23:17 |
| kfox1111 | but if you could template it in kube, you could put config in one, secret in the other, and joint them at the last second before hitting the container. that would work around it farily nicely. | 23:17 |
| wirehead_ | It’s like the scene in the 1980 Flash Gordon movie where you have to stick your hand in the tree stump containing a monster that will sting you if you put your hand in the wrong part of the tree stump. | 23:17 |
| kfox1111 | hehe. yeah. | 23:17 |
| wirehead_ | Yeah. Pretty much, you should be able to template env variables and a few facts. That’s a reasonable boundary. | 23:19 |
| wirehead_ | If we did that, either on the kube side, or the kolla side, we can just take advantage of how 1.3 lets you put secrets in env variables. | 23:20 |
| sbezverk | kfox1111: what is interesting as soon as I added strace to the command executed by Daemonset | 23:21 |
| sbezverk | it starts working | 23:22 |
| sbezverk | it looks like some sort of racing issue | 23:22 |
| kfox1111 | Until k8s actually treats secrets differently then configmaps, it probably doesn't matter too much. I think we can wait to see what they come up with. | 23:22 |
| kfox1111 | sbezverk: yeah. that even more makes me think its a race and we will probably see it with rc's too. | 23:22 |
| wirehead_ | Yah. | 23:22 |
| kfox1111 | sometimes its exasterbated by env though. if you capture the env between the two, is anything different? | 23:23 |
| sbezverk | well I tried multiple more than 10 iterations and I got 100% positive results wit RC | 23:23 |
| sdake_ | sbezverk a backtracee will take literally 10 minutes at most to make if that screen approach works | 23:23 |
| sdake_ | sbezverk backtrace = solution to problem | 23:24 |
| sbezverk | ok I have 30 minutes before my wife will shoot me | 23:24 |
| kfox1111 | :) | 23:24 |
| sbezverk | I can start webex | 23:24 |
| sdake_ | i am otp atm | 23:24 |
| sdake_ | just use that screen command | 23:25 |
| sdake_ | and i'll tell you the two commands to run | 23:25 |
| sdake_ | its either going to work or not going to work | 23:25 |
| sbezverk | "screen" needs to be installed right? | 23:26 |
| sdake_ | yes you need ot rebuild that image with screen | 23:26 |
| sdake_ | if it doen't work i am out of ideas to debug | 23:26 |
| sdake_ | backtrace = gold :) | 23:27 |
| kfox1111 | I asked the kube folks for a way to run a debug sidecar container in the same namespace as another container. so you could gdb dump it when gdb wasn't in your main container. | 23:29 |
| kfox1111 | really would like to see that... :) | 23:29 |
| sdake_ | in this case that wouldn't help because the container segfaults immediately | 23:30 |
| kfox1111 | sleep 10; exec thingy? | 23:30 |
| kfox1111 | :) | 23:30 |
| sdake_ | sleep 1 causes the problem not to reproduce | 23:30 |
| kfox1111 | really? | 23:30 |
| sdake_ | really | 23:30 |
| kfox1111 | ... | 23:30 |
| sbezverk | not really, because when we specify sleep | 23:31 |
| sbezverk | ovsdb ignores all setting | 23:31 |
| sbezverk | we provide in command line | 23:31 |
| kfox1111 | I wonder if theres a condition where the process is started before all the volumes are mounted then. | 23:32 |
| kfox1111 | does it have many volumes? | 23:32 |
| sdake_ | my money is on that scenario kfox | 23:33 |
| sdake_ | if i had money to bet :) | 23:33 |
| *** yee_ has quit IRC | 23:33 | |
| kfox1111 | 1.3.4 fixed a major problem where volumes wouldn't always mount and the container would wedge. | 23:33 |
| kfox1111 | I think. | 23:34 |
| kfox1111 | I havent' seen it since, but haven't tried it too heavily either yet. | 23:34 |
| sbezverk | kfox1111: I am running 1.3.4 for customer dns net=host issue | 23:37 |
| sbezverk | I meant custom dns | 23:37 |
| kfox1111 | ah. | 23:38 |
| *** berendt has quit IRC | 23:38 | |
| sbezverk | sdake_ it does not work, that command Sean gave.. | 23:39 |
| sbezverk | the container get into the completed state | 23:39 |
| sbezverk | but it is supposed to be running in the foreground | 23:40 |
| sbezverk | Running command: 'screen -dms ovsdb bash -c /usr/sbin/ovsdb-server /etc/openvswitch/conf.db -vconsole:emer -vsyslog:err --remote=punix:/run/openvswitch/db.sock -vfile:info --log-file=/var/log/kolla/openvswitch/ovsdb-server.log' | 23:40 |
| sbezverk | openvswitch-db-1vb6e 0/1 CrashLoopBackOff 5 4m | 23:41 |
| sbezverk | openvswitch-db-2c65j 0/1 CrashLoopBackOff 5 4m | 23:41 |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Add customizations for keystone https://review.openstack.org/349736 | 23:42 |
| kfox1111 | kubectl describe pod openvswitch-db-1vb6e ? whats the reason? | 23:42 |
| *** Jeffrey4l_ has quit IRC | 23:42 | |
| *** sdake has joined #openstack-kolla | 23:44 | |
| sbezverk | http://paste.openstack.org/show/549320/ | 23:45 |
| *** Jeffrey4l has joined #openstack-kolla | 23:45 | |
| *** sdake_ has quit IRC | 23:46 | |
| kfox1111 | hmm.. not really very useful. :/ any logs in kubectl logs openvswitch-db-1vb6e ? | 23:48 |
| kfox1111 | if not, then tweak the container to have | 23:49 |
| kfox1111 | command: [/bin/bash, -c, 'while true; do sleep 100; done'] | 23:49 |
| kfox1111 | then exec in and run the same command and see if it runs or errors. maybe you can see the error that way. | 23:50 |
| sbezverk | kfox1111 kubectl logs does not work is pod is not in running state | 23:50 |
| sbezverk | kfox1111 the issue here is json, I think it does not lke these manipulations | 23:51 |
| kfox1111 | try with the sleep thing. should be able to see if the command works that way. its a complicated enough command line string I can see it failing. | 23:52 |
| *** haplo37__ has joined #openstack-kolla | 23:53 | |
| sbezverk | kfox1111 by itself sleep works, but combaining two together never worked | 23:54 |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Add customizations for keystone https://review.openstack.org/349736 | 23:54 |
| *** zhurong has joined #openstack-kolla | 23:55 | |
| openstackgerrit | Naren Narendra proposed openstack/kolla: Add customizations for keystone https://review.openstack.org/349736 | 23:56 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!
