| *** goldyfruit has quit IRC | 00:00 | |
| kbaegis | inc0: Quorum is >50 percent, and ceph uses PAXOS for electing the master monitor | 00:00 |
|---|---|---|
| inc0 | all 3 mons does the same | 00:00 |
| kbaegis | So for 3 nodes, 2 must be up | 00:01 |
| kbaegis | 1 can be failed | 00:01 |
| inc0 | yeah, all 3 are messed upo | 00:01 |
| kbaegis | The monitors shouldn't formally care with paxos though | 00:01 |
| inc0 | I mean | 00:01 |
| kbaegis | Just building the map every epoch | 00:01 |
| inc0 | they can't handle themselves | 00:01 |
| kbaegis | So your storage pools are up and clean? | 00:02 |
| openstackgerrit | Merged openstack/kolla-ansible master: Add a Kolla 'devstack' mode https://review.openstack.org/454690 | 00:02 |
| kbaegis | I still can't figure out this issue: https://hastebin.com/qalaqofafa.php | 00:02 |
| *** yingjun has joined #openstack-kolla | 00:03 | |
| inc0 | kbaegis: iface doesn't have IP | 00:03 |
| inc0 | tunnel_interface to be precise | 00:03 |
| kbaegis | Does tun interface need one? I thought that defaulted to your network one, which isn't supposed to have one allocated? | 00:04 |
| inc0 | it does | 00:04 |
| inc0 | well | 00:04 |
| inc0 | it seems like it didn't default for some reason | 00:04 |
| inc0 | did you specify tunnel_iface? | 00:04 |
| kbaegis | Yes. Should that be the api_interface as well instead? | 00:04 |
| kbaegis | Or the neutron_external one? In my AIO deployments, it was always the second interface without an IP | 00:04 |
| inc0 | well, it's fine to specify tunnel_interface | 00:05 |
| inc0 | but it has to have an address | 00:05 |
| inc0 | neutron_external is different, it doesn't need an IP | 00:05 |
| inc0 | but tunnel needs it | 00:05 |
| kbaegis | inc0: It makes since, since you're terminating a L3 tunnel | 00:05 |
| kbaegis | K | 00:05 |
| inc0 | exactly | 00:05 |
| inc0 | anyway, I'm off, ceph hates me | 00:07 |
| kbaegis | ceph hates everyone | 00:07 |
| *** jtriley has joined #openstack-kolla | 00:08 | |
| kbaegis | https://www.youtube.com/watch?v=0QaAKi0NFkA | 00:08 |
| inc0 | nasty little mollusc | 00:08 |
| inc0 | ah, stabby crabby | 00:08 |
| kbaegis | The sea creatures are uniting against us | 00:09 |
| kbaegis | And cuttlefish shall inherit the earth | 00:09 |
| inc0 | shellfish shall shellshock | 00:09 |
| inc0 | anyway, have a great afternoon, I'm going to do brainless shit for rest of today | 00:10 |
| inc0 | or play with machine learning | 00:10 |
| kbaegis | Sure, night | 00:10 |
| inc0 | cyas | 00:10 |
| *** TxGirlGeek has quit IRC | 00:13 | |
| *** jtriley has quit IRC | 00:22 | |
| *** ducttape_ has joined #openstack-kolla | 00:30 | |
| *** trozet has quit IRC | 00:30 | |
| *** spsurya has joined #openstack-kolla | 00:37 | |
| *** zhurong has joined #openstack-kolla | 00:45 | |
| *** hongbin has quit IRC | 00:54 | |
| *** tovin07_ has joined #openstack-kolla | 00:59 | |
| *** lucasxu has joined #openstack-kolla | 01:00 | |
| *** lucasxu has quit IRC | 01:01 | |
| *** salv-orlando has joined #openstack-kolla | 01:06 | |
| duonghq | moring guys | 01:10 |
| *** zhubingbing__ has joined #openstack-kolla | 01:11 | |
| *** inc0 has quit IRC | 01:11 | |
| *** inc0 has joined #openstack-kolla | 01:11 | |
| *** iniazi has joined #openstack-kolla | 01:13 | |
| * inc0 bragging - yisss, ceph is working in gate https://review.openstack.org/#/c/466815/ | 01:14 | |
| kbaegis | :) | 01:15 |
| kbaegis | duonghq: Morning! | 01:15 |
| *** ducttape_ has quit IRC | 01:15 | |
| duonghq | moring kbaegis | 01:15 |
| *** ducttape_ has joined #openstack-kolla | 01:15 | |
| *** salv-orlando has quit IRC | 01:16 | |
| *** ducttape_ has quit IRC | 01:16 | |
| *** jtriley has joined #openstack-kolla | 01:17 | |
| *** goldyfruit has joined #openstack-kolla | 01:18 | |
| *** zhurong has quit IRC | 01:33 | |
| *** zhurong has joined #openstack-kolla | 01:35 | |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Fix mechanism_drivers configuration item repeats https://review.openstack.org/473753 | 01:41 |
| kbaegis | This is awesome | 01:41 |
| *** kbaegis has quit IRC | 01:42 | |
| openstackgerrit | Dave McCowan proposed openstack/kolla-kubernetes master: [WIP] Add support for HTTPS on external API network https://review.openstack.org/471992 | 01:42 |
| *** trozet has joined #openstack-kolla | 01:45 | |
| *** xinliang has quit IRC | 01:46 | |
| *** xinliang has joined #openstack-kolla | 01:59 | |
| *** trozet has quit IRC | 02:12 | |
| *** unicell has quit IRC | 02:13 | |
| *** goldyfruit has quit IRC | 02:16 | |
| *** salv-orlando has joined #openstack-kolla | 02:16 | |
| *** ducttape_ has joined #openstack-kolla | 02:17 | |
| *** salv-orlando has quit IRC | 02:25 | |
| *** eanylin has quit IRC | 02:29 | |
| *** kbaegis has joined #openstack-kolla | 02:30 | |
| *** kbaegis has left #openstack-kolla | 02:35 | |
| *** ducttape_ has quit IRC | 02:43 | |
| *** ducttape_ has joined #openstack-kolla | 02:43 | |
| *** japestinho has joined #openstack-kolla | 02:52 | |
| *** eanylin has joined #openstack-kolla | 03:02 | |
| *** jtriley has quit IRC | 03:05 | |
| *** ducttape_ has quit IRC | 03:08 | |
| *** jtriley has joined #openstack-kolla | 03:23 | |
| openstackgerrit | jimmygc proposed openstack/kolla-ansible master: Make glance filesystem_store_datadir configurable. https://review.openstack.org/452094 | 03:26 |
| *** jtriley has quit IRC | 03:29 | |
| *** dave-mccowan has quit IRC | 03:31 | |
| *** kbaegis has joined #openstack-kolla | 03:35 | |
| *** itlinux has joined #openstack-kolla | 03:41 | |
| *** itlinux_ has joined #openstack-kolla | 03:42 | |
| kbaegis | duonghq: Have you played with skydive yet? | 03:47 |
| duonghq | kbaegis, not yet :( | 03:49 |
| *** salv-orlando has joined #openstack-kolla | 03:52 | |
| *** jtriley has joined #openstack-kolla | 03:55 | |
| *** jamesbenson has joined #openstack-kolla | 03:56 | |
| *** jamesbenson has quit IRC | 04:01 | |
| *** salv-orlando has quit IRC | 04:02 | |
| *** jtriley has quit IRC | 04:03 | |
| *** caowei has joined #openstack-kolla | 04:03 | |
| *** daidv_ has quit IRC | 04:14 | |
| spsurya | morning | 04:16 |
| *** kbaegis has quit IRC | 04:18 | |
| *** zhurong has quit IRC | 04:28 | |
| *** EmilienM has quit IRC | 04:30 | |
| *** zhurong has joined #openstack-kolla | 04:38 | |
| *** EmilienM has joined #openstack-kolla | 04:39 | |
| *** janki has joined #openstack-kolla | 04:40 | |
| *** cuongnv has joined #openstack-kolla | 04:42 | |
| *** unicell has joined #openstack-kolla | 04:43 | |
| *** salv-orlando has joined #openstack-kolla | 04:51 | |
| *** jtriley has joined #openstack-kolla | 04:52 | |
| *** jtriley has quit IRC | 04:57 | |
| *** jtriley has joined #openstack-kolla | 04:58 | |
| *** japestinho has quit IRC | 05:02 | |
| *** jtriley has quit IRC | 05:03 | |
| *** salv-orl_ has joined #openstack-kolla | 05:03 | |
| *** salv-orlando has quit IRC | 05:06 | |
| *** imcsk8 has quit IRC | 05:06 | |
| *** hyakuhei has quit IRC | 05:11 | |
| *** itlinux has quit IRC | 05:12 | |
| *** itlinux_ has quit IRC | 05:12 | |
| *** skramaja has joined #openstack-kolla | 05:12 | |
| *** caowei_ has joined #openstack-kolla | 05:12 | |
| *** imcsk8 has joined #openstack-kolla | 05:14 | |
| *** caowei has quit IRC | 05:15 | |
| *** caowei_ is now known as caowei | 05:15 | |
| *** zhurong has quit IRC | 05:17 | |
| *** zhurong has joined #openstack-kolla | 05:21 | |
| *** caowei has quit IRC | 05:24 | |
| *** caowei has joined #openstack-kolla | 05:25 | |
| *** kornicameister has joined #openstack-kolla | 05:47 | |
| *** cuongnv has quit IRC | 05:50 | |
| kornicameister | duonghq: hi, have small question I asked in PM | 05:52 |
| *** jtriley has joined #openstack-kolla | 05:52 | |
| *** jtriley has quit IRC | 06:02 | |
| *** caowei_ has joined #openstack-kolla | 06:02 | |
| *** caowei has quit IRC | 06:05 | |
| *** caowei_ is now known as caowei | 06:05 | |
| *** salv-orl_ has quit IRC | 06:08 | |
| *** salv-orlando has joined #openstack-kolla | 06:09 | |
| *** krtaylor has quit IRC | 06:10 | |
| *** salv-orlando has quit IRC | 06:13 | |
| *** caowei_ has joined #openstack-kolla | 06:16 | |
| *** caowei has quit IRC | 06:19 | |
| *** krtaylor has joined #openstack-kolla | 06:23 | |
| *** caowei_ has quit IRC | 06:23 | |
| *** daidv_ has joined #openstack-kolla | 06:25 | |
| *** caowei has joined #openstack-kolla | 06:26 | |
| *** manheim has joined #openstack-kolla | 06:27 | |
| manheim | Hi | 06:28 |
| *** sudipto has joined #openstack-kolla | 06:33 | |
| *** sudipto has quit IRC | 06:33 | |
| *** sudipto has joined #openstack-kolla | 06:33 | |
| *** sudipto_ has joined #openstack-kolla | 06:33 | |
| *** manheim_ has joined #openstack-kolla | 06:33 | |
| *** manheim_ has quit IRC | 06:38 | |
| *** manheim_ has joined #openstack-kolla | 06:38 | |
| *** manheim_ has quit IRC | 06:42 | |
| *** spsurya has quit IRC | 06:47 | |
| *** david-lyle has quit IRC | 06:48 | |
| *** dklyle has joined #openstack-kolla | 06:48 | |
| *** dklyle has quit IRC | 06:49 | |
| *** david-lyle_ has joined #openstack-kolla | 06:49 | |
| *** jaosorior has joined #openstack-kolla | 06:49 | |
| *** jtriley has joined #openstack-kolla | 06:51 | |
| *** mewald has joined #openstack-kolla | 06:55 | |
| *** jtriley has quit IRC | 06:56 | |
| *** manheim_ has joined #openstack-kolla | 06:56 | |
| *** jtriley has joined #openstack-kolla | 06:57 | |
| *** manheim_ has quit IRC | 06:59 | |
| *** manheim__ has joined #openstack-kolla | 06:59 | |
| *** jtriley has quit IRC | 07:02 | |
| *** manheim__ has quit IRC | 07:02 | |
| *** manheim_ has joined #openstack-kolla | 07:03 | |
| *** manheim_ has quit IRC | 07:03 | |
| *** manheim_ has joined #openstack-kolla | 07:03 | |
| *** manheim_ has quit IRC | 07:10 | |
| *** sudipto_ has quit IRC | 07:16 | |
| *** sudipto has quit IRC | 07:16 | |
| *** manheim_ has joined #openstack-kolla | 07:16 | |
| *** sudipto has joined #openstack-kolla | 07:17 | |
| *** sudipto has quit IRC | 07:17 | |
| *** sudipto has joined #openstack-kolla | 07:17 | |
| *** sudipto_ has joined #openstack-kolla | 07:17 | |
| openstackgerrit | Yu Chaochao proposed openstack/kolla master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/474090 | 07:17 |
| openstackgerrit | Yu Chaochao proposed openstack/kolla master: Using assertIsNone() instead of assertEqual(None) https://review.openstack.org/474090 | 07:18 |
| *** salv-orlando has joined #openstack-kolla | 07:19 | |
| *** mewald has quit IRC | 07:22 | |
| *** XantiaNotDead has quit IRC | 07:25 | |
| *** jamesbenson has joined #openstack-kolla | 07:33 | |
| *** reidrac has joined #openstack-kolla | 07:35 | |
| *** jamesbenson has quit IRC | 07:37 | |
| *** mgoddard has joined #openstack-kolla | 07:37 | |
| *** sbhojpur has joined #openstack-kolla | 07:41 | |
| sbhojpur | need help with Kolla Magnum support | 07:42 |
| *** shardy has joined #openstack-kolla | 07:43 | |
| sbhojpur | Cluster is stuck in create in progress state and finally it fails | 07:43 |
| sbhojpur | Magnum conductor logs show | 07:44 |
| sbhojpur | loopingcall HTTPUnauthorized: ERROR: Authentication failed: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} 2017-06-13 05:27:35.689 7 ERROR oslo.service.loopingcall 2017-06-13 05:53:00.072 7 INFO magnum.service.periodic [req-bbe86451-7a47-43fb-ab0f-6d3de55e70e7 bae4f93c-cf38-4245-aec8-d2251e73218c_c763595944534f1ebf44a53b03764a10 - - - -] Sync up cluster wi | 07:44 |
| sbhojpur | CREATE_FAILED | 07:45 |
| *** egonzalez has joined #openstack-kolla | 07:46 | |
| *** karimb has joined #openstack-kolla | 07:50 | |
| *** jtriley has joined #openstack-kolla | 07:51 | |
| *** egonzalez has quit IRC | 07:55 | |
| *** jtriley has quit IRC | 07:56 | |
| *** jtriley has joined #openstack-kolla | 07:56 | |
| *** egonzalez has joined #openstack-kolla | 07:58 | |
| *** daidv_ has quit IRC | 07:59 | |
| *** jtriley has quit IRC | 08:01 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Bump min Ansible version to 2.2.0 https://review.openstack.org/452067 | 08:08 |
| *** blallau has joined #openstack-kolla | 08:10 | |
| *** tovin07_ has quit IRC | 08:14 | |
| *** tovin07_ has joined #openstack-kolla | 08:24 | |
| *** daidv has quit IRC | 08:27 | |
| *** manheim has quit IRC | 08:37 | |
| *** daidv has joined #openstack-kolla | 08:42 | |
| *** omenv has quit IRC | 08:44 | |
| *** jtriley has joined #openstack-kolla | 08:50 | |
| *** lpetrut has joined #openstack-kolla | 08:50 | |
| *** tvignaud has quit IRC | 08:54 | |
| *** jtriley has quit IRC | 08:55 | |
| *** jtriley has joined #openstack-kolla | 08:56 | |
| openstackgerrit | Dan Ardelean proposed openstack/kolla-ansible master: Add Hyper-V role https://review.openstack.org/455684 | 08:56 |
| *** manheim_ has quit IRC | 08:57 | |
| *** manheim has joined #openstack-kolla | 08:57 | |
| *** manheim_ has joined #openstack-kolla | 08:59 | |
| *** jtriley has quit IRC | 09:01 | |
| *** manheim has quit IRC | 09:01 | |
| *** manheim_ is now known as manheim | 09:01 | |
| *** tvignaud has joined #openstack-kolla | 09:07 | |
| *** manheim_ has joined #openstack-kolla | 09:14 | |
| *** manheim_ has quit IRC | 09:18 | |
| *** jmccarthy has joined #openstack-kolla | 09:38 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/kolla master: Add Apache packages to ironic-api container https://review.openstack.org/474141 | 09:38 |
| *** yingjun has quit IRC | 09:39 | |
| *** zhurong has quit IRC | 09:40 | |
| *** omenv has joined #openstack-kolla | 09:45 | |
| *** pbourke has joined #openstack-kolla | 09:46 | |
| *** omenvl has joined #openstack-kolla | 09:48 | |
| *** omenv has quit IRC | 09:50 | |
| *** jtriley has joined #openstack-kolla | 09:50 | |
| *** jtriley has quit IRC | 09:55 | |
| *** jtriley has joined #openstack-kolla | 09:55 | |
| *** sambetts|afk is now known as sambetts | 09:55 | |
| *** omenvl has quit IRC | 09:56 | |
| *** kbaegis has joined #openstack-kolla | 09:57 | |
| *** magicboiz has quit IRC | 09:57 | |
| *** jamesbenson has joined #openstack-kolla | 09:57 | |
| *** omenvl has joined #openstack-kolla | 09:58 | |
| *** jtriley has quit IRC | 10:00 | |
| *** omenvlad has joined #openstack-kolla | 10:01 | |
| *** kbaegis has left #openstack-kolla | 10:02 | |
| *** kbaegis has joined #openstack-kolla | 10:02 | |
| *** jamesbenson has quit IRC | 10:02 | |
| *** tovin07_ has quit IRC | 10:02 | |
| *** omenvlad has quit IRC | 10:02 | |
| *** omenvl has quit IRC | 10:02 | |
| *** cuongnv has joined #openstack-kolla | 10:08 | |
| *** ducttape_ has joined #openstack-kolla | 10:09 | |
| *** ducttape_ has quit IRC | 10:13 | |
| sbhojpur | need help with Kolla Magnum support Cluster is stuck in create in progress state and finally it fails Magnum conductor logs show loopingcall HTTPUnauthorized: ERROR: Authentication failed: {"error": {"message": "The request you have made requires authentication.", "code": 401, "title": "Unauthorized"}} 2017-06-13 05:27:35.689 7 ERROR oslo.service.loopingcall 2017-06-13 05:53:00.072 7 INFO magnum.service.periodic [req-bbe86 | 10:14 |
| reidrac | ls | 10:17 |
| *** omenv has joined #openstack-kolla | 10:19 | |
| *** sbhojpur has quit IRC | 10:31 | |
| *** magicboiz has joined #openstack-kolla | 10:33 | |
| kornicameister | hi all | 10:45 |
| kornicameister | does kolla supports concept of upgrading already running containers | 10:45 |
| kornicameister | with some graceful chery pick on top of everything | 10:45 |
| kornicameister | or sth ? | 10:45 |
| *** kbaegis1 has joined #openstack-kolla | 10:47 | |
| *** kbaegis has quit IRC | 10:48 | |
| *** jtriley has joined #openstack-kolla | 10:49 | |
| *** wagiel has joined #openstack-kolla | 10:50 | |
| *** kbaegis has joined #openstack-kolla | 10:51 | |
| *** kbaegis1 has quit IRC | 10:53 | |
| *** jtriley has quit IRC | 10:54 | |
| *** Akarki has joined #openstack-kolla | 10:54 | |
| Akarki | Cannot create images when CEPH is used | 10:55 |
| Akarki | Error finding address for http://172.16.109.183:9292/v2/images/df798428-099d-405e-ad86-4bebaa5bcaf0/file: Unable to establish connection to http://172.16.109.183:9292/v2/images/df798428-099d-405e-ad86-4bebaa5bcaf0/file | 10:55 |
| *** jtriley has joined #openstack-kolla | 10:55 | |
| *** salv-orlando has quit IRC | 10:55 | |
| egonzalez | kornicameister, not getting the question, mean applying direct patch into running container? | 10:57 |
| egonzalez | Akarki, is glance running? check logs at /var/lib/docker/volumes/kolla_logs/_data/glance/ | 10:58 |
| Akarki | ok let me check | 10:59 |
| *** jtriley has quit IRC | 11:00 | |
| Akarki | glance is running but when i tried to check CEPH status it shows haelth err | 11:00 |
| egonzalez | Akarki, many pgs ? | 11:00 |
| Akarki | i am trieng to use a loobback file as CEPH drive | 11:00 |
| Akarki | [6/14/2017 4:30 PM] Ashish Karki: docker exec ceph_mon ceph -s cluster 4752dfaf-e7fe-4629-9269-8797eac533c4 health HEALTH_ERR 584 pgs are stuck inactive for more than 300 seconds 584 pgs stuck inactive 584 pgs stuck unclean no osds monmap e1: 1 mons at {172.16.109.220=172.16.109.220:6789/0} election epoch 4, quorum 0 172.16.109.220 osdm | 11:00 |
| egonzalez | Akarki, there says no OSDs, loopback devices were implemented yesterday, so is only available in master branch | 11:02 |
| *** spsurya has joined #openstack-kolla | 11:02 | |
| spsurya | sup all | 11:02 |
| *** Akarki has quit IRC | 11:02 | |
| *** Illuminati has joined #openstack-kolla | 11:04 | |
| *** Illuminati is now known as Guest94222 | 11:04 | |
| *** Guest94222 has left #openstack-kolla | 11:05 | |
| *** Rex_clark has joined #openstack-kolla | 11:07 | |
| Rex_clark | Hi we are facing some issue with kolla magnum | 11:08 |
| *** Akarki has joined #openstack-kolla | 11:09 | |
| Rex_clark | creation of magnum cluster gets timeout | 11:09 |
| Rex_clark | Heat engine and magnum API shows authentication failed.. | 11:10 |
| egonzalez | Rex_clark, is the user with heat_stack_owner role? | 11:12 |
| Rex_clark | Yes it is | 11:12 |
| *** rwallner has joined #openstack-kolla | 11:12 | |
| Akarki | egonzalez, sorry I was accidentally locked out. You mean support for loopback file is not there yet | 11:14 |
| Rex_clark | Actually master instance gets created but master stack resource gets stuck atcreate in progress | 11:15 |
| *** Akarki has quit IRC | 11:18 | |
| *** Rex_clark has quit IRC | 11:20 | |
| egonzalez | Rex_clark, can you share heat-engine logs and heat event-list <stack> and heat resource-list <stack> | 11:20 |
| *** ghs39ghsggf has joined #openstack-kolla | 11:20 | |
| *** jmccarthy has quit IRC | 11:23 | |
| *** Rex_clark has joined #openstack-kolla | 11:23 | |
| egonzalez | Rex_clark, can you share heat-engine logs and heat event-list <stack> and heat resource-list <stack> | 11:23 |
| *** Rex_clark_ has joined #openstack-kolla | 11:25 | |
| Rex_clark_ | Heat engine logs | 11:27 |
| Rex_clark_ | INFO magnum.common.cert_manager.barbican_cert_manager [req-f48907df-98dc-4d3c-bc65-eb41ec8e3436 - - - - -] Storing certificate container 'Magnum-Conductor' in Barbican. 2017-06-13 05:27:35.689 7 ERROR oslo.service.loopingcall [-] Fixed interval looping call 'magnum.conductor.handlers.cluster_conductor.HeatPoller.poll_and_check' failed 2017-06-13 05:27:35.689 7 ERROR oslo.service.loopingcall Traceback (most recent call las | 11:27 |
| Rex_clark_ | sorry the logs were for magnum conductor | 11:28 |
| Rex_clark_ | Heat engine logs.. | 11:29 |
| Rex_clark_ | HEAT-ENG LOGS----------------------------------------------------------------------------------------------------- 2017-06-13 04:52:47.539 22 INFO heat.engine.stack [req-b589937b-e8c6-4779-9dc7-3fb3abbf7c47 f0f7ff12b07a444c857948d6708106e8 c763595944534f1ebf44a53b03764a10 - - -] Stack CREATE COMPLETE (cluster_1-yynhf4i5vztw-kube_masters-i6eud4bxjvtb-0-lxketu3logql-api_address_switch-q2dwpx5kkimo): Stack CREATE completed | 11:29 |
| *** jaosorior has quit IRC | 11:29 | |
| *** wagiel has quit IRC | 11:29 | |
| Rex_clark_ | 2017-06-13 04:52:47.755 20 INFO heat.engine.resource [req-b589937b-e8c6-4779-9dc7-3fb3abbf7c47 f0f7ff12b07a444c857948d6708106e8 c763595944534f1ebf44a53b03764a10 - - -] creating SoftwareConfig "write_heat_params" Stack "cluster_1-yynhf4i5vztw-kube_masters-i6eud4bxjvtb-0-lxketu3logql" [9eb4c7c9-bff9-44b2-a107-20df94243239] 2017-06-13 04:52:47.983 19 INFO heat.engine.resource [req-b589937b-e8c6-4779-9dc7-3fb3abbf7c47 f0f7ff1 | 11:29 |
| *** Rex_clark has quit IRC | 11:29 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Support for increasing thread cache size in TCMalloc. https://review.openstack.org/468284 | 11:31 |
| kornicameister | egonzalez: I mean, sth like we cut new release and want to flawlessly intergrate that into current deployment | 11:31 |
| kornicameister | egonzalez: have version 0.0.1 and would like to have 0.0.2 for components x-{something} | 11:32 |
| kornicameister | egonzalez: regardless of the scale of x-{deployment} | 11:32 |
| kornicameister | egonzalez: for now I collected some blueprint from launchpad about graceful-upgrade and some wiki page that tells me that it is theoretically possible, but it mentions only 'openstack_version' | 11:33 |
| kornicameister | egonzalez: so it's not possible to upgrade only subset of components ? | 11:33 |
| kornicameister | egonzalez: can it be controlled through...maybe specific /etc/kolla/globals.yaml ? | 11:33 |
| kornicameister | egonzalez: also while we're on it, there is 2nd question about scaling up/scalling down | 11:34 |
| *** Lex_Luther has joined #openstack-kolla | 11:34 | |
| Rex_clark_ | HELP | 11:35 |
| *** mewald has joined #openstack-kolla | 11:35 | |
| egonzalez | Rex_clark_, please share logs in paste.openstack.org | 11:35 |
| egonzalez | kornicameister, each service and each container have its own tag avaiable to customize https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/senlin/defaults/main.yml#L39 | 11:35 |
| egonzalez | example of senlin, you can set senlin_tag for all senlin images or senlin_api_tag if only want to change senlin api image | 11:36 |
| *** wahnsinn has joined #openstack-kolla | 11:36 | |
| egonzalez | well, senlin_tag was introduced in master, but senlin_api_tag is present before | 11:37 |
| *** rwallner has quit IRC | 11:37 | |
| *** mewald1 has joined #openstack-kolla | 11:37 | |
| *** rhallisey has joined #openstack-kolla | 11:38 | |
| kornicameister | ok, I get that | 11:38 |
| kornicameister | and tag evaluates to some upstream tag | 11:39 |
| kornicameister | a.k.a. github.com/openstack/senlin-api | 11:39 |
| kornicameister | ? | 11:39 |
| *** mewald has quit IRC | 11:39 | |
| egonzalez | kornicameister, nope, that tag evaluates to a kolla image tag | 11:39 |
| kornicameister | and how is image correlated to the upstream repo ? | 11:40 |
| kornicameister | egonzalez: there has to be sth | 11:41 |
| kornicameister | egonzalez: otherwise each image version would build for master | 11:41 |
| *** Rex_clark_ has quit IRC | 11:42 | |
| *** rwallner_ has joined #openstack-kolla | 11:42 | |
| egonzalez | kornicameister, using source images guess | 11:42 |
| egonzalez | kornicameister, if are binary all that depends on the repositories, in case of centos rdo ocata repo as example | 11:42 |
| kornicameister | egonzalez: getting dark here...ah...you answered the question I mean to ask | 11:43 |
| kornicameister | s/mean/meant/g | 11:43 |
| *** sudipto has quit IRC | 11:43 | |
| *** sudipto_ has quit IRC | 11:43 | |
| kornicameister | but still that's rather tricky in case of binary images. | 11:43 |
| egonzalez | kornicameister, if using source, kolla-build use this config file with service versions https://github.com/openstack/kolla/blob/stable/ocata/kolla/common/config.py#L491 | 11:43 |
| *** Lex_Luther has quit IRC | 11:44 | |
| *** rwallner_ has quit IRC | 11:44 | |
| kornicameister | egonzalez: I mean repository can be upgraded with newer version, right | 11:44 |
| kornicameister | ? | 11:44 |
| *** rwallner has joined #openstack-kolla | 11:44 | |
| egonzalez | kornicameister, well, if you have a local mirror yes, otherwise depends on packagers, rdo as example | 11:45 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla-ansible master: Add /lib/modules to neutron_openvswitch_agent https://review.openstack.org/474167 | 11:46 |
| *** Rex_clark has joined #openstack-kolla | 11:46 | |
| Rex_clark | egonzalez, please find the logs at http://paste.openstack.org/show/612531/ | 11:46 |
| Rex_clark | egonzalez, heat engine and Magnum conductor logs are there, please let me know if you need any other logs | 11:47 |
| *** omenv has quit IRC | 11:47 | |
| *** omenv has joined #openstack-kolla | 11:47 | |
| kornicameister | egonzalez: ok, I will try to think this through, if I have any questions I'll try to find you on friday | 11:47 |
| kornicameister | egonzalez: but what about scalling up and scalling down | 11:47 |
| kornicameister | ? | 11:47 |
| kornicameister | I couldn't find anything about that | 11:48 |
| kornicameister | perhaps bad query in google or sth | 11:48 |
| *** Nomen has joined #openstack-kolla | 11:48 | |
| egonzalez | kornicameister, just add hosts in inventory and kolla-ansible deploy, you can also limit scope with --limit or --tags | 11:48 |
| *** jtriley has joined #openstack-kolla | 11:49 | |
| kornicameister | ok, so pure ansible here | 11:49 |
| egonzalez | kornicameister, take care of tags because some services depends on other configs, ie: API will need to reconfigure haproxy to add its backend | 11:49 |
| kornicameister | that's neat | 11:49 |
| kornicameister | egonzalez: something more ? just brief idea what you meant here | 11:50 |
| egonzalez | kornicameister, nope, i use to deploy the service with tags <service> | 11:50 |
| egonzalez | and after seing the service fine, reconfigure haproxy | 11:50 |
| kornicameister | ok, I think I get that | 11:51 |
| kornicameister | egonzalez: can I ask a favor, I am trying to note all that down | 11:52 |
| kornicameister | http://paste.openstack.org/show/u4JnfSsKOITtBwiifmlJ/ | 11:52 |
| kornicameister | is is something close to what you have in your own mind about upgrading ? | 11:52 |
| kornicameister | or did I mess it up ? | 11:52 |
| egonzalez | kornicameister, thats overall idea, just add this doc https://github.com/openstack/kolla-ansible/blob/master/doc/operating-kolla.rst | 11:53 |
| *** jtriley has quit IRC | 11:54 | |
| kornicameister | oh my lord...that investigating tasks are pain in the *** :D | 11:54 |
| kornicameister | ok, thx for that | 11:54 |
| *** jtriley has joined #openstack-kolla | 11:54 | |
| kornicameister | btw: does kolla-kubernetes operate in similar way regarding upgrades/scalling or does it delegate everything to kubernetes | 11:55 |
| kornicameister | ? | 11:55 |
| egonzalez | Rex_clark, are you using barbican? | 11:55 |
| Rex_clark | yes | 11:57 |
| *** Nomen has quit IRC | 11:57 | |
| Rex_clark | also we had to manually add heat_stack_owner option to admin project otherwise it showed heat_stack_owner not found in magnum stack creation | 11:59 |
| egonzalez | kornicameister, afik, kolla-k8s has not implemented release upgrades, probably minor version upgrades 1.0.n+1 will behave on k8s features | 11:59 |
| egonzalez | Rex_clark, yep, thats was fixed in master iirc | 11:59 |
| kornicameister | egonzalez: sounds more and more like kolla-k8s is not sth that should be considered for any production usage (that's just beta, alpha ?) | 11:59 |
| egonzalez | Rex_clark, check barbican logs if there is any error or more info | 11:59 |
| Rex_clark | ok will do that now but may i know when this fix was released | 12:00 |
| *** jtriley has quit IRC | 12:01 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Add heat_stack_owner role to admin project https://review.openstack.org/474173 | 12:01 |
| *** caowei has quit IRC | 12:01 | |
| egonzalez | Rex_clark, ^^ thats the backport, only add role to admin user | 12:02 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix service_type of Octavia https://review.openstack.org/474174 | 12:03 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Update loopback instructions in cinder docs https://review.openstack.org/474176 | 12:05 |
| *** omenv has quit IRC | 12:06 | |
| *** omenv has joined #openstack-kolla | 12:06 | |
| *** omenv has quit IRC | 12:06 | |
| *** ducttape_ has joined #openstack-kolla | 12:07 | |
| *** ducttape_ has quit IRC | 12:08 | |
| *** manheim has quit IRC | 12:09 | |
| Rex_clark | egonzalev logs for barbican service : http://paste.openstack.org/show/612538/ | 12:09 |
| egonzalez | Rex_clark, there is something wrong there with barbican database | 12:10 |
| *** manheim has joined #openstack-kolla | 12:11 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible stable/ocata: Fix wrong variable used in trove role https://review.openstack.org/474179 | 12:13 |
| Rex_clark | Actually without barbican service we are facing the same issue | 12:15 |
| *** rwallner has quit IRC | 12:20 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Update loopback instructions in cinder docs https://review.openstack.org/474176 | 12:20 |
| *** kornicameister has quit IRC | 12:22 | |
| *** mewald1 has quit IRC | 12:25 | |
| *** hrw has quit IRC | 12:28 | |
| *** hrw has joined #openstack-kolla | 12:30 | |
| *** schwicht has joined #openstack-kolla | 12:30 | |
| *** omenv has joined #openstack-kolla | 12:33 | |
| *** schwicht has quit IRC | 12:37 | |
| rwellum | sbezverk: hey trying ubuntu for the first time and iscsid is in crashloopbackoff - is this a known issue? | 12:39 |
| *** schwicht has joined #openstack-kolla | 12:40 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix cron restarting issue https://review.openstack.org/473863 | 12:45 |
| *** jtriley has joined #openstack-kolla | 12:49 | |
| *** kbaegis has quit IRC | 12:50 | |
| *** sudipto_ has joined #openstack-kolla | 12:50 | |
| *** sudipto has joined #openstack-kolla | 12:50 | |
| *** sbezverk has quit IRC | 12:53 | |
| *** eaguilar has joined #openstack-kolla | 12:53 | |
| *** sbezverk has joined #openstack-kolla | 12:54 | |
| *** kbaegis has joined #openstack-kolla | 12:54 | |
| *** rwallner has joined #openstack-kolla | 12:55 | |
| *** ddyer has joined #openstack-kolla | 12:56 | |
| *** jtriley has quit IRC | 12:56 | |
| *** schwicht has quit IRC | 12:58 | |
| *** goldyfruit has joined #openstack-kolla | 13:02 | |
| *** eanylin has quit IRC | 13:03 | |
| hawi | hi there: question - what is the keyword to enable powerdns as backend for designate in kolla-ansible config | 13:04 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Add Skydive startup guide documentation https://review.openstack.org/473853 | 13:07 |
| *** manheim has quit IRC | 13:08 | |
| egonzalez | hawi, powerdns is not implemented in kolla for now | 13:08 |
| *** schwicht has joined #openstack-kolla | 13:10 | |
| *** manheim has joined #openstack-kolla | 13:10 | |
| *** magicboiz has quit IRC | 13:12 | |
| *** omenv has quit IRC | 13:14 | |
| *** lucasxu has joined #openstack-kolla | 13:15 | |
| *** openstackgerrit has quit IRC | 13:18 | |
| *** dave-mccowan has joined #openstack-kolla | 13:19 | |
| *** ghs39ghsggf has quit IRC | 13:20 | |
| *** wahnsinn has quit IRC | 13:24 | |
| *** openstackgerrit has joined #openstack-kolla | 13:25 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-ansible master: Add instructions to shutdown mariadb cluster https://review.openstack.org/466881 | 13:25 |
| *** dmsimard has joined #openstack-kolla | 13:27 | |
| tvignaud | mandre: new cinder-volume's Dockerfile is doing "+RUN yum-config-manager --add-repo http://people.redhat.com/mbaldess/rpms/container-repo/pacemaker-bundle.repo && yum install ..." :-( | 13:27 |
| *** goldyfruit has quit IRC | 13:27 | |
| tvignaud | that's not great for downstream who want to use distro pkgs... | 13:27 |
| sbezverk | rwellum : did you get my reply? by some reason my irc client got disconnected | 13:30 |
| rwellum | sbezverk: no I did not | 13:30 |
| sbezverk | rwellum : you need to stop iscsid and iscsid.socket on your host, then kill iscsid pod it will gets restarted properly | 13:30 |
| hawi | egonzalez: thanks! | 13:30 |
| rwellum | For the second one did yo mean iscsid.service sbezverk ? | 13:34 |
| sbezverk | rwellum nope iscsd.socket service | 13:34 |
| *** skramaja has quit IRC | 13:37 | |
| rwellum | That one was not running - but .service was | 13:37 |
| *** pradk has quit IRC | 13:38 | |
| kbaegis | Is there a typo in the cron container? | 13:39 |
| kbaegis | It looks for mongdb.conf instead of mongodb.conf? | 13:39 |
| sbezverk | rwellum : hm, must be ubuntu specificity | 13:39 |
| kbaegis | Has that been fixed already? | 13:39 |
| *** eanylin has joined #openstack-kolla | 13:39 | |
| sbezverk | on centos it is iscsdi.socket | 13:39 |
| kbaegis | Had to symlink my files to fix it | 13:39 |
| openstackgerrit | Merged openstack/kolla-ansible master: Add /lib/modules to neutron_openvswitch_agent https://review.openstack.org/474167 | 13:40 |
| rwellum | sbezverk: I'll play with it a bit more. | 13:40 |
| egonzalez | kbaegis, just was fixed one hour ago ;) | 13:41 |
| kbaegis | Dang, beat me to it :) | 13:42 |
| egonzalez | kbaegis, ups, was fixed other cron issue, this one is the fix https://review.openstack.org/#/c/471316/ | 13:42 |
| *** zhurong has joined #openstack-kolla | 13:42 | |
| *** jtriley has joined #openstack-kolla | 13:44 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Deploy etcd in compute nodes when kuryr https://review.openstack.org/473857 | 13:45 |
| *** spsurya has quit IRC | 13:50 | |
| hawi | egonzalez: is there something more needed for enabling designate - some neutron config etc? | 13:51 |
| hawi | in addition to dns_interface and backend and enable_designate? | 13:51 |
| egonzalez | hawi, enable_designate and configure a couple of values https://github.com/openstack/kolla-ansible/blob/master/doc/designate-guide.rst | 13:51 |
| egonzalez | hawi, then create a zone and reconfigure designate-sink with the zone id | 13:52 |
| *** omenv has joined #openstack-kolla | 13:52 | |
| egonzalez | hawi, neutron config is done while enabling designate | 13:52 |
| hawi | egonzalez: thanks, i had designate enabled back some time, i had feeling that nothing more is needed. ok, thanks | 13:53 |
| hawi | ok | 13:53 |
| *** klindgren_ has joined #openstack-kolla | 13:54 | |
| *** iniazi_ has joined #openstack-kolla | 13:55 | |
| *** klindgren has quit IRC | 13:55 | |
| *** iniazi has quit IRC | 13:58 | |
| *** omenvl has joined #openstack-kolla | 13:58 | |
| *** omenv has quit IRC | 14:00 | |
| dave-mccowan | sbezverk thanks for the help yesterday. i have a new patch up. https://review.openstack.org/471992 but, the gate jobs are failing (several containers never finish coming up). i don't know my way around enough yet to debug why they are stuck. any tips? | 14:02 |
| *** rwallner has quit IRC | 14:03 | |
| sbezverk | dave-mccowan : I will check the gate results in a bit, I have a fire like situation to deal with .. | 14:05 |
| kbaegis | will reconfigure remove services that I've since disabled, or simply reshuffle the config files and restart the containers? | 14:06 |
| *** masber has quit IRC | 14:06 | |
| *** masber has joined #openstack-kolla | 14:07 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/kolla master: Add Apache packages to ironic-api container https://review.openstack.org/474141 | 14:08 |
| egonzalez | kbaegis, will remove any reference in config files in other services about the removed service, but containers or config files will remain there, also endpoints and databases | 14:09 |
| *** dmsimard has quit IRC | 14:10 | |
| inc0 | good morning | 14:12 |
| *** omenvl has quit IRC | 14:12 | |
| kbaegis | egonzalez: So I have to stop the containers manually? Run deploy? | 14:12 |
| kbaegis | inc0: morning | 14:12 |
| *** omenvl has joined #openstack-kolla | 14:14 | |
| *** omenvlad has joined #openstack-kolla | 14:16 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 14:17 |
| *** rwallner has joined #openstack-kolla | 14:18 | |
| *** omenvl has quit IRC | 14:18 | |
| zhubingbing__ | hi guys | 14:18 |
| inc0 | good morning zhubingbing__ | 14:19 |
| zhubingbing__ | hi inc0 | 14:20 |
| manheim | hi | 14:21 |
| *** manheim has quit IRC | 14:21 | |
| *** manheim has joined #openstack-kolla | 14:22 | |
| *** pradk has joined #openstack-kolla | 14:22 | |
| *** bmace has quit IRC | 14:26 | |
| *** manheim has quit IRC | 14:26 | |
| *** bmace has joined #openstack-kolla | 14:26 | |
| *** jaosorior has joined #openstack-kolla | 14:28 | |
| *** rwallner has quit IRC | 14:28 | |
| *** srwilkers has joined #openstack-kolla | 14:29 | |
| *** jmccarthy has joined #openstack-kolla | 14:35 | |
| jmccarthy | Hi core reviewers, any free to take a look at this here ? https://review.openstack.org/#/c/453738 | 14:36 |
| *** hongbin has joined #openstack-kolla | 14:37 | |
| sbezverk | dave-mccowan http://logs.openstack.org/92/471992/4/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/c513afb/logs/pods/kolla-cinder-api-2238004965-cxsfj.txt | 14:38 |
| sbezverk | pod fails to mount config map with your certificate | 14:38 |
| *** janki has quit IRC | 14:38 | |
| sbezverk | make sure naming is right and this config map gets created in the gate script | 14:39 |
| dave-mccowan | sbezverk got it. thanks. | 14:42 |
| jmccarthy | And maybe https://review.openstack.org/#/c/469511 and https://review.openstack.org/#/c/464675/ ? | 14:43 |
| *** eaguilar has quit IRC | 14:43 | |
| dave-mccowan | sbezverk if i'm running locally how would i find that output (logs/pods/cinder-api...) | 14:44 |
| sbezverk | dave-mccowan : you run kubectl describe pod {pod name} -n kolla | 14:47 |
| *** zhurong has quit IRC | 14:49 | |
| *** ansmith has joined #openstack-kolla | 14:51 | |
| *** jamesbenson has joined #openstack-kolla | 14:52 | |
| jamesbenson | morning all | 14:52 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 14:53 |
| egonzalez | pbourke, added another comment in the outward change | 14:58 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 14:59 |
| pbourke | egonzalez: thanks | 15:00 |
| *** omenvlad has quit IRC | 15:01 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 15:03 |
| *** Rex_clark has quit IRC | 15:03 | |
| *** k_mouza has joined #openstack-kolla | 15:04 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable https://review.openstack.org/453735 | 15:05 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla master: Make rabbitmq log dir configurable https://review.openstack.org/453735 | 15:06 |
| *** manheim has joined #openstack-kolla | 15:06 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 15:06 |
| pbourke | egonzalez: any idea why this seems to be stuck? https://review.openstack.org/#/c/437920/ | 15:07 |
| *** manheim has quit IRC | 15:07 | |
| egonzalez | pbourke, depends on https://review.openstack.org/#/c/465515/ | 15:07 |
| pbourke | egonzalez: oh nevermind, noticed there's a dependson | 15:07 |
| pbourke | :) | 15:07 |
| pbourke | thanks | 15:07 |
| *** duonghq has quit IRC | 15:10 | |
| *** duonghq has joined #openstack-kolla | 15:10 | |
| *** duonghq_ has joined #openstack-kolla | 15:11 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Change Solum keystone v2 to v3 https://review.openstack.org/472931 | 15:11 |
| hawi | egonzalez: is there some known way to have openstack client (or designate client) working properly? | 15:13 |
| inc0 | does it work improperly? | 15:13 |
| egonzalez | hawi, pip install python-designateclient==<RELEASE_VERSION> | 15:14 |
| egonzalez | hawi, if using master, designate client is not available since only support designate v2(openstack client only) | 15:15 |
| hawi | what is proper version for ocata | 15:16 |
| *** salv-orlando has joined #openstack-kolla | 15:16 | |
| egonzalez | hawi, 2.6.* | 15:18 |
| hawi | ERROR: Unable to establish connection to http://10.66.0.254:9001/domains: HTTPConnectionPool(host='10.66.0.254', port=9001): Max retries exceeded with url: /domains (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7fa271041b50>: Failed to establish a new connection: [Errno 111] Connection refused',)) | 15:18 |
| hawi | hmmm | 15:18 |
| inc0 | haproxy is up and healthy? | 15:19 |
| hawi | yes | 15:20 |
| inc0 | hmm ... ss -plant? | 15:20 |
| hawi | LISTEN 0 128 10.79.2.8:9001 *:* | 15:20 |
| hawi | it's from one controller only | 15:21 |
| inc0 | there you go, address is wrong | 15:21 |
| inc0 | looks like keystone endpoint has wrong address | 15:21 |
| *** manheim has joined #openstack-kolla | 15:21 | |
| egonzalez | yep, http://10.66.0.254:9001 should be the haproxy vip | 15:21 |
| inc0 | haproxy vip as in keepalived terminated | 15:22 |
| hawi | kolla_internal_vip_address: "10.79.3.253" kolla_external_vip_address: "10.66.0.254" kolla_external_vip_interface: "bond0.7" dns_interface: "bond0.7" | 15:23 |
| reidrac | keystone deployment is failing, in "keystone : Creating default user role"; with "Could not determine a suitable URL for the plugin"; any suggestions? | 15:23 |
| egonzalez | hrm, using external vip, is ssl properly configured? | 15:24 |
| inc0 | right, so check on network nodes if haproxy+keepalived terminates .254 properly | 15:24 |
| inc0 | egonzalez: you don't need ssl for external | 15:24 |
| hawi | kolla_enable_tls_external: "no" | 15:24 |
| hawi | it's ok, as i had no proper cert to put there | 15:25 |
| *** ArchiFleKs has quit IRC | 15:26 | |
| *** blallau has quit IRC | 15:30 | |
| egonzalez | hawi, any error in designate-api logs? container up and running? | 15:31 |
| *** mewald has joined #openstack-kolla | 15:32 | |
| mewald | Hi guys, any feedback on https://bugs.launchpad.net/kolla-ansible/+bug/1695875 would be awesome since I would like to patch this within the week. | 15:33 |
| openstack | Launchpad bug 1695875 in kolla-ansible pike "Ceilometer-api not starting: Name duplicates previous WSGI daemon definition" [High,Triaged] | 15:33 |
| inc0 | mewald: hmm...to me (very first glance) it looks like that apt installs some configs for ceilometer api | 15:34 |
| inc0 | try to do docker run -it ceilometer-api ls /etc/apache2 | 15:34 |
| inc0 | as in, before kolla copies configs, see if configs exist on their own | 15:35 |
| mewald | inc0: yes that's right, in the end we get the default-000.conf and ceilometer-api.conf with the same content | 15:35 |
| *** bmace has quit IRC | 15:35 | |
| *** ArchiFleKs has joined #openstack-kolla | 15:35 | |
| *** dmsimard has joined #openstack-kolla | 15:35 | |
| inc0 | I'd remove default config in images | 15:35 |
| inc0 | on build time | 15:36 |
| inc0 | so RUN rm default-000.conf | 15:36 |
| mewald | alright, that's what egonzalez suggested, too but I wanted at least a 2nd opinion before working on the patch | 15:37 |
| inc0 | egonzalez usually gets it right;) but there you go | 15:37 |
| mewald | no offense, egonzalez ;) | 15:39 |
| *** spsurya has joined #openstack-kolla | 15:39 | |
| mewald | inc0: actually, default-000.conf is the one that comes from kolla-ansible, so I will "RUN rm /etc/apache2/sites-available/ceilometer-api.conf" | 15:40 |
| inc0 | right, you know what I mean | 15:40 |
| mewald | I am not familiar at all with the kolla-kubernetes deployment but they use the same images, right? Are we sure it's going to break their deployment? | 15:43 |
| *** ArchiFleKs has quit IRC | 15:46 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Add a new 'outward' rabbitmq instance https://review.openstack.org/453738 | 15:47 |
| *** omenvlad has joined #openstack-kolla | 15:48 | |
| *** bmace has joined #openstack-kolla | 15:48 | |
| inc0 | mewald: shouldn't | 15:48 |
| *** omenvl has joined #openstack-kolla | 15:49 | |
| *** omenvl has quit IRC | 15:51 | |
| *** omenvlad has quit IRC | 15:52 | |
| jamesbenson | morning inc0 | 15:53 |
| manheim | i'll give my kolla prez in like 5 mins, will share the recording :D | 15:54 |
| zhubingbing__ | meeting in openstack-meeting-4 ? | 15:55 |
| inc0 | manheim: good luck! | 15:55 |
| inc0 | and yes zhubingbing__ in 5min | 15:55 |
| zhubingbing__ | ;) | 15:55 |
| zhubingbing__ | good job | 15:55 |
| *** ArchiFleKs has joined #openstack-kolla | 15:57 | |
| *** itlinux has joined #openstack-kolla | 16:00 | |
| *** itlinux_ has joined #openstack-kolla | 16:00 | |
| k_mouza | hello all! Just double checking on something. I want to make a custom horizon image with kolla and add some extra css files etc. in it. The way to do that is to tar everything up and add an "ADD <tar file>" option in a j2 file, right? | 16:01 |
| *** jascott1 has joined #openstack-kolla | 16:02 | |
| *** kbaegis has quit IRC | 16:02 | |
| *** vhosakot has joined #openstack-kolla | 16:13 | |
| *** ducttape_ has joined #openstack-kolla | 16:21 | |
| kfox1111 | morning. | 16:23 |
| *** mewald has quit IRC | 16:23 | |
| kfox1111 | https://github.com/kubernetes-incubator/external-storage/tree/master/local-volume looks quite interesting. | 16:23 |
| *** mewald has joined #openstack-kolla | 16:24 | |
| *** mewald has quit IRC | 16:25 | |
| *** mewald has joined #openstack-kolla | 16:26 | |
| *** mewald has joined #openstack-kolla | 16:26 | |
| *** mewald has quit IRC | 16:26 | |
| *** masber has quit IRC | 16:27 | |
| *** mewald1 has joined #openstack-kolla | 16:28 | |
| *** masber has joined #openstack-kolla | 16:28 | |
| *** david-lyle_ is now known as david-lyle | 16:35 | |
| openstackgerrit | Merged openstack/kolla master: Remove bootstrap in etcd https://review.openstack.org/473858 | 16:35 |
| *** reidrac has quit IRC | 16:35 | |
| *** mewald1 has quit IRC | 16:36 | |
| openstackgerrit | Merged openstack/kolla master: Fix kuryr docker images https://review.openstack.org/473490 | 16:39 |
| *** mewald has joined #openstack-kolla | 16:40 | |
| *** lucasxu has quit IRC | 16:43 | |
| jamesbenson | I'm getting an odd error with the elasticsearch : Get container facts; mentioning: File \"/usr/local/lib/python2.7/dist-packages/docker/client.py\", line 124, in _retrieve_server_version\r\n 'Error while fetching server API version: {0}'.format(e)\r\ndocker.errors.DockerException: Error while fetching server API version: Timeout value connect was Timeout(connect=60, read=60, total=None), but it must be an int, float or None | 16:43 |
| jamesbenson | .\r\n", "msg": "MODULE FAILURE", "rc": 0} | 16:43 |
| *** daidv_ has joined #openstack-kolla | 16:44 | |
| *** mgoddard has quit IRC | 16:44 | |
| *** sambetts is now known as sambetts|Afk | 16:44 | |
| jamesbenson | This is during the prechecks | 16:44 |
| *** sambetts|Afk is now known as sambetts|afk | 16:44 | |
| spsurya | kfox1111: morning | 16:49 |
| *** k_mouza has quit IRC | 16:51 | |
| *** jascott1 has quit IRC | 16:51 | |
| *** jascott1 has joined #openstack-kolla | 16:51 | |
| *** sudipto has quit IRC | 16:51 | |
| *** sudipto_ has quit IRC | 16:51 | |
| *** eaguilar has joined #openstack-kolla | 16:52 | |
| *** unicell has quit IRC | 16:52 | |
| kfox1111 | spsurya: good morning. :) | 16:53 |
| *** egonzalez has quit IRC | 16:55 | |
| *** jascott1 has quit IRC | 16:56 | |
| sbezverk | kfox1111: it is good to see that k8s community making storage more flexible! | 16:56 |
| kfox1111 | +1 | 16:57 |
| kfox1111 | glad to see the external storage stuff starting to fly too. | 16:57 |
| kfox1111 | trying to build it all into the main controller container is a real pain. :/ | 16:57 |
| karimb | hello buddies | 16:57 |
| karimb | do you run neutron agents in with -net= host always ? | 16:58 |
| inc0 | karimb: in kolla-ansible we run everything with -net=host | 16:58 |
| karimb | how come ? | 16:58 |
| inc0 | much easier to handle and by the time we made it docker networking was night awful | 16:59 |
| inc0 | not sure if it's any better now | 16:59 |
| karimb | what i d like is to bind a nic of a host to br-ex running in my neutron agents container | 16:59 |
| * karimb has no clue whether it can be done ( without -net host ) | 17:00 | |
| inc0 | well, you have net host:) | 17:00 |
| inc0 | so it should be straightforward | 17:00 |
| *** jascott1 has joined #openstack-kolla | 17:00 | |
| kfox1111 | neutron shoudl pretty much always be run net=host. | 17:00 |
| inc0 | in kolla-k8s afair neutron agents also run net=host | 17:00 |
| kfox1111 | the agents I mean. | 17:00 |
| inc0 | yeah that would be accurate | 17:00 |
| inc0 | all sorts of funny things can happen without it | 17:01 |
| kfox1111 | neutron's all about getting access to the network stuff. you don't want docker's simple abstraction in the way. | 17:01 |
| vhosakot | doubt OpenStack works without -net=host. karimb if you don't want -net=host and networking between containers, you could look into k8s+CNI route :) | 17:01 |
| inc0 | most of services won't care | 17:01 |
| kfox1111 | most of openstack works without net=host. | 17:01 |
| *** rstarmer has joined #openstack-kolla | 17:01 | |
| karimb | vhosakot actually it almost works in my case | 17:01 |
| zhubingbing__ | good night guys | 17:02 |
| zhubingbing__ | bye | 17:02 |
| kfox1111 | kolla-kubernetes only has a few things net=host. | 17:02 |
| inc0 | for kolla-ansible lack of net host would be problematic because we need IPs of things | 17:02 |
| karimb | i mean it works without it, but now i want to reach vms with floating ips | 17:02 |
| kfox1111 | the rest can float around/self heal pretty easily. | 17:02 |
| jamesbenson | inc0 any idea on why prechecks are failing on: Error while fetching server API version: {0}'.format(e)\r\ndocker.errors.DockerException: Error while fetching server API version: Timeout value connect was Timeout(connect=60, read=60, total=None), but it must be an int, float or None. | 17:02 |
| *** eaguilar has quit IRC | 17:02 | |
| kfox1111 | yeah. the lack of sdn in kolla-ansible means things have to be pretty static. | 17:02 |
| inc0 | jamesbenson: first time I see it | 17:02 |
| kfox1111 | I think thats one of kolla-kubernetes' major features. | 17:03 |
| jamesbenson | all prechecks that need to get container facts | 17:03 |
| inc0 | that's why k8s even works;) | 17:03 |
| *** salv-orl_ has joined #openstack-kolla | 17:03 | |
| vhosakot | karimb: neutron and dhcp-agent may be angry, very angry if you don't do -net=host :) | 17:03 |
| karimb | vhosakot it surprisingly works for me | 17:03 |
| vhosakot | karimb: in kolla-ansible and k8s+CNI? | 17:04 |
| karimb | my own custom deployment | 17:04 |
| karimb | only docker at the moment | 17:04 |
| karimb | but i dont see too much differences with using kubernetes honestly | 17:04 |
| karimb | in both cases, i ll get an ip from the "sdn" | 17:04 |
| inc0 | karimb: vxlan+ovs+dhcp works without net=host? | 17:04 |
| * karimb probably missing something ;) | 17:05 | |
| inc0 | you need to share namespace between these | 17:05 |
| karimb | inc0 well that works | 17:05 |
| karimb | but yeah you re right, | 17:05 |
| karimb | my container contains all neutron agents together with the ovs agent | 17:05 |
| *** salv-orlando has quit IRC | 17:05 | |
| karimb | so that s why it works | 17:05 |
| inc0 | interesting, well, sharing /run might be enough for net namespace sharing | 17:06 |
| karimb | but my point is rather than my other ovs agent connect together using snd ips | 17:06 |
| *** mewald has quit IRC | 17:06 | |
| karimb | sdn | 17:06 |
| inc0 | ah yeah, if you have fat containers as we call it | 17:07 |
| inc0 | so more than one agent in same container | 17:07 |
| inc0 | that will work | 17:07 |
| *** jmccarthy has left #openstack-kolla | 17:07 | |
| *** rstarmer has quit IRC | 17:07 | |
| inc0 | but we don't do that, for us is 1pid per container | 17:07 |
| inc0 | with floating ip container has to meddle with iptables | 17:08 |
| inc0 | which is kernel module | 17:08 |
| inc0 | so that might be issue you're experiencing | 17:08 |
| inc0 | and I have no idea how that would work | 17:08 |
| inc0 | since docker does iptables itself | 17:08 |
| *** manheim has quit IRC | 17:08 | |
| *** mewald has joined #openstack-kolla | 17:09 | |
| inc0 | in any case, without net=host...you're asking for it. Just sayin' | 17:09 |
| *** lpetrut has quit IRC | 17:10 | |
| *** rwallner has joined #openstack-kolla | 17:12 | |
| jamesbenson | inc0: I've got the stable ocata branch on kolla and kolla-ansible, ansible 2.3.0.0... bootstrap works perfect. prechecks fail consistently on container facts. | 17:12 |
| jamesbenson | inc0: http://paste.openstack.org/show/612575/ | 17:13 |
| *** rwallner_ has joined #openstack-kolla | 17:13 | |
| jamesbenson | I've disabled services, but this happens when it tries to get the container facts regardless... not sure how to debug this. any pointers? | 17:14 |
| *** manheim has joined #openstack-kolla | 17:15 | |
| inc0 | what version of Docker? | 17:16 |
| inc0 | same as bootstrap? | 17:16 |
| jamesbenson | Docker version 1.12.6, build 78d1802 | 17:16 |
| *** rwallner has quit IRC | 17:16 | |
| inc0 | hmm | 17:16 |
| jamesbenson | yes | 17:16 |
| inc0 | seems like a bug | 17:16 |
| jamesbenson | same as bootstrap | 17:16 |
| inc0 | strange that it doesn't show on gates tho | 17:16 |
| jamesbenson | agreed | 17:17 |
| *** cuongnv has quit IRC | 17:17 | |
| *** rwallner_ has quit IRC | 17:18 | |
| *** manheim has quit IRC | 17:19 | |
| jamesbenson | inc0: I installed kolla-ansible==4.0.1 and happened there as well.. | 17:20 |
| inc0 | this looks like somethign on nodes | 17:20 |
| inc0 | docker-py version issue is my bet | 17:20 |
| *** zhubingbing__ has quit IRC | 17:23 | |
| *** unicell has joined #openstack-kolla | 17:23 | |
| *** shardy has quit IRC | 17:24 | |
| *** rstarmer has joined #openstack-kolla | 17:26 | |
| jamesbenson | yeah, doing an uninstall on docker-py and pip and reinstall from bootstrap | 17:27 |
| jamesbenson | inc0: same issue. | 17:28 |
| *** pcaruana has quit IRC | 17:29 | |
| spsurya | night all | 17:34 |
| *** rstarmer has quit IRC | 17:34 | |
| *** rstarmer has joined #openstack-kolla | 17:35 | |
| *** eaguilar has joined #openstack-kolla | 17:35 | |
| *** rstarmer has quit IRC | 17:37 | |
| *** lucasxu has joined #openstack-kolla | 17:39 | |
| mewald | Have a strange issue: Trying to build images on stable/ocata and build.py fails instantly with "no such option ec2api-user in group [DEFAULT]" Does that ring a bell by any chance? | 17:40 |
| *** itlinux_ has quit IRC | 17:41 | |
| *** itlinux has quit IRC | 17:41 | |
| *** duonghq has quit IRC | 17:42 | |
| *** sudipto has joined #openstack-kolla | 17:45 | |
| *** sudipto_ has joined #openstack-kolla | 17:46 | |
| inc0 | mewald: hmm...custom build.conf? | 17:47 |
| *** schwicht has quit IRC | 17:49 | |
| mewald | only stuff I changed was base, namespace, push, registry, tag, maintainer and profile and added the profile under [profiles] | 17:49 |
| inc0 | but did you generate it in newton or prior to newton? | 17:49 |
| mewald | I know it worked a few days ago on the same commit so it has to be some dumb typo or something ... | 17:50 |
| inc0 | ah | 17:50 |
| inc0 | well it definetly looks like kolla-build.conf issue | 17:50 |
| mewald | ok thanks for the advice, I was more on the track of wrong version of some python package … | 17:51 |
| mewald | hmm, same with default kolla-build.conf … I had a 14 hour day so far with 6 hours on the train. maybe I should just stop for today xD | 17:52 |
| inc0 | mewald: it might one of *these* days;) | 17:53 |
| *** ghs39ghsggf has joined #openstack-kolla | 17:56 | |
| *** ghs39ghsggf has left #openstack-kolla | 17:56 | |
| openstackgerrit | Merged openstack/kolla master: Support OVN docker images https://review.openstack.org/459179 | 17:57 |
| *** itlinux has joined #openstack-kolla | 17:57 | |
| *** itlinux_ has joined #openstack-kolla | 17:57 | |
| *** iniazi_ has quit IRC | 17:59 | |
| *** sudipto_ has quit IRC | 18:01 | |
| *** sudipto has quit IRC | 18:01 | |
| *** schwicht has joined #openstack-kolla | 18:02 | |
| *** daidv_ has quit IRC | 18:03 | |
| *** vhosakot has quit IRC | 18:13 | |
| *** itlinux has quit IRC | 18:14 | |
| *** itlinux_ has quit IRC | 18:14 | |
| *** itlinux has joined #openstack-kolla | 18:16 | |
| *** itlinux_ has joined #openstack-kolla | 18:16 | |
| *** salv-orl_ has quit IRC | 18:31 | |
| *** egonzalez has joined #openstack-kolla | 18:31 | |
| sbezverk | kfox1111: ping | 18:34 |
| *** mewald has quit IRC | 18:45 | |
| *** iniazi has joined #openstack-kolla | 18:51 | |
| kfox1111 | sbezverk: pong | 18:52 |
| sbezverk | kfox1111: are you aware of any mechanism for the pod to select a specific network if cni offers multiple network | 19:00 |
| sbezverk | I configured multus cni plugin with multiple networks, and kube-dns consumes ALL interfaces | 19:02 |
| kolla-slack | <kfox1111> Cni supports it but k8s doesn't as far as I know. Search kubernetes multi Ip pod for the issue | 19:03 |
| sbezverk | kfox1111: thanks will do | 19:03 |
| jamesbenson | inc0: something must have gone south on my maas deployment of the node... once I redeployed the node, it worked again. | 19:07 |
| inc0 | jamesbenson: good to hear | 19:07 |
| jamesbenson | inc0: too many moving parts :-p | 19:08 |
| inc0 | ikr | 19:08 |
| *** pbourke has quit IRC | 19:10 | |
| EmilienM | inc0: hey | 19:15 |
| inc0 | hey EmilienM what's up? | 19:15 |
| EmilienM | inc0: what is the magic that takes params from kolla-ansible (ie: nova_api_port) and push it in the nova.conf template? What module do you use? | 19:16 |
| inc0 | we wrote our own https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/nova/tasks/config.yml#L48 | 19:17 |
| EmilienM | ok where is code of merge_configs? | 19:17 |
| inc0 | https://github.com/openstack/kolla-ansible/blob/master/ansible/action_plugins/merge_configs.py | 19:17 |
| EmilienM | thanks a lot | 19:18 |
| EmilienM | perfect | 19:18 |
| inc0 | np | 19:18 |
| inc0 | recently we found one limitation for it - multiple instances of same varialb | 19:18 |
| inc0 | e | 19:18 |
| inc0 | like in pci passthrough + multiple devices | 19:18 |
| EmilienM | inc0: I'm doing some research on re: etcd/confd things, see the thread. We're still trying to think with Doug, how we can make config management done in a single way in OpenStack | 19:18 |
| EmilienM | instead of relying on templates | 19:18 |
| inc0 | so might need some more work for edge use cases, but other thant that we are happy with it | 19:18 |
| inc0 | yeah I follow this, good work, just gonna be complex imho | 19:19 |
| inc0 | devil's in the details | 19:19 |
| EmilienM | yeah | 19:19 |
| EmilienM | bbl | 19:19 |
| inc0 | let me know if you have any more questiosn | 19:19 |
| *** rcrit has joined #openstack-kolla | 19:21 | |
| *** ducttape_ has quit IRC | 19:21 | |
| *** ducttape_ has joined #openstack-kolla | 19:21 | |
| kfox1111 | sbezverk: weird... it shouldn't grab all of the nics. | 19:26 |
| kfox1111 | I'd almost call that a bug in the cni driver. | 19:26 |
| *** rwallner has joined #openstack-kolla | 19:29 | |
| *** rwallner has quit IRC | 19:33 | |
| *** mewald has joined #openstack-kolla | 19:40 | |
| *** omenvl has joined #openstack-kolla | 19:41 | |
| *** eaguilar has quit IRC | 19:41 | |
| *** eaguilar has joined #openstack-kolla | 19:45 | |
| kfox1111 | sbezverk: does just loading the sriov driver work? | 19:45 |
| sdake | kfox1111 can you pleae review this: https://review.openstack.org/#/c/473588/ | 19:57 |
| *** eaguilar has quit IRC | 20:02 | |
| *** spsurya has quit IRC | 20:03 | |
| jamesbenson | inc0: Quick Random question about upgrading for example from 4.0.2 to 5.0.0 (when it's ready). If you have a deployment node, separate from the cluster. Would you have to upgrade the kolla/kolla-ansible packages on the deployment node, or just have the latest docker images and do upgrade? | 20:04 |
| *** ansmith has quit IRC | 20:04 | |
| inc0 | you have to | 20:04 |
| inc0 | kolla-ansible will have release-specific upgrae procedures | 20:04 |
| kfox1111 | sdake: will have a look. | 20:05 |
| inc0 | which are few, but they are there | 20:05 |
| sdake | kfox1111 thanks | 20:05 |
| jamesbenson | okay, so everything will need to be upgraded then? Would a fresh bootstrap be needed for any dependancies? | 20:05 |
| sdake | inc0 service charts contain all the dependency content, not th emicrocharts | 20:06 |
| inc0 | sdake: I replied to you, entrypoint is thing that actually makes it work | 20:06 |
| inc0 | and entrypoint is built into containers | 20:06 |
| inc0 | I mean pods | 20:06 |
| sdake | you are mistaken - sorry | 20:07 |
| sdake | kfox1111 can you please clarify | 20:07 |
| inc0 | https://github.com/openstack/kolla-kubernetes/blob/c7a45fb0dde4d73ac0aa7174d77653bb6e3e006a/helm/kolla-common/templates/_common_dependency_container.yaml I am not | 20:08 |
| kfox1111 | sdake: clerify how? | 20:09 |
| sdake | whether or not entrypoint is built into the containers is 100% irrelevant, the fact is service charts contain the dependency information | 20:09 |
| kfox1111 | sdake's correct. | 20:09 |
| inc0 | ok, then this works in a way that race codntions makes containers break and respin | 20:09 |
| sdake | i didint' say his implementaiton was race free | 20:10 |
| inc0 | as in, in playboook there is no notion of waiting for stuff to finish | 20:10 |
| inc0 | sorry then, I misunderstood this | 20:10 |
| kfox1111 | inc0: yeah, it probably will converge eventually. just will probably take a bit longer. | 20:10 |
| sdake | the sync points re what is left | 20:10 |
| *** itlinux_ has quit IRC | 20:13 | |
| *** itlinux has quit IRC | 20:13 | |
| sdake | also kfox1111 perhaps you could clarify another point - I thought the plan was not to use galera and just let mariadb restart if it failed | 20:14 |
| inc0 | I also think playbooks should be separate for service level charts | 20:14 |
| sdake | why do yo uthink that inc0 | 20:14 |
| inc0 | more composability | 20:15 |
| inc0 | and more pararell processing | 20:15 |
| sdake | sounds like gold plating | 20:15 |
| sdake | how about we get something working | 20:15 |
| inc0 | so what could happen is you have job with compute-kit | 20:15 |
| sdake | and then gold plate it | 20:15 |
| inc0 | that will spin neutron which will have job for spinning all neutron | 20:15 |
| inc0 | well, this is arch | 20:16 |
| inc0 | not gold plating, also nto much more work imho | 20:16 |
| inc0 | but agree, let's get prototype working | 20:16 |
| inc0 | however we get it to work | 20:16 |
| inc0 | if we do it in single play it's gonna be all sequential | 20:17 |
| inc0 | and k8s is bad at sequential | 20:17 |
| inc0 | as in, deployment will end up being lot slower than kolla-ansible | 20:17 |
| sdake | i t takes 4 minutes to deploy kolla-kubernetes | 20:17 |
| inc0 | if we do it in pararell, much faster | 20:17 |
| inc0 | we also need to think about how to put plays inside container...git clone whole repo? | 20:18 |
| sdake | much faster than 4 minutes -who cares? :) | 20:18 |
| inc0 | 4 minutes on how many nodes tho | 20:18 |
| sdake | any number of nodes | 20:19 |
| inc0 | kfox1111: you think config should be separate? I don't think that's good idea | 20:19 |
| sdake | ansible doesn't login to each node and do work | 20:19 |
| inc0 | maybe overrides | 20:19 |
| inc0 | but I see little value in having config as separate step | 20:19 |
| sdake | ansible launches helm microcharts on one node, kuberntes distributes the workload | 20:19 |
| inc0 | sdake: ok, so an idea for config management | 20:20 |
| inc0 | we can modify merge_config so instead of looking into /etc/kolla/config it will look into configmaps of k8s | 20:20 |
| sdake | mege_config isn't needed | 20:21 |
| sdake | we ha d 3 hours of sessions at summit on this otpic | 20:21 |
| inc0 | nova --config-file 1 --config-file 2? | 20:21 |
| sdake | yup | 20:21 |
| *** tonanhngo has joined #openstack-kolla | 20:23 | |
| sdake | although i do think in the short term sticking with what we have re genconfig is appropriate | 20:23 |
| inc0 | well using merge_config from k8s configmaps is actually quite easy | 20:23 |
| inc0 | and quick change | 20:23 |
| inc0 | https://github.com/openstack/kolla-kubernetes/blob/master/ansible/action_plugins/merge_configs.py#L92 replace this line with k8s logic and it all just going to work | 20:25 |
| inc0 | dims: how's python-kubernetes going?:) | 20:25 |
| dims | inc0 : haven't touched it in a while, seeing any issues? | 20:27 |
| *** mewald has quit IRC | 20:29 | |
| rcrit | I'm working on a container for the novajoin service and it works fine when run as root, but when running as a user it cannot read a required bind mounted file (a Kerberos keytab). | 20:32 |
| rcrit | is there some common fix for this? | 20:32 |
| inc0 | dims: no, about to put is into core of kolla-k8s | 20:33 |
| inc0 | :P | 20:34 |
| dims | +1 | 20:34 |
| *** tonanhngo has quit IRC | 20:34 | |
| sbezverk | kfox1111: I cannot use just sriov as in this case kube-dns pods does not come up, its liveness check fails as it cannot communicate with dns container | 20:36 |
| sbezverk | I really need a possibility to tell which pod uses which network and in addition how many.. | 20:37 |
| sdake | kfox1111 see pm plz | 20:45 |
| *** itlinux_ has joined #openstack-kolla | 20:45 | |
| *** itlinux has joined #openstack-kolla | 20:45 | |
| *** itlinux_ has quit IRC | 20:46 | |
| *** itlinux has quit IRC | 20:46 | |
| *** jtriley has quit IRC | 20:48 | |
| kfox1111 | sbezverk: weird... is there not a route from kube-apiserver to the sriov nic? | 20:51 |
| kfox1111 | k8s requires the ip's to all be routable. | 20:51 |
| sbezverk | kfox1111: right that is why I want combination if nics 1 - management for api server and the rest if required data plane, even without ipam | 20:52 |
| sbezverk | since with sriov it is application configures ip | 20:52 |
| *** dmsimard has quit IRC | 20:56 | |
| *** itlinux has joined #openstack-kolla | 21:03 | |
| *** itlinux_ has joined #openstack-kolla | 21:04 | |
| *** lucasxu has quit IRC | 21:06 | |
| *** rwallner has joined #openstack-kolla | 21:08 | |
| *** egonzalez has quit IRC | 21:11 | |
| kfox1111 | sbezverk: I don't think k8s really supports multinic well at all. :/ | 21:13 |
| kfox1111 | but if you had sriov deployed in a way k8s expected, I think you could do it all with sriov. | 21:13 |
| kfox1111 | though your pod capacity would be relatively small... | 21:13 |
| inc0 | hey, what are we doing with this? https://review.openstack.org/#/c/457317/ | 21:15 |
| kfox1111 | kind of related to this one: https://review.openstack.org/#/c/473588/ | 21:16 |
| inc0 | kinda | 21:23 |
| inc0 | but not really, it's quick setup of dev env | 21:23 |
| inc0 | jascott1: thoughts? | 21:23 |
| sbezverk | kfox1111: well they should support it eventually at least.. | 21:24 |
| kfox1111 | inc0: I think they could/should probably share a bunch of code. we are really good at reinventing the wheel with extra code, then letting it bitrot. would really like to see that stop. | 21:25 |
| kfox1111 | sbezverk: yeah. they are wroking on it. have a whole sig devoted to it. but they don't always move very fast. :/ | 21:25 |
| *** dave-mccowan has quit IRC | 21:26 | |
| kfox1111 | sbezverk: one thing I've not really seen, but suspect that will work, | 21:26 |
| kfox1111 | is that the network docs say all things gotta be able to talk to all other things. but not specifically how, | 21:26 |
| kfox1111 | so I think you probably could do some nodes with the sriov backend, and some with flannel in flat network mode, and if everything was on the same network or at least routable, it would work I think. | 21:27 |
| jascott1 | inc0 thoughtson? | 21:28 |
| jascott1 | ah the microservice one uses ansible to wait on everything | 21:30 |
| kfox1111 | "OpenLDAP is designed for massive scalability | 21:30 |
| *** schwicht has quit IRC | 21:30 | |
| kfox1111 | but aggressively shuns ease of administration." heh. isn't that the truth.... | 21:30 |
| inc0 | jascott1: and relation to playbook you wrote to deploy k8s and kolla-k8s | 21:30 |
| jascott1 | mine just follows the guide or used to... its probably outdated slightly | 21:31 |
| *** dave-mcc_ has joined #openstack-kolla | 21:31 | |
| inc0 | I'm thinking if we should propose just k8s deployment somewhere upstream | 21:31 |
| jascott1 | and take out of the guide? | 21:32 |
| jascott1 | BYOK8S | 21:32 |
| kfox1111 | yeah. maybe a pointer for deployment guides for kubeadm/kargo? | 21:32 |
| kfox1111 | (someone heare deployed a working k8s with kargo the other day. so seems viable.) | 21:33 |
| *** dmsimard has joined #openstack-kolla | 21:33 | |
| kfox1111 | always liked the BYOK8S idea. | 21:33 |
| inc0 | I tried it some time ago and it was "meh +" | 21:33 |
| *** jtriley has joined #openstack-kolla | 21:34 | |
| jascott1 | i like having a playbook version of the guide... if its removed from guide then yeah take it out of playbook | 21:34 |
| kfox1111 | sounds reasonable. | 21:34 |
| jascott1 | it works for me tho, i have vagrant and terraform setups that write my inventory and then I use that playbook | 21:35 |
| inc0 | jascott1: btw might sharing them? | 21:35 |
| jascott1 | NO THEY ARE MINE | 21:35 |
| jascott1 | ;D | 21:35 |
| jascott1 | yeah I need to put them somewhere anyway | 21:35 |
| kfox1111 | jascott1: cool. :) | 21:36 |
| jascott1 | think I was asking sdake his opinion on if we should have those kinda things in our repo | 21:36 |
| inc0 | I think terraform for k8s would be awesome in general | 21:36 |
| jascott1 | i can push to my github and you can see | 21:36 |
| jascott1 | yeah having both vagrant and terraform covers it pretty well for dev env pov | 21:37 |
| jascott1 | but ansible does all the real k8s work | 21:37 |
| jascott1 | brb | 21:38 |
| *** jascott1 has quit IRC | 21:38 | |
| kfox1111 | still would like to see building blocks split out from orchestration. | 21:38 |
| *** jascott1 has joined #openstack-kolla | 21:38 | |
| inc0 | well, entrypoint is quasi-orchiestration | 21:40 |
| *** jtriley has quit IRC | 21:42 | |
| *** rhallisey has quit IRC | 21:42 | |
| *** jascott1 has quit IRC | 21:43 | |
| kfox1111 | inc0: yeah, kind of. | 21:47 |
| kfox1111 | but currently all the rules are outside of the microservices. | 21:47 |
| kfox1111 | and its an optional feature in the microservices defaulted off. | 21:47 |
| inc0 | yeah that's not going to change | 21:48 |
| kfox1111 | so there really is still a pretty good seperateion of orchestration from the low level charts. | 21:48 |
| inc0 | that's the plan | 21:48 |
| kfox1111 | keeping them seperate would really keep the orchestration/religious arguments away from the heart of kolla-kubernetes | 21:49 |
| inc0 | my personal favorite today is to have k8s-job in service-level chart that will orchiestrate microservices | 21:49 |
| kfox1111 | the same way splitting out kolla-ansible/kolla-kubernetes has done for kolla-containers | 21:49 |
| kfox1111 | inc0: thats kind of the operator aproach. | 21:49 |
| inc0 | yeah | 21:49 |
| inc0 | operators being written in ansible | 21:49 |
| *** rwallner has quit IRC | 21:50 | |
| kfox1111 | certianly one way to do it. :) | 21:50 |
| kfox1111 | (that I think I recommended very early on. ;) | 21:50 |
| inc0 | I can't think of better way to for example do mariadb multinode | 21:50 |
| kfox1111 | I would really like to see a galera cluster operator similar to etcd-operator. | 21:51 |
| inc0 | well galera is harder than etcd | 21:51 |
| kfox1111 | is it? | 21:51 |
| inc0 | I think it is | 21:51 |
| kfox1111 | I'm not so sure. | 21:51 |
| inc0 | well, regardless, we need to write it somehow | 21:52 |
| kfox1111 | even still. I think it would be of great value to the greator community. not just kolla | 21:52 |
| sdake | i sort of thoguht our plan was to forgo galera entirely | 21:52 |
| sdake | since kubernetes just restarts mariadb | 21:52 |
| kfox1111 | was hoping the trove guys would retool on top of k8s and do things with operators. | 21:52 |
| kfox1111 | would fill a real nich. | 21:52 |
| inc0 | sdake: you don't want to forego galera | 21:52 |
| *** rwallner_ has joined #openstack-kolla | 21:52 | |
| inc0 | unless you're ok with ~1min downs of your "not so ha" cluster | 21:53 |
| kfox1111 | sort term/long term. | 21:53 |
| sdake | mariadb takes seconds to stat up | 21:53 |
| kfox1111 | sdake: yeah, but might take a few minutes to fail over. | 21:53 |
| inc0 | restarts are much slower because it takes time to figure out it's down | 21:53 |
| kfox1111 | detection is different then start. | 21:53 |
| inc0 | and without fencing you're in bad shape anyway | 21:53 |
| inc0 | in case of split brain | 21:53 |
| kfox1111 | right. | 21:53 |
| kfox1111 | so, for galera, I'm kind of cool with just pinning the nodes to physical nodes. | 21:55 |
| kfox1111 | and using hostPath. | 21:55 |
| kfox1111 | the one thing I'm leary of, is startup after lights out. | 21:55 |
| kfox1111 | I think I'd rather it block until an admin steps in and tells it which is to be the master. | 21:56 |
| kfox1111 | I think a specially crafted init-container could do that. | 21:56 |
| kfox1111 | but would have to be written. | 21:56 |
| *** rwallner_ has quit IRC | 21:57 | |
| inc0 | or we can try cocroach db;) | 21:57 |
| kfox1111 | like, if the init-container starts, and the cluster is already corrate, let it join. if its not, block until an admin kubectl exec's in, and picks a winner. | 21:57 |
| kfox1111 | -1 :) | 21:57 |
| kfox1111 | I really like the idea of cocroachdb. | 21:57 |
| kfox1111 | I don't trust it with my data yet. way way too new. | 21:58 |
| kfox1111 | bugs in regular software are somewhat bad, but usually not perminant. bugs in a storage system, thats a whole nother level of bad day. | 21:58 |
| *** jascott1 has joined #openstack-kolla | 21:59 | |
| *** schwicht has joined #openstack-kolla | 22:00 | |
| *** jamesbenson has quit IRC | 22:01 | |
| *** salv-orlando has joined #openstack-kolla | 22:01 | |
| kfox1111 | you know, one thing that would be really cool for an inital mysql operator, would be db/user creation via 3rd party resources. | 22:03 |
| *** jamesbenson has joined #openstack-kolla | 22:03 | |
| *** jamesbenson has quit IRC | 22:07 | |
| *** schwicht has quit IRC | 22:31 | |
| *** schwicht has joined #openstack-kolla | 22:33 | |
| *** schwicht has quit IRC | 22:36 | |
| *** srwilkers has quit IRC | 22:38 | |
| *** salv-orlando has quit IRC | 22:45 | |
| *** itlinux_ has quit IRC | 22:59 | |
| *** itlinux has quit IRC | 22:59 | |
| jascott1 | kfox1111 whats the advantage of TPR there? | 23:00 |
| *** iniazi has quit IRC | 23:04 | |
| *** iniazi has joined #openstack-kolla | 23:08 | |
| *** rwallner has joined #openstack-kolla | 23:09 | |
| *** rwallner has quit IRC | 23:13 | |
| *** jamesbenson has joined #openstack-kolla | 23:24 | |
| *** jamesbenson has quit IRC | 23:29 | |
| *** srwilkers has joined #openstack-kolla | 23:29 | |
| *** ducttape_ has quit IRC | 23:32 | |
| *** manheim has joined #openstack-kolla | 23:35 | |
| *** ansmith has joined #openstack-kolla | 23:35 | |
| *** manheim has quit IRC | 23:40 | |
| *** manheim has joined #openstack-kolla | 23:41 | |
| *** hongbin has quit IRC | 23:46 | |
| *** manheim has quit IRC | 23:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!