| masber | I would like to add dhcp_domain parameter to my nova.conf file, however kolla installs multiple of these files for different containers. How do I know which file to edit? | 00:00 |
|---|---|---|
| *** yingjun has joined #openstack-kolla | 00:01 | |
| *** bmaltais has quit IRC | 00:04 | |
| *** powerd has quit IRC | 00:05 | |
| *** MarginHu has joined #openstack-kolla | 00:06 | |
| *** caoyuan has joined #openstack-kolla | 00:09 | |
| dims | sdake : the current review is a overall policy statement, so i don't think a specific exception works in there, we don't even mention kolla even once. So you should probably follow up with ttx on that one | 00:11 |
| sdake | dims i left a comment in the review | 00:12 |
| dims | y responding there | 00:12 |
| sdake | dims the probelm is the policy statement locks out releasing ocata zstreams via the automted release process | 00:12 |
| *** ducttape_ has joined #openstack-kolla | 00:12 | |
| sdake | as we can't backport manifest code | 00:12 |
| dims | it does not say anything about releases | 00:12 |
| *** caoyuan has quit IRC | 00:12 | |
| dims | or the release process change | 00:13 |
| sdake | dims perhaps i misunderstodo the language - i readi t earlyl this mronign and injured myself last night - so not feeling weel | 00:13 |
| dims | one sec sdake | 00:13 |
| dims | sdake : fresh read from the rendered doc would help i think - http://docs-draft.openstack.org/65/469265/2/check/gate-governance-docs-ubuntu-xenial/85c307b//doc/build/html/resolutions/20170530-binary-artifacts.html | 00:14 |
| sdake | but i dont really want ot discuss it atm, as i'm heading to bed | 00:14 |
| sdake | think i broke my hand or injuredit badly | 00:14 |
| dims | sdake : ouch! no worries, when you get to it | 00:14 |
| *** mnasiadka has joined #openstack-kolla | 00:15 | |
| *** mnasiadka has quit IRC | 00:19 | |
| *** salv-orlando has joined #openstack-kolla | 00:21 | |
| *** Pavo has quit IRC | 00:24 | |
| *** krtaylor has joined #openstack-kolla | 00:28 | |
| *** salv-orlando has quit IRC | 00:30 | |
| *** manheim has joined #openstack-kolla | 00:34 | |
| *** manheim has quit IRC | 00:37 | |
| kfox1111 | sdake: I don't think the manifest generating code needs to live inside the containers. that avoids the issue alltogether. | 00:38 |
| kfox1111 | see: https://review.openstack.org/#/c/458174/5/tools/gate_run.sh does it all from within the container without changing it. | 00:39 |
| kfox1111 | gotta head out. bbl. | 00:39 |
| *** manheim has joined #openstack-kolla | 00:39 | |
| *** rwsu has quit IRC | 00:41 | |
| *** hongbin has quit IRC | 00:44 | |
| *** tonanhngo has quit IRC | 00:47 | |
| *** ducttape_ has quit IRC | 00:50 | |
| *** jtriley has quit IRC | 00:51 | |
| *** cuongnv has joined #openstack-kolla | 00:54 | |
| *** tonanhngo has joined #openstack-kolla | 00:54 | |
| *** daidv has joined #openstack-kolla | 00:55 | |
| *** tonanhngo has quit IRC | 00:58 | |
| *** tovin07_ has joined #openstack-kolla | 01:02 | |
| *** Pavo has joined #openstack-kolla | 01:11 | |
| *** Pavo has quit IRC | 01:13 | |
| *** duonghq has joined #openstack-kolla | 01:23 | |
| duonghq | morning guys | 01:25 |
| *** zhubingbing_ has joined #openstack-kolla | 01:25 | |
| zhubingbing_ | hi guys | 01:26 |
| duonghq | hi zhubingbing_ | 01:26 |
| duonghq | how are you | 01:26 |
| *** ducttape_ has joined #openstack-kolla | 01:27 | |
| *** hrw has quit IRC | 01:32 | |
| *** hrw has joined #openstack-kolla | 01:34 | |
| *** kbyrne has quit IRC | 01:39 | |
| *** chas has joined #openstack-kolla | 01:40 | |
| *** kbyrne has joined #openstack-kolla | 01:42 | |
| *** chas has quit IRC | 01:45 | |
| *** manheim has quit IRC | 01:51 | |
| *** manheim has joined #openstack-kolla | 01:51 | |
| zhubingbing_ | hi duonghq | 01:52 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Apply neutron database migration https://review.openstack.org/407922 | 01:56 |
| *** MarginHu has quit IRC | 01:59 | |
| *** MarginHu has joined #openstack-kolla | 01:59 | |
| *** ducttape_ has quit IRC | 01:59 | |
| *** mnasiadka has joined #openstack-kolla | 02:00 | |
| *** MarginHu has quit IRC | 02:00 | |
| *** kbaegis has joined #openstack-kolla | 02:04 | |
| *** mnasiadka has quit IRC | 02:05 | |
| *** salv-orlando has joined #openstack-kolla | 02:05 | |
| *** unicell has quit IRC | 02:08 | |
| *** caowei has joined #openstack-kolla | 02:08 | |
| *** manheim has quit IRC | 02:14 | |
| *** caoyuan has joined #openstack-kolla | 02:14 | |
| *** caoyuan has quit IRC | 02:15 | |
| *** rwsu has joined #openstack-kolla | 02:16 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 02:18 |
| *** goldyfruit has joined #openstack-kolla | 02:18 | |
| *** manheim has joined #openstack-kolla | 02:18 | |
| *** caoyuan has joined #openstack-kolla | 02:18 | |
| *** salv-orlando has quit IRC | 02:20 | |
| *** manheim has quit IRC | 02:39 | |
| *** manheim has joined #openstack-kolla | 02:39 | |
| *** manheim has quit IRC | 02:39 | |
| *** manheim has joined #openstack-kolla | 02:40 | |
| *** goldyfruit has quit IRC | 02:41 | |
| *** sayantan_ has joined #openstack-kolla | 02:43 | |
| *** manheim has quit IRC | 02:44 | |
| *** masber has quit IRC | 02:46 | |
| *** sayantan_ has quit IRC | 02:48 | |
| *** caoyuan has quit IRC | 02:50 | |
| *** caoyuan has joined #openstack-kolla | 02:51 | |
| *** caoyuan has quit IRC | 02:59 | |
| *** yingjun has quit IRC | 02:59 | |
| spsurya | morning all | 03:04 |
| spsurya | sup zhubingbing_ | 03:04 |
| spsurya | sup duonghq | 03:04 |
| *** yingjun has joined #openstack-kolla | 03:15 | |
| duonghq | sup spsurya | 03:15 |
| *** dave-mccowan has quit IRC | 03:16 | |
| *** salv-orlando has joined #openstack-kolla | 03:17 | |
| *** xinliang has joined #openstack-kolla | 03:21 | |
| *** xinliang has quit IRC | 03:21 | |
| *** xinliang has joined #openstack-kolla | 03:21 | |
| *** salv-orl_ has joined #openstack-kolla | 03:24 | |
| *** salv-orlando has quit IRC | 03:24 | |
| *** janki has joined #openstack-kolla | 03:26 | |
| *** zhurong has joined #openstack-kolla | 03:27 | |
| *** jamesPR has joined #openstack-kolla | 03:27 | |
| *** salv-orl_ has quit IRC | 03:29 | |
| *** caowei has quit IRC | 03:39 | |
| *** caoyuan has joined #openstack-kolla | 03:39 | |
| *** chas has joined #openstack-kolla | 03:41 | |
| *** yingjun has quit IRC | 03:44 | |
| *** chas has quit IRC | 03:46 | |
| *** yingjun has joined #openstack-kolla | 03:53 | |
| *** zhubingbing_ has quit IRC | 03:55 | |
| *** zhubingbing has joined #openstack-kolla | 03:56 | |
| openstackgerrit | Steve Baker proposed openstack/kolla master: kolla-build --list-dependencies use json, not pprint https://review.openstack.org/469741 | 04:03 |
| *** sayantan_ has joined #openstack-kolla | 04:06 | |
| *** rwsu has quit IRC | 04:07 | |
| *** zhubingbing has quit IRC | 04:11 | |
| *** caoyuan has quit IRC | 04:14 | |
| *** eanylin has quit IRC | 04:18 | |
| *** tovin07_ has quit IRC | 04:22 | |
| *** Serlex has joined #openstack-kolla | 04:24 | |
| *** salv-orlando has joined #openstack-kolla | 04:26 | |
| *** zhurong has quit IRC | 04:27 | |
| *** jamesPR has quit IRC | 04:28 | |
| *** sgrasley has joined #openstack-kolla | 04:28 | |
| *** sgrasley has quit IRC | 04:33 | |
| *** zhurong has joined #openstack-kolla | 04:36 | |
| *** unicell has joined #openstack-kolla | 04:39 | |
| *** salv-orlando has quit IRC | 04:40 | |
| *** manheim has joined #openstack-kolla | 04:42 | |
| *** pcaruana has joined #openstack-kolla | 04:43 | |
| *** eanylin has joined #openstack-kolla | 04:51 | |
| *** caoyuan has joined #openstack-kolla | 04:53 | |
| *** salv-orlando has joined #openstack-kolla | 04:53 | |
| *** zhubingbing has joined #openstack-kolla | 04:53 | |
| *** skramaja has joined #openstack-kolla | 05:08 | |
| *** caoyuan has quit IRC | 05:09 | |
| *** caoyuan has joined #openstack-kolla | 05:10 | |
| *** MarginHu has joined #openstack-kolla | 05:10 | |
| *** manheim has quit IRC | 05:14 | |
| *** kfir has quit IRC | 05:15 | |
| *** zhurong has quit IRC | 05:16 | |
| *** zhurong has joined #openstack-kolla | 05:17 | |
| *** jaosorior_away is now known as jaosorior | 05:24 | |
| *** rwsu has joined #openstack-kolla | 05:38 | |
| *** chas has joined #openstack-kolla | 05:42 | |
| *** unicell has quit IRC | 05:44 | |
| *** unicell has joined #openstack-kolla | 05:45 | |
| *** pcaruana has quit IRC | 05:49 | |
| *** chas has quit IRC | 05:49 | |
| *** masber has joined #openstack-kolla | 06:00 | |
| *** pcaruana has joined #openstack-kolla | 06:02 | |
| *** chas has joined #openstack-kolla | 06:07 | |
| *** salv-orlando has quit IRC | 06:11 | |
| *** manheim has joined #openstack-kolla | 06:11 | |
| *** salv-orlando has joined #openstack-kolla | 06:11 | |
| *** salv-orlando has quit IRC | 06:16 | |
| *** manheim has quit IRC | 06:24 | |
| *** caowei has joined #openstack-kolla | 06:27 | |
| *** korzen has joined #openstack-kolla | 06:27 | |
| *** yingjun has quit IRC | 06:29 | |
| openstackgerrit | Hanxi Liu proposed openstack/kolla-ansible master: Stop using old gnocchi reference https://review.openstack.org/469771 | 06:47 |
| openstackgerrit | Hanxi Liu proposed openstack/kolla master: Stop using old gnocchi reference https://review.openstack.org/469772 | 06:47 |
| *** mnasiadka has joined #openstack-kolla | 06:49 | |
| *** sayantan_ has quit IRC | 06:50 | |
| *** caoyuan has quit IRC | 06:52 | |
| *** chas has quit IRC | 06:55 | |
| *** salv-orlando has joined #openstack-kolla | 06:58 | |
| *** tovin07_ has joined #openstack-kolla | 06:59 | |
| *** jmccarthy has joined #openstack-kolla | 06:59 | |
| *** zhubingbing has quit IRC | 07:04 | |
| *** zhubingbing has joined #openstack-kolla | 07:06 | |
| *** n0isyn0ise has quit IRC | 07:12 | |
| *** slagle has quit IRC | 07:16 | |
| *** shardy has joined #openstack-kolla | 07:16 | |
| *** mgoddard has joined #openstack-kolla | 07:19 | |
| *** caoyuan has joined #openstack-kolla | 07:19 | |
| *** caoyuan_ has joined #openstack-kolla | 07:19 | |
| *** caoyuan has quit IRC | 07:23 | |
| *** manheim has joined #openstack-kolla | 07:25 | |
| *** salv-orl_ has joined #openstack-kolla | 07:28 | |
| hrw | morning | 07:28 |
| *** manheim has quit IRC | 07:30 | |
| *** salv-orlando has quit IRC | 07:30 | |
| Serlex | hello | 07:32 |
| hrw | https://www.redhat.com/archives/rdo-list/2017-June/msg00000.html - setuptools v36 (released today) fails to work | 07:37 |
| *** cuongnv has quit IRC | 07:37 | |
| *** cuongnv has joined #openstack-kolla | 07:38 | |
| openstackgerrit | Juan Badia Payno proposed openstack/kolla master: Modified fluentd Image https://review.openstack.org/467219 | 07:45 |
| *** tvignaud has quit IRC | 08:03 | |
| *** egonzalez has joined #openstack-kolla | 08:16 | |
| *** tvignaud has joined #openstack-kolla | 08:16 | |
| *** mgoddard has quit IRC | 08:16 | |
| hrw | https://github.com/pypa/setuptools/issues/1042 as well | 08:16 |
| *** zhurong has quit IRC | 08:29 | |
| *** athomas has joined #openstack-kolla | 08:30 | |
| reidrac | egonzalez: thanks for the review! \o/ | 08:33 |
| egonzalez | yw | 08:33 |
| *** manheim has joined #openstack-kolla | 08:42 | |
| *** chas has joined #openstack-kolla | 08:46 | |
| openstackgerrit | Hanxi Liu proposed openstack/kolla master: Stop using old gnocchi reference https://review.openstack.org/469772 | 08:46 |
| *** chas has quit IRC | 08:51 | |
| *** mgoddard has joined #openstack-kolla | 08:52 | |
| *** pbourke_ has quit IRC | 08:53 | |
| *** pbourke_ has joined #openstack-kolla | 08:53 | |
| *** zhubingbing has quit IRC | 08:56 | |
| *** zhubingbing has joined #openstack-kolla | 08:57 | |
| *** manheim has quit IRC | 09:00 | |
| openstackgerrit | Hanxi Liu proposed openstack/kolla master: Stop using old gnocchi reference https://review.openstack.org/469772 | 09:00 |
| *** manheim has joined #openstack-kolla | 09:00 | |
| *** powerd has joined #openstack-kolla | 09:06 | |
| *** chas has joined #openstack-kolla | 09:07 | |
| *** zhubingbing_ has joined #openstack-kolla | 09:11 | |
| *** chas has quit IRC | 09:12 | |
| *** zhubingbing has quit IRC | 09:12 | |
| *** caoyuan_ has quit IRC | 09:12 | |
| *** sambetts|afk is now known as sambetts | 09:13 | |
| egonzalez | Hi guys, regarding gnocchi not being an openstack project anymore | 09:17 |
| egonzalez | should we still supporting gnocchi source installs or move source images to consume binary packages? | 09:17 |
| openstackgerrit | Hanxi Liu proposed openstack/kolla master: Stop using old gnocchi reference https://review.openstack.org/469772 | 09:18 |
| masber | good afternoon all, I would like to edit nova.conf file, I see that kolla creates multiple ones, does it matters which one to use? I want to add a dhcp_domain entry to change my FQDN through cloud-init http://paste.openstack.org/raw/611180/ | 09:19 |
| openstackgerrit | Hanxi Liu proposed openstack/kolla-ansible master: Stop using old gnocchi reference https://review.openstack.org/469771 | 09:19 |
| egonzalez | masber, all depends on which service what to change the config | 09:21 |
| egonzalez | if is for all nova services, /etc/kolla/config/nova.conf | 09:22 |
| egonzalez | ie: nova-compute /etc/kolla/config/nova/nova-compute.conf | 09:22 |
| masber | I just want to change the FQDN for my vms | 09:23 |
| masber | I am not sure which nova service is managing that, I assume nova-api? | 09:23 |
| masber | or should I just change it for all (/etc/kolla/config/nova.conf)? | 09:23 |
| egonzalez | actually is not nova who manages fqdn, is neutron https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/templates/neutron.conf.j2#L54 | 09:25 |
| egonzalez | but I opened a but in nova 2 days ago because also need to enforce use_neutron: true in nova.conf | 09:26 |
| masber | is this outdated then http://lists.openstack.org/pipermail/openstack/2017-March/019000.html??? | 09:26 |
| *** caoyuan has joined #openstack-kolla | 09:27 | |
| egonzalez | masber, sth like this http://paste.openstack.org/show/611182/ | 09:28 |
| *** chas has joined #openstack-kolla | 09:28 | |
| masber | egonzalez, I assume I have to create /etc/kolla/config/nova.conf in my network nodes? or just openstack controller? | 09:31 |
| masber | because I cana;t find it | 09:31 |
| egonzalez | masber, in the deployer node (ansible) | 09:31 |
| egonzalez | those are custom files that are merged with nova or neutron conf files | 09:31 |
| reidrac | looks like there's an issue with reno 3.2.1; it uses six but fails to include it in the requirements | 09:31 |
| *** caoyuan has quit IRC | 09:31 | |
| masber | do I need to reconfigure afterwards? | 09:31 |
| reidrac | eh, 2.3.1 | 09:31 |
| egonzalez | masber, this way you can customize your services | 09:32 |
| Serlex | Ditto reidrac | 09:32 |
| *** chas has quit IRC | 09:32 | |
| egonzalez | masber, it depends of what changed, you can do it afterwards if is just a config option change, if need a db sync something else will be needed | 09:32 |
| egonzalez | masber, ie, here is done after deploy to include a config option https://github.com/openstack/kolla-ansible/blob/master/doc/designate-guide.rst#neutron-and-nova-integration | 09:33 |
| reidrac | looks like is a regression: https://bugs.launchpad.net/reno/+bug/1596731 | 09:34 |
| openstack | Launchpad bug 1596731 in reno "Undeclared dependency on six" [Undecided,Fix released] - Assigned to Jay Faulkner (jason-oldos) | 09:34 |
| egonzalez | reidrac, is a setuptools bug https://github.com/pypa/setuptools/issues/1042 | 09:34 |
| reidrac | egonzalez: good one, thanks | 09:35 |
| *** openstackstatus has quit IRC | 09:36 | |
| egonzalez | thanks to hrw ;) | 09:37 |
| *** openstackstatus has joined #openstack-kolla | 09:38 | |
| *** ChanServ sets mode: +v openstackstatus | 09:38 | |
| masber | egonzalez, yes I did that already new vms' got the domain from designate, however it reverts back to novalocal after reboot | 09:39 |
| masber | I also have use_neutron = True on mostly all nova.conf files --> http://paste.openstack.org/raw/611184/ | 09:42 |
| -openstackstatus- NOTICE: There is a known issue with setuptools 36.0.0 and errors about the "six" package. For current details see https://github.com/pypa/setuptools/issues/1042 and monitor #openstack-infra | 09:44 | |
| egonzalez | masber, thats right, the only missing is the cron file | 09:46 |
| egonzalez | masber, are you using master or ocata? | 09:47 |
| *** shardy is now known as shardy_afk | 09:47 | |
| *** chas has joined #openstack-kolla | 09:49 | |
| *** zhubingbing_ has quit IRC | 09:52 | |
| *** chas has quit IRC | 09:53 | |
| *** yangyapeng has joined #openstack-kolla | 09:56 | |
| *** yangyapeng has quit IRC | 09:56 | |
| *** tovin07_ has quit IRC | 09:57 | |
| *** salv-orl_ has quit IRC | 10:04 | |
| *** cuongnv has quit IRC | 10:04 | |
| *** shardy_afk is now known as shardy | 10:05 | |
| *** caowei has quit IRC | 10:05 | |
| *** chas has joined #openstack-kolla | 10:10 | |
| *** iniazi has joined #openstack-kolla | 10:10 | |
| *** iniazi_ has quit IRC | 10:13 | |
| *** chas has quit IRC | 10:14 | |
| *** chas has joined #openstack-kolla | 10:21 | |
| *** kolla-slack has quit IRC | 10:24 | |
| *** kolla-slack has joined #openstack-kolla | 10:24 | |
| *** chas has quit IRC | 10:26 | |
| bjolo | howdy | 10:28 |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla-ansible master: [WIP] Add dragonflow ansible role https://review.openstack.org/465862 | 10:31 |
| bjolo | gnocchi not openstack project anymore? | 10:33 |
| *** zhurong has joined #openstack-kolla | 10:34 | |
| bjolo | what has replaced it? | 10:34 |
| masber | egonzalez, not sure centos binary 4.0.0 | 10:49 |
| egonzalez | bjolo, they decided go outside openstack https://julien.danjou.info/blog/2017/gnocchi-independence | 10:50 |
| *** caoyuan has joined #openstack-kolla | 10:52 | |
| *** powerd has quit IRC | 10:55 | |
| *** caoyuan has quit IRC | 10:56 | |
| *** mnasiadka has quit IRC | 10:57 | |
| hrw | egonzalez: gnocchi? still allow source | 10:58 |
| *** powerd has joined #openstack-kolla | 10:59 | |
| *** mnasiadka has joined #openstack-kolla | 10:59 | |
| hrw | egonzalez: it is important module and debian lacks up-to-date packages | 10:59 |
| egonzalez | hrw, thats why asked, we dont use to support non openstack or infrastructure services as databases in the source install, only binary packages | 11:00 |
| hrw | egonzalez: how devstack devs went? | 11:02 |
| *** matrohon has joined #openstack-kolla | 11:07 | |
| *** dave-mccowan has joined #openstack-kolla | 11:08 | |
| *** kbaegis has quit IRC | 11:09 | |
| *** Pavo has joined #openstack-kolla | 11:19 | |
| *** rwallner has joined #openstack-kolla | 11:22 | |
| *** rwallner has quit IRC | 11:24 | |
| *** rwallner has joined #openstack-kolla | 11:30 | |
| *** rhallisey has joined #openstack-kolla | 11:31 | |
| *** skramaja_ has joined #openstack-kolla | 11:33 | |
| *** skramaja has quit IRC | 11:33 | |
| *** janki has quit IRC | 11:35 | |
| *** shardy has quit IRC | 11:36 | |
| *** pradk has joined #openstack-kolla | 11:45 | |
| *** rwallner has quit IRC | 11:48 | |
| openstackgerrit | Merged openstack/kolla master: Stop using old gnocchi reference https://review.openstack.org/469772 | 11:53 |
| *** shardy has joined #openstack-kolla | 11:58 | |
| *** skramaja_ has quit IRC | 11:59 | |
| *** skramaja has joined #openstack-kolla | 11:59 | |
| *** goldyfruit has joined #openstack-kolla | 11:59 | |
| *** goldyfruit has quit IRC | 12:00 | |
| *** rwallner has joined #openstack-kolla | 12:02 | |
| *** manheim has quit IRC | 12:03 | |
| *** slagle has joined #openstack-kolla | 12:09 | |
| *** rwallner has quit IRC | 12:15 | |
| *** zhurong has quit IRC | 12:16 | |
| *** krtaylor has quit IRC | 12:19 | |
| *** chas has joined #openstack-kolla | 12:22 | |
| *** rwallner has joined #openstack-kolla | 12:23 | |
| *** rmart04 has joined #openstack-kolla | 12:25 | |
| *** chas has quit IRC | 12:26 | |
| *** ducttape_ has joined #openstack-kolla | 12:34 | |
| *** salv-orlando has joined #openstack-kolla | 12:39 | |
| *** goldyfruit has joined #openstack-kolla | 12:46 | |
| *** ducttape_ has quit IRC | 12:46 | |
| *** eanylin has quit IRC | 12:47 | |
| *** ducttape_ has joined #openstack-kolla | 12:47 | |
| *** salv-orlando has quit IRC | 12:47 | |
| *** janki has joined #openstack-kolla | 12:49 | |
| *** ducttape_ has quit IRC | 12:52 | |
| *** salv-orlando has joined #openstack-kolla | 12:52 | |
| *** xinliang has quit IRC | 12:57 | |
| *** xinliang has joined #openstack-kolla | 12:58 | |
| *** xinliang has quit IRC | 12:58 | |
| *** xinliang has joined #openstack-kolla | 12:58 | |
| *** ansiwen is now known as ansiwen|lunch | 12:59 | |
| *** rwallner has quit IRC | 13:03 | |
| *** rwallner has joined #openstack-kolla | 13:03 | |
| *** ducttape_ has joined #openstack-kolla | 13:04 | |
| *** eanylin has joined #openstack-kolla | 13:05 | |
| *** mewald has joined #openstack-kolla | 13:06 | |
| *** GonZo2000 has joined #openstack-kolla | 13:08 | |
| *** GonZo2000 has quit IRC | 13:08 | |
| *** GonZo2000 has joined #openstack-kolla | 13:08 | |
| *** lucasxu has joined #openstack-kolla | 13:08 | |
| *** ducttape_ has quit IRC | 13:09 | |
| *** ducttape_ has joined #openstack-kolla | 13:12 | |
| *** ducttape_ has quit IRC | 13:14 | |
| *** powerd has quit IRC | 13:15 | |
| Reepicheep | qq from ansible newbe. how intergrade is the kolla_docker comand .. can I call it from my own playbook? | 13:15 |
| *** ducttap__ has joined #openstack-kolla | 13:16 | |
| Reepicheep | here is kinda what I'm trying to do: http://paste.openstack.org/show/611198/ | 13:16 |
| Reepicheep | my goal is to deploy a new cert quickly so I can call it from my acme client and have it verify the cert | 13:17 |
| *** skramaja has quit IRC | 13:17 | |
| *** bjolo has quit IRC | 13:18 | |
| Reepicheep | I started to do it with ad-hoc commands from a bash script but I figured if I half way there I may as well try to do it more proper | 13:18 |
| egonzalez | Reepicheep, why not copy the file to /etc/kolla/haproxy/ in haproxy nodes and restart or reconfigure haproxy containers? | 13:19 |
| egonzalez | when the container is restarted, config.json will replace old .pem inside containers | 13:19 |
| Reepicheep | oh.. so I can skip the "Copy haproxy.pem to existing container"? | 13:20 |
| * Reepicheep admits this is the first time I have actually attempted to write any ansible | 13:21 | |
| egonzalez | Reepicheep, have 2 options, copy the pem into /etc/kolla/certificates and do a reconfigure (all this in deploy node) | 13:21 |
| egonzalez | or copy pem to /etc/kolla/haproxy on each haproxy node and manually restart containers | 13:21 |
| Reepicheep | egonzalez: that second option is what I was attempting to do | 13:22 |
| Reepicheep | calling kolla-ansible reconfigure or deploy take way to long even with the --tag haproxy | 13:22 |
| Reepicheep | I could replace the kolla_docker: command with "command: docker restart haproxy" | 13:23 |
| Reepicheep | but I thought eventually if I make this more generic with ansible parameters someone else may want to use it to do something similar | 13:25 |
| *** salv-orl_ has joined #openstack-kolla | 13:28 | |
| egonzalez | Reepicheep, estrange, haproxy reconfigures not take too long in my env with --tags haproxy | 13:29 |
| egonzalez | Reepicheep, where hangs? | 13:29 |
| *** rhallisey has quit IRC | 13:30 | |
| *** rhallisey has joined #openstack-kolla | 13:30 | |
| Reepicheep | hmm. mine always seems to be slow. I timed a deploy --tags haproxy last night and it took 15 minutes .. the full deploy was 35minutes | 13:30 |
| *** salv-orlando has quit IRC | 13:31 | |
| Reepicheep | for my needs I need it down to preferably less then 30 seconds :| | 13:31 |
| *** ducttap__ has quit IRC | 13:32 | |
| Reepicheep | maybe my kolla machine needs more resources.. I'm not sure what the bottleneck is | 13:32 |
| egonzalez | Reepicheep, in which task hangs? | 13:33 |
| Reepicheep | egonzalez: I'm trying to figure that out now | 13:34 |
| *** ducttape_ has joined #openstack-kolla | 13:35 | |
| Reepicheep | So I'm watching a redeploy and it has been about a minute before it even gets trough prechecks gathering facts | 13:37 |
| Reepicheep | sorry s/redeploy/reconfigure/ ^ ^ | 13:37 |
| egonzalez | Reepicheep, how many nodes you have? | 13:37 |
| egonzalez | maybe take too long to stablish ssh connection | 13:38 |
| Reepicheep | currently I'm at 20 | 13:38 |
| Reepicheep | I still have more to add | 13:38 |
| Reepicheep | let me do some timing of ssh connections | 13:38 |
| sdake | morning | 13:39 |
| *** daidv_ has joined #openstack-kolla | 13:40 | |
| egonzalez | morning sdake | 13:40 |
| Reepicheep | egonzalez: I'm not thinking it's ssh: http://paste.openstack.org/show/611201/ | 13:41 |
| openstackgerrit | Andy Smith proposed openstack/kolla-ansible master: Add qdrouterd role for messaging infrastructure component https://review.openstack.org/469462 | 13:41 |
| Reepicheep | that is a random sample of my nodes | 13:42 |
| egonzalez | Reepicheep, a reconfigure --tags haproxy (no load in the env) | 13:42 |
| egonzalez | real0m36.390s | 13:42 |
| egonzalez | user0m26.812s | 13:42 |
| egonzalez | sys0m8.804s | 13:42 |
| Reepicheep | how many nodes? | 13:43 |
| *** rwellum has joined #openstack-kolla | 13:43 | |
| egonzalez | 3, im not rich ;) | 13:43 |
| *** rwallner has quit IRC | 13:43 | |
| *** GonZo2000 has quit IRC | 13:44 | |
| sdake | Reepicheep which part of the world are you in | 13:44 |
| Reepicheep | interestingly my reconfigure failed after 4 minutes :( | 13:44 |
| Reepicheep | fatal: [luigi]: FAILED! => {"failed": true, "reason": "the file_name '/usr/share/kolla-ansible/ansible/start.yml' does not exist, or is not readable"} | 13:44 |
| Reepicheep | I'm in central US | 13:44 |
| egonzalez | but sometimes some VMs hangs in a weird state where take like 1m to stablish ssh connection | 13:45 |
| sdake | Reepicheep intenet speed and storage speed are the two factors | 13:45 |
| sdake | two performance factors | 13:45 |
| *** ducttape_ has quit IRC | 13:45 | |
| sdake | if your running in a vm dont' expect things to be speedy | 13:45 |
| Reepicheep | well my nodes are all in two racks right next to each other | 13:45 |
| *** ducttape_ has joined #openstack-kolla | 13:45 | |
| Reepicheep | the kolla machine is the only VM | 13:46 |
| sdake | i see | 13:46 |
| *** eaguilar has joined #openstack-kolla | 13:46 | |
| Reepicheep | and my internet probably isn't the issue I have multilple multi gig links leaving this building | 13:46 |
| Reepicheep | with pretty good latancy considering I'm in the middle of Nebraska ;) | 13:47 |
| sdake | ssdstorage for /var/lib/docker? | 13:47 |
| sdake | and using a proper backing store filesystem? | 13:47 |
| Reepicheep | yes on all but a few nodes | 13:47 |
| Reepicheep | the only nodes that do not have all SSD are the storage nodes and those are only for the ceph osd | 13:48 |
| sdake | cool | 13:48 |
| rwellum | Morning sdake | 13:48 |
| sdake | not using lvm loopback i assuem? | 13:48 |
| sdake | morning rwellum | 13:48 |
| sdake | Reepicheep kolla takes 20 minutes to eploy on 120 nodes | 13:49 |
| openstackgerrit | Andy Smith proposed openstack/kolla-ansible master: Add support for hybrid messaging backends https://review.openstack.org/468966 | 13:49 |
| Reepicheep | nope my /var/lib/docker is all on btrfs with at least it's own partition if not it's own drive (all ssds) | 13:49 |
| *** jtriley has joined #openstack-kolla | 13:49 | |
| sdake | Reepicheep sounds like your doing everything right :) | 13:49 |
| rwellum | sdake: I didn't get to backing out that patch, sbezverk had some questions and the fact that it was more than one fix in the patch stopped me - then I had to leave for the evening. Can we discuss if/when sbezverk is available? | 13:50 |
| *** ansmith has joined #openstack-kolla | 13:50 | |
| sdake | rwellum sure | 13:50 |
| Reepicheep | well I have another issue now. my reconfigure did fail.. do you see the error I posted? | 13:50 |
| sdake | rwellum the patch looked like one change to me | 13:50 |
| sdake | Reepicheep dont know why that occured sorry | 13:50 |
| rwellum | sdake: https://review.openstack.org/#/c/466536/ - the description has three things: | 13:52 |
| rwellum | https://www.irccloud.com/pastebin/kSPzcP1h/ | 13:52 |
| *** powerd has joined #openstack-kolla | 13:52 | |
| Reepicheep | hmm. it was telegraf it failed on .. let me disable that | 13:52 |
| *** janki has quit IRC | 13:54 | |
| *** rwallner has joined #openstack-kolla | 13:54 | |
| Reepicheep | should it run the telegraf playbooks when I used --tag haproxy? | 13:54 |
| *** janki has joined #openstack-kolla | 13:54 | |
| sbezverk | rwellum : I am working on additional check which will check nova-api for registered compute resource before starting cell job | 13:55 |
| egonzalez | Reepicheep, there is a known issue with telegraf reconfiguration | 13:56 |
| egonzalez | let me find the review | 13:56 |
| *** sgrasley has joined #openstack-kolla | 13:56 | |
| blallau | https://review.openstack.org/#/c/464039/ | 13:57 |
| Reepicheep | egonzalez: thanks.. I'm going to just disable it for now.. actually I'm disabling everything but the bare min until I get my slowness figured out | 13:57 |
| egonzalez | Reepicheep, was merged yesterday the backport https://review.openstack.org/#/c/450846/2 | 13:57 |
| Reepicheep | ok.. I my move to stable/ocata instead of the kolla-ansible 4.0.1 that pip installed | 13:59 |
| *** caoyuan has joined #openstack-kolla | 14:00 | |
| *** ducttape_ has quit IRC | 14:01 | |
| *** emccormick has joined #openstack-kolla | 14:01 | |
| *** ducttape_ has joined #openstack-kolla | 14:01 | |
| *** janki has quit IRC | 14:02 | |
| *** ducttape_ has quit IRC | 14:04 | |
| *** manheim has joined #openstack-kolla | 14:04 | |
| *** mnasiadka has quit IRC | 14:05 | |
| *** ducttape_ has joined #openstack-kolla | 14:06 | |
| *** korzen has quit IRC | 14:07 | |
| *** ducttape_ has quit IRC | 14:08 | |
| *** manheim has quit IRC | 14:08 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Use labels in pv/pvc https://review.openstack.org/469870 | 14:10 |
| *** sgrasley has quit IRC | 14:12 | |
| *** sgrasley has joined #openstack-kolla | 14:12 | |
| egonzalez | can I get eyes on cinder binary ubuntu fix https://review.openstack.org/#/c/457993/ zun deploy fix https://review.openstack.org/#/c/456256/ | 14:14 |
| mewald | Is ist possible cinder-api container is broken? | 14:16 |
| *** sgrasley has quit IRC | 14:16 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Use labels in pv/pvc https://review.openstack.org/469870 | 14:16 |
| mewald | I build on stable ocata and get "/usr/local/bin/kolla_start: line 24: exec: cinder-api: not found" | 14:16 |
| *** sgrasley has joined #openstack-kolla | 14:17 | |
| egonzalez | mewald, yep only ubuntu binary is broken | 14:17 |
| egonzalez | mewald, ubuntu packages removed support for cinder-api command and only works under apache now :/ | 14:17 |
| mewald | Do we have a bug report for that? I couldnt finde any that would match | 14:19 |
| egonzalez | mewald, https://bugs.launchpad.net/kolla/+bug/1676751 | 14:20 |
| openstack | Launchpad bug 1676751 in kolla-ansible pike "cinder-api keeps restarting after the deployment" [Critical,In progress] - Assigned to Eduardo Gonzalez (egonzalez90) | 14:20 |
| egonzalez | and many marked as duplicated of this | 14:20 |
| openstackgerrit | Merged openstack/kolla-ansible master: Avoid facts gathered twice when deploying with --limit https://review.openstack.org/464963 | 14:21 |
| mewald | egonzalez: your patch definitely has enough +2 :D What's blocking? | 14:22 |
| egonzalez | mewald, only have +1 atm https://review.openstack.org/#/c/457993/ | 14:22 |
| mewald | ah I was looking at the container only | 14:23 |
| mewald | we have to get this out fast, broken cinder is really really bad :D | 14:23 |
| hrw | 206 debian/source images built in 51 minutes | 14:24 |
| hrw | nice | 14:24 |
| egonzalez | hrw, nice | 14:24 |
| hrw | egonzalez: on aarch64 machine ;D | 14:25 |
| hrw | without ssd ;D | 14:25 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Fix zun deployment https://review.openstack.org/456256 | 14:26 |
| Reepicheep | it seems to be the "Gathering Facts" that is slowest | 14:33 |
| Reepicheep | on almost all roles | 14:33 |
| -openstackstatus- NOTICE: python-setuptools 36.0.1 has been released and now making its way into jobs. Feel free to 'recheck' your failures. If you have any problems, please join #openstack-infra | 14:33 | |
| mewald | egonzalez: Also your Docker container patch was not backported into ocata ... | 14:34 |
| *** openstackgerrit has quit IRC | 14:34 | |
| egonzalez | mewald, yep, i dont want to backport both changes until master are merged | 14:35 |
| Reepicheep | do the facts get writen to the kolla machine? could that be the bottle neck? | 14:35 |
| mewald | yeah makes sense | 14:35 |
| mewald | I am testing your patches right now in my lab | 14:36 |
| egonzalez | mewald, thanks, let me know any issue | 14:36 |
| *** jaosorior has quit IRC | 14:36 | |
| *** openstackgerrit has joined #openstack-kolla | 14:38 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/newton: [DNM]: test kolla newton branch https://review.openstack.org/469879 | 14:38 |
| hrw | am I the only one who thinks that hub.docker.com documentation suxx? | 14:38 |
| *** kbaegis has joined #openstack-kolla | 14:41 | |
| *** jamesbenson has joined #openstack-kolla | 14:44 | |
| *** ducttape_ has joined #openstack-kolla | 14:45 | |
| jamesbenson | Good morning everyone | 14:45 |
| hrw | hi jamesbenson | 14:45 |
| *** sean-k-m1 is now known as sean-k-mooney | 14:47 | |
| jamesbenson | inc0, are you free? | 14:47 |
| hrw | https://hub.docker.com/r/haerwu/kolla-aarch64-debian-source-base/ - does such naming has sense? | 14:47 |
| kolla-slack | <inc0> Barely awake:) | 14:47 |
| *** hongbin has joined #openstack-kolla | 14:47 | |
| jamesbenson | ^_^ | 14:48 |
| rwellum | sbezverk: sounds good - let me know if i can help? | 14:48 |
| hrw | ~curse docker cli | 14:52 |
| mewald | egonzalez: the patches work like a charm, I will +1 although it's worth nothing :D | 14:52 |
| egonzalez | mewald, every review are good | 14:53 |
| *** chas has joined #openstack-kolla | 14:53 | |
| hrw | what you guys use to mass tag/push images to registry? | 14:54 |
| hrw | mewald: every review matters | 14:54 |
| mewald | reminds me of the "black lives matter" thing :D | 14:55 |
| jamesbenson | hrw: kolla-ansible pull -i multinode | 14:55 |
| hrw | jamesbenson: push not pull | 14:56 |
| jamesbenson | ah sorry, I thought you wanted to pull the images to the machines.... | 14:56 |
| hrw | jamesbenson: I built 206 images and now have to tag them and push | 14:56 |
| jamesbenson | hrw: wow, just a few.... | 14:56 |
| hrw | jamesbenson: current solution was "docker images|grep kolla-aarch64|sort|vim -, then make a script for tag and next one for push | 14:57 |
| jamesbenson | probably a good approach.... | 14:57 |
| *** chas has quit IRC | 14:58 | |
| hrw | jamesbenson: imho 'docker push kolla-aarch64/*:4.0.0' would be better ;D | 14:59 |
| *** mewald has quit IRC | 14:59 | |
| openstackgerrit | Merged openstack/kolla stable/ocata: Remove sudo operations that are no longer necessary https://review.openstack.org/468655 | 15:00 |
| *** manheim has joined #openstack-kolla | 15:02 | |
| *** reidrac_ has joined #openstack-kolla | 15:02 | |
| *** reidrac has quit IRC | 15:02 | |
| *** reidrac has joined #openstack-kolla | 15:05 | |
| *** manheim has quit IRC | 15:06 | |
| jamesbenson | So I'm having an issue with Fernet keys failing in HA environment.... it's yelling at me for a rsync protocol version mismatch, is your shell clean? | 15:09 |
| jamesbenson | It's under the TASK [keystone : Initialise fernet key authentication] | 15:10 |
| *** zhubingbing has joined #openstack-kolla | 15:12 | |
| zhubingbing | hi guys | 15:12 |
| egonzalez | sup zhubingbing | 15:13 |
| *** Serlex has left #openstack-kolla | 15:14 | |
| *** Serlex has quit IRC | 15:14 | |
| *** kbaegis has quit IRC | 15:14 | |
| *** sgrasley has left #openstack-kolla | 15:15 | |
| zhubingbing | hi egonzalez | 15:15 |
| zhubingbing | ;) | 15:15 |
| jamesbenson | hi zhubingbing | 15:15 |
| zhubingbing | hi jamesbenson | 15:15 |
| zhubingbing | ；） | 15:15 |
| *** manheim has joined #openstack-kolla | 15:17 | |
| *** klindgren__ has joined #openstack-kolla | 15:18 | |
| *** klindgren_ has quit IRC | 15:21 | |
| jamesbenson | I need to manually run a command, what's keystone's groupname? keystone? | 15:21 |
| zhubingbing | kolla-ansible post-deploy | 15:22 |
| zhubingbing | source /etc/kolla/admin-openrc | 15:22 |
| kfox1111 | morning | 15:24 |
| *** reidrac_ has quit IRC | 15:29 | |
| *** reidrac has quit IRC | 15:29 | |
| *** ansiwen|lunch is now known as ansiwen | 15:29 | |
| *** reidrac has joined #openstack-kolla | 15:29 | |
| *** rmart04 has quit IRC | 15:30 | |
| *** openstackgerrit has quit IRC | 15:33 | |
| inc0 | ok, now I'm back | 15:35 |
| jamesbenson | morning inc0 | 15:35 |
| jamesbenson | Do you have time to continue helping to debug that fernet issue from yesterday? | 15:36 |
| inc0 | hrw repushing images takes some "bash-fu" | 15:36 |
| inc0 | jamesbenson: happily, but I'm not good at fernet | 15:36 |
| inc0 | but let's try | 15:36 |
| sbezverk | inc0: do you have by any chance a working example of curl requesting keystone for a token? | 15:36 |
| jamesbenson | well currently the docker image is constantly rebooting and I can't find anything in the logs besides that "rsync protocol version mismatch, is your shell clean?" | 15:36 |
| inc0 | sbezverk: no | 15:37 |
| sbezverk | kfox1111: good morning | 15:37 |
| sbezverk | inc0: ok will keep looking then | 15:37 |
| inc0 | jamesbenson: in docker logs? | 15:37 |
| jamesbenson | yeah | 15:37 |
| inc0 | sbezverk: if you do nova --debug list it will print out curls | 15:38 |
| sdake | sbezverk if that works, that sounds like a wonderful solution! | 15:38 |
| inc0 | including keystone | 15:38 |
| sdake | sbezverk re your msg about 1 hr go | 15:38 |
| sbezverk | inc0: thanks for the idea!! | 15:38 |
| sdake | sup kfox1111 | 15:38 |
| sdake | sup inc0 | 15:39 |
| sdake | sup sbezverk :) | 15:39 |
| jamesbenson | http://paste.openstack.org/show/611219/ | 15:39 |
| jamesbenson | inc0: and remember, this was a completely clean environment, just redeployed the OS's.... | 15:41 |
| *** manheim has quit IRC | 15:41 | |
| inc0 | jamesbenson: try googling this error, first time I see it | 15:42 |
| inc0 | brb, food | 15:42 |
| *** manheim has joined #openstack-kolla | 15:42 | |
| *** chas has joined #openstack-kolla | 15:43 | |
| *** Mannheim has joined #openstack-kolla | 15:43 | |
| *** unicell1 has joined #openstack-kolla | 15:44 | |
| *** unicell has quit IRC | 15:44 | |
| jamesbenson | inc0: I might have found a solution... it's been a while since I've contributed to openstack, and never to kolla. I'd love to help build/add to the pre-tests to help produce more successful deployments out of the box, any suggestions? Or should I just do bug fixes, etc? | 15:45 |
| *** manheim has quit IRC | 15:46 | |
| *** openstackgerrit has joined #openstack-kolla | 15:47 | |
| openstackgerrit | Juan J. MartÃnez proposed openstack/kolla master: Remove dead code related to the Vagrant dev environment https://review.openstack.org/469930 | 15:47 |
| *** matrohon has quit IRC | 15:49 | |
| *** zhubingbing has quit IRC | 15:58 | |
| *** ntpttr has quit IRC | 15:59 | |
| *** iniazi_ has joined #openstack-kolla | 16:00 | |
| Reepicheep | how important is it to configure ansible with piplining=True and forks? | 16:01 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix outdated keystone-paste.ini file https://review.openstack.org/469934 | 16:02 |
| *** bmaltais has joined #openstack-kolla | 16:03 | |
| *** iniazi has quit IRC | 16:03 | |
| bmaltais | Hi. I am trying to figure out how to apply kvm/qemu config on a per node basis in kolla-ansible. | 16:05 |
| bmaltais | I am putting the needed changes in file /etc/kolla/config/nova/{node name}/nova-compute.conf | 16:05 |
| egonzalez | bmaltais, per node or per service? | 16:06 |
| bmaltais | Is it where it is supposed to go? | 16:06 |
| egonzalez | per node is /etc/kolla/config/nova/{node name}/nova.conf | 16:06 |
| bmaltais | Per node. Some need to be set as qemu | 16:06 |
| bmaltais | ok... so not nova-compute.conf | 16:06 |
| bmaltais | I tried to use the same file name as the "global" way of setting it | 16:07 |
| *** egonzalez has quit IRC | 16:09 | |
| *** caoyuan has quit IRC | 16:09 | |
| *** lucasxu has quit IRC | 16:10 | |
| *** zhubingbing has joined #openstack-kolla | 16:10 | |
| bmaltais | egonzalez: Working. I had two issues. file name and was using "node1" instead as "node1.fqdn.name" for the folder... | 16:12 |
| *** mgoddard has quit IRC | 16:12 | |
| *** ntpttr has joined #openstack-kolla | 16:13 | |
| jamesbenson | inc0: no luck...I tried enabled ssh between the nodes; didn't fix it. rsync versions are the same as well... | 16:17 |
| *** chas has quit IRC | 16:18 | |
| *** powerd has quit IRC | 16:22 | |
| *** mnasiadka has joined #openstack-kolla | 16:23 | |
| *** sambetts is now known as sambetts|afk | 16:23 | |
| *** mnasiadka has quit IRC | 16:27 | |
| inc0 | kfox1111: around? | 16:28 |
| *** jascott1 has joined #openstack-kolla | 16:28 | |
| inc0 | jamesbenson: we use ssh in container | 16:28 |
| inc0 | there is keystone-ssh container | 16:28 |
| *** lucasxu has joined #openstack-kolla | 16:29 | |
| kfox1111 | inc0: meeting in 5 min. back at about 10:00. | 16:29 |
| kfox1111 | it quick? :) | 16:29 |
| inc0 | kk I was hoping you could help jamesbenson debug the thing | 16:29 |
| kfox1111 | ah. k. | 16:30 |
| inc0 | I don't know anything about keystone fernet | 16:30 |
| *** sayantan_ has joined #openstack-kolla | 16:30 | |
| kfox1111 | kk. I know a thing or two about it. but not how kolla-ansible distributes the keys. | 16:30 |
| kfox1111 | what is the problem? | 16:30 |
| *** lucasxu has quit IRC | 16:30 | |
| inc0 | I'll help on that front | 16:30 |
| inc0 | jamesbenson: ^ | 16:31 |
| inc0 | kfox1111: http://paste.openstack.org/show/611219/ | 16:31 |
| kfox1111 | that does not apear to be a fernet issue, but a distribution issue? | 16:32 |
| kfox1111 | rsync not working. | 16:32 |
| inc0 | yeah, my question would be, have you seen anything like that? | 16:32 |
| inc0 | does kolla-k8s solve distribution already? | 16:32 |
| kfox1111 | usually when rsync versions are two far out of sync. | 16:32 |
| kfox1111 | it says whta t he server is. what about the client version? | 16:33 |
| kfox1111 | k8s solves it a completely different way, using k8s natively for shipping. | 16:33 |
| kfox1111 | cant reuse it I'm afraid. | 16:33 |
| *** tonanhngo has joined #openstack-kolla | 16:33 | |
| kfox1111 | it does a k8s job to do the rotation that updates a k8s secret, and the updated secrets automaticaly get pushed by k8s to the pods. | 16:34 |
| *** blallau has quit IRC | 16:35 | |
| inc0 | right, we can use configmaps for distribution | 16:35 |
| inc0 | or secret rather | 16:35 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix outdated barbican-api-paste.ini file https://review.openstack.org/469939 | 16:36 |
| *** rwallner has quit IRC | 16:39 | |
| *** mnasiadka has joined #openstack-kolla | 16:40 | |
| *** mnasiadka has quit IRC | 16:40 | |
| inc0 | can I have more reviews please? https://review.openstack.org/#/c/466007/ | 16:41 |
| inc0 | this is really really important to me... | 16:41 |
| *** rwallner has joined #openstack-kolla | 16:41 | |
| jamesbenson | kfox1111: the clients are all the same version.. | 16:41 |
| *** hyakuhei has quit IRC | 16:41 | |
| *** hyakuhei has joined #openstack-kolla | 16:41 | |
| *** hyakuhei has quit IRC | 16:41 | |
| *** hyakuhei has joined #openstack-kolla | 16:41 | |
| inc0 | jamesbenson: can you check if ssh from inside keystone-ssh works? | 16:41 |
| *** rmart04 has joined #openstack-kolla | 16:42 | |
| *** rwallner_ has joined #openstack-kolla | 16:42 | |
| jamesbenson | r2-710-6:~# docker exec keystone_ssh "ssh r2-710-7" | 16:43 |
| jamesbenson | rpc error: code = 13 desc = invalid header field value "oci runtime error: exec failed: container_linux.go:247: starting container process caused \"exec: \\\"ssh r2-710-7\\\": executable file not found in $PATH\"\n" | 16:43 |
| jamesbenson | inc0: something like: ? | 16:43 |
| *** eaguilar has quit IRC | 16:43 | |
| *** rwallne__ has joined #openstack-kolla | 16:43 | |
| inc0 | hmm | 16:43 |
| inc0 | that doesn't look good | 16:44 |
| inc0 | hold on, let me deploy setup with fernet | 16:44 |
| jamesbenson | outside of the containers, I can ssh without issue.. | 16:44 |
| inc0 | yeah, fernet will use ssh in container | 16:44 |
| jamesbenson | I've got 3 controllers if that helps | 16:44 |
| jamesbenson | inc0 Do you want my multinode? | 16:45 |
| jamesbenson | globals? | 16:45 |
| *** rwallner has quit IRC | 16:45 | |
| inc0 | show me globals please | 16:46 |
| inc0 | I'll try exactlyu same env | 16:46 |
| *** unicell1 has quit IRC | 16:46 | |
| jamesbenson | https://github.com/JamesOBenson/Kolla-misc/blob/master/globals.yml | 16:47 |
| *** rwallner_ has quit IRC | 16:47 | |
| jamesbenson | https://github.com/JamesOBenson/Kolla-misc/blob/master/multinode | 16:48 |
| kfox1111 | back. meeting was short. | 16:49 |
| jamesbenson | inc0: Separating the components like that is okay, in the multinode | 16:49 |
| inc0 | yeah looks good | 16:50 |
| inc0 | hold on let me setup something very similar | 16:50 |
| *** rmart04 has quit IRC | 16:51 | |
| *** rmart04 has joined #openstack-kolla | 16:52 | |
| *** MasterOfBugs has joined #openstack-kolla | 16:52 | |
| *** shardy has quit IRC | 16:53 | |
| sbezverk | kfox1111: can you check this reply from keystone: paste.openstack.org/show/611227 | 16:53 |
| inc0 | jamesbenson: what images did you use? | 16:53 |
| jamesbenson | inc0 and kfox1111: in the fernet-node-synch.sh file, the rsync commands there don't seem to work manually on the system or keystone containers | 16:53 |
| jamesbenson | 4.0.0 | 16:53 |
| sbezverk | kfox1111: by some reason I do not get token id | 16:53 |
| inc0 | built by yourself of pulled from dockerhub? | 16:53 |
| jamesbenson | pulled from dockerhub | 16:54 |
| kfox1111 | sbezverk: looking | 16:54 |
| inc0 | mhm so you probably want to build them yourself;) | 16:54 |
| inc0 | dockerhub 4.0.0 are very old | 16:54 |
| jamesbenson | :-( | 16:54 |
| inc0 | we're working on clean pipeline to provide fresh images | 16:55 |
| jamesbenson | eta? | 16:55 |
| inc0 | in fact this is my personal #1 priority now | 16:55 |
| jamesbenson | *high five* | 16:55 |
| kfox1111 | sbezverk: do a -v on the curl and look at the return code? | 16:55 |
| inc0 | well, lots of moving pieces that involve more than just Kolla community | 16:55 |
| inc0 | so hard to tell | 16:55 |
| kfox1111 | did it succeed? | 16:55 |
| inc0 | for now, if you want fresh images you can use small tool I made | 16:56 |
| inc0 | https://github.com/inc0/kolla_dockerhub_pusher | 16:56 |
| sbezverk | 201 created | 16:56 |
| kfox1111 | hmm... weird. | 16:57 |
| kfox1111 | any error in the keystone logs? | 16:57 |
| *** sayantan_ has quit IRC | 16:57 | |
| inc0 | jamesbenson: install this thing and do kolla-dockerhub-pushed download --base ubuntu --type source --release ocata | 16:57 |
| sbezverk | kfox1111: can you check this reply from keystone: paste.openstack.org/show/611229 | 16:57 |
| *** sayantan_ has joined #openstack-kolla | 16:57 | |
| kfox1111 | sbezverk: is that the token in the response headers? | 16:58 |
| sbezverk | kfox1111: maybe it has something with fernet? | 16:58 |
| kfox1111 | X-Subject-Token? | 16:58 |
| kfox1111 | it looks about right to me. | 16:58 |
| jamesbenson | inc0: how do I install it? I've cloned it down | 16:59 |
| sbezverk | kfox1111: I see it, it is just different from documented api | 16:59 |
| kfox1111 | yeah. | 16:59 |
| kfox1111 | also looks like they may have a bug. | 16:59 |
| inc0 | jamesbenson: pip install -e . | 16:59 |
| inc0 | from dir | 16:59 |
| kfox1111 | tehy have the Vary: X-Auth-Token set but return X-Subject-Token. | 16:59 |
| kfox1111 | I think those shoudl match up. | 17:00 |
| kfox1111 | though doesn't llook like they turn on any caching anyway, so may be ignored all together. | 17:01 |
| jamesbenson | inc0: kolla-dockerhub-pushed command not found after the pip install | 17:01 |
| inc0 | pusher | 17:01 |
| sbezverk | kfox1111: I guess then I need to parse header and not returning json | 17:01 |
| inc0 | try kolla-doc<<tab>>;) | 17:01 |
| jamesbenson | ah got it | 17:01 |
| inc0 | what it will do is it'll pull latest ocata images built in gates | 17:02 |
| kfox1111 | sbezverk: this related to the previous topic we had about deps? | 17:02 |
| inc0 | so last week | 17:02 |
| Reepicheep | so it looks like my system takes about 5-8 seconds for each role when it is "Gathering Facts", everything else flys by | 17:02 |
| sbezverk | kfox1111: yes | 17:02 |
| inc0 | oh sorry, today;) | 17:02 |
| kfox1111 | sbezverk: you trying to avoid using the cli tools? | 17:03 |
| sbezverk | kfox1111: I did not wanted to depend on openstack tools | 17:03 |
| jamesbenson | yeah, downloading the tarball now... | 17:03 |
| kfox1111 | sbezverk: can be done but a bit hard to get all the bits right. | 17:03 |
| Reepicheep | that seems to be at least where my stuff is takin so long.. now I need to figure out why | 17:03 |
| inc0 | after that you can do kolla-dockerhub-pusher ls to see what do you have | 17:04 |
| inc0 | I think tag will be 4.0.2 | 17:04 |
| jamesbenson | ok | 17:04 |
| inc0 | so mind the tag | 17:04 |
| sbezverk | kfox1111: yeah, it might be problematic as they do not follow their own documentation | 17:04 |
| inc0 | make sure to put it into openstack_release in globals | 17:04 |
| kfox1111 | sbezverk: indeed. | 17:04 |
| kfox1111 | openstack does not make it easy on users to use. :/ | 17:05 |
| sbezverk | I guess relying on client tool moves responsibilityy ;) | 17:05 |
| kfox1111 | yup. :) | 17:05 |
| *** rmart04 has quit IRC | 17:06 | |
| kfox1111 | less code to maintain ourselves. | 17:06 |
| kfox1111 | and gets more testing then we would give it. | 17:06 |
| jamesbenson | inc0 if I find bugs in the code for that, how do you want me to let you know? through github? here? | 17:07 |
| sbezverk | kfox1111: sound good, will do it with cli | 17:09 |
| inc0 | jamesbenson: https://bugs.launchpad.net/kolla-ansible | 17:09 |
| *** lucasxu has joined #openstack-kolla | 17:12 | |
| *** salv-orl_ has quit IRC | 17:14 | |
| *** unicell has joined #openstack-kolla | 17:15 | |
| jamesbenson | inc0: thanks, finding a few small ones.. | 17:15 |
| *** jmccarthy has quit IRC | 17:15 | |
| *** dmsimard is now known as dmsimard|off | 17:16 | |
| inc0 | jamesbenson: there is good chance they're already fixed in master | 17:16 |
| *** chas has joined #openstack-kolla | 17:18 | |
| inc0 | ahh jamesbenson you were asking for bugs in dockerhub pusher?;) | 17:18 |
| kfox1111 | sbezverk: did you get an answer to your fernet question? | 17:18 |
| inc0 | github then please | 17:18 |
| jamesbenson | yeah | 17:19 |
| jamesbenson | sorry | 17:19 |
| kfox1111 | sbezverk: looking at it, it looks like k8s is trying to authenticate the client with a client ssl cert. | 17:19 |
| kfox1111 | sbezverk: which it shouldn't be doing if its using a service account token. | 17:19 |
| jamesbenson | inc0: thought that was an odd place for errors in the pusher | 17:19 |
| kfox1111 | seems like it may be rejecting the service account token for some reason. | 17:19 |
| kfox1111 | guessing rbac related? | 17:19 |
| inc0 | jamesbenson: so few things I forgot to mention | 17:20 |
| inc0 | it needs python3 | 17:20 |
| sbezverk | kfox1111: if you refer to my email, then strangely reboot cluster helped :( | 17:20 |
| openstackgerrit | Merged openstack/kolla stable/newton: Fix Creating Ceilometer MongoDB database failed https://review.openstack.org/456513 | 17:20 |
| inc0 | tarball lib in python2.7 is no bueno:() | 17:20 |
| kfox1111 | sbezverk: weird... selinux get enabled by and update? or maybe token rotation failed. | 17:21 |
| *** daidv_ has quit IRC | 17:21 | |
| *** Slower has quit IRC | 17:23 | |
| jamesbenson | python3 is installed by default in ubuntu16.. Got 3.5.1-3 | 17:24 |
| jamesbenson | inc0: is it smart enough to use 3 vs 2 thought? python -V pulled python2 | 17:24 |
| inc0 | negative | 17:25 |
| *** eaguilar has joined #openstack-kolla | 17:25 | |
| jamesbenson | ok, updated an alias... | 17:26 |
| jamesbenson | once it's complete with the pull, and issue an ls what happens? | 17:26 |
| kfox1111 | inc0: all the more reason to containerize stuff. :) | 17:28 |
| kfox1111 | include the right python in the container and no more fighting with versions. :) | 17:28 |
| inc0 | that actually makes sense | 17:29 |
| inc0 | I'll make Dockerfile for this thing:) | 17:29 |
| kfox1111 | hehe. :) | 17:29 |
| kfox1111 | containers turn everythign inside out. :) | 17:29 |
| jamesbenson | yeah, sort of fighting small bugs in this, but I love the idea/concept.. a docker might help a bunch | 17:29 |
| inc0 | jamesbenson: it's quite literally 2 days of me hacking around | 17:29 |
| jamesbenson | and thank you so much inc0 and kfox1111 for all of the help! | 17:30 |
| jamesbenson | damn impressive for 2 days. :-) | 17:30 |
| *** eaguilar has quit IRC | 17:30 | |
| inc0 | it's not that complex | 17:30 |
| jamesbenson | I consider myself a programmer, but would be able to hack this in 2 days. | 17:30 |
| inc0 | it really downloads a file, untars it | 17:30 |
| jamesbenson | well, looks more impressive then :-p | 17:30 |
| inc0 | and runs docker registry with -b /path/to/untar:/var/lib/registry | 17:30 |
| inc0 | jamesbenson: all because of progress bars | 17:31 |
| inc0 | everything is better with progress bar | 17:31 |
| jamesbenson | yeppers | 17:31 |
| jamesbenson | inc0: | 17:31 |
| jamesbenson | Conflict. The container name "/kolla-registry" is already in use by container | 17:31 |
| inc0 | you can override this I think | 17:32 |
| jamesbenson | ? | 17:32 |
| jamesbenson | rm the container? | 17:32 |
| inc0 | that will definetly help;) | 17:32 |
| jamesbenson | inc0: Want me to assist with patches on this on your github page? | 17:33 |
| jamesbenson | inc0: after all of this is done, do you think this will work? Did the deploy succeed on your side | 17:34 |
| inc0 | if you want, I'll be happy to:) | 17:34 |
| inc0 | downloading registry now, I found few bugs in this tool myself;) | 17:35 |
| *** rwallne__ has quit IRC | 17:35 | |
| jamesbenson | in python2 at least, /tmp/kolla dir needs to be verified exists; and numeric_owner=True in tarballs it didn't like... | 17:36 |
| jamesbenson | issuing the ls command it doesn't like because the _manifests/tags doesn't exist | 17:36 |
| *** rwallner has joined #openstack-kolla | 17:37 | |
| jamesbenson | inc0: ls giving me a IndexError: list index out of range | 17:38 |
| inc0 | jamesbenson: that's exactly why you need python3:) | 17:39 |
| jamesbenson | I'm using 3 now. :-). still giving me it | 17:39 |
| inc0 | alias won't do I think | 17:39 |
| inc0 | but you can always manually untar it | 17:39 |
| inc0 | go to /tmp/kolla | 17:39 |
| inc0 | and untar tarball | 17:39 |
| *** rwallner has quit IRC | 17:42 | |
| *** mgoddard has joined #openstack-kolla | 17:42 | |
| jamesbenson | the first command works now...., it's the ls that fails | 17:44 |
| *** kelv_ has joined #openstack-kolla | 17:44 | |
| jamesbenson | I did the alias in my bashrc and re sourced it | 17:44 |
| *** Manheim has joined #openstack-kolla | 17:46 | |
| *** eaguilar has joined #openstack-kolla | 17:46 | |
| jamesbenson | inc0: i know why it's yelling, it's because of the lokolla directory | 17:48 |
| jamesbenson | that's where it stops the tagging in the ls function. | 17:48 |
| inc0 | lokolla is our "gate" registry | 17:48 |
| *** Mannheim has quit IRC | 17:48 | |
| inc0 | mv lokolla kolla | 17:48 |
| kfox1111 | inc0: any reason to not fix that at the gate? | 17:49 |
| kfox1111 | have it do kolla instead of lokolla? | 17:49 |
| inc0 | there is reason we did it in the gate in the first place | 17:49 |
| kfox1111 | yeah? | 17:50 |
| inc0 | to ensure that images are pulled from local registry not dockerhub, can't remember exact scenario tho | 17:50 |
| kfox1111 | ah. could that be resolved by just double tagging? | 17:50 |
| *** salv-orlando has joined #openstack-kolla | 17:50 | |
| kfox1111 | tag the images both kolla and lokolla? | 17:51 |
| kfox1111 | meeting. bbiab. | 17:51 |
| inc0 | well, this can be resolved while retagging to push | 17:51 |
| inc0 | it's really internal | 17:51 |
| jamesbenson | running into a bunch of issues, do you have a manual way of doing it? | 17:52 |
| jamesbenson | or guide? | 17:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/469567 | 17:52 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/469568 | 17:52 |
| inc0 | so you untarred it? | 17:52 |
| *** zhubingbing has quit IRC | 17:53 | |
| inc0 | but yeah, I see issues with fernet myself | 17:53 |
| inc0 | different issues tho | 17:53 |
| jamesbenson | inc0: ? | 17:54 |
| jamesbenson | yes, untarred | 17:55 |
| inc0 | 6docker run -d --name registry --restart=always -p 4000:5000 -v /tmp/kolla/registry/:/var/lib/registry registry:2 | 17:56 |
| jamesbenson | k | 17:57 |
| *** ducttap__ has joined #openstack-kolla | 18:00 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding extra condition for simple cell0 create job https://review.openstack.org/469963 | 18:01 |
| *** ducttape_ has quit IRC | 18:04 | |
| *** rwallner has joined #openstack-kolla | 18:07 | |
| rwellum | sbezverk: if the cell0 issue doesn't show up in the gates anyway - then the gate tests for #469963 won't reveal whether this is fixed? | 18:11 |
| *** eaguilar has joined #openstack-kolla | 18:12 | |
| *** mgoddard has quit IRC | 18:13 | |
| *** eaguilar has quit IRC | 18:17 | |
| hrw | inc0: noticed ;D | 18:18 |
| hrw | inc0: building 206 images: 51 minutes. pushing them: 164 minutes (+5 for scripting) | 18:18 |
| *** ducttape_ has joined #openstack-kolla | 18:19 | |
| *** ducttap__ has quit IRC | 18:22 | |
| *** rwallner has quit IRC | 18:23 | |
| *** salv-orlando has quit IRC | 18:25 | |
| jamesbenson | inc0: I verified my registry is there (curl -X GET http://10.245.122.31:5000/v2/kolla/kolla/ubuntu-source-keystone/tags/list) and looked at the tag of keystone, 4.0.2.... | 18:26 |
| jamesbenson | trying to figure out what I need to do next for the local registry... modify the docker.server, /etc/default/docker, and the restart the service? | 18:27 |
| *** rwallner has joined #openstack-kolla | 18:29 | |
| sbezverk | rwellum: the condition will be bullet proof if it passes it will guarantee that you will not hit it.. | 18:29 |
| sdake | kfox1111 i dont know how a storage provider works, and i've read numerous pv/pvc docs and blogs etc | 18:30 |
| sdake | kfox1111 mind giving me a quick explination? | 18:30 |
| *** eaguilar has joined #openstack-kolla | 18:30 | |
| kfox1111 | ok. but not sure which part your confused about. what are you wanting to know? | 18:31 |
| sdake | you said use a storage provider instead of a label? | 18:31 |
| sdake | i understand how labels select things | 18:31 |
| kfox1111 | oh. | 18:31 |
| sdake | not storage providers | 18:31 |
| kfox1111 | not 100% sure on this one, but pv's are assigned to storage providers. Some are dynamic, | 18:32 |
| kfox1111 | where there is something watching on that for creation requests and deleting them when they are done. | 18:32 |
| kfox1111 | I think there is a static verision too. | 18:32 |
| trozet | mgkwill: hi | 18:32 |
| kfox1111 | where an admin can create pv's that users can ask for a particular type. | 18:32 |
| kfox1111 | like you may create some 'fast' and some 'slow' pv's, that users can request from. | 18:33 |
| kfox1111 | I'm thinking, for the use case your trying to cover, maybe we make a 'kolla' type. | 18:33 |
| sdake | yes - the use csae i want is to make sure mariadb maps to mariadb every time | 18:33 |
| kfox1111 | so a random pvc request won't ask for the 'kolla' pool. | 18:33 |
| kfox1111 | just the ones from kolla. | 18:34 |
| kfox1111 | that shoudl allow the names to always match up I think. | 18:34 |
| sbezverk | sdake: I suspect main issue will happen when we start having multiple mariadb or rabbitmq, your labeling system will fail.. | 18:34 |
| kfox1111 | sbezverk: true. element name will need to go along with it. | 18:34 |
| kfox1111 | the mariadb service chart would do pv/pvc/svc/statefulset all with the same element name | 18:35 |
| sdake | at pressent with the serach path, isn't there only one mariadb that may be deployed? | 18:36 |
| kfox1111 | each instantiation of the chart would change the elment name to ensure uniqueness. | 18:36 |
| sbezverk | kfox1111: right, but I am not sure adding complexity to pv naming is really worth it, anonymous PV sounds better to me ;) | 18:36 |
| sdake | anonymous pv is not good - as if i delte the pvc, and then later recreate it, it connects to hte wrong thing in some circumstances | 18:36 |
| * kfox1111 shrugs. | 18:36 | |
| kfox1111 | sdake: you should never delete a pvc unless your done done with it. | 18:37 |
| kfox1111 | there is not a garantee you can ever get the data back after that event. | 18:37 |
| kfox1111 | same as with cinder. | 18:37 |
| sbezverk | sdake: right and then we need to recycle if this service is really gone.. | 18:37 |
| kfox1111 | sbezverk: right now we're in a bad state I think. | 18:39 |
| kfox1111 | we half implement pv creation. | 18:39 |
| kfox1111 | if we want to do it right by your way, we should drop pv microservices entirely. | 18:39 |
| kfox1111 | or we should fix them so they work right, which is what sdake's trying to do. | 18:40 |
| kfox1111 | I'm kind of good either way, as long as its optional. | 18:40 |
| trozet | inc0, mgkwill : there is some issue where for me, opendaylight doesnt start with the dumb-init bash shell when the exec runs in kolla start. It will run if i just use a regular bash shell though | 18:40 |
| sbezverk | kfox1111: I would vote for excluding PV from microservices | 18:41 |
| sbezverk | as this most likely the optiom will be used in REAL production | 18:41 |
| kfox1111 | I'm in that boat, so I don't disagree with that. | 18:42 |
| mgkwill | trozet: hmm. weird. I don't know if I have any helpful info there. | 18:45 |
| trozet | mgkwill: well how do you start it in kolla-ansible? | 18:45 |
| mgkwill | trozet: I wonder what the difference is between your start and kolla-ansible start. | 18:45 |
| mgkwill | trozet: I'll have to look but I have to head to offsite. Will get back to you later. | 18:46 |
| trozet | mgkwill: exactly | 18:46 |
| trozet | mgkwill: ok np | 18:46 |
| *** emccormick has quit IRC | 18:48 | |
| *** chas has quit IRC | 18:55 | |
| *** chas has joined #openstack-kolla | 18:55 | |
| *** egonzalez has joined #openstack-kolla | 18:57 | |
| trozet | does anyone else know what dumb-init does in kolla start? egonzalez ? | 18:58 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding extra condition for simple cell0 create job https://review.openstack.org/469963 | 19:00 |
| *** chas has quit IRC | 19:03 | |
| egonzalez | trozet good explanation https://engineeringblog.yelp.com/2016/01/dumb-init-an-init-for-docker.html | 19:05 |
| trozet | egonzalez: yeah i found this one and was just reading it https://github.com/Yelp/dumb-init | 19:06 |
| trozet | egonzalez: but the opendaylight start script launches a java process, and for some reason this doesnt work with the dumb-init | 19:06 |
| egonzalez | trozet odl server starts with mgkwill's ansible change | 19:07 |
| trozet | egonzalez: it tries to run another exec inside of the odl start script, i wonder if that is the issue | 19:08 |
| trozet | egonzalez: yeah i was having trouble finding where in his patch the opendaylight start command is | 19:08 |
| egonzalez | trozet line 2 https://review.openstack.org/#/c/416367/57/ansible/roles/opendaylight/templates/opendaylight.json.j2 | 19:09 |
| trozet | egonzalez: oh he doesnt use start, he just starts karaf directly...let me try that | 19:10 |
| egonzalez | trozet yeah, we use to put the command to execute the process directly | 19:10 |
| openstackgerrit | Artom Lifshitz proposed openstack/kolla master: DNM: testing Nova fix for 1694666 https://review.openstack.org/469980 | 19:11 |
| egonzalez | same as other services | 19:11 |
| trozet | egonzalez: ok cool, it started for me | 19:11 |
| *** Pavo has quit IRC | 19:11 | |
| trozet | egonzalez: let me try it with full openstack deployment now, thanks for your help! | 19:12 |
| *** sayantan_ has quit IRC | 19:20 | |
| *** sayantan_ has joined #openstack-kolla | 19:27 | |
| *** sayantan_ has quit IRC | 19:32 | |
| *** dave-mccowan has quit IRC | 19:32 | |
| *** sayantan_ has joined #openstack-kolla | 19:40 | |
| *** ducttape_ has quit IRC | 19:42 | |
| *** sayantan_ has quit IRC | 19:44 | |
| *** dave-mccowan has joined #openstack-kolla | 19:46 | |
| *** randylev has joined #openstack-kolla | 19:47 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding extra condition for simple cell0 create job https://review.openstack.org/469963 | 19:52 |
| *** bmaltais has quit IRC | 19:54 | |
| *** ducttape_ has joined #openstack-kolla | 19:54 | |
| *** salv-orlando has joined #openstack-kolla | 19:55 | |
| *** rwallner has quit IRC | 19:58 | |
| *** sayantan_ has joined #openstack-kolla | 19:59 | |
| *** klowse1 has left #openstack-kolla | 19:59 | |
| *** rwallner has joined #openstack-kolla | 20:02 | |
| *** spsurya has quit IRC | 20:03 | |
| jamesbenson | very odd fail in the prechecks, TASK [rabbitmq : fail] : Hostname has to resolve to IP address of api_interface | 20:06 |
| *** rwallner has quit IRC | 20:07 | |
| *** sayantan_ has quit IRC | 20:15 | |
| *** sayantan_ has joined #openstack-kolla | 20:16 | |
| *** mgoddard has joined #openstack-kolla | 20:16 | |
| Manheim | they to put the IP in /etc/hosts | 20:22 |
| Manheim | try* | 20:23 |
| Manheim | i remember I had this issue in older releases | 20:23 |
| *** dave-mccowan has quit IRC | 20:32 | |
| jamesbenson | yeah, debugging it now, they IP's are all in the /etc/hosts... the IP's seem to be there in the rabbitmq_hostnames.results, the ansible function "{{ hostvars[item['item']]['ansible_' + hostvars[item['item']]['api_interface']]['ipv4']['address'] }}" seems to have an error in it.... | 20:38 |
| *** madhukar has joined #openstack-kolla | 20:44 | |
| egonzalez | jamesbenson verify if pinging to the hostname pings the correct IP | 20:44 |
| *** jrist has quit IRC | 20:45 | |
| egonzalez | sometimes resolv to localhost | 20:45 |
| jamesbenson | echo $HOSTNAME is good in each machine...ping is good | 20:46 |
| madhukar | Hi All, IS there a document which tells which docker hub container tag maps to which openstack-release? | 20:46 |
| madhukar | For example docker.io/kolla/ubuntu-source-neutron-server:3.0.3 ==> newton | 20:47 |
| egonzalez | madhukar nope afik, 4.0.* ocata, 3.0.* newton, etc | 20:47 |
| *** jtriley has quit IRC | 20:47 | |
| jemcevoy | Has anyone seen this error "Could not resolve hostname none hostname none" when kolla is generating ceph keys? http://paste.openstack.org/show/611248/ | 20:48 |
| madhukar | Thanks @egonzalez! It also means mitaka ==> 2.0.* | 20:48 |
| madhukar | Thats what I was looking for :) | 20:48 |
| *** eaguilar has quit IRC | 20:48 | |
| *** jrist has joined #openstack-kolla | 20:49 | |
| jamesbenson | jemcevoy: I haven't seen that one before, but have had issues with ceph as well, which is a bummer... | 20:50 |
| egonzalez | jamesbenson check if the api interface ip appears at getent ahostsv4 <hostname> | 20:51 |
| jamesbenson | 192.168.2.5 STREAM r2-710-6 | 20:53 |
| jamesbenson | egonzalez: This fails on the three controllers.... http://paste.openstack.org/show/611249/ | 20:54 |
| jamesbenson | egonzalez: Is that what is expected? | 20:54 |
| egonzalez | jamesbenson if those are the IPs in api_interface yes | 20:55 |
| jamesbenson | api_interface is just eno1 | 20:55 |
| *** manheim_ has joined #openstack-kolla | 20:56 | |
| egonzalez | jamesbenson eno1 in r2-710-6 have IP 192.168.2.5 ? | 20:57 |
| jamesbenson | 192.168.2. CIDR is for eno1 | 20:57 |
| jamesbenson | yep | 20:57 |
| jamesbenson | egonzales: I have a 10. for br0 | 20:59 |
| *** manheim_ has quit IRC | 21:01 | |
| *** manheim_ has joined #openstack-kolla | 21:02 | |
| jamesbenson | egonzalez: sorry for the typo. | 21:02 |
| *** chas has joined #openstack-kolla | 21:03 | |
| inc0 | hey sorry, I'm back | 21:04 |
| inc0 | lots of meetings ad hoc | 21:04 |
| jamesbenson | no worries inc0 | 21:06 |
| jamesbenson | :-) | 21:06 |
| inc0 | this day is killing me, I can't think straight:/ did you get your issue solved jamesbenson? | 21:06 |
| *** manheim_ has quit IRC | 21:06 | |
| *** jamesPR has joined #openstack-kolla | 21:06 | |
| jamesPR | sup everyone | 21:06 |
| *** lucasxu has quit IRC | 21:06 | |
| egonzalez | jamesbenson share prechecks with -vvv in the command | 21:07 |
| jamesbenson | nope, blew it out and trying again... lol, now failing in prechecks! ... rabbit can't resolve all hostnames ?!? | 21:07 |
| egonzalez | jamesbenson should be sth like this http://paste.openstack.org/show/611251/ | 21:07 |
| inc0 | jamesbenson: riiiiight | 21:07 |
| inc0 | did you run kolla-ansible bootstrap servers? | 21:07 |
| inc0 | kolla-ansible -i multinode bootstrap_servers | 21:07 |
| randylev | We are running into some issue with Kolla and Magnum. I'm curious to know if anyone has or had Kolla-ansible / Magnum / Newton working. | 21:08 |
| jamesPR | ppl every deployment I have been testing deploy without issues but when I stop all container and try to deploy again mariadb : Waiting for MariaDB service to be ready and it fails | 21:08 |
| inc0 | hey Randy, so, what kind of issues? | 21:08 |
| *** chas has quit IRC | 21:09 | |
| egonzalez | jamesPR stop or destroy? | 21:09 |
| inc0 | also I know we've put quite a bit of work into it during ocata timeframe | 21:09 |
| jamesbenson | inc0: I saw something about ssh not working between servers in a forum, which failed for me too, but corrected it, passing root ssh key to everyone, but still failed... | 21:09 |
| jamesPR | egonzalez, stop | 21:09 |
| jamesbenson | egonzalez: here's the output... http://paste.openstack.org/show/611252/ | 21:10 |
| egonzalez | jamesPR to recover mariadb cluster need to follow this steps (doc under review yet) https://review.openstack.org/#/c/466881/14/doc/operating-kolla.rst | 21:10 |
| *** manheim_ has joined #openstack-kolla | 21:11 | |
| randylev | inc0: I'm trying to get details from the dev team. Before we spend too much time on it, I was just wondering if other have gotten it up and running. The ppl working on this feature are in India. | 21:11 |
| inc0 | randylev: yes, we did, including keynote in Barcelona:) | 21:11 |
| inc0 | but that was ocata | 21:11 |
| jamesbenson | inc0: I did, but going to run it again just for giggles... | 21:11 |
| inc0 | or was it? can't remember really | 21:11 |
| inc0 | jamesbenson: cool, what was the issue? | 21:12 |
| randylev | inc0: ok. I was wondering if we would have to move to Octa. | 21:12 |
| *** manheim_ has quit IRC | 21:12 | |
| jamesbenson | very odd fail in the prechecks, TASK [rabbitmq : fail] : Hostname has to resolve to IP address of api_interface | 21:12 |
| egonzalez | was newton, ocata came in atlanta | 21:13 |
| jamesPR | egonzalez, ok trying | 21:13 |
| egonzalez | jamesPR cannot see the execution of that task, only the ssh connection stuff | 21:14 |
| egonzalez | ^^ jamesbenson | 21:14 |
| inc0 | randylev: lots of good things happened in ocata, but if you can't let's try to fix it | 21:14 |
| randylev | inc0: Thanks. I'll get more information from the dev team and have them ask again. They must asked on this channel when folks were not paying attention. | 21:15 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding extra condition for simple cell0 create job https://review.openstack.org/469963 | 21:16 |
| inc0 | well we have quite a lot of contributors in Asia | 21:16 |
| inc0 | just ask them to be persistent;) | 21:16 |
| randylev | We are trying to wrap up some final features on Newton now, and then jump to Pike later on. | 21:17 |
| randylev | inc0: I will...many thanks for your help. | 21:17 |
| inc0 | oh, that's gonna be quite a ride randylev | 21:17 |
| jamesbenson | egonzalez: rabbitmq : Check if all rabbit hostnames are resolvable task reports roughly the same.... the fail test next, actually fails instead of skips | 21:17 |
| inc0 | be very careful with upgrading | 21:17 |
| inc0 | as N->O was not a straightforward upgrade | 21:18 |
| *** Mannheim has joined #openstack-kolla | 21:18 | |
| inc0 | if you'll be running Kolla-ansible, I strongly encourage to do upgrade to Ocata and Ocata to Pike right after that | 21:18 |
| inc0 | but don't skip Ocata | 21:18 |
| egonzalez | jemcevoy the previous task failed and keys was not generated, may find more info doing docker logs bootstrap_ceph_mon | 21:19 |
| *** sayantan_ has quit IRC | 21:19 | |
| randylev | inc0: Fortunately, I can re-install in my lab and don't have to do an upgrade :D | 21:19 |
| inc0 | right, that makes it easier;) | 21:20 |
| *** sayantan_ has joined #openstack-kolla | 21:20 | |
| *** Manheim has quit IRC | 21:20 | |
| jamesbenson | egonzalez: http://paste.openstack.org/show/611254/ | 21:21 |
| *** manheim has joined #openstack-kolla | 21:22 | |
| jamesbenson | inc0: never could get the docker repo working right though.... | 21:23 |
| jamesbenson | still on 4.0.0 | 21:23 |
| *** krtaylor has joined #openstack-kolla | 21:23 | |
| inc0 | jamesbenson: meh, well, bear with us, soon | 21:24 |
| jamesbenson | :-) | 21:24 |
| jamesbenson | I tried once before and had issues too :-( | 21:24 |
| inc0 | speaking of which, I really really need reviews on this https://review.openstack.org/#/c/466007/ | 21:24 |
| egonzalez | jamesbenson seems the hostname is right, what jinja version have installed? | 21:25 |
| *** pramodrj07 has joined #openstack-kolla | 21:26 | |
| *** manheim has quit IRC | 21:26 | |
| *** sayantan_ has quit IRC | 21:26 | |
| jamesbenson | Jinja2 (2.9.6) | 21:26 |
| *** manheim has joined #openstack-kolla | 21:27 | |
| *** manheim has quit IRC | 21:28 | |
| jamesbenson | egonzalez: ^^ | 21:28 |
| *** manheim has joined #openstack-kolla | 21:28 | |
| egonzalez | hrm, ansible doesn't like 2.9.6 iirc, try with 2.8.1 | 21:28 |
| *** MasterOfBugs has quit IRC | 21:29 | |
| *** manheim has quit IRC | 21:32 | |
| jamesPR | egonzalez, doesn't work | 21:32 |
| *** powerd has joined #openstack-kolla | 21:33 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multi-scenerio gates https://review.openstack.org/466815 | 21:33 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 21:33 |
| jamesbenson | egonzalez: fixed version, still fails... could the {{ }} that are required for ansible be the issue for jinja? | 21:34 |
| jamesbenson | got to run... debug some more... | 21:34 |
| *** jamesbenson has quit IRC | 21:35 | |
| jemcevoy | Will check docker logs egonzalez Have not had a problem here after kickstarting cluster but this tile I just tried a cleanup. Made some temp network changes that I did want to code in my ansible prep for this one case... | 21:35 |
| *** myoung is now known as myoung|gone | 21:36 | |
| egonzalez | jamesbenson maybe something changed in ansible, i use 2.3.0.0 with jinja 2.8.1 and prechecks dont complain about syntax (only warnings) | 21:36 |
| egonzalez | jamesPR what are the erors or where is failing? | 21:38 |
| egonzalez | jamesPR when do stop, are you trying to deploy a new env or recover the stopped? | 21:38 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 21:39 |
| *** goldyfruit has quit IRC | 21:39 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding extra condition for simple cell0 create job https://review.openstack.org/469963 | 21:40 |
| *** salv-orlando has quit IRC | 21:40 | |
| sbezverk | kfox1111: rwellum: it is still WIP but appreciate your review. ^^^ | 21:41 |
| sbezverk | I tested it locally and it seems to be working.. | 21:41 |
| jamesPR | egonzalez, I'm just testing kolla for future production environment, I just simply deploy the cluster without issues then I test stoping the cluster with kolla-ansible -i inventory stop and then deploy again, then mysql is not starting | 21:41 |
| *** jamesbenson has joined #openstack-kolla | 21:41 | |
| inc0 | jamesPR: my question is, you sure you want to stop cluster? | 21:42 |
| inc0 | it's not great for mariadb | 21:42 |
| egonzalez | jamesPR to stop databases need a clean stop, following the previous link guide you thorough the process to clean stop databases | 21:42 |
| inc0 | it might happen, but shouldn't be a rule | 21:42 |
| *** emccormick has joined #openstack-kolla | 21:44 | |
| *** jascott1 has quit IRC | 21:45 | |
| jamesPR | inc0, if happends what can I do? ;) that's the question | 21:45 |
| jamesPR | stress test ;) | 21:45 |
| *** jamesbenson has quit IRC | 21:46 | |
| inc0 | jamesPR: well, just follow regular galera maintenance then;) | 21:46 |
| *** jascott1 has joined #openstack-kolla | 21:47 | |
| jamesPR | inc0, good point ;) | 21:49 |
| jemcevoy | egonzalez: old config left around. Looks like I did not do a good cleanup between deploys. I deleted all docker instances, images and volumes but what else? rm -rf /etc/kolla? http://paste.openstack.org/show/611258/ | 21:50 |
| jemcevoy | any other places to clean? | 21:50 |
| egonzalez | jemcevoy if using ceph clean up osd disks | 21:50 |
| egonzalez | kolla-ansible destroy do most of the cleanups except the disk | 21:51 |
| jemcevoy | I did that,,, Entries in the fstab, umount and relabel disks... | 21:51 |
| jemcevoy | Should I wipe /etc/kolla/... across the cluster? | 21:52 |
| jemcevoy | I will try the destroy/// | 21:52 |
| egonzalez | just use destroy command, will remove containers, config files | 21:52 |
| *** ipsecguy_ has joined #openstack-kolla | 21:56 | |
| *** ipsecguy has quit IRC | 21:59 | |
| *** jascott1 has quit IRC | 22:02 | |
| *** jascott1 has joined #openstack-kolla | 22:03 | |
| *** jascott1 has quit IRC | 22:04 | |
| *** jascott1 has joined #openstack-kolla | 22:06 | |
| *** rwallner has joined #openstack-kolla | 22:06 | |
| *** powerd has quit IRC | 22:07 | |
| *** jascott1 has quit IRC | 22:10 | |
| *** rwallner has quit IRC | 22:11 | |
| *** egonzalez has quit IRC | 22:12 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla master: Updated from global requirements https://review.openstack.org/469567 | 22:12 |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/469568 | 22:12 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Fix fernet deployment on certain configurations https://review.openstack.org/470040 | 22:20 |
| *** powerd has joined #openstack-kolla | 22:28 | |
| jemcevoy | egonzalez: Thanks for the pointers on ceph... After running the kolla destroy... I had to my docker cleanup which found a few images and 2 volumes to delete on the controllers. Then I relabeled all the osd drives and got further... All the OSDs are rebuilding :-) | 22:39 |
| jemcevoy | inc0 and sean-k-mooney: I ended up having to manually edit the /etc/kolla/nova-scheduler/nova.conf to have the two separate pci_alias entries to allow both GPU types to work with the scheduler. Attempts to configure pci_aliases as a single list entry failed when the nova scheduler tried to read the entry. | 22:44 |
| sdake | sbezverk in production, we are using pv_enabled: false and modifying the PVCs to have labels associated with them - indepdnent of what upstream does - woudl be nice if upstream matched what was going into production | 22:46 |
| *** Mannheim has quit IRC | 22:46 | |
| *** manheim has joined #openstack-kolla | 22:54 | |
| *** powerd has quit IRC | 22:56 | |
| *** jascott1 has joined #openstack-kolla | 22:58 | |
| *** madhukar has quit IRC | 22:59 | |
| *** rhallisey has quit IRC | 23:00 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Fix fernet deployment on certain configurations https://review.openstack.org/470040 | 23:01 |
| *** jamesbenson has joined #openstack-kolla | 23:07 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 23:08 |
| *** manheim_ has joined #openstack-kolla | 23:09 | |
| *** jamesbenson has quit IRC | 23:11 | |
| *** manheim has quit IRC | 23:12 | |
| *** rwellum has quit IRC | 23:13 | |
| *** emccormick has quit IRC | 23:13 | |
| inc0 | beep beep red alert, prechecks are broken | 23:21 |
| inc0 | https://review.openstack.org/#/c/470040/ seems like something bad happened today | 23:22 |
| jascott1 | :| | 23:23 |
| kfox1111 | jascott1: hey. :) | 23:25 |
| jascott1 | hey kfox1111 | 23:25 |
| kfox1111 | will you have time soon to work on the --node-selector -o json thing? | 23:25 |
| inc0 | ansible 2.3.1 broken | 23:25 |
| inc0 | fun life | 23:25 |
| jascott1 | yeah hopefully I can start on it tomorrow | 23:25 |
| kfox1111 | ok. cool. | 23:25 |
| jascott1 | working helm issue 2331 atm | 23:25 |
| inc0 | yeah, released 7hrs ago | 23:26 |
| jascott1 | bug in importValues | 23:26 |
| kfox1111 | if you don't have time and don't mind, I can take a quick stab at a first pass too. | 23:26 |
| jascott1 | inc0 hows it borken? | 23:26 |
| kfox1111 | the windows folks are chomping at the bit for a fix, | 23:26 |
| kfox1111 | and I'd rather it be a proper one then a hack. | 23:26 |
| jascott1 | i definitely dont mind | 23:26 |
| inc0 | http://logs.openstack.org/40/470040/2/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/c459bc6/console.html#_2017-06-01_23_09_15_397283 | 23:27 |
| kfox1111 | ok. no promices I can get to it, but we'll see if any time frees up. | 23:27 |
| kfox1111 | I'll ping you before I start just in case. | 23:27 |
| jascott1 | kfox1111 ill check with ye if I can start | 23:27 |
| *** jamesbenson has joined #openstack-kolla | 23:27 | |
| kfox1111 | ok. cool. thanks. :) | 23:27 |
| inc0 | https://bugs.launchpad.net/kolla-ansible/+bug/1695111 I have to run, be back later | 23:28 |
| openstack | Launchpad bug 1695111 in kolla-ansible "Ansible 2.3.1 broke preckecks" [Critical,Triaged] | 23:28 |
| inc0 | in the meantime if someone could take a look, it will be greatly appreciated | 23:28 |
| *** jamesbenson has quit IRC | 23:31 | |
| *** athomas has quit IRC | 23:38 | |
| *** ducttape_ has quit IRC | 23:41 | |
| *** jamesbenson has joined #openstack-kolla | 23:48 | |
| *** tonanhngo has quit IRC | 23:52 | |
| *** jamesbenson has quit IRC | 23:52 | |
| *** Pavo has joined #openstack-kolla | 23:52 | |
| *** manheim_ has quit IRC | 23:53 | |
| *** dklyle has joined #openstack-kolla | 23:55 | |
| *** hongbin has quit IRC | 23:56 | |
| *** tonanhngo has joined #openstack-kolla | 23:58 | |
| *** david-lyle has quit IRC | 23:58 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!