| Mannheim | i'll have to do it in the kolla | 00:00 |
|---|---|---|
| inc0 | not sure | 00:04 |
| inc0 | well, let's do this | 00:05 |
| inc0 | do what you need to do | 00:05 |
| inc0 | write a blog | 00:05 |
| inc0 | or at least notes | 00:05 |
| inc0 | and I'll review it and we'll streamline process | 00:05 |
| inc0 | how about that? | 00:05 |
| Mannheim | sure thing | 00:09 |
| *** Pavo has quit IRC | 00:09 | |
| Mannheim | i thing for the first blog post i'll just want it as a running env, manually deployment, same for the nova-compute on windows | 00:10 |
| *** jrobinson has joined #openstack-kolla | 00:10 | |
| Mannheim | think will manually do the configs | 00:10 |
| Mannheim | but my end goal i sto contribute to kolla | 00:10 |
| inc0 | yeah kolla on windows sounds like fun | 00:10 |
| inc0 | great:) | 00:10 |
| Mannheim | think kolla is very cool | 00:10 |
| inc0 | depends on what we need to do at very least we can write a doc page for hyperv | 00:11 |
| Mannheim | also, my company does openstack with windows stuff, so it will be fubn | 00:11 |
| inc0 | I'm glad to hear that | 00:11 |
| Mannheim | actually we did port nova to hyperv :d | 00:11 |
| inc0 | yeah, I'm pretty sure no other deploy tools works with windows | 00:12 |
| Mannheim | so thus the interest | 00:12 |
| inc0 | but it is interesting | 00:12 |
| *** Pavo has joined #openstack-kolla | 00:12 | |
| inc0 | whats your company btw? | 00:12 |
| Mannheim | oh, there is stuff like juju | 00:12 |
| Mannheim | cloudbase | 00:12 |
| inc0 | ah cool:) | 00:12 |
| Mannheim | we wrote juju charms for nova on windows, so there are methods | 00:13 |
| inc0 | I didn't know that juju works on windows | 00:13 |
| inc0 | cool | 00:13 |
| inc0 | well, welcome to kolla:) | 00:13 |
| Mannheim | but i am free to experiment other tools, and i like very much hwo kolla sounds like | 00:13 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 00:13 |
| Mannheim | thanks, hope i'm here to stay and contribute | 00:13 |
| inc0 | bbiaf | 00:15 |
| *** gfhellma has quit IRC | 00:17 | |
| *** n0isyn0ise has quit IRC | 00:22 | |
| *** n0isyn0ise has joined #openstack-kolla | 00:24 | |
| jemcevoy | inc0 Samyaple Mech422: We have kolla deployed on the new cluster for Penguin-on-Demand cloud. It is not quite in production yet as we work out integration with the beowulf cluster but it is in heavy testing. | 00:32 |
| *** Pavo has quit IRC | 00:33 | |
| jemcevoy | Has anyone seen this error in kibana from cron? And know how to fix it? pam_systemd(crond:session): Failed to connect to system bus: No such file or directory | 00:33 |
| jemcevoy | Also seeing thousands of ceilometer errors : Ignoring instance instance-00000107 (9b5675b0-c90c-4630-b324-9e3aeec08c91) : can't set attribute | 00:35 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 00:38 |
| *** tonanhngo has joined #openstack-kolla | 00:38 | |
| *** jtriley has joined #openstack-kolla | 00:40 | |
| openstackgerrit | Balaji G Pattewar proposed openstack/kolla-kubernetes master: Helm charts for heat/heat-cfn microservices https://review.openstack.org/414496 | 00:42 |
| *** tonanhngo has quit IRC | 00:43 | |
| *** Margin has joined #openstack-kolla | 00:44 | |
| Jeffrey4l | sdake, yep. i saw it. i have left a mail in the ML and will prepare and push the rc2 tag today. | 00:48 |
| Jeffrey4l | thanks for the reminding. | 00:48 |
| *** saneax is now known as saneax-_-|AFK | 00:53 | |
| *** salv-orlando has joined #openstack-kolla | 00:53 | |
| *** cuongnv has joined #openstack-kolla | 00:55 | |
| *** duonghq has joined #openstack-kolla | 00:56 | |
| *** Pavo has joined #openstack-kolla | 00:56 | |
| duonghq | hello | 00:57 |
| Jeffrey4l | hey kollaers, we need prepare kolla/kolla-ansible rc2 tag today. if is there any patch should be push/merged in ocata, please do it ASAP. then ping the core member to review it. | 00:58 |
| *** qiliang27 has joined #openstack-kolla | 00:58 | |
| *** salv-orlando has quit IRC | 00:58 | |
| Jeffrey4l | and please add it into this etherpad https://etherpad.openstack.org/p/kolla-rc2-tag | 00:59 |
| *** Manheim has quit IRC | 01:01 | |
| *** Mannheim has quit IRC | 01:01 | |
| *** manheim has joined #openstack-kolla | 01:05 | |
| *** Pavo has quit IRC | 01:05 | |
| *** zhurong has joined #openstack-kolla | 01:05 | |
| *** zhubingbing has joined #openstack-kolla | 01:10 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 01:17 |
| sp_ | duonghq: hi... | 01:18 |
| duonghq | hi sp_ | 01:18 |
| sp_ | duonghq: what we are planning to do for https://review.openstack.org/#/c/425446/ ? | 01:19 |
| *** jtriley has quit IRC | 01:19 | |
| duonghq | Jeffrey4l, do you have any idea about Ubuntu gate is broken | 01:19 |
| Jeffrey4l | duonghq, no idea. i tried some fix, but failed ;( | 01:20 |
| duonghq | Jeffrey4l, I cannot figure out why ara failed | 01:21 |
| duonghq | sp_, fix the gate, of course | 01:21 |
| sp_ | duonghq: ok, i am also looking into this | 01:21 |
| Jeffrey4l | i think libvirt is trying to write the console.log file, but failed due to the permission issue. | 01:22 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Implement Keystone zero-downtime upgrade https://review.openstack.org/425446 | 01:22 |
| duonghq | sp_, I just rebase it | 01:22 |
| duonghq | Jeffrey4l, roger | 01:22 |
| sp_ | duonghq: Jeffrey4l yes, i see | 01:23 |
| Jeffrey4l | duonghq, sp_ are u using ubuntu or centos general? | 01:23 |
| duonghq | Jeffrey4l, for host: Ubuntu, for image: CentOS | 01:24 |
| sp_ | duonghq: Jeffrey4l do we support rolling upgrade for other services too like neutron and nova ? | 01:24 |
| sp_ | Jeffrey4l: ubuntu | 01:24 |
| duonghq | sp_, it's in dev | 01:24 |
| Jeffrey4l | sp_, we support upgrade, but not rolling. duonghq is trying to fix this. | 01:24 |
| Jeffrey4l | duonghq, interesting. | 01:25 |
| *** caowei has joined #openstack-kolla | 01:25 | |
| Jeffrey4l | sp_, could u debug the permission issue in you env? | 01:25 |
| Jeffrey4l | duonghq, sp_ check https://bugs.launchpad.net/bugs/1667741 | 01:25 |
| openstack | Launchpad bug 1667741 in kolla ocata "deploy ubuntu source failed due to file permission issue" [Critical,New] | 01:25 |
| duonghq | Jeffrey4l, sp_ seem that somebody fixed ara, last time I check the log, ara failed | 01:26 |
| *** jtriley has joined #openstack-kolla | 01:26 | |
| sp_ | Jeffrey4l: So this would be first for rolling upgrade https://blueprints.launchpad.net/kolla-ansible/+spec/ks-rolling-upgrade-role ? If i got you perfectly ? | 01:26 |
| Jeffrey4l | for ara, yes. | 01:27 |
| dmsimard | Is there a problem with ara? :) | 01:27 |
| sp_ | Jeffrey4l: I just moved to Japan | 01:27 |
| sp_ | so may last dev env so for way | 01:27 |
| Jeffrey4l | dmsimard, no. we are talking the issue you fixed recently about ara. ( cli is changed ) | 01:28 |
| Jeffrey4l | sp_, got. | 01:28 |
| *** lucasxu has joined #openstack-kolla | 01:28 | |
| dmsimard | ah | 01:28 |
| *** tonanhngo has joined #openstack-kolla | 01:28 | |
| *** Pavo has joined #openstack-kolla | 01:29 | |
| *** tonanhngo has quit IRC | 01:33 | |
| *** yee379 has quit IRC | 01:33 | |
| *** trinaths has joined #openstack-kolla | 01:33 | |
| *** britthouser6 has joined #openstack-kolla | 01:34 | |
| *** britthouser has quit IRC | 01:34 | |
| *** yee379 has joined #openstack-kolla | 01:34 | |
| *** trinaths has quit IRC | 01:34 | |
| *** unicell has quit IRC | 01:34 | |
| *** trinaths has joined #openstack-kolla | 01:34 | |
| *** trinaths1 has joined #openstack-kolla | 01:35 | |
| *** trinaths has quit IRC | 01:39 | |
| *** salv-orlando has joined #openstack-kolla | 01:55 | |
| *** qiliang28 has joined #openstack-kolla | 01:58 | |
| *** salv-orlando has quit IRC | 01:59 | |
| *** qiliang27 has quit IRC | 02:00 | |
| *** qiliang28 is now known as qiliang27 | 02:00 | |
| Jeffrey4l | duonghq, could u update your patch https://review.openstack.org/#/c/438886/3 | 02:03 |
| duonghq | Jeffrey4l, sure | 02:05 |
| Jeffrey4l | thanks. | 02:09 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: Fix missing python modules in Ceilometer https://review.openstack.org/438886 | 02:10 |
| duonghq | Jeffrey4l, I updated the ps, thank you | 02:11 |
| duonghq | can you re-review? | 02:11 |
| duonghq | Jeffrey4l, how about this ps: https://review.openstack.org/#/c/438838/ | 02:11 |
| Jeffrey4l | duonghq, reviewed. -1 , nit. | 02:12 |
| Jeffrey4l | duonghq, for the second one, LGTM, but need talk with pbourke and sp_ | 02:13 |
| duonghq | Jeffrey4l, roger | 02:13 |
| Jeffrey4l | hrm, add the info to task name may be better. | 02:14 |
| *** erlon has quit IRC | 02:15 | |
| Jeffrey4l | like "checking ceilometer mysql database with given credential. | 02:15 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla master: Fix missing python modules in Ceilometer https://review.openstack.org/438886 | 02:19 |
| *** MasterOfBugs has quit IRC | 02:21 | |
| *** pramodrj07 has quit IRC | 02:22 | |
| duonghq | Jeffrey4l, this task both trying to checking and creating db | 02:22 |
| duonghq | Maybe trying to connect and create database ... | 02:23 |
| duonghq | "Trying to connect and create database with given... | 02:23 |
| Jeffrey4l | duonghq, creating ceilometer database using user ceilometer? | 02:24 |
| duonghq | Jeffrey4l, the intent of this module is make sure the db exist (by trying to create one if not existed) | 02:26 |
| duonghq | so, even if we do not have permission to create db, it still success if there is db already existed | 02:26 |
| Jeffrey4l | yes. so the result of the module is the database exist. | 02:27 |
| Jeffrey4l | msyql_db with admin user is the same with ceilometer. | 02:27 |
| Jeffrey4l | if the database exist, do nothing. | 02:27 |
| duonghq | maybe "Trying to connect/create mysql database with custom credential"? | 02:28 |
| Jeffrey4l | there is no big difference bewteen user ceilometer and admin ceilometer, right? | 02:28 |
| duonghq | in which facet? | 02:29 |
| Jeffrey4l | so no need mention the "trying to connect", imo. | 02:29 |
| Jeffrey4l | as the result. | 02:29 |
| Jeffrey4l | there is no big difference bewteen user ceilometer and USER admin , right? | 02:29 |
| duonghq | hmm, but creating is somewhat misleading | 02:29 |
| duonghq | "Creating" | 02:30 |
| duonghq | It make user think that we are trying to create the db | 02:30 |
| Jeffrey4l | duonghq, if the database do not exist, but the user ceilometer have permission to create on, it really create a new database by using ceilometer user. | 02:30 |
| duonghq | but if the given credential doesn't have create db permission, it will be diff | 02:31 |
| Jeffrey4l | if database do not exist and user ceilometer haven't the permission, this task is failed and create the database using user admin. | 02:31 |
| duonghq | no, in this case, the following task will use internal db | 02:32 |
| duonghq | "internal" meaning Kolla deployed db | 02:32 |
| Jeffrey4l | hrm, wait. i missed something. | 02:33 |
| Jeffrey4l | go on. | 02:34 |
| duonghq | so, I'm finding way to describe the intent of mysql | 02:34 |
| duonghq | mysql_db module | 02:34 |
| Jeffrey4l | duonghq, for "Checking Ceilometer mysql database" it will use external database when 1. the external ceilometer database exist or 2. the given ceilometer user can create the database. | 02:35 |
| *** manheim has quit IRC | 02:36 | |
| *** huzhengchuan has joined #openstack-kolla | 02:36 | |
| duonghq | yup | 02:36 |
| Jeffrey4l | so this name should be " creating external ceilometer database by using ceilometer user" | 02:36 |
| Jeffrey4l | the following task is: " creating internal ceilometer database by using root user" | 02:37 |
| Jeffrey4l | admin user* | 02:37 |
| Jeffrey4l | creating external ceilometer database: means create it or do nothing if it exist. | 02:37 |
| duonghq | I think it should be "default user" | 02:37 |
| duonghq | ack | 02:37 |
| Jeffrey4l | btw, this is worth a bp. we can implement such patten for all role. can we can support external rabbit, external mariadb, even external neutron, | 02:39 |
| duonghq | Jeffrey4l, sure, I thought about this some time | 02:39 |
| duonghq | I'll draft one? | 02:40 |
| Jeffrey4l | re "default user", whatever. or simple call it as "creating internal ceilometer database" | 02:40 |
| Jeffrey4l | cool, please. | 02:40 |
| duonghq | Jeffrey4l, ack | 02:40 |
| Jeffrey4l | thanks. | 02:41 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Fix Ceilometer cannot create MySQL database https://review.openstack.org/438838 | 02:41 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-ansible master: Fix Ceilometer cannot create MySQL database https://review.openstack.org/438838 | 02:43 |
| duonghq | sp_, ping | 02:43 |
| sp_ | duonghq: pong | 02:43 |
| sp_ | duonghq: i will check https://review.openstack.org/438838 | 02:44 |
| duonghq | sp_, Jeffrey4l about the Keystone zero-downtime upgrade, do you think it's better to implement in-upgrade testing as inc0 said in last meeting right in the ps, or let it be a separate bp and we'll revive it later? | 02:44 |
| Jeffrey4l | hrm, what the details of "in upgrade testing"? | 02:45 |
| duonghq | Jeffrey4l, this one https://blueprints.launchpad.net/kolla-ansible/+spec/in-upgrade-test | 02:45 |
| Jeffrey4l | this should be separate bp, imo. otherwise rolling-upgrade will become more complicated. | 02:47 |
| sp_ | Jeffrey4l: +1 | 02:47 |
| sp_ | duonghq: I think https://blueprints.launchpad.net/kolla-ansible/+spec/in-upgrade-test is related to testing and all | 02:47 |
| sp_ | we should keep it in separate BP so that it we can drive it efficiently | 02:48 |
| duonghq | Jeffrey4l, sp_ nice | 02:49 |
| *** yuanying_ has joined #openstack-kolla | 02:49 | |
| *** yuanying has quit IRC | 02:49 | |
| *** oomichi has quit IRC | 02:50 | |
| sp_ | duonghq: I think, in future, we will support more rolling upgrade for other services too like nova and neutron and many more | 02:50 |
| *** oomichi has joined #openstack-kolla | 02:50 | |
| duonghq | sp_, sure | 02:50 |
| duonghq | sp_, this bp https://blueprints.launchpad.net/kolla-ansible/+spec/apply-service-upgrade-procedure | 02:51 |
| duonghq | addresses it | 02:51 |
| sp_ | duonghq: and every component would be having there own step by steps process of rolling upgrade | 02:51 |
| duonghq | sp_, yup | 02:52 |
| sp_ | duonghq: I jst seen it | 02:53 |
| *** bpattewar has joined #openstack-kolla | 02:53 | |
| bpattewar | Hi all | 02:53 |
| duonghq | hi bpattewar | 02:53 |
| *** jtriley has quit IRC | 02:55 | |
| sp_ | duonghq: I am getting more understanding of it i.e rolling upgrade role addition. Lets see what i can add more to it ASAP. | 02:55 |
| sp_ | bpattewar: hi.. | 02:56 |
| duonghq | sp_, nice | 02:56 |
| bpattewar | sp hi | 02:56 |
| bpattewar | duonghq I made some progress but the heat-manage-db service is still not working. First it failed because of of missing configmap, when I updated serviceName to heat-engine | 02:57 |
| bpattewar | I have commented out ceph_manage_db from ceph_workflow.sh so most of the test are passing, but I am not able to figure out why it is failing for this | 02:59 |
| bpattewar | http://logs.openstack.org/96/414496/31/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/0d94d6f/ | 02:59 |
| duonghq | bpattewar, seen | 02:59 |
| duonghq | bpattewar, maybe it need some info in the configmap | 03:00 |
| duonghq | one template and some vars | 03:01 |
| bpattewar | the heat-manage-db fails on my local machine with log message ERROR:__main__:The source to copy does not exist: /var/lib/kolla/config_files/_deprecated.yaml | 03:03 |
| bpattewar | any idea what configmap we need for this.. | 03:03 |
| duonghq | not yet | 03:03 |
| *** rwellum has quit IRC | 03:03 | |
| duonghq | also looking for which configmap we should use | 03:04 |
| duonghq | also which container | 03:04 |
| bpattewar | heat-engine container | 03:04 |
| duonghq | I used heat-engine as container due to it's the only module connect to db | 03:04 |
| bpattewar | yes the servicename was set to heat so i changed to heat-engine and it fixed configmap issue | 03:05 |
| duonghq | ah, just recall why I put heat in the serviceName | 03:05 |
| duonghq | Jeffrey4l, I see heat in kolla-ansbile as one config template is _deprecated.yaml, does the underscore serve some meaning? | 03:07 |
| duonghq | *has one | 03:07 |
| *** trinaths1 has quit IRC | 03:11 | |
| Jeffrey4l | duonghq, i am not sure. not guess _ means nothing. | 03:12 |
| duonghq | Jeffrey4l, ack | 03:12 |
| *** lucasxu has quit IRC | 03:15 | |
| *** sudipto has joined #openstack-kolla | 03:15 | |
| *** sudipto_ has joined #openstack-kolla | 03:15 | |
| *** sp_ has quit IRC | 03:16 | |
| *** jtriley has joined #openstack-kolla | 03:23 | |
| *** jtriley has quit IRC | 03:28 | |
| *** sp_ has joined #openstack-kolla | 03:29 | |
| *** sp_ has quit IRC | 03:36 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Release note for reconfiguration optimizaiton https://review.openstack.org/431107 | 03:36 |
| *** jtriley has joined #openstack-kolla | 03:41 | |
| *** sp_ has joined #openstack-kolla | 03:49 | |
| *** salv-orlando has joined #openstack-kolla | 03:55 | |
| *** salv-orlando has quit IRC | 04:00 | |
| *** jtriley has quit IRC | 04:06 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 04:08 |
| *** tonanhngo has joined #openstack-kolla | 04:08 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 04:11 |
| *** tonanhngo has quit IRC | 04:13 | |
| *** skramaja has joined #openstack-kolla | 04:21 | |
| *** sudipto has quit IRC | 04:32 | |
| *** sudipto_ has quit IRC | 04:32 | |
| *** sudipto has joined #openstack-kolla | 04:33 | |
| *** sudipto_ has joined #openstack-kolla | 04:33 | |
| *** bpattewar has quit IRC | 04:34 | |
| *** unicell has joined #openstack-kolla | 04:40 | |
| *** unicell has quit IRC | 04:40 | |
| *** unicell has joined #openstack-kolla | 04:40 | |
| *** unicell has quit IRC | 04:43 | |
| *** sudipto_ has quit IRC | 04:43 | |
| *** sudipto has quit IRC | 04:43 | |
| *** unicell has joined #openstack-kolla | 04:47 | |
| openstackgerrit | Merged openstack/kolla master: Fix missing python modules in Ceilometer https://review.openstack.org/438886 | 04:47 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 04:47 |
| *** jtriley has joined #openstack-kolla | 04:48 | |
| *** mdnadeem has joined #openstack-kolla | 04:55 | |
| *** trinaths has joined #openstack-kolla | 04:57 | |
| *** jtriley has quit IRC | 04:57 | |
| *** tonanhngo has joined #openstack-kolla | 04:59 | |
| *** trinaths1 has joined #openstack-kolla | 05:00 | |
| *** trinaths1 has quit IRC | 05:01 | |
| *** trinaths has quit IRC | 05:02 | |
| *** tonanhngo_ has joined #openstack-kolla | 05:02 | |
| *** tonanhngo has quit IRC | 05:03 | |
| *** trinaths1 has joined #openstack-kolla | 05:04 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix the failure when running precheck after deploy https://review.openstack.org/441788 | 05:04 |
| *** trinaths has joined #openstack-kolla | 05:06 | |
| *** trinaths1 has quit IRC | 05:06 | |
| *** tonanhngo_ has quit IRC | 05:07 | |
| *** trinaths has quit IRC | 05:08 | |
| *** trinaths has joined #openstack-kolla | 05:08 | |
| *** trinaths1 has joined #openstack-kolla | 05:09 | |
| *** trinaths has quit IRC | 05:12 | |
| *** trinaths has joined #openstack-kolla | 05:13 | |
| *** trinaths1 has quit IRC | 05:13 | |
| *** trinaths has quit IRC | 05:14 | |
| *** takashi_sogabe has joined #openstack-kolla | 05:16 | |
| *** cuongnv has quit IRC | 05:28 | |
| *** cuongnv has joined #openstack-kolla | 05:32 | |
| *** huzhengchuan has quit IRC | 05:33 | |
| *** lyan has quit IRC | 05:35 | |
| *** salv-orlando has joined #openstack-kolla | 05:39 | |
| *** sudipto has joined #openstack-kolla | 05:39 | |
| *** sudipto_ has joined #openstack-kolla | 05:39 | |
| *** salv-orlando has quit IRC | 05:43 | |
| *** jtriley has joined #openstack-kolla | 05:47 | |
| *** jtriley has quit IRC | 05:52 | |
| *** Jeffrey4l has quit IRC | 05:52 | |
| *** jtriley has joined #openstack-kolla | 05:52 | |
| Margin | hi, I have an question | 05:52 |
| berendt | good [morning,noon,evening] | 05:52 |
| Margin | If I want to scale out a node, which command should be use ? still is "kolla-ansible deploy" ? | 05:53 |
| berendt | Margin: feel free to ask your question :) | 05:55 |
| berendt | what do you mean with "scale out a node"? | 05:55 |
| Margin | for example , new add a compute node | 05:55 |
| berendt | first you have to add your new node to the inventory file, afterwards you can run the deploy command | 05:56 |
| *** Jeffrey4l has joined #openstack-kolla | 05:56 | |
| *** jtriley has quit IRC | 05:57 | |
| *** jrobinson is now known as jrobinson-afk | 05:57 | |
| *** tovin07 is now known as tovin07_at_work | 06:08 | |
| duonghq | hi berendt | 06:12 |
| *** salv-orlando has joined #openstack-kolla | 06:16 | |
| *** jrobinson-afk has quit IRC | 06:17 | |
| *** jrobinson-afk has joined #openstack-kolla | 06:21 | |
| *** jrobinson-afk is now known as jrobinson | 06:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Run precheck after deployed openstack https://review.openstack.org/442289 | 06:37 |
| *** nathharp has joined #openstack-kolla | 06:37 | |
| berendt | Jeffrey4l: why do you want to re-run the prechecks after the deployment? | 06:38 |
| *** yee379 has quit IRC | 06:38 | |
| Jeffrey4l | berendt, prechecks only works before deploying before. | 06:38 |
| *** yee379 has joined #openstack-kolla | 06:39 | |
| *** nathharp has quit IRC | 06:39 | |
| Jeffrey4l | but this cycle, we add the ability to let prechecks works after deployed. | 06:39 |
| berendt | then the command/role should be renamed | 06:39 |
| Jeffrey4l | this is useful when enable another service after first deploy. | 06:40 |
| Jeffrey4l | or even run prechecks before upgrading. | 06:40 |
| berendt | yes, it is useful, but a little bit confusing | 06:40 |
| Jeffrey4l | command/role means? | 06:40 |
| berendt | also i think the checks behave differently when running before and after a deployment | 06:40 |
| berendt | at the moment the command is kolla-ansible precheck | 06:40 |
| berendt | i think this should be renamed to something different when it is possible to run it post deployment | 06:41 |
| Jeffrey4l | prechecks is OK. it means, check something before deploying and upgradeing. | 06:41 |
| Jeffrey4l | " checks behave differently when running before and after a deployment" this should use "kolla-ansible --check deploy/upgrade" | 06:42 |
| Jeffrey4l | even though we do not support "--check" parameter now. | 06:42 |
| berendt | ok so prechecks command will be removed after adding the --check parameter? | 06:43 |
| Jeffrey4l | no. they are different things. | 06:43 |
| Jeffrey4l | prechecks just check whether your environment is ready to deploy/upgrade. | 06:43 |
| *** unicell1 has joined #openstack-kolla | 06:43 | |
| Jeffrey4l | --check mean what will be changed during deploy. but it is dry run. | 06:44 |
| *** unicell has quit IRC | 06:44 | |
| berendt | e.g. the port_checks.yml tasks make no sense after the deployment | 06:45 |
| Jeffrey4l | berendt, no. it is meaningful still. | 06:45 |
| Jeffrey4l | imaging, after first deploy, you want to enable cinder then | 06:46 |
| Jeffrey4l | ( which is not enabled at first deploy ) | 06:46 |
| Jeffrey4l | then you can enable_cinder: yes. | 06:46 |
| *** jtriley has joined #openstack-kolla | 06:46 | |
| Jeffrey4l | and run prechecks again, | 06:46 |
| berendt | Checking free port for HAProxy stats | 06:46 |
| berendt | this task will fail when you deployed haproxy? | 06:46 |
| Jeffrey4l | it will not run when you deployed haproxy | 06:47 |
| *** satyar has joined #openstack-kolla | 06:47 | |
| Jeffrey4l | that is what we added this cycle. | 06:47 |
| Jeffrey4l | run the prechecks tasks conditionally. | 06:47 |
| berendt | Jeffrey4l: https://review.openstack.org/#/c/441788/3/ansible/roles/prechecks/tasks/port_checks.yml | 06:48 |
| berendt | I missed this open review.. | 06:48 |
| berendt | after the removal of the tasks it makes sense | 06:48 |
| Jeffrey4l | yes. this patch should be pushed. | 06:48 |
| Jeffrey4l | i made several test with this patch. | 06:49 |
| Jeffrey4l | it works as expected. | 06:49 |
| Jeffrey4l | btw, i also wanna to check the password variables and inventory group expected. | 06:50 |
| Jeffrey4l | in the precheck stage. | 06:50 |
| berendt | Jeffrey4l: there is a nit in the commit message | 06:51 |
| berendt | Close-Bug: #1670286 | 06:51 |
| openstack | bug 1670286 in kolla-ansible "precheck still failed when running on a deployed env" [Undecided,Triaged] https://launchpad.net/bugs/1670286 - Assigned to Jeffrey Zhang (jeffrey4l) | 06:51 |
| berendt | will change this to closes | 06:51 |
| *** jtriley has quit IRC | 06:51 | |
| *** lamt has joined #openstack-kolla | 06:51 | |
| Jeffrey4l | roger. will fix it. thanks. | 06:51 |
| *** jtriley has joined #openstack-kolla | 06:51 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix the failure when running precheck after deploy https://review.openstack.org/441788 | 06:57 |
| *** jtriley has quit IRC | 06:57 | |
| *** lamt has quit IRC | 07:06 | |
| *** jrist has quit IRC | 07:10 | |
| *** cuongnv_ has joined #openstack-kolla | 07:12 | |
| *** jrist has joined #openstack-kolla | 07:12 | |
| *** cuongnv has quit IRC | 07:14 | |
| *** jrobinson has quit IRC | 07:16 | |
| *** salv-orl_ has joined #openstack-kolla | 07:23 | |
| *** salv-orlando has quit IRC | 07:26 | |
| *** yee379 has quit IRC | 07:31 | |
| *** yee379 has joined #openstack-kolla | 07:31 | |
| *** saneax-_-|AFK is now known as saneax | 07:31 | |
| *** magicboiz has joined #openstack-kolla | 07:32 | |
| *** salv-orl_ has quit IRC | 07:35 | |
| *** bmace has quit IRC | 07:45 | |
| *** bmace has joined #openstack-kolla | 07:45 | |
| *** jtriley has joined #openstack-kolla | 07:46 | |
| *** cuongnv has joined #openstack-kolla | 07:48 | |
| *** unicell1 has quit IRC | 07:48 | |
| *** pcaruana has joined #openstack-kolla | 07:50 | |
| *** cuongnv_ has quit IRC | 07:50 | |
| *** jtriley has quit IRC | 07:51 | |
| *** jtriley has joined #openstack-kolla | 07:51 | |
| *** shardy has joined #openstack-kolla | 07:52 | |
| *** salv-orlando has joined #openstack-kolla | 07:53 | |
| *** jtriley has quit IRC | 07:56 | |
| *** tonanhngo has joined #openstack-kolla | 07:57 | |
| *** dosaboy has quit IRC | 07:58 | |
| *** dosaboy has joined #openstack-kolla | 07:59 | |
| berendt | sdake: i cannot join todays meeting, have to visit a new office space, can we delay the "how to push docker images to docker hub" topic to the next meeting or is it an urgent topic from your pov? | 08:11 |
| *** gfidente has joined #openstack-kolla | 08:12 | |
| duonghq | berendt, which meeting? you mean weekly meeting? | 08:12 |
| berendt | duonghq: yes | 08:12 |
| duonghq | iirc, it's organized tomorrow | 08:12 |
| berendt | duonghq: doh, it is on wednesday, of course | 08:12 |
| duonghq | ya, | 08:13 |
| berendt | i need more coffee and more sleep :( | 08:13 |
| duonghq | quite large tz shifting | 08:13 |
| duonghq | ? | 08:13 |
| berendt | yes i want to add a job to my jenkins system | 08:13 |
| berendt | i think the best is to run a daily job that pulls the tarbals from http://tarballs.openstack.org/kolla/images/ and pushs to docker hub | 08:14 |
| berendt | i have a system with 1 gbit outgoing connectivity for such tasks, should be sufficient | 08:14 |
| berendt | we only have to make a daily frozen copy of the tarballs because the download of the tarballs will take several minutes and it is possible that a new commit will change the tarballs while downloading them | 08:15 |
| *** tonanhngo_ has joined #openstack-kolla | 08:15 | |
| berendt | duonghq: tz shifting should be fine this week, lot of work at the moment :( | 08:16 |
| *** tonanhngo has quit IRC | 08:16 | |
| duonghq | where are you now? | 08:17 |
| berendt | in Germany at the moment | 08:17 |
| duonghq | ack | 08:17 |
| *** shardy has quit IRC | 08:26 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 08:30 | |
| *** Jeffrey4l has quit IRC | 08:34 | |
| *** Jeffrey4l_ has quit IRC | 08:37 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 08:38 | |
| *** mgoddard has joined #openstack-kolla | 08:43 | |
| *** jtriley has joined #openstack-kolla | 08:45 | |
| *** tvignaud has quit IRC | 08:52 | |
| jmccarthy | I found this in the release notes: openstack-kolla/releasenotes/notes/add-zookeeper-2454cdfbfa7047b4.yaml: - Add zookeeper image, used in monasca | 08:56 |
| jmccarthy | Just curious, any ideas about when monasca will be in the mix ? | 08:56 |
| *** athomas has joined #openstack-kolla | 09:00 | |
| *** Manheim has joined #openstack-kolla | 09:04 | |
| *** Mannheim has joined #openstack-kolla | 09:04 | |
| *** jtriley has quit IRC | 09:07 | |
| *** egonzalez has joined #openstack-kolla | 09:15 | |
| *** tvignaud has joined #openstack-kolla | 09:20 | |
| *** pc_m has quit IRC | 09:25 | |
| *** mgoddard has quit IRC | 09:29 | |
| *** Andrew_jedi has joined #openstack-kolla | 09:30 | |
| *** pc_m has joined #openstack-kolla | 09:31 | |
| *** duonghq has quit IRC | 09:35 | |
| *** tonanhngo_ has quit IRC | 09:38 | |
| *** hrw has quit IRC | 09:39 | |
| *** Margin has quit IRC | 09:41 | |
| *** Margin has joined #openstack-kolla | 09:41 | |
| *** Margin has quit IRC | 09:43 | |
| *** jtriley has joined #openstack-kolla | 09:44 | |
| *** manheim_ has joined #openstack-kolla | 09:45 | |
| *** hrw has joined #openstack-kolla | 09:45 | |
| *** tovin07_at_work has quit IRC | 09:46 | |
| zhubingbing | sup egonzalez | 09:49 |
| egonzalez | hey zhubingbing | 09:49 |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Magnum Kolla demo in contrib https://review.openstack.org/442347 | 09:50 |
| *** jtriley has quit IRC | 09:52 | |
| *** tonanhngo has joined #openstack-kolla | 09:52 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Magnum Kolla demo in contrib https://review.openstack.org/442347 | 09:53 |
| *** tonanhngo has quit IRC | 09:57 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Fix Heat Kolla demo in contrib https://review.openstack.org/442350 | 10:00 |
| *** saneax is now known as saneax-_-|AFK | 10:07 | |
| *** cuongnv has quit IRC | 10:12 | |
| *** pbourke has quit IRC | 10:19 | |
| *** pbourke has joined #openstack-kolla | 10:20 | |
| *** satyar has quit IRC | 10:23 | |
| *** salv-orl_ has joined #openstack-kolla | 10:23 | |
| *** sambetts|afk is now known as sambetts | 10:25 | |
| *** zhubingbing has quit IRC | 10:25 | |
| *** salv-orlando has quit IRC | 10:27 | |
| *** salv-orl_ has quit IRC | 10:27 | |
| *** openstackgerrit has quit IRC | 10:33 | |
| *** caowei has quit IRC | 10:33 | |
| *** jtriley has joined #openstack-kolla | 10:42 | |
| *** Guest6667 is now known as Guest6666 | 10:43 | |
| *** tovin07 has joined #openstack-kolla | 10:43 | |
| *** shardy has joined #openstack-kolla | 10:45 | |
| *** tovin07 has left #openstack-kolla | 10:46 | |
| *** jtriley has quit IRC | 10:47 | |
| *** jtriley has joined #openstack-kolla | 10:47 | |
| *** jtriley has quit IRC | 10:52 | |
| *** saneax-_-|AFK is now known as saneax | 11:10 | |
| *** duankebo has joined #openstack-kolla | 11:20 | |
| *** duankebo has quit IRC | 11:20 | |
| *** duankebo has joined #openstack-kolla | 11:20 | |
| *** duankebo has quit IRC | 11:20 | |
| *** duankebo has joined #openstack-kolla | 11:20 | |
| *** duankebo has joined #openstack-kolla | 11:21 | |
| des_consolado | hey I'm getting this oAuth error when deploying http://paste.openstack.org/show/601739/ | 11:21 |
| *** duankebo has joined #openstack-kolla | 11:21 | |
| *** duankebo has joined #openstack-kolla | 11:21 | |
| *** duankebo has quit IRC | 11:22 | |
| *** duankebo has joined #openstack-kolla | 11:22 | |
| des_consolado | There is nothing listening on 16509 | 11:22 |
| *** duankebo has quit IRC | 11:22 | |
| *** duankebo has joined #openstack-kolla | 11:22 | |
| *** duankebo has quit IRC | 11:23 | |
| *** duankebo has joined #openstack-kolla | 11:23 | |
| des_consolado | grep for 16509 in the kolla-ansible repo shows nothing | 11:25 |
| *** caowei has joined #openstack-kolla | 11:27 | |
| duankebo | I'm using kolla-ansible to install openstack. Could anyone tell me whether the "enable_central_logging" feature is usable or not? | 11:28 |
| duankebo | When I set it to "Yes". Two containers are started, elasticsearch and kibana. But I can't find any log on Kibana's. Do I miss something? | 11:31 |
| *** mliima has joined #openstack-kolla | 11:31 | |
| *** mliima has quit IRC | 11:31 | |
| *** mgoddard has joined #openstack-kolla | 11:32 | |
| *** mgoddard_ has joined #openstack-kolla | 11:32 | |
| *** sudipto has quit IRC | 11:36 | |
| *** sudipto_ has quit IRC | 11:36 | |
| *** jtriley has joined #openstack-kolla | 11:41 | |
| *** jtriley has quit IRC | 11:46 | |
| *** jtriley has joined #openstack-kolla | 11:47 | |
| des_consolado | duankebo: https://github.com/openstack/kolla-ansible/blob/master/doc/troubleshooting.rst#debugging-kolla | 11:55 |
| des_consolado | you're supposed to just browse to some URI | 11:57 |
| des_consolado | default kibana port is 5601 it looks like | 11:57 |
| *** qiliang27 has quit IRC | 12:00 | |
| *** qiliang27 has joined #openstack-kolla | 12:01 | |
| *** sps_ has joined #openstack-kolla | 12:06 | |
| *** jtriley has quit IRC | 12:08 | |
| *** salv-orlando has joined #openstack-kolla | 12:09 | |
| japestinho | morning folks, I was trying deployment kolla-kubernetes from this https://etherpad.openstack.org/p/kolla-kubernetes-deploy-guide-BP, and got Error: failed to parse ./cloud.yaml: error converting YAML to JSON: yaml: line 49: did not find expected key | 12:14 |
| japestinho | here is my interface http://pastebin.com/5xsgMaxB | 12:15 |
| *** shardy is now known as shardy_lunch | 12:15 | |
| *** rwallner has joined #openstack-kolla | 12:15 | |
| japestinho | which interface should be tunnel_interface on cloud.yaml, need an advice :-) | 12:16 |
| *** rwellum has joined #openstack-kolla | 12:19 | |
| *** dgonzalez has quit IRC | 12:22 | |
| *** openstackgerrit has joined #openstack-kolla | 12:25 | |
| openstackgerrit | Mark Goddard proposed openstack/kolla-ansible master: Improve developer documentation for Bifrost https://review.openstack.org/436462 | 12:25 |
| *** jtriley has joined #openstack-kolla | 12:26 | |
| *** salv-orlando has quit IRC | 12:29 | |
| *** jtriley has quit IRC | 12:30 | |
| *** zhubingbing has joined #openstack-kolla | 12:37 | |
| duankebo | des_consolado, thank you! I have read the document. | 12:38 |
| duankebo | des_consolado, It may be a stupid question, who will forward log to elasticsearch? heka? | 12:40 |
| *** rhallisey has joined #openstack-kolla | 12:41 | |
| *** erlon has joined #openstack-kolla | 12:43 | |
| *** zhurong has quit IRC | 12:45 | |
| sdake | berendt morning - I am not going to be at the wednesday meeting | 12:45 |
| sdake | berendt although I'm sure you can handle it without me | 12:46 |
| *** dave-mccowan has joined #openstack-kolla | 12:56 | |
| *** shardy_lunch is now known as shardy | 12:59 | |
| *** eaguilar has joined #openstack-kolla | 13:00 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 13:01 |
| *** sudipto has joined #openstack-kolla | 13:09 | |
| *** sudipto_ has joined #openstack-kolla | 13:09 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Run precheck after deployed openstack https://review.openstack.org/442289 | 13:10 |
| Jeffrey4l_ | egonzalez, could u review this one https://review.openstack.org/441788 the following patch works ( run precheck after deploy ) | 13:11 |
| *** mnasiadka has joined #openstack-kolla | 13:14 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Access placement service through internal interface rather than public https://review.openstack.org/442466 | 13:18 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ironic-pxe: handle non-x86 architectures https://review.openstack.org/434817 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: enable all images enabled for Ubuntu https://review.openstack.org/432787 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: Add support for non-x86 architectures (aarch64, ppc64le) https://review.openstack.org/430940 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-libvirt: handle ppc64le https://review.openstack.org/434810 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: ceph: move ceph-fuse package to cephfs-fuse image https://review.openstack.org/440634 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: nova-compute: handle rtslib(-fb) for debian-binary builds too https://review.openstack.org/435941 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: move to stretch https://review.openstack.org/434453 | 13:20 |
| openstackgerrit | Marcin Juszkiewicz proposed openstack/kolla master: debian: install gnupg and dirmngr needed for apt-key https://review.openstack.org/434431 | 13:20 |
| *** caowei has quit IRC | 13:25 | |
| *** salv-orlando has joined #openstack-kolla | 13:28 | |
| *** Margin has joined #openstack-kolla | 13:28 | |
| *** Margin has quit IRC | 13:30 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Use internal interface instead of public interface https://review.openstack.org/441752 | 13:30 |
| *** Margin has joined #openstack-kolla | 13:30 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 13:33 |
| *** zhurong has joined #openstack-kolla | 13:35 | |
| *** sps_ has quit IRC | 13:36 | |
| *** salv-orl_ has joined #openstack-kolla | 13:36 | |
| *** Andrew_jedi has quit IRC | 13:38 | |
| *** salv-orlando has quit IRC | 13:39 | |
| *** rhallisey_ has joined #openstack-kolla | 13:46 | |
| *** salv-orl_ has quit IRC | 13:46 | |
| *** rhallisey has quit IRC | 13:50 | |
| *** tonanhngo has joined #openstack-kolla | 13:50 | |
| *** rwallner has quit IRC | 13:50 | |
| *** rwallner has joined #openstack-kolla | 13:50 | |
| *** rhallisey_ is now known as rhallisey | 13:53 | |
| *** sudipto_ has quit IRC | 13:54 | |
| *** sudipto has quit IRC | 13:54 | |
| *** tonanhngo has quit IRC | 13:55 | |
| *** crushil has joined #openstack-kolla | 13:55 | |
| *** yingjun has joined #openstack-kolla | 14:03 | |
| *** fguillot has joined #openstack-kolla | 14:04 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix the failure when running precheck after deploy https://review.openstack.org/441788 | 14:05 |
| openstackgerrit | Merged openstack/kolla-ansible master: Run precheck after deployed openstack https://review.openstack.org/442289 | 14:06 |
| *** mdnadeem has quit IRC | 14:06 | |
| *** goldyfruit has joined #openstack-kolla | 14:12 | |
| *** jtriley has joined #openstack-kolla | 14:13 | |
| *** lamt has joined #openstack-kolla | 14:14 | |
| *** zhurong has quit IRC | 14:16 | |
| rwellum | portdirect, bmace hi guys - can you take a look at: http://paste.openstack.org/show/601772/ - I've tried as root and non root but I always get permission errors, when running: run_dev_image.sh | 14:17 |
| *** schwicht has joined #openstack-kolla | 14:21 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix booting from volume failure https://review.openstack.org/442519 | 14:24 |
| *** tonanhngo has joined #openstack-kolla | 14:24 | |
| Jeffrey4l_ | inc0, around? | 14:24 |
| *** Deys has joined #openstack-kolla | 14:25 | |
| kfox1111 | morning. | 14:27 |
| *** tonanhngo has quit IRC | 14:29 | |
| *** sps_ has joined #openstack-kolla | 14:29 | |
| *** satyar has joined #openstack-kolla | 14:30 | |
| *** lamt has quit IRC | 14:32 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Access placement service through internal interface rather than public https://review.openstack.org/442466 | 14:32 |
| *** Margin has quit IRC | 14:35 | |
| *** Margin has joined #openstack-kolla | 14:35 | |
| *** rhallisey_ has joined #openstack-kolla | 14:36 | |
| *** Andrew_jedi has joined #openstack-kolla | 14:37 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 14:38 |
| *** rhallisey has quit IRC | 14:39 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 14:46 |
| sbezverk | kfox1111: good morning | 14:47 |
| sbezverk | kfox1111: been busy? | 14:47 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Adding nova-compute-ironic service chart https://review.openstack.org/439841 | 14:50 |
| *** tovin07 has joined #openstack-kolla | 14:53 | |
| *** tovin07 has left #openstack-kolla | 14:54 | |
| duankebo | des_consolado, it's because of using the old version. Thank you! | 14:54 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add debian related test env https://review.openstack.org/437761 | 14:55 |
| Jeffrey4l_ | egonzalez, could u review https://review.openstack.org/442519 | 14:55 |
| Jeffrey4l_ | which is critical for ocata branch. | 14:55 |
| *** lamt has joined #openstack-kolla | 15:03 | |
| *** rwallner has quit IRC | 15:06 | |
| *** pcaruana has quit IRC | 15:09 | |
| kfox1111 | sbezverk: yeah. still dealing with fallout from that ups. :/ | 15:09 |
| kfox1111 | should be a little more free today I hope. | 15:10 |
| *** yingjun has quit IRC | 15:11 | |
| *** rwallner has joined #openstack-kolla | 15:11 | |
| *** lucasxu has joined #openstack-kolla | 15:11 | |
| *** rwallner has quit IRC | 15:11 | |
| sbezverk | kfox1111: good to hear that you are coming back to life .. :) | 15:12 |
| *** rwallner has joined #openstack-kolla | 15:12 | |
| *** rwallner has quit IRC | 15:12 | |
| kfox1111 | I'm really close to having a cncf corperate cla signed too. | 15:13 |
| kfox1111 | not having one has really tied my hands. :/ | 15:13 |
| *** TxGirlGeek has joined #openstack-kolla | 15:13 | |
| kfox1111 | I was able to get prometheus/ganglia/nginx-ingress deployed here with some minor tweaks to the charts. want to get those upstreamed. | 15:14 |
| sbezverk | kfox1111: cool, what is ganglia? | 15:15 |
| kfox1111 | arg. meant grafana. | 15:15 |
| kfox1111 | they chose too many names similar to each other in monitoring. gangia/grafana/graphite. | 15:16 |
| sbezverk | kfox1111: :) I am about to deploy VM with ironic | 15:16 |
| kfox1111 | sweet. :) | 15:17 |
| sbezverk | kfox1111: when you have time, check my wip, appreciate suggestions to make things better.. | 15:18 |
| kfox1111 | sure. :) | 15:19 |
| *** matrohon has joined #openstack-kolla | 15:21 | |
| *** pcaruana has joined #openstack-kolla | 15:23 | |
| *** dgonzalez has joined #openstack-kolla | 15:24 | |
| *** Margin has quit IRC | 15:24 | |
| *** salv-orlando has joined #openstack-kolla | 15:24 | |
| *** Margin has joined #openstack-kolla | 15:25 | |
| *** sps_ has quit IRC | 15:27 | |
| *** rwallner has joined #openstack-kolla | 15:28 | |
| *** goldyfruit1 has joined #openstack-kolla | 15:28 | |
| *** goldyfruit has quit IRC | 15:28 | |
| *** Margin has quit IRC | 15:31 | |
| *** saneax is now known as saneax-_-|AFK | 15:35 | |
| kfox1111 | sbezverk: in general, the gate job is looking great. :) | 15:36 |
| *** aolwas has quit IRC | 15:38 | |
| kfox1111 | couple of optional tweaks that could make it a little simpler. but not critical. | 15:38 |
| *** nhlfr has quit IRC | 15:38 | |
| *** manheim_ has quit IRC | 15:39 | |
| *** mattmceuen has joined #openstack-kolla | 15:43 | |
| *** GK___1wm____SU has joined #openstack-kolla | 15:46 | |
| *** GK___1wm____SU has left #openstack-kolla | 15:46 | |
| *** GK___1wm____SU has quit IRC | 15:46 | |
| *** adrian_otto has joined #openstack-kolla | 15:46 | |
| *** lamt has quit IRC | 15:50 | |
| *** mattmceuen has quit IRC | 15:50 | |
| *** nhlfr has joined #openstack-kolla | 15:50 | |
| *** mattmceuen has joined #openstack-kolla | 15:50 | |
| *** tvignaud has quit IRC | 15:52 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 15:52 |
| sbezverk | kfox1111: thank you for review.. | 15:54 |
| *** manheim_ has joined #openstack-kolla | 15:55 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix booting from volume failure https://review.openstack.org/442519 | 15:55 |
| *** manheim__ has joined #openstack-kolla | 15:55 | |
| zhubingbing | sup guys | 15:55 |
| *** manheim_ has quit IRC | 15:59 | |
| *** manheim__ has quit IRC | 16:00 | |
| *** manheim_ has joined #openstack-kolla | 16:00 | |
| *** lamt has joined #openstack-kolla | 16:00 | |
| egonzalez | Jeffrey4l_, when was the change to nova/cinder/ceph included? was made in ceph or nova? | 16:00 |
| Jeffrey4l_ | o cycle. | 16:00 |
| *** hieulq_ has joined #openstack-kolla | 16:01 | |
| Jeffrey4l_ | it is related a change in nova. | 16:01 |
| Jeffrey4l_ | let me show you the code. | 16:01 |
| Jeffrey4l_ | egonzalez, check https://github.com/openstack/nova/blob/master/nova/virt/libvirt/volume/net.py#L56,L80 | 16:02 |
| egonzalez | Jeffrey4l_, thanks | 16:02 |
| Jeffrey4l_ | np. | 16:02 |
| *** tvignaud has joined #openstack-kolla | 16:04 | |
| egonzalez | Jeffrey4l_, looks good | 16:05 |
| Jeffrey4l_ | my patch? | 16:05 |
| egonzalez | yep | 16:08 |
| Jeffrey4l_ | yep. i am running the last test for it. should be OK now. | 16:09 |
| *** dave-mccowan has quit IRC | 16:10 | |
| *** dave-mccowan has joined #openstack-kolla | 16:11 | |
| *** rhallisey_ is now known as rhallisey | 16:12 | |
| *** tonanhngo has joined #openstack-kolla | 16:20 | |
| *** kristian__ has joined #openstack-kolla | 16:21 | |
| *** crushil has quit IRC | 16:22 | |
| *** tonanhngo has quit IRC | 16:25 | |
| *** hieulq_ has quit IRC | 16:26 | |
| *** pcaruana has quit IRC | 16:26 | |
| Jeffrey4l_ | egonzalez, please review https://review.openstack.org/442531 and its following patch. | 16:26 |
| Jeffrey4l_ | https://review.openstack.org/442532 and https://review.openstack.org/442559 | 16:27 |
| *** gfhellma has joined #openstack-kolla | 16:36 | |
| kfox1111 | sbezverk: np. thanks for working on it. thats a very cool piece of code. :) | 16:36 |
| *** lucasxu has quit IRC | 16:38 | |
| *** JoseMello has joined #openstack-kolla | 16:39 | |
| bmace | rwellum: instead of executing the run_dev_image script you can just docker run --entrypoint=/bin/bash the image and then examine the properties on that file.. tweak / fix them if need be, and then you can execute /opt/kolla-kubernetes/tools/setup_dev_env.sh manually rather than having it run from that run_dev_image script.. should work just as well | 16:39 |
| *** lucasxu has joined #openstack-kolla | 16:40 | |
| *** shardy has quit IRC | 16:47 | |
| *** crushil has joined #openstack-kolla | 16:48 | |
| *** shardy has joined #openstack-kolla | 16:49 | |
| *** strigazi is now known as strigazi_AFK | 16:53 | |
| *** mnasiadka has quit IRC | 17:05 | |
| *** skramaja has quit IRC | 17:05 | |
| *** manheim_ has quit IRC | 17:05 | |
| Jeffrey4l_ | egonzalez, any idea about ubuntu gate failed now? | 17:10 |
| *** magicboiz has quit IRC | 17:11 | |
| openstackgerrit | Bertrand Lallau proposed openstack/kolla-ansible master: Update searchlight configuration https://review.openstack.org/442628 | 17:14 |
| *** tonanhngo has joined #openstack-kolla | 17:15 | |
| *** ntpttr has joined #openstack-kolla | 17:15 | |
| *** lucasxu has quit IRC | 17:16 | |
| *** gfhellma_ has joined #openstack-kolla | 17:18 | |
| *** gfhellma has quit IRC | 17:18 | |
| *** zhubingbing has quit IRC | 17:18 | |
| *** tonanhngo has quit IRC | 17:20 | |
| *** vhosakot has joined #openstack-kolla | 17:20 | |
| *** egonzalez has quit IRC | 17:20 | |
| *** gfidente has quit IRC | 17:22 | |
| *** gfhellma_ has quit IRC | 17:23 | |
| *** gfhellma_ has joined #openstack-kolla | 17:28 | |
| *** matrohon has quit IRC | 17:29 | |
| *** MasterOfBugs has joined #openstack-kolla | 17:30 | |
| *** pramodrj07 has joined #openstack-kolla | 17:30 | |
| *** lucasxu has joined #openstack-kolla | 17:31 | |
| openstackgerrit | Alicja Kwasniewska proposed openstack/kolla-ansible master: Update Multinode deployment of kolla guide https://review.openstack.org/426852 | 17:32 |
| *** gfhellma_ has quit IRC | 17:38 | |
| *** erhudy has joined #openstack-kolla | 17:42 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 17:43 |
| *** kristian__ has quit IRC | 17:46 | |
| *** Andrew_jedi has quit IRC | 17:48 | |
| *** sudipto has joined #openstack-kolla | 17:48 | |
| *** sudipto_ has joined #openstack-kolla | 17:48 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Update Multinode deployment of kolla guide https://review.openstack.org/426852 | 17:49 |
| inc0 | good morning | 17:50 |
| *** david-lyle_ has joined #openstack-kolla | 17:52 | |
| *** david-lyle has quit IRC | 17:54 | |
| *** david-lyle_ is now known as david-lyle | 17:55 | |
| inc0 | Jeffrey4l_: you there? | 17:56 |
| Jeffrey4l_ | inc0, yep. | 17:56 |
| inc0 | I'm looking at this perma red ubuntu gate | 17:56 |
| Jeffrey4l_ | me too. but still no idea ;( | 17:56 |
| inc0 | you have any idea why that happends? I think it became red around cells | 17:56 |
| Jeffrey4l_ | no. simple about console.log file permission. | 17:57 |
| Jeffrey4l_ | nova create a empty console.log but libvirt failed to write to it. | 17:57 |
| Jeffrey4l_ | no idea why centos works. | 17:57 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 17:57 |
| *** sudipto has quit IRC | 18:00 | |
| *** sudipto_ has quit IRC | 18:00 | |
| inc0 | hmm | 18:02 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DEBUGING Ubuntu owner issue https://review.openstack.org/438341 | 18:03 |
| *** david-lyle has quit IRC | 18:04 | |
| *** david-lyle has joined #openstack-kolla | 18:05 | |
| *** mgoddard has quit IRC | 18:05 | |
| *** sambetts is now known as sambetts|afk | 18:08 | |
| *** gfhellma has joined #openstack-kolla | 18:10 | |
| *** gfhellma has quit IRC | 18:10 | |
| *** gfhellma has joined #openstack-kolla | 18:11 | |
| *** athomas has quit IRC | 18:15 | |
| *** gfhellma has quit IRC | 18:15 | |
| *** rwallner has quit IRC | 18:17 | |
| *** david-lyle has quit IRC | 18:23 | |
| *** tonanhngo has joined #openstack-kolla | 18:23 | |
| *** david-lyle has joined #openstack-kolla | 18:25 | |
| inc0 | sean-k-mooney: around? | 18:28 |
| sean-k-mooney | hi yes | 18:28 |
| *** tonanhngo has quit IRC | 18:28 | |
| sean-k-mooney | inc0: how are things :) | 18:29 |
| inc0 | correct me if I'm wrong, but this simplke thing will allow you to deploy dpdk right? https://review.openstack.org/#/c/440834/ | 18:29 |
| *** david-lyle has quit IRC | 18:29 | |
| inc0 | just provide playbook to deploy ovs+dpdk on host and let Kolla do the rest? | 18:29 |
| inc0 | not ideal but will get us somewhere without big problems | 18:30 |
| sean-k-mooney | if we have external ovs yes | 18:30 |
| sean-k-mooney | it would also enable a few other usecases | 18:30 |
| sean-k-mooney | inc0 i would still like to reuse kolla build system for the image but i could host a playbook in networking-ovs-dpdk for deploying it and use a templte overide to replace the packages in the default kolla ovs image with ovs-dpdk | 18:32 |
| sean-k-mooney | inc0: i could put the playbook into the contrib repo with the templete override instead of networking-ovs-dpdk also | 18:33 |
| Jeffrey4l_ | inc0, Ubuntu ocata is using libvirt 2.5.0. after fallback to libvirt 1.3.1, it works ;). no idea what's happening. | 18:33 |
| sean-k-mooney | Jeffrey4l_: really i taught ocata was going to use 2.7 | 18:34 |
| sean-k-mooney | e.g. ovs 2.7 | 18:34 |
| Jeffrey4l_ | i am sayings libvirt. it is using 2.5.0 | 18:34 |
| sean-k-mooney | Jeffrey4l_: have you enabled the could archive? | 18:34 |
| *** shardy has quit IRC | 18:34 | |
| Jeffrey4l_ | yes. | 18:34 |
| Jeffrey4l_ | 2.5.0-3ubuntu2~cloud0 500 | 18:35 |
| sean-k-mooney | and the only have ovs 2.5 for the ocata branch? | 18:35 |
| Jeffrey4l_ | 500 http://ubuntu-cloud.archive.canonical.com/ubuntu xenial-updates/ocata/main amd64 Packages | 18:35 |
| Jeffrey4l_ | i am saying libvirt , not ovs. | 18:35 |
| *** david-lyle has joined #openstack-kolla | 18:36 | |
| inc0 | damn -.- | 18:36 |
| sean-k-mooney | Jeffrey4l_: oh well ovs is 2.6.1 http://ubuntu-cloud.archive.canonical.com/ubuntu/dists/xenial-proposed/ocata/main/binary-amd64/Packages libvrit is 2.5.0-3ubuntu2~cloud0 | 18:36 |
| Jeffrey4l_ | yep. | 18:37 |
| *** rwellum_ has joined #openstack-kolla | 18:37 | |
| sean-k-mooney | Jeffrey4l_: do we run the new libvirt virtlog deamon in kolla | 18:37 |
| Jeffrey4l_ | sean-k-mooney, sorry? libvirt virt log ? | 18:38 |
| inc0 | Jeffrey4l_: soo....just pin libvirt version to <2 for now? | 18:39 |
| sean-k-mooney | im trying to rembere the name of the service let me check one of my dev enviornments | 18:39 |
| Jeffrey4l_ | inc0, i am trying to push a testing patch for this. | 18:39 |
| Jeffrey4l_ | but it is not a good idea. | 18:39 |
| *** salv-orlando has quit IRC | 18:39 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Test fallback to libvirt 1.3 on Ubuntu https://review.openstack.org/442673 | 18:40 |
| *** rwellum has quit IRC | 18:40 | |
| Jeffrey4l_ | inc0, ^^ check. | 18:40 |
| *** mgoddard has joined #openstack-kolla | 18:40 | |
| Jeffrey4l_ | i am going to bed. will check this when get up. | 18:40 |
| inc0 | I'll take over, thanks | 18:41 |
| Jeffrey4l_ | and please check this critical issue https://review.openstack.org/442519 inc0 | 18:41 |
| Jeffrey4l_ | thanks ; ) | 18:41 |
| sean-k-mooney | Jeffrey4l_: the virtlogd service | 18:42 |
| sean-k-mooney | Jeffrey4l_: libvirt 2.x requires it for libvrt to run as far as i know | 18:43 |
| Jeffrey4l_ | hrm. i have no idea for this.. | 18:43 |
| Jeffrey4l_ | what will happen if this service do not exist? | 18:43 |
| sean-k-mooney | Jeffrey4l_: libvirt uses it for all its loging | 18:44 |
| *** salv-orlando has joined #openstack-kolla | 18:44 | |
| sean-k-mooney | Jeffrey4l_: without it i dont thikn you cans start the libvirt deamon anymore | 18:44 |
| sean-k-mooney | this is an old change though we fixed it in devstack over a year ago | 18:44 |
| Jeffrey4l_ | ok. it may be matter. | 18:44 |
| Jeffrey4l_ | sean-k-mooney, the gate is using libvirt > 2 one year ago? | 18:45 |
| sean-k-mooney | Jeffrey4l_: i may be configurable it loogs like the change happended in 1.3.1. sorry this is proabbly a tangent that is unrealted to your issue | 18:46 |
| Jeffrey4l_ | my issue is related to "console.log" | 18:46 |
| sean-k-mooney | libvirt 2.5.0 however works fine with devstack on ubuntu 16.04 | 18:46 |
| Jeffrey4l_ | libvirt is complaining he have no permission to this file. | 18:47 |
| Jeffrey4l_ | ok. let me test with virtlog service. | 18:47 |
| sean-k-mooney | Jeffrey4l_: perhaps apparmor is to blame? did you remove the apparmor profile for libvirt? | 18:48 |
| Jeffrey4l_ | sean-k-mooney, no. i am running ubuntu image on centos. ( selinux is disabled ) | 18:48 |
| Jeffrey4l_ | sean-k-mooney, libvirtError: internal error: process exited while connecting to monitor: 2017-03-07T18:49:11.579207Z qemu-system-x86_64: -chardev socket,id=charserial0,host=10.3.0.20,port=10000,server,nowait,logfile=/var/lib/nova/instances/de33a522-a568-4724-8461-9ae296f0867e/console.log,logappend=off: Unable to open logfile /var/lib/nova/instances/de33a522-a568-4724-8461-9ae296f0867e/console.log: Permission denied | 18:49 |
| Jeffrey4l_ | got this. get the same error when enabled virtlog service in libvirt container. | 18:49 |
| sean-k-mooney | interesting... i know i have got that to work in the past but i also know their have been a lot of issue mixing ubunut and centos | 18:50 |
| Jeffrey4l_ | at least, this kind of mix should not cause console.log permission issue. ;) | 18:51 |
| sean-k-mooney | Jeffrey4l_: if you exec into the container can you cat/touch the file | 18:51 |
| sean-k-mooney | Jeffrey4l_: just wondering if there a user/group id issue or something | 18:51 |
| Jeffrey4l_ | what i know is: libvirt ( run as root ) should chown the file . | 18:52 |
| Jeffrey4l_ | to the write permissions. | 18:52 |
| Jeffrey4l_ | to the right permissions. | 18:52 |
| Jeffrey4l_ | this is called dynamic ownership in libvirt. | 18:52 |
| Jeffrey4l_ | and libvirt 1.3.x works anb libvirt 2.5.0 doesn't . | 18:53 |
| Jeffrey4l_ | i guess libvirt changed something. | 18:53 |
| sean-k-mooney | that is not how it works as far as i understood. in anycase though does the qemu user have permission to create files in that folder? | 18:53 |
| Jeffrey4l_ | sean-k-mooney, check this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2922 | 18:54 |
| Jeffrey4l_ | absolution mo. that folder is nova:nova with 755 | 18:55 |
| sean-k-mooney | and if you checkout your /etc/libvirt/qemu.conf is the qemu user set to nova:nova? | 18:56 |
| Jeffrey4l_ | nova. user/group is not set. | 18:57 |
| Jeffrey4l_ | hrm ? nova:nova is better? | 18:57 |
| sean-k-mooney | so libvirt should not be running the qemu instaces as root but by default if you install form apt it will run them in the qemu group | 18:58 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DEBUGING Ubuntu owner issue https://review.openstack.org/438341 | 18:58 |
| sean-k-mooney | i dont know how we do it in kolla as i never checked that out | 18:58 |
| inc0 | I don't think we do anything with qemu groupo | 18:58 |
| Jeffrey4l_ | i am curiosity why it works ( without kolla ) | 18:59 |
| inc0 | Jeffrey4l_: most of the time people run it all as root | 18:59 |
| Jeffrey4l_ | for libvirt case, should be not. | 19:00 |
| Jeffrey4l_ | wow. change the qemu user/group to nova:nova works ;( | 19:01 |
| sean-k-mooney | well by default the libvirt deamon runs as root but the qemu process does not | 19:01 |
| Jeffrey4l_ | sean-k-mooney, correct. | 19:01 |
| sean-k-mooney | at leas if you instlal from the package mangaer | 19:01 |
| inc0 | Jeffrey4l_: do we create qemu group? | 19:02 |
| inc0 | with static gid? | 19:02 |
| sean-k-mooney | inc0: no i dont think so | 19:02 |
| Jeffrey4l_ | inc0, yesp. | 19:02 |
| sean-k-mooney | Jeffrey4l_: with a static gid | 19:02 |
| Jeffrey4l_ | we created qemu user with static gid. | 19:03 |
| inc0 | sean-k-mooney: we create groups and users with static gids | 19:03 |
| inc0 | qemu user...but qemu group too? | 19:03 |
| Jeffrey4l_ | libvirt run process with libvirt-qemu user in ubuntu+latest libvirt. | 19:04 |
| sean-k-mooney | inc0: since when. i know we chmod them but i have neever see the code where we create them | 19:04 |
| Jeffrey4l_ | but i still donot think it is cause by this. | 19:04 |
| inc0 | sean-k-mooney: early ocata I think | 19:04 |
| *** david-lyle has quit IRC | 19:04 | |
| sean-k-mooney | inc0: in docker image or in ansible? | 19:04 |
| Jeffrey4l_ | libvirt should re-chown the console.log file to correct user/group. | 19:04 |
| inc0 | sean-k-mooney: image | 19:04 |
| inc0 | let me show you code | 19:04 |
| sean-k-mooney | oh i was looking at the ansible scripts | 19:05 |
| Jeffrey4l_ | libvirt 1.3.1 does, but libvirt 2.5 doesn't | 19:05 |
| inc0 | https://github.com/openstack/kolla/blob/master/docker/base/Dockerfile.j2#L7-L11 | 19:05 |
| inc0 | https://github.com/openstack/kolla/blob/master/kolla/common/config.py#L535-L768 | 19:05 |
| *** david-lyle has joined #openstack-kolla | 19:06 | |
| sean-k-mooney | that good to know i had a horrible hack https://review.openstack.org/#/c/408872/3/ansible/roles/nova/templates/qemu.conf.j2 for testing because i needed to make the group match between ovs and the qemu process | 19:06 |
| inc0 | Jeffrey4l_: how fun...so ubuntu broken their upgrade of libvirt right? | 19:06 |
| inc0 | you need to re-chown vm files | 19:06 |
| Jeffrey4l_ | i guess so. but i am not sure. | 19:06 |
| inc0 | sean-k-mooney: issue we've hit is if you install mariadb from apt, it will add user | 19:06 |
| inc0 | and if you run it again, uid might not be the same | 19:07 |
| inc0 | so during upgrade your mariadb datafiles were permission-denied'ed | 19:07 |
| sean-k-mooney | inc0: i see. well we are missing a group for openvswitch | 19:07 |
| sean-k-mooney | and user | 19:07 |
| inc0 | I believe ovs is running in privileged container | 19:08 |
| *** crushil has quit IRC | 19:08 | |
| sean-k-mooney | yes but for ovs-dpdk to work the shared unix socket need to be readable and wriable by both the vm and ovs | 19:08 |
| sean-k-mooney | anyway that a problem for another day | 19:08 |
| inc0 | yeah | 19:09 |
| sean-k-mooney | usably we handle that by running ovs in the same group as qemu and changing the umask so that group member have rw to files by default | 19:09 |
| *** rwallner has joined #openstack-kolla | 19:09 | |
| inc0 | well adding user/gid for ovs is fine | 19:10 |
| inc0 | we could then pin it to the same gid | 19:10 |
| sean-k-mooney | inc0: Jeffrey4l_ is there any reason not to run qemu in the nova group? | 19:10 |
| sean-k-mooney | inc0: ya that would work | 19:10 |
| Jeffrey4l_ | sean-k-mooney, there is no one recommend this. | 19:10 |
| inc0 | sean-k-mooney: not that I can think of... adding user to second group is not a big deal | 19:11 |
| inc0 | question....does centos run on libvirt 1.3 or 2.5? | 19:11 |
| Jeffrey4l_ | inc0, 2.0 | 19:11 |
| inc0 | right...why does it work then? | 19:11 |
| inc0 | can we pin version of libvirt to same between ubuntu and centos please? | 19:11 |
| Jeffrey4l_ | -rw-r--r-- 1 qemu qemu 19K Mar 8 02:03 console.log | 19:11 |
| sean-k-mooney | inc0: no | 19:12 |
| Jeffrey4l_ | libvirt chown the file to the qemu:qemu from nova:nova. | 19:12 |
| sean-k-mooney | it would be nice but they dont ship the same version in the package manager | 19:12 |
| *** eaguilar has quit IRC | 19:12 | |
| Jeffrey4l_ | inc0, i do not think it is a good idea. | 19:12 |
| inc0 | oh joy | 19:12 |
| inc0 | Jeffrey4l_: soo..add nova user to qemu group? | 19:13 |
| sean-k-mooney | Jeffrey4l_: ya that is what i would expect unless you set the config value | 19:13 |
| *** matrohon has joined #openstack-kolla | 19:13 | |
| inc0 | or config change | 19:13 |
| Jeffrey4l_ | or fallback to 1.13 ;( | 19:13 |
| Jeffrey4l_ | my patch seems works. | 19:13 |
| inc0 | I'd rather fix user and not use ancient libvrt with different major version than centos | 19:13 |
| Jeffrey4l_ | we may need ask for help from ubuntu. | 19:14 |
| Jeffrey4l_ | i guess there is something wrong with libvirt 2.5. | 19:14 |
| sean-k-mooney | well ubuntu is doing the right thing | 19:14 |
| sean-k-mooney | it should be running under the qemu group | 19:14 |
| inc0 | yeah I guess they didn't think of user separation way we do | 19:15 |
| sean-k-mooney | again uless you set the config value which i think is the better option. or add nova to the qemu group | 19:15 |
| Jeffrey4l_ | use nova:nova to run qemu is a little weird. | 19:15 |
| Jeffrey4l_ | fyi, libvirt 1.13 work as expect | 19:15 |
| Jeffrey4l_ | sean-k-mooney, i tried. won't works. | 19:15 |
| sean-k-mooney | Jeffrey4l_: you would proably need to change the umask too | 19:15 |
| *** satyar has quit IRC | 19:15 | |
| Jeffrey4l_ | i added all users related to libvirt into nova group , and nova user to related libvirt groups. | 19:16 |
| sean-k-mooney | why is the libvrt demaon runing as nova anyway? | 19:16 |
| Jeffrey4l_ | the issue is still happend. | 19:16 |
| sean-k-mooney | this is not a libvirt thing | 19:16 |
| Jeffrey4l_ | sean-k-mooney, that is not how libvirt works. | 19:16 |
| Jeffrey4l_ | i want to know: why libvirt dynamic ownship do not works | 19:16 |
| Jeffrey4l_ | libvirt run as root. he know and he should chown the log file. | 19:17 |
| sean-k-mooney | as far as i understand the behavior you are seeing is the correct one | 19:17 |
| sean-k-mooney | i should be changin it to the qemu group | 19:17 |
| sean-k-mooney | that is the default expected behavior | 19:17 |
| Jeffrey4l_ | sean-k-mooney, agree. | 19:17 |
| Jeffrey4l_ | libvirt 1.13, 2.0 works like this. | 19:18 |
| Jeffrey4l_ | but 2.5 doesn't . | 19:18 |
| sean-k-mooney | so did you add the qemu user to the nova group? | 19:18 |
| *** crushil has joined #openstack-kolla | 19:18 | |
| Jeffrey4l_ | yes. tryied. | 19:18 |
| inc0 | can we add nova user to qemu group? | 19:18 |
| Jeffrey4l_ | add user to another group won't work. | 19:18 |
| inc0 | files are not gid-readable? | 19:19 |
| Jeffrey4l_ | drwxr-xr-x 1 nova nova 22 Mar 8 02:02 206e65da-070a-4558-8eee-9c43e5183cc5 | 19:19 |
| Jeffrey4l_ | see? only nova user have wirte permission. | 19:19 |
| inc0 | on directory...how about console.log iteself? | 19:19 |
| inc0 | also maybe tha'ts configurable? | 19:19 |
| Jeffrey4l_ | nova create it by using nova:nova . | 19:20 |
| inc0 | ehh | 19:20 |
| *** mgoddard has quit IRC | 19:20 | |
| Jeffrey4l_ | inc0, check this https://github.com/openstack/nova/blob/master/nova/virt/libvirt/driver.py#L2922 | 19:20 |
| rwellum_ | bmace, sorry missed your response earlier. Talking about docker - which image do you mean - when you say: "docker run --entrypoint=/bin/bash the image" | 19:20 |
| bmace | the dev image that you built? | 19:21 |
| Jeffrey4l_ | so, only two solution: 1. run qemu with nova:nova 2. fix dynamic ownership on libvirt | 19:21 |
| sean-k-mooney | Jeffrey4l_: yes to fix that you would have to alter the umask to - | 19:21 |
| sean-k-mooney | Jeffrey4l_: yes to fix that you would have to alter the umask to 002 | 19:21 |
| inc0 | does ubuntu ship libvirt 2.0? | 19:21 |
| Jeffrey4l_ | sean-k-mooney, it is not related ti umask ( base on my two solution ) | 19:22 |
| Jeffrey4l_ | inc0, i think no. | 19:22 |
| rwellum_ | bmace, ok right... | 19:22 |
| *** salv-orl_ has joined #openstack-kolla | 19:22 | |
| *** rwallner has quit IRC | 19:23 | |
| Jeffrey4l_ | # Whether libvirt should dynamically change file ownership | 19:23 |
| Jeffrey4l_ | # to match the configured user/group above. Defaults to 1. | 19:23 |
| Jeffrey4l_ | # Set to 0 to disable file ownership changes. | 19:23 |
| Jeffrey4l_ | #dynamic_ownership = 1 | 19:23 |
| sean-k-mooney | well the umask for the service will alter the file permission but again i would just set the user and group in the /etc/libvirt/qemu.conf that will work for all version and keep the same behavior | 19:24 |
| *** salv-orlando has quit IRC | 19:25 | |
| Jeffrey4l_ | sean-k-mooney, it may works. | 19:26 |
| inc0 | yeah...1.3 or 2.5 | 19:26 |
| Jeffrey4l_ | but this is a issue from libvirt 2.5. | 19:26 |
| Jeffrey4l_ | that just a dirty trick. | 19:26 |
| Jeffrey4l_ | really need go to sleep. bye guys. | 19:26 |
| inc0 | np Jeffrey4l_ thanks | 19:27 |
| inc0 | I'll keep digging | 19:27 |
| sean-k-mooney | Jeffrey4l_: not that is the correct way to specify the ownership fo files created by qemu | 19:27 |
| Jeffrey4l_ | cool | 19:27 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 19:31 |
| sbezverk | Jeffrey4l_: qq, if I need to change default nova scheduler, the change goes into [scheduler] section in nova.conf, right? | 19:32 |
| sean-k-mooney | sbezverk: in the [DEFAULT] section in the nova.conf set scheduler_default_filters and scheduler_driver | 19:36 |
| sean-k-mooney | sbezverk: if you jsut want to chagne the filters that are used scheduler_default_filters is enough | 19:37 |
| sean-k-mooney | sbezverk: if you want to use a different scheduler implnetation you need to chaange scheduler_driver | 19:37 |
| sbezverk | sean-k-mooney: it is for baremetal host aggregate | 19:42 |
| *** tonanhngo has joined #openstack-kolla | 19:42 | |
| sbezverk | does it go under default? | 19:43 |
| sean-k-mooney | inc0: just back to https://review.openstack.org/#/c/440834/2 if i added a template override and docs to the contibe folder in kolla for ovs-dpdk and a playbook the contrib repo for kolla-ansible to deploy would that work for you interms of ovs-dpdk support assuming external ovs support was merged? | 19:44 |
| Slower | inc0: heya! | 19:44 |
| inc0 | sean-k-mooney: I think so | 19:44 |
| inc0 | that's really low hanging fruit to have first proper contrib | 19:44 |
| inc0 | at least until we figure out ansible overrides | 19:45 |
| Slower | hehe, I read that as "send money"! :) | 19:45 |
| inc0 | hey Slower | 19:45 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 19:45 |
| Slower | send-money-k? :) | 19:45 |
| inc0 | :D | 19:45 |
| sean-k-mooney | Slower: i wont object if it is sent to me :) | 19:45 |
| *** rwallner has joined #openstack-kolla | 19:45 | |
| Slower | inc0: question for you on kolla-k8s.. remind me how you guys are dealing with config files there? | 19:45 |
| inc0 | sean-k-mooney: you don't know of any Nigerian royality family connections right? | 19:45 |
| inc0 | Slower: right now poorly - we generate them with ansible and add to configmaps | 19:46 |
| Slower | haha | 19:46 |
| sean-k-mooney | inc0: its funny you say that cause i have a cosin that recently got stranded abroad ... | 19:46 |
| Slower | inc0: ok, and you aren't using the kolla copy config stuff right? | 19:46 |
| inc0 | we do in fact | 19:46 |
| *** tonanhngo has quit IRC | 19:46 | |
| Slower | ah hmm interesting | 19:46 |
| inc0 | we mount configmaps to place where ansible would mount it | 19:47 |
| inc0 | and cp them inside | 19:47 |
| Slower | hmm | 19:47 |
| Slower | trying to figure out if we should keep using hte kolla copy thing in tripleo or just mount the configs directly | 19:47 |
| Slower | we don't have a permissions/ownership issue because the configs are generated with puppet using the same container that later runs them | 19:47 |
| sean-k-mooney | inc0: dose kolla-k8 reuse the config generation from kolla-ansible or does it have its own? | 19:47 |
| Slower | and it generally just makes things very verbose | 19:47 |
| inc0 | sean-k-mooney: right now it uses ansible, but we want to get rid of it | 19:48 |
| inc0 | as soon as we provide feature-parity mechanism in k8s | 19:48 |
| inc0 | with stuff like overrides | 19:48 |
| sean-k-mooney | inc0: is the plan to put it into its own repo and share or two different implentations | 19:48 |
| inc0 | Slower: permissions become an issue with upgrades | 19:48 |
| inc0 | but not any more as we use static uid/gid | 19:49 |
| Slower | hmm | 19:49 |
| inc0 | reason we did all that chown magic is if mariadb installs, it creates user | 19:49 |
| Slower | inc0: any other reasons that might be a bad idea? (to not use the copy thing) | 19:49 |
| inc0 | and uid today might be different with uid tomorrow | 19:49 |
| Slower | ah I see | 19:49 |
| inc0 | sean-k-mooney: right now plan is to have it in kolla-k8s | 19:50 |
| inc0 | Slower: it's really about safety of reconfigure | 19:50 |
| inc0 | too | 19:50 |
| inc0 | so if you mess with your files | 19:50 |
| inc0 | they won't be used until you specifically tell them to be | 19:50 |
| Slower | right, that was brought up before | 19:50 |
| Slower | in our conversations about this | 19:50 |
| Slower | I had forgotten | 19:51 |
| inc0 | it might be issue or not...really opinionated one | 19:51 |
| inc0 | so for example node dies during reconfigure | 19:51 |
| inc0 | and restarts after a while | 19:51 |
| inc0 | with our arch it will restart with old config until you specifically recreate container | 19:51 |
| sean-k-mooney | inc0: if you just precreated the mysql/mariadb users/groups would you need the chown logic or will the install fail if the user already exist? | 19:51 |
| inc0 | or it will use new config | 19:51 |
| inc0 | sean-k-mooney: once we precreated it shouldn't be problem | 19:52 |
| inc0 | chown was for upgrade from non-precreated to precreated | 19:52 |
| sean-k-mooney | ah that makes sense | 19:52 |
| sean-k-mooney | inc0: im going to head home and have dinner. my ovs-dpdk patches are in merge conflict so ill be respining theam this week anyway. | 19:54 |
| sean-k-mooney | inc0: ill split them out as you suggested when i respin | 19:54 |
| inc0 | kk sean-k-mooney, test out with external ovs approach | 19:54 |
| sean-k-mooney | inc0: yep though as i just said in teh review you should prabole rebase your patch on https://review.openstack.org/#/c/416793/ | 19:55 |
| inc0 | cool | 19:55 |
| jemcevoy | inc0 Samyaple Mech422: We have kolla deployed on the new cluster for Penguin-on-Demand cloud. Replacing br-ex with ovs running on the host fixed the neutron issues I was seeing with the VLAN provider network. | 19:57 |
| *** lucasxu has quit IRC | 19:57 | |
| inc0 | cool | 19:57 |
| inc0 | bbiaf, lunch | 19:57 |
| jemcevoy | It is up and running with neutron DVR... | 19:57 |
| sean-k-mooney | jemcevoy: sound like https://review.openstack.org/#/c/440834/2 would be useful for you then | 19:59 |
| *** Andrew_jedi has joined #openstack-kolla | 20:00 | |
| jemcevoy | sean-k-mooney Absolutely - I had to comment out the first 75 lines of https://github.com/openstack/kolla/blob/stable/newton/ansible/roles/neutron/tasks/start.yml#L2-L75 | 20:01 |
| Mech422 | jemcevoy: yeah - I use OVS for everything on the host - including mgmt access | 20:01 |
| Mech422 | jemcevoy: hoping to move to OVN in pike | 20:01 |
| jemcevoy | I am using it to bond the 10gig nics too | 20:02 |
| Mech422 | yep | 20:02 |
| sean-k-mooney | Mech422: ya i sould love to add supports for bonds to the kolla host playbook | 20:02 |
| Mech422 | sean-k-mooney: yeah - kolla-ansible loves to muck with your OVS config... | 20:02 |
| jemcevoy | CentOS does it way different than Ubuntu. | 20:02 |
| sean-k-mooney | e.g. kolla-ansible bootstrap-servers would also set up you bonds for you in ovs/linuxbridge | 20:02 |
| Mech422 | sean-k-mooney: I have about 20 interfaces defined with like 30 vlans + vxlan - so it gets annoying | 20:03 |
| sean-k-mooney | jemcevoy: it depends on if you like using network manager to manage ovs | 20:03 |
| *** salv-orl_ has quit IRC | 20:03 | |
| *** bpattewar has joined #openstack-kolla | 20:04 | |
| bpattewar | Hi | 20:04 |
| Mannheim | hi | 20:04 |
| sean-k-mooney | Mech422: 30 i assume you are spliging each of the openstack service into there own vlan. e.g. 1 vlan for db, 1 for rabbit .... | 20:04 |
| *** lucasxu has joined #openstack-kolla | 20:04 | |
| jemcevoy | It works with the olde network init scripts too... I'll write up what I had to do in the next couple weeks... | 20:04 |
| sean-k-mooney | jemcevoy: ya i just dont like mix the network init scripts / interface file and ovs | 20:05 |
| bpattewar | I am working on https://review.openstack.org/#/c/414496/ changeset. My changes are passing most of the test except one. I am not able to find any reason for failures. http://logs.openstack.org/96/414496/31/check/gate-kolla-kubernetes-deploy-centos-binary-2-ceph-multi-nv/0d94d6f/ | 20:05 |
| Mech422 | sean-k-mooney: not really - api is 1 vlan, but stuff like ceph replication/mysql replication get a VLAN | 20:05 |
| bpattewar | I am looking for some help to debug it | 20:06 |
| sean-k-mooney | you can but it can get a little annoying sometimes. that said i use it to move the tunnel enpoint ip to the br-ex by default so it has its usese | 20:06 |
| *** gfhellma has joined #openstack-kolla | 20:06 | |
| jemcevoy | No all in the api vlan.. Separate vlan for ceph, public and vxlan. and admin access on the 1Gig net | 20:07 |
| Mech422 | sean-k-mooney: I _DID_ reserve a couple of (currently unused) VLANs for the api external/admin networks | 20:07 |
| sean-k-mooney | Mech422: ya that makes sense. its kindof like how fule deploys. | 20:08 |
| Mech422 | sean-k-mooney: also thinking Ceilo and MongoDB might need their own vlan/replication vlan depending on load? | 20:08 |
| jemcevoy | sean-k-mooney... I am seeing 35MB/s coming into the VMs but only 112 KB outbound... Using 9000 MTU... any ideas where the bottleneck might be? | 20:08 |
| sean-k-mooney | well one advantage of use vlans for the different replication protoclas and rpc busses is even dumb switch suport the valn proity bit for basic qos | 20:09 |
| jemcevoy | bonding hash on the switches? | 20:09 |
| sean-k-mooney | jemcevoy: sorry i dont follow | 20:10 |
| sean-k-mooney | you only have 112KB/s upload for the vms but 35MB/s download? | 20:10 |
| bmace | lol, sounds like satellite internet | 20:11 |
| jemcevoy | I thing the datacenter has 1GB up down | 20:11 |
| jemcevoy | I an thing the fragmentation/reassembly gets out of order somewhere | 20:11 |
| sean-k-mooney | is the MTU set correctly on the veth pair that links the br-int to the linux bridge with the vm tap? | 20:11 |
| jemcevoy | All the way through... getting 135MB/s between VMs | 20:12 |
| jemcevoy | local | 20:12 |
| sean-k-mooney | jemcevoy: well that is maxing out you 1G link for local traffic | 20:13 |
| jemcevoy | ping -M do -s 8972 works everywhere local | 20:13 |
| sean-k-mooney | jemcevoy: i assume when you say local you mean inter server in the data ceneter ya? | 20:13 |
| sean-k-mooney | jemcevoy: what about between 2 vms in different neutron networks connected via a neutron router | 20:14 |
| sean-k-mooney | the most like place for the slowdown onto the net is the network node when going from the internal network to the external provider network connected to the wan | 20:15 |
| jemcevoy | Yes local is on the local 10Gig/40gig switches between local datacenters and VMs or hosts | 20:15 |
| *** gfhellma has quit IRC | 20:15 | |
| *** gfhellma has joined #openstack-kolla | 20:15 | |
| *** eaguilar has joined #openstack-kolla | 20:16 | |
| jemcevoy | It is just the internet that is slow... | 20:16 |
| jemcevoy | and only outbound | 20:17 |
| jemcevoy | where the 9000 MTU needs to fit into 1500 MTU | 20:17 |
| Mech422 | jemcevoy: I've never used neutron DVR... when I set network up, it was still a bit 'experimental' and now, I figured I skip straight to OVN instead | 20:17 |
| Mech422 | jemcevoy: sorry I cant be more help | 20:18 |
| *** tonanhngo has joined #openstack-kolla | 20:18 | |
| Mech422 | sean-k-mooney: oh! if your thinking about kolla-ansible networking... We've been thinking about Midonet | 20:20 |
| Mech422 | sean-k-mooney: OVN seems like a simpler/cleaner setup for now, but not sure if we'll need all the BGP goodness and stuff going forward | 20:20 |
| Mech422 | prolly be people doing calico and stuff too | 20:21 |
| *** rhallisey has quit IRC | 20:21 | |
| sean-k-mooney | Mech422: i like OVN but im hoping Service Function Chaining with NSH will meen i will not have to learn how BGP works | 20:21 |
| Mech422 | sean-k-mooney: I recongized 3 of those letters! | 20:23 |
| jemcevoy | I am going to have switches check to see what type of hash they are using for the bonds I have seen some cause out of order packets which combined with fragmentation would be nasty... | 20:23 |
| jemcevoy | One more thing... We had a power failure in my dev lab over the weekend... How do I restart galera/mariadb? Without a Primary... | 20:24 |
| *** Andrew_jedi has quit IRC | 20:24 | |
| sean-k-mooney | Mech422: OVNHFC are all good leters but when you combine the to OVN with NSH based SFC you have a hipster product that people would be excited for | 20:24 |
| *** tonanhngo_ has joined #openstack-kolla | 20:25 | |
| sean-k-mooney | i missed S | 20:25 |
| *** tonanhngo has quit IRC | 20:25 | |
| sean-k-mooney | Mech422: honestly thogh the simpler the networking is the better. i disslike BGP simply because its complicated to setup yourself and the specs are complicated to read | 20:26 |
| Mech422 | sean-k-mooney: dam - does that mean I have to start drinking Starbucks to project the proper hipster form ? | 20:26 |
| jemcevoy | Mech422 Do you know how to restart galera/mariadb after power failure? | 20:27 |
| Mech422 | sean-k-mooney: yeah - but we're basically a Networking company (CDN) so I have to bow to the gods of NetArch/NetEng | 20:27 |
| sean-k-mooney | Mech422: only if you want to increase you inteak of overrated suggray coffee | 20:27 |
| sean-k-mooney | Mech422: what company are you at again? | 20:28 |
| Mech422 | jemcevoy: I would start one manually with that bootstrap flag | 20:28 |
| Mech422 | jemcevoy: then boot the 2nd node - wait for it to sync, boot 3rd wait | 20:28 |
| jemcevoy | How do I pass it through docker? | 20:28 |
| Mech422 | jemcevoy: then finally restart the 1st node with no bootstrap flag | 20:28 |
| sean-k-mooney | cachfly? i rember you were at one of the top 3 cdns right? | 20:28 |
| Mech422 | sean-k-mooney: yeah - Limelight Networks | 20:28 |
| Mech422 | jemcevoy: sorry - off the top of my head, I don't remember... | 20:29 |
| jemcevoy | That needs to be documented... | 20:29 |
| Mech422 | jemcevoy: my percona cluster isn't dockerized.... | 20:29 |
| Mech422 | jemcevoy: I roll my own ceph, networking and DB | 20:29 |
| *** rwallner has quit IRC | 20:30 | |
| sean-k-mooney | ah yes yes that rings a bell. well in your case it really does pay to have as efficent a network as possible. if ovn works for you thats awsome to hear. if you need calico that good to know too. im sure other people will hit similar issue to you if you find it does not meet your needs | 20:30 |
| sean-k-mooney | Mech422: so what do you use kolla for :P | 20:31 |
| Mech422 | sean-k-mooney: yeah - not sure calico works for us with no L2 support - we were leaning towards midonet - but calico seems to be the favorite | 20:31 |
| sean-k-mooney | Mech422: but ya it makes sense you use kolla for just the openstack service and tune your infra service yourself | 20:32 |
| jemcevoy | Who might know how to start galera? I have resorted to editing some dat file to set a flag from zero to one... | 20:32 |
| Mech422 | sean-k-mooney: yeah - thats pretty much what we do - DB group was supposed to be in charge of the percona stuff, so I couldn't do kolla... | 20:32 |
| Mech422 | sean-k-mooney: but now it appears it got pushed off to me - so maybe that part can go to bog standard kolla | 20:32 |
| *** rwallner has joined #openstack-kolla | 20:33 | |
| Mech422 | sean-k-mooney: I'll prolly inherit the mongo stuff too - don't think we use it in production for anything else | 20:33 |
| *** bpattewar has quit IRC | 20:33 | |
| sean-k-mooney | Mech422: mongo for ceilometer? it would be better to move to gnocci or mysql if you are using it in production | 20:34 |
| sean-k-mooney | jemcevoy: maybe inc0 can help | 20:34 |
| Mech422 | sean-k-mooney: I thought mongo would have better perf. for Ceilo ? If I can get rid of it, that'd be awesome! | 20:35 |
| Mech422 | one less moving part! | 20:35 |
| *** JoseMello has quit IRC | 20:35 | |
| sean-k-mooney | anyway its 8:30 and i skipped lunch so off to dinner for real this time. | 20:35 |
| Mech422 | sean-k-mooney: Enjoy! | 20:35 |
| sean-k-mooney | Mech422: oh for calico not sure i assumed it only supported etcd | 20:35 |
| sean-k-mooney | sorry missread that | 20:36 |
| Mech422 | yeah - problem with percona for ceilo is its syncronous commit... | 20:36 |
| sean-k-mooney | ya ceilometer supprot mysql and gnocci. gnocci is the recommended backend now | 20:36 |
| jemcevoy | The kolla-ansible mariadb-recover has never worked for me cuz the transaction id gets reset to -1 when the initial power on fails to form a cluster and it cannot decide which is the newest | 20:36 |
| Mech422 | sean-k-mooney: AHH! I'll have to look at that | 20:36 |
| sean-k-mooney | Mech422: kolla can deploy it too | 20:37 |
| sean-k-mooney | at least kolla-ansible | 20:37 |
| Mech422 | sean-k-mooney: Awesome dude - thanks for the pointer! | 20:37 |
| * Mech422 frantically starts googling | 20:37 | |
| jemcevoy | thanks sean-k-mooney , have a healthy dinner | 20:38 |
| *** tonanhngo_ has quit IRC | 20:38 | |
| sdake | inc0 sean-k-mooney slower - the plan kfox1111 outlined is it use configmaps to provide a completely helm native implementation - no need for anything except helm and kubernetes | 20:48 |
| sdake | reference: http://eavesdrop.openstack.org/irclogs/%23openstack-kolla/%23openstack-kolla.2017-03-04.log.html#t2017-03-04T18:59:15 | 20:50 |
| sdake | needs someone to write the code :) | 20:51 |
| jemcevoy | inc0 I have two dev cluster that had a power failure over the weekend... How do I start mariadb/galera on them? | 20:55 |
| sdake | slower fwiw this is the same plan we outlined at the openstack PTG in February | 20:56 |
| *** tonanhngo has joined #openstack-kolla | 20:57 | |
| inc0 | I'm back | 20:58 |
| *** athomas has joined #openstack-kolla | 20:59 | |
| inc0 | jemcevoy: we have playbook for that | 20:59 |
| inc0 | kolla-ansible mariadb-recovery | 20:59 |
| jemcevoy | I have tried that but after the poweron and mariadb autostarts which resets all the transaction IDs on all the three servers to -1 in the dat file... It never really works... | 21:01 |
| *** tonanhngo has quit IRC | 21:01 | |
| *** rwallner has quit IRC | 21:01 | |
| jemcevoy | Maybe mariadb should not autstart | 21:01 |
| *** rhallisey has joined #openstack-kolla | 21:02 | |
| SamYaple | jemcevoy: mariadb autostarting isn't whats reseting the stuff to -1 | 21:03 |
| SamYaple | it wont update the grastate.dat until it successfully joins a cluster | 21:03 |
| SamYaple | its like the 10 second timeout from a 'docker stop mariadb' that is crashing the mariadb server | 21:03 |
| *** rwallner has joined #openstack-kolla | 21:03 | |
| *** rwallner_ has joined #openstack-kolla | 21:05 | |
| jemcevoy | SamYaple So the restart is to "ipmitool power on" then login and stop all the mariadb instances, then run kolla-ansible mariadb-recovery | 21:05 |
| vhosakot | bmace: ping, I finally increased my RAM and disk of my machine (:)) and ready to run build_dev_image.sh and run_dev_image.sh in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html. Can you confirm if 4GB RAM is still needed for kube2 per the diff you sent the other day http://paste.openstack.org/show/598475/ ? | 21:06 |
| bmace | vhosakot: hey. 4 actually worked last time i ran it. if you want to give it 6 it probably wouldn't hurt? depends on how much you actually plan to do with the env beyond just deploying it and how much ram you have available. | 21:07 |
| vhosakot | bmace: thanks for the info. so, I have 32 G RAM and 64 G disk space on my host. | 21:08 |
| SamYaple | jemcevoy: you could do it that way i suppose. you may very well be losing a transaction or two depending on how active it was if you go that path | 21:08 |
| SamYaple | and if node1 was done _before_ the cluster shutoff, you could lose _alot_ of data | 21:08 |
| *** rwallner has quit IRC | 21:08 | |
| jemcevoy | What is the best path for our production cluster? | 21:08 |
| bmace | vhosakot: kk, you should be good to go with either 4 or 6 on kube2.. if you actually plan to try to bring up guests or whatever else might increase memory usage in the ostk control services more than 4gb might be good | 21:09 |
| jemcevoy | I have a perfect opportunity to test with two dev cluster down | 21:09 |
| *** rwallner_ has quit IRC | 21:10 | |
| vhosakot | bmace: is kube2 the compute node? yes, I'm planning to boot at least one nova instance and do some basic sanity checks (like ping gateway/DHCP/DNS server/google from inside the nova instance). do you think 6 G RAM on kube2 is good for this? | 21:11 |
| bmace | vhosakot: i think the default labeling should have kube2 as control and 1 / 3 / 4 as computes? | 21:12 |
| *** matrohon has quit IRC | 21:14 | |
| jemcevoy | SamYaple What is the best path for our production cluster? I have a perfect opportunity to test with two dev cluster down. | 21:15 |
| jemcevoy | I do not know how to restart the mariadb cluster... | 21:16 |
| *** fguillot has quit IRC | 21:16 | |
| jemcevoy | I have always had to hack it back to life | 21:16 |
| vhosakot | bmace: I see the labels here.. cool setup_dev_env.sh | 21:17 |
| vhosakot | bmace: https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_dev_env.sh#L48-L52 | 21:17 |
| bmace | vhosakot: ah, yup, if you don't want to do compute on kube2 i expect you can yank the label | 21:18 |
| bmace | vhosakot: or just give it more ram.. you have a bunch | 21:18 |
| sbezverk | sean-k-mooney: have you worked with host aggregates? | 21:22 |
| *** gfhellma has quit IRC | 21:22 | |
| vhosakot | bmace: cool.. yes, I'll assign 6G RAM to kube2, np.. so, per your latest change in the docs https://review.openstack.org/#/c/429295/26/doc/source/development-environment.rst, I'll run build_dev_image.sh and run_dev_image.sh, and will keep you posted. | 21:22 |
| sbezverk | or anybody on this channelm any experience with host aggregats? | 21:23 |
| bmace | vhosakot: cool, good luck! i think people have had hit or miss success with weave for networking in their k8s env btw.. i think canal / calico have been used consistently with success. | 21:24 |
| bmace | if you are using weave and it works though, all the better | 21:24 |
| vhosakot | bmace: cool, I'll deploy weave (what comes OTB) and let you know, and then try calico/canal/flannel. I see the labels are called here in the Dockerfile.. cool... https://github.com/openstack/kolla-kubernetes/blob/master/tools/Dockerfile#L74 | 21:25 |
| *** gfhellma has joined #openstack-kolla | 21:25 | |
| SamYaple | jemcevoy: if the clusters are already down, then i can't tell you the best path. but if you know things like "they were all up and active before the cluster stop, and node1 was what haproxy was loadbalancing to", then starting from node1 is what you want to do | 21:26 |
| jemcevoy | Sure... The power failed when the only updates would be the status heartbearts from the services... | 21:28 |
| jemcevoy | How do I feed the container the flag to tell it to form a new cluster? | 21:29 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 21:29 |
| vhosakot | bmace: is 2G RAM good on the other nodes kube{1,3,4} ? | 21:30 |
| bmace | vhosakot: depends on what kind of vms you want to bring on them.. as long as it is 1 or 2 tiny, probably. | 21:30 |
| vhosakot | bmace: yeah, I'll just spin a cirros tiny for my test, thanks for the info. | 21:31 |
| bmace | vhosakot: most of my work was just to make sure all the env / pods were coming up cleanly. i never did much with it once it was all up so not sure how big things need to be for a really "usable" environment, and usable really depends on what you want to do with it :) | 21:31 |
| vhosakot | bmace: true, 100% agreed :) can I update the docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html to add 4G or 6G RAM for kube2 so the user knows about it? | 21:32 |
| bmace | vhosakot: i don't own the docs or really any of the code. feel completely free to make any improvements you think will be helpful :) | 21:33 |
| vhosakot | bmace: cool, will do! | 21:33 |
| *** tonanhngo has joined #openstack-kolla | 21:34 | |
| *** Jeffrey4l_ has quit IRC | 21:34 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:37 | |
| *** jrobinson has joined #openstack-kolla | 21:38 | |
| *** salv-orlando has joined #openstack-kolla | 21:38 | |
| *** tonanhngo has quit IRC | 21:39 | |
| *** rwellum_ has quit IRC | 21:40 | |
| Slower | sdake: inc0: thanks guys :) | 21:43 |
| *** MasterOfBugs has quit IRC | 21:47 | |
| *** pramodrj07 has quit IRC | 21:47 | |
| *** MasterOfBugs has joined #openstack-kolla | 21:47 | |
| *** pramodrj07 has joined #openstack-kolla | 21:47 | |
| *** rhallisey has quit IRC | 21:48 | |
| *** athomas has quit IRC | 21:49 | |
| *** dave-mcc_ has joined #openstack-kolla | 21:53 | |
| *** dave-mccowan has quit IRC | 21:54 | |
| *** jtriley_ has joined #openstack-kolla | 21:56 | |
| *** jtriley has quit IRC | 21:59 | |
| *** crushil has quit IRC | 22:00 | |
| *** jtriley_ has quit IRC | 22:01 | |
| *** lucasxu has quit IRC | 22:02 | |
| *** lucasxu has joined #openstack-kolla | 22:03 | |
| *** dave-mcc_ has quit IRC | 22:03 | |
| *** rhallisey has joined #openstack-kolla | 22:05 | |
| openstackgerrit | Balaji G Pattewar proposed openstack/kolla-kubernetes master: Helm charts for heat/heat-cfn microservices https://review.openstack.org/414496 | 22:06 |
| vhosakot | bmace: have you seen this error when doing "vagrant up"? - http://paste.openstack.org/show/601834/ | 22:13 |
| vhosakot | bmace: looks like the command "kubeadm init" is getting a "404 not found" error since https://storage.googleapis.com/kubernetes-release/release/v.txt returns "NoSuchKeyThe specified key does not exist." in the browser. | 22:14 |
| vhosakot | v1k0d3n: ^^^ | 22:15 |
| bmace | vhosakot: no, i don't recall seeing anything like that :(, maybe a halcyon-vagrant-kubernetes issue that was introduced since my last update, or some sort of change in how google exposes what packages are available.. idk | 22:15 |
| vhosakot | bmace: right, https://storage.googleapis.com/kubernetes-release/release/v.txt and https://storage.googleapis.com/kubernetes-release/release does not exist, but https://storage.googleapis.com/kubernetes-release exists. | 22:16 |
| vhosakot | v1k0d3n: portdirect: bmace: I'm running into this - https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/63 | 22:17 |
| v1k0d3n | interesting. | 22:18 |
| v1k0d3n | hey vhosakot :) | 22:18 |
| vhosakot | v1k0d3n: hey, sorry, got pulled into other things, and just got back to kolla-k8s. | 22:18 |
| vhosakot | v1k0d3n: :) | 22:18 |
| v1k0d3n | no worries. haven't really used halcyon in a while. | 22:19 |
| v1k0d3n | so this is a new cluster? | 22:19 |
| vhosakot | v1k0d3n: cool, yep, brand new halcyon-vagrant-kubernetes repo. I'm following stock docs in https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html, and see error in the task that initializes kubernetes master - http://paste.openstack.org/show/601834/ | 22:20 |
| vhosakot | v1k0d3n: is the "--use-kubernetes-version v" part in "kubeadm init --token ac7da3.b2cofcda6ab01976 --use-kubernetes-version v --api-advertise-addresses 172.16.35.11" finding "v.txt" in https://storage.googleapis.com/kubernetes-release/release/v.txt that does not exist? | 22:21 |
| *** tonanhngo has joined #openstack-kolla | 22:21 | |
| vhosakot | v1k0d3n: don't know why it is v instead of v1.5.1. I'll dig more. | 22:23 |
| *** tonanhngo has quit IRC | 22:25 | |
| *** eaguilar has quit IRC | 22:26 | |
| vhosakot | v1k0d3n: I see "kube_version: v" in my halcyon-kubernetes/kube-deploy/group_vars/all.yml. | 22:27 |
| v1k0d3n | vhosakot: yeah, so you would need to put a version there... | 22:27 |
| v1k0d3n | i.e. v1.5.2 or v1.5.2 | 22:27 |
| v1k0d3n | somehow yours is missing the actual version number, which is needed. | 22:27 |
| v1k0d3n | somehow just showing v i guess? | 22:28 |
| vhosakot | v1k0d3n: sercanacar saw it too looks like - https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/63 | 22:28 |
| v1k0d3n | but in default in the halcyon it is v1.5.1 | 22:28 |
| v1k0d3n | https://github.com/att-comdev/halcyon-kubernetes/blob/master/kube-deploy/group_vars/all.yml#L27 | 22:28 |
| vhosakot | v1k0d3n: yep, master has it right. cool | 22:29 |
| v1k0d3n | does that resolve it? | 22:29 |
| vhosakot | v1k0d3n: I'll try now, doing "vagrant destroy", but still thinking why it is "v" in the first place :) | 22:30 |
| *** financeMann has quit IRC | 22:30 | |
| vhosakot | brb 5 mins | 22:30 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP Adding virtualbmc to the gate https://review.openstack.org/441550 | 22:30 |
| v1k0d3n | not sure | 22:31 |
| v1k0d3n | you said master has the correct version, identified, right? | 22:31 |
| vhosakot | v1k0d3n: yes, both halcyon-kubernetes master and the e8d4319 submodule that the halcyon-vagrant-kubernetes master pulls have "kube_version: v1.5.1", I verified. | 22:35 |
| portdirect | vhosakot: could you confirm the versions of vagrant and ansible you are using? | 22:36 |
| *** lucasxu has quit IRC | 22:36 | |
| portdirect | vhosakot: also platform would be great :) | 22:36 |
| vhosakot | portdirect: sure :) so, I'm using vagrant 1.8.1, Ansible 2.2.1.0, CentOS 7.3, with libvirt. | 22:37 |
| *** Margin has joined #openstack-kolla | 22:38 | |
| *** Margin has quit IRC | 22:38 | |
| v1k0d3n | so vhosakot you still have the same issue with master is what you're saying? | 22:38 |
| *** Margin has joined #openstack-kolla | 22:38 | |
| portdirect | vhosakot: any chance you could paste the ./config.rb and ./halcyon-kubernetes/kube-deploy/group_vars/all.yml? | 22:39 |
| vhosakot | v1k0d3n: with halcyon-vagrant-kubernetes master yes and in e8d4319 the detached branch that halcyon-vagrant-kubernetes thru submodule. | 22:40 |
| *** lucasxu has joined #openstack-kolla | 22:41 | |
| vhosakot | portdirect: http://paste.openstack.org/show/601838/ | 22:42 |
| vhosakot | portdirect: as we can see in ^^^^, I see "kube_version: v" in halcyon-kubernetes/kube-deploy/group_vars/all.yml in the detached branch e8d4319 (that halcyon-vagrant-kubernetes pulls) | 22:42 |
| portdirect | vhosakot: theres your issue :/ | 22:43 |
| vhosakot | portdirect: well, I didn't change it :) it is what is pulled | 22:44 |
| vhosakot | should the submodule be updated? | 22:44 |
| portdirect | did you set these from the dev guide? https://github.com/openstack/kolla-kubernetes/blob/master/doc/source/development-environment.rst#set-environment-variable-versions-for-easy-copy-and-pasting | 22:44 |
| sdake | vhosakot did you set the environment variable in the docs? | 22:44 |
| portdirect | ^^ | 22:44 |
| sdake | jinx :) | 22:44 |
| sdake | i'm pretty sure we need ot remove those env vars, that is the 3rd or 4th person that has been hit by that problem | 22:45 |
| sdake | i'm off for the week - enjoy :) | 22:45 |
| vhosakot | sdake: that was it.. I do not have the env set.... sorry guys... ROFL.... I exited from the shell, and the env var was gone... thanks for all the help | 22:46 |
| vhosakot | v1k0d3n: portdirect ^^^ my env var issue, thanks for your help. | 22:46 |
| vhosakot | looks like sercanacar also did the same in https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/63 .. lol, I might submit a PR to check this... | 22:47 |
| vhosakot | v1k0d3n: portdirect like a check in https://github.com/att-comdev/halcyon-vagrant-kubernetes/blob/master/setup-halcyon.sh#L30 to check if the en var is not set.. | 22:48 |
| inc0 | right...so I got libvirt to work | 22:48 |
| v1k0d3n | ah, ok...so that's why i wasn't getting the "v" issue vhosakot and portdirect | 22:48 |
| inc0 | and I'm sad today | 22:48 |
| inc0 | we either can run qemu as nova user or downgrade libvirt to 1.3 | 22:49 |
| vhosakot | v1k0d3n: portdirect, cool.. all good now :) I'll submit PR later. | 22:50 |
| portdirect | vhosakot: if you could make a PR for that it would be awesome :) | 22:50 |
| v1k0d3n | PR for what? | 22:50 |
| portdirect | v1k0d3n: adding a check into the kolla setup script | 22:50 |
| v1k0d3n | ah, ok. | 22:51 |
| sdake | qemu should ideally run as qemu user inc0 | 22:51 |
| inc0 | sdake: "ideally" | 22:51 |
| sdake | pretty sure that is how centos operates- but not certain it still does | 22:51 |
| vhosakot | portdirect: yes, will, do! I might also file a bug againt upstream k8s since the "kubeadm init" command does not validate what is --use-kubernetes-version. | 22:51 |
| inc0 | issue is nova creates files with umask 722 | 22:51 |
| inc0 | so qemu user has no access | 22:51 |
| inc0 | and libvirt used to chown files | 22:51 |
| sdake | groan | 22:52 |
| inc0 | but it's not doing this in 2.5 as it seems | 22:52 |
| inc0 | and 2.5 rolled in ubuntu cloud archive | 22:52 |
| sdake | centos is libvirt 2.5 iirc | 22:52 |
| inc0 | it is not | 22:52 |
| inc0 | it's 2.0 | 22:52 |
| sdake | ahh ok :) | 22:52 |
| vhosakot | portdirect: the fix would be in https://github.com/att-comdev/halcyon-vagrant-kubernetes/blob/master/setup-halcyon.sh#L30, right? | 22:52 |
| inc0 | so when centos bumps this version, same issue | 22:52 |
| sdake | right | 22:52 |
| inc0 | anyway, I running qemu with nova user works | 22:52 |
| sdake | that seems ok but suboptimal | 22:53 |
| sdake | there has to be a way to override the nova umask | 22:53 |
| sdake | or should be atleast | 22:53 |
| inc0 | alternative would to teach nova to create dirs with 772 instad of 722 | 22:53 |
| inc0 | but nova doesn't | 22:53 |
| sdake | hmm well your on it - i'm certain you will figure it out :) | 22:54 |
| inc0 | then we could just add libvirt-qemu user to nova group, but not the case | 22:54 |
| inc0 | already figured it out | 22:54 |
| sdake | i mean how to solve it properly :) | 22:54 |
| inc0 | gonna push patchset in a second | 22:54 |
| sdake | cool | 22:54 |
| *** Pavo has joined #openstack-kolla | 22:55 | |
| *** Margin has quit IRC | 22:56 | |
| portdirect | vhosakot: thats just the help text | 22:58 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Make qemu use nova user https://review.openstack.org/442850 | 22:58 |
| portdirect | you'd want to check for an empty var somewhere around L56 | 22:58 |
| vhosakot | portdirect: cool, thanks for the info.. yeah, I'll add it in that script to check if the env var KUBERNETES_VERSION is first set before running setup-halcyon.sh for kolla. cool | 22:59 |
| vhosakot | portdirect: my SSH window timed out and auto-logged out, and when I ssh'ed back in, forgot to reset the KUBERNETES_VERSION env var, might add to me .bashrc to it alwasy exists. | 23:01 |
| *** TxGirlGeek has quit IRC | 23:03 | |
| portdirect | Personally I wonder if we should just remove that var from the docs - and revert to specifying it in the command? | 23:04 |
| *** erlon has quit IRC | 23:05 | |
| *** tonanhngo has joined #openstack-kolla | 23:07 | |
| *** Pavo has quit IRC | 23:08 | |
| vhosakot | portdirect: the same env var is used to install the kubectl client as well --> https://docs.openstack.org/developer/kolla-kubernetes/development-environment.html#installing-clients-on-centos-or-ubuntu | 23:09 |
| portdirect | vhosakot: yeah - though we have the choice: get the doc maintainer to specify a value twice in a document or use the var. The var has caused multiple people problems/confusion and now your looking at adding things to your .bashrc to ensure smooth running | 23:11 |
| *** tonanhngo has quit IRC | 23:12 | |
| portdirect | that's a totally acceptable approach (the .bashrc) but I think we have perhaps optimized to far here? | 23:12 |
| *** lucasxu has quit IRC | 23:12 | |
| vhosakot | portdirect: yes, using the var multiple times is fine, yeah, addind in .bashrc is good... I think missing env var must be validated by halcyon-vagrant-kubernetes (for which I'll submit a PR soon) and invalid value for -use-kubernetes-version must be validated by "kubeadm init" (for which I'll open a bug againt kubeadmin upstream). | 23:15 |
| vhosakot | brb 20 mins... rebooting my laptop. | 23:18 |
| *** vhosakot has quit IRC | 23:19 | |
| *** goldyfruit1 has quit IRC | 23:21 | |
| *** lamt has quit IRC | 23:23 | |
| *** manheim_ has joined #openstack-kolla | 23:24 | |
| *** saneax-_-|AFK is now known as saneax | 23:29 | |
| jascott1 | woah 20 mins.. helluva boot cycle | 23:33 |
| *** Pavo has joined #openstack-kolla | 23:33 | |
| *** eaguilar has joined #openstack-kolla | 23:41 | |
| *** gfhellma has quit IRC | 23:47 | |
| *** sps_ has joined #openstack-kolla | 23:47 | |
| *** manheim_ has quit IRC | 23:48 | |
| *** adrian_otto has quit IRC | 23:52 | |
| inc0 | sdake: around? | 23:56 |
| jemcevoy | inc0: The root file systems are filling up there is a giant 64GB json log from the heka container... | 23:57 |
| inc0 | lol interesting...you didnt happen to have debug enabled? | 23:58 |
| inc0 | or something is spamming error messages | 23:58 |
| inc0 | never seen it this big tho | 23:58 |
| *** yingjun has joined #openstack-kolla | 23:59 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!