| sdake | tonyb in #openstack-requirements will help once the requirements gate is unwedged | 00:00 |
|---|---|---|
| sdake | i dont know when that will be | 00:00 |
| sbezverk | that sucks | 00:01 |
| sbezverk | now gerrit is unresponsive | 00:01 |
| sbezverk | what a day | 00:02 |
| *** Pavo has quit IRC | 00:03 | |
| *** lamt has quit IRC | 00:03 | |
| *** Pavo has joined #openstack-kolla | 00:04 | |
| sdake | sbezverk kfox1111 | 00:05 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing broken kolla-kubernetes gate https://review.openstack.org/433332 | 00:05 |
| sdake | http://logs.openstack.org/95/429295/20/check/gate-kolla-kubernetes-python35/eb53d52/console.html#_2017-02-13_22_34_46_823972 | 00:05 |
| sdake | after talking with the requirements team, they seemed to indicate the j2 code is broken somewhere | 00:05 |
| sdake | at that line # | 00:05 |
| sbezverk | sdake: we do not change that code | 00:06 |
| sbezverk | have no idea how it could have happened | 00:06 |
| sdake | jinja2 changed | 00:06 |
| sdake | the code parses jinja2 obivously | 00:06 |
| sdake | if its a bug in jinja2, we can get a top pin in | 00:06 |
| *** g3ek has joined #openstack-kolla | 00:06 | |
| sdake | if its a bug in our jinja2 syntax, we need to fix that | 00:06 |
| sbezverk | sdake: it looks like kfox1111 template validating code | 00:08 |
| sdake | yes | 00:08 |
| sbezverk | will wait for him to come back and check | 00:08 |
| sdake | my guess is you are usign // as a comment | 00:08 |
| sdake | somewhere in the jinja2 | 00:08 |
| sdake | and jinja2 may ahve deprecated that as a comment | 00:08 |
| sbezverk | sdake: I checked the code I do not see // used for comments | 00:11 |
| *** sdake_ has joined #openstack-kolla | 00:11 | |
| *** sdake has quit IRC | 00:12 | |
| *** Margin has joined #openstack-kolla | 00:13 | |
| *** Pavo has quit IRC | 00:14 | |
| *** dasTor has quit IRC | 00:17 | |
| *** dasTor has joined #openstack-kolla | 00:17 | |
| *** Margin has quit IRC | 00:18 | |
| *** adrian_otto has joined #openstack-kolla | 00:24 | |
| *** srwilkers_ has joined #openstack-kolla | 00:27 | |
| *** srwilkers has quit IRC | 00:27 | |
| *** srwilkers_ is now known as srwilkers | 00:27 | |
| *** Margin has joined #openstack-kolla | 00:28 | |
| *** MarginHu has joined #openstack-kolla | 00:29 | |
| *** srwilkers has quit IRC | 00:30 | |
| *** srwilkers has joined #openstack-kolla | 00:31 | |
| *** Margin has quit IRC | 00:32 | |
| MarginHu | J | 00:33 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Test of removing ceph-mon service https://review.openstack.org/433335 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Bootstrap osd removal https://review.openstack.org/433352 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove ceph-mon-pod.yml.j2 https://review.openstack.org/433353 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: ceph-osd-pod https://review.openstack.org/433354 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: common deployment https://review.openstack.org/433355 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: configmap https://review.openstack.org/433356 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: es https://review.openstack.org/433357 | 00:34 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: horizon service https://review.openstack.org/433358 | 00:34 |
| *** dave-mccowan has joined #openstack-kolla | 00:34 | |
| sdake_ | sup MarginHu | 00:34 |
| sdake_ | sbezverk why on earth do we still have the services dir in the tree | 00:34 |
| MarginHu | hello,sdake.sorry, I'm studying IRC | 00:35 |
| sbezverk | sdake_: because we treasure that part of our life | 00:37 |
| sdake_ | you can treasure it with git history | 00:37 |
| sbezverk | ;) | 00:37 |
| sdake_ | once code base has moved on dead code should be removed | 00:38 |
| sbezverk | we will remove it once all jinja2 services are moved to helm | 00:38 |
| sdake_ | i think we can remove a whole lot of it now | 00:39 |
| sdake_ | most everything has been helm'ized | 00:39 |
| sbezverk | sdake_: not yet, we still use it from time to time for reference | 00:39 |
| sdake_ | it is confusing tonew contribs | 00:39 |
| sdake_ | yes - git show xyz | 00:39 |
| sdake_ | there is your reference ;) | 00:39 |
| sdake_ | git checkout 0.4.0 | 00:39 |
| sdake_ | etc | 00:39 |
| sbezverk | sdake_: not sure it is that urgent, we have broken gate | 00:40 |
| sdake_ | this will fix the broken gate | 00:40 |
| sbezverk | nope | 00:40 |
| sdake_ | 1 dollar vs 100 | 00:40 |
| sbezverk | template check does not check that code, only temapltes | 00:40 |
| sdake_ | where are the templates? | 00:41 |
| sbezverk | helm/ | 00:41 |
| sdake_ | somethign is checking j2 files | 00:41 |
| sbezverk | kfox1111 validates helm temapltes | 00:41 |
| sdake_ | the only j2 files in the tree are in service dir | 00:41 |
| sdake_ | helm templatess are in gotpl | 00:41 |
| *** zhuzeyu has quit IRC | 00:41 | |
| sdake_ | maining jinja2 as a dep is untenable | 00:42 |
| sdake_ | because of jinja2 madness upstream and ansible's dependency on it | 00:42 |
| sbezverk | sdake_ ok got it, we just need to remove that check since it is checking | 00:42 |
| sbezverk | j2 | 00:42 |
| sdake_ | the rationale i just heard to keep the j2 files is completely invalid | 00:42 |
| sbezverk | sdake_: as a explained we still have some services which are not converted removing them now is -2 | 00:43 |
| *** adrian_otto has quit IRC | 00:43 | |
| sdake_ | git checkout 0.4.0 | 00:43 |
| sdake_ | problem solved | 00:43 |
| sbezverk | we need to disable the check | 00:43 |
| sdake_ | if you insist on keepign them in tree, then we can remove the things that have been helmized | 00:43 |
| sbezverk | sdake_: sorry but I will -2 | 00:43 |
| sdake_ | you insist on keepi gdead code in tree? | 00:44 |
| sbezverk | I think it makes more sense to disable unnecessary check | 00:44 |
| *** adrian_otto has joined #openstack-kolla | 00:44 | |
| sdake_ | answer my question pls | 00:44 |
| sbezverk | sdake_: it is not dead code, it is not yet transfered code | 00:44 |
| sdake_ | most of it has been transferred | 00:44 |
| sdake_ | true or flase? | 00:44 |
| sbezverk | I do not want to jump between branches | 00:44 |
| sbezverk | probably 2/3 | 00:44 |
| sdake_ | ok then we can remove that | 00:45 |
| sbezverk | but to solve this issue you need to remove all and it is -2 | 00:45 |
| sdake_ | no i dont need to remove all | 00:45 |
| kfox1111 | back | 00:45 |
| sdake_ | this is orthognal to the problem | 00:45 |
| sdake_ | the problem is the jinja2 formatting is incorrect | 00:45 |
| sdake_ | or jinja2 upstream parses incorrectly | 00:46 |
| sdake_ | however there is no good reason to keep dead code in tree | 00:46 |
| sbezverk | kfox1111: how can we disable jinja2 temaplte validation? | 00:46 |
| sdake_ | its not like we kept 5 revisions of the docker files in tree | 00:46 |
| sdake_ | that is what git history is for | 00:46 |
| sdake_ | that is *WHY* we use an rcs | 00:46 |
| sbezverk | remember you and David worked on it? | 00:46 |
| kfox1111 | I think deployment is going to have the same problem the validation will have. | 00:46 |
| kfox1111 | so once we have ceph and the neutron-set-bridge-ip stuff converted, we might be able to? | 00:47 |
| sdake_ | gotpl | 00:47 |
| kfox1111 | oh. and the configmaps I guess are generated too. :/ | 00:47 |
| sdake_ | if i can hear a valid reason to keep dead code in tree i'd cnsider keeping it | 00:47 |
| sdake_ | if the answer is some of it isn't dead, then that wfm we cna keep that in tree | 00:47 |
| kfox1111 | I'm ok removing the templates that have been converted to helm. | 00:47 |
| sdake_ | cool | 00:48 |
| kfox1111 | that won't eliminate the issue though. | 00:48 |
| sdake_ | it will | 00:48 |
| sdake_ | or it may | 00:48 |
| sbezverk | no it will not | 00:48 |
| sdake_ | i am finding out now | 00:48 |
| kfox1111 | it wont. | 00:48 |
| sbezverk | all j2 temapltes use the same format | 00:48 |
| sdake_ | here is my rationale; j2 is used in services dir | 00:48 |
| sbezverk | it is all or nothing | 00:48 |
| sdake_ | only certain tyeps of j2 are offensive to jinja2 | 00:48 |
| kfox1111 | kubectl template create foo and kubectl res create foo generate the exact same templates. on purpose. | 00:48 |
| sbezverk | we still use it | 00:48 |
| sbezverk | for other things | 00:48 |
| sdake_ | explain on purpose | 00:49 |
| sbezverk | kfox1111: I think we need to find a way to use older jinja2 version | 00:49 |
| sdake_ | sbezverk that is a nonstarter | 00:49 |
| kfox1111 | I'd stick a kubectl res create secret ceph in the gate and see what it spits out. | 00:49 |
| kfox1111 | maybe its a simple thing to reformat. | 00:49 |
| sdake_ | let me ask this question and i want a straight answer please | 00:50 |
| kfox1111 | lets just figurure out the real problem before trying to hack around it. | 00:50 |
| sdake_ | what uses the /services dir | 00:50 |
| sdake_ | i know what the real problem is | 00:50 |
| sdake_ | or am in the midst of debugging it | 00:50 |
| kfox1111 | ceph and a tiny bit of neutron, and a bit of libvirt? | 00:50 |
| sdake_ | i take offense at the dead code in the tree | 00:50 |
| sdake_ | dead code is unmaintained | 00:51 |
| kfox1111 | I agree we should get rid of the older templates. git history is enough. | 00:51 |
| sdake_ | if its not dead then fine | 00:51 |
| kfox1111 | its not all dead though. | 00:51 |
| sdake_ | i hear ya | 00:51 |
| sbezverk | sdake_: then you will have to yorself develop lost services | 00:51 |
| sdake_ | what services are lost - we are only removing stuff already implemented in helm | 00:51 |
| Jeffrey4l_ | sdake_, cloud u ack this one https://review.openstack.org/433128 | 00:52 |
| sbezverk | kfox1111: here is an idea, we do not change j2 files right | 00:53 |
| sbezverk | no there is no reason to validate them. let's try to for now disable test_template.py | 00:53 |
| sbezverk | and see | 00:53 |
| *** duonghq has joined #openstack-kolla | 00:53 | |
| kfox1111 | sbezverk: I think it will fail on the secret the same way. | 00:54 |
| kfox1111 | you can try though. | 00:54 |
| *** adrian_otto has quit IRC | 00:54 | |
| kfox1111 | but the res create secret ceph should fail in the same way as template create secret ceph | 00:54 |
| duonghq | morning kfox1111 sbezverk | 00:54 |
| kfox1111 | I think. | 00:54 |
| sbezverk | kfox1111: I am not sure where it shold be one | 00:54 |
| sbezverk | done | 00:54 |
| sbezverk | duonghq: morning | 00:54 |
| sdake_ | sbezverk there is no reason to hoard dead code | 00:55 |
| kfox1111 | sbezverk: try deleting the file outright. | 00:55 |
| kfox1111 | shuld work I think. | 00:55 |
| sbezverk | so we kill ./kolla_kubernetes/tests/test_templates.py | 00:55 |
| sbezverk | and pyc? | 00:56 |
| kfox1111 | yeah | 00:56 |
| kfox1111 | that would remove the check automatically. it just scans that dir for tests. | 00:56 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing broken kolla-kubernetes gate https://review.openstack.org/433332 | 00:58 |
| sbezverk | kfox1111: I think it will fail since it will not find command | 00:58 |
| sbezverk | it must be called somewhere | 00:58 |
| sbezverk | but will see | 00:58 |
| sdake_ | wrong answer | 00:59 |
| *** dave-mcc_ has joined #openstack-kolla | 00:59 | |
| kfox1111 | sbezverk: its called out of tox, but thats not the root of the tests. its imported as a module, then automatically introspected for tests. | 00:59 |
| sdake_ | i still haven't heard why we need to keep the dead code | 00:59 |
| *** huzhengchuan has joined #openstack-kolla | 01:00 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: ceph-osd-pod https://review.openstack.org/433354 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: common deployment https://review.openstack.org/433355 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Bootstrap osd removal https://review.openstack.org/433352 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove ceph-mon-pod.yml.j2 https://review.openstack.org/433353 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: horizon service https://review.openstack.org/433358 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: configmap https://review.openstack.org/433356 | 01:01 |
| *** dave-mccowan has quit IRC | 01:01 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: es https://review.openstack.org/433357 | 01:01 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Test of removing ceph-mon service https://review.openstack.org/433335 | 01:01 |
| sdake_ | reminded of that pink floyd song hello is there anyone out there | 01:01 |
| sdake_ | here is the root cause | 01:03 |
| sbezverk | kfox1111: deleting the file helped | 01:03 |
| sdake_ | one of the files is using \\ in their templates | 01:03 |
| sdake_ | and jinja2 doesn't like that | 01:04 |
| kfox1111 | sdake_: glance, horizon cinder, keystone, mariadb, memcached, neutron, nova, rabbit, and skydns can go I think. | 01:04 |
| sdake_ | or the formatting of the dead unmaintained code is incorrect | 01:04 |
| sdake_ | kfox1111 cool | 01:04 |
| *** cuongnv has joined #openstack-kolla | 01:04 | |
| kfox1111 | sbezverk: yeah, but it will still fail. | 01:04 |
| kfox1111 | \\ is used in ceph-bootstrap-osd.yml.j2 | 01:04 |
| sdake_ | i dont know which file it is kfox1111 | 01:05 |
| kfox1111 | sdake_: so, jinja2's interpreting \\'s now? | 01:05 |
| sdake_ | that is what that stack o patches is | 01:05 |
| sdake_ | to find the offending file | 01:05 |
| Jeffrey4l_ | sdake_, cloud u check and ack https://review.openstack.org/433128 which blocks the gate. | 01:05 |
| sdake_ | kfox1111 not sure - jinja 2.9.5 is busted | 01:05 |
| kfox1111 | sdake_: grep -r '\\\\' services/ | 01:06 |
| kfox1111 | looks like ceph and keepalived use it. | 01:06 |
| kfox1111 | we could: | 01:06 |
| sdake_ | Jeffrey4l_ when are you branching kolla-ansible? | 01:06 |
| kfox1111 | 1. finish the migration of those to helm. | 01:06 |
| kfox1111 | 2. escape them so that jinja likes them again. | 01:06 |
| Jeffrey4l_ | sdake_, when the gate is green. | 01:07 |
| kfox1111 | I'm for #1 if we can do it quickly. | 01:07 |
| Jeffrey4l_ | is there any patch need to merge? | 01:07 |
| sdake_ | i think quickly is hard here - 2 days is not good to keep the gate blocked | 01:07 |
| kfox1111 | there are reviews open already for the ceph bits. though stale. | 01:07 |
| sdake_ | right | 01:07 |
| kfox1111 | we could dust them off and finish them. | 01:07 |
| sdake_ | well lets confirm that is what the offending files are | 01:08 |
| Jeffrey4l_ | fyi, requirements ocata branch is created. now ocata use jinja==2.8.x and master use jinja=2.9.5 | 01:08 |
| kfox1111 | we cant without "fixing them" or fixing jinja | 01:09 |
| *** zhubingbing_ has joined #openstack-kolla | 01:09 | |
| sbezverk | kfox1111: still need keepalived | 01:10 |
| kfox1111 | how about this. lets pin to older jinja. commit ceph migration. unpin jinja. | 01:10 |
| kfox1111 | sbezverk: agreed. we need to port it. but its not launched in the gate currently. | 01:10 |
| kfox1111 | the keepalive chart is so useful I want to use it in a non openstack system too. | 01:11 |
| *** Pavo has joined #openstack-kolla | 01:14 | |
| sdake_ | kfox1111 that wont work - requirements gate job fails | 01:15 |
| openstackgerrit | Merged openstack/kolla-ansible master: Disable revoke_by_id in keystone https://review.openstack.org/433058 | 01:16 |
| kfox1111 | sdake_: oh. there is a test for it? fun... | 01:16 |
| sdake_ | yup | 01:16 |
| kfox1111 | it it our gate test? | 01:16 |
| sdake_ | and its voting - mandatory | 01:16 |
| sdake_ | no its infra mandatory | 01:16 |
| sdake_ | we have no input over it | 01:16 |
| sdake_ | and the requirements sycn job would be afu as a result of that change | 01:17 |
| sdake_ | master needs to test master | 01:17 |
| sdake_ | i dont know how many times i have to say it :) | 01:17 |
| sdake_ | for it to sink in - we can't test old dependencies with new software | 01:17 |
| kfox1111 | we can't commit fixes though properly... I'm sure we're not the first project to hit this issue. | 01:18 |
| sbezverk | kfox1111: I heard other project also reported some issue this morning | 01:20 |
| *** qiliang27 has joined #openstack-kolla | 01:20 | |
| sdake_ | sbezverk that was because equalto was not implemented in jinja2 | 01:21 |
| sdake_ | equalto is now implemented in jinja2- no other projects rae having problems but us | 01:21 |
| sbezverk | kfox1111: can we first line in gate test remove new jinja and install 2.8 | 01:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove ceph-bootstrap-initial-mon https://review.openstack.org/433374 | 01:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: remove ceph-bootstrap-osd https://review.openstack.org/433375 | 01:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove ceph mon pod https://review.openstack.org/433376 | 01:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: remove ceph-osd-pod https://review.openstack.org/433377 | 01:21 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Remove keepalive daemonset https://review.openstack.org/433378 | 01:22 |
| sdake_ | this patchset will identify which file is offensive to jinja 2.9.5 | 01:22 |
| *** caowei has joined #openstack-kolla | 01:22 | |
| sbezverk | sdake_ you cannot remove these | 01:22 |
| sdake_ | sbezverk i get it | 01:23 |
| sdake_ | ITS A TEST | 01:23 |
| kfox1111 | sdake_: we have a solution. | 01:23 |
| kfox1111 | common requirements caused the breakage, so we fix common requirements in infra first. | 01:23 |
| kfox1111 | fix our code, | 01:23 |
| kfox1111 | then we allow them to update it again. | 01:23 |
| sdake_ | yes i talked to tonyb about it already | 01:24 |
| sdake_ | he is busy unwedging the requirements gate | 01:24 |
| sdake_ | because apparently the requirements gate is broken too | 01:24 |
| sdake_ | so he is unable to proces any such requeest | 01:24 |
| sbezverk | kfox1111: next time I see req update ps I will NEVER ack it | 01:24 |
| sdake_ | sbezverk the pip cache was reloaded | 01:24 |
| sdake_ | acking or not acking the requirements patch would not have any imapct in this case | 01:25 |
| kfox1111 | yeah. its infra's patch that broke things. | 01:25 |
| sdake_ | who cares who broke it | 01:25 |
| sdake_ | lets stop assigning blame | 01:25 |
| sdake_ | and fix it so the gate is unblocked | 01:25 |
| sdake_ | Jeffrey4l_ this needs an ack plz: https://review.openstack.org/#/c/433118/ | 01:26 |
| sdake_ | before the branch | 01:26 |
| kfox1111 | I'm not blaming anyone. | 01:26 |
| kfox1111 | just stating that they said, the proper fix was to rever their own patch. | 01:26 |
| kfox1111 | the common requirements checks are intended to check on this exact case. | 01:26 |
| kfox1111 | now that it founda breakage, its an issue to fix in the common requirements. | 01:27 |
| *** MasterOfBugs has joined #openstack-kolla | 01:27 | |
| sdake_ | or altenately fix in the jinja 2 templates and the global requirements at the same time | 01:27 |
| sbezverk | sdake_: in this case we need to push a fix to linux foundation to replace sed syntax | 01:28 |
| Jeffrey4l_ | sdake_, done. | 01:29 |
| sbezverk | oh and one nore to kubernetes because their template requires double \\ | 01:29 |
| sbezverk | whole world needs to run around what openstack-requirment team comes up with | 01:30 |
| *** rhallisey has quit IRC | 01:31 | |
| Jeffrey4l_ | sdake_, should we merge this before branching? https://review.openstack.org/432297 | 01:32 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fixing ironic configs for kolla-kubernetes https://review.openstack.org/433118 | 01:32 |
| *** saneax is now known as saneax-_-|AFK | 01:35 | |
| sdake_ | Jeffrey4l_ is cross repo gating functional? | 01:35 |
| sdake_ | Jeffrey4l_ if so, i'd say yes, if not, i'd say does it work :) | 01:35 |
| Jeffrey4l_ | for kolla and kolla-ansible, yes. | 01:36 |
| sdake_ | life was so much easier with one repository :) | 01:36 |
| Jeffrey4l_ | hrm, wait | 01:36 |
| sdake_ | kolla-kubernetes has no cross repo gating at present | 01:36 |
| Jeffrey4l_ | are u mean in kolla, to use kolla-ansible certain ref? this can not be happen. because there is no ocata rc tag or branch in kolla-ansible project. | 01:37 |
| kfox1111 | what version of jinja2 fixes the issue please? | 01:37 |
| sdake_ | 2.8.0-2.8.z | 01:37 |
| sdake_ | sbezverk huh re sed syntax? | 01:37 |
| *** unicell has quit IRC | 01:38 | |
| sdake_ | sbezverk openstack-requirements team is entirely cross-project dirven | 01:38 |
| sdake_ | sbezverk each project negotiates for dependency versions | 01:38 |
| sdake_ | and theoretically we have a requirements liason to handle that process | 01:39 |
| sdake_ | just as jeffrey is the release liason with the release team | 01:39 |
| sdake_ | kfox1111 join #openstack-requirements | 01:39 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fix for the gate to work around gate issue. https://review.openstack.org/433383 | 01:41 |
| *** tovin07 has joined #openstack-kolla | 01:41 | |
| *** ssurana has quit IRC | 01:42 | |
| sdake_ | kfox1111 ya that approaach looks feasible - although i'd like ot verify there isn't a formatting error in the jinja2 templates | 01:42 |
| sdake_ | we have not proven that is true | 01:43 |
| kfox1111 | they worked before. | 01:44 |
| sbezverk | kfox1111 +2 worked last night | 01:44 |
| kfox1111 | so somethings probably stricter now. | 01:44 |
| sdake_ | if you reaad the srollback in openstack-requirements | 01:44 |
| *** rhallisey has joined #openstack-kolla | 01:44 | |
| sdake_ | you would see what tony has to say about jinja2 ;) | 01:45 |
| kfox1111 | so, I need to head home. but thinking, if that ps works, lets merge it, quick, and then I'll work on getting the ceph reviewes finished up enough to merge. then we can unpin it again. | 01:45 |
| sdake_ | how does that fix the pin? | 01:45 |
| sdake_ | the test cases stil lrun against the j2 files that are in the codebase | 01:46 |
| kfox1111 | cause the solution to the jinja2 problem is to get out of jinja2. :/ | 01:47 |
| kfox1111 | if they are going to make breaking changes in a point release. :/ | 01:47 |
| kfox1111 | this buys us just enough time to do so. | 01:47 |
| kfox1111 | bbiab | 01:49 |
| *** srwilkers has quit IRC | 01:49 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: [TEST][MASTER][KOLLA] test build speed w/o cleaning yum cache https://review.openstack.org/432901 | 01:57 |
| *** jrist has quit IRC | 01:58 | |
| sdake_ | kfox1111 same problem every time: 2017-02-14 01:25:06.129590 | jinja2.exceptions.TemplateSyntaxError: unexpected char u'\\' at 19492 | 01:58 |
| sdake_ | even on removal of some of the files | 01:58 |
| sdake_ | kfox1111 jinja2 has been broken in 2.9.0-2.9.5 as 2.9 is a development branch | 01:59 |
| sdake_ | not a stable branch | 01:59 |
| openstackgerrit | Tony Breeds proposed openstack/kolla-kubernetes master: [DNM] Testing Jinja2 https://review.openstack.org/433393 | 02:02 |
| sbezverk | sdake_ \\ is used not directly in jinja but for SED command | 02:03 |
| sdake_ | sbezverk i removed every instance of \\ and the problem was not resolved | 02:03 |
| sbezverk | there are two \\ becuase kubernbetes requires escape \ hence we put \\ | 02:04 |
| openstackgerrit | Tony Breeds proposed openstack/kolla-kubernetes master: [DNM] Testing Jinja2 https://review.openstack.org/433393 | 02:04 |
| *** zhubingbing__ has joined #openstack-kolla | 02:08 | |
| *** sdake_ is now known as sdake | 02:09 | |
| *** huzhengchuan has quit IRC | 02:09 | |
| *** zhubingbing_ has quit IRC | 02:11 | |
| *** jrist has joined #openstack-kolla | 02:11 | |
| *** huzhengchuan has joined #openstack-kolla | 02:14 | |
| sdake | sbezverk its parsing the secrets file | 02:15 |
| *** sdake has quit IRC | 02:17 | |
| *** sdake has joined #openstack-kolla | 02:19 | |
| *** Pavo has quit IRC | 02:23 | |
| *** tfukushima has joined #openstack-kolla | 02:23 | |
| *** tfukushima has quit IRC | 02:23 | |
| kfox1111 | back. | 02:24 |
| kfox1111 | yeah, yaml requires a \ to be escaped. | 02:24 |
| kfox1111 | so \ needs to be \\ | 02:24 |
| kfox1111 | maybe jinja2 now needs another level of \\. | 02:25 |
| duonghq | sdake, around? it seem that the heat microservice was not fully implemented? | 02:26 |
| duonghq | or the create-db job does not belong to the microservice's scope? | 02:26 |
| *** dtturner has quit IRC | 02:27 | |
| sdake | duonghq yes it is incompletey implemented | 02:27 |
| kfox1111 | duonghq:yeah. heat needs the creation jobs still, and a gate test. | 02:27 |
| sdake | kfox1111 note tonyb tested with jinja 2.8.1 and its not jinja | 02:27 |
| duonghq | sdake, kfox1111 roger, but it's marked as done in blueprint, got some confuse | 02:28 |
| sdake | duonghq not sure why it was marked done - maybe an error | 02:28 |
| sdake | mistakes happen :) | 02:28 |
| sdake | kfox1111 ok so we know its not jinja2 now | 02:28 |
| sdake | what else could it be | 02:28 |
| kfox1111 | how did he test it? | 02:29 |
| sdake | I notice its processing something with secret ceph ceph | 02:29 |
| kfox1111 | I'm getting much further once I pinned back to 2.8.0. | 02:29 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Use ocata tag in config.py https://review.openstack.org/432297 | 02:29 |
| duonghq | sdake, sorry, I mis-recalled, seem that jascott1 is working on that | 02:29 |
| duonghq | jascott1, ping | 02:29 |
| *** japestinho has joined #openstack-kolla | 02:30 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Fix for the gate to work around gate issue. https://review.openstack.org/433383 | 02:32 |
| sdake | kfox1111 in a way that isn't a gross hack :) | 02:33 |
| kfox1111 | just a little gross. :) | 02:34 |
| sdake | ;-) | 02:34 |
| sdake | there is scrollback in openstack-infra showing his test example | 02:35 |
| sdake | and i validated that it returns same results | 02:35 |
| sdake | its some other change or dep | 02:35 |
| kfox1111 | did kube make a release or something? | 02:39 |
| sdake | how would that affect the unit tests? | 02:40 |
| kfox1111 | maybe the different k8s client pulls ome different deps? | 02:41 |
| sdake | did the devenv merge? | 02:41 |
| sdake | i thought it had -1's on it | 02:41 |
| kfox1111 | no. not yet. | 02:41 |
| sdake | if you mean kubectl, that is a self-contained binary | 02:41 |
| sdake | same story with all go programs | 02:42 |
| kfox1111 | we pull latest though I think. | 02:42 |
| sdake | go has no external dependencies whatsoever | 02:42 |
| sdake | yes - however - the unit tests are failing? | 02:42 |
| sdake | is kubectl used in the unit tests? | 02:42 |
| kfox1111 | it really seems like its failing on jinja2 though.... | 02:42 |
| kfox1111 | lets render one and see what it looks like. | 02:43 |
| sdake | here is where it dies in a fire: http://logs.openstack.org/83/433383/1/check/gate-kolla-kubernetes-python27-ubuntu-xenial/72ddf6c/console.html#_2017-02-14_01_43_35_806987 | 02:43 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: [WIP]Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 02:45 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Helm charts for heat-create-db and heat-manage-db https://review.openstack.org/414496 | 02:48 |
| *** sudipto has joined #openstack-kolla | 02:48 | |
| *** sudipto_ has joined #openstack-kolla | 02:48 | |
| *** MasterOfBugs has quit IRC | 02:49 | |
| kfox1111 | sdake: taht one looks pretty weird. | 02:49 |
| kfox1111 | its trying to process a tgz? | 02:49 |
| sdake | kfox1111 check this out: | 02:49 |
| sdake | tonyb says this works in his local enviornment: https://review.openstack.org/#/c/432721/ | 02:50 |
| sdake | rather that patch merged into kolla-ansible | 02:50 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: [WIP]Optimize reconfiguration for haproxy https://review.openstack.org/424156 | 02:50 |
| sdake | he says a patch after that likely broke kolla-kubernetes | 02:50 |
| kfox1111 | thats really weird. | 02:51 |
| sdake | its not that wierd | 02:52 |
| sdake | genconfig may be failing in some way and producing odd results | 02:52 |
| kfox1111 | yeah. could be introducing other \\'s... | 02:52 |
| *** tvignaud has quit IRC | 02:54 | |
| kfox1111 | I'm not sure how reverting that ps could have that effect.... | 03:01 |
| kfox1111 | oh. the check date on that review is 6:57.... | 03:01 |
| kfox1111 | lets try and revert it though... I'm skeptical | 03:02 |
| sdake | revert which? | 03:03 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Test a critical kolla-ansible defect resolution https://review.openstack.org/432721 | 03:04 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patchset. https://review.openstack.org/433410 | 03:04 |
| sbezverk | which PS broke? | 03:05 |
| sdake | nah its not my patch | 03:05 |
| sdake | its one after it | 03:06 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Helm charts for heat-create-db and heat-manage-db https://review.openstack.org/414496 | 03:06 |
| kfox1111 | sdake: e121ecb380d7f7e962fbd1afe5c402ef6b277171 ? | 03:07 |
| sdake | ya its that ironic patch that broke things | 03:07 |
| *** tvignaud has joined #openstack-kolla | 03:07 | |
| sdake | -transport_url = rabbit://{% for host in groups['rabbitmq'] %}{{ rabbitmq_user }}:{{ rabbitmq_password }}@{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}:{{ rabbitmq_port }}{% if not loop.last %},{% endif %}{% endfor %} | 03:08 |
| sdake | +transport_url = rabbit://{% for host in groups['rabbitmq'] %}{{ rabbitmq_user }}:{{ rabbitmq_password }}@{% if orchestration_engine == 'KUBERNETES' %}rabbitmq{% else %}{{ hostvars[host]['ansible_' + hostvars[host]['api_interface']]['ipv4']['address'] }}{% endif %}:{{ rabbitmq_port }}{% if not loop.last %},{% endif %}{% endfor %} | 03:08 |
| sdake | or maybe not | 03:08 |
| sdake | i dunno tonyb is bisecting it now | 03:08 |
| *** Jeffrey4l_ has quit IRC | 03:08 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:09 | |
| *** schwicht has quit IRC | 03:11 | |
| *** Jeffrey4l_ has quit IRC | 03:13 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:14 | |
| kfox1111 | I'm going to be very sad if this is the second kolla-ansible gate breakages in 1 week of unpinning. | 03:14 |
| sdake | if i knew where kolla-ansible was installed in tox -e py27 i'd bisect it | 03:14 |
| sdake | the reason that happens is we don't have cross repo gating | 03:14 |
| kfox1111 | that too. | 03:14 |
| *** Jeffrey4l_ has quit IRC | 03:16 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:16 | |
| kfox1111 | so, 4.0.0.0b2 use to work. | 03:18 |
| kfox1111 | maybe we pin back to that and see if it works or not. | 03:18 |
| sdake | here is the problem: https://review.openstack.org/#/c/424504/ | 03:18 |
| kfox1111 | if not, its probably something outside kolla-ansible | 03:18 |
| kfox1111 | hmm... | 03:18 |
| sdake | tonyb ran a bisect | 03:18 |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Add the missing condition for config.yml https://review.openstack.org/433419 | 03:22 |
| sbezverk | sdake: how ironic ps in kolla could break kolla kubernets? | 03:23 |
| sbezverk | also the string you are refering was copied form glance config | 03:24 |
| sbezverk | so I am positive it is good | 03:24 |
| sdake | Jeffrey4l_ wtb your help :) | 03:24 |
| sdake | sbezverk it isn't the ironic ps | 03:25 |
| sdake | it is this patchset: https://review.openstack.org/#/c/424504/ | 03:25 |
| sdake | i'm so beat i can barely se estraight | 03:25 |
| kfox1111 | yeah. its in here somewhere: https://review.openstack.org/#/c/424504/4/ansible/roles/neutron/defaults/main.yml | 03:25 |
| Jeffrey4l_ | sdake, sup. | 03:25 |
| sdake | something about that patch breaks kolla-genconfig Jeffrey4l_ | 03:26 |
| sdake | it results in the following: | 03:26 |
| sdake | moemnt | 03:26 |
| Jeffrey4l_ | sdake, any error message? | 03:26 |
| sbezverk | kfox1111: has it been proven? | 03:27 |
| sbezverk | still does not make sense, the break was way before genconfig got called | 03:27 |
| sdake | http://paste.fedoraproject.org/557849/70428731/ -> http://paste.fedoraproject.org/557849/70428731 | 03:28 |
| sdake | Jeffrey4l_ rtaher it breaks kolla-kubernetes in some wierd odd way | 03:28 |
| sdake | yo ucan see right at the bottom something about hostvars | 03:28 |
| sdake | with that patch reverted, unit tetts pass | 03:28 |
| Jeffrey4l_ | yep. checking. | 03:28 |
| *** jrobinson is now known as jrobinson-afk | 03:28 | |
| sdake | sbezverk ya i'm not clear on what hte problem is | 03:28 |
| sdake | however its a damn good thing the unit tests were there to catch it :) | 03:29 |
| sdake | shows we need m0ar of those | 03:29 |
| sdake | as well as cross repo gating | 03:29 |
| *** rhallisey has quit IRC | 03:30 | |
| kfox1111 | sdake: m0ar tests! :) | 03:30 |
| sdake | we have great functional testing better then most projects | 03:30 |
| sdake | our unit testing is sort of weak | 03:30 |
| sdake | kolla-ansible cores are working on gating tests for a bunch of stuff | 03:31 |
| sdake | it looks really good | 03:31 |
| sbezverk | sdake: I have not see any Jenkins with +1 on gate jobs | 03:31 |
| sdake | sbezverk yes -the bisect was done outside of the gate | 03:32 |
| sdake | in 15 minutes | 03:32 |
| sdake | tox -e py27 of kolla-kubernetes | 03:33 |
| sdake | bisecting the kolla-ansible that gets moved to the kolla dir | 03:33 |
| sdake | not sure why there is a mv there in the tests | 03:33 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 03:33 |
| sdake | anyway it doesn't matter however perhaps it will help jeffrey4l sort out the problem | 03:33 |
| Jeffrey4l_ | cloud u reproduce this locally? | 03:34 |
| kfox1111 | sdake: a mv? | 03:34 |
| sdake | kfox1111 yup there is a mv kolla-ansible to kolla | 03:34 |
| sdake | https://github.com/openstack/kolla-kubernetes/blob/master/tools/test.sh#L2 | 03:34 |
| kfox1111 | there was a while ago, but shoudl have been removed.... | 03:34 |
| sdake | Jeffrey4l_ yes | 03:34 |
| kfox1111 | ah. still there I guess. | 03:35 |
| kfox1111 | that was a quick fix right after the repo rename. | 03:35 |
| kfox1111 | the rest of the gate got a further refactoring but I guess that file didnt. | 03:35 |
| sdake | no big deal | 03:35 |
| Jeffrey4l_ | cloud u check which template it is rendering? rendered_template = j2env.get_template(name).render(dict_) | 03:35 |
| sdake | i was pointing it out so jeffrey4l could find kolla-ansible :) | 03:36 |
| sdake | Jeffrey4l_ its not during the genconfig stage | 03:36 |
| sdake | genconfig seems to work | 03:36 |
| kfox1111 | ah. still. probably should fix that issue. | 03:36 |
| sdake | the probelm is genconfig renderst he wrong thing | 03:36 |
| sdake | kfox1111 agreed | 03:36 |
| Jeffrey4l_ | sdake, we should try to find which template has issue. try to add some print in rendered_template = JinjaUtils.render_jinja(d, template) | 03:37 |
| sbezverk | kfox1111: still fails | 03:38 |
| Jeffrey4l_ | i have on idea how to set up this locally. | 03:38 |
| sdake | Jeffrey4l_ i'll walk you through it | 03:38 |
| sdake | Jeffrey4l_ because honestly I have no idea what your tlaking about | 03:38 |
| sdake | git clone http://github.com/openstack/kolla-kubernetes | 03:38 |
| Jeffrey4l_ | done | 03:38 |
| sdake | cd kolla-kubernetes | 03:39 |
| sdake | tox -e py27 | 03:39 |
| *** mliima has quit IRC | 03:40 | |
| *** bpattewar has joined #openstack-kolla | 03:40 | |
| sdake | kfox1111 where precisely does the pushd and popd with teh gitmove i linked above put the kolla dir? | 03:40 |
| Jeffrey4l_ | running. | 03:40 |
| sdake | I dont see it locally, however, tonyb had it | 03:40 |
| kfox1111 | sdake: parent dir. kind of sketchy. :/ | 03:41 |
| sdake | which parent dir, ~? | 03:41 |
| kfox1111 | wherever kolla-kubernetes's checked out of, ../kolla | 03:41 |
| sdake | got it | 03:41 |
| kfox1111 | so say, ~/git/kolla-kubernetes | 03:41 |
| sdake | we need to fix that ;-) | 03:41 |
| kfox1111 | ~/git kolla | 03:41 |
| kfox1111 | ~/git/kolla | 03:41 |
| sdake | so jeffrey after tests run open a new terminal | 03:41 |
| kfox1111 | A | 03:42 |
| sdake | Jeffrey4l_ then after that you can modify kolla-ansible in ../kolla dir wher eyou checked out kolla-kubernetes | 03:42 |
| sdake | Jeffrey4l_ then replay tox -e py27 in kolla-kubernetes | 03:42 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 03:42 |
| sdake | and the changes will be active | 03:42 |
| kfox1111 | sdake: I don't think that ps is fully responsible. | 03:43 |
| kfox1111 | http://logs.openstack.org/22/433422/1/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-ceph-nv/df51730/console.html | 03:43 |
| kfox1111 | reverting it didn't fix the issue. | 03:43 |
| Jeffrey4l_ | sdake, add print(name) in https://github.com/openstack/kolla-kubernetes/blob/master/kolla_kubernetes/utils.py#L163 and have a try. | 03:44 |
| Jeffrey4l_ | i am trying set up the unittest env. | 03:44 |
| sdake | Jeffrey4l_ dinner is served - i'll be back in 10 :) | 03:44 |
| Jeffrey4l_ | but no idea when it will be successful. i saw some issue. | 03:44 |
| Jeffrey4l_ | ok | 03:45 |
| *** mliima has joined #openstack-kolla | 03:45 | |
| sbezverk | kfox1111: it just did not make sense to me how neutron kolla-ansible patch break something in kolla-kube | 03:45 |
| sbezverk | even BEFORE kolla-ansible got cloned | 03:45 |
| kfox1111 | lets pin back to 4.0.0.0b2, which use to work. | 03:46 |
| sdake | we can't do that | 03:46 |
| kfox1111 | sbezverk: got an example log of that? | 03:46 |
| sbezverk | kfox1111: which log? | 03:46 |
| kfox1111 | sbezverk: of a failure beforoe genconfig? | 03:46 |
| sbezverk | sure | 03:46 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 03:47 |
| kfox1111 | I think all the tests happen after genconfig. | 03:47 |
| Jeffrey4l_ | why kolla-k8s tox -epy27 depends on /home/jeffrey/openstack/kolla/ansible/roles/neutron/defaults/main.yml this localtion? rather then kolla-ansible? | 03:48 |
| sbezverk | kfox1111: scratch that I was thinking about jinja sorry | 03:48 |
| kfox1111 | oh. actualy ,thats not strictly true... | 03:48 |
| kfox1111 | it uses the same kind of engine gencnofig does, but not entirely. | 03:48 |
| *** tyrola has quit IRC | 03:49 | |
| sdake | Jeffrey4l_ we went over that earlier - it was a temporary thing that never got f ixed after the repo split | 03:49 |
| sdake | Jeffrey4l_ have to keep in mind kolla-kubernetes is only at 0.5.0 :) | 03:50 |
| *** tyrola has joined #openstack-kolla | 03:50 | |
| kfox1111 | hmm... | 03:51 |
| Jeffrey4l_ | sdake, how can i change this path? | 03:51 |
| kfox1111 | ok. pulling back to 4.0.0.0b2 does seem to fix it. | 03:52 |
| sdake | Jeffrey4l_ yo ucant that I know of but you can work in that path and modify stuff | 03:52 |
| kfox1111 | so not an external tooling issue. | 03:52 |
| kfox1111 | Jeffrey4l_: see the top of tools/test.sh | 03:52 |
| Jeffrey4l_ | ok | 03:52 |
| sdake | then if you run tox -e py27 it doesn't delete what is there | 03:52 |
| sdake | you can verify that patch is the problem by reverting it in ../kolla and then running tox -e py27 in kolla-kubernetes | 03:53 |
| *** david-lyle_ has joined #openstack-kolla | 03:56 | |
| *** david-lyle has quit IRC | 03:58 | |
| duonghq | sdake, bmace have we had document for developer env? | 04:00 |
| sdake | duonghq i hear there is a review up for the docs - let me see if i can find it | 04:00 |
| sdake | hear/heard | 04:00 |
| *** mdnadeem has joined #openstack-kolla | 04:00 | |
| kfox1111 | sdake: http://logs.openstack.org/22/433422/1/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/5b5a77a/console.html | 04:02 |
| kfox1111 | that really looks like that ps isn't the issue. or not the only issue. | 04:02 |
| sbezverk | kfox1111: I think I found where the issue is in json which is passed to jinja | 04:03 |
| sbezverk | check this line | 04:03 |
| sbezverk | {{ neutron_linuxbridge_agent_tag }}', u'nova_consoleauth_replicas': u'1', u'neutron_dhcp_agent_image': u"{{ dock | 04:03 |
| sdake | kfox1111 what does that gate job refer to | 04:04 |
| sbezverk | nova_console and neutron dhcp are missing {{ brackets | 04:04 |
| kfox1111 | sdake: thats from: https://review.openstack.org/#/c/433422/1 | 04:04 |
| sbezverk | kfox1111: what do you think? | 04:05 |
| sdake | sbezverk how did you ge tthat output | 04:05 |
| kfox1111 | sdake: not sure how to interprete that paste? looks like python? | 04:05 |
| kfox1111 | sry. sbezverk I mean. | 04:05 |
| *** huzhengchuan has quit IRC | 04:05 | |
| sdake | duonghq no idea where the docs rae althoguh bmace did mention i think that they were published | 04:06 |
| sbezverk | kfox1111: I printed data which gets passed to jinja for rendering | 04:06 |
| sbezverk | and then run json analyzer | 04:06 |
| duonghq | sdake, I also think so | 04:06 |
| sbezverk | it showed that data is malformed | 04:06 |
| kfox1111 | Ok. so, yeah, python dict... | 04:06 |
| kfox1111 | hmm.. its like its not doing a substitute but just leaving the block in place? | 04:07 |
| sbezverk | kfox1111: do you want the the whole thing? | 04:07 |
| kfox1111 | sbezverk: yes please. | 04:07 |
| *** dave-mcc_ has quit IRC | 04:07 | |
| kfox1111 | looks like pulling back to 4.0.0.0b2 is working. | 04:08 |
| kfox1111 | so likely kolla-ansible. | 04:08 |
| sbezverk | kfox1111: http://paste.openstack.org/show/598733/ | 04:08 |
| sbezverk | it is at position 19492 | 04:08 |
| kfox1111 | oh. thats the strings that ps introduced I think. | 04:09 |
| sbezverk | kfox1111: I am not so sure, we gather data which then gets passed to jinja rendering | 04:10 |
| sdake | kfox1111 why rebase this patch? https://review.openstack.org/#/c/432721/ | 04:11 |
| sbezverk | kfox1111: but we do use config info generated by kolla-ansible | 04:11 |
| kfox1111 | sbezverk: https://review.openstack.org/#/c/424504/4/ansible/roles/neutron/defaults/main.yml | 04:11 |
| kfox1111 | it pulls in all that stuff into the env I think. | 04:11 |
| kfox1111 | sdake: I was testing reverting it, and rebased what I thoguth was my patch, but ended up rebasing the parent. | 04:12 |
| sdake | its ok | 04:12 |
| sdake | just curious :) | 04:12 |
| kfox1111 | still not sure its that patch alone. | 04:12 |
| kfox1111 | reverting it didn't fix the issue. | 04:12 |
| sdake | it works locally | 04:13 |
| sdake | so its part of the problem | 04:13 |
| kfox1111 | are there any other, similar patchess? | 04:13 |
| sdake | yup a whole slew of em | 04:13 |
| kfox1111 | oh. :/ | 04:14 |
| sbezverk | kfox1111: that patch does not touch nova_consoleauth_replicas | 04:14 |
| sdake | i can tell you kolla-ansible was good at the revision whre i fixed them | 04:14 |
| japestinho | Hi guys, I still experienced this error https://bugs.launchpad.net/kolla/+bug/1634712, I'm using stable/newton | 04:14 |
| openstack | Launchpad bug 1634712 in kolla "neutron bootstrap container crashes when optional plugins are enabled" [Undecided,Fix released] | 04:14 |
| sbezverk | but it seems where the problem starts | 04:14 |
| sdake | this patch is good: 6cb4507267e0a368a69c0ba1d02a6e16301ddf43 | 04:15 |
| sbezverk | kfox11111: nova_consoleauth_replicas is not even defined in ansible | 04:15 |
| kfox1111 | sbezverk: log? | 04:15 |
| sbezverk | kfox1111: I was just trying to find patterns from the broken fragment where they are defined | 04:16 |
| sbezverk | first incorrect entry comes from kolla-kube etc/kolla-kubernetes/kolla-kubernetes.yml | 04:16 |
| *** tvignaud has quit IRC | 04:17 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 04:18 |
| kfox1111 | sdake: that one pulls it up to 6cb4507267e0a368a69c0ba1d02a6e16301ddf43. | 04:18 |
| kfox1111 | sdake: are many more patches expected to land before branching? | 04:19 |
| *** huzhengchuan has joined #openstack-kolla | 04:19 | |
| sdake | we are trying to branch asap | 04:20 |
| sdake | hard to do that wen every gate in the project is cratered | 04:20 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Add VENV variable in bifrost base Dockerfile https://review.openstack.org/433434 | 04:20 |
| kfox1111 | k. just trying to figure out how worth it would it be for me to work on a cross gate check. | 04:20 |
| kfox1111 | but if 4.x branch in immenent, then it doesn't matter too much. | 04:20 |
| sdake | i'm not sure - too hard to make a judgement call on that, been up since 5am and its 10 with about 1 15 minute break all day | 04:21 |
| sdake | on that note, I need to hit the rack guys | 04:21 |
| kfox1111 | yeah. | 04:21 |
| sdake | i'd recommend the same | 04:21 |
| kfox1111 | k. nite. | 04:21 |
| *** sdake has quit IRC | 04:22 | |
| sbezverk | kfox1111: so we are back to 4.0.0.0b2? | 04:23 |
| kfox1111 | sbezverk: just tested that to get a base line. | 04:24 |
| kfox1111 | the whole gate works with that. so its not an external requirement changing | 04:24 |
| kfox1111 | the claim is 6cb4507267e0a368a69c0ba1d02a6e16301ddf43 works. | 04:25 |
| kfox1111 | I'm testing that now. | 04:25 |
| kfox1111 | the ps after that suposed to break things. | 04:25 |
| kfox1111 | reverting just that one ps didn't fix the gate though for me. | 04:26 |
| kfox1111 | so I'm going to walk forward and revert ps's as they break. | 04:26 |
| kfox1111 | so I can find them all. | 04:26 |
| sbezverk | kfox1111: I am very sceptical that it is that PS | 04:26 |
| sbezverk | because after it was merged I run my PS at least 100 times | 04:26 |
| sbezverk | the issue started only this morning | 04:27 |
| kfox1111 | yeah. | 04:27 |
| sbezverk | kfox1111: just checked one the gate jobs and it passed that template check | 04:28 |
| kfox1111 | I'm waiting for some of the non py jobs too. | 04:30 |
| kfox1111 | looks like this one's going to pass. | 04:30 |
| kfox1111 | so, 6cb4507267e0a368a69c0ba1d02a6e16301ddf43 does seem happy. | 04:30 |
| duonghq | kfox1111, what is "...manage-db-job" for? | 04:30 |
| kfox1111 | duonghq: it does a db schema update or initialization. | 04:30 |
| kfox1111 | comes from the corresponding openstack commands. | 04:31 |
| duonghq | thanks | 04:31 |
| sbezverk | I am off too, let me know if you need me to ack review, will do it first thing tomorrow morning | 04:32 |
| kfox1111 | k. | 04:32 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 04:32 |
| kfox1111 | testing the suspect now in isolation. | 04:32 |
| *** tvignaud has joined #openstack-kolla | 04:35 | |
| kfox1111 | yup... it fails the gate... | 04:36 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 04:38 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 04:42 |
| kfox1111 | test reverting... | 04:42 |
| *** jrobinson-afk is now known as jrobinson | 04:43 | |
| *** ssurana has joined #openstack-kolla | 04:46 | |
| *** sudipto_ has quit IRC | 04:50 | |
| *** sudipto has quit IRC | 04:50 | |
| kfox1111 | uh... | 04:54 |
| kfox1111 | http://logs.openstack.org/22/433422/7/check/gate-kolla-kubernetes-deploy-ubuntu-binary-2-iscsi-nv/4e6a206/console.html | 04:54 |
| kfox1111 | that should have been with that one ps reverted.... | 04:55 |
| kfox1111 | oh.. cause I didn't pipe it in. :/ | 04:57 |
| kfox1111 | ok... | 04:57 |
| kfox1111 | alright. lets push forward a bit. | 04:58 |
| *** yuanying has quit IRC | 04:58 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 04:58 |
| *** yuanying has joined #openstack-kolla | 04:58 | |
| *** unicell has joined #openstack-kolla | 05:13 | |
| *** unicell has quit IRC | 05:14 | |
| *** unicell has joined #openstack-kolla | 05:18 | |
| *** sudipto_ has joined #openstack-kolla | 05:26 | |
| *** sudipto has joined #openstack-kolla | 05:26 | |
| *** saneax-_-|AFK is now known as saneax | 05:26 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Fix Kolla gate https://review.openstack.org/433434 | 05:31 |
| *** skramaja_afk is now known as skramaja | 05:32 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: Test revert patch https://review.openstack.org/433422 | 05:33 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: Use ocata tag in config.py https://review.openstack.org/432297 | 05:34 |
| *** Jeffrey4l_ has quit IRC | 05:37 | |
| kfox1111 | arg... | 05:38 |
| kfox1111 | so I didn't test that patch before. :/ | 05:38 |
| kfox1111 | patch -p1 -R foo.patch != patch -p1 -R < foo.patch. :/ | 05:39 |
| *** Jeffrey4l_ has joined #openstack-kolla | 05:40 | |
| *** unicell1 has joined #openstack-kolla | 05:40 | |
| *** bpattewar has quit IRC | 05:41 | |
| *** unicell has quit IRC | 05:41 | |
| openstackgerrit | OpenStack Proposal Bot proposed openstack/kolla-ansible master: Updated from global requirements https://review.openstack.org/431962 | 05:52 |
| *** hyakuhei has quit IRC | 06:06 | |
| *** bpattewar has joined #openstack-kolla | 06:08 | |
| bpattewar | Hi | 06:10 |
| bpattewar | Duonghq hi | 06:10 |
| *** salv-orlando has joined #openstack-kolla | 06:12 | |
| duonghq | hi bpattewar | 06:15 |
| bpattewar | Duonghq I am looking to contribute where it needs. | 06:20 |
| bpattewar | duonghq I am new to kolla-kubernetes | 06:20 |
| bpattewar | duonghq sdake told me that you are working heat services. | 06:21 |
| duonghq | bpattewar, I'm working on microservice atm | 06:25 |
| duonghq | and move to service when the microservices are ready | 06:26 |
| *** sudipto_ has quit IRC | 06:26 | |
| *** sudipto has quit IRC | 06:26 | |
| bpattewar | duonghq atm ? | 06:27 |
| duonghq | bpattewar, at this moment | 06:27 |
| duonghq | currently | 06:27 |
| duonghq | can you make some microservices role? | 06:28 |
| *** satyar has joined #openstack-kolla | 06:29 | |
| bpattewar | duonghq I am new to kolla-kubernetes but I will be happy to learn and do it. So you are developing microservices for heat ? | 06:30 |
| duonghq | I'm working on create-db and manage-db job | 06:30 |
| duonghq | we have had some job already in the master | 06:30 |
| bpattewar | can you share your existing patches. I can take look at them and start working.. | 06:31 |
| *** tvignaud has quit IRC | 06:35 | |
| duonghq | bpattewar, we have some ps here: https://review.openstack.org/#/q/project:openstack/kolla-kubernetes+heat | 06:36 |
| duonghq | I'm taking over dupengfei work on create-db and manage-db | 06:36 |
| *** jrobinson has quit IRC | 06:38 | |
| bpattewar | duonghq shall I look for " Helm charts for heat/heat-cfn create user and endpoints." ? | 06:40 |
| bpattewar | duonghq / | 06:42 |
| bpattewar | or do you suggest any other microservice ? | 06:42 |
| duonghq | bpattewar, sure | 06:42 |
| duonghq | you can take this ps | 06:42 |
| bpattewar | duonhq I can take look at it. | 06:45 |
| *** tvignaud has joined #openstack-kolla | 06:49 | |
| *** sudipto has joined #openstack-kolla | 06:56 | |
| *** sudipto_ has joined #openstack-kolla | 06:56 | |
| *** salv-orl_ has joined #openstack-kolla | 07:00 | |
| *** salv-orlando has quit IRC | 07:03 | |
| *** brad[] has quit IRC | 07:11 | |
| *** fragatin_ has joined #openstack-kolla | 07:13 | |
| *** fragatina has quit IRC | 07:17 | |
| *** fragatin_ has quit IRC | 07:18 | |
| Jeffrey4l_ | zhubingbing__, pbourke please review https://review.openstack.org/433434 which unblocks the gate. | 07:20 |
| *** magicboiz has quit IRC | 07:21 | |
| *** magicboiz has joined #openstack-kolla | 07:33 | |
| *** prameswar has joined #openstack-kolla | 07:36 | |
| *** fragatina has joined #openstack-kolla | 07:36 | |
| *** fragatina has quit IRC | 07:37 | |
| *** fragatina has joined #openstack-kolla | 07:37 | |
| *** brad[] has joined #openstack-kolla | 07:38 | |
| *** fragatina has quit IRC | 07:44 | |
| *** fragatina has joined #openstack-kolla | 07:44 | |
| *** ssurana has quit IRC | 07:49 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla master: [TEST][MASTER][KOLLA] test build speed w/o cleaning yum cache https://review.openstack.org/432901 | 07:50 |
| *** shardy has joined #openstack-kolla | 07:58 | |
| *** MarginHu has quit IRC | 07:59 | |
| openstackgerrit | caoyuan proposed openstack/kolla-ansible master: Optimize reconfiguration for mariadb https://review.openstack.org/433480 | 08:08 |
| *** pcaruana has joined #openstack-kolla | 08:11 | |
| *** shardy has quit IRC | 08:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: [TEST][NEWTON][KOLLA] Run periodic https://review.openstack.org/417033 | 08:25 |
| *** matrohon has joined #openstack-kolla | 08:27 | |
| *** mgoddard has joined #openstack-kolla | 08:44 | |
| *** egonzalez has joined #openstack-kolla | 08:45 | |
| *** strigazi_AFK is now known as strigazi | 08:46 | |
| *** fragatina has quit IRC | 08:47 | |
| *** athomas has joined #openstack-kolla | 08:51 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla-kubernetes master: Helm charts for heat-create-db and heat-manage-db https://review.openstack.org/414496 | 08:53 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/newton: This feature is supporting for workers config in swift servers. https://review.openstack.org/433503 | 08:57 |
| *** gfidente has joined #openstack-kolla | 08:59 | |
| *** skramaja has quit IRC | 09:02 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: Disable gpgcheck in grafana repo https://review.openstack.org/433507 | 09:04 |
| openstackgerrit | Merged openstack/kolla-ansible master: Move Kuryr to start after neutron has started https://review.openstack.org/433328 | 09:11 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla stable/newton: Move Kuryr to start after neutron has started https://review.openstack.org/433516 | 09:15 |
| satyar | Hi pbourke | 09:16 |
| *** Serlex has joined #openstack-kolla | 09:32 | |
| *** duonghq has quit IRC | 09:34 | |
| *** hrw has quit IRC | 09:38 | |
| *** hrw has joined #openstack-kolla | 09:38 | |
| *** jmccarthy has quit IRC | 09:46 | |
| *** mannamne has joined #openstack-kolla | 09:50 | |
| *** hfu has quit IRC | 09:50 | |
| egonzalez | does neutron-sfc-agent ever work? there is not entrypoint in docker image nor config.json | 09:51 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Make service_plugins and extensions more configurable https://review.openstack.org/426709 | 09:53 |
| *** tovin07 has quit IRC | 10:01 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: bump neutron version to 9.2.0 https://review.openstack.org/433554 | 10:02 |
| *** tyrola has quit IRC | 10:03 | |
| *** tyrola has joined #openstack-kolla | 10:04 | |
| *** cuongnv has quit IRC | 10:09 | |
| *** sambetts|afk is now known as sambetts | 10:12 | |
| pbourke | satyar: morning | 10:13 |
| *** caowei has quit IRC | 10:23 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Add missing dashboard policies https://review.openstack.org/433570 | 10:25 |
| *** mgoddard has quit IRC | 10:32 | |
| *** sdake has joined #openstack-kolla | 10:34 | |
| *** sdake_ has joined #openstack-kolla | 10:37 | |
| *** sdake has quit IRC | 10:39 | |
| *** sdake__ has joined #openstack-kolla | 10:39 | |
| *** sdake_ has quit IRC | 10:42 | |
| *** satyar has quit IRC | 10:44 | |
| sdake__ | mroing peeps | 10:47 |
| sdake__ | Jeffrey4l_ any progress on the regressoin probelm in kolla-ansible? | 10:47 |
| *** shardy has joined #openstack-kolla | 10:48 | |
| egonzalez | morning sdake__ | 10:50 |
| *** rafael_sisweb has joined #openstack-kolla | 10:50 | |
| *** sdake has joined #openstack-kolla | 10:51 | |
| *** sdake__ has quit IRC | 10:53 | |
| *** Serlex has quit IRC | 11:01 | |
| hrw | going extreme: debian/jessie-backports source build of all | 11:02 |
| *** matrohon has quit IRC | 11:10 | |
| *** jmccarthy has joined #openstack-kolla | 11:11 | |
| *** mgoddard has joined #openstack-kolla | 11:15 | |
| *** zhubingbing__ has quit IRC | 11:26 | |
| *** jrist has quit IRC | 11:27 | |
| *** rhallisey has joined #openstack-kolla | 11:38 | |
| *** jrist has joined #openstack-kolla | 11:40 | |
| *** magicboiz has quit IRC | 11:51 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Run simple_cell_setup after the new container is up during upgrade https://review.openstack.org/431508 | 11:51 |
| *** sudipto has quit IRC | 11:52 | |
| *** sudipto_ has quit IRC | 11:52 | |
| *** Serlex has joined #openstack-kolla | 11:52 | |
| *** magicboiz has joined #openstack-kolla | 11:55 | |
| *** haplo37 has quit IRC | 11:59 | |
| hrw | debian/jessie-backports source: 201 built | 12:00 |
| Jeffrey4l_ | sdake, no. still can not set up the test env ;( | 12:02 |
| sdake | ok well I wont have time to look at it until THURSDAY | 12:03 |
| sdake | have 2 hour meetup and I am way behind the 8 ball in prep | 12:03 |
| sdake | (meetup is Wednesday) | 12:03 |
| Jeffrey4l_ | ok.. | 12:03 |
| *** tyrola has quit IRC | 12:07 | |
| *** haplo37 has joined #openstack-kolla | 12:08 | |
| *** tyrola has joined #openstack-kolla | 12:08 | |
| *** pbourke has quit IRC | 12:09 | |
| *** pbourke has joined #openstack-kolla | 12:09 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: Bump service version number https://review.openstack.org/433554 | 12:10 |
| mliima | need act here | 12:22 |
| mliima | https://review.openstack.org/#/c/432405/ | 12:22 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix haproxy configure for external placement api listen https://review.openstack.org/433618 | 12:23 |
| *** zhubingbing_ has joined #openstack-kolla | 12:25 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add how configure backend to cinder HNAS NFS driver guide https://review.openstack.org/432405 | 12:29 |
| *** matrohon has joined #openstack-kolla | 12:30 | |
| *** rwallner has joined #openstack-kolla | 12:33 | |
| *** matrohon has quit IRC | 12:36 | |
| *** satyar has joined #openstack-kolla | 12:36 | |
| Jeffrey4l_ | egonzalez, mliima could u review https://review.openstack.org/#/c/433507/ the source gate red is unrelated. | 12:41 |
| *** sdake_ has joined #openstack-kolla | 12:41 | |
| egonzalez | Jeffrey4l_: re cells upgrade, the workaround from grenade works | 12:42 |
| *** sdake has quit IRC | 12:43 | |
| Jeffrey4l_ | egonzalez, but the gate is red ;(. | 12:44 |
| Jeffrey4l_ | check http://logs.openstack.org/08/431508/2/check/gate-kolla-ansible-dsvm-deploy-centos-source-centos-7-nv/5581953/console.html#_2017-02-14_12_17_56_533577 egonzalez | 12:46 |
| egonzalez | Jeffrey4l_: yeah, went to lunch after pushing that, didn't look at gates | 12:46 |
| sdake_ | gate is also red wrt kolla-kubernetes | 12:46 |
| sdake_ | totally cratered | 12:46 |
| sdake_ | and its a regression in kolla-ansible that is the cause | 12:47 |
| sdake_ | I know where it last works | 12:47 |
| mliima | done Jeffrey4l_ | 12:47 |
| Jeffrey4l_ | sdake_, we are talking upgrade cell patch. | 12:47 |
| Jeffrey4l_ | which gate is red now. | 12:47 |
| Jeffrey4l_ | the master code for kolla and kolla-ansible are green. | 12:47 |
| sdake_ | Jeffrey4l_ i know, I hijacked the conversatoin into the fact that we can't tag rc1 utntil all gates are green | 12:48 |
| sdake_ | kolla-ansible has no cross gating with kolla-kubernetes or hte gate would be red too | 12:48 |
| Jeffrey4l_ | i still confused the reconfigure neutron do not touch the "hostvars" variables. | 12:50 |
| sdake_ | the last commit that we know works properly is: 6cb4507267e0a368a69c0ba1d02a6e16301ddf43 | 12:51 |
| sdake_ | it may not be reconfigure neutron, tonyb ran that in his local env with a git bisect | 12:51 |
| sdake_ | the local env and gating env are vastly different wrt testing | 12:51 |
| sdake_ | once tonyb reverted that patch, we were good to go | 12:52 |
| sdake_ | (in local) | 12:52 |
| sdake_ | however when it was reverted in the kolla-kubernetes gate, the gate was red | 12:52 |
| sdake_ | it may be a combination of patches | 12:53 |
| *** mgiles has joined #openstack-kolla | 12:53 | |
| sdake_ | sup mgiles | 12:53 |
| sdake_ | as we want to tag kolla-kubernetes 0.5.0 wednesday, and I have a meetup wednesday and need to fit 50 hours of work into the next 30 hours, I'm not sure how I can help wrt unblocking the tag | 12:54 |
| mgiles | good morning sdake_ | 12:55 |
| sbezverk | good morning | 12:56 |
| *** huzhengchuan has quit IRC | 12:59 | |
| *** matrohon has joined #openstack-kolla | 12:59 | |
| *** salv-orlando has joined #openstack-kolla | 13:00 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: Bump service version number https://review.openstack.org/433554 | 13:00 |
| sdake_ | sup sbezverk | 13:01 |
| *** athomas has quit IRC | 13:01 | |
| sdake_ | sbezverk the gate is not yet fixed | 13:01 |
| sbezverk | sdake_ I know :( could not be better timing for this to happen.. | 13:02 |
| sdake_ | yup | 13:02 |
| sdake_ | i saw kfox1111 mentioned he would consider doing cross-repo gating for kolla-ansible | 13:02 |
| sdake_ | I think that makes alot of sense to prevent this in the future | 13:02 |
| sdake_ | even though genconfig is temporary I am not sure how temporary it is | 13:03 |
| *** salv-orl_ has quit IRC | 13:03 | |
| sbezverk | sdake_: it is either this or we stop using ansible and will have to implement config gen asap.. | 13:06 |
| *** dave-mccowan has joined #openstack-kolla | 13:06 | |
| *** mdnadeem has quit IRC | 13:06 | |
| *** athomas has joined #openstack-kolla | 13:06 | |
| sdake_ | sbezverk ok - well we can pause the cross repo gating idea until after PTG which is only 4 days away - and make a decision based upon how to generate configuration | 13:07 |
| sdake_ | if generating configuration is something that is easy to tackle, then we can ditch genconfig after ocata | 13:07 |
| sdake_ | and that will save us the cross-repo gating on kolla-ansible | 13:07 |
| sdake_ | in the meantime, we will have to suck up the pain | 13:08 |
| sdake_ | sbezverk fwiw kolla-kubernetes gate being red also blocks kolla-ansible from tagging | 13:08 |
| sdake_ | so this is a serious serious problem as all deliverables need to tag wednesday in prep for the PTG! | 13:08 |
| sdake_ | finally I dont have time to fix the gate or investigate why its fubared - have meetup where I need to fit something like 50 hours of work into the next 30 hours of my day | 13:09 |
| sbezverk | sdake_: if we suspect that neutron PS, let's cut it in small chunks and try one by one until we find ofending chunk | 13:09 |
| sdake_ | sbezverk it is unclear to me if that is the only patch that breaks it | 13:10 |
| sbezverk | sdake_: I do not know how else we can make progress on this nightmare | 13:10 |
| sdake_ | Jeffrey4l_ - WTB HELP :) | 13:10 |
| sdake_ | sbezverk I suspect it may be multiple packages | 13:10 |
| sdake_ | rather multiple patches | 13:10 |
| sbezverk | sdake_: maybe but we need to start doing something | 13:11 |
| sdake_ | when toby git bisected, he bisected in a local env | 13:11 |
| sbezverk | if there is no better plan, let's start this one | 13:11 |
| Jeffrey4l_ | sdake_, yep. | 13:11 |
| sdake_ | the gate env is different from the local env | 13:11 |
| Jeffrey4l_ | sdake_, i got this when running "tox -epy27" | 13:12 |
| Jeffrey4l_ | http://paste.openstack.org/show/598814/ | 13:12 |
| sbezverk | Jeffrey4l_: it matches to what we see at the gate | 13:15 |
| Jeffrey4l_ | sbezverk, how can i fix this? | 13:15 |
| Jeffrey4l_ | locally | 13:15 |
| sbezverk | Jeffrey4l_: it is million dollar question :) we can try to revert neutron PS and see the result | 13:16 |
| Jeffrey4l_ | sbezverk, i think you can try revert that patch in gate, right? | 13:17 |
| *** huzhengchuan has joined #openstack-kolla | 13:17 | |
| Jeffrey4l_ | in k8s's gate. | 13:17 |
| sdake_ | sbezverk Jeffrey4l_ it was reverted in teh gate and the gate failed - in a different way | 13:17 |
| sdake_ | so it is just not that patch | 13:17 |
| sbezverk | Jeffrey4l_: kfox1111 tried last night it did not help | 13:17 |
| sdake_ | wait a minute: https://review.openstack.org/#/c/433422/ | 13:18 |
| Jeffrey4l_ | sbezverk, so i can say neutron patch is not the root cause? | 13:18 |
| sdake_ | this has a green gate | 13:18 |
| sdake_ | Jeffrey4l_ w edont know for sure | 13:18 |
| sdake_ | Jeffrey4l_ i think the root cause INVOLVES that patch | 13:18 |
| sdake_ | check that review - i am not sure what the git commits re that kevin did, however, the gate is green | 13:19 |
| sdake_ | re/are | 13:19 |
| sbezverk | sdake_: he removed several ps | 13:19 |
| openstackgerrit | Merged openstack/kolla stable/newton: Disable gpgcheck in grafana repo https://review.openstack.org/433507 | 13:19 |
| sdake_ | ok - i think we need a gate bisect, rather then a local bisect | 13:20 |
| sdake_ | moment let me get that started | 13:20 |
| Jeffrey4l_ | sdake_, kervin's patch just reject neutron reconfigure patch, rather then several patches. | 13:22 |
| sdake_ | so it is the neutron patch? | 13:22 |
| Jeffrey4l_ | i guess yes. but can not explain, still. | 13:23 |
| Jeffrey4l_ | we are using a format in neutron reconfigure path | 13:23 |
| Jeffrey4l_ | + host_in_groups: >- | 13:23 |
| Jeffrey4l_ | + {{ | 13:23 |
| Jeffrey4l_ | + inventory_hostname in groups['compute'] | 13:23 |
| Jeffrey4l_ | + or (enable_manila | bool and inventory_hostname in groups['manila-share']) | 13:23 |
| Jeffrey4l_ | + or inventory_hostname in groups['neutron-dhcp-agent'] | 13:23 |
| Jeffrey4l_ | + or inventory_hostname in groups['neutron-l3-agent'] | 13:24 |
| Jeffrey4l_ | + or inventory_hostname in groups['neutron-metadata-agent'] | 13:24 |
| Jeffrey4l_ | + or inventory_hostname in groups['neutron-vpnaas-agent'] | 13:24 |
| Jeffrey4l_ | + }} | 13:24 |
| Jeffrey4l_ | may be related? | 13:24 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: DNM Bisecting in the gate https://review.openstack.org/433643 | 13:26 |
| *** sudipto_ has joined #openstack-kolla | 13:26 | |
| *** sudipto has joined #openstack-kolla | 13:26 | |
| Jeffrey4l_ | could u give me a link which have the issue log? sdake_ | 13:26 |
| sdake_ | Jeffrey4l_ let me bisect in gate - it will take roughly 3 steps or about 1 hour | 13:26 |
| *** Pavo has joined #openstack-kolla | 13:27 | |
| sdake_ | Jeffrey4l_ nobody has filed a bug | 13:27 |
| sdake_ | sbezverk can you file a bug pls | 13:27 |
| Jeffrey4l_ | any logs i means. | 13:27 |
| Jeffrey4l_ | logs.openstack.org? | 13:27 |
| sbezverk | Jeffrey4l_: one sec. will get you the log | 13:27 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Run simple_cell_setup after the new container is up during upgrade https://review.openstack.org/431508 | 13:27 |
| sdake_ | thanks sbezverk | 13:27 |
| sdake_ | sbezverk fwiw, that is how you do a proper gate bisect above ^^ | 13:28 |
| sdake_ | sbezverk with this approach evyeroen can see the good and bad commits | 13:28 |
| sdake_ | sbezverk i'm not sure if a local bisect is viable or not - i guess we will find out :) | 13:28 |
| sbezverk | Jeffrey4l_: http://logs.openstack.org/56/431156/26/check/gate-kolla-kubernetes-python27-ubuntu-xenial/13c0d41/console.html#_2017-02-13_14_41_56_420641 | 13:28 |
| sdake_ | if the gate bisect matches the local bisect, then i think a local bisect is viable (and much faster) | 13:29 |
| Jeffrey4l_ | this is different with that one i saw yesterday. | 13:29 |
| sdake_ | Jeffrey4l_ you will notice the tox -e py27 is different from the gate log | 13:29 |
| Jeffrey4l_ | i saw KeyERROR iirc. | 13:30 |
| sdake_ | yup looks like you noticed it :) | 13:30 |
| sdake_ | ya it was ah ostvars keyerror | 13:30 |
| sdake_ | which led me to believe we were using keyerror without an {%if ORCHESTRATION_ENGINE == KUBERNETES} block in j2 | 13:30 |
| sdake_ | hwoever your patch has no j2 file | 13:30 |
| Jeffrey4l_ | so keyerror is gone? and the only issue is: jinja2.exceptions.TemplateSyntaxError: unexpected char u'\\' at 19492 | 13:30 |
| sdake_ | Jeffrey4l_ keyerror still happens locally | 13:31 |
| Jeffrey4l_ | sdake_, i can not reproduce it. | 13:31 |
| Jeffrey4l_ | is the keyerror caused by neutron patch? | 13:31 |
| sdake_ | Jeffrey4l_ i can make a script which reproduces it so you can follow the work - would that help? | 13:31 |
| sdake_ | (a typescript) | 13:31 |
| *** eaguilar has quit IRC | 13:31 | |
| sdake_ | it is confirmed by tonyb the keyerror is caused by the neutron patch | 13:31 |
| Jeffrey4l_ | i want to know what the currrent error message? | 13:32 |
| Jeffrey4l_ | shouldn't we fix the jinja2.exceptions.TemplateSyntaxError: unexpected char u'\\' at 19492 right now? | 13:32 |
| sdake_ | yup - let me run through it locally moment | 13:32 |
| sdake_ | Jeffrey4l_ in some way that is coming because of a change in kolla-ansible | 13:32 |
| Jeffrey4l_ | at least now, i can not see "keyerror" in gate. | 13:32 |
| Jeffrey4l_ | OK. | 13:32 |
| sdake_ | Jeffrey4l_ moment - let me run locally | 13:32 |
| sbezverk | Jeffrey4l_: I will recheck one of the ps to get the latest error | 13:32 |
| Jeffrey4l_ | cool. | 13:33 |
| *** shardy has quit IRC | 13:33 | |
| sbezverk | Jeffrey4l_: here is the sdake's ps just finished: Jeffrey4l_ | 13:33 |
| *** eaguilar has joined #openstack-kolla | 13:34 | |
| sbezverk | http://logs.openstack.org/43/433643/1/check/gate-kolla-kubernetes-python27-ubuntu-xenial/9082208/console.html#_2017-02-14_13_29_32_753118 | 13:34 |
| *** shardy has joined #openstack-kolla | 13:34 | |
| Jeffrey4l_ | ok. so i can reproduce this now. let me debug it to check what's going on. | 13:35 |
| sbezverk | Jeffrey4l_: curly braces in neutron ps jinja thinks as its statements | 13:36 |
| *** eaguilar_ has joined #openstack-kolla | 13:37 | |
| Jeffrey4l_ | scripts in kolla-kubernetes can not parse " "host_in_groups": "{{\n inventory_hostname in groups['neutron-l3-agent']\n or (inventory_hostname in groups['compute'] and enable_neutron_dvr | bool)\n}}" | 13:39 |
| Jeffrey4l_ | " | 13:39 |
| Jeffrey4l_ | let me try to fix this in kolla-k8s. ;) | 13:39 |
| *** eaguilar has quit IRC | 13:40 | |
| sbezverk | Jeffrey4l_: we should be able to escape curly braces | 13:40 |
| Jeffrey4l_ | sbezverk, hrm, can not get what do u mean. this should not related to curly braces | 13:41 |
| *** eaguilar_ has quit IRC | 13:42 | |
| sbezverk | Jeffrey4l_: nvm | 13:43 |
| hrw | does someone uses that debug output when kolla ends work? http://paste.fedoraproject.org/557994/87079786/ I mean | 13:43 |
| *** huzhengchuan has quit IRC | 13:46 | |
| *** lamt has joined #openstack-kolla | 13:47 | |
| *** sudipto has quit IRC | 13:52 | |
| *** sudipto_ has quit IRC | 13:52 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-kubernetes master: [TEST][MASTER][KOLLA-KUBERNETES] Simple Test Master https://review.openstack.org/433136 | 13:56 |
| Jeffrey4l_ | sbezverk, sdake_ ^^ | 13:56 |
| Jeffrey4l_ | i removed "\\n" in the template_str. | 13:57 |
| Jeffrey4l_ | then the "\\" can not parse error gone. | 13:57 |
| Jeffrey4l_ | and get hostvars issue, which is easy to fix. | 13:57 |
| sdake_ | Jeffrey4l_ so your saying you think the gate can be fixed shortly? | 13:57 |
| Jeffrey4l_ | maybe sdake_ | 13:58 |
| sdake_ | Jeffrey4l_ +++++++++++++++++++++++++++++++++100 | 13:58 |
| *** fguillot has joined #openstack-kolla | 13:58 | |
| sdake_ | Jeffrey4l_ thanks a bunch dude, I am totally stuck - and have zero time | 13:58 |
| sdake_ | I know you ahve zero time too | 13:58 |
| sdake_ | I think we need m0ar people :) | 13:59 |
| Jeffrey4l_ | jinja2 script get '"host_in_groups": "{{\n inventory_h ' string, which can not parser by jinja2. | 13:59 |
| Jeffrey4l_ | i guess ansible may do some trick to handle this . | 13:59 |
| sdake_ | ya master ansible has unpinned from 2.9 | 13:59 |
| sdake_ | of jinja2 | 13:59 |
| sdake_ | however the current stable release pins on <2.9 | 13:59 |
| Jeffrey4l_ | this should not related to jinja2 | 13:59 |
| Jeffrey4l_ | i tested this with jinja2 2.8,which do not works too. | 14:00 |
| sdake_ | ok :) | 14:00 |
| *** tovin07 has joined #openstack-kolla | 14:00 | |
| sdake_ | well I dont toatlly understand what that patch fixes, but if you say it works that works for me :) | 14:00 |
| Jeffrey4l_ | let's wait the gate result ;) | 14:01 |
| sbezverk | Jeffrey4l_: http://logs.openstack.org/36/433136/2/check/gate-kolla-kubernetes-python27-ubuntu-xenial/0e89779/console.html#_2017-02-14_14_00_59_080132 | 14:02 |
| Jeffrey4l_ | ;/ | 14:03 |
| Jeffrey4l_ | is that possible for kolla-kuberenetes to inject a groups variables? | 14:04 |
| sdake_ | Jeffrey4l_ sbezverk gets those results via zuul somehow I think - notsure - I'd like to know so I can see premature gate results | 14:04 |
| Jeffrey4l_ | in the neutron patch ,we are using ' "host_in_groups": "{{ inventory_hostname in groups['neutron-lbaas-agent'] }}"' such statement. | 14:05 |
| Jeffrey4l_ | which requires groups variables. | 14:05 |
| sbezverk | sdake_: right you can get logs from zuul | 14:05 |
| sdake_ | sbezverk all I get is a telnet window | 14:05 |
| sdake_ | sbezverk anyway we can discuss when the fire is out :) | 14:05 |
| sdake_ | and my fire is out - which will be thursday after i pass out for 15 hours :) | 14:05 |
| sdake_ | to top it off my wife is travleing this morning for 2 days | 14:06 |
| sdake_ | last minute travel | 14:06 |
| *** tvignaud has quit IRC | 14:10 | |
| *** hachi has quit IRC | 14:10 | |
| sbezverk | Jeffrey4l_: you can try to define it in etc/kolla-kubernetes/kolla-kubernetes.yml but I doubt it will help | 14:10 |
| sdake_ | ok - well sounds like I am not helpful for this conversation at this time - if someone needs me ping me pls | 14:12 |
| sdake_ | I've got to prepare for wednesday | 14:12 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-kubernetes master: [TEST][MASTER][KOLLA-KUBERNETES] Simple Test Master https://review.openstack.org/433136 | 14:16 |
| *** lamt has quit IRC | 14:16 | |
| *** kristian__ has joined #openstack-kolla | 14:16 | |
| *** hachi has joined #openstack-kolla | 14:17 | |
| *** tvignaud has joined #openstack-kolla | 14:18 | |
| sbezverk | Jeffrey4l_: still the same :( | 14:20 |
| sbezverk | http://logs.openstack.org/36/433136/3/check/gate-kolla-kubernetes-python27-ubuntu-xenial/b8e1176/ | 14:20 |
| Jeffrey4l_ | sbezverk, patch is not working. trying again. | 14:20 |
| sbezverk | Jeffrey4l_: the log is for your latest ps | 14:20 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-kubernetes master: [TEST][MASTER][KOLLA-KUBERNETES] Simple Test Master https://review.openstack.org/433136 | 14:21 |
| Jeffrey4l_ | sbezverk, use patch in wrong. try again. | 14:21 |
| kfox1111 | morning. | 14:21 |
| sbezverk | kfox1111: good morning | 14:22 |
| Jeffrey4l_ | morning. | 14:22 |
| sbezverk | kfox1111: you are early today ;) | 14:22 |
| kfox1111 | couldn't sleep. | 14:23 |
| kfox1111 | how goes the battle? | 14:23 |
| sbezverk | kfox1111: Jeffrey4l_ found a piece of code which breaks our gate | 14:24 |
| kfox1111 | cthe neutron change? | 14:26 |
| Jeffrey4l_ | kfox1111, yep. | 14:26 |
| kfox1111 | is the neutron_services chunk used by anything in the config? | 14:26 |
| kfox1111 | can we just put a single if != kubernetes around the whole thing? | 14:27 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-kubernetes master: [TEST][MASTER][KOLLA-KUBERNETES] Simple Test Master https://review.openstack.org/433136 | 14:28 |
| Jeffrey4l_ | kfox1111, yep. we added a host_in_groups variable in neutron defaults. which cause the issue. | 14:28 |
| Jeffrey4l_ | i am trying to fix it, see above. | 14:28 |
| *** saneax is now known as saneax-_-|AFK | 14:28 | |
| kfox1111 | k | 14:30 |
| sbezverk | sdake: please workflow it https://review.openstack.org/#/c/433036/ | 14:33 |
| sbezverk | kfox1111: when entrypoint gets merged do we push it with the same tag 4.0.0? | 14:34 |
| Jeffrey4l_ | sbezverk, py35 is gree. | 14:34 |
| Jeffrey4l_ | green. | 14:34 |
| kfox1111 | sbezverk: probably ok I think. | 14:35 |
| sbezverk | Jeffrey4l_ it looks better, but we need to make this fix global as it fails at the different place now | 14:35 |
| sbezverk | 2017-02-14 14:34:26.511698 | 'groups' is undefined | 14:35 |
| Jeffrey4l_ | groups should be disappear after applied this patch https://review.openstack.org/#/c/433136/5/a.patch | 14:37 |
| sbezverk | Jeffrey4l_: you apply it in tests.sh but somehow it does not help for other case where the same code runs | 14:37 |
| sbezverk | Jeffrey4l_ or you mean apply this path on ansible side? | 14:38 |
| Jeffrey4l_ | sbezverk, i applied that into kolla(kolla-ansible)'s code. | 14:38 |
| Jeffrey4l_ | yep. | 14:38 |
| sbezverk | got it | 14:38 |
| kfox1111 | sbezverk: it needs o be appliet to setup_gate.sh too. | 14:40 |
| *** srwilkers has joined #openstack-kolla | 14:42 | |
| *** NobodyCam has quit IRC | 14:42 | |
| *** alanmeadows has quit IRC | 14:42 | |
| *** lamt has joined #openstack-kolla | 14:43 | |
| *** lucker has quit IRC | 14:43 | |
| *** lucker has joined #openstack-kolla | 14:44 | |
| zhubingbing_ | ping sdake | 14:45 |
| *** NobodyCam has joined #openstack-kolla | 14:47 | |
| *** kristian__ has quit IRC | 14:47 | |
| Jeffrey4l_ | sbezverk, why we still get groups is undefined? is there other place where clone kolla-ansible's code? | 14:48 |
| sbezverk | Jeffrey4l_: yes we clone kolla-ansible 1 more time for gate setup | 14:48 |
| Jeffrey4l_ | ok. | 14:48 |
| kfox1111 | tools/setup_gate_common does too. | 14:48 |
| sbezverk | Jeffrey4l_: it would make more sense I think if your neutron change will be applied in kolla-ansible repo | 14:49 |
| Jeffrey4l_ | sbezverk, i will when this test patch is green. | 14:49 |
| sbezverk | in this case it will cover all possible cases | 14:49 |
| Jeffrey4l_ | so, only two place, right? kfox1111 ? | 14:49 |
| kfox1111 | yes | 14:50 |
| sbezverk | Jeffrey4l_: yes | 14:50 |
| Jeffrey4l_ | ok. | 14:50 |
| kfox1111 | maybe we should break that out and make it shared. | 14:50 |
| *** eaguilar has joined #openstack-kolla | 14:50 | |
| *** jtriley has joined #openstack-kolla | 14:52 | |
| sbezverk | kfox1111: if it is temporary for testing it will be easier just add second patch command in common gate | 14:54 |
| sbezverk | script | 14:54 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-kubernetes master: [TEST][MASTER][KOLLA-KUBERNETES] Simple Test Master https://review.openstack.org/433136 | 14:54 |
| Jeffrey4l_ | why not use zuul-cloner to clone the kolla-ansible project? in this way, it will support cross repo test. | 14:56 |
| kfox1111 | Jeffrey4l_: yeah. the py27 script isnt as advanced as the setup_gate one. | 15:01 |
| kfox1111 | setup_gate does use zuul cloner. | 15:01 |
| kfox1111 | like I said, we probably should share code between them. | 15:01 |
| Jeffrey4l_ | +1 | 15:02 |
| sbezverk | kfox1111: Jeffrey4l_: monitoring one job and it went past the place where it normally failed.. | 15:03 |
| Jeffrey4l_ | nice ;)( | 15:03 |
| kfox1111 | cool. | 15:03 |
| Jeffrey4l_ | and in current solution, only need change kolla-ansible. | 15:04 |
| Jeffrey4l_ | check the patch. https://review.openstack.org/#/c/433136/5/a.patch | 15:04 |
| sbezverk | Jeffrey4l_: could you push it since we know now that it fixes the issue? | 15:06 |
| kfox1111 | looks good. but would it work to just have one big {{ if orchestration_engine == 'KUBERNETES' }} around the whole chunk? | 15:06 |
| Jeffrey4l_ | i am pushing the patch. | 15:06 |
| kfox1111 | k | 15:07 |
| *** ipsecguy has quit IRC | 15:07 | |
| Jeffrey4l_ | two issue, actually, 1. is what you said. 2 is use the same indent for host_in_group. | 15:08 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix the host_in_groups error in kolla-ansible. https://review.openstack.org/433694 | 15:10 |
| Jeffrey4l_ | kfox1111, sbezverk check ^^ | 15:10 |
| sbezverk | Jeffrey4l_: would it be safe to add replace for \n into our code as well? just in case? | 15:13 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible master: Fix the host_in_groups error in kolla-kubernetes. https://review.openstack.org/433694 | 15:13 |
| *** mgiles has quit IRC | 15:14 | |
| Jeffrey4l_ | sbezverk, no now. we will careful to use such format. | 15:14 |
| Jeffrey4l_ | when using the same indent for all string, yaml will not add extra '\n' in the value. | 15:14 |
| sbezverk | Jeffrey4l_: sounds good | 15:14 |
| Jeffrey4l_ | egonzalez, mliima could u review https://review.openstack.org/433554 | 15:15 |
| Jeffrey4l_ | we need bump the openstack service version periodic | 15:15 |
| sdake_ | Jeffrey4l_ thanks for firefighting :) | 15:20 |
| kfox1111 | +1 | 15:20 |
| Jeffrey4l_ | ;) | 15:20 |
| sdake_ | if someone needs a review, pls ping me | 15:21 |
| sdake_ | we need to branch asap | 15:21 |
| sdake_ | and tag kolla-kubernetes wednesday | 15:21 |
| sdake_ | hopefully we can jam the dev env into kolla-kubernetes without breaking it prior to 0.5.0 | 15:21 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/newton: Bump service version number https://review.openstack.org/433554 | 15:22 |
| sdake_ | can we get some reviews going on that after the gate is functional? | 15:22 |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix haproxy configure for external placement api listen https://review.openstack.org/433618 | 15:22 |
| sdake_ | https://review.openstack.org/#/c/429295/ | 15:23 |
| sdake_ | note htis patch was passing the gate - and should pass the gate again, I think if bmace can get some reviews on it, that would be fantastic! | 15:23 |
| sdake_ | kfox1111 sbezverk especially since it touches lots of the gating implementation | 15:24 |
| sbezverk | sdake_: compute kit goes in first :) | 15:24 |
| sdake_ | reviews | 15:24 |
| openstackgerrit | Merged openstack/kolla-ansible master: update dispatcher configurations for database backend https://review.openstack.org/427639 | 15:24 |
| sdake_ | bmace any chance you can handle that waitforpods rework as a seprate patch - we can really merge that after 0.5.0 imo | 15:24 |
| *** crushil has joined #openstack-kolla | 15:24 | |
| sdake_ | i was pretty sure compute kit was ready to go | 15:25 |
| sdake_ | are we just blocked o n the gate there? | 15:25 |
| sbezverk | sdake_: right | 15:25 |
| sdake_ | right so blocked on gate on merging - are reviews completed? | 15:25 |
| sdake_ | sbezverk I was pretty sure i reviewed it several times as well as srwilkers reviewed it and found something i missed | 15:26 |
| sbezverk | sdake_: srwinkler removed his +2 so I am missing one +2 | 15:26 |
| sdake_ | why did srwilkers remove his +2? | 15:26 |
| sdake_ | srwilkers ? | 15:26 |
| sbezverk | sdake_: he got scared by red gate | 15:26 |
| sdake_ | i see | 15:26 |
| srwilkers | yep, removed due to the gate issues | 15:26 |
| sdake_ | when gate is red, we ccan keep +2's in place and just not +w it | 15:26 |
| sdake_ | we do this all the time | 15:26 |
| sdake_ | everyone knows not to merge a patch with a red gate (i.e. press the +w) | 15:26 |
| sdake_ | srwilkers if you left yoru +2 in place, I could +w it or you could depending on who was around at the time | 15:27 |
| *** mgiles has joined #openstack-kolla | 15:27 | |
| sdake_ | I sually leave a note as well "Please do not workflow until gate is green" | 15:27 |
| sdake_ | morning mgiles | 15:27 |
| srwilkers | sdake_: thats fine. removed it before seeing the issue was related to the jinja problems | 15:27 |
| sdake_ | srwilkers roger | 15:27 |
| sdake_ | srwilkers ya the gate is a disaster although i think it will be fixed shortly | 15:27 |
| sdake_ | disaster is a bit of wrong term there but you get the ide a:) | 15:28 |
| sdake_ | its BDWNF :) | 15:28 |
| srwilkers | yeah, was watching the discussions last night and digging through the jinja stuff on the side | 15:28 |
| kfox1111 | yeah. kolla-ansible fix is in process. | 15:28 |
| sbezverk | Jeffrey4l_: Super job your test patch came out all GREEEEENNNNN | 15:28 |
| kfox1111 | I'm going to try and get the ceph stuff to helm figured out today. | 15:28 |
| sdake_ | Jeffrey4l_ for president! | 15:28 |
| openstackgerrit | Merged openstack/kolla master: Update for kubernetes entrypoint https://review.openstack.org/433036 | 15:28 |
| Jeffrey4l_ | yeah ;) | 15:28 |
| sdake_ | sounds good kfox1111 | 15:28 |
| sdake_ | nice kubernetes entrypoitn merged | 15:29 |
| sdake_ | hopefull ytaht doesn't braek something | 15:29 |
| Jeffrey4l_ | then please review this patch ,which unblock kolla-kubernetes gate https://review.openstack.org/433694 | 15:29 |
| sdake_ | say kfox1111 - you weren't aroudn earlier when i was takling about corss-repo gating | 15:29 |
| sdake_ | I think if your willing to take that work on that is somethingthat needs to be done shortly after the PTG, depending on when we think we can remove the kolla-ansible dependency | 15:29 |
| sdake_ | we can discuss more at the PTG | 15:30 |
| sdake_ | if we think we can rmeove kolla-ansible dependency quickly and clean up the mess we currently have, then cross repo gating may be a waste of time with kolla-ansible | 15:30 |
| sdake_ | although with kolla it would be very valuable | 15:31 |
| srwilkers | +1 | 15:31 |
| srwilkers | also, +1 | 15:31 |
| sdake_ | ok enough armchair desing from me :) | 15:31 |
| sdake_ | I'll either write code or GTFOW | 15:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 15:32 |
| sdake_ | srwilkers kfox1111 sbezverk can all of you review the following patch pls and make commentary | 15:33 |
| sdake_ | https://review.openstack.org/#/c/429295/ | 15:33 |
| sdake_ | if its good to go +1 it if not -1 it | 15:33 |
| sdake_ | i'd really like to get this sorted out before Jeffrey4l_ tags 0.5.0 | 15:33 |
| kfox1111 | when is the tagging? | 15:33 |
| sdake_ | the 15th | 15:34 |
| sdake_ | wednesday | 15:34 |
| sdake_ | the same day as branch day | 15:34 |
| sdake_ | althoguh we will branch sooner | 15:34 |
| sdake_ | we are tagging kolla-ansible rc1 on wednesday | 15:34 |
| sdake_ | and tagging kolla-kubernetes 0.5.0 on wednesday | 15:34 |
| sdake_ | the branch is created from the tag | 15:34 |
| sdake_ | so thats why the branch happens when teh tag happens | 15:34 |
| kfox1111 | sdake_: I did review. one outstanding issue. | 15:34 |
| sdake_ | kfox1111 thanks :) | 15:34 |
| sdake_ | kfox1111 other then that gtg? | 15:35 |
| sbezverk | sdake: do not forget this one https://review.openstack.org/#/c/433694/2 | 15:35 |
| sbezverk | sdake: should be first priority ;) | 15:35 |
| kfox1111 | sdake_: yeah. I think so. | 15:35 |
| sdake_ | Jeffrey4l_ re https://review.openstack.org/#/c/429295/ - can you communicate that to the team - that that work needs to stay in place until we eject kolla-ansible as a dep from kolla-kubernetes | 15:36 |
| sdake_ | Jeffrey4l_ and future reviews that use that syntax need to accommodate that issue | 15:36 |
| Jeffrey4l_ | sdake_, confused. no idea what this patch is doing https://review.openstack.org/#/c/429295/ | 15:37 |
| Jeffrey4l_ | did u paste the wrong link? | 15:37 |
| sdake_ | Jeffrey4l_ ya - I wasn't asking yo uto reivew that patch - its sort of a complicated thign that requires alot of context in kolla-kubernetes | 15:38 |
| sdake_ | oh right | 15:38 |
| sdake_ | sorry wrong link | 15:38 |
| sdake_ | mybad | 15:38 |
| openstackgerrit | Merged openstack/kolla-ansible master: Do not check security opt in some case in kolla_docker module https://review.openstack.org/429459 | 15:38 |
| sdake_ | Jeffrey4l_ thisis the correct link: https://review.openstack.org/#/c/433694/2/ansible/roles/neutron/defaults/main.yml | 15:38 |
| Jeffrey4l_ | yes. i will. | 15:39 |
| sdake_ | Jeffrey4l_ thanks dude - i'm not sure how you figured that out | 15:39 |
| sdake_ | Jeffrey4l_ good work tho :) | 15:40 |
| Jeffrey4l_ | made some debug ;) | 15:40 |
| sdake_ | Jeffrey4l_ the kolla-kubernetes plan is to get rid of the mess there in kolla-ansible so it can be more tidy by handling config generation within kolla-kubernetes | 15:40 |
| *** unicell has joined #openstack-kolla | 15:41 | |
| *** unicell1 has quit IRC | 15:41 | |
| Jeffrey4l_ | that will be cool. | 15:41 |
| sdake_ | Jeffrey4l_ how we do that remains to be seen - although it involves from my understanding external config maps (external being a helm package of some sort) | 15:41 |
| Jeffrey4l_ | any plan for this? kolla-kuberneters will be maintain its own configures? | 15:41 |
| Jeffrey4l_ | for move the configs into another place , then used by kolla-ansible and kolla-kubernetes? | 15:42 |
| *** rwallner has quit IRC | 15:42 | |
| *** tonanhngo has joined #openstack-kolla | 15:43 | |
| sdake_ | Jeffrey4l_ i think for the short term we may end up with differing base configs | 15:43 |
| sdake_ | and then unify them later in the spirit of velocity | 15:43 |
| *** matrohon has quit IRC | 15:44 | |
| sdake_ | IOW we will copy kolla-ansible's configs, un-injafy them, and package them in helm in some way | 15:44 |
| Jeffrey4l_ | cool. kolla-kubernetes should maintain its own config. ;) | 15:44 |
| sdake_ | Jeffrey4l_ April 31st is really our deadline for having everything highly functional | 15:44 |
| Jeffrey4l_ | for kolla-kubernetes? what's the special for April 31st? prepare for summit? | 15:45 |
| *** rwallner has joined #openstack-kolla | 15:45 | |
| *** skramaja has joined #openstack-kolla | 15:46 | |
| *** adrian_otto has joined #openstack-kolla | 15:48 | |
| sbezverk | kfox1111: new entrypoint is at the docker hub | 15:49 |
| *** jtriley_ has joined #openstack-kolla | 15:50 | |
| *** hieulq_ has joined #openstack-kolla | 15:52 | |
| *** jtriley has quit IRC | 15:53 | |
| sbezverk | Jeffrey4l_: could you ask another ansible core to ack your patch? | 15:55 |
| *** kristian__ has joined #openstack-kolla | 15:56 | |
| Jeffrey4l_ | sbezverk, np. | 15:56 |
| Jeffrey4l_ | egonzalez, could u ack https://review.openstack.org/433694 | 15:57 |
| Jeffrey4l_ | sbezverk, done. | 15:57 |
| sbezverk | Jeffrey4l_: thanks so much! | 15:58 |
| *** matrohon has joined #openstack-kolla | 15:58 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Fix the host_in_groups error in kolla-kubernetes. https://review.openstack.org/433694 | 16:00 |
| *** zhubingbing_ has quit IRC | 16:00 | |
| *** erlon has joined #openstack-kolla | 16:03 | |
| *** unicell has quit IRC | 16:08 | |
| *** ccrouch has joined #openstack-kolla | 16:13 | |
| sdake_ | @jeff | 16:14 |
| sdake_ | Jeffrey4l_ yes summit prep | 16:14 |
| sdake_ | say folks - here is the agenda for the meetup i am presenting wednesday locally | 16:15 |
| sdake_ | any feedback welcome | 16:15 |
| sdake_ | https://etherpad.openstack.org/p/sdake-feb-meetup-agenda | 16:15 |
| *** financeman has quit IRC | 16:19 | |
| *** tovin07 has quit IRC | 16:20 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Add the missing condition for config.yml https://review.openstack.org/433419 | 16:23 |
| *** brad[] has quit IRC | 16:23 | |
| *** srwilkers has quit IRC | 16:24 | |
| *** david-lyle has joined #openstack-kolla | 16:26 | |
| *** david-lyle_ has quit IRC | 16:29 | |
| openstackgerrit | Merged openstack/kolla-ansible master: Checking certificate files for octavia https://review.openstack.org/432279 | 16:29 |
| *** matrohon has quit IRC | 16:30 | |
| *** vi01et03 has joined #openstack-kolla | 16:30 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing service dependencies https://review.openstack.org/433761 | 16:31 |
| *** vhosakot has joined #openstack-kolla | 16:31 | |
| sbezverk | kfox1111: after entrypoint started working right, we need to remove service dependency when the process depends on its own service | 16:32 |
| sbezverk | please see PS https://review.openstack.org/433761 | 16:32 |
| *** jtriley_ has quit IRC | 16:35 | |
| *** prameswar has quit IRC | 16:35 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Stabilizing the gate, fixing resolv.conf issue https://review.openstack.org/432764 | 16:37 |
| *** srwilkers has joined #openstack-kolla | 16:40 | |
| *** alanmeadows has joined #openstack-kolla | 16:41 | |
| bmace | sdake_: what waitforpods rework? | 16:43 |
| kfox1111 | k | 16:44 |
| sdake_ | bmace at some point in the review, i dont know if this is the latest bits, kevin had asked for the waitforpods py code to be broken out | 16:44 |
| openstackgerrit | Merged openstack/kolla master: Using CI mirror host https://review.openstack.org/365721 | 16:44 |
| *** jtriley has joined #openstack-kolla | 16:44 | |
| sdake_ | bmace if kfox1111 can wait on that, then it doesn't matter to me :) | 16:44 |
| sdake_ | bmace kfox1111 had said the patch only has 1 problem left | 16:44 |
| kfox1111 | bmace: yeah. the waitforpod thing in services might cause gate instability as it doesn't handle the same cases the older version does. | 16:44 |
| kfox1111 | is it possible to break that out on its own? | 16:45 |
| sdake_ | I haven't had a chance to look at the numerous reviews - DRINKINGFROMFIREHOSE | 16:45 |
| sdake_ | and we tag tomorrow | 16:45 |
| sdake_ | so i want that patch merged today | 16:45 |
| sdake_ | what i want may not happen :) | 16:45 |
| bmace | kfox1111: i would be a little surprised if it was an issue.. and i don't know if i can break it out because i'm not sure how well devenv will work with the old behavior | 16:45 |
| kfox1111 | sbezverk: is the new entrypoint on both centos and ubuntu? | 16:45 |
| bmace | kfox1111: that pull stuff is certainly problematic when running remotely | 16:46 |
| bmace | kfox1111: at one point tests were passing... before the gate went crazy | 16:46 |
| kfox1111 | bmace: It took me a very long time to get the old version working reliably. over a month at least. thats why I'm skeptical of the new version. | 16:46 |
| kfox1111 | it may pass one iteration, and then fail the next. the bringup is very nondeterministic. | 16:47 |
| openstackgerrit | Merged openstack/kolla master: Multi-Regions Support https://review.openstack.org/431588 | 16:47 |
| bmace | kfox1111: the gate feels like it is failing semi-randomly quite well without my changes | 16:47 |
| sdake_ | bmace yes - although that is being corrected | 16:47 |
| kfox1111 | bmace: yeah. and I'm trying to fix it, and prevent more of that. | 16:47 |
| kfox1111 | thats the issue. :/ | 16:48 |
| bmace | kfox1111: the old wait behavior wasn't working well in the devenv bringup and that is why i removed it.. if you can find a case where mine doesn't work i'm happy to either wait or fix it | 16:48 |
| bmace | removed = replaced | 16:48 |
| sdake_ | ok how about this | 16:48 |
| kfox1111 | bmace: then can you make it conditional? don't use it in the gate? | 16:49 |
| sdake_ | we merge it as is, and then fix it post ptg to include the use cases kfox1111 outlines | 16:49 |
| sdake_ | kfox1111 thta is also another fine suggestion | 16:49 |
| sdake_ | whatever it takes to get it merged today - lets do that | 16:49 |
| kfox1111 | I really want to get the gate back to stable. | 16:49 |
| sdake_ | yup getting gate back to stable makes perfect sense to me, although with most people atteh ptg next week we can deal with some temporary apin | 16:50 |
| kfox1111 | sdake_: if we can't get it stable, we cant landn patches. thats a big problem. | 16:50 |
| sdake_ | right | 16:50 |
| kfox1111 | how far out is the kolla-ansible patch? | 16:50 |
| sdake_ | ok - bmace any thoughts on the conditionalizing it? | 16:51 |
| sbezverk | kfox1111: it has been nerged | 16:51 |
| bmace | that is fine.. i'll use the first arg.. if gate job comes in as "devenv" then i will set a devenv var to true and also set ceph-multi because that is what devenv uses. | 16:51 |
| sbezverk | kfox1111: but fixing entrypoint it uncovered issue in our code | 16:51 |
| kfox1111 | bmace: k. sounds good. | 16:52 |
| kfox1111 | sbezverk: whats that? | 16:52 |
| sbezverk | some services were depending on its own service, like mariadb | 16:52 |
| bmace | kfox1111: do you want me to put the less than helpful pulls back in the non-devenv cases as well? | 16:52 |
| sbezverk | but now entrypoint really checks for presence of endpoints | 16:52 |
| sbezverk | before dependency is resolved | 16:52 |
| sbezverk | so chicken and egg issue | 16:53 |
| sbezverk | I just pushed PS to clean that up | 16:53 |
| sbezverk | https://review.openstack.org/433761 | 16:53 |
| sbezverk | there were 4 services total with this problem | 16:53 |
| *** dave-mcc_ has joined #openstack-kolla | 16:53 | |
| *** jtriley has quit IRC | 16:53 | |
| kfox1111 | bmace: how about maybe we do two bash functions at the top. one for pull/wait, one for wait. | 16:53 |
| kfox1111 | on the devenv, pull/wait switches to your wait. and the second to your wait. | 16:54 |
| bmace | kfox1111: both to my wait? ;) lol, i know what you mean.. it will be fine. | 16:54 |
| kfox1111 | :) | 16:55 |
| kfox1111 | k | 16:55 |
| *** jtriley has joined #openstack-kolla | 16:55 | |
| sbezverk | kfox1111: could you review it? | 16:55 |
| *** rcherrueau has joined #openstack-kolla | 16:55 | |
| kfox1111 | sbezverk: gotcha. lets merge it quick. | 16:55 |
| vhosakot | team, if anyone has deployed halcyon-vagrant-kubernetes, can they check if they see https://github.com/att-comdev/halcyon-vagrant-kubernetes/issues/62 ? | 16:55 |
| vhosakot | bmace: ^^^ | 16:55 |
| des_consolado | hmmm this is confusing. I've got the ironic service enabled and I've got 2 controller nodes. I've just destroyed ironic_pxe and the ironic_pxe volume and then deployed them again. There's a difference between the file contents on both of these nodes, I'll paste the ls -lah | 16:56 |
| kfox1111 | sbezverk: did the kolla-ansible fix land yet? | 16:56 |
| sbezverk | yes | 16:56 |
| kfox1111 | ok. cool. thanks. | 16:56 |
| sbezverk | basically we just need that service PS and then life should be easier | 16:56 |
| des_consolado | http://paste.openstack.org/show/598866/ | 16:56 |
| vhosakot | has host has not disk space looks like :( adding more disk space now. | 16:56 |
| *** dave-mccowan has quit IRC | 16:56 | |
| vhosakot | has host==my host | 16:56 |
| vhosakot | not disk==no disk | 16:57 |
| bmace | vhosakot: i haven't had any issues like that.. sorry | 16:57 |
| vhosakot | bmace: cool, thanks for checking... how much is the disk space of each vagrant VM? 10 GB? | 16:57 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Fixing service dependencies https://review.openstack.org/433761 | 16:57 |
| bmace | vhosakot: yeah, i left kube_disk at 10 | 16:57 |
| vhosakot | bmace: cool | 16:58 |
| vhosakot | bmace: how much disk space do you have on the host? | 16:58 |
| vhosakot | > 40 GB? | 16:58 |
| bmace | vhosakot: though usually i see some sort of message at startup about the disk size, and i think 10 may be below some "minimum" so it might be slightly more.. i can jump into one and check real quick | 16:58 |
| vhosakot | bmace: cool, four vagrant VMs each with 10 G disk space, means the host must have at least >~40 disk space, is this right? | 16:59 |
| bmace | vhosakot: lol, on the host that i'm running the vms on? a bit more than 40, yeah.. like 900gig | 16:59 |
| *** Pavo has quit IRC | 16:59 | |
| vhosakot | bmace: wow, 900G, you on bare metal? lucky you :) | 16:59 |
| bmace | vhosakot: if your host OS doesn't take up any room at all 40 would be fine.. but that is unlikely so you probably want a fair bit more | 16:59 |
| vhosakot | bmace: yeah, so, I'm thiking like 60 G for he disk, assuming it needs 20G for the host OS and vagrant | 17:00 |
| vhosakot | he=the | 17:00 |
| bmace | vhosakot: lol, yeah, i know it is odd to work on an honest to goodness real computer in this virtualized day and age. | 17:00 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 17:00 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 17:01 |
| *** sdake has joined #openstack-kolla | 17:01 | |
| vhosakot | bmace: lol, cool, I'll add disk space and keep you posted.. thanks for the info! | 17:01 |
| *** sdake_ has quit IRC | 17:02 | |
| bmace | vhosakot: you also need room for the code repos and to do the build, etc, on the host system unless you are doing it somewhere else.. personally i would be pessimistic and give it a bit more space than it likely needs than run out of disk space a bunch of times trying to determine the minimal viable configuration, but up to you. | 17:02 |
| *** japestinho__ has joined #openstack-kolla | 17:02 | |
| vhosakot | bmace: yeah, good point, I might add like 100G or something | 17:02 |
| bmace | vhosakot: i would expect that would be safer, yes, if that is an option. | 17:02 |
| vhosakot | bmace: if the vagrant host has disk space _less_ than the sum of the disk spaces of the VMs booted by vagrant, wouldn't you expect vagrant to complain with an error like "not enough disk space to boot 4 VMs each with 10 G disk" or something like that? | 17:03 |
| vhosakot | I don;t see this error, I'll dig my logs and env, and keep you posted. | 17:04 |
| *** jpstnh has quit IRC | 17:04 | |
| bmace | vhosakot: seems like a nice feature to add / toss upstream.. a size check on the host system of the combined disk space needed. | 17:04 |
| bmace | vhosakot: and if it isn't met, fail early | 17:04 |
| *** sdake has quit IRC | 17:05 | |
| vhosakot | bmace: yes, I might open a bug against vagrant once I have enough info.. right, fail early instead of booting with less disk space and crashing services inside the VMs.. right | 17:05 |
| *** hieulq_ has quit IRC | 17:05 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Remove unused task from mariadb https://review.openstack.org/433789 | 17:06 |
| openstackgerrit | Nenad Radojevic proposed openstack/kolla-ansible master: Add default roles used by Barbican https://review.openstack.org/422643 | 17:07 |
| *** sdake has joined #openstack-kolla | 17:08 | |
| *** rwallner has quit IRC | 17:09 | |
| *** rwallner has joined #openstack-kolla | 17:09 | |
| *** rwallner has quit IRC | 17:09 | |
| *** egonzalez has quit IRC | 17:10 | |
| des_consolado | hey when I delete this docker volume for ironic_pxe, I can still locate files that were in the volume using updatedb/locate | 17:14 |
| des_consolado | I've also ran `sudo docker volume prune -f` | 17:14 |
| des_consolado | this is obviously just a gap in my docker knowledge but do you know how to completely destroy a volume once and for all? | 17:14 |
| des_consolado | oh wait no I think it's fine | 17:17 |
| des_consolado | it's just that the file I'm looking for is part of the distro stuff in aufs | 17:17 |
| *** skramaja has quit IRC | 17:21 | |
| *** rwallner has joined #openstack-kolla | 17:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 17:22 |
| *** dis_consolate has joined #openstack-kolla | 17:23 | |
| dis_consolate | hi des_consolado | 17:23 |
| dis_consolate | i'm your brother | 17:23 |
| *** dis_consolate has quit IRC | 17:24 | |
| *** sudipto has joined #openstack-kolla | 17:28 | |
| *** sudipto_ has joined #openstack-kolla | 17:28 | |
| *** rcherrueau has quit IRC | 17:29 | |
| bmace | is sbezverk's compute kit change good to go? if so can we get it workflowed in? i need / conflict with part of it for my devenv final testing. | 17:30 |
| *** pcaruana has quit IRC | 17:31 | |
| sbezverk | bmace: I think it is good to go, just waiting for one fix which had to push for new kubernetes entrypoint | 17:31 |
| sbezverk | bmace: new entrypoint will affect you as well | 17:31 |
| pbourke | hmm is the voting open for the summit talks? | 17:31 |
| bmace | of course it will :/ | 17:31 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 17:32 |
| sbezverk | kolla-kube cores please ack this one urgently https://review.openstack.org/433761 | 17:32 |
| sbezverk | it is all green | 17:32 |
| *** rwallner has quit IRC | 17:32 | |
| srwilkers | just did | 17:32 |
| sbezverk | srwilkers: thank you | 17:33 |
| sbezverk | bmace: once it is merged compute kit can go in | 17:34 |
| bmace | sbezverk: looks like it has a workflow +1.. odd that it looks like it doesn't have a jenkins verified yet the jobs ran? | 17:35 |
| inc0 | good morning | 17:36 |
| inc0 | b 11 | 17:36 |
| bmace | morning inc0 | 17:36 |
| srwilkers | morning inc0 | 17:36 |
| inc0 | disregard, my irc client is still...a bit of a black magic for me | 17:36 |
| sbezverk | bmace: it is still running | 17:37 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Fixing service dependencies https://review.openstack.org/433761 | 17:37 |
| bmace | there we go :) | 17:37 |
| *** fragatina has joined #openstack-kolla | 17:38 | |
| sbezverk | bmace: with new entrypoint we should not see massive containers crashes | 17:39 |
| sdake | inc0 which irc client you using | 17:40 |
| sdake | I use textual for mac and love it | 17:40 |
| *** lamt has quit IRC | 17:40 | |
| inc0 | weechat with vim bindings | 17:41 |
| inc0 | in tmux | 17:41 |
| sbezverk | srwilkers: please ack compute kit now | 17:41 |
| sdake | yikes old school irc ;) | 17:41 |
| inc0 | on vm on top of openstack | 17:41 |
| inc0 | I ssh to my irc | 17:41 |
| sdake | I am semi-fond of slack | 17:41 |
| srwilkers | sbezverk: done | 17:41 |
| sbezverk | srwilkers: thank you | 17:41 |
| sdake | inc0 is that an intel policy thing or a your choice kind of thing :) | 17:41 |
| srwilkers | sbezverk: np | 17:41 |
| sdake | inc0 sounds like a PITA | 17:41 |
| inc0 | sdake: it's me | 17:42 |
| inc0 | this way I can have irc constantly running | 17:42 |
| inc0 | so people can ping me when I'm away | 17:42 |
| inc0 | and I can connect to same irc from my home pc too | 17:42 |
| bmace | sbezverk: less container crashes is a good thing.. i pulled some of your changes right into my review so i don't think we should conflict and i can get on with my testing. | 17:42 |
| inc0 | actually it's pretty conveninent - by now I do all my dev/reviews in this vm over ssh | 17:42 |
| inc0 | I just ssh back, tmux a and I'm back exactly where I left off | 17:43 |
| *** fragatina has quit IRC | 17:43 | |
| sbezverk | bmace: cool, compute kit should merge momentarely | 17:45 |
| bmace | sbezverk: if all my changes test well i hope to be following you in shortly :) | 17:46 |
| des_consolado | also whenever I want to destroy and re-deploy, I get errors with kolla_toolbox, Unable to remove filesystem blah blah blah /var/lib/docker/containers/blahblahblah/shm device or resource busy | 17:46 |
| des_consolado | then when I docker rm -f manually it moans again but then if I do it again it says no such container, so I guess it worked | 17:46 |
| des_consolado | this is on master from like last week, I should probably pull again | 17:47 |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Adding compute kit based on iscsi backend https://review.openstack.org/431156 | 17:49 |
| jascott1 | yay compute kit! | 17:50 |
| sbezverk | Yeeeessssss !!!! | 17:51 |
| srwilkers | sbezverk: good work on that | 17:53 |
| *** unicell has joined #openstack-kolla | 17:53 | |
| *** gfidente is now known as gfidente|afk | 17:55 | |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: [WIP] Convert remaining kolla_toolbox tasks to use the module https://review.openstack.org/433832 | 17:56 |
| *** mgoddard has quit IRC | 17:57 | |
| *** pbourke has quit IRC | 17:59 | |
| *** rwallner has joined #openstack-kolla | 18:01 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 18:02 |
| *** vi01et03 has quit IRC | 18:03 | |
| kfox1111 | jascott1: sbezverk: would be fasinating to try the conditional support on iscsi/ceph on the compute kit. :) | 18:03 |
| jascott1 | yes I will be playing with that today | 18:03 |
| jascott1 | release hasnt dropped yet | 18:04 |
| inc0 | and local path | 18:04 |
| inc0 | we can get rid of microservice dir | 18:04 |
| inc0 | it's scary looking | 18:04 |
| kfox1111 | jascott1: cool. :) | 18:05 |
| kfox1111 | inc0: no getting rid of microservice dir. | 18:05 |
| kfox1111 | inc0: +1 to local path. | 18:05 |
| jascott1 | it would be a lot of work at this point | 18:05 |
| *** krtaylor has quit IRC | 18:05 | |
| inc0 | kfox1111: why? wouldn't you want to have microservies sorted per parent service? | 18:06 |
| jascott1 | lets settle that in the ATL octagon/thunderdome | 18:06 |
| inc0 | :D | 18:06 |
| inc0 | at least have mircoservices/nova/nova-compute.... | 18:06 |
| kfox1111 | inc0: that only wworks if it can nicely fit into a tree. it cant | 18:06 |
| inc0 | how so? | 18:07 |
| kfox1111 | say you have a mariadb for neutron and nova. | 18:07 |
| kfox1111 | where does it live? | 18:07 |
| inc0 | in mariadb | 18:07 |
| inc0 | I assume local path will work with ../../mariadb | 18:08 |
| kfox1111 | say you have neutron-openvswitch-agent but its deployed by nova-compute? | 18:08 |
| inc0 | why would it be deployed by nova compute | 18:08 |
| inc0 | ? | 18:08 |
| kfox1111 | perhaps you want a service packge per host aggregate. | 18:08 |
| kfox1111 | so you can easily launch new host aggregates. | 18:08 |
| inc0 | also... ../../neutron/neutron-openvswitch-agent | 18:08 |
| *** fragatina has joined #openstack-kolla | 18:09 | |
| kfox1111 | so, you just want to group them all per service name, | 18:09 |
| kfox1111 | or do it in a chart fasion? | 18:09 |
| kfox1111 | neutron/charts/neutron-openvswitch-agent? | 18:09 |
| inc0 | we already have 115 dirs there and we barely scrached the surface of big tent | 18:10 |
| kfox1111 | maybe you can write up a spec on how you think it shoudl be aranged? | 18:10 |
| kfox1111 | so is it just an attempt to have less dirs per dir? | 18:11 |
| inc0 | yeah, wft, but I don't think current approach is scallable once we get into implementation of big tent | 18:11 |
| inc0 | not only | 18:11 |
| *** kristian__ has quit IRC | 18:11 | |
| kfox1111 | ok. maybe spec it with the problems your trying to solve and a proposed solution and we can discuss it easier. | 18:11 |
| inc0 | readibility counts and I want to avoid cross-service dependency spaghetti as much as possible | 18:11 |
| inc0 | like nova-compute depending on neutron | 18:12 |
| *** sambetts is now known as sambetts|afk | 18:12 | |
| inc0 | I get why it can be convenient but we could easily end up with stuff depending on each other in unmaintainable fashon | 18:12 |
| *** jtriley has quit IRC | 18:12 | |
| inc0 | it will be only matter of time before we end up with circular dependencies and such | 18:13 |
| kfox1111 | thats why I think microservices shoudl be seperate. it seperates orcestration from implementation. | 18:14 |
| kfox1111 | things can use them as needed. | 18:14 |
| kfox1111 | helm doesn't support circular deps. so we're good. ;) | 18:14 |
| inc0 | anyway, with local path we can segregate it into subfolders | 18:16 |
| inc0 | that's a start | 18:16 |
| kfox1111 | yeah. | 18:16 |
| * srwilkers reads | 18:16 | |
| inc0 | also what stops us from having /helm/serivce/nova-control/microservices/* structure? | 18:17 |
| inc0 | I mean microservice charts don't have requirements | 18:17 |
| kfox1111 | helm doesn't support that. | 18:17 |
| inc0 | charts within chart dir? | 18:17 |
| kfox1111 | they will in a day or two. | 18:17 |
| kfox1111 | microservices will have local path deps on kolla-common. | 18:17 |
| inc0 | ok, I'm talking about local path | 18:17 |
| *** athomas has quit IRC | 18:18 | |
| kfox1111 | inc0: it can, but only for dependencies. | 18:18 |
| kfox1111 | so you could do foo/charts/bar | 18:18 |
| *** ccrouch has quit IRC | 18:19 | |
| srwilkers | the charts/ directory gets created and populated on `helm dep up <chart>` | 18:19 |
| kfox1111 | yeah | 18:19 |
| kfox1111 | I'm really not surie its a good fit to rearange things in that way. | 18:19 |
| srwilkers | with where we’re at now, i agree | 18:19 |
| kfox1111 | right now, we have a clean split between charts that are building blocks, and charts that are orchestration. | 18:20 |
| inc0 | wfm, just at least lets segregate building blocks into multiple subdirs | 18:22 |
| kfox1111 | k. | 18:23 |
| inc0 | issue I'm having is that it looks bad from readability standpoint | 18:24 |
| kfox1111 | from doing an ls and seeing too many things? | 18:24 |
| inc0 | and aesthetics counts;) if we want to attract new contributors | 18:24 |
| inc0 | yeah | 18:24 |
| kfox1111 | +1 | 18:24 |
| inc0 | it's just intimidating | 18:25 |
| inc0 | and at some point it'll go exponential | 18:25 |
| kfox1111 | though I think thats a much smaller worry to ensuring we have a working/tested dev env | 18:25 |
| inc0 | yeah, true, priorities | 18:26 |
| inc0 | just this might be pretty easy chage with localpath in place, and easily scallable | 18:26 |
| inc0 | I mean work-scallable | 18:26 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 18:26 |
| inc0 | repeatable tasks over multiple services - good ones for onboarding new contributors | 18:26 |
| kfox1111 | it will have to touch the same amount of code before/after local path I think. | 18:27 |
| kfox1111 | prebuild_microservices and build_microservices need to be tought about 2 levels of dirs. | 18:27 |
| kfox1111 | not a hard thing. but a thing. | 18:27 |
| inc0 | we'll figure it out, all I'm saying | 18:28 |
| kfox1111 | yup | 18:28 |
| kfox1111 | just, non trivial. thats all I'm saying. :) | 18:28 |
| srwilkers | if only we could all be in the same place to discuss this in person or something ;) | 18:28 |
| kfox1111 | +1 :) | 18:29 |
| kfox1111 | localpath won't just make it 0 work. | 18:29 |
| inc0 | we'll discuss it, to me it's part of "path to kubernetes/charts" session | 18:29 |
| kfox1111 | yup. | 18:29 |
| kfox1111 | which I kind of think is a misnomer. | 18:29 |
| kfox1111 | more of a "how do we be as helm native as possible" | 18:30 |
| inc0 | well, yeah | 18:30 |
| srwilkers | yes, exactly kfox1111 | 18:30 |
| srwilkers | lets not jump the shark | 18:30 |
| kfox1111 | agreed | 18:30 |
| inc0 | but it's a play;) I don't want to say we're not helm native, because there is no such thing yet | 18:30 |
| inc0 | we're paving roads | 18:31 |
| kfox1111 | true. but stating the goal of entering kubernetes/charts, when thats not clear thats a desirable goal is equally a problem. | 18:31 |
| srwilkers | kfox1111: agree | 18:31 |
| inc0 | I'd like to think it's actually desirable ;) one way or another | 18:31 |
| bmace | kfox1111: remember the ceph issue i hit a few days back.. you mentioned something about pv / pvc and the way stuff behaves right now not getting the right disks in the right order or some such? is someone working on that? i still find bringing up the devenv stuff i will randomly get ceph failures where it seems the containers are borked because they have the wrong disks, or are both fighting over the same ones, idk.. | 18:32 |
| inc0 | well, it is desirable by me:P | 18:32 |
| inc0 | anyway, afk, gonna deploy kolla-k8s | 18:32 |
| inc0 | ;) | 18:32 |
| inc0 | need to check out compute kit patch;) | 18:32 |
| kfox1111 | inc0: if our stated goal is to deliver stable openstack to users, that may fly counter to the stated goals of kubernetes/charts | 18:32 |
| jascott1 | tis merged | 18:32 |
| inc0 | well, both can be goals but in case of priority first one is higher up | 18:32 |
| inc0 | thing is, this goal is 2-fold | 18:33 |
| inc0 | us being helm native and helm being closer to what we need | 18:33 |
| *** sudipto_ has quit IRC | 18:33 | |
| *** sudipto has quit IRC | 18:33 | |
| kfox1111 | bmace: its mostly a problem for mariadb/rabbit/etc. if ceph itself can't find its disks, thats a seperate issue. | 18:33 |
| inc0 | what I'm saying is, ideally we need to figure out it from both fronts, us and helm | 18:33 |
| inc0 | which we are doing currently:) | 18:33 |
| bmace | kfox1111: it is definitely only an issue for rabbitmq / mariadb bringup | 18:33 |
| bmace | kfox1111: at least that is the only time i have seen it | 18:33 |
| kfox1111 | bmace: ah. ok. | 18:34 |
| kfox1111 | bmace: yeah, not 100% sure what the right solution to it is, and as far as I know no one is currently working on it. | 18:34 |
| kfox1111 | bmace: I really think the issue is we're mixing two seperate users into one. | 18:34 |
| kfox1111 | pv's are k8s admin activities. pvc's are user activities. | 18:34 |
| kfox1111 | and we're kind of assuming they are all user activites. | 18:35 |
| kfox1111 | which may be true sometimes, perhaps. | 18:35 |
| *** sudipto has joined #openstack-kolla | 18:35 | |
| *** sudipto_ has joined #openstack-kolla | 18:35 | |
| *** sudipto_ has quit IRC | 18:35 | |
| *** sudipto has quit IRC | 18:35 | |
| bmace | kfox1111: is that really the problem though? it seemed more like you can't actually have a named pv so you end up pulling them in a non-deterministic order when doing the pvcs? or you said something like that | 18:36 |
| *** adrian_otto1 has joined #openstack-kolla | 18:36 | |
| kfox1111 | right. | 18:36 |
| kfox1111 | its expected in k8s, that and admin creates a bunch of pv's. | 18:37 |
| kfox1111 | then its expected a user puts in a bunch of pvc's that bind to random pv's. | 18:37 |
| kfox1111 | so the names aren't ever related as the users nameing them are different. | 18:37 |
| *** adrian_otto has quit IRC | 18:39 | |
| des_consolado | are you guys using AUFS? | 18:39 |
| bmace | kfox1111: so do we really need to get a specific pv? can't we just name them pv1 / pv2, etc.. and as long as mariadb can get to one on the host it wants to sit on its all good? it feels like there is some tighter binding somehow that, when it doesn't get the one it wants it blows up. | 18:39 |
| kfox1111 | bmace: yeah. thats what I argued for initially. just call them pvX. it doesn't matter at all to the pvc's. | 18:39 |
| bmace | kfox1111: but obviously it must matter in some way, at least right now, since sometimes things won't work if they don't get the ones they want | 18:40 |
| kfox1111 | have a look at a pvc. it doesn't specify anything identifying to a pv at all. | 18:40 |
| kfox1111 | bmace: I think its unrelated. something sometimes breaks, and it looks like its pv name related. | 18:40 |
| bmace | kk, certainly could be | 18:40 |
| kfox1111 | with 3 pv's, the likelyhood of at least 2 being named wrong is high. | 18:41 |
| *** DuncanT has quit IRC | 18:41 | |
| bmace | well, my failure rate seems like it hovers around 50% and maybe i have just been lucky :/ | 18:42 |
| *** MasterOfBugs has joined #openstack-kolla | 18:42 | |
| bmace | consistently around ceph / disk access from mariadb / rabbitmq | 18:42 |
| kfox1111 | yup. we need to dig further though. | 18:43 |
| kfox1111 | can you get it into that state again, and we can debug it? | 18:43 |
| kfox1111 | a kubectl describe pod podname would help. | 18:44 |
| inc0 | so, I assume our quickstart guide is totally outdated | 18:44 |
| inc0 | do we have docs for moving from installed k8s + installed ceph to running kolla? | 18:44 |
| kfox1111 | inc0: bmace has a ps up for a dev env. thats by far the closest docs we have now. | 18:45 |
| kfox1111 | everything else is very bitrotten/pre helm. | 18:45 |
| *** lamt has joined #openstack-kolla | 18:46 | |
| *** jtriley has joined #openstack-kolla | 18:47 | |
| inc0 | can I have link please?:) | 18:47 |
| bmace | inc0: https://review.openstack.org/#/c/429295/ | 18:48 |
| bmace | inc0: working on a couple of changes based on the recent compute kit merge.. testing now. hope to have a closer to last review up in an hour if all goes well | 18:48 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 18:51 |
| *** jtriley has quit IRC | 18:52 | |
| *** krtaylor has joined #openstack-kolla | 18:56 | |
| inc0 | sooo....my problem is | 18:58 |
| inc0 | I have k8s setup and I have ceph setup | 18:59 |
| inc0 | where do they connect? | 18:59 |
| inc0 | do I need to setup ceph creds in every k8s node or kolla configs are enough? | 18:59 |
| *** shardy has quit IRC | 19:00 | |
| kfox1111 | that use case is not very well documented. :/ | 19:01 |
| *** salv-orl_ has joined #openstack-kolla | 19:01 | |
| kfox1111 | you can be our guinipig. :) | 19:01 |
| kfox1111 | or post ptg, I can be a guinipig. | 19:01 |
| bmace | inc0: you using the halcyon-vagrant stuff? | 19:01 |
| *** mannamne has quit IRC | 19:02 | |
| bmace | inc0: the setup_dev_env.sh script sets up ceph on kube2 via loopbacks like we do in the gate | 19:02 |
| *** jtriley has joined #openstack-kolla | 19:03 | |
| kfox1111 | bmace: he said he has an existing ceph. | 19:03 |
| bmace | kfox1111: ah, well, good luck then :) | 19:03 |
| kfox1111 | yeah. | 19:03 |
| *** salv-orlando has quit IRC | 19:03 | |
| kfox1111 | we've got a couple of scripts that do that kind of setup, but not broken out enough yet that it would be simple to just call that one part. | 19:04 |
| inc0 | bmace: no, I'm using vms deployed with kargo | 19:04 |
| kfox1111 | inc0: see tools/setup_gate_ceph.sh | 19:04 |
| kfox1111 | from right after the line that says "tests/bin/build_test_ceph.sh" | 19:04 |
| inc0 | I'll just use host for now | 19:04 |
| kfox1111 | up to the kubectl res create configmap. | 19:04 |
| bmace | a process to do consistent bringups on bare metal / existing systems is likely to be my next adventure | 19:05 |
| kfox1111 | thats the logic that sets up the stuff in ceph/k8s for ceph. | 19:05 |
| kfox1111 | we probably shoudl break up that chunk into its own function so we can reuse it easily. | 19:05 |
| kfox1111 | bmace: cool. | 19:05 |
| *** saneax-_-|AFK is now known as saneax | 19:06 | |
| sbezverk | inc0: if you are ok to follow some steps, which are not scripted I can share my appraoch | 19:12 |
| *** srwilkers has quit IRC | 19:13 | |
| sbezverk | kfox1111: the gate is still green ;) ? | 19:14 |
| kfox1111 | sbezverk: sweet. lets try and keep it that way. ;) :) | 19:16 |
| *** rwallner has quit IRC | 19:16 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 19:17 |
| *** ccrouch has joined #openstack-kolla | 19:17 | |
| kfox1111 | inc0: can you ask the mascot team for a koala that's borgized image? ;) | 19:19 |
| *** jtriley has quit IRC | 19:19 | |
| inc0 | borgized? | 19:21 |
| *** salv-orl_ has quit IRC | 19:21 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 19:21 |
| kfox1111 | inc0: https://www.google.com/search?q=locutus+of+borg&safe=active&biw=981&bih=701&tbm=isch&imgil=1_gIigjLxEsS8M%253A%253BW9OTvHBtvXAEcM%253Bhttp%25253A%25252F%25252Fmemory-beta.wikia.com%25252Fwiki%25252FLocutus&source=iu&pf=m&fir=1_gIigjLxEsS8M%253A%252CW9OTvHBtvXAEcM%252C_&usg=__H3jsjgLswNxoG8GRiJGElx67GtI%3D&ved=0ahUKEwiDyqnKp5DSAhXDxlQKHcb6ALAQyjcIYA&ei=xVijWMPTGcON0wLG9YOACw#imgrc=1_gIigjLxE | 19:21 |
| *** salv-orlando has joined #openstack-kolla | 19:21 | |
| kfox1111 | sS8M: | 19:21 |
| kfox1111 | bad link... try: http://memory-beta.wikia.com/wiki/Locutus | 19:22 |
| inc0 | you want collective of koalas? | 19:22 |
| kfox1111 | just kind of fitting for kolla-kubernetes. a borgized koala, since google's own k8s is called borg. :) | 19:23 |
| kfox1111 | I'm mostly joking. | 19:24 |
| *** salv-orlando has quit IRC | 19:24 | |
| inc0 | well we could make blue-eyed koala | 19:24 |
| inc0 | after all, spice must flow | 19:24 |
| inc0 | and god eucalyptus is full of it | 19:24 |
| kfox1111 | indeed | 19:24 |
| inc0 | our koala indeed looks like it sees the universe | 19:25 |
| bmace | the fact that the kolla mascot is a koala feels odd.. it is as if we don't know how to spell koala :) | 19:25 |
| inc0 | there is secret behind koala | 19:25 |
| inc0 | our secret version of koala | 19:25 |
| *** rwallner has joined #openstack-kolla | 19:27 | |
| sbezverk | kfox1111: I need to use one of the gate job temporarely to test ironic containers. I do not intend to have them permanently. I will use centos iscsi microservices job for this purposes. | 19:28 |
| *** rwallner has quit IRC | 19:29 | |
| *** rwallner has joined #openstack-kolla | 19:31 | |
| *** srwilkers has joined #openstack-kolla | 19:31 | |
| *** salv-orlando has joined #openstack-kolla | 19:41 | |
| *** fragatina has quit IRC | 19:42 | |
| *** jtriley has joined #openstack-kolla | 19:43 | |
| *** fragatina has joined #openstack-kolla | 19:43 | |
| sdake | hey folks | 19:44 |
| bmace | yo sdake | 19:44 |
| sdake | in the marketing update - we added kubernetes support to Kolla in the Ocata cycle ;-) | 19:44 |
| sdake | sup bmace | 19:44 |
| bmace | sdake: meh, testing out a hopefully last pass at devenv changes after another trunk merge after the compute commit went in. | 19:45 |
| sdake | bmace that should be tteh last of the merge collisions | 19:45 |
| sdake | bmace we need to tag 0.5.0 on wednesday - as the release team will be at PTG and may not be around thursday and friday | 19:46 |
| *** jtriley has quit IRC | 19:48 | |
| *** matrohon has joined #openstack-kolla | 19:48 | |
| kfox1111 | sbezverk: ok. | 19:48 |
| kfox1111 | bmace: I'm afraid breakages will probably keep happening after merge if its not tested somehow. :/ | 19:49 |
| sdake | how about a recheck recheck recheck over and over | 19:49 |
| kfox1111 | sdake: can't recheck something thats not in the gate. :/ | 19:50 |
| bmace | sdake: i think he means the devenv stuff? which is indeed problematic, if we can't spin up a bunch of halcyon-vagrant vms in the gate. | 19:50 |
| kfox1111 | yeah. | 19:50 |
| bmace | as long as people actually use it we should find out | 19:50 |
| sdake | i see - you want to gat eethe devenv | 19:50 |
| bmace | what other real choice is there? | 19:50 |
| kfox1111 | people actually using it isn't the problem. | 19:50 |
| kfox1111 | new people freshly checking it out is. | 19:50 |
| sdake | i think we can rely on humans to do that job | 19:51 |
| kfox1111 | sdake: that way lies madness. ;) | 19:51 |
| sdake | devenv is all about human interaction with the dev env | 19:51 |
| sdake | anyway we are not blocking on the devenv to wait for gating | 19:51 |
| sdake | we have been blocked for over a month on it | 19:51 |
| *** egonzalez90 has joined #openstack-kolla | 19:51 | |
| sdake | and its slowing down community growth significantly | 19:51 |
| kfox1111 | +1 for not waiting on a gate. | 19:51 |
| kfox1111 | just saying, it will be hard to keep it unbroken. | 19:52 |
| egonzalez90 | inc0 ping re https://bugs.launchpad.net/kolla-ansible/+bug/1664674 | 19:52 |
| openstack | Launchpad bug 1664674 in kolla-ansible "kolla-ansible tries to restart containers while genconfig" [Critical,New] | 19:52 |
| *** adrian_otto1 has quit IRC | 19:52 | |
| sdake | agreed - that is why we have a team :) | 19:52 |
| kfox1111 | heck. without a kolla-ansible gate, it took 3 days to break kolla-kubernetes. | 19:52 |
| kfox1111 | twice now. | 19:52 |
| *** adrian_otto has joined #openstack-kolla | 19:53 | |
| *** jtriley has joined #openstack-kolla | 19:57 | |
| *** kristian__ has joined #openstack-kolla | 19:58 | |
| openstackgerrit | Borne Mace proposed openstack/kolla-kubernetes master: Added support for containerized dev bringup https://review.openstack.org/429295 | 19:58 |
| bmace | kfox1111: feel free to take a look at the ceph_workflow_service changes i put in there. i make sure that both the old and new workflow work | 19:58 |
| sbezverk | kfox1111: I think we will not get new people between now and next summit, so as long as by the summit it is nice and clean, I think it should be ok | 19:59 |
| sdake | kfox1111 right - cross repo gating would have prevented that | 20:00 |
| sdake | so here is the deal - we need to unblock devs entirely and completely | 20:01 |
| sdake | if we block them with "we wont merge it because there is no gating" that sucks | 20:01 |
| sdake | if we block them with "its broken because there is no gating" atleast they ahve something to fix :) | 20:01 |
| sbezverk | bmace: sorry if you answered that question before but what was the driver to start using python wait for container? | 20:01 |
| *** kristian__ has quit IRC | 20:02 | |
| bmace | sbezverk: i personally found it a lot cleaner just to use the kubernetes api instead of parsing stuff off the command line, and i hate bash | 20:02 |
| kfox1111 | bmace: looks good to me. | 20:03 |
| kfox1111 | sdake: I keep saying, I don't want to block the thing on lack of gating. | 20:03 |
| bmace | sbezverk: there was some other code around as part of the process that also uses python, maybe one of the secret generation things, don't recall exactly, but it was also python | 20:03 |
| kfox1111 | sdake: I am saying, I don't know how long we can keep it not broken without a gate. | 20:04 |
| sbezverk | bmace: right, that was my first python commit to kube ;) | 20:04 |
| vhosakot | bmace: do you see docker-storage-setup.service running happily on all your nodes? | 20:05 |
| vhosakot | bmace: can you send "systemctl status -l docker-storage-setup.service" so I can compare it with my output. | 20:05 |
| kfox1111 | sdake: I am worried that there is a ballance between getting new devs in, and having devs scared to commit because they may break something thats not easily testable. | 20:05 |
| kfox1111 | I was generally scared to commit stuff to kolla-kubernetes until I started a gate. | 20:06 |
| kfox1111 | it mattered less then too, as no one was using it yet. | 20:06 |
| bmace | vhosakot: doesn't seem like it is running, seems to be disabled? | 20:06 |
| sbezverk | kfox1111: qq to generate new configmaps I need to change service_resource.yml right? | 20:07 |
| kfox1111 | sbezverk: yeah | 20:07 |
| bmace | vhosakot: which, on the Loaded line, looks to be the vendor default | 20:07 |
| vhosakot | bmace: ah ok, do you see it failed when you do "systemctl --failed" | 20:07 |
| sbezverk | it has been so long since I touched it | 20:07 |
| kfox1111 | well... | 20:07 |
| kfox1111 | maybe not. I think if it wasnt in the list, and the type was configmap, it assumed it would be built with kubectl out of /etc/kolla. | 20:07 |
| kfox1111 | yeah. its been a long time since I touched it too. :/ | 20:08 |
| bmace | vhosakot: no.. probably because it is disabled? | 20:08 |
| bmace | vhosakot: disabled != failed, really, even though Active: shows as failed in status -l.. Active: should probably show disabled, but systemd is what it is :) | 20:09 |
| bmace | vhosakot: why the question? what sort of issue are you hitting? the only one i hit consistently is the ceph one where mariadb / rabbitmq seem to fight over disks.. | 20:10 |
| vhosakot | bmace: I see... can you please send the outputs of "systemctl | grep docker-storage-setup.service", "systemctl list-unit-files | grep docker-storage-setup.service" and "systemctl status -l docker-storage-setup.service" so I can compare. | 20:10 |
| vhosakot | bmace: well, I'm just trying to check if the failure is fine/harmless... | 20:11 |
| mgiles | vhosakot bmace fyi, i see failed for that service in my vagrant environment, but the devenv stuff still runs successfully | 20:12 |
| bmace | vhosakot: first two outputs are empty. last one shows Loaded: not-found (no such file or directory) and Active: is inactive | 20:12 |
| bmace | yup, ditto mgiles | 20:12 |
| vhosakot | mgiles: bmace cool, thanks for the info... | 20:12 |
| bmace | i don't think it is a concern | 20:12 |
| vhosakot | bmace: cool, good to know! | 20:13 |
| vhosakot | mgiles: "devenv stuff still runs successfully" meaning you k8s is functional and you can create/delete pods, services, etc ? | 20:13 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 20:14 |
| openstackgerrit | Borne Mace proposed openstack/kolla-kubernetes master: Added support for containerized dev bringup https://review.openstack.org/429295 | 20:14 |
| mgiles | vhosakot yes. I also can successfully run the docker container from bmace's review | 20:14 |
| *** saneax is now known as saneax-_-|AFK | 20:14 | |
| mgiles | vhosakot which runs to completion successfully for me | 20:15 |
| vhosakot | mgiles: sweet, thanks for the confirmation! :) | 20:15 |
| vhosakot | bmace: thanks for all the help! | 20:15 |
| bmace | :) | 20:15 |
| mgiles | vhosakot well, did as of rev 22 of the ps - haven't tried the very newest stuff | 20:15 |
| bmace | it only gets better ;) | 20:15 |
| bmace | ok, not much different from 22 really, just dealing with recent merges mostly and some tweaks to keep the behavior the same as it used to be for pull / wait in the gate | 20:16 |
| *** openstackgerrit has quit IRC | 20:18 | |
| kfox1111 | yeah. the devenv stuff looks really nice. :) | 20:19 |
| *** salv-orlando has quit IRC | 20:20 | |
| *** adrian_otto has quit IRC | 20:21 | |
| *** sdake has quit IRC | 20:22 | |
| *** sdake has joined #openstack-kolla | 20:22 | |
| *** openstackgerrit has joined #openstack-kolla | 20:24 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 20:24 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 20:31 |
| *** hyakuhei has joined #openstack-kolla | 20:31 | |
| *** hyakuhei has quit IRC | 20:31 | |
| *** hyakuhei has joined #openstack-kolla | 20:31 | |
| *** hyakuhei has quit IRC | 20:31 | |
| *** hyakuhei has joined #openstack-kolla | 20:31 | |
| v1k0d3n | hey vhosakot :) | 20:32 |
| vhosakot | v1k0d3n: hey, thanks for all the reviews/commantes! :) | 20:33 |
| v1k0d3n | glad things are working out for you guys with the repo | 20:33 |
| vhosakot | comments* | 20:33 |
| v1k0d3n | of course :) | 20:33 |
| v1k0d3n | happy to help | 20:33 |
| v1k0d3n | i didn't really know anyone was still using it that much | 20:33 |
| vhosakot | v1k0d3n: yep, the repo is great! | 20:34 |
| v1k0d3n | glad it's working out for you guys | 20:34 |
| *** lamt has quit IRC | 20:34 | |
| *** lamt has joined #openstack-kolla | 20:36 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 20:38 |
| *** mannamne has joined #openstack-kolla | 20:39 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Helm Ceph Initial Mon https://review.openstack.org/410641 | 20:39 |
| *** mannamne has quit IRC | 20:40 | |
| *** bmace has quit IRC | 20:41 | |
| *** bmace has joined #openstack-kolla | 20:41 | |
| *** egonzalez90 has quit IRC | 20:41 | |
| *** krtaylor has quit IRC | 20:46 | |
| *** lamt has quit IRC | 20:48 | |
| *** athomas has joined #openstack-kolla | 20:49 | |
| *** kristian__ has joined #openstack-kolla | 20:52 | |
| *** kristian__ has quit IRC | 20:52 | |
| *** kristian__ has joined #openstack-kolla | 20:52 | |
| kfox1111 | helm 2.2.0 :) | 20:53 |
| bmace | full deploy on my lastest devenv change worked like a charm so any feedback is appreciated and lets get this puppy in ;) | 20:54 |
| kfox1111 | bmace: looks like there is a slight gate breakage left. | 20:55 |
| kfox1111 | but really really close. :) | 20:55 |
| bmace | kfox1111: i looked, doesn't look like it is from my change | 20:55 |
| kfox1111 | bmace: worked before your change. | 20:56 |
| bmace | well then possibly inconsistent? guess i can recheck.. but no reason the helm command would fail with any of my changes | 20:56 |
| kfox1111 | and its isolated to the ceph_workflow_service file. | 20:56 |
| kfox1111 | not a recheck issue. | 20:56 |
| kfox1111 | actual brokenness. pretty sure. | 20:56 |
| sbezverk | kfox1111: bmace: you are talking about helm-service failure in your ps? | 20:59 |
| bmace | yeah | 20:59 |
| sbezverk | bmace: run gate test to confirm | 20:59 |
| bmace | idk.. i can fumble around in the gate output.. the error isn't super clear | 20:59 |
| sbezverk | bmace: usually this error means missing some variable and instead parser finds either empty value or "" | 21:00 |
| *** ssurana has joined #openstack-kolla | 21:02 | |
| kfox1111 | sbezverk: bmace's review. | 21:02 |
| bmace | i think i see it.. i think some sort of arg usage somewhere to this script changed.... again.. and of course they aren't documented anywhere | 21:02 |
| kfox1111 | bmace: yeah. any help to cleanup the scripts would be appreciated. | 21:03 |
| bmace | i see... did my other review not get merged yet? | 21:03 |
| *** Pavo has joined #openstack-kolla | 21:03 | |
| bmace | my change where we yank out the no longer valid "2" arg needs to go in | 21:03 |
| kfox1111 | but thats also why we are worried about breaking it even after merged. | 21:03 |
| kfox1111 | I thought it merged... let me look. | 21:03 |
| bmace | https://review.openstack.org/#/c/432509/ | 21:04 |
| *** rhallisey has quit IRC | 21:04 | |
| bmace | that will fix this | 21:04 |
| kfox1111 | doesn't look like it. :/ | 21:05 |
| bmace | doesn't look like what? like it will fix it? | 21:05 |
| kfox1111 | oh. it was cause of kolla-ansible broken at the time. rechecking. | 21:05 |
| kfox1111 | doesn't look like it merged. | 21:05 |
| bmace | no, didn't merge, no workflow. all it does is get rid of completely unused args to the ceph workflow script.. | 21:06 |
| kfox1111 | k. I'll wf it once its done testing. | 21:07 |
| kfox1111 | can you rebase your devenv one on it in the mean time? | 21:08 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 21:08 |
| *** kristian__ has quit IRC | 21:09 | |
| bmace | i can just drop the other patch and roll it into my devenv change | 21:10 |
| bmace | if timing is what you are concerened about | 21:10 |
| *** dave-mcc_ has quit IRC | 21:10 | |
| kfox1111 | no. just thinking if you rebased the one on the other, the test might clear right away. | 21:11 |
| kfox1111 | and a rebase will probably need to happen anyway once the other one merges. | 21:11 |
| *** bpattewar has quit IRC | 21:12 | |
| *** bpattewar has joined #openstack-kolla | 21:13 | |
| *** rwallner has quit IRC | 21:14 | |
| *** rwallner has joined #openstack-kolla | 21:15 | |
| *** bpattewar has quit IRC | 21:15 | |
| *** bpattewar has joined #openstack-kolla | 21:16 | |
| *** rwallner has quit IRC | 21:18 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 21:20 |
| *** rwallner has joined #openstack-kolla | 21:21 | |
| *** mgiles has quit IRC | 21:23 | |
| inc0 | so, "set is not defined" after trying to do helm install rabbitmq | 21:25 |
| *** rwallner has quit IRC | 21:25 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 21:25 |
| *** krtaylor has joined #openstack-kolla | 21:27 | |
| *** jrobinson has joined #openstack-kolla | 21:28 | |
| *** athomas has quit IRC | 21:32 | |
| *** lamt has joined #openstack-kolla | 21:34 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 21:34 | |
| *** satyar has quit IRC | 21:35 | |
| openstackgerrit | Borne Mace proposed openstack/kolla-kubernetes master: Added support for containerized dev bringup https://review.openstack.org/429295 | 21:35 |
| openstackgerrit | Borne Mace proposed openstack/kolla-kubernetes master: Remove no longer relevant argument to ceph_workflow* https://review.openstack.org/432509 | 21:35 |
| *** Jeffrey4l_ has quit IRC | 21:35 | |
| *** l4yerffeJ has quit IRC | 21:35 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 21:35 | |
| inc0 | sooo guys you are aware that we *need* to move to latest kolla with kolla-k8s? and that also means ansible until we figure out configs | 21:41 |
| inc0 | stuff currently doesn't work with master | 21:42 |
| *** crushil has quit IRC | 21:44 | |
| *** adrian_otto has joined #openstack-kolla | 21:44 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-ansible master: Changing ironic-inspector configuration for kolla-kubernetes https://review.openstack.org/433941 | 21:49 |
| *** zioproto has joined #openstack-kolla | 21:50 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-ansible master: Changing ironic-inspector configuration for kolla-kubernetes https://review.openstack.org/433941 | 21:50 |
| *** salv-orlando has joined #openstack-kolla | 21:51 | |
| *** Pavo has quit IRC | 21:53 | |
| sbezverk | kfox1111: I have a qq if you have 2 minutes please ping back.. | 21:55 |
| *** zioproto has quit IRC | 21:57 | |
| *** fguillot has quit IRC | 21:58 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 21:59 |
| *** srwilkers has quit IRC | 22:00 | |
| *** matrohon has quit IRC | 22:03 | |
| *** sdake has quit IRC | 22:04 | |
| bmace | all green on the argument removal review | 22:05 |
| *** lamt has quit IRC | 22:06 | |
| *** crushil has joined #openstack-kolla | 22:06 | |
| *** lamt has joined #openstack-kolla | 22:11 | |
| *** gfidente|afk has quit IRC | 22:12 | |
| *** lamt has quit IRC | 22:21 | |
| *** jtriley has quit IRC | 22:24 | |
| *** lamt has joined #openstack-kolla | 22:24 | |
| *** krtaylor has quit IRC | 22:26 | |
| openstackgerrit | Merged openstack/kolla-kubernetes master: Remove no longer relevant argument to ceph_workflow* https://review.openstack.org/432509 | 22:27 |
| *** sdake has joined #openstack-kolla | 22:28 | |
| bmace | thanks sbezverk | 22:28 |
| sbezverk | bmace: I like greeeennn ;) | 22:29 |
| bmace | lol.. any idea if the failure in my devenv run is something that is hit once in a while? looks like a "no route to host" very late in the process | 22:30 |
| *** sdake has quit IRC | 22:30 | |
| sbezverk | bmace: we saw the same several times at the gate as well | 22:30 |
| bmace | kk, i'll do recheck then | 22:30 |
| sbezverk | did not have cycles to hunt it | 22:31 |
| *** sdake has joined #openstack-kolla | 22:31 | |
| bmace | sbezverk: just wanted to make sure it was a known thing and unlikely related to my change. | 22:31 |
| sbezverk | bmace: very unlikely it is related to yours | 22:32 |
| bmace | sbezverk: yeah, sounds like it, very happy to hear it. hopefully the recheck runs all green | 22:32 |
| *** bpattewar_ has joined #openstack-kolla | 22:33 | |
| *** tonanhngo_ has joined #openstack-kolla | 22:34 | |
| *** imcsk8_ has joined #openstack-kolla | 22:34 | |
| *** rafael_sisweb_ has joined #openstack-kolla | 22:35 | |
| *** SaMnCo_ has joined #openstack-kolla | 22:35 | |
| *** sdake_ has joined #openstack-kolla | 22:36 | |
| sdake_ | inc0 if you man the kolla containers, I've been beating that drum for months with no effect | 22:36 |
| *** mannamne has joined #openstack-kolla | 22:36 | |
| sdake_ | inc0 if you maen kolla-ansible, the gate is currently using and depends on kolla-ansible master | 22:36 |
| *** cliles_ has joined #openstack-kolla | 22:36 | |
| sdake_ | inc0 i think there is an impedence mismatch between some of our devs that think a non 1.0.0 version is somehow supportable and backward compatible | 22:37 |
| *** tonyb_ has joined #openstack-kolla | 22:38 | |
| *** Serlex has quit IRC | 22:40 | |
| *** dgonzalez_ has joined #openstack-kolla | 22:40 | |
| *** dims has joined #openstack-kolla | 22:41 | |
| *** sdake__ has joined #openstack-kolla | 22:41 | |
| *** sdake has quit IRC | 22:42 | |
| *** hyakuhei has quit IRC | 22:42 | |
| *** tonanhngo has quit IRC | 22:42 | |
| *** rafael_sisweb has quit IRC | 22:42 | |
| *** jemcevoy has quit IRC | 22:42 | |
| *** dims_ has quit IRC | 22:42 | |
| *** vcn[m] has quit IRC | 22:42 | |
| *** mnaser has quit IRC | 22:42 | |
| *** dgonzalez has quit IRC | 22:42 | |
| *** imcsk8 has quit IRC | 22:42 | |
| *** wirehead_ has quit IRC | 22:42 | |
| *** SaMnCo has quit IRC | 22:42 | |
| *** cliles has quit IRC | 22:42 | |
| *** tonyb has quit IRC | 22:42 | |
| *** saneax-_-|AFK has quit IRC | 22:42 | |
| *** dgonzalez_ is now known as dgonzalez | 22:42 | |
| *** rafael_sisweb_ is now known as rafael_sisweb | 22:42 | |
| *** SaMnCo_ is now known as SaMnCo | 22:43 | |
| *** mnaser has joined #openstack-kolla | 22:44 | |
| *** sdake_ has quit IRC | 22:44 | |
| *** tonanhngo_ has quit IRC | 22:46 | |
| *** MasterOfBugs has quit IRC | 22:46 | |
| *** vhosakot has quit IRC | 22:46 | |
| *** hachi has quit IRC | 22:46 | |
| *** hrw has quit IRC | 22:46 | |
| *** v1k0d3n has quit IRC | 22:46 | |
| *** klindgren has quit IRC | 22:46 | |
| *** zigo has quit IRC | 22:46 | |
| *** qwang has quit IRC | 22:46 | |
| *** jascott1 has quit IRC | 22:46 | |
| *** masber has quit IRC | 22:46 | |
| *** devananda has quit IRC | 22:46 | |
| *** jowisz has quit IRC | 22:46 | |
| *** clayton has quit IRC | 22:46 | |
| *** Slower has quit IRC | 22:46 | |
| *** gema has quit IRC | 22:46 | |
| *** otavio has quit IRC | 22:46 | |
| *** yee379 has quit IRC | 22:46 | |
| *** tonyb_ is now known as tonyb | 22:46 | |
| *** hrw has joined #openstack-kolla | 22:46 | |
| *** MasterOfBugs has joined #openstack-kolla | 22:46 | |
| *** otavio has joined #openstack-kolla | 22:46 | |
| *** devananda has joined #openstack-kolla | 22:47 | |
| *** hachi has joined #openstack-kolla | 22:48 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 22:48 |
| *** vcn[m] has joined #openstack-kolla | 22:50 | |
| *** qwang has joined #openstack-kolla | 22:50 | |
| *** jemcevoy has joined #openstack-kolla | 22:51 | |
| *** zigo has joined #openstack-kolla | 22:52 | |
| *** v1k0d3n has joined #openstack-kolla | 22:52 | |
| *** klindgren has joined #openstack-kolla | 22:52 | |
| *** masber has joined #openstack-kolla | 22:52 | |
| *** jowisz has joined #openstack-kolla | 22:52 | |
| *** clayton has joined #openstack-kolla | 22:52 | |
| *** Slower has joined #openstack-kolla | 22:52 | |
| *** gema has joined #openstack-kolla | 22:52 | |
| *** yee379 has joined #openstack-kolla | 22:52 | |
| *** tonanhngo has joined #openstack-kolla | 22:52 | |
| kfox1111 | sbezverk: ping | 22:53 |
| *** vhosakot has joined #openstack-kolla | 22:53 | |
| *** zigo is now known as Guest27057 | 22:54 | |
| sbezverk | kfox1111: I was wondering if you could show me how to from the script to clone a specicif not merged ps | 22:54 |
| kfox1111 | gotta go :/ another meeting. | 22:56 |
| kfox1111 | real quick though, | 22:56 |
| kfox1111 | if you lok at the review you wat to pul from, | 22:56 |
| kfox1111 | there is a "Download" link in the corner. | 22:56 |
| kfox1111 | if you click it you get a bunch of options. | 22:56 |
| kfox1111 | the quickest way I've found, is to copy the format patch line, | 22:56 |
| sbezverk | kfox1111: no rush | 22:56 |
| kfox1111 | and pipe that to a file. so like, | 22:56 |
| kfox1111 | git fetch https://git.openstack.org/openstack/kolla-kubernetes refs/changes/09/432509/1 && git format-patch -1 --stdout FETCH_HEAD > foo.patch | 22:57 |
| kfox1111 | then at the root of the repo, apply it: | 22:57 |
| kfox1111 | patch -p1 < foo.patch | 22:57 |
| kfox1111 | or unapply it, | 22:57 |
| kfox1111 | patch -p1 -R < foo.patch | 22:57 |
| sbezverk | kfox1111: cool got it, need to save iit | 22:57 |
| openstackgerrit | Merged openstack/kolla-ansible master: Changing ironic-inspector configuration for kolla-kubernetes https://review.openstack.org/433941 | 22:57 |
| sbezverk | so I will not forget next time | 22:57 |
| *** MasterOfBugs has quit IRC | 22:58 | |
| *** MasterOfBugs has joined #openstack-kolla | 22:59 | |
| *** adrian_otto has quit IRC | 23:01 | |
| klindgren | Afternoon | 23:02 |
| klindgren | Question - what check does kolla do to make sure that a deployed container actually started successfully? | 23:02 |
| klindgren | Specifically I dont see anything dealing with the newish docker healthcheck feature | 23:03 |
| klindgren | I believe that a number of services have implemented the systemd healthcheck feature. Seems like this would go hand in hand with the docker healthcheck feature | 23:05 |
| inc0 | klindgren: we don't really have healthchecks | 23:06 |
| inc0 | good idea too look at tho | 23:06 |
| inc0 | k8s could use it | 23:06 |
| kfox1111 | kolla-kube does do some health checks, but not all yet. | 23:07 |
| klindgren | kfox1111, is that using some internal kube function? | 23:09 |
| bmace | devenv review all green... the beautiful green | 23:09 |
| kfox1111 | kubernetes has health checks as a first class resource in pods. | 23:11 |
| sbezverk | bmace: niiiiccceee | 23:11 |
| inc0 | ok I'll resume my fight with deployment tomorrow;) | 23:11 |
| bmace | nice work on fixing the gates btw guys.. was looking really rough the last couple of days | 23:11 |
| *** crushil has quit IRC | 23:12 | |
| klindgren | kfox1111, I assume you are talking about: https://github.com/openstack/kolla-kubernetes/blob/9a50bb5d514a3302ccc68ead0ffe5ae13474ef45/helm/kolla-common/templates/_common_api_python_deployment.yaml#L94-L97 | 23:12 |
| klindgren | ? | 23:12 |
| *** jascott1 has joined #openstack-kolla | 23:12 | |
| kfox1111 | yeah. | 23:14 |
| kfox1111 | we're not doing it too much yet, but there's a livenss version too, that will actually kill pods if they aren't responsive enough. | 23:15 |
| *** crushil has joined #openstack-kolla | 23:16 | |
| bmace | kfox1111: that seems both interesting and dangerous, with great power and all that. if there is a common back end resource that is effecting responsiveness of all pods of a particular kind you don't want to nuke them all, etc. | 23:16 |
| kfox1111 | bmace: yup. :) | 23:18 |
| kfox1111 | yeah, that is the risk of that model. | 23:18 |
| klindgren | kfox1111, is kube doing anything with rpc based containers? IE services that just connect to rabbit and do work, vs's api services which actually bind to a port and are easily interegatable for health. | 23:18 |
| kfox1111 | if the db locks up, it could shoot everything. | 23:18 |
| klindgren | we have had the db lock in prod | 23:19 |
| kfox1111 | no health checks on them currently. but they could be added. | 23:19 |
| bmace | yeah, i guess if you have some poorly written pods that just slow down or lock up for no external reason it makes sense.. but feels more the exception rather than the rule. | 23:19 |
| klindgren | which impacted other services like keystone | 23:19 |
| kfox1111 | I think it was intended to heal around bad nodes, in very large clusters. | 23:19 |
| klindgren | when neutron was chewing db like a mofo | 23:19 |
| kfox1111 | yeah. its a mixed bag. long term, I think they shoudl be totally configurable. | 23:20 |
| klindgren | due to dhcp_agent update notification and exetensive use of vm's on provider networks | 23:20 |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 23:20 |
| kfox1111 | leave it up to the operator to decide how to set the settings or distable them. | 23:20 |
| bmace | kfox1111: makes sense.. a node behaving really badly from a disk or networking or whatever perspective. would think maybe some node health checks might be just as effective and perhaps a bit safer.. but it is what it is :) | 23:21 |
| klindgren | is anyone aware of how openstack hooks into systemd currently for health checking? | 23:21 |
| klindgren | I know this works: https://github.com/openstack/oslo.service/blob/master/oslo_service/systemd.py#L59-L67 | 23:21 |
| *** fragatin_ has joined #openstack-kolla | 23:22 | |
| *** fragatina has quit IRC | 23:25 | |
| *** Pavo has joined #openstack-kolla | 23:27 | |
| kfox1111 | there are those too I think. if anode is deamed sick, k8s will reschedule its pods. | 23:29 |
| kfox1111 | I think that support is farily limited at the moment though, so sick is more like "kubelet is no longer talking to me for too long. it must be dead" | 23:30 |
| kfox1111 | I konw they are working on making that much finer grained. | 23:30 |
| kfox1111 | so like, "attached ssd died, but networking is fine, so dont evict network only pods" | 23:31 |
| *** l4yerffeJ_ has quit IRC | 23:36 | |
| *** l4yerffeJ_ has joined #openstack-kolla | 23:37 | |
| *** dave-mccowan has joined #openstack-kolla | 23:42 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: WIP adding Ironic microservices packages https://review.openstack.org/433322 | 23:45 |
| *** Pavo has quit IRC | 23:45 | |
| *** salv-orlando has quit IRC | 23:47 | |
| kfox1111 | sbezverk: ironic's looking nice. :) | 23:49 |
| kfox1111 | we need to figure out just what we can/can't test. | 23:49 |
| kfox1111 | and test what we can. | 23:50 |
| sdake__ | bmace re debugging gates, huge team effort including several people in different teams | 23:50 |
| bmace | sdake__: and that is why i didn't call out anyone specifically :P | 23:51 |
| *** sdake__ is now known as sdake | 23:52 | |
| sdake | bmace :) | 23:52 |
| *** vhosakot has quit IRC | 23:52 | |
| sdake | re tag of 0.5.0 - where we at with the dev script | 23:52 |
| sdake | i think we are nearly ready to tag | 23:53 |
| *** ssurana has quit IRC | 23:54 | |
| sbezverk | kfox1111: :) qq there is dnsmasq service in ironic | 23:55 |
| sbezverk | which runs dhcp | 23:55 |
| sbezverk | kfox1111: do you think doing it as a deployment make sense? | 23:55 |
| kfox1111 | might depend on what its doing. | 23:56 |
| sdake | bmace looks like requirements team added kuberenetes to global requirements | 23:56 |
| kfox1111 | dns seems to work well through a svc/deployment. I do it on my home system. | 23:56 |
| *** wirehead_ has joined #openstack-kolla | 23:57 | |
| kfox1111 | tftp might work but unsure. but it needs a shared space I think? | 23:57 |
| kfox1111 | dhcp... pretty sure that will need to be net=host | 23:57 |
| bmace | sdake: yeah, i know, tests wouldn't be green without it :) | 23:57 |
| sdake | bmace ya - sadly there is a whitespace problem which may crater reno | 23:57 |
| sbezverk | kfox1111: right I plan to run conductor and tftp in the same pod | 23:58 |
| kfox1111 | sbezverk: do you have an archetecture diagram for ironic? | 23:58 |
| *** ssurana has joined #openstack-kolla | 23:58 | |
| openstackgerrit | Borne Mace proposed openstack/kolla-kubernetes master: Added support for containerized dev bringup https://review.openstack.org/429295 | 23:58 |
| sbezverk | kfox1111: http://docs.openstack.org/developer/ironic/deploy/user-guide.html | 23:58 |
| kfox1111 | if they are in the same pod, with a emptydir, it might work, but only if net=host and replica's hardcoded to 1? | 23:58 |
| bmace | huh? what white space? it magically disappeared ;) | 23:58 |
| kfox1111 | or maybe better as a statefulset. | 23:59 |
| sbezverk | kfox1111: cannot as there might be multiple conductors | 23:59 |
| sbezverk | they will be daemonsets | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!