| *** rwellum has quit IRC | 00:00 | |
| *** stefan_bo has joined #openstack-kolla | 00:07 | |
| *** fguillot has joined #openstack-kolla | 00:07 | |
| *** schwicht has joined #openstack-kolla | 00:08 | |
| *** sindhu has quit IRC | 00:09 | |
| *** manheim_ has quit IRC | 00:11 | |
| *** stefan_bo has quit IRC | 00:11 | |
| *** eaguilar has joined #openstack-kolla | 00:13 | |
| *** tonanhngo has quit IRC | 00:17 | |
| *** kbaegis1 has joined #openstack-kolla | 00:23 | |
| *** schwicht has quit IRC | 00:23 | |
| openstackgerrit | sean mooney proposed openstack/kolla master: enable the kolla build system to use out of tree docker files https://review.openstack.org/466132 | 00:25 |
|---|---|---|
| *** tonanhngo has joined #openstack-kolla | 00:26 | |
| *** ducttape_ has joined #openstack-kolla | 00:26 | |
| *** kbaegis has quit IRC | 00:27 | |
| sean-k-mooney | inc0: what do you think of https://review.openstack.org/466132 its a very small change but its all that is need to support builing external docker files with kolla build | 00:28 |
| sean-k-mooney | inc0: instead of a direct replacement i could also model it as an addtional serch path for images so you could build the standard images and external at the same time but i dont know if that is overkill | 00:30 |
| *** mattmceuen has quit IRC | 00:30 | |
| *** tonanhngo has quit IRC | 00:30 | |
| *** ducttape_ has quit IRC | 00:31 | |
| sean-k-mooney | inc0: personally i would be fine with running kolla build twice once for openstack + once for external images not related to openstack. | 00:31 |
| *** salv-orlando has joined #openstack-kolla | 00:33 | |
| *** salv-orlando has quit IRC | 00:34 | |
| *** salv-orlando has joined #openstack-kolla | 00:34 | |
| *** hongbin has quit IRC | 00:46 | |
| *** cuongnv has joined #openstack-kolla | 00:57 | |
| *** salv-orlando has quit IRC | 00:59 | |
| *** tovin07_ has joined #openstack-kolla | 01:02 | |
| *** Slower has quit IRC | 01:04 | |
| *** dmellado has quit IRC | 01:04 | |
| *** dave-mccowan has quit IRC | 01:06 | |
| *** sgrasley has quit IRC | 01:06 | |
| *** sgrasley has joined #openstack-kolla | 01:08 | |
| *** srwilkers has quit IRC | 01:09 | |
| *** rhallisey has quit IRC | 01:10 | |
| kfox1111 | sbezverk: I know why its failing I think. | 01:14 |
| kfox1111 | theres a nova placement database now. | 01:14 |
| kfox1111 | with that fixed I think the v4 centos gate might be ready. | 01:14 |
| *** Slower has joined #openstack-kolla | 01:16 | |
| openstackgerrit | Dai Dang Van proposed openstack/kolla-ansible master: Mixing binary and source images (1) https://review.openstack.org/460436 | 01:18 |
| spsurya | morning all | 01:19 |
| *** qiliang28 has joined #openstack-kolla | 01:20 | |
| *** qiliang27 has quit IRC | 01:21 | |
| *** qiliang28 is now known as qiliang27 | 01:21 | |
| *** dmellado has joined #openstack-kolla | 01:22 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 01:22 |
| sdake | kfox1111 sbezverk can you fellas review this and tell me what is wrong with the log capture: https://review.openstack.org/#/c/464031/ | 01:25 |
| sdake | also why is helm-repo PV not present? | 01:25 |
| sdake | sup spsurya | 01:25 |
| *** gfidente has quit IRC | 01:26 | |
| *** daidv has joined #openstack-kolla | 01:27 | |
| inc0 | nice Sean | 01:28 |
| inc0 | sean-k-mooney: | 01:28 |
| inc0 | Jeffrey4l: around? | 01:28 |
| kfox1111 | sdake: the whole script is run with -x so I think sbezverk was mistaken. it should gather logs on failure. | 01:28 |
| sdake | kfox1111 i think it doesn't gather the PV/helm/etc objects | 01:29 |
| sdake | kfox1111 which I parsed his comment to mean he wanted to see | 01:29 |
| sdake | kfox1111 however there is no file in the logs dir | 01:29 |
| kfox1111 | oh. to see if those are left over? | 01:29 |
| sdake | if somehthing is left over, its helpful to know what :) | 01:30 |
| kfox1111 | yeah. I don't think we currently gather that. | 01:30 |
| sdake | kfox1111 https://review.openstack.org/#/c/464031/16/tests/bin/destroy_tests.sh | 01:30 |
| kfox1111 | would be good to add that to tests/bin/gate_capture_logs.sh | 01:30 |
| *** yee379 has quit IRC | 01:31 | |
| *** zul has joined #openstack-kolla | 01:31 | |
| *** yee379 has joined #openstack-kolla | 01:31 | |
| sdake | any idea why the job times out? | 01:32 |
| sdake | (ironic is the job) | 01:32 |
| *** hrw has quit IRC | 01:32 | |
| kfox1111 | once in a blue moon we see that. | 01:32 |
| kfox1111 | does it happen again on recheck? | 01:33 |
| *** hrw has joined #openstack-kolla | 01:34 | |
| sdake | unclear | 01:34 |
| sdake | i'll try again | 01:34 |
| sdake | helm-repo is the pv name? | 01:34 |
| sdake | it seems to not be found | 01:34 |
| kfox1111 | I think so. | 01:35 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 01:35 |
| sdake | kfox1111 ubuntu jobs broken atm? | 01:36 |
| kfox1111 | not sure. | 01:36 |
| kfox1111 | I know sbezverk put in a fix for the 1.6.3 thing, but not sure if it fixed just centos or both. | 01:37 |
| Jeffrey4l | inc0, yep. | 01:37 |
| kfox1111 | I was rather medicated at the time and havent checked back up on it yet. | 01:37 |
| openstackgerrit | shaofeng cheng proposed openstack/kolla-ansible master: Add VMware DataStore support to glance https://review.openstack.org/452176 | 01:37 |
| kfox1111 | 1.6.4 is immenent though. | 01:37 |
| inc0 | Jeffrey4l: so when we pip install Kolla it installs 4.0.0 | 01:38 |
| inc0 | shouldn't we be in 5.0.0 now? | 01:38 |
| sdake | inc0 pip is a pile of garb? | 01:38 |
| sdake | inc0 pip pips up the git tag | 01:38 |
| Jeffrey4l | inc0, we missed 5.0.0b1. it will be still 4.0.0 until 5.0.0b2 is tagged. | 01:38 |
| sdake | there is no 5.0.0 tag as pike is unreleased | 01:38 |
| inc0 | yeah that's what I was thinking | 01:39 |
| inc0 | too much chaos | 01:39 |
| openstackgerrit | pangliye proposed openstack/kolla master: Install docker_client and groupadd https://review.openstack.org/465880 | 01:40 |
| inc0 | I'm thinking whether or not we should maually set up default tags instead of autogenerating from pbr | 01:41 |
| *** zhubingbing has joined #openstack-kolla | 01:41 | |
| inc0 | as in, now default tag of images would be pike | 01:41 |
| Jeffrey4l | inc0, use pike as default? | 01:43 |
| Jeffrey4l | the issue is when tagged, we need change the "pike" to pbr value. | 01:43 |
| Jeffrey4l | and there is no way to ensure this. | 01:43 |
| Jeffrey4l | if we forget to change it when tagged, big issue will happen. | 01:44 |
| *** fguillot has quit IRC | 01:44 | |
| inc0 | on the other hand, if we actually push stuff to dockerhub, people won't be able to use images easily | 01:45 |
| inc0 | as default will never point to them | 01:45 |
| inc0 | in fact, today if they just use master code they'll pull ocata | 01:45 |
| sdake | Jeffrey4l is this ansible syntax correct: | 01:45 |
| sdake | - name: Delete Kolla PVs | 01:45 |
| sdake | command: kubectl delete {{ item }} | 01:45 |
| sdake | with_items: | 01:45 |
| sdake | - "mariadb" | 01:45 |
| sdake | - "glance" | 01:45 |
| sdake | - "rabbitmq" | 01:45 |
| sdake | - "helm-repo" | 01:45 |
| inc0 | and if we'd tag in time, they wouldn't be able to pull anything | 01:45 |
| inc0 | sdake: looks good | 01:45 |
| Jeffrey4l | about pike, it is master image stuff. we do not push/support master image right now. | 01:45 |
| Jeffrey4l | sdake, looks good. | 01:46 |
| inc0 | right now no, but I want to if TC agrees | 01:46 |
| sdake | does [{ item }} need to be in "? | 01:46 |
| inc0 | sdake: it does | 01:46 |
| inc0 | all of it should | 01:46 |
| inc0 | as in "kubectl delete {{item}}" | 01:46 |
| sdake | groan | 01:46 |
| Jeffrey4l | sdake, right. need. other with, ansible will complain something. | 01:46 |
| sdake | so all command operations need to be in full quotes? | 01:46 |
| Jeffrey4l | yes. | 01:47 |
| inc0 | sdake: most of stuff should be in quotes | 01:47 |
| *** stefan_bo has joined #openstack-kolla | 01:47 | |
| inc0 | most of arguments that is | 01:47 |
| Jeffrey4l | inc0, re tag name, if we push master image to docker hub, we should re-consider the tag name. | 01:48 |
| Jeffrey4l | fix it to pike is not good too. | 01:49 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 01:49 |
| inc0 | Jeffrey4l: why? | 01:49 |
| Jeffrey4l | cause issue when tagging. | 01:49 |
| Jeffrey4l | unless you are think use the same tag ( pike ) for all tags ( 5.0.1, 5.0.2) | 01:49 |
| inc0 | well, with pushes after tagging | 01:49 |
| inc0 | we can manually retag on pushing | 01:50 |
| inc0 | as in, default will never point to tagged image, but it's there | 01:50 |
| Jeffrey4l | we can assign the tag name when building for dockerhub or retag. | 01:50 |
| inc0 | also, depends if TC agrees to even pushing images | 01:50 |
| inc0 | Jeffrey4l: but it won't ever default | 01:50 |
| inc0 | reason I want these tags is | 01:50 |
| *** unicell has quit IRC | 01:50 | |
| inc0 | when you do pip install kolla-ansible | 01:50 |
| inc0 | you'll get set of images immediatly | 01:51 |
| inc0 | no matter if you want master or ocata or what | 01:51 |
| Jeffrey4l | then what will happen i run "pip install kolla-ansible==4.0.1"? | 01:51 |
| inc0 | kolla-ansible will be in version 4.0.1 but default images will be ocata | 01:51 |
| inc0 | well, that will change in pike | 01:52 |
| inc0 | thing is | 01:52 |
| Jeffrey4l | kolla-ansible 4.0.0 may not work with kolla image 4.0.1, there is no way to ensure this. | 01:52 |
| inc0 | I don't think it's good to have images frozen at 4.0.1 for example | 01:52 |
| inc0 | but having 4.0.1 image means you'll not get security updates on anything | 01:53 |
| inc0 | so at this ponit I think we shouldn't even push tags at all | 01:53 |
| Jeffrey4l | then we have to add a NOTE that: "please update the kolla-ansible into latest version" | 01:53 |
| Jeffrey4l | This is not *immediatly* either. | 01:54 |
| inc0 | well, do you follow all of CVEs of all packages in all containers? | 01:54 |
| inc0 | because I don't | 01:54 |
| Jeffrey4l | kolla can not ensure include all CVEs. unless you built it again. | 01:55 |
| sdake | speaking of 4.0.1 is anyone planning to push these to dockerhub | 01:55 |
| inc0 | so dhellman has point on this one, tagged fixed images are bad idea from security standpoint | 01:55 |
| Jeffrey4l | sdake, good point | 01:55 |
| Jeffrey4l | we can build 4.0.1 4.0.2 by using kolla 4.0.1 and 4.0.2 daily to ensure it include all CVEs. | 01:56 |
| Jeffrey4l | but only build ocata branch / latest tag image may not good. | 01:56 |
| Jeffrey4l | inc0, how about this " build 4.0.1 4.0.2 by using kolla 4.0.1 and 4.0.2 daily" | 01:59 |
| kfox1111 | we've used the same objects in kolla-kubernetes to deply 2.x through 4.x. | 02:00 |
| kfox1111 | is it really that hard guys? | 02:00 |
| Jeffrey4l | kfox1111, the point is: no body and no way to ensure that it works. | 02:01 |
| kfox1111 | with kolla-kubernetes we can. | 02:01 |
| inc0 | Jeffrey4l: it won't break within stable that often | 02:01 |
| kfox1111 | not sure why its hard in kolla-ansible, but don't know it very well. | 02:01 |
| inc0 | and then we can say "upgrade your kolla-ansible" | 02:01 |
| Jeffrey4l | inc0, yes. | 02:01 |
| inc0 | since all published images will have to pass deploy gates | 02:01 |
| kfox1111 | inc0: yeah. | 02:02 |
| inc0 | and deploy gates will use latest stable deploy code | 02:02 |
| kfox1111 | trunk is an enitirely different animal. if you are on the bleading edge, you should expect to bleed from time to time. | 02:02 |
| *** sgrasley has quit IRC | 02:02 | |
| kfox1111 | if using stable, it should be stable. | 02:02 |
| Jeffrey4l | inc0, the gate just test image 4.0.1 with kolla-ansible 4.0.1 , no gate to test image 4.0.0 with kolla-ansible 4.0.1 | 02:02 |
| inc0 | so what you'll do is pip install -U kolla-ansible and that's it | 02:02 |
| inc0 | we'll build set of gates dedicated for this pushes | 02:02 |
| *** sgrasley has joined #openstack-kolla | 02:03 | |
| Jeffrey4l | pip install -U kolla-ansible, no. this will install latest version. at least, please use "pip install -U kolla-ansible < 5" | 02:03 |
| inc0 | well, you know what I mean | 02:03 |
| Jeffrey4l | ;) | 02:03 |
| inc0 | but bottom line, whatever we propose | 02:03 |
| inc0 | we'll gate test it before publishing | 02:03 |
| kfox1111 | +1 | 02:03 |
| inc0 | in exactly the way we're proposing it | 02:03 |
| inc0 | so unless you skip the step, it's all going to work | 02:03 |
| inc0 | if new images fails the gate, they're not going to be pushed | 02:04 |
| inc0 | until we fix it | 02:04 |
| Jeffrey4l | are u meaning: \only push latest image ( i.e image:ocata) by using branch? which is tested in gate. | 02:04 |
| inc0 | so my idea goes like that | 02:06 |
| inc0 | we build special set of gates that will be run once a day | 02:06 |
| inc0 | only these gates will publish to tarballs.o.o | 02:06 |
| inc0 | these gates will have to pass (voting) all deploy gates | 02:07 |
| Jeffrey4l | branch tag is not that stable. for example no one use nova/stable/ocata direct in prod env. | 02:07 |
| inc0 | binary will be voting too | 02:07 |
| inc0 | so only if set of containers passes build gates and deploy gates for both ansible and k9s | 02:07 |
| inc0 | gets pushed to tarballs | 02:07 |
| inc0 | Jeffrey4l: lots of people do | 02:08 |
| inc0 | those who don't usually don't because it's hard to orchiestrate | 02:08 |
| sdake | Jeffrey4l tend to disagree slews of people use our images | 02:08 |
| sdake | whether that is a good idea or not, we leave to them :) | 02:08 |
| inc0 | so I'd make it easy to locally build stable tags | 02:09 |
| inc0 | but not host them due to security updates being missed | 02:09 |
| inc0 | and only host freshest of the fresh builds | 02:09 |
| inc0 | I think we need to reverse our approach to dockerhub publishing | 02:09 |
| inc0 | becaue on that I agree with dhellman, we can't ensure safety of tagged images | 02:10 |
| inc0 | and I'd rather have failed deploys that can be fixed by pip install -U than missed critical security update in one of packages | 02:10 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 02:12 |
| sdake | in other news, kolla-kubernetes requires 3.7gb of ram :) | 02:12 |
| Jeffrey4l | tbh, i never used dockerhub. and for any prod env, use self-build image is recommended. imo, dockerhub should be only used for newbies. | 02:12 |
| inc0 | Jeffrey4l: frankly? more I think about it more I disagree with you | 02:13 |
| sdake | Jeffrey4l you dont use dockerhub becausez your in china and dockerhub is slow as mud there | 02:13 |
| Jeffrey4l | inc0, aha, that the key point between us. | 02:13 |
| inc0 | Jeffrey4l: http://lists.openstack.org/pipermail/openstack-dev/2017-May/116906.html | 02:14 |
| inc0 | that's why:P) | 02:14 |
| inc0 | I mean I used to think that self-built is good | 02:14 |
| inc0 | but now I don't really | 02:14 |
| sdake | self built is the way to go | 02:14 |
| *** goldyfruit has joined #openstack-kolla | 02:14 | |
| inc0 | kfox1111 swayed me otherwise;) | 02:14 |
| Jeffrey4l | sdake, slow it one reason. and personally, i will not use it even if it is fast. | 02:14 |
| sdake | however, some nerds in kolla wanted images on dockerhub, so now we are stuck with em :) | 02:14 |
| inc0 | that's the thing, if we do good CI and automatic builds, self-building is quite literally worse | 02:15 |
| inc0 | simply because, if we do this with CI I envision | 02:15 |
| inc0 | containers we build will be deployed before publishing, so we have guarantee that it's deployable | 02:15 |
| inc0 | if you build yourself, that's not the case | 02:15 |
| Jeffrey4l | inc0, immutable is one issue. tag:ocata is change for time to time. it may cause different nova version on my whole openstack env. | 02:16 |
| inc0 | Jeffrey4l: we'll upload ocata and ocata-44 | 02:16 |
| *** zul has quit IRC | 02:16 | |
| inc0 | where 44 is autoincremental | 02:16 |
| inc0 | so if you force todays 44 it won't upgrade on it's own any more | 02:17 |
| Jeffrey4l | ok.. | 02:17 |
| inc0 | until you bump incremental | 02:17 |
| inc0 | you won't be able to download 44 again as we won't store it, but you'll have it locally and it won't get overriden | 02:17 |
| inc0 | archiving of images is up to you | 02:17 |
| inc0 | dims: btw ^ | 02:18 |
| inc0 | good discussion about our ideas how that might work in secure manner | 02:18 |
| inc0 | also that's argument why we shouldn't follow release team policy on that strictly:) actually if we follow releases, that's where security becomes bad | 02:19 |
| sdake | inc0 what it really comes down to is nobody wants to take responsibility for the release of the images except the kolla team | 02:21 |
| sdake | inc0 and the release team is the "gatekeeper" of releasing artifacts | 02:21 |
| Jeffrey4l | what i expecting is: 1. build kolla image 4.0.1 and push it into docker hub daily 2. build kolla image 4.0.2 and push it to docker hub daily. | 02:21 |
| inc0 | Jeffrey4l: that's gonna be hell to maintain | 02:21 |
| sdake | inc0 therefore I believe its the release team's position they own the greenlight on any activity you ahve proposed | 02:21 |
| inc0 | because if 4.0.1 breaks due to bug | 02:21 |
| inc0 | it won't ever be fixed, 4.0.1 is not updated, security is compromised | 02:22 |
| Jeffrey4l | we can also to add :<branch> tag to point to latest tag ( i.e 4.0.2 ) | 02:22 |
| Jeffrey4l | nova 15.0.1 is never be updated. tag is never be changed. | 02:22 |
| kfox1111 | that but with release numbers. | 02:22 |
| kfox1111 | 15.0.1 wont ever be updated. but deps might, ensuring 15.0.1-2 | 02:23 |
| Jeffrey4l | in my way: kolla image 4.0.1 is mutable for ( nova version / neutron version and all it's running env ) | 02:23 |
| kfox1111 | that doesnt work well with k8s. | 02:24 |
| Jeffrey4l | hrm, all this talk is talk is based on source type. binary is hard.. | 02:24 |
| kfox1111 | I really want to see atomic upgrades. | 02:24 |
| kfox1111 | binary's the easy one I think. it won't change frequently. | 02:24 |
| kfox1111 | fingerprint current version. build new one. fingerprint it. only if different, increment release number and push. | 02:25 |
| Jeffrey4l | is that OK to use diffferent openstack ( nova / neutron ) y/z stream on you env? | 02:25 |
| kfox1111 | to some extent, yeah. | 02:25 |
| kfox1111 | live upgrades need that sort of thing. | 02:25 |
| kfox1111 | not that openstack supports too much of that yet. | 02:25 |
| kfox1111 | but it is actively being worked on. | 02:26 |
| Jeffrey4l | i prefer to save all images for the future extend usage. | 02:26 |
| kfox1111 | I don't disagree. but thats being a harder sale then the current only most recent. | 02:26 |
| kfox1111 | and thats a hard sale. :/ | 02:26 |
| Jeffrey4l | anyway, this is still my opinion: and for any prod env, use self-build image is recommended. dockerhub should be only used for newbies. | 02:27 |
| kfox1111 | +1 | 02:28 |
| kfox1111 | alwasys best to self-build for production. | 02:28 |
| inc0 | Jeffrey4l: but nova doesn't ship with all the deps | 02:28 |
| kfox1111 | for proof of concept/dev/testing though, should be a much lower bar. | 02:28 |
| inc0 | we ship containers witl all the deps | 02:28 |
| inc0 | ok, that's different conversation, but we all agree that there is benefit from having :ocata | 02:29 |
| inc0 | that is not horribly hole'y, fresh and updated | 02:29 |
| Jeffrey4l | kfox1111, yes. include these purpose. | 02:29 |
| inc0 | and easy to download | 02:29 |
| kfox1111 | inc0: yeah. | 02:29 |
| Jeffrey4l | on the other hand, it is very hard to download image from docker hub from China ;( | 02:30 |
| inc0 | although I'll be adamant that imho dockerhub the way we will do it will be better than self built:) | 02:30 |
| inc0 | Jeffrey4l: does China has equivalent? | 02:30 |
| Jeffrey4l | and we ( company ) provide this https://hub.trystack.cn/ lol | 02:30 |
| inc0 | we can push there too | 02:30 |
| kfox1111 | Jeffrey4l: yeah. but china's great firewall is not reason enough to not do it. | 02:30 |
| Jeffrey4l | inc0, no right now. not large/free equivalent at least. | 02:30 |
| inc0 | Jeffrey4l: maybe 99cloud would want to host it for kolla?;) | 02:31 |
| inc0 | then we'll just document that for China, use this address | 02:31 |
| *** goldyfruit has quit IRC | 02:31 | |
| Jeffrey4l | kfox1111, seem docker hub's image is on CDN. and the CDN provider is blocked. | 02:31 |
| *** eanylin has quit IRC | 02:31 | |
| Jeffrey4l | inc0, good idea. | 02:31 |
| kfox1111 | even still. being able to download some prebuild container images should be much easier on the great firewall, | 02:31 |
| kfox1111 | then building from source which pulls from all over the place. :/ | 02:31 |
| inc0 | then for China it will be even better | 02:32 |
| inc0 | since building containers behind great firewall is painful too | 02:32 |
| kfox1111 | right. | 02:32 |
| Jeffrey4l | we(99cloud) also provide a ISO to install kolla right now too. http://openstack.99cloud.com.cn/ | 02:33 |
| inc0 | there you go, registry.99cloud.com.cn:) | 02:33 |
| kfox1111 | that would work. :) | 02:33 |
| inc0 | and a page in docs where are mirrors are | 02:33 |
| kfox1111 | I've always said we hsould push to multiple repo's. | 02:33 |
| Jeffrey4l | https://hub.trystack.cn/ should be OK. | 02:33 |
| inc0 | dockerhub, quay.co, 99cloud | 02:33 |
| kfox1111 | +1 | 02:34 |
| inc0 | maybe docker.openstack.org one day | 02:34 |
| Jeffrey4l | inc0, ++1 | 02:34 |
| inc0 | anyway, we still need to wait for TC resolution | 02:34 |
| kfox1111 | inc0: we must wait for the TC resolution to get the cronjob in place. | 02:35 |
| kfox1111 | all the rest we can develop anyway. | 02:35 |
| inc0 | that's what I'm doing | 02:35 |
| kfox1111 | all the rest is still needed. | 02:35 |
| kfox1111 | yeah. me too. | 02:35 |
| inc0 | I started with simple tool | 02:35 |
| inc0 | and now I'm working on gates | 02:35 |
| inc0 | since as we said, we don't want to take chances with these images | 02:36 |
| kfox1111 | inc0: maybe we shoudl come up with a final workflow diagram of all the pieces. | 02:36 |
| kfox1111 | that may make folks feel better. | 02:36 |
| inc0 | probably | 02:36 |
| inc0 | gliffy or something | 02:36 |
| kfox1111 | I gotta head out now. have a good evening. | 02:37 |
| inc0 | you too | 02:37 |
| kfox1111 | yeah. maybe we can talk more about that tomorrow. | 02:37 |
| inc0 | btw Jeffrey4l take a look at that https://review.openstack.org/#/c/466007/15 | 02:37 |
| inc0 | I want to refactor our gates quite a bit | 02:37 |
| Jeffrey4l | got. | 02:38 |
| inc0 | for example drop setup_node | 02:38 |
| inc0 | and use kolla-ansible bootstrap-servers instead | 02:38 |
| inc0 | I already have piece that will prep ansible,inventory | 02:38 |
| inc0 | and I'm close with basic globals.yml | 02:38 |
| inc0 | after that we'll be able to add scenerios super easily, just add different globals.yml template | 02:39 |
| Jeffrey4l | thats cool. | 02:39 |
| inc0 | we'll also be able to effectively remove aio gates | 02:39 |
| inc0 | as aio will be same as multinode | 02:39 |
| inc0 | just with less nodes | 02:39 |
| inc0 | :) | 02:40 |
| inc0 | so if we add for example ceph | 02:40 |
| inc0 | it will automatically be tested aio and multinode | 02:40 |
| inc0 | all within project_config | 02:40 |
| inc0 | much like kolla-k8s has today, I like this approach | 02:40 |
| Jeffrey4l | that will be better. | 02:41 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 02:44 |
| *** zhubingbing has quit IRC | 02:46 | |
| *** eanylin has joined #openstack-kolla | 02:46 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla-ansible master: Removing ovs bridge before destroying container https://review.openstack.org/464998 | 02:46 |
| *** qiliang28 has joined #openstack-kolla | 02:48 | |
| *** xinliang has quit IRC | 02:49 | |
| openstackgerrit | Merged openstack/kolla stable/newton: Bump OpenStack service https://review.openstack.org/465849 | 02:50 |
| *** qiliang27 has quit IRC | 02:51 | |
| *** qiliang28 is now known as qiliang27 | 02:51 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 02:53 |
| *** xinliang has joined #openstack-kolla | 03:01 | |
| *** zhubingbing has joined #openstack-kolla | 03:03 | |
| sdake | kfox1111 i've sorted out what causes http://logs.openstack.org/31/464031/18/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/f3b113c/console.html#_2017-05-19_02_37_58_150789 | 03:05 |
| sdake | kfox1111 the cloud is destroyed - hence libvirt is no longer present | 03:05 |
| sdake | suggestions? | 03:05 |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Delete unnecessary code and insert a blank lines. https://review.openstack.org/465919 | 03:08 |
| sdake | sbezverk ^6 | 03:11 |
| *** cosy294 has joined #openstack-kolla | 03:11 | |
| cosy294 | hello | 03:13 |
| sbezverk | sdake: what are you truing to do with libvirt? | 03:14 |
| sdake | cpature_logs does a virsh dump | 03:14 |
| sdake | capture logs seems to run after desetroy | 03:15 |
| sdake | sbezverk as you know capture logs does a whoe lbunch of stuff, it all happens after destroy it appears | 03:16 |
| cosy294 | I install kolla-kubernetes with AIO environments. | 03:16 |
| cosy294 | now, i want to install multi node | 03:16 |
| cosy294 | help me | 03:16 |
| sbezverk | sdake: I think there is a check for ironic gate job.. Only ironic gate job has bits to communicate with virsh.. | 03:16 |
| sdake | sbezverk even if i don't run destroy ony in the ironic job (which is where it is run presently) capture_logs still runs after destroy | 03:17 |
| sbezverk | right right, libvirt runs in the container and you kill it | 03:17 |
| sbezverk | so virsh command stop working.. | 03:17 |
| sdake | hence capture_logs wont capture any state, as it has been erased | 03:18 |
| sbezverk | but log capure should be run only if there is a failure | 03:18 |
| sbezverk | you do not call it unless something gets broken | 03:18 |
| sdake | right - so we can't have jobs timign out at 1.05 hours just because capture_logs is ordered incorrectly | 03:19 |
| sbezverk | we can move virsh command in log capture to be the last ones | 03:19 |
| sbezverk | and make is || true | 03:19 |
| sdake | that would be great if the gate didn't block | 03:19 |
| sbezverk | but regardless if your script worked then capture log should not be called | 03:20 |
| sdake | do you know why the gate blocks?? | 03:20 |
| sdake | sbezverk i hear you saying my code is broken and I agree | 03:20 |
| sbezverk | it hapens on rare ocasions | 03:20 |
| sdake | sbezverk it should not cause a trap, however ,it does, and we don't want flakey gates when something goes wrong | 03:20 |
| sdake | its 100% with my patchset ;) | 03:21 |
| sdake | good chance to debug the problem :) | 03:21 |
| sbezverk | sdake: why you always start blaming something else, before makeing sure that you do right thing?? | 03:21 |
| sdake | sbezverk its pretty straightforward, I could make my code work correctly | 03:21 |
| sdake | sbezverk and then capture logs wouldn't run | 03:21 |
| sdake | and then the day somethign breaks and capture logs runs, the gate craters | 03:21 |
| sdake | and everyone goes "oh thats a random failure" | 03:21 |
| sdake | no failures are random | 03:21 |
| sbezverk | sdake: perfect, you are on the right track to nail it then!! | 03:22 |
| sdake | sadly I don't know shell well enoug hto debug this problem | 03:22 |
| sbezverk | I mean to prove your point, right?? | 03:22 |
| sdake | i think back to back traps are occuring | 03:22 |
| sdake | which creates a trap loop | 03:22 |
| sdake | i could use your help to debug | 03:23 |
| sbezverk | can you run in ansible commands and use its output as an input to next action? | 03:23 |
| sdake | yup | 03:24 |
| sbezverk | example the issue here is with PVs | 03:24 |
| sdake | i understand there is a problem with the pvs :) | 03:24 |
| sbezverk | what then you want to debug? | 03:24 |
| sdake | the gate lockup | 03:24 |
| sdake | that happens outside of my code | 03:24 |
| sdake | that happens in capture logs | 03:25 |
| sbezverk | it get stuck trying to communicate with non-existing libvirt | 03:25 |
| sbezverk | I mean it is not our bits | 03:25 |
| sbezverk | we cannot change that | 03:25 |
| *** zhubingbing has quit IRC | 03:25 | |
| sbezverk | all we can do is work around it | 03:25 |
| sdake | here is my theory - destroy raises trap, triggers capture logs | 03:26 |
| sdake | libvirt raises trap - bash implodes | 03:26 |
| sbezverk | just move virsh commands to the end of log capturing script | 03:26 |
| sbezverk | and add || true to each of them | 03:27 |
| sbezverk | in this case even if it fails it will not prevent further processing.. | 03:27 |
| sdake | sbezverk they are at the end already | 03:27 |
| sdake | there must surely be a way to ignore traps inside a script? | 03:27 |
| sdake | https://github.com/openstack/kolla-kubernetes/blob/master/tools/setup_gate.sh#L10 | 03:28 |
| sdake | that needs to be disabled here: https://github.com/openstack/kolla-kubernetes/blob/master/tests/bin/gate_capture_logs.sh#L3 | 03:30 |
| sdake | this scenario happens all the time in signal handling in the c language | 03:31 |
| *** stefan_bo has quit IRC | 03:32 | |
| sdake | i guess i'll try this trap - signal | 03:33 |
| sbezverk | sdake: I think last part which is ironic specific just needs to be wrapped into if | 03:33 |
| sdake | the problem exists independent of my work :) | 03:33 |
| sdake | this is why you see it from time to time in the gate jobs | 03:33 |
| sbezverk | so it is not executed if failure happen in your destroy script | 03:33 |
| sbezverk | sdake: if you can consistently reproduce it, then we can debug, other than that I see failure becuase we try to access already non-existing services.. | 03:35 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 03:35 |
| sdake | sbezverk its reproducible | 03:35 |
| sdake | 100% with this code | 03:35 |
| sdake | anyway I just tested my theory if the internet is correct here: https://bash.cyberciti.biz/guide/How_to_clear_trap | 03:35 |
| sbezverk | sdake: hm, is it 100% reproducable why we have green gate all the time ??? | 03:36 |
| sdake | sbezverk 100% with my code (which raises a trap in gate_setup.sh) | 03:36 |
| sbezverk | and why do you raise trap? | 03:37 |
| sdake | because ansible raises a signal | 03:37 |
| sdake | and that causes bash to receive a signal | 03:37 |
| sdake | and trpa catches that signal | 03:37 |
| sdake | and triggers signal handling | 03:37 |
| sdake | signal handling in bash = evil :) | 03:37 |
| sbezverk | we had gate log capture script gets called on every failure | 03:38 |
| sbezverk | no evil, everything was working fine until you declaried it to be evil.. that does not compute will | 03:38 |
| sdake | right and when the second failure happens, the gate deadlocks | 03:38 |
| sdake | we will see if the gate lbocks this time. :) | 03:39 |
| sdake | sbezverk if you raise a signal in a signal handler in unix, bad bad things happen (TM) | 03:39 |
| sdake | its not like || true magically ignores signals :) | 03:39 |
| sbezverk | it is not a C it is a script called from another script, it is done many time | 03:40 |
| sdake | sbezverk i'll be you 1 american dollar | 03:40 |
| sdake | that if you cause some kind of error in capture_gate_logs, th gat ewill lock up | 03:40 |
| sdake | be/bet :) | 03:40 |
| sbezverk | there is OS signal generated like sigterm or sigabort | 03:40 |
| sbezverk | it looks up because virsh cannot get in touch with libvirt | 03:41 |
| sdake | that is a second possible theory althoug hthe gate logs don't indicate that | 03:41 |
| sbezverk | we have bunch of commands inluding ipmi commands trying and then re-trying to communicate with libvirt | 03:41 |
| sdake | well we will know in approximately 30 minutes :) | 03:42 |
| sbezverk | it clearly indicates that: sdake | 03:42 |
| sbezverk | http://logs.openstack.org/31/464031/18/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/f3b113c/console.html#_2017-05-19_02_37_58_155818 | 03:42 |
| sbezverk | it is damn last message in the log | 03:43 |
| sdake | sbezverk look at the gate delta in times | 03:43 |
| sdake | 20 minutes | 03:43 |
| sdake | between that and the gate timeout | 03:43 |
| sdake | that operation completely locks up | 03:43 |
| sdake | whereas the job exited and returned... | 03:43 |
| sdake | rather command | 03:43 |
| sbezverk | because there are multiple commands and we do not know how virtual bmc reacts on this connection | 03:44 |
| sbezverk | I meant on this condition.. | 03:44 |
| sdake | 30 minutes | 03:44 |
| sdake | gate will either block or not | 03:44 |
| sdake | if it doesn't block, my theory is correct :) | 03:45 |
| sbezverk | ok, have fun, I am off.. | 03:45 |
| sdake | later | 03:45 |
| sdake | shame you didn't take me up on my bet :) | 03:45 |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Test: insert a blank https://review.openstack.org/466169 | 03:50 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Demonstrate trap failure https://review.openstack.org/466170 | 03:51 |
| *** unicell has joined #openstack-kolla | 03:51 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Demonstrate trap failure https://review.openstack.org/466170 | 03:52 |
| sdake | sbezverk that review should fail in about 1-2 minutes :) | 03:52 |
| sdake | sbezverk or alternately block for 1.05 hours | 03:52 |
| *** zhubingbing has joined #openstack-kolla | 03:56 | |
| sdake | Jeffrey4l how does one make ansible ignore errors | 03:59 |
| *** stefan_bo has joined #openstack-kolla | 04:05 | |
| *** iniazi has joined #openstack-kolla | 04:07 | |
| *** imcsk8 has quit IRC | 04:12 | |
| *** vhosakot has joined #openstack-kolla | 04:12 | |
| openstackgerrit | Anthony Lin proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 04:24 |
| Seyong | hello sdake, just greeting | 04:24 |
| *** eaguilar has quit IRC | 04:29 | |
| inc0 | sdake: add ignore_errors: true | 04:30 |
| inc0 | to task | 04:30 |
| *** stefan_bo has quit IRC | 04:32 | |
| *** yuanying has quit IRC | 04:32 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 04:35 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 04:43 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 04:44 |
| *** yuanying has joined #openstack-kolla | 04:48 | |
| *** skramaja has joined #openstack-kolla | 04:48 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 04:50 |
| *** salv-orlando has joined #openstack-kolla | 04:50 | |
| *** cosy294 has quit IRC | 04:54 | |
| inc0 | Jeffrey4l: any ideas why docker failed to install? http://logs.openstack.org/07/466007/21/experimental/gate-kolla-ansible-dsvm-deploy-multinode-ubuntu-source-ubuntu-xenial-2-node-nv/aca66de/console.html | 04:55 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 05:06 |
| *** salv-orlando has quit IRC | 05:07 | |
| *** cuongnv has quit IRC | 05:15 | |
| *** stefan_bo has joined #openstack-kolla | 05:15 | |
| *** vhosakot has quit IRC | 05:15 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 05:16 |
| *** cuongnv has joined #openstack-kolla | 05:19 | |
| Jeffrey4l | sdake, what's kinda error? | 05:24 |
| Jeffrey4l | inc0, not sure, environments may be. try to use "rpm -ql docker-engine" to list all file? | 05:26 |
| *** caowei has joined #openstack-kolla | 05:33 | |
| *** unicell has joined #openstack-kolla | 05:39 | |
| *** jaosorior_away has quit IRC | 05:44 | |
| *** stefan_bo has quit IRC | 05:44 | |
| *** mkoderer has quit IRC | 05:47 | |
| *** janki has joined #openstack-kolla | 05:51 | |
| *** jbadiapa has quit IRC | 05:53 | |
| *** caowei has quit IRC | 05:53 | |
| *** stefan_bo has joined #openstack-kolla | 05:59 | |
| *** aagate has quit IRC | 06:00 | |
| *** yuanying has quit IRC | 06:06 | |
| *** jbadiapa has joined #openstack-kolla | 06:08 | |
| *** yuanying has joined #openstack-kolla | 06:08 | |
| *** salv-orlando has joined #openstack-kolla | 06:08 | |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Insert a blank line. https://review.openstack.org/466195 | 06:10 |
| *** salv-orlando has quit IRC | 06:13 | |
| openstackgerrit | Surya Prakash (spsurya) proposed openstack/kolla master: Fixing the running of keystone https://review.openstack.org/464953 | 06:16 |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Delete useless code https://review.openstack.org/466199 | 06:21 |
| *** sgrasley1 has joined #openstack-kolla | 06:27 | |
| *** sgrasley has quit IRC | 06:27 | |
| *** Serlex has joined #openstack-kolla | 06:32 | |
| *** chas has joined #openstack-kolla | 06:38 | |
| *** pcaruana has joined #openstack-kolla | 06:39 | |
| *** korzen has joined #openstack-kolla | 06:42 | |
| *** athomas has joined #openstack-kolla | 06:42 | |
| *** pcaruana is now known as pcaruana|tve| | 06:44 | |
| *** pcaruana|tve| is now known as pcaruana|rtve| | 06:44 | |
| openstackgerrit | serlex proposed openstack/kolla-ansible master: Add index pattern to kibana once https://review.openstack.org/466101 | 06:46 |
| *** matrohon has joined #openstack-kolla | 06:58 | |
| openstackgerrit | zhuangzejian proposed openstack/kolla-ansible master: Delete useless code https://review.openstack.org/466199 | 07:03 |
| *** Serlex has quit IRC | 07:04 | |
| *** salv-orlando has joined #openstack-kolla | 07:09 | |
| openstackgerrit | Mathieu Rohon proposed openstack/kolla master: Add skydive service https://review.openstack.org/459995 | 07:11 |
| *** salv-orlando has quit IRC | 07:13 | |
| *** shardy has quit IRC | 07:16 | |
| *** shardy has joined #openstack-kolla | 07:17 | |
| *** jmccarthy has joined #openstack-kolla | 07:21 | |
| *** unicell has quit IRC | 07:22 | |
| *** unicell has joined #openstack-kolla | 07:24 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla stable/ocata: Catch exception during calling os.chmod in set_configs.py https://review.openstack.org/466215 | 07:29 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla-ansible stable/ocata: Remove verbose log in memcached container https://review.openstack.org/466216 | 07:30 |
| *** chas has quit IRC | 07:31 | |
| *** egonzalez has joined #openstack-kolla | 07:32 | |
| *** gfidente has joined #openstack-kolla | 07:38 | |
| *** gfidente has quit IRC | 07:38 | |
| *** gfidente has joined #openstack-kolla | 07:38 | |
| *** chas has joined #openstack-kolla | 07:41 | |
| *** blallau has joined #openstack-kolla | 07:43 | |
| *** chas has quit IRC | 07:45 | |
| *** chas has joined #openstack-kolla | 07:47 | |
| *** Serlex has joined #openstack-kolla | 07:49 | |
| *** manheim_ has joined #openstack-kolla | 07:51 | |
| *** manheim_ has quit IRC | 07:51 | |
| *** manheim_ has joined #openstack-kolla | 07:52 | |
| *** chas has quit IRC | 08:00 | |
| *** chas has joined #openstack-kolla | 08:00 | |
| openstackgerrit | Dan Ardelean proposed openstack/kolla-ansible master: Add Hyper-V role https://review.openstack.org/455684 | 08:04 |
| *** salv-orlando has joined #openstack-kolla | 08:10 | |
| *** salv-orlando has quit IRC | 08:16 | |
| *** mgoddard__ has joined #openstack-kolla | 08:20 | |
| *** chas has quit IRC | 08:23 | |
| *** chas has joined #openstack-kolla | 08:23 | |
| *** powerd has joined #openstack-kolla | 08:27 | |
| *** jaosorior has joined #openstack-kolla | 08:29 | |
| *** lpetrut has joined #openstack-kolla | 08:39 | |
| *** stefan_bo has quit IRC | 08:49 | |
| *** salv-orlando has joined #openstack-kolla | 09:00 | |
| *** xinliang has quit IRC | 09:00 | |
| *** caowei has joined #openstack-kolla | 09:01 | |
| *** c06 has joined #openstack-kolla | 09:05 | |
| *** awebber has joined #openstack-kolla | 09:05 | |
| *** c06 has left #openstack-kolla | 09:05 | |
| *** c06 has joined #openstack-kolla | 09:05 | |
| *** c06 has left #openstack-kolla | 09:05 | |
| *** salv-orlando has quit IRC | 09:06 | |
| *** ArminderSingh- has joined #openstack-kolla | 09:06 | |
| *** chas has quit IRC | 09:09 | |
| *** xinliang has joined #openstack-kolla | 09:11 | |
| *** caowei has quit IRC | 09:14 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Implement blazar Docker images https://review.openstack.org/454745 | 09:16 |
| *** kolla-slack1 has joined #openstack-kolla | 09:17 | |
| *** kolla-slack has quit IRC | 09:17 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Implement blazar Docker images https://review.openstack.org/454745 | 09:20 |
| *** caowei has joined #openstack-kolla | 09:23 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: Allow nova plugins install, add blazar plugin https://review.openstack.org/454743 | 09:23 |
| kolla-slack1 | <archifleks> hi | 09:25 |
| *** jascott1 has joined #openstack-kolla | 09:26 | |
| hrw | morning | 09:30 |
| Serlex | top of the morning | 09:34 |
| openstackgerrit | Paul Bourke (pbourke) proposed openstack/kolla-ansible master: Kibana post_config checks only need to run locally https://review.openstack.org/463798 | 09:39 |
| *** sambetts|afk is now known as sambetts | 09:46 | |
| *** jascott1 has quit IRC | 09:48 | |
| pbourke | Serlex: did anyone say that in boston? | 09:50 |
| *** stefan_bo has joined #openstack-kolla | 09:51 | |
| *** openstack has joined #openstack-kolla | 10:00 | |
| *** salv-orlando has joined #openstack-kolla | 10:02 | |
| Serlex | Unfortunately no pbourke, I clearly didn't hit the rights part of the city :) | 10:02 |
| *** salv-orlando has quit IRC | 10:06 | |
| pbourke | well that's the thing, im not sure anyone actually says it :p | 10:16 |
| *** stefan_bo has quit IRC | 10:20 | |
| *** caowei has quit IRC | 10:28 | |
| *** zul has joined #openstack-kolla | 10:31 | |
| *** mkoderer has joined #openstack-kolla | 10:31 | |
| *** skramaja has quit IRC | 10:35 | |
| *** shardy has quit IRC | 10:36 | |
| *** cuongnv has quit IRC | 10:37 | |
| *** shardy has joined #openstack-kolla | 10:38 | |
| *** daidv has quit IRC | 10:47 | |
| *** blallau has quit IRC | 11:01 | |
| *** YuYangWang has joined #openstack-kolla | 11:02 | |
| *** salv-orlando has joined #openstack-kolla | 11:02 | |
| *** tovin07_ has quit IRC | 11:03 | |
| *** imcsk8 has joined #openstack-kolla | 11:05 | |
| manheim_ | guys, do you know any good guide for k8s networking? i've failed to find a good one, i have hard time comprehending k8s networking | 11:06 |
| *** salv-orlando has quit IRC | 11:07 | |
| *** chas has joined #openstack-kolla | 11:09 | |
| *** feisky has quit IRC | 11:10 | |
| *** rhallisey has joined #openstack-kolla | 11:12 | |
| *** shardy has quit IRC | 11:12 | |
| *** chas has quit IRC | 11:13 | |
| *** chas has joined #openstack-kolla | 11:15 | |
| *** zul has quit IRC | 11:19 | |
| *** shardy has joined #openstack-kolla | 11:19 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: WIP: Implement blazar ansible role https://review.openstack.org/454752 | 11:19 |
| *** Serlex has quit IRC | 11:22 | |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Implement blazar ansible role https://review.openstack.org/454752 | 11:24 |
| *** blallau has joined #openstack-kolla | 11:27 | |
| *** schwicht has joined #openstack-kolla | 11:29 | |
| *** Serlex has joined #openstack-kolla | 11:30 | |
| *** schwicht has quit IRC | 11:35 | |
| *** schwicht has joined #openstack-kolla | 11:38 | |
| *** YuYangWang has quit IRC | 11:42 | |
| *** zul has joined #openstack-kolla | 11:54 | |
| *** zhubingbing has quit IRC | 11:55 | |
| *** zhubingbing has joined #openstack-kolla | 11:55 | |
| *** srwilkers has joined #openstack-kolla | 11:59 | |
| *** Serlex has quit IRC | 12:00 | |
| *** salv-orlando has joined #openstack-kolla | 12:03 | |
| *** schwicht has quit IRC | 12:05 | |
| *** janki has quit IRC | 12:06 | |
| *** janki has joined #openstack-kolla | 12:06 | |
| *** tloeb has joined #openstack-kolla | 12:08 | |
| *** salv-orlando has quit IRC | 12:08 | |
| *** tloeb has quit IRC | 12:10 | |
| *** egonzalez has quit IRC | 12:10 | |
| openstackgerrit | Juan Antonio Osorio Robles proposed openstack/kolla master: Add images for novajoin services https://review.openstack.org/466268 | 12:13 |
| *** schwicht has joined #openstack-kolla | 12:16 | |
| *** jistr is now known as jistr|mtg | 12:16 | |
| *** sean-k-mooney has quit IRC | 12:18 | |
| *** jrist has joined #openstack-kolla | 12:23 | |
| *** dave-mccowan has joined #openstack-kolla | 12:26 | |
| *** shardy has quit IRC | 12:27 | |
| *** shardy has joined #openstack-kolla | 12:27 | |
| openstackgerrit | Serguei Bezverkhi proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 12:27 |
| *** eanylin has quit IRC | 12:30 | |
| *** egonzalez has joined #openstack-kolla | 12:31 | |
| *** rwellum has joined #openstack-kolla | 12:34 | |
| *** fguillot has joined #openstack-kolla | 12:36 | |
| *** jchhatbar has joined #openstack-kolla | 12:42 | |
| *** eaguilar has joined #openstack-kolla | 12:43 | |
| *** janki has quit IRC | 12:45 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 12:45 |
| *** jchhatbar has quit IRC | 12:48 | |
| *** eaguilar has quit IRC | 12:49 | |
| *** goldyfruit has joined #openstack-kolla | 12:52 | |
| *** zhubingbing has quit IRC | 12:52 | |
| *** janki has joined #openstack-kolla | 12:56 | |
| *** eaguilar has joined #openstack-kolla | 12:57 | |
| *** salv-orlando has joined #openstack-kolla | 12:59 | |
| *** jistr|mtg is now known as jistr | 12:59 | |
| *** schwicht has quit IRC | 13:03 | |
| *** janki has quit IRC | 13:04 | |
| *** goldyfruit has quit IRC | 13:05 | |
| *** zhubingbing_ has joined #openstack-kolla | 13:07 | |
| *** schwicht has joined #openstack-kolla | 13:08 | |
| *** shardy has quit IRC | 13:12 | |
| *** shardy has joined #openstack-kolla | 13:13 | |
| *** salv-orlando has quit IRC | 13:16 | |
| *** salv-orlando has joined #openstack-kolla | 13:16 | |
| *** schwicht has quit IRC | 13:19 | |
| *** salv-orlando has quit IRC | 13:21 | |
| *** zul has quit IRC | 13:27 | |
| *** schwicht has joined #openstack-kolla | 13:30 | |
| *** krtaylor has joined #openstack-kolla | 13:31 | |
| *** mattmceuen has joined #openstack-kolla | 13:37 | |
| *** duonghq has joined #openstack-kolla | 13:46 | |
| *** salv-orlando has joined #openstack-kolla | 13:54 | |
| *** Serlex has joined #openstack-kolla | 13:54 | |
| *** blallau has quit IRC | 13:54 | |
| *** srwilkers has quit IRC | 13:55 | |
| *** aagate has joined #openstack-kolla | 13:55 | |
| *** goldyfruit has joined #openstack-kolla | 13:58 | |
| *** eanylin has joined #openstack-kolla | 13:59 | |
| *** iniazi has quit IRC | 14:01 | |
| *** amurali has joined #openstack-kolla | 14:05 | |
| amurali | I'm trying to build neutron-server image from master branch with networking-6wind plugin...but, I ended up with error "Could not find a version that satisfies the requirement nova (from networking-6wind==0.0.1.dev49) (from versions: )" | 14:06 |
| amurali | Anybody has any idea about this ? | 14:06 |
| *** shardy has quit IRC | 14:08 | |
| *** shardy has joined #openstack-kolla | 14:08 | |
| *** zhubingbing_ has quit IRC | 14:08 | |
| *** schwicht has quit IRC | 14:09 | |
| *** duonghq has quit IRC | 14:09 | |
| *** korzen has quit IRC | 14:10 | |
| egonzalez | amurali cannot find a pip package with that name | 14:12 |
| *** zhubingbing has joined #openstack-kolla | 14:16 | |
| egonzalez | you will need to install from github or tarball, follow this guide https://github.com/openstack/kolla/blob/master/doc/image-building.rst#plugin-functionality | 14:17 |
| *** ducttape_ has joined #openstack-kolla | 14:27 | |
| *** srwilkers has joined #openstack-kolla | 14:34 | |
| *** eaguilar has quit IRC | 14:34 | |
| *** ducttape_ has quit IRC | 14:35 | |
| *** iniazi has joined #openstack-kolla | 14:36 | |
| *** lpetrut has quit IRC | 14:36 | |
| *** ducttape_ has joined #openstack-kolla | 14:36 | |
| *** schwicht has joined #openstack-kolla | 14:38 | |
| *** manheim_ has quit IRC | 14:39 | |
| openstackgerrit | Bernard Maltais proposed openstack/kolla-kubernetes master: Script to Deploy Kolla Kubernetes AIO https://review.openstack.org/464312 | 14:39 |
| sdake | morning folks | 14:42 |
| *** sgrasley has joined #openstack-kolla | 14:44 | |
| bmaltais | Morning | 14:44 |
| *** emccormick has joined #openstack-kolla | 14:44 | |
| bmaltais | eanylin: The script is failing when running with sudo | 14:45 |
| *** sgrasley1 has quit IRC | 14:45 | |
| bmaltais | eanylin: Slowly fixing it but I am stuck at the step "Generate the Kubernetes secrets and register them with Kubernetes" | 14:46 |
| bmaltais | eanylin: The strange thing is that it work when run by hand... but not via the script. | 14:46 |
| bmaltais | eanylin: Also, since this script will eventually be run from within a git clone kolla-kubernetes... a lot of the content will need to change | 14:47 |
| bmaltais | eanylin: If a used download it to another location vs /opt the script should handle that... right now it would just download another copy in /opt | 14:48 |
| *** schwicht has quit IRC | 15:00 | |
| *** zhubingbing has quit IRC | 15:00 | |
| *** manheim_ has joined #openstack-kolla | 15:01 | |
| *** hongbin has joined #openstack-kolla | 15:01 | |
| *** zhubingbing has joined #openstack-kolla | 15:02 | |
| inc0 | good morning | 15:03 |
| *** gmann has quit IRC | 15:05 | |
| *** gmann has joined #openstack-kolla | 15:05 | |
| *** zhubingbing has quit IRC | 15:06 | |
| *** srwilkers has quit IRC | 15:06 | |
| manheim_ | hello | 15:07 |
| *** salv-orlando has quit IRC | 15:09 | |
| *** salv-orlando has joined #openstack-kolla | 15:10 | |
| *** srwilkers has joined #openstack-kolla | 15:12 | |
| *** jaosorior has quit IRC | 15:13 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:13 |
| *** salv-orlando has quit IRC | 15:14 | |
| *** schwicht has joined #openstack-kolla | 15:18 | |
| bmaltais | eanylin: Found the issue... but it is now failing with another error. I think it is due to the fact that there is now an all.yml file | 15:20 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:26 |
| openstackgerrit | Martin André proposed openstack/kolla master: Install ODL driver from package where appropriate https://review.openstack.org/466325 | 15:30 |
| *** vhosakot has joined #openstack-kolla | 15:32 | |
| *** eaguilar has joined #openstack-kolla | 15:34 | |
| *** manheim_ has quit IRC | 15:35 | |
| *** manheim_ has joined #openstack-kolla | 15:36 | |
| *** salv-orlando has joined #openstack-kolla | 15:36 | |
| *** awebber has quit IRC | 15:39 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:39 |
| *** powerd has quit IRC | 15:41 | |
| *** salv-orlando has quit IRC | 15:41 | |
| *** sgrasley1 has joined #openstack-kolla | 15:42 | |
| *** sgrasley has quit IRC | 15:42 | |
| *** powerd has joined #openstack-kolla | 15:44 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:47 |
| *** matrohon has quit IRC | 15:50 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 15:50 |
| *** k_mouza has joined #openstack-kolla | 15:51 | |
| k_mouza | hello all!. Quick one please. I want to make some changes in the /etc/neutron/plugins/ml2/ml2_conf.ini file on all the neutron-openvswitch-agent nodes | 15:53 |
| rwellum | bmaltais: running a script as sudo and using sudo per command is much different right? Basically you're running as root the first way. Second way you're running certain commands as root - but home directory is still your user etc, | 15:53 |
| k_mouza | what file should I put my changes in, on the deployment node? | 15:53 |
| k_mouza | it should be something like /etc/kolla/config/neutron/openvswitch-agent/... | 15:54 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla master: ODL installed for source images https://review.openstack.org/466336 | 15:55 |
| k_mouza | I've done a similar configuration for the nova.conf file, but this is a slightly different directory structure | 15:55 |
| *** eaguilar has quit IRC | 15:59 | |
| egonzalez | k_mouza custom config for ml2 file is only supported for all services(neutron-server, ovs-agent or linuxbridge-agent) or pointing to a host | 15:59 |
| openstackgerrit | Martin André proposed openstack/kolla master: Install ODL driver from package for binary distros https://review.openstack.org/466325 | 15:59 |
| *** janki has joined #openstack-kolla | 16:00 | |
| egonzalez | config/neutron/ml2_conf.ini or config/neutron/<inventory_hostname>/ml2_conf.ini | 16:00 |
| egonzalez | k_mouza ^^ | 16:00 |
| k_mouza | thanks a lot @egonzalez ! I'll try that now | 16:02 |
| egonzalez | k_mouza for other service, you can look at config.yml of each role and see what merge_config support, ex of ml2 https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/config.yml#L117 | 16:03 |
| openstackgerrit | Stephanie Miller proposed openstack/kolla-ansible master: Fix vagrant dev env build https://review.openstack.org/466337 | 16:03 |
| egonzalez | most services share the same pattern but in the case of neutron with a ton of different config files is harder | 16:03 |
| openstackgerrit | Stephanie Miller proposed openstack/kolla master: Fix vagrant dev env https://review.openstack.org/466339 | 16:04 |
| bmaltais | rwellum: So perhaps I have made changes that were not needed in the script... I tought the idea was that it would be run as a regular user and that each elevated prix functions would be called via sudo in the script... | 16:04 |
| *** sgrasley1 has quit IRC | 16:05 | |
| rwellum | bmaltais: haven't had a chance to look at the script - but I think what you're describing is the correct way imo. | 16:05 |
| bmaltais | rwellum: Are you saying the the script should be run like: sudo ./setup_AIO.sh vs ./setup_AIO.sh | 16:05 |
| rwellum | No - before it was the former, I think it should be the latter | 16:06 |
| *** manheim_ has quit IRC | 16:06 | |
| *** sgrasley has joined #openstack-kolla | 16:06 | |
| bmaltais | rwellum: OK then. This is what I have been working on to fix it so it work that way based on what was started. | 16:07 |
| rwellum | Great - that in the latest ps? | 16:07 |
| bmaltais | rwellum: Now I am running on another issue not related to sudo due to a change in how the script will no longer use globals.yml for some of the things. | 16:07 |
| rwellum | Ah the all.yml? Yeah I was curious how that would work. | 16:08 |
| bmaltais | rwellum: I am not sure either. I will let eanylin: look into it as he started with the changes. At least most of the sudo issues are cleared now... | 16:08 |
| rwellum | Ok | 16:09 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla-ansible master: Implement blazar ansible role https://review.openstack.org/454752 | 16:10 |
| rwellum | bmaltais: there's one issue with running as individual sudo commands I've discovered. If the script take a long time, then sudo will time out - and mid-way through the script you'll have to enter sudo creds again. | 16:11 |
| *** eaguilar has joined #openstack-kolla | 16:15 | |
| *** stefan_bo has joined #openstack-kolla | 16:17 | |
| vhosakot | k_mouza: I did it recently. Just place your new OVS config in https://github.com/openstack/kolla-ansible/blob/master/ansible/roles/neutron/tasks/config.yml#L118-L119 and run "kolla-ansible reconfigure". node_custom_config defaults to /etc/kolla/config in https://github.com/openstack/kolla-ansible/blob/master/ansible/group_vars/all.yml#L10. | 16:19 |
| *** stefan_bo has quit IRC | 16:22 | |
| *** jmccarthy has left #openstack-kolla | 16:23 | |
| *** unicell has quit IRC | 16:23 | |
| *** eaguilar has quit IRC | 16:26 | |
| *** manheim_ has joined #openstack-kolla | 16:26 | |
| *** manheim_ has quit IRC | 16:27 | |
| vhosakot | amurali: don't think kolla has containerized 6wind yet. Does neutron+networking-6wind work outside the container? :) | 16:30 |
| vhosakot | inc0: another candidate for the new kolla-sdn repo that does fast data - https://github.com/openstack/networking-6wind | 16:31 |
| amurali | Actually, i'm installing it inside 'neutron-server' container as a plugin | 16:31 |
| inc0 | dpdk is cool;) | 16:32 |
| vhosakot | amurali: I have not deployed it, but I've heard it it similar to vpp that I've deployed with neutron, right, as an ml2 plugin for neutron that does fast data in user-space. | 16:32 |
| *** pbourke has quit IRC | 16:33 | |
| vhosakot | amurali: "installing it inside", I think ml2 plugin is a separate process that does fast L2, I think it must be in a separate container like how kolla deploys OTB l2 services like OVS and linux bridge is separate containers. | 16:35 |
| vhosakot | s/ml2 plugin/ml2 plugin for 6wind | 16:35 |
| sdake | morning | 16:37 |
| vhosakot | morning sdake | 16:38 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 16:39 |
| sdake | payday | 16:41 |
| *** pcaruana|rtve| has quit IRC | 16:42 | |
| *** imcsk8 has quit IRC | 16:42 | |
| *** k_mouza has quit IRC | 16:42 | |
| *** imcsk8 has joined #openstack-kolla | 16:42 | |
| egonzalez | amurali last commit into the networking-6wind repo is from oct2016, not sure how maintained is the plugin, unless they moved to other repo, I wouldn't use with ocata | 16:43 |
| *** AzazeL_ has joined #openstack-kolla | 16:44 | |
| vhosakot | some vendor plugins are not as fast moving as other core OpenStack projects | 16:44 |
| eanylin | bmaltais rwellum: I made changes based on what I understand. I will let kfox1111 comment on that. But yeah, with the new proposed change, I believe we need to update all.yml as well for it to work properly. | 16:46 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 16:46 |
| *** sambetts is now known as sambetts|afk | 16:47 | |
| bmaltais | rwellum: This is why I always use a nopassword sudo user so that timeout is not an issue | 16:51 |
| rwellum | bmaltais: ack. | 16:52 |
| rwellum | bmaltais: other work-arounds too. like 'sudo -v' - or setting timeout=-1 in sudoers | 16:52 |
| bmaltais | rwellum: Perhaps it should be indicated in the guide at some point. | 16:52 |
| rwellum | bmaltais: yeah | 16:53 |
| sbezverk | sdake: I have a suggestion. instead of running ansible clean up, let's run just a regular bash script clean up with the same commands adn see if it fails the same way or not | 16:54 |
| sbezverk | at least we will know if it is general issue or specific to ansible | 16:55 |
| bmaltais | eanylin: The script will be intended to run, when ready, from the kolla-kubernetes git clone... right? | 16:55 |
| sdake | sbezverk i could try that - although i scrweed up my knee yesterday so struggling to get my day going | 16:55 |
| bmaltais | eanylin: Right now the script is not build on that basis... | 16:55 |
| bmaltais | eanylin: It is almost build as if it would be downloaded seperatly and not be bundled as part of the tools folder | 16:56 |
| sbezverk | sdake: sorry to hear that.. I might have time later tonight to try it | 16:56 |
| bmaltais | eanylin: What was your original intention? | 16:56 |
| sbezverk | if you do not do it earlier | 16:56 |
| sdake | sbezverk i'll see if i can dig into it - i jammed my knee right into the corner of a door | 16:56 |
| sdake | you wouldn't think that would hurt all tha tmuch | 16:56 |
| *** mgoddard__ has quit IRC | 16:57 | |
| sbezverk | sdake: yeah, knee is a tricky thing too many moving and delicate parts there | 16:57 |
| *** salv-orlando has joined #openstack-kolla | 16:57 | |
| *** unicell has joined #openstack-kolla | 17:02 | |
| eanylin | bmaltais: It is not in the folder for sure as its still in review. This was done in respond to feedback from kfox1111 from patch set $9 | 17:02 |
| eanylin | #9 | 17:03 |
| *** salv-orlando has quit IRC | 17:03 | |
| sdake | sbezverk re https://review.openstack.org/#/c/464031/20/ansible/roles/destroy/tasks/main.yml - I tend to agree re not deleting PVs | 17:03 |
| sdake | sbezverk those are generally created by the cluster admin (in a perfect world) | 17:04 |
| *** srwilkers has quit IRC | 17:04 | |
| sbezverk | sdake: right, but let's keep then for now as we need consistent error | 17:04 |
| sdake | you mean to debug the other gate problem? | 17:04 |
| sbezverk | sdake: well I am not sure if it is gate or combination your ansible script and gate | 17:05 |
| sdake | right thats what i mean | 17:05 |
| sdake | how about i split the review into two | 17:06 |
| sdake | so we can merge one and I can suffer a sense of accomplishment :) | 17:06 |
| *** chas has quit IRC | 17:06 | |
| sbezverk | sdake: if by eleminating ansible it starts working, then ansible experts will nee dto look into it | 17:06 |
| *** chas has joined #openstack-kolla | 17:07 | |
| sdake | genconfig has the same issue | 17:07 |
| sdake | i actually have a way to fix er, I think | 17:07 |
| sdake | if traps return to bash | 17:07 |
| *** chas has quit IRC | 17:11 | |
| rwellum | eanylin: in the latest script, you have as an input the subnet size of management network - will that ever be anything other than 24? | 17:11 |
| *** sgrasley has quit IRC | 17:11 | |
| eanylin | rwellum: Not too sure too but it will be /24 in most cases I believe | 17:12 |
| rwellum | Did you test with other ranges? | 17:12 |
| *** sgrasley has joined #openstack-kolla | 17:12 | |
| eanylin | rwellum: I have not, seems like /24 is the default value, so I used /24 | 17:12 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Test of gate deadlock with bash https://review.openstack.org/466358 | 17:13 |
| rwellum | Ok. Wasn't sure why we made it an option. | 17:13 |
| rwellum | s/you/we | 17:13 |
| eanylin | rwellum: Hmm, cause its an option in the cloud.yml | 17:13 |
| eanylin | rwellum: So I exposed it as well, just in case | 17:14 |
| rwellum | Ok. wfm | 17:15 |
| sdake | egonzalez you about | 17:18 |
| *** mkoderer has quit IRC | 17:24 | |
| *** salv-orlando has joined #openstack-kolla | 17:25 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Crater the deploy using genconfig https://review.openstack.org/466363 | 17:25 |
| sdake | rwellum can you confirm the current deployment guide works as is? | 17:26 |
| *** salv-orlando has quit IRC | 17:28 | |
| rwellum | sdake - no I think the cloud.yaml is kaputt | 17:28 |
| sdake | there is a review in the queue for a canal change | 17:28 |
| sdake | unless it merged | 17:28 |
| *** amurali has quit IRC | 17:28 | |
| rwellum | Urgh - sorry meant canal | 17:28 |
| rwellum | Hmm - interesting - let me look. | 17:29 |
| *** eaguilar has joined #openstack-kolla | 17:29 | |
| rwellum | Don't see it - can you point me to it? | 17:30 |
| *** mgoddard__ has joined #openstack-kolla | 17:31 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 17:35 |
| rwellum | sdake: https://review.openstack.org/#/c/465754/1/doc/source/deployment-guide.rst | 17:40 |
| *** fguillot has quit IRC | 17:40 | |
| *** srwilkers has joined #openstack-kolla | 17:41 | |
| rwellum | sdake: that looks great - missed that. I have now figured out that: project:openstack/kolla-kubernetes works | 17:41 |
| sdake | rwellum nice ;) | 17:42 |
| *** ducttape_ has quit IRC | 17:43 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 17:48 |
| *** powerd has quit IRC | 17:48 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 17:48 |
| sdake | sbezverk https://review.openstack.org/#/c/466358/1/tools/setup_gate.sh appears to fail also | 17:49 |
| sbezverk | sdake: I do not see gate results for that ps | 17:51 |
| sdake | ya zuul is locked up | 17:51 |
| sdake | so looks like its timing out | 17:51 |
| sdake | i can't onnect to these machines as I dont' have ipv6 | 17:51 |
| sbezverk | it will fail because 4th kubectl delete pv does not exist and will fail | 17:52 |
| sdake | right - althoug htaht shouldn't cause the gate to lock up | 17:52 |
| sdake | that should cause the gate to fail fast | 17:52 |
| sbezverk | sdake: you see it does not lock up | 17:52 |
| sbezverk | it did, 36min instead of 1.5 hour | 17:52 |
| sdake | yup | 17:52 |
| sbezverk | so it is ansible then | 17:52 |
| sdake | i think it is a relation between ansible and trap | 17:57 |
| sdake | although i had genconfig generate a similar trap and nothing bad happened | 17:57 |
| *** eaguilar has quit IRC | 17:58 | |
| sdake | i think kubectl is not generating signals when it fails outside of ansible | 17:58 |
| sdake | however ansible is | 17:58 |
| sdake | hard to tell for certain | 17:58 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Crater the deploy using genconfig https://review.openstack.org/466363 | 17:59 |
| sbezverk | sdake: you are ready to suspect bash, kubectl, but not obvious that it is ansible glitch 99.99% | 18:00 |
| sdake | sbezverk i don't know what precisely the problem is, however, my training says its a trap error | 18:01 |
| sdake | signal handling is broken | 18:01 |
| sdake | i have spent years of my life debugging signal handling | 18:01 |
| sdake | and i hae seen this problem hundreds of times in various code | 18:01 |
| sdake | so forgive me if years of my life wasted debugging points to signal handling and i am biased towards that :) | 18:02 |
| sdake | sbezverk remember the gate fails in this way sans destroy :) | 18:02 |
| sdake | (sans ansible) | 18:03 |
| vhosakot | sdake: I saw a similar issue where a Linux command wasn't generating the right failed signal/trap when it failed and my ansible playbook happily treated it as a successful play. I debugged it, opened https://bugzilla.redhat.com/show_bug.cgi?id=1288528 with RH, and RH updated their ERRTA now https://rhn.redhat.com/errata/RHEA-2016-0603.html in OSP8. | 18:03 |
| openstack | bugzilla.redhat.com bug 1288528 in python-oslo-service "services can´t stop due to broken oslo service" [Urgent,Closed: errata] - Assigned to vstinner | 18:03 |
| sbezverk | sdake: right, it fails exactly the way it is supposed to fail | 18:03 |
| sbezverk | because incorrect operation was performed | 18:03 |
| sbezverk | if you add || true to that kubectl command | 18:04 |
| sbezverk | then it will not fail | 18:04 |
| sdake | which one in particular | 18:04 |
| sdake | the destroy? | 18:04 |
| sdake | the gate should never deadlock one a failure | 18:04 |
| sbezverk | no, in your last PS you have line kubectl delete pv helm-repo-wrong | 18:04 |
| sdake | right, i was expecting the gate to lock up there | 18:05 |
| sbezverk | this command tries to delete non-existing object | 18:05 |
| sdake | not for the destroy to work | 18:05 |
| sdake | i get that dude | 18:05 |
| sdake | how do you think i can reproduce the problem without reproducing the problem? | 18:05 |
| sbezverk | as a result bash catches it and calls another bash script which properly collect data | 18:05 |
| sdake | sbezverk http://logs.openstack.org/63/466363/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/80e5644/console.html#_2017-05-19_18_02_36_619018 | 18:06 |
| sbezverk | maybe the issue is the ansible pass errorness to calling parent bash script? | 18:06 |
| sbezverk | I mean, how | 18:06 |
| sbezverk | instead of just returning error code it just dies | 18:07 |
| vhosakot | I hit this too where ethtool was returning the wrong signal to ansible and had to open https://bugzilla.redhat.com/show_bug.cgi?id=1434236. RH hasn't fixed it yet. | 18:07 |
| openstack | bugzilla.redhat.com bug 1434236 in ethtool "ethtool returns incorrect exit code for error (stderr)" [Medium,New] - Assigned to ivecera | 18:07 |
| sbezverk | in a non graceful way | 18:07 |
| sdake | sbezverk ansible doesn't die | 18:07 |
| sdake | capture logs dies | 18:07 |
| sdake | or its parent thereof | 18:07 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:08 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:09 |
| sbezverk | sdake: we know this works: bash (parent) hit error, call child bash to collect logs, do you agree? | 18:10 |
| sdake | we know that a bash parent hit error, this calls bash child to collect logs, that child suffers a virsh error, and then exits immidately: http://logs.openstack.org/63/466363/2/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/80e5644/console.html#_2017-05-19_18_02_36_619018 | 18:12 |
| sdake | we know that a bash parent hits an error (running ansible), this calls bash child to collect logs, that child suffers a virsh error, and then exits | 18:12 |
| *** harlowja has joined #openstack-kolla | 18:13 | |
| sbezverk | sdake: what does not work: bash script (parent) calls ansible script where we hit error and bash script log collector gets stuck. | 18:13 |
| *** salv-orlando has joined #openstack-kolla | 18:13 | |
| sdake | i think the blame of what gets stuck is an open quesiton | 18:13 |
| sdake | + is returned - indcating the parent shell is back in action | 18:13 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:14 |
| *** Jeffrey4l has quit IRC | 18:14 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Crater the deploy using genconfig https://review.openstack.org/466363 | 18:15 |
| sbezverk | sdake: why, clearly you can see from your PS that the reaction is exactly as expected | 18:15 |
| sdake | we also know that this review here will test if genconfig fails in same way | 18:16 |
| *** ddyer has quit IRC | 18:16 | |
| sbezverk | we hit an issue during the execution, and then call log caputre, it hit another issue but it does not get stuck | 18:16 |
| sdake | before debating me to deat hon the root cause, why not let me debug it a bit ;) | 18:16 |
| sbezverk | job does not get hanging for another 40 minutes to be killed | 18:16 |
| sbezverk | sure, I was just highlighting the log your PS was generating and that behavior makes perfect sense | 18:17 |
| sdake | agreed that particular bash log maeks perfect sense | 18:18 |
| sdake | what doesn't make sense is the ansible log | 18:18 |
| sdake | and that is what we are after solving iiuc :) | 18:18 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:18 |
| sdake | the fact that this fails quickly provides extra information: https://review.openstack.org/#/c/466363/ | 18:20 |
| *** Jeffrey4l has joined #openstack-kolla | 18:20 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:24 |
| rwellum | sbezverk: I am trying to get a loopback LVM for Cinder running - as per the gate. But nova is complaining that: " 4: no space left on device". df -h shows plenty of space. | 18:24 |
| * inc0 working up patchset stats | 18:24 | |
| * rwellum daily question for sbezverk | 18:25 | |
| sbezverk | sdake: in this case you run a different ansible script, so it means whatever you do you destroy script is causing the issue and not just ansible in general | 18:26 |
| sdake | that is a possibility | 18:26 |
| sbezverk | rwellum: vgs cinder-volumes | 18:26 |
| sbezverk | rwellum: lvs | 18:26 |
| rwellum | Nothing returned sbezverk | 18:27 |
| sbezverk | rwellum: well I guess that is the answer | 18:28 |
| rwellum | Sorry.. | 18:28 |
| rwellum | lvs nothing | 18:28 |
| rwellum | vgs: https://www.irccloud.com/pastebin/Fd4EfU2S/ | 18:28 |
| openstackgerrit | Numan Siddique proposed openstack/kolla master: Support openvswitch-ovn docker images https://review.openstack.org/459179 | 18:28 |
| rwellum | lvs: https://www.irccloud.com/pastebin/81zqv9LD/ | 18:29 |
| rwellum | I read your commands wrong. | 18:29 |
| *** JoseMello has joined #openstack-kolla | 18:29 | |
| sbezverk | rwellum: how much space you are requesting? | 18:29 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:29 |
| sbezverk | cinder service-list | 18:30 |
| rwellum | What is my '--os-username'? | 18:32 |
| rwellum | sudo dd if=/dev/zero of=/data/kolla/cinder-volumes.img bs=5M count=2048 | 18:33 |
| sbezverk | source keystone_adminrc | 18:33 |
| *** emccormick has quit IRC | 18:35 | |
| rwellum | Ok so have not generated one yet as my cluster didn't come all the way up. But when I try to I am getting space issues. So seems I have really hurt this VM for space somehow. | 18:35 |
| *** Jeffrey4l has quit IRC | 18:36 | |
| sbezverk | rwellum: not sure then what to say, cinder-volume VG takes 10G | 18:36 |
| sbezverk | rwellum: how much space you allocated to you VM? | 18:37 |
| *** numans has quit IRC | 18:37 | |
| rwellum | 50G - and it's all gone :) | 18:37 |
| sbezverk | rwellum: right, because images takes good chunk of memory | 18:38 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 18:38 |
| *** ddyer has joined #openstack-kolla | 18:38 | |
| rwellum | Then I added 10G and it fell over - quite possible | 18:38 |
| sbezverk | rwellum: I normally allocate 100G (OS + Docker images) for VM and extra voume like vdb for cinder-volume vg | 18:39 |
| *** zul has joined #openstack-kolla | 18:41 | |
| rwellum | sbezverk: ok makes sense. | 18:42 |
| rwellum | Where are the images stored btw? | 18:42 |
| *** Jeffrey4l has joined #openstack-kolla | 18:43 | |
| sbezverk | rwellum: depends how your docker is configured | 18:44 |
| sbezverk | rwellum: some time ago I started to use Storage Driver: btrfs as a storage driver and keep images on yet separate vdc disk | 18:45 |
| sbezverk | instead of docker's default loopback | 18:46 |
| sbezverk | it seems a bit better in terms of performance and reliability | 18:46 |
| sbezverk | but require additional configuration and planning | 18:46 |
| rwellum | Interesting. Yeah the VM is harder to create then. | 18:47 |
| rwellum | Where is the default loopback? | 18:47 |
| sbezverk | rwellum: well not that much, instead of 1 disk you pass 3 disks to VM | 18:47 |
| sbezverk | rwellum: I think it is /var/lib/docker | 18:48 |
| rwellum | Yeah and I've done that for other deployments. But for this was just going with default centos. Which never seems to work that well long-term. | 18:48 |
| *** emccormick has joined #openstack-kolla | 18:50 | |
| *** mattmceuen has quit IRC | 18:50 | |
| *** mgoddard__ has quit IRC | 18:50 | |
| rwellum | Yeah looks like /var/lib/docker. Is that something that should be cleaned up between deployments? | 18:53 |
| sbezverk | rwellum: note really, if you pull the same images, then they will just take space they need | 18:55 |
| rwellum | ok | 18:55 |
| sbezverk | but if you deploy different version then, potentially they will eat a lot ;) | 18:55 |
| rwellum | Ok makes sense. | 18:56 |
| rwellum | Sorry sdake didn't mean to de-rail your debug session... | 18:56 |
| rwellum | sbezverk: how much RAM normally for your VM? | 19:05 |
| *** athomas has quit IRC | 19:09 | |
| *** itlinux has joined #openstack-kolla | 19:09 | |
| *** ducttape_ has joined #openstack-kolla | 19:12 | |
| *** mgoddard__ has joined #openstack-kolla | 19:15 | |
| *** powerd has joined #openstack-kolla | 19:16 | |
| *** powerd has quit IRC | 19:21 | |
| *** celo_r has joined #openstack-kolla | 19:26 | |
| *** salv-orlando has quit IRC | 19:38 | |
| *** powerd has joined #openstack-kolla | 19:39 | |
| *** celo_r has quit IRC | 19:43 | |
| *** shardy has quit IRC | 19:46 | |
| *** myoung is now known as myoung|pto | 19:50 | |
| *** mgoddard__ has quit IRC | 19:53 | |
| *** mgoddard__ has joined #openstack-kolla | 19:55 | |
| inc0 | https://review.openstack.org/#/c/466007/ \o/ | 20:01 |
| inc0 | multinode gate green for kolla-ansible | 20:01 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 20:03 |
| *** Serlex has quit IRC | 20:03 | |
| *** cali_box has quit IRC | 20:03 | |
| kfox1111 | 1.6.4 :) | 20:05 |
| *** sgrasley has quit IRC | 20:08 | |
| *** sgrasley has joined #openstack-kolla | 20:08 | |
| *** matrohon has joined #openstack-kolla | 20:09 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 20:09 |
| matrohon | Jeffrey4l, hi | 20:10 |
| inc0 | kfox1111: not quite your league yet, but I'm getting there (talking about number of patchsets) https://review.openstack.org/#/c/466007/40 | 20:10 |
| *** jrist has quit IRC | 20:11 | |
| kfox1111 | inc0: hehe. looking good. :) | 20:11 |
| kfox1111 | and a really important use case. :) | 20:11 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: Remove workaround for broken 1.6.3 https://review.openstack.org/466398 | 20:12 |
| inc0 | well at the same time I'm refactoring gates quite a bit | 20:12 |
| kfox1111 | it would be nice if we could generalize the kkolla-kubernetes gates a bit and reuse chunks. | 20:12 |
| inc0 | so I hope at the end we'll be able to add use cases easily | 20:12 |
| kfox1111 | the part that gathers logs is really really good. | 20:12 |
| inc0 | agree | 20:12 |
| kfox1111 | but could be broken up into a docker vs non docker logging section. | 20:12 |
| matrohon | Does anyone know if "kolla-ansible upgrade" is supposed to work to upgrade form newton to ocata? | 20:12 |
| inc0 | matrohon: it is supposed to | 20:13 |
| matrohon | inc0: good to know, I'm having some troubles | 20:13 |
| inc0 | what kind of problems? | 20:14 |
| matrohon | I've try to upgrade from 3.0.3 to 4.0.2 | 20:14 |
| *** itlinux has quit IRC | 20:14 | |
| matrohon | may be the gap is to big... | 20:14 |
| sdake | rwellum sup dude its ok - i was taking a nap | 20:16 |
| matrohon | inc0 : the cmd "nova-manage cell_v2 simple_cell_setup" is run from the nova-api container in the 3.0.3 version | 20:16 |
| rwellum | eanylin: still around? | 20:18 |
| inc0 | manheim: it should be run from 4.x | 20:18 |
| inc0 | did you bump container versions in your globals.yml? | 20:18 |
| matrohon | and the task "running nova bootstrap containers" fails, saying the "nova manage cell..." command should be run first, but it should have been run.... | 20:18 |
| matrohon | inc0, I did | 20:19 |
| *** mgoddard__ has quit IRC | 20:20 | |
| SamYaple | yea newton is all funky and stuff | 20:20 |
| SamYaple | im pretty sure its broken | 20:20 |
| SamYaple | like the instructions work for ocata, but newton doesnt and i havent had a chance to figure out why | 20:20 |
| SamYaple | which is wierd because it _was_ working at one point | 20:20 |
| matrohon | inc0, I feel like the "nova manage cell..." should be run in a up-to-date nova-api container, it can't work in a old nova-api container | 20:21 |
| inc0 | matrohon: it should | 20:21 |
| inc0 | that's why I'm asking about globals.yml, and openstack_release conf | 20:21 |
| matrohon | inc0, it should work even if it is run in the old nova-api container? | 20:22 |
| inc0 | no, simple cell setup is done in new container | 20:23 |
| inc0 | we might have bug. | 20:24 |
| inc0 | but it's strange | 20:25 |
| inc0 | also, looking at code and comments it should actually be runnable in newton cotnainers | 20:25 |
| matrohon | inc0, ok that's what I was guessing, but if I look at the upgrade task, it seems the exec command is done in the old container | 20:25 |
| matrohon | http://git.openstack.org/cgit/openstack/kolla-ansible/tree/ansible/roles/nova/tasks/upgrade.yml?h=stable/ocata#n11 | 20:26 |
| inc0 | yeah | 20:26 |
| inc0 | that's what I'm looking at | 20:26 |
| inc0 | if you try to add cell0 like that manually | 20:26 |
| *** janki has quit IRC | 20:26 | |
| *** salv-orlando has joined #openstack-kolla | 20:26 | |
| inc0 | does it work? | 20:26 |
| inc0 | (I hate what nova did to N->O upgrades)( | 20:26 |
| rwellum | sdake: you are doing the vip keepalive in your deployments? | 20:26 |
| inc0 | especially that it merged 2 weeks before Newton release:/ | 20:26 |
| matrohon | inc0, I'm not sure how to figure it did work | 20:26 |
| sdake | rwellum indeed | 20:27 |
| inc0 | matrohon: do docker exec -it nova-api bash | 20:27 |
| inc0 | and run simple cell setup manually from within container | 20:27 |
| kfox1111 | ok.... so... looks like we just need way to delete the nova_cell0 db now... | 20:28 |
| rwellum | sdake: For the bridge to bind the VIP to - anything special? Which int - or just a dump listening bridge? | 20:28 |
| sdake | rwellum moment | 20:28 |
| matrohon | inc0, I need to figure out what was the transport-url parameter! | 20:31 |
| *** dave-mccowan has quit IRC | 20:31 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: WIP: v4 gates. https://review.openstack.org/454841 | 20:31 |
| sdake | rwellum https://da.gd/El5P | 20:31 |
| sdake | kfox1111 a bit of help on https://review.openstack.org/#/c/464031/ | 20:32 |
| *** zul has quit IRC | 20:32 | |
| inc0 | matrohon: you can add - debug task to ansible | 20:32 |
| sdake | kfox1111 the ironic gate locks up 100% of the time and times out when running that workflow with ansibulls | 20:33 |
| sdake | kfox1111 same workflow in bash doesn't lock up | 20:33 |
| kfox1111 | thats a little weird... | 20:33 |
| sdake | kfox1111 suspicion is it has to do with the trap operation | 20:33 |
| sdake | the save logs code is also generating an error | 20:33 |
| sdake | which is like a double trap | 20:33 |
| sdake | typcially when you generatea signal from within a signal handler (in c) the system implodes | 20:34 |
| sdake | i suspect bash uses the same mechanics but not certain | 20:34 |
| matrohon | incO BTW I also had to change permission on the file /var/log/kolla/nova/nova-manage.log to 777 | 20:34 |
| kfox1111 | so, the trap should not be set inside that file. | 20:35 |
| matrohon | otherwise the nova-manage command fails with permission denied | 20:35 |
| kfox1111 | so disabling it should be a noop. | 20:35 |
| sbezverk | sdake: -1 because another ansible script works fine | 20:35 |
| kfox1111 | sorry. in gate_capture_logs.sh. | 20:35 |
| *** zul has joined #openstack-kolla | 20:35 | |
| sbezverk | it is something inside of you destroy script | 20:35 |
| sdake | other ansible script doesn't generate double trap | 20:35 |
| kfox1111 | line 3 of destroy_tests.sh starts with 'stray' ? | 20:36 |
| kfox1111 | that a typo? | 20:36 |
| sbezverk | if you remove everything from it, i bet it will not gets locked up | 20:36 |
| kfox1111 | oh. yeah. it looks like it might be a comment that got munged. | 20:36 |
| matrohon | inc0, do you know how I can tell kolla-ansible to only run nova tasks? | 20:37 |
| sdake | kfox1111 oh ya comment got fubared it appears | 20:37 |
| inc0 | matrohon: kolla-ansible upgrade -t nova | 20:37 |
| matrohon | inc0 I've tried --tags, but it runs a lot of other tasks | 20:37 |
| inc0 | it will also run common role | 20:37 |
| sdake | sbezverk i need to be able to preserve the double trap to genreate the problem -that means nova has to be deleted | 20:37 |
| matrohon | inc0 -t is an alias of --tags? | 20:37 |
| inc0 | yes | 20:37 |
| sdake | if ndeed that is the problem | 20:38 |
| rwellum | kfox1111: did you try 1.6.4 - or you making the change and seeing if the gate works? | 20:38 |
| matrohon | incO I'll continue this way | 20:38 |
| sbezverk | sdake remove everything from ansible except kubectl that gen error | 20:38 |
| sdake | sbezverk there are two interactions - 1 trap occurs in ansible, 1 occurs in virsh | 20:38 |
| kfox1111 | rwellum: the latter. | 20:38 |
| kfox1111 | the gate's my test. :) | 20:38 |
| sdake | removing everything except kubectl i would expect to work | 20:38 |
| rwellum | kfox1111: cool | 20:39 |
| sbezverk | excep then two things | 20:39 |
| kfox1111 | rwellum: very easy to just poke in a test, then forget about it for 30 min, and check back on it later. | 20:39 |
| sdake | sbezverk you mean keep the two operations that result in a failure? | 20:39 |
| sbezverk | yes | 20:39 |
| sdake | ok | 20:39 |
| sbezverk | but remove everything else | 20:39 |
| sdake | kfox1111 is there a wa yto disable a venv | 20:40 |
| kfox1111 | sdake: deactivate | 20:40 |
| sdake | sbezverk we got a different failure this time: http://logs.openstack.org/31/464031/21/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/ef09ca9/console.html#_2017-05-19_17_53_06_151968 | 20:42 |
| sdake | sbezverk this was with a typo in the destroy code - although the destory code was never run | 20:42 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 20:43 |
| matrohon | inc0, done | 20:43 |
| matrohon | inc0, the nova-manage cmd says "Already setup, nothing to do." | 20:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/466404 | 20:46 |
| sdake | sbezverk kfox1111 - running two different things now - one with deactive - one with just the two operations that result in a crater in destroy to see if its some other part of destroy causing the problem | 20:47 |
| sdake | kfox1111 do you have any other specualtino as to what could be the problem? | 20:48 |
| matrohon | inc0, but the nova-bootstrap container keeps on failing saying "ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing." | 20:49 |
| *** salv-orlando has quit IRC | 20:49 | |
| matrohon | inc0, I'll keep on digging around and report a bug if needed | 20:49 |
| rwellum | As an aside - went for a walk today down by a lake by our office, and a huge snake shot up the path and stopped an inch from my foot. Then followed me about 20ft down the path... I never saw a snake do that before... Someone told me people have been feeding them... | 20:49 |
| kfox1111 | sdake: not really sure... off hand. if its taking that long, usually thats something is wedged. | 20:49 |
| kfox1111 | the trap code in the gate log capture shoudl be a no op. | 20:50 |
| sdake | it is wedged for sure | 20:50 |
| kfox1111 | a double trap shouldn't really be a problem as it crosses bash process bounderies. | 20:50 |
| sdake | thanks for that peice of information - iddin't know that | 20:50 |
| kfox1111 | I'm thinking something in the destory_tests.sh script is just g etting stuck and not returning. | 20:50 |
| kfox1111 | then getting killed by the gate timeout. | 20:50 |
| sdake | hadn't thought of that | 20:51 |
| sdake | worth a go to verify | 20:51 |
| kfox1111 | might remove most of the stuff out of the ansible script. then try. if it works, add chunks back till it hangs. | 20:51 |
| sdake | kfox1111 ya i've got a line of debugging on that that sbezverk suggested | 20:51 |
| kfox1111 | cool. | 20:51 |
| kfox1111 | gotoa head out to another meeting. | 20:51 |
| sdake | i like the destroy_tests.shsanalysis | 20:51 |
| sdake | later kfox1111 thanks | 20:51 |
| kfox1111 | bbiab. | 20:51 |
| kfox1111 | np. :) | 20:52 |
| sdake | rwellum gah who would feed a snake :) | 20:52 |
| sdake | rwellum actualy i don't mind snakes | 20:52 |
| sdake | as long as they are not rattlers | 20:52 |
| sdake | rwellum when i was in BSA, summer camp was filled with 20+ hours playing with snakes | 20:53 |
| rwellum | sdake: me neither but they aren't supposed to approach you and ask for snack! | 20:53 |
| sdake | even in BSA wee were not permitted to feed the snakes | 20:53 |
| sdake | (unless it was their meal) | 20:53 |
| sdake | they let us do that - kind of crazy to watch | 20:53 |
| * rwellum shuddering | 20:53 | |
| sdake | there is a zoo in arizona that does live lion feedings | 20:54 |
| sdake | that is friegenting | 20:54 |
| sdake | actually tis raw meat so the lions aren't feeding on live animals | 20:54 |
| sdake | its live in the sense that its right in front of you :) | 20:54 |
| rwellum | Oh phew.. | 20:55 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Kevins' line of debug https://review.openstack.org/466408 | 20:57 |
| sdake | i took my kids once | 20:57 |
| sdake | when they were too young | 20:57 |
| sdake | (first itme) | 20:57 |
| sdake | bad idea | 20:57 |
| sdake | although they still love that zoo | 20:58 |
| rwellum | Yeah I bet! :) | 20:58 |
| *** itlinux has joined #openstack-kolla | 20:58 | |
| sdake | its really cool - about 5 square miles | 20:58 |
| sdake | all the animals are in natural habitats etc | 20:58 |
| sdake | they have a ride in the zoo area where the animals are | 20:58 |
| sdake | and a giraffe licked my daughter | 20:59 |
| rwellum | Similar with Lions - but it was mating season. One of those driving through a park deals. Some hard explaining had to occur. :/ | 20:59 |
| openstackgerrit | Marcus Williams proposed openstack/kolla-ansible master: Add OpenDaylight role https://review.openstack.org/416367 | 20:59 |
| rwellum | Guess she was tasty. | 20:59 |
| sdake | lol - once iin this zoo saw a pair of turtles doing their thing | 20:59 |
| sdake | allt he kids wer elike"what is that":) | 20:59 |
| sdake | like 30 kids on this truck | 20:59 |
| sdake | it was a little emberassing for the adults | 20:59 |
| sdake | because of course the kids knew what was going on :) | 21:00 |
| rwellum | :) | 21:00 |
| *** ssurana has joined #openstack-kolla | 21:00 | |
| *** bmace__ has quit IRC | 21:03 | |
| *** salv-orlando has joined #openstack-kolla | 21:03 | |
| *** bmace__ has joined #openstack-kolla | 21:03 | |
| *** salv-orlando has quit IRC | 21:04 | |
| egonzalez | inc0 matrohon may be late, N to O need cells created before upgrade thats why command is executed before upgrade containers | 21:04 |
| matrohon | egonzalez, ack, but the nova-api container with the name nova-bootstrap fails as if the command has never been run... | 21:05 |
| rwellum | kfox1111: 1.6.4 seems to be working smoothly for me locally. So far anyway | 21:05 |
| egonzalez | nova-manage in 3.0.2 had a bug with the behaviour of the command, cannot remember what was, first need upgrade to 3.0.3 | 21:05 |
| *** zul has quit IRC | 21:05 | |
| sbezverk | rwellum: gate is still pulling 1.6.3 at least for Ubunut | 21:06 |
| *** ssurana has quit IRC | 21:07 | |
| sbezverk | we should wait, maybe until EOB today and then re-check | 21:07 |
| rwellum | sbezverk: https://review.openstack.org/#/c/466398/1 | 21:07 |
| matrohon | egonzalez, ok, but I'm already in 3.0.3 | 21:07 |
| sbezverk | rwellum: I know, I commented already.. | 21:07 |
| vhosakot | matrohon: "kolla-ansible upgrade" should work and runs upgrade.yml depending on your openstack_release. here is how the gate runs upgrade http://logs.openstack.org/52/454752/6/check/gate-kolla-ansible-dsvm-deploy-centos-binary-centos-7-nv/f2ed72a/console.html#_2017-05-19_16_51_06_716478. | 21:08 |
| rwellum | sbezverk: oops sorry! | 21:08 |
| sbezverk | rwellum> no worries | 21:08 |
| *** JoseMello has quit IRC | 21:08 | |
| egonzalez | matrohon what was the error? | 21:10 |
| matrohon | egonzalez, the "nova-manage sync" cmd run in the nova-bootstrap container fails with : ValidationError: Cell mappings are not created, but required for Ocata. Please run nova-manage cell_v2 simple_cell_setup before continuing. | 21:13 |
| vhosakot | ocata and its cells :) | 21:13 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: DONOTMERGE: test multinode gate https://review.openstack.org/466007 | 21:13 |
| *** zul has joined #openstack-kolla | 21:15 | |
| egonzalez | matrohon can you share ansible logs in the nova part? | 21:15 |
| egonzalez | matrohon have you built your images or pulled from dockerhub? | 21:16 |
| matrohon | egonzalez, I'm running an new upgrade cmd in vernose mode | 21:17 |
| matrohon | egonzalez, I've build from source | 21:17 |
| matrohon | and pushed in my local repo | 21:17 |
| vhosakot | devstack stable/ocata OTB was also broken for a long time until recently as devstack's stack.sh failed to create nova cells OTB --> https://github.com/openstack-dev/devstack/blob/stable/ocata/stack.sh#L1369-L1383 | 21:20 |
| vhosakot | it is fixed now. | 21:20 |
| emccormick | Hey all! Quick (hopefully) question. I need to override tunnel_interface and cluster_interface for a couple hosts as they are not identical. Do I need to override it via neutron configs in /etc/kolla/config or is there a way to specify those options per host? | 21:20 |
| egonzalez | emccormick add the variable in the inventory file | 21:21 |
| egonzalez | emccormick example https://github.com/openstack/kolla-ansible/blob/master/doc/multinode.rst#edit-the-inventory-file | 21:22 |
| emccormick | egonzalez: Ah I thought that was just for ansible-specific options. Cool. I'll give it a go. Thanks! | 21:24 |
| *** itlinux has quit IRC | 21:25 | |
| *** salv-orlando has joined #openstack-kolla | 21:25 | |
| vhosakot | emccormick: right, tunnel_interface is not a neutron config, it is a kolla variable. You could also set via command line like appending -e "tunnel_interface=<interface other than cluster_interface>" at the end of your ansible command. | 21:26 |
| egonzalez | remember one of the issues, simple_cell_setup looked up for flavors in the db, if no flavor was present took the db sync as a fresh deploy, that caused false positives in many upgrade gates without load | 21:26 |
| vhosakot | never used cluster_interface much really, think it is for ceph storage.. | 21:26 |
| *** zul has quit IRC | 21:27 | |
| egonzalez | matrohon just to be sure, can you verify if ansible roles are in ocata version, the cell setup in current master was removed from upgrade phase since is not needed anymore | 21:28 |
| *** dave-mccowan has joined #openstack-kolla | 21:30 | |
| matrohon | egonzalez, good idea, I'll run the script directly from my kolla-ansible clone, in the stable/ocata branch | 21:32 |
| emccormick | vhosakot: Sorry, meant api_interface. Old habit from my previous puppet stuff. sluster_interface is for the private network of a ceph cluster, yes. | 21:33 |
| matrohon | egonzalez, unfortunately, kolla-ansible doesn't have a --version option | 21:33 |
| egonzalez | yeah, is a bash script ;) | 21:33 |
| egonzalez | may be a good feature to have | 21:34 |
| *** dave-mccowan has quit IRC | 21:34 | |
| matrohon | egonzalez, same error.. | 21:34 |
| vhosakot | emccormick: ah ok cool | 21:35 |
| matrohon | egonzalez, I'm not sure to understand how the "nova-manage db-sync " is triggered | 21:35 |
| egonzalez | matrohon im running a deploy/upgrade | 21:35 |
| matrohon | egonzalez, cool, thanks | 21:35 |
| egonzalez | https://github.com/openstack/kolla-ansible/blob/stable/ocata/ansible/roles/nova/tasks/upgrade.yml | 21:36 |
| egonzalez | first cell0 is created in newton | 21:36 |
| *** schwicht has quit IRC | 21:36 | |
| egonzalez | do simple_cell_setup which sync cell0 tables in current db version | 21:36 |
| egonzalez | then do the db sync for the schema version upgrade | 21:37 |
| matrohon | egonzalez, I'd like to give a try on running simple_cell_setuo cmd during the nova_bootstrap phase, before the db_sync cmd... | 21:38 |
| *** salv-orlando has quit IRC | 21:39 | |
| egonzalez | issue was, in newton cell_setup pointed to nova_api_cell0 database instead of nova_cell0 | 21:39 |
| egonzalez | thats why first is created the cell | 21:39 |
| *** salv-orlando has joined #openstack-kolla | 21:41 | |
| *** srwilkers has quit IRC | 21:41 | |
| matrohon | egonzalez, inc0, gtg, thanks for your help, I'll keep on working on that on monday | 21:42 |
| inc0 | sorry to hear you had issues matrohon | 21:43 |
| vhosakot | emccormick: I just use the default api_interface (which defaults to network_interface) in group vars. | 21:43 |
| matrohon | inc0, no worry, it's not production, only testing, and I'm learning a lot about ansible :) | 21:43 |
| inc0 | it's good knowledge to have if you want to stick with kolla-ansible | 21:43 |
| *** salv-orlando has quit IRC | 21:45 | |
| inc0 | egonzalez: check that out :) https://review.openstack.org/#/c/466007/42 | 21:45 |
| vhosakot | inc0: 42 patch sets.. I think sbezverk crashed gerrit once with his 829th patch set ;) | 21:46 |
| egonzalez | a big w00t! | 21:46 |
| emccormick | vhosakot: I spread things out a bit more and then I have a bunch of gear from all manner of vendors with all manner of different nic configs so I can't generalize. Beggars can't be choosers ;) | 21:46 |
| inc0 | not only enabled multinode | 21:47 |
| inc0 | but also simplify a lot to add new scenerios | 21:47 |
| inc0 | https://review.openstack.org/#/c/466415/ when this merges I'll work on having super easy mechanism to add new scenerios | 21:47 |
| vhosakot | emccormick: cool, good to isolate network segments/NICs for specific traffic | 21:48 |
| sdake | sbezverk check this out: http://logs.openstack.org/31/464031/22/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/b2206d4/console.html#_2017-05-19_21_45_28_094397 | 21:50 |
| vhosakot | sdake: touch it lol.. touch /var/run/libvirt/libvirt-sock-ro... I just touch all missing things lol | 21:51 |
| vhosakot | j/k | 21:52 |
| *** ssurana has joined #openstack-kolla | 21:52 | |
| sdake | kfox1111 this no worky: https://review.openstack.org/#/c/464031/21..22/tests/bin/destroy_tests.sh - although looking very positive (no gate lockup after the ansible code is run) | 21:52 |
| sdake | kfox1111 which implies that the script is locking up related to testing | 21:52 |
| sdake | kfox1111 any suggestions on how to get that deactivate to work? | 21:53 |
| sdake | inc0 ^^ | 21:53 |
| * sdake never uses virtualenv | 21:53 | |
| inc0 | sdake: just call deactivate | 21:54 |
| sdake | inc0 so ./deactivate? | 21:55 |
| sdake | (what is the path after an activation) | 21:55 |
| inc0 | let me check my envs | 21:55 |
| sdake | inc0 http://logs.openstack.org/31/464031/22/check/gate-kolla-kubernetes-deploy-centos-source-4-ironic-nv/b2206d4/console.html#_2017-05-19_21_45_05_293645 | 21:55 |
| inc0 | it normally is available just as $deactivate | 21:55 |
| sdake | so just run from the cmdline $deactivate? | 21:59 |
| vhosakot | sdake: exit also exits from venv, but deactivate is much more graceful and cleaner. | 22:00 |
| sdake | vhosakot which command should be run | 22:01 |
| sdake | vhosakot ./deactivate? | 22:01 |
| vhosakot | there should be files activate and deactivate inside the venv.... find . | grep 'activate' should show the path. | 22:02 |
| *** schwicht has joined #openstack-kolla | 22:02 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 22:02 |
| sdake | vhosakot i am in a gate job - it takes 45 minute sto run - i really don't want to hunt for it which is why i am asking | 22:03 |
| *** matrohon has quit IRC | 22:03 | |
| sdake | spend 30 minutes writing code - spend 5 days making the gate work - winning | 22:03 |
| *** salv-orlando has joined #openstack-kolla | 22:04 | |
| sdake | inc0 is this correct: https://review.openstack.org/#/c/464031/23/tests/bin/destroy_tests.sh | 22:04 |
| inc0 | sdake: no | 22:05 |
| inc0 | but try just calling deactivate | 22:05 |
| sdake | that wont be in the path... | 22:05 |
| inc0 | well it is, I think it's an alias | 22:06 |
| inc0 | try iot | 22:06 |
| *** sambetts|afk has quit IRC | 22:06 | |
| sdake | that makes sense thnaks i'll try | 22:06 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 22:08 |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 22:10 |
| vhosakot | sdake: ah, I don't know what's happening in the review in gate, sorry... just said how I'm able to deactivate my venv... I use activate and deactivate my venv like this --> virtualenv test ; cd test/ ; source bin/activate ; deactivate | 22:10 |
| sdake | thanks vhosakot | 22:10 |
| *** sambetts_ has joined #openstack-kolla | 22:10 | |
| sdake | espp FTW | 22:11 |
| *** mgoddard__ has joined #openstack-kolla | 22:11 | |
| *** harlowja has quit IRC | 22:12 | |
| vhosakot | sdake: yes, deactivate won't be in the path inside the venv, it is not an alias too, but the command "deactivate" works fine and deactivates the activated venv. | 22:19 |
| vhosakot | sdake: "which deactivate" inside the venv prints "/usr/bin/which: no deactivate in (/root/test/bin:/sbin:/bin:/usr/sbin:/usr/bin)" even though the command deactivate works.. | 22:19 |
| *** stefan_bo has joined #openstack-kolla | 22:20 | |
| vhosakot | don't know how it works, need to read up on it.. deactivate will not be in the path and not an alias too.. some sort of runtime thing happening inside venv | 22:20 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Kevins' line of debug https://review.openstack.org/466408 | 22:21 |
| sdake | vhosakot its a shell fucntion | 22:21 |
| vhosakot | inc0: deactivate is not an alias too, "alias | grep deactivate" prints nothing in my venv :) | 22:22 |
| vhosakot | sdake: ah ok, yeah, some sort of shell-level parser invocation thing inside the venv. | 22:22 |
| egonzalez | inc0 just upgraded centos source 3.0.3-> 4.0.0 and went fine, will try with 4.0.2 just in case something changed | 22:22 |
| inc0 | vhosakot: right, I don't know where it came from;) | 22:22 |
| inc0 | thanks egonzalez | 22:23 |
| vhosakot | inc0: me too :) | 22:23 |
| inc0 | upgrade gates btw are next after ceph on my gate list | 22:23 |
| *** ducttape_ has quit IRC | 22:23 | |
| sdake | inc0 a focus of kolla-ansible on gating would be in order i suspect - as the implementation is pretty solid at this point - so more power to ya :) | 22:24 |
| egonzalez | ceph w/ cinder will fail until this get merged https://review.openstack.org/#/c/457993/ | 22:24 |
| sbezverk | sdake: you need to remove hard coding PV's name even in the gate code we will change them at one point when we need to run multiple Rabbitmq | 22:24 |
| inc0 | sdake: https://review.openstack.org/#/c/466007/ | 22:24 |
| inc0 | so my order of work I'm going to do for next few weeks | 22:24 |
| *** stefan_bo has quit IRC | 22:25 | |
| inc0 | this, ceph gates, upgrade gates | 22:25 |
| inc0 | then, if we actually get permission to push images to dockerhub | 22:25 |
| inc0 | a voting gate set for push job daily | 22:25 |
| inc0 | if we won't get this permission | 22:25 |
| inc0 | I'll do it anyway just for tarballs | 22:25 |
| inc0 | so we can have deploy-tested images in tarballs at all times | 22:26 |
| inc0 | after that we can make voting deploy gates | 22:26 |
| inc0 | and that will make our gates really really solid | 22:26 |
| inc0 | btw please review https://review.openstack.org/#/c/466007/ | 22:27 |
| inc0 | it's ready to be merged imho | 22:27 |
| inc0 | or at least reviewed;) | 22:27 |
| *** powerd has quit IRC | 22:29 | |
| *** salv-orlando has quit IRC | 22:29 | |
| *** mgoddard__ has quit IRC | 22:30 | |
| *** schwicht has quit IRC | 22:32 | |
| *** sgrasley has quit IRC | 22:32 | |
| *** powerd has joined #openstack-kolla | 22:35 | |
| *** sgrasley has joined #openstack-kolla | 22:36 | |
| kfox1111 | sbezverk: the name is totally erelevent. there's nothing pinning pvc's to pv's by name anyway. | 22:38 |
| kfox1111 | we could just change it to name: {{random()}} | 22:38 |
| *** amurali has joined #openstack-kolla | 22:38 | |
| sbezverk | kfox1111: right but in sdake's destroy, there is a lost of PVs | 22:39 |
| sbezverk | which gets removed | 22:39 |
| sbezverk | I think that whole part needs to be removed | 22:39 |
| kfox1111 | he's deleting everything I thoguht. | 22:39 |
| kfox1111 | "burn it all down" | 22:39 |
| sbezverk | what if I choose a different name for PV | 22:39 |
| sbezverk | then his script will not delete | 22:39 |
| kfox1111 | I thought he was for looping and deleting all pv's? mabye Im wrong. | 22:40 |
| sbezverk | it is either to make it flexible or just not delete PVs | 22:40 |
| sbezverk | nope, he hard coded 4 PVs | 22:40 |
| kfox1111 | depends on the use case. | 22:40 |
| sbezverk | which is not right I think | 22:40 |
| kfox1111 | if its "user owns the whole k8s and wants to hit the reset button so can deploy kolla-kubernetes again", | 22:40 |
| kfox1111 | thats easy to just delete all the things. | 22:40 |
| sbezverk | or ALL | 22:41 |
| kfox1111 | if its more complicated then that, it probably should use your cleanout stuff. | 22:41 |
| *** chas has joined #openstack-kolla | 22:41 | |
| kfox1111 | then its very precice. | 22:41 |
| *** amurali has quit IRC | 22:42 | |
| sbezverk | kfox1111: example is running multiple rabbitmq, then we will need PV per rabbitmq instance. I would really like to see this scenario tested and working.. | 22:45 |
| kfox1111 | +1 | 22:45 |
| kfox1111 | but I'm saying, his cleanup stuff should either just delete all pv's, so thats not a problem, | 22:45 |
| kfox1111 | or call the cleanup stuff you wrote, which should delete them ok? | 22:45 |
| *** chas has quit IRC | 22:46 | |
| sbezverk | kfox1111: right | 22:46 |
| *** Pavo has joined #openstack-kolla | 22:46 | |
| *** sgrasley has quit IRC | 22:46 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla-ansible master: Enable multinode gate https://review.openstack.org/466007 | 22:47 |
| *** sgrasley has joined #openstack-kolla | 22:49 | |
| *** Pavo has quit IRC | 22:49 | |
| *** harlowja has joined #openstack-kolla | 22:57 | |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 23:08 |
| sdake | kfox1111 i don't want to burn down the pvs, as the cluster may have other pvs in it unrelated to kolla and pvs are enot namespaced | 23:09 |
| kfox1111 | sdake: then I wonder if we should do something like sbezverk's cleanup script to handle those... | 23:10 |
| sdake | kfox1111 in fact in that last review i removed the pvs | 23:10 |
| sdake | which script kfox1111 ? | 23:10 |
| kfox1111 | like, helm/service/*-cleanup | 23:10 |
| sdake | the pvs are to be managed by teh admin and hopefully not deleted at all I think | 23:10 |
| *** emccormick has quit IRC | 23:11 | |
| kfox1111 | the whole pv thing is a mess. we should get to dynamic pv's and let the admin deal with manual entirely. | 23:11 |
| *** powerd has quit IRC | 23:11 | |
| sdake | i think for now we can just punt on pvs | 23:11 |
| sdake | the alternative is for me to list them out and delete those with a mapping of mariadb/glance/rabbitmq/etc | 23:11 |
| sdake | the cleanup helm jobs are naother optoin as well | 23:12 |
| sdake | although last i looked at those, they simply clean up the database entries/users/etc | 23:12 |
| kfox1111 | yeah. helm delete <thingy> --purge does most of the rest. | 23:13 |
| kfox1111 | the cleanup stuff is for cleaning up stuff helm itself cant. | 23:13 |
| sdake | deleting pvs vai helm works | 23:14 |
| sbezverk | kfox1111: here is universal way of deleting pvc and pv | 23:15 |
| sbezverk | kubectl get pvc -n kolla | grep Bound | awk '{print "kubectl get pvc -n kolla -o jsonpath={.spec.volumeName} "$1}' | 23:15 |
| sbezverk | we get all pvc in kolla namespace then get attached volumes and then delete them | 23:15 |
| sdake | sbezverk does that operation print out a list of pvs? | 23:15 |
| kfox1111 | that works, but only if the pvcs havent been deleted yet. | 23:15 |
| sbezverk | so we do not hard code anything and we know since it seats in kolla | 23:15 |
| kfox1111 | I though helm deleted pvc's? | 23:16 |
| sdake | kfox1111 i can work with that model :) | 23:16 |
| sdake | kfox1111 i can capture the pvs prior to deletion of the helm charts | 23:16 |
| kfox1111 | sdake: the other option is to run that, then delete all the charts, then the pv's listed. | 23:16 |
| sbezverk | kfox1111: right in the loop we get info, then delete pvc and then pv | 23:16 |
| kfox1111 | yeah. | 23:16 |
| kfox1111 | :) | 23:16 |
| kfox1111 | isn't it wonderful when a plan comes together. :) | 23:16 |
| sbezverk | kfox1111: man I like so much bash :) really good thing to have | 23:17 |
| kfox1111 | ages and ages ago, when I was on the dos platform, I really liked the cli over windows. could do so many more things. | 23:17 |
| kfox1111 | oh how wrong I felt when I finally discovered bash. :) | 23:18 |
| sbezverk | kfox1111: :) | 23:18 |
| sbezverk | now windows provides power shell | 23:18 |
| kfox1111 | bash is in a nice middle ground between shell and programming language. | 23:18 |
| kfox1111 | yeah. I have really mixed feelings about power shell. | 23:19 |
| kfox1111 | it has some really cool features, above and beyond bash. | 23:19 |
| sbezverk | never used it, but something tells me it is not as good as bash | 23:19 |
| kfox1111 | and some really uglys. :) | 23:19 |
| kfox1111 | since it uses .net, | 23:19 |
| kfox1111 | its pipes are actually more powerful then bash. | 23:19 |
| kfox1111 | in bash, everythings a file stream. | 23:19 |
| kfox1111 | in power shell, they are data types, so you can push more complicated things through much faster. | 23:19 |
| kfox1111 | tehy actually learned a thing or two from unix rather then just igorning it for once. | 23:20 |
| sbezverk | nice to hear that microsoft is still capable to learn from others | 23:21 |
| kfox1111 | they aren't what they use to be. | 23:22 |
| kfox1111 | they sell a lot of linux now. | 23:22 |
| kfox1111 | linux broke their monopoly. | 23:22 |
| sbezverk | have you tried docker on windows? | 23:22 |
| kfox1111 | no. | 23:22 |
| kfox1111 | don't realy want to get locked into that. :) | 23:22 |
| kfox1111 | but glad they are trying to adapt to the modern world. :) | 23:22 |
| kfox1111 | and its good to see them play catch up rather then the open source community trying to play catch up all the time. :) | 23:23 |
| sdake | sbezverk what is $1 in your 1 liner? | 23:23 |
| sbezverk | sdake: pvc name | 23:23 |
| *** bmaltais has quit IRC | 23:24 | |
| kfox1111 | its the first column of the input data. | 23:25 |
| kfox1111 | by default, space delimited. | 23:25 |
| sbezverk | sdake: it is also specific to awk, not to confuse with bash parameters | 23:25 |
| kfox1111 | dudes..... | 23:26 |
| kfox1111 | https://review.openstack.org/#/c/454841/ | 23:26 |
| kfox1111 | working v4 ceph multi gate ! :) | 23:26 |
| inc0 | yay kfox1111 | 23:26 |
| sbezverk | kfox1111: yeah!!! | 23:26 |
| inc0 | I'll prep ansible play to setup disks and such for ceph | 23:27 |
| inc0 | we could use it in k8s gates too | 23:27 |
| kfox1111 | inc0: why the big push to take existing working bash and rewrite in ansible? | 23:27 |
| sbezverk | kfox1111: what was final change to fix authentication? | 23:27 |
| kfox1111 | so... should we merge the ps even though it doesn't do helm-compute and helm-entrypoint v4 yet, or wait for those? | 23:28 |
| kfox1111 | sbezverk: it was uploading the nova key in the cinder name by mistake. | 23:28 |
| kfox1111 | and v2/3 was always using the nova name, not cinder. | 23:28 |
| kfox1111 | v4 always used "cinder" for the user name. | 23:28 |
| sdake | [admin@falkor-node0 oscore]$ kubectl get pvc -n kolla | grep Bound | cut -f1-2 -d ' ' | xargs -i kubectl get pv -n kolla -o jsonpath={.spec.volumeName} {} | 23:29 |
| sdake | Error from server (NotFound): persistentvolumes "glance " not found | 23:29 |
| sdake | Error from server (NotFound): persistentvolumes "mariadb " not found | 23:29 |
| sdake | Error from server (NotFound): persistentvolumes "rabbitmq " not found | 23:29 |
| inc0 | kfox1111: well, I'm just writing it from scrach in kolla-ans | 23:29 |
| sbezverk | sdake: because you deleted them before | 23:29 |
| inc0 | if you don't think it's worth taking, fair enough | 23:29 |
| inc0 | :) | 23:29 |
| kfox1111 | inc0: yeah. I know. something we like to do a lot. :/ | 23:29 |
| sdake | sbezverk the pvs are still rpesent | 23:29 |
| sdake | sbezverk the extra space is the problem | 23:29 |
| inc0 | kfox1111: I need to write it from scratch anyway | 23:30 |
| kfox1111 | why? | 23:30 |
| inc0 | and orchiestrating these things with ansible instead of ssh to node and do it just seems simpler | 23:30 |
| sbezverk | sdake: why you run kubectl get pv????? | 23:30 |
| sdake | kfox1111 i have no interest in rewriting all the gates in ansible | 23:30 |
| inc0 | well, I'll copy code from k8s ofc | 23:30 |
| sbezverk | check the command I pasted | 23:30 |
| inc0 | but execute it with ansible so I won't need to ssh to other nodes | 23:30 |
| sdake | sbezverk will check - ltitle faded today because of knee | 23:30 |
| kfox1111 | inc0: I thinhk we can just take the script and wrap it up in a container. | 23:30 |
| sdake | kfox1111 i am not keen to wrap the gate scripts in a container - however | 23:31 |
| kfox1111 | I'd much prefer stuff running in k8s rather then needing to install stuff on the system like ansible. | 23:31 |
| sdake | [admin@falkor-node0 oscore]$ kubectl get pvc -n kolla | grep Bound | cut -f1-2 -d ' ' | xargs -i kubectl get pvc -n kolla -o jsonpath={.spec.volumeName} {} | 23:31 |
| sdake | Error from server (NotFound): persistentvolumeclaims "glance " not found | 23:31 |
| sdake | Error from server (NotFound): persistentvolumeclaims "mariadb " not found | 23:31 |
| sdake | Error from server (NotFound): persistentvolumeclaims "rabbitmq " not found | 23:31 |
| sbezverk | also people use bits from gate to do their stuff | 23:31 |
| sdake | kfox1111 the ansible is being wrapped in a container | 23:31 |
| inc0 | well just to be fair, ansible needs only to be installed on deploy node;) | 23:31 |
| sbezverk | if gates are re-written in ansible | 23:32 |
| sbezverk | then it becomes dependency for other people | 23:32 |
| sdake | i am not rewriting gates in ansibe | 23:32 |
| sdake | i am writing two workflows | 23:32 |
| kfox1111 | inc0: I want to be able to deploy with just helm. | 23:32 |
| inc0 | I'm nto rewritting gates in ansible too | 23:32 |
| sdake | (just to be clear on the objective of my work) | 23:32 |
| *** kbaegis1 has quit IRC | 23:32 | |
| *** kbaegis has joined #openstack-kolla | 23:32 | |
| sbezverk | then let's not replace what we already have, but ADD to it | 23:32 |
| sdake | kfox1111 i think deploying just with helm is a good objective - although requires serious ehavy lifting | 23:33 |
| sdake | sbezverk who said anytihng about replacing? | 23:33 |
| sdake | sbezverk i am adding here - not deleting stuff | 23:33 |
| sdake | sbezverk (unless your tlaking to inc) | 23:33 |
| kfox1111 | the balancing act is, if we don't test it in gate, it breaks a lot. | 23:33 |
| kfox1111 | but its alot of work, of unknown real benifit rewriting it all in ansible. | 23:33 |
| sdake | yes, which is why i've spent the week writing the gate tests for destroy | 23:33 |
| sbezverk | sdake: inc0 mentioned replacing ceph prep bits | 23:33 |
| inc0 | but this https://review.openstack.org/#/c/466007/44/tests/templates/globals-default.j2 is much more tidy than this https://github.com/openstack/kolla-ansible/blob/master/tools/deploy_aio.sh#L80-L94 | 23:33 |
| inc0 | sbezverk: in kolla-ansible | 23:33 |
| vhosakot | Friday past 7:30pm here in Boston area, going for beer, peace pals! go for a beer and do it on Monday ;) | 23:34 |
| *** harlowja has quit IRC | 23:34 | |
| sbezverk | inc0: Oh I am sorry, I thought you meant in k8s gates | 23:34 |
| inc0 | noo | 23:34 |
| kfox1111 | inc0: I don't see them too different. its just what your use to reading. ansible or shell. | 23:34 |
| inc0 | I just said since I'm writing whole slew of plays | 23:34 |
| kfox1111 | inc0: I guess, if you write gate tests for them, then I don't really care too much. | 23:35 |
| kfox1111 | I just don't want to have to keep worrying about what I'm breaking when I code stuff in gate. | 23:35 |
| inc0 | np, well, just sayin' | 23:35 |
| kfox1111 | I want to know what I'm breaking so I can fix it. :) | 23:36 |
| *** vhosakot has quit IRC | 23:36 | |
| inc0 | in ansible next step would be to automatically pick config/additional logic per scenerio | 23:36 |
| sbezverk | kfox1111: Have you heard anything about Docker Swarm becoming more popular then k8s? I took a training and the instructor was trying to convience me that it is world wide case :) | 23:37 |
| *** iniazi_ has joined #openstack-kolla | 23:38 | |
| kfox1111 | its a shell game. | 23:38 |
| kfox1111 | since docker swarm is now called "docker", | 23:38 |
| kfox1111 | and users only know to ask for "docker" historically, | 23:38 |
| sdake | sbezverk pretty sure that instructor was high on crack | 23:38 |
| kfox1111 | they are playing "statistics" and saying people are asking for docker swarm. | 23:39 |
| kfox1111 | (and most of the instructors are hired by docker inc ;) | 23:39 |
| kfox1111 | https://trends.google.com/trends/explore?q=kubernetes,docker%20swarm,mesos | 23:39 |
| sbezverk | kfox1111: :) he must be ex docker | 23:39 |
| kfox1111 | for a slightlyl less biased answer. | 23:40 |
| *** iniazi has quit IRC | 23:40 | |
| kfox1111 | yeah. they tried to do the same to us here for "docker training" | 23:40 |
| sdake | what is wrong with this bash: [admin@falkor-node0 oscore]$ kubectl get pvc -n kolla | grep Bound | awk 'print { $1 }' | 23:40 |
| sdake | awk: cmd. line:1: print { $1 } | 23:40 |
| sdake | awk: cmd. line:1: ^ syntax error | 23:40 |
| kfox1111 | mostly that means " docker inc training" | 23:40 |
| kfox1111 | sdake: | 23:40 |
| kfox1111 | awk '{print $1}' | 23:40 |
| sdake | thanks | 23:41 |
| sdake | i guess i need to go to bed :) | 23:41 |
| kfox1111 | np. | 23:41 |
| sdake | victory: | 23:42 |
| sdake | [admin@falkor-node0 oscore]$ kubectl get pvc -n kolla | grep Bound | awk '{print $1}' | xargs -i kubectl get pvc -n kolla -o jsonpath={.spec.volumeName} {} | 23:42 |
| sdake | glancemariadbrabbitmq[admin@falkor-node0 oscore]$ | 23:42 |
| kfox1111 | cool. | 23:43 |
| *** rwellum has quit IRC | 23:43 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes master: v4 gate. https://review.openstack.org/454841 | 23:44 |
| kfox1111 | ok. this one should be good to go. :) | 23:44 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Kevins' line of debug https://review.openstack.org/466408 | 23:45 |
| sdake | kfox1111 i hope mine is ready to go too :) | 23:46 |
| *** zul has joined #openstack-kolla | 23:46 | |
| sbezverk | kfox1111: other 2 V4 gates do not use the same ceph that is why they are failing, or those are different issues? | 23:46 |
| kfox1111 | sbezverk: they need cell0 stuff integrated into the service charts. | 23:47 |
| kfox1111 | shouldn't be too hard, but extra work. | 23:47 |
| sbezverk | kfox1111: ahhh cell0, I was starting to forget about it ;) | 23:47 |
| kfox1111 | sorry to remind you. ;) | 23:47 |
| sbezverk | kfox1111: yeah it is like a wound ;) | 23:48 |
| sdake | ya cells - pretty much drives me batty | 23:48 |
| kfox1111 | especially bad when they dont really work with more then cell1. | 23:49 |
| kfox1111 | a lot of extra work with 0 possible benifit. :/ | 23:49 |
| sbezverk | kfox1111: they will figure it out around 'z' time frame ;) | 23:50 |
| kfox1111 | heh. I know, right? :/ | 23:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 23:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla-kubernetes master: Add a destroy workflow for kolla-kubernetes https://review.openstack.org/464031 | 23:54 |
| inc0 | ok, I'm calling a Friday too | 23:55 |
| inc0 | have a good weekend everyone | 23:55 |
| kfox1111 | you too. :) | 23:55 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!