| sbezverk_ | kfox1111: if ovs is used this config line must be present | 00:00 |
|---|---|---|
| *** david-lyle has quit IRC | 00:01 | |
| kfox1111 | yeah. should be handled by lines 29-31 here: https://review.openstack.org/#/c/356114/9/services/neutron/neutron-openvswitch-agent-daemonset.yml.j2 | 00:01 |
| kfox1111 | sbezverk_: oh... you haven't hooked these into the services files though. | 00:04 |
| sbezverk_ | kfox1111: nope | 00:05 |
| adnavare | sdake: yeah it is pretty strange | 00:05 |
| kfox1111 | so your patch can land at any time. but we need a follow on to actuall hook hem up. | 00:05 |
| *** zhurong has quit IRC | 00:05 | |
| sdake | adnavare makes sense to me - they probably released a new version that ONLY works on xenial | 00:05 |
| sbezverk_ | if you have cycles to hook it up, I see you are very familiar with modifying it ;-) would be great | 00:05 |
| sdake | but I am not an ubuntu dude | 00:05 |
| kfox1111 | k. I can do that. :) | 00:06 |
| sdake | fwiw it looks like kolla is dropping 14.04 support in newton as well | 00:06 |
| *** david-lyle has joined #openstack-kolla | 00:07 | |
| *** gfhellma1 has joined #openstack-kolla | 00:10 | |
| sbezverk_ | sdake: how is your tomorrow look like? need help with nova.. | 00:11 |
| *** wally__ has joined #openstack-kolla | 00:11 | |
| wally__ | @sdake, I just send you ean email | 00:11 |
| wally__ | please check | 00:11 |
| *** Jeffrey4l_ has joined #openstack-kolla | 00:12 | |
| adnavare | sdake: oh is it | 00:12 |
| adnavare | sdake: hmm then need to see how to migrate | 00:12 |
| *** gfhellma has quit IRC | 00:13 | |
| *** gfhellma1 has quit IRC | 00:13 | |
| sdake | wally__ i just responded with more detials | 00:14 |
| sdake | sbezverk_ hae time now? | 00:14 |
| sdake | sbezverk_ or need to roll | 00:14 |
| sdake | my afternoons are always beetter hten mornings | 00:14 |
| sdake | adnavare at the moment it is - and I'm not sure I like it | 00:15 |
| sdake | adnavare centos has long overlaps of working together | 00:15 |
| sdake | ubuntu doesn't though from my understnading | 00:15 |
| sdake | i.e. centos6 is STILL around and centos7 is rolling along | 00:15 |
| Pavo | sdake that network diagram you posted, does the target nodes have 3 nics or do they have 2 and you are using openstack with ovs to do external routing for internet for tenent networks? | 00:16 |
| sdake | pavo that is how i think it hsould be setup | 00:16 |
| sdake | pavo i have never actually deployed in that way | 00:17 |
| sdake | ;-) | 00:17 |
| Pavo | ah | 00:17 |
| Pavo | yeah seems ideal | 00:17 |
| sdake | ya it was for a security analysis | 00:17 |
| sdake | that is secure | 00:17 |
| adnavare | sdake: ohh then goodluck to you :) | 00:17 |
| sbezverk_ | sdake: it is a bit late here, so if you could block 30 minutes for me tomorrow afernoon that would be great | 00:18 |
| Pavo | but couldn't ovs take care of the network traffic for both external and api networks? | 00:18 |
| adnavare | sdake: It is not complaining if i get 1.8.3 though | 00:18 |
| sdake | adnavare not sure - there is probablyl an older version of docker 1.10 you can get that does work | 00:18 |
| sdake | it just may take some manual downloading | 00:19 |
| adnavare | sdake: okay | 00:19 |
| sdake | liberty/mitaka still support 14.04 | 00:19 |
| Pavo | so.... technically the deplyer node can be a VM, correct? | 00:21 |
| Pavo | deployer* | 00:21 |
| sdake | pavo sure | 00:21 |
| sdake | or a container... :) | 00:21 |
| Pavo | hmmm never thought about it being a container | 00:21 |
| sdake | ya kolla in a box | 00:21 |
| sdake | my wife says its a dumb idea | 00:21 |
| sdake | that people send out armies of folks to install openstack to deployment sites | 00:22 |
| Pavo | would be interesting to see it automatically start deploying nodes from config on host machine when you start the container | 00:22 |
| sdake | i think there is a problem with that thinking but anyway too busy to try to explore it atm :) | 00:22 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 | 00:22 |
| Pavo | well...... they is a HUGE need right now for openstack engineers mainly for deployment | 00:23 |
| wally__ | hi sdake | 00:23 |
| wally__ | I got your email | 00:23 |
| *** zhiwei has joined #openstack-kolla | 00:23 | |
| wally__ | so let me swap eth0 with eth1 and try | 00:23 |
| sdake | make sure to use destroy operation | 00:23 |
| Pavo | sdake so exactly what does kolla do? I know it uses ansible to deploy openstack services as containers, but what exactly does kolla itself do? | 00:24 |
| sdake | pavo i dont get it | 00:24 |
| Pavo | nothing clear about kolla itself in docs | 00:24 |
| sdake | you mean how does it do it? | 00:24 |
| wally__ | and no ip for the eth1 right | 00:25 |
| sdake | wally__ right | 00:25 |
| Pavo | kind of and kind of not, I know it utilizes ansible, which it seems that ansible is doing most of the work, just wondering exactly what kolla does | 00:25 |
| sdake | kolla is the name of the project | 00:25 |
| wally__ | after chanage , and then re run the kolla-ansible deploy again? | 00:26 |
| sdake | kolla combines ansible playbooks, docker jinja2 files, and a build system | 00:26 |
| Pavo | ah ok now I understand | 00:26 |
| sdake | with a cli | 00:26 |
| *** david-lyle has quit IRC | 00:26 | |
| sdake | kolla is greek for glue | 00:26 |
| Pavo | thank you, have to conduct a breif tomorrow on this project and needed exactly that | 00:26 |
| sdake | wally__ run kolla-ansible destroy | 00:26 |
| wally__ | ok, run this first | 00:26 |
| *** vhosakot has quit IRC | 00:27 | |
| Pavo | wally__ you need to destroy your first attempt befor trying to deploy another one | 00:27 |
| Pavo | is what I think sdake is trying to say | 00:27 |
| sdake | Pavo the playbooks have logic encoded in them which deploy and manage openstack | 00:27 |
| kfox1111 | Pavo: and there is a kolla-ansible which switches out some parts with kubernetes. | 00:27 |
| kfox1111 | kolla-kubernetes I mean. | 00:27 |
| wally__ | ok i will destroy and then change and then deploy again | 00:27 |
| kfox1111 | heh. about time to go home I guess. :) | 00:27 |
| wally__ | right | 00:27 |
| Pavo | sdake yeah I understood that, kfox1111 I understand that also thank you | 00:28 |
| sdake | pavo the 4 things i talked about are only part of the composition | 00:29 |
| *** v1k0d3n has quit IRC | 00:29 | |
| Pavo | was just trying to finish up some key points for Kolla, gonna be conducting a brief with some high ranking officials tomorrow and we all know how office need things broken down for them lol | 00:30 |
| sdake | pavo but the nice thing is everything is fairly decuopled | 00:30 |
| Pavo | officers* | 00:30 |
| sdake | wally__ after ou desstroy just deploy again - your configs wont be destroyed | 00:31 |
| *** duonghq has joined #openstack-kolla | 00:31 | |
| wally__ | there is kolla-ansible destory command | 00:33 |
| wally__ | no this command ? | 00:33 |
| Pavo | if you are using kolla stable no | 00:33 |
| wally__ | shit | 00:34 |
| Pavo | don't think it was added until ansible 1.9.9 correct? | 00:34 |
| wally__ | then what do u | 00:34 |
| duonghq | wally_ with Kolla stable, there are script in ./tools/ | 00:34 |
| wally__ | where | 00:34 |
| duonghq | kolla/tools | 00:34 |
| sdake | oh stable | 00:34 |
| sdake | tools/cleanup-containers | 00:34 |
| sdake | tools/cleanup-host | 00:35 |
| Pavo | or you can clone the tools directory from Kolla master to target nodes and run cleanup-containers, cleanup-hosts scripts | 00:35 |
| sdake | tools/cleanup-images | 00:35 |
| sdake | sudo reboot | 00:35 |
| sdake | pavo that is too complciated for this case ;) | 00:35 |
| Pavo | oh lol mybad | 00:35 |
| Pavo | I could never find the cleanup scripts from stable | 00:35 |
| Pavo | so thats what I did | 00:35 |
| *** v1k0d3n has joined #openstack-kolla | 00:35 | |
| *** sdake has quit IRC | 00:36 | |
| duonghq | sdake: it seems that many ops prefer kolla:kolla? | 00:36 |
| Pavo | kolla:kolla? | 00:37 |
| *** sdake has joined #openstack-kolla | 00:37 | |
| duonghq | ah, for the /etc/kolla permission | 00:37 |
| wally__ | so , run all these <sdake> tools/cleanup-containers [17:35] <sdake> tools/cleanup-host tools/cleanup-images | 00:37 |
| Pavo | containers first then images and then I would do host last | 00:37 |
| Pavo | then reboot | 00:37 |
| wally__ | ok | 00:38 |
| duonghq | Pavo: why host last but not images? | 00:38 |
| Pavo | just seems cleaner logically to me, host cleanup deletes alot of directories and restores network interfaces back to their original state | 00:39 |
| Pavo | would think deleting images before that would be cleaner | 00:39 |
| duonghq | Pavo: images is in docker local repo, so the order should not matter, IMO | 00:40 |
| Pavo | I agree | 00:40 |
| duonghq | when all container is down, the images removal is safe (IMO) | 00:41 |
| *** caowei has quit IRC | 00:41 | |
| Pavo | yeah but wouldn't removing anything logical before physical seem cleaner? | 00:41 |
| sdake | tools/cleanup-containers -f actually wally__ | 00:41 |
| Pavo | either way don't think it matters | 00:42 |
| sdake | cleanup-continers must always be done irst | 00:42 |
| duonghq | Pavo: agree | 00:42 |
| wally__ | where is the tool folder, cannot fild it | 00:48 |
| wally__ | cannot find it | 00:48 |
| sdake | wally__ you remember hwere those etc files where | 00:49 |
| sdake | /usr/local/share/kolla | 00:49 |
| wally__ | yeah | 00:49 |
| sdake | look for a tools directory there | 00:49 |
| wally__ | but there is no tools directory | 00:49 |
| sdake | or those files i nthat directory | 00:49 |
| sdake | look for those files there | 00:49 |
| sdake | wally__ bad news | 00:53 |
| sdake | wally__ 2.0.2 does not include those tools | 00:53 |
| sdake | wally__ follow these directions PRECISELY | 00:53 |
| Pavo | sdake yeah I don't think stable included those cleanup scripts | 00:53 |
| sdake | git clone http://github.com/openstack/kolla | 00:53 |
| sdake | cp -aR kolla/tools ~/kolla-tools | 00:54 |
| sdake | rm -f kolla | 00:54 |
| sdake | now in your directory kolla-tools yo should have what you need to cleanup | 00:54 |
| sdake | in your home dir | 00:54 |
| Pavo | sdake can those scripts be ran remotely? | 00:54 |
| sdake | Pavo that is what the destroy playbook doess | 00:54 |
| Pavo | as in to a target machine without having them on the host? | 00:55 |
| Pavo | yeah I understand that | 00:55 |
| sdake | pavo not that i know of | 00:55 |
| Pavo | didn't think so | 00:56 |
| Pavo | might look into building an ansible playbook for them | 00:56 |
| sdake | we have one | 00:57 |
| sdake | its called destroy | 00:57 |
| Pavo | lol yeah for master, not stable | 00:57 |
| Pavo | I meant for stable | 00:57 |
| *** huikang has joined #openstack-kolla | 00:57 | |
| *** adnavare has quit IRC | 00:59 | |
| wally__ | ok let me try | 01:00 |
| wally__ | i have to go to the lab again | 01:01 |
| sdake | pavo openstack has a no backport policy on stable branches | 01:01 |
| Pavo | ah | 01:01 |
| Pavo | makes sense | 01:01 |
| sdake | sorry no feature packbport | 01:02 |
| *** ravig has joined #openstack-kolla | 01:03 | |
| sdake | we backport high and critical fixes onlhy | 01:03 |
| *** ravig has quit IRC | 01:06 | |
| *** ravig has joined #openstack-kolla | 01:06 | |
| *** ravig has quit IRC | 01:07 | |
| *** jtriley has quit IRC | 01:08 | |
| *** JoseMello has quit IRC | 01:10 | |
| *** jtriley has joined #openstack-kolla | 01:12 | |
| *** zhurong has joined #openstack-kolla | 01:13 | |
| duonghq | Pavo: are you there? | 01:16 |
| *** jtriley has quit IRC | 01:18 | |
| *** ravig has joined #openstack-kolla | 01:23 | |
| *** jtriley has joined #openstack-kolla | 01:29 | |
| *** mark-casey has joined #openstack-kolla | 01:36 | |
| *** salv-orlando has joined #openstack-kolla | 01:40 | |
| duonghq | Pavo: did you try deploy to multinode with different NIC configuration (i.e: different name) | 01:40 |
| *** haplo37__ has quit IRC | 01:41 | |
| *** daneyon has joined #openstack-kolla | 01:41 | |
| openstackgerrit | Merged openstack/kolla: Add mariadb client binaries to kolla-toolbox https://review.openstack.org/359440 | 01:45 |
| *** daneyon has quit IRC | 01:46 | |
| *** adnavare has joined #openstack-kolla | 01:47 | |
| *** salv-orlando has quit IRC | 01:51 | |
| *** v1k0d3n has quit IRC | 01:54 | |
| *** wally__ has quit IRC | 01:56 | |
| *** caowei has joined #openstack-kolla | 02:02 | |
| *** v1k0d3n has joined #openstack-kolla | 02:03 | |
| *** vhosakot has joined #openstack-kolla | 02:06 | |
| *** phuongnh has joined #openstack-kolla | 02:08 | |
| sdake | Jeffrey4l_ gate busted? | 02:18 |
| Jeffrey4l_ | sdake, which job? | 02:19 |
| sdake | https://review.openstack.org/359440 | 02:19 |
| Jeffrey4l_ | btw i added the ubuntu binary job. so it is failed no. | 02:19 |
| Jeffrey4l_ | now | 02:19 |
| sdake | centos source | 02:19 |
| sdake | centos binary works | 02:19 |
| sdake | centos binary goesthrough extra ci copared to source releasse | 02:19 |
| *** michauds has joined #openstack-kolla | 02:20 | |
| *** unicell has quit IRC | 02:20 | |
| Jeffrey4l_ | hmm. no idea why it failed. | 02:21 |
| Jeffrey4l_ | But other source install succeed | 02:21 |
| sdake | i caught tht | 02:21 |
| sdake | i dont have access to hardware gear | 02:22 |
| sdake | did libvirt binary change versions? | 02:22 |
| Jeffrey4l_ | I will push another PS to test it. | 02:22 |
| sdake | https://bugs.launchpad.net/nova/+bug/1616240 | 02:22 |
| openstack | Launchpad bug 1616240 in OpenStack Compute (nova) "Traceback in vif.py execv() arg 2 must contain only strings" [Undecided,New] | 02:22 |
| duonghq | sdake: it seems that Kolla hasnot had anything like pre-deploy role? | 02:22 |
| sdake | may or may not be rleated | 02:23 |
| sdake | duonghq its in proces sof merging shortly actually | 02:23 |
| duonghq | sdake: sean's ones? | 02:23 |
| Jeffrey4l_ | sdake, http://logs.openstack.org/40/359440/1/check/gate-kolla-dsvm-deploy-centos-source-centos-7-nv/f155314/logs/kolla/nova/nova-compute.log | 02:23 |
| sdake | duonghq yes | 02:23 |
| sdake | yes i saw those deprecation Jeffrey4l_ | 02:24 |
| sdake | deprecations | 02:24 |
| duonghq | but sean works focus mainly con baremetal, not really general? | 02:25 |
| Jeffrey4l_ | it is not relate the bug. it is not the same error message. | 02:25 |
| *** huikang has quit IRC | 02:25 | |
| *** huikang has joined #openstack-kolla | 02:25 | |
| duonghq | sdake: this topic https://review.openstack.org/#/q/topic:kolla-host ? | 02:26 |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: DO NOT MERGE: TEST MASTER BRANCH https://review.openstack.org/326307 | 02:26 |
| sdake | duonghq yes | 02:26 |
| duonghq | thank sdake | 02:26 |
| Jeffrey4l_ | a test PS is pushed. lets check whether it will happen again. | 02:26 |
| sdake | Jeffrey4l_ have you been using the osic cluster much | 02:27 |
| Jeffrey4l_ | sdake, make some test. but not that much. | 02:28 |
| sdake | Jeffrey4l_ - need any tutoring you can provide - we are way behind schedule for where id like to be on the scale testing | 02:28 |
| sdake | i'm not even sure which scenario we are on | 02:29 |
| *** jamesdenton has joined #openstack-kolla | 02:32 | |
| *** adnavare has quit IRC | 02:35 | |
| duonghq | sdake: I need adding user and group beforce config playbook is play, but not every roles depend on common, so how can I achieved it | 02:36 |
| duonghq | I can add this task to every config.yml, but it quite messy | 02:37 |
| duonghq | the sean-k-money work is on baremetal role | 02:39 |
| *** ravig has quit IRC | 02:40 | |
| *** caowei has quit IRC | 02:40 | |
| *** caowei has joined #openstack-kolla | 02:41 | |
| *** ravig has joined #openstack-kolla | 02:41 | |
| *** ravig has quit IRC | 02:42 | |
| *** ravig has joined #openstack-kolla | 02:42 | |
| *** sdake has quit IRC | 02:46 | |
| duonghq | pbourke: are you there? does the baremetal role is mean for every host, include non-bm machine? | 02:49 |
| *** sdake has joined #openstack-kolla | 02:49 | |
| duonghq | sdake: does the baremetal role is mean for every host, include non-bm machine? | 02:49 |
| sdake | Jeffrey4l_ connected to osic vpn | 02:50 |
| sdake | duonghq not sure on that point | 02:50 |
| *** vhosakot has quit IRC | 02:50 | |
| Jeffrey4l_ | me is connecting. | 02:50 |
| *** salv-orlando has joined #openstack-kolla | 02:50 | |
| sdake | what i the ip of the deployment machine | 02:51 |
| sdake | i/is | 02:51 |
| Jeffrey4l_ | root@172.21.8.253 | 02:51 |
| *** yuanying_ has joined #openstack-kolla | 02:51 | |
| *** yuanying has quit IRC | 02:52 | |
| sdake | right just found it | 02:52 |
| duonghq | sdake: in this file: https://review.openstack.org/#/c/325631/8/ansible/inventory/multinode | 02:52 |
| sdake | nova list works | 02:52 |
| Jeffrey4l_ | sdake, can u ssh into that machine? I am blocked during `ssh ` | 02:53 |
| duonghq | seem that every host is added to baremetal role | 02:53 |
| *** yuanying has joined #openstack-kolla | 02:53 | |
| sdake | duonghq i'm sorry dude- but i dont have time at this point in time to help - i am really focused on something else atm | 02:53 |
| Jeffrey4l_ | sdake, can u ssh into that machine? I am blocked during `ssh ` | 02:53 |
| sdake | duonghq we have osic gear for another 7 or 8 days, and we need to finish using it | 02:53 |
| duonghq | sorry sdake | 02:53 |
| sdake | ya ssh works for me, make sure to connect via vpn | 02:53 |
| sdake | duonghq no problem - just unable to help for awhile - just try digging into the code base - i'm sure you will figure it out :) | 02:53 |
| *** salv-orlando has quit IRC | 02:53 | |
| duonghq | thanks | 02:54 |
| sdake | root@172.21.8.253 | 02:55 |
| sdake | wfm | 02:55 |
| sdake | Jeffrey4l_ you hae to open the vpn in a web browser tho | 02:55 |
| sdake | i have eto use safari on my end | 02:55 |
| *** yuanying has quit IRC | 02:55 | |
| *** ravig has quit IRC | 02:55 | |
| Jeffrey4l_ | sdake, yes. I can pint that ip. but can not ssh into. It happened once. no idea why it happen | 02:56 |
| *** yuanying has joined #openstack-kolla | 02:56 | |
| Jeffrey4l_ | ping | 02:56 |
| sdake | sounds like firewall issue | 02:56 |
| sdake | GFW? | 02:56 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 | 02:56 |
| *** yuanying_ has quit IRC | 02:57 | |
| *** ravig has joined #openstack-kolla | 02:57 | |
| Jeffrey4l_ | no idea. :( maybe | 02:57 |
| sdake | dont you have a vpn tunnel or something | 02:59 |
| Jeffrey4l_ | it just blocked here http://imgur.com/a/lFUqt | 02:59 |
| sdake | try restarting your vpn software (the f5 vpn softwre) | 03:01 |
| sdake | it hsould have a green light in our rowser | 03:01 |
| sdake | and be printing out data indicating data is transmitting | 03:01 |
| sdake | our/your | 03:01 |
| *** Jeffrey4l_ has quit IRC | 03:03 | |
| *** Jeffrey4l_ has joined #openstack-kolla | 03:08 | |
| sdake | Jeffrey4l_ your ip didn't change | 03:09 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Check if ansible_user is in docker group https://review.openstack.org/358432 | 03:12 |
| *** aswadr_ has joined #openstack-kolla | 03:17 | |
| *** gfhellma has joined #openstack-kolla | 03:22 | |
| sdake | Jeffrey4l_ this one looks pretty good | 03:25 |
| *** gfhellma has quit IRC | 03:25 | |
| openstackgerrit | Juan L. Negron proposed openstack/kolla: Fix ceph permissions after ansible run https://review.openstack.org/359570 | 03:26 |
| openstackgerrit | Juan L. Negron proposed openstack/kolla: Fix ceph permissions after ansible run https://review.openstack.org/359570 | 03:29 |
| *** daneyon has joined #openstack-kolla | 03:30 | |
| *** daneyon has quit IRC | 03:34 | |
| *** huikang has quit IRC | 03:40 | |
| *** huikang has joined #openstack-kolla | 03:45 | |
| *** huikang has quit IRC | 03:46 | |
| *** huikang has joined #openstack-kolla | 03:46 | |
| *** senk has joined #openstack-kolla | 03:47 | |
| *** yuanying has quit IRC | 03:48 | |
| *** yuanying has joined #openstack-kolla | 03:51 | |
| *** salv-orlando has joined #openstack-kolla | 03:58 | |
| *** jtriley has quit IRC | 04:01 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add missing container image names to build config file https://review.openstack.org/356854 | 04:04 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (2) https://review.openstack.org/359031 | 04:06 |
| *** salv-orlando has quit IRC | 04:09 | |
| *** severion has joined #openstack-kolla | 04:09 | |
| *** michauds has quit IRC | 04:09 | |
| *** v1k0d3n has quit IRC | 04:10 | |
| *** mark-casey has quit IRC | 04:11 | |
| duonghq | sean-k-mooney: are you there? | 04:12 |
| *** severion has quit IRC | 04:12 | |
| *** v1k0d3n has joined #openstack-kolla | 04:17 | |
| *** v1k0d3n has quit IRC | 04:17 | |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd container https://review.openstack.org/355156 | 04:26 |
| *** unicell has joined #openstack-kolla | 04:29 | |
| *** huikang has quit IRC | 04:30 | |
| *** unicell has quit IRC | 04:35 | |
| *** unicell has joined #openstack-kolla | 04:37 | |
| *** senk has quit IRC | 04:38 | |
| *** unicell has quit IRC | 04:43 | |
| sdake | Jeffrey4l_ what is a uec image | 04:44 |
| sdake | i heard pual say something about using the uec image | 04:44 |
| Jeffrey4l_ | no idea about the uec image sdake | 04:44 |
| Jeffrey4l_ | why use that? | 04:44 |
| sdake | i didnt ask | 04:44 |
| sdake | its not in the doc | 04:44 |
| Jeffrey4l_ | ok. i think the root cause it 0.23 is alive, which shouldn't be used now. | 04:45 |
| *** unicell has joined #openstack-kolla | 04:46 | |
| *** dave-mccowan has quit IRC | 04:49 | |
| sdake | 0.23 is alive? | 04:51 |
| sdake | you mean not alive? | 04:51 |
| sdake | or 0.23 is broken | 04:51 |
| sdake | 0.23 of which - the osic machine? | 04:52 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 | 04:53 |
| sdake | glance just lock up? | 04:53 |
| Jeffrey4l_ | 0.23 shouldn't be used. the p4p1 have some issue. | 04:53 |
| Jeffrey4l_ | sdake, wait, uploading image. | 04:54 |
| sdake | take it out - lets clean up and redeploy | 04:54 |
| *** salv-orlando has joined #openstack-kolla | 04:54 | |
| Jeffrey4l_ | sdake, no need. | 04:54 |
| sdake | why not? | 04:54 |
| Jeffrey4l_ | the vip is on the 0.23. | 04:54 |
| Jeffrey4l_ | remove it. it works now. | 04:54 |
| sdake | so bug in what then? | 04:54 |
| Jeffrey4l_ | vip exist on 0.23 and one of (0.24, 0.25, 0.26) | 04:55 |
| Jeffrey4l_ | i deploy using 0.24/5/6 0.23 has some uncleaned data. | 04:56 |
| Jeffrey4l_ | so after cleaned it, it works now | 04:56 |
| sdake | ok | 04:56 |
| sdake | so user error | 04:56 |
| sdake | if we rebooted the boxes for exampel - this problem wouldn't occur | 04:57 |
| sdake | can you produce tempesst resultss for scenario 2? | 04:57 |
| Jeffrey4l_ | sdake, no. it will happend. the container on 0.23 is not cleaned. | 04:57 |
| Jeffrey4l_ | after rebooted, it happend again. | 04:57 |
| sdake | i see | 04:58 |
| Jeffrey4l_ | sdake, trying to make the dd test works. | 04:58 |
| sdake | wfm | 04:58 |
| sdake | sorry hae alot of things going throug my brain atm i'd like to run through | 04:59 |
| openstackgerrit | Steven Dake proposed openstack/kolla: Add OSIC Scale Testing Documentation https://review.openstack.org/352101 | 05:02 |
| sdake | order will be ubuntu 14.04 -> centos 7.2 source - > ubuntu 16.04 xenial | 05:03 |
| sdake | since newton wont release with 14.04 | 05:03 |
| sdake | 14.04 results not all that useful | 05:03 |
| Jeffrey4l_ | sdake, so just test the deployment process? | 05:03 |
| sdake | lets stick with 14.04 for now | 05:04 |
| sdake | i don't want to repave atm | 05:04 |
| sdake | lets get our test cases and documentaiton format sorted out | 05:04 |
| sdake | we can burn through each distro in a day | 05:04 |
| sdake | once we have the basic stuff we want to collect sorted out | 05:04 |
| Jeffrey4l_ | sdake, do u still on the osic node? | 05:05 |
| Jeffrey4l_ | sdake, check the dd test result i made. | 05:05 |
| sdake | i am looking now | 05:05 |
| Jeffrey4l_ | sdake, the test report could get from /root/lei/output.html | 05:05 |
| sdake | or was looking | 05:05 |
| sdake | isn't it on the screen? | 05:06 |
| sdake | Command output | 05:06 |
| sdake | +-----------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+ | 05:06 |
| sdake | | Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Count | | 05:06 |
| sdake | +-----------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+ | 05:06 |
| sdake | | read_seq_5000m | 9.3 | 9.965 | 10.281 | 10.555 | 10.83 | 5.474 | 10 | | 05:06 |
| sdake | | write_seq_5000m | 28.16 | 34.53 | 34.889 | 34.93 | 34.97 | 17.001 | 10 | | 05:06 |
| sdake | +-----------------+-----------+--------------+--------------+--------------+-----------+-----------+-------+ | 05:06 |
| Jeffrey4l_ | sdake, it is. But the html has more data. like each instance data. | 05:06 |
| *** ibmchas has joined #openstack-kolla | 05:06 | |
| sdake | ok -w e can record that in json | 05:07 |
| sdake | and sort out how to get it out later | 05:07 |
| sdake | is dd good enough? | 05:07 |
| Jeffrey4l_ | sdake, no. dd is bad. fio/bonnie should be use. | 05:07 |
| sdake | ok lets do bonnie++ = standard benchmark tool for file io | 05:07 |
| sdake | dd good for block i/o i think | 05:08 |
| sdake | what does fio do? | 05:08 |
| Jeffrey4l_ | we need test 4k randwrite | 05:08 |
| Jeffrey4l_ | fio is the same tool. I used that tool | 05:08 |
| sdake | there is hundreds of testss in rally | 05:09 |
| sdake | why are only 6 or 7 showing up there? | 05:09 |
| Jeffrey4l_ | sdake, where? | 05:10 |
| sdake | +-----------------------------------------------------------------------------------------------------------------------------+ | 05:10 |
| sdake | | Response Times (sec) | | 05:10 |
| sdake | +--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ | 05:10 |
| sdake | | Action | Min (sec) | Median (sec) | 90%ile (sec) | 95%ile (sec) | Max (sec) | Avg (sec) | Success | Count | | 05:10 |
| sdake | +--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ | 05:10 |
| sdake | | nova.boot_server | 19.229 | 21.074 | 28.605 | 29.225 | 29.844 | 22.87 | 100.0% | 10 | | 05:10 |
| sdake | | vm.run_command_over_ssh | 38.099 | 45.354 | 45.694 | 45.718 | 45.741 | 43.721 | 100.0% | 10 | | 05:10 |
| sdake | | cinder.create_volume | 2.534 | 2.711 | 2.979 | 3.026 | 3.074 | 2.749 | 100.0% | 10 | | 05:10 |
| sdake | | vm.wait_for_ssh | 2.91 | 3.504 | 4.414 | 6.091 | 7.767 | 3.865 | 100.0% | 10 | | 05:10 |
| sdake | | nova.force_delete_server | 4.53 | 4.632 | 4.84 | 4.846 | 4.852 | 4.659 | 100.0% | 10 | | 05:10 |
| sdake | | vm.wait_for_ping | 8.033 | 8.038 | 8.14 | 8.59 | 9.04 | 8.137 | 100.0% | 10 | | 05:10 |
| sdake | | vm.attach_floating_ip | 1.463 | 1.745 | 1.862 | 1.901 | 1.939 | 1.712 | 100.0% | 10 | | 05:10 |
| sdake | | vm.delete_floating_ip | 1.225 | 1.374 | 1.447 | 1.475 | 1.502 | 1.362 | 100.0% | 10 | | 05:10 |
| sdake | | total | 81.32 | 89.066 | 95.896 | 96.407 | 96.918 | 89.12 | 100.0% | 10 | | 05:10 |
| sdake | +--------------------------+-----------+--------------+--------------+--------------+-----------+-----------+---------+-------+ | 05:10 |
| sdake | those^^ | 05:10 |
| Jeffrey4l_ | sdake, i just run one test: boot vm, test the io with dd, | 05:10 |
| sdake | that must have been an old test run then? | 05:11 |
| sdake | we need to be able to charactize what we are benchmarking ;) | 05:11 |
| Jeffrey4l_ | sdake, yes. | 05:11 |
| sdake | having numberes without what makes them is not helpful | 05:11 |
| Jeffrey4l_ | sdake, i do not think we need run all the rally tests. | 05:12 |
| sdake | the problem is then which ones do we run vs not run? | 05:12 |
| Jeffrey4l_ | like: create-user-and-list is meaningless for kolla. | 05:12 |
| Jeffrey4l_ | sdake, the only two i can tell is: boot-and-delete-server and boot-and-io-test (with custom value ) | 05:13 |
| Jeffrey4l_ | i think | 05:13 |
| sdake | that tests out the performance of openstack that kolla deploys | 05:13 |
| sdake | cinder.create_volume seems interesting | 05:13 |
| sdake | vm.wait_for-ssh seems interesting | 05:13 |
| Jeffrey4l_ | maybe need some help for guys | 05:13 |
| sdake | wait or ping | 05:13 |
| sdake | maybe we can caputre it all | 05:13 |
| sdake | an dfilter it later | 05:14 |
| Jeffrey4l_ | create_volume count | 05:14 |
| Jeffrey4l_ | sdake, yes. | 05:14 |
| sdake | the problem is the defaults are too low for 100 nodes ;) | 05:14 |
| sdake | what is boot-and-delete-server | 05:15 |
| sdake | is that the 300 vm boot and delete cusstom tet? | 05:15 |
| Jeffrey4l_ | and we need test this will in continuous. | 05:15 |
| Jeffrey4l_ | sdake, we can combine it. | 05:15 |
| sdake | can combine what | 05:15 |
| sdake | so here is my thinking | 05:16 |
| sdake | the rally cats went to alot of trouble to create baseline scenarios | 05:16 |
| sdake | lets use those, but jut amp up the numbers a notch or two so they are measurable | 05:16 |
| sdake | and create a custom bonnie++ test and submit to rally upstream | 05:16 |
| Jeffrey4l_ | (combine the details of the test process for one job.) | 05:16 |
| sdake | bonnie++ will be huge indicator of performance - i/o is always the laggard in any system ;) | 05:17 |
| Jeffrey4l_ | yes. bonnie++ is needed. | 05:17 |
| sdake | ok - so we have two options | 05:18 |
| sdake | a) try to sort out a bonnie integration now | 05:18 |
| sdake | or b) try to sort out the default valules to capture | 05:18 |
| Jeffrey4l_ | for the boot server test: we need boot 100 server, 200 server 400 server and 1000 server with different concurrency. ( which is continuous.) | 05:18 |
| sdake | is that built in? | 05:19 |
| sdake | i want to limit number of custom tests we make | 05:19 |
| sdake | we only hve 7 days | 05:19 |
| Jeffrey4l_ | sdake, need some small work to implement this. | 05:19 |
| Jeffrey4l_ | sdake, just like this /tmp/boot-and-delete.yaml test does | 05:19 |
| Jeffrey4l_ | a) b) are two option or two must? | 05:20 |
| sdake | ok so continuous boot and delete is one custom ttest | 05:20 |
| sdake | bonnie++ is another custom test | 05:20 |
| sdake | any other custom tests? | 05:21 |
| Jeffrey4l_ | create_volume you mentioned above | 05:21 |
| sdake | its already implemented | 05:21 |
| Jeffrey4l_ | yes. | 05:21 |
| Jeffrey4l_ | But need some custom too. | 05:21 |
| Jeffrey4l_ | like create 100, 200 300 volumes at the same time. | 05:21 |
| sdake | custom for what purpose | 05:21 |
| Jeffrey4l_ | create_volume | 05:21 |
| Jeffrey4l_ | just like the boot server | 05:22 |
| sdake | 7 days - need 4 to run through test cases and that is cutting it close | 05:22 |
| sdake | lets allocate 2 days to sorting out the correct values for rally | 05:22 |
| Jeffrey4l_ | boonie++ is the hardest one. :( | 05:22 |
| sdake | and making custom tests | 05:22 |
| Jeffrey4l_ | others are just change the yml file | 05:23 |
| sdake | i'd rather have correct values or bonnie then custom tests personally :) | 05:23 |
| sdake | yes I saw that | 05:23 |
| *** ssurana1 has quit IRC | 05:23 | |
| sdake | rather I'd rather have correct values for rally then custom tests | 05:23 |
| sdake | I think a correct value is something that produces a result over 30 seconds | 05:23 |
| sdake | so its something measurable | 05:24 |
| sdake | but doens't tke forever to run | 05:24 |
| sdake | then the json can print out which vars we ran with | 05:24 |
| Jeffrey4l_ | the correct value mean: the rally should have the correct value for easy use? | 05:24 |
| sdake | god willing | 05:24 |
| sdake | correct value = 30 seconds for avg runtime for a testcase | 05:24 |
| sdake | with a good balance on the parameters | 05:25 |
| Jeffrey4l_ | for the IO test, 30 seconds is not enough. | 05:25 |
| sdake | greater then 30 seconds is fine | 05:25 |
| sdake | minimum 30 seconds | 05:25 |
| Jeffrey4l_ | OK. we can adjust it. it is easy. | 05:25 |
| sdake | so lets do that first | 05:25 |
| sdake | then do custom next | 05:25 |
| Jeffrey4l_ | yep. | 05:26 |
| sdake | wow tired | 05:26 |
| sdake | ok lets get cracking :) | 05:26 |
| sdake | what is running now? | 05:26 |
| Jeffrey4l_ | I will try to implement the bonnie++ script first | 05:26 |
| Jeffrey4l_ | /tmp/boot-and-delete.yaml this task | 05:26 |
| sdake | so you want to do custom test cases firt then rally optimization second? | 05:27 |
| Jeffrey4l_ | just make a test. | 05:27 |
| *** senk has joined #openstack-kolla | 05:27 | |
| Jeffrey4l_ | rally optimization second is easy, just change the value, like how many instance do you want to launch for test. | 05:27 |
| *** unicell has quit IRC | 05:28 | |
| Jeffrey4l_ | sdake, now we only have 3 testcase, so the running seconds is not a big deal | 05:28 |
| sdake | rally has hundreds of tets cases iiuc | 05:28 |
| *** unicell has joined #openstack-kolla | 05:28 | |
| sdake | atleat from that git diff | 05:28 |
| sdake | i want to capture that data | 05:28 |
| sdake | even if we dont present it ddirectly | 05:28 |
| Jeffrey4l_ | sdake, we only test: 1. create servers 2. io test 3 create_volume | 05:29 |
| Jeffrey4l_ | isn't we? | 05:29 |
| Jeffrey4l_ | aren't we? | 05:29 |
| sdake | i'd like ot see all the rally tests run as well | 05:29 |
| sdake | and highlight our special teseting of what we think is important | 05:29 |
| Jeffrey4l_ | we really do not need run all the test result. | 05:29 |
| Jeffrey4l_ | hmm. if we run all the test, then we get one conclusion: kolla works. | 05:30 |
| sdake | there is already a create_vollume test in the raly repo | 05:30 |
| Jeffrey4l_ | and the pbourke's script does this already. | 05:30 |
| sdake | Jeffrey4l_ yup you got it - it works at scale under load | 05:31 |
| sdake | which script? | 05:31 |
| Jeffrey4l_ | sdake, /root/rally.git/kolla-rally-tests.sh | 05:32 |
| sdake | ok openplz | 05:32 |
| Jeffrey4l_ | opened on osic node | 05:32 |
| Jeffrey4l_ | see? | 05:32 |
| Jeffrey4l_ | it should work | 05:32 |
| sdake | yup | 05:33 |
| sdake | can we run quickly | 05:33 |
| sdake | iwant to see what minimum time is | 05:33 |
| sdake | on a test | 05:33 |
| sdake | what is all-scenarios.txt? | 05:33 |
| Jeffrey4l_ | sdake, opened | 05:33 |
| sdake | looking good | 05:33 |
| Jeffrey4l_ | sdake, there should no such option to set the minimum time. | 05:34 |
| sdake | wonder how far pual is for this | 05:34 |
| sdake | Jeffrey4l_ we can crank up the numbers to 1k iterations and that will increase minimum time | 05:34 |
| sdake | if we measure 0.00001 seconds for 100 iterations not all that helpful :) | 05:34 |
| Jeffrey4l_ | sdake, we need change the `times` value in scrip to adjust the running time. | 05:35 |
| sdake | looks like this file is good to go | 05:35 |
| Jeffrey4l_ | in the txt file, it is all the rally jobs. | 05:35 |
| sdake | Jeffrey4l_ right -that is the number of iterations | 05:35 |
| Jeffrey4l_ | yes. | 05:35 |
| sdake | lets find out the minimum time first | 05:35 |
| sdake | that will give us guidance on what to set the max value to | 05:35 |
| openstackgerrit | Merged openstack/kolla: Add missing \n at end of file https://review.openstack.org/359443 | 05:35 |
| sdake | how do you run the scenarios pual has pointed out - that script earlier you had loaded? | 05:36 |
| Jeffrey4l_ | sdake for the functional test, i prefer to use tempest. it can test more api interface. | 05:36 |
| sdake | Jeffrey4l_ we iwll do tempest as well | 05:37 |
| sdake | from what i can tell temest sort of works out of the box for us | 05:37 |
| sdake | lets run pauls' script and see what happens? | 05:37 |
| Jeffrey4l_ | sdake, never ran the script. but i think just ./script should work | 05:37 |
| Jeffrey4l_ | OK | 05:37 |
| sdake | lets look at it first | 05:37 |
| sdake | make sure it wont crater his data ;) | 05:37 |
| Jeffrey4l_ | sdake, need ctrl+b ] | 05:39 |
| Jeffrey4l_ | to scroll up and down | 05:39 |
| sdake | thanks | 05:39 |
| *** harlowja_at_home has joined #openstack-kolla | 05:39 | |
| sdake | goign to run in script | 05:39 |
| sdake | moemnt | 05:39 |
| Jeffrey4l_ | ok | 05:39 |
| *** ravig has quit IRC | 05:39 | |
| sdake | bunch o ffial not responding to ctrl-c | 05:40 |
| sdake | the rally tests should be taking 1 hr minimum i htink | 05:42 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Fix bug in Elasticsearch role https://review.openstack.org/359596 | 05:42 |
| sdake | if there is 100 test cases at 30 seconds a case | 05:42 |
| sdake | lets look at typescript when it wraps up | 05:42 |
| sdake | concurrency in rally is what? | 05:43 |
| sdake | how many threads it runs at once? | 05:43 |
| Jeffrey4l_ | sdake, one by one. | 05:43 |
| Jeffrey4l_ | one test case by one test case | 05:44 |
| sdake | i dont understand | 05:44 |
| Jeffrey4l_ | in certain one test case, the concurrency value control the threads | 05:44 |
| Jeffrey4l_ | sdake, each file is a test case samples/tasks/scenarios/keystone/create-and-delete-service.yaml | 05:45 |
| sdake | yes i saw that | 05:45 |
| Jeffrey4l_ | the concurrency=25 , for one test case, there are 25 threads | 05:46 |
| Jeffrey4l_ | sdake, the gate is green https://review.openstack.org/326307 | 05:48 |
| sdake | good news | 05:48 |
| sdake | :) | 05:48 |
| openstackgerrit | Swapnil Kulkarni (coolsvap) proposed openstack/kolla: Update Dockerfiles for footer related changes https://review.openstack.org/357746 | 05:49 |
| *** coolsvap has joined #openstack-kolla | 05:49 | |
| sdake | coolsvap we are banging on osic | 05:50 |
| sdake | care to join | 05:50 |
| coolsvap | sdake: just a min | 05:50 |
| sdake | coolsvap we are creating some custom tests | 05:51 |
| sdake | a bonnie++ test for rally | 05:51 |
| sdake | and a server boot test for rally | 05:51 |
| sdake | and running the rest as pauls (or yours?) script sets them | 05:51 |
| sdake | root@729494-comp-s3500-002:/openstack# bonnie++ -d /var/lib/nova -s 520G -n 0 -m TEST -f -b -u root | 05:59 |
| Jeffrey4l_ | ok | 06:00 |
| sdake | /var/lib/nvoa is the scratch dir | 06:00 |
| sdake | -s 520g should be double ram | 06:00 |
| sdake | not sure how your going to get bonnie on those images | 06:00 |
| Jeffrey4l_ | sdake, we can install after booted. Or use a pre-installed image. | 06:02 |
| sdake | cirros doesn't hve a pkg manager does it? | 06:02 |
| Jeffrey4l_ | sdake, yes. We can use centos image. | 06:02 |
| Jeffrey4l_ | i have uploaded it. | 06:02 |
| sdake | i wonder how much of a pita it would be to use centos for all rally testing | 06:03 |
| sdake | i am not keen on cirros as a benchmark platform | 06:03 |
| sdake | if rally can run concurrent jobs - might s well try it out | 06:03 |
| sdake | : Error OverLimit: VolumeLimitExceeded: Maximum number of volumes allowed (10) exceeded for quota 'volumes'. (HTTP 413) (Request-ID: req-82ba0c61-e29e-4934-8c23-b9e92f2c32a4) | 06:04 |
| sdake | 2016-08-24 01:04:04.376 13037 INFO rally.task.runner [-] Task 509cb7a7-df60-476e-a242-88e510005057 | ITER: 100 EN | 06:04 |
| Jeffrey4l_ | sdake, job need be change. add a quota context | 06:04 |
| sdake | les look for errors after this job finishes | 06:04 |
| sdake | its in a script session | 06:04 |
| Jeffrey4l_ | sdake, rally is a performance tool. So running jobs in concurrent is not a good idea. | 06:05 |
| coolsvap | sdake: we are changing concurrency in rally tests | 06:05 |
| sdake | coolsvap the goal is to get ot a minimum runtime per test of 30seconds | 06:06 |
| coolsvap | the scenarios have option to increase the load and concurrency | 06:06 |
| sdake | by balancing out iterations and concurrency | 06:06 |
| sdake | we think we can do this by doubling the number of iterations | 06:07 |
| sdake | concurrency is # of threads running concurrently coolsvap ? | 06:07 |
| sdake | we have 24 cores, should probably be running 24 concurrent jobs | 06:08 |
| coolsvap | sdake: are you talking about different rally threads? | 06:08 |
| sdake | coolsvap i am talking about the concurrency field in the yaml files | 06:08 |
| coolsvap | the concurrency is concurrent requests, say in nova case if the concurrency is 25, then 25 instance creation requests | 06:09 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Fix bug in Elasticsearch role https://review.openstack.org/359596 | 06:10 |
| sdake | so 24, 48, 72? | 06:10 |
| sdake | that shows min load, hyperthread load, and overload ;) | 06:10 |
| sdake | ok thiinking about structure of document here - wrt time left to test | 06:11 |
| sdake | lets crank up iterations to 300, and run 24, 48, and 96 concurrency tests | 06:12 |
| sdake | for each scenario | 06:12 |
| sdake | feasible or not feasible? | 06:12 |
| sdake | coolsvap ^6 | 06:12 |
| coolsvap | i think it should be feasible | 06:13 |
| *** mdnadeem_ has joined #openstack-kolla | 06:13 | |
| sdake | ok so Jeffrey4l_ is working on a bonnie++ test scenario | 06:14 |
| sdake | and a rolling boot and delete server test | 06:14 |
| sdake | thee will be our custom test cases | 06:14 |
| *** egonzalez90 has joined #openstack-kolla | 06:14 | |
| egonzalez90 | o/ morning | 06:14 |
| sdake | sup egonzalez90 | 06:14 |
| sdake | egonzalez90 we are working on osic atm, your welcome to join if you want | 06:15 |
| sdake | but i think we have enough epople power on it atm | 06:15 |
| sdake | coolsvap is there a way to abort a job in progress in rally? | 06:16 |
| *** mdnadeem has quit IRC | 06:17 | |
| egonzalez90 | sadly I can't join atm, im at the office | 06:18 |
| sdake | nice zombie process collection :) | 06:20 |
| coolsvap | sdake: did you start new task? | 06:20 |
| sdake | no | 06:21 |
| sdake | its running puals' script | 06:21 |
| sdake | or yours | 06:21 |
| sdake | not sure who the author is | 06:21 |
| egonzalez90 | sdake, i have a technical debt. I'm working on serialconsole for nova, and the only way I found is binding serialproxy_host at 0.0.0.0. I know is a security problem | 06:21 |
| duonghq | morning egonzalez90 | 06:21 |
| egonzalez90 | morning duonghq | 06:21 |
| sdake | there we go | 06:23 |
| sdake | how do we make shell accept ctrl-c ;) | 06:23 |
| sdake | ok 96*3=288 | 06:23 |
| sdake | so lets try that out | 06:23 |
| coolsvap | there is no option for that unfortunately | 06:23 |
| sdake | and see wht happens | 06:23 |
| coolsvap | alright lets se | 06:24 |
| sdake | 24 concurrency | 06:24 |
| sdake | 288 iterations | 06:24 |
| duonghq | egonzalez90: can you review my ps https://review.openstack.org/#/c/359596/ | 06:24 |
| sdake | lets disable all broken teste casess now | 06:26 |
| sdake | or fix the quotas | 06:27 |
| sdake | in the case of these keystone tests, some of thm probably onl ywork wwith v2 | 06:27 |
| sdake | i am in vi btw :) | 06:27 |
| sdake | pbourke has a file with disabled test casess | 06:28 |
| sdake | lets just fix that up first | 06:28 |
| *** zhubingbing has joined #openstack-kolla | 06:30 | |
| zhubingbing | hello | 06:31 |
| duonghq | hello | 06:31 |
| sdake | Volume LimitExeceeded | 06:31 |
| sdake | i think we need to increase quotas here | 06:32 |
| sdake | groan | 06:36 |
| sdake | nee dto do tha for all cinder cses? | 06:36 |
| *** senk has quit IRC | 06:36 | |
| Jeffrey4l_ | sdake, yes. | 06:36 |
| sdake | i think there must be an override - the sh script that is there overrides | 06:36 |
| Jeffrey4l_ | do not think so. | 06:38 |
| sdake | what do we ddo with this particular file | 06:39 |
| coolsvap | sdake: what if we do it in the service rather than doing it in scripts | 06:39 |
| sdake | coolsvap expand? you mean using the cli? | 06:39 |
| coolsvap | yes | 06:40 |
| sdake | as long sa it is documented | 06:40 |
| coolsvap | yeah we need as many resources created as possible | 06:40 |
| coolsvap | we can unblock quotas | 06:40 |
| Jeffrey4l_ | coolsvap, the test is ran under new user/project. we need change the default quota, which may work | 06:42 |
| bjolo | morning | 06:42 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 | 06:42 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 | 06:43 |
| sdake | google would hepfu lhere | 06:43 |
| coolsvap | Jeffrey4l_: ? | 06:43 |
| coolsvap | sdake: hmm | 06:43 |
| sdake | i've been up since 6am and its midnight | 06:43 |
| *** harlowja_at_home has quit IRC | 06:44 | |
| *** bootsha has joined #openstack-kolla | 06:44 | |
| sdake | and i plan to stya up another 8-10 hr ;) | 06:44 |
| *** fragatina has quit IRC | 06:44 | |
| sdake | so if I could get some help along the way i'd super appreciate it ;) | 06:44 |
| sdake | http://docs.openstack.org/admin-guide/cli-cinder-quotas.html | 06:45 |
| sdake | quota-defaults shoudl do it | 06:45 |
| sdake | need cinder clinet | 06:45 |
| *** fragatina has joined #openstack-kolla | 06:46 | |
| sdake | dhttp://docs.openstack.org/ops-guide/ops-quotas.html | 06:50 |
| *** ibmchas has quit IRC | 06:50 | |
| duonghq | thank egonzalez90 | 06:51 |
| *** fragatina has quit IRC | 06:51 | |
| *** Mr_Broken has joined #openstack-kolla | 06:51 | |
| coolsvap | finally | 07:00 |
| coolsvap | we are actually uncovering different problems :/ | 07:01 |
| sdake | we had another broken keystone test | 07:01 |
| sdake | lets let this finish | 07:01 |
| sdake | and look ath te typescript | 07:01 |
| sdake | and fix allthe errors | 07:01 |
| sdake | don't press ctrl-c | 07:01 |
| coolsvap | yes | 07:01 |
| sdake | all those backtraces were rom my ctrl-cs :) | 07:01 |
| coolsvap | might be lets see | 07:02 |
| *** daneyon has joined #openstack-kolla | 07:07 | |
| *** coolsvap is now known as coolsvap_ | 07:09 | |
| *** zhubingbing has quit IRC | 07:09 | |
| *** daneyon has quit IRC | 07:11 | |
| sdake | coolsvap_ how long does rally typically take - about an hour? | 07:13 |
| *** zhubingbing has joined #openstack-kolla | 07:13 | |
| sdake | still getting volume limit exceeded | 07:20 |
| sdake | Jeffrey4l_ can you fix up the cinder exmaples o have the correct quotas | 07:20 |
| *** dmueller-evoila has joined #openstack-kolla | 07:21 | |
| Jeffrey4l_ | sdake, changing the default quota not work? | 07:21 |
| sdake | no | 07:21 |
| Jeffrey4l_ | so we need change all the yaml files.. | 07:22 |
| Jeffrey4l_ | sdake, how about write a script to load all the yaml file and append the quota to all files? | 07:24 |
| sdake | groan | 07:24 |
| sdake | i think the issue is the syntax in some of the files wont work in that model | 07:25 |
| sdake | and manual edit of cinder dir probaby faster | 07:25 |
| sdake | i just dont now where the quota sstuff goes | 07:25 |
| Jeffrey4l_ | sdake, all file support that. | 07:25 |
| sdake | in relationship to the other yamlz | 07:25 |
| sdake | need snapshots unlimited as well | 07:25 |
| *** gfidente has joined #openstack-kolla | 07:35 | |
| *** athomas has joined #openstack-kolla | 07:37 | |
| sdake | well that sucked | 07:40 |
| sdake | ok looks like snapshots may be bust | 07:40 |
| sdake | i think that fialed because we were out of vollumes | 07:43 |
| sdake | after this wraps up, i'm going to hunt down thosee keystonefailures reove them | 07:43 |
| sdake | and give it another spin | 07:43 |
| Jeffrey4l_ | sdake, i fixed one | 07:46 |
| sdake | nice | 07:46 |
| sdake | i put quotas i all the files | 07:46 |
| sdake | but possibly in the wrong place ;) | 07:46 |
| Jeffrey4l_ | sdake, 1. syntax error 2. there are some yamls has two task. | 07:47 |
| sdake | Jeffrey4l_ can you go throug hthe files nd fix them all | 07:47 |
| sdake | i probably broke ehalf of them | 07:47 |
| sdake | only needed in cinder dir | 07:47 |
| Jeffrey4l_ | ok | 07:47 |
| sdake | i now know more about rally then i ever needed to know | 07:47 |
| sdake | need a tty for that Jeffrey4l_ | 07:50 |
| sdake | xargs wont give you one iirc | 07:50 |
| duonghq | egonzalez90: it seems that gate failed on binary-ubuntu, due to missing binary for ubuntu | 07:50 |
| sdake | off by on errors - ttired :( | 07:51 |
| sdake | binary ubuntu gate is broken atm | 07:52 |
| sdake | if folks wnat ot fix it - that would be great :) | 07:52 |
| Jeffrey4l_ | sdake, https://review.openstack.org/358488 here | 07:52 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 07:53 |
| duonghq | oops | 07:53 |
| sdake | needs a bug id - i'll open one for you | 07:53 |
| zhubingbing | hi | 07:53 |
| *** Serlex has joined #openstack-kolla | 07:54 | |
| openstackgerrit | Dennis Mueller proposed openstack/kolla: Added neutron_metering_agent to ansible tasks for neutron https://review.openstack.org/346630 | 07:54 |
| Jeffrey4l_ | ok | 07:54 |
| duonghq | Jeffrey4l_: not only designate failed 'cause of binary missing | 07:54 |
| sdake | Jeffrey4l_ lets get ot binary gate later | 07:55 |
| sdake | its not critical at this time | 07:55 |
| Jeffrey4l_ | duonghq, the gate is green now. check https://review.openstack.org/358488 | 07:55 |
| duonghq | Jreffrey4l_: it did not run the kuryr build | 07:57 |
| duonghq | http://logs.openstack.org/96/359596/2/check/gate-kolla-dsvm-build-ubuntu-binary-ubuntu-xenial-nv/a0538aa/console.html | 07:57 |
| duonghq | for example | 07:57 |
| sdake | Jeffrey4l_ the reason it needs a bug id is because its to be bckported | 07:58 |
| duonghq | but I agree with sdake, source build is ok so it good enough atm, seems that we have many tasks pending | 07:58 |
| sdake | duonghq that is now in the excluded imae list | 07:58 |
| egonzalez90 | is intended to not running kuryr build, kuryr dont have binary packages yet https://github.com/openstack/kolla/blob/master/docker/kuryr/Dockerfile.j2 | 07:58 |
| Jeffrey4l_ | sdake, no need backport .mitaka ubuntu binary is green now. | 07:58 |
| Jeffrey4l_ | I fixed that. | 07:58 |
| sdake | fixes should go to master first | 07:58 |
| sdake | but ok wasn't aware of that | 07:58 |
| sdake | is librty green? | 07:59 |
| Jeffrey4l_ | libery? no idea. there are lots of red on liberty. | 07:59 |
| Jeffrey4l_ | sdake, in that day, we haven't ubuntu binary gate for master. | 07:59 |
| sdake | right | 07:59 |
| sdake | i recall | 07:59 |
| sdake | probably need to backport to liberty | 08:00 |
| sdake | backports -> later | 08:00 |
| sdake | osic -> now :) | 08:00 |
| Jeffrey4l_ | sdake, much quota should be added. | 08:01 |
| sdake | what do you mean | 08:01 |
| duonghq | sdake: do you mean nv list or totally exclude? | 08:01 |
| Jeffrey4l_ | sdake, check the current file. | 08:01 |
| Jeffrey4l_ | it will create 72 instances. | 08:01 |
| Jeffrey4l_ | the nova default quota is 10 | 08:01 |
| Jeffrey4l_ | duonghq, non-voting | 08:02 |
| sdake | duonghq binary has an exlusion list in the build tet scripts | 08:02 |
| duonghq | sdake: thanks | 08:02 |
| sdake | Jeffrey4l_ roger | 08:04 |
| *** matrohon has joined #openstack-kolla | 08:04 | |
| sdake | we shoudl just create a diff at the end and publish with our docs ;-) | 08:04 |
| Jeffrey4l_ | sdake, that will be a plenty of works... i will try the default quota again. it should work | 08:06 |
| *** matrohon has quit IRC | 08:06 | |
| sdake | init-runonce gets the nova quotas right i think | 08:06 |
| Jeffrey4l_ | sdake, that is for admin user. not the default | 08:06 |
| sdake | Jeffrey4l_ i thought of this | 08:08 |
| sdake | but thought it would require a reconfigure | 08:08 |
| Jeffrey4l_ | sdake, yes. nova default quota need changed by nova.conf at least. | 08:08 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 08:08 |
| Jeffrey4l_ | so need a reconfiguration. | 08:08 |
| Jeffrey4l_ | wip | 08:09 |
| sdake | and neutron ports | 08:09 |
| sdake | reconfig may be busted in master - may haveto deploy again | 08:09 |
| sdake | guess we will find out | 08:09 |
| Jeffrey4l_ | sdake, no reconfigure work. we tried | 08:09 |
| sdake | set to -1 | 08:09 |
| sdake | -1 = no quota | 08:09 |
| Jeffrey4l_ | hmm.. ok | 08:10 |
| *** matrohon has joined #openstack-kolla | 08:10 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 | 08:11 |
| *** dmueller-evoila has quit IRC | 08:13 | |
| openstackgerrit | Jeffrey Zhang proposed openstack/kolla: Fix Ubuntu binary build and deploy gate https://review.openstack.org/358488 | 08:14 |
| sdake | Jeffrey4l_ how do i just run the keystone rally tests? | 08:18 |
| Jeffrey4l_ | sdake, find the yml file you want to run | 08:18 |
| sdake | got that one | 08:19 |
| sdake | add and remove role | 08:19 |
| sdake | maybe thats it | 08:19 |
| sdake | sad no keystone v3 support | 08:19 |
| Jeffrey4l_ | sdake, can not see... | 08:20 |
| openstackgerrit | zhubingbing proposed openstack/kolla: ≈Add Barbican container dockerfile https://review.openstack.org/351822 | 08:20 |
| sdake | lets abort any rally runs | 08:20 |
| sdake | andd after reconfigure goes, I'll give it another go | 08:20 |
| egonzalez90 | ping zhubingbing | 08:21 |
| Jeffrey4l_ | sdake, should this ./samples/tasks/scenarios/keystone/add-and-remove-user-role.yaml | 08:21 |
| sdake | i already removed it from the .txt file | 08:21 |
| sdake | rather commented it out | 08:21 |
| zhubingbing | hi engonzal2z90 | 08:23 |
| *** dmueller-evoila has joined #openstack-kolla | 08:24 | |
| Jeffrey4l_ | sdake, is the typescript out of date? | 08:25 |
| sdake | yes someone didn't exit when the screen was closedi think | 08:26 |
| sdake | so script is still running | 08:26 |
| sdake | script the command | 08:26 |
| sdake | reconfigure takes so long because of pid 1 problem? Jeffrey4l_ ? | 08:30 |
| Jeffrey4l_ | sdake, it should be related. | 08:31 |
| Jeffrey4l_ | sdake, and we have 100+ nodes. so it takes long. | 08:32 |
| Jeffrey4l_ | moreover, we have a common role is ran when reconfiguring, it shouldn't be ran imho | 08:33 |
| sdake | ok well our peformnace is smoking anyway | 08:34 |
| sdake | so no need to tackle that right during this osic benchmark gathering session :) | 08:34 |
| sdake | don't know what to make of that | 08:36 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 | 08:36 |
| Jeffrey4l_ | sdake, i got some idea to shorten the time of reconfiguration | 08:40 |
| Jeffrey4l_ | will try to push some PS later | 08:40 |
| sdake | possible reconfigure is busted for nova ? | 08:44 |
| sdake | i saw an awful lot of skipping | 08:44 |
| Jeffrey4l_ | sdake, now. 1. we need serial 30%, 2. there are much more task need to run when reconfigure. ( handle the copy_once and copy_always) | 08:45 |
| Jeffrey4l_ | now/no | 08:45 |
| Jeffrey4l_ | need to optimize | 08:45 |
| sdake | Jeffrey4l_ can you hack in quotas -1 into neutron and cinder | 08:46 |
| Jeffrey4l_ | sdake, ok. | 08:47 |
| sdake | that should make things more smooth | 08:47 |
| sdake | pbourke awake yet? :) | 08:48 |
| Jeffrey4l_ | sdake, got a idea to remove the skip. use ansible handlers | 08:48 |
| sdake | well we are pretty much at the end of the dev cycle | 08:48 |
| sdake | and we have a million patches to merge | 08:48 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (1) https://review.openstack.org/358539 | 08:51 |
| sdake | dont know what i'd do without you ejff | 08:54 |
| sdake | Jeffrey4l_ | 08:54 |
| sdake | ;) | 08:54 |
| Jeffrey4l_ | ;) | 08:54 |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add gnocchi ansible role https://review.openstack.org/349351 | 08:59 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 08:59 |
| sdake | osc is lacking most of the quota operations Jeffrey4l_ | 09:01 |
| Jeffrey4l_ | sdake, no idea about the default value for neutron and cinder. | 09:02 |
| openstackgerrit | zhubingbing proposed openstack/kolla: repair gnocchi dockerfile https://review.openstack.org/348388 | 09:03 |
| *** senk has joined #openstack-kolla | 09:05 | |
| zhubingbing | ping Hui Kang | 09:06 |
| *** dmueller-evoila has quit IRC | 09:10 | |
| *** phuongnh has quit IRC | 09:11 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' to neccesary tasks (1) https://review.openstack.org/358539 | 09:15 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (2) https://review.openstack.org/359031 | 09:15 |
| openstackgerrit | Christian Berendt proposed openstack/kolla: Remove fedora from Dockerfile templates https://review.openstack.org/359447 | 09:26 |
| pbourke | sdake: hey | 09:29 |
| sdake | sup | 09:30 |
| sdake | pbourke - i think we got rally about beat into submission | 09:30 |
| pbourke | nice | 09:30 |
| pbourke | just catching up on what you guys have been up to | 09:30 |
| sdake | gtting ready to pull the trigger on a full run now | 09:30 |
| sdake | to make sure ther eare no more annoying errors or quota problems | 09:30 |
| sdake | pbourke - i pused a new doc up for review | 09:31 |
| sdake | can you check it out please | 09:31 |
| pbourke | looks like you've been busy :) | 09:31 |
| sdake | the structure has changed slighlty | 09:31 |
| pbourke | will check now | 09:31 |
| sdake | also, we are doing 3 rally tests | 09:31 |
| sdake | 24, 48, 96 concurrency with 288 iterations | 09:31 |
| sdake | per scenario | 09:31 |
| sdake | cpu count, hyperthread cpu count, overload | 09:32 |
| pbourke | good stuff | 09:32 |
| pbourke | sdake: https://review.openstack.org/#/c/352379/ - just wondering, this is not a critical security or bugfix | 09:32 |
| pbourke | so why backport? | 09:32 |
| sdake | i thought i t was a bugfix | 09:33 |
| sdake | not a feature | 09:33 |
| sdake | i thought we determined running haproy not on the control nodes didn't work | 09:33 |
| pbourke | it works it's just ineffecient | 09:33 |
| pbourke | and non standard | 09:33 |
| pbourke | maybe I should add a reno? | 09:34 |
| sdake | possibly | 09:34 |
| sdake | up to you :) | 09:34 |
| pbourke | its not big enough to need one I think | 09:34 |
| sdake | so on osic | 09:35 |
| sdake | we have until the 30th to finish our testing | 09:35 |
| sdake | since 14.04 is no longer in master | 09:36 |
| sdake | we will need to switch to xenial at some point | 09:36 |
| sdake | i'd rather get 14.04 data | 09:36 |
| sdake | just so we have it | 09:36 |
| sdake | incae xenial is doa in some way | 09:36 |
| sdake | and I know centos works | 09:36 |
| sdake | so basically, we are doing ubuntu 14.04->centos7.2->xenial | 09:36 |
| sdake | tempest + 3 rally types of testing + 2 or 3 custom tests | 09:37 |
| sdake | per scenario | 09:37 |
| sdake | impossible or possible? | 09:37 |
| pbourke | well what scenario are we on right now | 09:37 |
| sdake | ceph... | 09:37 |
| sdake | but on the plus side all this stuff is automated | 09:37 |
| sdake | all we need is data capture n ot a full document by then | 09:37 |
| pbourke | there's still quite a bit to do is the only thing | 09:38 |
| pbourke | with the current pace I just dont see there been enough time to redo with ubuntu | 09:38 |
| sdake | ok well lets hold of on that for now | 09:38 |
| sdake | and focus on getting 14.04 and centos data | 09:38 |
| sdake | within those prameters | 09:38 |
| sdake | i think we have rallly in an operational state now however | 09:39 |
| *** zhubingbing has quit IRC | 09:39 | |
| sdake | pbourke - i'm going to be full time on this from nwo until the 30th | 09:40 |
| sdake | but I need some instruction on setup | 09:40 |
| pbourke | great | 09:40 |
| pbourke | ok I say we get rally results for scenario 1 into the review | 09:40 |
| *** salv-orlando has quit IRC | 09:40 | |
| *** bootsha has quit IRC | 09:40 | |
| pbourke | do we have those generated? | 09:40 |
| pbourke | scenario 1 is standard deploy, no ceph | 09:40 |
| sdake | no, we have been jerkingaround trying to get rally tow ork correctly | 09:40 |
| sdake | so our scenario 2 (ceph) will have data for rally | 09:40 |
| *** zhubingbing has joined #openstack-kolla | 09:40 | |
| sdake | but not sscenario #1 | 09:40 |
| sdake | that will require a redeploy | 09:41 |
| *** salv-orlando has joined #openstack-kolla | 09:41 | |
| pbourke | ok, generate ceph rally data | 09:41 |
| sdake | i don't know how to setup the networks/etc | 09:41 |
| pbourke | then we'll tear and do #1 | 09:41 |
| pbourke | tools/init-run-once-osic | 09:41 |
| pbourke | inc0 has it ready to go | 09:41 |
| sdake | cool | 09:41 |
| pbourke | i'll jump on tmux now | 09:41 |
| sdake | btw my screen is broken | 09:41 |
| sdake | and i am on a laptop | 09:41 |
| sdake | so if the screeen is small, my fault :( | 09:41 |
| pbourke | :) | 09:41 |
| pbourke | its not too bad | 09:41 |
| pbourke | so ceph rally running atm? | 09:42 |
| *** berendt has joined #openstack-kolla | 09:42 | |
| sdake | just got a error 500 | 09:42 |
| * sdake groans | 09:42 | |
| berendt | hello everybody | 09:42 |
| sdake | ya ceph rally running with iter 30 and 24 for concurrency | 09:42 |
| sdake | hey berendt | 09:42 |
| sdake | to test the config | 09:43 |
| Jeffrey4l_ | samples/tasks/scenarios/cinder/create-nested-snapshots-and-attach-volume.yaml | 09:43 |
| sdake | a 500 error is a bug in nova imo | 09:43 |
| pbourke | worry about it after,let the rest run on | 09:43 |
| sdake | we should be running these with typecript | 09:43 |
| Jeffrey4l_ | maybe | 09:43 |
| sdake | to caputre the output of rally run | 09:43 |
| Jeffrey4l_ | captured. | 09:43 |
| Jeffrey4l_ | ops....it is not in the stdout.. | 09:44 |
| sdake | is it jammed up? | 09:45 |
| sdake | or cleaning up | 09:45 |
| Jeffrey4l_ | no idea. .. | 09:47 |
| sdake | maybe out of disk space | 09:48 |
| *** daneyon has joined #openstack-kolla | 09:49 | |
| Jeffrey4l_ | blocked? | 09:49 |
| *** daneyon has quit IRC | 09:53 | |
| sdake | looks like rally finally timing it out | 09:53 |
| sdake | rally eventually gets impatient and drops the task | 09:53 |
| sdake | i think cinder snapshots are fubared | 09:54 |
| sdake | nothign can be done to unblock rally | 09:57 |
| Jeffrey4l_ | may be | 09:57 |
| sdake | except kill it | 09:57 |
| sdake | maye we dont need to redeploy though | 09:57 |
| Jeffrey4l_ | sdake, killed one task | 09:57 |
| *** zhurong has quit IRC | 09:59 | |
| *** HyperJohnGraham has quit IRC | 10:01 | |
| openstackgerrit | Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 | 10:02 |
| *** mgoddard_ has joined #openstack-kolla | 10:02 | |
| openstackgerrit | Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 | 10:04 |
| *** mgoddard has quit IRC | 10:05 | |
| sdake | now Jeffrey4l_ is jusst torturign me ;) | 10:06 |
| sdake | Jeffrey4l_ call the files somethign unique like tyepscript-scenario2-24 | 10:07 |
| sdake | and typecript-scearntio2-48 | 10:07 |
| sdake | etc | 10:07 |
| sdake | pbourke btw I backed up our working dir | 10:08 |
| Jeffrey4l_ | sdake, ok | 10:08 |
| sdake | since if we lost that, we would be in bad shape | 10:08 |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 | 10:08 |
| sdake | i'd encourage others to do the same - redundancy ftw ;) | 10:08 |
| *** zhiwei has quit IRC | 10:10 | |
| Jeffrey4l_ | including the rally samples ;) | 10:10 |
| sdake | Jeffrey4l_ copy /etc/kolla in /root as well plz | 10:10 |
| sdake | since you made those cusstomizations | 10:11 |
| Jeffrey4l_ | OK. | 10:11 |
| sdake | and a destroy might wipe them out | 10:11 |
| sdake | i dont recall if destroy runs on the deploy hot or not | 10:11 |
| sdake | the total run has been 13 minutes? | 10:12 |
| Jeffrey4l_ | yes. | 10:13 |
| sdake | my recommendation is to go straight to 288+96 | 10:13 |
| sdake | and see how long that takes after this is finished | 10:13 |
| sdake | I shoud lbe awake from my nap y then | 10:13 |
| Jeffrey4l_ | OK | 10:13 |
| sdake | 288 iterations, 96 concurrency | 10:13 |
| sdake | if that fails, that will save us alot of time | 10:14 |
| Jeffrey4l_ | sdake, so try this later? when the current task finished? | 10:14 |
| sdake | after this task finishes | 10:14 |
| Jeffrey4l_ | OK | 10:14 |
| sdake | if its successful | 10:14 |
| sdake | try 288+96 | 10:14 |
| sdake | if its not succesful | 10:14 |
| sdake | make it so :) | 10:14 |
| Jeffrey4l_ | got | 10:14 |
| sdake | i think we have how to debug it down | 10:14 |
| sdake | run time on 288+96 | 10:15 |
| sdake | in the meantime if you don twant to wath numbers scroll, might think about the bonnie test | 10:15 |
| sdake | thatwould be rockin | 10:15 |
| Jeffrey4l_ | debug it down? means | 10:15 |
| sdake | debug the problem and fix it | 10:16 |
| sdake | sorry | 10:16 |
| sdake | speaking in jibberish :) | 10:16 |
| sdake | oh what i mean | 10:16 |
| sdake | we know how ot debug | 10:16 |
| Jeffrey4l_ | sdake, in fact i want to use fio. which is powerful and it can print json report directly. | 10:16 |
| Jeffrey4l_ | yep | 10:16 |
| sdake | when you have something down, you understand it well | 10:16 |
| sdake | fio wfm | 10:16 |
| Jeffrey4l_ | OK. | 10:16 |
| sdake | we will need to chnage the baremetal characterization in that case | 10:16 |
| sdake | or addd fio to the results | 10:16 |
| Jeffrey4l_ | OKl | 10:17 |
| sdake | no rush on that | 10:17 |
| sdake | and dont do it while rally running pls :) | 10:17 |
| Jeffrey4l_ | i will . | 10:17 |
| Jeffrey4l_ | i will test it locally first. | 10:17 |
| sdake | sweet | 10:17 |
| sdake | ok - alarm sset for t-4 hrs, see you all in a bit | 10:18 |
| Jeffrey4l_ | see you :) | 10:18 |
| *** duonghq has left #openstack-kolla | 10:19 | |
| openstackgerrit | Duong Ha-Quang proposed openstack/kolla: Specify 'become' for only neccesary tasks (last) https://review.openstack.org/359096 | 10:19 |
| *** bootsha has joined #openstack-kolla | 10:20 | |
| *** sdake has quit IRC | 10:28 | |
| *** chas has joined #openstack-kolla | 10:32 | |
| *** chas is now known as ibmchas | 10:33 | |
| *** athomas has quit IRC | 10:49 | |
| *** athomas has joined #openstack-kolla | 10:54 | |
| *** zhubingbing has quit IRC | 11:00 | |
| *** Mr_Broken has quit IRC | 11:02 | |
| *** Mr_Broken has joined #openstack-kolla | 11:02 | |
| *** JoseMello has joined #openstack-kolla | 11:05 | |
| *** Mr_Broken has quit IRC | 11:06 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Allow use of database for Horizon sessions https://review.openstack.org/359794 | 11:08 |
| *** egonzalez90 has quit IRC | 11:11 | |
| *** caowei has quit IRC | 11:12 | |
| *** rhallisey has joined #openstack-kolla | 11:14 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Customizations for Nova https://review.openstack.org/351676 | 11:18 |
| pbourke | Jeffrey4l_: can you revisit https://review.openstack.org/#/c/352379/ please? | 11:19 |
| pbourke | Jeffrey4l_: I dont think it should be backported | 11:19 |
| *** salv-orlando has quit IRC | 11:25 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Check volume group exists for Cinder in prechecks https://review.openstack.org/354154 | 11:25 |
| *** salv-orlando has joined #openstack-kolla | 11:25 | |
| *** egonzalez90 has joined #openstack-kolla | 11:29 | |
| *** mgoddard has joined #openstack-kolla | 11:31 | |
| *** mgoddard_ has quit IRC | 11:31 | |
| *** dwalsh has joined #openstack-kolla | 11:36 | |
| *** Mr_Broken has joined #openstack-kolla | 11:36 | |
| *** bootsha has quit IRC | 11:37 | |
| *** bootsha has joined #openstack-kolla | 11:40 | |
| *** Mr_Broken has quit IRC | 11:41 | |
| *** bootsha has quit IRC | 11:43 | |
| *** sean-k-mooney has quit IRC | 11:44 | |
| *** Mr_Broken has joined #openstack-kolla | 11:46 | |
| openstackgerrit | Merged openstack/kolla: Fix Ubuntu binary build and deploy gate https://review.openstack.org/358488 | 11:46 |
| openstackgerrit | Merged openstack/kolla: ironic.conf file missing enabled_drivers info https://review.openstack.org/356290 | 11:47 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Add option to configure routers in HA mode https://review.openstack.org/359819 | 11:51 |
| *** egonzalez90 has quit IRC | 12:00 | |
| openstackgerrit | Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 | 12:14 |
| openstackgerrit | Prithiv proposed openstack/kolla: Dockerfiles for collectd Container https://review.openstack.org/358632 | 12:15 |
| Jeffrey4l_ | pbourke, iirc, i talked with coolsvap_ or inc0 once about this. I think the right solution is control. | 12:16 |
| *** coolsvap_ has quit IRC | 12:21 | |
| *** hieulq_ has joined #openstack-kolla | 12:22 | |
| *** ravig has joined #openstack-kolla | 12:28 | |
| *** ravig has quit IRC | 12:29 | |
| bjolo | anyone time for a horizon Q? | 12:36 |
| bjolo | what is the intended way to get lbaas tab in horizon? | 12:36 |
| bjolo | for ex lbaas. I belive it is pretty much the same concern for all "extra" services | 12:36 |
| *** caowei has joined #openstack-kolla | 12:42 | |
| *** bootsha has joined #openstack-kolla | 12:43 | |
| *** jtriley has joined #openstack-kolla | 12:46 | |
| *** duonghq has joined #openstack-kolla | 12:47 | |
| *** bootsha has quit IRC | 12:48 | |
| duonghq | Do we have meeting today? | 12:51 |
| pbourke | Jeffrey4l_: why | 12:59 |
| pbourke | Jeffrey4l_: wait, that's what I have | 12:59 |
| pbourke | Jeffrey4l_: do you mean network? | 12:59 |
| pbourke | bjolo: is it a horizon plugin/ | 13:00 |
| Jeffrey4l_ | pbourke, first, do u think that fix is correct? | 13:03 |
| pbourke | Jeffrey4l_: yes | 13:03 |
| Jeffrey4l_ | me too. | 13:03 |
| pbourke | so +2 it please :) | 13:03 |
| Jeffrey4l_ | pbourke, why it shouldn't be backport? | 13:04 |
| pbourke | because its not critical | 13:04 |
| Jeffrey4l_ | i still think haproxy on the network is not right.. | 13:04 |
| pbourke | it depends really what our backport policy is | 13:04 |
| Jeffrey4l_ | But.. wait, how can we handle the upgrade? | 13:04 |
| pbourke | my understanding was it's security and critical bug fixes | 13:04 |
| Jeffrey4l_ | pbourke, hmm.. you are correct. | 13:05 |
| bjolo | pbourke, believe so, reading up | 13:05 |
| Jeffrey4l_ | pbourke, it may break something if we backport it. | 13:05 |
| pbourke | I'll add a note to the reno that operators upgrading will likely want to stop the haproxy running on their network nodes | 13:06 |
| *** zhiwei has joined #openstack-kolla | 13:07 | |
| Jeffrey4l_ | pbourke, cool . so could u add the releasenote in your PS? | 13:07 |
| *** salv-orlando has quit IRC | 13:07 | |
| pbourke | i will | 13:07 |
| pbourke | thanks Jeffrey4l_ | 13:07 |
| Jeffrey4l_ | then i will +2 | 13:07 |
| Jeffrey4l_ | ;) | 13:07 |
| bjolo | pbourke, https://github.com/openstack/neutron-lbaas-dashboard | 13:08 |
| pbourke | bjolo: it looks similar to the murano-dashboard | 13:08 |
| pbourke | bjolo: you can install it using the template-overrides | 13:09 |
| bjolo | pbourke, yes | 13:09 |
| bjolo | i can try, have not done a template overide before (newbie) | 13:09 |
| bjolo | but it is the same issue for all "extra" services. They often have a plugin/dashboard for horizon | 13:10 |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Move haproxy to control nodes in sample inventories https://review.openstack.org/352379 | 13:10 |
| pbourke | bjolo: it is documented here: http://docs.openstack.org/developer/kolla/image-building.html#dockerfile-customisation | 13:11 |
| *** zhiwei has quit IRC | 13:11 | |
| bjolo | kewl | 13:11 |
| pbourke | specifically http://docs.openstack.org/developer/kolla/image-building.html#plugin-functionality | 13:11 |
| bjolo | will try | 13:11 |
| *** Mr_Broken has quit IRC | 13:12 | |
| *** lrensing_ has joined #openstack-kolla | 13:12 | |
| *** Mr_Broken has joined #openstack-kolla | 13:12 | |
| bjolo | another general question. there is a sahara image, but no ansible role for it. is anyone working on that? | 13:13 |
| *** bootsha has joined #openstack-kolla | 13:14 | |
| pbourke | bjolo: https://review.openstack.org/#/c/351294/ | 13:14 |
| bjolo | tnx again :) | 13:14 |
| *** dave-mccowan has joined #openstack-kolla | 13:15 | |
| *** Mr_Broken has quit IRC | 13:18 | |
| *** senk has quit IRC | 13:24 | |
| *** openstackgerrit has quit IRC | 13:26 | |
| *** openstackgerrit has joined #openstack-kolla | 13:26 | |
| *** Mr_Broken has joined #openstack-kolla | 13:34 | |
| *** sdake has joined #openstack-kolla | 13:34 | |
| sdake | yo | 13:35 |
| sdake | Jeffrey4l_ how did it go | 13:35 |
| Jeffrey4l_ | sdake, runing the 288+96 test. | 13:36 |
| *** sdake_ has joined #openstack-kolla | 13:37 | |
| Jeffrey4l_ | the 24 concurrency has two failed. and it takes 2.5 hours :( | 13:37 |
| *** zhurong has joined #openstack-kolla | 13:38 | |
| sdake_ | 2.5 hours in tlta? | 13:38 |
| sdake_ | 2 fails is ok | 13:38 |
| sdake_ | fialures are ok at 288x96 | 13:38 |
| sdake_ | are expected | 13:38 |
| sdake_ | how far along is it Jeffrey4l_ | 13:39 |
| sdake_ | Jeffrey4l_ can you estiate total time? | 13:39 |
| Jeffrey4l_ | let me check | 13:39 |
| *** sdake has quit IRC | 13:40 | |
| *** ravig has joined #openstack-kolla | 13:40 | |
| Jeffrey4l_ | sdake_, need 3 more i think. | 13:40 |
| Jeffrey4l_ | but may be less. | 13:41 |
| sdake_ | 5 hrs total? | 13:41 |
| Jeffrey4l_ | maybe.. | 13:41 |
| Jeffrey4l_ | who knows :( | 13:41 |
| *** zhurong has quit IRC | 13:41 | |
| britthouser | pbourke: ping! | 13:41 |
| pbourke | britthouser: hey | 13:42 |
| sdake_ | Jeffrey4l_ we hae 4 hours total we can dedicate to each test to finish by the 30th | 13:42 |
| britthouser | did ou wanna go through some scenarios...or is that what sdake_ and Jeffrey4l_ are discussing? | 13:42 |
| *** zhurong has joined #openstack-kolla | 13:43 | |
| sdake_ | britthouser yup we got a full run of rally udnerway atm | 13:43 |
| britthouser | Ok. I'm coming late to the game today it seems. =) | 13:43 |
| britthouser | My morning is pretty much open until the weekly kolla meeting. | 13:44 |
| *** zhubingbing has joined #openstack-kolla | 13:44 | |
| zhubingbing | hello | 13:44 |
| Jeffrey4l_ | sdake_, 4 hours? what's mean? | 13:44 |
| sdake_ | well i just woke up sos my math by be off | 13:44 |
| sdake_ | 6 days left | 13:45 |
| sdake_ | we have 24 total scenarios to get through | 13:45 |
| Jeffrey4l_ | correct :) | 13:45 |
| sdake_ | at minimum | 13:45 |
| sdake_ | what math do you make of that | 13:45 |
| *** Mr_Broken has quit IRC | 13:45 | |
| britthouser | we need todo 4 scenarios/day | 13:45 |
| sdake_ | oh - 4 sceanrio a day | 13:45 |
| sdake_ | ya thats better | 13:46 |
| sdake_ | i'm going back tobed :) | 13:46 |
| britthouser | I ate my wheaties this morning. =) | 13:46 |
| sdake_ | britthouser here is what we are doingwith each test case | 13:47 |
| sdake_ | we are running tempest | 13:47 |
| sdake_ | we are running rally at 288x96, 288x48, and 288x24 | 13:47 |
| *** ravig has quit IRC | 13:47 | |
| sdake_ | the first number is iterations the second numberis concurrency | 13:47 |
| sdake_ | we are doing all scenarios on ubuntu 14.04 first | 13:47 |
| sdake_ | then switchign to centos | 13:47 |
| sdake_ | then if time remains switching to xenial | 13:48 |
| sdake_ | it may be that 288 is to olarge ofa nubmer | 13:48 |
| britthouser | And the scenarios are the same from the etherpad (1-5?) | 13:48 |
| *** mgoddard_ has joined #openstack-kolla | 13:48 | |
| sdake_ | same sceanrios | 13:48 |
| *** zhubingbing has quit IRC | 13:49 | |
| sdake_ | rally = whole lot of waiting | 13:49 |
| sdake_ | Jeffrey4l_ how long did that 30x20 take? | 13:49 |
| *** zhubingbing has joined #openstack-kolla | 13:50 | |
| Jeffrey4l_ | sdake_, 2.5 hours | 13:50 |
| sdake_ | are you serious? | 13:50 |
| Jeffrey4l_ | yes. but two job hanged for some time. | 13:51 |
| Jeffrey4l_ | 9:00 -> 11:00 | 13:51 |
| Jeffrey4l_ | 9:00 -> 11:30 | 13:51 |
| Jeffrey4l_ | oh. sorry. | 13:52 |
| Jeffrey4l_ | 1:30 | 13:52 |
| Jeffrey4l_ | 10:00 -> 11:30 | 13:52 |
| Jeffrey4l_ | sdake_, ^ | 13:52 |
| *** mgoddard has quit IRC | 13:52 | |
| sdake_ | the original one we were doing - I thought that would finish in an hour o rso | 13:53 |
| zhubingbing | what are you talking about？ | 13:53 |
| Jeffrey4l_ | zhubingbing, osic test | 13:53 |
| zhubingbing | sure | 13:53 |
| *** prithiv has joined #openstack-kolla | 13:53 | |
| britthouser | ok sdake_ and Jeffery4l_ I'm following along in the tmux | 13:54 |
| sdake_ | Jeffrey4l_ hhat does cpu utilization look like on one of the controllers? | 13:57 |
| sdake_ | Jeffrey4l_ can you paste top or something like that? :) | 13:57 |
| sdake_ | ori guess i can connect to the vpn | 13:57 |
| Jeffrey4l_ | sdake_, http://paste.openstack.org/show/563007/ | 13:58 |
| sdake_ | wow lots of zombies | 13:59 |
| britthouser | What is beam.smp? | 13:59 |
| sdake_ | we gotta fix that | 13:59 |
| sdake_ | beam.smp i tihnk is a smp kernel thing | 13:59 |
| *** dwalsh has quit IRC | 14:00 | |
| sdake_ | looks related to java | 14:00 |
| sdake_ | wonder why java running on control nodes | 14:00 |
| Jeffrey4l_ | beam.smp is rabbitmq staff. | 14:01 |
| Jeffrey4l_ | stuff | 14:01 |
| sdake_ | an oprofile would be fantastic of this controller under this test | 14:02 |
| sdake_ | maybe if we hvae time i'll get a trace | 14:02 |
| sdake_ | Jeffrey4l_ used oprofile before? | 14:02 |
| Jeffrey4l_ | no.. | 14:03 |
| sdake_ | well i can tell we are leaking zombie processes | 14:03 |
| sdake_ | i hadn't thought it would be this bad | 14:03 |
| sdake_ | but it looks pretty bad | 14:03 |
| sdake_ | oprofile is a systemwide profilign tool | 14:03 |
| sdake_ | it tells you every bitof usage of your entire system including kernel | 14:03 |
| Jeffrey4l_ | sdake_, zombie processes related with the pid 1 issue. | 14:03 |
| sdake_ | it basically gives a breakdown of what is sucking the most cpu | 14:03 |
| sdake_ | Jeffrey4l_ yes of course | 14:04 |
| Jeffrey4l_ | the pid 1 do not reaper the process properly. | 14:04 |
| sdake_ | aware of problem | 14:04 |
| *** mgoddard_ has quit IRC | 14:06 | |
| openstackgerrit | zhubingbing proposed openstack/kolla: Add Barbican container dockerfile https://review.openstack.org/351822 | 14:06 |
| sdake_ | wonder if those zombies hapepn on a fresh reboot of controllers - since yu restarted a bucnh of service a bunch of times | 14:06 |
| *** mgoddard has joined #openstack-kolla | 14:06 | |
| Jeffrey4l_ | sdake_, it happen. I tested that. | 14:07 |
| Jeffrey4l_ | but no idea why it happen | 14:07 |
| sdake_ | Jeffrey4l_ so - a fresh reboot would result in 1200 zombiess again after 2.5 hrs of runtime? | 14:07 |
| *** sbezverk_ has quit IRC | 14:07 | |
| Jeffrey4l_ | after restart the container, the zombie process will be reaper by the kernel. | 14:07 |
| sdake_ | good point | 14:08 |
| *** sbezverk has joined #openstack-kolla | 14:08 | |
| sdake_ | again ju woke up | 14:08 |
| Jeffrey4l_ | sdake_, it should be related to the namespace created. | 14:08 |
| sdake_ | and not alot o sleep | 14:08 |
| sdake_ | wonder if they happen to grow | 14:08 |
| sdake_ | over time | 14:08 |
| sdake_ | if they grow and fill process table, no bueno | 14:09 |
| Jeffrey4l_ | based on whether u create new namespace, it will increase. | 14:09 |
| Jeffrey4l_ | So we should introduce dumb-init to | 14:09 |
| Jeffrey4l_ | how big the process table be, normally? | 14:09 |
| sdake_ | not sure | 14:09 |
| sdake_ | but at max 65k entries | 14:10 |
| sdake_ | 64k entries | 14:10 |
| sdake_ | that is all the pids there are available | 14:10 |
| Jeffrey4l_ | OK. we have <2K now. | 14:10 |
| *** inc0 has joined #openstack-kolla | 14:10 | |
| britthouser | my_init can be used in container to reap pids | 14:11 |
| sdake_ | britthouser we are thinking of something non-python for init | 14:11 |
| *** dmueller-evoila has joined #openstack-kolla | 14:11 | |
| inc0 | good morning | 14:11 |
| Jeffrey4l_ | morning :) | 14:11 |
| inc0 | how are you guys? | 14:12 |
| *** ravig has joined #openstack-kolla | 14:12 | |
| britthouser | Why no python sdake_ ? | 14:12 |
| britthouser | morning inc0 | 14:12 |
| Jeffrey4l_ | there are lots of options for this issue. britthouser | 14:12 |
| inc0 | whats the issue? | 14:12 |
| britthouser | Agreed Jeffrey4l_ | 14:12 |
| britthouser | zombies | 14:12 |
| inc0 | what kind of zombies? | 14:13 |
| Jeffrey4l_ | dumb-init is simple and with most number of star :) | 14:13 |
| Jeffrey4l_ | inc0, https://review.openstack.org/357015 | 14:13 |
| sdake_ | britthouser - my main rationale is i dont feel we need the need for a full python runtime at pid1 | 14:15 |
| inc0 | Jeffrey4l_, but dumbinint takes pid1 right? so docker logs will be impossible | 14:15 |
| inc0 | and other stuff that docker uses for pid 1 | 14:15 |
| britthouser | I gotcha sdake_ something more lightweight. | 14:15 |
| sdake_ | i like that it is surgical change | 14:16 |
| sdake_ | easy to backpoort | 14:16 |
| sdake_ | this is not really a feature but more like a bug | 14:16 |
| Jeffrey4l_ | inc0, it still possible. the logs still print to the stdout | 14:16 |
| *** ravig has quit IRC | 14:16 | |
| inc0 | so I talked to Sam when he was here | 14:16 |
| inc0 | and there is issue when some process spawns children, and these children spawns grandchildren | 14:17 |
| inc0 | if process dies, grandchildren are zombified | 14:17 |
| sdake_ | yes that is the pid 1 problem | 14:17 |
| sdake_ | well known | 14:17 |
| sdake_ | i keep hoping docker will fix it | 14:17 |
| inc0 | for us it's neutron | 14:17 |
| sdake_ | its fixable in docker itsself | 14:17 |
| sdake_ | but vrey complex | 14:17 |
| inc0 | neutron spawns dnsmasq, dnsmasq spawns it's runners | 14:18 |
| Jeffrey4l_ | inc0, so dumb-init solve such issue, right? | 14:18 |
| inc0 | might | 14:18 |
| sdake_ | it does - i looked at the implementation | 14:18 |
| inc0 | although | 14:18 |
| inc0 | ok, if it reaps, it's cool | 14:19 |
| *** dmueller-evoila has quit IRC | 14:19 | |
| sdake_ | the problem is it is buit on the fly | 14:19 |
| sdake_ | rather then prebuilt | 14:19 |
| sdake_ | (atleast with current approach) | 14:19 |
| sdake_ | Jeffrey4l_ actually that is just an assumption - is that correct? | 14:20 |
| inc0 | built on the fly? | 14:21 |
| sdake_ | its c softwre in a pip package | 14:21 |
| Jeffrey4l_ | sdake_, the dumb-init? i download the binary from github now. | 14:21 |
| sdake_ | oh i see | 14:21 |
| sdake_ | so not super optimal | 14:21 |
| sdake_ | but we can live with it | 14:21 |
| inc0 | but prebuilt | 14:21 |
| sdake_ | Jeffrey4l_ i looked at your version of the patch when it used git | 14:22 |
| *** egonzalez90 has joined #openstack-kolla | 14:23 | |
| Jeffrey4l_ | ok | 14:23 |
| inc0 | Jeffrey4l_, so we'd need to use it in toolbox and neutron | 14:23 |
| Jeffrey4l_ | inc0, i add it in all images. | 14:23 |
| inc0 | ok | 14:23 |
| inc0 | makes sense | 14:23 |
| inc0 | actually I could use it in upgrades | 14:23 |
| Jeffrey4l_ | add it in base image and run the start.sh scrip by using the dumb-init | 14:23 |
| sdake_ | implementation looks good | 14:24 |
| inc0 | as sending sighup to services is funky | 14:24 |
| *** michauds has joined #openstack-kolla | 14:24 | |
| Jeffrey4l_ | inc0, https://review.openstack.org/#/c/357015/4/docker/base/start.sh check this. | 14:24 |
| Jeffrey4l_ | inc0, that's cool | 14:24 |
| inc0 | ok, I see what you mean | 14:24 |
| sdake_ | inc0 are you up to speed wit h osic | 14:26 |
| inc0 | yesterday I didn't touch it | 14:27 |
| *** michauds_ has joined #openstack-kolla | 14:28 | |
| *** michauds has quit IRC | 14:28 | |
| *** dwalsh has joined #openstack-kolla | 14:29 | |
| inc0 | rally is running now? | 14:29 |
| Jeffrey4l_ | inc0, yep | 14:29 |
| sdake_ | we are doing the following | 14:30 |
| sdake_ | we are doing ubuntu 14.04->centos->xenial | 14:30 |
| sdake_ | we are doing all scnearios in order | 14:30 |
| inc0 | did you add new images to cobbler? | 14:30 |
| sdake_ | we are doing tempesst runs as well as srally runs with 288x96, 288x48 and 288x24 loads | 14:30 |
| sdake_ | we are not repaving eahch time inc0 so I haven't as no need for centos yet | 14:31 |
| *** ayoung has joined #openstack-kolla | 14:31 | |
| sdake_ | Jeffrey4l_ and I spent the day getting rallly woring mostly well | 14:31 |
| inc0 | did you solve issues with rally? | 14:31 |
| sdake_ | yup i ts fixed | 14:31 |
| *** michauds_ is now known as michauds__ | 14:31 | |
| inc0 | cool | 14:31 |
| *** mgoddard_ has joined #openstack-kolla | 14:31 | |
| sdake_ | requirees customizations | 14:31 |
| sdake_ | for quota forone ething | 14:32 |
| *** michauds__ is now known as michauds | 14:32 | |
| sdake_ | and comented out some more test cases | 14:32 |
| sdake_ | we have to do 4 scenarios a day | 14:32 |
| inc0 | ok, I'll help today | 14:32 |
| inc0 | although I need to figure out base customizations | 14:32 |
| inc0 | kinda important;) | 14:32 |
| sdake_ | everything is rady to go - jsut need to run script captur e data | 14:32 |
| sdake_ | and redeploy | 14:33 |
| inc0 | and figure out centos with cobbler;) | 14:33 |
| sdake_ | i could do all of it if i knew how to deploy | 14:33 |
| sdake_ | pbourke says there is an osic init runonce | 14:33 |
| openstackgerrit | Eduardo Gonzalez proposed openstack/kolla: Change source with dot at extend_start files https://review.openstack.org/359926 | 14:33 |
| sdake_ | i am entirely dedicated to this problem until the 30th | 14:34 |
| inc0 | errors started to show up | 14:34 |
| *** dwalsh has quit IRC | 14:34 | |
| *** mgoddard has quit IRC | 14:34 | |
| *** bootsha has quit IRC | 14:34 | |
| *** dwalsh has joined #openstack-kolla | 14:35 | |
| sdake_ | i'm glad after 3 weeks we hae a properly working test bed and test cases ;-) | 14:36 |
| *** prithiv has quit IRC | 14:36 | |
| *** berendt has quit IRC | 14:37 | |
| sdake_ | when we near finish on centos, i want to grab an oproifle trace | 14:38 |
| sdake_ | community may find that fascinating | 14:38 |
| inc0 | tbh we could prepare better - making rally work prior to osic | 14:38 |
| *** bootsha has joined #openstack-kolla | 14:39 | |
| sdake_ | no shit sherlock | 14:40 |
| sdake_ | i guess i was in unicorn goldilocks land of "it will just work" | 14:40 |
| inc0 | like it ever happends sdake | 14:40 |
| sdake_ | well I knew kolla wouold work out of the box | 14:41 |
| *** dwalsh has quit IRC | 14:41 | |
| sdake_ | its already been tested at 64 compute nodes :) | 14:41 |
| sdake_ | next cycle with bifrost, ought to take on that timing challenge of openstack deploy | 14:42 |
| *** bootsha has quit IRC | 14:43 | |
| sdake_ | we may go down to 192x96, 192x48, 192x24 | 14:43 |
| sdake_ | depends how long jeffrey's run takes | 14:44 |
| openstackgerrit | Vladislav Belogrudov proposed openstack/kolla: Add defaults for interfaces to all.yml https://review.openstack.org/354173 | 14:45 |
| *** hieulq_ has quit IRC | 14:45 | |
| *** dwalsh has joined #openstack-kolla | 14:53 | |
| *** zhubingbing has quit IRC | 14:59 | |
| *** haplo37__ has joined #openstack-kolla | 15:03 | |
| *** HyperJohnGraham has joined #openstack-kolla | 15:05 | |
| sdake_ | Jeffrey4l_ I think we need a file in the kolla dir on osic thats called "NEXT STEP" | 15:06 |
| sdake_ | or "NEXTSTEP" | 15:06 |
| sdake_ | to point people in proper direction | 15:07 |
| *** lrensing_ has quit IRC | 15:07 | |
| Jeffrey4l_ | that's will be better. | 15:07 |
| Jeffrey4l_ | need go to bed. bye guys | 15:08 |
| *** zhurong has quit IRC | 15:08 | |
| *** mark-casey has joined #openstack-kolla | 15:09 | |
| sdake_ | bye Jeffrey4l_ | 15:12 |
| britthouser | gnight Jeffrey4l_ | 15:13 |
| *** caowei has quit IRC | 15:15 | |
| *** egonzalez90 has quit IRC | 15:15 | |
| *** david-lyle has joined #openstack-kolla | 15:17 | |
| *** salv-orlando has joined #openstack-kolla | 15:18 | |
| *** zhubingbing has joined #openstack-kolla | 15:18 | |
| *** egonzalez90 has joined #openstack-kolla | 15:19 | |
| *** egonzalez90 has quit IRC | 15:19 | |
| *** zhubingbing has quit IRC | 15:21 | |
| *** zhubingbing has joined #openstack-kolla | 15:21 | |
| *** vbel has quit IRC | 15:22 | |
| *** vbel has joined #openstack-kolla | 15:22 | |
| openstackgerrit | Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 | 15:23 |
| *** prithiv has joined #openstack-kolla | 15:23 | |
| *** salv-orlando has quit IRC | 15:25 | |
| *** david-lyle_ has joined #openstack-kolla | 15:25 | |
| *** sdake_ has quit IRC | 15:25 | |
| *** david-lyle_ has quit IRC | 15:26 | |
| *** vhosakot has joined #openstack-kolla | 15:26 | |
| zhubingbing | Is there half an hour for week meeting？ | 15:27 |
| duonghq | zhubinghing: huh? | 15:28 |
| inc0 | yeah, meeting is in 30min if that's what you mean | 15:28 |
| zhubingbing | ok | 15:29 |
| *** prithiv has quit IRC | 15:30 | |
| openstackgerrit | Li Yingjun proposed openstack/kolla: Add option to support dhcp/l3 agent ha https://review.openstack.org/352180 | 15:30 |
| *** fragatina has joined #openstack-kolla | 15:31 | |
| *** fragatin_ has joined #openstack-kolla | 15:32 | |
| zhubingbing | who know what is the theme of meeting today？ | 15:33 |
| *** dwalsh has quit IRC | 15:33 | |
| *** mgoddard_ has quit IRC | 15:34 | |
| *** mgoddard has joined #openstack-kolla | 15:34 | |
| *** fragatina has quit IRC | 15:35 | |
| *** srwilkers_ has joined #openstack-kolla | 15:35 | |
| *** sdake has joined #openstack-kolla | 15:36 | |
| sdake | 75% done | 15:37 |
| sdake | we have 3*24 rally runs to do | 15:37 |
| sdake | pbourke ping re rally qeustion | 15:39 |
| pbourke | sdake: yup | 15:39 |
| sdake | pbourke is there a way to get all json fora specific deployment? | 15:39 |
| sdake | as in 1 step export | 15:40 |
| pbourke | sdake: not that we've found :( | 15:40 |
| *** Serlex has quit IRC | 15:40 | |
| *** ibmchas has quit IRC | 15:40 | |
| pbourke | sdake: time to break out your bash skills | 15:40 |
| sdake | how have youbeen collecting data in the eviews up to now? | 15:41 |
| sdake | one by one export manually? | 15:41 |
| pbourke | some sort of grep + for loop | 15:41 |
| pbourke | i'll check if its still in the history | 15:41 |
| pbourke | sdake: i think that will do the trick (the one currently on screen) | 15:42 |
| *** mdnadeem_home has joined #openstack-kolla | 15:43 | |
| *** egonzalez90 has joined #openstack-kolla | 15:44 | |
| pbourke | sdake: just checking something super quick | 15:47 |
| *** salv-orlando has joined #openstack-kolla | 15:48 | |
| *** vhosakot has quit IRC | 15:49 | |
| *** vhosakot has joined #openstack-kolla | 15:49 | |
| sdake | we will have to write a json to rst parser for rally | 15:51 |
| sdake | should be easy enough | 15:51 |
| *** dwalsh has joined #openstack-kolla | 15:52 | |
| jrich523 | morning guys | 15:53 |
| sdake | sup jrich523 | 15:53 |
| jrich523 | im back at it! | 15:53 |
| jrich523 | more errors | 15:53 |
| jrich523 | ansible is failing to install now, which is kinda odd since i didnt have this problem the first time, and im duplicating steps... | 15:54 |
| sdake | jrich523 - can't help yo utoday - sorry | 15:54 |
| jrich523 | ah no worries | 15:54 |
| sdake | jrich523 - we only ahve osic cluster for 6 more days - and we are short on time | 15:55 |
| jrich523 | its a sunny wed, i dont plan on getting much done :) | 15:55 |
| jrich523 | id help.. but... you know.... kinda useless until i learn more :) | 15:55 |
| *** berendt has joined #openstack-kolla | 15:58 | |
| *** dwalsh has quit IRC | 16:00 | |
| *** mgoddard_ has joined #openstack-kolla | 16:00 | |
| sdake | team meeting now in openstack-meeting-4 | 16:02 |
| zhubingbing | ok | 16:03 |
| kfox1111 | sbezverk: how about pod-config or pod-local-config? | 16:03 |
| *** mgoddard has quit IRC | 16:04 | |
| zhubingbing | snake, what is the theme of today meeting | 16:06 |
| *** daneyon has joined #openstack-kolla | 16:08 | |
| *** prithiv has joined #openstack-kolla | 16:08 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: neutron openvswitch agent ip https://review.openstack.org/356114 | 16:09 |
| *** daneyon has quit IRC | 16:13 | |
| *** imcsk8 has quit IRC | 16:15 | |
| zhubingbing | No one is talking about the problem？ | 16:15 |
| *** zhubingbing has quit IRC | 16:20 | |
| *** athomas has quit IRC | 16:21 | |
| *** fragatin_ has quit IRC | 16:21 | |
| openstackgerrit | Paul Bourke proposed openstack/kolla: Import the Murano core libary as part of setup https://review.openstack.org/360002 | 16:22 |
| *** matrohon has quit IRC | 16:22 | |
| *** zhubingbing has joined #openstack-kolla | 16:22 | |
| *** vhosakot has quit IRC | 16:27 | |
| *** vhosakot has joined #openstack-kolla | 16:28 | |
| *** kklimonda_ is now known as kklimonda | 16:28 | |
| *** vinkman has joined #openstack-kolla | 16:33 | |
| *** haplo37__ has quit IRC | 16:33 | |
| *** prithiv has quit IRC | 16:34 | |
| zhubingbing | startmeeting kolla | 16:35 |
| zhubingbing | #startmeeting kolla | 16:35 |
| openstack | Meeting started Wed Aug 24 16:35:46 2016 UTC and is due to finish in 60 minutes. The chair is zhubingbing. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:35 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:35 |
| openstack | The meeting name has been set to 'kolla' | 16:35 |
| rhallisey | zhubingbing, meeting is in #openstack-meeting-4 | 16:36 |
| Mech422 | rhallisey: it doesn't appear to be ? | 16:36 |
| *** ssurana has joined #openstack-kolla | 16:37 | |
| *** jtriley has quit IRC | 16:40 | |
| *** jtriley has joined #openstack-kolla | 16:43 | |
| *** unicell has quit IRC | 16:47 | |
| *** gfhellma has joined #openstack-kolla | 16:49 | |
| *** dwalsh has joined #openstack-kolla | 16:50 | |
| *** mdnadeem_home has quit IRC | 16:52 | |
| *** jemcevoy has joined #openstack-kolla | 16:52 | |
| *** aNupoisc has joined #openstack-kolla | 16:56 | |
| *** mgoddard_ has quit IRC | 16:57 | |
| *** mgoddard has joined #openstack-kolla | 16:57 | |
| *** srwilkers_ has quit IRC | 16:58 | |
| berendt | zhubingbing wrong room ... :) | 16:59 |
| berendt | zhubingbing can you please end meeting here | 17:00 |
| duonghq | so bad, when somebody start meeting out of room, the bot will crazy | 17:00 |
| duonghq | somebody should backup meeting log | 17:00 |
| duonghq | sdake: can you do that | 17:00 |
| sdake | duonghq that is infra's job | 17:01 |
| berendt | maybe we should make the backup first... :) | 17:01 |
| sdake | it is recorded | 17:01 |
| sdake | like irc is | 17:01 |
| duonghq | but when somebody end meeting here, we'll lose the log | 17:01 |
| sdake | duonghq meetings are timestamped | 17:01 |
| sdake | #endmeeting | 17:01 |
| berendt | why is it possible to start the kolla meeting here? should this be possible? | 17:01 |
| * sdake groans | 17:01 | |
| sdake | berendt yes its possible | 17:01 |
| berendt | but it is not possible to end it :) | 17:02 |
| *** wally__ has joined #openstack-kolla | 17:02 | |
| duonghq | berendt: this is what happen last weeks | 17:02 |
| duonghq | berendt: you can see last week log | 17:02 |
| inc0 | zhubingbing, please do #endmeeting kolla | 17:02 |
| duonghq | inc0: anybody can endmeeting kolla | 17:03 |
| inc0 | ok | 17:03 |
| duonghq | quite disappoint about the meeting bot | 17:03 |
| berendt | duonghq sdake tried it but no end messages appeared? | 17:03 |
| sdake | onlly the chairs can control the meeting ot | 17:03 |
| sdake | anyway dont sweat it | 17:04 |
| sdake | lets focus on reviews and osic plz :) | 17:04 |
| sdake | pbourke how do you run tempest | 17:04 |
| pbourke | sdake: that im not sure of | 17:04 |
| duonghq | sdake, berendt, http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-17-16.00.txt | 17:04 |
| berendt | I am out for today, will spend some time on reviews tomorrow | 17:04 |
| pbourke | sdake: best to check with coolsvap when he's on | 17:05 |
| pbourke | heading home now be back on tomorrow | 17:05 |
| duonghq | berendt: just last week irc meeting log, somewhat funny | 17:05 |
| openstackgerrit | Hui Kang proposed openstack/kolla: Add etcd container https://review.openstack.org/355156 | 17:05 |
| egonzalez90 | logs from today still there http://eavesdrop.openstack.org/meetings/kolla/2016/kolla.2016-08-24-16.00.log.txt | 17:06 |
| duonghq | sdake: iirc: the baremetal role get in master already | 17:06 |
| pbourke | inc0: could you have another look at https://review.openstack.org/#/c/337594/ ? | 17:06 |
| duonghq | egozales90: good news, still cannot understand | 17:06 |
| pbourke | inc0: I dont like the idea of pause | 17:06 |
| pbourke | inc0: would like to see it merge for n3 | 17:06 |
| pbourke | inc0: keep in mind this is a very real scenario that can cause data loss | 17:07 |
| *** wally__ has quit IRC | 17:07 | |
| pbourke | sdake: ^ | 17:07 |
| sdake | pbourke can you mvoe the iwnodws around in the osic cluster | 17:07 |
| sdake | so they appear properly | 17:07 |
| duonghq | egonzalez90: https://review.openstack.org/#/c/359596/ -> gate is already green, would you mind re-review it. | 17:07 |
| pbourke | sdake: i cant handle all the splits either :) | 17:08 |
| inc0 | pbourke, I know, however having this as a config option will mean people who actually wants to redeploy will have a hard time | 17:08 |
| inc0 | and it won't be easy to figure out | 17:08 |
| *** berendt has quit IRC | 17:08 | |
| inc0 | I'd be ok with Jeffrey's suggestion of a mixture - you can configure behaviour with config | 17:08 |
| inc0 | but unless config is set, pasue will kick in | 17:08 |
| sdake | where did the splits go | 17:08 |
| *** vinkman has left #openstack-kolla | 17:08 | |
| inc0 | like ceph_osd_behaviour: "keep" or "wipe" | 17:09 |
| *** gfidente is now known as gfidente|afk | 17:09 | |
| inc0 | and unless it's one of these it will pause "do you really want to wipe you drives? to automate set ceph_osd_behaviour" | 17:09 |
| pbourke | sdake: i thought you asked me clean them up | 17:09 |
| sdake | no | 17:09 |
| sdake | i wanted them fixed not deleted :( | 17:10 |
| pbourke | for me fixed == deleted :p | 17:10 |
| pbourke | sure you can create them again the way you want them | 17:10 |
| *** JoseMello has quit IRC | 17:10 | |
| pbourke | inc0: ok thanks for that, I'll take a look at doing it that way | 17:10 |
| duonghq | I almost finish the bp ansible-specific-task-become, hope that somebody review my works when you have free time | 17:11 |
| sdake | pbourke there all fixed | 17:11 |
| egonzalez90 | duonghq sent a recheck. Gates fix was merged today, all should be green after recheck | 17:12 |
| *** pbourke has quit IRC | 17:13 | |
| *** pbourke has joined #openstack-kolla | 17:13 | |
| egonzalez90 | but ubuntu xenial tox :( | 17:13 |
| duonghq | egonzalez90: I have some doubts, but let see | 17:14 |
| *** unicell has joined #openstack-kolla | 17:15 | |
| *** thumpba has joined #openstack-kolla | 17:17 | |
| *** wally__ has joined #openstack-kolla | 17:17 | |
| wally__ | hi sdake, I just sent you an email regarding some question, can you take a look | 17:18 |
| *** egonzalez90 has quit IRC | 17:19 | |
| duonghq | egonzalez90: I need sleep now, see you tomorrow | 17:20 |
| sdake | wally__ i don't provide support from my inbox sorry | 17:22 |
| sdake | the community provides limited support to get you going via irc however | 17:22 |
| wally__ | ok my quesiton is | 17:22 |
| *** thumpba has quit IRC | 17:22 | |
| *** thumpba has joined #openstack-kolla | 17:22 | |
| wally__ | i just re-do again, and I ran the kolla-ansible prechecks | 17:23 |
| wally__ | i got this error | 17:23 |
| wally__ | http://pastebin.com/jD3BJXM4 | 17:23 |
| wally__ | can you please take a look | 17:24 |
| wally__ | i have configured my network as you said | 17:24 |
| sdake | wrong version of ansible | 17:24 |
| wally__ | root@controller:/etc/kolla# ansible --version ansible 1.9.4 configured module search path = /usr/share/ansible | 17:24 |
| wally__ | the same as yeterday | 17:25 |
| wally__ | TASK: [prechecks | Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes] *** fatal: [localhost] => Failed to template {% if '169.254.' not in kolla_internal_vip_address and 'False' in kolla_internal_vip_address | ipaddr(ip_addr_output.stdout.split()[3]) %} True {% else %} False {% endif %}: an unexpected type error occurred. Error was argument of type 'NoneType' is not iterable | 17:25 |
| sdake | not sure then | 17:25 |
| wally__ | this error, is ansible version peoblem? | 17:25 |
| sdake | too busy to help | 17:25 |
| sdake | t-6 days to milestone 3 and end of osic gear | 17:26 |
| sdake | so you will either have to wait or find other help in channel o fwhich there is plenty :) | 17:26 |
| wally__ | ok, thanks a lot, | 17:26 |
| wally__ | anyone knows what the error is ? | 17:27 |
| jrich523 | im looking now | 17:27 |
| jrich523 | can you post your globals.yml | 17:28 |
| wally__ | ok one sec | 17:28 |
| wally__ | http://pastebin.com/BUuAixZv | 17:29 |
| *** vhosakot_ has joined #openstack-kolla | 17:29 | |
| jrich523 | what are the IPs for eth0 and eth1 | 17:30 |
| wally__ | i post my ifconfig here | 17:30 |
| wally__ | http://pastebin.com/qbcmBE3e | 17:30 |
| wally__ | there is no ip for eth1, i did not set ip | 17:30 |
| jrich523 | so you need two IPs, one for mgmt and one for 'externa' | 17:30 |
| jrich523 | so two adapters need to be configured | 17:31 |
| wally__ | ok, i see | 17:31 |
| jrich523 | and the VIP belongs on the external (i think) | 17:31 |
| jrich523 | so set an IP on both adapters | 17:31 |
| wally__ | so now I have eth0, which is 10.145.213.51. that is management? | 17:31 |
| jrich523 | based on what you have in globals | 17:31 |
| jrich523 | line 61 and 74 | 17:32 |
| jrich523 | when i set it up, it was all virtual so, same vlan | 17:32 |
| jrich523 | but, im certain this is whats going on :) | 17:32 |
| *** vhosakot has quit IRC | 17:32 | |
| *** senk has joined #openstack-kolla | 17:32 | |
| wally__ | ok, let me try | 17:32 |
| jrich523 | eth1 is def your external, and eth0 is mgmt, | 17:32 |
| jrich523 | just not sure where the vip actually belongs | 17:33 |
| duonghq | jrich523: iirc the external interface should be no ip is set, neutron will do its work | 17:33 |
| jrich523 | hmm wel lthen he needs to change the interface i think | 17:33 |
| duonghq | the Internet is laggy here, cannot look at his link | 17:34 |
| duonghq | but if it's kolla_internal_vip, so it belongs to network_interface | 17:34 |
| jrich523 | thats some serious lag, but pastbin is horribly covered in ads | 17:34 |
| jrich523 | yup, which in his case is eth0 | 17:34 |
| wally__ | well, then need 2 ips or 1 ip set? | 17:35 |
| jrich523 | also way different | 17:35 |
| wally__ | confused | 17:35 |
| duonghq | wally_ you set up all-in-one or multinode? | 17:35 |
| jrich523 | his IP on eth0 is :10.145.213.51 and the VIP is 10.10.10.254 | 17:35 |
| wally__ | all in one | 17:35 |
| wally__ | i am going to setup eth1: 10.10.10.51 | 17:35 |
| duonghq | so you can safely disable haproxy (by add enable_haproxy: "no") | 17:36 |
| jrich523 | according to duonghq you dont need to, so, you're probably ok there i think | 17:36 |
| duonghq | and set the VIP to your real ip of network_interface | 17:36 |
| jrich523 | wait, what? | 17:36 |
| duonghq | just for test, because of if you enable haproxy, the VIP must different than the NIC IP | 17:37 |
| jrich523 | the VIP should be its own IP, on top of Network_interface's (eth0) IP | 17:37 |
| wally__ | this is the ip 10.145.213.51 , I ssh into the server, | 17:37 |
| *** vhosakot_ has quit IRC | 17:37 | |
| jrich523 | ahhhh | 17:37 |
| jrich523 | well unless you've got a crazy netmask | 17:37 |
| jrich523 | the VIP and eth0 ip are WAY different | 17:37 |
| wally__ | auto eth0 iface eth0 inet static address 10.145.213.51 netmask 255.255.255.0 dns-nameservers 8.8.8.8 gateway 10.145.213.1 | 17:37 |
| *** vhosakot has joined #openstack-kolla | 17:37 | |
| wally__ | this is my interfaces file | 17:37 |
| duonghq | jrich523: if you disable haproxy, so VIP == network_interface | 17:37 |
| jrich523 | yeah i suppose thats fine for what he's doing | 17:38 |
| duonghq | not too bad setting for 1st run | 17:38 |
| wally__ | so what is the global file | 17:38 |
| jrich523 | you still have to provide the VIP in globals tho right, just happens to be the same IP? | 17:38 |
| jrich523 | globals.yml that you posted for me | 17:38 |
| duonghq | right | 17:38 |
| wally__ | but you said what do i need to edit in the global.yaml | 17:38 |
| jrich523 | two things, but im looking for the first :) | 17:39 |
| duonghq | wally__: your 2 IP is in different subnet, but the error you post is not relate the that problem (iirc) | 17:39 |
| duonghq | *the -> to | 17:39 |
| wally__ | i only setup one 1p now, which is 10.145.213.51. and I use this ip to ssh into the server | 17:40 |
| duonghq | I guess that the problem is in " ipaddr(ip_addr_output.stdout.split()[3])" part | 17:40 |
| duonghq | no idea | 17:40 |
| sdake | ok folks working on osic - i've got it down to 8 primary scenarios | 17:40 |
| duonghq | wally_ yeah,... I think it is fine | 17:40 |
| wally__ | how is the network diagram setup in kolla, I am confused | 17:41 |
| jrich523 | no no | 17:41 |
| jrich523 | so based on the test name: Checking if kolla_internal_vip_address is in the same network as network_interface on all nodes | 17:41 |
| duonghq | it's too much for today, need some sleep, sorry wally_, jrich523 | 17:41 |
| jrich523 | his eth0 ip is a different subnet from the VIP | 17:41 |
| duonghq | bye | 17:41 |
| jrich523 | make the VIP something on the same subnet as eth0 | 17:41 |
| wally__ | but I got the error when do the pre-check, | 17:41 |
| jrich523 | right | 17:41 |
| jrich523 | its checking to see if the VIP are on the same network | 17:42 |
| wally__ | ok let me see | 17:42 |
| jrich523 | which, they are not | 17:42 |
| *** duonghq has left #openstack-kolla | 17:42 | |
| *** duonghq has quit IRC | 17:42 | |
| jrich523 | also, it oddly assumes a /24 network, kinda odd | 17:42 |
| jrich523 | change the VIP so that its in the same subnet as eth0 | 17:42 |
| wally__ | ok let me try | 17:43 |
| jrich523 | eth0: 10.145.213.51 / VIP 10.10.10.254 - different subnets | 17:43 |
| *** HyperJohnGraham has quit IRC | 17:43 | |
| jrich523 | make the VIP something like 10.145.213.52 or whatever is free | 17:43 |
| wally__ | thx | 17:44 |
| sdake | jrich523 ya /24 is a bad assumption - ip interface shoudl be read | 17:51 |
| *** aswadr_ has quit IRC | 17:51 | |
| jrich523 | still early, the assumption is fine for now :) | 17:51 |
| jrich523 | but yeah, at it to that long list of things to do :) | 17:52 |
| *** daneyon has joined #openstack-kolla | 17:56 | |
| *** daneyon has quit IRC | 18:01 | |
| *** vdo has left #openstack-kolla | 18:03 | |
| sdake | inc0 - where is the tempest .testr. file | 18:11 |
| *** jamesdenton is now known as Guest20708 | 18:11 | |
| inc0 | no clue sdake - I don't know tempest at all | 18:14 |
| * sdake groans | 18:14 | |
| inc0 | find is your hope | 18:14 |
| sdake | coolsvap is the one who got that setup originallly | 18:14 |
| sdake | i already used find | 18:14 |
| inc0 | $find -n .testr | 18:14 |
| inc0 | but I think rally runs tempest | 18:15 |
| inc0 | tempest is kinda embedded in rally somehow | 18:15 |
| inc0 | not sure how tho | 18:15 |
| *** dims has quit IRC | 18:18 | |
| *** dims has joined #openstack-kolla | 18:18 | |
| sdake | inc0 i find that highly unbelievable | 18:19 |
| *** rhallisey has quit IRC | 18:20 | |
| sdake | hrm | 18:20 |
| sdake | i guess it does integrate with it | 18:20 |
| *** wally__ has quit IRC | 18:22 | |
| inc0 | sdake, look at tmux | 18:22 |
| sdake | ya who do you think is typign that in | 18:23 |
| inc0 | well fuck. | 18:23 |
| sdake | oh i see what you mean | 18:23 |
| sdake | say | 18:23 |
| *** HyperJohnGraham has joined #openstack-kolla | 18:23 | |
| sdake | did the last run of stuff | 18:23 |
| sdake | have a time on it? | 18:23 |
| sdake | can you scrol up? | 18:23 |
| inc0 | it;s in rally db | 18:24 |
| sdake | 6 hours | 18:24 |
| openstackgerrit | Merged openstack/kolla-kubernetes: Introduces new approach in starting OVS in Kube https://review.openstack.org/358895 | 18:24 |
| sdake | what is in rally db | 18:24 |
| inc0 | results of all ran tests | 18:24 |
| sdake | yes | 18:24 |
| sdake | now we need tempest in the mix ;) | 18:24 |
| sdake | and work can actually begin | 18:25 |
| inc0 | this is how you run tempest | 18:25 |
| inc0 | but something is bad | 18:25 |
| openstackgerrit | Merged openstack/kolla-kubernetes: neutron openvswitch agent ip https://review.openstack.org/356114 | 18:26 |
| sdake | deps busted | 18:27 |
| sdake | sudo pip -U pip | 18:27 |
| inc0 | yeah....syntax error \o/ | 18:27 |
| sdake | sudo pip install -U pip | 18:27 |
| *** wally__ has joined #openstack-kolla | 18:27 | |
| wally__ | hi,I just done the deloyment in AIO, it worked, I can login to the horizon, | 18:28 |
| jrich523 | awesome, way to go wally__ | 18:28 |
| inc0 | wally__, do one woot for kolla:) | 18:28 |
| wally__ | I am just wondering how can I change the admin login password. it is very long, I changed the .openrs.sh file , but not work | 18:28 |
| *** intr1nsic has quit IRC | 18:28 | |
| inc0 | wally__, /etc/kolla/passwords.yml | 18:28 |
| inc0 | but once it's set you need to change it in horizon | 18:29 |
| inc0 | or keystone | 18:29 |
| wally__ | i run this kolla-ansible post-deploy, and get the password | 18:29 |
| inc0 | but basically passwords.yml is config file for well...passwords | 18:29 |
| *** intr1nsic has joined #openstack-kolla | 18:29 | |
| wally__ | i just want to change admin login passsword | 18:29 |
| inc0 | wally__, log in to horizon with the long one | 18:29 |
| wally__ | :( | 18:30 |
| inc0 | top right corner afair, you can change password there | 18:30 |
| inc0 | but you need to login first | 18:30 |
| wally__ | you mean insided horizon | 18:30 |
| inc0 | yeah | 18:30 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 | 18:30 |
| wally__ | ok worked | 18:31 |
| wally__ | thanks | 18:31 |
| inc0 | :) | 18:31 |
| inc0 | np | 18:31 |
| inc0 | soo...how about this woot for kolla?;) | 18:31 |
| inc0 | j/k don't woot unless you really believe in the woot | 18:31 |
| wally__ | nice, fast | 18:32 |
| wally__ | question, after i reboot the server, all the docker image will start auto? | 18:32 |
| inc0 | yeah | 18:32 |
| sdake | inc0 is this how you were doing it previously with reporting? | 18:32 |
| inc0 | I just dumped all into files | 18:34 |
| inc0 | but don't quote me on that, I'm newbie in rally | 18:34 |
| sdake | someone ran tmepest reports | 18:35 |
| inc0 | to me it seems deps are broken | 18:35 |
| inc0 | there was a freaking syntax error there | 18:35 |
| sdake | build/temp.linux-x86_64-2.7/check_libyaml.c:2:18: fatal error: yaml.h: No such file or directory^M | 18:37 |
| sdake | can you make that file appear please | 18:37 |
| sdake | probaby PyYaml-dev or something | 18:37 |
| *** senk has quit IRC | 18:41 | |
| *** senk has joined #openstack-kolla | 18:41 | |
| *** HyperJohnGraham has quit IRC | 18:42 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 | 18:45 |
| *** fragatina has joined #openstack-kolla | 18:46 | |
| *** gfhellma has quit IRC | 18:48 | |
| *** gfhellma has joined #openstack-kolla | 18:49 | |
| *** HyperJohnGraham has joined #openstack-kolla | 18:51 | |
| *** gfhellma1 has joined #openstack-kolla | 18:52 | |
| *** zhubingbing has quit IRC | 18:52 | |
| *** gfhellma has quit IRC | 18:54 | |
| wally__ | question, I create a vm, but the vm canot go outside, and I cannot ping and ssh into the vm, how is the network for the vm? | 18:54 |
| sdake | damn thought you wer ein a venv inc0 | 18:54 |
| sdake | sorry | 18:54 |
| inc0 | let's try now | 18:54 |
| *** lrensing_ has joined #openstack-kolla | 18:55 | |
| sdake | yup | 18:55 |
| sdake | source creds first | 18:55 |
| inc0 | nah | 18:55 |
| inc0 | rally has it covered | 18:55 |
| sdake | sweet it works | 18:55 |
| *** HyperJohnGraham has quit IRC | 18:55 | |
| inc0 | but we'll need to hack around it every time we do new deployment | 18:55 |
| inc0 | which sux | 18:55 |
| sdake | bummer its not easy to install | 18:56 |
| inc0 | we can cp -R this working venv | 18:56 |
| inc0 | to new env | 18:56 |
| inc0 | hacky but should work | 18:56 |
| sdake | ya i'll create a taraall of root | 18:56 |
| inc0 | just this venv | 18:56 |
| sdake | i already baked up /root | 18:56 |
| inc0 | and extract it to .venv in dir created | 18:56 |
| sdake | but will do again | 18:56 |
| inc0 | we're running out of space btw;) | 18:56 |
| sdake | not surprising | 18:57 |
| inc0 | also it's you who look at this terminal on a poststamp? | 18:57 |
| sdake | what do we have left | 18:57 |
| inc0 | it's like 1/6 of my screen | 18:57 |
| sdake | ya laptop - busted screen | 18:57 |
| sdake | you know the story | 18:57 |
| inc0 | yeah | 18:57 |
| inc0 | "dont stand left of me" | 18:57 |
| sdake | haha | 18:58 |
| sdake | say do you know how we mae rally get a new database | 18:58 |
| inc0 | yeah it's just sqlite somewhere | 18:58 |
| sdake | i'd like to remove the db and start fresh | 18:58 |
| sdake | but waotor this tempest run | 18:58 |
| sdake | i need to see how it is reported | 18:58 |
| sdake | inc0 i see | 19:00 |
| sdake | thanks | 19:00 |
| sdake | is the temepst job still running? | 19:00 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 | 19:02 |
| inc0 | it is. | 19:02 |
| *** aNupoisc has left #openstack-kolla | 19:03 | |
| sdake | inc0 back killing me - going to lay dow for an hour or two - and then i'll be ready to go with a full run | 19:03 |
| inc0 | kk | 19:03 |
| sdake | i'll take care of running and getting output for scenario2 | 19:04 |
| sdake | we need then scenario 3 setup | 19:04 |
| inc0 | I need to get back to patches | 19:04 |
| sdake | scenario runtime is probably 4-8 hours | 19:04 |
| *** Mech422 has quit IRC | 19:04 | |
| sdake | right | 19:04 |
| inc0 | let me know when you're back up | 19:04 |
| sdake | i struggle with setup because I don't know the network environment | 19:04 |
| sdake | perhaps you can show me how you go thorugh that so I can just get this done | 19:04 |
| inc0 | everyone struggles with setups, that's why kolla exist | 19:04 |
| sdake | we have all the pieces in place | 19:04 |
| sdake | well I struggle because I lack network informatio ;-) | 19:05 |
| sdake | anyway bbiaf | 19:05 |
| *** haplo37__ has joined #openstack-kolla | 19:05 | |
| sdake | need tha t tempest to finish before i can procee | 19:05 |
| *** Mech422 has joined #openstack-kolla | 19:06 | |
| wally__ | i created a vm . but the VM canot reach outside, how is the network setup for a VM? | 19:09 |
| *** senk has quit IRC | 19:10 | |
| *** jemcevoy has quit IRC | 19:11 | |
| *** senk has joined #openstack-kolla | 19:11 | |
| *** lrensing_ has quit IRC | 19:11 | |
| *** rhallisey has joined #openstack-kolla | 19:14 | |
| *** sdake has quit IRC | 19:20 | |
| *** senk has quit IRC | 19:27 | |
| *** senk has joined #openstack-kolla | 19:28 | |
| *** adnavare has joined #openstack-kolla | 19:33 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 | 19:34 |
| adnavare | Hi Folks, has anyone tried setting up docker on ubuntu 14.04? When I am trying to install I am getting hell of dependencies which I never saw earlier and unable to find a way around | 19:34 |
| adnavare | *recently | 19:34 |
| mark-casey | adnavare: can you create a paste with the output of sudo apt-mark showhold as well as sudo apt-mark showmanual | 19:36 |
| *** senk has quit IRC | 19:36 | |
| mark-casey | wondering if there are any packages on the hold list that are the sam ebeing complained about | 19:36 |
| adnavare | mark-casey: showhold did not returned anything | 19:38 |
| adnavare | showmanual output is here http://pastebin.com/n0zgZSQP | 19:38 |
| *** thumpba_ has joined #openstack-kolla | 19:39 | |
| *** thumpba_ has quit IRC | 19:39 | |
| *** gfhellma1 has quit IRC | 19:39 | |
| adnavare | mark-casey: yesterday i got to know docker has a new release that only supports ubuntu xenial, but i am wondering there has to be a way for trusty | 19:41 |
| *** thumpba has quit IRC | 19:41 | |
| mark-casey | re: Xenial they've stated that publicly or that was a theory? | 19:43 |
| *** gfhellma has joined #openstack-kolla | 19:47 | |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: l3/metadata DaemonSets + DVR, & labels https://review.openstack.org/357557 | 19:56 |
| openstackgerrit | Kevin Fox proposed openstack/kolla-kubernetes: Deployment, Readiness, Safe Shutdown, & Scaling for glance https://review.openstack.org/354895 | 19:58 |
| openstackgerrit | Merged openstack/kolla-kubernetes: openvswitch service hookup/rename https://review.openstack.org/360051 | 19:59 |
| adnavare | mark-casey: I am not sure, I got to know from sdake | 20:04 |
| adnavare | mark-casey: can we actually try to get docker 1.10 specifically? So that I don't need these dependencies | 20:05 |
| adnavare | mark-casey: i have this in my sources.list.d/docker.list "deb https://apt.dockerproject.org/repo ubuntu-trusty main" | 20:07 |
| adnavare | and then when i try to do apt-get install docker-engine it is throwing all these dependencies | 20:07 |
| *** vhosakot has quit IRC | 20:08 | |
| *** vhosakot has joined #openstack-kolla | 20:12 | |
| *** egonzalez90 has joined #openstack-kolla | 20:15 | |
| wally__ | hi guys, after I created VM, but the VM cannot go to internet, how is the network setup? | 20:19 |
| wally__ | i cannot ping and ssh into the VM | 20:19 |
| Pavo | wally__ does your secuity groups in openstack have SSH and ICMP ingress added? | 20:25 |
| Pavo | and for the instance to have internet access you have to have an external network inside openstack configured with DNS to resolve | 20:26 |
| wally__ | yeah, i have ssh an dcmp ingress added | 20:30 |
| wally__ | I setup all in one, and my sever has internet access, | 20:31 |
| wally__ | how do u mean by external network inside openstack configured | 20:31 |
| *** Jeffrey4l__ has joined #openstack-kolla | 20:34 | |
| *** Jeffrey4l_ has quit IRC | 20:34 | |
| *** jtriley_ has joined #openstack-kolla | 20:35 | |
| openstackgerrit | Michal Jastrzebski (inc0) proposed openstack/kolla: Customization for base https://review.openstack.org/358042 | 20:36 |
| *** jtriley has quit IRC | 20:38 | |
| mark-casey | adnavare: IIRC the errors you were getting don't mean it doesn't have the dependencies, they mean it is holding on to older versions of those dependencies for an unknown reason. If you want to see which versions are currently available you could 'sudo apt-cache show docker-engine | grep Version' | 20:38 |
| *** daneyon has joined #openstack-kolla | 20:39 | |
| *** jtriley_ has quit IRC | 20:40 | |
| *** bootsha has joined #openstack-kolla | 20:42 | |
| *** egonzalez90 has quit IRC | 20:42 | |
| *** daneyon has quit IRC | 20:43 | |
| mark-casey | adnavare: if 1.10 is listed then installing it may be as simple as sudo apt-get install docker-engine=1.10.0-0~trusty | 20:43 |
| mark-casey | or there are also a few point revisions like 1.10.3 | 20:44 |
| *** vhosakot has quit IRC | 20:49 | |
| *** vhosakot has joined #openstack-kolla | 20:50 | |
| inc0 | jrich523, just finished meddling with headphones, it was easy indeed | 20:59 |
| *** HyperJohnGraham has joined #openstack-kolla | 21:02 | |
| *** vhosakot has quit IRC | 21:11 | |
| *** vhosakot has joined #openstack-kolla | 21:11 | |
| *** bootsha has quit IRC | 21:13 | |
| adnavare | mark-casey: Okay yes install docker-engine=1.10.0-0~trusty did solved the problem. Somehow earlier it did not worked. | 21:14 |
| adnavare | mark-casey: may be i tried something wrong the last time. | 21:14 |
| *** srwilkers_ has joined #openstack-kolla | 21:15 | |
| *** vhosakot has quit IRC | 21:17 | |
| *** salv-orl_ has joined #openstack-kolla | 21:17 | |
| *** adnavare has quit IRC | 21:18 | |
| *** salv-orlando has quit IRC | 21:20 | |
| *** dwalsh has quit IRC | 21:20 | |
| *** rhallisey has quit IRC | 21:23 | |
| *** aNupoisc has joined #openstack-kolla | 21:23 | |
| *** HyperJohnGraham_ has joined #openstack-kolla | 21:33 | |
| *** HyperJohnGraham has quit IRC | 21:34 | |
| *** HyperJohnGraham_ is now known as HyperJohnGraham | 21:35 | |
| *** lamt has joined #openstack-kolla | 21:37 | |
| *** lrensing_ has joined #openstack-kolla | 21:38 | |
| lrensing_ | would someone mind reviewing this when they have some spare time? https://review.openstack.org/#/c/358305/ | 21:39 |
| *** haplo37__ has quit IRC | 21:43 | |
| *** bootsha has joined #openstack-kolla | 21:43 | |
| *** sdake has joined #openstack-kolla | 21:48 | |
| *** HyperJohnGraham has quit IRC | 21:48 | |
| *** sdake_ has joined #openstack-kolla | 21:50 | |
| *** lrensing_ has quit IRC | 21:51 | |
| *** srwilkers_ has quit IRC | 21:51 | |
| *** bootsha has quit IRC | 21:52 | |
| *** sdake has quit IRC | 21:54 | |
| *** sdake_ has quit IRC | 22:01 | |
| *** HyperJohnGraham has joined #openstack-kolla | 22:03 | |
| *** sdake has joined #openstack-kolla | 22:03 | |
| lamt | Hello, I am planning to work on https://bugs.launchpad.net/kolla/+bug/1616387 - and am wondering if there is any corner cases that I should be aware of. Besides removing all references to fedora. | 22:06 |
| openstack | Launchpad bug 1616387 in kolla "Remove base_distro type fedora from Dockerfile templates" [Undecided,New] | 22:06 |
| *** zhiwei has joined #openstack-kolla | 22:06 | |
| *** zhiwei has quit IRC | 22:11 | |
| *** vhosakot has joined #openstack-kolla | 22:13 | |
| sdake | inc0 around? | 22:19 |
| inc0 | yeah | 22:19 |
| *** dave-mccowan has quit IRC | 22:19 | |
| sdake | can you make rally verify install work again | 22:19 |
| sdake | i am not ure what you did | 22:19 |
| sdake | i hae the virtual env | 22:20 |
| sdake | but i'm not sure where it goes | 22:20 |
| sdake | its for-deployment-id-buildenv | 22:20 |
| inc0 | here | 22:20 |
| *** lamt has quit IRC | 22:27 | |
| *** HyperJohnGraham has quit IRC | 22:29 | |
| *** michauds has quit IRC | 22:30 | |
| *** jrist has quit IRC | 22:33 | |
| sdake | inc0 after i recreated the databasea | 22:35 |
| sdake | and played with a few things | 22:35 |
| sdake | now rally isn'tworking at all | 22:36 |
| sdake | inc0 almost got it entirey scripted on the gather data front | 22:36 |
| inc0 | ehh I don't knwo rally:S | 22:38 |
| *** mark-casey has quit IRC | 22:38 | |
| inc0 | there was another script here right? | 22:39 |
| sdake | ya i renamed it tot his one | 22:39 |
| sdake | and going ot make a new one ot run everything | 22:39 |
| sdake | also temest works but fais | 22:39 |
| openstackgerrit | Wei Cao proposed openstack/kolla: Add solum container https://review.openstack.org/355408 | 22:40 |
| sdake | inc0 i'm goin to go get lunch or breakfasat or whatever | 22:41 |
| sdake | inc0 bbi20-30 mins - please see if yo u can get that working | 22:41 |
| inc0 | let's hoep | 22:41 |
| jrich523 | inc0: glad to hear it went smoothly :) | 22:44 |
| inc0 | yeah thanks for pointers | 22:45 |
| *** jrist has joined #openstack-kolla | 22:45 | |
| *** bmace has quit IRC | 22:46 | |
| *** fguillot has joined #openstack-kolla | 22:46 | |
| *** vhosakot has quit IRC | 22:50 | |
| inc0 | sdake, works, somebody messed up admin openrc. Also moved deployment creation out of forloop. | 22:50 |
| jrich523 | glad to help inc0 :) | 22:50 |
| *** dwalsh has joined #openstack-kolla | 22:54 | |
| *** zigo has quit IRC | 22:56 | |
| *** sdake has quit IRC | 22:57 | |
| *** zigo has joined #openstack-kolla | 22:57 | |
| *** fguillot has quit IRC | 23:00 | |
| *** fguillot has joined #openstack-kolla | 23:03 | |
| sbezverk | sdake: ping | 23:06 |
| *** sdake has joined #openstack-kolla | 23:07 | |
| sdake | inc0 thanks so ready to go then on the test run? | 23:08 |
| inc0 | 1 conc and 1 times is runnihng now | 23:08 |
| inc0 | feel free to trash db again and redo whole thing | 23:08 |
| sdake | i always use /etc/kolla/admin-openrc | 23:09 |
| sdake | was that file modified? | 23:09 |
| inc0 | it was invalid | 23:09 |
| sdake | nova list worked with those creds ;-) | 23:09 |
| inc0 | yeah | 23:10 |
| inc0 | but not everything | 23:10 |
| inc0 | OS_PROJECT_DOMAIN_ID=default | 23:10 |
| inc0 | I changed it to DOMAIN_NAME | 23:10 |
| sdake | ok 3 tests are running | 23:21 |
| sdake | in serial order | 23:21 |
| sdake | then we need to run tempest manually | 23:21 |
| sdake | because the install is manual | 23:21 |
| sdake | inc0 re https://etherpad.openstack.org/p/kolla-N-midcycle-osic | 23:22 |
| sdake | inc0 how many storage/compute/control nodes does scenario 2 have? | 23:23 |
| sdake | and what networing is in use? | 23:23 |
| inc0 | just pick a numbe | 23:23 |
| inc0 | r | 23:23 |
| sdake | inc0 i want to consolidate some scnearios | 23:23 |
| sdake | how many do we have now | 23:23 |
| inc0 | and as networking goes, it should be ok as it is | 23:24 |
| sdake | inc0 one of our scenarios is linuxbridge another is ovs | 23:24 |
| sdake | inc0 what is the current inventory file in use? | 23:28 |
| inc0 | ~/inventory-ips-only afaik | 23:28 |
| inc0 | brb gonna eat sth | 23:28 |
| *** fguillot has quit IRC | 23:29 | |
| sdake | cool 3 control 100 compute 20 storage | 23:30 |
| sdake | can consolidate | 23:30 |
| *** fguillot has joined #openstack-kolla | 23:33 | |
| *** lamt has joined #openstack-kolla | 23:34 | |
| sdake | we basicalaly have 6 data captures in total per distro now | 23:39 |
| sdake | much mor etidy | 23:39 |
| sdake | scenario 2 is capturing now | 23:39 |
| sdake | once we run tempest need to deploy scenario 3 | 23:39 |
| wally__ | hi guys, Is there a network diagram for kolla deployment, my vm still cannot reach outside, I dot know how does a VM can reach outside? | 23:48 |
| sdake | inc0 2016-08-24 18:48:22.205 12258 INFO rally.task.runner [-] Task 477b86f1-3e08-4a15-a692-a5fd55ce1eb7 | ITER: 111 END: Error IpAddressGenerationFailureClient: No more IP addresses available on network b73d8375-e54b-46cf-8070-7d5c7966444c. | 23:48 |
| inc0 | lol | 23:49 |
| inc0 | we should redeploy whole thing | 23:49 |
| sdake | i think that seems reasonable | 23:49 |
| sdake | how long will that take you :) | 23:49 |
| sdake | dont delete files in /etc dir | 23:50 |
| sdake | they contain custom quota config | 23:50 |
| inc0 | sdake, should be about 20min;) | 23:50 |
| inc0 | wanna do the honors? | 23:50 |
| sdake | including ceph? | 23:51 |
| sdake | nah - not sure i woudl get ceph right | 23:51 |
| inc0 | ahh...with ceph | 23:51 |
| inc0 | then 21min | 23:51 |
| sdake | i see some special playbooks for dealign with the disk labeling | 23:51 |
| inc0 | I can watch | 23:51 |
| sdake | how about i wtch you drive | 23:51 |
| inc0 | kk | 23:51 |
| inc0 | I'm killing this thing | 23:52 |
| sdake | what about disk labeling? | 23:52 |
| sdake | oh aftre destroy | 23:52 |
| sdake | cmdfailed? | 23:54 |
| sdake | jeffrey had soem special playbook to deal with the disk partitions iirc | 23:54 |
| *** zhurong has joined #openstack-kolla | 23:54 | |
| aNupoisc | is 2.0.3 tag associated with kolla stable/mitika branch? | 23:59 |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!