Tuesday, 2014-07-01

*** alazarev has joined #openstack-sahara00:01
*** RobLevas has joined #openstack-sahara00:07
*** RobLevas has quit IRC00:15
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Fixed volumes mount in case of existing volumes  https://review.openstack.org/10369100:21
*** alazarev has quit IRC00:24
*** IlyaE has quit IRC00:45
*** robsparker has quit IRC00:54
*** robsparker has joined #openstack-sahara01:06
*** RobLevas has joined #openstack-sahara01:10
*** RobLevas has quit IRC01:14
*** ViswaV_ has joined #openstack-sahara01:43
*** ViswaV has quit IRC01:46
*** RobLevas has joined #openstack-sahara02:10
*** RobLevas has quit IRC02:14
*** alazarev has joined #openstack-sahara02:21
*** alazarev has quit IRC02:28
*** IlyaE has joined #openstack-sahara02:37
*** zhiyan_ is now known as zhiyan02:38
*** zhiyan is now known as zhiyan_02:50
*** zhiyan_ is now known as zhiyan02:51
*** IlyaE has quit IRC03:02
*** RobLevas has joined #openstack-sahara03:10
*** RobLevas has quit IRC03:14
*** IlyaE has joined #openstack-sahara03:21
*** IlyaE has quit IRC03:39
*** ViswaV has joined #openstack-sahara04:11
*** ekarlso has quit IRC04:11
*** rhodgin has quit IRC04:11
*** ViswaV_ has quit IRC04:15
*** rhodgin has joined #openstack-sahara04:22
*** ekarlso has joined #openstack-sahara04:22
*** robsparker has quit IRC04:47
*** RobLevas has joined #openstack-sahara05:09
*** RobLevas has quit IRC05:14
*** Longgeek has joined #openstack-sahara05:22
*** Longgeek has quit IRC05:25
*** IlyaE has joined #openstack-sahara05:29
*** Longgeek has joined #openstack-sahara05:35
*** IlyaE has quit IRC05:58
*** tnovacik has joined #openstack-sahara05:58
*** ViswaV has quit IRC06:08
*** RobLevas has joined #openstack-sahara06:10
*** RobLevas has quit IRC06:14
*** Longgeek has quit IRC06:16
*** Longgeek has joined #openstack-sahara06:19
*** IlyaE has joined #openstack-sahara06:24
*** yiconglu has joined #openstack-sahara06:25
*** sballe_ has joined #openstack-sahara06:49
*** sballe has quit IRC06:53
*** ViswaV has joined #openstack-sahara06:59
*** julienvey has joined #openstack-sahara07:21
*** skolekonov has joined #openstack-sahara07:25
*** IlyaE has quit IRC07:40
*** jpich has joined #openstack-sahara07:55
*** IlyaE has joined #openstack-sahara07:56
openstackgerritA change was merged to openstack/sahara-specs: [EDP] Refactor job manager to support multiple implementations  https://review.openstack.org/10067807:59
*** IlyaE has quit IRC08:01
openstackgerritA change was merged to openstack/python-saharaclient: Updated from global requirements  https://review.openstack.org/10303408:03
openstackgerritlawrancejing proposed a change to openstack/sahara: Use immutable arg rather mutable arg  https://review.openstack.org/10379308:25
openstackgerritA change was merged to stackforge/sahara-ci-config: Resolve problem with java downloading  https://review.openstack.org/10317608:26
*** tmckay has quit IRC08:33
openstackgerritDmitry Mescheryakov proposed a change to openstack/sahara: Fix docs to use sahara-all instead of sahara-api  https://review.openstack.org/10051408:34
*** tosky has joined #openstack-sahara08:37
openstackgerritIvan Berezovskiy proposed a change to stackforge/sahara-ci-config: Use sudo to delete /tmp/sahara directory  https://review.openstack.org/10380008:41
openstackgerritA change was merged to stackforge/sahara-ci-config: Add the .gitignore file and update the pbr version  https://review.openstack.org/10360708:45
*** tmckay has joined #openstack-sahara08:57
*** IvanBerezovskiy1 has joined #openstack-sahara09:05
*** yiconglu has quit IRC09:13
openstackgerritDmitry Mescheryakov proposed a change to openstack/python-saharaclient: Make Cluster Template accept default_image_id field  https://review.openstack.org/10093309:14
*** witlessb has joined #openstack-sahara09:18
*** RobLevas has joined #openstack-sahara09:25
*** Longgeek has quit IRC09:49
*** Longgeek has joined #openstack-sahara09:50
*** Longgeek_ has joined #openstack-sahara10:00
*** witlessb has quit IRC10:02
*** witlessb has joined #openstack-sahara10:02
*** Longgeek has quit IRC10:03
*** skolekonov has quit IRC10:09
openstackgerritA change was merged to openstack/sahara: Fix docs to use sahara-all instead of sahara-api  https://review.openstack.org/10051410:13
openstackgerritSergey Reshetnyak proposed a change to openstack/sahara: Refactoring vanilla 2 plugin  https://review.openstack.org/9096110:21
*** RobLevas has quit IRC10:26
*** openstack has joined #openstack-sahara10:51
*** witlessb has quit IRC11:04
*** tmckay has quit IRC11:05
*** Longgeek_ has quit IRC11:08
*** Longgeek has joined #openstack-sahara11:09
*** witlessb has joined #openstack-sahara11:09
*** witlessb has quit IRC11:16
*** witlessb_ has joined #openstack-sahara11:16
*** RobLevas has joined #openstack-sahara11:28
*** RobLevas has quit IRC11:32
*** Longgeek_ has joined #openstack-sahara11:39
*** Longgeek has quit IRC11:43
*** ylobankov has joined #openstack-sahara11:49
*** skolekonov has joined #openstack-sahara11:49
*** witlessb_ has quit IRC11:53
*** witlessb has joined #openstack-sahara11:55
*** tmckay has joined #openstack-sahara12:04
openstackgerritA change was merged to stackforge/sahara-ci-config: Use sudo to delete /tmp/sahara directory  https://review.openstack.org/10380012:05
*** tmckay has quit IRC12:06
*** Longgeek_ has quit IRC12:08
*** Longgeek has joined #openstack-sahara12:08
*** Timotey has joined #openstack-sahara12:12
*** Longgeek has quit IRC12:13
*** tmckay has joined #openstack-sahara12:15
*** Longgeek has joined #openstack-sahara12:19
*** RobLevas has joined #openstack-sahara12:26
*** RobLevas has quit IRC12:30
*** Longgeek has quit IRC12:34
*** Longgeek has joined #openstack-sahara12:34
*** Longgeek_ has joined #openstack-sahara12:39
*** Longgee__ has joined #openstack-sahara12:40
*** Longgee__ has quit IRC12:40
*** Longgee__ has joined #openstack-sahara12:41
*** Longgeek_ has quit IRC12:41
*** Longgeek has quit IRC12:42
*** Longgeek has joined #openstack-sahara12:44
*** Longgee__ has quit IRC12:44
*** Longgeek has quit IRC12:52
*** Longgeek_ has joined #openstack-sahara12:52
*** witlessb has quit IRC12:54
*** Longgeek has joined #openstack-sahara12:56
*** xianghuihui has joined #openstack-sahara12:56
*** witlessb has joined #openstack-sahara12:57
*** Longgeek_ has quit IRC12:59
*** xianghui has quit IRC12:59
*** Longgeek has quit IRC13:03
*** RobLevas has joined #openstack-sahara13:04
*** Longgeek has joined #openstack-sahara13:04
*** _elmiko is now known as elmiko13:07
*** Longgeek_ has joined #openstack-sahara13:10
*** Longgeek has quit IRC13:14
*** _crobertsrh is now known as crobertsrh13:17
tmckayanybody remember the url for monitoring ambari progress during an  hdp cluster install?  I know there is something you can hit to show the actions that are pending/executing/completed but I forget14:06
elmikotmckay: i think its http://ambari-server-ip:844014:10
elmikoi think14:10
*** RobLevas has quit IRC14:14
*** RobLevas has joined #openstack-sahara14:15
*** RobLevas has quit IRC14:17
*** Longgeek_ has quit IRC14:25
*** Longgeek has joined #openstack-sahara14:26
*** IlyaE has joined #openstack-sahara14:27
*** RobLevas has joined #openstack-sahara14:40
openstackgerritLonggeek proposed a change to stackforge/sahara-guestagent: Update requirements file and rename savanna  https://review.openstack.org/10360814:49
tmckayelmiko, fyi, I think I was looking for something like this http://10.3.11.225:8080/api/v1/clusters/test-hdp/requests/14:49
elmikoahh, gotcha14:49
elmikoi had gotten into the ambari server page when i was having difficulty with the hdp stuff, and that server was showing me the status of all the nodes14:50
tmckay(followed by a number like /1 or /2 gives you individual tasks and subtasks.  I found some labeled "install all services" and "start all services"14:50
elmikooh yea, those are usually what we see in the sahara.log coming back as json from the server14:50
tmckayyeah, this is json status stuff that John showed me in Atlanta.  trusnak was having a problem with an install bailing on hive-server14:51
*** Longgeek has quit IRC14:51
tmckayelmiko, ^^ if you have any insight I'm sure he would love to talk to you tomorrow :)14:51
*** Longgeek has joined #openstack-sahara14:51
elmikoyea, John showed me how to make queries against that server to find out the tasks. interesting stuff14:51
elmikotmckay: i'm certinaly willing to listen in, not sure i can add much. but i'll give it my best :)14:52
RobLevastmckay: what do you mean by "bailing" ?  Are you using DevStack?14:53
RobLevasI am working on a problem where it appears that Sahara is failing to communicate with Ambari due to a network issue14:54
*** xianghuihui has quit IRC14:54
tmckayRobLevas, I mean the cluster install failed in Sahara, and the only thing in the logs shows that the ambari server had a problem installing the hive-server but there is not a lot of info on what happened14:54
tmckay(hdp cluster, obviously :) )14:54
RobLevastmckay: Ah... I heard there is a lack of error messages going around bummer.14:54
* tmckay looks for log messages14:55
toskyand the instances should have proper external network access14:55
tmckayhmm, not worth posting.  Looks to me like ambari reported that a subtask failed for hive-server during service install14:56
tmckayI don't have the whole log.14:56
RobLevasI am not familiar with the hive install... else I would gladly try to help out14:56
tmckaybut, sahara and ambari definitely communicated.  I'm thinking config issue or missing dependency or something.14:56
tmckayI'm not directly involved either, just received some questions :)14:57
RobLevasIf you had my issue, the sahara log would indicate a network error14:57
*** Longgeek has quit IRC15:03
*** shakamunyi has joined #openstack-sahara15:05
*** shakamunyi has quit IRC15:05
*** jpich has quit IRC15:15
*** jpich has joined #openstack-sahara15:15
dmitrymetmckay: re “Looks to me like ambari reported that a subtask failed for hive-server during service install”. I know that you can actually look at the specific task and see what exactly failed.15:26
tmckaydmitryme, through the REST interface?15:27
dmitrymeyep15:27
dmitrymeAmbari REST15:27
tmckayhttp://10.3.11.225:8080/api/v1/clusters/test-hdp/requests/2/tasks/5215:29
tmckaylike that, I think15:29
dmitrymeyep,that is what I mean15:29
dmitrymeif you access it, it can give a description15:29
elmikotmckay: apparently you can also issue commands to query tasks too15:29
tmckaydmitryme, thank you, that might be enough15:29
dmitrymethough it can be vague like “”puppet timed out” in my case15:29
elmikotmckay: https://github.com/apache/ambari/blob/trunk/ambari-server/docs/api/v1/index.md15:32
*** skolekonov has quit IRC15:33
tmckayelmiko, thank you15:33
*** IvanBerezovskiy1 has left #openstack-sahara15:38
*** ViswaV_ has joined #openstack-sahara15:47
*** ViswaV has quit IRC15:49
*** tnovacik has quit IRC16:01
*** zhiyan is now known as zhiyan_16:19
openstackgerritRobert Nettleton proposed a change to openstack/sahara: Adds DataNode decommissioning support to HDP Plugin  https://review.openstack.org/10359616:29
*** witlessb has quit IRC16:33
openstackgerritpritesh proposed a change to openstack/python-saharaclient: Fixes ambiguous cli output between "None" and NoneType  https://review.openstack.org/10295516:39
openstackgerritTrevor McKay proposed a change to openstack/sahara: Allow plugins to choose the EDP implementation  https://review.openstack.org/10101616:55
*** jpich has quit IRC17:05
*** witlessb has joined #openstack-sahara17:16
*** bandarji has joined #openstack-sahara17:21
*** bandarji has quit IRC17:23
*** bandarji has joined #openstack-sahara17:36
*** bandarji has quit IRC18:17
*** shakamunyi has joined #openstack-sahara18:41
*** shakamunyi has quit IRC18:41
*** shakamunyi has joined #openstack-sahara18:42
RobLevasHas anyone experienced intermittent network issues using DevStack (pulled from github a week ago)?18:47
elmikonot me, but i haven't done much provisioning this week18:48
RobLevasFrom my DevStack host, pinging a VM instance, I get this from time to time:18:48
RobLevas2014-07-01 11:19:00: 64 bytes from 172.21.194.4: icmp_req=499 ttl=63 time=0.335 ms18:48
RobLevas2014-07-01 11:19:01: 64 bytes from 172.21.194.4: icmp_req=500 ttl=63 time=0.540 ms18:48
RobLevas2014-07-01 11:19:02: 64 bytes from 172.21.194.4: icmp_req=501 ttl=63 time=0.205 ms18:48
RobLevas2014-07-01 11:19:37: 64 bytes from 172.21.194.4: icmp_req=536 ttl=63 time=0.387 ms18:48
RobLevas2014-07-01 11:19:38: 64 bytes from 172.21.194.4: icmp_req=537 ttl=63 time=0.327 ms18:48
RobLevas2014-07-01 11:19:39: 64 bytes from 172.21.194.4: icmp_req=538 ttl=63 time=0.471 ms18:48
RobLevasNotice the jump from 11:19:02 to 11:19:37?18:49
elmikohmm18:49
elmikonot sure why that would happen18:49
RobLevasme neither. :(18:49
elmikooh, dropped packets18:49
elmikoicmp_req goes up by 3518:49
RobLevasMaybe.. this is causing issue provisioning a cluster18:50
elmikocould be18:50
elmikois this still all on one machine?18:50
RobLevasyup18:50
RobLevasAround the same time I get this error from Sahara:18:51
RobLevas2014-07-01 11:19:59.703 30758 TRACE sahara.service.ops ConnectionError: HTTPConnectionPool(host='172.21.194.4', port=8080): Max retries exceeded with url: /api/v1/clusters/rlevas/requests/2/tasks?fields=Tasks/status (Caused by <class 'httplib.BadStatus18:51
elmikomakes sense given the ping results18:51
elmikowhat distro you running?18:52
RobLevasIt is possible the VM instance is halted or hung?18:52
elmikooh yea vm, could be having issues switching contexts. poor performance causing dropouts18:53
RobLevasUbuntu host, CentOS VM18:53
elmikoi've seen poor network performance caused by vm issues, but nothing concrete enough to make a bug out of it18:53
RobLevasbut the box doesnt appear to be loaded18:53
elmikomine didn't either, cpu never went above %30 load18:54
elmikohow big is the cluster you are creating?18:54
RobLevasthis happens all the time..18:54
RobLevasa single node cluster18:54
elmikowow...18:54
RobLevasit never fails to fail.. just at different points.18:55
openstackgerritVitaly Gridnev proposed a change to openstack/sahara: Corrected a number of pep8 errors  https://review.openstack.org/10184518:55
elmikohave you tried running devstack on your host? (i realize you might need to test centos, just curious)18:55
RobLevasOn my mac?18:55
elmikoi meant on the ubuntu host, or is that the vm?18:56
RobLevasubuntu is the base OS on the server.. that is where DevStack is running18:56
elmikook, and it's creating centos instances for the cluster?18:57
RobLevasyup.. sorry my terminology may not be up to speed yet18:57
elmikono worries, just trying to understand the setup18:57
elmikojust out of curiousity, is the ubuntu kernel using kvm?18:59
RobLevasI assume so, how do I tell?19:00
elmikoyou could `dmesg | grep kvm`19:00
RobLevasempty19:00
elmikowhat kernel is it?19:00
RobLevas3.2.0-64-generic #97-Ubuntu SMP Wed Jun 4 22:04:21 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux19:01
toskybetter try lsmod | grep kvm I guess19:01
elmikoyea, what tosky said19:01
RobLevas# lsmod | grep kvm19:01
RobLevaskvm_intel             137721  019:01
RobLevaskvm                   415622  1 kvm_intel19:01
RobLevastosky: thanks19:01
elmikothat looks good19:01
elmikois that server doing other stuff in addition to devstack?19:02
RobLevasnope.. im the only one on it. I have DevStack and a stand-along Sahara running... Sahara dashboard in running on my Mac19:03
elmikowow, strange19:03
RobLevasyup19:03
elmikomy guess would be some configuration related issue on the server19:04
elmikobut that's pretty generic19:04
elmikoif you want to get really deep, you could start your ping, and then monitor the network traffic on the virtual bridge to see what's happening19:04
RobLevashow do I monitor the traffic on the virtual bridge?19:05
elmikosec19:05
*** julienvey has quit IRC19:06
RobLevassorry, this may be getting too involved for you. I dont mean to take this much of your time,.19:06
elmikonah, it's ok. i need a break from working on this spec19:06
RobLevasfeel free to tell me to go and Google more stuff. :)19:06
RobLevasok.. thanks19:06
RobLevasspec writing can suck the life out of you19:07
elmikoi would recommend watching Phil Hopkins talk from the last summit about troubleshooting virtual networks. he shows some great techniques for attaching virtual interfaces to the bridge and then monitoring all traffic.19:08
elmikohttps://www.openstack.org/summit/openstack-summit-atlanta-2014/session-videos/presentation/troubleshooting-neutron-virtual-networks19:08
elmikoit's a really neat talk19:08
RobLevasawesome.. thanks I will take a look at that19:08
*** julienvey has joined #openstack-sahara19:08
elmikoit's involved, and you will want to know the tools available19:08
RobLevasi bet19:08
elmikobasically, what you can do is add a virtual interface to the ovs bridge, and then use tcpdump to watch all the traffic.19:10
elmikoyou should be able to watch your ping requests and see when they get rejected. it could provide insight as to why the network stops.19:10
elmikothis is basically, one more avenue of debugging19:11
elmikounfortunately no simple answer from me :/19:11
RobLevaswould i be able to tell if the virtual network hiccuped or the VM instance hung?19:12
*** julienvey has quit IRC19:12
elmikoyou should be able to see if all traffic stops, or the vm starts rejecting packets, or something similar19:12
RobLevasI think you have provided me with some great information on how I can start to debug this... its much appreciated.19:12
RobLevascool beans.19:13
elmikocool, i know debugging these things can be a serious pain19:13
RobLevasyeah.. I just hope the data will make sense to me once I get to the point where I can read it. :)19:13
elmikoyeeeeaaa....19:13
elmikothat's the next part :)19:13
RobLevas:)19:14
elmikohopefully you can determine which part is causing the breakdown, then figure out what to do19:14
elmikoi mean, if you could determine the vm is misbehaving, or the network is stopping, you could fix it19:14
*** IlyaE has quit IRC19:24
RobLevasi hope... it looks like kvm eats about 105% of the CPU and then things start to get funky..19:25
crobertsrhHere's some low hanging fruit (hopefully) reviews for the Sahara stuff in Horizon (well, at least the non-merge ones).  This should take care of all the additional changes that went in since the merge process started.19:35
crobertsrhhttps://review.openstack.org/#/q/owner:%22Chad+Roberts%22+status:open,n,z19:35
elmikogot yer back crobertsrh ;)19:43
crobertsrhThese reviews are considerably shorter :)19:43
crobertsrhProbably only 25-30 patch sets required19:44
elmikocool, some of the merge ones were real bears...19:44
elmikoLOL19:44
crobertsrhYeah.  I couldn't really come up with a better way to merge that much code and keep it in meaningful chunks.19:44
elmikohopefully we're finally rounding third base on those19:46
crobertsrhI'm really starting to have second, even third, thoughts about merging our goods into horizon.  It seems to be quite a black whole even for reviews of "core horizon" things that a lot of people should care about.  I'm losing confidence that future "data processing" changes will catch the attention of a critical mass of people to get it pushed through in a timely manner.19:51
elmikoi had a similar thought as i was looking at the changes that will be needed for the auth stuff19:51
crobertsrhThese "extra" patches that I just sent up should serve as a good canary to see how well "more manageable" changes can get merged.19:51
elmikocrobertsrh: were some of these changes you pulled out of the bigger merges?20:01
*** IlyaE has joined #openstack-sahara20:02
*** tmckay has quit IRC20:14
*** shakamunyi has quit IRC20:17
*** shakamunyi has joined #openstack-sahara20:18
*** shakamunyi has quit IRC20:30
*** IlyaE has quit IRC20:31
*** IlyaE has joined #openstack-sahara20:32
crobertsrhNo, they were added to sahara-dashboard after the merge started20:35
crobertsrhsort of a "forward porting" of those changes.20:35
elmikoahh, ok. i thought they looked familiar20:37
crobertsrhFor the record, you +1 https://review.openstack.org/#/c/92833/ :)  All of the sudden you're all picky :)20:38
* elmiko checking20:39
elmikoyea, it didn't occur to me then i guess lol20:39
elmikoi think it's because i looked at too many of the pep8 cleanup reviews20:39
crobertsrhHa!20:39
elmikomight as well cleanup now lol20:40
elmikoi feel so bad putting a -1 on any of these horizon reviews because i know what a pita it's been :/20:40
crobertsrhNah, these are much easier....no monster dep chain tagging along on the backend of these20:41
openstackgerritAndrew Lazarev proposed a change to openstack/sahara: Fixed volumes mount in case of existing volumes  https://review.openstack.org/10369120:42
elmikoyea, much easier to digest20:43
*** shakamunyi has joined #openstack-sahara20:45
*** IlyaE has quit IRC20:46
RobLevaselmiko: just got done watching that video you sent me a link to. Awesome, thanks.  Now to try out some of the tips. :)20:48
*** IlyaE has joined #openstack-sahara20:52
elmikoRobLevas: cool, i'm glad you enjoyed it :)20:53
elmikoand good luck!20:53
*** crobertsrh is now known as _crobertsrh20:56
RobLevasthanks20:57
*** ViswaV_ has quit IRC20:57
*** ViswaV has joined #openstack-sahara20:58
*** shakamunyi has quit IRC21:02
*** shakamunyi has joined #openstack-sahara21:03
*** shakamunyi has quit IRC21:03
*** shakamunyi has joined #openstack-sahara21:04
*** ghenriks has quit IRC21:05
*** IlyaE has quit IRC21:08
*** shakamunyi has quit IRC21:13
*** IlyaE has joined #openstack-sahara21:14
*** shakamunyi has joined #openstack-sahara21:14
*** tmckay has joined #openstack-sahara21:26
*** ViswaV_ has joined #openstack-sahara21:44
*** IlyaE has quit IRC21:44
*** IlyaE has joined #openstack-sahara21:45
*** ViswaV has quit IRC21:47
sballe_elmiko, Hi I now have my hadoop cluster working and running in devstack. Is there a place I can get a quick program to test taht it is actually working? I see reference in the docs to edp-wordcount but cannot find it. Thanks for anyhelp22:04
RobLevassballe_: Are you running an instance of HDP? If so, you can install Hue and run through the tutorials - http://hortonworks.com/tutorials/22:06
elmikoyea, that's a good place to start22:07
sballe_no I am running the vanilla plugin vanilla 1.2.122:07
RobLevassballe_: The latest version of the HDP plugin will install Hue for you.22:07
sballe_RobLevas, ok. Maybe I should switch22:08
RobLevassballe_: oh.. maybe the tutorials will work with that, but you may need to manually install Hue22:08
sballe_RobLevas, ok22:08
sballe_let me try... and I'll get back to yu :) thanks22:09
openstackgerritA change was merged to openstack/sahara: Use immutable arg rather mutable arg  https://review.openstack.org/10379322:12
sballe_RobLevas, What version of the HDP should I use? 1.3 or 2.0.6?22:12
elmikosballe_: also look at https://github.com/openstack/sahara-extra , it has some examples22:13
RobLevassballe_: 2.0.6 should have the option to install Hue22:13
sballe_oh cool! I'll start with wordcount and then move on to HDF22:14
*** IlyaE has quit IRC22:15
*** IlyaE has joined #openstack-sahara22:16
*** ViswaV has joined #openstack-sahara22:21
*** ViswaV_ has quit IRC22:23
*** elmiko is now known as _elmiko22:33
*** IlyaE has quit IRC22:39
*** IlyaE has joined #openstack-sahara22:40
*** IlyaE has quit IRC22:50
*** witlessb has quit IRC23:10
*** IlyaE has joined #openstack-sahara23:18
*** tosky has quit IRC23:26
*** RobLevas has quit IRC23:38

Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!