| *** crushil_ has joined #openstack-cyborg | 00:35 | |
| *** crushil_ has quit IRC | 01:36 | |
| *** crushil_ has joined #openstack-cyborg | 01:36 | |
| *** crushil_ has quit IRC | 02:23 | |
| *** crushil_ has joined #openstack-cyborg | 02:50 | |
| -openstackstatus- NOTICE: Due to an unexpected outage with Zuul (1 hour), you'll need to recheck any jobs that were in progress. Sorry for the inconvenience. | 02:56 | |
| *** masber has joined #openstack-cyborg | 07:53 | |
| masber | good afternoon, I just found you project, is cybord ready to be installed? | 08:03 |
|---|---|---|
| masber | I would like to be able to use FPGAs on openstack, can I get some help within this channel? | 08:04 |
| zhipengh[m] | Hi masber , thx for the interest ! | 08:21 |
| zhipengh[m] | Could u introduce a little bit about yourself and your requirement ? | 08:22 |
| masber | oh yes sorry | 08:22 |
| masber | I work for a medical research intitute | 08:22 |
| masber | and we are playing with openstack/ceph in order to see if it can fit out use cases | 08:23 |
| masber | now we are doing a project with intel | 08:23 |
| masber | and we will borrow some FPGA cards from them | 08:23 |
| zhipengh[m] | That sounds interesting :) | 08:24 |
| masber | I would like to see if we can make those cards to work on openstack, I mean been able to access the FPGAs from the instance | 08:24 |
| masber | I did a talk in the openstack summit in sydney | 08:24 |
| masber | I talk about what we do etc | 08:25 |
| masber | I am not an openstack expert just, installed it and played with it a little bit | 08:25 |
| masber | but now with these FPGAs coming I feel I should start getting my hands dirty | 08:26 |
| zhipengh[m] | Cool ! Can you provide the title of your Sydney topic ? So that I could search on youtube :) | 08:26 |
| zhipengh[m] | As for cyborg | 08:26 |
| masber | so, if I am not mistaken we are receiving some nallatech 385a | 08:27 |
| zhipengh[m] | Our current code base cannot function yet, and we are undergoing active development | 08:27 |
| masber | the name of the talk was "building a genome future with openstack" | 08:27 |
| masber | https://www.youtube.com/watch?v=HSAJpAxORAw | 08:28 |
| zhipengh[m] | I will definitely check it out :) | 08:28 |
| masber | it is quite basic talk | 08:28 |
| zhipengh[m] | So at the moment you could run up the three main cyborg processes, but no function yet | 08:28 |
| masber | I see | 08:29 |
| zhipengh[m] | We are now try to define data models | 08:29 |
| zhipengh[m] | And would be great to have input from you | 08:29 |
| zhipengh[m] | For example on FPGA | 08:29 |
| masber | ok, I am not an user with lots of expertise on these kind of things, but happy to give it a go | 08:29 |
| zhipengh[m] | What is your main use case for that and what would you like to see FPGA resource represented | 08:30 |
| zhipengh[m] | That'd be great ! | 08:30 |
| masber | this is to run a tool called GATK4 | 08:30 |
| masber | it is a best practice set of tools to run genome analysis | 08:30 |
| masber | so this tool can work on HPC or in a SPARK cluster | 08:31 |
| masber | our goal is to see how much efficiency we can get by running these tools on FPGAs | 08:31 |
| masber | compared to running on Apache Spark | 08:31 |
| masber | the thing is I am receiving the cards tomorrow from the intel guy who is coming from overseas (not sure from singapose, china or US) | 08:32 |
| masber | *singapore | 08:32 |
| masber | and I am not sure for how long I can keep the cards either | 08:32 |
| zhipengh[m] | that is very interesting | 08:33 |
| masber | another problem is that currently I am deploying openstack using kolla... | 08:34 |
| zhipengh[m] | Ah~ | 08:34 |
| zhipengh[m] | I think we should first look into GATK4 , see what are the resource requirements for running it on the FPGA card | 08:36 |
| zhipengh[m] | Those requirements could compose what we call accelerator capability in cyborg | 08:37 |
| masber | I think I can get that information from intel I think | 08:37 |
| masber | I see | 08:37 |
| zhipengh[m] | And then I think our team could come up the very first data model outline for FPGA | 08:37 |
| masber | I think intel got GATK4 running on FPGAs | 08:38 |
| zhipengh[m] | With a real use case :) | 08:38 |
| masber | yes my use case is quite real because we are planing to deploy this pipeline in production | 08:38 |
| masber | we are actually running GATK3 in production | 08:38 |
| zhipengh[m] | BTW we will have video conference about 6:20 hour later | 08:38 |
| zhipengh[m] | Would it be possible for you to join ? | 08:38 |
| masber | in 6.20 hours | 08:39 |
| masber | let me check | 08:39 |
| zhipengh[m] | Haha right | 08:39 |
| masber | 2:00 am my time | 08:39 |
| masber | yes I can join | 08:39 |
| zhipengh[m] | Wow that is pretty late | 08:39 |
| zhipengh[m] | Anyways I will send out the details later on openstack-dev mailing list | 08:40 |
| masber | yep, well as I said Im not a quite competent openstack user so most of the time I have to stay till late so I can get help from the community | 08:40 |
| masber | many nights working until 4am | 08:40 |
| masber | but it is ok, I am happy to try and see if I can help you | 08:41 |
| zhipengh[m] | Man that is hardcore | 08:41 |
| masber | my boss gives me free days | 08:41 |
| masber | it is ok | 08:41 |
| zhipengh[m] | :) | 08:41 |
| masber | plus I'm a pragmatic person which means I learn by experience so for me it is quite good to get in contact with people in get things done | 08:42 |
| masber | anyway | 08:42 |
| masber | yeah I will join the call so if there any doubt I can talk with the intel guy tomorrow | 08:42 |
| masber | he is giving me the cards and then we will have a meeting | 08:43 |
| masber | so maybe I can ask questions in your behalf | 08:43 |
| masber | so how do you do the video conference? | 08:46 |
| masber | skype? | 08:47 |
| masber | google hangouts? | 08:47 |
| zhipengh[m] | ZOOM | 08:48 |
| zhipengh[m] | Very good quality | 08:50 |
| masber | ok | 08:50 |
| masber | ok I will see you later, Im going home now | 08:53 |
| zhipengh[m] | Take care man ! | 08:59 |
| *** crushil_ has quit IRC | 10:30 | |
| *** mpaolino has joined #openstack-cyborg | 14:36 | |
| *** zhipeng has joined #openstack-cyborg | 14:48 | |
| *** crushil has joined #openstack-cyborg | 14:49 | |
| *** masuberu has joined #openstack-cyborg | 14:55 | |
| masuberu | hi | 14:56 |
| masuberu | is the meeting happening? | 14:56 |
| zhipeng | hi will start momentarily | 14:57 |
| masuberu | ok | 14:58 |
| masuberu | I don't know how to access tho | 14:58 |
| masuberu | and also I can't talk as it is 2am and everybody is sleeping at home but I can chat and listen | 14:59 |
| masuberu | just ping me when people is ready | 15:00 |
| zhipeng | okey no problem, let me send the zoom link here | 15:00 |
| zhipeng | #startmeeting openstack-cyborg | 15:00 |
| openstack | Meeting started Wed Nov 15 15:00:29 2017 UTC and is due to finish in 60 minutes. The chair is zhipeng. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| *** openstack changes topic to " (Meeting topic: openstack-cyborg)" | 15:00 | |
| openstack | The meeting name has been set to 'openstack_cyborg' | 15:00 |
| zhipeng | #link https://zoom.us/j/973513277 | 15:01 |
| zhipeng | just did a little bit chatting with crushil, that we will keep irc meeting today, and move the zoom meeting to a later time slot for dev specific issue | 15:06 |
| zhipeng | will keep everybody posted | 15:06 |
| masuberu | so no zoom, just irc for now? | 15:07 |
| *** shaohe_feng_ has joined #openstack-cyborg | 15:08 | |
| zhipeng | masuberu yep :) | 15:08 |
| zhipeng | crushil you there ? | 15:08 |
| *** Li has joined #openstack-cyborg | 15:08 | |
| masuberu | ok | 15:09 |
| shaohe_feng_ | zhipeng: what's the zoom conference ID? | 15:09 |
| *** Li is now known as Guest69871 | 15:09 | |
| zhipeng | shaohe_feng_ will do zoom tomorrow morning | 15:09 |
| shaohe_feng_ | zhipeng: got it. | 15:09 |
| zhipeng | could everyone log their names ? | 15:10 |
| zhipeng | #info Howard | 15:10 |
| zhuli | Can’t get my pc connected, have to use cellphone | 15:10 |
| mpaolino | #info Michele | 15:10 |
| Guest69871 | #info Li | 15:10 |
| zhipeng | zhuli we will just irc tonight | 15:10 |
| zhuli | #info zhuli | 15:10 |
| zhuli | Got it | 15:10 |
| Guest69871 | Do we have another meeting tomorrow? | 15:11 |
| crushil | I'm here | 15:11 |
| zhipeng | okey seems like full house almost :) | 15:11 |
| crushil | #info Rushil | 15:11 |
| zhipeng | #topic GATK4 FPGA requirement discussion | 15:11 |
| *** openstack changes topic to "GATK4 FPGA requirement discussion (Meeting topic: openstack-cyborg)" | 15:11 | |
| zhipeng | the first order of business today is that we have masuberu | 15:12 |
| masuberu | o/ | 15:12 |
| zhipeng | system engineer from one of the largest genome testing reserach testing centers in the world | 15:12 |
| zhuli | Welcome masuberu | 15:13 |
| masuberu | thank you all for having me here | 15:13 |
| zhipeng | to talk with us about their upcoming trail with Intel FPGA card to run GTAK4 application for genome analysis | 15:13 |
| zhipeng | the floor is yours masuberu :) | 15:13 |
| masuberu | than you | 15:13 |
| masuberu | I work for a medical research intitute | 15:14 |
| *** shaohe_feng__ has joined #openstack-cyborg | 15:14 | |
| masuberu | and as sequence and analyse genomes | 15:14 |
| shaohe_feng__ | #info shaohe | 15:14 |
| *** shaohe_feng_ has quit IRC | 15:14 | |
| masuberu | right now we are in a project with intel | 15:14 |
| masuberu | they are helping us to build an environment to run GATK4 which a set of tools of best practice genome analysis | 15:15 |
| masuberu | tomorrow a guy from intel is coming to my office to give me FPGA cards | 15:15 |
| masuberu | because GATK4 can either work on HPC cluster, Spark cluster and GDC | 15:15 |
| masuberu | so my idea is to test GATK4 on FPGAs through openstack and compare performance with apache spark | 15:16 |
| masuberu | obviously FPGA should be much faster but we want to see how much faster | 15:16 |
| masuberu | and I was looking for documentation about how to integrate FPGAs on openstack and I found your project | 15:17 |
| zhipeng | shaohe_feng_ is from Intel :) | 15:17 |
| masuberu | so wanted to say hi and see if this would be a possibility of collaboration | 15:17 |
| masuberu | ok | 15:18 |
| masuberu | well intel is quite big | 15:18 |
| masuberu | I guess | 15:18 |
| shaohe_feng__ | masuberu: hi | 15:18 |
| zhipeng | haha yes | 15:18 |
| Guest69871 | Hi Masuberu, This is Li from Huawei, our team has a lot of experience using FPGAs in Openstack | 15:18 |
| masuberu | the person coming tomorrow is from either China or Singapore I think | 15:18 |
| zhipeng | masuberu could you describe a possible setup ? | 15:18 |
| zhipeng | For example if you are gonna have a VM connected to the FPGA | 15:19 |
| zhipeng | or just feeding data directly to GTAK4 on that FPGA ? | 15:19 |
| masuberu | what ever is easier, I was thinking of giving a go and see if could be possible to integrate FPGAs with nova and see if the vms could see it | 15:20 |
| shaohe_feng__ | masuberu: I'm also working on FPGA. And some of us has started to support FPGA on k8s | 15:20 |
| masuberu | otherwise I am happy of using different methods | 15:20 |
| masuberu | ok | 15:20 |
| masuberu | I am totally open to suggestions | 15:20 |
| masuberu | the setup is | 15:20 |
| masuberu | 4 compute nodes with AMD cpus (56 cores) 512 RAM and ~4.5TB spinning disks 10k | 15:21 |
| Guest69871 | Do you have the developed IPs to use for the FPGA? | 15:21 |
| zhipeng | #info masuberu from Garvan Institute of Medical Research | 15:21 |
| masuberu | and 3 nodes with intel cpus (numa 2 sockets 14 cpus each) 512 GB RAM and 20 TB nvmes | 15:21 |
| zhipeng | #link https://www.youtube.com/watch?v=HSAJpAxORAw | 15:21 |
| masuberu | all connected through 2x25GB bond network | 15:21 |
| masuberu | network is mellanox ethernet | 15:22 |
| masuberu | mellanos switch 100GB | 15:22 |
| *** Guest69871 has quit IRC | 15:22 | |
| masuberu | it is not big setup but I think it is a good starting point | 15:22 |
| *** LiLiu_ has joined #openstack-cyborg | 15:23 | |
| LiLiu_ | #info Li | 15:23 |
| masuberu | I don't know how many FPGAs I will get, I will know tomorrow | 15:23 |
| zhipeng | shaohe_feng_ any tips on nova integrates with FPGA ? | 15:23 |
| masuberu | so I am not sure if that explains most of the questions? please feel free to ask | 15:24 |
| zhipeng | masuberu will GTAK4 run as an IP for the FPGA ? | 15:24 |
| masuberu | tomorrow I have a meeting with Albert from intel and he will brief me about how it works | 15:25 |
| LiLiu_ | Question: is there multiple IPs to be used in the GATK4 | 15:25 |
| masuberu | LiLiu_, what do you mean by multiple IPs? | 15:25 |
| LiLiu_ | I mean different bitstreams | 15:26 |
| masuberu | so GATK4 is a bunch of tools bioinformatitians use to create pipelines | 15:26 |
| masuberu | you can run a tool at a time or multiple ones at the same time, depends on what you are doing | 15:27 |
| zhipeng | #link https://github.com/broadinstitute/gatk | 15:27 |
| LiLiu_ | OK, I assume you will have multiple types of IPs in your system. Do you chain the FPGAs together or each FPGA only talk to the tied CPU ? | 15:28 |
| masuberu | ok, I will ask that question to the intel guy tomorrow | 15:29 |
| masuberu | I honestly never used a FPGA before | 15:29 |
| zhipeng | shaohe_feng__ any tips on nova integrates with FPGA ? | 15:29 |
| LiLiu_ | I see, my team works a lot on FPGA heterogeneous system in Openstack, we may talk more offline if you are interested, masuberu. | 15:31 |
| masuberu | sure | 15:31 |
| LiLiu_ | liliu1@huawei.com | 15:31 |
| zhipeng | did some googling | 15:33 |
| LiLiu_ | I wanna know more about the scope/size of the system you guys are targeting | 15:33 |
| masuberu | it will depends on the performance of this tests | 15:33 |
| zhipeng | seems like the pairHMM will be the part that need acceleration from FPGA or GPU ? | 15:33 |
| masuberu | for instance we have capacity to sequence 1200 genomes per month so it all depends how much hardware we need for that | 15:34 |
| zhipeng | #link https://www.altera.com/content/dam/altera-www/global/en_US/pdfs/literature/wp/wp-accelerating-genomics-opencl-fpgas.pdf | 15:34 |
| shaohe_feng__ | zhipeng: actually, every accelerator in Intel's FPGA is a sriov device. | 15:34 |
| masuberu | zhipeng, yep I got that document yesterday from Albert | 15:35 |
| zhipeng | shaohe_feng__ how about the altera ones ? | 15:35 |
| zhipeng | the same ? | 15:35 |
| shaohe_feng__ | zhipeng: altera means intel :) | 15:35 |
| zhipeng | haha | 15:35 |
| masuberu | nallatech 385A is what I am getting | 15:36 |
| shaohe_feng__ | zhipeng: there maybe 3 method to support FPGA. | 15:36 |
| masuberu | getting for testing | 15:36 |
| shaohe_feng__ | zhipeng: 1. pre-programming FPGA on the host. | 15:36 |
| shaohe_feng__ | zhipeng: 2. the orchestration program FPGA | 15:38 |
| shaohe_feng__ | zhipeng: 3. VM program FPGA. | 15:38 |
| shaohe_feng__ | zhipeng: I have got a FPGA server, and I'm trying to do some test on it | 15:38 |
| masuberu | shaohe_feng__, the person coming tomorrow is called AlbertZQ Wang, not sure if you know him... | 15:38 |
| shaohe_feng__ | zhipeng: AlbertZQ Wang from intel? | 15:39 |
| shaohe_feng__ | ^ masuberu: | 15:39 |
| masuberu | yes | 15:39 |
| shaohe_feng__ | which site? China or US | 15:40 |
| masuberu | I guess China but not 100% sure | 15:40 |
| masuberu | maybe either of them | 15:40 |
| masuberu | any questions? | 15:41 |
| shaohe_feng__ | masuberu: haha, we can connect each other convenience in intel's internal connection tool. | 15:41 |
| masuberu | yeah they showed me that tool last time they came to my office | 15:42 |
| masuberu | any plan? | 15:42 |
| zhipeng | i think this is definitely a very interesting use case for us | 15:42 |
| zhipeng | it would be great if you could come back with more details from the Intel guys tomorrow | 15:43 |
| zhipeng | and we could hammer out details for future work | 15:43 |
| zhipeng | i think getting Nova connect with FPGA is easy | 15:43 |
| zhipeng | but how to sched and manage your FPGAs is what we will be working on | 15:44 |
| masuberu | zhipeng, you mean because of SRIOV? | 15:44 |
| masuberu | it can be nova or k8s | 15:44 |
| masuberu | or another method | 15:44 |
| zhipeng | yes because the current support for Nova is good enough | 15:45 |
| zhipeng | to set it up | 15:45 |
| masuberu | ok | 15:45 |
| masuberu | what about overhead? | 15:45 |
| zhipeng | what type of overhead you are thinking about ? | 15:45 |
| masuberu | is it going to have much overhead due to virtualization? | 15:45 |
| zhipeng | i think that depends | 15:45 |
| zhipeng | how you setup the workload between VM and FPGA | 15:46 |
| masuberu | ok, is there anything I can read about that? because I don't know much to be honest | 15:46 |
| zhipeng | for example if you gonna have FPGA do all the GATK heavy lifting and VM just for running a data pipe application | 15:46 |
| masuberu | and I want to get the most of this project | 15:46 |
| zhipeng | i don't think there will be much overhead | 15:47 |
| masuberu | I don't know for how long I can have the FPGAs | 15:47 |
| masuberu | lets say we decide to run on the numa nodes, then I can even enable cpu afinity | 15:47 |
| masuberu | to make sure process use local memory | 15:48 |
| masuberu | but that is a different story | 15:48 |
| zhipeng | yes | 15:48 |
| masuberu | I am more worry about the lack of documentation about openstack+FPGAs | 15:48 |
| zhipeng | you can just go through the nova pci support docs | 15:49 |
| masuberu | also we have another project coming to run GPUs for machine learning stuff | 15:49 |
| masuberu | but that will come in 3-4 months time | 15:49 |
| masuberu | ok | 15:49 |
| zhipeng | cool :) | 15:49 |
| masuberu | another thing I normally deploy openstack using kolla... | 15:50 |
| masuberu | kolla-ansible, not sure if that will be an issue | 15:50 |
| zhipeng | crushil | 15:50 |
| zhipeng | any thoughts on making cyborg kolla-able ? :P | 15:50 |
| crushil | zhipeng, My coworker who presented for us is kolla-kubernetes core | 15:51 |
| masuberu | nice | 15:51 |
| crushil | I'm working with him to get Cyborg kollafied | 15:51 |
| zhipeng | masuberu there you go :) | 15:51 |
| masuberu | that's cool | 15:51 |
| zhipeng | we have everyone you need in cyborg :P | 15:51 |
| crushil | It's already on my plate for a future sprint | 15:51 |
| masuberu | I am quite flexible we can do whatever we want with this nodes as they are running nothing | 15:51 |
| zhipeng | masuberu just give a howler any time you have an issue on this channel | 15:52 |
| zhipeng | core members are always logged in , so as long as any of us are awake, we will respond :) | 15:52 |
| crushil | Btw you should play with this tool https://review.openstack.org/#/c/487972/. I will be integrating Cyborg into this tool | 15:52 |
| zhipeng | you could also just send email to the openstack-dev mailinglist, or openstack-ops | 15:53 |
| masuberu | I think I know rwellum | 15:55 |
| masuberu | ok | 15:55 |
| masuberu | by the way GATK4 is just a jar file | 15:56 |
| masuberu | very simple to install and run | 15:56 |
| masuberu | is it possible to share an FPGU with multiple instances? | 15:57 |
| masuberu | or do I need to assign 1 FPGU card per vm? | 15:57 |
| LiLiu_ | if the FPGA has multiple PFs/VFs, then yes | 15:57 |
| *** zhipeng has quit IRC | 15:57 | |
| LiLiu_ | It really depends on how the IP is developed | 15:58 |
| *** zhipeng has joined #openstack-cyborg | 15:58 | |
| masuberu | ok, and the FPGUs has cores? | 15:58 |
| LiLiu_ | nope | 15:59 |
| masuberu | I mean can I break the resources of a FPGU card and assign it to multiple vms? | 15:59 |
| LiLiu_ | FPGA can be virtualized and shared by multiple VMs | 15:59 |
| LiLiu_ | you are right | 16:00 |
| masuberu | for example with memory and cpus I can "define" flavors, can I do same with FPGUs? | 16:00 |
| LiLiu_ | yup | 16:01 |
| LiLiu_ | That's one of the most common ways of managing FPGA in Openstack | 16:01 |
| masuberu | like for instance 0.3Teraflops for one instance and 0.7Teraflops to another | 16:01 |
| masuberu | ok | 16:01 |
| LiLiu_ | That's totally doable | 16:02 |
| masuberu | do you have experience with openhpc? | 16:02 |
| LiLiu_ | I don | 16:02 |
| masuberu | we run rocks cluster but the development is quite slow, I am wondering whether openhpc would be better option | 16:02 |
| masuberu | ok | 16:02 |
| LiLiu_ | but I know someone in my team might do | 16:02 |
| masuberu | ok, thats ok, let try and focus on these tests comparing GATK4 on FPGUs vs Apache Spark | 16:03 |
| masuberu | maybe this type of tests doesn't even makes sense, we will see | 16:03 |
| masuberu | so I will get the cards tomorrow and connected to the servers | 16:04 |
| LiLiu_ | sure | 16:04 |
| masuberu | and from there if anyone know any document about how to integrate nova with fpgus or if there is any document/tool within cyborg project just let me know | 16:05 |
| LiLiu_ | before cyborg is ready to use, I think the most naive way is just use PCI passthrough | 16:06 |
| masuberu | sure | 16:07 |
| crushil | zhipeng, Don't forget to close the meeting | 16:13 |
| zhipeng | of course :) | 16:13 |
| zhipeng | if nothing else, we had a great discussion today | 16:14 |
| zhipeng | the meeting is adjourned | 16:14 |
| zhipeng | and I will let everyone know the new ZOOM meeting time | 16:14 |
| zhipeng | #endmeeting | 16:14 |
| *** openstack changes topic to "#openstack-cyborg" | 16:14 | |
| openstack | Meeting ended Wed Nov 15 16:14:28 2017 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:14 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-11-15-15.00.html | 16:14 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-11-15-15.00.txt | 16:14 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/openstack_cyborg/2017/openstack_cyborg.2017-11-15-15.00.log.html | 16:14 |
| masuberu | thank you all | 16:22 |
| masuberu | and sorry I hijack the whole meeting, I guess you probably had other things to talk | 16:22 |
| masuberu | it is 3.22am here so I am going to sleep now | 16:22 |
| masuberu | please feel free to send me an email if any idea pops up manuel.sb@garvan.org.au or masber@hotmail.com | 16:23 |
| masuberu | nite | 16:23 |
| zhipeng | nite :) | 16:26 |
| *** zhipeng has quit IRC | 16:28 | |
| jkilpatr | did we start at 10am this time? I was off by an hour :( | 16:50 |
| *** shaohe_feng__ has quit IRC | 16:56 | |
| *** masuberu has quit IRC | 16:57 | |
| *** masuberu has joined #openstack-cyborg | 17:06 | |
| *** masber has quit IRC | 17:15 | |
| *** crushil has quit IRC | 17:21 | |
| *** crushil has joined #openstack-cyborg | 17:32 | |
| *** crushil has quit IRC | 17:49 | |
| *** mpaolino has quit IRC | 17:51 | |
| *** crushil has joined #openstack-cyborg | 18:13 | |
| *** LiLiu_ has quit IRC | 18:54 | |
| *** crushil has quit IRC | 19:31 | |
| *** crushil has joined #openstack-cyborg | 19:47 | |
| *** masber has joined #openstack-cyborg | 23:32 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!