| *** dtrainor_ has quit IRC | 00:04 | |
| *** macza has quit IRC | 00:04 | |
| *** mriedem has quit IRC | 00:05 | |
| *** slaweq has joined #openstack-meeting | 00:11 | |
| *** slaweq has quit IRC | 00:15 | |
| *** rossella_s has quit IRC | 00:18 | |
| *** rossella_s has joined #openstack-meeting | 00:19 | |
| *** tetsuro has joined #openstack-meeting | 00:21 | |
| *** longkb has joined #openstack-meeting | 00:24 | |
| *** rossella_s has quit IRC | 00:36 | |
| *** rossella_s has joined #openstack-meeting | 00:37 | |
| *** longkb has quit IRC | 00:48 | |
| *** rossella_s has quit IRC | 00:54 | |
| *** rossella_s has joined #openstack-meeting | 00:54 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:55 | |
| *** jamesmcarthur has quit IRC | 01:03 | |
| *** rossella_s has quit IRC | 01:03 | |
| *** rossella_s has joined #openstack-meeting | 01:04 | |
| *** efried has joined #openstack-meeting | 01:15 | |
| *** ijw has joined #openstack-meeting | 01:16 | |
| *** ijw has quit IRC | 01:21 | |
| *** cloudrancher has quit IRC | 01:22 | |
| *** hongbin has joined #openstack-meeting | 01:23 | |
| *** cloudrancher has joined #openstack-meeting | 01:23 | |
| *** Leo_m has joined #openstack-meeting | 01:29 | |
| *** Leo_m has quit IRC | 01:29 | |
| *** rossella_s has quit IRC | 01:30 | |
| *** rossella_s has joined #openstack-meeting | 01:30 | |
| *** yamahata has quit IRC | 01:39 | |
| *** iyamahat has quit IRC | 01:39 | |
| *** efried has quit IRC | 02:10 | |
| *** efried1 has joined #openstack-meeting | 02:10 | |
| *** slaweq has joined #openstack-meeting | 02:11 | |
| *** efried1 is now known as efried | 02:12 | |
| *** dmacpher_ has joined #openstack-meeting | 02:13 | |
| *** rossella_s has quit IRC | 02:14 | |
| *** rossella_s has joined #openstack-meeting | 02:14 | |
| *** slaweq has quit IRC | 02:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:23 | |
| *** rossella_s has quit IRC | 02:25 | |
| *** rossella_s has joined #openstack-meeting | 02:26 | |
| *** yamamoto has quit IRC | 02:32 | |
| *** yamamoto has joined #openstack-meeting | 02:33 | |
| *** yamamoto has quit IRC | 02:37 | |
| *** rbudden has joined #openstack-meeting | 02:46 | |
| *** rossella_s has quit IRC | 02:52 | |
| *** rossella_s has joined #openstack-meeting | 02:52 | |
| *** jamesmcarthur has quit IRC | 03:01 | |
| *** longkb has joined #openstack-meeting | 03:04 | |
| *** yamamoto has joined #openstack-meeting | 03:05 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:09 | |
| *** yamamoto has quit IRC | 03:12 | |
| *** ykatabam has quit IRC | 03:13 | |
| *** hongbin has quit IRC | 03:15 | |
| *** rossella_s has quit IRC | 03:22 | |
| *** rossella_s has joined #openstack-meeting | 03:23 | |
| *** diablo_rojo has quit IRC | 03:27 | |
| *** jamesmcarthur has quit IRC | 03:28 | |
| *** rossella_s has quit IRC | 03:28 | |
| *** rossella_s has joined #openstack-meeting | 03:29 | |
| *** rbudden has quit IRC | 03:32 | |
| *** ijw has joined #openstack-meeting | 03:36 | |
| *** sagara has joined #openstack-meeting | 03:46 | |
| *** yamamoto has joined #openstack-meeting | 03:47 | |
| *** tpatil has joined #openstack-meeting | 03:58 | |
| tpatil | Hi Masakari members | 04:01 |
|---|---|---|
| tpatil | Today Sampath has taken day-off so I would like to chair today’s masakari meeting | 04:01 |
| *** rossella_s has quit IRC | 04:02 | |
| *** rossella_s has joined #openstack-meeting | 04:02 | |
| sagara | Hi | 04:03 |
| tpatil | #startmeeting Masakari | 04:04 |
| openstack | Meeting started Tue Oct 2 04:04:08 2018 UTC and is due to finish in 60 minutes. The chair is tpatil. Information about MeetBot at http://wiki.debian.org/MeetBot. | 04:04 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 04:04 |
| *** openstack changes topic to " (Meeting topic: Masakari)" | 04:04 | |
| openstack | The meeting name has been set to 'masakari' | 04:04 |
| tpatil | In the last meeting, I had requested to review few patches | 04:05 |
| tpatil | sagara: did you get time to review those patches? | 04:06 |
| *** tpatil_ has joined #openstack-meeting | 04:06 | |
| sagara | Yes, I remembered. But not yet. Could you write the those gerrit link? | 04:07 |
| tpatil_ | #link : https://review.openstack.org/#/c/590735/ | 04:07 |
| tpatil_ | #link : https://review.openstack.org/#/c/591557/ | 04:08 |
| sagara | I'll check them after IRC | 04:09 |
| tpatil_ | Thank you | 04:09 |
| tpatil_ | #topic : Bugs (stuck/critical) | 04:10 |
| *** tpatil has quit IRC | 04:10 | |
| tpatil_ | https://bugs.launchpad.net/masakari/+bug/1779752 | 04:11 |
| openstack | Launchpad bug 1779752 in masakari-monitors "masakari segment-list returns error (openstack queens)" [Critical,Confirmed] - Assigned to Shilpa Devharakar (shilpasd) | 04:11 |
| *** slaweq has joined #openstack-meeting | 04:11 | |
| tpatil_ | This bug is fixed in the above listed proposed patches. It's marked as critical so we should merge it ASAP | 04:11 |
| *** rossella_s has quit IRC | 04:12 | |
| tpatil_ | There are two other high priority bugs assigned to samP | 04:12 |
| tpatil_ | He is not available today so we can skip discussion about these two bugs | 04:12 |
| tpatil_ | Is there any other bugs that you want to discuss now? | 04:13 |
| *** rossella_s has joined #openstack-meeting | 04:14 | |
| sagara | Yes, let's skip discussion. But could you write down those samP's gerrit urls just in case. | 04:14 |
| tpatil_ | sagara: I missed to inform you about this patch : https://review.openstack.org/#/c/562973/ | 04:14 |
| tpatil_ | Please review the above patch as well | 04:15 |
| *** ykatabam has joined #openstack-meeting | 04:15 | |
| tpatil_ | From my end, we are making slow progress on adding notifications and also writing functional tests in openstacksdk | 04:15 |
| *** slaweq has quit IRC | 04:16 | |
| sagara | OK, I will review 590735, 591557, 562973 as soon as possible | 04:16 |
| tpatil_ | Thank you | 04:17 |
| tpatil_ | Let's end this meeting if there is no other topic left for discussion | 04:19 |
| sagara | I don't have other special topics | 04:19 |
| tpatil_ | Ok, Let's end this meeting then | 04:20 |
| sagara | Bye | 04:20 |
| tpatil_ | Thank you | 04:20 |
| *** tpatil has joined #openstack-meeting | 04:21 | |
| tpatil | #endmeeting | 04:22 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 04:22 | |
| openstack | Meeting ended Tue Oct 2 04:22:03 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 04:22 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-10-02-04.04.html | 04:22 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-10-02-04.04.txt | 04:22 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/masakari/2018/masakari.2018-10-02-04.04.log.html | 04:22 |
| *** tpatil has quit IRC | 04:22 | |
| *** tpatil_ has quit IRC | 04:26 | |
| *** rossella_s has quit IRC | 04:26 | |
| *** rossella_s has joined #openstack-meeting | 04:28 | |
| *** pcaruana has joined #openstack-meeting | 04:36 | |
| *** pcaruana has quit IRC | 04:43 | |
| *** rossella_s has quit IRC | 04:44 | |
| *** rossella_s has joined #openstack-meeting | 04:45 | |
| *** sagara has quit IRC | 04:57 | |
| *** rossella_s has quit IRC | 05:02 | |
| *** rossella_s has joined #openstack-meeting | 05:02 | |
| *** e0ne has joined #openstack-meeting | 05:06 | |
| *** e0ne has quit IRC | 05:06 | |
| *** slaweq has joined #openstack-meeting | 05:11 | |
| *** slaweq has quit IRC | 05:15 | |
| *** rossella_s has quit IRC | 05:16 | |
| *** rossella_s has joined #openstack-meeting | 05:16 | |
| *** rossella_s has quit IRC | 05:22 | |
| *** rossella_s has joined #openstack-meeting | 05:24 | |
| *** rossella_s has quit IRC | 05:34 | |
| *** rossella_s has joined #openstack-meeting | 05:34 | |
| *** rossella_s has quit IRC | 05:44 | |
| *** zaneb has quit IRC | 05:45 | |
| *** rossella_s has joined #openstack-meeting | 05:50 | |
| *** zaneb has joined #openstack-meeting | 05:50 | |
| *** awaugama has quit IRC | 05:56 | |
| *** rossella_s has quit IRC | 06:01 | |
| *** Luzi has joined #openstack-meeting | 06:02 | |
| *** rossella_s has joined #openstack-meeting | 06:03 | |
| *** rossella_s has quit IRC | 06:03 | |
| *** rossella_s has joined #openstack-meeting | 06:04 | |
| *** slaweq has joined #openstack-meeting | 06:11 | |
| *** slaweq has quit IRC | 06:16 | |
| *** yamamoto has quit IRC | 06:27 | |
| *** yamamoto has joined #openstack-meeting | 06:27 | |
| *** d0ugal has joined #openstack-meeting | 06:32 | |
| *** Emine has quit IRC | 06:36 | |
| *** dims has quit IRC | 06:38 | |
| *** dims has joined #openstack-meeting | 06:44 | |
| *** dims has quit IRC | 06:48 | |
| *** dims has joined #openstack-meeting | 06:51 | |
| *** tetsuro has quit IRC | 06:53 | |
| *** e0ne has joined #openstack-meeting | 07:00 | |
| *** rcernin has quit IRC | 07:01 | |
| *** pcaruana has joined #openstack-meeting | 07:01 | |
| *** awaugama has joined #openstack-meeting | 07:01 | |
| *** ykatabam has quit IRC | 07:01 | |
| *** ykatabam has joined #openstack-meeting | 07:04 | |
| *** slaweq has joined #openstack-meeting | 07:06 | |
| *** aloga has quit IRC | 07:15 | |
| *** aloga has joined #openstack-meeting | 07:15 | |
| *** kopecmartin|off is now known as kopecmartin|ruck | 07:17 | |
| *** rockyg has quit IRC | 07:20 | |
| *** mpiwowarczy has quit IRC | 07:32 | |
| *** psachin has joined #openstack-meeting | 07:35 | |
| *** tetsuro has joined #openstack-meeting | 07:38 | |
| *** slaweq has quit IRC | 07:39 | |
| *** ralonsoh has joined #openstack-meeting | 07:40 | |
| *** slaweq has joined #openstack-meeting | 07:48 | |
| *** priteau has joined #openstack-meeting | 07:51 | |
| *** iyamahat has joined #openstack-meeting | 07:51 | |
| *** d0ugal has quit IRC | 07:53 | |
| *** zaneb has quit IRC | 08:12 | |
| *** zaneb has joined #openstack-meeting | 08:12 | |
| *** Luzi has quit IRC | 08:28 | |
| *** Luzi has joined #openstack-meeting | 08:40 | |
| *** ttsiouts has joined #openstack-meeting | 08:51 | |
| *** aojea has joined #openstack-meeting | 08:53 | |
| *** ttsiouts has quit IRC | 09:00 | |
| *** ttsiouts has joined #openstack-meeting | 09:02 | |
| *** hyunsikyang has joined #openstack-meeting | 09:12 | |
| *** slaweq has quit IRC | 09:19 | |
| *** yamamoto has quit IRC | 09:19 | |
| *** yamamoto has joined #openstack-meeting | 09:20 | |
| *** slaweq has joined #openstack-meeting | 09:21 | |
| *** yamamoto has quit IRC | 09:24 | |
| *** yamamoto has joined #openstack-meeting | 09:42 | |
| *** electrofelix has joined #openstack-meeting | 09:45 | |
| *** psachin has quit IRC | 09:59 | |
| *** longkb has quit IRC | 10:00 | |
| *** ttsiouts has quit IRC | 10:01 | |
| *** ttsiouts has joined #openstack-meeting | 10:03 | |
| *** yamamoto has quit IRC | 10:40 | |
| *** yamamoto has joined #openstack-meeting | 10:40 | |
| *** d0ugal has joined #openstack-meeting | 11:01 | |
| *** ttsiouts has quit IRC | 11:03 | |
| *** ttsiouts has joined #openstack-meeting | 11:03 | |
| *** jamesmcarthur has joined #openstack-meeting | 11:06 | |
| *** ttsiouts has quit IRC | 11:08 | |
| *** e0ne has quit IRC | 11:10 | |
| *** jamesmcarthur has quit IRC | 11:11 | |
| *** ttsiouts has joined #openstack-meeting | 11:13 | |
| *** tetsuro has quit IRC | 11:18 | |
| *** tbarron has joined #openstack-meeting | 11:24 | |
| *** ttsiouts has quit IRC | 11:25 | |
| *** bobh has joined #openstack-meeting | 11:49 | |
| *** e0ne has joined #openstack-meeting | 11:50 | |
| *** bobh has quit IRC | 11:53 | |
| *** dtrainor has joined #openstack-meeting | 11:54 | |
| *** ssbarnea|bkp has joined #openstack-meeting | 12:04 | |
| *** yamamoto has quit IRC | 12:07 | |
| *** yamamoto has joined #openstack-meeting | 12:08 | |
| *** ttsiouts has joined #openstack-meeting | 12:10 | |
| *** yamamoto has quit IRC | 12:12 | |
| *** mschuppert has joined #openstack-meeting | 12:15 | |
| *** ykatabam has quit IRC | 12:16 | |
| *** beagle is now known as beagles | 12:16 | |
| *** Emine has joined #openstack-meeting | 12:23 | |
| *** haleyb has quit IRC | 12:23 | |
| *** ssbarnea|bkp has quit IRC | 12:26 | |
| *** ssbarnea|bkp2 has joined #openstack-meeting | 12:27 | |
| *** longkb has joined #openstack-meeting | 12:27 | |
| *** _hemna has quit IRC | 12:27 | |
| *** haleyb has joined #openstack-meeting | 12:27 | |
| *** Luzi has quit IRC | 12:29 | |
| *** _pewp_ has quit IRC | 12:30 | |
| *** _pewp_ has joined #openstack-meeting | 12:31 | |
| *** rossella_s has quit IRC | 12:32 | |
| *** raildo has joined #openstack-meeting | 12:32 | |
| *** markvoelker has joined #openstack-meeting | 12:32 | |
| *** rossella_s has joined #openstack-meeting | 12:34 | |
| *** raildo_ has joined #openstack-meeting | 12:34 | |
| *** Bhujay has joined #openstack-meeting | 12:36 | |
| *** raildo has quit IRC | 12:37 | |
| *** jamesmcarthur has joined #openstack-meeting | 12:40 | |
| *** Luzi has joined #openstack-meeting | 12:43 | |
| *** mriedem has joined #openstack-meeting | 12:46 | |
| *** yamamoto has joined #openstack-meeting | 12:51 | |
| *** longkb has quit IRC | 12:53 | |
| *** tssurya has joined #openstack-meeting | 12:53 | |
| *** longkb has joined #openstack-meeting | 12:56 | |
| *** longkb has quit IRC | 12:56 | |
| *** pcaruana has quit IRC | 12:57 | |
| *** d0ugal has quit IRC | 13:00 | |
| *** jamesmcarthur has quit IRC | 13:02 | |
| *** Luzi has quit IRC | 13:06 | |
| *** bobh has joined #openstack-meeting | 13:08 | |
| *** awaugama has quit IRC | 13:09 | |
| *** rbudden has joined #openstack-meeting | 13:14 | |
| *** d0ugal has joined #openstack-meeting | 13:21 | |
| *** Luzi has joined #openstack-meeting | 13:23 | |
| *** ttsiouts has quit IRC | 13:23 | |
| *** rbudden has quit IRC | 13:23 | |
| *** pcaruana has joined #openstack-meeting | 13:27 | |
| *** dustins has joined #openstack-meeting | 13:27 | |
| *** rbudden has joined #openstack-meeting | 13:29 | |
| *** rossella_s has quit IRC | 13:30 | |
| *** e0ne has quit IRC | 13:32 | |
| *** ttsiouts has joined #openstack-meeting | 13:33 | |
| *** rossella_s has joined #openstack-meeting | 13:34 | |
| *** dangtrinhnt_x has joined #openstack-meeting | 13:34 | |
| *** d0ugal has quit IRC | 13:36 | |
| *** wznoinsk has joined #openstack-meeting | 13:38 | |
| *** tpsilva has joined #openstack-meeting | 13:39 | |
| *** raildo_ is now known as raildo | 13:39 | |
| *** awaugama has joined #openstack-meeting | 13:52 | |
| *** awaugama has quit IRC | 13:52 | |
| *** awaugama has joined #openstack-meeting | 13:53 | |
| *** d0ugal has joined #openstack-meeting | 13:54 | |
| *** hongbin has joined #openstack-meeting | 13:57 | |
| *** dustins has quit IRC | 13:58 | |
| *** ttsiouts has quit IRC | 13:58 | |
| *** mriedem has quit IRC | 14:00 | |
| *** dustins has joined #openstack-meeting | 14:01 | |
| *** e0ne has joined #openstack-meeting | 14:02 | |
| *** amotoki_ is now known as amotoki | 14:04 | |
| *** mriedem has joined #openstack-meeting | 14:05 | |
| *** cloudrancher has quit IRC | 14:07 | |
| *** cloudrancher has joined #openstack-meeting | 14:08 | |
| *** ttsiouts has joined #openstack-meeting | 14:12 | |
| *** Luzi has quit IRC | 14:19 | |
| *** munimeha1 has joined #openstack-meeting | 14:24 | |
| *** liuyulong has joined #openstack-meeting | 14:29 | |
| *** liuyulong has left #openstack-meeting | 14:41 | |
| *** ttsiouts has quit IRC | 14:46 | |
| *** Emine has quit IRC | 14:55 | |
| *** jamesmcarthur has joined #openstack-meeting | 14:55 | |
| *** dangtrinhnt_x has quit IRC | 14:56 | |
| *** ttsiouts has joined #openstack-meeting | 15:00 | |
| *** toabctl has joined #openstack-meeting | 15:03 | |
| *** yamamoto has quit IRC | 15:10 | |
| *** yamamoto has joined #openstack-meeting | 15:10 | |
| *** d0ugal has quit IRC | 15:12 | |
| *** d0ugal has joined #openstack-meeting | 15:12 | |
| *** Leo_m has joined #openstack-meeting | 15:15 | |
| *** e0ne has quit IRC | 15:19 | |
| *** ircuser-1 has joined #openstack-meeting | 15:25 | |
| *** bobh has quit IRC | 15:31 | |
| *** ttsiouts has quit IRC | 15:42 | |
| *** macza has joined #openstack-meeting | 15:43 | |
| *** felipemonteiro has joined #openstack-meeting | 15:44 | |
| *** macza_ has joined #openstack-meeting | 15:47 | |
| *** gyee has joined #openstack-meeting | 15:49 | |
| *** macza has quit IRC | 15:51 | |
| *** Sigyn has quit IRC | 15:56 | |
| *** manjeets_ has joined #openstack-meeting | 15:57 | |
| *** Sigyn has joined #openstack-meeting | 15:57 | |
| *** mlavalle has joined #openstack-meeting | 16:00 | |
| slaweq | #startmeeting neutron_ci | 16:00 |
| openstack | Meeting started Tue Oct 2 16:00:26 2018 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 16:00 | |
| slaweq | hi | 16:00 |
| openstack | The meeting name has been set to 'neutron_ci' | 16:00 |
| mlavalle | o/ | 16:00 |
| bcafarel | o/ | 16:01 |
| slaweq | lets wait few minutes for haleyb, njohnston and others | 16:01 |
| mlavalle | no problem | 16:02 |
| haleyb | hi | 16:02 |
| njohnston | o/ | 16:02 |
| slaweq | ok, I think we can start now | 16:02 |
| slaweq | #topic Actions from previous meetings | 16:03 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 16:03 | |
| slaweq | mlavalle will work on logstash query to find if issues with test_attach_volume_shelved_or_offload_server still happens | 16:03 |
| manjeets_ | hi | 16:03 |
| mlavalle | I did work on that | 16:03 |
| mlavalle | http://logstash.openstack.org/#/dashboard/file/logstash.json?query=build_status:%5C%22FAILURE%5C%22%20AND%20project:%5C%22openstack%2Fneutron%5C%22%20AND%20message:%5C%22test_attach_volume_shelved_or_offload_server%5C%22%20AND%20tags:%5C%22console%5C%22&from=7d | 16:03 |
| mlavalle | I watched it over the past 7 days | 16:04 |
| mlavalle | as of today no hits whatsoever | 16:04 |
| mlavalle | last hit was https://review.openstack.org/#/c/601336 | 16:04 |
| mlavalle | and as you can see, the problem is the patch | 16:04 |
| mlavalle | that test creates an instance and the ssh's to it | 16:05 |
| mlavalle | if we mess something in a patch, sometime we break ssh | 16:05 |
| mlavalle | and that's what we see | 16:05 |
| mlavalle | so it's not a bug | 16:05 |
| slaweq | ok, so I think we can stop work on this one, at least for now | 16:06 |
| mlavalle | yes | 16:06 |
| njohnston | +1 | 16:06 |
| slaweq | I hope it will not back in the future :) | 16:06 |
| mlavalle | I'll preseve the query in case we need it in the future | 16:06 |
| slaweq | ok | 16:06 |
| slaweq | thx mlavalle for checking that | 16:06 |
| slaweq | next one: | 16:06 |
| slaweq | njohnston will debug why fullstack-py36 job is using py35 | 16:06 |
| *** aojea has quit IRC | 16:06 | |
| njohnston | I think bcafarel beat me to the punch a bit on this one | 16:07 |
| *** michael-beaver has joined #openstack-meeting | 16:07 | |
| njohnston | https://review.openstack.org/#/c/604749/ | 16:07 |
| njohnston | but I think more change is needed | 16:08 |
| njohnston | specifically I need to check this, but my impression was that you had to use the ubuntu-bionic nodeset to get a host that had 3.6 installed | 16:08 |
| njohnston | so I | 16:09 |
| slaweq | it looks that there is not py36 in xenial indeed: http://logs.openstack.org/49/604749/1/check/neutron-fullstack-python36/490122b/job-output.txt.gz#_2018-09-24_12_36_02_778606 | 16:09 |
| njohnston | so I will update that and see if we have more success with that nodeset | 16:09 |
| bcafarel | the review is all yours njohnston :) | 16:09 |
| njohnston | I'll get that done during the meeting | 16:10 |
| slaweq | maybe we should just do it as fullstack-python3 and not specify exact version of python used in tests? | 16:10 |
| njohnston | I would be all in favor of that... except that it is known that python 3.7 includes some non-backwards-compatible changes that may break things | 16:11 |
| njohnston | at least that is what I heard from the infra guys | 16:11 |
| njohnston | that is my only reservation | 16:11 |
| slaweq | yep, but for now py37 is even not released, right? | 16:12 |
| *** Bhujay has quit IRC | 16:12 | |
| slaweq | if it will be and we will have to test on distros with this version, we will add new job then | 16:12 |
| njohnston | It was released 2018-06-27 https://www.python.org/downloads/release/python-370/ | 16:12 |
| bcafarel | not in current CI distros but it will be there at some point | 16:12 |
| slaweq | now, many people still uses xenial for e.g. development and locally to run fullstack tests (me for example) and it will not work for me if we specify exactly python36 in tox ini, right? | 16:13 |
| slaweq | ok, it's released but is it in any distro which we support? | 16:14 |
| njohnston | That is a good point. So we should keep the old tox target around so people can test locally. | 16:14 |
| bcafarel | isn't the idea to leave generic python3 in tox.ini and set specific versions in gates/zuul? | 16:15 |
| slaweq | bcafarel: I think that this would be better than specify exact version in tox.ini | 16:16 |
| njohnston | bcafarel: I can alter your change to do that; right now it calls out 3.6 specifically: https://review.openstack.org/#/c/604749/1/tox.ini | 16:16 |
| njohnston | bcafarel: but once we switch to bionic probably we can omit the dot-version | 16:16 |
| slaweq | maybe we should consider to just replace neutron-fullstack-python36 job with neutron-fullstack job simply and run it with python3 | 16:17 |
| slaweq | what You think about that? | 16:17 |
| mlavalle | yes, that makes sense | 16:17 |
| mlavalle | the exeception now python 2.7 | 16:17 |
| mlavalle | we should get used to it | 16:18 |
| njohnston | ok, I will change things so that we have a neutron-fullstack job that calls python3 in tox.ini but that runs on bionic nodeset. Is that an accurate capturing of the consensus? | 16:18 |
| bcafarel | since we will drop the py2 one soon, makes sense indeed | 16:19 |
| slaweq | njohnston: for me yes, if bionic is the release which OpenStack Stein will support of course :) | 16:19 |
| *** toabctl has quit IRC | 16:19 | |
| mlavalle | yes | 16:19 |
| * slaweq don't knows exactly what versions of Ubuntu should be supported in Stein | 16:20 | |
| njohnston | slaweq: IIUC it is, I can get clarification on that from... I don't know who, I guess the infra guys. | 16:20 |
| slaweq | ok, so let's do that and I hope it will work fine | 16:20 |
| slaweq | ok, thx njohnston | 16:21 |
| mlavalle | njohnston: infra guys are a good source | 16:21 |
| njohnston | If infra says that Stein needs to work on both bionic and xenial then we may need two jobs in a few cases | 16:21 |
| njohnston | #action njohnston Reduce to a single fullstack job named 'neutron-fullstack' that calls python3 in tox.ini and runs on bionic nodeset. | 16:21 |
| slaweq | thx njohnston for writing action, I just wanted to do it :) | 16:21 |
| njohnston | #action njohnston Ask the infra team for what ubuntu release Stein will be supported | 16:22 |
| slaweq | ok, let's move on to the next one | 16:22 |
| slaweq | njohnston will send a patch to remove fullstack py27 job completly | 16:22 |
| njohnston | That is in, but it depends on the change to fix the fullstack 3.6 job | 16:23 |
| slaweq | ok, in fact it will be done together if You replace python version to py3 in this job :) | 16:24 |
| njohnston | So it has been sitting there. Given the previous discussion, I'll probably abandon it since we'll be changing the 2.7 job to 3.x instead | 16:24 |
| njohnston | right-o | 16:24 |
| slaweq | njohnston++ | 16:24 |
| slaweq | ok, lets move to the next topic then | 16:24 |
| slaweq | #topic Grafana | 16:24 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 16:24 | |
| slaweq | Quick info | 16:24 |
| slaweq | I have 2 patches: | 16:24 |
| slaweq | https://review.openstack.org/#/c/607285/ - to update fullstack-python35 to py36, but it will be probably changed/abandoned as we will change those jobs | 16:25 |
| slaweq | and second: https://review.openstack.org/#/c/583870/ to add tempest-slow job to grafana which we are now missing in dashboard | 16:25 |
| *** rfolco has quit IRC | 16:26 | |
| slaweq | also, speaking about grafana I noticed that we are probably missing some other jobs also, e.g. openstack-tox-py36 | 16:26 |
| njohnston | slaweq: I also had a change to update fullstack-python35 to 36, I'll abandon it as well: https://review.openstack.org/#/c/605128/ | 16:26 |
| slaweq | is there any volunteer to check that and add missing jobs to grafana? | 16:26 |
| njohnston | o/ | 16:26 |
| slaweq | njohnston: thx :) | 16:27 |
| slaweq | #action njohnston to check and add missing jobs to grafana dashboard | 16:27 |
| slaweq | ahh, I forgot about link | 16:28 |
| njohnston | FYI, answer from fungi on the previous question: "the infra team isn't really in charge of making it happen, but https://governance.openstack.org/tc/reference/project-testing-interface.html#linux-distributions is interpreted to mean that projects should be switching their master branches to use ubuntu-bionic instead of ubuntu-xenial for jobs soon so they work out any remaining kinks before stein releases" | 16:28 |
| slaweq | http://grafana.openstack.org/d/Hj5IHcSmz/neutron-failure-rate?orgId=1 | 16:28 |
| slaweq | njohnston: thx for checking that, so we should be good to switch fullstack to py36 and bionic | 16:28 |
| fungi | yep, one thing we probably want to consider is whether this should be treated like a de facto cycle goal | 16:28 |
| * slaweq wonders how many new issues we will have on new Ubuntu release :) | 16:29 | |
| fungi | (making sure projects are testing against bionic instead of xenial on master before some point ni this cycle) | 16:29 |
| mlavalle | fungi: so are we switching in this cycle? | 16:29 |
| mlavalle | or just preparing for the switch | 16:29 |
| mlavalle | ? | 16:29 |
| fungi | mlavalle: our governance documents state we test on the latest ubuntu lts release | 16:30 |
| mlavalle | ack | 16:30 |
| fungi | so if we're not, we either need to fix that or update governance | 16:30 |
| fungi | previously we've considered it reasonable to stick with whatever the latest ubuntu lts version was at the start of the cycle, which for rocky was ubuntu xenial | 16:31 |
| fungi | ubuntu bionic was released partway into the rocky cycle, so we stuck with xenial through rocky but should switch to bionic in stein | 16:32 |
| slaweq | so we should switch all our jobs to bionic in this cycle, right? | 16:32 |
| mlavalle | thanks for the clarification | 16:32 |
| fungi | ideally yes | 16:32 |
| slaweq | ok, thx fungi | 16:32 |
| bcafarel | that would be in line with community goal py3.6 tests too (they need a distro with it) | 16:32 |
| fungi | during the ubuntu trusty to xenial transition trove ended up with a lot of their jobs still on trusty after other projects released using xenial and that made integration testing really challenging for them. ultimately they needed to update their stable branch testing to xenial which was nontrivial | 16:33 |
| slaweq | so I can prepare some etherpad with list of jobs which we should switch and we will track progress there, what You think? | 16:34 |
| mlavalle | that's a good idea | 16:35 |
| mlavalle | we can review it weekly in this meeting | 16:35 |
| slaweq | yes :) | 16:35 |
| njohnston | fungi: I was going to write an email to the ML with the neutron steadium as the intended audience, but as a member of the TC perhaps this is something that should come from that level, at least to remind everyone to start making plans for an orderly transition | 16:36 |
| slaweq | #action slaweq to prepare etherpad with list of jobs to move to Bionic | 16:36 |
| njohnston | s/as a member of the TC/since you are a member of the TC/ | 16:36 |
| slaweq | ok, moving on? | 16:38 |
| njohnston | yes go ahead | 16:38 |
| slaweq | so getting back to grafana, we have some issues with I want to talk about but generally it's not very bad | 16:39 |
| slaweq | what is worth to mention is fact that tempest jobs and scenario jobs are doing quite good last week | 16:39 |
| mlavalle | that's great! | 16:40 |
| slaweq | so let's talk with some issues which we have there | 16:40 |
| slaweq | #topic fullstack/functional | 16:40 |
| *** openstack changes topic to "fullstack/functional (Meeting topic: neutron_ci)" | 16:40 | |
| * mlavalle flinches for the but.... | 16:40 | |
| slaweq | LOL | 16:41 |
| slaweq | There is many errors like reported in https://bugs.launchpad.net/neutron/+bug/1795482 recently in fullstack | 16:41 |
| openstack | Launchpad bug 1795482 in neutron "Deleting network namespaces sometimes fails in check/gate queue with ENOENT" [High,In progress] - Assigned to Brian Haley (brian-haley) | 16:41 |
| slaweq | and since new pyroute2 version was released (0.5.3) there is issue like in https://bugs.launchpad.net/neutron/+bug/1795548 | 16:42 |
| openstack | Launchpad bug 1795548 in neutron "neutron.tests.functional.agent.l3.test_legacy_router.L3AgentTestCase.test_legacy_router_lifecycle* fail" [High,In progress] - Assigned to Brian Haley (brian-haley) | 16:42 |
| slaweq | I mention both of them together as haleyb already proposed one patch for both: https://review.openstack.org/#/c/607009/ | 16:42 |
| slaweq | so we should be good after this will be merged | 16:42 |
| slaweq | anything else related to fullstack/functional jobs? | 16:43 |
| * mlavalle will review today | 16:44 | |
| slaweq | thx mlavalle | 16:44 |
| slaweq | if there is nothing else, lets go to the next one | 16:45 |
| slaweq | #topic Tempest/Scenario | 16:45 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 16:45 | |
| slaweq | Patch https://review.openstack.org/#/c/578796/ waits for review - it's last job which is now done using legacy zuul templates, | 16:45 |
| slaweq | please review it :) | 16:45 |
| *** _hemna has joined #openstack-meeting | 16:46 | |
| * mlavalle added it to his pile | 16:47 | |
| slaweq | thx mlavalle | 16:47 |
| slaweq | and one more thing, recently I saw few times that one of trunk tests is failing still | 16:47 |
| slaweq | for example: One of trunk tests is failing from time to time, example | 16:47 |
| slaweq | http://logs.openstack.org/85/606385/4/check/neutron-tempest-plugin-dvr-multinode-scenario/e7a983b/logs/testr_results.html.gz | 16:47 |
| slaweq | so my fix for long bash command helped but there is something else also still | 16:48 |
| njohnston | slaweq: Since no more legacy jobs, does that mean that playbooks/legacy will go away? | 16:48 |
| *** manjeets_ has quit IRC | 16:48 | |
| slaweq | njohnston: maybe I forgot to remove it :/ | 16:49 |
| njohnston | it'll be a satisfying follow-on change | 16:51 |
| slaweq | njohnston: ok, thx | 16:51 |
| bcafarel | looks good git-checking out the review, legacy/ went away here | 16:51 |
| bcafarel | oh it's in neutron | 16:52 |
| slaweq | speaking about trunk test issue, I will open new bug for it but I don't know if I will have time to investigate it | 16:52 |
| mlavalle | I can take it | 16:52 |
| slaweq | #action slaweq will report bug about failing trunk scenario test | 16:52 |
| slaweq | ok, thx mlavalle | 16:52 |
| mlavalle | please ping once you file the bug | 16:52 |
| slaweq | #action mlavalle will check failing trunk scenario test | 16:53 |
| slaweq | mlavalle: sure | 16:53 |
| slaweq | thx a lot for help | 16:53 |
| slaweq | ok, moving on to the next one | 16:53 |
| slaweq | #topic grenade | 16:53 |
| *** openstack changes topic to "grenade (Meeting topic: neutron_ci)" | 16:53 | |
| slaweq | we still have this issue with grenade-multinode-dvr job | 16:53 |
| slaweq | what I observed yesterday was that this instance created by grenade test started pinging after around 240-250 seconds after it was created | 16:54 |
| njohnston | wow, that is pretty long | 16:54 |
| mlavalle | 4 minutes | 16:54 |
| slaweq | today I'm trying to add some additional logs to this job to check iptables/openflow/arp tables on both nodes and in all namespaces but I didn't do it yet properly | 16:55 |
| slaweq | yes, it's around 4 minutes and I don't know why it's like that | 16:55 |
| slaweq | I will continue work on this one | 16:56 |
| slaweq | ok, so that's all from me | 16:57 |
| mlavalle | I have one comment | 16:57 |
| mlavalle | boden is trying to add a vmware-nsx job to checking queue. But for non statium projects we said 3rd party CY, right? | 16:58 |
| mlavalle | CI^^^^ | 16:58 |
| munimeha1 | hi | 16:58 |
| slaweq | I think it would be good to add it as 3rd party job | 16:58 |
| slaweq | munimeha1: hi | 16:58 |
| munimeha1 | for tap as service inclusion to Neutron | 16:58 |
| munimeha1 | require CI for Grenade coverage | 16:58 |
| munimeha1 | what need to be done | 16:58 |
| njohnston | We did, and I think the assumption was that we wouldn't have hardware to properly check 3rd party (i.e. no NSX switches). | 16:59 |
| mlavalle | correct | 16:59 |
| slaweq | munimeha1: I think we are out of time now, can You ask about it on neutron-channel? | 17:00 |
| munimeha1 | sure | 17:00 |
| slaweq | #endmeeting | 17:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 17:00 | |
| openstack | Meeting ended Tue Oct 2 17:00:26 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 17:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-10-02-16.00.html | 17:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-10-02-16.00.txt | 17:00 |
| mlavalle | o/ | 17:00 |
| njohnston | All we could reasonably do in Zuul is cross-check the more basic things like unit tests, which would probably not be too horrible | 17:00 |
| slaweq | thx for attending | 17:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2018/neutron_ci.2018-10-02-16.00.log.html | 17:00 |
| slaweq | o/ | 17:00 |
| njohnston | o/ | 17:00 |
| *** iyamahat has quit IRC | 17:01 | |
| bcafarel | \o | 17:01 |
| *** mlavalle has left #openstack-meeting | 17:02 | |
| *** rf0lc0 has joined #openstack-meeting | 17:12 | |
| *** rossella_s has quit IRC | 17:12 | |
| *** rossella_s has joined #openstack-meeting | 17:12 | |
| *** jamesmcarthur has quit IRC | 17:13 | |
| *** jamesmcarthur has joined #openstack-meeting | 17:18 | |
| *** cloudrancher has quit IRC | 17:24 | |
| *** cloudrancher has joined #openstack-meeting | 17:25 | |
| *** iyamahat has joined #openstack-meeting | 17:26 | |
| *** jamesmcarthur has quit IRC | 17:27 | |
| *** rossella_s has quit IRC | 17:30 | |
| *** rossella_s has joined #openstack-meeting | 17:34 | |
| *** Emine has joined #openstack-meeting | 17:43 | |
| *** electrofelix has quit IRC | 17:45 | |
| *** munimeha1 has quit IRC | 17:47 | |
| *** priteau has quit IRC | 17:47 | |
| *** diablo_rojo has joined #openstack-meeting | 17:48 | |
| *** tssurya has quit IRC | 17:52 | |
| *** pcaruana has quit IRC | 18:02 | |
| *** zaneb has quit IRC | 18:13 | |
| *** zaneb has joined #openstack-meeting | 18:13 | |
| *** Emine has quit IRC | 18:15 | |
| *** gary_perkins has quit IRC | 18:15 | |
| *** bobh has joined #openstack-meeting | 18:18 | |
| *** imacdonn has quit IRC | 18:21 | |
| *** imacdonn has joined #openstack-meeting | 18:21 | |
| *** zaneb has quit IRC | 18:27 | |
| *** zaneb has joined #openstack-meeting | 18:28 | |
| *** AJaeger has joined #openstack-meeting | 18:53 | |
| *** ralonsoh has quit IRC | 18:56 | |
| fungi | anybody want to have an infra meeting? | 19:02 |
| fungi | i guess i can chair, but if i'm the only one around there may be little point to it | 19:02 |
| AJaeger | I'm here but it feels kind of lonely, fine with skipping | 19:02 |
| *** Shrews has joined #openstack-meeting | 19:02 | |
| *** Swami has joined #openstack-meeting | 19:03 | |
| AJaeger | fungi, it's just us, let's cancel... | 19:05 |
| Shrews | ++ | 19:05 |
| fungi | no infra team meeting this week, but if you happen to be in austin, tx right now go check out the zuul booth at ansiblefest! | 19:06 |
| AJaeger | ;) | 19:06 |
| fungi | i am not in austin, but i've seen photos. it looks awesome | 19:07 |
| *** AJaeger has left #openstack-meeting | 19:11 | |
| *** awaugama has quit IRC | 19:31 | |
| *** e0ne has joined #openstack-meeting | 19:32 | |
| *** Shrews has left #openstack-meeting | 19:33 | |
| *** diablo_rojo has quit IRC | 20:00 | |
| *** dustins has quit IRC | 20:16 | |
| *** ykatabam has joined #openstack-meeting | 20:31 | |
| *** ykatabam has quit IRC | 20:32 | |
| *** ykatabam has joined #openstack-meeting | 20:32 | |
| *** cloudrancher has quit IRC | 20:41 | |
| *** oneswig has joined #openstack-meeting | 20:49 | |
| *** trandles has joined #openstack-meeting | 20:55 | |
| *** ttsiouts has joined #openstack-meeting | 20:55 | |
| *** raildo has quit IRC | 20:58 | |
| *** bobh has quit IRC | 20:58 | |
| *** cloudrancher has joined #openstack-meeting | 20:59 | |
| *** janders has joined #openstack-meeting | 20:59 | |
| *** bobh has joined #openstack-meeting | 21:00 | |
| oneswig | #startmeeting scientific-sig | 21:00 |
| openstack | Meeting started Tue Oct 2 21:00:26 2018 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 21:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 21:00 |
| oneswig | I made it. Greetings all. | 21:00 |
| janders | g'day oneswig :) | 21:00 |
| oneswig | Hey janders! | 21:00 |
| janders | do you have any experience with NVME-oF in the OpenStack context? | 21:01 |
| trandles | hello all | 21:01 |
| janders | (cinder/nova/swift/... backend) | 21:01 |
| oneswig | ooh, straight down to it :-) | 21:01 |
| oneswig | hey trandles! | 21:01 |
| janders | yes! :) | 21:01 |
| oneswig | Not direct experience of NVMEoF | 21:01 |
| *** e0ne has quit IRC | 21:02 | |
| oneswig | keep looking for a good opportunity. | 21:02 |
| oneswig | janders: what's come up? | 21:02 |
| oneswig | Is this in bare metal or virt? | 21:02 |
| janders | I was chatting to Moshe and the Intel guys about it in Vancouver and they said that 1) the cinder driver is ready and will be supported in Pike but 2) they haven't really gone beyond a scale of a single NVMe device with their testing (eg no NVMe-oF arrays etc) | 21:03 |
| janders | now I was chatting to my storage boss colleague about specing out a backend for the cyber system and he pulled out some array type options that do do NVMe-oF | 21:03 |
| oneswig | I recently saw PR from Excelero - a German cloud (teuto.net?) have built a substantial data platform upon it. | 21:04 |
| janders | found it very interesting - I didn't know anyone offers NVMe-of appliances just yet | 21:04 |
| janders | he quoted E8 | 21:04 |
| oneswig | Ah yes, I've heard of them too. | 21:04 |
| *** bobh has quit IRC | 21:04 | |
| oneswig | Not sure on the unique selling points to distinguish them (I am sure there are some) | 21:05 |
| *** Qiming has quit IRC | 21:05 | |
| janders | you asked what is this for - our cyber system (which will do both baremetal and VM) | 21:05 |
| oneswig | new project? | 21:06 |
| janders | no 1) use case would be persistent volumes, but the more OpenStack services can use the backend the better | 21:06 |
| janders | long story. Short version is what I was talking about in Vancouver has been split up in few parts and this is in the context of one of them | 21:06 |
| oneswig | I've previously used iSER in a similar context. It worked really well for large (>64K) transfers, was pretty lame below that. This was serving Cinder volumes via RDMA to VMs. | 21:07 |
| *** jmlowe has joined #openstack-meeting | 21:07 | |
| janders | was that IB or RoCE? | 21:07 |
| jmlowe | Hi everybody | 21:07 |
| oneswig | RoCE in this case. | 21:07 |
| oneswig | Hey Mike - good to see you | 21:07 |
| janders | ok! that is encouraging - and important given that quite a few NVME-oF players do go RoCE and not IB | 21:07 |
| *** martial has joined #openstack-meeting | 21:08 | |
| janders | and on a related but slightly different front - if you don't mind me asking | 21:08 |
| oneswig | Are you thinking of buying it in or building yourself? | 21:08 |
| janders | for your ironic ethernet+IB system, what ethernet NICs do you use? | 21:08 |
| janders | are those Mellanox or non-Mellanox? | 21:08 |
| oneswig | Mellanox ConnectX4 VPI works well - dual 100G, 1 Ethernet, 1 IB | 21:09 |
| janders | ok! | 21:09 |
| oneswig | Although in this specific case the Ethernet was down-clocked to 25G | 21:09 |
| janders | did you need to do anything to make sure that the Ethernet port doesn't steal bandwidth away from IB? | 21:09 |
| janders | reading my mind :) | 21:09 |
| oneswig | No, nothing major | 21:09 |
| janders | was the 25G cap due to PCIe3x16 bandwidth limitations? | 21:10 |
| oneswig | If you want to use Secure Host for the multi-tenant IB, I believe you're stuck with ConnectX3-pro - unless you know otherwise? | 21:10 |
| *** ttsiouts has quit IRC | 21:10 | |
| janders | all my POC nodes are either 2xFDR or 4xFDR so haven't had the PCIe3x16 problem just yet | 21:10 |
| oneswig | janders: no, switch budget, and I think it was considered more representative of the use case requirements | 21:11 |
| *** ttsiouts has joined #openstack-meeting | 21:11 | |
| janders | you are right regarding Secure Host - I hope to work this out with Erez and his team | 21:11 |
| janders | do you know if mlnx have any good reason NOT to implement SecureHost for CX4 and above? | 21:11 |
| janders | or is it that no one kept asking persistently enough? :) | 21:12 |
| oneswig | I don't know. | 21:12 |
| janders | I will have a go at this soon :) hopefully I will get something out and it will be useful to you guys too | 21:12 |
| oneswig | I think your theory's good. | 21:12 |
| *** cloudrancher has quit IRC | 21:12 | |
| janders | regarding the appliance vs roll your own for NVME-oF - we'll consider both approaches | 21:12 |
| janders | the idea of hyperconverged storage in a bare-metal cloud context is quite interesting actually, but I think this is a potentially endless topic | 21:13 |
| janders | first we discouraged it as not feasible but it came back and it might actually work | 21:13 |
| oneswig | Our thinking has been more towards disaggregated storage recently. | 21:14 |
| oneswig | Possibly a single block device hyperconverged - to keep the resource footprint low on the compute nodes. | 21:15 |
| oneswig | Might be interesting if also nailed down with cgroups | 21:15 |
| janders | we're not there yet - but in a fully containerised context where the whole deployment is baremetal openstack running k8s, it could be just as simple as it used to be in the hypervisor centric reality | 21:16 |
| *** Qiming has joined #openstack-meeting | 21:16 | |
| *** Emine has joined #openstack-meeting | 21:16 | |
| oneswig | We did a little testing on resource footprint with hyperconverged (non-containerised) on bare metal. I think the containers could work. | 21:16 |
| oneswig | On the extreme bandwidth topic, one of our team has been getting really great numbers from BeeGFS at scale recently. Servers with NVME and dual OPA exceeding 21GBytes/s for reads. Yeee-ha | 21:18 |
| janders | wow! | 21:18 |
| oneswig | And there are lots of them in this setup too... | 21:18 |
| janders | our BeeGFS is being built (after some shipping delays). It's 24x NVMe and 2xEDR per box so hoping for some good numbers, too | 21:19 |
| oneswig | There was talk of offering this resource in a NVMEoF mode, but it's secondary to scratch BeeGFS filesystems | 21:19 |
| janders | again our thinking is quite closely aligned :) | 21:19 |
| oneswig | Sounds pretty similar, in which case it should fly | 21:19 |
| janders | our guys are thinking 1) build up the BeeGFS on half of the nodes and see how hard the users will push it | 21:20 |
| janders | if it's not being pushed too hard, we can use some of the NVMes for NVMe-oF, BeeGFS-OND etc | 21:20 |
| jmlowe | Is BeeGFS taking root in Europe? I'm not aware of any deployments in the US | 21:20 |
| *** awaugama has joined #openstack-meeting | 21:21 | |
| oneswig | jmlowe: It seems to be gaining ground over here. Some people got shaken with recent events in Lustre I guess | 21:21 |
| janders | good BeeGFS-OpenStack integration would be an amazing thing to have | 21:21 |
| oneswig | The performance we've seen from it is as good or better than Lustre, on the same hardware. | 21:22 |
| oneswig | janders: Manila! Lets do this! | 21:22 |
| janders | I don't really have much experience with non-IB BeeGFS deployments, but I do know it's quite flexible so it could be quite useful outside HPC too | 21:22 |
| janders | +1! :) | 21:22 |
| *** diablo_rojo has joined #openstack-meeting | 21:22 | |
| janders | the tricky bit is no kerberos support (or prospects for it) last time I checked | 21:22 |
| jmlowe | I'm really happy with nfs-ganesha/cephfs/manila so far, going to try to start rolling it out | 21:23 |
| janders | this can be worked around, but it does make life much harder | 21:23 |
| jmlowe | so all you'd really need to get bootstrapped is a BeeGFS FASL for ganesha | 21:23 |
| oneswig | jmlowe: I'd be interested to hear how that flies when it's under heavy load, really interested. | 21:24 |
| jmlowe | benchmarking from a single node it's faster than native cephfs for smaller files | 21:24 |
| oneswig | This week we've been experimenting with a kubernetes CSI driver for BeeGFS but it's really early days for that. | 21:24 |
| oneswig | jmlowe: an extra layer of caching at play there? Any theories? | 21:25 |
| jmlowe | that was my go to theory | 21:25 |
| oneswig | That's pretty interesting. Does Ganesha run in the hypervisor or separate provisioned nodes? | 21:27 |
| jmlowe | separate | 21:27 |
| *** ttsiouts has quit IRC | 21:27 | |
| *** ttsiouts has joined #openstack-meeting | 21:28 | |
| oneswig | so what's new with Jetstream jmlowe? | 21:28 |
| oneswig | I've had a frustrating day debugging bizarre stubbornness with getting Heat to do more things | 21:31 |
| *** bobh has joined #openstack-meeting | 21:31 | |
| janders | what specifically? | 21:31 |
| janders | heat is always "fun" (even though tight integration with all the other APIs is pretty cool) | 21:31 |
| trandles | +1 frustrating day...but for non-fun non-technical reasons | 21:31 |
| oneswig | Creating resource groups of nodes with parameterised numbers of network interfaces. | 21:32 |
| oneswig | trandles: feel your pain. | 21:32 |
| janders | ouch | 21:32 |
| oneswig | Did you want to mention those open positions? | 21:32 |
| janders | if I remember correctly, that was non-trivial even in ansible.. | 21:32 |
| oneswig | (somewhat select audience here, mostly of non-US nationals...) | 21:33 |
| trandles | ah... | 21:33 |
| trandles | https://jobszp1.lanl.gov/OA_HTML/OA.jsp?OAFunc=IRC_VIS_VAC_DISPLAY&OAMC=R&p_svid=68995&p_spid=3171345&p_lang_code=US | 21:33 |
| trandles | I need to get it up on the openstack jobs board | 21:33 |
| jmlowe | We are getting some new IU only hardware it looks like, 20 v100's and 16 of dell's latest blades | 21:33 |
| oneswig | janders: they appear to share an issue with handling parameters that are yaml-structured compound data objects | 21:33 |
| *** felipemonteiro has quit IRC | 21:33 | |
| oneswig | trandles: the jobs board seems to work | 21:34 |
| jmlowe | I need to figure out a good way to partition so that NSF users only get the hardware the NSF purchased and IU users get only the hardware IU purchased | 21:34 |
| trandles | we've done zero advertising and are pulling in resumes | 21:34 |
| trandles | so fingers crossed we get the right candidate | 21:34 |
| oneswig | jmlowe: there are ways and means for that. | 21:35 |
| jmlowe | I was thinking via private flavors and host aggregates | 21:35 |
| trandles | jmlowe: I hate that hard partitioning problem. We almost always try to argue "you'll get the equivalent amount of time for the dollars you provided" but then end up in arguments about normalization | 21:35 |
| oneswig | I forget the current best practice, but linking project flavors and custom resources seems to be in fashion | 21:35 |
| jmlowe | It's a generation newer cpu, so they really are different, skylake vs haswell I think | 21:36 |
| janders | jmlowe: do you need to worry about idle resources in this scenario, or is the demand greater than the supply across the board? | 21:37 |
| oneswig | jmlowe: what will you do for network fabric between those V100s? | 21:38 |
| jmlowe | not terribly concerned about idle resources | 21:38 |
| jmlowe | 2x10GigE | 21:38 |
| janders | briefly jumping back to the SDN discussion - do you guys know if it's possible to have a single port virtual function on a dual port CX4/5/6? | 21:39 |
| oneswig | Had a question on this - does anyone have experience on mixing SR-IOV with bonded NICs? Is there a way to make that work? | 21:39 |
| jmlowe | looking at nvidia grid and slicing n dicing them, let's say to run a workshop on the openmp like automagical gp-gpu compiler thingy whose name escapes me right now | 21:39 |
| oneswig | janders: snap :-) | 21:40 |
| *** Leo_m has quit IRC | 21:40 | |
| jmlowe | I would think you would need to do the bonding inside the vm in order for SR-IOV to work with bonding | 21:40 |
| janders | oneswig: I think Mellanox do have some clever trick to do in-hardware LACP in CX5 / Ethernet. But that's me repeating their words, I haven't tested that | 21:40 |
| trandles | SR-IOV and bonded NICs sounds like evil voodoo | 21:41 |
| oneswig | trandles: it does indeed. | 21:41 |
| jmlowe | if you consider SR-IOV is just multiplexed pci passthrough then it doesn't sound so scary | 21:41 |
| trandles | but across multiple physical devices? | 21:42 |
| oneswig | trandles: there's two scenarios - 2 nics or 1 dual-port nic - perhaps the second might work, the first, I am not confident | 21:43 |
| janders | http://www.hpcadvisorycouncil.com/events/2018/swiss-workshop/pdf/Monday09April/Cohen_NetworkingAdvantage_Mon090418.pdf | 21:43 |
| janders | slide 10 | 21:43 |
| janders | I think this is what Erez was talking about | 21:43 |
| oneswig | No way - I was there :-) | 21:44 |
| trandles | dual-port NICs sounds like something manageable...multiple NICs sounds like driver hell | 21:44 |
| janders | I wasn't but got a good debrief in Vancouver | 21:44 |
| janders | hmm now I wonder if I should ask for 1xHDR and 1xdual port EDR for my new blades :) | 21:45 |
| jmlowe | trandles never had fun in the bad old days altering a nic's mac address to defeat mac address filtering | 21:45 |
| janders | ..or even 2xEDR and 2xHDR all active/passive for maximum availability | 21:46 |
| oneswig | janders: might push out the delivery date, beyond an event horizon | 21:46 |
| janders | oneswig: this is a very valid point | 21:46 |
| *** bobh has quit IRC | 21:47 | |
| janders | I do have some kit that can be used in the interim, but perhaps retrofitting HDR later isn't a bad idea | 21:47 |
| *** bobh_ has joined #openstack-meeting | 21:47 | |
| jmlowe | Did live migration with melanox sr-iov ever come to fruition? | 21:47 |
| janders | not that I know of | 21:47 |
| oneswig | DK Panda presented something on this | 21:48 |
| janders | however cold migration is now supposed to work (historically wasn't the case) | 21:48 |
| oneswig | But it's never clear if his work will be generally available (or when) | 21:48 |
| jmlowe | damn, that would have been killer, DL Panda hinted at it during the 2017 HPC Advisory Council | 21:48 |
| jmlowe | well, that's something at least | 21:48 |
| oneswig | He came back with numbers, just no github URL (as far as I remember) | 21:48 |
| oneswig | jmlowe: if you look for his video from Lugano 2018, he covered some results. It's possible he was presenting on using RDMA to migrate the VM, I don't recall exactly. | 21:50 |
| *** janders_ has joined #openstack-meeting | 21:50 | |
| oneswig | trandles: we were talking Charliecloud over here the other day. Any news from the project? | 21:51 |
| jmlowe | That's an entirely different beast, no to say it's not useful for something like postcopy live migration, but it's not what I was promised with my flying car | 21:51 |
| oneswig | jmlowe: transporting your physical person, I wouldn't trust RDMA for that :-) | 21:52 |
| janders_ | cmon... isn't IB loseless ;)? | 21:52 |
| *** janders has quit IRC | 21:52 | |
| trandles | Charliecloud: Reid has been kicking ass on implementing some shared object voodoo to get MPI support automagically inside a containerized application on both OpenMPI installations and Cray MPICH | 21:52 |
| jmlowe | see star trek quote | 21:53 |
| *** Emine has quit IRC | 21:53 | |
| oneswig | trandles: this is for the kind of problems that Shifter tackles by grafting in system MPI libraries? | 21:54 |
| trandles | we have also had quite a bit of success putting some large multi-physics lagrangian applications into containers and running automated workflows of parameter studies | 21:54 |
| trandles | Reid had a few chats with the Shifter guys (Shane and Doug) about the gotchas they found and then implemented his own method of doing something similar | 21:55 |
| oneswig | trandles: do you know of people successfully using MPI + Kubernetes? It keeps coming up here and I'm suspicious it's missing some key PMI glue | 21:55 |
| trandles | MPI + Kubernetes: A Story of Oil and Water | 21:56 |
| janders_ | LOL | 21:56 |
| oneswig | trandles: so nearly vinaigrette | 21:56 |
| trandles | not nearly as delicious | 21:56 |
| janders_ | no.. | 21:56 |
| jmlowe | just needs some egg | 21:56 |
| trandles | I know more than I ever thought I'd know (or wanted to know) about how MPI starts up jobs | 21:56 |
| jmlowe | mmm, sausage | 21:57 |
| jmlowe | it's dinner time here | 21:57 |
| janders_ | I don't think there's an easy way of plumbing RDMA into k8s managed docker containers at this point in time - please correct me if I am wrong | 21:57 |
| trandles | long story short, ORTE is dead (not a joke...), PRTE is specialized and will be unsupported, PMIx is the future (maybe...) | 21:57 |
| *** ijw has quit IRC | 21:57 | |
| trandles | what that all means is in the near future (once OpenMPI 2.x dies off) there won't be an mpirun to start jobs | 21:58 |
| trandles | what we do is compile slurm with PMIx support and srun wires it all up | 21:59 |
| trandles | k8s would need that PMIx stuff to properly start a job IMO | 22:00 |
| oneswig | It's a missing piece for AI at scale, I suspect it'll happen one way or another and the deep learning crowd will drive it. | 22:00 |
| oneswig | trandles: any evidence of k8s getting extended with that support? | 22:00 |
| trandles | I don't follow k8s enough right now to know | 22:00 |
| oneswig | hmmm. OK the quest continues but that's more doors closed. | 22:01 |
| trandles | for MPI jobs, k8s runtime model feels like an anti-pattern | 22:01 |
| trandles | ie. when a rank dies, MPI dies | 22:01 |
| oneswig | All that careful just-so placement... | 22:01 |
| trandles | yeah | 22:01 |
| oneswig | Alas we are out of time. | 22:01 |
| oneswig | final comments? | 22:01 |
| trandles | for things like DASK, multi-node TensorFlow, other ML/AI frameworks, k8s makes more sense | 22:02 |
| janders_ | thanks guys, great chat! | 22:02 |
| trandles | janders_: +1 | 22:02 |
| trandles | later everyone | 22:02 |
| jmlowe | I'll miss most of you in Dallas | 22:02 |
| oneswig | jmlowe: you're not going? | 22:02 |
| trandles | jmlowe: you're in Dallas or not in Dallas? | 22:02 |
| jmlowe | I'm going, was under the impression most were going to the Berlin Summit instead | 22:02 |
| oneswig | Ich bin eine Berliner | 22:03 |
| trandles | ah, ok, I'll be in Dallas Monday night, all day Tuesday...leaving early Wednesday morning | 22:03 |
| jmlowe | and I need to buy plane tickets RSN | 22:03 |
| *** mriedem has quit IRC | 22:03 | |
| oneswig | OK y'all, until next time | 22:03 |
| oneswig | #endmeeting | 22:03 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:04 | |
| openstack | Meeting ended Tue Oct 2 22:03:59 2018 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:04 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-10-02-21.00.html | 22:04 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-10-02-21.00.txt | 22:04 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2018/scientific_sig.2018-10-02-21.00.log.html | 22:04 |
| *** ttsiouts has quit IRC | 22:04 | |
| *** oneswig has quit IRC | 22:05 | |
| *** trandles has quit IRC | 22:06 | |
| *** jesusaur has quit IRC | 22:06 | |
| *** bobh_ has quit IRC | 22:07 | |
| *** jesusaur has joined #openstack-meeting | 22:11 | |
| martial | Was a cool meeting ;) | 22:12 |
| *** michael-beaver has quit IRC | 22:33 | |
| *** rbudden has quit IRC | 22:54 | |
| *** tpsilva has quit IRC | 22:55 | |
| *** hongbin has quit IRC | 23:02 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:05 | |
| *** _alastor_ has quit IRC | 23:08 | |
| *** jamesmcarthur has quit IRC | 23:10 | |
| *** awaugama has quit IRC | 23:11 | |
| *** rbudden has joined #openstack-meeting | 23:17 | |
| *** njohnston has quit IRC | 23:17 | |
| *** yamamoto has quit IRC | 23:21 | |
| *** yamamoto has joined #openstack-meeting | 23:22 | |
| *** yamamoto has quit IRC | 23:24 | |
| *** yamamoto has joined #openstack-meeting | 23:24 | |
| *** yamamoto has quit IRC | 23:29 | |
| *** yamamoto has joined #openstack-meeting | 23:30 | |
| *** yamamoto has quit IRC | 23:34 | |
| *** rbudden has quit IRC | 23:36 | |
| *** erlon has joined #openstack-meeting | 23:36 | |
| *** macza_ has quit IRC | 23:37 | |
| *** macza has joined #openstack-meeting | 23:37 | |
| *** rbudden has joined #openstack-meeting | 23:41 | |
| *** janders_ has quit IRC | 23:41 | |
| *** macza has quit IRC | 23:42 | |
| *** Swami has quit IRC | 23:42 | |
| *** rbudden has quit IRC | 23:57 | |
| *** rcernin has joined #openstack-meeting | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!