| *** jamesmcarthur has quit IRC | 00:09 | |
| *** mattw4 has quit IRC | 00:12 | |
| *** diablo_rojo has joined #openstack-meeting | 00:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 00:16 | |
| *** gyee has quit IRC | 00:19 | |
| *** diablo_rojo has quit IRC | 00:25 | |
| *** diablo_rojo__ has joined #openstack-meeting | 00:25 | |
| *** mriedem has quit IRC | 00:38 | |
| *** jamesmcarthur has quit IRC | 00:38 | |
| *** markvoelker has joined #openstack-meeting | 00:41 | |
| *** markvoelker has quit IRC | 00:51 | |
| *** diablo_rojo__ has quit IRC | 00:54 | |
| *** igordc has quit IRC | 01:00 | |
| *** yamamoto has joined #openstack-meeting | 01:01 | |
| *** markvoelker has joined #openstack-meeting | 01:17 | |
| *** yamamoto has quit IRC | 01:24 | |
| *** yamamoto has joined #openstack-meeting | 01:25 | |
| *** mhen has quit IRC | 01:26 | |
| *** altlogbot_2 has quit IRC | 01:37 | |
| *** altlogbot_3 has joined #openstack-meeting | 01:38 | |
| *** markvoelker has quit IRC | 01:38 | |
| *** tetsuro has joined #openstack-meeting | 01:39 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:44 | |
| *** jamesmcarthur has quit IRC | 01:49 | |
| *** nitinuikey has joined #openstack-meeting | 01:55 | |
| *** yamamoto has quit IRC | 02:02 | |
| *** apetrich has quit IRC | 02:08 | |
| *** tetsuro has quit IRC | 02:12 | |
| *** ricolin has joined #openstack-meeting | 02:16 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:18 | |
| *** yamamoto has joined #openstack-meeting | 02:28 | |
| *** slaweq has quit IRC | 02:35 | |
| *** markvoelker has joined #openstack-meeting | 02:42 | |
| *** tetsuro has joined #openstack-meeting | 02:44 | |
| *** nitinuikey has quit IRC | 02:48 | |
| *** hongbin has joined #openstack-meeting | 02:49 | |
| *** tetsuro_ has joined #openstack-meeting | 02:51 | |
| *** jamesmcarthur has quit IRC | 02:53 | |
| *** tetsuro has quit IRC | 02:53 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:54 | |
| *** jamesmcarthur has quit IRC | 02:59 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:04 | |
| *** whoami-rajat has joined #openstack-meeting | 03:07 | |
| *** markvoelker has quit IRC | 03:15 | |
| *** jamesmcarthur has quit IRC | 03:17 | |
| *** tetsuro_ has quit IRC | 03:17 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:40 | |
| *** tetsuro has joined #openstack-meeting | 03:55 | |
| *** enriquetaso has quit IRC | 03:56 | |
| *** nitinuikey has joined #openstack-meeting | 04:00 | |
| *** cheng1 has quit IRC | 04:04 | |
| *** hongbin has quit IRC | 04:11 | |
| *** ircuser-1 has joined #openstack-meeting | 04:34 | |
| *** jamesmcarthur has quit IRC | 04:40 | |
| *** Lucas_Gray has joined #openstack-meeting | 04:41 | |
| *** ykatabam has quit IRC | 04:51 | |
| *** jhesketh has joined #openstack-meeting | 04:54 | |
| *** nitinuikey has quit IRC | 04:54 | |
| *** janki has joined #openstack-meeting | 05:04 | |
| *** Luzi has joined #openstack-meeting | 05:05 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:09 | |
| *** jamesmcarthur has quit IRC | 05:15 | |
| *** tetsuro has quit IRC | 05:23 | |
| *** tetsuro has joined #openstack-meeting | 05:24 | |
| *** tetsuro has quit IRC | 05:27 | |
| *** markvoelker has joined #openstack-meeting | 05:28 | |
| *** tetsuro has joined #openstack-meeting | 05:30 | |
| *** markvoelker has quit IRC | 05:33 | |
| *** Lucas_Gray has quit IRC | 05:35 | |
| *** jchhatbar has joined #openstack-meeting | 05:38 | |
| *** jchhatbar has quit IRC | 05:41 | |
| *** janki has quit IRC | 05:41 | |
| *** jchhatbar has joined #openstack-meeting | 05:42 | |
| *** jchhatbar has quit IRC | 05:43 | |
| *** links has joined #openstack-meeting | 05:47 | |
| *** links has quit IRC | 06:00 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:11 | |
| *** yamamoto has quit IRC | 06:12 | |
| *** jamesmcarthur has quit IRC | 06:16 | |
| *** yamamoto has joined #openstack-meeting | 06:18 | |
| *** yamamoto has quit IRC | 06:24 | |
| *** yamamoto has joined #openstack-meeting | 06:30 | |
| *** vishalmanchanda has joined #openstack-meeting | 06:36 | |
| *** markvoelker has joined #openstack-meeting | 06:36 | |
| *** belmoreira has joined #openstack-meeting | 06:36 | |
| *** belmoreira has quit IRC | 06:37 | |
| *** belmoreira has joined #openstack-meeting | 06:37 | |
| *** yamamoto has quit IRC | 06:42 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:46 | |
| *** apetrich has joined #openstack-meeting | 06:47 | |
| *** yamamoto has joined #openstack-meeting | 06:47 | |
| *** jamesmcarthur has quit IRC | 06:51 | |
| *** kopecmartin|off is now known as kopecmartin | 06:58 | |
| *** jamesmcarthur has joined #openstack-meeting | 07:00 | |
| *** tssurya has joined #openstack-meeting | 07:04 | |
| *** jamesmcarthur has quit IRC | 07:05 | |
| *** hyunsik_m has joined #openstack-meeting | 07:06 | |
| *** slaweq has joined #openstack-meeting | 07:06 | |
| *** hyunsikyang__ has quit IRC | 07:08 | |
| *** hyunsikyang has joined #openstack-meeting | 07:08 | |
| *** markvoelker has quit IRC | 07:09 | |
| *** jbadiapa has joined #openstack-meeting | 07:18 | |
| *** dkushwaha has joined #openstack-meeting | 07:21 | |
| *** jbadiapa has quit IRC | 07:27 | |
| *** tesseract has joined #openstack-meeting | 07:30 | |
| *** keiko-k has joined #openstack-meeting | 07:30 | |
| *** rubasov has joined #openstack-meeting | 07:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 07:39 | |
| *** ralonsoh has joined #openstack-meeting | 07:43 | |
| *** jamesmcarthur has quit IRC | 07:43 | |
| *** hyunsik_m has quit IRC | 07:53 | |
| *** hyunsik_m has joined #openstack-meeting | 07:54 | |
| *** hyunsik_m has quit IRC | 07:54 | |
| *** iyamahat has quit IRC | 07:56 | |
| *** hyunsik_m has joined #openstack-meeting | 07:58 | |
| *** joxyuki has joined #openstack-meeting | 07:58 | |
| *** nitinuikey has joined #openstack-meeting | 08:01 | |
| dkushwaha | #startmeeting tacker | 08:02 |
|---|---|---|
| openstack | Meeting started Tue Aug 6 08:02:13 2019 UTC and is due to finish in 60 minutes. The chair is dkushwaha. Information about MeetBot at http://wiki.debian.org/MeetBot. | 08:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 08:02 |
| *** openstack changes topic to " (Meeting topic: tacker)" | 08:02 | |
| openstack | The meeting name has been set to 'tacker' | 08:02 |
| dkushwaha | #topic Roll Call | 08:02 |
| *** openstack changes topic to "Roll Call (Meeting topic: tacker)" | 08:02 | |
| dkushwaha | who is here for Tacker weekly meeting ? | 08:02 |
| nitinuikey | Hi | 08:02 |
| keiko-k | o/ | 08:02 |
| *** takahashi-tsc has joined #openstack-meeting | 08:02 | |
| hyunsikyang | Hi | 08:02 |
| takahashi-tsc | Hi | 08:03 |
| joxyuki | hi | 08:03 |
| dkushwaha | howdy all | 08:03 |
| *** rcernin has quit IRC | 08:04 | |
| dkushwaha | #chair joxyuki | 08:04 |
| openstack | Current chairs: dkushwaha joxyuki | 08:04 |
| dkushwaha | #topic BP | 08:04 |
| *** openstack changes topic to "BP (Meeting topic: tacker)" | 08:04 | |
| *** hjwon has joined #openstack-meeting | 08:05 | |
| *** ociuhandu has joined #openstack-meeting | 08:05 | |
| *** hokeeeeun has joined #openstack-meeting | 08:05 | |
| dkushwaha | enable updating VNF parameters | 08:05 |
| dkushwaha | #link https://review.opendev.org/#/c/672199/ | 08:05 |
| *** jaewook_oh_ has joined #openstack-meeting | 08:05 | |
| *** ociuhandu has quit IRC | 08:05 | |
| dkushwaha | joxyuki, I don't have no comment on that | 08:06 |
| joxyuki | dkushwaha, thanks for your review | 08:06 |
| dkushwaha | joxyuki, could you please update it, so we will merge it | 08:06 |
| joxyuki | ok. will do it soon | 08:06 |
| *** peschk_l has joined #openstack-meeting | 08:07 | |
| dkushwaha | joxyuki, I have one thing, like how to update image, I mean abot software update things, which we needs to handle, but do not have comment on it for now | 08:07 |
| *** jamesmcarthur has joined #openstack-meeting | 08:08 | |
| joxyuki | dkushwaha, it depends on heat. As for image, I think target instance is re-created. | 08:09 |
| joxyuki | so user have to be carefully what he is going to do. | 08:10 |
| *** hyunsik_m has quit IRC | 08:10 | |
| *** lpetrut has joined #openstack-meeting | 08:10 | |
| *** keiko-k has quit IRC | 08:11 | |
| dkushwaha | joxyuki, yes, but the cases like: 1: how to handle already attached block storages. 2: what if user want to just update some patch on its Os. | 08:11 |
| *** electrofelix has joined #openstack-meeting | 08:11 | |
| *** keiko-k has joined #openstack-meeting | 08:12 | |
| *** tpatil has joined #openstack-meeting | 08:12 | |
| *** jamesmcarthur has quit IRC | 08:12 | |
| dkushwaha | joxyuki, so, for now I suggest to support update parameters, and later we can work for such other cases | 08:12 |
| dkushwaha | thoughts.. | 08:13 |
| joxyuki | dkushwaha, what do you mean other cases? | 08:14 |
| *** jamesmcarthur has joined #openstack-meeting | 08:14 | |
| *** tetsuro has quit IRC | 08:14 | |
| joxyuki | Are they case1 and 2 just you mentioned above? | 08:14 |
| dkushwaha | joxyuki, not sure about all cases, but among them 2 as i mentioned in above comment | 08:15 |
| *** hokeen has joined #openstack-meeting | 08:15 | |
| joxyuki | dkushwaha, got it. | 08:15 |
| joxyuki | As for case 2, I think tacker need to issue commands, such as apt/yum/patch, in the instance. | 08:17 |
| *** ociuhandu has joined #openstack-meeting | 08:17 | |
| joxyuki | because heat doesn't support such use case, maybe. | 08:18 |
| *** hokeeeeun has quit IRC | 08:18 | |
| *** shubham_potale has joined #openstack-meeting | 08:18 | |
| dkushwaha | joxyuki, I see. | 08:18 |
| *** jamesmcarthur has quit IRC | 08:20 | |
| dkushwaha | joxyuki, ok, so please update spec, I will be give my +2, and if further no any comment byt others, we willb merge | 08:20 |
| joxyuki | dkushwaha, yes | 08:21 |
| tpatil | dkushwaha: I want to discuss about VNF packages support for VNF onboarding specs | 08:21 |
| dkushwaha | tpatil, sure | 08:22 |
| tpatil | specs: https://review.opendev.org/#/c/582930 | 08:22 |
| tpatil | We are planning to add new RPC API in tacker-conductor for processing vnf packages | 08:22 |
| tpatil | so I would like to ask question whether tacker-conductor service is installed on controller node or on a separate node in the production env. | 08:23 |
| *** JangwonLee has joined #openstack-meeting | 08:23 | |
| dkushwaha | tpatil, its installed on controller | 08:24 |
| tpatil | Generally for HA, tacker.service will be installed on multiple controller nodes ( 2 or 3) | 08:24 |
| tpatil | I have seen one patch where monitoring is moved to tacker.conductor | 08:24 |
| dkushwaha | tpatil, which patch? you mean mistral-monitoring patch? | 08:25 |
| tpatil | that patch is not yet merged, but monitoring same vnF from 2 or 3 controller nodes would be problematic | 08:25 |
| tpatil | yes | 08:25 |
| *** hokeeeeun has joined #openstack-meeting | 08:26 | |
| dkushwaha | tpatil, i needs to re-check that patch, but as I remember, conductor is to communicate with, not for monitoring | 08:27 |
| tpatil | in our specs, we want to process vnf packages in tacker conductor, for that, we need to extract the csar zip in a folder which will be made configurable. | 08:28 |
| tpatil | once the csar zip is extracted, we want to keep the files as is until the vnf package is deleted | 08:28 |
| *** hokeen has quit IRC | 08:29 | |
| tpatil | now if tacker-conductor is running on multiple nodes for HA, we will need to cleanup the extracted data from folder when vnf package will be deleted from tacker.conductor from all nodes | 08:29 |
| *** ociuhandu has quit IRC | 08:30 | |
| dkushwaha | tpatil, just trying to understand, why new API on conductor? | 08:31 |
| tpatil | for that we will need to introduce periodic tasks in conductor for clean up of deleted VNF packages | 08:31 |
| tpatil | we want to add processing of vnf package code in conductor | 08:32 |
| tpatil | as it would be lengthy tasks | 08:32 |
| dkushwaha | make sense | 08:32 |
| tpatil | and also conductor manager, we can introduce the periodic task for cleanup | 08:32 |
| joxyuki | tpatil, why is it priodic? when VNF package delete is called, tacker will delete it. | 08:34 |
| joxyuki | s/priodic/periodic/ | 08:34 |
| tpatil | but if you run multiple tacker.conductor service, the request will be processed by only one service | 08:35 |
| *** e0ne has joined #openstack-meeting | 08:35 | |
| joxyuki | understand | 08:35 |
| tpatil | in that case, some of the extracted csar data from one of the controller node won't be deleted | 08:36 |
| dkushwaha | tpatil, seems, we missed this case in spec. | 08:38 |
| dkushwaha | i needs to re-llok into it | 08:38 |
| nitinuikey | @tpatil so you mean periodic task will clean up vnf data from all the tacker conductor nodes? | 08:38 |
| dkushwaha | re-look | 08:38 |
| tpatil | yes, I will update the specs as now I'm clear that tacker.conductor will be installed on the controller node | 08:38 |
| hyunsikyang | IMO, If you want to change the conductor architecture, it is anoother issue. | 08:39 |
| hyunsikyang | dkushwaha, now is tacker support multiple conductor and service? | 08:39 |
| tpatil | nitinuikey: it will be deleted from one of the controller node when user will delete the vnf package, and from other controller node, if any data is there ,it will be cleaned up in the periodic tasks | 08:39 |
| nitinuikey | tpatil understood | 08:40 |
| dkushwaha | hyunsikyang, some actions cannot access tacker database directly | 08:43 |
| dkushwaha | hyunsikyang, so conductor server was introduced to do database access for those actions | 08:43 |
| dkushwaha | hyunsikyang, but yes, it looks an issue to have multiple conductor | 08:44 |
| *** tpatil has quit IRC | 08:44 | |
| *** jamesmcarthur has joined #openstack-meeting | 08:44 | |
| hyunsikyang | dkushwaha, yes. I think so. thanks | 08:45 |
| shubham_potale | FYI tpatil lost internet connection | 08:46 |
| dkushwaha | tpatil, please update spec, i will check again | 08:46 |
| dkushwaha | oh | 08:46 |
| nitinuikey | dkushwaha we will inform them if he will not able to connect | 08:46 |
| shubham_potale | dkushwaha: tpatil here, sure i will update the specs | 08:47 |
| dkushwaha | hyunsikyang, could you please help to review https://review.opendev.org/#/c/582930 | 08:47 |
| dkushwaha | tpatil, thanks | 08:47 |
| *** panda|pubholiday is now known as panda | 08:49 | |
| dkushwaha | moving next.. | 08:49 |
| dkushwaha | Prometheus plugin support | 08:49 |
| dkushwaha | #link https://review.opendev.org/#/c/540416/ | 08:49 |
| *** iyamahat has joined #openstack-meeting | 08:50 | |
| dkushwaha | jaewook_oh_, any update from your side? | 08:50 |
| jaewook_oh_ | Umm I updated the bp and I checked your comments | 08:50 |
| *** ociuhandu has joined #openstack-meeting | 08:51 | |
| dkushwaha | jaewook_oh_, I just commented some nits. | 08:52 |
| jaewook_oh_ | Yes, and I've updated the bp from patch set 29 to patch set 30. Some new comments from the reviewers would be appreciated. | 08:54 |
| *** sridharg has joined #openstack-meeting | 08:54 | |
| dkushwaha | Folks, As we have to freeze spec soon, so please help to review specs on priority. | 08:54 |
| dkushwaha | jaewook_oh_, ok | 08:54 |
| jaewook_oh_ | And as you said it is not for container-based vnf only, and I've changed the commit title, but that made some error. | 08:55 |
| jaewook_oh_ | That's why I couldn't change it... and I think creating new bp would be nice in this case :( | 08:56 |
| dkushwaha | moving next.. | 08:56 |
| *** tpatil has joined #openstack-meeting | 08:57 | |
| dkushwaha | #topic Open Discussion | 08:57 |
| *** openstack changes topic to "Open Discussion (Meeting topic: tacker)" | 08:57 | |
| dkushwaha | tpatil, as in last meeting discussion about cp-auto-heal | 08:58 |
| *** tetsuro has joined #openstack-meeting | 08:58 | |
| dkushwaha | https://github.com/openstack/tacker/blame/master/tacker/vnfm/policy_actions/vdu_autoheal/vdu_autoheal.py#L51 | 08:58 |
| dkushwaha | tpatil, it does not heal CP values, but its name only. | 08:59 |
| *** Lucas_Gray has joined #openstack-meeting | 08:59 | |
| dkushwaha | so once a vnf(i.e. VDU) heal, it lost there ip, and created new one | 08:59 |
| tpatil | yes, but if Mac address is there, it would assign same ip address | 09:00 |
| dkushwaha | so every time we comes with new ip | 09:00 |
| *** tetsuro has quit IRC | 09:00 | |
| *** tetsuro has joined #openstack-meeting | 09:01 | |
| dkushwaha | oh, time up folks | 09:01 |
| dkushwaha | Closing this meeting | 09:01 |
| takahashi-tsc | FYI, We checked how it works, and IP address is not changed. | 09:02 |
| dkushwaha | thanks all for joining | 09:02 |
| tpatil | I don't recollect every thing at this point, will update later | 09:02 |
| dkushwaha | takahashi-tsc, tpatil we can continue on tacker channel for further discussion | 09:02 |
| dkushwaha | #endmeeting | 09:02 |
| tpatil | sure | 09:03 |
| *** tetsuro has quit IRC | 09:03 | |
| *** hokeen has joined #openstack-meeting | 09:03 | |
| *** tpatil has left #openstack-meeting | 09:05 | |
| *** ociuhandu has quit IRC | 09:05 | |
| *** hokeeeeun has quit IRC | 09:06 | |
| *** hokeen has quit IRC | 09:08 | |
| *** hjwon has quit IRC | 09:17 | |
| *** hyunsikyang__ has joined #openstack-meeting | 09:17 | |
| *** jamesmcarthur has quit IRC | 09:18 | |
| *** hyunsikyang has quit IRC | 09:20 | |
| *** hyunsikyang has joined #openstack-meeting | 09:21 | |
| *** hyunsikyang__ has quit IRC | 09:22 | |
| *** hyunsikyang has quit IRC | 09:23 | |
| *** yamamoto has quit IRC | 09:27 | |
| *** Lucas_Gray has quit IRC | 09:27 | |
| *** yamamoto has joined #openstack-meeting | 09:28 | |
| *** dtrainor_ has quit IRC | 09:28 | |
| *** Lucas_Gray has joined #openstack-meeting | 09:29 | |
| *** dtrainor has joined #openstack-meeting | 09:31 | |
| *** keiko-k has quit IRC | 09:32 | |
| *** yamamoto has quit IRC | 09:33 | |
| *** abishop_ has joined #openstack-meeting | 09:40 | |
| *** abishop has quit IRC | 09:41 | |
| *** ociuhandu has joined #openstack-meeting | 09:47 | |
| *** yamamoto has joined #openstack-meeting | 09:51 | |
| *** jamesmcarthur has joined #openstack-meeting | 09:55 | |
| *** jamesmcarthur has quit IRC | 09:59 | |
| *** yamamoto has quit IRC | 10:00 | |
| *** yamamoto has joined #openstack-meeting | 10:05 | |
| *** yamamoto has quit IRC | 10:06 | |
| *** yaawang has quit IRC | 10:21 | |
| *** dkushwaha has quit IRC | 10:30 | |
| *** takahashi-tsc has quit IRC | 10:31 | |
| *** rfolco|ruck is now known as rfolco|doctor | 10:33 | |
| *** bbowen has quit IRC | 10:38 | |
| *** yaawang has joined #openstack-meeting | 10:38 | |
| *** yaawang has quit IRC | 10:44 | |
| *** yamamoto has joined #openstack-meeting | 10:48 | |
| *** rakhmerov has joined #openstack-meeting | 10:55 | |
| *** jamesmcarthur has joined #openstack-meeting | 10:55 | |
| *** jamesmcarthur has quit IRC | 11:00 | |
| *** nitinuikey has quit IRC | 11:05 | |
| *** links has joined #openstack-meeting | 11:12 | |
| *** ociuhandu has quit IRC | 11:13 | |
| *** ociuhandu has joined #openstack-meeting | 11:16 | |
| *** ociuhandu has quit IRC | 11:21 | |
| *** jamesmcarthur has joined #openstack-meeting | 11:31 | |
| *** jamesmcarthur has quit IRC | 11:35 | |
| *** jamesmcarthur has joined #openstack-meeting | 11:35 | |
| *** ociuhandu has joined #openstack-meeting | 11:35 | |
| *** links has quit IRC | 11:36 | |
| *** bbowen has joined #openstack-meeting | 11:36 | |
| *** carloss has joined #openstack-meeting | 11:37 | |
| *** ociuhandu has quit IRC | 11:40 | |
| *** yamamoto has quit IRC | 11:42 | |
| *** yaawang has joined #openstack-meeting | 11:43 | |
| *** ociuhandu has joined #openstack-meeting | 11:49 | |
| *** panda is now known as panda|lunch | 11:49 | |
| *** brinzhang_ has joined #openstack-meeting | 11:50 | |
| *** brinzhang has quit IRC | 11:50 | |
| *** brinzhang has joined #openstack-meeting | 11:50 | |
| *** dviroel has joined #openstack-meeting | 11:53 | |
| *** brinzhang_ has quit IRC | 11:54 | |
| *** rubasov has quit IRC | 12:00 | |
| *** ociuhandu has quit IRC | 12:01 | |
| *** zbr is now known as zbr|lunch | 12:01 | |
| *** Lucas_Gray has quit IRC | 12:02 | |
| *** markvoelker has joined #openstack-meeting | 12:04 | |
| *** Lucas_Gray has joined #openstack-meeting | 12:04 | |
| *** markvoelker has quit IRC | 12:06 | |
| *** markvoelker has joined #openstack-meeting | 12:06 | |
| *** rubasov has joined #openstack-meeting | 12:07 | |
| *** yamamoto has joined #openstack-meeting | 12:14 | |
| *** links has joined #openstack-meeting | 12:18 | |
| *** jamesmcarthur has quit IRC | 12:20 | |
| *** Lucas_Gray has quit IRC | 12:22 | |
| *** hongbin has joined #openstack-meeting | 12:34 | |
| *** zbr|lunch is now known as zbr | 12:38 | |
| *** yamamoto has quit IRC | 12:43 | |
| *** enriquetaso has joined #openstack-meeting | 12:57 | |
| *** mriedem has joined #openstack-meeting | 13:00 | |
| *** eharney has joined #openstack-meeting | 13:00 | |
| *** enriquetaso has quit IRC | 13:03 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:07 | |
| *** enriquetaso has joined #openstack-meeting | 13:08 | |
| *** ociuhandu has joined #openstack-meeting | 13:08 | |
| *** panda|lunch is now known as panda | 13:12 | |
| *** lseki has joined #openstack-meeting | 13:14 | |
| *** jamesmcarthur has joined #openstack-meeting | 13:17 | |
| *** yamamoto has joined #openstack-meeting | 13:19 | |
| *** hongbin has quit IRC | 13:24 | |
| *** yamamoto has quit IRC | 13:34 | |
| *** ociuhandu has quit IRC | 13:40 | |
| *** belmoreira has quit IRC | 13:41 | |
| *** belmoreira has joined #openstack-meeting | 13:42 | |
| *** Lucas_Gray has quit IRC | 13:43 | |
| *** yamamoto has joined #openstack-meeting | 13:44 | |
| *** yamamoto has quit IRC | 13:44 | |
| *** yamamoto has joined #openstack-meeting | 13:45 | |
| *** yamamoto has quit IRC | 13:46 | |
| *** yamamoto has joined #openstack-meeting | 13:46 | |
| *** yamamoto has quit IRC | 13:46 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:47 | |
| *** yamamoto has joined #openstack-meeting | 13:47 | |
| *** yamamoto has quit IRC | 13:51 | |
| *** Luzi has quit IRC | 13:56 | |
| *** abishop_ is now known as abishop | 13:57 | |
| *** shintaro has joined #openstack-meeting | 14:02 | |
| *** JangwonLee has quit IRC | 14:08 | |
| *** liuyulong has joined #openstack-meeting | 14:10 | |
| *** ociuhandu has joined #openstack-meeting | 14:11 | |
| *** altlogbot_3 has quit IRC | 14:12 | |
| *** altlogbot_2 has joined #openstack-meeting | 14:14 | |
| *** ociuhandu has quit IRC | 14:17 | |
| *** Lucas_Gray has quit IRC | 14:17 | |
| *** rfolco|doctor is now known as rfolco | 14:19 | |
| *** rfolco is now known as rfolco|ruck | 14:19 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:19 | |
| *** brinzhang has quit IRC | 14:24 | |
| *** brinzhang has joined #openstack-meeting | 14:25 | |
| *** brinzhang has quit IRC | 14:26 | |
| *** ociuhandu has joined #openstack-meeting | 14:27 | |
| *** brinzhang has joined #openstack-meeting | 14:29 | |
| *** brinzhang has quit IRC | 14:29 | |
| *** brinzhang has joined #openstack-meeting | 14:30 | |
| *** Lucas_Gray has quit IRC | 14:31 | |
| *** Lucas_Gray has joined #openstack-meeting | 14:32 | |
| *** yamamoto has joined #openstack-meeting | 14:45 | |
| *** belmoreira has quit IRC | 14:49 | |
| *** yamamoto has quit IRC | 14:50 | |
| *** Lucas_Gray has quit IRC | 14:52 | |
| *** links has quit IRC | 14:58 | |
| *** shintaro has quit IRC | 15:00 | |
| *** cheng1 has joined #openstack-meeting | 15:01 | |
| *** diablo_rojo has joined #openstack-meeting | 15:02 | |
| *** belmoreira has joined #openstack-meeting | 15:05 | |
| *** belmoreira has quit IRC | 15:13 | |
| *** Lucas_Gray has joined #openstack-meeting | 15:14 | |
| *** nfakhir has quit IRC | 15:26 | |
| *** gyee has joined #openstack-meeting | 15:29 | |
| *** enriquetaso has quit IRC | 15:35 | |
| *** ociuhandu has quit IRC | 15:36 | |
| *** sridharg has quit IRC | 15:47 | |
| *** igordc has joined #openstack-meeting | 15:48 | |
| *** ociuhandu has joined #openstack-meeting | 15:48 | |
| *** rsimai is now known as rsimai_away | 15:51 | |
| *** Wryhder has joined #openstack-meeting | 16:00 | |
| slaweq | #startmeeting neutron_ci | 16:00 |
| openstack | slaweq: Error: Can't start another meeting, one is in progress. Use #endmeeting first. | 16:00 |
| slaweq | hi | 16:00 |
| *** mlavalle has joined #openstack-meeting | 16:00 | |
| slaweq | #endmeeting | 16:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:00 | |
| openstack | Meeting ended Tue Aug 6 16:00:47 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-08-06-08.02.html | 16:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-08-06-08.02.txt | 16:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tacker/2019/tacker.2019-08-06-08.02.log.html | 16:00 |
| slaweq | #startmeeting neutron_ci | 16:00 |
| openstack | Meeting started Tue Aug 6 16:00:55 2019 UTC and is due to finish in 60 minutes. The chair is slaweq. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: neutron_ci)" | 16:00 | |
| openstack | The meeting name has been set to 'neutron_ci' | 16:01 |
| ralonsoh | hello | 16:01 |
| slaweq | hi | 16:01 |
| mlavalle | LO, I thought I had missed the meeting by 1 hour | 16:01 |
| mlavalle | LOL | 16:01 |
| slaweq | :D | 16:01 |
| *** Lucas_Gray has quit IRC | 16:01 | |
| slaweq | mlavalle: You're just in time | 16:01 |
| *** Wryhder is now known as Lucas_Gray | 16:01 | |
| mlavalle | yes, I realized that right a way, but for a split second I got confused | 16:02 |
| slaweq | Nate is no PTO this week | 16:02 |
| mlavalle | on | 16:02 |
| slaweq | but lets wait 2 more minutes for haleyb and others | 16:02 |
| haleyb | hi | 16:02 |
| slaweq | hi haleyb | 16:03 |
| *** ociuhandu has quit IRC | 16:03 | |
| haleyb | hi, will need to leave about :15 early | 16:03 |
| slaweq | haleyb: ok | 16:03 |
| slaweq | so lets start this meeting | 16:03 |
| slaweq | Grafana dashboard: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:03 |
| slaweq | #topic Actions from previous meetings | 16:04 |
| *** openstack changes topic to "Actions from previous meetings (Meeting topic: neutron_ci)" | 16:04 | |
| slaweq | mlavalle to report bug with router migrations | 16:04 |
| mlavalle | I did report the bug | 16:04 |
| mlavalle | https://bugs.launchpad.net/neutron/+bug/1838449 | 16:04 |
| openstack | Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel) | 16:04 |
| mlavalle | and I spent time debugging it | 16:04 |
| mlavalle | to make a long story short | 16:04 |
| mlavalle | the neutron server never receives notification that the router_interface_distributed is down in the controller ovs agent | 16:06 |
| mlavalle | I can see that L3 agent removed the interface | 16:06 |
| mlavalle | I can also see that the ovs agent removes the port | 16:06 |
| mlavalle | but the notification never gets to the server | 16:06 |
| mlavalle | and therefore the port remains in status UP from the point of the neutron server and the API | 16:07 |
| slaweq | and do You see that notification was sent by ovs-agent? | 16:07 |
| mlavalle | checking that is my next step | 16:07 |
| mlavalle | I kight need to add some LOG statements in a DNM patch | 16:08 |
| slaweq | yes, some additional logs might be useful for that | 16:08 |
| mlavalle | but definitely the problem is somewhere in the notifcation path | 16:09 |
| slaweq | so sounds like real bug, not testonly issue | 16:09 |
| mlavalle | yes it is | 16:09 |
| *** mattw4 has joined #openstack-meeting | 16:09 | |
| slaweq | thx mlavalle for update and for working on this | 16:10 |
| mlavalle | in my dev system the test always succeeds | 16:10 |
| mlavalle | but I am executing it without load | 16:10 |
| mlavalle | so it must be related with the system under load | 16:10 |
| slaweq | yes, probably | 16:10 |
| mlavalle | this type is tough to debug | 16:10 |
| slaweq | as many of our gate failures | 16:10 |
| slaweq | I think we already fixed easy ones ;) | 16:11 |
| slaweq | ok, lets move on to the next action | 16:11 |
| slaweq | ralonsoh to report a bug about failed test neutron.tests.fullstack.test_qos.TestMinBwQoSOvs.test_bw_limit_qos_port_removed | 16:11 |
| ralonsoh | slaweq, I talked to you last week | 16:12 |
| *** igordc has quit IRC | 16:12 | |
| ralonsoh | The logs are gone and I didn't see this error reproduced | 16:12 |
| mlavalle | #action mlavalle will continue debugging router migration bug | 16:12 |
| *** Lucas_Gray has quit IRC | 16:12 | |
| slaweq | ralonsoh: ok, I probably forgot about it | 16:12 |
| ralonsoh | so I didn't open a bug because we don't have a log to report | 16:12 |
| slaweq | now I remember | 16:12 |
| slaweq | ok, sounds good for me | 16:12 |
| ralonsoh | (but I'm still checking the CI) | 16:13 |
| slaweq | mlavalle: thx for adding action for Yourself | 16:13 |
| slaweq | ralonsoh: thx :) | 16:13 |
| slaweq | ok, next one than | 16:13 |
| slaweq | slaweq to check midonet job and report bug(s) related to it | 16:13 |
| slaweq | I checked that there was issue with expired SSL cert for some midonet domain | 16:13 |
| slaweq | yamamoto workarounded that with https://review.opendev.org/#/c/674538/ | 16:14 |
| slaweq | but today I found out that there is still one test failing in this job, so I reported bug https://bugs.launchpad.net/networking-midonet/+bug/1839169 | 16:14 |
| openstack | Launchpad bug 1839169 in networking-midonet "Tempest test tempest.api.compute.admin.test_migrations.MigrationsAdminTest is failing 100% times" [Undecided,New] | 16:14 |
| slaweq | and I just sent patch to skip this test in this midonet job running on neutron repo | 16:15 |
| *** tssurya has quit IRC | 16:15 | |
| slaweq | https://review.opendev.org/674858 | 16:15 |
| slaweq | and the last one from last week was | 16:16 |
| slaweq | slaweq to report and try to fix bug in neutron_tempest_plugin.api.test_port_forwardings.PortForwardingTest | 16:16 |
| slaweq | I reported this bug, started looking into it and than found out that this was releated to the patch on which it was running, so no bug at all :P | 16:17 |
| slaweq | I just lost about 30minutes of my life on that one :D | 16:17 |
| mlavalle | LOL | 16:18 |
| slaweq | mlavalle: yeah, that was my reaction when I noticed that :) | 16:18 |
| slaweq | ok, any questions/comments? | 16:18 |
| *** mattw4 has quit IRC | 16:19 | |
| slaweq | ok, lets move on than | 16:19 |
| slaweq | #topic Stadium projects | 16:20 |
| *** openstack changes topic to "Stadium projects (Meeting topic: neutron_ci)" | 16:20 | |
| slaweq | python 3 migration | 16:20 |
| *** ricolin has quit IRC | 16:20 | |
| slaweq | Stadium projects etherpad: https://etherpad.openstack.org/p/neutron_stadium_python3_status | 16:20 |
| *** mattw4 has joined #openstack-meeting | 16:20 | |
| slaweq | I think there is nothing to update here today | 16:20 |
| slaweq | ok, next one | 16:21 |
| slaweq | tempest-plugins migration | 16:21 |
| slaweq | Etherpad: https://etherpad.openstack.org/p/neutron_stadium_move_to_tempest_plugin_repo | 16:21 |
| slaweq | I know that tidwellr is working on patch for neutron-dynamic-routing | 16:22 |
| slaweq | and still have some problems with some tests there | 16:22 |
| mlavalle | I did spend some time with the vpn one | 16:22 |
| slaweq | any update about vpnaas mlavalle ? | 16:22 |
| mlavalle | didn't make much progress though | 16:22 |
| mlavalle | I'll continue this week | 16:23 |
| slaweq | ok, thx for taking care of it | 16:23 |
| *** enriquetaso has joined #openstack-meeting | 16:23 | |
| slaweq | any questions/comments about stadium projects? | 16:24 |
| *** hongbin has joined #openstack-meeting | 16:24 | |
| slaweq | ok, lets move on then | 16:26 |
| slaweq | next topic | 16:26 |
| slaweq | #topic Grafana | 16:26 |
| *** openstack changes topic to "Grafana (Meeting topic: neutron_ci)" | 16:26 | |
| slaweq | link was given earlier: http://grafana.openstack.org/dashboard/db/neutron-failure-rate | 16:26 |
| slaweq | in overall it looks quite good IMO | 16:27 |
| mlavalle | and summer vacation has hit us | 16:27 |
| mlavalle | it seems there is not a lot of activity | 16:28 |
| slaweq | we had problem with functional/fullstack jobs at the end of last week but it's fixed now by frickler https://review.opendev.org/#/c/674426/ | 16:28 |
| slaweq | mlavalle: right, we have much less patches, especially in gate queue | 16:28 |
| slaweq | that is my impression too | 16:28 |
| *** hongbin has quit IRC | 16:29 | |
| slaweq | any other comments related to grafana in overall? | 16:30 |
| mlavalle | none from me | 16:30 |
| slaweq | ok, so lets move on | 16:30 |
| slaweq | I don't have anything related to functional/fullstack jobs for today | 16:30 |
| slaweq | so lets skip to | 16:31 |
| slaweq | #topic Tempest/Scenario | 16:31 |
| *** openstack changes topic to "Tempest/Scenario (Meeting topic: neutron_ci)" | 16:31 | |
| *** jamesmcarthur has quit IRC | 16:31 | |
| slaweq | here I have only one thing which I want to mention | 16:31 |
| slaweq | recently I realized that we are not running neutron-tempest-plugin tests on any non-dvr openvswitch environment | 16:31 |
| slaweq | so I proposed 2 new jobs: | 16:31 |
| slaweq | neutron-tempest-plugin-scenario-openvswitch - https://review.opendev.org/#/c/670738/ - merged already | 16:31 |
| slaweq | neutron-tempest-plugin-scenario-openvswitch-iptables-hybrid https://review.opendev.org/#/c/674274/ | 16:32 |
| *** markvoelker has quit IRC | 16:32 | |
| slaweq | both are proposed as voting and gating because this is in fact our most common and most supported configuration | 16:32 |
| slaweq | so it should works fine | 16:32 |
| slaweq | Description of those jobs is proposed in https://review.opendev.org/#/c/674272/ | 16:32 |
| *** enriquetaso has quit IRC | 16:32 | |
| slaweq | please review those patches and tell me what You think about them :) | 16:33 |
| slaweq | anything else You want to talk about, related to tempest/scenario jobs? | 16:34 |
| mlavalle | the description is already merged | 16:34 |
| slaweq | no, it's not as it depends-on on other patches | 16:34 |
| slaweq | so it's approved but not merged yet | 16:34 |
| mlavalle | ok | 16:36 |
| mlavalle | approved the other one | 16:36 |
| slaweq | mlavalle: thx a lot | 16:36 |
| slaweq | I will push also patch to update grafana and add those 2 new jobs | 16:37 |
| mlavalle | thanks | 16:37 |
| slaweq | Last one thing which I have for today is new neutron-tempest-plugin release | 16:38 |
| slaweq | I proposed it yesterday: https://review.opendev.org/#/c/674573/ | 16:39 |
| slaweq | the reason why I did it so fast after the last one is that it has some quite important new test cases, like multicast traffic test | 16:39 |
| slaweq | so mlavalle if You can look at it too, that would be nice | 16:39 |
| slaweq | amotoki already gave +1 for it :) | 16:40 |
| mlavalle | ok | 16:40 |
| mlavalle | done | 16:40 |
| slaweq | mlavalle: thx | 16:41 |
| slaweq | ok, that's all from my side for today | 16:41 |
| mlavalle | I don't have anything else | 16:41 |
| slaweq | (faster than ever) | 16:41 |
| mlavalle | it must be the summer vacation spirit | 16:41 |
| slaweq | if You don't have anything else to talk about, we can finish this meeting earlier today :) | 16:41 |
| slaweq | mlavalle: that's right :) | 16:41 |
| mlavalle | at least for us in the Northern hemisphere | 16:41 |
| slaweq | yeah :) | 16:42 |
| mlavalle | The Aussies are in the middle of their winter | 16:42 |
| ralonsoh | (I'm going to the pool right now) | 16:42 |
| slaweq | but do we have anyone from Southern hemispere here? | 16:42 |
| mlavalle | no | 16:42 |
| slaweq | :) | 16:42 |
| slaweq | ralonsoh: have fun | 16:42 |
| mlavalle | I am probably the one farthest south | 16:42 |
| slaweq | ok, thx for attending | 16:43 |
| mlavalle | both now and where I was born, whihch is even further south | 16:43 |
| mlavalle | o/ | 16:43 |
| ralonsoh | bye! | 16:43 |
| slaweq | have a great day and week :) | 16:43 |
| slaweq | see You online | 16:43 |
| slaweq | #endmeeting | 16:43 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:43 | |
| slaweq | o/ | 16:43 |
| openstack | Meeting ended Tue Aug 6 16:43:47 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:43 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-08-06-16.00.html | 16:43 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-08-06-16.00.txt | 16:43 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_ci/2019/neutron_ci.2019-08-06-16.00.log.html | 16:43 |
| *** markvoelker has joined #openstack-meeting | 16:44 | |
| *** mlavalle has left #openstack-meeting | 16:44 | |
| *** e0ne has quit IRC | 16:48 | |
| *** ociuhandu has joined #openstack-meeting | 16:49 | |
| *** ociuhandu has quit IRC | 16:57 | |
| *** ociuhandu has joined #openstack-meeting | 16:58 | |
| *** yamahata has quit IRC | 17:03 | |
| *** iyamahat has quit IRC | 17:03 | |
| *** ralonsoh has quit IRC | 17:08 | |
| *** ociuhandu has quit IRC | 17:22 | |
| *** iyamahat has joined #openstack-meeting | 17:22 | |
| *** enriquetaso has joined #openstack-meeting | 17:23 | |
| *** ociuhandu has joined #openstack-meeting | 17:23 | |
| *** iyamahat_ has joined #openstack-meeting | 17:26 | |
| *** ociuhandu has quit IRC | 17:27 | |
| *** ociuhandu has joined #openstack-meeting | 17:28 | |
| *** iyamahat has quit IRC | 17:29 | |
| *** senrique_ has joined #openstack-meeting | 17:33 | |
| *** enriquetaso has quit IRC | 17:35 | |
| *** yamahata has joined #openstack-meeting | 17:42 | |
| *** lpetrut has quit IRC | 17:44 | |
| *** diablo_rojo has quit IRC | 17:45 | |
| *** ociuhandu has quit IRC | 17:48 | |
| *** diablo_rojo has joined #openstack-meeting | 17:55 | |
| *** ociuhandu has joined #openstack-meeting | 18:02 | |
| *** lpetrut has joined #openstack-meeting | 18:03 | |
| *** ociuhandu has quit IRC | 18:04 | |
| *** ociuhandu has joined #openstack-meeting | 18:05 | |
| *** igordc has joined #openstack-meeting | 18:08 | |
| *** ociuhandu has quit IRC | 18:09 | |
| *** armax has joined #openstack-meeting | 18:15 | |
| *** bbowen_ has joined #openstack-meeting | 18:15 | |
| *** bbowen has quit IRC | 18:17 | |
| *** e0ne has joined #openstack-meeting | 18:17 | |
| *** igordc has quit IRC | 18:25 | |
| *** e0ne has quit IRC | 18:31 | |
| *** senrique_ has quit IRC | 18:41 | |
| *** igordc has joined #openstack-meeting | 18:46 | |
| *** jbadiapa has joined #openstack-meeting | 18:47 | |
| *** tesseract has quit IRC | 18:52 | |
| *** Shrews has joined #openstack-meeting | 18:55 | |
| *** jamesmcarthur has joined #openstack-meeting | 18:57 | |
| clarkb | anyone else here for the infra meeting? I'll get things started in a few minutes | 18:59 |
| Shrews | o/ | 19:00 |
| ianw | o/ | 19:00 |
| *** jgriffith has joined #openstack-meeting | 19:00 | |
| clarkb | #startmeeting infra | 19:01 |
| openstack | Meeting started Tue Aug 6 19:01:07 2019 UTC and is due to finish in 60 minutes. The chair is clarkb. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:01 |
| *** openstack changes topic to " (Meeting topic: infra)" | 19:01 | |
| openstack | The meeting name has been set to 'infra' | 19:01 |
| clarkb | #link http://lists.openstack.org/pipermail/openstack-infra/2019-August/006437.html Today'd Agenda | 19:01 |
| clarkb | #link https://wiki.openstack.org/wiki/Meetings/InfraTeamMeeting Edit the agenda at least 24 hours before our scheduled meeting to get items on the agenda | 19:02 |
| clarkb | #topic Announcements | 19:02 |
| *** openstack changes topic to "Announcements (Meeting topic: infra)" | 19:02 | |
| clarkb | Next week I will be attending foundation staff meetings and will not be able to run our weekly meeting. I expect fungi is in the same boat. We will need a non clarkb or fungi volunteer to chair the meeting | 19:03 |
| clarkb | or we can decide to skip it if people prefer that | 19:03 |
| fungi | yup | 19:03 |
| *** jbadiapa has quit IRC | 19:03 | |
| clarkb | Also expect that I won't be much help next week in general | 19:03 |
| fungi | the boat is probably a metaphor | 19:03 |
| *** e0ne has joined #openstack-meeting | 19:04 | |
| fungi | i won't be bringing any fishing gear | 19:04 |
| clarkb | #topic Actions from last meeting | 19:04 |
| *** openstack changes topic to "Actions from last meeting (Meeting topic: infra)" | 19:04 | |
| clarkb | #link http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-07-30-19.01.txt minutes from last meeting | 19:04 |
| clarkb | I think mordred did github things last week | 19:05 |
| fungi | he did indeed | 19:05 |
| clarkb | github.com/openstack-infra repos should all be updated with a note on where they can now be found as well as archived | 19:05 |
| diablo_rojo | o/ | 19:05 |
| clarkb | mordred: was the opendev admin account created too? | 19:05 |
| fungi | s/updated with/replaced by/ | 19:05 |
| corvus | i think i saw mordred say he did that | 19:06 |
| clarkb | awesome | 19:06 |
| clarkb | The other action listed was updating the gitea sshd container to log its sshd logs | 19:06 |
| clarkb | I do not think this happened; however, I can take a look at that today so I'll assign the action to myself | 19:07 |
| clarkb | #action clarkb Have gitea sshd logs recorded somewhere | 19:07 |
| *** mriedem has quit IRC | 19:08 | |
| clarkb | #topic Priority Efforts | 19:08 |
| *** openstack changes topic to "Priority Efforts (Meeting topic: infra)" | 19:08 | |
| mordred | o/ | 19:08 |
| clarkb | #topic OpenDev | 19:08 |
| *** openstack changes topic to "OpenDev (Meeting topic: infra)" | 19:08 | |
| *** jamesmcarthur has quit IRC | 19:08 | |
| clarkb | That is a good jump into recent opendev things | 19:09 |
| mordred | yes - I did github things | 19:09 |
| clarkb | we do still have the OOM problem however it seems less prevalent | 19:09 |
| mordred | and the opendevadmin account | 19:09 |
| clarkb | mordred: tyty | 19:09 |
| clarkb | #link https://etherpad.openstack.org/p/debugging-gitea08-OOM | 19:09 |
| clarkb | Last week I dug into that a bit and tried to collect my thoughts there | 19:09 |
| *** mriedem has joined #openstack-meeting | 19:09 | |
| clarkb | it would probably be good if someone else could review that and see if I missed anything obvious and consider my ideas there | 19:10 |
| corvus | we have no reason to think that gitea 1.9.0 will improve anything, but mordred has a patch to upgrade; assuming we move forward that will be a variable change | 19:10 |
| mordred | clarkb: I pushed up a patch just a little bit ago to upgrade us to gitea 1.9 - it's also possible that 1.9 magically fixes the oom problems | 19:10 |
| mordred | corvus: yes - I agree - I have no reason to believe it fixes anything | 19:10 |
| fungi | right, it's also possible magical elves will fix the oom ;) | 19:11 |
| mordred | so we could also hold off so as not to move variables | 19:11 |
| clarkb | ya I think the memory issues largely come down to big git repos being a problem and gitea holding open requests against big git repos for significant periods of time so they pile up | 19:11 |
| corvus | yeah, but unfounded optimism is great, i endorse it :) | 19:11 |
| mordred | \o/ | 19:11 |
| corvus | i don't think we need to hold back for further debugging; i think we should do the 190 upgrade and just be aware of the change | 19:11 |
| clarkb | we have a 2 minute haproxy timeout (and haproxy seemed to time out these requests because i could not map them to gitea logs based on timestamps) but gitea logs show requests going on for hours | 19:11 |
| clarkb | corvus: ++ | 19:11 |
| clarkb | one idea I had was maybe digging into having gitea timeout requiests because a single 500 error is better than OOMing and crashing gitea | 19:12 |
| fungi | an option there, just spitballing, might be to use an haproxy health check which includes some resource metrics like memory | 19:12 |
| corvus | clarkb: oh that's an interesting data point i missed | 19:12 |
| clarkb | (I have not done that yet, but possibly can later this week) | 19:12 |
| fungi | probably involves running a check agent on the gitea servers though | 19:12 |
| corvus | clarkb: if the remote side has hung up, i don't see why gitea should continue whatever it was doing | 19:12 |
| clarkb | corvus: ya that is at the bottom of the etherpad | 19:12 |
| clarkb | corvus: exactly | 19:12 |
| fungi | but that would force additional requests to get redistributed if there's a pileup on one of the backends | 19:13 |
| fungi | on the other hand it could just end up taking the entire pool offline | 19:13 |
| *** e0ne has quit IRC | 19:13 | |
| clarkb | fungi: ya I think if this continues to be a problem (we've failed at making gitea/git more efficient first) then improving haproxy load balancing methods is our next step | 19:13 |
| clarkb | corvus: you have some grasp of the gitea code base maybe I can take a look at it later this week and if I run into problems ask for help? | 19:14 |
| *** e0ne has joined #openstack-meeting | 19:14 | |
| corvus | clarkb: absolutely | 19:14 |
| clarkb | great. Any other opendev related business before we move on? | 19:15 |
| *** e0ne has quit IRC | 19:15 | |
| corvus | oh one thing | 19:15 |
| corvus | i think tobiash identified the underlying cause of the zuul executors ooming; the fix has merged and if we restart them, things should be better | 19:16 |
| corvus | this is the thing ianw discovered | 19:16 |
| *** jbadiapa has joined #openstack-meeting | 19:16 | |
| clarkb | corvus: is that related to the executor gearman worker class fix? | 19:16 |
| corvus | https://review.opendev.org/674762 is the fix | 19:16 |
| corvus | yes | 19:16 |
| corvus | uneven distribution of jobs makes executors use too much memory and either the log streaming process gets killed, or the executor itself (making the problem worse) | 19:17 |
| corvus | what's really cool is -- | 19:17 |
| corvus | if you take a look at the graphs right now, you can actually see that some of them are graphs of noisy neighbors | 19:17 |
| corvus | (they have oscillations which have no relationship to zuul itself) | 19:18 |
| corvus | because absent our leveling algorithm, external inputs like hypervisor load and network topology have an outsize effect | 19:18 |
| clarkb | because gearman is a "you get jobs as quickly as you can service them" system | 19:18 |
| corvus | yep | 19:19 |
| corvus | nanoseconds count | 19:19 |
| clarkb | #topic Update Config Management | 19:20 |
| *** openstack changes topic to "Update Config Management (Meeting topic: infra)" | 19:20 | |
| clarkb | ianw has changes up to deploy an ansible based/managed backup server | 19:20 |
| * clarkb finds links | 19:20 | |
| ianw | #link https://review.opendev.org/674549 | 19:20 |
| clarkb | ianw wins | 19:20 |
| ianw | #link https://review.opendev.org/674550 | 19:20 |
| ianw | i can fiddle that today if i can get some eyes and see if we can get something backing up to it | 19:20 |
| ianw | should run in parallel with existing backups, so no flag day etc | 19:21 |
| clarkb | ianw: ya the original backup design had us backing up to two location anyway | 19:22 |
| ianw | (well, no client is opted into it yet either, the first one i'll babysit closely) | 19:22 |
| clarkb | I expect that the puppetry will handle that fine | 19:22 |
| clarkb | I think we are in a good spot to start pushing on the CD stuff too? | 19:23 |
| clarkb | corvus: ^ I unfortunately tend to page that out more than I should. You probably know what the next step is there | 19:23 |
| corvus | for jobs triggered by changes to system-config, yes | 19:23 |
| corvus | for the dns stuff, no | 19:23 |
| corvus | https://review.opendev.org/671637 is the current hangup for that | 19:24 |
| clarkb | #link https://review.opendev.org/671637 Next step for CD'ing changes to system-config | 19:24 |
| corvus | that's how i wanted to solve the problem, but logan pointed out a potentially serious problem | 19:24 |
| corvus | so we either need to put more brainpower into that, or adopt one of our secondary plans (such as, opendev takes over the zuul-ci.org zone from the zuul project). that could be a temporary thing until we have the brainpower to solve it better. | 19:25 |
| clarkb | er that is for the dns stuff not system-config right? | 19:26 |
| clarkb | #undo | 19:26 |
| openstack | Removing item from minutes: #link https://review.opendev.org/671637 | 19:26 |
| corvus | clarkb: correct | 19:26 |
| clarkb | #link https://review.opendev.org/671637 Next step for CD'ing changes to DNS zones | 19:26 |
| corvus | no known obstacles to triggering cd jobs from changes to system-config | 19:26 |
| clarkb | got it | 19:26 |
| clarkb | Anything else on this subject? | 19:26 |
| corvus | (project-config is probably ok too) | 19:27 |
| clarkb | #topic Storyboard | 19:28 |
| *** openstack changes topic to "Storyboard (Meeting topic: infra)" | 19:28 | |
| fungi | no updates for sb this week that i'm aware of | 19:28 |
| clarkb | fungi: I know mnaser reported some slowness with the dev server, but I believe that was tracked back to sql queries actually being slow? | 19:28 |
| * mordred did not help on the sql this past week | 19:28 | |
| clarkb | (so there isn't an operational change we need to be aware of?) | 19:28 |
| * mordred will endeavor to do so again this week | 19:29 | |
| fungi | it seemed to be the same behavior, yes. if i tested the same query i saw mysql occupy 100% a vcpu until the query returned | 19:29 |
| fungi | (api query, i mean) | 19:29 |
| *** kopecmartin is now known as kopecmartin|off | 19:29 | |
| clarkb | diablo_rojo: Do you have anything to add? | 19:29 |
| diablo_rojo | mordred, should I just actively bother you in like...two days or something? Would that be helpful? | 19:29 |
| fungi | so the bulk of the wait was in one or more database queries presumably | 19:30 |
| diablo_rojo | clarkb, the only other thing we did during the meeting last week was start to talk about how we want to try to do onboarding in SHanghai of users and another for contributors | 19:30 |
| diablo_rojo | That's all. | 19:30 |
| clarkb | diablo_rojo: related to that can I assume that you have or will handle space allocation for storyboard? or should I make a formal request similar to what I did for infra? | 19:30 |
| fungi | ahh, yep, and covered that trying to facilitate remote participation in shanghai might be harder than normal | 19:31 |
| mordred | diablo_rojo: yeah - actually - if you don't mind | 19:31 |
| mordred | diablo_rojo: I keep remembering on tuesday morning - when I look at the schedule and think "oh, infra meeting" | 19:31 |
| diablo_rojo | mordred, happy to be annoying ;) I'll try to find/create a quality gif or meme for your reminder | 19:31 |
| *** jbadiapa has quit IRC | 19:31 | |
| mordred | hah | 19:31 |
| diablo_rojo | mordred, lol | 19:31 |
| clarkb | diablo_rojo: just let me know if I need to do anything official like for storyboard presence in shanghai. Happy to do so | 19:32 |
| diablo_rojo | clarkb, I will handle space for StoryBoard :) | 19:32 |
| clarkb | awesome | 19:32 |
| diablo_rojo | clarkb, I know the person with the form ;) | 19:32 |
| clarkb | indeed | 19:32 |
| diablo_rojo | ^^ bad joke I will continue to make | 19:32 |
| clarkb | #topic General Topics | 19:33 |
| *** openstack changes topic to "General Topics (Meeting topic: infra)" | 19:33 | |
| clarkb | First up is trusty server replacements. | 19:33 |
| clarkb | fungi: are you planning to do the testing of wiki-dev02? | 19:33 |
| clarkb | iirc planned next steps was to redeploy it to make sure puppet works from scratch? | 19:33 |
| fungi | yep, have been sidetracked by other responsibilities unfortunately, but that's still high on my to do list | 19:33 |
| fungi | wiki-dev02 can simply be deleted and re-launched at any time | 19:34 |
| fungi | nothing is using it | 19:34 |
| fungi | i'll try to get to that this week | 19:34 |
| clarkb | thank you | 19:34 |
| clarkb | corvus has also made great progress with the swift log storage (whcih means we can possibly get rid of logs.openstack.org) | 19:34 |
| clarkb | corvus: at this point you are workign through testing of individual cloud behaviors? | 19:35 |
| corvus | clarkb: yes, i believe rax, and vexxhost are ready, confirming ovh now (i expect it's good) | 19:35 |
| corvus | so we'll be able to randomly store logs in one of six regions | 19:36 |
| clarkb | andI know you intended to switch over to the zuul logs tab with logs.o.o backing it first. Are we ready to start planning that move or do we want to have the swift stuff ready to happen shortly after? | 19:36 |
| corvus | (job/log region proximity would be nice, but not relevant at the moment since our logs still go through the executor) | 19:36 |
| corvus | yeah, we're currently waiting out a deprecation period for one of the roles which ends monday | 19:37 |
| clarkb | exciting we might be switched over next week then? | 19:37 |
| corvus | after that, i think we can switch to zuul build page as the reporting target (but we need a change to zuul to enable that behavior) | 19:37 |
| corvus | and then i think we'll have the swift stuff ready almost immediately after that | 19:38 |
| clarkb | that is great news | 19:38 |
| corvus | maybe we plan for a week between the two changes, just to give time for issues to shake out | 19:38 |
| clarkb | wfm | 19:38 |
| mordred | ++ | 19:39 |
| fungi | yes, it's timely, given we've had something like 3 disruptions to the current log storage in a couple weeks time | 19:39 |
| corvus | though... hrm, timing might be tight on that cause i leave for gerrit user summit soon | 19:39 |
| *** lpetrut has quit IRC | 19:39 | |
| corvus | i leave on aug 22 | 19:40 |
| clarkb | we can probably take our time then and do the switches we are comfortable with bit by bit as people are around to monitor | 19:40 |
| corvus | assuming we don't want to merge it the day before i leave, we really only have next week to work with | 19:40 |
| corvus | i return sept 3 | 19:40 |
| clarkb | k | 19:40 |
| corvus | so we either do both things next week, or build page next week and swift in september | 19:41 |
| * mnaser curious which swifts are being used | 19:41 | |
| clarkb | and we can probably decide on when to do swift based on how smoothly the build logs tag change goes? | 19:41 |
| clarkb | s/tag/tab/ | 19:41 |
| corvus | mnaser: vexxhost, rax, ovh | 19:42 |
| mnaser | just wondering how much data is expected to be likely hosted? | 19:42 |
| clarkb | and fortnebula has hinted that a swift install there might happen too | 19:42 |
| *** bbowen__ has joined #openstack-meeting | 19:42 | |
| corvus | mnaser: i think we're currently estimating about 2GB for starters (much less than when we initially discussed it with you) | 19:42 |
| corvus | er 2TB | 19:43 |
| mnaser | cool, thank you for that info | 19:43 |
| * mnaser hides again | 19:43 | |
| clarkb | Which is a good lead into the next agenda item. State of the clouds | 19:43 |
| clarkb | I wanted to quickly give a status update on fn and was hoping mordred could fill us in on any changes with MOC | 19:43 |
| clarkb | fn is now providing 100 test instances and we seem to be quite stable there now | 19:44 |
| clarkb | We have noticed odd mirror throughput when pulling things from afs | 19:44 |
| mordred | app-creds are working in moc now - so next steps are getting teh second account created and creating the mirror node | 19:44 |
| clarkb | if we manually pull cold files we get about 1MBps and if we pull a warm file we get about 270MBps. But yum installing packages reports 12MBps | 19:44 |
| *** jamesmcarthur has joined #openstack-meeting | 19:44 | |
| *** bbowen_ has quit IRC | 19:45 | |
| clarkb | I am not sure that the afs mirror performance behavior is a major issue as the impact on job runtimes is low | 19:45 |
| clarkb | but something I wanted to make note of | 19:45 |
| clarkb | mordred: exciting | 19:45 |
| corvus | clarkb: only yum? | 19:45 |
| mnaser | yum being slow is nothing new :( | 19:45 |
| clarkb | corvus: I haven't looked at the other package managers yet, but the examples donnyd dug up were yum | 19:45 |
| clarkb | corvus: but that is a good point we should check apt-get too | 19:45 |
| mnaser | OSA's centos jobs take almost twice as long and there isn't a lot of different things happening | 19:46 |
| mnaser | for context | 19:46 |
| clarkb | mnaser: good to know | 19:46 |
| clarkb | mordred: do you need anything to push MOC along or is that mostly you filing a ticket/request for the second account? | 19:46 |
| mordred | nope- just filing a ticket | 19:47 |
| donnyd | Im a little late to the party, but swift will surely be happening. Just a matter of when | 19:47 |
| *** bbowen__ has quit IRC | 19:47 | |
| clarkb | great | 19:48 |
| clarkb | next up is making note of a couple of our distro mirrors' recent struggles | 19:48 |
| ianw | re mirrors i think overall some macro apache throughput stats would be useful, for this and also for kafs comparisons. working on some ideas | 19:48 |
| clarkb | ianw: thanks! | 19:48 |
| clarkb | fungi has found reprepro won't create a repo until it has packages (even if an empty repo exists upstream) | 19:49 |
| clarkb | this is causing problems for debian buster jobs as buster updates does not exist | 19:49 |
| clarkb | fungi: ^ have we managed to workaroudn that yet? | 19:49 |
| ianw | oh ... maybe a skip like we added for the security when it wasn't there? | 19:49 |
| fungi | yeah, the first buster stable point release is scheduled to happen a month from tomorrow, so the buster-updates suite won't exist until then | 19:49 |
| fungi | or rather it will have no packages in it until then | 19:50 |
| fungi | ianw: well, except we also add it to sources.list on test nodes | 19:50 |
| fungi | so would need to actually omit that | 19:50 |
| fungi | or find a way to convince reprepro to generate an empty suite, but i haven't been able to identify a solution in that direction | 19:50 |
| *** e0ne has joined #openstack-meeting | 19:51 | |
| ianw | ahh .. can we just fake an empty something? | 19:51 |
| clarkb | fungi: if we touch a couple files does that result in a valid empty repo or is it more involved than that? | 19:51 |
| clarkb | or maybe even mirror empty repos exactly as upstream rather than building from scratch | 19:51 |
| fungi | it's mostly that... | 19:51 |
| *** senrique_ has joined #openstack-meeting | 19:51 | |
| fungi | i mean, sure we could fake one but need to then find a way to prevent reprepro from removing it | 19:52 |
| clarkb | ah | 19:52 |
| fungi | since it's a suite in an existing mirror | 19:52 |
| fungi | existing package repository i mean | 19:53 |
| fungi | so not as simple as something like debian-security which is a different repository we're mirroring separately | 19:53 |
| clarkb | ok something to dig into more outside of the meeting I guess | 19:53 |
| clarkb | we are almost at time and have a few more thing to bring up really quickly | 19:54 |
| *** jamesmcarthur has quit IRC | 19:54 | |
| fungi | yeah, we can move on | 19:54 |
| clarkb | the fedora mirror has also been struggling. It did not update for about a month because a vos release timed out (presumably that is why the lock on the volume was held) | 19:54 |
| clarkb | I have since manaully updated it and returned the responsibility for updates to the mirror update server. | 19:54 |
| clarkb | One thing I did though was to reduce the size of that mirror by removing virtualbox and vagrant image files, old atomic release files, and power pc files | 19:55 |
| clarkb | That dropped repo size by about 200GB whihc should make vos releases quicker | 19:55 |
| fungi | yeah, the debian mirror was similarly a month stale until we worked out which keys we should be verifying buster-backports with | 19:55 |
| clarkb | that said it is still a large repo and we may be want to further exclude thigns we don't need | 19:55 |
| ianw | thanks; i haven't quite got f30 working which is why i guess nobody noticed ... we should be able to drop f28 then | 19:55 |
| clarkb | I'm watching it now to make sure automatic updates work | 19:56 |
| clarkb | ianw: I think tripleo depends on 28 to stand in for rhel8/centos8 | 19:56 |
| clarkb | ianw: so we might not be able to drop 28 until they also drop it, but ya that will also reduce the size | 19:56 |
| fungi | also... magnum? uses f27 still right? | 19:56 |
| clarkb | fungi: the atomic image only | 19:56 |
| fungi | aha | 19:56 |
| clarkb | fungi: which I don't think uses our mirrors | 19:56 |
| fungi | got it | 19:56 |
| clarkb | And finally we have PTG prep as a topic | 19:57 |
| clarkb | friendly reminder we can start brainstorming topics if we have them | 19:57 |
| clarkb | #link https://etherpad.openstack.org/p/OpenDev-Shanghai-PTG-2019 | 19:57 |
| clarkb | #topic Open Discussion | 19:58 |
| *** openstack changes topic to "Open Discussion (Meeting topic: infra)" | 19:58 | |
| clarkb | we have a couple minutes for any remaining items | 19:58 |
| clarkb | I will be doing family things tomorrow so won't be around | 19:58 |
| Shrews | fwiw, i think i've identified the sdk bug that is causing us to leak swift objects from uploading images to rax. if we fail to upload the final manifest after the segments, we don't retry and don't cleanup after ourselves. seems to happen at least once every few days or so according to what logs we have. | 19:58 |
| clarkb | Shrews: the manifest is the special object that tells swift about the multiobject file? | 19:58 |
| mordred | yah | 19:59 |
| Shrews | clarkb: correct. uploaded last | 19:59 |
| corvus | \o/ | 19:59 |
| clarkb | and we are at time. That is an excellent find re image uploads. Thank you everyone! | 20:00 |
| clarkb | #endmeeting | 20:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 20:00 | |
| openstack | Meeting ended Tue Aug 6 20:00:23 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-08-06-19.01.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-08-06-19.01.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/infra/2019/infra.2019-08-06-19.01.log.html | 20:00 |
| diablo_rojo | Thanks clarkb! | 20:00 |
| fungi | thanks clarkb!!! | 20:00 |
| *** Shrews has left #openstack-meeting | 20:00 | |
| *** lpetrut has joined #openstack-meeting | 20:08 | |
| *** slaweq has quit IRC | 20:09 | |
| *** diablo_rojo has quit IRC | 20:12 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:14 | |
| *** jamesmcarthur has quit IRC | 20:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:17 | |
| *** jamesmcarthur has quit IRC | 20:19 | |
| *** eharney has quit IRC | 20:22 | |
| *** jamesmcarthur has joined #openstack-meeting | 20:25 | |
| *** slaweq has joined #openstack-meeting | 20:25 | |
| *** slaweq has quit IRC | 20:30 | |
| *** yamamoto has joined #openstack-meeting | 20:36 | |
| *** eharney has joined #openstack-meeting | 20:37 | |
| *** yamamoto has quit IRC | 20:41 | |
| *** mriedem is now known as mriedem_afk | 20:46 | |
| *** e0ne has quit IRC | 20:48 | |
| *** priteau has joined #openstack-meeting | 20:51 | |
| *** oneswig has joined #openstack-meeting | 20:54 | |
| *** whoami-rajat has quit IRC | 20:56 | |
| *** b1airo has joined #openstack-meeting | 21:00 | |
| oneswig | #startmeeting scientific-sig | 21:00 |
| openstack | Meeting started Tue Aug 6 21:00:18 2019 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 21:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 21:00 |
| *** janders has joined #openstack-meeting | 21:00 | |
| *** armax has quit IRC | 21:00 | |
| oneswig | away we go | 21:00 |
| b1airo | Morning | 21:00 |
| janders | g'day guys | 21:00 |
| oneswig | #link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_August_6th_2019 | 21:00 |
| *** strigazi has joined #openstack-meeting | 21:01 | |
| oneswig | Morning b1airo janders how's Wednesday going? | 21:01 |
| *** armax has joined #openstack-meeting | 21:01 | |
| janders | oneswig starting slowly :) how are you? | 21:01 |
| b1airo | I'm just back from holiday and organising late start with the kids etc, so will be a bit fleeting | 21:01 |
| priteau | Hello everyone | 21:01 |
| oneswig | Hi priteau! | 21:01 |
| oneswig | janders: I'm well, thanks. Been a fairly intense week working on some customer projects, but a good one. | 21:02 |
| janders | that's good to hear | 21:02 |
| oneswig | I was anticipating jmlowe and trandles coming along today | 21:03 |
| oneswig | janders: what's the latest on supercloud? | 21:03 |
| janders | not much of an update - we're at this time in the year where all the projects are getting reshuffled - so more paperwork than anything else really | 21:04 |
| janders | some interesting challenges with dual port ConnectX6es - happy to touch on this in AOB if there is time | 21:04 |
| oneswig | was talking with somebody recently with an interest in a Manila driver for BeeGFS | 21:05 |
| oneswig | janders: always happy to have some grunty networking issues to round the hour out. | 21:05 |
| janders | :) sounds good | 21:06 |
| b1airo | Yeah Manila BeeGFS sounds useful | 21:06 |
| janders | +1!!! | 21:06 |
| oneswig | #topic OpenStack user survey | 21:06 |
| *** openstack changes topic to "OpenStack user survey (Meeting topic: scientific-sig)" | 21:06 | |
| oneswig | #link yer tiz https://www.openstack.org/user-survey/survey-2019/landing | 21:06 |
| oneswig | Get it filled in and make sure your scientific clouds are stood up and counted. | 21:07 |
| *** trandles has joined #openstack-meeting | 21:07 | |
| oneswig | That's all about that. | 21:07 |
| oneswig | #topic Monitoring for Chargeback and Accounting | 21:07 |
| *** openstack changes topic to "Monitoring for Chargeback and Accounting (Meeting topic: scientific-sig)" | 21:07 | |
| tbarron | just need someone to write and maintain a manila driver for BeeGFS and we commit (manila maintainers) to help it integrate/merge/etc. | 21:07 |
| * tbarron apologizes for jumping before current topic, is done | 21:08 | |
| oneswig | Hi tbarron - ears burning :-) | 21:08 |
| oneswig | Thanks for joining in. | 21:08 |
| *** lpetrut has quit IRC | 21:08 | |
| tbarron | oneswig: :) | 21:08 |
| trandles | o/ I'm late but oh well | 21:09 |
| oneswig | I fear the maintaining bit is often overlooked. But I think there's a good opportunity here to do something good. | 21:09 |
| oneswig | hi trandles, you made it! | 21:10 |
| oneswig | tbarron: we'll see if we can get an interested group together | 21:11 |
| *** igordc has quit IRC | 21:12 | |
| oneswig | #action oneswig to canvas for interest on Manila driver for BeeGFS | 21:12 |
| oneswig | OK, let's return to topic, ok? | 21:13 |
| oneswig | We actually discussed this last week when priteau did some interesting work investigating CloudKitty drawing data from Monasca instead of the Ceilometer family. | 21:13 |
| priteau | oneswig: Telemetry family :) | 21:14 |
| oneswig | I stand corrected | 21:14 |
| oneswig | #link CloudKitty and Monasca (episode 1) https://www.stackhpc.com/cloudkitty-and-monasca-1.html | 21:14 |
| oneswig | This article sets the scene | 21:15 |
| *** senrique_ has quit IRC | 21:15 | |
| oneswig | priteau: I think you've been busy since and assume you've no further developments to report? | 21:15 |
| priteau | I am afraid I've been otherwise engaged | 21:16 |
| priteau | But hopefully the summer is not yet over for a sequel blog post | 21:16 |
| oneswig | I hope not - you can't leave us hanging off this cliff-edge! | 21:17 |
| oneswig | b1airo: were you using CloudKitty at Monash? | 21:17 |
| b1airo | No, looked at it a few times, but never attempted putting it all together | 21:19 |
| oneswig | Ah ok, I remember you mentioning it. | 21:20 |
| *** mriedem_afk is now known as mriedem | 21:20 | |
| oneswig | We have an interest in generating billing data but an aversion to pulling in additional telemetry to do it. | 21:21 |
| priteau | CloudKitty itself isn't very complex to configure, it's more having the right data collected that can be tricky | 21:21 |
| oneswig | I'll be interested to see how it works with data from the OpenStack exporter, or nova instance data | 21:24 |
| *** bbowen__ has joined #openstack-meeting | 21:24 | |
| priteau | Nova is actually the easiest service to charge because there are various ways to collect usage metrics. Other services, like charge for image or volume storage, will be more challenging. | 21:26 |
| oneswig | A pity jmlowe's not around to tell us how they use xdmod (I assume) for this | 21:28 |
| *** shubham_potale has quit IRC | 21:28 | |
| *** igordc has joined #openstack-meeting | 21:31 | |
| oneswig | OK, time for janders ConnectX6 issue? | 21:31 |
| oneswig | #topic AOB | 21:31 |
| *** openstack changes topic to "AOB (Meeting topic: scientific-sig)" | 21:31 | |
| janders | ok! | 21:32 |
| oneswig | janders: what's been going on? | 21:32 |
| janders | does any of you have any experience with dual-port CX6s? | 21:32 |
| oneswig | only CX-5, alas | 21:32 |
| clarkb | Thought I would point out http://lists.openstack.org/pipermail/openstack-discuss/2019-August/008304.html as some of you may be able to answer those questions | 21:33 |
| janders | I think CX5s are a bit easier to work with :) | 21:33 |
| janders | I wanted to use one CX6 port as 50GE and the other as HDR200 | 21:33 |
| oneswig | Hi clarkb - saw that earlier tonight, thanks for raising it. | 21:34 |
| janders | it seems that with the current firmware it's hard to get eth/ib the ports to work concurrently | 21:34 |
| janders | it's a bit of "one or the other" | 21:34 |
| oneswig | What happened to VPI? | 21:34 |
| janders | another angle: do you guys use splitter cables (eg 100GE> 2x50GE)? | 21:35 |
| b1airo | Yes | 21:35 |
| oneswig | janders: yes, on a CX-4 system, had a good deal of trouble initially | 21:35 |
| janders | interesting.. hitting the same | 21:35 |
| janders | what sorts of issues did you have? | 21:36 |
| janders | so far I've seen that the support for splitters sometimes varies across firmware versions (say 1.1 supports it than you upgrade to 1.2 and stuff stops working and support say this version doesn't support splitters) | 21:36 |
| janders | also support guys seem very confused about splitters in general (once I was told these are meant to connect switches not nodes, which seems insane) | 21:37 |
| oneswig | We had 100G-4x25G splitter with 25G SFP+, but the NICs had QSFP sockets. Mellanox do a passive slipper that would take the SFP cable in the QSFP socket - needed some firmware tweaks to get it going. | 21:38 |
| *** markvoelker has quit IRC | 21:38 | |
| janders | right! | 21:38 |
| janders | did they make these tweaks mainstream in the end, or did these remain tweaks? | 21:39 |
| *** jamesmcarthur has quit IRC | 21:39 | |
| oneswig | I think they were mainstream but I'll check the fw version now | 21:39 |
| janders | IIRC our "splitters" are QSFP on both sides | 21:40 |
| oneswig | The CX4 NICs are running 12.20.1010 | 21:41 |
| oneswig | Probably ~1.5 years old | 21:41 |
| janders | ok! | 21:41 |
| janders | so - I think between dual-port CX6es and the splitter cables we might have a bit more "fun" before everything works | 21:41 |
| oneswig | What's preventing running the NIC dual-protocol? Do you think it's the splitter? | 21:42 |
| janders | I was told there are missing bits in the CX6 firmware but that came through a reseller not Mellanox directly | 21:42 |
| janders | so not 100% sure if this is accurate | 21:42 |
| oneswig | That would be a surprising piece to be missing. | 21:43 |
| janders | also given we're using the dual-PCI-slot form factor I have a sneaking suspicion that VPI might be tricker than it used to | 21:43 |
| oneswig | What do you get with CX-6, apart from 200G? | 21:43 |
| janders | thanks, PCIe3.0 | 21:43 |
| janders | nothing | 21:43 |
| janders | I'm actually considering asking to have some cards replaced with CX5s if this can't be resolved promptly | 21:43 |
| janders | HDR200 would be handy for my GPFS cluster though | 21:43 |
| janders | it's NVMe based (similar design to our BeeGFS) so could definitely use that bandwidth | 21:44 |
| janders | but... 2x CX5 could do that, too :) | 21:44 |
| oneswig | You'd get to put one on each socket as well, and perhaps exploit some numa locality | 21:45 |
| janders | indeed | 21:45 |
| janders | I'm anticipating an update from the reseller & Mellanox this week, I will report back if I learn anything intresting | 21:45 |
| janders | it's very useful to know you guys had issues with splitters, too | 21:46 |
| janders | I will be less keen to put them in any larger systems now :( | 21:46 |
| oneswig | We did but I think the QSA28 slipper was part of our problem | 21:46 |
| janders | it's a shame cause these could be used to build some really funky IO-centric topologies | 21:46 |
| janders | I haven't given up on them entirely just yet but it's definitely an amber light in terms of resilient systems design | 21:47 |
| oneswig | I wouldn't give up on them just yet, it would rule out many options | 21:47 |
| janders | one thing seems certain: taking up a new HCA generation + splitters at the same time is painful | 21:48 |
| janders | that's pretty much all I have on this topic for now. Thank you for sharing thoughts, I will keep you posted! :) | 21:49 |
| oneswig | Thanks janders, feel your pain but slightly envy your kit at the same time :-) | 21:50 |
| oneswig | OK, shall we wrap up? Any more to add today? | 21:50 |
| b1airo | I think being an early adopter of anything new from MLNX has a certain degree of pain and perplexedness involved | 21:50 |
| janders | +1 :) | 21:50 |
| oneswig | how about being an early adopter of OPA, in fairness? | 21:51 |
| janders | LOL! | 21:51 |
| janders | I suppose you can be among the first and the last ones, all at the same time | 21:52 |
| b1airo | Hehe | 21:53 |
| b1airo | You guys see ARDC has released a call for proposals for new Research Cloud infra | 21:53 |
| oneswig | What's going on there b1airo? | 21:54 |
| b1airo | Only ~$5.5m worth though it seems | 21:54 |
| b1airo | Probably just aiming to replace existing capacity. Was $20m 6-7 years ago, so $5m probably buys at least the same number of vcores now | 21:55 |
| oneswig | ARDC is New Nectar? | 21:56 |
| b1airo | Yeah, Nectar-ANDS-RDS conglomerate | 21:57 |
| *** raildo has quit IRC | 21:57 | |
| b1airo | Also, quickly on topic of public vs private/hybrid | 21:58 |
| b1airo | Was thinking of reaching out to some server vendors to see if they have some good base modelling on costs, anyone know of some? | 21:58 |
| oneswig | I think Cambridge Uni had some models for cost per core hour that might be relevant but I'm not sure how general (or public) they are | 21:59 |
| oneswig | I'll check | 21:59 |
| oneswig | OK, time to close | 22:00 |
| b1airo | Sounds like I'll need to do some direct legwork | 22:00 |
| oneswig | Thanks all | 22:00 |
| oneswig | #endmeeting | 22:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 22:00 | |
| openstack | Meeting ended Tue Aug 6 22:00:29 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 22:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-08-06-21.00.html | 22:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-08-06-21.00.txt | 22:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-08-06-21.00.log.html | 22:00 |
| b1airo | Cheers gang | 22:01 |
| oneswig | cheers b1airo, have a good one | 22:01 |
| *** trandles has left #openstack-meeting | 22:03 | |
| *** oneswig has quit IRC | 22:09 | |
| *** jamesmcarthur has joined #openstack-meeting | 22:20 | |
| *** efried has left #openstack-meeting | 22:21 | |
| *** priteau has quit IRC | 22:21 | |
| *** ekcs has quit IRC | 22:23 | |
| *** armax has quit IRC | 22:42 | |
| *** joxyuki has quit IRC | 22:43 | |
| *** igordc has quit IRC | 22:56 | |
| *** yamamoto has joined #openstack-meeting | 23:00 | |
| *** markvoelker has joined #openstack-meeting | 23:13 | |
| *** janders has quit IRC | 23:14 | |
| *** markvoelker has quit IRC | 23:18 | |
| *** igordc has joined #openstack-meeting | 23:20 | |
| *** rcernin has joined #openstack-meeting | 23:26 | |
| *** armax has joined #openstack-meeting | 23:33 | |
| *** mriedem has quit IRC | 23:40 | |
| *** mattw4 has quit IRC | 23:42 | |
| *** jamesmcarthur has quit IRC | 23:42 | |
| *** mattw4 has joined #openstack-meeting | 23:42 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:47 | |
| *** jamesmcarthur_ has joined #openstack-meeting | 23:54 | |
| *** jamesmcarthur has quit IRC | 23:54 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!