| *** yamamoto has quit IRC | 00:04 | |
| *** slaweq has joined #openstack-meeting | 00:11 | |
| *** slaweq has quit IRC | 00:16 | |
| *** bobh has quit IRC | 00:44 | |
| *** diablo_rojo has joined #openstack-meeting | 00:53 | |
| *** tetsuro has joined #openstack-meeting | 00:55 | |
| *** ricolin has joined #openstack-meeting | 01:01 | |
| *** enriquetaso has joined #openstack-meeting | 01:10 | |
| *** zhangchi has joined #openstack-meeting | 01:17 | |
| zhangchi | #startmeeting tricircle | 01:18 |
|---|---|---|
| openstack | Meeting started Wed Jul 31 01:18:15 2019 UTC and is due to finish in 60 minutes. The chair is zhangchi. Information about MeetBot at http://wiki.debian.org/MeetBot. | 01:18 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 01:18 |
| *** openstack changes topic to " (Meeting topic: tricircle)" | 01:18 | |
| openstack | The meeting name has been set to 'tricircle' | 01:18 |
| zhangchi | Morning all | 01:18 |
| *** SergeyLukjanov has quit IRC | 01:20 | |
| zhangchi | Welcome back to tricircle meeting | 01:21 |
| *** SergeyLukjanov has joined #openstack-meeting | 01:22 | |
| *** ricolin has quit IRC | 01:31 | |
| *** enriquetaso has quit IRC | 01:32 | |
| *** zhangchi has quit IRC | 01:32 | |
| *** jamesmcarthur has joined #openstack-meeting | 01:35 | |
| *** zhangchi has joined #openstack-meeting | 01:46 | |
| zhangchi | Bye | 01:49 |
| zhangchi | #endmeeting | 01:49 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 01:49 | |
| openstack | Meeting ended Wed Jul 31 01:49:15 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 01:49 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/tricircle/2019/tricircle.2019-07-31-01.18.html | 01:49 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/tricircle/2019/tricircle.2019-07-31-01.18.txt | 01:49 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/tricircle/2019/tricircle.2019-07-31-01.18.log.html | 01:49 |
| *** zhangchi has quit IRC | 01:51 | |
| *** ykatabam has quit IRC | 01:54 | |
| *** yamamoto has joined #openstack-meeting | 01:54 | |
| *** diablo_rojo has quit IRC | 01:59 | |
| *** yamamoto has quit IRC | 01:59 | |
| *** slaweq has joined #openstack-meeting | 02:11 | |
| *** hjwon has quit IRC | 02:13 | |
| *** slaweq has quit IRC | 02:16 | |
| *** tetsuro has quit IRC | 02:16 | |
| *** baojg has joined #openstack-meeting | 02:36 | |
| *** jamesmcarthur has quit IRC | 02:38 | |
| *** jamesmcarthur has joined #openstack-meeting | 02:45 | |
| *** armax has quit IRC | 02:48 | |
| *** ykatabam has joined #openstack-meeting | 02:48 | |
| *** njohnston has quit IRC | 02:51 | |
| *** tetsuro has joined #openstack-meeting | 02:52 | |
| *** rcernin has quit IRC | 02:57 | |
| *** rcernin has joined #openstack-meeting | 03:13 | |
| *** jamesmcarthur has quit IRC | 03:22 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:23 | |
| *** jamesmcarthur has quit IRC | 03:27 | |
| *** tetsuro has quit IRC | 03:28 | |
| *** psachin has joined #openstack-meeting | 03:37 | |
| *** tetsuro has joined #openstack-meeting | 03:46 | |
| *** cheng1 has quit IRC | 03:52 | |
| *** cheng1 has joined #openstack-meeting | 03:52 | |
| *** jamesmcarthur has joined #openstack-meeting | 03:53 | |
| *** imsurit has joined #openstack-meeting | 04:00 | |
| *** jamesmcarthur has quit IRC | 04:00 | |
| *** slaweq has joined #openstack-meeting | 04:11 | |
| *** slaweq has quit IRC | 04:16 | |
| *** whoami-rajat has joined #openstack-meeting | 04:19 | |
| *** apetrich has quit IRC | 04:20 | |
| *** imsurit has quit IRC | 04:25 | |
| *** imsurit has joined #openstack-meeting | 04:26 | |
| *** Luzi has joined #openstack-meeting | 04:30 | |
| *** jamesmcarthur has joined #openstack-meeting | 04:30 | |
| *** jamesmcarthur has quit IRC | 04:38 | |
| *** nitinuikey has joined #openstack-meeting | 04:46 | |
| *** nitinuikey has quit IRC | 04:54 | |
| *** vishalmanchanda has joined #openstack-meeting | 05:13 | |
| *** radeks has joined #openstack-meeting | 05:17 | |
| *** gyee has quit IRC | 05:26 | |
| *** jamesmcarthur has joined #openstack-meeting | 05:34 | |
| *** nitinuikey has joined #openstack-meeting | 05:37 | |
| *** jamesmcarthur has quit IRC | 05:39 | |
| *** baojg has quit IRC | 05:47 | |
| *** links has joined #openstack-meeting | 05:58 | |
| *** jraju__ has joined #openstack-meeting | 06:02 | |
| *** links has quit IRC | 06:03 | |
| *** ykatabam has quit IRC | 06:05 | |
| *** ykatabam has joined #openstack-meeting | 06:07 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:08 | |
| *** slaweq has joined #openstack-meeting | 06:11 | |
| *** rubasov has quit IRC | 06:12 | |
| *** belmoreira has joined #openstack-meeting | 06:13 | |
| *** nitinuikey has quit IRC | 06:13 | |
| *** baojg has joined #openstack-meeting | 06:14 | |
| *** jamesmcarthur has quit IRC | 06:15 | |
| *** slaweq has quit IRC | 06:16 | |
| *** artom has quit IRC | 06:32 | |
| *** artom has joined #openstack-meeting | 06:33 | |
| *** jamesmcarthur has joined #openstack-meeting | 06:57 | |
| *** slaweq has joined #openstack-meeting | 06:58 | |
| *** ykatabam has quit IRC | 07:02 | |
| *** rcernin has quit IRC | 07:02 | |
| *** jamesmcarthur has quit IRC | 07:03 | |
| *** takamatsu has quit IRC | 07:17 | |
| *** brault has joined #openstack-meeting | 07:19 | |
| *** brault has quit IRC | 07:23 | |
| *** apetrich has joined #openstack-meeting | 07:26 | |
| *** rubasov has joined #openstack-meeting | 07:26 | |
| *** tesseract has joined #openstack-meeting | 07:27 | |
| *** belmoreira has quit IRC | 07:28 | |
| *** tssurya has joined #openstack-meeting | 07:31 | |
| *** pcaruana has quit IRC | 07:35 | |
| *** ociuhandu has joined #openstack-meeting | 07:35 | |
| *** igordc has joined #openstack-meeting | 07:37 | |
| *** igordc has quit IRC | 07:38 | |
| *** ociuhandu has quit IRC | 07:40 | |
| *** ociuhandu has joined #openstack-meeting | 07:44 | |
| *** ociuhandu has quit IRC | 07:44 | |
| *** e0ne has joined #openstack-meeting | 07:58 | |
| *** belmoreira has joined #openstack-meeting | 08:01 | |
| *** electrofelix has joined #openstack-meeting | 08:03 | |
| *** lpetrut has joined #openstack-meeting | 08:04 | |
| *** e0ne has quit IRC | 08:07 | |
| *** peschk_l has joined #openstack-meeting | 08:11 | |
| *** tetsuro has quit IRC | 08:11 | |
| *** ttsiouts has joined #openstack-meeting | 08:11 | |
| *** pcaruana has joined #openstack-meeting | 08:13 | |
| *** iyamahat has quit IRC | 08:14 | |
| *** jgriffith has quit IRC | 08:14 | |
| *** ociuhandu has joined #openstack-meeting | 08:15 | |
| *** jgriffith has joined #openstack-meeting | 08:16 | |
| *** imsurit has quit IRC | 08:16 | |
| *** tetsuro has joined #openstack-meeting | 08:17 | |
| *** ttsiouts has quit IRC | 08:21 | |
| *** ttsiouts has joined #openstack-meeting | 08:22 | |
| *** cuseinovic has joined #openstack-meeting | 08:25 | |
| *** ttsiouts has quit IRC | 08:27 | |
| *** takamatsu has joined #openstack-meeting | 08:32 | |
| *** cheng1 has quit IRC | 08:36 | |
| *** ttsiouts has joined #openstack-meeting | 08:36 | |
| *** cheng1 has joined #openstack-meeting | 08:37 | |
| *** ricolin has joined #openstack-meeting | 08:40 | |
| *** takamatsu_ has joined #openstack-meeting | 08:48 | |
| *** [yann] has joined #openstack-meeting | 08:48 | |
| *** takamatsu has quit IRC | 08:49 | |
| *** cheng1 has quit IRC | 08:55 | |
| *** cheng1 has joined #openstack-meeting | 08:58 | |
| *** panda has quit IRC | 08:58 | |
| *** panda has joined #openstack-meeting | 08:59 | |
| *** belmoreira has quit IRC | 09:03 | |
| *** ttsiouts has quit IRC | 09:05 | |
| *** priteau has joined #openstack-meeting | 09:05 | |
| *** tetsuro has quit IRC | 09:06 | |
| *** e0ne has joined #openstack-meeting | 09:06 | |
| *** cheng1 has quit IRC | 09:07 | |
| *** ttsiouts has joined #openstack-meeting | 09:09 | |
| *** cheng1 has joined #openstack-meeting | 09:11 | |
| *** ociuhandu has quit IRC | 09:12 | |
| *** ociuhandu has joined #openstack-meeting | 09:12 | |
| *** ralonsoh has joined #openstack-meeting | 09:13 | |
| *** rfolco|ruck has joined #openstack-meeting | 09:17 | |
| *** imsurit has joined #openstack-meeting | 09:20 | |
| *** nfakhir has joined #openstack-meeting | 09:20 | |
| *** ykatabam has joined #openstack-meeting | 09:22 | |
| *** kopecmartin|off is now known as kopecmartin | 09:23 | |
| *** ttsiouts has quit IRC | 09:23 | |
| *** ttsiouts has joined #openstack-meeting | 09:24 | |
| *** ttsiouts has quit IRC | 09:28 | |
| *** takamatsu_ has quit IRC | 09:30 | |
| *** takamatsu has joined #openstack-meeting | 09:37 | |
| *** [yann] has quit IRC | 09:37 | |
| *** cuseinovic has quit IRC | 09:44 | |
| *** ttsiouts has joined #openstack-meeting | 09:45 | |
| *** ttsiouts has quit IRC | 10:00 | |
| *** ttsiouts has joined #openstack-meeting | 10:01 | |
| *** apetrich has quit IRC | 10:05 | |
| *** ttsiouts has quit IRC | 10:06 | |
| *** takamatsu has quit IRC | 10:08 | |
| *** takamatsu has joined #openstack-meeting | 10:24 | |
| *** lpetrut has quit IRC | 10:24 | |
| *** bbowen has joined #openstack-meeting | 10:34 | |
| *** pcaruana has quit IRC | 10:45 | |
| *** belmoreira has joined #openstack-meeting | 10:50 | |
| *** oneswig has joined #openstack-meeting | 10:50 | |
| *** noggin143 has joined #openstack-meeting | 10:55 | |
| *** witek has joined #openstack-meeting | 10:59 | |
| *** dh3 has joined #openstack-meeting | 10:59 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
| openstack | Meeting started Wed Jul 31 11:00:22 2019 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | Greetings | 11:00 |
| verdurin | Afternoon. | 11:00 |
| oneswig | #link Agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_July_31st_2019 | 11:00 |
| dh3 | hi | 11:00 |
| peschk_l | o/ | 11:00 |
| belmoreira | o/ | 11:00 |
| *** ricolin_ has joined #openstack-meeting | 11:01 | |
| *** Lucas_Gray has joined #openstack-meeting | 11:01 | |
| priteau | \o | 11:01 |
| oneswig | Hi all | 11:01 |
| oneswig | Two quick points to cover first. | 11:02 |
| oneswig | #topic OpenStack user survey | 11:02 |
| *** openstack changes topic to "OpenStack user survey (Meeting topic: scientific-sig)" | 11:02 | |
| oneswig | #link add your scientific openstack cloud here and be counted https://www.openstack.org/user-survey/survey-2019/landing | 11:02 |
| oneswig | It's true, we've yet to do ours... | 11:02 |
| *** ricolin has quit IRC | 11:03 | |
| oneswig | #topic Open Infra days Rome / Milan | 11:03 |
| *** openstack changes topic to "Open Infra days Rome / Milan (Meeting topic: scientific-sig)" | 11:03 | |
| oneswig | Two infra days in one | 11:03 |
| oneswig | #link CFP closes today (not sure what time) https://openinfraday.it/call-for-paper/ | 11:04 |
| oneswig | what better accompaniment to the delights of Italy than the delights of open infrastructure :-) | 11:04 |
| oneswig | Ok, let's move to today's main event | 11:05 |
| oneswig | #topic Monitoring for Chargeback and Accounting | 11:05 |
| *** openstack changes topic to "Monitoring for Chargeback and Accounting (Meeting topic: scientific-sig)" | 11:05 | |
| oneswig | priteau: thanks for doing this investigation and coming along to talk about it. | 11:06 |
| priteau | Happy to share :) | 11:06 |
| oneswig | #link CloudKitty+Monasca for accounting and chargeback https://www.stackhpc.com/cloudkitty-and-monasca-1.html | 11:06 |
| *** njohnston has joined #openstack-meeting | 11:06 | |
| oneswig | ^^ That's Pierre's first findings | 11:06 |
| witek | nice summary | 11:07 |
| oneswig | Hi witek, thanks for joining | 11:07 |
| peschk_l | I've read it this morning, we'd be glad to help with this :) | 11:07 |
| oneswig | excellent | 11:08 |
| priteau | Nice to see that you joined peschk_l. I am still fairly new to CloudKitty so hopefully I didn't write anything completely absurd. | 11:08 |
| witek | priteau: have you tried editing `monasca_field_definitions.yaml` to include additional metadata? | 11:08 |
| oneswig | We've been looking for integrated options for accounting for usage and this is the first contender investigated | 11:09 |
| priteau | witek: I did, but didn't manage yet to get the flavor info in there. There should be a way to do it though, because I see actual values for `created_at`, `launched_at`, `host`… | 11:09 |
| priteau | I also discovered in the process that Monasca has a restriction of 16 metadata values per metric | 11:10 |
| verdurin | Impressive work. I noted the caveat about account not being taken of instance flavour for Ceilosca. That's a shame. | 11:11 |
| priteau | verdurin: It should be possible to pass flavor data by changing the configuration, just haven't found the perfect incantation yet. | 11:12 |
| priteau | I will update the post once I get it to work. | 11:13 |
| verdurin | Great, thanks. | 11:13 |
| oneswig | I think the most interesting piece is the final section and the different ways that usage data can be collected | 11:13 |
| witek | priteau: I assume you've seen that example already: https://opendev.org/openstack/monasca-ceilometer/src/branch/master/etc/ceilometer/examples/monasca_field_definitions.yaml.full-example | 11:13 |
| priteau | Gnocchi stores this metadata by default, so it would be good to have feature parity when using Monasca | 11:13 |
| belmoreira | priteau: nice post. Instance flavor would be a great addition. Did you look into volume types? | 11:14 |
| priteau | witek: Yes, I tried importing monasca_field_definitions.yaml from master, and also adapting meter definitions in Ceilometer. I am sure it must be something minor missing, just need some more debugging to figure it out. | 11:15 |
| priteau | belmoreira: Not yet looked at charging for volumes, and the cloud I am using doesn't actually run Cinder (it's a bare-metal cloud). But you can check peschk_l's post about it: https://www.objectif-libre.com/en/blog/2018/03/14/integration-monasca-et-cloudkitty/ | 11:16 |
| oneswig | The model of polling for usage is subject to rounding errors and coarse granularity. Can you cover the options for using notification events? | 11:16 |
| dh3 | presumably when flavor type is in there, Cloudkitty can charge differently for different flavors? I'm thinking dedicated vs overcommitted CPUs | 11:17 |
| peschk_l | note that the configuration format of cloudkitty has changed since this post, the reference is available here: https://docs.openstack.org/cloudkitty/latest/admin/configuration/collector.html#metric-collection | 11:17 |
| peschk_l | dh3: yes, that's a pretty standard usecase | 11:18 |
| priteau | dh3: Yes, that's the idea. Check the official CloudKitty documentation for an example: https://docs.openstack.org/cloudkitty/latest/user/rating/hashmap.html#examples | 11:18 |
| dh3 | OK, I see, is there a way to scale based on flavor name (e.g. "all m1.* flavors cost 8 times what o1.* flavors cost") or are the flavor costs all set explicitly? | 11:19 |
| priteau | oneswig: Ceilometer is the main option for capturing and publishing notification events, as it already supports many OpenStack services. However, it looks like we may have other options in the future, such as Monasca capturing events directly: http://specs.openstack.org/openstack/monasca-specs/specs/stein/approved/monasca-events-listener.html | 11:19 |
| witek | yes, the the pipeline for persisting notifications is missing the collecting agent only | 11:20 |
| noggin143 | FYI, there is also an initiative going on in the public cloud working group looking at alternatives to ceilometer - https://etherpad.openstack.org/p/publiccloud-sig-billing-implementation-proposal | 11:20 |
| oneswig | priteau: thanks, I was wondering about that. | 11:20 |
| peschk_l | dh3: you need to explicitly set the price of each flavor. But globing would be a nice addition | 11:20 |
| priteau | dh3: I don't think you can do this with the hashmap rating module, but there is also a pyscript one which should allow complete customisation | 11:21 |
| dh3 | peschk_l priteau thanks! | 11:21 |
| oneswig | noggin143: thanks - it would be good to connect to that effort. I added Pierre's blog to their etherpad | 11:21 |
| witek | they're meeting Thursdays, bi-weekly | 11:22 |
| priteau | dh3: But presumably you want to charge differently for each m1.* flavor, so you would have to enter multiple costs explicitly anyway | 11:22 |
| dh3 | yes | 11:22 |
| peschk_l | I beliieve they've a meeting tomorrow at 14h UTC | 11:23 |
| priteau | noggin143: thanks for the reminder about the public cloud SIG effort, I will try to join their meetings | 11:23 |
| *** ttsiouts has joined #openstack-meeting | 11:23 | |
| oneswig | do you know if there has been further discussion beyond the content of the etherpad? | 11:23 |
| noggin143 | there was a discussion at the Denver summit if I remember rightly. | 11:24 |
| peschk_l | (dh3 CK has a meeting friday at 15h UTC, if you want to ask usquestions) | 11:24 |
| priteau | #link http://eavesdrop.openstack.org/#Public_Cloud_SIG | 11:24 |
| *** cheng1 has quit IRC | 11:25 | |
| oneswig | unfortunately eavesdrop does not collect meeting logs for their channel | 11:25 |
| noggin143 | there was also a good presentation at Denver comparing the different agents - https://www.openstack.org/videos/summits/denver-2019/monasca-openstack-monitoring-1 | 11:25 |
| witek | http://eavesdrop.openstack.org/irclogs/%23openstack-publiccloud/ | 11:26 |
| oneswig | I recognise that guy :-) | 11:26 |
| witek | :) | 11:26 |
| priteau | Log of their last meeting: http://eavesdrop.openstack.org/irclogs/%23openstack-publiccloud/%23openstack-publiccloud.2019-07-18.log.html#t2019-07-18T14:00:31 | 11:26 |
| *** cheng1 has joined #openstack-meeting | 11:27 | |
| witek | there have been several discussion following the Denver PTG/Summit | 11:27 |
| oneswig | priteau: there was also the option of having the Monasca Agent draw data from prometheus openstack exporters, what's the latest there? | 11:28 |
| priteau | oneswig: I got this alternative solution to work using a recent version of prometheus-openstack-exporter, which gathers server status. I extended the exporter to report the flavor ID and was able to charge based on this information. | 11:30 |
| priteau | I need to figure out what limitations there are compared to using Ceilometer and notifications, will report in another blog post soon. | 11:31 |
| oneswig | Thanks priteau, I think that would be useful information. | 11:32 |
| oneswig | noggin143: does CERN collect usage data using APEL? | 11:32 |
| peschk_l | one thing with the prometheus openstack exporters is that they do not provide many chageable metrics | 11:32 |
| peschk_l | they're very convenient for monitoring, but do not provide enough information for rating | 11:33 |
| noggin143 | CERN exports usage data to APEL using a combination of data sources and cASO | 11:33 |
| noggin143 | #link https://caso.readthedocs.io/en/stable/ | 11:33 |
| priteau | peschk_l: https://github.com/openstack-exporter/openstack-exporter now has metrics such as `openstack_nova_server_status`, which I've used for rating with CloudKitty. | 11:34 |
| peschk_l | prtieau: good to know, thanks | 11:34 |
| noggin143 | we tried ceilometer for nearly 2 years but there were many issues so belmoreira wrote a sensor to pull the data from libvirt | 11:34 |
| noggin143 | INFN are also interested in this area (Christina Duma) | 11:36 |
| oneswig | noggin143: do they use a similar setup to CERN? | 11:36 |
| priteau | noggin143: did you have issues with Ceilometer extracting the data or its metrics storage backend? (originally MongoDB, then Gnocchi) | 11:37 |
| *** raildo has joined #openstack-meeting | 11:38 | |
| belmoreira | priteau we only used MongoDB (Gnocchi was still not available). Extracting data as nearly impossible | 11:39 |
| belmoreira | currently we only keep ceilometer for the notifications | 11:40 |
| priteau | I had the same experience on Chameleon. We had to purge the MongoDB database regularly if we wanted queries to return a result. | 11:40 |
| noggin143 | For INFN, they did not have a good solution and were looking for advice | 11:40 |
| priteau | oneswig: I just found that the publiccloud meeting logs are actually at http://eavesdrop.openstack.org/meetings/publiccloud_wg/ | 11:41 |
| *** raub has joined #openstack-meeting | 11:41 | |
| priteau | They still use publiccloud_wg instead of publiccloud_sig for their meeting name. | 11:42 |
| noggin143 | ceilometer at scale also created very high keystone load | 11:42 |
| oneswig | ah ok, thanks priteau | 11:42 |
| oneswig | noggin143: I think we've been using the caso collector, or something similar to it, for parts of the IRIS project. | 11:43 |
| oneswig | Did anyone have more to add on this, or suggestions for other areas to explore? | 11:45 |
| priteau | Something nice about CloudKitty is that it's very easy to extend. I wrote a proof of concept collector that uses the Nova os-simple-tenant-usage API to charge for historical Nova usage. | 11:46 |
| peschk_l | Well, we'd love some feedback about people using CK with monasca, as there are only few existing integrations. priteau 's article was nice for that | 11:48 |
| peschk_l | priteau: is the code available somewhere ? | 11:49 |
| *** noggin143 has quit IRC | 11:49 | |
| *** ociuhandu has quit IRC | 11:49 | |
| priteau | peschk_l: I'll clean it up a bit and share it | 11:49 |
| witek | one thing I'd like to move forward in future is to instrument the OpenStack services itself | 11:50 |
| witek | so that they can provide valuable measurements about their state (and usage) without the need of external black-box monitoring | 11:51 |
| oneswig | witek: sounds like really useful insights. | 11:51 |
| oneswig | I am sure there are a lot of "golden signals" that are buried deep within these services. | 11:51 |
| priteau | This would be great for monitoring large OpenStack deployments. You can monitor things externally, like API response time, but having internal insight would be even better. | 11:52 |
| dh3 | witek: that sounds good, we do some ad-hoc monitoring of e.g. rabbitmq queue sizes, would be nice to have that instrumented "out of the box" | 11:52 |
| oneswig | We are nearing time to close. Final points for discussion? | 11:54 |
| oneswig | #topic AOB | 11:55 |
| *** openstack changes topic to "AOB (Meeting topic: scientific-sig)" | 11:55 | |
| dh3 | just a quick call for anyone with experience of Spark/Hail talking to Ceph rgw/S3 - got a user getting unexpected Content-Length errors which I can't reproduce - would like to talk to someone else who has done this | 11:56 |
| oneswig | Might be one for verdurin? | 11:57 |
| verdurin | dh3: we have some interest here, but no-one running it yet | 11:58 |
| dh3 | verdurin: ah OK, thanks anyway | 11:58 |
| oneswig | dh3: how has that rolling ceph upgrade gone? | 11:58 |
| dh3 | oneswig: complete :) | 11:58 |
| oneswig | I'm watching one deep-scrub before moving to Nautilus, hopefully this afternoon | 11:58 |
| dh3 | (now we are battling large omap objects and orphaned bits and pieces...........) | 11:58 |
| oneswig | dh3 - that's a story for another day, surely | 11:59 |
| oneswig | Are you going to CERN Ceph day in September? | 11:59 |
| dh3 | no but my colleague Matthew is | 11:59 |
| dh3 | he will probably have war stories to share :) | 12:00 |
| oneswig | Great - hopefully I'll see him there. | 12:00 |
| *** dviroel has joined #openstack-meeting | 12:00 | |
| oneswig | OK we are out of time | 12:00 |
| verdurin | Thanks, bye. | 12:00 |
| oneswig | thanks priteau and everyone for joining | 12:00 |
| oneswig | #endmeeting | 12:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:00 | |
| openstack | Meeting ended Wed Jul 31 12:00:36 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-07-31-11.00.html | 12:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-07-31-11.00.txt | 12:00 |
| *** ykatabam has left #openstack-meeting | 12:00 | |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2019/scientific_sig.2019-07-31-11.00.log.html | 12:00 |
| priteau | Good bye | 12:00 |
| *** oneswig has quit IRC | 12:00 | |
| dh3 | ttfn | 12:01 |
| peschk_l | thanks oneswig | 12:01 |
| priteau | Thanks witek and peschk_l for joining, I'll keep you posted of my progress. | 12:01 |
| witek | thanks priteau, good work | 12:02 |
| *** links has joined #openstack-meeting | 12:07 | |
| *** jraju__ has quit IRC | 12:07 | |
| *** takamatsu has quit IRC | 12:10 | |
| *** pcaruana has joined #openstack-meeting | 12:10 | |
| peschk_l | thanks priteau | 12:14 |
| *** carloss has joined #openstack-meeting | 12:19 | |
| *** ociuhandu has joined #openstack-meeting | 12:22 | |
| *** takamatsu has joined #openstack-meeting | 12:31 | |
| *** priteau has quit IRC | 12:39 | |
| *** psachin has quit IRC | 12:41 | |
| *** bobh has joined #openstack-meeting | 12:43 | |
| *** apetrich has joined #openstack-meeting | 12:43 | |
| *** apetrich has quit IRC | 12:44 | |
| *** apetrich has joined #openstack-meeting | 12:45 | |
| *** jamesmcarthur has joined #openstack-meeting | 12:46 | |
| *** bobh has quit IRC | 12:47 | |
| *** lpetrut has joined #openstack-meeting | 12:50 | |
| *** ricolin_ is now known as ricolin | 12:50 | |
| *** Luzi has quit IRC | 12:54 | |
| *** dh3 has quit IRC | 12:59 | |
| *** lpetrut has quit IRC | 12:59 | |
| *** lpetrut has joined #openstack-meeting | 12:59 | |
| *** brtknr has left #openstack-meeting | 13:16 | |
| *** tesseract has quit IRC | 13:20 | |
| *** enriquetaso has joined #openstack-meeting | 13:22 | |
| *** tesseract has joined #openstack-meeting | 13:24 | |
| *** lseki has joined #openstack-meeting | 13:25 | |
| *** ttsiouts has quit IRC | 13:25 | |
| *** ttsiouts has joined #openstack-meeting | 13:26 | |
| *** bbowen has quit IRC | 13:26 | |
| *** mriedem has joined #openstack-meeting | 13:27 | |
| *** bbowen has joined #openstack-meeting | 13:27 | |
| *** bbowen has quit IRC | 13:29 | |
| *** bbowen has joined #openstack-meeting | 13:30 | |
| *** ttsiouts has quit IRC | 13:31 | |
| *** Lucas_Gray has quit IRC | 13:33 | |
| *** Lucas_Gray has joined #openstack-meeting | 13:38 | |
| *** ociuhandu has quit IRC | 13:39 | |
| *** ociuhandu has joined #openstack-meeting | 13:39 | |
| *** ttsiouts has joined #openstack-meeting | 13:41 | |
| *** belmoreira has quit IRC | 13:43 | |
| *** belmoreira has joined #openstack-meeting | 13:45 | |
| *** liuyulong has joined #openstack-meeting | 13:46 | |
| *** eharney has joined #openstack-meeting | 13:55 | |
| *** jamesmcarthur has quit IRC | 13:55 | |
| *** belmoreira has quit IRC | 13:56 | |
| liuyulong | #startmeeting neutron_l3 | 14:00 |
| openstack | Meeting started Wed Jul 31 14:00:59 2019 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:01 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:01 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:01 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:01 |
| liuyulong | Hello | 14:01 |
| slaweq | hi | 14:02 |
| liuyulong | Today will be a quick short meeting, IMO. | 14:02 |
| njohnston | o/ | 14:02 |
| liuyulong | #topic Announcements | 14:02 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:02 | |
| haleyb | hi | 14:03 |
| liuyulong | #link https://etherpad.openstack.org/p/Shanghai-Neutron-Planning | 14:03 |
| liuyulong | Just added my name here ^ | 14:03 |
| slaweq | great, we will finally meet in person liuyulong :) | 14:03 |
| liuyulong | #chair haleyb | 14:03 |
| openstack | Current chairs: haleyb liuyulong | 14:03 |
| *** Lucas_Gray has quit IRC | 14:04 | |
| liuyulong | I have no more announcements. IMO, we have reminded all of them yesterday in team meeting. | 14:05 |
| liuyulong | OK, let's move on. | 14:06 |
| liuyulong | #topic Bugs | 14:06 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:06 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008089.html | 14:06 |
| liuyulong | Boden Russell was our bug deputy last week, thanks. | 14:06 |
| liuyulong | And again, I will skip all the bugs which were fixed or the related patches are getting merged now. | 14:07 |
| liuyulong | First one: | 14:07 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1837635 | 14:07 |
| openstack | Launchpad bug 1837635 in neutron "HA router state change from "standby" to "master" should be delayed" [Undecided,In progress] - Assigned to Rodolfo Alonso (rodolfo-alonso-hernandez) | 14:07 |
| liuyulong | For the fix, IMO, it looks good to me, but I'd like to see more deep test results. | 14:08 |
| liuyulong | I just added two scenarios here: https://review.opendev.org/#/c/672533/ | 14:08 |
| *** jraju__ has joined #openstack-meeting | 14:09 | |
| *** links has quit IRC | 14:09 | |
| liuyulong | Sometimes, the actual results of the running program may be different from what you expected. | 14:10 |
| liuyulong | Next: | 14:10 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1834308 | 14:10 |
| openstack | Launchpad bug 1834308 in neutron "[DVR][DB] too many slow query during agent restart" [Medium,In progress] - Assigned to LIU Yulong (dragon889) | 14:10 |
| liuyulong | I submitted the patch set yesterday. | 14:11 |
| liuyulong | It is here: https://review.opendev.org/#/c/673557/ | 14:11 |
| liuyulong | A pep8 failure... | 14:11 |
| liuyulong | All the DB query in this patch has the highest frequency of call when restart ovs-agent. | 14:12 |
| haleyb | do you have numbers on the improvement? | 14:13 |
| liuyulong | And it is time-consuming, when your 'ports' table is getting larger and lager. These query will have a worse results. | 14:14 |
| *** iyamahat has joined #openstack-meeting | 14:14 | |
| liuyulong | 40 nodes of ovs-agent restart will call these DB query about 300K+ times. | 14:14 |
| *** belmoreira has joined #openstack-meeting | 14:15 | |
| liuyulong | And these query costs about 0.1s+ seconds logged by our mariadb cluster. | 14:17 |
| haleyb | .1s per query? | 14:17 |
| *** brinzhang_ has quit IRC | 14:17 | |
| liuyulong | Yes, one of them is about 0.5s+. Let me link it in the gerrit. | 14:18 |
| *** brinzhang_ has joined #openstack-meeting | 14:18 | |
| *** brinzhang_ has quit IRC | 14:18 | |
| liuyulong | https://review.opendev.org/#/c/673557/1/neutron/db/dvr_mac_db.py@145 | 14:18 |
| *** brinzhang_ has joined #openstack-meeting | 14:18 | |
| liuyulong | get_ports_on_host_by_subnet | 14:18 |
| liuyulong | This one. | 14:19 |
| liuyulong | haleyb, the results is when the ports table has about 10-20K records. | 14:20 |
| *** brinzhang has quit IRC | 14:21 | |
| haleyb | so _get_ports_query() is really slow | 14:23 |
| liuyulong | The scale of resource is about: 17000+ VMs, 3000+ DVR routers, 3000+ network, 3000+ subnets and 3000+ security groups; 40 security group rules for each security group. | 14:25 |
| liuyulong | After this change, the ovs-agent restart time has a very significant improvement, it's about 40-50mins to 15mins. | 14:26 |
| njohnston | I wonder if it would be further optimized by adding an index specifically on the Port.device_owner field. I'll comment on the change. | 14:26 |
| tidwellr | interesting | 14:26 |
| tidwellr | hi, btw | 14:27 |
| haleyb | liuyulong: that's quite an improvement, even if 15mins is still a long time :) | 14:27 |
| *** AlanClark has joined #openstack-meeting | 14:27 | |
| liuyulong | 40 - 50 mins, I can't believe it once, but indeed it is. | 14:27 |
| *** takamatsu has quit IRC | 14:28 | |
| liuyulong | rpc_loop 1 it will scan the ports and process it. 40-50mins....... | 14:28 |
| haleyb | liuyulong: it looks like you have lots of reviewers now | 14:30 |
| liuyulong | More detail about our test deployment is: 3 neutron-server with about (172 workers), its 3 nodes DB and 3 nodes MQ, are all in dedicated server. | 14:31 |
| liuyulong | Yes, neutron has its own DB and MQ. | 14:31 |
| *** trident has quit IRC | 14:32 | |
| *** jraju__ has quit IRC | 14:34 | |
| liuyulong | Last one: | 14:35 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1838431 | 14:35 |
| openstack | Launchpad bug 1838431 in neutron "[scale issue] ovs-agent port processing time increases linearly and eventually timeouts" [Undecided,New] | 14:35 |
| liuyulong | More like a L2 issue... | 14:35 |
| *** trident has joined #openstack-meeting | 14:36 | |
| slaweq | liuyulong: this one looks like related to already known problem with "remote_security_group" rules in SG | 14:37 |
| slaweq | there was bug reported for it already IIRC | 14:37 |
| liuyulong | The test have not get successfully yet. | 14:37 |
| liuyulong | I will test it again today. | 14:38 |
| liuyulong | One more interesting thing is that we disable the DHCP for this test. No DHCP agent in this test. I can image if DHCP is enabled the vif-plug-timeout may get more... | 14:40 |
| liuyulong | That's all bugs from me. | 14:41 |
| liuyulong | any other bugs that need the team to pay attention? | 14:41 |
| haleyb | there was one miguel filed yesterday | 14:41 |
| slaweq | liuyulong: I can't find any bug reported for Your last issue but please check https://etherpad.openstack.org/p/openstack-networking-train-ptg in L347 | 14:41 |
| slaweq | njohnston: raised this problem on last PTG | 14:41 |
| liuyulong | slaweq, OK, great | 14:41 |
| haleyb | https://bugs.launchpad.net/neutron/+bug/1838449 | 14:42 |
| openstack | Launchpad bug 1838449 in neutron "Router migrations failing in the gate" [Medium,Confirmed] - Assigned to Miguel Lavalle (minsel) | 14:42 |
| slaweq | liuyulong: please do Your test without security group rules which reference to remote_group_id | 14:42 |
| slaweq | than it should be much, much faster | 14:42 |
| haleyb | liuyulong: that was the only other bug i had, was going to try and reproduce locally today for miguel | 14:44 |
| slaweq | haleyb: yes, this one hurts us quite lot in CI jobs | 14:45 |
| liuyulong | slaweq, actually I refactor may test to 27 tenants yesterday. It looks better now. | 14:45 |
| liuyulong | haleyb, thanks for bring up this, seems Miguel has found the issue code. | 14:46 |
| slaweq | liuyulong: because if You have more tenants, there is less IPs (ports) using same security group probably and thus it's faster | 14:46 |
| liuyulong | slaweq, yes, and I'm trying to add more security group for each tenant, or network. | 14:47 |
| *** trident has quit IRC | 14:47 | |
| liuyulong | For one tenant and one default security group, it is a disaster. | 14:48 |
| liuyulong | IMO, every one try to test this will be very easy to encounter this problem. | 14:49 |
| liuyulong | njohnston's PTG summary looks very similar to this. | 14:49 |
| slaweq | liuyulong: yes, we had this issue too | 14:49 |
| *** trident has joined #openstack-meeting | 14:50 | |
| liuyulong | And maybe some security group DB query also need some optimizing work. | 14:51 |
| liuyulong | OK, next topic | 14:51 |
| liuyulong | #topic Routed Networks | 14:51 |
| *** openstack changes topic to "Routed Networks (Meeting topic: neutron_l3)" | 14:51 | |
| *** belmoreira has quit IRC | 14:51 | |
| liuyulong | mlavalle, tidwellr, wwriverrat: your turn now. | 14:51 |
| tidwellr | if mlavalle and wwriverrat don't have anything, we can talk briefly floating IP's for routed networks | 14:53 |
| tidwellr | https://review.opendev.org/#/c/486450/ and the POC code https://review.opendev.org/#/c/669395/ | 14:53 |
| njohnston | I don't see mlavalle online | 14:55 |
| tidwellr | if it isn't obvious by my nagging folks to take a look at these, this has turned into my pet project :) | 14:55 |
| tidwellr | I've spun up a little lab where I've tested the POC code, it seems to work nicely and it's not terribly invasive. What I'm interested in is feedback about the approach in the spec | 14:57 |
| liuyulong | tidwellr, thank you for replying to my question in the patch sets. | 15:00 |
| liuyulong | OK, let's end the meeting. | 15:00 |
| liuyulong | #endmeeting | 15:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:00 | |
| openstack | Meeting ended Wed Jul 31 15:00:17 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2019/neutron_l3.2019-07-31-14.00.html | 15:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2019/neutron_l3.2019-07-31-14.00.txt | 15:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2019/neutron_l3.2019-07-31-14.00.log.html | 15:00 |
| *** trident has quit IRC | 15:00 | |
| liuyulong | Thank you guys for attending. | 15:00 |
| liuyulong | Bye | 15:00 |
| slaweq | bye | 15:00 |
| slaweq | o/ | 15:00 |
| *** armax has joined #openstack-meeting | 15:02 | |
| *** trident has joined #openstack-meeting | 15:03 | |
| *** belmoreira has joined #openstack-meeting | 15:03 | |
| *** rosmaita has joined #openstack-meeting | 15:06 | |
| *** zhengMa has joined #openstack-meeting | 15:07 | |
| *** trident has quit IRC | 15:08 | |
| *** yamahata has quit IRC | 15:15 | |
| *** iyamahat has quit IRC | 15:15 | |
| *** trident has joined #openstack-meeting | 15:16 | |
| *** jamesmcarthur has joined #openstack-meeting | 15:18 | |
| *** diablo_rojo has joined #openstack-meeting | 15:19 | |
| *** trident has quit IRC | 15:24 | |
| *** bbowen has quit IRC | 15:26 | |
| *** trident has joined #openstack-meeting | 15:27 | |
| *** tosky has joined #openstack-meeting | 15:27 | |
| *** lpetrut has quit IRC | 15:34 | |
| *** trident has quit IRC | 15:35 | |
| *** trident has joined #openstack-meeting | 15:38 | |
| *** ttsiouts has quit IRC | 15:42 | |
| *** ttsiouts has joined #openstack-meeting | 15:42 | |
| *** jamesmcarthur has quit IRC | 15:44 | |
| *** tssurya has quit IRC | 15:46 | |
| *** ttsiouts has quit IRC | 15:47 | |
| *** e0ne has quit IRC | 15:49 | |
| *** e0ne has joined #openstack-meeting | 15:50 | |
| *** belmoreira has quit IRC | 15:50 | |
| *** gyee has joined #openstack-meeting | 15:52 | |
| *** woojay has joined #openstack-meeting | 15:53 | |
| *** sfernand has joined #openstack-meeting | 15:53 | |
| *** sfernand has left #openstack-meeting | 15:54 | |
| *** zaneb has quit IRC | 15:56 | |
| *** efried is now known as efried_rollin | 15:57 | |
| *** witek has quit IRC | 15:59 | |
| *** ociuhandu has quit IRC | 15:59 | |
| *** sfernand has joined #openstack-meeting | 16:00 | |
| jungleboyj | #startmeeting Cinder | 16:00 |
| openstack | Meeting started Wed Jul 31 16:00:18 2019 UTC and is due to finish in 60 minutes. The chair is jungleboyj. Information about MeetBot at http://wiki.debian.org/MeetBot. | 16:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 16:00 |
| *** openstack changes topic to " (Meeting topic: Cinder)" | 16:00 | |
| openstack | The meeting name has been set to 'cinder' | 16:00 |
| jungleboyj | courtesy ping: jungleboyj whoami-rajat rajinir lseki carloss pots woojay erlon geguileo eharney rosmaita enriquetaso e0ne smcginnis davidsha walshh_ xyang hemna _hemna tosky | 16:00 |
| whoami-rajat | Hi | 16:00 |
| walshh_ | hi | 16:00 |
| pots | o/ | 16:00 |
| rosmaita | o/ | 16:00 |
| jungleboyj | @! | 16:00 |
| _pewp_ | jungleboyj ＼( ･_･) | 16:00 |
| tosky | o/ | 16:00 |
| woojay | \o | 16:00 |
| smcginnis | o/ | 16:01 |
| carloss | hi | 16:01 |
| e0ne | hi | 16:01 |
| eharney | hi | 16:01 |
| *** thgcorrea has joined #openstack-meeting | 16:01 | |
| lseki | ji | 16:01 |
| lseki | hi | 16:02 |
| jungleboyj | Cinderers Assemble! | 16:02 |
| jungleboyj | Give people one more minute here. | 16:03 |
| *** andrewar_ has joined #openstack-meeting | 16:04 | |
| rosmaita | i don't see how anyone would dare to be late after hearing "Cinderers Assemble!" | 16:04 |
| jungleboyj | rosmaita: :-) | 16:04 |
| jungleboyj | Secretly I needed more coffee. I am the late one. | 16:04 |
| jungleboyj | ;-) | 16:04 |
| jungleboyj | Ok. | 16:05 |
| jungleboyj | #topic announcements | 16:05 |
| *** openstack changes topic to "announcements (Meeting topic: Cinder)" | 16:05 | |
| jungleboyj | I have submitted a request to be part of the Shanghai PTG. | 16:05 |
| jungleboyj | I know that rosmaita and I are going to be there. | 16:05 |
| rosmaita | :) | 16:05 |
| jungleboyj | Well, I don't have travel approved yet, but the Lenovo offices are hosting OUI, so they better get me there. | 16:06 |
| * e0ne didn't get approval yet | 16:06 | |
| jungleboyj | Anyway, I estimated our attendance at 20 people, half what we have had in the past, and asked for 2 days for technical discussion and .25 days for onboarding. | 16:06 |
| jungleboyj | Haven't heard any confirmation on the request yet. | 16:07 |
| rajinir | o/ | 16:07 |
| jungleboyj | So, that is the plan there. | 16:07 |
| jungleboyj | We have the mid-cycle coming up in Raleigh in less than a month now: https://etherpad.openstack.org/p/cinder-train-mid-cycle-planning | 16:07 |
| jungleboyj | #link https://etherpad.openstack.org/p/cinder-train-mid-cycle-planning | 16:08 |
| jungleboyj | We have 4 people confirmed. | 16:08 |
| enriquetaso | o/ | 16:08 |
| jungleboyj | 4 or 5 remote attendees. | 16:08 |
| jungleboyj | I am hoping to have zoom to use to host the meeting this time. | 16:08 |
| jungleboyj | *fingers crossed* | 16:08 |
| jungleboyj | Please add topics to the etherpad. | 16:09 |
| jungleboyj | I will work on getting topics organized before the meeting. | 16:09 |
| whoami-rajat | jungleboyj: hangout was good last time :) | 16:09 |
| jungleboyj | We will have hangouts if nothing else. :-) | 16:10 |
| *** zaneb has joined #openstack-meeting | 16:10 | |
| jungleboyj | That is all I had for announcements | 16:10 |
| jungleboyj | #topic Milestone-2 follow-up | 16:10 |
| *** openstack changes topic to "Milestone-2 follow-up (Meeting topic: Cinder)" | 16:10 | |
| jungleboyj | I think we got the possible drivers and specs merged. | 16:11 |
| jungleboyj | Thank you to all who helped make that happen. | 16:11 |
| jungleboyj | hemna_: and rosmaita The last minute help was greatly appreciated. | 16:11 |
| whoami-rajat | Thanks rosmaita | 16:12 |
| rosmaita | de nada | 16:12 |
| jungleboyj | So, I wanted to ask about the spec for updating the scheduler to use the DB. | 16:12 |
| jungleboyj | e0ne: Wants to delay that to the Unicorn release so he can focus on the generic backup work. | 16:12 |
| jungleboyj | That sounds like a good plan to me. | 16:13 |
| e0ne | jungleboyj: thanks for mentioning this | 16:13 |
| eharney | more time for design thought on that sounds like it would be beneficial anyway, so, cool | 16:13 |
| jungleboyj | Yeah ... I am fine with that. | 16:13 |
| jungleboyj | e0ne: Thanks for being realistic about what you can get done. | 16:13 |
| *** lpetrut has joined #openstack-meeting | 16:14 | |
| e0ne | jungleboyj: thanks for supporting me. I appreciate your help | 16:14 |
| jungleboyj | e0ne: Any time. | 16:14 |
| jungleboyj | So, was there anything else that people wanted to bring up about Milestone-2? | 16:14 |
| *** lpetrut has quit IRC | 16:15 | |
| jungleboyj | Ok. | 16:15 |
| *** lpetrut has joined #openstack-meeting | 16:15 | |
| jungleboyj | #topic Leverage hw accelerator in image compression | 16:15 |
| *** openstack changes topic to "Leverage hw accelerator in image compression (Meeting topic: Cinder)" | 16:15 | |
| jungleboyj | zhengMa: | 16:15 |
| zhengMa | hi | 16:16 |
| *** gyee has quit IRC | 16:16 | |
| jungleboyj | zhengMa: Hey. | 16:16 |
| jungleboyj | #link https://review.opendev.org/#/c/668825/ | 16:16 |
| *** trident has quit IRC | 16:16 | |
| *** gyee has joined #openstack-meeting | 16:17 | |
| zhengMa | Happy for spec merged | 16:17 |
| jungleboyj | Yes. Good news. | 16:17 |
| zhengMa | And we have implemented according to the spec | 16:17 |
| *** baojg has quit IRC | 16:17 | |
| jungleboyj | Good. Looks like you patch is currently failing the check. | 16:17 |
| zhengMa | #spec link https://review.opendev.org/#/c/652275/ | 16:18 |
| zhengMa | yes, because of the unit test.. | 16:18 |
| jungleboyj | You are also missing an update to cinder/opts.py | 16:18 |
| zhengMa | we have not implemented yet :( | 16:18 |
| zhengMa | tox right? | 16:19 |
| jungleboyj | tox -e genopts | 16:19 |
| zhengMa | sorry, my mistake | 16:19 |
| jungleboyj | Check the pep8 check for details. | 16:19 |
| zhengMa | Ok, got it | 16:20 |
| jungleboyj | Otherwise, it does appear that you have a UT issue. | 16:20 |
| jungleboyj | zhengMa: Is this just a request for reviews? | 16:21 |
| zhengMa | yes, the newly added container_format property DOES impact some unit test | 16:21 |
| *** trident has joined #openstack-meeting | 16:21 | |
| jungleboyj | If so, noted. If you can get the items I have noted fixed we will take a look. | 16:21 |
| zhengMa | jungleboyj: yes, we want some comments so we can quickly fix our code:) | 16:22 |
| jungleboyj | Ok. Will do. Start by getting the check passing please. | 16:22 |
| zhengMa | Ok, thank you! | 16:22 |
| eharney | does it implement the non-hardware compression path? | 16:22 |
| *** electrofelix has quit IRC | 16:22 | |
| zhengMa | eharney: yes, we implemented a gzip class for software compression | 16:23 |
| eharney | ok | 16:23 |
| zhengMa | jungleboyj: got it, thank you! | 16:23 |
| jungleboyj | Cool. Any other questions? | 16:23 |
| hemna_ | flerm | 16:23 |
| * jungleboyj marks hemna_ tardy | 16:23 | |
| hemna_ | :P | 16:24 |
| rosmaita | zhengMa: those failures look like a test fixture problem | 16:24 |
| jungleboyj | @!h | 16:24 |
| _pewp_ | jungleboyj (/ .□.) ︵╰(゜Д゜)╯︵ /(.□. ) | 16:24 |
| jungleboyj | rosmaita: ++ | 16:24 |
| jungleboyj | #topic question about the cross-project GPG encryption effort | 16:25 |
| *** openstack changes topic to "question about the cross-project GPG encryption effort (Meeting topic: Cinder)" | 16:25 | |
| zhengMa | rosmaita: yes, I have just read the log. And we're going to fix it | 16:25 |
| jungleboyj | rosmaita: | 16:25 |
| rosmaita | we have an approved spec for train, "image encryption and decryption" | 16:25 |
| rosmaita | #link http://specs.openstack.org/openstack/cinder-specs/specs/train/image-encryption.html | 16:25 |
| rosmaita | it says that "This spec requires the implementation of appropriate encryption/decryption functionality in a global library shared between the components involved in image encryption workflows (Nova, Cinder, OSC), like os-brick" | 16:25 |
| rosmaita | i was reviewing the glance spec for this, and that one says about the global library, "We determined to use os-brick". | 16:25 |
| rosmaita | #link https://review.opendev.org/#/c/609667/6 | 16:25 |
| rosmaita | which is a bit stronger statement | 16:25 |
| rosmaita | so this is partly an awareness notice | 16:26 |
| rosmaita | but also, i'd think we'd want a spec outlining the proposed changes in os-brick? | 16:26 |
| rosmaita | how do we usually handle that? | 16:26 |
| hemna_ | rosmaita:do we have a cinder spec? | 16:27 |
| jungleboyj | I don't think we have required a separate spec for os-brick in the past. | 16:27 |
| jungleboyj | I could be wrong though. | 16:27 |
| rosmaita | hemna_: we have just the general spec | 16:27 |
| rosmaita | i just want to make sure everyone is on board with using os-brick for this | 16:28 |
| jungleboyj | Yes, we did a lot of discussion around this and everyone reached agreement on os-brick rather than trying to create some other library. | 16:29 |
| smcginnis | It would be good if the spec detailed what changes are needed in os-brick to support this since cinder and glance are going to both dependent on those changes. | 16:29 |
| rosmaita | i've been attending the encryption team pop-up meeting and can put it on the agenda | 16:30 |
| rosmaita | can ask for a revision to the spec | 16:30 |
| rosmaita | #link http://eavesdrop.openstack.org/#Image_Encryption_Popup-Team_Meeting | 16:30 |
| jungleboyj | rosmaita: ++ | 16:30 |
| jungleboyj | Sounds like a good way to proceed. | 16:30 |
| rosmaita | ok, that's all from me, then | 16:30 |
| jungleboyj | rosmaita: Thank you! | 16:31 |
| jungleboyj | #topic py3.7 review underway | 16:31 |
| *** openstack changes topic to "py3.7 review underway (Meeting topic: Cinder)" | 16:31 | |
| jungleboyj | So, for awareness I have started the py37 review: | 16:31 |
| jungleboyj | #link https://etherpad.openstack.org/p/cinder-py3-ci-review | 16:32 |
| *** zbitter has joined #openstack-meeting | 16:32 | |
| jungleboyj | Since I am going through all the CIs I am taking general notes in the etherpad as well. | 16:32 |
| *** zaneb has quit IRC | 16:32 | |
| jungleboyj | Have a fear that the results of my investigation will become a long mid-cycle discussion. | 16:32 |
| rosmaita | good thing we have an otherwise light agenda | 16:33 |
| jungleboyj | Regardless, I am going to keep working through the review and will present the results here next week before starting to push up 'unsupported' patches. | 16:33 |
| jungleboyj | rosmaita: Yeah. For better or worse. | 16:33 |
| jungleboyj | Anyway, that was all I had on that. Was for awareness more than anything and you can watch the results develop in that etherpad. | 16:34 |
| smcginnis | Do we need to restate the motivation and importance of this effort for anyone? | 16:34 |
| *** ociuhandu has joined #openstack-meeting | 16:35 | |
| eharney | i'm pretty sure a few drivers have dependencies that don't state py3.7 support, that might be something to scan while looking through the drivers | 16:35 |
| hemna_ | rosmaita:brick seems like the right place for shared code for stuff related to volumes | 16:35 |
| jungleboyj | eharney: Guidance on how to stop that if they are running py37? | 16:35 |
| eharney | jungleboyj: not sure what you mean | 16:36 |
| jungleboyj | Not sure what you are saying maybe. | 16:36 |
| *** igordc has joined #openstack-meeting | 16:36 | |
| jungleboyj | Do you mean there are a few drivers that don't support 3.7 because of dependencies that aren't ready? | 16:36 |
| eharney | i mean we can probably identify some drivers that are likely to have issues with this, if we want to | 16:37 |
| hemna_ | if we identify drivers that can't py3.7, we can just mark them unsupported for the release and let them fix their stuffs | 16:37 |
| jungleboyj | Ok. If you know of ones that are going to have issues putting them into the etherpad would be helpful. | 16:37 |
| *** lpetrut has quit IRC | 16:37 | |
| jungleboyj | hemna_: Based on the stance we have made, that would seem to be waht we need to do. | 16:38 |
| *** jamesmcarthur has joined #openstack-meeting | 16:38 | |
| jungleboyj | Anything else on that discussion. | 16:38 |
| jungleboyj | #topic open discussion | 16:39 |
| *** openstack changes topic to "open discussion (Meeting topic: Cinder)" | 16:39 | |
| *** panda has quit IRC | 16:39 | |
| jungleboyj | Any other topics for discussion today? | 16:40 |
| *** panda has joined #openstack-meeting | 16:41 | |
| jungleboyj | Going once. | 16:42 |
| jungleboyj | Going twice .. | 16:42 |
| jungleboyj | Ok. I guess that is all we have for today. | 16:43 |
| smcginnis | Thanks jungleboyj | 16:43 |
| whoami-rajat | jungleboyj: Thanks! | 16:43 |
| jungleboyj | Thanks everyone for joining. Talk to you all next week! | 16:43 |
| jungleboyj | #endmeeting | 16:44 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 16:44 | |
| openstack | Meeting ended Wed Jul 31 16:44:00 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 16:44 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-07-31-16.00.html | 16:44 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-07-31-16.00.txt | 16:44 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/cinder/2019/cinder.2019-07-31-16.00.log.html | 16:44 |
| zhengMa | bye | 16:44 |
| *** woojay has left #openstack-meeting | 16:44 | |
| *** rosmaita has left #openstack-meeting | 16:45 | |
| *** diablo_rojo has quit IRC | 16:46 | |
| *** diablo_rojo has joined #openstack-meeting | 16:46 | |
| *** takamatsu has joined #openstack-meeting | 16:47 | |
| *** trident has quit IRC | 16:48 | |
| *** zhengMa has quit IRC | 16:48 | |
| *** diablo_rojo has quit IRC | 16:48 | |
| *** lpetrut has joined #openstack-meeting | 16:48 | |
| *** tosky has left #openstack-meeting | 16:48 | |
| *** diablo_rojo has joined #openstack-meeting | 16:49 | |
| *** ricolin has quit IRC | 16:50 | |
| *** trident has joined #openstack-meeting | 16:51 | |
| *** ociuhandu has quit IRC | 16:52 | |
| *** jgriffith has quit IRC | 16:57 | |
| *** takamatsu has quit IRC | 16:59 | |
| *** andrewar_ has left #openstack-meeting | 16:59 | |
| *** jamesmcarthur has quit IRC | 17:00 | |
| *** lpetrut has quit IRC | 17:01 | |
| *** diablo_rojo_ has joined #openstack-meeting | 17:01 | |
| *** ekcs has joined #openstack-meeting | 17:04 | |
| *** diablo_rojo has quit IRC | 17:04 | |
| *** diablo_rojo_ has quit IRC | 17:10 | |
| *** diablo_rojo has joined #openstack-meeting | 17:11 | |
| *** e0ne has quit IRC | 17:18 | |
| *** ociuhandu has joined #openstack-meeting | 17:21 | |
| *** ociuhandu has quit IRC | 17:26 | |
| *** igordc has quit IRC | 17:32 | |
| *** imsurit has quit IRC | 17:40 | |
| *** yamahata has joined #openstack-meeting | 17:44 | |
| *** iyamahat has joined #openstack-meeting | 17:45 | |
| *** ralonsoh has quit IRC | 17:57 | |
| *** tesseract has quit IRC | 18:05 | |
| *** radeks has quit IRC | 18:11 | |
| *** altlogbot_0 has quit IRC | 18:19 | |
| *** altlogbot_3 has joined #openstack-meeting | 18:20 | |
| *** radeks has joined #openstack-meeting | 18:22 | |
| *** bbowen has joined #openstack-meeting | 18:25 | |
| *** njohnston has quit IRC | 18:35 | |
| *** igordc has joined #openstack-meeting | 18:36 | |
| *** e0ne has joined #openstack-meeting | 18:37 | |
| *** takamatsu has joined #openstack-meeting | 18:43 | |
| *** sfernand has quit IRC | 18:53 | |
| *** radeks_ has joined #openstack-meeting | 18:56 | |
| *** radeks has quit IRC | 18:58 | |
| *** thgcorrea has quit IRC | 19:01 | |
| *** e0ne has quit IRC | 19:02 | |
| *** whoami-rajat has quit IRC | 19:08 | |
| *** e0ne has joined #openstack-meeting | 19:12 | |
| SotK | #startmeeting storyboard | 19:13 |
| openstack | Meeting started Wed Jul 31 19:13:53 2019 UTC and is due to finish in 60 minutes. The chair is SotK. Information about MeetBot at http://wiki.debian.org/MeetBot. | 19:13 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 19:13 |
| *** openstack changes topic to " (Meeting topic: storyboard)" | 19:13 | |
| openstack | The meeting name has been set to 'storyboard' | 19:13 |
| SotK | #link https://wiki.openstack.org/wiki/Meetings/StoryBoard Agenda | 19:14 |
| SotK | or something approximating one anyway | 19:14 |
| SotK | I don't think there's much to discuss this week, so this could be short | 19:14 |
| fungi | sounds good | 19:15 |
| fungi | thanks! | 19:15 |
| SotK | #topic Migration Updates | 19:15 |
| *** openstack changes topic to "Migration Updates (Meeting topic: storyboard)" | 19:15 | |
| SotK | #link https://storyboard.openstack.org/#!/board/45 Progress Board | 19:15 |
| SotK | the status/progress board is up to date now I believe | 19:16 |
| SotK | I don't know that there are any other updates here though | 19:17 |
| fungi | yeah, i'm unaware of any | 19:18 |
| SotK | #topic PTG + Forum | 19:18 |
| *** openstack changes topic to "PTG + Forum (Meeting topic: storyboard)" | 19:18 | |
| SotK | we should start thinking about what we want to do at the PTG + Forum | 19:19 |
| SotK | last PTG there was some discussion of having onboarding sessions for contributors and users | 19:19 |
| SotK | I don't think I'll be there but I'm happy to call in like we've done in the past assuming that its possible and at a reasonable time | 19:21 |
| fungi | yes, do we have a brainstorming etherpad chosen yet? | 19:21 |
| fungi | i don't know how reasonable shanghai times will be for you, and also am somewhat skeptical that we'd manage decent voice quality from behind the great firewall, but it's worth a try | 19:22 |
| SotK | it'd need to be mid-afternoon in Shanghai time at the earliest really I think | 19:23 |
| SotK | I don't think we have an etherpad yet, I'll try to remember to make one in the near future unless someone beats me to it | 19:23 |
| fungi | i can just make one now | 19:24 |
| *** e0ne has quit IRC | 19:24 | |
| fungi | #link https://etherpad.openstack.org/p/storyboard-shanghai-ptg-planning | 19:25 |
| fungi | presto! | 19:25 |
| SotK | thanks :) | 19:25 |
| *** njohnston has joined #openstack-meeting | 19:25 | |
| SotK | #topic In Progress Work | 19:26 |
| *** openstack changes topic to "In Progress Work (Meeting topic: storyboard)" | 19:26 | |
| SotK | #link https://storyboard.openstack.org/#!/board/1 Board for tracking active development | 19:27 |
| SotK | we updated and populated that board the other week, to give a rough overview of what's currently being worked on and what our priorities are | 19:27 |
| SotK | #link https://storyboard.openstack.org/#!/board/115 Board for notable lists | 19:28 |
| *** njohnston has left #openstack-meeting | 19:28 | |
| SotK | we also made that board which contains stories sorted by some interesting tags, to get a view of the triaged work we have available | 19:29 |
| SotK | I've still not got round to doing the reviews I keep promising, I'll continue to try to get to that in the next couple of days | 19:30 |
| *** bbowen has quit IRC | 19:31 | |
| SotK | similarly the attachments work is still waiting on me finding some free time and energy, hopefully I should get to that in the next couple of weeks | 19:31 |
| SotK | anything else to mention here? | 19:32 |
| *** radeks_ has quit IRC | 19:32 | |
| fungi | i think that about covers it | 19:33 |
| fungi | i was hoping to run some more slow query analysis this week | 19:34 |
| fungi | but haven't found time yet | 19:34 |
| fungi | the week is still somewhat young, at least | 19:34 |
| diablo_rojo_phon | Sorry I'm late. | 19:34 |
| diablo_rojo_phon | Went for a run that lasted longer than it should have. | 19:34 |
| diablo_rojo_phon | Sorry! | 19:34 |
| fungi | we can take a few to let you catch up, though it's been brief | 19:34 |
| SotK | heh, np :) | 19:34 |
| diablo_rojo_phon | Coolio | 19:34 |
| * diablo_rojo_phon reada | 19:35 | |
| diablo_rojo_phon | Reads | 19:35 |
| *** liuyulong has quit IRC | 19:35 | |
| diablo_rojo_phon | Caught up. | 19:36 |
| diablo_rojo_phon | I don't think I have anything to add to what you said aside from encouraging everyone to think of how we want to do onboarding. At the ptg? Or trying to do a forum session? | 19:37 |
| SotK | maybe contributors at the ptg and users at the forum? idk | 19:38 |
| diablo_rojo_phon | That would make a lot of sense lol | 19:39 |
| diablo_rojo_phon | I can probably draft a forum proposal for the user onboarding in the etherpad | 19:39 |
| SotK | sounds good | 19:40 |
| *** kopecmartin is now known as kopecmartin|off | 19:41 | |
| diablo_rojo_phon | As for ptg..I can put in a request for space. I know the person with the request survey ;) | 19:41 |
| fungi | do we think there would be more use for forum interaction than ptg this time? | 19:42 |
| fungi | or vice versa? | 19:42 |
| diablo_rojo_phon | I don't think I'd have any reason for ptg things aside from onboarding contributors | 19:43 |
| * SotK agrees | 19:45 | |
| fungi | yeah, if we don't expect to have critical mass of folks on hand in shanghai to hack on sb development or documenting or something then that makes sense | 19:46 |
| *** takamatsu has quit IRC | 19:47 | |
| *** efried_rollin is now known as efried | 19:47 | |
| *** eharney has quit IRC | 19:48 | |
| fungi | corvus: tarballs? | 19:49 |
| fungi | er, wrong channel | 19:49 |
| diablo_rojo_phon | Lol | 19:49 |
| corvus | tarballs! | 19:49 |
| diablo_rojo_phon | tarballs! | 19:49 |
| fungi | everybody loves a tarball | 19:49 |
| SotK | tarballs! | 19:49 |
| diablo_rojo_phon | SotK: there is travel support..if codethink will give you the time.. | 19:50 |
| diablo_rojo_phon | It's open through the 8th. | 19:50 |
| fungi | we can even provide tarballs | 19:50 |
| SotK | its a long way to travel though, and I don't have sufficient holidays spare really | 19:52 |
| * mordred hands SotK a tarball | 19:53 | |
| SotK | :D | 19:54 |
| SotK | actually, speaking of tarballs, we should fix http://tarballs.openstack.org/storyboard-webclient/ | 19:54 |
| fungi | we probably ought to move to tarballs.opendev.org anyway (which was the actual context of the contextless "tarballs!" interjection earlier) | 19:55 |
| *** e0ne has joined #openstack-meeting | 19:55 | |
| SotK | yeah, that sounds like a good idea | 19:56 |
| fungi | but yes, i thought i'd cleaned that tree up once already | 19:56 |
| fungi | maybe we're continuing to upload unpacked content there? | 19:56 |
| fungi | i'll double-check dates | 19:56 |
| diablo_rojo_phon | SotK: yeah definitely don't take vacation for it, but if they would let you work from Shanghai you might be able to get TSP. | 19:57 |
| fungi | -rwxrwxr-x 1 jenkins jenkins 1224 Mar 15 19:30 /srv/static/tarballs/storyboard-webclient/index.html | 19:57 |
| fungi | we've merged content as recently as june for that repo, so it's no longer happening (i think) | 19:58 |
| fungi | will clean it up now | 19:58 |
| * SotK doesn't think that would be likely | 19:59 | |
| SotK | Thanks fungi | 19:59 |
| fungi | cleaned up | 19:59 |
| SotK | alright, we're out of time, thanks for coming folks | 20:00 |
| SotK | #endmeeting | 20:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 20:00 | |
| openstack | Meeting ended Wed Jul 31 20:00:20 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 20:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-07-31-19.13.html | 20:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-07-31-19.13.txt | 20:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/storyboard/2019/storyboard.2019-07-31-19.13.log.html | 20:00 |
| diablo_rojo_phon | Thanks SotK! | 20:00 |
| fungi | thanks SotK! | 20:01 |
| *** e0ne has quit IRC | 20:12 | |
| *** eharney has joined #openstack-meeting | 20:35 | |
| *** AlanClark has quit IRC | 20:35 | |
| *** witek has joined #openstack-meeting | 20:36 | |
| *** trident has quit IRC | 20:44 | |
| *** mriedem has quit IRC | 20:44 | |
| *** mriedem has joined #openstack-meeting | 20:45 | |
| *** trident has joined #openstack-meeting | 20:46 | |
| *** mriedem has quit IRC | 20:54 | |
| *** witek has quit IRC | 20:55 | |
| *** mriedem has joined #openstack-meeting | 20:55 | |
| *** bobh has joined #openstack-meeting | 21:00 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Jul 31 21:00:13 2019 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| *** tdasilva has joined #openstack-meeting | 21:00 | |
| timburke | who's here for the swift meeting? | 21:00 |
| kota_ | o/ | 21:00 |
| mattoliverau | o/ | 21:00 |
| tdasilva | o/ | 21:00 |
| *** patchbot has joined #openstack-meeting | 21:01 | |
| rledisez | o/ | 21:01 |
| clayg | ohai | 21:01 |
| timburke | agenda hasn't changed too much | 21:01 |
| timburke | #link https://wiki.openstack.org/wiki/Meetings/Swift | 21:01 |
| timburke | #topic 404s from handoffs | 21:02 |
| *** openstack changes topic to "404s from handoffs (Meeting topic: swift)" | 21:02 | |
| timburke | i mentioned https://review.opendev.org/#/c/672186/ last week and asked that people start to think about it and form some opinions | 21:02 |
| patchbot | patch 672186 - swift - Ignore 404s from handoffs for objects when calcula... - 7 patch sets | 21:02 |
| timburke | rledisez helpfully suggested that i make the new behavior configurable | 21:03 |
| rledisez | *for few versions, to help transition | 21:03 |
| timburke | ...but as i tried to write up an explanation for why you'd enable the new option... i'm having some doubts... | 21:03 |
| timburke | regardless, i think we've got a bit of time -- clayg pointed out that it really ought to have some unit tests to cover the part of the change that only affects replicated | 21:04 |
| clayg | oh dear, what did I do :'( | 21:05 |
| timburke | but i'll try to get that done sooner rather than later | 21:05 |
| *** bobh has quit IRC | 21:05 | |
| clayg | oh, right - I don't like that knob at all | 21:05 |
| clayg | I don't see any comments on the review that explain the change, so I'm guessing there was a discussion in irc that I missed? | 21:06 |
| timburke | rledisez, what do we really expect operators to *do* with the option? ok, we upgraded -- crap, there's a bunch more 503s... twiddle the knob and make them 404s? | 21:06 |
| clayg | rledisez: am I correct that the primary concern is some graph that tracks 5XX errors going up under load/failure/rebalance? | 21:06 |
| rledisez | I think it should basically say "the new behavior might breaks your clients because the error code changed. to keep the old behavior while you upgrade your clients, set foo = false. the old behavior will be removed in version 2.Y" | 21:07 |
| rledisez | the point is not about the metrics. i just won't get my bonus this year ;) | 21:07 |
| timburke | what remediation would allow them to turn it back *off* | 21:07 |
| clayg | but I want rledisez to get his bonus! he buys me beers sometimes! | 21:08 |
| clayg | I don't think a client that was "correctly" handling a 404 with a retry would be surprised by needing to retry a 503 | 21:08 |
| clayg | maybe there's a more subtle status code change I'm not considering with proper reference - "might break clients" is a good reason *not* to change it all, but I'm not sure I understand the risk? | 21:09 |
| timburke | yeah... idk -- clients have to be able to handle 5xx, even now... | 21:09 |
| rledisez | I know some internal users that retry on 404 because I taught them about eventual consistency. I'm not sure what will happen on 503 | 21:09 |
| clayg | 🤔 | 21:10 |
| rledisez | (not saying they are right to retry, but that's an other discussion) | 21:10 |
| timburke | rledisez, what sdk do they use? or is it pretty hand-rolled? | 21:10 |
| rledisez | mostly homemade for what I know | 21:11 |
| rledisez | languages are Go, Perl and Java | 21:11 |
| timburke | yeah... that does make it tricky... | 21:11 |
| notmyname | rledisez: but I think we all tought the clients incorrectly! a 404 should only maybe retry. most of the time, 404 should be the right answer if it's what's given. swift was doing the wrong thing by giving 404 instead of 503 | 21:12 |
| timburke | kota_, mattoliverau: have you had a chance to think about the issue? | 21:13 |
| rledisez | notmyname: i totally agree with that | 21:13 |
| notmyname | but if they have been taught to retry on 404 (and we always said retry on 5xx), and we change 404->503, aren't they even more likely to retry if there's a failure? | 21:13 |
| kota_ | hmm... interesting | 21:13 |
| notmyname | so even if clients today have been retrying on 404, they will still work after the 404->503 change (because they better already be handling 5xx anyway) | 21:13 |
| mattoliverau | I won't lie, I forgot to look | 21:13 |
| clayg | well, we certainly could have been more specific when we want to say "i couldn't find your object, but also might be relevant I couldn't talk to anyone that might have been authoratative" | 21:13 |
| timburke | fwiw, i'd written up https://bugs.launchpad.net/swift/+bug/1837819 to try to describe the issue | 21:13 |
| openstack | Launchpad bug 1837819 in OpenStack Object Storage (swift) "Overloaded object primaries cause 404s on GET" [Medium,New] - Assigned to Tim Burke (1-tim-z) | 21:13 |
| clayg | we want the client to retry, and a 5XX to me is a more RESTful way to indicate to *most* clients what they should do next... | 21:14 |
| rledisez | as I said, I'm don't think they should retry, but the most common I case I see is people doing an upload, and checking it's there right after (even if they got a 2xx). I tried to explain "trust swift", but no, they don't trust | 21:14 |
| kota_ | IMHO, we could retry on both cases. if the users knows swift error statement absolutely retry with 503, not sure on 404. | 21:15 |
| timburke | rledisez, we recently had a customer doing the same thing, but looking for the object in *listings* 🤦 | 21:15 |
| notmyname | rledisez: yeah. unfortunately I haven't been able to figure out how to fix users yet ;-) | 21:15 |
| kota_ | imo, it's not 5xx, just 503 | 21:15 |
| kota_ | because we should not retry on 500 Internal Server Error, that wouldn't be fixed in the near future. | 21:16 |
| mattoliverau | The users surely just need a sleep(10) or setting :p | 21:16 |
| mattoliverau | *Something | 21:16 |
| rledisez | maybe i'm just too cautious, because I don't think to enable that flag in my clusters. i'll do a proper communication before upgrading. but i'm thinking of somebody who would want to rollback quickly | 21:17 |
| clayg | rledisez: yeah... if someone was already running on the edge and they didn't get enough communication before the upgrade they might be really confused/nervous about what the new status code is telling them | 21:17 |
| clayg | "everything was working FINE!!!" - yeah... no... it wasn't. | 21:17 |
| timburke | this is highlighting for me that this probably ought to have an UpgradeImpact regardless of whether we keep the config option | 21:18 |
| kota_ | sounds reasonable | 21:19 |
| rledisez | agree | 21:19 |
| clayg | honestly this change shouldn't even be as contreversal as p 667235 (in that case the proxy could really be doing an inline retry) | 21:20 |
| patchbot | https://review.opendev.org/#/c/667235/ - swift - Don't handle object without container (MERGED) - 1 patch set | 21:20 |
| clayg | well, I don't want the config option - but won't -2 it or anything w/o it - but it might be the ONLY CONFIG OPTION EVAR that I set a reminder to make sure we deprecate it in the next release 😉 | 21:21 |
| timburke | ...and *that* makes me wonder if maybe the guy that wrote the bug shouldn't have been the one clicking +A... | 21:21 |
| timburke | (well, insofar as that guy was *me*) | 21:21 |
| clayg | timburke: you're PTL you can +A whenver you want ;) | 21:22 |
| kota_ | lol | 21:22 |
| notmyname | lol, not it! | 21:23 |
| mattoliverau | Lol | 21:23 |
| timburke | we've got some precedent for adding known-terrible-idea config options: https://github.com/openstack/swift/commit/94bac4a | 21:24 |
| timburke | i'll keep thinking on it, but i'm kinda leaning toward clayg's position personally | 21:24 |
| mattoliverau | lol | 21:25 |
| timburke | let's keep moving | 21:26 |
| timburke | #topic py3 | 21:26 |
| *** openstack changes topic to "py3 (Meeting topic: swift)" | 21:26 | |
| kota_ | config opt_out/in might be terrible | 21:26 |
| timburke | not much to report | 21:26 |
| kota_ | i'm imagine s3acl... | 21:26 |
| mattoliverau | I need to look closer at it, and sorry I didn't. Like kota_ says, 503 should be retry. But its the contract changing that is the impact. And one of our dev and Ops wants an escape clause, for a release, I think I'm ok with that. | 21:26 |
| kota_ | imaging | 21:26 |
| mattoliverau | anyway, move on :) | 21:26 |
| kota_ | ok | 21:26 |
| timburke | ugh, yeah... s3_acl... | 21:27 |
| timburke | https://review.opendev.org/#/c/672610/ landed! hopefully zaitcev's cluster will be happier now :-) | 21:27 |
| patchbot | patch 672610 - swift - py3: fix non-ascii metadata handling in account-se... (MERGED) - 2 patch sets | 21:27 |
| mattoliverau | \o/ | 21:27 |
| timburke | as did https://review.opendev.org/#/c/672803/ -- there may be a bit of a long tail of patches like that :-/ | 21:28 |
| patchbot | patch 672803 - swift - py3: Fix title-casing in HeaderKeyDict (MERGED) - 3 patch sets | 21:28 |
| timburke | it'll be nice to get https://review.opendev.org/#/c/671333/ so i can run probe tests locally again :-P | 21:30 |
| patchbot | patch 671333 - swift - py3: (mostly) port probe tests - 2 patch sets | 21:30 |
| timburke | that's about it | 21:30 |
| timburke | #topic lots of small files | 21:30 |
| *** openstack changes topic to "lots of small files (Meeting topic: swift)" | 21:30 | |
| timburke | rledisez, kota_ i haven't seen too much lately on the branch (but that's ok) | 21:31 |
| kota_ | ya, sorry | 21:31 |
| timburke | like i said, that's ok! no need to apologize | 21:31 |
| clayg | tdasilva: is going to be all up in the losf pretty soon! | 21:31 |
| rledisez | yeah. alecuyer is off so don't expect a lot from OVH for few weeks | 21:31 |
| timburke | i know people here at swiftstack have been getting increasingly interested -- i think tdasilva has been taking a look recently? | 21:31 |
| *** zaitcev has joined #openstack-meeting | 21:32 | |
| kota_ | nice, tdasilva!!! | 21:32 |
| timburke | fwiw, i feel like this ought to be the next big item of work that we're all focused on | 21:32 |
| kota_ | +1 | 21:33 |
| *** tdasilva has quit IRC | 21:33 | |
| kota_ | oh, tdasilva has left. | 21:33 |
| clayg | +2 | 21:33 |
| timburke | bah. i was just about to ask him if there was anything he'd like to bring up about it... oh well | 21:34 |
| timburke | #topic sharding | 21:34 |
| *** openstack changes topic to "sharding (Meeting topic: swift)" | 21:34 | |
| timburke | mattoliverau, i know you've got a few patches up now -- anything we ought to be doing besides reviewing them? | 21:35 |
| mattoliverau | Nah just reviewing them and point out the obvious flaws and edge cases :) | 21:35 |
| timburke | 👍 | 21:35 |
| timburke | #topic symlinks and versioning | 21:36 |
| *** openstack changes topic to "symlinks and versioning (Meeting topic: swift)" | 21:36 | |
| mattoliverau | I last patch is a complete POC were we send the ranges from the scanner via UPDATE to do a reverse rollback stratergy. Not sure I like it, but was an idea I had. | 21:36 |
| mattoliverau | that's all | 21:36 |
| timburke | thanks | 21:36 |
| clayg | on the hardlinks I was looking at some comments this morning, and adding some more tests for behaviors that we want to better specify - I think we're still not 100% clear on how hardlinks to manifests/symlinks should look | 21:37 |
| clayg | I think timburke had the most experience/insight - so i'm hoping to get his feedback on some of the new tests I drafted | 21:37 |
| *** e0ne has joined #openstack-meeting | 21:37 | |
| timburke | i'll be sure to take a look :-) | 21:38 |
| timburke | any other blockers for you, or places that you need more input? | 21:38 |
| *** zaitcev has quit IRC | 21:39 | |
| clayg | i also managed to get up the s3 versioning patch at the end of the chain, p 673682 | 21:39 |
| patchbot | https://review.opendev.org/#/c/673682/ - swift - s3api: Implement versioning status API - 1 patch set | 21:39 |
| clayg | one thing that starts to shine through on that one is how much it just assumes it knows how versioning works and does what it needs to implement the aws api | 21:39 |
| clayg | I think in a more perfect world we'd have looked at the s3 versioning features and added them to versioned writes - then s3api is just doing *translation* from aws apis to swift apis | 21:40 |
| *** takamatsu has joined #openstack-meeting | 21:40 | |
| kota_ | +1 | 21:40 |
| clayg | but since we don't have spellings for ... e.g. "copy version_id XYZ" we just "do it" in s3api | 21:40 |
| clayg | but I don't think moving forward with something that works really prevents us in any meaningful way from doing that work later (except that maybe we'd have less motivation to do so) | 21:41 |
| clayg | OTOH, I'm not sure much moving to symlink versionsing is really going to throw off clients that sort of had to learn how stack & history versoining worked already so they could do things like "restore version X" or "delete version Y" | 21:42 |
| clayg | had we had an API for that all along it would make less of a difference to clients when we decide to change the underlying implementation | 21:43 |
| timburke | makes sense | 21:43 |
| clayg | anyway, is what is... something I'll be a little more aware of as we flesh more of the s3api matrix down the road... | 21:44 |
| clayg | I don't think i'm blocked right now and I feel like i'm making progress | 21:44 |
| clayg | all feedback is appreciated! | 21:44 |
| timburke | 👍 | 21:45 |
| timburke | #topic shanghai | 21:45 |
| *** openstack changes topic to "shanghai (Meeting topic: swift)" | 21:45 | |
| timburke | i know i dropped it from the agenda, but i've been looking at the order in which i'll have to do things and i thought i ought to share | 21:45 |
| timburke | keeping in mind that it's a bit US-centric, but may be more-or-less applicable for you, too mattoliverau, kota_, and rledisez | 21:46 |
| kota_ | something like VISA? | 21:46 |
| mattoliverau | its probably pretty similar I suspect. Ie get a letter, get visa, etc. | 21:47 |
| timburke | looks like to get the visa (http://www.china-embassy.org/eng/visas/hrsq/#M, $140) i need the invitation letter (https://openstackfoundation.formstack.com/forms/visa_form_shanghai_summit), and to get *that* i need to register (https://app.eventxtra.link/registrations/6640a923-98d7-44c7-a623-1e2c9132b402, $161 after the contributor discount) | 21:49 |
| timburke | as far as i can tell, airfare and hotel can be done at any point along there | 21:50 |
| *** zaitcev has joined #openstack-meeting | 21:50 | |
| kota_ | that registration with early bird will close around... 14 (maybe?) Aug. | 21:50 |
| rledisez | kota_: I was looking for that information earlier. do you have a link? | 21:50 |
| kota_ | I'm not sure the contribution discount will keep the $161 | 21:51 |
| timburke | https://www.openstack.org/summit/shanghai-2019/ says "Summit registration is open - get your tickets before prices increase on August 14 at 11:59pm PT! " | 21:51 |
| kota_ | https://www.openstack.org/summit/shanghai-2019/ | 21:51 |
| kota_ | same URL | 21:51 |
| kota_ | #link https://www.openstack.org/summit/shanghai-2019/ | 21:51 |
| rledisez | perfect. thx. how did i miss that :) | 21:51 |
| kota_ | no information about the standard price after early bird. | 21:51 |
| timburke | that's about it. i did send something to the mailing list to call out our etherpad (http://lists.openstack.org/pipermail/openstack-discuss/2019-July/008156.html) -- it was a bit behind when most projects did theirs | 21:53 |
| kota_ | 你好! | 21:54 |
| timburke | we'll see who else puts their name on https://etherpad.openstack.org/p/swift-ptg-shanghai :-D | 21:54 |
| timburke | that's all i've got | 21:54 |
| timburke | #topic open discussion | 21:54 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:55 | |
| clayg | i gotta bounce, ya'll be good | 21:55 |
| timburke | anyone have anything eles to bring up in the last five minutes? | 21:55 |
| timburke | all right. thank you all for coming, and thank you for working on swift! | 21:57 |
| timburke | #endmeeting | 21:57 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:57 | |
| openstack | Meeting ended Wed Jul 31 21:57:08 2019 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:57 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-07-31-21.00.html | 21:57 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-07-31-21.00.txt | 21:57 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2019/swift.2019-07-31-21.00.log.html | 21:57 |
| *** njohnston has joined #openstack-meeting | 21:59 | |
| *** raildo has quit IRC | 22:00 | |
| *** zaitcev has left #openstack-meeting | 22:02 | |
| *** slaweq has quit IRC | 22:04 | |
| *** rcernin has joined #openstack-meeting | 22:05 | |
| *** carloss has quit IRC | 22:19 | |
| *** patchbot has quit IRC | 22:36 | |
| *** mriedem has quit IRC | 22:43 | |
| *** panda has quit IRC | 23:03 | |
| *** panda has joined #openstack-meeting | 23:04 | |
| *** notmyname has quit IRC | 23:05 | |
| *** njohnston has quit IRC | 23:09 | |
| *** slaweq has joined #openstack-meeting | 23:11 | |
| *** slaweq has quit IRC | 23:15 | |
| *** jamesmcarthur has joined #openstack-meeting | 23:22 | |
| *** e0ne has quit IRC | 23:33 | |
| *** eharney has quit IRC | 23:45 | |
| *** notmyname has joined #openstack-meeting | 23:52 | |
| *** notmyname has quit IRC | 23:53 | |
| *** notmyname has joined #openstack-meeting | 23:53 | |
| *** takamatsu has quit IRC | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!