| *** brinzhang has joined #openstack-meeting | 00:01 | |
| *** tetsuro has joined #openstack-meeting | 00:05 | |
| *** verdurin has quit IRC | 00:07 | |
| *** ykatabam has quit IRC | 00:10 | |
| *** ykatabam has joined #openstack-meeting | 00:10 | |
| *** verdurin has joined #openstack-meeting | 00:14 | |
| *** brinzhang_ has joined #openstack-meeting | 00:18 | |
| *** brinzhang has quit IRC | 00:21 | |
| *** jmasud has joined #openstack-meeting | 00:32 | |
| *** jmasud has quit IRC | 00:44 | |
| *** jmasud has joined #openstack-meeting | 00:55 | |
| *** Liang__ has joined #openstack-meeting | 01:01 | |
| *** gyee has quit IRC | 01:04 | |
| *** jmasud has quit IRC | 01:15 | |
| *** yasufum has joined #openstack-meeting | 01:24 | |
| *** rh-jelabarre has quit IRC | 01:28 | |
| *** jmasud has joined #openstack-meeting | 01:39 | |
| *** masahito has joined #openstack-meeting | 01:42 | |
| *** masahito has quit IRC | 01:47 | |
| *** sfernand has quit IRC | 01:48 | |
| *** masahito has joined #openstack-meeting | 02:01 | |
| *** ociuhandu has joined #openstack-meeting | 02:08 | |
| *** apetrich has quit IRC | 02:09 | |
| *** tinwood has quit IRC | 02:10 | |
| *** tinwood has joined #openstack-meeting | 02:12 | |
| *** ociuhandu has quit IRC | 02:12 | |
| *** jmasud has quit IRC | 02:16 | |
| *** viks____ has joined #openstack-meeting | 02:17 | |
| *** jmasud has joined #openstack-meeting | 02:19 | |
| *** ricolin has joined #openstack-meeting | 02:33 | |
| *** jmasud has quit IRC | 02:40 | |
| *** tetsuro has quit IRC | 03:02 | |
| *** masahito has quit IRC | 03:15 | |
| *** tetsuro has joined #openstack-meeting | 03:18 | |
| *** tetsuro has quit IRC | 03:19 | |
| *** psachin has joined #openstack-meeting | 03:31 | |
| *** dmacpher_ has quit IRC | 03:56 | |
| *** tetsuro has joined #openstack-meeting | 03:57 | |
| *** jmasud has joined #openstack-meeting | 04:00 | |
| *** masahito has joined #openstack-meeting | 04:01 | |
| *** ykatabam has quit IRC | 04:13 | |
| *** ykatabam has joined #openstack-meeting | 04:15 | |
| *** yamamoto has quit IRC | 04:18 | |
| *** kevinz has quit IRC | 04:24 | |
| *** jmasud has quit IRC | 04:35 | |
| *** yamamoto has joined #openstack-meeting | 04:37 | |
| *** jmasud has joined #openstack-meeting | 04:42 | |
| *** ociuhandu has joined #openstack-meeting | 04:50 | |
| *** ociuhandu has quit IRC | 04:55 | |
| *** jmasud has quit IRC | 05:02 | |
| *** rfolco has quit IRC | 05:09 | |
| *** bnemec has quit IRC | 05:22 | |
| *** rcernin has quit IRC | 05:24 | |
| *** jmasud has joined #openstack-meeting | 05:32 | |
| *** ysandeep|away is now known as ysandeep | 05:34 | |
| *** masahito has quit IRC | 05:43 | |
| *** ricolin has quit IRC | 05:51 | |
| *** ricolin has joined #openstack-meeting | 05:53 | |
| *** ttsiouts has joined #openstack-meeting | 05:56 | |
| *** ociuhandu has joined #openstack-meeting | 05:59 | |
| *** rcernin has joined #openstack-meeting | 06:09 | |
| *** rcernin has quit IRC | 06:10 | |
| *** belmoreira has joined #openstack-meeting | 06:10 | |
| *** rcernin has joined #openstack-meeting | 06:11 | |
| *** belmoreira has quit IRC | 06:12 | |
| *** jmasud has quit IRC | 06:17 | |
| *** jmasud has joined #openstack-meeting | 06:19 | |
| *** tetsuro has quit IRC | 06:21 | |
| *** belmoreira has joined #openstack-meeting | 06:23 | |
| *** rcernin has quit IRC | 06:25 | |
| *** rcernin has joined #openstack-meeting | 06:26 | |
| *** psachin has quit IRC | 06:26 | |
| *** tetsuro has joined #openstack-meeting | 06:37 | |
| *** psachin has joined #openstack-meeting | 06:44 | |
| *** dklyle has quit IRC | 06:50 | |
| *** slaweq has joined #openstack-meeting | 06:58 | |
| *** slaweq has quit IRC | 07:02 | |
| *** tetsuro has quit IRC | 07:03 | |
| *** apetrich has joined #openstack-meeting | 07:04 | |
| *** slaweq has joined #openstack-meeting | 07:05 | |
| *** ricolin has quit IRC | 07:06 | |
| *** maciejjozefczyk has joined #openstack-meeting | 07:08 | |
| *** tetsuro has joined #openstack-meeting | 07:11 | |
| *** tetsuro has quit IRC | 07:11 | |
| *** rcernin has quit IRC | 07:13 | |
| *** ricolin has joined #openstack-meeting | 07:13 | |
| *** tetsuro has joined #openstack-meeting | 07:20 | |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| *** ralonsoh has joined #openstack-meeting | 07:25 | |
| *** jmasud has quit IRC | 07:42 | |
| *** vishalmanchanda has joined #openstack-meeting | 07:54 | |
| *** ociuhandu has quit IRC | 07:57 | |
| *** ociuhandu has joined #openstack-meeting | 08:00 | |
| *** oneswig has joined #openstack-meeting | 08:00 | |
| *** links has joined #openstack-meeting | 08:01 | |
| ttx | jiaopengju1: large scale SIG meeting in #openstack-meeting-3 if interested | 08:02 |
|---|---|---|
| *** tetsuro_ has joined #openstack-meeting | 08:08 | |
| *** tetsuro has quit IRC | 08:11 | |
| *** tetsuro_ has quit IRC | 08:27 | |
| *** masahito has joined #openstack-meeting | 08:41 | |
| *** ysandeep is now known as ysandeep|lunch | 08:46 | |
| *** ociuhandu has quit IRC | 08:55 | |
| *** e0ne has joined #openstack-meeting | 08:56 | |
| *** ociuhandu has joined #openstack-meeting | 08:57 | |
| *** witek has joined #openstack-meeting | 09:02 | |
| *** ociuhandu has quit IRC | 09:02 | |
| *** witek has left #openstack-meeting | 09:02 | |
| *** ysandeep|lunch is now known as ysandeep | 09:06 | |
| *** yasufum has quit IRC | 09:10 | |
| *** ricolin has quit IRC | 09:22 | |
| *** ricolin has joined #openstack-meeting | 09:23 | |
| *** ricolin has quit IRC | 09:24 | |
| *** ricolin has joined #openstack-meeting | 09:25 | |
| *** ricolin has quit IRC | 09:25 | |
| *** ricolin_ has joined #openstack-meeting | 09:25 | |
| *** manuvakery has quit IRC | 09:32 | |
| *** e0ne has quit IRC | 09:34 | |
| *** e0ne has joined #openstack-meeting | 09:35 | |
| *** ricolin_ is now known as ricolin | 09:36 | |
| *** hyunsikyang__ has joined #openstack-meeting | 09:46 | |
| *** ociuhandu has joined #openstack-meeting | 09:48 | |
| *** hyunsikyang has quit IRC | 09:49 | |
| *** ociuhandu has quit IRC | 09:51 | |
| *** ociuhandu has joined #openstack-meeting | 09:52 | |
| *** yamamoto has quit IRC | 09:57 | |
| *** JangwonLee__ has joined #openstack-meeting | 10:04 | |
| *** JangwonLee has joined #openstack-meeting | 10:06 | |
| *** JangwonLee_ has quit IRC | 10:07 | |
| *** JangwonLee_ has joined #openstack-meeting | 10:08 | |
| *** JangwonLee__ has quit IRC | 10:09 | |
| *** JangwonLee has quit IRC | 10:12 | |
| *** hemna has quit IRC | 10:18 | |
| *** hemna has joined #openstack-meeting | 10:19 | |
| *** Liang__ has quit IRC | 10:21 | |
| *** takamatsu has joined #openstack-meeting | 10:28 | |
| *** masahito has quit IRC | 10:28 | |
| *** yamamoto has joined #openstack-meeting | 10:34 | |
| *** priteau has joined #openstack-meeting | 10:39 | |
| *** rpittau is now known as rpittau|bbl | 10:42 | |
| *** yamamoto has quit IRC | 10:44 | |
| *** yamamoto has joined #openstack-meeting | 10:59 | |
| *** JangwonLee__ has joined #openstack-meeting | 10:59 | |
| *** witek has joined #openstack-meeting | 11:00 | |
| oneswig | #startmeeting scientific-sig | 11:00 |
| openstack | Meeting started Wed Apr 22 11:00:34 2020 UTC and is due to finish in 60 minutes. The chair is oneswig. Information about MeetBot at http://wiki.debian.org/MeetBot. | 11:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 11:00 |
| *** openstack changes topic to " (Meeting topic: scientific-sig)" | 11:00 | |
| openstack | The meeting name has been set to 'scientific_sig' | 11:00 |
| oneswig | hello | 11:01 |
| witek | hi | 11:01 |
| oneswig | Hi witek, how are you? | 11:02 |
| oneswig | #link agenda for today https://wiki.openstack.org/wiki/Scientific_SIG#IRC_Meeting_April_22nd_2020 | 11:02 |
| oneswig | small agenda. Must try harder :-) | 11:02 |
| *** JangwonLee_ has quit IRC | 11:02 | |
| *** yamamoto has quit IRC | 11:03 | |
| *** janders has joined #openstack-meeting | 11:03 | |
| janders | g'day | 11:04 |
| oneswig | #topic Sig at virtual PTG | 11:04 |
| *** openstack changes topic to "Sig at virtual PTG (Meeting topic: scientific-sig)" | 11:04 | |
| witek | I'm fine, thanks | 11:04 |
| oneswig | Hi janders, evening | 11:04 |
| oneswig | janders: had a question for you about how you were using vxlan - on the IB NIC was that right? | 11:05 |
| janders | I've done this some time back | 11:05 |
| janders | I used this method for the fully virtualised part of the cluster running on IB kit | 11:07 |
| oneswig | #link virtual PTG planning https://ethercalc.openstack.org/126u8ek25noy | 11:07 |
| janders | I wouldnt recommend vxlan over IB for large scale, performance sensitive cases | 11:09 |
| janders | but for small-medium it works pretty well | 11:09 |
| oneswig | I put in for 1500UTC-1700UTC and 2100UTC-2300UTC for virtual PTG, hopefully the split session will work for more people. | 11:11 |
| *** janders24 has joined #openstack-meeting | 11:11 | |
| janders | comms issues sorry | 11:12 |
| witek | the afternoon session should work for me | 11:12 |
| oneswig | apologies, had a call | 11:13 |
| oneswig | afternoon for me too, I'll see how much I've got left in the tank for the evening. | 11:14 |
| *** njohnston has quit IRC | 11:14 | |
| oneswig | We should think about how to advocate scientific sig use cases | 11:15 |
| oneswig | (and who to?) | 11:15 |
| oneswig | #topic AOB | 11:17 |
| *** openstack changes topic to "AOB (Meeting topic: scientific-sig)" | 11:17 | |
| *** dmacpher has joined #openstack-meeting | 11:17 | |
| oneswig | Just sitting in on a presentation on OpenStack changes for supporting CentOS8 | 11:18 |
| oneswig | plenty of changes to accommodate | 11:18 |
| *** njohnston has joined #openstack-meeting | 11:18 | |
| oneswig | Quite a big piece of work but CentOS 7 will rapidly go off, likely | 11:20 |
| janders | top #3 challenges? | 11:20 |
| janders | which release will support both el7 and el8? | 11:21 |
| oneswig | I joined late, but DNF, PFtables, Python 3, and network scripts being deprecated all have to be resolved. | 11:21 |
| oneswig | In the Kolla world, Train spans both. | 11:21 |
| oneswig | Assume it's the same for TripleO | 11:22 |
| janders | how well is el7->el8 migration going for people? | 11:22 |
| janders | or is it irrelevant for OS? | 11:22 |
| oneswig | It's a reinstall | 11:22 |
| *** yamamoto has joined #openstack-meeting | 11:22 | |
| janders | wasn't that supposed to be an in-place? | 11:23 |
| oneswig | I'm not sure for RHEL but for CentOS it's reinstall | 11:23 |
| janders | I've done interesting el6->el7 migrations on Icehouse I think it was | 11:23 |
| janders | back in the day | 11:23 |
| janders | SRIOV cluster so no live migration | 11:24 |
| janders | but /var/lib/nova/instances as a separate LV worked a treat | 11:24 |
| oneswig | Right. I occasionally hear things about support for SRIOV live migration... | 11:25 |
| janders | in the "maybe one day" context or ready for testing context? | 11:25 |
| *** yamamoto has quit IRC | 11:25 | |
| *** yamamoto has joined #openstack-meeting | 11:25 | |
| oneswig | Trying to remember where I saw it recently. | 11:27 |
| oneswig | It's quiet today, anyone with AOB to add? | 11:33 |
| janders | playing around with tripleo updates on OSP13 | 11:33 |
| janders | seems updates are heaps better than installs :) | 11:34 |
| janders | mostly ansible | 11:34 |
| oneswig | Good to hear it, although usually it's the other way around | 11:34 |
| janders | as next versions come out, who knows, maybe tripleo will get usable | 11:34 |
| janders | promise is less puppet more ansible, should be all ansible soon | 11:34 |
| janders | another interesting find - CX6 drivers are in overcloud images, but not in IPA | 11:35 |
| oneswig | Seems like a good path and one that converges with OS-A and Kolla. | 11:35 |
| janders | we were rebuilding IPA with CX6 drivers this week | 11:35 |
| oneswig | From our team Doug did our first production deploy with the Ironic SW RAID driver | 11:35 |
| janders | nice! | 11:36 |
| janders | i may move to that before I get VROC working in IPA :P | 11:36 |
| oneswig | Those things are harder than they ought to be, I've heard. | 11:37 |
| janders | confirmed :/ | 11:37 |
| oneswig | The price paid for always having the shiny toys! | 11:37 |
| janders | indeed | 11:37 |
| oneswig | janders: are you still deploying supercloud nodes with only one NIC? | 11:38 |
| janders | for the cyber project im doing right now I have two | 11:39 |
| janders | 100/200 HDR for storage only | 11:39 |
| janders | and 100GE for everything else | 11:39 |
| janders | but this cluster is a little HPC and more security focused | 11:39 |
| *** ociuhandu has quit IRC | 11:39 | |
| oneswig | Are you using the m-key stuff on the IB, or is that outside of the user-facing world? | 11:41 |
| janders | m-key? | 11:42 |
| oneswig | There was a fix for bare metal multi-tenant IB security, I believe it related to preventing IB NICs from sending management datagrams. | 11:43 |
| janders | right! | 11:43 |
| janders | for older cards it was special locked-down firmware | 11:44 |
| janders | unsure for the latest ones - in this cluster, IB fabric is not user facing (at least for now) | 11:44 |
| oneswig | ok, makes sense. | 11:44 |
| janders | so it's just the standard FW | 11:44 |
| oneswig | Are your tenants untrusted in this system janders? | 11:45 |
| janders | im less worried about the tenants and more about the stuff they will be working on | 11:45 |
| janders | malware analysis | 11:45 |
| janders | so IB is mostly used to connect volumes with their data to hypervisors at decent performance | 11:45 |
| oneswig | NVMEoF? | 11:46 |
| *** ysandeep is now known as ysandeep|brb | 11:46 | |
| janders24 | GPFS | 11:46 |
| janders24 | though NVMEoF is a good idea actually :) | 11:47 |
| janders24 | I've gone with GPFS cause it can do everything (glance/cinder backend + parallel FS) | 11:47 |
| janders24 | but im starting to lean towards splitting openstack backends off the parallel FS | 11:47 |
| oneswig | What's the state of the OpenStack support? | 11:47 |
| janders24 | for OSP13 it *just works* | 11:48 |
| oneswig | Nice, good to hear it. | 11:48 |
| janders24 | for OSP16 it's uncertain, sometimes I'm thinking about forward-porting it myself :) | 11:48 |
| oneswig | Assuming you didn't mean it *just* works :-) | 11:48 |
| *** rh-jelabarre has joined #openstack-meeting | 11:48 | |
| janders24 | yeah... there is some fiddling to get it to work | 11:48 |
| janders24 | and then it *just works* :) | 11:49 |
| janders24 | containerisation isn't helping to make things simple | 11:49 |
| oneswig | how so? | 11:49 |
| janders | passing through extra subtrees into a number of containers via tripleo isnt trivial | 11:50 |
| *** rpittau|bbl is now known as rpittau | 11:50 | |
| *** evrardjp has joined #openstack-meeting | 11:52 | |
| *** raildo has joined #openstack-meeting | 11:53 | |
| oneswig | Something like /var/lib/nova? | 11:53 |
| janders | spot on | 11:53 |
| janders | and glance | 11:53 |
| janders | cinder is even more interesting cause its pacemaker powered in OSP | 11:54 |
| oneswig | Back in the day it was a huge problem upgrading a hypervisor or controller with OFED packages installed. How's that going now? | 11:55 |
| janders24 | hahaha | 11:55 |
| janders24 | it was killing us just this morning | 11:55 |
| janders24 | I fixed this some time back with elaborate repo management tactics | 11:55 |
| janders24 | but these days more and more stuff is in kernel | 11:55 |
| janders24 | so I usually get away with running without OFED | 11:56 |
| janders24 | i probably wouldnt do that on GPFS servers | 11:56 |
| janders24 | but GPFS clients are running fine | 11:56 |
| *** janders has quit IRC | 11:56 | |
| janders24 | and as mentioned earlier recent OSP13 overcloud images come with CX6 drivers included | 11:56 |
| oneswig | Interestingly, it does seem possible to upgrade in-place, but not easy: https://www.centlinux.com/2020/01/how-to-upgrade-centos-7-to-8-server.html | 11:56 |
| janders24 | IPA may need injection of bits of OFED but that doesn't need to be upgradable, it's rebuild-as-needed | 11:57 |
| oneswig | I remember benchmarking the iSER cinder driver, it was really good (but also a significant SPOF) | 11:57 |
| janders24 | tahts the beauty of GPFS | 11:58 |
| janders24 | no SPOFs | 11:58 |
| oneswig | yes, very nice. | 11:58 |
| janders24 | yeah with el7>el8 I was expecting to see Fedora-like in-place upgrade mechanism | 11:58 |
| janders24 | unsure to what degree its applicable to OpenStack (or how worthwhile this would be) | 11:59 |
| janders24 | likes of tripleo will probably rebuild node after node | 11:59 |
| janders24 | but having said that i havent explored that path so not sure... yet | 11:59 |
| janders24 | OSP13 has a fair bit of life left on it and works well | 11:59 |
| oneswig | Ah, we are out of time. | 12:00 |
| janders24 | thanks guys | 12:00 |
| oneswig | Thanks janders24 witek good talking with you both | 12:00 |
| janders24 | stay safe! | 12:00 |
| oneswig | and you | 12:00 |
| oneswig | #endmeeting | 12:00 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 12:00 | |
| openstack | Meeting ended Wed Apr 22 12:00:30 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 12:00 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-04-22-11.00.html | 12:00 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-04-22-11.00.txt | 12:00 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/scientific_sig/2020/scientific_sig.2020-04-22-11.00.log.html | 12:00 |
| janders24 | im going for walks via drone - and baking a lot of bread! :) | 12:00 |
| oneswig | walks via drone, whatever next! :-) | 12:01 |
| *** oneswig has quit IRC | 12:02 | |
| *** janders24 has quit IRC | 12:03 | |
| *** manuvakery has joined #openstack-meeting | 12:04 | |
| *** yamamoto has quit IRC | 12:15 | |
| *** andrebeltrami has joined #openstack-meeting | 12:16 | |
| *** ttsiouts_ has joined #openstack-meeting | 12:16 | |
| *** JangwonLee has joined #openstack-meeting | 12:19 | |
| *** ttsiouts has quit IRC | 12:20 | |
| *** rfolco has joined #openstack-meeting | 12:22 | |
| *** JangwonLee__ has quit IRC | 12:22 | |
| *** ociuhandu has joined #openstack-meeting | 12:23 | |
| *** JangwonLee_ has joined #openstack-meeting | 12:23 | |
| *** JangwonLee__ has joined #openstack-meeting | 12:24 | |
| *** JangwonLee has quit IRC | 12:27 | |
| *** JangwonLee_ has quit IRC | 12:28 | |
| *** ociuhandu has quit IRC | 12:29 | |
| *** ociuhandu has joined #openstack-meeting | 12:29 | |
| *** ysandeep|brb is now known as ysandeep | 12:41 | |
| *** ykatabam has quit IRC | 12:46 | |
| *** yamamoto has joined #openstack-meeting | 12:48 | |
| *** yamamoto has quit IRC | 12:56 | |
| *** psachin has quit IRC | 13:14 | |
| *** e0ne has quit IRC | 13:44 | |
| *** Liang__ has joined #openstack-meeting | 13:44 | |
| *** e0ne has joined #openstack-meeting | 13:44 | |
| *** moguimar has joined #openstack-meeting | 13:46 | |
| *** yamamoto has joined #openstack-meeting | 13:47 | |
| *** TrevorV has joined #openstack-meeting | 13:50 | |
| *** harsha24 has joined #openstack-meeting | 13:52 | |
| *** abhishek has joined #openstack-meeting | 13:55 | |
| abhishek | #startmeeting neutron_l3 | 13:56 |
| openstack | Meeting started Wed Apr 22 13:56:46 2020 UTC and is due to finish in 60 minutes. The chair is abhishek. Information about MeetBot at http://wiki.debian.org/MeetBot. | 13:56 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 13:56 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 13:56 | |
| openstack | The meeting name has been set to 'neutron_l3' | 13:56 |
| *** abhishek has left #openstack-meeting | 13:57 | |
| *** harsha24 has quit IRC | 13:57 | |
| *** yamamoto has quit IRC | 13:58 | |
| *** abhishek has joined #openstack-meeting | 13:58 | |
| abhishek | #endmeeting | 13:58 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 13:58 | |
| openstack | Meeting ended Wed Apr 22 13:58:56 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 13:58 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-13.56.html | 13:59 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-13.56.txt | 13:59 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-13.56.log.html | 13:59 |
| *** Liang__ is now known as LiangFang | 13:59 | |
| *** abhishek has left #openstack-meeting | 13:59 | |
| *** e0ne has quit IRC | 14:00 | |
| *** abhishek has joined #openstack-meeting | 14:00 | |
| *** e0ne has joined #openstack-meeting | 14:01 | |
| *** yamamoto has joined #openstack-meeting | 14:01 | |
| *** yamamoto has quit IRC | 14:01 | |
| *** liuyulong has joined #openstack-meeting | 14:01 | |
| *** yamamoto has joined #openstack-meeting | 14:01 | |
| *** bnemec has joined #openstack-meeting | 14:01 | |
| liuyulong | #startmeeting neutron_l3 | 14:02 |
| openstack | Meeting started Wed Apr 22 14:02:04 2020 UTC and is due to finish in 60 minutes. The chair is liuyulong. Information about MeetBot at http://wiki.debian.org/MeetBot. | 14:02 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 14:02 |
| *** openstack changes topic to " (Meeting topic: neutron_l3)" | 14:02 | |
| openstack | The meeting name has been set to 'neutron_l3' | 14:02 |
| liuyulong | Sorry a bit late | 14:02 |
| slaweq | hi | 14:02 |
| ralonsoh | hi | 14:02 |
| liuyulong | hi | 14:02 |
| *** yamamoto has quit IRC | 14:02 | |
| liuyulong | OK, let's start | 14:02 |
| liuyulong | #topic Announcements | 14:03 |
| *** openstack changes topic to "Announcements (Meeting topic: neutron_l3)" | 14:03 | |
| *** yamamoto has joined #openstack-meeting | 14:03 | |
| liuyulong | #link https://etherpad.opendev.org/p/neutron-victoria-ptg | 14:03 |
| liuyulong | I have one question the virtual PTG will be place in the UTC timezone or else? | 14:05 |
| liuyulong | #link http://eavesdrop.openstack.org/meetings/networking/2020/networking.2020-04-21-14.00.log.html#l-63 | 14:05 |
| *** thgcorrea has joined #openstack-meeting | 14:05 | |
| liuyulong | I see some comments from last team meeting | 14:05 |
| slaweq | liuyulong: there are 3 different slots during the day | 14:06 |
| slaweq | one of them at least is good for APAC TZ | 14:07 |
| slaweq | please check https://doodle.com/poll/xcd82ea6svs5dben and fill which time slots would be good for You | 14:07 |
| slaweq | I will try to book some at the end of this week | 14:07 |
| liuyulong | It's almost 12 hours time difference between Beijing and NY city. | 14:07 |
| slaweq | liuyulong: I hope we will somehow manage to do it :) | 14:10 |
| *** liuyulong_ has joined #openstack-meeting | 14:10 | |
| *** liuyulong_ has quit IRC | 14:11 | |
| *** liuyulong_ has joined #openstack-meeting | 14:12 | |
| liuyulong_ | Sorry lost my network connection | 14:12 |
| *** liuyulong has quit IRC | 14:13 | |
| *** liuyulong_ is now known as liuyulong | 14:13 | |
| liuyulong | OK | 14:13 |
| liuyulong | let's move on. | 14:13 |
| *** manuvakery has quit IRC | 14:14 | |
| liuyulong | I have no more announcement. And this time slots tool should be applied to all you want to attend the ptg. | 14:14 |
| liuyulong | #topic Bugs | 14:14 |
| *** openstack changes topic to "Bugs (Meeting topic: neutron_l3)" | 14:14 | |
| liuyulong | #link http://lists.openstack.org/pipermail/openstack-discuss/2020-April/014332.html | 14:14 |
| liuyulong | Ryan Tidwell (tidwellr) was our bug deputy last week, thanks. | 14:15 |
| liuyulong | First one | 14:15 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1872407 | 14:15 |
| openstack | Launchpad bug 1798577 in neutron "duplicate for #1872407 [FWaas-DVR]wrong port name in iptables rules" [Medium,In progress] - Assigned to Wang Weijia (wangweij) | 14:15 |
| liuyulong | It is duplicated to this: https://bugs.launchpad.net/neutron/+bug/1798577 | 14:15 |
| openstack | Launchpad bug 1798577 in neutron "[FWaas-DVR]wrong port name in iptables rules" [Medium,In progress] - Assigned to Wang Weijia (wangweij) | 14:15 |
| liuyulong | And there is a fix: https://review.opendev.org/#/c/606007/ | 14:15 |
| liuyulong | The fix actually is from our local deployment. Wang Weijia is my colleague. | 14:16 |
| liuyulong | So maybe we can move forward about the fix. | 14:17 |
| liuyulong | OK, next one | 14:18 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1873761 | 14:18 |
| openstack | Launchpad bug 1873761 in neutron "Internal IP leak to physical interface from qrouter in DVR mode" [Undecided,New] | 14:18 |
| slaweq | liuyulong: sure, but problem is that we are lack of core reviewers for fwaas | 14:18 |
| slaweq | but I will try to check that patch | 14:18 |
| liuyulong | slaweq, OK, thank you. | 14:19 |
| liuyulong | I was trying to reproduce the problem locally today. | 14:19 |
| liuyulong | But unfortunately, not successfully. | 14:19 |
| slaweq | that's interesting | 14:20 |
| slaweq | IMO it has to be some rare corner case when this issue happens | 14:20 |
| liuyulong | I will try to reproduce again in a new devstack installation. | 14:20 |
| slaweq | or some misconfiguration issue | 14:20 |
| slaweq | as I think lajoskatona or rubasov already tried to reproduce it too and also failed | 14:21 |
| slaweq | or maybe I mixed 2 different LPs - sorry :) | 14:21 |
| liuyulong | slaweq, np | 14:22 |
| rubasov | I was not working with the 'internal ip leak' bug :-) | 14:22 |
| slaweq | rubasov: ok, I probaby made some mistake there :) | 14:22 |
| rubasov | np | 14:23 |
| liuyulong | slaweq, about the config options, I guess the security group driver can be another potential leak point for the bug. | 14:23 |
| liuyulong | Maybe not | 14:24 |
| liuyulong | At least the openflow security group driver has the conntrack functionality. | 14:25 |
| liuyulong | For the bug itself, it could be a security risk for shared networks. | 14:26 |
| liuyulong | And for the access of floating IPs from the out side world, the Internal IP may also be visible due to such leak. | 14:28 |
| liuyulong | OK, let's try to reproduce it first. | 14:29 |
| liuyulong | Next | 14:29 |
| liuyulong | https://bugs.launchpad.net/neutron/+bug/1873375 | 14:29 |
| openstack | Launchpad bug 1873375 in neutron "Can not use vrrp in a dvr openstack environment " [Undecided,New] | 14:29 |
| liuyulong | IMO, it is duplicated to this: https://bugs.launchpad.net/neutron/+bug/1859638 | 14:29 |
| openstack | Launchpad bug 1774459 in neutron "duplicate for #1859638 Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:29 |
| liuyulong | Then the final bug: https://bugs.launchpad.net/neutron/+bug/1774459 | 14:30 |
| openstack | Launchpad bug 1774459 in neutron "Update permanent ARP entries for allowed_address_pair IPs in DVR Routers" [High,In progress] - Assigned to Brian Haley (brian-haley) | 14:30 |
| liuyulong | This should be a known issue "VIP between dvr east-west networks does not work at all" | 14:31 |
| slaweq | liuyulong: isn't this patch https://review.opendev.org/#/c/717319/ addressing this issue? | 14:32 |
| slaweq | sorry, https://review.opendev.org/#/c/716302/ is patch for master branch | 14:32 |
| liuyulong | No, it's not the same. | 14:33 |
| liuyulong | Due to the dvr_host_mac, there are some cases that the VIP's ARP may not work when the traffic is crossing the east-west subnets. | 14:37 |
| slaweq | ok, I see now | 14:38 |
| liuyulong | This is the fix which is trying to address these issues: https://review.opendev.org/#/c/601336/ | 14:40 |
| liuyulong | Swaminathan is not so much activity now... | 14:40 |
| liuyulong | OK, let's move on | 14:42 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1873708 | 14:42 |
| openstack | Launchpad bug 1873708 in neutron "Floating IP was not removed from the router when the port forwarding deletion was completed" [Low,In progress] - Assigned to ZhouHeng (zhouhenglc) | 14:42 |
| liuyulong | I'm be able to reproduce the bug locally. | 14:42 |
| liuyulong | Yes, after "Delete port forwarding", the floating IP is still in the snat-namespace. | 14:43 |
| liuyulong | #link https://review.opendev.org/#/c/721129/ | 14:43 |
| liuyulong | This is the fix, it's simple, and it works. | 14:44 |
| liuyulong | Next one, it is not in the deputy's list. | 14:45 |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1874211 | 14:45 |
| openstack | Launchpad bug 1874211 in neutron "[L3HA] Keepalived 2.x.x tracks state of virtual_ipaddresses interfaces and router now" [Critical,In progress] - Assigned to Slawek Kaplonski (slaweq) | 14:45 |
| slaweq | this one is fresh :) | 14:46 |
| liuyulong | This fix should be this: https://review.opendev.org/#/c/721799/ | 14:46 |
| *** ttsiouts_ has quit IRC | 14:46 | |
| liuyulong | LP has another link https://review.opendev.org/721805 | 14:46 |
| slaweq | the second one is just modification of our ci jobs related to this bug a bit | 14:47 |
| slaweq | real fix is https://review.opendev.org/#/c/721799/ | 14:47 |
| slaweq | and it works, as can be seen in https://review.opendev.org/#/c/721574/ when checking results of tripleo-ci-centos-8-scenario007-standalone job | 14:47 |
| *** ttsiouts has joined #openstack-meeting | 14:48 | |
| liuyulong | slaweq, nice work! Neutron always needs such work when the underlay software changes... | 14:49 |
| slaweq | yes, and we need CI coverage to catch it :) | 14:49 |
| *** links has quit IRC | 14:49 | |
| liuyulong | OK, no more bugs from me now. | 14:50 |
| liuyulong | Any updates? | 14:50 |
| liuyulong | OK, next topic | 14:51 |
| liuyulong | #topic OVN_L3 | 14:51 |
| *** openstack changes topic to "OVN_L3 (Meeting topic: neutron_l3)" | 14:51 | |
| liuyulong | #link https://bugs.launchpad.net/neutron/+bug/1871730 | 14:52 |
| openstack | Launchpad bug 1871730 in neutron "[OVN] local conf devstack for a ovn-northd (DB only) node does not work" [Low,In progress] - Assigned to Maciej Jozefczyk (maciej.jozefczyk) | 14:52 |
| liuyulong | I've run the fix https://review.opendev.org/#/c/719972/ locally, it really works! | 14:52 |
| maciejjozefczyk | liuyulong, thanks to Lajos :) | 14:52 |
| liuyulong | Thanks to Lajos Katona | 14:52 |
| liuyulong | maciejjozefczyk, Yep : ) | 14:53 |
| liuyulong | I have a topic about the OVN L3. | 14:53 |
| maciejjozefczyk | yes? | 14:54 |
| liuyulong | It is possible to disable built-in router service plugin (and L3-agent), then just use ovn-router to apply the L3 functions? | 14:54 |
| liuyulong | Impossible I guess. | 14:55 |
| maciejjozefczyk | liuyulong, we explicitly disable q-l3 | 14:56 |
| maciejjozefczyk | https://github.com/openstack/neutron/blob/master/devstack/ovn-local.conf.sample#L43 | 14:56 |
| liuyulong | I mean, for the compute node, it is still run ovs-agent, but DVR should not be used. And a centralized ovn-gateway cluster can be run with ovn related stuff only. | 14:57 |
| liuyulong | Then the floating IPs, internal subnet gateway and east-west traffic could work? | 14:58 |
| maciejjozefczyk | liuyulong, for compute node we don't use ovs-agent at all | 14:59 |
| maciejjozefczyk | liuyulong, have you noticed it somehow running? | 14:59 |
| liuyulong | maciejjozefczyk, no | 15:00 |
| liuyulong | maciejjozefczyk, I guess it is impossible, OVN L3 functions is highly coupled to L2 implementation. | 15:00 |
| liuyulong | maciejjozefczyk, if no ovn-controller in compute node, the L3 functionalities can not work in ovn-gateway node. | 15:01 |
| liuyulong | OK, time is up... | 15:01 |
| maciejjozefczyk | liuyulong, in call cases while using OVN we need to have running ovn-controller thats a must have | 15:01 |
| liuyulong | #endmeeting | 15:01 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 15:01 | |
| openstack | Meeting ended Wed Apr 22 15:01:47 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:01 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-14.02.html | 15:01 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-14.02.txt | 15:01 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/neutron_l3/2020/neutron_l3.2020-04-22-14.02.log.html | 15:01 |
| ralonsoh | bye | 15:01 |
| *** abhishek has quit IRC | 15:03 | |
| *** ttsiouts has quit IRC | 15:03 | |
| *** manuvakery has joined #openstack-meeting | 15:10 | |
| *** dklyle has joined #openstack-meeting | 15:11 | |
| *** gyee has joined #openstack-meeting | 15:11 | |
| *** LiangFang has quit IRC | 15:17 | |
| *** links has joined #openstack-meeting | 15:19 | |
| *** ttsiouts has joined #openstack-meeting | 15:23 | |
| *** e0ne has quit IRC | 15:28 | |
| *** e0ne has joined #openstack-meeting | 15:28 | |
| *** belmoreira has quit IRC | 15:40 | |
| *** ttsiouts has quit IRC | 15:50 | |
| *** links has quit IRC | 15:58 | |
| *** jmasud has joined #openstack-meeting | 16:10 | |
| *** seco has joined #openstack-meeting | 16:11 | |
| *** igordc has joined #openstack-meeting | 16:12 | |
| *** links has joined #openstack-meeting | 16:20 | |
| *** jmasud has quit IRC | 16:20 | |
| *** ociuhandu has quit IRC | 16:27 | |
| *** ociuhandu has joined #openstack-meeting | 16:28 | |
| *** ttsiouts has joined #openstack-meeting | 16:28 | |
| *** rpittau is now known as rpittau|afk | 16:29 | |
| *** jmasud has joined #openstack-meeting | 16:31 | |
| *** ociuhandu has quit IRC | 16:32 | |
| *** yaawang has joined #openstack-meeting | 16:32 | |
| *** yaawang_ has quit IRC | 16:33 | |
| *** evrardjp has quit IRC | 16:35 | |
| *** evrardjp has joined #openstack-meeting | 16:35 | |
| *** eharney has quit IRC | 16:39 | |
| *** ttsiouts has quit IRC | 16:44 | |
| *** e0ne has quit IRC | 16:47 | |
| *** e0ne has joined #openstack-meeting | 16:48 | |
| *** ociuhandu has joined #openstack-meeting | 16:49 | |
| *** yasufum has joined #openstack-meeting | 16:53 | |
| *** eharney has joined #openstack-meeting | 16:53 | |
| *** e0ne has quit IRC | 16:56 | |
| *** e0ne has joined #openstack-meeting | 16:56 | |
| *** yasufum has quit IRC | 16:57 | |
| *** yasufum has joined #openstack-meeting | 16:57 | |
| *** liuyulong has quit IRC | 17:02 | |
| *** eharney has quit IRC | 17:06 | |
| *** eharney has joined #openstack-meeting | 17:08 | |
| *** links has quit IRC | 17:10 | |
| *** seco has quit IRC | 17:13 | |
| *** ttsiouts has joined #openstack-meeting | 17:16 | |
| *** priteau has quit IRC | 17:18 | |
| *** ttsiouts has quit IRC | 17:20 | |
| *** ociuhandu has quit IRC | 17:24 | |
| *** jmasud has quit IRC | 17:35 | |
| *** jmasud_ has joined #openstack-meeting | 17:35 | |
| *** witek has quit IRC | 17:38 | |
| *** ttsiouts has joined #openstack-meeting | 17:49 | |
| *** ralonsoh has quit IRC | 17:56 | |
| *** ociuhandu has joined #openstack-meeting | 18:02 | |
| *** jmasud_ has quit IRC | 18:10 | |
| *** jmasud has joined #openstack-meeting | 18:17 | |
| *** ttsiouts has quit IRC | 18:22 | |
| *** e0ne has quit IRC | 18:34 | |
| *** e0ne has joined #openstack-meeting | 18:34 | |
| *** ociuhandu has quit IRC | 18:38 | |
| *** vishalmanchanda has quit IRC | 18:43 | |
| *** e0ne_ has joined #openstack-meeting | 18:43 | |
| *** e0ne has quit IRC | 18:43 | |
| *** ociuhandu has joined #openstack-meeting | 18:44 | |
| *** ociuhandu has quit IRC | 18:48 | |
| *** ociuhandu has joined #openstack-meeting | 18:48 | |
| *** ttsiouts has joined #openstack-meeting | 18:58 | |
| *** ttsiouts has quit IRC | 19:03 | |
| *** e0ne_ has quit IRC | 19:04 | |
| *** manuvakery has quit IRC | 19:19 | |
| *** ociuhandu has quit IRC | 19:22 | |
| *** jmasud has quit IRC | 19:27 | |
| *** ttsiouts has joined #openstack-meeting | 19:32 | |
| *** ociuhandu has joined #openstack-meeting | 19:33 | |
| *** ociuhandu has quit IRC | 19:38 | |
| *** diablo_rojo has joined #openstack-meeting | 19:41 | |
| *** ttsiouts has quit IRC | 20:06 | |
| *** Lucas_Gray has joined #openstack-meeting | 20:09 | |
| *** Lucas_Gray has quit IRC | 20:12 | |
| *** ociuhandu has joined #openstack-meeting | 20:17 | |
| *** seco has joined #openstack-meeting | 20:23 | |
| *** ociuhandu has quit IRC | 20:28 | |
| *** yasufum has quit IRC | 20:28 | |
| *** seco has quit IRC | 20:32 | |
| *** jmasud has joined #openstack-meeting | 20:43 | |
| *** alecuyer has joined #openstack-meeting | 20:56 | |
| *** patchbot has joined #openstack-meeting | 20:58 | |
| *** ociuhandu has joined #openstack-meeting | 21:00 | |
| timburke | #startmeeting swift | 21:00 |
| openstack | Meeting started Wed Apr 22 21:00:22 2020 UTC and is due to finish in 60 minutes. The chair is timburke. Information about MeetBot at http://wiki.debian.org/MeetBot. | 21:00 |
| openstack | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 21:00 |
| *** openstack changes topic to " (Meeting topic: swift)" | 21:00 | |
| openstack | The meeting name has been set to 'swift' | 21:00 |
| timburke | who's here for the swift meeting? | 21:00 |
| seongsoocho | o/ | 21:00 |
| mattoliverau | o/ | 21:00 |
| alecuyer | o/ | 21:00 |
| kota_ | o/ | 21:01 |
| rledisez | o/ | 21:01 |
| timburke | as usual, agenda's at https://wiki.openstack.org/wiki/Meetings/Swift | 21:02 |
| timburke | #topic PTG | 21:02 |
| *** openstack changes topic to "PTG (Meeting topic: swift)" | 21:02 | |
| timburke | another reminder that we've got an etherpad to collect things to talk about | 21:02 |
| timburke | #link https://etherpad.openstack.org/p/swift-ptg-victoria | 21:02 |
| tdasilva | hi | 21:03 |
| timburke | i also recently tried to get together a list of all our *prior* etherpads | 21:03 |
| timburke | #link https://wiki.openstack.org/wiki/Swift/Etherpads | 21:03 |
| timburke | thought it might be worth revisiting some of them :-) | 21:03 |
| mattoliverau | Oh nice | 21:03 |
| tdasilva | wow, nice! | 21:03 |
| rledisez | awesome! | 21:04 |
| kota_ | good | 21:04 |
| clayg | i'm here too 😁 | 21:04 |
| timburke | #topic swift release | 21:04 |
| *** openstack changes topic to "swift release (Meeting topic: swift)" | 21:04 | |
| timburke | we had one! | 21:04 |
| clayg | 🥳 | 21:04 |
| mattoliverau | Yay | 21:04 |
| timburke | 2.25.0 is out and is expected to be our ussuri release | 21:04 |
| timburke | ...which leads to... | 21:05 |
| timburke | #topic rolling upgrade gate is broken | 21:05 |
| *** openstack changes topic to "rolling upgrade gate is broken (Meeting topic: swift)" | 21:05 | |
| timburke | i burned a day or so figuring out what went wrong, but our upgrade job broke following 2.25.0 | 21:06 |
| timburke | the key was that i wrote a func test for the etag-quoter that left the thing on for the account | 21:06 |
| timburke | which broke a whole bunch of tests after that one | 21:07 |
| timburke | https://review.opendev.org/#/c/721518/ should fix the test to not leave the account dirty | 21:07 |
| patchbot | patch 721518 - swift - func tests: Allow test_etag_quoter to be run multi... - 1 patch set | 21:07 |
| clayg | but the multi node upgrade job is still failing 🤔 | 21:08 |
| timburke | https://review.opendev.org/#/c/695131/ both fixes the func tests to accept either quoted or unquoted etags *and* makes an in-process job run with etag-quoter in the pipeline | 21:09 |
| patchbot | patch 695131 - swift - func tests: work with etag-quoter on by default - 7 patch sets | 21:09 |
| timburke | clayg, 2.25.0 is out, and the func tests for that job come from that tag :-( | 21:09 |
| timburke | plus, the fix isn't merged yet :P | 21:09 |
| timburke | which was why i had to make it non-voting in https://review.opendev.org/#/c/721519/ | 21:09 |
| patchbot | patch 721519 - swift - Make rolling-upgrade job non-voting (MERGED) - 1 patch set | 21:09 |
| clayg | ic, so the rolling update does setup using code form a tag - so once the latest tag has fixed code future changes will work! Brilliant! | 21:10 |
| *** ociuhandu has quit IRC | 21:10 | |
| clayg | so basically "tim fixed everything and needs people to click buttons"??? | 21:10 |
| clayg | or just "tim fixed everything" and we can just 👏 | 21:10 |
| timburke | maybe? there's going to be more work coming, too -- in particular, i really want to have a "kitchen sink" func test job where we have *no* (or exceedingly few) skips | 21:11 |
| timburke | i was thinking the dsvm jobs might be a good place for it, so we see all features passing against both keystone and tempauth | 21:12 |
| timburke | (plus there are a bunch of tests that skip if you're not running with some extra keystone users defined) | 21:12 |
| timburke | so i dusted off https://review.opendev.org/#/c/620189/ | 21:13 |
| patchbot | patch 620189 - swift - WIP: swift-dsvm: Create more Keystone users so we ... - 11 patch sets | 21:13 |
| timburke | and started poking at https://review.opendev.org/#/c/722120/ | 21:13 |
| patchbot | patch 722120 - swift - dsvm: Enable more middlewares - 1 patch set | 21:13 |
| timburke | and i'm thinking about ways to make sure that we never have a release break that job again -- but i'm not entirely sure how to get that 🤔 | 21:14 |
| timburke | maybe a rolling-upgrade job that uses origin/master? could add it to the experimental checks that i always run when putting the authors/changelog patch together | 21:15 |
| timburke | if anybody has ideas on that, feel free to reach out! | 21:15 |
| timburke | on to updates! | 21:16 |
| *** ttsiouts has joined #openstack-meeting | 21:16 | |
| timburke | #topic waterfall ec | 21:16 |
| *** openstack changes topic to "waterfall ec (Meeting topic: swift)" | 21:16 | |
| timburke | clayg, still trying to find time to think more about it? | 21:17 |
| clayg | i mean i guess kinda... i want to put it behind working on better tracing support | 21:17 |
| timburke | 👍 | 21:18 |
| timburke | should i take it off the agenda for next meeting? | 21:18 |
| clayg | but if I was for sure where the problem was and that I could do something to make it better I guess we could skip that part | 21:18 |
| clayg | yeah you don't need to carry it forward | 21:18 |
| timburke | #topic lots of small files | 21:18 |
| *** openstack changes topic to "lots of small files (Meeting topic: swift)" | 21:18 | |
| timburke | alecuyer, rledisez how's it going? | 21:18 |
| alecuyer | Still working on it, but I've had to spend some time on other things too, I think I can post a first patch for the new key format this week | 21:19 |
| timburke | sounds good | 21:20 |
| timburke | #topic CORS | 21:20 |
| *** openstack changes topic to "CORS (Meeting topic: swift)" | 21:20 | |
| timburke | i still need to clean up https://review.opendev.org/#/c/720098/ | 21:21 |
| patchbot | patch 720098 - swift - WIP: s3api: Allow MPUs via CORS requests - 6 patch sets | 21:21 |
| timburke | but i think the other three in the chain are good to go (p 533028, p 710330, p 710355) | 21:21 |
| patchbot | https://review.opendev.org/#/c/710330/ - swift - s3api: Pass through CORS headers - 13 patch sets | 21:21 |
| patchbot | https://review.opendev.org/#/c/710355/ - swift - s3api: Allow CORS preflight requests - 16 patch sets | 21:21 |
| timburke | any chance someone would have an opportunity to look at them? | 21:22 |
| timburke | patchbot, p 533028 | 21:22 |
| patchbot | https://review.opendev.org/#/c/533028/ - swift - Add some functional CORS tests - 17 patch sets | 21:22 |
| timburke | better :D | 21:22 |
| *** zaitcev has joined #openstack-meeting | 21:22 | |
| timburke | well, worth asking ;-) | 21:24 |
| timburke | #topic sharding split brain | 21:24 |
| *** openstack changes topic to "sharding split brain (Meeting topic: swift)" | 21:24 | |
| * mattoliverau not good at js. But I can look.. not sure it'll be useful :P | 21:24 | |
| mattoliverau | uh-oh | 21:25 |
| timburke | so we had a customer that accidentally got two sets of shard ranges inserted from different nodes | 21:25 |
| timburke | oops | 21:25 |
| timburke | this was through swift-manage-shard-ranges (not the auto-sharding stuff), it just got run twice for the same container | 21:25 |
| timburke | i got as far as writing a couple probe tests in p 721376, one of them gets it into the bad state | 21:26 |
| patchbot | https://review.opendev.org/#/c/721376/ - swift - sharding: Add probe test that exercises swift-mana... - 2 patch sets | 21:26 |
| timburke | next up i need to figure out how to get back to a *good* state :P | 21:26 |
| mattoliverau | I'll definitely look at that test. | 21:27 |
| timburke | thanks! | 21:27 |
| mattoliverau | maybe we need to make sure we can id a set of ranges inserted by a tool. | 21:27 |
| rledisez | on that topic, I saw some activity about auto sharding mattoliverau, anything we can do to help? | 21:28 |
| timburke | well, they'll all have the same epoch & timestamp as i recall, so *that's* good | 21:28 |
| mattoliverau | I did have some code somewhere, in the old POC that scanned ranges in a table and found the most recent contiguas full set of ranges (no gaps) | 21:28 |
| timburke | i'm thinking i probably want to mark one set's shard ranges as collapsed, so any already-cleaved DBs move their rows to the other set's shards? | 21:28 |
| timburke | fortunately, i'm pretty sure each set is complete -- it was using find_and_replace to do the whole set at once | 21:30 |
| mattoliverau | oh yeah I'm sure, just if we want to pick one set then collpase or whatever the rest. | 21:30 |
| mattoliverau | if we can't as easily define set by metadata. though surely we can. I'll need to look at the code again | 21:31 |
| timburke | yeah, i'll try that out, see how far i get. definitely more code coming in the next few weeks | 21:31 |
| timburke | #topic open discussion | 21:31 |
| *** openstack changes topic to "open discussion (Meeting topic: swift)" | 21:32 | |
| mattoliverau | rledisez: thanks, just started blowing the dust off it, rebasing it. Wrote up a braindump: https://docs.google.com/document/d/17NllKQmH6tfTsKm5nAx3KCKUvs0zs_qamXtkreOQDWg/edit# | 21:32 |
| mattoliverau | ^ auto-sharding | 21:32 |
| mattoliverau | next step was to maybe create a probetest(s) to attempt to find other edgecases I can fix. | 21:32 |
| timburke | oh, i realized etag-quoter isn't happy on py3: https://review.opendev.org/#/c/721714/ | 21:34 |
| patchbot | patch 721714 - swift - py3: Make etag-quoter work - 1 patch set | 21:34 |
| mattoliverau | Thinking about an exclusive UPDATE, so only add shards if there not already there to other primary nodes. Fail if there are nodes, to fix a potential edge case I could see. | 21:34 |
| mattoliverau | I'll add it to the doc when I get a chance. | 21:34 |
| mattoliverau | *fail if there are shards.. | 21:34 |
| mattoliverau | but anyway, all just brainstorming atm, getting ready for the PTG ;) | 21:35 |
| rledisez | mattoliverau: I'll carefully read your document. I'm really interested in this feature | 21:35 |
| rledisez | still related to container sharding: say I have few (tens) of thousands of containers to shard. would you recommend that I shard them all at once and let the sharder do it's job, or shard some of them, wait for the sharder, shard more, etc… | 21:35 |
| timburke | yeah, i really need to catch up on emails for that :-( | 21:35 |
| timburke | rledisez, at first, i'd say do one at a time, just to see how it goes, check logs, etc. | 21:36 |
| *** raildo has quit IRC | 21:37 | |
| mattoliverau | +1 | 21:37 |
| rledisez | yeah, we already did about 15 containers, it went well so now I want to go further :) | 21:37 |
| timburke | after doing that a couple times, though, i think it should be pretty safe to do a bunch at once -- we recently started offering that, i can ask around a bit to see how it went | 21:37 |
| timburke | i've come to rather appreciate the cleave limit -- makes sure the sharder isn't stuck on any single container for too long | 21:38 |
| rledisez | what I'm afraid if we do a lot at once is maybe the disk space usage on container server. It could a lot more space until all are done | 21:38 |
| timburke | true -- how full are disks? | 21:38 |
| rledisez | well filled, so I should do small steps probably :) | 21:39 |
| timburke | time to buy more disks! damn supply-chain issues... | 21:39 |
| clayg | i don't think the process really requires a bunch of scratch space | 21:39 |
| rledisez | to be exact, as you probably know, some of them are almost full while others are quite empty, big containers, uh… :D | 21:40 |
| rledisez | clayg: I though it deleted the original sqlite only when all is done, so for each container it would consume the double space in the cluster until it's done. am I wrong? | 21:40 |
| clayg | i guess the root being split won't be removed until after all the shards are populated - so yeah probably one a time is better | 21:40 |
| timburke | ah, yeah -- so actually, it might be fine. sharding's *great* at evening out that lumpiness a bit | 21:41 |
| rledisez | it's probably something to keep in mind for the auto-sharding. a limit on the number of containers being sharded at a time | 21:41 |
| *** ociuhandu has joined #openstack-meeting | 21:41 | |
| rledisez | mattoliverau: ^ | 21:41 |
| mattoliverau | rledisez: good idea! | 21:42 |
| timburke | could probably even have an optimization where it goes to check recon dumps *first* to see what's currently sharding, then go straight to the DBs... skip the treewalk. hmm... | 21:43 |
| rledisez | the best would be to estimate the size of each shard and check there enough space on the devices holding these shards | 21:43 |
| rledisez | timburke: totally, that's what we do know: for db in $(cat | jq … Í …); do | 21:44 |
| *** geguileo has quit IRC | 21:45 | |
| *** tobiash has quit IRC | 21:45 | |
| *** pescobar has quit IRC | 21:45 | |
| *** Guest91194 has joined #openstack-meeting | 21:46 | |
| timburke | all right, i think i'm going to call it | 21:46 |
| *** eharney has quit IRC | 21:46 | |
| timburke | thank you all for coming, and thank you for working on swift! | 21:47 |
| timburke | #endmeeting | 21:47 |
| *** openstack changes topic to "OpenStack Meetings || https://wiki.openstack.org/wiki/Meetings/" | 21:47 | |
| openstack | Meeting ended Wed Apr 22 21:47:06 2020 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 21:47 |
| openstack | Minutes: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-04-22-21.00.html | 21:47 |
| openstack | Minutes (text): http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-04-22-21.00.txt | 21:47 |
| openstack | Log: http://eavesdrop.openstack.org/meetings/swift/2020/swift.2020-04-22-21.00.log.html | 21:47 |
| *** patchbot has left #openstack-meeting | 21:47 | |
| *** tobiash has joined #openstack-meeting | 21:47 | |
| *** haleyb has quit IRC | 21:47 | |
| *** Guest91194 is now known as geguileo | 21:47 | |
| *** eharney has joined #openstack-meeting | 21:49 | |
| *** ttsiouts has quit IRC | 21:50 | |
| *** haleyb has joined #openstack-meeting | 21:50 | |
| *** ociuhandu has quit IRC | 21:52 | |
| *** ociuhandu has joined #openstack-meeting | 21:52 | |
| *** moguimar has quit IRC | 21:53 | |
| *** seco has joined #openstack-meeting | 21:55 | |
| *** ociuhandu has quit IRC | 21:57 | |
| *** slaweq has quit IRC | 21:58 | |
| *** jmasud has quit IRC | 21:58 | |
| *** jmasud has joined #openstack-meeting | 22:03 | |
| *** jmasud has quit IRC | 22:17 | |
| *** jmasud has joined #openstack-meeting | 22:18 | |
| *** ttsiouts has joined #openstack-meeting | 22:20 | |
| *** jmasud has quit IRC | 22:22 | |
| *** ykatabam has joined #openstack-meeting | 22:29 | |
| *** ttsiouts has quit IRC | 22:29 | |
| *** jmasud has joined #openstack-meeting | 22:30 | |
| *** rcernin has joined #openstack-meeting | 22:31 | |
| *** ociuhandu has joined #openstack-meeting | 22:44 | |
| *** seco has quit IRC | 22:54 | |
| *** ttsiouts has joined #openstack-meeting | 22:57 | |
| *** TrevorV has quit IRC | 22:59 | |
| *** ociuhandu has quit IRC | 23:01 | |
| *** zaitcev has left #openstack-meeting | 23:06 | |
| *** ttsiouts has quit IRC | 23:31 | |
| *** ociuhandu has joined #openstack-meeting | 23:33 | |
| *** jmasud has quit IRC | 23:41 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!