| timburke | so i noticed that there's nothing new on the meeting agenda and i know *i* don't have any meaningful updates ATM -- i'm going to propose we skip tomorrow's meeting and see what next week brings | 02:09 |
|---|---|---|
| timburke | i'll bring it up again (my) tomorrow morning, but kota_, mattoliverau, seongsoocho you should sleep in :-) | 02:09 |
| mattoliverau | \o/ | 02:10 |
| seongsoocho | \o/ | 02:10 |
| kota_ | \o/ | 02:51 |
| * kota_ trust timburke even it's in April 1st :P | 02:51 | |
| *** psachin has joined #openstack-swift | 03:20 | |
| *** evrardjp has quit IRC | 04:36 | |
| *** evrardjp has joined #openstack-swift | 04:36 | |
| zaitcev | Well, okay. I keep forgetting about the meeting anyway. | 05:34 |
| *** irclogbot_2 has quit IRC | 06:49 | |
| *** spotz has quit IRC | 06:51 | |
| *** psachin has quit IRC | 06:53 | |
| *** tonyb has quit IRC | 06:54 | |
| *** irclogbot_3 has joined #openstack-swift | 06:55 | |
| *** tonyb has joined #openstack-swift | 06:59 | |
| *** rcernin has quit IRC | 07:15 | |
| *** rpittau|afk is now known as rpittau | 07:37 | |
| *** tkajinam has quit IRC | 08:18 | |
| *** rdejoux has quit IRC | 09:22 | |
| *** rdejoux has joined #openstack-swift | 09:48 | |
| *** rpittau is now known as rpittau|bbl | 10:16 | |
| *** rcernin has joined #openstack-swift | 10:34 | |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Optimize obj replicator/reconstructor healthchecks https://review.opendev.org/715298 | 11:17 |
| *** markguz_ has joined #openstack-swift | 11:43 | |
| markguz_ | Hi quick question, we setting up swift i used multipath drive names "/dev/mapper/mpathx" and the replicator daemons are reporting | 11:44 |
| markguz_ | container-replicator[31851]: Skipping: /dev/mapper/mpathx is not a valid drive name | 11:44 |
| markguz_ | I couldn't find anywhere in the docs that forbids the use of that drive naming scheme. and googling for the error gets 0 hits. | 11:45 |
| markguz_ | is there some sort of filter i need to set? | 11:46 |
| openstackgerrit | Romain LE DISEZ proposed openstack/swift master: Optimize obj replicator/reconstructor healthchecks https://review.opendev.org/715298 | 12:03 |
| *** rpittau|bbl is now known as rpittau | 12:07 | |
| *** spotz has joined #openstack-swift | 12:07 | |
| *** manuvakery has joined #openstack-swift | 12:32 | |
| alecuyer | markguz_: I'm guessing /dev/mapper/mpathx is a block device? For swift, you would need to format these devices (use XFS, probably), mount them somewhere, and configure swift to use that, rather than /dev/mapper/xxx. Check https://docs.openstack.org/swift/latest/development_saio.html , "Configuring storage", which should help. | 12:41 |
| markguz_ | alecuyer: i've done that, as per the swift deployment docs. whole block device is formatted and mount as xfs | 12:42 |
| markguz_ | each drive is mount under /srv/node/mpathXX | 12:44 |
| markguz_ | i've followed the docs faithfully. What i'm suspecting is that something in swift is hardcoded to look for /dev/sdX and not multipath /dev/mapper/mpathX | 12:45 |
| alecuyer | markguz_: what is "devices" set to in your configuration ? | 12:47 |
| markguz_ | alecuyer: /srv/node as per docs | 12:47 |
| alecuyer | markguz_: I haven't seen that before, let me check how that could happen | 12:50 |
| manuvakery | hi guys . my storage node (ACO) memory usage is gradually going up and used memory becomes 95% or more and the node become unresponsive . i grep for the processes which consumes memory and i can see all object-servers are on the top . i have 32GB ram on each node, there are not too many requests per sec (less than 20) . I guess if the gc works well then i dont get to that saturation point where i had | 12:51 |
| manuvakery | to do a hard reboot of the node | 12:51 |
| rledisez | manuvakery: what version of Swift are you running? | 12:55 |
| manuvakery | rledisez: 2.24.0 the latest | 12:56 |
| rledisez | when you say object-servers are "on the top", if you do the sum of all processes memory, is it equal to 32GB ? | 12:57 |
| rledisez | i'm asking because I add a lot of kernel memory leak with XFS and the only solution to free memory was to umount/remount all devices | 12:58 |
| manuvakery | not completly but close to 60 % of the tottal memory is consumed by object-servers | 13:00 |
| manuvakery | rledisez: http://paste.openstack.org/show/791460/ this is how it looks now | 13:05 |
| rledisez | manuvakery: hum, I never saw that. Do you have stale connections (from proxy or replicator) on your process? | 13:10 |
| manuvakery | rledisez: i dont see too many stale connection | 13:41 |
| manuvakery | https://www.irccloud.com/pastebin/NVbdx0yD/ | 13:42 |
| alecuyer | markguz_: so I don't yet understand how the "drive" is "/dev/mapper/mpathxx" when it should just be "mpathxx". That's the cause of the exception, but I'm not sure how/why that happens yet | 13:48 |
| markguz_ | wait a mo | 13:49 |
| markguz_ | swift-ring-builder container.builder add --region 1 --zone 1 --ip 192.168.129.254 --port 6201 --device /dev/mapper/mpathw --weight 100 | 13:49 |
| markguz_ | it's there isn't it | 13:49 |
| markguz_ | i put the full path to the device name, not the name of the folder in /srv/node/ | 13:50 |
| alecuyer | Oh yes, you should just put that single name there, not a whole path | 13:51 |
| markguz_ | d-oh! | 13:52 |
| markguz_ | it said device, and i instinctively put the path to the device. | 13:52 |
| alecuyer | I can see how that would be confusing :) sorry I didn't think of that earlier | 13:52 |
| markguz_ | sorry i didn't spot that myself. Sometimes it takes two heads ... i appreciate the time you took. I rebuilt the env and it is replicating... | 13:53 |
| alecuyer | np, I'm glad it works now | 13:54 |
| DHE | and the device name is its mount point subdirectory in /srv/node and not necessarily a literal device name | 14:24 |
| timburke | good morning | 15:29 |
| *** gyee has joined #openstack-swift | 15:36 | |
| *** viks___ has quit IRC | 15:51 | |
| *** rdejoux has quit IRC | 16:01 | |
| alecuyer | hello tim | 16:04 |
| alecuyer | I forgot I was home and now rebuilding SAIO on a slow DSL line :/ Don't be me | 16:04 |
| *** rpittau is now known as rpittau|afk | 16:15 | |
| timburke | :-( sorry | 16:19 |
| timburke | interesting! http://lists.openstack.org/pipermail/openstack-discuss/2020-April/013745.html | 16:19 |
| timburke | oh yeah, alecuyer, rledisez, clayg, tdasilva: in case you missed it, no meeting today | 16:31 |
| alecuyer | timburke: Oh yes I saw it, thanks still for the reminder! | 16:33 |
| *** evrardjp has quit IRC | 16:36 | |
| *** evrardjp has joined #openstack-swift | 16:36 | |
| *** rdejoux has joined #openstack-swift | 16:37 | |
| *** rdejoux has quit IRC | 17:15 | |
| *** rdejoux has joined #openstack-swift | 17:21 | |
| *** gmann is now known as gmann_lunch | 17:56 | |
| DHE | tempurl can't be used for container listings can it... | 18:12 |
| *** gmann_lunch is now known as gmann | 18:28 | |
| *** diablo_rojo has quit IRC | 18:32 | |
| *** diablo_rojo has joined #openstack-swift | 18:33 | |
| *** rdejoux has quit IRC | 19:26 | |
| *** rdejoux has joined #openstack-swift | 19:33 | |
| *** early` has quit IRC | 19:36 | |
| *** early has joined #openstack-swift | 19:44 | |
| *** jv has quit IRC | 20:51 | |
| clayg | oh right, no meeting today! | 21:02 |
| *** jv has joined #openstack-swift | 21:05 | |
| *** jv has quit IRC | 21:30 | |
| *** jv has joined #openstack-swift | 21:30 | |
| mattoliverau | morning | 22:28 |
| *** tkajinam has joined #openstack-swift | 22:55 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!