| *** diablo_rojo has quit IRC | 00:01 | |
| timburke | seongsoocho, o/ | 00:02 |
|---|---|---|
| timburke | interesting: http://lists.openstack.org/pipermail/openstack-discuss/2020-March/012950.html | 00:02 |
| timburke | with more discussion going on around http://eavesdrop.openstack.org/irclogs/%23openstack-tc/%23openstack-tc.2020-03-02.log.html#t2020-03-02T20:00:22 | 00:02 |
| *** threestrands has joined #openstack-swift | 00:03 | |
| *** DHE has joined #openstack-swift | 00:32 | |
| *** diablo_rojo has joined #openstack-swift | 00:47 | |
| DHE | minor question. a lot of stale partition directories remain after some significant rebalancing with large hashes.invalid files, but otherwise largely empty. what does that file actually do? | 00:53 |
| timburke | DHE, it flags suffix directories that need to have their hash (recorded in hashes.pkl) recalculated -- i'm a little surprised it got left there, i thought we had code to clean it up... | 00:56 |
| DHE | (apologies if I crash out again, power outage here) | 00:56 |
| timburke | i'd expect the next replication run to clear it all out, though | 00:57 |
| DHE | I've been rather aggressively running replicators in handsoff-only mode after a rebalance... that might be related... | 00:57 |
| DHE | actually that could be double replicators... | 00:57 |
| DHE | well it's not causing any real harm so I'll leave it be and wait | 01:04 |
| *** gyee has quit IRC | 02:08 | |
| *** diablo_rojo has quit IRC | 02:33 | |
| *** jvisser has quit IRC | 02:46 | |
| *** rcernin_ has joined #openstack-swift | 02:52 | |
| *** rcernin has quit IRC | 02:55 | |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: tests: Use timedelta to adjust dates, not string manipulations https://review.opendev.org/710901 | 05:03 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/stein: tests: Use timedelta to adjust dates, not string manipulations https://review.opendev.org/710902 | 05:04 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/train: Ensure domain stored in memcached gets utf8 decoded on py2 https://review.opendev.org/710904 | 05:14 |
| openstackgerrit | Tim Burke proposed openstack/swift stable/stein: Ensure domain stored in memcached gets utf8 decoded on py2 https://review.opendev.org/710905 | 05:20 |
| *** rahul_sg has joined #openstack-swift | 05:21 | |
| *** rcernin_ has quit IRC | 05:24 | |
| rahul_sg | Is there any bucket policy patch available for s3api? | 05:32 |
| *** ccamel has quit IRC | 05:34 | |
| *** evrardjp has quit IRC | 05:35 | |
| *** evrardjp has joined #openstack-swift | 05:35 | |
| *** camelCaser has joined #openstack-swift | 05:37 | |
| pawan-gupta | clayg: thanks for the response, I was looking bucket and object operations e.g https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-buckets and https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html#using-with-s3-actions-related-to-objects | 06:20 |
| *** diablo_rojo has joined #openstack-swift | 06:33 | |
| *** diablo_rojo has quit IRC | 07:09 | |
| *** rcernin has joined #openstack-swift | 07:13 | |
| *** rdejoux has quit IRC | 07:17 | |
| *** rdejoux has joined #openstack-swift | 08:16 | |
| *** manuvakery has quit IRC | 08:27 | |
| *** tesseract has joined #openstack-swift | 08:30 | |
| *** rpittau|afk is now known as rpittau | 08:41 | |
| *** rcernin has quit IRC | 08:41 | |
| *** ccamacho has quit IRC | 09:02 | |
| *** FlorianFa has quit IRC | 09:05 | |
| *** threestrands has quit IRC | 09:09 | |
| *** tesseract has quit IRC | 09:25 | |
| *** tesseract has joined #openstack-swift | 09:26 | |
| *** rcernin has joined #openstack-swift | 10:05 | |
| *** rcernin has quit IRC | 10:37 | |
| *** rcernin has joined #openstack-swift | 10:38 | |
| *** ccamacho has joined #openstack-swift | 10:52 | |
| *** ccamacho has quit IRC | 10:53 | |
| *** ccamacho has joined #openstack-swift | 10:54 | |
| *** openstackstatus has quit IRC | 10:59 | |
| *** rpittau is now known as rpittau|bbl | 11:17 | |
| *** rcernin has quit IRC | 11:52 | |
| *** jvisser has joined #openstack-swift | 12:33 | |
| *** rpittau|bbl is now known as rpittau | 13:04 | |
| *** DHE has quit IRC | 13:04 | |
| *** jvisser_ has joined #openstack-swift | 13:18 | |
| *** jvisser has quit IRC | 13:19 | |
| *** DHE has joined #openstack-swift | 13:24 | |
| *** jvisser_ has quit IRC | 13:25 | |
| *** jvisser_ has joined #openstack-swift | 13:38 | |
| *** zigo has quit IRC | 13:58 | |
| *** tkajinam has quit IRC | 14:30 | |
| *** tkajinam has joined #openstack-swift | 14:31 | |
| *** zigo has joined #openstack-swift | 14:33 | |
| *** tkajinam has quit IRC | 15:09 | |
| *** tesseract has quit IRC | 15:43 | |
| *** tesseract has joined #openstack-swift | 15:53 | |
| *** gyee has joined #openstack-swift | 16:27 | |
| *** rdejoux has quit IRC | 16:43 | |
| *** tesseract has quit IRC | 16:58 | |
| *** rdejoux has joined #openstack-swift | 17:02 | |
| *** rpittau is now known as rpittau|afk | 17:10 | |
| *** rdejoux has quit IRC | 17:20 | |
| *** evrardjp has quit IRC | 17:35 | |
| *** evrardjp has joined #openstack-swift | 17:35 | |
| *** guimaluf has joined #openstack-swift | 17:42 | |
| guimaluf | clayg, timburke hi folks, I'm here available to give all needed info for idenitifying and fixing this O_TMPFILE issue... let me know if you guys are available | 17:45 |
| *** diablo_rojo has joined #openstack-swift | 17:57 | |
| clayg | booo p 710492 - do I just type recheck again? | 18:32 |
| patchbot | https://review.opendev.org/#/c/710492/ - swift - Apply limit to list versioned containers - 4 patch sets | 18:32 |
| timburke | hmm... maybe i should just squash in the follow-up... | 19:22 |
| openstackgerrit | Tim Burke proposed openstack/swift master: Apply limit to list versioned containers https://review.opendev.org/710492 | 19:27 |
| *** mvkr has quit IRC | 19:28 | |
| clayg | yeah, squash is good 👍 | 19:51 |
| *** mvkr has joined #openstack-swift | 19:53 | |
| guimaluf | clayg, timburke how can I enable/disable that O_TMPFILE support on xfs? | 20:33 |
| DHE | I think you have to modify swift/obj/diskfile.py and indicate that tmpfile support isn't available | 20:34 |
| guimaluf | DHE, that mean I'll have to change each of my storage node code and change that part? isn't there another more elegant solution? | 20:36 |
| DHE | nothing I noticed... | 20:37 |
| DHE | (I needed to disable this myself for ZFS) | 20:37 |
| clayg | well that's at least two use-cases for a patch to make it a config option | 20:37 |
| guimaluf | :( what is the cause of this O_TMPFILE been disabled? | 20:37 |
| guimaluf | I tought it was something in my environment... some missing kernel, lib or somethingselse. | 20:38 |
| * clayg FWIW originally it was a config option: | 20:40 | |
| clayg | https://review.opendev.org/#/c/162243/11/etc/object-server.conf-sample | 20:40 |
| patchbot | patch 162243 - swift - Make object creation more atomic in Linux (MERGED) - 21 patch sets | 20:40 |
| clayg | but everyone got behind the auto identification | 20:41 |
| clayg | guimaluf: it's still not clear to me why o_tmpfile support or your system doesn't seem to be working like we've observed on other systems | 20:41 |
| clayg | it's a nice improvement and should be quite mainstream by now - just trying to not use it might be a good strategy, but may not be the best long term | 20:42 |
| DHE | The ZFS thing was just a performance issue. https://github.com/openzfs/zfs/pull/10005 (accepted, so I guess the next version will fix it...) | 20:42 |
| * clayg guimaluf: are you the same: | 20:43 | |
| clayg | vagrant@saio:~$ uname -a | 20:43 |
| clayg | Linux saio 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 20:43 |
| clayg | vagrant@saio:~$ ldd --version | 20:43 |
| clayg | ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27 | 20:43 |
| guimaluf | clayg, Linux storage-dev-01 4.15.0-88-generic #88-Ubuntu SMP Tue Feb 11 20:11:34 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux | 20:48 |
| guimaluf | ldd (Ubuntu GLIBC 2.27-3ubuntu1) 2.27 | 20:49 |
| guimaluf | clayg, I would like to try to identify why my system doesn't have that support... this is so wierd | 20:49 |
| guimaluf | the only "detail" is that system was upgraded from 14.04 to 18.04 | 20:50 |
| *** jvisser_ has quit IRC | 20:53 | |
| *** rdejoux has joined #openstack-swift | 21:05 | |
| guimaluf | DHE, clayg do you guys know where does the TMPFILE is created? /tmp /var/tmp or something similar? | 21:05 |
| guimaluf | isn't this permission denied a sign that the place being available for the TMPFILE hasn't the right permissions? | 21:06 |
| DHE | guimaluf: the feature involves making a file appear out of nowhere with all file contents already intact. you start with a file that's already deleted, populate it, then use hardlinking to make it appear where you really want it. | 21:18 |
| DHE | so basically it involves undeleting a file | 21:18 |
| DHE | but the hardlink means it MUST be done in the filesystem where it's being kept. so /srv/node/$DEVICENAME is mandatory, not /tmp | 21:19 |
| guimaluf | make sense, does this mean to /srv/node/object/tmp for instance? | 21:19 |
| DHE | umm.. maybe. I'm not actually sure. I'm not a dev, just a user who's not afraid to get his hands any dirtier. | 21:20 |
| guimaluf | the permissions are right... that's odd... | 21:20 |
| guimaluf | DHE, I like that kind of user :) | 21:20 |
| guimaluf | the object/tmp permissions are right | 21:22 |
| *** jvisser has joined #openstack-swift | 21:26 | |
| timburke | so the tempfile should have a parent directory of the object's hash dir -- ie, the directory that will container the .data once we've called linkat | 21:39 |
| timburke | guimaluf, i think you already left when i posted it last night, but mind adding something like st = os.fstat(dfw._fd); print('%r (mode=0o%o)' % (st, st.st_mode))` between the write() and the put() in that test script? | 21:40 |
| * guimaluf doing... | 21:41 | |
| guimaluf | timburke, posix.stat_result(st_mode=32768, st_ino=140288475, st_dev=51776, st_nlink=0, st_uid=106, st_gid=113, st_size=4, st_atime=1583271804, st_mtime=1583271804, st_ctime=1583271804) (mode=0o100000) | 21:43 |
| mattoliverau | Morning | 21:44 |
| timburke | !! mode=0o100000 🤔 | 21:44 |
| openstack | timburke: Error: "!" is not a valid command. | 21:44 |
| timburke | quiet, you | 21:44 |
| guimaluf | hahahahaha | 21:45 |
| guimaluf | timburke, what that means? | 21:45 |
| timburke | well, i can see how that'd explain the EACCES... but how did the write succeed?? | 21:45 |
| guimaluf | the write didn't | 21:45 |
| guimaluf | I'm getting the same error | 21:45 |
| guimaluf | IOError: [Errno 13] Permission denied | 21:46 |
| guimaluf | `sudo -u swift python /var/tmp/otmpfile.py /var/storage/node/xvdf/tmp/` | 21:46 |
| guimaluf | that's how I'm executing | 21:46 |
| timburke | sorry, i meant the write() -- like, we could put down the data, just couldn't do the metadata, the put() | 21:46 |
| timburke | right? | 21:46 |
| guimaluf | yes. exactly. the put() is failing | 21:47 |
| guimaluf | dfw.put() that raise the execption | 21:48 |
| timburke | so the mode says that *no one* gets to read, write, or execute -- if i try the script on my system, i see something more like mode=0o100775 | 21:48 |
| guimaluf | and where this mode come from? | 21:48 |
| guimaluf | who sets this? mount time? xfs? | 21:50 |
| timburke | that's my question, too | 21:50 |
| guimaluf | everything works fine as root hehehe | 21:50 |
| guimaluf | the put() and write() | 21:50 |
| guimaluf | but I don't think I would like to run swift-object server as root | 21:51 |
| timburke | maybe try doing the print before the write(), too? just to spot-check that we didn't somehow manage to change perms there | 21:51 |
| timburke | heh, yeah -- that tends to lead to accidentally full root drives... | 21:51 |
| guimaluf | timburke, same output, only difference: st_size=0 | 21:53 |
| guimaluf | same mode | 21:53 |
| openstackgerrit | Merged openstack/swift master: middlewares: Clean up app iters better https://review.opendev.org/700959 | 21:53 |
| timburke | i'm confused about why https://github.com/openstack/swift/blob/master/swift/obj/diskfile.py#L1764 doesn't *also* raise EACCES | 21:53 |
| guimaluf | I would love to help more, but I'm really with my hands tie... | 21:55 |
| *** rcernin has joined #openstack-swift | 21:56 | |
| timburke | guimaluf, when you said your umask was set to 0022, was that as your swift user? or as the user running your `sudo -u swift ...` commands? | 22:05 |
| guimaluf | timburke, was with `sudo -u swift /bin/bash; umask -l` | 22:06 |
| guimaluf | without the -l heheh | 22:06 |
| timburke | try `sudo -u swift bash -c umask` | 22:06 |
| guimaluf | the same | 22:07 |
| guimaluf | 0022 | 22:07 |
| timburke | just double checking :-) | 22:07 |
| guimaluf | sure :) ask whatever you want I'm here to help | 22:08 |
| guimaluf | and to be helped hehehe | 22:08 |
| timburke | any chance you're running with selinux enabled? | 22:09 |
| guimaluf | no | 22:10 |
| guimaluf | Command 'getenforce' not found, but can be installed with: | 22:10 |
| guimaluf | Command 'sestatus' not found | 22:10 |
| timburke | 👍 | 22:10 |
| *** jvisser has quit IRC | 22:22 | |
| timburke | maybe time to try running `strace -e trace=%file` on our repro-script? there's going to be a bunch of extra info to dig through, though | 22:27 |
| guimaluf | that's getting interesting... xD | 22:27 |
| guimaluf | timburke, http://paste.openstack.org/show/790278/ | 22:32 |
| guimaluf | timburke, uuu not the full file there... :/ | 22:33 |
| timburke | bah, truncated. just as well, though -- i realized i needed *even more* info -- we want a -f on that strace, since we do the writes in pthreads | 22:33 |
| *** ormandj has joined #openstack-swift | 22:34 | |
| ormandj | anybody know if WORM/immutability is in the near/mid-term plans? : https://blueprints.launchpad.net/swift/+spec/immutability-middleware | 22:34 |
| ormandj | we're seeing a lot more requests from this from legitimate need/users (not just people that have read about it in an industry blog) | 22:35 |
| guimaluf | timburke, https://gist.github.com/guimaluf/a4f4bdbbae20df5edd78a0c861e282fa | 22:35 |
| ormandj | they even still want it even after discussing costs :) | 22:35 |
| *** tkajinam has joined #openstack-swift | 22:48 | |
| guimaluf | timburke, FYI I've just launched a new VM with ubuntu 18.04, dist-upgraded with the same kernel 4.15.0-88, mkfs.xfs with a disk and mount with same options. running as swift. I'm getting the error on `with df.create()` | 22:54 |
| guimaluf | ops, my bad... forgot to change permissions after mounting | 22:55 |
| guimaluf | timburke, the same scenario above, the script run smoothly: (mode=0o100755) | 22:56 |
| timburke | weird! | 22:56 |
| guimaluf | some issue related with the upgrade.... | 22:56 |
| guimaluf | I'll swap disks between vms | 22:56 |
| timburke | and maybe check `grep swift /etc/passwd` between the two? | 22:57 |
| timburke | and `mount | grep xvdf` maybe? | 22:58 |
| timburke | ormandj, i think the new object-versioning mode may get you close -- since it uses reserved names for the data, users can't normally poke at data after it's written; they have to go through the new api | 23:03 |
| timburke | might "just" be a matter of having a middleware that disables the put & delete w/ version_id api calls for protected containers? | 23:05 |
| guimaluf | timburke, I could write in the xfs filesystem when move the disk to the new vm... I'll try the opposite now | 23:07 |
| guimaluf | timburke, same error... so seems like is something strongly related to my system... very wierd... well, time to go... I'll be back tomorrow :) cya | 23:17 |
| *** guimaluf has quit IRC | 23:22 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!