| timburke | clayg, oh right! of *course* symlink's fine with the override footer -- it's zero bytes and doesn't *use* footers! looks like it broke slo, though: https://logs.opendev.org/51/675451/1/check/swift-tox-func-ec/8401cd9/job-output.txt.gz#_2019-08-08_23_01_33_282857 | 00:00 |
|---|---|---|
| *** gyee has quit IRC | 00:43 | |
| *** spsurya has joined #openstack-swift | 01:05 | |
| mattoliverau | belated morning :) | 01:08 |
| mattoliverau | clayg, timburke: yeah, symlinks feel close to posix symlinks, but suddendly hardlink's aren't the same thing.. so makes me now really like the name. static link is _much_ better. | 01:09 |
| *** baojg has joined #openstack-swift | 01:23 | |
| *** altlogbot_0 has quit IRC | 02:44 | |
| *** altlogbot_2 has joined #openstack-swift | 02:46 | |
| *** redrobot has quit IRC | 03:22 | |
| *** zaitcev has quit IRC | 03:27 | |
| *** psachin has joined #openstack-swift | 03:36 | |
| *** zaitcev has joined #openstack-swift | 03:39 | |
| *** ChanServ sets mode: +v zaitcev | 03:39 | |
| *** baojg has quit IRC | 04:03 | |
| *** zaitcev_ has joined #openstack-swift | 04:38 | |
| *** ChanServ sets mode: +v zaitcev_ | 04:38 | |
| *** zaitcev has quit IRC | 04:42 | |
| *** ianychoi_ has quit IRC | 04:44 | |
| *** baojg has joined #openstack-swift | 04:44 | |
| *** [diablo]9 has quit IRC | 05:25 | |
| *** zaitcev__ has joined #openstack-swift | 05:27 | |
| *** ChanServ sets mode: +v zaitcev__ | 05:27 | |
| *** zaitcev_ has quit IRC | 05:31 | |
| *** e0ne has joined #openstack-swift | 05:40 | |
| *** e0ne has quit IRC | 05:59 | |
| *** zaitcev__ has quit IRC | 06:47 | |
| *** zaitcev__ has joined #openstack-swift | 07:00 | |
| *** ChanServ sets mode: +v zaitcev__ | 07:00 | |
| *** tesseract has joined #openstack-swift | 07:04 | |
| *** takamatsu has joined #openstack-swift | 07:35 | |
| *** zaitcev__ has quit IRC | 07:43 | |
| *** rcernin has quit IRC | 07:46 | |
| *** zaitcev__ has joined #openstack-swift | 07:55 | |
| *** ChanServ sets mode: +v zaitcev__ | 07:55 | |
| *** psachin has quit IRC | 08:07 | |
| *** mikecmpbll has joined #openstack-swift | 08:07 | |
| *** psachin has joined #openstack-swift | 08:08 | |
| *** zaitcev_ has joined #openstack-swift | 08:16 | |
| *** ChanServ sets mode: +v zaitcev_ | 08:16 | |
| openstackgerrit | Merged openstack/swift master: py3: Add swift-tox-func-encryption-py37 gate job https://review.opendev.org/674878 | 08:19 |
| *** zaitcev__ has quit IRC | 08:19 | |
| *** jistr is now known as jistr|afk | 08:24 | |
| *** e0ne has joined #openstack-swift | 09:24 | |
| *** tesseract has quit IRC | 09:28 | |
| *** tesseract has joined #openstack-swift | 09:28 | |
| *** tesseract has quit IRC | 09:29 | |
| *** tesseract has joined #openstack-swift | 09:29 | |
| *** zaitcev_ has quit IRC | 09:33 | |
| *** zaitcev_ has joined #openstack-swift | 09:47 | |
| *** ChanServ sets mode: +v zaitcev_ | 09:47 | |
| *** hoonetorg has quit IRC | 09:57 | |
| *** jistr|afk is now known as jistr | 10:18 | |
| *** hoonetorg has joined #openstack-swift | 10:20 | |
| viks___ | hi, i upgraded from queens to stein, but i see object server service cpu usage is around 40% even if no load is there? Anyone have any idea why this is happening? Not sure what % of cpu earlier for object server process i.e. in queens. Also my overall cpu usage(user) of storage node is around 80% after upgrade, Earlier it was around 45%. | 10:51 |
| *** tdasilva has joined #openstack-swift | 11:34 | |
| *** ChanServ sets mode: +v tdasilva | 11:34 | |
| DHE | can you identify the specific process consuming the CPU? | 12:36 |
| *** zaitcev_ has quit IRC | 12:43 | |
| viks___ | object server | 12:52 |
| *** zaitcev_ has joined #openstack-swift | 12:58 | |
| *** ChanServ sets mode: +v zaitcev_ | 12:58 | |
| *** henriqueof has joined #openstack-swift | 12:58 | |
| *** Guest1857 has joined #openstack-swift | 12:59 | |
| *** zaitcev_ has quit IRC | 13:19 | |
| *** BjoernT has joined #openstack-swift | 13:28 | |
| *** jistr is now known as jistr|call | 13:29 | |
| *** zaitcev_ has joined #openstack-swift | 13:32 | |
| *** ChanServ sets mode: +v zaitcev_ | 13:32 | |
| *** zaitcev__ has joined #openstack-swift | 13:33 | |
| *** ChanServ sets mode: +v zaitcev__ | 13:33 | |
| *** zaitcev_ has quit IRC | 13:37 | |
| *** jistr|call is now known as jistr | 13:38 | |
| *** tdasilva has quit IRC | 13:44 | |
| *** tdasilva has joined #openstack-swift | 13:45 | |
| *** ChanServ sets mode: +v tdasilva | 13:45 | |
| viks___ | Is it safe to upgrade from one version to another version keeping the conf files intact? | 13:49 |
| DHE | generally yes, but do read the changelog for anything you should be aware of | 13:53 |
| *** spsurya has quit IRC | 13:54 | |
| *** zaitcev__ is now known as zaitcev | 14:14 | |
| *** psachin has quit IRC | 14:16 | |
| *** zaitcev has quit IRC | 14:21 | |
| *** zaitcev has joined #openstack-swift | 14:35 | |
| *** ChanServ sets mode: +v zaitcev | 14:35 | |
| viks___ | ok | 14:49 |
| clayg | viks___: keep an eye on it, it might go down | 15:01 |
| clayg | Mostly the work done by the object-server is related to client traffic - so you wouldn't expect a resource change unless it's related to a change in your traffic patterns | 15:02 |
| clayg | however, unless you're running seperate dedicated replication object-server processes - they're also involved in REPLICATE requests | 15:02 |
| clayg | and those requests can take up cpu time (although really it's just cpu time waiting on os calls talking to disks) | 15:03 |
| clayg | I don't remember the exact time frames but there were some bug fixes and optimizations made to hash recalculation (REPLICATE requests) that might cause a temporary (?) spike in hash recalculation of old hash files... maybe... | 15:05 |
| clayg | depending on the size of your cluster is could go away after a few days or a week - but if keeps up, it'd be interesting to me at least to know more about what's going on | 15:06 |
| *** zaitcev has quit IRC | 15:27 | |
| *** zaitcev has joined #openstack-swift | 15:40 | |
| *** ChanServ sets mode: +v zaitcev | 15:40 | |
| *** e0ne has quit IRC | 15:44 | |
| *** gyee has joined #openstack-swift | 15:50 | |
| *** mikecmpbll has quit IRC | 16:03 | |
| *** guimaluf has quit IRC | 16:09 | |
| *** henriqueof has quit IRC | 16:14 | |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Consolidate Container-Update-Override headers https://review.opendev.org/675451 | 16:14 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: WIP: Use Sysmeta Container Override in footers https://review.opendev.org/675657 | 16:14 |
| *** Guest1857 is now known as redrobot | 16:17 | |
| *** tdasilva has quit IRC | 16:24 | |
| clayg | mattoliverau: timburke: I'm going to try redoing the docs with defining a "Static Symlink" | 16:29 |
| clayg | oh this is great, I'm even taking to refering to "dynamic Symlinks" to draw contrast where appropriate | 16:32 |
| openstackgerrit | Clay Gerrard proposed openstack/swift master: Allow "harder" symlinks https://review.opendev.org/633094 | 16:43 |
| clayg | hopefully if Zuul goes for it p 675451 will be non-controversial | 16:44 |
| patchbot | https://review.opendev.org/#/c/675451/ - swift - Consolidate Container-Update-Override headers - 2 patch sets | 16:44 |
| clayg | I think the docs on p 633094 have a much better shot of being understandable with static vs. dynamic symlinks (cc mattoliverau) | 16:45 |
| patchbot | https://review.opendev.org/#/c/633094/ - swift - Allow "harder" symlinks - 25 patch sets | 16:45 |
| *** csmart has quit IRC | 16:45 | |
| timburke | 👍 | 16:45 |
| *** irclogbot_1 has quit IRC | 16:47 | |
| *** irclogbot_1 has joined #openstack-swift | 16:50 | |
| *** zaitcev has quit IRC | 16:56 | |
| *** csmart has joined #openstack-swift | 16:56 | |
| viks___ | clayg: Thanks.. Not running dedicated replication object-server processes(i think you meant replication_server = false under [app:object-server]).. cluster size is not so big.. it's 2 storage nodes each having 9 object storage disks occupying around 25Gb of disk usage | 16:58 |
| *** mikecmpbll has joined #openstack-swift | 16:59 | |
| *** tesseract has quit IRC | 17:00 | |
| *** zaitcev has joined #openstack-swift | 17:07 | |
| *** ChanServ sets mode: +v zaitcev | 17:07 | |
| openstackgerrit | Merged openstack/swift master: slo: Add X-Manifest-Etag to responses https://review.opendev.org/675469 | 17:21 |
| clayg | viks___: well all of that sounds fine except for the inrecased cpu load then 🤣 | 17:25 |
| clayg | viks___: you could stop the object-replicators and see if cpu changes - that might hint at something... or maybe see how many worker processes is being spawned? I think the default is "auto" but that's probably not a recent change... | 17:26 |
| viks___ | clayg: 4 obkect-server workers because of 4 CPU s.. each process has 21 threads including main thread.. I tried changing reducing and increasing no. of threads but CPU usage is around 45%.. if I increase workers, per CPU usage reduces but overall CPU usage remains around 80%.. earlier overall CPU usage itself was around 45%.. | 17:37 |
| *** mikecmpbll has quit IRC | 17:58 | |
| *** mikecmpb_ has joined #openstack-swift | 17:59 | |
| clayg | timburke: I'm thinking about the tempest tests and p 633857 | 18:01 |
| patchbot | https://review.opendev.org/#/c/633857/ - swift - symlink-backed versioned_writes - 10 patch sets | 18:01 |
| clayg | I feel like the right answer is to go ahead and make use_symlinks default to False? But I'd like the versioned_writes test work in at least most of the test enviornments 🤔 | 18:01 |
| clayg | At first I thought if it defaults to false, but I enabled it etc/proxy-server.conf (like we'd want all new clusters to do) then the tempest test would pass and the functional environments we configure on our own would work... but I'm not sure how all of our test environments are configured | 18:02 |
| clayg | like... do the dsvm deployments use our example config - or they just pass because we were defaulting to on? | 18:03 |
| clayg | i don't think I can very easily make tests work with *either* config (basicaly bifricate all the assertions that the diff updates) | 18:03 |
| clayg | worse if we have both branches, we'd probably want to run both configs in the gate, and that seems like a lot of overhead for maintaining tests... seems like a not great situation... | 18:04 |
| clayg | not obvious to me at all which direction I should be going | 18:04 |
| clayg | the threads are the eventlet tpool of 20 and the main thread | 18:05 |
| clayg | viks___: ^ the 4 workers sounds right | 18:05 |
| clayg | viks___: I guess I'm still guessing it's somehow related to replication somehow (temporarily stopping replicators would test hypothosis) | 18:06 |
| clayg | viks___: I don't *remember* merging any changes that said "make object-server half as efficient" 🤣 | 18:06 |
| clayg | maybe less tonge in cheek, we monitor that sort of thing in our clusters and haven't seen any other complaints - but we wouldn't have upgraded directly from Q to S so it could be a frog in water effect 🤔 | 18:08 |
| clayg | viks___: try upgrading to an even NEWER swift!!! new swift is best. | 18:08 |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: py3: Fix testUtf8Container (test.functional.tests.TestContainer) https://review.opendev.org/675244 | 18:20 |
| clayg | maybe I could mostly get the old tests passing on either config... I might even be able to minimize overhead with an `_assertContainerCount(n)` that says `if not self.use_symlinks: n -= 1` 🙄 | 18:22 |
| viks___ | clayg: Ok.. I'll try to investigate by stopping the replicators maybe on Monday, and update here my observations.. | 18:24 |
| clayg | viks___: thanks! | 18:28 |
| clayg | GL | 18:28 |
| timburke | clayg, it might be reasonable to have some func tests running use_symlinks=true and some use_symlinks=false... we already have a bunch of overlap between, say, swift-tox-func and swift-tox-func-encryption -- we could just pick one and say, "ok, you get symlinks" | 19:29 |
| timburke | i'm kinda amazed that swift-multinode-rolling-upgrade passes on that patch... | 19:29 |
| clayg | Well, that runs old code old func new code new func yeah? | 19:30 |
| clayg | I think with enough work I can get most of the existing tests passing on both. And the new tests can Skip 🤷♂️ | 19:31 |
| clayg | Then I just default false and try to get some env to run with use_symlinks - does that sound like a reasonable direction? | 19:31 |
| *** e0ne has joined #openstack-swift | 19:41 | |
| *** BjoernT has quit IRC | 19:44 | |
| timburke | oh! right. it's new code for storage, old code for proxy, old func tests. so proxy doesn't support the new stuff and it's fine | 19:45 |
| timburke | yeah, sounds reasonable | 19:45 |
| clayg | ha, found a bug with use_symlinks = false already! 😢 | 19:53 |
| openstackgerrit | Pete Zaitcev proposed openstack/swift master: Allow Connection: value to be case-insensitive https://review.opendev.org/675696 | 20:07 |
| *** diablo_rojo has joined #openstack-swift | 20:16 | |
| *** e0ne has quit IRC | 20:18 | |
| *** diablo_rojo has quit IRC | 20:51 | |
| clayg | timburke: ok, fixed that bug - i'll need to keep working through tests to see what else pops up | 21:01 |
| zaitcev | Ugh | 21:21 |
| *** e0ne has joined #openstack-swift | 21:21 | |
| zaitcev | We have a functest that does this: sends PUT with Transfer-Encoding: gzip,chunked but omits Content-Length. It expects 501. | 21:22 |
| zaitcev | Guess what | 21:22 |
| zaitcev | Apache LB turns it around with 500 before it hits Swift proxy. | 21:23 |
| zaitcev | At this point I stopped Apache and returned to stunnel to terminate SSL. | 21:23 |
| timburke | whee! | 21:26 |
| timburke | yeah, personally i much prefer an ssl terminator that *just* terminates | 21:27 |
| timburke | maybe https://github.com/openstack/swift/blob/2.22.0/test/functional/__init__.py#L822 would be useful? | 21:30 |
| timburke | i know we use it in some places like https://github.com/openstack/swift/blob/2.20.0/test/functional/tests.py#L2527-L2529 (though that skip message seems to be exactly wrong...) | 21:30 |
| *** e0ne has quit IRC | 21:32 | |
| *** e0ne has joined #openstack-swift | 21:36 | |
| *** e0ne has quit IRC | 21:45 | |
| zaitcev | Aug 9 18:08:14 rhev-a24c-01 object-server[29481]: STDERR: Exception ignored in: | 22:09 |
| zaitcev | Aug 9 18:08:14 rhev-a24c-01 object-server[29481]: STDERR: <function _after_fork at 0x7fb2975e9620> | 22:09 |
| *** mikecmpb_ has quit IRC | 22:09 | |
| *** mikecmpbll has joined #openstack-swift | 22:14 | |
| zaitcev | Seems working fine though. | 22:15 |
| timburke | zaitcev, any more context to that? like what the exception was, or maybe even better, the stack? | 22:22 |
| zaitcev | No, that's all there is to it. Every server spews it on start, just once. | 22:23 |
| *** mikecmpbll has quit IRC | 22:24 | |
| zaitcev | If you didn't see it, it's probably Fedora specific | 22:24 |
| zaitcev | Functests pass | 22:25 |
| zaitcev | Except unittest2.loader._FailedTest.test.functional.s3api, which has "ImportError: No module named boto3" | 22:26 |
| zaitcev | I don't know where to get that module. | 22:26 |
| *** mikecmpbll has joined #openstack-swift | 22:27 | |
| *** mikecmpb_ has joined #openstack-swift | 22:31 | |
| *** mikecmpbll has quit IRC | 22:32 | |
| timburke | https://sjc.edge.kernel.org/fedora-buffet/fedora/linux/releases/30/Everything/x86_64/os/Packages/p/python3-boto3-1.9.101-1.fc30.noarch.rpm won't do? | 22:40 |
| zaitcev | Hmm. | 22:53 |
| zaitcev | I forgot that the _client_ is actually on RHEL 7.5. Sorry that I misled you. It's the py3 server that's on F30. | 22:55 |
| timburke | ah, gotcha | 22:56 |
| timburke | there's always pip... | 22:56 |
| zaitcev | Yeah. | 22:56 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: port test/functional/test_versioned_writes.py https://review.opendev.org/675710 | 23:00 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: port test/functional/test_versioned_writes.py https://review.opendev.org/675710 | 23:01 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: mostly port s3 func tests https://review.opendev.org/674716 | 23:10 |
| openstackgerrit | Tim Burke proposed openstack/swift master: py3: Finish porting s3 func tests https://review.opendev.org/675227 | 23:10 |
| *** zaitcev has quit IRC | 23:24 | |
| *** zaitcev has joined #openstack-swift | 23:36 | |
| *** ChanServ sets mode: +v zaitcev | 23:36 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!