Tuesday, 2018-07-31

*** rlandy has quit IRC00:31
*** swest has quit IRC01:10
*** swest has joined #zuul01:25
*** threestrands has quit IRC02:10
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver  https://review.openstack.org/53555702:48
openstackgerritTristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenShift resource provider  https://review.openstack.org/57066702:48
openstackgerritTristan Cacqueray proposed openstack-infra/zuul master: executor: add support for generic build resource  https://review.openstack.org/57066802:49
*** bhavik1 has joined #zuul03:12
*** threestrands has joined #zuul03:26
*** threestrands has quit IRC03:26
*** threestrands has joined #zuul03:26
*** lennyb has quit IRC04:08
*** yolanda has quit IRC04:09
*** lennyb has joined #zuul04:22
*** yolanda has joined #zuul04:24
*** bhavik1 has quit IRC04:48
*** nchakrab has joined #zuul05:02
*** nchakrab_ has joined #zuul05:04
*** nchakrab has quit IRC05:07
*** hwoarang has quit IRC05:07
*** hwoarang has joined #zuul05:09
*** nchakrab has joined #zuul05:11
*** nchakrab_ has quit IRC05:15
*** nchakrab has quit IRC05:41
*** nchakrab has joined #zuul05:59
*** nchakrab_ has joined #zuul06:03
*** nchakrab has quit IRC06:07
*** dtruong2 has joined #zuul06:09
*** dtruong2 has quit IRC06:20
*** pcaruana has joined #zuul06:37
*** hashar has joined #zuul07:23
*** electrofelix has joined #zuul07:40
*** threestrands has quit IRC07:45
*** jpena|off is now known as jpena07:53
*** threestrands has joined #zuul08:00
*** threestrands has quit IRC08:03
openstackgerritJoshua Hesketh proposed openstack-infra/zuul-jobs master: Some small improvements to zuul_swift_upload  https://review.openstack.org/58734508:49
openstackgerritJoshua Hesketh proposed openstack-infra/zuul-jobs master: Remove unused code path in zuul_swift_upload  https://review.openstack.org/58734608:49
openstackgerritJoshua Hesketh proposed openstack-infra/zuul-jobs master: Add --dry-run option to zuul_swift_upload  https://review.openstack.org/58734708:49
openstackgerritJoshua Hesketh proposed openstack-infra/zuul-jobs master: Enable options for generating the parent links  https://review.openstack.org/58734808:49
openstackgerritTobias Henkel proposed openstack-infra/zuul-jobs master: Allow predefined zuul_log_path during swift upload  https://review.openstack.org/58734908:55
tobiashjhesketh: added a comment on 58734509:00
*** sshnaidm|afk has quit IRC09:02
*** sshnaidm has joined #zuul09:04
*** jimi|ansible has quit IRC09:21
jheskethtobiash: thanks, I've repliedc09:37
*** panda|rover|off is now known as panda|rover09:48
*** electrofelix has quit IRC10:12
*** hashar is now known as hasharAway10:22
*** electrofelix has joined #zuul10:47
*** swest has quit IRC11:07
*** swest has joined #zuul11:21
*** jesusaur has quit IRC11:36
*** jesusaur has joined #zuul11:40
*** jpena is now known as jpena|lunch11:46
*** tobiash has quit IRC12:01
*** rcarrillocruz has quit IRC12:04
*** nchakrab_ has quit IRC12:17
*** panda|rover is now known as panda|rover|lunc12:18
*** dkranz has joined #zuul12:30
*** nchakrab has joined #zuul12:34
*** rlandy has joined #zuul12:35
*** elyezer has joined #zuul12:36
*** jpena|lunch is now known as jpena12:48
*** samccann has joined #zuul12:56
*** tobiash has joined #zuul12:57
*** dkranz has quit IRC13:09
*** elyezer has quit IRC13:27
*** samccann has quit IRC13:31
*** nchakrab_ has joined #zuul13:32
*** jimi|ansible has joined #zuul13:33
*** jimi|ansible has joined #zuul13:33
*** nchakrab has quit IRC13:35
*** myoung has joined #zuul13:37
*** samccann has joined #zuul13:39
*** elyezer has joined #zuul13:40
*** panda|rover|lunc is now known as panda|rover14:01
*** nchakrab_ has quit IRC14:08
*** nchakrab has joined #zuul14:09
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Enable options for generating the parent links  https://review.openstack.org/58734814:14
jheskethcorvus: heh, I was just adding that whitespace ;-)14:14
jheskeththanks14:14
*** elyezer has quit IRC14:15
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Swift logs: rename POST_RETRIES POST_ATTEMPTS  https://review.openstack.org/58747914:15
corvusjhesketh: oh you're still up!  hi!  :)14:16
corvusjhesketh, tobiash: ^ i think we can push that whole stack through now14:16
jheskethcorvus: I was just about to fix up the retries thing... do you want me to do that quickly or push through the stack first?14:17
*** samccann has quit IRC14:17
corvusjhesketh: i just renamed it in a followup patch; i say we ask tobiash to +3 the first patch and you to +2 the last one14:18
jheskethsure14:18
tobiashcorvus, mordred: looks like the latest openstacksdk release fixed my USE_SHADE issue :)14:18
corvustobiash: yep, and our initial tests using that are bearing fruit: https://object-storage-ca-ymq-1.vexxhost.net/swift/v1/86bbbcfa8ad043109d2d7af530225c72/logs_78/587178/1/check/tox-py35/e88b9ea/14:18
corvusjhesketh: ^ not sure if you saw this in -infra scrollback14:19
*** nchakrab has quit IRC14:19
corvus(probably so if you wrote the "../" patch :)14:19
*** nchakrab_ has joined #zuul14:19
pabelangerre: https://review.openstack.org/584488/ anybody have suggestions how I can best test indefinite launch retries? We don't seem to track retries in zk at all14:19
jheskethcorvus: I hadn't, but it's looking good :-)14:19
jheskethnah I had picked up on the series of nits I posted when I reviewed the earlier patch, but only just pushed them up14:20
corvusjhesketh: oh i'm sorry, i don't remember seeing those!14:20
jheskethcorvus: no, I didn't add them because they weren't blockers14:21
corvuswell, all good points :)14:21
jheskethwe'd spend forever nitpicking over what is a wip tool if I did ;-)14:21
jheskethI was just a bit slow on the followup sorry14:21
corvusthen i have successfully made room for collaboration :)14:21
jhesketh\o/14:22
corvuspabelanger: maybe you should try a white-box approach -- where you poke at the internal data structures of the requesthandler to find out how many attempts it has tried14:23
tobiashcorvus: yay :)14:23
*** dkranz has joined #zuul14:23
corvuspabelanger: (most of our testing is black-box, where we don't rely on knowledge of internal state, but occasionally we have to, and that's okay)14:24
tobiashcorvus: I have it in a base-test job in prod too now14:24
tobiashcorvus: what I noticed is that the x-delete-after seems to be totally ignored by our ceph-radosgw14:24
corvusmnaser: ^ know anything about that?14:25
tobiashcorvus: any objection about adding this info explicitly to the metadata of the object too?14:25
corvustobiash: i confess, i haven't tried it on vexxhost with a really small value.  i guess i should?14:25
tobiashcorvus: I know that ceph-radosgs doesn't have the automated cleanup but I hoped that I get this implicitly as metadata14:26
tobiashso I can run a cleanup script periodically14:26
corvustobiash: that's a thing?  oh, the metadata isn't for ceph, it's for you?14:26
tobiashbut that doesn't seem to be the case so I think I need to add that timestamp also to the metadata14:26
mordredtobiash: oh yuck - radosgw doesn't honor the expires? :(14:26
tobiashI found no x-delete-* header in the ceph docs14:27
tobiashso I'm pretty sure that there is no autodeleter14:27
tobiashso my way would be to run a maybe nightly job that does the cleanup14:28
tobiash(could be even part of zuul-jobs as well)14:28
pabelangercorvus: sure, let me see if I can access that data14:28
corvustobiash: can we set arbitrary metadata in the headers when we upload the object?14:28
mordredhttp://tracker.ceph.com/issues/409914:28
mordredfwiw14:29
corvusmordred, tobiash: is it possible it only supports x-delete-at rather than x-delete-after?14:29
corvusthat would be super easy to work around14:29
corvushttp://tracker.ceph.com/projects/ceph/repository/revisions/4f9a84310e847f10b6c3d0fc573e6dd06bc095c6/diff14:30
tobiashcorvus: I think the additional metadata is there as this is really part of the swift api14:30
corvustobiash: i think it would be fine to add metadata as long as it doesn't add an extra API call.  however, can you test whether x-delete-at works first?14:31
tobiashcorvus: yes, testing it14:31
*** nchakrab has joined #zuul14:31
corvus(it looks like the GC interval is 10 minutes, based on http://tracker.ceph.com/projects/ceph/repository/revisions/db27ea92494201e127d81498f5918834e04e3d02/diff )14:32
*** samccann has joined #zuul14:33
tobiashif that would work that would be awesome14:34
*** nchakrab_ has quit IRC14:35
*** elyezer has joined #zuul14:35
corvushrm, it also looks like it should support x-delete-after http://tracker.ceph.com/projects/ceph/repository/revisions/master/entry/src/rgw/rgw_rest_swift.cc#L76714:38
openstackgerritMerged openstack-infra/zuul-jobs master: Some small improvements to zuul_swift_upload  https://review.openstack.org/58734514:39
mordredcorvus: I don't see that code in the actual ceph repo14:39
corvushttp://tracker.ceph.com/projects/ceph/repository/revisions/fa347d8f69b8eff2e246d35a127c4bfa5a50b5e0/diff14:39
mordredcorvus: https://github.com/ceph/ceph/search?q=DELETE_AFTER&unscoped_q=DELETE_AFTER14:39
tobiashcorvus: hrm, ok I'll retest with just a small number. I expected that I get any info in any header about the expiration date.14:39
corvusmordred: ceph uses submodules, maybe it's in those?14:39
corvusmordred: not a submodule: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_swift.cc14:41
openstackgerritMerged openstack-infra/zuul-jobs master: Remove unused code path in zuul_swift_upload  https://review.openstack.org/58734614:41
mordredcorvus: ah - there we go yeah: https://github.com/ceph/ceph/blob/master/src/rgw/rgw_rest_swift.cc#L77114:41
mordredI was apparently just bad at searching14:41
openstackgerritMerged openstack-infra/zuul-jobs master: Add --dry-run option to zuul_swift_upload  https://review.openstack.org/58734714:42
mordredhttp://lists.ceph.com/pipermail/ceph-users-ceph.com/2017-June/018367.html14:43
corvustobiash: are you basing your assertion that x-delete-after doesn't work on having tried it, or just reading the docs, or it not having returned the headers back to you when you performed a GET or HEAD?14:43
mordredat least one year ago someone else was having issues with ceph not expiring things14:43
tobiashcorvus: I'm based on reading the docs and getting no info about the expiration date14:44
mordredhttp://tracker.ceph.com/issues/2017714:44
tobiashso I'll try it now with 5 minutes and check later if it's still there14:44
mordred"14:44
mordredobjects are checked for expiration once per day. The start time for processing is configured by "rgw lifecycle work time", and defaults to midnight."14:44
tobiashmordred: so I have to check tomorrow probably14:45
mordredyes, it seems so14:45
corvustobiash: in that case, it's worth testing both delete-after and delete-at at the same time on two different objects :)14:45
tobiashcorvus: yah, it probably makes sense to do that manually instead of via zuul14:46
corvustobiash: you can run zuul_swift_upload from the cli; it will use your clouds.yaml file14:46
tobiashcorvus: I know, I just deleted that from the workdir an hour ago ;)14:47
tobiashhave to set it up again first14:47
*** elyezer has quit IRC14:48
*** elyezer has joined #zuul14:49
tobiashok, uploaded the test fixtures once with delete-at and once with delete-after...14:54
tobiashlet's see tomorrow if that worked14:54
* mordred hopes14:54
tobiashoh, looks like delete-after is already gone14:55
tobiashdelete-at now too14:55
corvus\o/ easy resolution!14:55
mordredyay!14:56
openstackgerritTobias Henkel proposed openstack-infra/zuul-jobs master: Allow predefined zuul_log_path during swift upload  https://review.openstack.org/58734914:57
openstackgerritTobias Henkel proposed openstack-infra/zuul-jobs master: Add delete_after to swift upload role  https://review.openstack.org/58749314:57
tobiashso we should plumb that into a role var ^14:57
*** nchakrab_ has joined #zuul14:59
*** nchakrab has quit IRC15:02
*** pcaruana has quit IRC15:10
*** nchakrab_ has quit IRC15:32
*** sshnaidm is now known as sshnaidm|off15:36
*** elyezer has quit IRC15:36
*** elyezer has joined #zuul15:57
mnasercorvus: it should work, did it work with a small value?15:59
mnaserit is supported afaik and there is even tempest tests which test against that15:59
mnaserdid it end up working?16:00
clarkbmnaser: tobiash's tests were successful16:03
openstackgerritMerged openstack-infra/zuul-jobs master: Enable options for generating the parent links  https://review.openstack.org/58734816:04
openstackgerritMerged openstack-infra/zuul-jobs master: Swift logs: rename POST_RETRIES POST_ATTEMPTS  https://review.openstack.org/58747916:11
*** myoung is now known as myoung|lunch16:24
*** sdake has quit IRC16:33
corvusmnaser: yep, sorry, false alarm! :)16:33
*** sdake has joined #zuul16:33
*** sdake has joined #zuul16:33
*** dtruong2 has joined #zuul16:36
*** sshnaidm|off has quit IRC16:43
*** panda|rover is now known as panda|rover|off16:49
openstackgerritMerged openstack-infra/zuul-jobs master: Allow predefined zuul_log_path during swift upload  https://review.openstack.org/58734917:02
openstackgerritMerged openstack-infra/zuul-jobs master: Add delete_after to swift upload role  https://review.openstack.org/58749317:03
*** jpena is now known as jpena|off17:05
corvusmordred: i don't see that swift has support for symlinks.  there is a trick you can do with manifests to have an object return the content of a different existing object, which sort of approximates symlinked files, but i don't think that's going to work for symlinked directories (unless you also did it for every file in the target directory).17:07
corvusmordred: given that the use case you're presenting is a recursive symlink ( symlink -> . ), that may not be the best idea.  we could decide to do it for one level deep and stop, i guess.  but that's starting to feel weird.17:08
corvusmordred: i wonder if we should just change that to a copy instead of a symlink in the post playbook?17:09
corvusclarkb: are there other symlink use cases we should consider?  anything in devstack you know of?17:09
clarkbI think following symlinks and copying the "result" is probablu fine17:09
corvusclarkb: for files only, or also directories?17:09
clarkbI'm not sure if yhere are specific cases of one or the other17:10
clarkbbut if foo -> bar/ and bar/ has A and B in it we can copy those as foo/A foo/B ?17:10
corvusyeah, we could do that.  that works for files and directories.  what about recursive directory symlinks though?  copy one level or ignore them?17:11
clarkbfoo -> baz -bar/ ?17:11
clarkbhrm17:12
corvus(mutually recursive directory symlinks also need to be considered, of course: a->b b->a)17:12
corvus(projecting symlinks onto a system that doesn't support them is hard)17:13
clarkbfoo -> baz -> bar/ would still be foo/A and foo/B I think17:14
clarkbfoo -> baz/C -> bar becomes foo/C/A ?17:14
clarkbor do we flatten flattening may be the easiest consistent thing to do17:15
clarkbI want to say that is how os.walk returns them too17:15
tobiashhttps://specs.openstack.org/openstack/swift-specs/specs/in_progress/symlinks.html17:16
tobiashThat's probably unimplemented?17:16
corvusi'm assuming so.  i can't find any reference to it in the docs17:17
tobiashIs there a need for symlinks?17:18
mordredtobiash: zuul-build-dashboard-multi-tenant uses them17:19
corvustobiash: we have one use case in the zuul-build-dashboard-multi-tenant job.  but i think that can be replaced with copying files.  i'm unsure if there are others.17:19
mordredcorvus: I agree re: copying17:19
clarkbcorvus: I think loops we ignore (just delete them or add a file at the first "entrypoint" to loop that says "this is a symlink loop, not copying") then for everything else do what os.walk does and return files relative to the logs/ dir and not their abspaths17:20
clarkbthat way the tree looks as presented to zuul when uploaded17:20
corvusclarkb: os.walk doesn't descend into symlink targets which are directories17:21
clarkbcorvus: my local testing shows it follows them17:24
tobiashIt can but it cannot detect loops17:24
corvusBy default, walk() will not walk down into symbolic links that resolve to directories. Set followlinks to True to visit directories pointed to by symlinks, on systems that support them.17:24
corvushttps://docs.python.org/2/library/os.html17:24
clarkbyup, I'm saying we should do the followlinks=True behavior then check for loops like zuul does17:25
corvusclarkb: it will return the symlink as a directory, but won't descend into it17:25
clarkbhttp://paste.openstack.org/show/726961/ we would copy ./testing_dir/bar2 and ./foo/baz/bar217:26
corvusclarkb: so basically copy the result, unless we end up looping, in which case we just stop descending?17:26
clarkbcorvus: ya I think if we detect a loop we can add a file with "symlink loop detected" at that point17:26
clarkband stop looping17:26
mordredyeah - and in zuul-build-dashboard-multi-tenant - we can switch the multi.yaml playbook to do copying itself instead of creating symlinks17:27
corvusokay.  i agree that's sensible.  however, i don't believe we have any current use cases which would benefit from that, so i'm tempted to leave that for someone else to implement.  :)17:28
corvusi can write that up as a TODO17:29
corvusclarkb: where's the os.walk loop detection in zuul you were thinking of?17:31
*** gchenuet has joined #zuul17:32
corvusclarkb: oh, it's the ansible path stuff.  got it17:32
clarkbcorvus: zuul.ansible.paths._is_safe_path()17:32
corvusokay, i'll just implement this.  the TODO comment is nearly as long as the implementation anyway.  :)17:32
gchenuetHi guys ! I'm working on TLS and Gearman with ansible-role-zule. But i faced some `[SSL: WRONG_VERSION_NUMBER] wrong version number` on gearman server log.17:33
openstackgerritMonty Taylor proposed openstack-infra/zuul master: Use copy instead of symlink for multi-tenant dashboard  https://review.openstack.org/58755417:33
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header  https://review.openstack.org/55794717:33
gchenuetI'm using self-signed cert17:33
gchenuetAm i missing something ?17:33
pabelangerclarkb: corvus: mordred: a little late, but updated ^ to add more nodepool information for emit-job-header17:34
*** myoung|lunch is now known as myoung17:34
pabelangergchenuet: it should work, we use SSL cert for gate jobs in windmill.17:35
pabelangergchenuet: https://git.zuul-ci.org/cgit/zuul/tree/tests/fixtures/gearman/README.rst might help17:36
gchenuetshould i need to used the openstack-gearman-role to setup gearman or only the zuul-role is good enough ?17:37
gchenuetthanks for the link !17:38
gchenuetWe used the Gearman SSL page to generate our cert: http://gearman.info/gearmand/ssl.html17:38
pabelangerdon't need ansible-role-gearman, that is only if you want a stand alone gearman server17:38
gchenuetok ! I'm trying your link !17:39
gchenuetthanks a lot !17:39
pabelangermake sure you have gearman_server.start=true in zuul.conf17:39
pabelangeralong with proper SSL certs17:39
*** gchenuet has quit IRC17:40
corvusit's more complicated than the code that's in the ansible stuff (since it needs to actually handle recursion).  i'm going to leave a note and call it done.17:47
corvus(the ansible path checks short-circuits if it's seen a directory already, but it's not a stack, it's just a list, so if we applied the same algorithm, we'd get different results based on ordering)17:49
clarkbcorvus: we would still short circuit though right? just maybe not at the same location every job run?17:51
corvusclarkb: yeah.  we'd end up randomly picking one directory to upload if it is the target of multiple symlinks17:53
clarkbI think that sort is stable based on locale fwiw17:53
corvus/s/randomly/arbitrarily/17:54
clarkbbut ya that won't fix the case of mordred needing multiple copies of the same thing17:56
mordredclarkb: https://review.openstack.org/587554 has fix for that17:57
clarkbfor the logs case I think that behavior is sufficient for the artifacts case it isn't. hrm17:59
clarkbthe arbitrary selection not mordreds fix17:59
corvusclarkb: what's the artifacts case?17:59
openstackgerritPaul Belanger proposed openstack-infra/zuul-jobs master: Add nodepool info to test-emit-job-header  https://review.openstack.org/55794718:00
clarkbcorvus: one example is mordreds symlink case I think, its the same files shared across multiple dirs for different tenants? but also say you tag a release and build docs for that but also want to update the latest docs publications you might symlink in the job to accomplish that18:00
clarkbcorvus: all of these cases can be solved job side by using a copy or hardlink instead of a symlink18:00
*** electrofelix has quit IRC18:07
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Swift logs: don't allow links outside of the supplied path  https://review.openstack.org/58758018:19
corvusclarkb, mordred, tobiash: ^ that closes a symlink vulnerability and adds a test for the current behavior18:19
openstackgerritJames E. Blair proposed openstack-infra/zuul-jobs master: Swift logs: don't allow links outside of the supplied path  https://review.openstack.org/58758018:20
*** dkranz has quit IRC20:03
*** samccann has quit IRC20:28
openstackgerritTobias Henkel proposed openstack-infra/zuul-jobs master: Compress svg graphics during swift upload  https://review.openstack.org/58761620:43
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped autohold, enqueue (change, ref)  https://review.openstack.org/57690720:44
tobiashcorvus: I have some big svg images (various system stats) and compressing would change them from 1.1M to 75K, so I suggest to also compress svg images ^20:45
corvus++21:02
*** yolanda has quit IRC21:25
*** myoung has quit IRC21:28
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped, JWT-protected actions  https://review.openstack.org/57690721:34
openstackgerritMatthieu Huin proposed openstack-infra/zuul master: web: add tenant-scoped, JWT-protected actions  https://review.openstack.org/57690721:58
openstackgerritMerged openstack-infra/zuul-jobs master: Compress svg graphics during swift upload  https://review.openstack.org/58761622:55
*** hasharAway has quit IRC23:06
*** threestrands has joined #zuul23:24

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!