| *** mattw4 has quit IRC | 00:08 | |
| *** jamesmcarthur has joined #zuul | 00:14 | |
| *** jamesmcarthur has quit IRC | 00:19 | |
| *** jamesmcarthur has joined #zuul | 00:24 | |
| *** jamesmcarthur has quit IRC | 00:29 | |
| tristanC | mordred: SF does deploy zuul using distro packages, and for executor ansible version, it's using a fake venv so that all the ansibles share the system libraries | 01:06 |
|---|---|---|
| clarkb | tristanC: does that mean changing the virtualenv version won't affect you? | 01:07 |
| tristanC | clarkb: it should not | 01:08 |
| tristanC | here is how the sake venv is assembled: https://softwarefactory-project.io/cgit/rpms/zuul-executor-ansible-29/tree/zuul-executor-ansible-29.spec#n57 | 01:08 |
| tristanC | fake* venv, turns out python looks for a `pyvenv.cfg` file, that when provided, satisfies the venv property | 01:09 |
| fungi | neat | 01:12 |
| fungi | i never knew that | 01:12 |
| tristanC | fungi: well, it's not very documented... it may break in future version | 01:13 |
| tristanC | clarkb: more generally, SF is not affected by updates coming from pypi, all the requirements are packaged and their update are gated by zuul | 01:16 |
| clarkb | ok, in this case I guess you hvae to worry about virtualenv changing its behavior | 01:18 |
| clarkb | but you'd control that on your side | 01:18 |
| *** rfolco has quit IRC | 01:19 | |
| tristanC | clarkb: yup, though it's worrysome that doing `pip install zuul` may results in a non working zuul until the requirements cap is tagged and available through pypi... | 01:20 |
| clarkb | tristanC: there is a fix up | 01:20 |
| clarkb | (I think we should probably prefer to tag that over the cap) | 01:21 |
| tristanC | clarkb: i meant, maybe we could freeze the requirements, at least when doing a release. 3.15.0 seems to be currently affected by the cheroot issue too | 01:24 |
| clarkb | tristanC: that has the opposite issue | 01:25 |
| clarkb | security updates won't be delivered | 01:25 |
| clarkb | also if someone deletes a release on pypi you are broken | 01:25 |
| webknjaz | @fungi @clarkb: FYI Bernat did a nice overview of how various venvs work last summer — https://www.youtube.com/watch?v=o1Vue9CWRxU | 01:30 |
| tristanC | clarkb: well with service like dependabot, it should be possible to update the requirements freeze when needed. but you raised good point, it's hard to tell what is the best strategy for the end-user. | 01:31 |
| *** wxy-xiyuan has joined #zuul | 01:52 | |
| tristanC | clarkb: makes me wonder if we shouldn't deprecate pip installation and promote the container image as the official install method? At least until those recurring issue are affecting pip deployment. | 01:55 |
| tristanC | or perhaps we should indicate that pip user should use the master version because tagged release may need version cap to be usable today | 01:57 |
| tristanC | and for user who needs reproducable deployment, we could document how to gate the container image using zuul itself. | 01:58 |
| clarkb | another option could be a constraints file and manage the pins separately so they arent hard requirements | 01:58 |
| clarkb | then you can still install if a release on pypi is deleted | 01:59 |
| tristanC | clarkb: yes, as long as we can somehow control those surprise release effect | 02:02 |
| tristanC | so, could we have a zuul/requirements? should we draft a spec first? | 02:05 |
| *** jamesmcarthur has joined #zuul | 02:09 | |
| clarkb | Im not sure it needs to be a separate repo | 02:09 |
| *** rlandy has quit IRC | 02:09 | |
| *** zxiiro has quit IRC | 02:25 | |
| *** jamesmcarthur has quit IRC | 02:28 | |
| openstackgerrit | Steve Baker proposed zuul/zuul master: Scheduler: make autohold hold_list configurable https://review.opendev.org/632498 | 02:31 |
| *** jamesmcarthur has joined #zuul | 02:31 | |
| *** jamesmcarthur has quit IRC | 02:42 | |
| fungi | what if each time we tagged a release and/or merged a commit, one of the release artifacts was a constraints file indicating the exact versions of dependencies that commit was tested with? | 02:47 |
| fungi | so rather than having to track a constraints file in git and deal with reviewing auto-proposed updates like openstack does, we did it the other way around | 02:48 |
| clarkb | I like that | 02:48 |
| fungi | it could be terrible, just a random saké-fueled ponderance | 02:49 |
| *** bhavikdbavishi has joined #zuul | 03:12 | |
| *** fdegir has quit IRC | 03:14 | |
| *** fdegir has joined #zuul | 03:14 | |
| *** bhavikdbavishi1 has joined #zuul | 03:21 | |
| *** bhavikdbavishi has quit IRC | 03:23 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:23 | |
| *** sgw has joined #zuul | 03:38 | |
| ianw | clarkb / Shrews : so linaro-london got fixed, and everything seems to have just magically cleared up | 05:01 |
| ianw | i don't know what the moral of the story is ... possibly it's just if your cloud is broken you can expect weird things | 05:03 |
| fungi | including blocking some image cleanup on other providers for the same builder? | 05:22 |
| *** raukadah is now known as chkumar|rover | 05:28 | |
| *** evrardjp has quit IRC | 05:34 | |
| *** evrardjp has joined #zuul | 05:34 | |
| *** igordc has joined #zuul | 05:51 | |
| *** saneax has joined #zuul | 05:54 | |
| *** igordc has quit IRC | 06:00 | |
| *** sanjayu_ has joined #zuul | 06:01 | |
| *** saneax has quit IRC | 06:03 | |
| *** marvs has joined #zuul | 06:05 | |
| openstackgerrit | Merged zuul/zuul master: Install kubectl/oc into executor container image https://review.opendev.org/706995 | 06:46 |
| openstackgerrit | Felix Schmidt proposed zuul/zuul master: Implement basic github checks API workflow https://review.opendev.org/705168 | 07:37 |
| *** NBorg has joined #zuul | 07:49 | |
| NBorg | Hello. How do you return a link from gerrit to the buildset when a job is finished? | 07:54 |
| *** Defolos has joined #zuul | 07:55 | |
| *** sanjayu_ has quit IRC | 07:58 | |
| fungi | NBorg: i don't entirely understand your question. are you asking how to have zuul comment on a gerrit review with a link to the buildset instead of individual job status links? | 08:05 |
| tobiash | NBorg: do you mean the buildset or build page? Build page is configured in the tenant config: https://zuul-ci.org/docs/zuul/reference/tenants.html#attr-tenant.report-build-page | 08:05 |
| fungi | by default the gerrit reporter will leave a comment with links to the individual build results for each job which was run, per https://zuul-ci.org/docs/zuul/reference/drivers/gerrit.html#attr-pipeline.reporter.%3Cgerrit%20reporter%3E.comment | 08:06 |
| tobiash | NBorg: afaik linking to the buildset page as a result is not possible atm | 08:08 |
| NBorg | Yes, sorry, that was a bit unclear. I want the <web>/t/<tenant>/buildset/<buildset> to be a comment in gerrit. If possible, also the individual builds. | 08:08 |
| NBorg | Ah, ok. | 08:08 |
| NBorg | I'll look into the report-build-page. That should hopefully be good enough. Thanks! | 08:10 |
| *** avass has joined #zuul | 08:11 | |
| tobiash | NBorg: note that report-build-page currently has unexpected side effects on mqtt reporting if you use that (see https://review.opendev.org/703983 for a wip fix) | 08:11 |
| *** avass has quit IRC | 08:16 | |
| NBorg | We are currently not using that, so no worries. | 08:18 |
| *** tosky has joined #zuul | 08:27 | |
| *** carli has joined #zuul | 08:27 | |
| *** avass has joined #zuul | 08:29 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make bindep role compatible with newer virtualenv https://review.opendev.org/707078 | 08:30 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: Make bindep role compatible with newer virtualenv https://review.opendev.org/707078 | 08:39 |
| *** jpena|off is now known as jpena | 08:54 | |
| *** sanjayu_ has joined #zuul | 09:08 | |
| openstackgerrit | Merged zuul/zuul master: Extract allow/disallow filter into util function https://review.opendev.org/706144 | 09:21 |
| *** sshnaidm|afk is now known as sshnaidm | 09:25 | |
| *** mhu has joined #zuul | 09:29 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 09:30 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Authorization rules: add templating https://review.opendev.org/705193 | 09:30 |
| *** avass has quit IRC | 10:05 | |
| tobiash | corvus: it appears that we have a hole in testing gerrit with ssh access. This broke jobs that rely on ssh access to gerrit but passed testing: https://review.opendev.org/706827 | 10:17 |
| tobiash | corvus: do you have an idea how we could test this? | 10:17 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Try quickstart with gerrit and pure ssh https://review.opendev.org/707092 | 10:21 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Offload repo reset to processes https://review.opendev.org/706827 | 10:22 |
| *** bhavikdbavishi has quit IRC | 10:24 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: Offload repo reset fixed with ssh https://review.opendev.org/707095 | 10:33 |
| *** mhu has quit IRC | 11:00 | |
| *** bhavikdbavishi has joined #zuul | 11:09 | |
| *** bhavikdbavishi1 has joined #zuul | 11:12 | |
| *** bhavikdbavishi has quit IRC | 11:14 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 11:14 | |
| *** fungi has quit IRC | 11:24 | |
| openstackgerrit | Merged zuul/zuul master: JWT drivers: Deprecate RS256withJWKS, introduce OpenIDConnect https://review.opendev.org/701972 | 11:24 |
| *** wxy-xiyuan has quit IRC | 11:25 | |
| tobiash | corvus: the quickstart test change reveals the error on 706827 and works as expected on 707095 | 11:27 |
| *** fungi has joined #zuul | 11:27 | |
| tobiash | corvus: so the test case works, the question now is should we duplicate/modify quickstart to use ssh or do something completely different? | 11:28 |
| *** rfolco has joined #zuul | 12:02 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: WIP: Store unparsed branch config in Zookeeper https://review.opendev.org/705716 | 12:03 |
| *** dpawlik has joined #zuul | 12:34 | |
| *** jpena is now known as jpena|lunch | 12:38 | |
| *** sanjayu__ has joined #zuul | 12:39 | |
| *** sanjayu_ has quit IRC | 12:41 | |
| *** sanjayu_ has joined #zuul | 12:51 | |
| *** rlandy has joined #zuul | 12:51 | |
| *** sanjayu__ has quit IRC | 12:53 | |
| *** carli has quit IRC | 13:01 | |
| *** jamesmcarthur has joined #zuul | 13:15 | |
| *** jamesmcarthur has quit IRC | 13:24 | |
| *** jamesmcarthur has joined #zuul | 13:25 | |
| *** jamesmcarthur has quit IRC | 13:28 | |
| *** jamesmcarthur has joined #zuul | 13:28 | |
| *** jpena|lunch is now known as jpena | 13:32 | |
| *** jamesmcarthur has quit IRC | 13:35 | |
| mordred | tristanC, clarkb: I don't think we should overthink/overcomplicate the requirements situation just yet. while I agree 3.15.0 is broken, we've been working on a fix and as soon as the fix is landed we can cut a new release. because of the gate, we should notice pip install zuul breaking pretty quickly. that said - I do get the impression that the container quick-start is the main way people are getting started | 13:37 |
| mordred | anyway | 13:38 |
| *** jamesmcarthur has joined #zuul | 13:43 | |
| *** sgw has quit IRC | 13:53 | |
| pabelanger | I just got hit by virtualenv 2.0.0 break on centos-8: ERROR:root:ImportError: cannot import name 'ensure_text', that's just doing pip3 install tox | 13:57 |
| pabelanger | I know we are working on zuul fix, but is there upstream bug? | 13:57 |
| mordred | pabelanger: dunno about a bug. https://review.opendev.org/#/c/706871/2 is the fix and is winding its way through the gate | 13:59 |
| pabelanger | mordred: I wonder how we can expose that into bindep role, because that is where I see breakage | 14:01 |
| pabelanger | from pip task | 14:01 |
| mordred | pabelanger: as a workaround until it lands you can pin virtualenv to <20.0.0 - it's an issue with virtualenv now symlinking pip and setuptools and pkg_resources | 14:01 |
| pabelanger | https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/ansible_52/352/859f1378d11b913870dee1a4f99a59a89ef35d90/check/ansible-tox-linters/edfb3af/job-output.html#l376 | 14:01 |
| mordred | OH - yeah - that's much more of a pita | 14:01 |
| mordred | there was discussion of that in -infra - I haven't caught up with the latest thinking | 14:02 |
| mordred | it's a really impressively bad bug though :) | 14:03 |
| mordred | pabelanger: frickler says that issue goes away if you upgrade pip - it's a combo bug - happens with pip<10 and virtualenv>=20 | 14:04 |
| pabelanger | thanks, let me look at that. in centos-8 case, we use pip from distro | 14:04 |
| pabelanger | but, there doesn't seem to be tox rpm package in centos8 | 14:04 |
| pabelanger | so we pip install | 14:05 |
| pabelanger | which is how we open this can of worms | 14:05 |
| mordred | yeah- where is virtualenv coing from? | 14:05 |
| pabelanger | distro | 14:05 |
| pabelanger | but, when I pip install to | 14:05 |
| pabelanger | tox* | 14:05 |
| mordred | really? then why is virtualenv 20 even at play? | 14:05 |
| pabelanger | it now pulls 20.0.0 from pypi | 14:05 |
| pabelanger | not sure | 14:05 |
| pabelanger | looking at tox now | 14:05 |
| pabelanger | it is a mess for sure | 14:06 |
| mordred | pabelanger: I'm also looking at your logs ... it seems like we need to make sure if someone is using pip and virtualenv from distro that we don't pip install things globally | 14:06 |
| pabelanger | mordred: yes, that is part of the issue too, we pip3 install tox global, so have tox role work | 14:07 |
| mordred | pabelanger: this: https://object-storage-ca-ymq-1.vexxhost.net/v1/a0b4156a37f9453eb4ec7db5422272df/ansible_52/352/859f1378d11b913870dee1a4f99a59a89ef35d90/check/ansible-tox-linters/edfb3af/job-output.html#l379 is using pip installed virtualenv, not distro installed | 14:07 |
| pabelanger | which goes back to same issue last week of pip install tox --user not working | 14:07 |
| pabelanger | so, it is a series of things, that has lead to this failure in zuul.a.c jobs and only on centos-8 | 14:07 |
| mordred | jeez | 14:07 |
| *** mhu has joined #zuul | 14:08 | |
| pabelanger | mordred: yes, that is right but on centos-8 /bin/virtualenv exists, from package | 14:08 |
| mordred | pabelanger: I'm just curious why /usr/local/bin/virtualenv even exists- maybe there's a dib element installing latest virtualenv for you from pip and that's actually what you're using on centos? | 14:09 |
| pabelanger | I believe https://github.com/ansible-network/windmill-config/pull/599 is my fix on centos-8 | 14:10 |
| pabelanger | but, I also don't know why pip3 pulls in virtuelenv, if distro exists | 14:10 |
| pabelanger | I guess, something about matching globals pip installs | 14:10 |
| mordred | pabelanger: OH | 14:12 |
| mordred | pabelanger: it's because tox depends on virtualenv and you're pip installing tox | 14:12 |
| pabelanger | yes | 14:12 |
| pabelanger | we only do, because i cannot find tox RPM | 14:12 |
| *** Goneri has joined #zuul | 14:12 | |
| pabelanger | for centos-7, it is in epel | 14:12 |
| mordred | pip *never* tries to not install something via pip just if there is a distro version | 14:12 |
| mordred | pabelanger: yes - I think that element change is your best bet | 14:14 |
| mordred | pabelanger: you might want to, for now, since that's an image build - do a similar thing and put a "pip3 install virtualenv<20" into your base job with an "if platform == centos" thing on it | 14:14 |
| mordred | just to fix things while waiting on that image to build and upload | 14:15 |
| pabelanger | yah, good ide | 14:15 |
| mordred | pabelanger: I *think* you might need a -U for the base job ... if virtualenv is already installed, I don't think pip will care what your version constraint it unless you give is -U | 14:16 |
| pabelanger | ++ | 14:16 |
| pabelanger | long term, is fix ensure-tox to install into --user then remove from images | 14:17 |
| mordred | pabelanger: I think that ensure-tox fix then requires switching all tox jobs to calling {{ tox_executable }} right? | 14:28 |
| pabelanger | yah | 14:28 |
| mordred | I really wish there was a good story for adding ~/.local/bin to the path | 14:28 |
| openstackgerrit | Merged zuul/zuul master: Uncap virtualenv https://review.opendev.org/706871 | 14:28 |
| fungi | i really wish ansible didn't hate shells (or at least replicated their envvar initialization from files on the system) | 14:31 |
| *** Defolos has quit IRC | 14:38 | |
| *** chkumar|rover is now known as raukadah | 14:50 | |
| tristanC | is it me or using `lookup("file", ...)` in zuul job task doesn't work, it's failing with "An unhandled exception occurred while running the lookup plugin 'file'. Error was a <class 'TypeError'>, original message: expected str, bytes or os.PathLike object, not NoneType" | 14:53 |
| mordred | tristanC: oh good | 14:55 |
| tristanC | i also saw this failure mode: "'NoneType' object has no attribute 'startswith'" | 14:56 |
| *** sgw has joined #zuul | 14:56 | |
| *** michael-beaver has joined #zuul | 14:56 | |
| corvus | tristanC: is it possible the argument isn't a string? | 14:57 |
| mordred | corvus, tristanC: is the file in quesiton perhaps one that doesn't exist? | 14:57 |
| tristanC | corvus: the lookup works when running ansible-playbook locally | 14:57 |
| mordred | we do: lookupfile = self.find_file_in_search_path() | 14:57 |
| mordred | then paths._fail_if_unsafe(lookupfile) | 14:57 |
| mordred | perhaps lookupfile is coming back as None? | 14:57 |
| corvus | i have used lookup('file') recently with zuul master in the untrusted context | 14:57 |
| pabelanger | I didn't know you could use it in untrusted | 14:58 |
| corvus | mordred: sounds plausible | 14:58 |
| tristanC | corvus: is your lookup happening on localhost? the errors i'm seeing is when using it on the test instance | 14:59 |
| tristanC | remote* test instance | 14:59 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Protect fail if unsafe from None value https://review.opendev.org/707167 | 15:00 |
| mordred | tristanC: that's a *maybe* there ^^ but obviously unproven/untested | 15:00 |
| corvus | tristanC: i thought lookups always happened on localhost | 15:00 |
| mordred | yes, lookups always happen on localhost | 15:01 |
| corvus | that might confirm the missing-file theory | 15:01 |
| mordred | so - that makes me think even more that it's the file not found thing - if you're expecting to read a remote file, it's not gfoing to find it | 15:01 |
| tristanC | oh, then how am i supposed to lookup a remote file? should i keep on registering the output of cat? | 15:01 |
| mordred | yeah | 15:01 |
| corvus | tristanC: slurp is one option | 15:01 |
| pabelanger | tristanC: slurp | 15:01 |
| mordred | tristanC: slurp | 15:02 |
| pabelanger | :) | 15:02 |
| corvus | tristanC: (but be careful, you have to base64decode it) | 15:02 |
| mordred | just so I didn't not say it | 15:02 |
| tristanC | ohhh, thanks, seems like slurp is the solution. i didn't knew about it | 15:03 |
| tristanC | not sure why should i use it instead of cat command though :) | 15:06 |
| corvus | tobiash: i think i can live with the ssh testing hole -- that doesn't change that often, and we can try to live-test changes that touch it. but if we think we're going to make a lot of changes and we do need testing, then i guess we could add a copy of quickstart as a functional test. or we could deprecate ssh-only gerrit. | 15:07 |
| *** sanjayu_ has quit IRC | 15:09 | |
| *** zxiiro has joined #zuul | 15:10 | |
| tristanC | mordred: regarding requirements, i'm in favor of deprecating pip install until we have a future proof solution to the issues we keep on having when a requirement release a buggy/incompatible version | 15:11 |
| mordred | tristanC: there is no future proof | 15:11 |
| mordred | the pip developers will always find new ways to break us | 15:12 |
| mordred | and I do not want to do the pin everything method | 15:12 |
| tristanC | mordred: what's the issue with the constrain thing fungi and clarkb suggested yesterday? | 15:12 |
| mordred | I think it's a horrible experience | 15:12 |
| corvus | this is a python program. it sucks, but pip breakage just comes with the territory. but i don't think we should stop supporting the standard way of installing python programs. | 15:13 |
| tristanC | mordred: then what about documenting that pip install should use the master version, e.g. pip install https://opendev.org/zuul/zuul#egg=zuul | 15:13 |
| mordred | we either have to do something similar to dependabot (which is gross and misses security updates) - or we have a constraints file that nobody uses | 15:13 |
| mordred | tristanC: I don't think that's what people should do | 15:13 |
| mordred | I think they should pip install zuul | 15:13 |
| mordred | and something that will break | 15:13 |
| mordred | and when it does we'll fix it | 15:14 |
| corvus | s/something/sometimes/ ? | 15:14 |
| mordred | yes | 15:14 |
| fungi | tristanC: the constraint suggestion i had was not to constrain our tests, but to publish a constraints list documenting what we tested with | 15:14 |
| tristanC | mordred: then we have to cut a release with things that are not related | 15:14 |
| fungi | possibly in a more directly consumable format than a pip freeze dump buried in some build logs that is | 15:15 |
| mordred | fungi: yeah - but that's just the thing - the pip tooling doesn't really support us doing that | 15:15 |
| mordred | in any way that's consumable for people doiung "pip install zuul" | 15:15 |
| mordred | they can't do a "pip install --upstream-constraints zuul" | 15:15 |
| mordred | so I doubt anyone would figure out a decent way to use that file | 15:16 |
| fungi | right, it would at best be for people who know something broke and where to find that filter | 15:16 |
| webknjaz | pip has `pip install -c constraints.txt` | 15:16 |
| mordred | right. but constraints.txt file has to exist | 15:16 |
| mordred | if the constraints.txt file is inside of the release as a file | 15:16 |
| mordred | pip can't do anything with it | 15:16 |
| mordred | so we can't publish it in any automatically consumable manner | 15:17 |
| mordred | which means nobody is going to use it and it's a waste of time | 15:17 |
| fungi | yep, nobody will know where to find that file or the special command to use without rereading zuul's documentation | 15:17 |
| mordred | I mean - we do publish constraints files for openstack and still deployers don't use them | 15:17 |
| fungi | my idea was to publish it as an artifact alongside our wheels and tarballs, but then i forgot nobody consumes the versions we host in opendev because they're all just retrieving them from pypi anyway | 15:18 |
| fungi | or pip installing the source from git | 15:18 |
| mordred | yah- I mean, it's not a terrible idea if it was supported by tooling | 15:19 |
| mordred | ALTHOUGH - it still suffers from lack of security updates etc | 15:20 |
| mordred | which is why I don't like the dependabot version of this | 15:20 |
| mordred | the number of times we are broken by an upstream thing shifting int his manner that can't be solved trivially by a followup patch are pretty minimal - and I so I don't think we should invent or institute structures to deal with them | 15:20 |
| corvus | speaking of which, 871 merged, should we cut a release now? | 15:23 |
| *** jamesmcarthur has quit IRC | 15:23 | |
| mordred | corvus: yeah, I think that's likely a good idea | 15:25 |
| clarkb | I'm not super concerned about it myself. Maybe I'm weord but if pip installing you cant assume it will just work(tm) | 15:25 |
| corvus | should we do an opendev restart/burn-in, or just go ahead and tag? | 15:25 |
| *** jamesmcarthur has joined #zuul | 15:26 | |
| clarkb | I definitely dont want to prevent users from getting security updates so the hard freeze idea is a no go | 15:26 |
| corvus | look good? commit a9eafb572568ecd10d8cdbdc654f9d5bbeac7248 (HEAD -> master, tag: 3.16.0, origin/master, origin/HEAD) | 15:27 |
| corvus | mordred, clarkb, tristanC, tobiash, fungi: ^ release this tag without opendev restart? | 15:30 |
| corvus | https://zuul-ci.org/docs/zuul/reference/releasenotes.html | 15:30 |
| mordred | corvus: there's a decent number of changes since 3.15 ... are we still running 3.15 ourselves? | 15:30 |
| corvus | mordred: we're pretty close to 3.15. i think it's still just 3.15 with the memleak revert | 15:31 |
| tobiash | lgtm | 15:32 |
| mordred | corvus: most of the substantive changesa re things bmw is already running | 15:32 |
| clarkb | did my change for opendev to set gerrit auth method land? | 15:32 |
| clarkb | we'll need that if restarting zuul | 15:32 |
| corvus | clarkb: i think so, but we should double check | 15:32 |
| tristanC | corvus: sounds good to me, though there is lots of change since 3.15 | 15:32 |
| mordred | most of the changes are docs or CI related - there's a few dashboard changes - then a couple that are meaningful based on things bmw ran in to | 15:33 |
| corvus | yeah. maybe we should go ahead and cut the release because we're pretty sure it's good, but also restart opendev and we can cut a .1 if any problems show up? | 15:34 |
| mordred | corvus: ++ | 15:34 |
| tristanC | corvus: +1 | 15:34 |
| mordred | corvus: if we cut the release then restart opendev - we'll be running on a tagged release for the first time possibly ever :) | 15:35 |
| corvus | haha | 15:35 |
| corvus | pushed 3.16.0 | 15:36 |
| mordred | ++ | 15:36 |
| tobiash | corvus: do you have any objection about pausing the merger patch as well when pausing the executor? It is currently hard to stop a misbehaving executor from trying to merge things. | 15:38 |
| tobiash | s/patch/path | 15:38 |
| *** jamesmcarthur has quit IRC | 15:38 | |
| corvus | tobiash: no objection! i think i assumed it already worked that way :) | 15:41 |
| tobiash | :), it doesn't (which caused problems today) | 15:42 |
| corvus | tobiash: i think pause is important for rolling restarts, so i think everything should support it eventually | 15:42 |
| corvus | mordred: i'm confused about something image related: i have built a local zuul-web image, but changes to the js web app do not seem to be reflected in it. can you think of any reason why that would be the case? | 15:42 |
| tobiash | ++ | 15:42 |
| mordred | corvus: uh | 15:44 |
| mordred | corvus: only thing I could think of is if you had stale built artifacts copied in with the source code and yarn decided to not rebuild" | 15:45 |
| *** jamesmcarthur has joined #zuul | 15:46 | |
| corvus | mordred: how could i confirm that? a shell in the container, or a build log? | 15:47 |
| corvus | or run a yarn build then rebuild the image and see if it works? | 15:48 |
| mordred | corvus: yeah - I'd do that | 15:48 |
| corvus | k. i'll try that after restart | 15:49 |
| mordred | corvus: I wonder if maybe we shouldn't do an explicit yarn build in the dockerfile | 15:49 |
| corvus | mordred: is there a 'yarn clean'? maybe i should do that instead of a yarn build (and maybe that's what we should do in the dockerfile?) | 15:50 |
| fungi | 3.16.0 sounds appropriate for a9eafb5, yes | 15:51 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 15:52 |
| mordred | corvus: not sure - let me check | 15:52 |
| mordred | corvus: there's a stab at an explicit yarn build | 15:52 |
| *** zbr_ has joined #zuul | 15:53 | |
| *** zbr_ has quit IRC | 15:53 | |
| corvus | mordred: i'll just try that change then | 15:53 |
| mordred | no - no yarn clean | 15:53 |
| tobiash | corvus: would you unregister the merger functions on executor based on the governor or just on pause? | 15:53 |
| mordred | corvus: one set - path is off | 15:53 |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 15:54 |
| mordred | corvus: try that version instead | 15:54 |
| corvus | tobiash: i think on pause we don't have to worry about registration, we can just stop accepting jobs? but if it's easy, then unregistering is a way to accomplish that. actually, unregistering everything would be nice because then the stats reported are more accurate. maybe we should unregister execute and merge jobs on pause? | 15:56 |
| fungi | all this release talk reminds me, i still need to look into how to add the bits we forgot to include in the new opendev release jobs we switched to, as we're still not publishing release artifacts to anywhere besides pypi currently | 15:57 |
| tobiash | corvus: the execute jobs are already unregistered on pause (controlled by the governor) | 15:57 |
| corvus | tobiash: then let's do the same for merge | 15:57 |
| corvus | tobiash: oh i think i just now fully understood your question | 15:58 |
| tobiash | corvus: yeah, my question was more about pause on pause request or pause triggered by the governor | 15:58 |
| corvus | tobiash: you're going to unregister the merge functions on pause. you're asking should you also unregister them based on the governor. huh. i would lean toward "no" -- always have the merger running if it's not paused. | 15:59 |
| tobiash | ok | 15:59 |
| corvus | tobiash: (the more mergers the better, and it's probably not a huge impact on the system -- though maybe it will be by the time you finish making it super-efficient :) | 16:00 |
| tobiash | yeah, there is still quite some room for improvement :) | 16:01 |
| tobiash | we also seem to set all refs twice in some cases (reset repo and later restore repo state) | 16:01 |
| tobiash | but that needs some more thoughts | 16:01 |
| corvus | ++ | 16:01 |
| pabelanger | fungi: could you share your script again, to generate equeue command for release jobs that failed? I cannot seem to find it | 16:02 |
| fungi | https://review.opendev.org/613676 | 16:02 |
| pabelanger | thanks! | 16:03 |
| fungi | pabelanger: if you want to polish that up feel free to take it over, i seem to not have found time to make it more generic | 16:03 |
| tobiash | corvus: re-registration of functions in gearman is safe is it? | 16:03 |
| openstackgerrit | Federico Ressi proposed zuul/zuul-jobs master: Allow to force bindep installation https://review.opendev.org/707185 | 16:04 |
| corvus | tobiash: i don't understand | 16:04 |
| tobiash | the executor registers and unregisters functions in gearman based on the governor, I guess we don't need state handling there to avoid that we send multiple cando packets to gearman? | 16:05 |
| tobiash | at least I don't see an already existing state handling there, but I just wanted to make sure that I understood this correctly | 16:06 |
| corvus | tobiash: i think multiple cando is ok | 16:06 |
| tobiash | k, thx | 16:07 |
| pabelanger | fungi: thanks, I'll look to push up more updates | 16:11 |
| fungi | appreciated! | 16:11 |
| *** rf0lc0 has joined #zuul | 16:11 | |
| *** rfolco has quit IRC | 16:13 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: kubernetes-operator: change attribute to camelCase https://review.opendev.org/707190 | 16:21 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 16:22 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-operator master: Update attributes to camelCase https://review.opendev.org/707193 | 16:24 |
| tristanC | pabelanger: mnaser: jhesketh: to review the zuul-operator implementation (topic:zuul-crd) I suggest you start by checkout https://review.opendev.org/#/c/705535/3/CONTRIBUTE.md | 16:28 |
| tristanC | SpamapS: you might be interested by this too ^ | 16:28 |
| *** ysastri has joined #zuul | 16:37 | |
| corvus | tobiash, pabelanger: we're seeing some errors on the opendev github connection: https://zuul.opendev.org/t/openstack/config-errors | 16:44 |
| tobiash | corvus: do you have a log from scheduler startup? | 16:44 |
| tobiash | I guess you did a scheduler restart? | 16:45 |
| corvus | tobiash: yes and yes | 16:45 |
| pabelanger | we are still on 3.15.0, and haven't see that yet | 16:45 |
| corvus | tobiash: i did a full restart | 16:45 |
| corvus | that error is from the branch-listing part of the process | 16:45 |
| corvus | 2020-02-11 15:57:26,044 ERROR zuul.GithubConnection: No installation ID available for project cherrypy/cherrypy | 16:46 |
| *** jpena is now known as jpena|brb | 16:46 | |
| tobiash | corvus: how went prime_installation_map? | 16:46 |
| tobiash | our last scheduler update was on feb 3 and we hadn't an issue with this | 16:47 |
| tobiash | checking git log | 16:47 |
| corvus | tobiash: no errors with prime | 16:47 |
| corvus | do we require installations now? | 16:47 |
| corvus | (because these are repos without a zuul install on them) | 16:47 |
| SpamapS | tristanC: I ported all of my stuff to Terraform, and I couldn't be happier. I'm pretty sure Helm and Operators are the devil. ;) | 16:49 |
| tobiash | corvus: no, we shouldn't | 16:49 |
| tobiash | at least not on purpose | 16:49 |
| tobiash | this looks most suspicious to me: https://review.opendev.org/705167 | 16:49 |
| tobiash | that might cause that side effect | 16:50 |
| tristanC | SpamapS: right, and I'm looking forward dhall binding for Terraform configuration. I guess what you have can't be easily shared and re-used by other? | 16:50 |
| tobiash | the old implementation probably just ignores the missing installation and falls back to anonymous access | 16:50 |
| tobiash | while the new one might break because it does more stuff there | 16:51 |
| corvus | tobiash: yeah, maybe an exception in that strftime ? | 16:51 |
| tobiash | probably, do you have a trace? | 16:51 |
| tristanC | SpamapS: because that seems to be the main reason for helm and operator; e.g. being able to build higher order configuration | 16:52 |
| corvus | tobiash: no, it's hitting the user-level exception handler, so it's masked :( | 16:52 |
| tobiash | oops | 16:52 |
| corvus | mordred: /bin/sh: 1: react-scripts: not found | 16:53 |
| corvus | tobiash: should we revert that? | 16:53 |
| tobiash | corvus: do you want to revert that locally? I can prepare the revert in the meantime | 16:54 |
| SpamapS | tristanC: Terraform modules are about as sharable as Ansible roles. | 16:54 |
| SpamapS | What I like is the full lifecycle management, and easy integration with other APIs. | 16:54 |
| corvus | tobiash: sounds good | 16:55 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 16:56 |
| tobiash | corvus: ^ | 16:56 |
| corvus | tobiash: thanks. it's going to be a while since we have releases in progress right now | 16:58 |
| tobiash | oh good timing, is only the zuul tenant affected? | 16:59 |
| *** ysastri has quit IRC | 17:00 | |
| mordred | corvus: wtf. ok - I'll work on that more | 17:00 |
| corvus | tobiash: no, we have github projects in zuul and openstack tenants at least | 17:01 |
| corvus | actually looks like just those 2 | 17:02 |
| corvus | the kata tenant has a reall installation id, so it's fine | 17:02 |
| pabelanger | sounds like we should hold off on zuul 3.16.0 for the moment. Are we thinking of doing a 3.16.1 release? Should we also add release node to revert | 17:04 |
| corvus | tobiash: ^ oh yes please add a reno | 17:14 |
| corvus | (so we don't have a release without a note) | 17:14 |
| corvus | pabelanger: and yeah, i expect to restart with a local patch in maybe 20 mins, then after confirming that fixes the problem, release .1 as soon as that lands | 17:14 |
| tobiash | kk | 17:15 |
| pabelanger | wfm! | 17:15 |
| *** michael-beaver has quit IRC | 17:16 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 17:17 |
| *** gmann is now known as gmann_afk | 17:20 | |
| mordred | assuming that all works, I guess the next step for that patch is to update it to still have the fallback for anonymous repos, yeah? | 17:24 |
| *** igordc has joined #zuul | 17:24 | |
| *** jpena|brb is now known as jpena | 17:25 | |
| *** mattw4 has joined #zuul | 17:29 | |
| *** evrardjp has quit IRC | 17:34 | |
| *** evrardjp has joined #zuul | 17:34 | |
| *** jamesmcarthur has quit IRC | 17:40 | |
| corvus | mordred: yeah, and probably some better exception handling since it's getting complicated :) | 17:45 |
| mordred | yeah | 17:47 |
| corvus | tobiash: opendev restart with your patch looks good, i'm approving it and will tag it as .1 after it merges | 17:48 |
| tobiash | :) | 17:48 |
| *** igordc has quit IRC | 18:02 | |
| *** igordc has joined #zuul | 18:06 | |
| *** bhavikdbavishi has quit IRC | 18:08 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 18:21 |
| mordred | corvus: ^^ | 18:21 |
| mordred | corvus: was issing yarn install :) | 18:21 |
| mordred | double checking locally - but I'm pretty sure that'll do it | 18:22 |
| corvus | mordred: cool, i kicked off a local build with that too, will check back in a bit | 18:22 |
| mordred | cool | 18:22 |
| mordred | corvus, clarkb: warning sha.js@2.4.11: Invalid bin entry for "sha.js" (in "sha.js"). | 18:23 |
| mordred | THAT'S a warning you always want to see | 18:24 |
| clarkb | who checks the checkers | 18:24 |
| *** bhavikdbavishi has joined #zuul | 18:25 | |
| *** jpena is now known as jpena|off | 18:36 | |
| *** bhavikdbavishi has quit IRC | 18:45 | |
| *** gmann_afk is now known as gmann | 18:49 | |
| *** NBorg has quit IRC | 18:50 | |
| corvus | mordred: that worked! | 18:56 |
| *** Defolos has joined #zuul | 18:56 | |
| mordred | corvus: woot! | 18:58 |
| SpamapS | Has anyone ever considered multi-inheritance for zuul jobs? | 19:00 |
| clarkb | yes, I believe it is possible already? I want to say its used somewhere but I can't remember where | 19:00 |
| SpamapS | We have several competing interests in our parent jobs, like with-ecr-upload and with-slack-notify and then we have to have with-ecr-upload-and-slack-notify and now we're adding more and it's making a messy matrix. | 19:01 |
| SpamapS | hah I hope once again Zuul just changed under me and I can just do this. | 19:02 |
| corvus | SpamapS: sorta -- there's a sneaky way you can do it by having multiple variants each with their own parent. | 19:02 |
| corvus | SpamapS: but we haven't make the "parent" attribute a list. | 19:02 |
| corvus | i think there is growing support for just doing that. | 19:02 |
| SpamapS | cool | 19:03 |
| SpamapS | just had the thought right now, glad it's not an outlier | 19:03 |
| corvus | SpamapS: fwiw, the multiple variants thing *is* intentional -- it has regression tests and everything. it's just not really advertised because it's so complicated. so i would say feel free to try that out with confidence that it's not an accidental feature subject to breakage. | 19:03 |
| SpamapS | corvus: is there a good starting place in the docs to figure that out? | 19:05 |
| SpamapS | I'm not sure I actually know all the ways to make a variant. | 19:05 |
| SpamapS | branches are the only one I'm really familiar with. ;) | 19:05 |
| corvus | SpamapS: no, and unfortunately we reverted our one use of it in opendev so i can't point. but let me mock up an example real quick | 19:06 |
| corvus | SpamapS: something like this should do: https://etherpad.openstack.org/p/VqP017oWn8 | 19:07 |
| corvus | SpamapS: and yeah, it's basically just multiple branch variants that match | 19:07 |
| corvus | and you don't change anything in the variant other than parent | 19:07 |
| corvus | SpamapS: if you use that, i'll be very interested in feedback :) | 19:08 |
| corvus | (maybe it's good and we just make 'parent' a list; maybe it's tricky and we tweak something before we make it a first-class feature | 19:08 |
| pabelanger | we use it in zuul.a.c | 19:09 |
| corvus | pabelanger: ooh cool, maybe you can share a link? | 19:09 |
| pabelanger | https://github.com/ansible-network/network_collections_migration/blob/master/.zuul.yaml#L20 and https://github.com/ansible-network/network_collections_migration/blob/master/.zuul.yaml#L2 | 19:09 |
| *** bhavikdbavishi has joined #zuul | 19:09 | |
| pabelanger | this is because we want to parent to both propose-github-updates and unittests | 19:10 |
| pabelanger | but not force unitests for propose-github-updates | 19:10 |
| corvus | i'm pretty sure it's a depth-first traversal without repetition, but that's just off the top of my head, i'm not positive. | 19:11 |
| *** bhavikdbavishi has quit IRC | 19:14 | |
| mordred | corvus: takea . look at http://zuul.opendev.org/t/openstack/status real quick | 19:16 |
| mordred | corvus: it's showing 707214 in check twice | 19:16 |
| mordred | corvus: I think there might be a bug - but I'm worried about capturing appropriate info | 19:17 |
| mordred | corvus: I've saved a copy of status.json | 19:17 |
| corvus | mordred: anything you can think of interesting about that? manual enqueue? extra recheck comment? | 19:17 |
| corvus | oh it's different patchsets | 19:18 |
| mordred | one copy is 707214,1 and one is 707214,2 | 19:18 |
| mordred | eyah | 19:18 |
| corvus | and the ,1 patchset was in my re-enqueue script | 19:18 |
| mordred | AH | 19:18 |
| mordred | maybe just a race-condition | 19:18 |
| corvus | so basically, i manually enqueued the ,1 after it was out of date | 19:18 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:18 |
| corvus | yeah, i think it's benign | 19:18 |
| mordred | corvus: PHEW | 19:18 |
| corvus | the fix is HA scheduler :) | 19:18 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:20 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add an ensure-tox test job https://review.opendev.org/706371 | 19:23 |
| corvus | zbr: ^ can you rebase on that ? | 19:23 |
| corvus | zbr: (also, feel free to add more testing for your change as appropriate) | 19:23 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: Install tox-venv alongside tox https://review.opendev.org/707237 | 19:24 |
| zbr | nice sync :) | 19:24 |
| zbr | all this testing does not cover other platforms.... | 19:27 |
| zbr | so is not of much use of indicating that the role would work fine on platforms other than ubuntu-bionic | 19:28 |
| fungi | for some reason that gave me a brief vision of zuul as a platformer-style arcade game | 19:28 |
| zbr | :] | 19:29 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Run ensure-tox on all platforms https://review.opendev.org/707238 | 19:30 |
| corvus | easily remedied | 19:30 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: tox: change default calling as a module https://review.opendev.org/690057 | 19:34 |
| *** hashar has joined #zuul | 19:35 | |
| webknjaz | \ | 19:52 |
| *** Goneri has quit IRC | 19:53 | |
| openstackgerrit | Merged zuul/zuul master: Extract the watcher from git driver https://review.opendev.org/706092 | 20:35 |
| mordred | tristanC: feel like a +A on https://review.opendev.org/#/c/707182 ? | 20:40 |
| tristanC | mordred: isn't this going to build the web file twice? | 20:44 |
| tristanC | nvm, it seems like the _setup_hook check if it's already built | 20:46 |
| mordred | tristanC: ++ | 20:48 |
| corvus | derp, we installed kubectl/oc on the builder image | 21:01 |
| mordred | corvus: derp | 21:08 |
| mordred | corvus: it's not useful on the builder image | 21:08 |
| corvus | mordred: this is what i'm finding to be true yes | 21:09 |
| mordred | corvus: that said - you could do the download in the builder image and do a copy-from to get the two files and skip the rm step if you wanted to | 21:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Fix kubectl/oc install in container image https://review.opendev.org/707249 | 21:09 |
| corvus | mordred: yep! that's what i was thinking and did | 21:09 |
| mordred | corvus: I do not know whether that would be better or not - but it's possible if you put the download early that in iterative def it could reduce duplicate fetching | 21:09 |
| mordred | woot! | 21:09 |
| corvus | but i also simplied the download a bit using a suggestion from tristanC | 21:10 |
| corvus | oh dear https://review.opendev.org/707205 failed tests? | 21:11 |
| mordred | corvus: I left a comment - but we can also skip that and just land it as-is if you prefer | 21:11 |
| mordred | oh dear indeed | 21:11 |
| corvus | mordred: sounds good | 21:12 |
| corvus | hrm, same test failed on both runs | 21:13 |
| *** Goneri has joined #zuul | 21:13 | |
| tristanC | the failure doesn't seem related to the change | 21:14 |
| corvus | agreed | 21:14 |
| corvus | oh wait it looks like it has a dep | 21:15 |
| corvus | yep https://review.opendev.org/707192 is failing too | 21:15 |
| corvus | i think that was just a rebase error, i will fix | 21:15 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Support pausing merge jobs https://review.opendev.org/707192 | 21:16 |
| corvus | derp | 21:16 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 21:16 |
| tristanC | corvus: that looks correct | 21:17 |
| *** hashar has quit IRC | 21:21 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul master: Move oc download to before src copy https://review.opendev.org/707250 | 21:22 |
| mordred | corvus, tristanC: ^^ I +A'd the corvus patch and pushed my suggestion up as a followup | 21:23 |
| corvus | cool, i have a build of my original patch running now, i should be able to confirm it worked soon | 21:24 |
| corvus | yep, it looks good -- kubectl and oc are in /usr/local/bin | 21:25 |
| mordred | corvus: woot | 21:35 |
| openstackgerrit | Merged zuul/zuul master: Run yarn explicitly in Dockerfile https://review.opendev.org/707182 | 21:47 |
| *** jamesmcarthur has joined #zuul | 22:06 | |
| *** mattw4 has quit IRC | 22:27 | |
| openstackgerrit | Merged zuul/zuul master: Fix kubectl/oc install in container image https://review.opendev.org/707249 | 22:40 |
| *** jamesmcarthur has quit IRC | 22:41 | |
| corvus | mordred, tristanC: huzzah! https://gerrit-zuul.inaugust.com/t/gerrit/build/867c84e630244ec4ab06b6126764c9aa/console | 23:01 |
| corvus | k8s config update and scheduler reconfiguration all from the executor | 23:01 |
| corvus | (that's using add_host with kubectl to do the reconfig) | 23:01 |
| *** jamesmcarthur has joined #zuul | 23:02 | |
| mordred | corvus: SWEET | 23:03 |
| corvus | mordred: i think i'll send a status update to repo-discuss and let folks know that it's running, self-deployed, we're just waiting on a dns name to declare it production-ready, and we can probably start working on some jobs | 23:04 |
| corvus | how does that sound? | 23:04 |
| mordred | corvus: ++ | 23:04 |
| *** jamesmcarthur_ has joined #zuul | 23:05 | |
| *** jamesmcarthur has quit IRC | 23:05 | |
| *** jamesmcarthur_ has quit IRC | 23:24 | |
| *** jamesmcarthur has joined #zuul | 23:25 | |
| *** jamesmcarthur has joined #zuul | 23:26 | |
| corvus | email sent | 23:30 |
| *** jamesmcarthur has quit IRC | 23:32 | |
| *** igordc has quit IRC | 23:46 | |
| *** Defolos has quit IRC | 23:46 | |
| openstackgerrit | Merged zuul/zuul master: Revert "Fix github app authentication to work with checks API endpoints" https://review.opendev.org/707205 | 23:58 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!