| *** armstrongs has joined #zuul | 00:01 | |
| *** igordc has joined #zuul | 00:02 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add ensure-virtualenv https://review.opendev.org/723309 | 00:10 |
|---|---|---|
| *** armstrongs has quit IRC | 00:16 | |
| clarkb | as a heads up the pinned version of mypy that zuul's tox -e linters runs pulls in an older typed-ast which doesn't work on python3.8 | 00:21 |
| clarkb | I'm trying to sort that out | 00:21 |
| clarkb | this is fun. If I change the mypy version I can make tox crash | 00:26 |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Update dib to 2.36.0 https://review.opendev.org/723761 | 00:33 |
| *** igordc has quit IRC | 00:34 | |
| *** rlandy has quit IRC | 00:42 | |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Bump mypy for py3.8 support https://review.opendev.org/723763 | 00:45 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Don't retry after Gerrit HTTP 409s https://review.opendev.org/723764 | 00:46 |
| clarkb | that second change is something I noticed onf riday when debugging fun happened | 00:47 |
| clarkb | the first change was what I noticed when trying to run linters locally under python3.8 | 00:47 |
| *** cdearborn has quit IRC | 01:04 | |
| *** rlandy has joined #zuul | 01:36 | |
| *** rlandy has quit IRC | 01:51 | |
| *** swest has quit IRC | 02:01 | |
| *** rf0lc0 has quit IRC | 02:03 | |
| *** rfolco has joined #zuul | 02:03 | |
| *** swest has joined #zuul | 02:15 | |
| *** bhavikdbavishi has joined #zuul | 03:01 | |
| *** bhavikdbavishi1 has joined #zuul | 03:04 | |
| *** bhavikdbavishi has quit IRC | 03:06 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:06 | |
| *** threestrands has joined #zuul | 03:12 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Allow disabling build-log-retention https://review.opendev.org/723782 | 03:20 |
| openstackgerrit | Merged zuul/zuul-jobs master: k8-logs: use failed_when: instead of ignore_errors: https://review.opendev.org/723647 | 03:56 |
| openstackgerrit | Merged zuul/zuul-jobs master: container-logs: use failed_when: instead of ignore_errors: https://review.opendev.org/723648 | 03:57 |
| *** sshnaidm|afk is now known as sshnaidm|off | 04:34 | |
| *** evrardjp has quit IRC | 04:35 | |
| *** evrardjp has joined #zuul | 04:35 | |
| *** bhavikdbavishi has quit IRC | 05:01 | |
| clarkb | https://zuul.opendev.org/t/zuul/build/231db3c20f8d4fa69c3e069e3d6d675c/log/job-output.txt#2114 is a weird failure in zuul's test suite. Seems like the change.message attribute wasn't looking like a string? | 05:05 |
| clarkb | I can'tdebug now but thought I'd call it out | 05:05 |
| *** bhavikdbavishi has joined #zuul | 05:17 | |
| *** ysandeep|away is now known as ysandeep | 05:21 | |
| *** zenkuro has quit IRC | 05:35 | |
| openstackgerrit | Ian Wienand proposed zuul/nodepool master: Allow disabling build-log-retention https://review.opendev.org/723782 | 05:47 |
| *** bhavikdbavishi has quit IRC | 05:54 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul master: Revert "Revert "Tune automatic garbage collection of git repos"" https://review.opendev.org/723800 | 05:55 |
| AJaeger | corvus, fungi, so this is fine now to revert, correct ^ | 05:57 |
| *** bhavikdbavishi has joined #zuul | 05:59 | |
| *** bhavikdbavishi has quit IRC | 06:05 | |
| *** dpawlik has joined #zuul | 06:06 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Add ensure-virtualenv https://review.opendev.org/723309 | 06:09 |
| *** saneax has joined #zuul | 06:20 | |
| *** jcapitao has joined #zuul | 07:03 | |
| *** yolanda has joined #zuul | 07:18 | |
| *** bhavikdbavishi has joined #zuul | 07:21 | |
| *** bhavikdbavishi1 has joined #zuul | 07:24 | |
| *** bhavikdbavishi has quit IRC | 07:26 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 07:26 | |
| *** yolanda has quit IRC | 07:26 | |
| *** yolanda has joined #zuul | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:34 | |
| *** tosky has joined #zuul | 07:36 | |
| *** bhavikdbavishi has quit IRC | 07:38 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Make task errors expandable https://review.opendev.org/723534 | 07:40 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Make task errors expandable https://review.opendev.org/723534 | 07:41 |
| *** jpena|off is now known as jpena | 07:54 | |
| *** bhavikdbavishi has joined #zuul | 08:22 | |
| *** ysandeep is now known as ysandeep|lunch | 08:22 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: POC: Add convenience Makefile https://review.opendev.org/723837 | 08:27 |
| zbr | did anyone run zuul unittests under macos? apparently i am not even able to make them start | 08:36 |
| *** sgw has quit IRC | 08:41 | |
| zbr | created https://storyboard.openstack.org/#!/story/2007601 | 08:42 |
| *** ysandeep|lunch is now known as ysandeep | 08:53 | |
| -openstackstatus- NOTICE: Zuul is currently failing testing, please refrain from recheck and submitting of new changes until this is solved. | 09:06 | |
| *** ChanServ changes topic to "Zuul is currently failing testing, please refrain from recheck and submitting of new changes until this is solved." | 09:06 | |
| -openstackstatus- NOTICE: Zuul is currently failing all testing, please refrain from approving, rechecking or submitting of new changes until this is solved. | 09:17 | |
| *** ChanServ changes topic to "Zuul is currently failing all testing, please refrain from approving, rechecking or submitting of new changes until this is solved." | 09:17 | |
| *** guillaumec has joined #zuul | 09:48 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Fix cherrypy dependency conflict https://review.opendev.org/723855 | 09:50 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Add DOCKER_* to passenv https://review.opendev.org/723856 | 09:56 |
| *** avass has quit IRC | 09:59 | |
| *** avass has joined #zuul | 09:59 | |
| *** rpittau is now known as rpittau|bbl | 10:10 | |
| *** bhavikdbavishi has quit IRC | 10:25 | |
| *** bhavikdbavishi has joined #zuul | 10:30 | |
| *** Tahvok has quit IRC | 10:31 | |
| *** Tahvok has joined #zuul | 10:33 | |
| *** jcapitao is now known as jcapitao_lunch | 11:06 | |
| *** ysandeep is now known as ysandeep|afk | 11:07 | |
| *** bhavikdbavishi has quit IRC | 11:09 | |
| *** bhavikdbavishi has joined #zuul | 11:18 | |
| *** tosky has quit IRC | 11:23 | |
| *** jpena is now known as jpena|lunch | 11:35 | |
| *** tosky has joined #zuul | 11:54 | |
| mordred | zbr: could you fix up 723855 real quick? | 11:54 |
| zbr | haha, sure | 11:55 |
| zbr | maybe we need to patch git-review to make it fail on messages not matching 50/72 rule | 11:56 |
| zbr | when you get the warning from the server is already too late | 11:56 |
| mordred | zbr: there's actually a gerrit setting to reject things that don't follow the rule - but we never enabled it beause folks thought it was too strict (sometimes a 54 character subject makes more sense) | 11:57 |
| mordred | but - yeah | 11:57 |
| avass | mordred, zbr: even though I like 'code golfing' with the commit message, 50 characters can be a bit hard to get the meaning across sometimes. especially if you add a topic in the commit message like "build-container-image-docker: now i only have 25 characters left" | 11:59 |
| avass | I'd see it as a guideline | 12:00 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Fix cherrypy dependency conflict https://review.opendev.org/723855 | 12:00 |
| zbr | in fact an interactive prompt would be fine too, the idea is to give user a chance to fix it. printing a warning after you already uploaded it is not. | 12:01 |
| zbr | indeed, there are cases where is ok to ignore the rule | 12:01 |
| mordred | yeah | 12:06 |
| avass | what's up with zuul by the way? | 12:08 |
| AJaeger | avass: see backscroll in #opendev | 12:08 |
| *** rpittau|bbl is now known as rpittau | 12:08 | |
| AJaeger | avass: zuul01 run out of memory ;( | 12:09 |
| avass | AJaeger: how do I reach the logs? I usually use the link in the topic :) | 12:10 |
| mordred | yeah - I'm trying to decide whether to restart it to get back up, or wait longer for corvus to look at it | 12:10 |
| *** guillaumec has quit IRC | 12:10 | |
| AJaeger | avass: http://eavesdrop.openstack.org/irclogs/%23opendev/latest.log.html | 12:12 |
| AJaeger | avass: so, not much information to gain | 12:12 |
| avass | should probably bookmark that | 12:12 |
| mordred | avass: should be back up | 12:17 |
| *** jcapitao_lunch is now known as jcapitao | 12:20 | |
| *** ChanServ changes topic to "Discussion of the project gating system Zuul | Website: https://zuul-ci.org/ | Docs: https://zuul-ci.org/docs/ | Source: https://git.zuul-ci.org/ | Channel logs: http://eavesdrop.openstack.org/irclogs/%23zuul/ | Weekly updates: https://etherpad.openstack.org/p/zuul-update-email" | 12:29 | |
| -openstackstatus- NOTICE: Zuul has been restarted, all events are lost, recheck or re-approve any changes submitted since 9:50 UTC. | 12:29 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 12:30 |
| *** rlandy has joined #zuul | 12:33 | |
| *** dpawlik has quit IRC | 12:36 | |
| *** hashar has joined #zuul | 12:37 | |
| *** jpena|lunch is now known as jpena | 12:38 | |
| *** dpawlik has joined #zuul | 12:40 | |
| *** Goneri has joined #zuul | 12:40 | |
| zbr | mordred: https://review.opendev.org/#/c/723534/ is also ready for review, loot for a "more" button on that expands the text. | 12:47 |
| mordred | zbr: sadly it totally works - so I cant see the more button ;) | 12:57 |
| zbr | find a failed build, sadly sharing links does not work with the dashboard | 12:58 |
| *** bhavikdbavishi has quit IRC | 12:58 | |
| mordred | oh - duh | 12:58 |
| zbr | the button will be present only if output is longer than the limit, but i was lucky, the first failure I checked was longer than the limit | 12:59 |
| mordred | zbr: neat! yes - I found one and it worked as advertised | 12:59 |
| mordred | zbr: we might want to style that more link - my eyes totally missed it even though I was looking for it | 13:00 |
| zbr | yep, i was sure someone would say that :D | 13:00 |
| mordred | :) | 13:01 |
| zbr | i will have a look, in fact we should be able to even make that pre-wrap setting in the global css | 13:01 |
| zbr | i am not yet aware of any case where we do not want the pre-wrap behavior. | 13:01 |
| avass | mordred, zbr: yep, I completely missed that 'more' button | 13:02 |
| avass | I also expected it to be on the bottom so I scrolled past it :) | 13:02 |
| zbr | sure, I will restyle it to make it look like a link. ok about the css question or you prefer in-line? | 13:02 |
| zbr | tbh, i hate in-line, but sometimes is more covenient. | 13:03 |
| *** sgw has joined #zuul | 13:07 | |
| *** yolanda has quit IRC | 13:08 | |
| *** avass is now known as Guest80970 | 13:10 | |
| *** avass has joined #zuul | 13:10 | |
| *** yolanda has joined #zuul | 13:14 | |
| *** rfolco is now known as rfolco|bbl | 13:34 | |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add haskell tool stack test https://review.opendev.org/723263 | 13:54 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-jobs master: haskell-stack-test: add haskell tool stack test https://review.opendev.org/723263 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: Use 'block: ... always: ...' instead of ignore_errors https://review.opendev.org/723640 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ensure-sphinx: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723642 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: go: Use 'block: ... always: ...' and failed_when instead of ignore_errors https://review.opendev.org/723643 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ara-report: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723644 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: fetch-subunit-output: use failed_when: instead of ignore_errors: https://review.opendev.org/723653 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: add-build-sshkey: use failed_when: instead of ignore_errors: https://review.opendev.org/723654 | 14:08 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: Use 'block: ... always: ...' instead of ignore_errors https://review.opendev.org/723640 | 14:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ensure-sphinx: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723642 | 14:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: go: Use 'block: ... always: ...' and failed_when instead of ignore_errors https://review.opendev.org/723643 | 14:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ara-report: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723644 | 14:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: fetch-subunit-output: use failed_when: instead of ignore_errors: https://review.opendev.org/723653 | 14:14 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: add-build-sshkey: use failed_when: instead of ignore_errors: https://review.opendev.org/723654 | 14:14 |
| *** ysandeep|afk is now known as ysandeep | 14:19 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: tox: Use 'block: ... always: ...' instead of ignore_errors https://review.opendev.org/723640 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ensure-sphinx: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723642 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: go: Use 'block: ... always: ...' and failed_when instead of ignore_errors https://review.opendev.org/723643 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: ara-report: use failed_when: false instead of ignore_errors: true https://review.opendev.org/723644 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: fetch-subunit-output: use failed_when: instead of ignore_errors: https://review.opendev.org/723653 | 14:20 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: add-build-sshkey: use failed_when: instead of ignore_errors: https://review.opendev.org/723654 | 14:20 |
| avass | sorry for that, should be ready now :) | 14:21 |
| *** fbo is now known as fbo|off | 14:21 | |
| corvus | mordred: we restarted on april 20 due to memory pressure, and that was stable for about a week. i don't think we've merged anything suspicious since then, so we may be looking at a memory leak that was merged before april 20 (possibly a long time ago) but doesn't always manifest. | 14:23 |
| mordred | corvus: awesome | 14:26 |
| mordred | corvus: I love issues that only show up sometimes | 14:27 |
| * AJaeger wonders why zuul01 is already using 10 GB of swap - but has 24 GB of memory free? | 14:28 | |
| *** rpittau is now known as rpittau|brb | 14:32 | |
| mordred | corvus: latest on the multi-arch build - I got through CA certs and /etc/hosts - and it couldn't push to the registry becaues there was no ipv6 address in the builder container ;) | 14:33 |
| mordred | corvus: latest patch adds net=host - we'll see if that fixes it | 14:34 |
| corvus | AJaeger: that's probably non-zuul related data the kernel previously swapped out; it's not going to load it in until it's used. | 14:36 |
| *** rfolco|bbl is now known as rfolco | 14:41 | |
| AJaeger | corvus: you're probably right - I forgot that we didn't reboot... | 14:47 |
| corvus | avass, AJaeger: i'm looking at approving avass's message to zuul-announce. do we want to remove the roles in 2 weeks, or do we want to merge a change that reports an error in 2 weeks then remove them 2 weeks after that? | 14:48 |
| fungi | interestingly, zuul-web is consuming 18.644g vmem with 7g resident according to top | 14:49 |
| fungi | so maybe the memory leak is in there? | 14:49 |
| corvus | clarkb found that we're hitting zuul-web for a lot more than we used to. | 14:49 |
| fungi | looks like the zuul-web container was restarted sometime yesterday | 14:50 |
| corvus | my guess is perhaps it's using more ram due to the additional load, but it's probably not enough to account for the kind of usage that prompted the scheduler restart | 14:51 |
| avass | corvus: the second alternative sounds a bit better, forgot that we discussed that yesterday | 14:52 |
| corvus | avass: okay, if you like that approach, how about i discard the message and you can send another one? | 14:52 |
| corvus | avass: (to be clear, the message is still in the moderation queue, so it hasn't been delivered to anyone yet) | 14:53 |
| avass | corvus: are we looking to implement a warning in the web for the build or so zuul could report 'build: warning' instead of succeeded or error in general? | 14:54 |
| avass | corvus: yeah go ahead :) | 14:54 |
| corvus | avass: i think a new build state would be a fine idea | 14:55 |
| *** rpittau|brb is now known as rpittau | 14:57 | |
| avass | corvus: cool | 14:58 |
| fungi | yeah, at the time of the oom, we seem to have had nearly 14g vmem used by the zuul-scheduler process and barely more than 4gb by zuul-web | 14:58 |
| fungi | (at least according to the process report in the kmesg) | 14:59 |
| *** timburke has quit IRC | 15:04 | |
| avass | crovus: oh, should we go with throwing a warning or fail the jobs to begin with? I'm not sure how long it would take to implementing a new build state | 15:05 |
| avass | corvus: ^ | 15:05 |
| *** hashar has quit IRC | 15:15 | |
| corvus | avass: i think fail the jobs; it'll take a long time for any warning system to actually get deployed by users to be useful | 15:18 |
| corvus | at least if we fail with a message, anyone who ignores the email will quickly know what to fix | 15:18 |
| avass | corvus: yeah that's what I thought | 15:18 |
| avass | corvus: I sent a new message | 15:19 |
| clarkb | are the test failures in tests.unit.test_cross_crd.TestGerritToGithubCRD.test_crd_check_unknown a known thing? | 15:23 |
| clarkb | https://zuul.opendev.org/t/zuul/build/b702c7815c944819a75300e356c30995/log/job-output.txt#2117 https://zuul.opendev.org/t/zuul/build/231db3c20f8d4fa69c3e069e3d6d675c/log/job-output.txt#2114 I'e caught it failing in two unrelated ways and was checking this isn'y already a known thing before digging further | 15:24 |
| corvus | clarkb: doesn't sound familiar | 15:31 |
| *** zxiiro has joined #zuul | 15:33 | |
| mordred | clarkb: \o/ the multi-arch job passed! | 15:41 |
| mordred | gah | 15:42 |
| mordred | corvus: ^^ that was menat for you - although hopefully clarkb also finds it exciting | 15:42 |
| corvus | mordred: that's great! did it run in an ipv6 cloud? we should make sure that earlier failure case is adequately tested (and it was fixed by the host net, not because it happened to run in ipv4-only) | 15:44 |
| zbr | a stupid question: why zuul uses stestr instead of pytest for running tests? | 15:44 |
| mordred | corvus: that's a great question | 15:45 |
| clarkb | zbr: some of it is history. Zuul existed before pytest. Personally I prefer using a standard complying runner by default which ensures you can use whatever test runner you want. Also testr allows running tests in parallel and is much quicker than pytest | 15:45 |
| fungi | zbr: likely because, owing to openstack legacy, we have existing tooling to deal with subunit | 15:45 |
| mordred | yeah. both of those | 15:45 |
| clarkb | you should be able to run zuul's tests with pytest | 15:46 |
| clarkb | but if we used pytest by default you'd only be able to use pytest | 15:46 |
| zbr | kindof, you can use pytest w/o using its own features, but like having a ferari with speed-limiter set to 20mph :) | 15:47 |
| mordred | yeah - pytest does the same thing nose did - it's both a test runner and a testing framework - so really there's two sides "why do we use stestr as the runner instead of pytest" and "why do we use testtools as the underlying framework instead of pytest" | 15:47 |
| fungi | some of the testing-cabal folks responsible for testrepository et al were heavily involved in openstack early on | 15:47 |
| fungi | and testtools | 15:47 |
| zbr | pytest used to lack on parallel part, but now both pytest-parallel and pytest-xdist are quite good. | 15:47 |
| mordred | I personally *FAR* prefer testtools to pytest for writing tests - the pytest fixture injection model is too magical to me | 15:47 |
| mordred | so even if I wanted to use pytest as a runner, I'd be very unpleased by using pytest as a test writing base class - at least personally | 15:48 |
| zbr | mordred: indeed, these magic shrooms... are not for everyone. | 15:48 |
| clarkb | mordred: for me its the nose concern | 15:48 |
| mordred | yeah | 15:48 |
| clarkb | mordred: what happens when pytest is abandoned for greener pastures | 15:48 |
| clarkb | now you're stuck with thousands of lines of useless code | 15:48 |
| clarkb | sticking to stdlib interfaces avoids that | 15:49 |
| mordred | yup! | 15:49 |
| zbr | i will have a look to see what is missing for pytest, mainly for dev convenience, like running single test, fail-fast, live progress. | 15:50 |
| zbr | all these features do not require changing the tests | 15:50 |
| clarkb | zbr: correct, but you have no way of enforcing it either | 15:50 |
| clarkb | because pytest will happily work with all of its injected non standard stuff | 15:50 |
| fungi | also you can run a single test with testr too | 15:50 |
| fungi | not sure what fail-fast and live progress are | 15:51 |
| clarkb | zbr: which is why if we test with testr we can enforce those things without preventing you from using pytest | 15:51 |
| AJaeger | mordred: want to abandon https://review.opendev.org/#/c/723109/ with https://review.opendev.org/#/c/723524/6 approved and the stack at https://review.opendev.org/723640 . or do we need 723109? | 15:51 |
| clarkb | pytest should work because we've stuck to standard interfaces | 15:51 |
| clarkb | you should feel free to use pytest | 15:51 |
| mordred | zbr: stestr tests.test_foo.TestFoo.test_foo runs a single test | 15:52 |
| mordred | sorry | 15:53 |
| mordred | zbr: stestr run tests.test_foo.TestFoo.test_foo runs a single test | 15:53 |
| zbr | but it lacks a "-x" option, or a way to display the overall progress(baR) | 15:53 |
| mordred | it can print out each test as it runs it | 15:53 |
| clarkb | what does -x do? | 15:53 |
| mordred | (which is how I run stestr myself) | 15:54 |
| zbr | -x is --exitfirst | 15:54 |
| zbr | mainly, the fail-fast | 15:54 |
| mordred | otoh - stestr has --failing - which will run only the tests that failed last time | 15:55 |
| mordred | but seriously - if you like pytest as a test runner - use it | 15:55 |
| clarkb | zbr: --until-failure | 15:55 |
| mordred | clarkb: that's different | 15:55 |
| clarkb | mordred: I think it works the same way though | 15:55 |
| mordred | clarkb: that runs the tests over and over ina. loop until they fail | 15:55 |
| clarkb | mordred: it will exit as soon as it fails | 15:56 |
| mordred | oh - that is a good point | 15:56 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: doc: add how to run a single test https://review.opendev.org/724072 | 15:56 |
| mordred | it'll keep running if nothing does fail though | 15:56 |
| zbr | aha! i forgot to look for help under "run" command. | 15:56 |
| clarkb | mordred: ya | 15:56 |
| zbr | it has most of the stuff i need, like --pdb :) | 15:56 |
| mordred | zbr: --failing is super cool - it's a great way to iterate on failing tests | 15:56 |
| clarkb | but ya I think the important point is by enforcing the use of standard interfaces in CI we allow you to use whatever standard compliant test runner you want including pytest | 15:57 |
| avass | mordred: ooh, nice | 15:57 |
| clarkb | if you want to use pytest yu should | 15:57 |
| zbr | out of curiority, when does it save the "failing" status? usually i press Ctrl-C if i see things starting to fail, not wanting to waste too much time. | 15:57 |
| mordred | corvus, clarkb: any idea why https://review.opendev.org/#/c/722339/ didn't collect /etc/hosts and /home/zuul/.docker/config.json ? | 15:58 |
| mordred | it has host_copy_output in https://review.opendev.org/#/c/722339/17/zuul-tests.d/container-roles-jobs.yaml | 15:58 |
| clarkb | zbr: its streamed to disk as it runs | 15:58 |
| clarkb | zbr: however it is streamed into a temp file until the process ends at which point it is moved into its final location | 15:59 |
| *** klindgren has quit IRC | 15:59 | |
| clarkb | zbr: you can look in .stestrepository and you'll see tmpfiles from when you killed it early | 15:59 |
| *** klindgren has joined #zuul | 15:59 | |
| zbr | okey, i will try to learn more about it, maybe i will write a comparison between the two. | 16:00 |
| mordred | oh - host_copy_output is system-config specific | 16:03 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 16:03 |
| *** sgw has quit IRC | 16:04 | |
| mordred | corvus: doesn't look like it was an on ipv6 host. any thoughts for how we'd test that? | 16:04 |
| *** sgw has joined #zuul | 16:05 | |
| clarkb | mordred: you want the job to run in openedge or linaro-us for that | 16:06 |
| clarkb | and limestone | 16:06 |
| mordred | clarkb: I don't suppose we've got a good way to simulate that in our test job | 16:07 |
| mordred | clarkb: making sure that the role in zuul-jobs works if the host it runs on is ipv6 only relying on zuul-jobs running its test jobs on ipv6 only hosts seems prone to failure | 16:07 |
| clarkb | mordred: ya, but we also have ipv4 only clouds so we can't really do ipv6 only testing reliable? | 16:08 |
| clarkb | this is one of those things wheer if the world had coalesced on ipv6 in 1998 everything would be great | 16:08 |
| mordred | right? | 16:10 |
| donnyd | clarkb: if you use the OE custom labels for the job, wouldn't that ensure it lands on an ipv6 host | 16:10 |
| donnyd | or maybe I am missing the point | 16:11 |
| clarkb | donnyd: it would but I'm not sure we want to force the job to run there as then we'd lose ipv4 testing ? | 16:11 |
| clarkb | somethign to think about | 16:11 |
| donnyd | maybe two jobs then? one targeted for ipv4 hosts and one for ipv6 ? | 16:11 |
| clarkb | ya that might work | 16:12 |
| corvus | mordred: maybe make a throwaway followup change that triggers that job and sets the oe node label, just so we can get a baseline? | 16:12 |
| donnyd | isn't vexxhost ipv4? and I think they have the same custom labels as OE | 16:12 |
| corvus | or 2 jobs | 16:12 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add serial pipeline manager https://review.opendev.org/722981 | 16:16 |
| mordred | donnyd: is the label in question nested-virt-ubuntu-bionic ? | 16:17 |
| corvus | mordred, tristanC: ^ serial pipeline updated to use a shared queue | 16:18 |
| *** rpittau is now known as rpittau|afk | 16:18 | |
| donnyd | i think you could also use the expanded-* labels | 16:18 |
| donnyd | I think this "ubuntu-bionic-expanded" | 16:19 |
| donnyd | unless you need nested virt | 16:19 |
| clarkb | expanded means more memory and is also provided by ipv4 only citycloud | 16:19 |
| clarkb | raelly if we need ipv6 only labels we might want a specific label for that? | 16:19 |
| donnyd | and then for ipv4 maybe use ubuntu-bionic-vexxhost | 16:19 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: WIP: Add WARNING build state https://review.opendev.org/724078 | 16:19 |
| donnyd | idk if vexxhost is ipv4 only or not though | 16:20 |
| corvus | tobiash: re https://review.opendev.org/722981 (serial pipeline manager) i could rebase that on 718531, or we could do it the other way around. the serial manager just has a copy of the dependent manager's queue methods, so it should be easy either way. | 16:20 |
| donnyd | Between those two labels is should make your jobs land exclusively on OE and Vexxhost | 16:20 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 16:21 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Run builder tests on expanded node https://review.opendev.org/724079 | 16:21 |
| mordred | donnyd: vexxhost has both - but I think it'll at least increase the chane that with some rechecks we'll hit OE | 16:21 |
| donnyd | well OE has both too - but I think it picks a primary interface | 16:22 |
| donnyd | OE has private ipv4 | 16:22 |
| openstackgerrit | Merged zuul/zuul-jobs master: Do not set buildset_fact if it's not present in results.json https://review.opendev.org/723524 | 16:22 |
| donnyd | in vexxhost it should pick the ipv4 addr to test against... but then again I know literally nothing about these jobs and how they work | 16:22 |
| mordred | yeha- I think in this case we want the builder node to pick the ipv6 address to talk to the registry node | 16:23 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Run builder tests on expanded node https://review.opendev.org/724079 | 16:23 |
| tobiash | corvus: as you wish. I'm out of office this week so if it blocks you I can rebase mine. | 16:24 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul master: Make task errors expandable https://review.opendev.org/723534 | 16:28 |
| *** rlandy is now known as rlandy|biab | 16:34 | |
| zbr | avass: i seen your WARNING feature and reminded me of the UNSTABLE status from jenkins. | 16:35 |
| *** evrardjp has quit IRC | 16:35 | |
| zbr | i like the idea, still there is one concern, if I remember well only SUCCESS is a success, anything else is a type of failure. | 16:35 |
| *** evrardjp has joined #zuul | 16:35 | |
| zbr | but for practical reasons, we may want to define minimum level needed for passing a build, which default should be SUCCESS. | 16:36 |
| zbr | some jobs may allow the WARNING/UNSTABLE but this should be optional IMHO. | 16:37 |
| avass | found out that ANSIBLE_EXTRA_PACKAGES doesn't work with the official docker images earlier unless you remove ansible from them first by the way | 16:37 |
| avass | since ManagedAnsible.validate() only checks if ansible is installed and not if the extra packages is installed | 16:38 |
| *** TomStappaerts has joined #zuul | 16:45 | |
| *** y2kenny has joined #zuul | 16:45 | |
| TomStappaerts | When merging several patchsets of the same project but of different branches default behavior is that all of these depend on each other. For my use case they actually do not. Is there a way to make Zuul think that patchsets of the same project are only dependent when they are of the same branch? | 16:47 |
| tristanC | corvus: thanks, left a couple of comment. Otherwise this is lgtm | 16:59 |
| tristanC | (re the serial pipeline) | 17:00 |
| clarkb | https://review.opendev.org/#/c/723763/ is a small change to update mypy in order to support python3.8 if anyone has a moment for that. Thank you tristanC for the quick review | 17:01 |
| *** jpena is now known as jpena|off | 17:05 | |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: Support multi-arch image builds with docker buildx https://review.opendev.org/722339 | 17:06 |
| openstackgerrit | Monty Taylor proposed zuul/zuul-jobs master: DNM Run builder tests on expanded node https://review.opendev.org/724079 | 17:06 |
| openstackgerrit | Guillaume Chauvel proposed zuul/zuul master: Add git name and email for quickstart executor https://review.opendev.org/724096 | 17:11 |
| *** guillaumec has joined #zuul | 17:15 | |
| *** jcapitao has quit IRC | 17:39 | |
| *** yolanda has quit IRC | 17:44 | |
| *** dpawlik has quit IRC | 17:48 | |
| *** cdearborn has joined #zuul | 17:51 | |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Validate ansible extra packages https://review.opendev.org/724110 | 18:01 |
| *** y2kenny has quit IRC | 18:03 | |
| avass | ^ that seems to do the trick | 18:06 |
| openstackgerrit | Albin Vass proposed zuul/zuul master: Validate ansible extra packages https://review.opendev.org/724110 | 18:09 |
| corvus | TomStappaerts: we're working on adding that right now actually: https://review.opendev.org/718531 | 18:11 |
| mnaser | hey -- is it possible that we merged something that broke jobs with buildset registries? | 18:17 |
| AJaeger | mordred: want to abandon https://review.opendev.org/#/c/723109/ with https://review.opendev.org/#/c/723524/6 approved and the stack at https://review.opendev.org/723640 . or do we need 723109? | 18:18 |
| mnaser | see: https://zuul.opendev.org/t/vexxhost/build/2c5e2247629946b18c0d7372def74006 | 18:18 |
| mnaser | Error initializing source docker://127.0.0.1:51295/vexxhost/openstack-operator:latest: Error reading manifest latest in 127.0.0.1:51295/vexxhost/openstack-operator: error parsing HTTP 404 response body: invalid character | 18:18 |
| mnaser | it looks like the local buildset registry is somehow returning a 404? | 18:18 |
| *** rlandy|biab is now known as rlandy | 18:20 | |
| mnaser | https://opendev.org/zuul/zuul-jobs/commit/6fb73060ec919d4e2364e418db84ce6aaa50492d seems to line up with the failures | 18:20 |
| corvus | mnaser: looking | 18:20 |
| mnaser | 12:33 pm est i had a change that was ok, this reported at 1:43 pm it was broken, that changed merged in that timeline-ish, i'm reading through it | 18:22 |
| mnaser | (btw my build/upload job also acts as a buildset registry to avoid building 2 vms, cause i always build the docker container anyways) | 18:22 |
| avass | mnaser, corvus: if something is failing there I would guess it's line 13 for some reason | 18:23 |
| mnaser | skipping: Conditional result was False | 18:24 |
| corvus | mnaser: yeah that's it: https://zuul.opendev.org/t/vexxhost/build/2c5e2247629946b18c0d7372def74006/console#3/0/0/ubuntu-bionic | 18:24 |
| mnaser | it skipped "Load information from zuul_return" | 18:24 |
| corvus | it didn't find results.json, but it should have | 18:24 |
| avass | corvus, mnsaer: aah I see it | 18:25 |
| mnaser | what generates results.json ? | 18:25 |
| mordred | the stat is running on the wrong host | 18:25 |
| avass | corvus, mnsaer: I missed a delegate_to: localhost on the stat | 18:25 |
| corvus | mnaser: zuul_return module | 18:25 |
| mordred | lookup runs on the executor - the stat is gonna be running on teh remote host | 18:25 |
| mordred | so the stat is goign to fail | 18:25 |
| mnaser | delegate_to: localhost | 18:25 |
| avass | pushing a fix | 18:25 |
| mnaser | yep that sounds like we all just agreed | 18:25 |
| mnaser | i can depends-on a change too once avass pushes that | 18:26 |
| TomStappaerts | corvus: Thx, will look at upgrading once it is in :D | 18:26 |
| corvus | mnaser: i think you'll have to wait for it to merge; i think that runs in a trusted context | 18:26 |
| mnaser | ahhhh yes you're right | 18:26 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: stat for result.json on the executor https://review.opendev.org/724116 | 18:26 |
| mordred | I agree | 18:26 |
| mnaser | +2 from me avass -- i'll let corvus +w to close it up :) | 18:27 |
| corvus | done | 18:27 |
| mnaser | ok, i'll report back when this lands | 18:27 |
| mnaser | thanks all D: | 18:28 |
| avass | looks like it fails | 18:35 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Revert "Do not set buildset_fact if it's not present in results.json" https://review.opendev.org/724120 | 18:35 |
| mnaser | avass: how come its failing? | 18:35 |
| * mnaser looks | 18:35 | |
| mnaser | ok yeah | 18:36 |
| mnaser | avass: i guess it's because there's nothing that is doing zuul_return in that test? | 18:36 |
| mnaser | but results.json seems to exist as an empty file | 18:37 |
| mnaser | not an empty json string | 18:37 |
| *** ysandeep is now known as ysandeep|away | 18:37 | |
| mordred | I'd be willing to bet it's not short-circuit evaluating the when list | 18:38 |
| avass | mordred: I think it is, since we didn't get the exception earlier since it didn't find the file with the stat | 18:39 |
| mordred | ah - nod, what mnaser said then | 18:39 |
| mnaser | i could be wrong though | 18:39 |
| mnaser | the error does seem to imply that the file is emtpy | 18:39 |
| mnaser | "Expecting value: line 1 column 1 (char 0)" | 18:40 |
| mordred | I agree - let's revert - then we can figure it out with less pressure | 18:40 |
| mordred | because I really do think this is an improvment | 18:40 |
| mnaser | yes i agree | 18:40 |
| avass | mnsaer, mordred: yeah looks like it throws that exception if it's not a json file | 18:43 |
| mordred | avass: so - maybe we should just check that stat.st_size > 0 | 18:45 |
| mordred | because it should either be empty or have json in it | 18:45 |
| avass | mordred: yeah it's that or somehow check if there's json in it, I'd like the latter | 18:47 |
| *** saneax has quit IRC | 18:49 | |
| mordred | yeah | 18:50 |
| mordred | I would to - if there was a way to do it other than just trying to load it and failing | 18:50 |
| avass | mordred: is the jmespath package installed on the executor? | 18:52 |
| avass | if so the json_query filter works | 18:52 |
| *** TomStappaerts has left #zuul | 18:53 | |
| avass | wait no, nevermind | 18:53 |
| mordred | avass: we need a new module | 18:54 |
| mordred | avass: "load_json_or_none" | 18:54 |
| mordred | that does a python try: return json.loads(path) except: return None | 18:54 |
| avass | mordred: yeah | 18:54 |
| mordred | avass: almost makes me want to write a custom zuul filter plugin that does that | 18:58 |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "Do not set buildset_fact if it's not present in results.json" https://review.opendev.org/724120 | 19:09 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Revert "Revert "Do not set buildset_fact if it's not present in results.json"" https://review.opendev.org/724132 | 19:13 |
| avass | mordred: I couldn't find a better way so lets just check the filesize | 19:13 |
| avass | mordred: actually, I found a better way | 19:20 |
| mordred | woot | 19:20 |
| avass | if only slightly | 19:21 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Revert "Revert "Do not set buildset_fact if it's not present in results.json"" https://review.opendev.org/724132 | 19:23 |
| avass | mordred: I guess it's checking for empty string vs checking if filesize > 0 | 19:23 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Revert "Revert "Do not set buildset_fact if it's not present in results.json"" https://review.opendev.org/724132 | 19:24 |
| avass | mordred: whichever we prefer, compare patchsets 1 vs 3 :) | 19:24 |
| openstackgerrit | Albin Vass proposed zuul/zuul-jobs master: Revert "Revert "Do not set buildset_fact if it's not present in results.json"" https://review.opendev.org/724132 | 19:26 |
| avass | I take it back, we need that variable undefined. I'm getting too tired to work.. | 19:27 |
| avass | I'll check in tomorrow :) | 19:27 |
| fungi | i woke up too tired to work. you deserve some rest! | 19:28 |
| mordred | fungi: we're supposed to be working? | 19:34 |
| *** zxiiro has quit IRC | 19:36 | |
| fungi | i'm not | 19:36 |
| fungi | depends on how you define "work" | 19:36 |
| *** threestrands has quit IRC | 20:09 | |
| tristanC | zuul-maint : there are a few refactoring change for the zuul-operator to reduce the size of the main resources file. Could you please have a look at the changes between https://review.opendev.org/719965 and https://review.opendev.org/#/c/720024 . Thank you in advance | 20:10 |
| tristanC | it seems like i'm currently the only one pushing for and using the zuul-operator, would it be possible to adopt a lighter approval process? | 20:25 |
| tristanC | i feel bad for requesting reviews repeatedly , but on the other hand jugling with open changes makes it difficult to develop the project | 20:27 |
| tristanC | so would it be possible to implement this workflow: https://github.com/dhall-lang/dhall-lang/blob/master/.github/CONTRIBUTING.md#how-do-changes-get-approved ? | 20:28 |
| *** dustinc has joined #zuul | 20:34 | |
| clarkb | I'll admit to being swamped with a lot of opendev stuff lately | 20:41 |
| clarkb | I personally would be ok with more relaxed operator reviews, but I'm also not a primary consumer/user of it | 20:41 |
| openstackgerrit | Merged zuul/zuul master: Bump mypy for py3.8 support https://review.opendev.org/723763 | 20:44 |
| tristanC | in the proposed rules, we could consider the changes that affects the CRD to requires review, then the rest could follow the simplified `rules for merging a change` | 20:45 |
| openstackgerrit | Merged zuul/zuul-jobs master: Add ensure-virtualenv https://review.opendev.org/723309 | 20:51 |
| corvus | tristanC: tobiash is out this week | 21:21 |
| corvus | tristanC: we could adopt a lighter process, but also, he's been a keen reviewer in the past, so i don't know if you want to wait so he doesn't get too far behind :) | 21:23 |
| corvus | tristanC: if you want a simplified process, i'd rather not do something complicated and time-based like what's on the dhall page; i'd rather you just ask for reviews of substantial changes and not worry about refactoring changes that don't change the inputs or outputs :) | 21:26 |
| corvus | tristanC: i think it's probably fine to merge those 5 changes now | 21:29 |
| corvus | tristanC: https://review.opendev.org/716300 apparently depends on https://review.opendev.org/716298 can you look at that? | 21:31 |
| fungi | that's one good thing about a project with comprehensive tests. when a cleanup/refactor doesn't require altering the tests, it's probably safe to approve without rigorous review | 21:31 |
| clarkb | corvus: what is the expected behavior from Zuul when you depends on a PR or change in an unknown repo? | 21:38 |
| fungi | i assumed it was like when you get the url wrong in a depends-on, it just gets ignored | 21:39 |
| corvus | i think what fungi said but not positive | 21:39 |
| clarkb | ok, our testing may not quite check that. The reason that our test_crd_check_unknown seems to be failing is that our fake github client doesn't know about the unknown repo and throws and exception | 21:40 |
| clarkb | this causes gerrit to ignore its known child change too | 21:41 |
| clarkb | when the test succeeds it seems to find that repo (which it shouldn't?) | 21:41 |
| clarkb | I'm still trying to unpuzzle what is going on but that helps me set some general expectations | 21:41 |
| clarkb | https://opendev.org/zuul/zuul/src/branch/master/tests/unit/test_cross_crd.py#L431-L449 in that test why would we assert B is merged on line 448 is what I think I'm getting at | 21:43 |
| clarkb | somehow that works most of the time but then like 1/48 test runs locally it fails (and I think the failure is correct) | 21:43 |
| clarkb | oh what thats assert false B is merged ok I'm not completely crazy :) | 21:43 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add serial pipeline manager https://review.opendev.org/722981 | 21:46 |
| fungi | 1:48 is a remarkably precise ratio | 21:48 |
| clarkb | fungi: well I only have one sample as I started looking at logs after my first failure | 21:52 |
| clarkb | I *think* I may know what the issue is | 21:52 |
| clarkb | I've made a change and am running it through the gauntlet | 21:52 |
| clarkb | however I don't know that this is asserting correct zuul behavior | 21:52 |
| openstackgerrit | Merged zuul/zuul-operator master: Use ensure-* roles https://review.opendev.org/719401 | 22:00 |
| clarkb | the issue seems related to how we recursively create fake githubclients and githubsessions. https://opendev.org/zuul/zuul/src/branch/master/tests/fakegithub.py#L621 and https://opendev.org/zuul/zuul/src/branch/master/tests/fakegithub.py#L596 | 22:06 |
| clarkb | however undoing that breaks in new exciting ways | 22:06 |
| fungi | this is the time of day where i tend to not prefer excitation, personally | 22:09 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Add more docs about pipelines https://review.opendev.org/724185 | 22:17 |
| openstackgerrit | Andy Ladjadj proposed zuul/zuul master: Add timezone select on topbar https://review.opendev.org/722653 | 22:20 |
| openstackgerrit | Andy Ladjadj proposed zuul/zuul master: Add new timezone selector in web interface https://review.opendev.org/722653 | 22:21 |
| clarkb | and I think Isorted out the race | 22:25 |
| clarkb | now rerunning through the gauntlet | 22:25 |
| openstackgerrit | Tristan Cacqueray proposed zuul/zuul-base-jobs master: base: skip role incompatible with kubectl connection https://review.opendev.org/716298 | 22:29 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Fix test_crd_check_unknown tests https://review.opendev.org/724191 | 22:47 |
| openstackgerrit | Clark Boylan proposed zuul/zuul master: Simplify FakeGithubClient and FakeGithubSession https://review.opendev.org/724192 | 22:47 |
| clarkb | ok that stack passed 100 local test runs | 22:47 |
| clarkb | I believe the fix is correct there | 22:48 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Make fake test Gerrit merger more realistic https://review.opendev.org/722982 | 23:00 |
| corvus | clarkb: thanks, i think one of my changes hit that too | 23:00 |
| *** Goneri has quit IRC | 23:39 | |
| *** tosky has quit IRC | 23:39 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!