*** rlandy|bbl is now known as rlandy | 02:21 | |
*** bhavikdbavishi has joined #zuul | 03:34 | |
SpamapS | Is there anything a casual developer can do to help with the test failures? | 04:28 |
---|---|---|
tobiash | SpamapS: the fix merged recently: https://review.opendev.org/659296 | 05:16 |
tobiash | so once puppet went through (is it already?) we can recheck, but we should start with https://review.opendev.org/659329 and https://review.opendev.org/659262 which are high priority fixes | 05:17 |
tobiash | SpamapS: btw, both could use a second review ;) | 05:18 |
tobiash | or any other zuul-maint ^ | 05:18 |
tobiash | oh wait, I can approve one of them :) | 05:18 |
tobiash | so only 659329 needs a second review | 05:19 |
*** saneax has quit IRC | 05:23 | |
*** saneax has joined #zuul | 05:29 | |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures https://review.opendev.org/657914 | 05:40 |
tristanC | tobiash: shouldn't those "pattern" in line be replaced by line == "pattern" as tried in https://review.opendev.org/#/c/657914/11/zuul/ansible/base/callback/zuul_stream.py@178 ? | 05:40 |
tristanC | tobiash: i wonder if the zuul-tox-stream is not failing because the task output get mangled with those pattern and incorectly skipped | 05:42 |
tobiash | tristanC: I don't know anymore, I thought there was a reason about that but I could be wrong | 05:42 |
tobiash | I think the 'in' is used as a safety mechanism that we detect the exit code reliably because that is the stop sign to stop streaming | 05:43 |
tobiash | tristanC: if we want to be more sophisticated we need to split the parts we look for from the string (if it's intermangled) and log them before | 05:45 |
*** gtema has joined #zuul | 05:45 | |
tobiash | tristanC: so under the assumption that the line could be intermangled we currently might drop the last part of the log | 05:45 |
tobiash | tristanC: but if so we might also want to analyse why it's intermangled | 05:46 |
tristanC | tobiash: that's why i've been trying to do in PS10 of 657914 but without luck | 05:47 |
tobiash | so you think it gets mangled? | 05:48 |
tristanC | tobiash: well i don't know, but if it is, then the task output may get discard and the test is failing because it can't find in the final log file | 05:48 |
tristanC | tobiash: though i'm not sure what's going on, i tried sending any traceback from zuul_console back to zuul_stream in https://review.opendev.org/#/c/657914/10/zuul/ansible/base/library/zuul_console.py@193 | 05:49 |
tristanC | tobiash: but for some reason, i only get "Log not found", not the extra "LOG NOT FOUND" added, e.g. in http://logs.openstack.org/14/657914/10/check/zuul-tox-remote/fbacac1/testr_results.html.gz | 05:50 |
tobiash | that's really weird | 05:51 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures https://review.opendev.org/657914 | 06:01 |
*** quiquell has quit IRC | 06:04 | |
*** quiquell has joined #zuul | 06:04 | |
*** gtema has quit IRC | 06:05 | |
*** quiquell has quit IRC | 06:05 | |
*** quiquell has joined #zuul | 06:06 | |
AJaeger | is this a new error http://logs.openstack.org/38/658738/6/check/zuul-quick-start/effaca0/job-output.txt.gz#_2019-05-16_05_30_18_288159 | 06:10 |
AJaeger | "remote: You need 'Create' rights to create new references." | 06:10 |
*** bjackman_ has joined #zuul | 06:12 | |
tobiash | AJaeger: so far I only saw zuul-quick-start failures where also the post_failures of zuul-build-image was present | 06:22 |
AJaeger | tobiash: yeah | 06:23 |
tobiash | so let's see if this gets fixed too when the first zuul-build-image jobs are working (I already see some in the pipeline) | 06:23 |
AJaeger | tobiash: good ;) | 06:26 |
tobiash | oh maybe I was wrong, the post failure happens after zuul-build-image gets resumed | 06:28 |
tristanC | do we know if the skopeo package got reverted on the executor nodes? | 06:29 |
AJaeger | tristanC: we expected that the usual puppet/ansible runs would do so - perhaps it didn't happen? | 06:31 |
AJaeger | ianw: could you check, please? ^ | 06:31 |
tristanC | grep "subuid mapping" $(which skopeo) should not match when an unaffected version of skopeo is installed | 06:34 |
*** gtema has joined #zuul | 06:40 | |
tristanC | tobiash: it seems like zuul-quick-start is now running with "Gerrit Code Review 3.0.0 ready" :) | 06:40 |
tobiash | so another update that breaks us? | 06:43 |
AJaeger | should we cap? | 06:44 |
tristanC | well, gerrit seems to need a new ACL to create project | 06:44 |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: quick-start: pin docker image to 2.16.8 https://review.opendev.org/659454 | 06:46 |
tobiash | +2 | 06:58 |
tobiash | any other zuul-maint awake to +3 ^? | 06:58 |
*** openstackgerrit has quit IRC | 07:03 | |
*** pcaruana has joined #zuul | 07:13 | |
AJaeger | tobiash: I suggest you do a single core approval on that one to move forward. Alternative: ianw might still be around, corvus might be awake by now but not sure whether he has time to review... | 07:15 |
tobiash | AJaeger: yes, that was my plan as soon as the test was successful | 07:16 |
AJaeger | ;) | 07:16 |
*** hashar has joined #zuul | 07:17 | |
tobiash | done | 07:17 |
AJaeger | zuul-quick-start succeeded, still post_failure for zuul-build-image ;( | 07:25 |
tobiash | what a wonderful day | 07:25 |
AJaeger | http://logs.openstack.org/54/659454/1/check/zuul-build-image/9b933f4/job-output.txt.gz#_2019-05-16_07_23_38_792642 | 07:25 |
AJaeger | subuid mapping? | 07:25 |
tobiash | AJaeger: that makes me think that skopeo is probably not (yet) downgraded on the executors | 07:26 |
tobiash | I guess we need an infra-root to check that | 07:26 |
AJaeger | ok | 07:27 |
ianw | hrm, i saw it running i thought | 07:27 |
*** openstackgerrit has joined #zuul | 07:27 | |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures https://review.opendev.org/657914 | 07:27 |
ianw | ii skopeo 0.1.36-1~dev~ubuntu16.04.2~ppa17 | 07:27 |
tobiash | but to keep us going we could squash 659218 and 659454 together | 07:28 |
ianw | hrm, hang on 17 there, not 14 | 07:28 |
tobiash | that should fix the quickstart and make the image build non-voting temporarily | 07:29 |
ianw | argh, hrm puppet has failed | 07:29 |
tobiash | AJaeger: want to do the squash? I'm busy with own operational topics atm, but can review | 07:30 |
AJaeger | can do... | 07:31 |
tobiash | thanks | 07:31 |
ianw | May 16 06:38:38 ze09 puppet-user[13943]: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install skopeo=0.1.36-1~dev~ubuntu16.04.2~ppa14' returned 100: Reading package lists... | 07:31 |
ianw | May 16 06:38:38 ze09 puppet-user[13943]: Building dependency tree... | 07:31 |
ianw | May 16 06:38:38 ze09 puppet-user[13943]: Reading state information... | 07:31 |
ianw | May 16 06:38:38 ze09 puppet-user[13943]: E: Version '0.1.36-1~dev~ubuntu16.04.2~ppa14' for 'skopeo' was not found | 07:31 |
ianw | it's kicked out of the repo? | 07:32 |
*** jpena|off is now known as jpena | 07:32 | |
AJaeger | Argh ;( | 07:32 |
ianw | ooohh, i think it keeps two .. and it seem another has built in the last hour or so | 07:33 |
openstackgerrit | Andreas Jaeger proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid https://review.opendev.org/659218 | 07:34 |
AJaeger | tobiash, ianw ^ | 07:35 |
openstackgerrit | Andreas Jaeger proposed zuul/zuul master: Make image build jobs voting again https://review.opendev.org/659242 | 07:35 |
tobiash | AJaeger: single core approved | 07:35 |
AJaeger | thanks | 07:36 |
ianw | so basically that skopeo pin can't work, because the ppa doesn't keep old packages | 07:36 |
tobiash | hopefully this passes so we have more time to figure out the skopeo thing | 07:36 |
*** saneax has quit IRC | 07:41 | |
ianw | i proposed a revert on the ~14 install ... i don't know where we could get that now. it's not in the apt archives | 07:41 |
ianw | like on the host apt archives | 07:41 |
ianw | it's not in the ppa | 07:41 |
AJaeger | what fun ;( Thanks, ianw | 07:42 |
*** bhavikdbavishi has quit IRC | 07:43 | |
*** themroc has joined #zuul | 07:44 | |
*** saneax has joined #zuul | 07:51 | |
*** themr0c has joined #zuul | 07:52 | |
*** themroc has quit IRC | 07:54 | |
ianw | i guess we could build our own in the opentack-ci-core ppa along with all the other things we have there. that's probably the next step but unforutnately i won't have time tonight to go down that path | 07:57 |
ianw | if someone wants to get the control file etc together (even build in their own ppa) we can easily migrate it in | 07:57 |
tristanC | tobiash: i don't understand where the "Log not found" in http://logs.openstack.org/14/657914/13/check/zuul-tox-remote/ee0855f/testr_results.html.gz is coming from. This should have been disabled by https://review.opendev.org/#/c/657914/13/zuul/ansible/base/library/zuul_console.py ... | 08:05 |
tristanC | could it be that the zuul.opendev.org executor library somehow be used by the job? | 08:07 |
tobiash | I don't think so, the test is executed on the node, not on the executor | 08:18 |
*** panda is now known as panda|rover | 09:17 | |
openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 10:11 |
*** pcaruana has quit IRC | 10:27 | |
openstackgerrit | Tristan Cacqueray proposed zuul/zuul master: [WIP] runner: enable reuse of a job-dir https://review.opendev.org/657955 | 10:27 |
*** sshnaidm|afk is now known as sshnaidm | 10:38 | |
*** bhavikdbavishi has joined #zuul | 11:00 | |
*** jpena is now known as jpena|lunch | 11:03 | |
*** panda|rover is now known as panda|rover|eat | 11:04 | |
*** gtema has quit IRC | 11:07 | |
*** pcaruana has joined #zuul | 11:37 | |
*** panda|rover|eat is now known as panda | 12:06 | |
*** panda is now known as panda|rover | 12:11 | |
*** saneax has quit IRC | 12:17 | |
*** themr0c has quit IRC | 12:24 | |
*** themr0c has joined #zuul | 12:25 | |
*** jpena|lunch is now known as jpena | 12:29 | |
*** rlandy has joined #zuul | 12:34 | |
openstackgerrit | Merged zuul/zuul master: bubblewrap: bind mount /etc/subuid https://review.opendev.org/659218 | 12:43 |
tobiash | \o/ | 12:45 |
AJaeger | tobiash, finally \o/ | 12:45 |
AJaeger | ;) | 12:45 |
tobiash | zuul-maint: someone here to review https://review.opendev.org/659329 ? | 12:46 |
*** saneax has joined #zuul | 12:47 | |
*** snapiri has quit IRC | 12:56 | |
*** bjackman_ has quit IRC | 13:03 | |
Shrews | tobiash: lgtm | 13:05 |
tobiash | \o/ | 13:05 |
*** markwork has joined #zuul | 13:06 | |
markwork | Hey, when I build the containers from my patchset I get an empty zuul-web page, with cherry py complaining that path '/' was not found :( | 13:07 |
*** gtema has joined #zuul | 13:16 | |
openstackgerrit | Mark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source https://review.opendev.org/658835 | 13:24 |
*** ianychoi has joined #zuul | 13:26 | |
clarkb | markwork: if not using co tainers you may have to jump through hoops to generate the js but I though the containers did all that for you. This was a recent build? | 13:34 |
markwork | Yes, I just ran the container build on my machine and just inserted a 'pip install --upgrade .' to install my changeset. Just an empty page | 13:36 |
markwork | But the container build runs | 13:36 |
clarkb | mordred: ^ any ideas? | 13:37 |
clarkb | actually I wonder if the pip install upgrade clears the old js content | 13:38 |
clarkb | I think if you dont modify the dockerfile and instead rebuild with your change checked out it will get included | 13:39 |
clarkb | maybe try that | 13:39 |
markwork | Yes, I think it's something like this. I assume I can just `docker build . ` from my checkout and it will take my current source? I think this may be stupidity on my side... | 13:39 |
*** gtema has quit IRC | 13:39 | |
mordred | markwork: yes - docker build . should Just Work™ | 13:41 |
*** pcaruana has quit IRC | 13:56 | |
markwork | mordred, clarkb: that was it, now it works :D so almost ready for demo day. | 14:04 |
mordred | \o/ | 14:05 |
mordred | markwork: I'm very excited about that and hope it goes well (also, thanks again for working on that bitbucket driver!) | 14:05 |
markwork | mordred: you're welcome, it's an interesting codebase and a lot of fun :) | 14:08 |
markwork | still need to push some tests and then the here-and-theres | 14:09 |
markwork | We now have a zuul-test instance that works without Gerrit, just using Bitbucket and it has just built its first project :D | 14:09 |
mordred | \o/ | 14:10 |
*** gtema has joined #zuul | 14:30 | |
markwork | What can be the reason that `builds` and `buildsets` are in state `Loading...` in the web ui? They never display anything... | 14:32 |
clarkb | you need a sql reporter for those to work | 14:33 |
clarkb | therehas been talk of just making the sql reporter mandatory but currently it is optional as those tabs are the only things that dont function without it | 14:34 |
markwork | clarkb: understood, very helpful, thanks :) | 14:35 |
*** pcaruana has joined #zuul | 14:37 | |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.8 https://review.opendev.org/631933 | 14:42 |
openstackgerrit | Merged zuul/zuul master: Fix missing check if logger is None https://review.opendev.org/659262 | 14:44 |
markwork | Anybody got an idea why I get this message: Project <ProjectConfig bitbucket.smaato.net/SYS-IC/animal source: SYS-IC/animal/.zuul.yaml@update_readme {ImpliedBranchMatcher:update_readme}> did not match item <QueueItem 0x7fefba0e2a58 for <Change 0x7fefb90c72e8 SYS-IC/animal None,05bcb585354268af13b8c44f050b1e53a5089547> in check> | 14:50 |
markwork | Apparently some branch matcher rejects the branch? What's the background? | 14:50 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: executor: use node python path https://review.opendev.org/637339 | 14:51 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Update quickstart nodepool node to python3 https://review.opendev.org/658486 | 14:51 |
pabelanger | tristanC: just a rebase to confirm working on quickstart, I'll try and add test coverage for my comment today | 14:51 |
pabelanger | Wouldn't mind landing that soon for python3 only images | 14:51 |
openstackgerrit | Merged zuul/zuul master: Handle artifacts on non change types. https://review.opendev.org/659329 | 14:57 |
tobiash | \o/ | 14:57 |
clarkb | markwork: if your zuul config is in a repo with branches (more than just master) zuul assumes the config on a branch is branch specific | 14:57 |
clarkb | markwork: we (opendev) use brnachless (master only) config repos to manage jobs that should apply globally | 14:58 |
*** bhavikdbavishi has quit IRC | 15:00 | |
*** hashar has quit IRC | 15:05 | |
clarkb | catching up, did we make the zuul build image/upload image jobs non voting to work around the skopeo problem? | 15:10 |
AJaeger | clarkb: yes, we did - since several problems were showing up | 15:10 |
clarkb | k thanks | 15:11 |
AJaeger | clarkb: change was https://review.opendev.org/659218 , the change to make these voting again is https://review.opendev.org/659242 | 15:15 |
openstackgerrit | Merged zuul/zuul master: Fix occasional gate resets with github https://review.opendev.org/650387 | 15:19 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.8 https://review.opendev.org/631933 | 15:28 |
pabelanger | Shrews: okay, ^ should remove copypasta of zuul_stream and add conditional logic | 15:29 |
pabelanger | then, once 2.8.0 is released later today (hopefully) we can remove WIP | 15:29 |
Shrews | pabelanger: is 'rescued' always present? If not, we could miss the 'skipped' portion, based on your conditional. | 15:35 |
openstackgerrit | Merged zuul/zuul master: Centralize logging adapters https://review.opendev.org/658644 | 15:38 |
*** pcaruana has quit IRC | 15:47 | |
openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 15:49 |
openstackgerrit | Fabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/ https://review.opendev.org/604404 | 15:50 |
openstackgerrit | Merged zuul/zuul master: web: upgrade react and react-scripts to ^2.0.0 https://review.opendev.org/631902 | 15:53 |
*** saneax has quit IRC | 16:01 | |
pabelanger | Shrews: resuced is always present in stable-2.8+ | 16:04 |
*** themr0c has quit IRC | 16:04 | |
pabelanger | and skipped, also only apply stable-2.8+ | 16:05 |
pabelanger | need to relocate, afk | 16:05 |
markwork | clarkb: Hmm, strangely this error has happened only for a branch that has an underscore in it | 16:06 |
markwork | (I think) | 16:06 |
markwork | And exactly that branch is not building | 16:06 |
openstackgerrit | Merged zuul/zuul master: Cleanup executor specific requirements https://review.opendev.org/649428 | 16:17 |
AJaeger | tobiash: zuul-build-image succeeded on https://review.opendev.org/580547 ;) | 16:19 |
tobiash | yay | 16:20 |
AJaeger | want to +A 659242 ? Hope it passes ;) | 16:21 |
AJaeger | ah, you did! | 16:21 |
AJaeger | release-zuul-python fails, see http://logs.openstack.org/ee/ee4b6b1a27d1de95a605e188ae9e36d7f1597ebb/post/release-zuul-python/f225995/job-output.txt.gz#_2019-05-16_16_10_21_009869 | 16:23 |
AJaeger | yarn install failed ;( | 16:23 |
AJaeger | zuul-maint, any ideas? ^ | 16:24 |
tobiash | Looks like we have a run... | 16:24 |
AJaeger | yeah ;( | 16:25 |
AJaeger | and still many failures due to tox-remote ;( | 16:25 |
tobiash | Actually I'd expect py36 to fail as well if that's broken | 16:25 |
*** mattw4 has joined #zuul | 16:28 | |
clarkb | I've just restarted opendev zuul to pull in the artifacts handling fix | 16:33 |
clarkb | looks like we also pulled ina github fix | 16:33 |
clarkb | and the react update | 16:33 |
clarkb | (an fyi mroe than anything else) | 16:33 |
fungi | and as of a few minutes ago we've been running with tristanC's pr for https://github.com/containers/skopeo/issues/649 applied on all our executors for opendev | 16:33 |
clarkb | tobiash: AJaeger we need to install newer nodejs for yarn looks like | 16:34 |
clarkb | we install it from nodesource so that is probably just a version number in the role somewhere that we bump | 16:34 |
tobiash | because of 631902? | 16:34 |
clarkb | ya that could be if that updated webpack | 16:34 |
clarkb | I think we should just use node 10 or whatever is latest stable and be happy with that | 16:35 |
AJaeger | that updated node_version - is that ignored? See https://review.opendev.org/#/c/631902/10/.zuul.yaml | 16:35 |
clarkb | AJaeger: it updated it on zuul-build-dashboard but not release-zuul-python | 16:36 |
*** sshnaidm is now known as sshnaidm|afk | 16:37 | |
AJaeger | clarkb: I see - needs updating in project-config. I'll try... | 16:37 |
*** markwork has quit IRC | 16:37 | |
clarkb | ok thanks | 16:37 |
AJaeger | clarkb, tobiash, https://review.opendev.org/659620 | 16:40 |
clarkb | +2 | 16:40 |
tobiash | +1 | 16:41 |
pabelanger | AJaeger: 659620 is not formatted correctly | 16:47 |
clarkb | pabelanger: AJaeger remove the - prefix? | 16:49 |
pabelanger | yup | 16:49 |
clarkb | edited | 16:51 |
pabelanger | +3 | 16:52 |
*** markwork has joined #zuul | 17:06 | |
AJaeger | thanks, clarkb ! | 17:06 |
*** jpena is now known as jpena|off | 17:11 | |
*** panda|rover is now known as panda|rover|off | 17:13 | |
openstackgerrit | Merged zuul/zuul master: web: honor allowed-labels setting in the REST API https://review.opendev.org/653895 | 17:14 |
AJaeger | tobiash: FYI, looks like quite a few of your changes are in merge conflict now and need a rebase... | 17:17 |
tobiash | AJaeger: yes, I've noticed that, will do in a bit | 17:17 |
AJaeger | sure, no urgency... | 17:17 |
openstackgerrit | Merged zuul/zuul master: docs: fix a typo in executor zone documentation https://review.opendev.org/653663 | 17:20 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate github logs with the event id https://review.opendev.org/658645 | 17:20 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate gerrit event logs https://review.opendev.org/658646 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Attach event to queue item https://review.opendev.org/658647 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate some logs in the scheduler with event id https://review.opendev.org/658648 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs in the zuul driver with event ids https://review.opendev.org/658649 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add event id to timer events https://review.opendev.org/658650 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate pipeline processing with event id https://review.opendev.org/658651 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate merger logs with event id https://review.opendev.org/658652 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id https://review.opendev.org/658888 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 17:21 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 17:21 |
tobiash | was just a minimal conflict... | 17:21 |
AJaeger | ;) | 17:22 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 17:24 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 17:24 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 17:25 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 17:25 |
*** electrofelix has quit IRC | 17:38 | |
-openstackstatus- NOTICE: Gerrit is being restarted to add gitweb links back to Gerrit. Sorry for the noise. | 17:38 | |
*** pcaruana has joined #zuul | 17:39 | |
openstackgerrit | Merged zuul/zuul master: gerrit: Add some quoting in 'gerrit query' commands https://review.opendev.org/649879 | 17:41 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: further zuul-remote debugging attempts https://review.opendev.org/659631 | 17:46 |
tobiash | hrm, again in merge conflict :/ | 17:48 |
tobiash | or just gerrit things it's in merge conflict | 17:49 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate github logs with the event id https://review.opendev.org/658645 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate gerrit event logs https://review.opendev.org/658646 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Attach event to queue item https://review.opendev.org/658647 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate some logs in the scheduler with event id https://review.opendev.org/658648 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate logs in the zuul driver with event ids https://review.opendev.org/658649 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add event id to timer events https://review.opendev.org/658650 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate pipeline processing with event id https://review.opendev.org/658651 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate merger logs with event id https://review.opendev.org/658652 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id https://review.opendev.org/658888 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Annotate node request processing with event id https://review.opendev.org/658889 | 17:51 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id https://review.opendev.org/658895 | 17:51 |
tobiash | nope, really conflict | 17:51 |
*** gtema has quit IRC | 17:56 | |
markwork | The shortname of a build in the ui, is that change.changeset? I set it to the commit Sha and get weird commit shas in the ui, can I change change.changeset to the branch? | 17:57 |
clarkb | markwork: for gerrit it is change.number,change.patchset (which might be aliased as change.changeset) and for tags, refs etc its the first 7 characters of the sha1 for that object | 17:58 |
fungi | also for some events it's a head name or a tag name | 17:59 |
fungi | er, wait, in which ui tab? | 18:00 |
fungi | are you asking about the status tab? | 18:01 |
openstackgerrit | Merged zuul/zuul master: github: add event filter debug https://review.opendev.org/580547 | 18:02 |
markwork | clarkb: thanks a lot, I'll fiddle with those | 18:02 |
fungi | in our tag-triggered pipelines we see the tag name there, in commit-triggered pipelines it's an abbreviation of the commit id, in gerrit-change-triggered pipelines it's the gerrit change number and patch number comma-separated, in our periodic pipelines it's the branch name... | 18:04 |
tobiash | markwork: the rendering is here: https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/status/ChangePanel.jsx#L117 | 18:04 |
fungi | oh, and for github-pr-triggered pipelines it's the pr number and an abbreviation of the commit id comma-separated | 18:05 |
tobiash | markwork: bitbucket probably has pull request numbers like github? | 18:05 |
markwork | tobiash: yes it has, I'm just building and update | 18:12 |
markwork | s/and/an/ | 18:12 |
tobiash | markwork: yeah, so you might want to handle it similar to github | 18:13 |
*** ianychoi has quit IRC | 18:15 | |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add foreground option https://review.opendev.org/635649 | 18:16 |
SpamapS | so, do I need to rebase 656541 to pick up whatever fixes the quick start and tox-remote fails? | 18:26 |
SpamapS | It's on recheck #5 | 18:26 |
SpamapS | Also, can we get rid of "clean check" for zuul so we don't have to do this madness? | 18:27 |
SpamapS | (since, I assume, check has to pass before it is tested merged on top of master) | 18:27 |
fungi | we need to move it into the zuul tenant in opendev, and then yes absolutely | 18:27 |
fungi | though check is also tested merged on tip of master always | 18:28 |
fungi | dropping the clean-check model would just mean we're okay approving changes which have no or negative verify vote from zuul | 18:28 |
clarkb | I don't know if tox-remote has been fixed | 18:29 |
clarkb | (I've still seen it fail here and there) | 18:29 |
fungi | quick-start should be fixed now, so simply rechecking should get that passing unless we still missed something | 18:30 |
tobiash | tox-remote is not yet fixed and hard to reproduce an debug | 18:30 |
fungi | a big part of what was breaking the quick-start job was not actually due to zuul's codebase, but a regression in skopeo which we've worked around by installing a temporary fork | 18:30 |
fungi | while waiting on the pr which solves it to merge | 18:31 |
clarkb | SpamapS: and no rebase requires due to the out of band nature of the fix | 18:32 |
Shrews | i've run the tox-remote jobs locally 100's of times, can't get it to fail | 18:32 |
clarkb | SpamapS: also zuul always rebases your change onto the target branch before testing it so you get that for free anyway | 18:32 |
tobiash | Shrews: me 2 | 18:33 |
openstackgerrit | Tobias Henkel proposed zuul/zuul master: DNM: further zuul-remote debugging attempts https://review.opendev.org/659631 | 18:41 |
mordred | SpamapS: other people have already said yes, but I want to echo the yes to removing clean-check from zuul | 18:42 |
mordred | as soon as we move it to the zuul tenant, we'll be good to go | 18:42 |
tobiash | is the replication to https://opendev.org/zuul/zuul lagging/broken after the gerrit restart? | 18:43 |
clarkb | tobiash: lagging | 18:43 |
tobiash | k | 18:43 |
clarkb | tobiash: gerrit does a full replication after a restart and takes a while to catch up from that | 18:43 |
clarkb | down to 8.2k tasks now | 18:43 |
tobiash | thanks, I just wondered | 18:43 |
mordred | clarkb: it would be really awesome if it would not do a full replication after a restart | 18:45 |
clarkb | mordred: ya updating the replication plugin to not do that by default is on my todo list | 18:46 |
clarkb | (you can always trigger full replication manually if desired) | 18:46 |
mordred | yeah | 18:46 |
openstackgerrit | Merged zuul/zuul master: Don't check out a commit if it is already checked out https://review.opendev.org/163922 | 18:47 |
AJaeger | wow, look at that number ^ | 18:48 |
mordred | AJaeger: that's amazing | 18:48 |
AJaeger | March 2015 - so, 4 years later... | 18:49 |
fungi | clarkb: off topic for this channel, but what if the gerrit replication plug-in stored a checksum of some sort (summing over the list of git object ids, or whatever) and checked that at start to decide whether it should rereplicate a given repository? | 18:50 |
fungi | though i guess that supposes that there weren't any queued replication events which hadn't completed when the service was stopped | 18:51 |
clarkb | ya it could keep track of whether or not updates have been replicated fully | 18:54 |
clarkb | that seems like a lot more work (and needing a database) | 18:54 |
fungi | well... it *has* a database... but point taken | 18:54 |
openstackgerrit | Merged zuul/zuul master: Use a more visible selection color https://review.opendev.org/648865 | 18:59 |
mordred | fungi: it's *really* trying to delete its database | 19:02 |
fungi | nah, just moving it to a different medium | 19:03 |
fungi | it's building databases of information in git repositories | 19:03 |
clarkb | can you safely store data about what has been replicated in the git repo itself :P | 19:04 |
clarkb | (you probably can) | 19:04 |
fungi | as long as you don't replicate that data. otherwise you get to argue with gödel about completeness | 19:05 |
openstackgerrit | Paul Belanger proposed zuul/zuul master: Support Ansible 2.8 https://review.opendev.org/631933 | 19:07 |
pabelanger | ansible 2.8.0 is out, ^ updated | 19:07 |
pabelanger | removing WIP | 19:07 |
pabelanger | zuul-maint: if you have spare review cycles^ :) | 19:08 |
paladox | clarkb fungi it should use All-Replications instead of a db :P | 19:10 |
Shrews | pabelanger: did you just change the version in the config in that last PS? it lgtm otherwise | 19:10 |
fungi | paladox: heh, just don't replicate the All-Replications repo then ;) | 19:11 |
paladox | yup! | 19:11 |
pabelanger | Shrews: yah, just version bump in ansible-config.conf | 19:17 |
openstackgerrit | Merged zuul/zuul master: Make image build jobs voting again https://review.opendev.org/659242 | 19:18 |
mordred | pabelanger: do we get a cookie for supporting 2.8 on release day? | 19:21 |
mordred | pabelanger: you have 3x+2 - land at will | 19:22 |
pabelanger | mordred: I hope so | 19:31 |
pabelanger | it will be such a yummy coffee | 19:31 |
pabelanger | +3'd | 19:32 |
pabelanger | yay | 19:32 |
openstackgerrit | Merged zuul/zuul master: zuul-quick-start: run docker-compose as regular user https://review.opendev.org/658738 | 19:32 |
pabelanger | zuul question, by default for all repos we set the remote to file:///dev/null. In check / gate I can understand that. However for doing CD with zuul, onto a static node, I'm wondering if that is needed? Given we aren't dealing with speculative code: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L915 | 19:38 |
SpamapS | pabelanger:is there a problem statement in there? | 19:39 |
pabelanger | :) | 19:39 |
pabelanger | so, problem is, I have bastion host I use to CD zuul deployment from, the repos are pushed to bastion, with remote set to file:///dev/null. Sometime, between zuul periodic job, I want to pull down latest branch tip, but remote is null | 19:40 |
mordred | yeah - I think you shouldn't do that | 19:40 |
mordred | :) | 19:40 |
SpamapS | eh.. yeah | 19:40 |
SpamapS | what mordred said | 19:40 |
SpamapS | The whole point is that you have what zuul thinks you have. | 19:40 |
mordred | (seriously - if you want to run something manually in-between CD runs, I'd do it from a different directory | 19:41 |
pabelanger | right, that is what I do today | 19:41 |
mordred | bcause you also wouldn't know if zuul is going to shell in and bork your pulled state | 19:41 |
pabelanger | different dir | 19:41 |
mordred | in the middle of a run | 19:41 |
pabelanger | also true | 19:41 |
SpamapS | Also, this is where I speculate if we should make manual triggering easier. | 19:41 |
SpamapS | So that we can just use the one path, every time. | 19:41 |
mordred | SpamapS: ++ | 19:41 |
mordred | I think that's coming along with the webtrigger work | 19:42 |
SpamapS | Right now, I trigger manual runs by opening and merging an empty merge commit. | 19:42 |
SpamapS | Which actually is kinda great.. because my git repo has paperwork in it. | 19:42 |
pabelanger | Yah, it isn't a big issue, and usually can wait for zuul to do the update (is 1 hour periodic). But there are some cases, I'd like to run it sooner. | 19:43 |
pabelanger | empty merge commit might work too | 19:43 |
SpamapS | The only thing about that is that I had to drop all my files matchers. | 19:45 |
SpamapS | It would be *GREAT* to have a header in commit message that would basically say "ignore files matchers" or something. | 19:45 |
clarkb | I'm running into that right now | 19:45 |
clarkb | had to put a comment .zuul.yaml to run all the jobs | 19:45 |
SpamapS | So now I build and push a docker image for every app we have on anything except docs updates. | 19:45 |
clarkb | and I'll push a new patchset up without that that won't run all the jobs but at least we'll hae the history in zuul saying it works | 19:46 |
SpamapS | Should be fairly simple to add similarly to Depends-On. | 19:46 |
SpamapS | Ignore-matchers: files | 19:46 |
SpamapS | Another thing that I kinda need is matcher templates. | 19:47 |
SpamapS | I have been using YAML anchors but that only works in a single file. | 19:47 |
SpamapS | Like, I want to say "files-templates: [ansible, docker, app1]" | 19:48 |
SpamapS | #MonoRepoProblems | 19:49 |
pabelanger | so, it looks like zuul.o.o dashboard is broken now | 19:49 |
pabelanger | I think it is react 2.0 stuff? | 19:49 |
pabelanger | http://paste.openstack.org/show/751488/ | 19:49 |
pabelanger | is that I am seeing | 19:49 |
pabelanger | https://review.opendev.org/631902/ is now live I guess | 19:50 |
pabelanger | looking to see what happened | 19:50 |
Shrews | it would appear the dashboard job for the react change shows similar: http://logs.openstack.org/02/631902/10/gate/zuul-build-dashboard/0b27362/npm/html/status | 19:52 |
*** pcaruana has quit IRC | 19:52 | |
Shrews | revert? | 19:52 |
mordred | Shrews: ++ | 19:54 |
pabelanger | Shrews: ++ | 19:55 |
openstackgerrit | David Shrewsbury proposed zuul/zuul master: Revert "web: upgrade react and react-scripts to ^2.0.0" https://review.opendev.org/659655 | 19:55 |
Shrews | ^^ | 19:55 |
pabelanger | +2 | 19:55 |
*** markwork has quit IRC | 19:59 | |
fungi | pabelanger: not sure if anyone pointed it out yet, but the reason we fabricate a remote when pushing onto the node is mostly to placate tools like pip which try to be smart when run on git repositories and then get confused by a repo with no remote | 20:08 |
fungi | ...even though that's a totally legitimate thing for a git repository | 20:08 |
daniel2 | Does zuul have any ubuntu packages? | 20:18 |
openstackgerrit | Merged zuul/zuul master: Add logs spec https://review.opendev.org/648714 | 20:18 |
mordred | daniel2: no, at least not anything official. most of the deployments I'm aware of are either installing from source via pip, or are using containers (which themselves are installing from source via pip in the image build) | 20:21 |
daniel2 | So disk image builder is still the same, right? Using elements. Where does ansible come into play, is that for actually running the jobs on the nodes? | 20:23 |
fungi | "still the same" compared to what? | 20:29 |
daniel2 | fungi: Sorry, so I'm upgrading from 0.5.0 to the latest nodepool. | 20:30 |
fungi | but dib has basically followed a consistent element model since its inception, so the answer is yes regardless | 20:30 |
fungi | got it, context helps | 20:30 |
daniel2 | 3 months down the drain with this project cause my coworker was too stubborn to let me upgrade, and we kept hitting major problems using the old version. Finally I was told to wipe out the old environment and do it with the newest versions. | 20:30 |
fungi | yes dib is the same as ever, and nodepool can use dib to create images to boot | 20:31 |
fungi | and ansible is basically just how zuul executes job payloads | 20:31 |
daniel2 | Yeah, the major problems we ran into were with jenkins. | 20:31 |
daniel2 | So ansible should be a lot better. | 20:31 |
fungi | so did we, which is why we stopped using it in favor of ansible ;) | 20:32 |
fungi | i'm quite sure jenkins is really great for a lot of things | 20:33 |
fungi | but remotely executing arbitrary scripts in an untrusted environment isn't one of them | 20:33 |
daniel2 | I use jenkins to automate stuff. Like I run FreeBSD at home and I setup my own package mirror using poudriere, I use jenkins to update and rebuild the packages nightly. I also manage my own ports collection with github so I use jenkins to keep that up to date. | 20:34 |
nickx-intel | "ports collection" as in, imported packages? | 20:38 |
clarkb | anyone have logs for dashboard builds I can use while we wait on the revert? | 20:39 |
fungi | nickx-intel: "ports" is basically how the *bsds (and gentoo) refer to out-of-tree software | 20:40 |
daniel2 | nickx-intel: FreeBSD offers building from source (ports) or using their official package mirror which is prebuild packages. Poudriere is their package building software, I use that because building from ports lets you change config options, add features, etc. So I get the best of both worlds. My server builds the ports into packages and I can distrubute that to all my computers at home. | 20:40 |
fungi | in bsdland it's not uncommon to have your kernel and basic userland all in a single source tree. ports is how you get other software outside the base system, either via automated builds from random external sources or prebuilt packages of the same redistributed | 20:41 |
openstackgerrit | Merged zuul/zuul master: web: add tags to jobs list https://review.opendev.org/633654 | 20:41 |
clarkb | also don't ever change root's shell to a dynamically linked shell like bash then try to rebuild the world | 20:42 |
clarkb | it is csh for a reason | 20:42 |
daniel2 | clarkb: I use zsh with static binaries. | 20:42 |
fungi | and "ports" basically started as a collection of patches and build recipes for other non-base software | 20:42 |
daniel2 | bash actually supprts that as well. | 20:42 |
fungi | software "ported" to (whatever)bsd | 20:42 |
clarkb | daniel2: ya the big issue wasn't bash vs csh it was static vs dynamically linked | 20:42 |
clarkb | and csh is what freebsd gave you statically linked by default at least once upon a time | 20:43 |
daniel2 | clarkb: it hasn't changed. | 20:43 |
daniel2 | Ports are just Makefiles that tell FreeBSD how to build the package from source, patches can be added to files/ but thats not always the case. | 20:44 |
daniel2 | I maintain about 50+ ports right now. | 20:44 |
pabelanger | I'd always switch to tcsh for some reason | 20:44 |
fungi | i rather like openbsd's ksh | 20:45 |
daniel2 | I used openbsd for a router for a month or so, ended up switching that to pfsense. | 20:45 |
daniel2 | I just got gigabit fiber in my home, had to buy a new box to host pfsense on as my old one's NICs couldn't handle the load. | 20:45 |
fungi | i've used openbsd for going on 20 years now. ran commercial products based on it for nearly half that time | 20:45 |
daniel2 | fungi: OpenBSD is great, lots of features and drivers that FreeBSD gets are brought over from OpenBSD | 20:46 |
daniel2 | PF was essentially from OpenBSD actually. | 20:46 |
fungi | if you want a combined firewall, layer-7 load balancer, web accelerator proxy and vpn concentrator, all redundantly clustered, it's hard to beat | 20:47 |
fungi | yeah, the history behind pf is a little fun. openbsd used to use a port of ipf from netbsd | 20:47 |
daniel2 | I never liked FreeBSD's old firewall. ipfw I think? | 20:48 |
fungi | due to some disagreements, the openbsd devs were caught having to rip ipfilter out and replace it in the span of a few months | 20:50 |
daniel2 | Did you know OpenBSD releases a song with every major release? | 20:50 |
fungi | that i do ;) | 20:50 |
fungi | it's an amusing bit of culture for the project | 20:51 |
daniel2 | My coworkers keep saying "You still use FreeBSD?". I more than use it. | 20:51 |
fungi | i ran commercial offerings on freebsd too. the provider i was working at replaced a bunch of symlink-farm mass hosting with freebsd jails back when it was still fresh and virtualization was really not something anyone thought would make a stable solution for separation of workloads | 20:54 |
fungi | though we did later migrate a lot of that to usermode linux once that became a viable option | 20:54 |
daniel2 | Virtualization on BSD is still meh. | 20:55 |
daniel2 | Bhyve is nice, but I feel still has a ways to go. | 20:55 |
openstackgerrit | Merged zuul/zuul master: Ensure correct state in MQTT connection https://review.opendev.org/652932 | 20:57 |
fungi | it's funny that the linux ecosystem eventually ended up going to virtualization and only recently came back around to the sorts of concepts that freebsd jails and solaris zones implemented 15+ years ago | 20:57 |
*** fdegir has quit IRC | 21:10 | |
*** fdegir has joined #zuul | 21:11 | |
openstackgerrit | Merged zuul/zuul master: Fix dequeue ref not handling the project name https://review.opendev.org/659110 | 21:19 |
SpamapS | I love OpenBSD so much... for firewalls. ;) | 21:23 |
fungi | its upgrade story is still a little rough around the edges | 21:23 |
fungi | like many projects i know | 21:23 |
SpamapS | Every OpenBSD I ever deployed was 1 of at least 2, usually 3. | 21:25 |
SpamapS | So, upgrade story was, wipe, install, copy config, repeat. | 21:26 |
SpamapS | And usually working via pf+carp | 21:26 |
SpamapS | Always loved having an air gap there.. openbsd bridged stateful firewall that doesn't even have an IP on any network reachable from anywhere except the box we use to update it. | 21:27 |
fungi | yeah, carp'd clusters and some sort of configuration management did make that way simpler | 21:29 |
fungi | the recommended in-place patching process was to cvsup and then make and make install and reboot or restart things | 21:30 |
fungi | keeping a full bsd source tree around on a firewall... silly | 21:31 |
openstackgerrit | Merged zuul/zuul master: Add support for submitting reviews on GitHub https://review.opendev.org/656541 | 21:32 |
daniel2 | I realized I'm super lazy when I spent as much time trying to find a nginx zuul sample config that it would had taken if I wrote it myself. | 21:33 |
openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Install latest git-review from PyPI in quickstart https://review.opendev.org/659674 | 21:45 |
fungi | paladox: ^ thanks for spotting that! | 21:45 |
fungi | (and for jogging my terrible memory) | 21:45 |
paladox | your welcome :) | 21:45 |
openstackgerrit | Merged zuul/zuul master: Support fail-fast in project pipelines https://review.opendev.org/652764 | 21:51 |
clarkb | the react revert failed on py36 in the gate I think | 21:55 |
clarkb | (I'm reading through czuul and its not the easiest to understand) | 21:55 |
paladox | fungi drafts was removed in gerrit 2.15 :) | 21:58 |
paladox | just that ref was deprecated. | 21:58 |
fungi | paladox: ahh, close enough | 22:00 |
paladox | yup. | 22:00 |
fungi | i'll update the commit message on my next iteration once i get some job results | 22:01 |
paladox | Drafts is split up into two features WIP and Private. | 22:01 |
openstackgerrit | Merged zuul/zuul master: Support Ansible 2.8 https://review.opendev.org/631933 | 22:04 |
paladox | though we disabled the "private" change feature at wikimedia to prevent users creating changes in private. | 22:04 |
fungi | yeah, we did the same (long, long ago via crippling it with acls) | 22:05 |
fungi | drafts was an attractive nuisance | 22:05 |
fungi | good riddance | 22:05 |
paladox | oh, yes! | 22:05 |
paladox | we allowed drafts, but once we upgraded, we disabled "private changes" immediately. | 22:07 |
paladox | WIP's are awesome though! | 22:08 |
fungi | oh, i see, you mean the drafts feature got split into two new features in 2.15, one of which was private changes | 22:09 |
paladox | yup | 22:09 |
fungi | the work in progress feature holds a special place in Shrews's heart | 22:09 |
fungi | vindication | 22:09 |
paladox | also you can now set changes in or out of WIP mode too | 22:09 |
paladox | so no longer a one time thing at creation. | 22:10 |
SpamapS | daniel2:did you find said config? | 22:47 |
SpamapS | We should have it in the docs. | 22:47 |
SpamapS | If we don't. | 22:47 |
fungi | hrm, zuul unit tests failing with: error An unexpected error occurred: "https://registry.yarnpkg.com/@xtuc/long/-/long-4.2.1.tgz: Request failed \"503 Service Unavailable\"". | 22:48 |
fungi | i seem to be able to fetch that, so maybe it was just a brief outage | 22:49 |
SpamapS | temp fail in yarnpkg.com is common | 22:49 |
SpamapS | :-P | 22:49 |
SpamapS | mirror time | 22:49 |
fungi | whee | 22:49 |
openstackgerrit | Jeremy Stanley proposed zuul/zuul master: Install latest git-review from PyPI in quickstart https://review.opendev.org/659674 | 22:54 |
openstackgerrit | Merged zuul/zuul master: Revert "web: upgrade react and react-scripts to ^2.0.0" https://review.opendev.org/659655 | 23:04 |
clarkb | yay | 23:05 |
clarkb | mirroring js things like npm or yarn isn't really practical (they've grown too large like pypi) | 23:06 |
clarkb | can probably do a caching proxy though to cut down on failures | 23:06 |
SpamapS | clarkb:yeah, as long as you configure with "serve stale if you can't refresh" | 23:07 |
* fungi gets the feeling SpamapS has dealt with a proxy or two before ;) | 23:08 | |
clarkb | for some things that would be safe, I'm not surei f it would be safe for the above resource though | 23:16 |
clarkb | (not sure about yarn's policy and replacing content) | 23:16 |
clarkb | Though Iguess usually you'd only serve stale on 500 not 400 so removal of content would still be an error that propagates thrgouh | 23:18 |
*** kepok__ has joined #zuul | 23:23 | |
*** rlandy has quit IRC | 23:34 | |
SpamapS | Right | 23:39 |
SpamapS | fungi: 1 or 2, yeah. :) | 23:39 |
*** kepok__ has quit IRC | 23:43 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!