Thursday, 2019-05-16

*** rlandy|bbl is now known as rlandy02:21
*** bhavikdbavishi has joined #zuul03:34
SpamapSIs there anything a casual developer can do to help with the test failures?04:28
tobiashSpamapS: the fix merged recently: https://review.opendev.org/65929605:16
tobiashso once puppet went through (is it already?) we can recheck, but we should start with https://review.opendev.org/659329 and https://review.opendev.org/659262 which are high priority fixes05:17
tobiashSpamapS: btw, both could use a second review ;)05:18
tobiashor any other zuul-maint ^05:18
tobiashoh wait, I can approve one of them :)05:18
tobiashso only 659329 needs a second review05:19
*** saneax has quit IRC05:23
*** saneax has joined #zuul05:29
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791405:40
tristanCtobiash: shouldn't those "pattern" in line be replaced by line == "pattern" as tried in https://review.opendev.org/#/c/657914/11/zuul/ansible/base/callback/zuul_stream.py@178 ?05:40
tristanCtobiash: i wonder if the zuul-tox-stream is not failing because the task output get mangled with those pattern and incorectly skipped05:42
tobiashtristanC: I don't know anymore, I thought there was a reason about that but I could be wrong05:42
tobiashI think the 'in' is used as a safety mechanism that we detect the exit code reliably because that is the stop sign to stop streaming05:43
tobiashtristanC: if we want to be more sophisticated we need to split the parts we look for from the string (if it's intermangled) and log them before05:45
*** gtema has joined #zuul05:45
tobiashtristanC: so under the assumption that the line could be intermangled we currently might drop the last part of the log05:45
tobiashtristanC: but if so we might also want to analyse why it's intermangled05:46
tristanCtobiash: that's why i've been trying to do in PS10 of 657914 but without luck05:47
tobiashso you think it gets mangled?05:48
tristanCtobiash: well i don't know, but if it is, then the task output may get discard and the test is failing because it can't find in the final log file05:48
tristanCtobiash: though i'm not sure what's going on, i tried sending any traceback from zuul_console back to zuul_stream in https://review.opendev.org/#/c/657914/10/zuul/ansible/base/library/zuul_console.py@19305:49
tristanCtobiash: but for some reason, i only get "Log not found", not the extra "LOG NOT FOUND" added, e.g. in http://logs.openstack.org/14/657914/10/check/zuul-tox-remote/fbacac1/testr_results.html.gz05:50
tobiashthat's really weird05:51
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791406:01
*** quiquell has quit IRC06:04
*** quiquell has joined #zuul06:04
*** gtema has quit IRC06:05
*** quiquell has quit IRC06:05
*** quiquell has joined #zuul06:06
AJaegeris this a new error http://logs.openstack.org/38/658738/6/check/zuul-quick-start/effaca0/job-output.txt.gz#_2019-05-16_05_30_18_28815906:10
AJaeger"remote: You need 'Create' rights to create new references."06:10
*** bjackman_ has joined #zuul06:12
tobiashAJaeger: so far I only saw zuul-quick-start failures where also the post_failures of zuul-build-image was present06:22
AJaegertobiash: yeah06:23
tobiashso let's see if this gets fixed too when the first zuul-build-image jobs are working (I already see some in the pipeline)06:23
AJaegertobiash: good ;)06:26
tobiashoh maybe I was wrong, the post failure happens after zuul-build-image gets resumed06:28
tristanCdo we know if the skopeo package got reverted on the executor nodes?06:29
AJaegertristanC: we expected that the usual puppet/ansible runs would do so - perhaps it didn't happen?06:31
AJaegerianw: could you check, please? ^06:31
tristanCgrep "subuid mapping" $(which skopeo)  should not match when an unaffected version of skopeo is installed06:34
*** gtema has joined #zuul06:40
tristanCtobiash: it seems like zuul-quick-start is now running with "Gerrit Code Review 3.0.0 ready" :)06:40
tobiashso another update that breaks us?06:43
AJaegershould we cap?06:44
tristanCwell, gerrit seems to need a new ACL to create project06:44
openstackgerritTristan Cacqueray proposed zuul/zuul master: quick-start: pin docker image to 2.16.8  https://review.opendev.org/65945406:46
tobiash+206:58
tobiashany other zuul-maint awake to +3 ^?06:58
*** openstackgerrit has quit IRC07:03
*** pcaruana has joined #zuul07:13
AJaegertobiash: I suggest you do a single core approval on that one to move forward. Alternative: ianw might still be around, corvus might be awake by now but not sure whether he has time to review...07:15
tobiashAJaeger: yes, that was my plan as soon as the test was successful07:16
AJaeger;)07:16
*** hashar has joined #zuul07:17
tobiashdone07:17
AJaegerzuul-quick-start succeeded, still post_failure for zuul-build-image ;(07:25
tobiashwhat a wonderful day07:25
AJaegerhttp://logs.openstack.org/54/659454/1/check/zuul-build-image/9b933f4/job-output.txt.gz#_2019-05-16_07_23_38_79264207:25
AJaegersubuid mapping?07:25
tobiashAJaeger: that makes me think that skopeo is probably not (yet) downgraded on the executors07:26
tobiashI guess we need an infra-root to check that07:26
AJaegerok07:27
ianwhrm, i saw it running i thought07:27
*** openstackgerrit has joined #zuul07:27
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] zuul_stream: add debug to investigate tox-remote failures  https://review.opendev.org/65791407:27
ianwii  skopeo                                0.1.36-1~dev~ubuntu16.04.2~ppa1707:27
tobiashbut to keep us going we could squash 659218 and 659454 together07:28
ianwhrm, hang on 17 there, not 1407:28
tobiashthat should fix the quickstart and make the image build non-voting temporarily07:29
ianwargh, hrm puppet has failed07:29
tobiashAJaeger: want to do the squash? I'm busy with own operational topics atm, but can review07:30
AJaegercan do...07:31
tobiashthanks07:31
ianwMay 16 06:38:38 ze09 puppet-user[13943]: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold --force-yes install skopeo=0.1.36-1~dev~ubuntu16.04.2~ppa14' returned 100: Reading package lists...07:31
ianwMay 16 06:38:38 ze09 puppet-user[13943]: Building dependency tree...07:31
ianwMay 16 06:38:38 ze09 puppet-user[13943]: Reading state information...07:31
ianwMay 16 06:38:38 ze09 puppet-user[13943]: E: Version '0.1.36-1~dev~ubuntu16.04.2~ppa14' for 'skopeo' was not found07:31
ianwit's kicked out of the repo?07:32
*** jpena|off is now known as jpena07:32
AJaegerArgh ;(07:32
ianwooohh, i think it keeps two .. and it seem another has built in the last hour or so07:33
openstackgerritAndreas Jaeger proposed zuul/zuul master: bubblewrap: bind mount /etc/subuid  https://review.opendev.org/65921807:34
AJaegertobiash, ianw ^07:35
openstackgerritAndreas Jaeger proposed zuul/zuul master: Make image build jobs voting again  https://review.opendev.org/65924207:35
tobiashAJaeger: single core approved07:35
AJaegerthanks07:36
ianwso basically that skopeo pin can't work, because the ppa doesn't keep old packages07:36
tobiashhopefully this passes so we have more time to figure out the skopeo thing07:36
*** saneax has quit IRC07:41
ianwi proposed a revert on the ~14 install ... i don't know where we could get that now.  it's not in the apt archives07:41
ianwlike on the host apt archives07:41
ianwit's not in the ppa07:41
AJaegerwhat fun ;( Thanks, ianw07:42
*** bhavikdbavishi has quit IRC07:43
*** themroc has joined #zuul07:44
*** saneax has joined #zuul07:51
*** themr0c has joined #zuul07:52
*** themroc has quit IRC07:54
ianwi guess we could build our own in the opentack-ci-core ppa along with all the other things we have there.  that's probably the next step but unforutnately i won't have time tonight to go down that path07:57
ianwif someone wants to get the control file etc together (even build in their own ppa) we can easily migrate it in07:57
tristanCtobiash: i don't understand where the "Log not found" in http://logs.openstack.org/14/657914/13/check/zuul-tox-remote/ee0855f/testr_results.html.gz is coming from. This should have been disabled by https://review.opendev.org/#/c/657914/13/zuul/ansible/base/library/zuul_console.py ...08:05
tristanCcould it be that the zuul.opendev.org executor library somehow be used by the job?08:07
tobiashI don't think so, the test is executed on the node, not on the executor08:18
*** panda is now known as panda|rover09:17
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883510:11
*** pcaruana has quit IRC10:27
openstackgerritTristan Cacqueray proposed zuul/zuul master: [WIP] runner: enable reuse of a job-dir  https://review.opendev.org/65795510:27
*** sshnaidm|afk is now known as sshnaidm10:38
*** bhavikdbavishi has joined #zuul11:00
*** jpena is now known as jpena|lunch11:03
*** panda|rover is now known as panda|rover|eat11:04
*** gtema has quit IRC11:07
*** pcaruana has joined #zuul11:37
*** panda|rover|eat is now known as panda12:06
*** panda is now known as panda|rover12:11
*** saneax has quit IRC12:17
*** themr0c has quit IRC12:24
*** themr0c has joined #zuul12:25
*** jpena|lunch is now known as jpena12:29
*** rlandy has joined #zuul12:34
openstackgerritMerged zuul/zuul master: bubblewrap: bind mount /etc/subuid  https://review.opendev.org/65921812:43
tobiash\o/12:45
AJaegertobiash, finally \o/12:45
AJaeger;)12:45
tobiashzuul-maint: someone here to review https://review.opendev.org/659329 ?12:46
*** saneax has joined #zuul12:47
*** snapiri has quit IRC12:56
*** bjackman_ has quit IRC13:03
Shrewstobiash: lgtm13:05
tobiash\o/13:05
*** markwork has joined #zuul13:06
markworkHey, when I build the containers from my patchset I get an empty zuul-web page, with cherry py complaining that path '/' was not found :(13:07
*** gtema has joined #zuul13:16
openstackgerritMark Meyer proposed zuul/zuul master: Create a basic Bitbucket event source  https://review.opendev.org/65883513:24
*** ianychoi has joined #zuul13:26
clarkbmarkwork: if not using co tainers you may have to jump through hoops to generate the js but I though the containers did all that for you. This was a recent build?13:34
markworkYes, I just ran the container build on my machine and just inserted a 'pip install --upgrade .' to install my changeset. Just an empty page13:36
markworkBut the container build runs13:36
clarkbmordred: ^ any ideas?13:37
clarkbactually I wonder if the pip install upgrade clears the old js content13:38
clarkbI think if you dont modify the dockerfile and instead rebuild with your change checked out it will get included13:39
clarkbmaybe try that13:39
markworkYes, I think it's something like this. I assume I can just `docker build . ` from my checkout and it will take my current source? I think this may be stupidity on my side...13:39
*** gtema has quit IRC13:39
mordredmarkwork: yes - docker build . should Just Work™13:41
*** pcaruana has quit IRC13:56
markworkmordred, clarkb: that was it, now it works :D so almost ready for demo day.14:04
mordred\o/14:05
mordredmarkwork: I'm very excited about that and hope it goes well (also, thanks again for working on that bitbucket driver!)14:05
markworkmordred: you're welcome, it's an interesting codebase and a lot of fun :)14:08
markworkstill need to push some tests and then the here-and-theres14:09
markworkWe now have a zuul-test instance that works without Gerrit, just using Bitbucket and it has just built its first project :D14:09
mordred\o/14:10
*** gtema has joined #zuul14:30
markworkWhat can be the reason that `builds` and `buildsets` are in state `Loading...` in the web ui? They never display anything...14:32
clarkbyou need a sql reporter for those to work14:33
clarkbtherehas been talk of just making the sql reporter mandatory but currently it is optional as those tabs are the only things that dont function without it14:34
markworkclarkb: understood, very helpful, thanks :)14:35
*** pcaruana has joined #zuul14:37
openstackgerritPaul Belanger proposed zuul/zuul master: Support Ansible 2.8  https://review.opendev.org/63193314:42
openstackgerritMerged zuul/zuul master: Fix missing check if logger is None  https://review.opendev.org/65926214:44
markworkAnybody got an idea why I get this message: Project <ProjectConfig bitbucket.smaato.net/SYS-IC/animal source: SYS-IC/animal/.zuul.yaml@update_readme {ImpliedBranchMatcher:update_readme}> did not match item <QueueItem 0x7fefba0e2a58 for <Change 0x7fefb90c72e8 SYS-IC/animal None,05bcb585354268af13b8c44f050b1e53a5089547> in check>14:50
markworkApparently some branch matcher rejects the branch? What's the background?14:50
openstackgerritPaul Belanger proposed zuul/zuul master: executor: use node python path  https://review.opendev.org/63733914:51
openstackgerritPaul Belanger proposed zuul/zuul master: Update quickstart nodepool node to python3  https://review.opendev.org/65848614:51
pabelangertristanC: just a rebase to confirm working on quickstart, I'll try and add test coverage for my comment today14:51
pabelangerWouldn't mind landing that soon for python3 only images14:51
openstackgerritMerged zuul/zuul master: Handle artifacts on non change types.  https://review.opendev.org/65932914:57
tobiash\o/14:57
clarkbmarkwork: if your zuul config is in a repo with branches (more than just master) zuul assumes the config on a branch is branch specific14:57
clarkbmarkwork: we (opendev) use brnachless (master only) config repos to manage jobs that should apply globally14:58
*** bhavikdbavishi has quit IRC15:00
*** hashar has quit IRC15:05
clarkbcatching up, did we make the zuul build image/upload image jobs non voting to work around the skopeo problem?15:10
AJaegerclarkb: yes, we did - since several problems were showing up15:10
clarkbk thanks15:11
AJaegerclarkb: change was https://review.opendev.org/659218 , the change to make these voting again is https://review.opendev.org/65924215:15
openstackgerritMerged zuul/zuul master: Fix occasional gate resets with github  https://review.opendev.org/65038715:19
openstackgerritPaul Belanger proposed zuul/zuul master: Support Ansible 2.8  https://review.opendev.org/63193315:28
pabelangerShrews: okay, ^ should remove copypasta of zuul_stream and add conditional logic15:29
pabelangerthen, once 2.8.0 is released later today (hopefully) we can remove WIP15:29
Shrewspabelanger: is 'rescued' always present? If not, we could miss the 'skipped' portion, based on your conditional.15:35
openstackgerritMerged zuul/zuul master: Centralize logging adapters  https://review.opendev.org/65864415:38
*** pcaruana has quit IRC15:47
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/  https://review.opendev.org/60440415:49
openstackgerritFabien Boucher proposed zuul/zuul master: Pagure driver - https://pagure.io/pagure/  https://review.opendev.org/60440415:50
openstackgerritMerged zuul/zuul master: web: upgrade react and react-scripts to ^2.0.0  https://review.opendev.org/63190215:53
*** saneax has quit IRC16:01
pabelangerShrews: resuced is always present in stable-2.8+16:04
*** themr0c has quit IRC16:04
pabelangerand skipped, also only apply stable-2.8+16:05
pabelangerneed to relocate, afk16:05
markworkclarkb: Hmm, strangely this error has happened only for a branch that has an underscore in it16:06
markwork(I think)16:06
markworkAnd exactly that branch is not building16:06
openstackgerritMerged zuul/zuul master: Cleanup executor specific requirements  https://review.opendev.org/64942816:17
AJaegertobiash: zuul-build-image succeeded on https://review.opendev.org/580547 ;)16:19
tobiashyay16:20
AJaegerwant to +A 659242 ? Hope it passes ;)16:21
AJaegerah, you did!16:21
AJaegerrelease-zuul-python fails, see http://logs.openstack.org/ee/ee4b6b1a27d1de95a605e188ae9e36d7f1597ebb/post/release-zuul-python/f225995/job-output.txt.gz#_2019-05-16_16_10_21_00986916:23
AJaegeryarn install failed ;(16:23
AJaegerzuul-maint, any ideas? ^16:24
tobiashLooks like we have a run...16:24
AJaegeryeah ;(16:25
AJaegerand still many failures due to tox-remote ;(16:25
tobiashActually I'd expect py36 to fail as well if that's broken16:25
*** mattw4 has joined #zuul16:28
clarkbI've just restarted opendev zuul to pull in the artifacts handling fix16:33
clarkblooks like we also pulled ina  github fix16:33
clarkband the react update16:33
clarkb(an fyi mroe than anything else)16:33
fungiand as of a few minutes ago we've been running with tristanC's pr for https://github.com/containers/skopeo/issues/649 applied on all our executors for opendev16:33
clarkbtobiash: AJaeger we need to install newer nodejs for yarn looks like16:34
clarkbwe install it from nodesource so that is probably just a version number in the role somewhere that we bump16:34
tobiashbecause of 631902?16:34
clarkbya that could be if that updated webpack16:34
clarkbI think we should just use node 10 or whatever is latest stable and be happy with that16:35
AJaegerthat updated node_version - is that ignored? See https://review.opendev.org/#/c/631902/10/.zuul.yaml16:35
clarkbAJaeger: it updated it on zuul-build-dashboard but not release-zuul-python16:36
*** sshnaidm is now known as sshnaidm|afk16:37
AJaegerclarkb: I see - needs updating in project-config. I'll try...16:37
*** markwork has quit IRC16:37
clarkbok thanks16:37
AJaegerclarkb, tobiash, https://review.opendev.org/65962016:40
clarkb+216:40
tobiash+116:41
pabelangerAJaeger: 659620 is not formatted correctly16:47
clarkbpabelanger: AJaeger remove the - prefix?16:49
pabelangeryup16:49
clarkbedited16:51
pabelanger+316:52
*** markwork has joined #zuul17:06
AJaegerthanks, clarkb !17:06
*** jpena is now known as jpena|off17:11
*** panda|rover is now known as panda|rover|off17:13
openstackgerritMerged zuul/zuul master: web: honor allowed-labels setting in the REST API  https://review.opendev.org/65389517:14
AJaegertobiash: FYI, looks like quite a few of your changes are in merge conflict now and need a rebase...17:17
tobiashAJaeger: yes, I've noticed that, will do in a bit17:17
AJaegersure, no urgency...17:17
openstackgerritMerged zuul/zuul master: docs: fix a typo in executor zone documentation  https://review.opendev.org/65366317:20
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate github logs with the event id  https://review.opendev.org/65864517:20
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate gerrit event logs  https://review.opendev.org/65864617:21
openstackgerritTobias Henkel proposed zuul/zuul master: Attach event to queue item  https://review.opendev.org/65864717:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate some logs in the scheduler with event id  https://review.opendev.org/65864817:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate logs in the zuul driver with event ids  https://review.opendev.org/65864917:21
openstackgerritTobias Henkel proposed zuul/zuul master: Add event id to timer events  https://review.opendev.org/65865017:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate pipeline processing with event id  https://review.opendev.org/65865117:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate merger logs with event id  https://review.opendev.org/65865217:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id  https://review.opendev.org/65888817:21
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888917:21
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889517:21
tobiashwas just a minimal conflict...17:21
AJaeger;)17:22
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888917:24
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889517:24
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888917:25
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889517:25
*** electrofelix has quit IRC17:38
-openstackstatus- NOTICE: Gerrit is being restarted to add gitweb links back to Gerrit. Sorry for the noise.17:38
*** pcaruana has joined #zuul17:39
openstackgerritMerged zuul/zuul master: gerrit: Add some quoting in 'gerrit query' commands  https://review.opendev.org/64987917:41
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: further zuul-remote debugging attempts  https://review.opendev.org/65963117:46
tobiashhrm, again in merge conflict :/17:48
tobiashor just gerrit things it's in merge conflict17:49
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate github logs with the event id  https://review.opendev.org/65864517:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate gerrit event logs  https://review.opendev.org/65864617:51
openstackgerritTobias Henkel proposed zuul/zuul master: Attach event to queue item  https://review.opendev.org/65864717:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate some logs in the scheduler with event id  https://review.opendev.org/65864817:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate logs in the zuul driver with event ids  https://review.opendev.org/65864917:51
openstackgerritTobias Henkel proposed zuul/zuul master: Add event id to timer events  https://review.opendev.org/65865017:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate pipeline processing with event id  https://review.opendev.org/65865117:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate merger logs with event id  https://review.opendev.org/65865217:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate job freezing logs with event id  https://review.opendev.org/65888817:51
openstackgerritTobias Henkel proposed zuul/zuul master: Annotate node request processing with event id  https://review.opendev.org/65888917:51
openstackgerritTobias Henkel proposed zuul/zuul master: WIP: Annotate builds with event id  https://review.opendev.org/65889517:51
tobiashnope, really conflict17:51
*** gtema has quit IRC17:56
markworkThe shortname of a build in the ui, is that change.changeset? I set it to the commit Sha and get weird commit shas in the ui, can I change change.changeset to the branch?17:57
clarkbmarkwork: for gerrit it is change.number,change.patchset (which might be aliased as change.changeset) and for tags, refs etc its the first 7 characters of the sha1 for that object17:58
fungialso for some events it's a head name or a tag name17:59
fungier, wait, in which ui tab?18:00
fungiare you asking about the status tab?18:01
openstackgerritMerged zuul/zuul master: github: add event filter debug  https://review.opendev.org/58054718:02
markworkclarkb: thanks a lot, I'll fiddle with those18:02
fungiin our tag-triggered pipelines we see the tag name there, in commit-triggered pipelines it's an abbreviation of the commit id, in gerrit-change-triggered pipelines it's the gerrit change number and patch number comma-separated, in our periodic pipelines it's the branch name...18:04
tobiashmarkwork: the rendering is here: https://opendev.org/zuul/zuul/src/branch/master/web/src/containers/status/ChangePanel.jsx#L11718:04
fungioh, and for github-pr-triggered pipelines it's the pr number and an abbreviation of the commit id comma-separated18:05
tobiashmarkwork: bitbucket probably has pull request numbers like github?18:05
markworktobiash: yes it has, I'm just building and update18:12
markworks/and/an/18:12
tobiashmarkwork: yeah, so you might want to handle it similar to github18:13
*** ianychoi has quit IRC18:15
openstackgerritTobias Henkel proposed zuul/zuul master: Add foreground option  https://review.opendev.org/63564918:16
SpamapSso, do I need to rebase 656541 to pick up whatever fixes the quick start and tox-remote fails?18:26
SpamapSIt's on recheck #518:26
SpamapSAlso, can we get rid of "clean check" for zuul so we don't have to do this madness?18:27
SpamapS(since, I assume, check has to pass before it is tested merged on top of master)18:27
fungiwe need to move it into the zuul tenant in opendev, and then yes absolutely18:27
fungithough check is also tested merged on tip of master always18:28
fungidropping the clean-check model would just mean we're okay approving changes which have no or negative verify vote from zuul18:28
clarkbI don't know if tox-remote has been fixed18:29
clarkb(I've still seen it fail here and there)18:29
fungiquick-start should be fixed now, so simply rechecking should get that passing unless we still missed something18:30
tobiashtox-remote is not yet fixed and hard to reproduce an debug18:30
fungia big part of what was breaking the quick-start job was not actually due to zuul's codebase, but a regression in skopeo which we've worked around by installing a temporary fork18:30
fungiwhile waiting on the pr which solves it to merge18:31
clarkbSpamapS: and no rebase requires due to the out of band nature of the fix18:32
Shrewsi've run the tox-remote jobs locally 100's of times, can't get it to fail18:32
clarkbSpamapS: also zuul always rebases your change onto the target branch before testing it so you get that for free anyway18:32
tobiashShrews: me 218:33
openstackgerritTobias Henkel proposed zuul/zuul master: DNM: further zuul-remote debugging attempts  https://review.opendev.org/65963118:41
mordredSpamapS: other people have already said yes, but I want to echo the yes to removing clean-check from zuul18:42
mordredas soon as we move it to the zuul tenant, we'll be good to go18:42
tobiashis the replication to https://opendev.org/zuul/zuul lagging/broken after the gerrit restart?18:43
clarkbtobiash: lagging18:43
tobiashk18:43
clarkbtobiash: gerrit does a full replication after a restart and takes a while to catch up from that18:43
clarkbdown to 8.2k tasks now18:43
tobiashthanks, I just wondered18:43
mordredclarkb: it would be really awesome if it would not do a full replication after a restart18:45
clarkbmordred: ya updating the replication plugin to not do that by default is on my todo list18:46
clarkb(you can always trigger full replication manually if desired)18:46
mordredyeah18:46
openstackgerritMerged zuul/zuul master: Don't check out a commit if it is already checked out  https://review.opendev.org/16392218:47
AJaegerwow, look at that number ^18:48
mordredAJaeger: that's amazing18:48
AJaegerMarch 2015 - so, 4 years later...18:49
fungiclarkb: off topic for this channel, but what if the gerrit replication plug-in stored a checksum of some sort (summing over the list of git object ids, or whatever) and checked that at start to decide whether it should rereplicate a given repository?18:50
fungithough i guess that supposes that there weren't any queued replication events which hadn't completed when the service was stopped18:51
clarkbya it could keep track of whether or not updates have been replicated fully18:54
clarkbthat seems like a lot more work (and needing a database)18:54
fungiwell... it *has* a database... but point taken18:54
openstackgerritMerged zuul/zuul master: Use a more visible selection color  https://review.opendev.org/64886518:59
mordredfungi: it's *really* trying to delete its database19:02
funginah, just moving it to a different medium19:03
fungiit's building databases of information in git repositories19:03
clarkbcan you safely store data about what has been replicated in the git repo itself :P19:04
clarkb(you probably can)19:04
fungias long as you don't replicate that data. otherwise you get to argue with gödel about completeness19:05
openstackgerritPaul Belanger proposed zuul/zuul master: Support Ansible 2.8  https://review.opendev.org/63193319:07
pabelangeransible 2.8.0 is out, ^ updated19:07
pabelangerremoving WIP19:07
pabelangerzuul-maint: if you have spare review cycles^ :)19:08
paladoxclarkb fungi it should use All-Replications instead of a db :P19:10
Shrewspabelanger: did you just change the version in the config in that last PS? it lgtm otherwise19:10
fungipaladox: heh, just don't replicate the All-Replications repo then ;)19:11
paladoxyup!19:11
pabelangerShrews: yah, just version bump in ansible-config.conf19:17
openstackgerritMerged zuul/zuul master: Make image build jobs voting again  https://review.opendev.org/65924219:18
mordredpabelanger: do we get a cookie for supporting 2.8 on release day?19:21
mordredpabelanger: you have 3x+2 - land at will19:22
pabelangermordred: I hope so19:31
pabelangerit will be such a yummy coffee19:31
pabelanger+3'd19:32
pabelangeryay19:32
openstackgerritMerged zuul/zuul master: zuul-quick-start: run docker-compose as regular user  https://review.opendev.org/65873819:32
pabelangerzuul question, by default for all repos we set the remote to file:///dev/null. In check / gate I can understand that. However for doing CD with zuul, onto a static node, I'm wondering if that is needed?  Given we aren't dealing with speculative code: https://opendev.org/zuul/zuul/src/branch/master/zuul/executor/server.py#L91519:38
SpamapSpabelanger:is there a problem statement in there?19:39
pabelanger:)19:39
pabelangerso, problem is, I have bastion host I use to CD zuul deployment from, the repos are pushed to bastion, with remote set to file:///dev/null. Sometime, between zuul periodic job, I want to pull down latest branch tip, but remote is null19:40
mordredyeah - I think you shouldn't do that19:40
mordred:)19:40
SpamapSeh.. yeah19:40
SpamapSwhat mordred said19:40
SpamapSThe whole point is that you have what zuul thinks you have.19:40
mordred(seriously - if you want to run something manually in-between CD runs, I'd do it from a different directory19:41
pabelangerright, that is what I do today19:41
mordredbcause you also wouldn't know if zuul is going to shell in and bork your pulled state19:41
pabelangerdifferent dir19:41
mordredin the middle of a run19:41
pabelangeralso true19:41
SpamapSAlso, this is where I speculate if we should make manual triggering easier.19:41
SpamapSSo that we can just use the one path, every time.19:41
mordredSpamapS: ++19:41
mordredI think that's coming along with the webtrigger work19:42
SpamapSRight now, I trigger manual runs by opening and merging an empty merge commit.19:42
SpamapSWhich actually is kinda great.. because my git repo has paperwork in it.19:42
pabelangerYah, it isn't a big issue, and usually can wait for zuul to do the update (is 1 hour periodic). But there are some cases, I'd like to run it sooner.19:43
pabelangerempty merge commit might work too19:43
SpamapSThe only thing about that is that I had to drop all my files matchers.19:45
SpamapSIt would be *GREAT* to have a header in commit message that would basically say "ignore files matchers" or something.19:45
clarkbI'm running into that right now19:45
clarkbhad to put a comment .zuul.yaml to run all the jobs19:45
SpamapSSo now I build and push a docker image for every app we have on anything except docs updates.19:45
clarkband I'll push a new patchset up without that that won't run all the jobs but at least we'll hae the history in zuul saying it works19:46
SpamapSShould be fairly simple to add similarly to Depends-On.19:46
SpamapSIgnore-matchers: files19:46
SpamapSAnother thing that I kinda need is matcher templates.19:47
SpamapSI have been using YAML anchors but that only works in a single file.19:47
SpamapSLike, I want to say "files-templates: [ansible, docker, app1]"19:48
SpamapS#MonoRepoProblems19:49
pabelangerso, it looks like zuul.o.o dashboard is broken now19:49
pabelangerI think it is react 2.0 stuff?19:49
pabelangerhttp://paste.openstack.org/show/751488/19:49
pabelangeris that I am seeing19:49
pabelangerhttps://review.opendev.org/631902/ is now live I guess19:50
pabelangerlooking to see what happened19:50
Shrewsit would appear the dashboard job for the react change shows similar: http://logs.openstack.org/02/631902/10/gate/zuul-build-dashboard/0b27362/npm/html/status19:52
*** pcaruana has quit IRC19:52
Shrewsrevert?19:52
mordredShrews: ++19:54
pabelangerShrews: ++19:55
openstackgerritDavid Shrewsbury proposed zuul/zuul master: Revert "web: upgrade react and react-scripts to ^2.0.0"  https://review.opendev.org/65965519:55
Shrews^^19:55
pabelanger+219:55
*** markwork has quit IRC19:59
fungipabelanger: not sure if anyone pointed it out yet, but the reason we fabricate a remote when pushing onto the node is mostly to placate tools like pip which try to be smart when run on git repositories and then get confused by a repo with no remote20:08
fungi...even though that's a totally legitimate thing for a git repository20:08
daniel2Does zuul have any ubuntu packages?20:18
openstackgerritMerged zuul/zuul master: Add logs spec  https://review.opendev.org/64871420:18
mordreddaniel2: no, at least not anything official. most of the deployments I'm aware of are either installing from source via pip, or are using containers (which themselves are installing from source via pip in the image build)20:21
daniel2So disk image builder is still the same, right?  Using elements.  Where does ansible come into play, is that for actually running the jobs on the nodes?20:23
fungi"still the same" compared to what?20:29
daniel2fungi: Sorry, so I'm upgrading from 0.5.0 to the latest nodepool.20:30
fungibut dib has basically followed a consistent element model since its inception, so the answer is yes regardless20:30
fungigot it, context helps20:30
daniel23 months down the drain with this project cause my coworker was too stubborn to let me upgrade, and we kept hitting major problems using the old version.  Finally I was told to wipe out the old environment and do it with the newest versions.20:30
fungiyes dib is the same as ever, and nodepool can use dib to create images to boot20:31
fungiand ansible is basically just how zuul executes job payloads20:31
daniel2Yeah, the major problems we ran into were with jenkins.20:31
daniel2So ansible should be a lot better.20:31
fungiso did we, which is why we stopped using it in favor of ansible ;)20:32
fungii'm quite sure jenkins is really great for a lot of things20:33
fungibut remotely executing arbitrary scripts in an untrusted environment isn't one of them20:33
daniel2I use jenkins to automate stuff.  Like I run FreeBSD at home and I setup my own package mirror using poudriere, I use jenkins to update and rebuild the packages nightly.  I also manage my own ports collection with github so I use jenkins to keep that up to date.20:34
nickx-intel"ports collection" as in, imported packages?20:38
clarkbanyone have logs for dashboard builds I can use while we wait on the revert?20:39
funginickx-intel: "ports" is basically how the *bsds (and gentoo) refer to out-of-tree software20:40
daniel2nickx-intel: FreeBSD offers building from source (ports) or using their official package mirror which is prebuild packages.  Poudriere is their package building software, I use that because building from ports lets you change config options, add features, etc.  So I get the best of both worlds.  My server builds the ports into packages and I can distrubute that to all my computers at home.20:40
fungiin bsdland it's not uncommon to have your kernel and basic userland all in a single source tree. ports is how you get other software outside the base system, either via automated builds from random external sources or prebuilt packages of the same redistributed20:41
openstackgerritMerged zuul/zuul master: web: add tags to jobs list  https://review.opendev.org/63365420:41
clarkbalso don't ever change root's shell to a dynamically linked shell like bash then try to rebuild the world20:42
clarkbit is csh for a reason20:42
daniel2clarkb: I use zsh with static binaries.20:42
fungiand "ports" basically started as a collection of patches and build recipes for other non-base software20:42
daniel2bash actually supprts that as well.20:42
fungisoftware "ported" to (whatever)bsd20:42
clarkbdaniel2: ya the big issue wasn't bash vs csh it was static vs dynamically linked20:42
clarkband csh is what freebsd gave you statically linked by default at least once upon a time20:43
daniel2clarkb: it hasn't changed.20:43
daniel2Ports are just Makefiles that tell FreeBSD how to build the package from source, patches can be added to files/ but thats not always the case.20:44
daniel2I maintain about 50+ ports right now.20:44
pabelangerI'd always switch to tcsh for some reason20:44
fungii rather like openbsd's ksh20:45
daniel2I used openbsd for a router for a month or so, ended up switching that to pfsense.20:45
daniel2I just got gigabit fiber in my home, had to buy a new box to host pfsense on as my old one's NICs couldn't handle the load.20:45
fungii've used openbsd for going on 20 years now. ran commercial products based on it for nearly half that time20:45
daniel2fungi: OpenBSD is great, lots of features and drivers that FreeBSD gets are brought over from OpenBSD20:46
daniel2PF was essentially from OpenBSD actually.20:46
fungiif you want a combined firewall, layer-7 load balancer, web accelerator proxy and vpn concentrator, all redundantly clustered, it's hard to beat20:47
fungiyeah, the history behind pf is a little fun. openbsd used to use a port of ipf from netbsd20:47
daniel2I never liked FreeBSD's old firewall.  ipfw I think?20:48
fungidue to some disagreements, the openbsd devs were caught having to rip ipfilter out and replace it in the span of a few months20:50
daniel2Did you know OpenBSD releases a song with every major release?20:50
fungithat i do ;)20:50
fungiit's an amusing bit of culture for the project20:51
daniel2My coworkers keep saying "You still use FreeBSD?".  I more than use it.20:51
fungii ran commercial offerings on freebsd too. the provider i was working at replaced a bunch of symlink-farm mass hosting with freebsd jails back when it was still fresh and virtualization was really not something anyone thought would make a stable solution for separation of workloads20:54
fungithough we did later migrate a lot of that to usermode linux once that became a viable option20:54
daniel2Virtualization on BSD is still meh.20:55
daniel2Bhyve is nice, but I feel still has a ways to go.20:55
openstackgerritMerged zuul/zuul master: Ensure correct state in MQTT connection  https://review.opendev.org/65293220:57
fungiit's funny that the linux ecosystem eventually ended up going to virtualization and only recently came back around to the sorts of concepts that freebsd jails and solaris zones implemented 15+ years ago20:57
*** fdegir has quit IRC21:10
*** fdegir has joined #zuul21:11
openstackgerritMerged zuul/zuul master: Fix dequeue ref not handling the project name  https://review.opendev.org/65911021:19
SpamapSI love OpenBSD so much... for firewalls. ;)21:23
fungiits upgrade story is still a little rough around the edges21:23
fungilike many projects i know21:23
SpamapSEvery OpenBSD I ever deployed was 1 of at least 2, usually 3.21:25
SpamapSSo, upgrade story was, wipe, install, copy config, repeat.21:26
SpamapSAnd usually working via pf+carp21:26
SpamapSAlways loved having an air gap there.. openbsd bridged stateful firewall that doesn't even have an IP on any network reachable from anywhere except the box we use to update it.21:27
fungiyeah, carp'd clusters and some sort of configuration management did make that way simpler21:29
fungithe recommended in-place patching process was to cvsup and then make and make install and reboot or restart things21:30
fungikeeping a full bsd source tree around on a firewall... silly21:31
openstackgerritMerged zuul/zuul master: Add support for submitting reviews on GitHub  https://review.opendev.org/65654121:32
daniel2I realized I'm super lazy when I spent as much time trying to find a nginx zuul sample config that it would had taken if I wrote it myself.21:33
openstackgerritJeremy Stanley proposed zuul/zuul master: Install latest git-review from PyPI in quickstart  https://review.opendev.org/65967421:45
fungipaladox: ^ thanks for spotting that!21:45
fungi(and for jogging my terrible memory)21:45
paladoxyour welcome :)21:45
openstackgerritMerged zuul/zuul master: Support fail-fast in project pipelines  https://review.opendev.org/65276421:51
clarkbthe react revert failed on py36 in the gate I think21:55
clarkb(I'm reading through czuul and its not the easiest to understand)21:55
paladoxfungi drafts was removed in gerrit 2.15 :)21:58
paladoxjust that ref was deprecated.21:58
fungipaladox: ahh, close enough22:00
paladoxyup.22:00
fungii'll update the commit message on my next iteration once i get some job results22:01
paladoxDrafts is split up into two features WIP and Private.22:01
openstackgerritMerged zuul/zuul master: Support Ansible 2.8  https://review.opendev.org/63193322:04
paladoxthough we disabled the "private" change feature at wikimedia to prevent users creating changes in private.22:04
fungiyeah, we did the same (long, long ago via crippling it with acls)22:05
fungidrafts was an attractive nuisance22:05
fungigood riddance22:05
paladoxoh, yes!22:05
paladoxwe allowed drafts, but once we upgraded, we disabled "private changes" immediately.22:07
paladoxWIP's are awesome though!22:08
fungioh, i see, you mean the drafts feature got split into two new features in 2.15, one of which was private changes22:09
paladoxyup22:09
fungithe work in progress feature holds a special place in Shrews's heart22:09
fungivindication22:09
paladoxalso you can now set changes in or out of WIP mode too22:09
paladoxso no longer a one time thing at creation.22:10
SpamapSdaniel2:did you find said config?22:47
SpamapSWe should have it in the docs.22:47
SpamapSIf we don't.22:47
fungihrm, zuul unit tests failing with: error An unexpected error occurred: "https://registry.yarnpkg.com/@xtuc/long/-/long-4.2.1.tgz: Request failed \"503 Service Unavailable\"".22:48
fungii seem to be able to fetch that, so maybe it was just a brief outage22:49
SpamapStemp fail in yarnpkg.com is common22:49
SpamapS:-P22:49
SpamapSmirror time22:49
fungiwhee22:49
openstackgerritJeremy Stanley proposed zuul/zuul master: Install latest git-review from PyPI in quickstart  https://review.opendev.org/65967422:54
openstackgerritMerged zuul/zuul master: Revert "web: upgrade react and react-scripts to ^2.0.0"  https://review.opendev.org/65965523:04
clarkbyay23:05
clarkbmirroring js things like npm or yarn isn't really practical (they've grown too large like pypi)23:06
clarkbcan probably do a caching proxy though to cut down on failures23:06
SpamapSclarkb:yeah, as long as you configure with "serve stale if you can't refresh"23:07
* fungi gets the feeling SpamapS has dealt with a proxy or two before ;)23:08
clarkbfor some things that would be safe, I'm not surei f it would be safe for the above resource though23:16
clarkb(not sure about yarn's policy and replacing content)23:16
clarkbThough  Iguess usually you'd only serve stale on 500 not 400 so removal of content would still be an error that propagates thrgouh23:18
*** kepok__ has joined #zuul23:23
*** rlandy has quit IRC23:34
SpamapSRight23:39
SpamapSfungi: 1 or 2, yeah. :)23:39
*** kepok__ has quit IRC23:43

Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!