| *** sgw has joined #zuul | 00:04 | |
| *** michael-beaver has quit IRC | 00:13 | |
| *** sgw has quit IRC | 00:20 | |
| *** threestrands has quit IRC | 00:49 | |
| *** tributarian has quit IRC | 01:12 | |
| *** igordc has quit IRC | 01:31 | |
| *** bhavikdbavishi has joined #zuul | 01:48 | |
| *** bhavikdbavishi1 has joined #zuul | 01:51 | |
| *** bhavikdbavishi has quit IRC | 01:52 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 01:52 | |
| *** bhavikdbavishi has quit IRC | 02:35 | |
| *** threestrands has joined #zuul | 03:02 | |
| *** rlandy|bbl has quit IRC | 03:10 | |
| *** bhavikdbavishi has joined #zuul | 03:29 | |
| *** jank has joined #zuul | 04:41 | |
| *** raukadah is now known as chandankumar | 04:59 | |
| *** sanjayu_ has joined #zuul | 05:16 | |
| *** jamesmcarthur has quit IRC | 05:26 | |
| ianw | i dunno if i've screwed something up trying to restart ze02, but it's just looping around showing | 05:31 |
|---|---|---|
| ianw | http://zuul.openstack.org/stream/ec45ac1d208343b58dee520053e8caee?logfile=console.loghttp://zuul.openstack.org/stream/ec45ac1d208343b58dee520053e8caee?logfile=console.log | 05:31 |
| ianw | 019-08-01 05:30:10,304 DEBUG zuul.ExecutorServer: [e: 598a301fc1b145d1b704b6098c7fdaca] Stop job with arguments: {'uuid': 'd09ce9f5b25c4878b531c0acd3d37274', 'zuul_event_id': '598a301fc1b145d1b704b6098c7fdaca'} | 05:31 |
| ianw | 2019-08-01 05:30:10,305 DEBUG zuul.ExecutorServer: [e: 598a301fc1b145d1b704b6098c7fdaca] Unable to find worker for job d09ce9f5b25c4878b531c0acd3d37274 | 05:31 |
| ianw | (not the url bits, sorry) | 05:31 |
| ianw | well, i cleared out /var/lib/zuul/builds, which included the d09ce9f5b25c4878b531c0acd3d37274 directory, and it's back alive i think | 05:35 |
| ianw | hrm, no ... | 05:38 |
| *** sanjayu_ has quit IRC | 05:41 | |
| *** sanjayu_ has joined #zuul | 05:41 | |
| ianw | ok, i think i've screwed something up; i've cleared everything, i've rebooted the host, and still no joy | 05:55 |
| *** sanjayu_ has quit IRC | 06:02 | |
| *** themroc has joined #zuul | 06:26 | |
| *** themroc has quit IRC | 06:41 | |
| *** themroc has joined #zuul | 06:42 | |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 06:56 |
| *** pcaruana has quit IRC | 07:12 | |
| *** fdegir has joined #zuul | 07:24 | |
| *** jpena|off is now known as jpena | 07:34 | |
| *** AJaeger has quit IRC | 07:37 | |
| *** pcaruana has joined #zuul | 07:42 | |
| *** AJaeger has joined #zuul | 07:48 | |
| *** tosky has joined #zuul | 07:54 | |
| *** electrofelix has joined #zuul | 07:58 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:28 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 08:31 |
| *** bhavikdbavishi has quit IRC | 09:56 | |
| *** themroc has quit IRC | 09:57 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: POC: Run linters via pre-commit https://review.opendev.org/667699 | 10:01 |
| *** sshnaidm|afk is now known as sshnaidm | 10:25 | |
| panda | hi, when I use a roles: - zuul: some_group/some_project configuration to include a role in a zuul job, where is some_group/some_project supposed to be defined so it can be used by zuul ? | 10:26 |
| panda | maybe in the tenant.source ? | 10:28 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 10:29 |
| panda | yep ok. | 10:31 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 10:33 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 10:55 |
| *** panda is now known as panda|eat | 11:25 | |
| *** bhavikdbavishi has joined #zuul | 11:35 | |
| *** bhavikdbavishi has quit IRC | 11:37 | |
| *** bhavikdbavishi has joined #zuul | 11:38 | |
| *** jamesmcarthur has joined #zuul | 11:51 | |
| *** jamesmcarthur has quit IRC | 12:06 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Optionally allow zoned executors to process unzoned jobs https://review.opendev.org/673840 | 12:19 |
| *** rlandy has joined #zuul | 12:31 | |
| *** panda|eat is now known as panda | 12:34 | |
| *** jpena is now known as jpena|off | 12:34 | |
| *** bhavikdbavishi1 has joined #zuul | 12:37 | |
| *** bhavikdbavishi has quit IRC | 12:37 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 12:37 | |
| *** jamesmcarthur has joined #zuul | 12:44 | |
| ofosos | I can't find it in the docs and I also can't remember: how can I trigger Zuul to re-read the tenant config? | 13:00 |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 13:07 |
| *** tosky has quit IRC | 13:10 | |
| *** tosky has joined #zuul | 13:10 | |
| *** jpena|off is now known as jpena | 13:16 | |
| *** jeliu_ has joined #zuul | 13:25 | |
| openstackgerrit | Jan Kubovy proposed zuul/zuul master: Evaluate CODEOWNERS settings during canMerge check https://review.opendev.org/644557 | 13:37 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Make tenant and pipeline optional in zuul-changes https://review.opendev.org/674034 | 13:38 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Move fingergw config to fingergw https://review.opendev.org/664949 | 13:45 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: WIP: Route streams to different zones via finger gateway https://review.opendev.org/664965 | 13:45 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Support ssl encrypted fingergw https://review.opendev.org/664950 | 13:45 |
| mordred | ofosos: https://zuul-ci.org/docs/zuul/admin/components.html#operation ... also, tobiash has a patch up to improve that: https://review.opendev.org/#/c/652114/ | 13:45 |
| mordred | ofosos: tl;dr - zuul-scheduler full-reconfigure | 13:45 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Add telnet to Docker Image https://review.opendev.org/672791 | 13:57 |
| ofosos | mordred: Thanks! | 14:00 |
| *** michael-beaver has joined #zuul | 14:02 | |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: Avoid openstacksdk image delete bug https://review.opendev.org/674043 | 14:14 |
| *** jhesketh has quit IRC | 14:23 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add spec for enhanced regional executor distribution https://review.opendev.org/663413 | 14:34 |
| *** jank has quit IRC | 14:51 | |
| *** dkehn_ has joined #zuul | 15:02 | |
| *** jpena is now known as jpena|off | 15:36 | |
| *** jank has joined #zuul | 15:37 | |
| corvus | clarkb: your eyes on https://review.opendev.org/673885 would be welcome | 15:38 |
| jeliu_ | tristanC, mordred: I was wondering what the correct namespace of the operator should be. In the specs it says "zuul-ci.org" and the current implementation has "operator.zuul-ci.org" | 15:41 |
| clarkb | corvus: change lgtm I didn't approve as I wasn't sure if you want to clean up that index_var | 15:43 |
| corvus | clarkb: meh | 15:43 |
| clarkb | ok I'll just approve it then | 15:43 |
| corvus | tosky: ^ we can fast-approve a fix cleaning that up if you want to push it | 15:44 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Create bindep_virtualenv_python for bindep role https://review.opendev.org/658439 | 15:46 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Add test-bindep job https://review.opendev.org/674078 | 15:46 |
| *** jamesmcarthur has quit IRC | 15:46 | |
| corvus | pabelanger: ^ rebased that on a change that adds a test job; if it looks good i'll +3 | 15:46 |
| tosky | corvus: oh, too fast - yes, it comes from the previous review; I can still push an update to it | 15:47 |
| corvus | tosky: either way :) | 15:48 |
| openstackgerrit | Luigi Toscano proposed zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 15:49 |
| corvus | +3 | 15:52 |
| *** tributarian has joined #zuul | 15:54 | |
| mordred | jeliu_: I think I'd vote for zuul-ci.org - what do you thnk corvus? | 15:58 |
| corvus | i'm not sure what the implications are | 15:58 |
| corvus | is the relevant question this: are we likely to have k8s objects other than the operator that might conflict? | 15:59 |
| AJaeger | fungi, want to rebase https://review.opendev.org/#/c/667697/ ? the change below it is not going to merge but those are independent... | 16:00 |
| *** hwangbo has joined #zuul | 16:00 | |
| *** sgw has joined #zuul | 16:00 | |
| corvus | mordred, jeliu_, tristanC: looks like the only object we have right now is Zuul; and even if we add a Nodepool object later, i'm not seeing a potential conflict, so i think i too am leaning toward zuul-ci.org | 16:01 |
| mordred | ++ | 16:01 |
| clarkb | mordred: corvus are there three distinct domains now: the operator itself, zuul, nodepool ? | 16:03 |
| *** michael-beaver has quit IRC | 16:04 | |
| *** jpena|off is now known as jpena | 16:05 | |
| corvus | clarkb: i think this question is in the context of the CRD namespace here: https://zuul-ci.org/docs/zuul/developer/specs/kubernetes-operator.html#custom-resource-definitions | 16:06 |
| corvus | clarkb: all of the CRD objects that the operator will create will be normal k8s stuff (pods, deployments, etc) | 16:06 |
| openstackgerrit | Merged zuul/zuul-jobs master: fetch-subunit-output: collect additional subunit files https://review.opendev.org/673885 | 16:07 |
| *** jpena is now known as jpena|off | 16:07 | |
| *** igordc has joined #zuul | 16:08 | |
| tristanC | jeliu_: mordred: corvus: "zuul-ci.org" lgtm | 16:14 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 16:19 |
| openstackgerrit | James E. Blair proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 16:19 |
| corvus | i figured out what was causing the test timeouts there; went ahead and fixed it | 16:20 |
| *** sshnaidm is now known as sshnaidm|afk | 16:46 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Don't compare to literal True/False https://review.opendev.org/667697 | 17:01 |
| AJaeger | fungi, rebased for you ^ | 17:02 |
| *** igordc has quit IRC | 17:03 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Be consistent about spaces before and after vars https://review.opendev.org/667698 | 17:04 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 17:04 |
| *** electrofelix has quit IRC | 17:06 | |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 17:07 |
| jeliu_ | corvus, mordred, tristanC: sorry Toronto office had a lunch meeting. "zuul-ci.org" sounds good! | 17:27 |
| *** igordc has joined #zuul | 17:27 | |
| *** igordc has quit IRC | 17:30 | |
| openstackgerrit | Merged zuul/zuul master: Try out reporting the build page https://review.opendev.org/673863 | 17:32 |
| openstackgerrit | Jeff Liu proposed zuul/zuul-operator master: Change operator namespace to zuul-ci.org https://review.opendev.org/674100 | 17:37 |
| *** igordc has joined #zuul | 17:42 | |
| openstackgerrit | Andreas Jaeger proposed zuul/zuul-jobs master: Revert "fetch-subunit-output: collect additional subunit files" https://review.opendev.org/674102 | 17:55 |
| *** tosky has quit IRC | 18:01 | |
| *** jank has quit IRC | 18:01 | |
| *** jeliu_ has quit IRC | 18:08 | |
| openstackgerrit | Merged zuul/zuul-jobs master: Revert "fetch-subunit-output: collect additional subunit files" https://review.opendev.org/674102 | 18:12 |
| *** jamesmcarthur has joined #zuul | 18:23 | |
| *** bhavikdbavishi has quit IRC | 19:02 | |
| *** bhavikdbavishi has joined #zuul | 19:04 | |
| *** michael-beaver has joined #zuul | 19:22 | |
| Shrews | Anyone care to push https://review.opendev.org/674043 through? I'd like to restart our builders with that | 19:26 |
| clarkb | Shrews: done | 19:27 |
| Shrews | thx | 19:28 |
| *** jeliu_ has joined #zuul | 19:33 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul Web: add /api/user/authorizations endpoint https://review.opendev.org/641099 | 19:39 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: authentication config: add optional token_expiry https://review.opendev.org/642408 | 19:39 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: [WIP] admin REST API: zuul-web integration https://review.opendev.org/643536 | 19:39 |
| *** bhavikdbavishi has quit IRC | 19:40 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Use a requests session to simplify auth'd calls https://review.opendev.org/670511 | 19:40 |
| corvus | jeliu_, mordred, tristanC: now that we have the k8s operator job testing that the operator pod is installed and running, should we (a) continue adding on to the job to install the zuul CRD and verify that the operator creates the deployment, and then start making changes to the operator to match the spec, or (b) start doing the changes and add testing as we approach parity with the spec? | 19:40 |
| mordred | corvus: I think b sounds better | 19:41 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 19:41 |
| mordred | corvus: because the current state isn't a supported state, so adding tests to assert it is just work that we're going to replace | 19:42 |
| corvus | (a) is nice because we get testing faster, and can make incremental tested changes to the spec, but it has the downside that we may have to "fix" some parts of the operator which we later end up removing just because they don't match (and may also force us into making some larger changes in a single commit) | 19:42 |
| mordred | that is - unless something about adding some of the testing now makes doing the changes easier | 19:42 |
| mordred | yeah. | 19:43 |
| corvus | mordred: yeah. | 19:43 |
| mordred | YEAH | 19:44 |
| corvus | (also, in parallel, regardless of which thing we do, we'll continue working on the basic setup of the openshift job) | 19:44 |
| tristanC | corvus: figuring out the rdms could be the next step | 19:45 |
| corvus | tristanC: good idea | 19:45 |
| tristanC | i mean, it's pretty easy to get zuul started, but without rdms (to test the build api) and a gerrit, it will be hard to ensure zuul is actually working | 19:46 |
| tristanC | and zookeeper too | 19:46 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: builder: Remove recency table logging https://review.opendev.org/674124 | 19:48 |
| openstackgerrit | Sorin Sbarnea proposed zuul/zuul-jobs master: WIP: add-build-sshkey: add centos/rhel-8 support https://review.opendev.org/674092 | 19:55 |
| Shrews | fungi: omg, i think i just found the image leak issue | 20:01 |
| fungi | ?!? | 20:01 |
| fungi | related to recency table logging? | 20:02 |
| Shrews | no | 20:02 |
| Shrews | that logging was just annoying | 20:02 |
| Shrews | but funnily enough, additional logging elsewhere would have helped | 20:02 |
| mordred | Shrews: oh yeah? I am excited to learn about it | 20:03 |
| fungi | hindsight. most bugs would be found easier with additional logging in the right places ;) | 20:03 |
| *** fbo has quit IRC | 20:03 | |
| mordred | fungi: but only that additional logging, and not additional logging in the wrong place | 20:03 |
| *** nhicher has quit IRC | 20:04 | |
| Shrews | fungi: mordred: so during image deletion, we get a Conflict exception for /reasons/ and expect nodepool to try to delete the image again later. BUT, we have this bit of code that deletes all non-recent uploads left around (https://opendev.org/zuul/nodepool/src/branch/master/nodepool/builder.py#L288-L294) | 20:04 |
| Shrews | the image we failed to delete is obviously non-recent because we are already trying to delete it | 20:05 |
| Shrews | so the zk record goes away, and we never try again | 20:05 |
| fungi | oops | 20:05 |
| fungi | overly-eager cleanup | 20:05 |
| Shrews | yeah. we obviously have that extra cleanup for a reason. just gotta remember why and how to consolidate the two things | 20:06 |
| openstackgerrit | David Shrewsbury proposed zuul/nodepool master: builder: Log all deletions of image upload records https://review.opendev.org/674126 | 20:10 |
| Shrews | fungi: mordred: ^^ that would have helped | 20:11 |
| *** jamesmcarthur has quit IRC | 20:16 | |
| *** tosky has joined #zuul | 20:17 | |
| jeliu_ | tristanC: Jim suggested that I should help with the openshift functional zuul operator job! I was wondering if I should work directly on the current change or download the current change but use a different change-Id | 20:18 |
| openstackgerrit | Merged zuul/nodepool master: Avoid openstacksdk image delete bug https://review.opendev.org/674043 | 20:18 |
| mordred | Shrews: oh FUN | 20:33 |
| *** armstrongs has joined #zuul | 20:42 | |
| *** jeliu_ has quit IRC | 20:46 | |
| *** jeliu_ has joined #zuul | 20:46 | |
| *** armstrongs has quit IRC | 20:51 | |
| pabelanger | anybody remember which job we have that support 2 different parents? | 21:03 |
| *** jeliu_ has quit IRC | 21:07 | |
| ianw | just checking if anyone's around ... the "Unable to find worker for job ..." flood that i saw when i restarted the executor last night, is that something expected now on startup? | 21:47 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Support Rackspace in upload-logs-swift https://review.opendev.org/674136 | 21:47 |
| ianw | i was guessing that while the executor is getting ready, the worker threads aren't up and that's why it was coming out, because the executor started to accept jobs and work after a while | 21:48 |
| corvus | ianw: that happens if zuul tries to stop or resume a job but there is no running thread for it; you might see a bunch of those if an executor stops (and therefore terminates all of the workers immediately) and then zuul later tries to stop some of those same jobs (ie, because the failures triggered a gate reset, etc). | 21:53 |
| corvus | ianw: under that circumstance, it's probably harmless (and correctly categorized as debug-level logging) | 21:54 |
| ianw | corvus: ok, thanks, that was the situation when i restarted the executor to get the streaming console back. i just got a little worried i'd missed something and send the executor into a death spiral :) | 21:58 |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Support Rackspace in upload-logs-swift https://review.opendev.org/674136 | 21:59 |
| corvus | ianw: nope, the ones you restarted looked good when i restarted the whole system this morning | 21:59 |
| corvus | ianw: there were others that were dead; i suspect several oom-kills happened, you caught the 2 that killed only the streaming process; some other ze's had the entire executor killed (which, it turns out is cleaner, but does nothing to alleviate the load) | 22:00 |
| ianw | ahhh, ok, that would explain it; i should have kept checking and it might have been clearer | 22:02 |
| *** tosky has quit IRC | 22:07 | |
| *** iamweswilson_ has joined #zuul | 22:29 | |
| *** samccann_ has joined #zuul | 22:29 | |
| *** jtanner_ has joined #zuul | 22:30 | |
| *** gundalow_ has joined #zuul | 22:30 | |
| *** mnasiadka_ has joined #zuul | 22:30 | |
| *** kmalloc_ has joined #zuul | 22:30 | |
| *** jpenag has joined #zuul | 22:36 | |
| *** jpena|off has quit IRC | 22:37 | |
| *** gundalow has quit IRC | 22:37 | |
| *** samccann has quit IRC | 22:37 | |
| *** iamweswilson has quit IRC | 22:37 | |
| *** jtanner has quit IRC | 22:37 | |
| *** mordred has quit IRC | 22:37 | |
| *** mnasiadka has quit IRC | 22:37 | |
| *** kmalloc has quit IRC | 22:37 | |
| *** iamweswilson_ is now known as iamweswilson | 22:37 | |
| *** samccann_ is now known as samccann | 22:37 | |
| *** gundalow_ is now known as gundalow | 22:37 | |
| *** mnasiadka_ is now known as mnasiadka | 22:37 | |
| *** kmalloc_ is now known as kmalloc | 22:37 | |
| *** jtanner_ is now known as jtanner | 22:37 | |
| *** panda has quit IRC | 22:41 | |
| *** panda has joined #zuul | 22:42 | |
| *** mordred has joined #zuul | 22:44 | |
| openstackgerrit | James E. Blair proposed zuul/zuul-jobs master: Support Rackspace in upload-logs-swift https://review.opendev.org/674136 | 23:10 |
| *** igordc has quit IRC | 23:10 | |
| *** ianychoi has quit IRC | 23:12 | |
| *** armstrongs has joined #zuul | 23:25 | |
| *** armstrongs has quit IRC | 23:34 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!