| *** JasonCL has joined #zuul | 00:24 | |
| *** JasonCL has quit IRC | 00:29 | |
| *** openstackgerrit has quit IRC | 00:33 | |
| *** JasonCL has joined #zuul | 01:19 | |
| *** JasonCL has quit IRC | 01:23 | |
| *** baiyi has joined #zuul | 02:11 | |
| *** JasonCL has joined #zuul | 02:14 | |
| *** JasonCL has quit IRC | 02:19 | |
| *** baiyi1 has joined #zuul | 02:21 | |
| *** baiyi has quit IRC | 02:23 | |
| *** baiyi1 is now known as baiyi | 02:23 | |
| *** baiyi has quit IRC | 02:39 | |
| *** JasonCL has joined #zuul | 03:09 | |
| *** JasonCL has quit IRC | 03:13 | |
| *** JasonCL has joined #zuul | 03:55 | |
| *** threestrands has joined #zuul | 04:14 | |
| *** threestrands has quit IRC | 04:14 | |
| *** threestrands has joined #zuul | 04:14 | |
| *** jappleii__ has quit IRC | 04:15 | |
| *** threestrands has quit IRC | 04:15 | |
| *** threestrands has joined #zuul | 04:16 | |
| *** threestrands has quit IRC | 04:16 | |
| *** threestrands has joined #zuul | 04:16 | |
| *** baiyi has joined #zuul | 04:28 | |
| *** threestrands has quit IRC | 04:36 | |
| *** threestrands has joined #zuul | 04:43 | |
| *** threestrands has quit IRC | 04:43 | |
| *** threestrands has joined #zuul | 04:43 | |
| *** threestrands has quit IRC | 04:44 | |
| *** threestrands has joined #zuul | 04:44 | |
| *** threestrands has quit IRC | 04:44 | |
| *** threestrands has joined #zuul | 04:44 | |
| *** threestrands has quit IRC | 04:45 | |
| *** threestrands has joined #zuul | 04:46 | |
| *** threestrands has quit IRC | 04:46 | |
| *** threestrands has joined #zuul | 04:46 | |
| *** threestrands has quit IRC | 04:47 | |
| *** threestrands has joined #zuul | 04:47 | |
| *** threestrands has quit IRC | 04:47 | |
| *** threestrands has joined #zuul | 04:47 | |
| tobiash | jlk, SpamapS: ghe 2.12 is released now | 04:53 |
|---|---|---|
| tobiash | And it includes github apps as tech preview :) | 04:53 |
| SpamapS | tobiash: that's good to hear! | 05:16 |
| SpamapS | No idea what version we even run | 05:17 |
| *** openstackgerrit has joined #zuul | 05:24 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/527579 | 05:24 |
| tristanC | jeblair: rcarrillocruz: 527579 adds the job's detail page we talked about, it even includes a tiny rendering of the nodeset with box colored based on label name hash | 05:26 |
| tristanC | more importantly, it also list source_context location to answer the "where is this job defined" question | 05:28 |
| *** baiyi1 has joined #zuul | 05:55 | |
| *** baiyi has quit IRC | 05:57 | |
| *** baiyi1 is now known as baiyi | 05:57 | |
| *** threestrands has quit IRC | 06:03 | |
| tobiash | tristanC: cool stuff | 06:15 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 06:44 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/527579 | 06:45 |
| tristanC | tobiash: thanks for the review, made me fixed my emacs conf... i was using setq instead of setq-default for indent-tabs-mode :) | 06:48 |
| *** flepied__ has quit IRC | 07:09 | |
| *** hashar has joined #zuul | 07:57 | |
| *** baiyi1 has joined #zuul | 07:57 | |
| *** baiyi has quit IRC | 08:00 | |
| *** baiyi has joined #zuul | 08:02 | |
| *** baiyi1 has quit IRC | 08:04 | |
| *** jpena|off is now known as jpena | 08:09 | |
| *** xinliang has quit IRC | 08:35 | |
| *** flepied__ has joined #zuul | 08:35 | |
| *** yolanda__ has joined #zuul | 08:37 | |
| rcarrillocruz | tristanC: good stuff :-) | 08:39 |
| *** yolanda has quit IRC | 08:40 | |
| *** yolanda__ has quit IRC | 08:46 | |
| *** yolanda__ has joined #zuul | 08:46 | |
| *** yolanda__ is now known as yolanda | 08:47 | |
| *** xinliang has joined #zuul | 08:47 | |
| *** xinliang has quit IRC | 08:47 | |
| *** xinliang has joined #zuul | 08:47 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: doc: refine zuul_return instruction https://review.openstack.org/527633 | 08:53 |
| *** flepied_ has joined #zuul | 09:35 | |
| *** flepied__ has quit IRC | 09:38 | |
| *** electrofelix has joined #zuul | 09:57 | |
| rcarrillocruz | tristanC: is zuul web going to host both websocket AND the dashboard via aiohttp ? or is supposedly to run a separate apache to serve the dashboard and aiohttp for the streaming stuff | 10:08 |
| kklimonda | it seems that nodepool logic for fulfilling node requests has a suboptimal behavior when you reach a quota and there are requests for multiple node types, some of which are ready | 10:11 |
| kklimonda | nodepool takes the next request, puts it into a pending state, and waits until its possible to spawn this node type, never processing requests for node types that are already available (via min-ready) | 10:12 |
| kklimonda | (it could be that it's been fixed in the last 2-3 weeks, I'm running a little behind feature/zuulv3) | 10:12 |
| tristanC | rcarrillocruz: zuul web is already serving both websocket and the dashboard via aiohttp | 10:14 |
| rcarrillocruz | Excellent thx | 10:14 |
| tristanC | kklimonda: there is a series of patch to greatly improve provider quota here: https://review.openstack.org/#/q/topic:quota-second-try | 10:15 |
| kklimonda | tristanC: oh, those are actually pretty cool - we could use cpu/ram instead of hardcoding number of servers per pool | 10:16 |
| tristanC | rcarrillocruz: actually zuul-web is going to also ingests github payloads when https://review.openstack.org/#/c/504267/ lands | 10:16 |
| rcarrillocruz | Neat | 10:17 |
| kklimonda | tristanC: but I don't think any of those will fix that particular issue - when we reach any quota, nodepool will just get "stuck" waiting for more resources for the node type it's trying to fulfill | 10:17 |
| tristanC | kklimonda: oh right, but then what would be logic to lock and wait for a nodeset with more than one node? | 10:23 |
| tristanC | if the provider is almost running at capacity, this nodeset may get delayed until all the one node request are fullfilled | 10:24 |
| fdegir | rcarrillocruz: regarding zuulv3 installation | 10:29 |
| fdegir | rcarrillocruz: did you install everything manually or tried something else or have something that installs stuff automatically which you can share? | 10:29 |
| rcarrillocruz | Well | 10:30 |
| rcarrillocruz | Let me link you | 10:30 |
| fdegir | ok | 10:30 |
| *** hashar is now known as hasharAway | 10:36 | |
| rcarrillocruz | on my first POC, i simply used pabelanger roles: ansible-role-zuul , ansible-role-nodepool | 10:39 |
| rcarrillocruz | which can be found on openstack git | 10:39 |
| rcarrillocruz | https://github.com/openstack/ansible-role-nodepool | 10:39 |
| rcarrillocruz | however, i'm working now on containerizing everything with an ansible-container app | 10:40 |
| rcarrillocruz | https://github.com/rcarrillocruz/zuul-ci-container | 10:40 |
| rcarrillocruz | with that, the workflow is | 10:40 |
| rcarrillocruz | 1. create your config files under config/ folder | 10:41 |
| rcarrillocruz | 2. ansible-container build | 10:41 |
| rcarrillocruz | 3a. if you want to run in your local docker env, ansible-container run | 10:41 |
| rcarrillocruz | 3b: if you want to run in an openshift cluster (i use oc cluster up), ansible-container --enginer deploy + ansible-playbook the deployment playbook created by deploy verb | 10:42 |
| rcarrillocruz | at this stage, i can completely run a very basic zuul on openshift | 10:42 |
| rcarrillocruz | i need to add more things to it to make it usable | 10:42 |
| rcarrillocruz | namely | 10:42 |
| rcarrillocruz | zuul-logs to hold job logs | 10:42 |
| rcarrillocruz | zuul-sqlstore to hold sql job log runs (with sql reporter) | 10:43 |
| rcarrillocruz | and zuul-web | 10:43 |
| rcarrillocruz | ONCE it's decent, then README | 10:43 |
| rcarrillocruz | and maybe announce it a bit more, haven't so far cos is very rough at the moment | 10:43 |
| fdegir | ok, thanks | 10:43 |
| rcarrillocruz | but, feel free to use it, as i said, it can be used on dev for both local docker and openshift | 10:44 |
| fdegir | I tried pabelanger windmill | 10:44 |
| fdegir | the gerrit event streaming and stuff works with my config for opnfv gerrit | 10:44 |
| fdegir | but having some errors in nodepool and openstack keystone | 10:44 |
| rcarrillocruz | yah, windmill is an AIO which uses the roles i pasted earlier | 10:44 |
| fdegir | failed to run noop even | 10:44 |
| rcarrillocruz | what nodepool errors | 10:44 |
| fdegir | I wiped out the installation and started again | 10:45 |
| fdegir | will come back with logs when i try again | 10:45 |
| fdegir | I'll try zuul-ci-container again | 10:45 |
| fdegir | what I need at this phase is the ability to run basic noop | 10:45 |
| fdegir | and then look at openstack base jobs | 10:45 |
| fdegir | slowly bringing up basic ci with zuulv3 in parallel to jenkins to have ability to compare outcome from both | 10:46 |
| fdegir | I think I have something to look at now | 10:46 |
| fdegir | thanks for the pointers | 10:46 |
| rcarrillocruz | sure, just ping me if you have any issues, with gerrit should be even easier, as you don't have the issue of the review system sending events to your zuul, it's zuul listeniing to gerrit event stream | 10:47 |
| rcarrillocruz | would be superawesome to have a mechanism to listen to webhooks, rathen than push | 10:48 |
| rcarrillocruz | tristanC: ^ any way you may think of ? | 10:48 |
| rcarrillocruz | cos i'm tied to deploying stuff on an oc cluster up in a public VM, like RDO or something, so zuul is reachable | 10:48 |
| rcarrillocruz | and well, i don't want to spend my day having a reverse proxy down to my local docker or mess with my home NAT :/ | 10:49 |
| tristanC | rcarrillocruz: it seems like having the fedmsg driver as a source trigger would let you "listen to webhook" | 10:57 |
| rcarrillocruz | AHA | 10:57 |
| rcarrillocruz | so | 10:57 |
| rcarrillocruz | webhook sends to fedmsg | 10:57 |
| rcarrillocruz | github driver listens to fedmsg notifications? | 10:58 |
| tristanC | rcarrillocruz: you could even dismiss the webhook and just follow project activity and trigger job as you want | 10:58 |
| rcarrillocruz | geez | 10:58 |
| rcarrillocruz | the future is BRIGHT | 10:58 |
| tristanC | rcarrillocruz: note that the fedmsg driver remains to be defined | 11:01 |
| rcarrillocruz | yah, | 11:02 |
| rcarrillocruz | lol | 11:06 |
| rcarrillocruz | it seems is a common thing amongst users | 11:06 |
| rcarrillocruz | http://www.ultrahook.com/ | 11:06 |
| kklimonda | tristanC: perhaps nodepool could consider "ready" unlocked nodes for deletion when trying to satisfy a request | 11:14 |
| kklimonda | it's still possible that other launcher will snatch the freed up resources for some other request, but at least those resources won't be "wasted" | 11:15 |
| *** jkilpatr has quit IRC | 11:23 | |
| *** JasonCL has quit IRC | 11:27 | |
| rcarrillocruz | tristanC: the native js toolchain, i take that will be installed with the zuul pypi package. Is that supposed to be a deliverable for 3.0 or will it be 3.1 ? | 11:33 |
| rcarrillocruz | like, i'm doing a role for zuul-web, i wonder if i should create tasks for creating www/static, pulling bootstrap, angular, etc as puppet-zuul does or just wait as that will come with 3.0 anyways | 11:34 |
| tristanC | rcarrillocruz: mordred probably knows what's the status of the js toolchain | 11:40 |
| tristanC | rcarrillocruz: fwiw, there is this change to add zuul-web to ansible-role-zuul: https://review.openstack.org/499417 | 11:42 |
| *** JasonCL has joined #zuul | 11:43 | |
| tristanC | rcarrillocruz: also there is this rpm package for zuulv3 that i meant to propose to fedora https://softwarefactory-project.io/r/gitweb?p=scl/zuul-distgit.git;a=blob;f=zuul.spec | 11:44 |
| tristanC | rcarrillocruz: this includes a zuul-webui sub package with all the stuff minified and ready to use for zuul-web... | 11:45 |
| *** jkilpatr has joined #zuul | 11:54 | |
| rcarrillocruz | cool | 11:56 |
| *** dmellado has quit IRC | 12:47 | |
| *** jpena is now known as jpena|lunch | 12:53 | |
| *** dmellado has joined #zuul | 12:58 | |
| *** toabctl has quit IRC | 13:00 | |
| *** dkranz has joined #zuul | 13:02 | |
| *** dmellado has quit IRC | 13:08 | |
| *** dmellado has joined #zuul | 13:09 | |
| *** flepied__ has joined #zuul | 13:53 | |
| *** flepied_ has quit IRC | 13:55 | |
| *** jpena|lunch is now known as jpena | 13:58 | |
| mordred | tristanC, rcarrillocruz: re: js toolchain patch, I've got a half-finished patch - if my head clears a bit more today I'll get it finished and pushed up | 13:59 |
| *** flepied_ has joined #zuul | 14:10 | |
| *** flepied__ has quit IRC | 14:13 | |
| *** JasonCL has quit IRC | 14:20 | |
| Shrews | aaaaaaarrrrrrrrrgh!!!! spent half a day yesterday and time this morning wondering why my gateway test no longer works. forgot i added a 'return' to short circuit things for a bit | 14:35 |
| Shrews | i iz teh dumb | 14:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 14:36 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 14:38 |
| *** openstack has quit IRC | 14:39 | |
| *** openstack has joined #zuul | 14:43 | |
| *** ChanServ sets mode: +o openstack | 14:43 | |
| *** openstack has quit IRC | 14:43 | |
| *** openstack has joined #zuul | 14:48 | |
| *** ChanServ sets mode: +o openstack | 14:48 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 15:04 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 15:07 |
| Shrews | ^^^ doc fixes | 15:07 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: WIP: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 15:19 |
| *** toabctl has joined #zuul | 15:27 | |
| tobiash | yay, upgraded ghe test instance to 2.12 and the api doesn't return protected branches anymore | 15:27 |
| tobiash | which makes zuul to completely ignore the github projects | 15:28 |
| openstackgerrit | Andreas Jaeger proposed openstack-infra/zuul-jobs master: Revert "Revert "Add sphinx_python variable to sphinx role and job"" https://review.openstack.org/526666 | 15:32 |
| pabelanger | tobiash: is that good or bad? | 15:34 |
| tobiash | pabelanger: in other words, zuul doesn't work with ghe 2.12 currently | 15:35 |
| tobiash | currently trying to figure out how to work around this | 15:35 |
| pabelanger | tobiash: boo | 15:35 |
| tobiash | looks like branch protection info is just gone from the api response :( | 15:36 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Convert back to zuul.projects https://review.openstack.org/524460 | 16:01 |
| *** maxamillion has quit IRC | 16:15 | |
| *** maxamillion has joined #zuul | 16:15 | |
| *** sshnaidm|ruck has quit IRC | 16:17 | |
| * sc68cal is currently on GHE 2.10 | 16:19 | |
| Shrews | tobiash: going to start looking at your quota changes now | 16:19 |
| Shrews | tobiash: those functional test timeouts don't look so good though :( | 16:19 |
| Shrews | although they're non-voting, we should see what's happening there | 16:21 |
| *** sshnaidm has joined #zuul | 16:21 | |
| Shrews | jeblair: fyi, i consider 525276 ready if you have time to review it before the end of the week | 16:22 |
| Shrews | no telling if i'll even remember what zuul is after the break :) | 16:23 |
| jeblair | Shrews: cool, i'll take a look, thanks | 16:55 |
| jlk | tobiash: neat! | 16:56 |
| jlk | tobiash: (re GHE) | 16:57 |
| *** flepied__ has joined #zuul | 17:01 | |
| *** hasharAway is now known as hashar | 17:01 | |
| *** flepied_ has quit IRC | 17:03 | |
| tobiash | Shrews: what functional test timeout? | 17:05 |
| tobiash | jlk: looks like we have different oppinions about what's neat ;) | 17:07 |
| *** jkilpatr has quit IRC | 17:07 | |
| jlk | tobiash: well, preview is better than not at all (for apps) | 17:08 |
| tobiash | jlk: ah this one, I thought you meant the broken branch protection api | 17:09 |
| jlk | oh, no I missed that. | 17:09 |
| tobiash | So we have apps now but no zuul... | 17:09 |
| jlk | GAH | 17:09 |
| *** tumbarka has joined #zuul | 17:12 | |
| tobiash | jlk: apps tech preview means that you can add apps via api with a special header | 17:13 |
| tobiash | seems that no ui is yet there | 17:13 |
| jlk | that's.. interesting. It at least lets us develop against it. | 17:14 |
| *** umbarkar has quit IRC | 17:15 | |
| tobiash | jlk: just saw, at least the user side of github apps is there | 17:15 |
| tobiash | so the user seems to be able to install apps into repos/orgs | 17:16 |
| tobiash | but the app itself needs to be installed via api | 17:16 |
| tobiash | but I think that's ok for starting | 17:17 |
| *** jkilpatr has joined #zuul | 17:23 | |
| *** jpena is now known as jpena|brb | 17:24 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: group matchers together and explain them https://review.openstack.org/527748 | 17:24 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: group matchers together and explain them https://review.openstack.org/527748 | 17:25 |
| Shrews | tobiash: timeouts are here with latest PS: https://review.openstack.org/503838 | 17:27 |
| Shrews | tobiash: and all dependent reviews, so appears to be a real thing | 17:28 |
| tobiash | Shrews: hm, any idea how to debug this? | 17:29 |
| tobiash | the log doesn't seem to have any advice | 17:29 |
| tobiash | jlk: ah no, I'm mistaken, the github apps ui is completely there, just didn't find it when I looked for it | 17:30 |
| jlk | woo! | 17:30 |
| tobiash | Shrews: found connection issues: http://logs.openstack.org/82/504282/16/check/nodepool-functional-py35-src/6e6d16d/job-output.txt.gz#_2017-12-12_08_37_00_074342 | 17:34 |
| tobiash | that might have been caused by the zuul problems yesterday | 17:34 |
| tobiash | I'll recheck a few changes to double check | 17:34 |
| Shrews | maybe | 17:36 |
| *** jkilpatr has quit IRC | 17:38 | |
| *** jpena|brb is now known as jpena | 17:43 | |
| *** jkilpatr has joined #zuul | 17:52 | |
| *** baiyi has quit IRC | 17:55 | |
| jeblair | Shrews: fingergw looks good -- i just left one tiny doc nit | 17:56 |
| Shrews | jeblair: cool. i'll fix that in a follow up | 17:57 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Fix nit in fingergw doc https://review.openstack.org/527751 | 18:02 |
| jlk | jeblair: I started poking at removing github3.py... I think you're right, it's a bit more involved than I thought. we'll need some basic things, like iterators for pagination of results for things, and re-thinking some data structures. I'm still poking away at it though. | 18:03 |
| jeblair | jlk: ack, thx | 18:03 |
| *** JasonCL has joined #zuul | 18:06 | |
| Shrews | tobiash: jeblair: reviewed the quota changes. only found an issue on the first review. still need to see the functional tests pass | 18:15 |
| tobiash | Shrews: thanks | 18:15 |
| Shrews | clarkb: i'm going to see if i can help with your test after lunch | 18:15 |
| clarkb | Shrews: thanks, I haven't had a chance to look at it yet myself. Been focused on sprint/onboarding new root stuff | 18:17 |
| clarkb | Shrews: feel free to push patchsets or whatever | 18:18 |
| *** myoung|rover is now known as myoung | 18:18 | |
| clarkb | tobiash: microsoft is adding or has added an ssh server to windows 10 | 18:25 |
| clarkb | tobiash: not sure if that changes anything for you, but thought I would point it out | 18:25 |
| tobiash | clarkb: thanks, but I think ansible has more problems than ssh with windows | 18:26 |
| *** flepied__ has quit IRC | 18:29 | |
| tobiash | jlk: further debugged the branch protection problem | 18:35 |
| tobiash | looks like the zuul user now needs admin rights on a repo in order to view branch protection | 18:35 |
| tobiash | write access is not enough | 18:36 |
| *** electrofelix has quit IRC | 18:43 | |
| *** jpena is now known as jpena|off | 18:49 | |
| jlk | hrm, I thought that was always the case. | 18:56 |
| jlk | might be better in app land where you can request specifically just access to those things | 18:57 |
| tobiash | probably | 19:06 |
| tobiash | Shrews: the upper three changes of the quota stack all succeeded the functional tests on the recheck | 19:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Update javascript jobs to take npm_command variable https://review.openstack.org/527770 | 19:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Install yarn if needed in javascript jobs https://review.openstack.org/527771 | 19:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage status and streaming https://review.openstack.org/487538 | 19:36 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Send open CORS header for jobs and builds https://review.openstack.org/527772 | 19:36 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Add cloud quota handling https://review.openstack.org/503838 | 19:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 19:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 19:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 19:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 19:44 |
| kklimonda | can zuulv3 tenants be used to completely separate projects, hiding some of them from public? | 19:56 |
| tobiash | kklimonda: yes that should be possible | 19:58 |
| tobiash | kklimonda: but you have to take care to isolate them on zuul-web (tenants are path separated there) | 19:58 |
| kklimonda | right | 19:58 |
| tobiash | that's also what we will be doing | 19:59 |
| tobiash | kklimonda: do you even need to hide the information of existence? | 20:00 |
| kklimonda | tobiash: lets say yes, I'm exploring our options right now and what can be done in zuul - whether we should recommend a separate deployment instead | 20:01 |
| tobiash | in this case you would need to block the tenants overview page and probably also merge the tenants.yaml from different sources | 20:01 |
| tobiash | the latter at least if you want your users to be able to propose changes to tenants.yaml | 20:02 |
| *** jkilpatr has quit IRC | 20:03 | |
| kklimonda | mhm, thanks - I haven't kept up with the latest changes to zuul-web to separate tenants, I'll have to read up a little on that | 20:04 |
| tobiash | kklimonda: and we would need to restrict labels to tenants (which I also need but had no time for this yet) if you want to leverage node image caching mechanisms | 20:05 |
| kklimonda | mhm | 20:05 |
| tobiash | that's not implemented yet | 20:05 |
| pabelanger | kklimonda: yah, shared deployments are much easier in zuulv3. While we haven't done so yet on zuulv3.o.o, I do expect us to have another tenant on it. It will still be public, but private between each other. | 20:06 |
| tobiash | but we're also going the way with one deployment and isolated projects via tenants | 20:06 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add finger gateway https://review.openstack.org/525276 | 20:08 |
| kklimonda | thanks, I'll keep it all in mind when we discuss it internally | 20:08 |
| pabelanger | Yah, shared zuul will make operations easier for sure | 20:11 |
| jlk | are nodepool node labels a per-tenant thing as well? | 20:13 |
| jlk | can you separate out the Zuul tenants by OpenStack tenants(projects) too? | 20:14 |
| tobiash | jlk: not yet, but some time ago there was a discussuin about letting them be restricted in tenants.yaml | 20:14 |
| pabelanger | you could have 1 nodepool-launcher per tenant today | 20:15 |
| pabelanger | but yah, think labels would be globak | 20:15 |
| pabelanger | global* | 20:15 |
| pabelanger | pretty exciting, seeing new uses for nodepool / zuul :D | 20:16 |
| tobiash | yes currently, but we will need some sort of restriction in the tenant config for labels | 20:16 |
| pabelanger | yah | 20:16 |
| tobiash | and if you distribute these labels to different providers you can separate them to different openstack tenants | 20:16 |
| tobiash | btw, had today an interesting discussion about how to handle a mandatory authenticating proxy for internet access | 20:22 |
| tobiash | that's the worst thing you can have for any type of deployment of anything | 20:22 |
| tobiash | the idea to handle this in check pipelines without disclosing proxy credentials is to install an unauthenticated squid within a trusted base job and revoke sudo for the zuul user | 20:23 |
| tobiash | that way the untrusted jobs can use unauthenticated localhost as proxy | 20:23 |
| tobiash | and with a little bit of care about the squid config permissions we can lock the untrusted job away from the proxy credentials | 20:24 |
| tobiash | Shrews: have you seen already such a build failure https://review.openstack.org/#/c/503838/21 ? | 20:29 |
| tobiash | looks to me like problems setting up zookeeper | 20:30 |
| tobiash | whole log full of kazoo.exceptions.ConnectionLoss | 20:31 |
| Shrews | tobiash: on rare occasion. just recheck | 20:31 |
| tobiash | ok | 20:31 |
| *** jkilpatr has joined #zuul | 20:39 | |
| Shrews | clarkb: so the reason your test still works w/o your fix is because the self.request object in poll() has a valid self.request.lock object, thus the exception is not being thrown as expected. | 20:41 |
| Shrews | clarkb: i'm not sure i understand what would be the cause of self.request.lock being None | 20:41 |
| clarkb | Shrews: gotcha so its not necesarily the zk state itself but the python process state that is affecting us? | 20:42 |
| tobiash | Shrews: that's where I also was stuck looking at this change | 20:42 |
| *** logan- has quit IRC | 20:43 | |
| *** logan- has joined #zuul | 20:43 | |
| Shrews | i think we need to figure out what causes the lock to be None during the race | 20:43 |
| *** JasonCL has quit IRC | 20:44 | |
| clarkb | this also means the lock could still be in place in zk right? | 20:44 |
| clarkb | its just the python object that is foobar potentially? | 20:44 |
| Shrews | clarkb: maybe? i honestly don't know | 20:44 |
| Shrews | if the cleanup thread removes the request lock, that removes the kazoo lock stuff too, making the Kazoo lock object invalid. how it handles that, i'm not sure | 20:46 |
| tobiash | unlocking an inexistent lock is handled gracefully | 20:47 |
| Shrews | looks like the kazoo release would get a NoNodeError and hide that fact from you | 20:47 |
| tobiash | yes | 20:47 |
| Shrews | yeah | 20:48 |
| tobiash | but our problem is that nodepool itself calls unlockNodeRequest somewhere in a different thread | 20:48 |
| tobiash | but I didn't find where | 20:48 |
| clarkb | tobiash: I thought I posted the two places in the commit message that I found | 20:50 |
| tobiash | maybe http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/launcher.py?h=feature/zuulv3#n203 is the source | 20:50 |
| clarkb | oh except now we are saying it may not be that | 20:51 |
| Shrews | tobiash: i don't think so. a handler already has the request locked, so the check above should handle that | 20:53 |
| tobiash | hrm | 20:53 |
| tobiash | maybe the lost requsts cleanup worker if there was a connection loss to zk | 20:54 |
| tobiash | then the locks would be invalid -> cleanup worker cleans the lock up with setting it to none | 20:54 |
| *** JasonCL has joined #zuul | 20:54 | |
| tobiash | but that would probably break more than that | 20:55 |
| *** JasonCL has quit IRC | 20:59 | |
| Shrews | hrm, if the request object is somehow the same one between the cleanup thread, and the request handling thread, that could be | 21:00 |
| Shrews | wouldn't be the first time python object references have hurt us :) | 21:01 |
| *** JasonCL has joined #zuul | 21:01 | |
| clarkb | so that I understand better are you saying that if there was no lock in zk the unlock in poll that is failing would not fail? | 21:01 |
| clarkb | (and as a result my test passes without the chagne as well) | 21:02 |
| Shrews | clarkb: yah, but there *should* be a lock | 21:02 |
| Shrews | that's what has me baffled | 21:02 |
| clarkb | right, thanks | 21:02 |
| *** hashar has quit IRC | 21:03 | |
| tobiash | well, the test explicitly deletes the lock from outside | 21:03 |
| tobiash | but that doesn't break the unlock | 21:03 |
| clarkb | tobiash: right because I think I assumed that the failures was due to the lock missing in zk | 21:03 |
| clarkb | but its not, its the python lock object | 21:03 |
| tobiash | yes | 21:03 |
| Shrews | to be clear, self.request.lock should ALWAYS be a Kazoo Lock() at that point | 21:04 |
| tobiash | Shrews: I think that is the case | 21:04 |
| clarkb | right its still a bug just not where I thought it was | 21:04 |
| Shrews | tobiash: but it wasn't the case for the thing clarkb and myself saw in production | 21:05 |
| Shrews | so clarkb is right... seems to be elsewhere | 21:06 |
| tobiash | now I'm confused | 21:06 |
| tobiash | so the error in production was a missing lock or the exception when trying to unlock? | 21:07 |
| jeblair | fbo_: the git driver looks really good; i left a comment inline about something we may want te restructure slightly | 21:07 |
| Shrews | tobiash: the error in production was an exception *caused* by self.request.lock being None | 21:08 |
| *** JasonCL has quit IRC | 21:08 | |
| Shrews | in the poll() method | 21:08 |
| tobiash | which is not the missing lock because that wouldn't have cause an exception | 21:09 |
| tobiash | so notepool itself unlocked it correctly but just from an unexpected time/codepath | 21:09 |
| Shrews | tobiash: correct. lock.release(), as we just determined, handles that gracefully | 21:09 |
| Shrews | tobiash: nodepool should NOT have unlocked it (which is the problem here). i suspect an overwrite of the object b/c of object references, but that's just a theory | 21:10 |
| * Shrews has to step away for a bit | 21:11 | |
| *** threestrands has joined #zuul | 21:18 | |
| *** threestrands has quit IRC | 21:18 | |
| *** threestrands has joined #zuul | 21:18 | |
| *** JasonCL has joined #zuul | 21:20 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix nit in fingergw doc https://review.openstack.org/527751 | 21:22 |
| *** JasonCL has quit IRC | 21:24 | |
| tobiash | mordred: looking at your yarn stuff | 21:28 |
| tobiash | what's the difference in package.json between the versions with '^' and the versions without? | 21:28 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Serve keys from canonical project name https://review.openstack.org/504807 | 21:31 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add cloud quota handling https://review.openstack.org/503838 | 21:33 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Don't fail on quota exceeded https://review.openstack.org/503051 | 21:33 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Make max-servers optional https://review.openstack.org/504282 | 21:33 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Support cores limit per pool https://review.openstack.org/504283 | 21:33 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Support ram limit per pool https://review.openstack.org/504284 | 21:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Send open CORS header for jobs and builds https://review.openstack.org/527772 | 21:36 |
| dmsimard | tobiash: ^ woot | 21:39 |
| dmsimard | best feature evar | 21:39 |
| dmsimard | <3 | 21:39 |
| tobiash | yay :):):) | 21:39 |
| *** hashar has joined #zuul | 21:40 | |
| dmsimard | tobiash: didn't we think of that back in denver ? | 21:44 |
| kklimonda | hmm, I just had zuul-merger fail to merge something (explode rather spectacularly: https://pastebin.com/vrER4DSR) but rerunning the same command (`git merge -s resolve FETCH_HEAD`) on the same HEAD worked fine: https://pastebin.com/621M1Jd0 | 21:44 |
| tobiash | that was most of my time in denver and refining the week after | 21:44 |
| *** JasonCL has joined #zuul | 21:46 | |
| kklimonda | I can probably ask author to rebase, but I'm curious as of why is this happening | 21:46 |
| clarkb | kklimonda: couple things, are you sure fetch head is the same ref there? | 21:47 |
| clarkb | kklimonda: but also I want to say that zuul attempts to approximate the jgit merge method so that it is similar to gerrit's behavior, that could possibly have an effect here | 21:47 |
| kklimonda | clarkb: yes | 21:47 |
| kklimonda | clarkb: can I do the same from commandline? i.e. approximate what jgit does? | 21:48 |
| clarkb | kklimonda: yes, but I'm not quite sure whee that happens in zuul so not sure how it is done | 21:48 |
| clarkb | it might be a git repository setting? or I could just be misremembering | 21:49 |
| kklimonda | (also, gerrit is not showing merge failure for this review) | 21:50 |
| jeblair | kklimonda, clarkb: the 'approximate jgit' behavior is in the pastebin -- it's 'git merge -s resolve' | 21:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Drop local fork of GitPython for 2.1.8 release https://review.openstack.org/527298 | 21:51 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix attribute syntax in docs https://review.openstack.org/526972 | 21:51 |
| jeblair | kklimonda: are there any other dependencies involved? | 21:53 |
| kklimonda | @jeblair there is another review depending on this one | 21:54 |
| jeblair | but this one doesn't have a depends-on: header or anything? | 21:54 |
| *** dkranz has quit IRC | 21:54 | |
| jeblair | kklimonda: what's the gerrit merge strategy for this repo? 'merge if necessary'? | 21:55 |
| kklimonda | jeblair: no, it doesn't have any depends-on itself, and its parent isn't in the review. the strategy is 'merge if necessary' indeed | 21:56 |
| jeblair | kklimonda: well, hrm. i'm not positive that gerrit's merge-conflict detection works exactly the same way as its actual merges. it's possible that everything is working as intended and if you did tell gerrit to submit this, it would fail. or it could be that this is just an edge case where jgit and git don't work quite the same. or perhaps jgit got better and we're now being too conservative. | 21:58 |
| jeblair | at any rate, for most of those cases, the best thing is probably to rebase. and if this starts happening more, we may need to look into whether we should still be using '-s resolve'. | 21:59 |
| kklimonda | thanks, I'll just ask author to do a rebase - we're having some issues with gerrit replication right now, so that's a low priority - if this happen again, I'll dig more into it | 22:00 |
| *** hashar has quit IRC | 22:39 | |
| *** harlowja has quit IRC | 22:54 | |
| *** flepied__ has joined #zuul | 22:54 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add implied-branches pragma directive https://review.openstack.org/527805 | 23:09 |
| *** harlowja has joined #zuul | 23:42 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!