| *** sshnaidm has joined #zuul | 00:40 | |
| *** sshnaidm is now known as sshnaidm|off | 00:40 | |
| SpamapS | weird.. I have a job "queued" for 20 minutes with plenty of ready nodes of the label type.. | 01:42 |
|---|---|---|
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:47 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:48 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 01:52 |
| tristanC | SpamapS: it happens sometime in our deployment too, didn't investigated yet, but using zuul promote usually unlock it | 01:55 |
| jhesketh | corvus: any objections to having the update_queue terminated as part of an executor stop? | 01:59 |
| jhesketh | should help turn things off quicker if we hit network issues in the future | 01:59 |
| corvus | jhesketh: yeah, i think there's some improvement that can be made there | 02:05 |
| jhesketh | corvus: cool, going to hack on that this afternoon. I'm currently looking to see if there'll be any races or anything that will care if an update fails | 02:05 |
| SpamapS | promote? | 02:35 |
| SpamapS | what's that? | 02:35 |
| tristanC | SpamapS: it's a zuul cli sub command | 02:39 |
| SpamapS | cool I didn't know | 02:40 |
| *** elyezer has quit IRC | 02:40 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/pipelines.json route https://review.openstack.org/541521 | 02:41 |
| *** sshnaidm_ has joined #zuul | 02:51 | |
| *** sshnaidm|off has quit IRC | 02:52 | |
| *** elyezer has joined #zuul | 02:53 | |
| *** sshnaidm_ has quit IRC | 03:08 | |
| *** elyezer has quit IRC | 03:22 | |
| *** elyezer has joined #zuul | 03:23 | |
| *** sshnaidm_ has joined #zuul | 03:23 | |
| *** jimi|ansible has quit IRC | 03:38 | |
| *** openstackgerrit has quit IRC | 04:04 | |
| *** harlowja has quit IRC | 04:32 | |
| *** openstackgerrit has joined #zuul | 05:06 | |
| openstackgerrit | Merged openstack-infra/zuul master: Sync when doing disk accountant testing https://review.openstack.org/541486 | 05:06 |
| openstackgerrit | Merged openstack-infra/zuul master: Use nested tempfile fixture for cleanups https://review.openstack.org/541487 | 05:13 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix stuck node requests across ZK reconnection https://review.openstack.org/541454 | 05:24 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Revert "Test Use item.checkout from zuul.projects when mirroring" https://review.openstack.org/540950 | 05:30 |
| *** harlowja has joined #zuul | 05:37 | |
| *** elyezer has quit IRC | 06:33 | |
| *** elyezer has joined #zuul | 06:35 | |
| *** sshnaidm_ has quit IRC | 06:37 | |
| *** sshnaidm_ has joined #zuul | 06:41 | |
| *** harlowja has quit IRC | 06:52 | |
| *** sshnaidm_ has quit IRC | 06:55 | |
| *** sshnaidm_ has joined #zuul | 07:10 | |
| Wei_Liu | SpamapS: hello, I have an issue to execute tox py27 job on one of our own project. tox -e py27 is working fine in local repo. but it failed in CI gate, the error message is "ImportError: Start directory is not importable: '/home/zuul/src/....../espresso/espresso/tests'", any idea? | 07:23 |
| *** weshay has quit IRC | 07:50 | |
| *** mhu has quit IRC | 07:50 | |
| *** pabelanger has quit IRC | 07:50 | |
| *** _ari__ has quit IRC | 07:51 | |
| *** myoung|off has quit IRC | 07:51 | |
| *** jpena|off is now known as jpena | 07:59 | |
| *** myoung has joined #zuul | 08:02 | |
| *** _ari_ has joined #zuul | 08:02 | |
| *** pabelanger has joined #zuul | 08:02 | |
| *** mhu has joined #zuul | 08:02 | |
| *** weshay has joined #zuul | 08:04 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 08:14 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 08:14 |
| *** sshnaidm_ is now known as sshnaidm | 08:19 | |
| *** threestrands has quit IRC | 08:22 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 08:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 08:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 08:28 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 08:29 |
| *** hashar has joined #zuul | 08:33 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 09:09 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Refactor status functions, add web endpoints, allow params https://review.openstack.org/536301 | 09:38 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 09:39 |
| *** bhavik1 has joined #zuul | 09:48 | |
| *** bhavik1 has quit IRC | 09:54 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Add tenant project group definition example and definition in the doc https://review.openstack.org/535730 | 10:34 |
| *** jimi|ansible has joined #zuul | 10:40 | |
| *** jimi|ansible has joined #zuul | 10:40 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 10:41 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] webapp: add optional admin endpoint https://review.openstack.org/536319 | 10:42 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 11:01 |
| openstackgerrit | Sam Betts proposed openstack-infra/zuul master: Ensure shlex.split takes comments into account https://review.openstack.org/541669 | 11:11 |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: Do not call merger:cat when all config items are excluded https://review.openstack.org/535509 | 11:12 |
| SpamapS | Wei_Liu: hm, no I've never seen that. | 11:17 |
| *** dmellado has quit IRC | 11:21 | |
| *** dmellado has joined #zuul | 11:27 | |
| *** JasonCL has quit IRC | 11:33 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: [WIP] webapp: add optional admin endpoint https://review.openstack.org/536319 | 11:41 |
| *** dmellado has quit IRC | 11:48 | |
| *** dmellado has joined #zuul | 11:59 | |
| *** elyezer has quit IRC | 12:17 | |
| *** elyezer has joined #zuul | 12:20 | |
| *** JasonCL has joined #zuul | 12:22 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** jpena|lunch is now known as jpena | 13:39 | |
| *** tobasco has joined #zuul | 14:12 | |
| tobasco | does zuul v3 still expose the ZUUL_* variables such as ZUUL_PROJECT? | 14:12 |
| tobasco | i.e would this work https://github.com/openstack/puppet-nova/blob/master/Gemfile#L4 | 14:13 |
| Shrews | tobasco: you should be able to access them via the zuul_legacy_vars ansible variable. Examples: http://codesearch.openstack.org/?q=zuul_legacy_vars&i=nope&files=&repos= | 14:23 |
| Shrews | but i wouldn't code anything new to use it | 14:24 |
| *** dkranz has joined #zuul | 14:27 | |
| *** elyezer has quit IRC | 14:28 | |
| *** elyezer has joined #zuul | 14:30 | |
| tobasco | Shrews: ok ty, then it isn't that atleast, thanks | 14:33 |
| AJaeger | tobasco: for new jobs: Don't use them - see what's configured instead. | 14:41 |
| *** hashar is now known as hasharAway | 14:41 | |
| tobasco | roger that | 14:46 |
| SpamapS | I believe the ansible equivalent of ZUU_PROJECT is {{ zuul.project.name }} | 15:02 |
| SpamapS | so.. just curious.. does anybody else use zuul as a test runner? I find myself just editting and committing and pushing until something passes/works/etc. | 15:03 |
| SpamapS | I only do local tests when I am at that early like "how does this even work" stage. | 15:04 |
| Shrews | SpamapS: For zuul dev, yes, b/c I had issues trying to get it all setup properly. For other projects, I *try* to remember to test locally first. I often fail at that though | 15:06 |
| SpamapS | I used to have more success on local tests | 15:08 |
| SpamapS | but then they gave me this stupid mac | 15:08 |
| kklimonda | I just use pycharm with remote python on my mac | 15:08 |
| SpamapS | and since now I have to do something in a VM anyway.. I just say screw that and let zuul do my VM management. ;) | 15:08 |
| SpamapS | yeah I *loathe* all remote editting and especially GUI editors. | 15:09 |
| SpamapS | :) | 15:09 |
| kklimonda | I just accepted the fact that I'll always have a VM with Linux running on my Mac to do work | 15:09 |
| SpamapS | I have that VM too.. but I find that it's enough trouble to interact with it that I just skip it and let zuul do my bidding in the cloud. | 15:09 |
| Shrews | SpamapS: oh yeah, when i was using a mac, i just gave up on local tests | 15:09 |
| Shrews | i mean, it was possible, just soooo much extra effort | 15:10 |
| SpamapS | yeah, there have been a few times where I need to get into the debugger for zuul | 15:10 |
| SpamapS | so then I go over to the linux VM and ttrun it up | 15:11 |
| SpamapS | but like right now I'm doing some integration tests for legacy puppet.. I want nothing to do with that crap.. let my disposable cloud VMs have it. | 15:11 |
| kklimonda | for one off changes sure | 15:11 |
| SpamapS | Eh.. I'm doing major refactors. :) | 15:11 |
| SpamapS | I admit, I have had to hold a node once because I was just at my wit's end at what was going on. | 15:12 |
| kklimonda | well, for major refactor, a cost of deploying a small "utility" VM you can run tests on is not that bad | 15:12 |
| SpamapS | (the python script we had building our hieradata would silently exit(0) if it didn't find templates it needed) | 15:12 |
| SpamapS | it's not so much cost, as focus | 15:12 |
| SpamapS | I will lose focus anyway | 15:12 |
| SpamapS | so I might as well just wrap it up in a zuul test, fire it off into the cloud, and switch focus for the next 10 minutes. | 15:13 |
| SpamapS | and then at the end, I usually have a nice test to keep | 15:13 |
| kklimonda | what I'd love to have is a good story for developing/testing zuul jobs locally | 15:14 |
| SpamapS | So I do that | 15:16 |
| SpamapS | I have a little script that feeds in a contrived zuul variable structure from zuul.yaml, and then I craft inventory as-needed | 15:16 |
| SpamapS | (and just calls ansible-playbook, I should say) | 15:17 |
| SpamapS | I also did at one time use zuul-bwrap to run it in a bwrap for better simulation | 15:17 |
| SpamapS | but I haven't tried that in a long time. | 15:17 |
| SpamapS | https://photos.app.goo.gl/VClcE6nENYzFsU1F2 | 15:18 |
| SpamapS | I tend to bomb the PR with stupid commits, and squish later... ;) | 15:19 |
| SpamapS | up to 28 now | 15:19 |
| kklimonda | I think everyone does that | 15:23 |
| kklimonda | sometimes I go one step further with commits like "fixing", "fix-up", "fix", "squash me".. | 15:23 |
| SpamapS | yep | 15:24 |
| SpamapS | this particular job is pretty annoying... | 15:25 |
| SpamapS | http://paste.openstack.org/show/664789/ | 15:26 |
| SpamapS | 5 nodes, 40+ required projects | 15:26 |
| SpamapS | takes a couple minutes to start up | 15:26 |
| kklimonda | testing puppet.. sigh :) | 15:30 |
| kklimonda | I've had some jobs take a couple of minutes to even process when I add many dependencies | 15:30 |
| kklimonda | such a build shows up in UI, but just sits there for a couple of minutes without even listing jobs to be ran | 15:31 |
| Shrews | mordred: so looking at 541434... not sure I get why you need to use socketserver there to handle the connection. Looks to me like you start one thread (LogStreamer), that will in turn only ever start one other thread (LogRecordSocketReceiver) to handle the one and only connection it will ever get for that job. | 15:32 |
| Shrews | mordred: still digging though, so maybe i've missed something obvious | 15:32 |
| Shrews | but seems excessive based on my current level of (mis)understanding | 15:33 |
| Shrews | that might, in fact, actually be spawning 3 threads to handle a single job stream. i forget if the main socketserver server starts its own thread too | 15:35 |
| *** bhavik1 has joined #zuul | 15:52 | |
| mordred | Shrews: it could handle multiple connections | 15:55 |
| mordred | Shrews: it's one server per run - but there are multiple playbooks and potentially multiple hosts per single playbook | 15:56 |
| mordred | Shrews: that said - I think we need to re-organize this anyway to happen in the connection plugin and the callback plugin ... | 15:57 |
| mordred | Shrews: because how it is in the patch now we'll lose context for which ansible inventory host is ... wait, nope - moving it won't fix that | 15:58 |
| corvus | i haven't reviewed yet (fires yesterday and all), so this is just abstract info: we should avoid adding more threads which scale with jobs -- it's okay to add single threads to the executor, but O(N) threads is bad and could cause us to get to the point where we're spending too much time in context-switching overhead. so if we need O(N) threads, we probably want to have a different process do that. | 16:06 |
| *** elyezer has quit IRC | 16:20 | |
| *** elyezer has joined #zuul | 16:24 | |
| *** bhavik1 has quit IRC | 16:29 | |
| *** jpena is now known as jpena|brb | 16:48 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 16:50 |
| clarkb | SpamapS: re local testing I do my best to force myself to do it as it seems to find all sorts of fun bugs that are useful to fix. Like the large logging based memory leaks that we had in the zuul test framework and more recently leaking files into /tmp and not working on btrfs /tmp. Its also not too bad to run the tests locally at least on linux. I just exclude all the sql tests because I don't want to run a | 16:53 |
| clarkb | mysql | 16:53 |
| clarkb | `tox -epy36 -- 'tests.unit.test_(?!connection).*'` is my command for that | 16:53 |
| mordred | corvus: yah - I think it'll wind up being better to do the log receiver in the playbook process instead of the executor - but I need to go spelunking to figure out the right way to stitch that in | 16:54 |
| mordred | corvus, Shrews so for now, I think reviewing that patch is about reviewing the conceptual plumbing from the command module across the wire to the receiver - but before we land it we should shift where the receiver runs | 16:55 |
| corvus | mordred: cool -- i just got to 541434 in the stack, is it worth me looking at now? should i do a detailed review? a 30-second review? | 16:55 |
| corvus | mordred: you just answered my q. thx :) | 16:55 |
| mordred | corvus: \o/ | 16:55 |
| mordred | corvus, Shrews: also - hopefully looking at that patch should provide context for us to discuss the mechanics of where/how the reciever should run | 16:56 |
| SpamapS | clarkb: no doubt there are things where a fire-and-forget doesn't work. But honestly, would still rather write python or ansible that ensures things are the way I expect, than peering into the top/du/ls/free abyss. | 17:00 |
| clarkb | oh definitely. The problem is you don't get any peering so it is easy to ignore/not know that you need to explicitly look | 17:00 |
| *** harlowja has joined #zuul | 17:11 | |
| corvus | mordred: i left a couple questions about things that weren't immediately apparent to me; but generally lg | 17:15 |
| *** jpena|brb is now known as jpena | 17:17 | |
| *** harlowja has quit IRC | 17:17 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 17:25 |
| tobiash | SpamapS: I'm running pycharm on mac with remote debugging in a docker container | 17:27 |
| tobiash | that took some time (a day) for setup but now I have it running together with tooling containers containing zk and mysql | 17:29 |
| tobiash | and a working debugger :) | 17:29 |
| tobiash | maybe I'm old fashioned, but I need this debugger ;) | 17:30 |
| tobiash | (and don't like working in a vm) | 17:30 |
| Shrews | mordred: yeah, i think down in the plugin is a better place | 17:41 |
| *** sshnaidm_ has joined #zuul | 17:54 | |
| Shrews | mordred: since you're probably the only other nodepool core that has experience with converting to devstack native jobs, mind reviewing https://review.openstack.org/535899 when you get the chance? | 17:57 |
| *** sshnaidm has quit IRC | 17:57 | |
| *** harlowja has joined #zuul | 18:11 | |
| *** harlowja has quit IRC | 18:18 | |
| *** sshnaidm_ is now known as sshnaidm|afk | 18:21 | |
| *** jpena is now known as jpena|off | 18:28 | |
| *** elyezer has quit IRC | 18:35 | |
| *** elyezer has joined #zuul | 18:38 | |
| mordred | Shrews: +A | 18:40 |
| *** elyezer has quit IRC | 18:45 | |
| dmsimard | corvus: replied on https://review.openstack.org/#/c/541452/, just want to confirm before sending a new patchset. | 18:46 |
| *** elyezer has joined #zuul | 18:46 | |
| corvus | dmsimard: i think your suggestion is fine too. | 18:47 |
| dmsimard | okay | 18:53 |
| dmsimard | thanks for catching it :) | 18:54 |
| openstackgerrit | David Moreau Simard proposed openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 18:54 |
| Shrews | tristanC: +1 on https://review.openstack.org/535563 but I don't see how that information could be useful (a count of node labels w/o any info as to node states). | 19:12 |
| mordred | Shrews: node-list can still be useful to provide dashboards for folks writing jobs to know what labels they can use | 19:27 |
| mordred | Shrews: I mean label-list | 19:28 |
| mordred | Shrews, tristanC: but I'd think that "Count" might want to be broken up in to "Total" and "Available" perhaps? | 19:29 |
| Shrews | mordred: ok, sure. but why count them? | 19:29 |
| mordred | Shrews: yah - the count without status seems less useful | 19:29 |
| Shrews | yeah, that's what i'm trying to say. the count holds no meaning as-is (at least to me) | 19:29 |
| mordred | yah. seems like having Label, and a count for each state would be th emost useful | 19:30 |
| Shrews | we don't limit by label, so an "Available" doesn't make sense either. but yeah, by state seems more sane | 19:31 |
| Shrews | s/sane/useful/ | 19:31 |
| mordred | ++ | 19:53 |
| openstackgerrit | Merged openstack-infra/nodepool master: Add /node-list to the webapp https://review.openstack.org/535562 | 19:53 |
| kklimonda | is there a technical reason for zuul not breaking dependency cycles? | 20:12 |
| kklimonda | one reason I can think of is it being rather hard to tell what the order of patches should be | 20:12 |
| clarkb | ya I don't think it can know what a valid order is if humans have asserted a circular order | 20:13 |
| openstackgerrit | Merged openstack-infra/zuul master: Add Executor Merger and Ansible execution statsd counters https://review.openstack.org/541452 | 20:14 |
| *** hasharAway has quit IRC | 20:15 | |
| tobiash | kklimonda: there was a discussion in that direction a year ago | 20:38 |
| kklimonda | tobiash: I saw a thread on ML, but it was mostly about non-technical reasons | 20:38 |
| kklimonda | while I subscribe to those reasons, the reality rears its ugly head and our developers just ask for force merge | 20:39 |
| clarkb | often the ask isnt to break the cycle but to somehow merge a cycle as if it were a single commit | 20:39 |
| clarkb | this is bad for continuius deployment | 20:39 |
| clarkb | kklimonda: we very rarely have issues with ut and I would argue our use is non trivial | 20:39 |
| kklimonda | wouldn't both reviews need their votes? | 20:39 |
| tobiash | one possible solution would be to merge such cycles into one buildset and have zuul to merge all related changes atomically | 20:39 |
| kklimonda | clarkb: openstack codebase has a much higher code quality than most | 20:40 |
| tobiash | but that would probably require zuul to push the merged changes instead of pressing the merge button in gerrit | 20:40 |
| clarkb | tobiash: ya | 20:41 |
| tobiash | I'm also bugged by this question every two weeks but until now I resisted successfully to open this can of worms :) | 20:41 |
| tobiash | we have the problem that there is some database containing api interface definitions (which we cannot really influence) and every few weeks there is a release of that (of course with plenty of incompatible changes) | 20:43 |
| kklimonda | well, we have had a single codebase, and now it's being split into multiple repositories | 20:44 |
| tobiash | and there are multiple projects updating together in an integration project | 20:44 |
| kklimonda | so I'm worried we'll be seeing more of those "cyclical reviews" during the transition | 20:45 |
| mordred | yah - it's a new mindset to get people into | 20:45 |
| mordred | "yes, you need to make three patches instead of one" | 20:45 |
| mordred | "add the new thing in project a, consume the new thing in project b, delete the old thing in project a" | 20:45 |
| mordred | and people will roll their eyes | 20:45 |
| mordred | but then, once they get used to it - they'll roll their eyes at anyone trying to do the otherthing :) | 20:46 |
| kklimonda | yes, and that gets harder when you have a C++ codebase that you're building with some -Werror's | 20:46 |
| tobiash | and if the C++ codebase is spread over 150 repos and needs to be built as it would be one repo :( | 20:47 |
| kklimonda | yep, maybe not 150 but 6 or so already makes it hard to do that properly | 20:48 |
| mordred | kklimonda: :) ... I miss -Werror ... before OpenStack I worked on Drizzle and we enforced -Werror -Weffc++ | 20:49 |
| tobiash | mordred: as everyone using c++ should :) | 20:50 |
| mordred | kklimonda: but I'd argue it's even more important in a complex C++ project like that, although also certainly more work - it makes each component treat all of their interfaces like fully external interfaces. it also become important to hide impl details behind smart pointers inside of a facade or something similar, so that ABI doesn't change on impl updates | 20:52 |
| mordred | kklimonda: that said - it has come up a few times as an idea to add support to zuul itself and then just not enable it for openstack (or other people who wish to not have it) | 20:52 |
| clarkb | too me that seems like a major indication you have too many repos | 20:53 |
| mordred | tobiash: yah. that's one of the reasons I like rust so much - it's like c++ with -Werror -Weffc++ on by default - and with a little syntax to make it harder to write things the 'wrong' way | 20:53 |
| clarkb | ignoring zuul and even CI if you must update all repos together maybe that should be one repo | 20:54 |
| tobiash | clarkb: that's what I always tell my people ;) | 20:54 |
| kklimonda | I mean, it's like listening to myself talking ;) | 20:54 |
| kklimonda | I agree on all counts, but I have to deal with the reality of having to support a codebase that is currently transitioning to separate repositories from a single codebase | 20:55 |
| tobiash | but they ignore me in that regards for some useless reasons and some valid reasons (like code ownership, legal stuff) | 20:55 |
| kklimonda | I can hope that at the end of this road we end up with clearly defined APIs and ABIs | 20:55 |
| kklimonda | but we have to get there first ;) | 20:55 |
| * mordred hands kklimonda a nice fluffy bunny rabbit to distract from the pain | 21:02 | |
| kklimonda | I should be able to special-case A->B->A at least, right? may have to do if we start getting more requests for force merging | 21:04 |
| corvus | notwithstanding all of that -- i think it would be okay for zuul to support simultaneous cross-repo-dependencies. we have thoughts about how to accomplish that, but it's not a priority for any of the folks working on zuul from openstack since we've decided we don't actually want to use the feature even if it exists. | 21:04 |
| kklimonda | are the thoughts written somewhere? | 21:05 |
| corvus | there are some pre-requisites for that, like having the zuul drivers (ie, gerrit and github) perform the actual final merges and pushes, rather than relying on the remote end (there are good reasons for this we can go into if needed) | 21:05 |
| kklimonda | mhm, that sounds like a great feature for zuulv4 ;) | 21:05 |
| corvus | if they are, i'm not sure, and they'd probably be out of date | 21:05 |
| corvus | if some folks wanted to work on it, i could work on writing a spec | 21:05 |
| tobiash | corvus: I fear this will land on my agenda within the next 1-2 years... | 21:06 |
| corvus | i think it's possible, but it's not trivial. anyone working on it would be a co-maintainer of the cross-repo-dependency system by the time they're done. :) | 21:06 |
| corvus | tobiash: yeah, we thought we really needed it, until we suddenly woke up one day and realized that it was the wrong thing for our development methodology, and how we expected people to deploy openstack. | 21:07 |
| corvus | there are really good reasons for wanting it, and really good reasons not to allow it :) | 21:08 |
| corvus | (so, any implementation will need to permit it to be disabled) | 21:08 |
| kklimonda | corvus: can you give a short brief why zuul should be the one merging the code, instead of relying on gerrit/github etc.? | 21:09 |
| corvus | i'll try to find some time after the ptg/v3 release to write up a spec and we can at least put that in the backlog for when someone has time | 21:09 |
| kklimonda | not saying I'll start writing this tomorrow, but I'd love to get a feeling for how much work a proper implementation is going to be | 21:10 |
| corvus | kklimonda: 1) so if the second merge fails, the first can be un-wound; 2) to reduce the likelihood of either one failing | 21:10 |
| corvus | (because we can't do a two-phase commit with git, we have to make sure the operation is reversible, and also very likely to succeed) | 21:10 |
| kklimonda | makes sense | 21:10 |
| tobiash | and the merge behavior of zuul and gerrit/github are only 98% identical as they use different algorithms, settings, libs | 21:11 |
| corvus | tobiash: yes, that too | 21:11 |
| corvus | so yeah, i think most of the implementation is that, plus updating zuul's internals so that a queue item can contain more than one change, and dealing with the graph | 21:12 |
| tobiash | corvus: not sure if that causes problems every now and then but I think the zuul pushes part could be even generally more stable | 21:13 |
| tobiash | (but currently with github as app possibly impossible due to an issue/missing feature in github branch protection) | 21:15 |
| corvus | tobiash: yes, i think once that's written, we would probably use that in openstack | 21:15 |
| corvus | tobiash: oh, what's missing? | 21:15 |
| tobiash | restricting pushes to an integration | 21:16 |
| corvus | (i know we did test that both gerrit and github will close the appropriate change/pull request when you push) | 21:16 |
| tobiash | you can restrict direct pushes to users but not integrations | 21:16 |
| corvus | tobiash: oh i see... | 21:16 |
| corvus | tobiash: do you know if that's been filed as an issue? that sounds generally useful... | 21:17 |
| tobiash | corvus: yes, there is a bug for that | 21:17 |
| * tobiash is looking for the link | 21:17 | |
| tobiash | corvus: there it is: https://platform.github.community/t/repositories-which-have-protected-branches-with-push-restrictions-have-no-ability-to-grant-push-rights-to-integrations/1376 | 21:18 |
| tobiash | it's open since a year | 21:19 |
| openstackgerrit | Merged openstack-infra/nodepool master: Convert from legacy to native devstack job https://review.openstack.org/535899 | 21:25 |
| mordred | tobiash, corvus: maybe jlk can get that fixed for us | 21:26 |
| jlk | ohai | 21:27 |
| jlk | yeah that's an ongoing debate | 21:27 |
| jlk | I haven't reached out internally to the folks on that issue yet. | 21:27 |
| jlk | I _suspect_ that it's tied up in some unreleased changes and so they can't really communicate about it in the public. | 21:27 |
| *** threestrands has joined #zuul | 21:39 | |
| corvus | tristanC: i'm experimenting with the static driver, and i get: | 21:41 |
| corvus | 2018-02-07 13:40:47,409 DEBUG nodepool.driver.static.StaticNodeRequestHandler[fuligin-12334-PoolWorker.local-main]: Declining node request 100-0000000008 because it would exceed quota | 21:41 |
| corvus | tristanC: that's with http://paste.openstack.org/show/665294/ | 21:42 |
| corvus | that's just the min-ready request | 21:42 |
| corvus | and there are no nodes | 21:43 |
| corvus | i'm confused :) | 21:43 |
| mordred | corvus: that's a cool config though | 21:44 |
| corvus | mordred: ain't it? i'm going through and trying to make sure things are simple for new users -- that's an ideal nodepool config for someone getting started | 21:45 |
| corvus | tristanC: oh i think i see the issue, it's due to the host key | 22:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 22:06 |
| corvus | mordred: copying over your logconfig stuff from zuul to nodepool ^ | 22:06 |
| corvus | tristanC: my issue was that i put the fingerprint in the host-key variable. that's obviously wrong, it *does* say "host-key" and not "host-key-fingerprint". :) though we may want to take this as UX feedback and consider whether we should use the fingerprint instead, or generally try to be smarter about that | 22:09 |
| corvus | (on my second attempt, i put in the entire host key, but that didn't work either because it had the comment, and nodepool doesn't want the comment. that's probably easy to fix too) | 22:09 |
| corvus | this was easy to diagnose though once i got the logging under control -- the error message made it very clear what the problem was | 22:10 |
| *** elyezer has quit IRC | 22:12 | |
| *** elyezer has joined #zuul | 22:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 22:39 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Add separate modules for management commands https://review.openstack.org/536303 | 23:02 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 23:02 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: webapp: add optional admin endpoint https://review.openstack.org/536319 | 23:05 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Make timeout value apply to entire job https://review.openstack.org/541485 | 23:25 |
| corvus | i don't understand this error: http://logs.openstack.org/30/541930/1/check/build-sphinx-docs/92baf7e/job-output.txt.gz#_2018-02-07_22_43_40_672819 | 23:25 |
| corvus | the docs don't build because nodepool-launcher --help exits with code 1 | 23:25 |
| corvus | but that command works fine for me locally, and 'tox -r -e docs' works as well | 23:26 |
| *** elyezer has quit IRC | 23:26 | |
| clarkb | corvus: the docs execute the cli help commands to populate portions of the docs iirc. I guess the sphinx plugin that does that doesn't like it when the command returns 1 | 23:26 |
| clarkb | I think it may be happening because the docs jobs no longer run tox | 23:26 |
| clarkb | so nodepool isn't installed. We may need to stop using that sphinx plugin | 23:26 |
| corvus | clarkb: we merged a change to nodepool recently though... | 23:27 |
| *** elyezer has joined #zuul | 23:27 | |
| clarkb | maybe the plugin updated to treat that as a failed build recently? | 23:28 |
| corvus | https://docs.openstack.org/infra/nodepool/operation.html#nodepool-launcher | 23:29 |
| corvus | worked the last time we landed a change | 23:29 |
| corvus | i mean, i want to assume that it's my change that broke it (no idea how, but it's not inconcievable) | 23:29 |
| corvus | i just don't know how to debug it | 23:29 |
| clarkb | ya I'm not sure if there is a suggested method to reproduce the doc builds locally now that they stopped using tox | 23:30 |
| clarkb | mordred: ^ | 23:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 23:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 23:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:30 |
| clarkb | http://logs.openstack.org/30/541930/1/check/build-sphinx-docs/92baf7e/ara/result/6f376940-c62f-4765-9146-88ec489af6ee/ ok so it does appear to install nodepool | 23:32 |
| clarkb | in that case maybe it is something in the change causing it to fail? | 23:32 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Noop https://review.openstack.org/541937 | 23:32 |
| mordred | clarkb: the suggested method ot reproduce doc builds locally is to either run 'sphinx-build -b html doc/source doc/build/html' or to set up a tox env called 'docs' that has that command in it | 23:33 |
| corvus | clarkb: sanity check ^ | 23:33 |
| * mordred looks at scrolback and builds | 23:33 | |
| mordred | corvus: I also do not understand that error - the logs look like all the appropriate things were done | 23:38 |
| corvus | i can't reproduce it with 'tox -r -e docs' or 'sphinx-build -b html doc/source doc/build/html' | 23:39 |
| corvus | my noop change works, so it's definitely my change which broke it | 23:40 |
| corvus | but i still have no idea how to figure out how | 23:40 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Update tox docs environment to match build-sphinx-docs https://review.openstack.org/541939 | 23:40 |
| mordred | corvus: wow. looking at your change it makes no sense that that would break in that way | 23:41 |
| clarkb | is it file permissions? | 23:42 |
| clarkb | eg is nodepool-launcher --help trying to setup logging and write to /var/log/nodepool? | 23:42 |
| clarkb | the default seems to be stdout though so I'm confused | 23:43 |
| clarkb | oh there is default server file handlers that may be used which wants /var/log/nodepool | 23:43 |
| corvus | clarkb: a local strace doesn't suggest anything like that, and there is no /var/log/nodepool on my workstation | 23:44 |
| corvus | do we need to hold a node? | 23:46 |
| clarkb | adding the nodaemon flag appears to be an easy way to test that theory but ya if there is no /var/log/nodepool on your workstation it seems unlikely | 23:46 |
| mordred | corvus: still poking | 23:46 |
| SpamapS | kklimonda: any chance you'll work on https://review.openstack.org/#/c/540035/ soon? I'm about to cherry pick it into our local zuul as our devs are stepping all over eachother with autoholds and need that. :-P | 23:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Add default logging configuration https://review.openstack.org/541921 | 23:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Launcher: exit immediately https://review.openstack.org/541930 | 23:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/nodepool master: Builder: exit immediately https://review.openstack.org/541935 | 23:49 |
| corvus | i set an autohold | 23:49 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Update tox docs environment to match build-sphinx-docs https://review.openstack.org/541939 | 23:56 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!