| jhesketh | corvus: Yep, +w'd | 00:00 |
|---|---|---|
| corvus | i'm noticing interesting behavior with openstack's zuul. it currently has a very large merge queue, and that seems to be having an effect on the executors | 00:47 |
| corvus | thing's i've noticed that we should look into later: | 00:47 |
| *** JasonCL has quit IRC | 00:48 | |
| corvus | * creating zuul references is slow and we should remove them. that would likely greatly reduce the impact of the large merge queue as working through it would be faster | 00:48 |
| corvus | * the merger built into the executor shares a lock with the executor, and new executed jobs can't start while a merge job is running. it seems that because lock allocation isn't always fair, the executor worker thread can get starved by the merger worker thread. | 00:49 |
| corvus | * there's a py3 bug with the geard 'workers' command | 00:50 |
| SpamapS | Indeed, I remember when we kept zuul refs and it was like "I hope we can remove these" | 00:54 |
| openstackgerrit | Merged openstack-infra/zuul master: Don't override allowed-projects https://review.openstack.org/538353 | 00:57 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix cross source dependencies for github https://review.openstack.org/537428 | 00:57 |
| *** JasonCL has joined #zuul | 01:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add memory awareness to system load governor https://review.openstack.org/539426 | 01:14 |
| *** JasonCL has quit IRC | 01:48 | |
| *** JasonCL has joined #zuul | 01:49 | |
| *** JasonCL has quit IRC | 01:53 | |
| openstackgerrit | Merged openstack-infra/nodepool master: nodeutils: use socket.getaddrinfo instead of ipaddress https://review.openstack.org/539086 | 02:05 |
| openstackgerrit | Merged openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 02:05 |
| *** threestrands has joined #zuul | 02:27 | |
| *** JasonCL has joined #zuul | 02:31 | |
| *** JasonCL has quit IRC | 02:32 | |
| *** JasonCL has joined #zuul | 02:32 | |
| *** JasonCL has quit IRC | 02:35 | |
| *** JasonCL has joined #zuul | 02:35 | |
| *** JasonCL has quit IRC | 02:37 | |
| *** rlandy|bbl is now known as rlandy | 02:58 | |
| openstackgerrit | Merged openstack-infra/zuul master: Remove python-keystoneclient from test-requirements https://review.openstack.org/536879 | 03:15 |
| openstackgerrit | Merged openstack-infra/zuul master: Remove pep8 and pyflakes from test-requirements https://review.openstack.org/536880 | 03:15 |
| *** JasonCL has joined #zuul | 03:19 | |
| *** JasonCL has quit IRC | 03:21 | |
| *** JasonCL has joined #zuul | 03:23 | |
| *** JasonCL has joined #zuul | 03:23 | |
| *** JasonCL has quit IRC | 03:24 | |
| *** JasonCL has joined #zuul | 03:25 | |
| *** JasonCL has quit IRC | 03:31 | |
| openstackgerrit | Merged openstack-infra/zuul master: Update docs to use sphinx-build https://review.openstack.org/536881 | 03:37 |
| *** rlandy has quit IRC | 04:10 | |
| *** harlowja has quit IRC | 04:12 | |
| *** harlowja has joined #zuul | 04:41 | |
| *** JasonCL has joined #zuul | 05:38 | |
| *** JasonCL has quit IRC | 05:42 | |
| *** threestrands has quit IRC | 05:48 | |
| *** harlowja has quit IRC | 06:12 | |
| *** bhavik1 has joined #zuul | 06:17 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 06:17 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 06:17 |
| *** bhavik1 has quit IRC | 06:20 | |
| tobiash | corvus: thanks, I know that the github (and especially apps) testing needs to be improved | 06:25 |
| tobiash | that's on my agenda as soon as I have time for this | 06:25 |
| tobiash | especially because we rely heavily on this | 06:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Don't gather host keys for non ssh connections https://review.openstack.org/535685 | 06:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 06:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix runtime stats reporting for noop job https://review.openstack.org/537432 | 07:08 |
| tobiash | corvus: commented on 539707 | 07:12 |
| *** sshnaidm|off is now known as sshnaidm | 07:21 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Zuul: Remove project name https://review.openstack.org/539853 | 08:15 |
| *** jpena|off is now known as jpena | 08:40 | |
| *** JasonCL has joined #zuul | 09:10 | |
| *** JasonCL has quit IRC | 09:11 | |
| *** JasonCL has joined #zuul | 09:16 | |
| *** JasonCL has quit IRC | 09:24 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 09:52 |
| *** JasonCL has joined #zuul | 10:06 | |
| *** JasonCL has quit IRC | 10:10 | |
| *** zaro_ has joined #zuul | 11:08 | |
| *** zaro has quit IRC | 11:11 | |
| *** bstinson has quit IRC | 11:11 | |
| *** pbrobinson has quit IRC | 11:11 | |
| *** ianw has quit IRC | 11:11 | |
| *** TheJulia has quit IRC | 11:11 | |
| *** sshnaidm is now known as sshnaidm|afk | 11:11 | |
| *** gothicmindfood has quit IRC | 11:12 | |
| *** gothicmindfood has joined #zuul | 11:12 | |
| *** ianw has joined #zuul | 11:12 | |
| *** TheJulia has joined #zuul | 11:12 | |
| *** pbrobinson has joined #zuul | 11:13 | |
| *** bstinson has joined #zuul | 11:14 | |
| *** JasonCL has joined #zuul | 11:20 | |
| tobiash | mhu: is it ok for you if I rebase https://review.openstack.org/#/c/535511 ? | 11:28 |
| tobiash | it has merge conflicts with current master | 11:28 |
| mhu | tobiash, you mean fbo :) | 11:28 |
| tobiash | mhu: ups sorry, fbo ^^ | 11:29 |
| rcarrillocruz | odyssey4me: https://github.com/ansible/ansible/pull/35602 | 11:30 |
| odyssey4me | orly? | 11:30 |
| * odyssey4me looks | 11:30 | |
| rcarrillocruz | mordred, Shrews : added you as reviewers, when you get a chance | 11:30 |
| rcarrillocruz | btw, i'm surprised there's no ssl on kazoo, | 11:31 |
| rcarrillocruz | i was assuming there was some suppor for client side cert | 11:31 |
| rcarrillocruz | but nothing ssl at all | 11:31 |
| rcarrillocruz | btw, i'm onto starting a nodepool ansible driver | 11:33 |
| rcarrillocruz | nothing has started it ? | 11:33 |
| rcarrillocruz | tristanC: you're the plugin machine | 11:33 |
| rcarrillocruz | i plan to get our vendors to use zuul as soon as we roll it out in prod (hint hint dmsimard|afk ) | 11:34 |
| rcarrillocruz | some use openstack | 11:34 |
| rcarrillocruz | some others physical devices | 11:35 |
| rcarrillocruz | others vmware | 11:35 |
| rcarrillocruz | etc | 11:35 |
| rcarrillocruz | so i rather have ansible provisioning those snowflakes | 11:35 |
| rcarrillocruz | odyssey4me: you mean inferring zookeeper envvar from plain nodepool.yaml right? | 11:36 |
| rcarrillocruz | erm, yaml var | 11:36 |
| rcarrillocruz | if so, that's good feedback, i'll take a look | 11:37 |
| odyssey4me | rcarrillocruz yep, basically if the nodepool config is on the same host, or even just the relevant subset of config that informs the zookeeper connection details, then why not make use of it | 11:37 |
| rcarrillocruz | ++ | 11:38 |
| tobiash | rcarrillocruz: feature/zuulv3 in zuul and nodepool had been merged into master so the correct doc url is now https://docs.openstack.org/infra/nodepool/ | 11:39 |
| AJaeger | tobiash: let's redirect from the old branch... | 11:44 |
| tobiash | AJaeger: good idea | 11:45 |
| rcarrillocruz | Oh neat | 11:46 |
| rcarrillocruz | Thx | 11:46 |
| electrofelix | At the PTG at the end of this month, are there any plans to discuss/hack on zuulv3 -> jenkins (https://etherpad.openstack.org/p/zuulv3-jenkins-integration)? mordred maybe this one is for you? | 11:51 |
| mordred | electrofelix: yes! I totally plan on discussing that with you :) | 11:59 |
| *** jpena is now known as jpena|lunch | 12:00 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Invalidate flavor and image cache on 400 errors https://review.openstack.org/441215 | 12:15 |
| AJaeger | tobiash: just pushed - https://review.openstack.org/539923 | 12:16 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Consume Task and TaskManager from shade https://review.openstack.org/414759 | 12:28 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove need for findNetwork https://review.openstack.org/398462 | 12:33 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul web: add admin endpoint, enqueue commands https://review.openstack.org/539004 | 12:40 |
| tobiash | yay, finally got approval for PTG :) | 12:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Run image object autocleanup after uploading images https://review.openstack.org/539933 | 12:43 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: zuul autohold: allow operator to specify nodes TTL https://review.openstack.org/539596 | 12:43 |
| *** jpena|lunch is now known as jpena|off | 12:56 | |
| *** Wei_Liu1 has joined #zuul | 13:04 | |
| *** Wei_Liu has quit IRC | 13:04 | |
| mordred | tobiash: \o/ | 13:05 |
| *** Wei_Liu1 is now known as Wei_Liu | 13:06 | |
| *** weshay|ruck is now known as weshay | 13:06 | |
| *** jpena|off is now known as jpena | 13:07 | |
| *** sshnaidm|afk is now known as sshnaidm | 13:10 | |
| Shrews | rcarrillocruz: i'm beginning to believe we have reached the point where we need to break the zk api out from nodepool as a separate library. I don't like that the module you've proposed needs to know zk paths, and others have expressed interest in it too | 13:24 |
| electrofelix | mordred: any days that will suit best, I may not be able to be present for the entire week, so trying to make sure I target the best days (which ever you are less busy for ;-) | 13:25 |
| Shrews | mhu: for 536295, i wasn't suggesting that zuul clean up the nodes. Nodepool should still do that. | 13:28 |
| Shrews | mhu: we should remove 'nodepool hold' totally | 13:29 |
| mhu | Shrews, aren't there cases where operators might want to hold a node manually? | 13:30 |
| Shrews | mhu: not via the nodepool command. zuul is totally in control of the node state when a build is happening. nodepool cannot intervene at that point | 13:31 |
| Shrews | that's why the hold option was moved to zuul | 13:32 |
| *** rlandy has joined #zuul | 13:37 | |
| Shrews | mhu: oh, i see the confusion. i made the comment on the wrong change. that comment was supposed to be on https://review.openstack.org/539493 | 13:37 |
| Shrews | not the parent | 13:37 |
| mhu | Shrews, makes much more sense to me now :) | 13:38 |
| odyssey4me | Shrews Would it perhaps make sense for that API to actually be a web service nodepool provides? Then zuul interacts via the web service, as can any other lib which uses nodepool. | 13:39 |
| Shrews | mhu: yeah sorry. yesterday was a crazy day for me | 13:39 |
| mhu | ok, in that case I can remove the CLI option, but keep the logic so that operators can specify a custom hold TTL when using autohold | 13:40 |
| odyssey4me | I ask because we're looking into doing a jenkins plugin for nodepool, and right now the zk model appears to be the only API - then there's a python lib in zuul which provides verbs. | 13:40 |
| mhu | Shrews, no worries, especially if it can be easily sorted by discussing | 13:40 |
| mordred | odyssey4me: yah - the zk api is the api for that because the requesting thing needs to be able to hold the lock on the node (which ensures if it goes away the lock is released and the resources freed) | 13:42 |
| Shrews | odyssey4me: i would think that would be quite difficult to do correctly because of the requirement to grab and hold zookeeper locks | 13:42 |
| mordred | Shrews: jinx | 13:43 |
| odyssey4me | ah, I guess I'm missing what makes that difficult - guess I need to look into how locks are held | 13:43 |
| mordred | Shrews: now, I *do* think we could expose a gRPC api if we wanted - since it's can be more stateful due to http/2 - but that would be a whole other conversation we'd need to have | 13:44 |
| Shrews | odyssey4me: zk locks are ephemeral and associated with a zk session. If that session goes away, the lock is released. Not very compatible with a REST API | 13:45 |
| odyssey4me | aha, so any implementation to integrate would have to do so directly to zookeeper, and be similar in implementation to the current python lib in zuul | 13:46 |
| mordred | yah | 13:46 |
| Shrews | odyssey4me: right | 13:46 |
| odyssey4me | ok - I'm not sure if there's anyone else looking into doing this, but we are and would love to collab on it. Our java skills are rusty - it's been 10+ years. :/ | 13:47 |
| mordred | odyssey4me: electrofelix and I are planning on sitting down at the PTG to talk jenkins integration - if I ever respond to him with when a good time is | 13:51 |
| odyssey4me | aha, ok - please let me know when that is as I'd like to represent our use-case, and if we've got any progress done then I'll let you guys know | 13:52 |
| * electrofelix waits in breathless anticipation... :p | 13:53 | |
| odyssey4me | I'm not personally doing the work, because my code sucks, but a colleague is. Unfortunately he won't be at the PTG. | 13:53 |
| Shrews | odyssey4me: i hear mordred LOVES writing java, fwiw | 13:53 |
| Shrews | too bad i know nothing about it | 13:53 |
| electrofelix | odyssey4me: I'm hoping to use a chunk of the time around the PTG to be hacking on some initial code either via groovy for a really dirty POC or maybe java | 13:54 |
| odyssey4me | electrofelix I'll try to get our guy to make himself available on IRC during the PTG to compare notes and work with you. | 13:54 |
| mordred | Shrews: speaking of java - the latest version of gerrit has a WIP state - and the docs for it give you a shout out | 13:55 |
| Shrews | mordred: eewwww... my shame is forever on the interwebs | 13:56 |
| *** dkranz has joined #zuul | 13:59 | |
| mordred | Shrews: http://bazaar.launchpad.net/~ndb-bindings/ndb-bindings/trunk/revision/400.1.1 | 14:00 |
| rcarrillocruz | Shrews: sure, given the interest in people using nodepool in non-zuul envs, it makes sense to decouple zk.py from zuul. Having said that, the chances that those paths will ever change are close to nil | 14:01 |
| mordred | Shrews: people may not fully realize or appreciate is just how long I've been abusing you in to shaming yourself on the internet | 14:01 |
| Shrews | that's not me. that's some other guy. i TOTALLY don't know java. nope. nuh uh. not me | 14:02 |
| Shrews | mordred: good grief, almost 10 years | 14:03 |
| mordred | Shrews: http://bazaar.launchpad.net/~ndb-bindings/ndb-bindings/trunk/view/400.1.1/java/com/mysql/cluster/ndbj/examples/JoinUsingScanAndIndexScan.java <-- at some point in time that made sense to us | 14:03 |
| rcarrillocruz | folks, has it been discussed the lack of SSL in kazoo? | 14:04 |
| rcarrillocruz | from what i see in docs, the only auth available is digest | 14:04 |
| rcarrillocruz | am I missing something | 14:04 |
| rcarrillocruz | ? | 14:04 |
| Shrews | rcarrillocruz: no, you are correct | 14:04 |
| Shrews | rcarrillocruz: though i did find a patch someone made for ssl support that didn't make it into the main trunk yet | 14:04 |
| rcarrillocruz | so, in environments where there are private and public nodepool launchers, I see difficult to operate it | 14:05 |
| rcarrillocruz | in my mind, i assume it would have client certs | 14:05 |
| rcarrillocruz | so | 14:05 |
| rcarrillocruz | a public ZK | 14:05 |
| mordred | https://github.com/aiven/kazoo/commit/faede3e6c0217155950ae264925b50340345480e | 14:05 |
| rcarrillocruz | a public provider for openstack | 14:05 |
| rcarrillocruz | erm launcher | 14:05 |
| rcarrillocruz | and a private launcher for esx | 14:05 |
| rcarrillocruz | that could work if end to end auth was in place | 14:06 |
| Shrews | rcarrillocruz: https://github.com/aiven/kazoo/commit/faede3e6c0217155950ae264925b50340345480e | 14:06 |
| Shrews | mordred: dammit | 14:06 |
| rcarrillocruz | as is right now, i can only think people putting zookeeper in the same nodepool host and/or in the same private lan | 14:06 |
| rcarrillocruz | is that accurate? | 14:06 |
| *** sshnaidm is now known as sshnaidm|mtg | 14:06 | |
| * Shrews apologizes for early morning publi cursing | 14:06 | |
| Shrews | rcarrillocruz: sounds about right | 14:07 |
| rcarrillocruz | :/ | 14:07 |
| * rcarrillocruz thinks gonna need VPN'ing if private launcher will be added to our setup | 14:07 | |
| mordred | I think we should work on getting that commit merged in to kazoo | 14:07 |
| rcarrillocruz | +10x | 14:07 |
| Shrews | rcarrillocruz: you could also try to convince harlowja to merge that commit | 14:07 |
| rcarrillocruz | will he be at PTG | 14:08 |
| rcarrillocruz | ? | 14:08 |
| rcarrillocruz | also, Shrews will you? haven't met you in confs in years | 14:08 |
| Shrews | rcarrillocruz: i won't be there this time | 14:08 |
| pabelanger | rcarrillocruz: yah, was thinking of trying at SSL support to zookeeper. lack of time so far | 14:08 |
| mordred | I made a PR for it | 14:09 |
| mordred | https://github.com/python-zk/kazoo/pull/493 | 14:09 |
| mordred | SpamapS: if you get bored, could you maybe point harlowja at that ^^ ? | 14:09 |
| pabelanger | oh, nice. houndci-bot is leaving per line commits for tox failures | 14:10 |
| pabelanger | comments* | 14:11 |
| mordred | one of us may want to take that commit and put it in to our own branch and submit it so that we can respond to review things | 14:12 |
| mordred | I can do it if nobody else wants to | 14:12 |
| *** jhesketh has quit IRC | 14:12 | |
| *** jhesketh has joined #zuul | 14:13 | |
| rcarrillocruz | other question folks: getting rid of zuul gearman, i assume the plan is to have a new path on zookeeper, just for jobs for executor to pull. Is that a 3.1 objective or further in development? | 14:14 |
| Shrews | rcarrillocruz: not a 3.0 thing for sure | 14:15 |
| mordred | rcarrillocruz: I think we'll start talking design for that once 3.0 is out the door | 14:15 |
| rcarrillocruz | k | 14:15 |
| Shrews | mordred: i can do that for us. i added ssl support to geard, so should be similar. would be nice to test it to see if it what is there actually works too | 14:19 |
| mordred | Shrews, SpamapS: https://github.com/python-zk/kazoo/pull/494 | 14:19 |
| Shrews | oh, well. nm :) | 14:19 |
| mordred | Shrews: oh - well, I went ahead and fixed the hound ci issues and made a new one | 14:19 |
| Shrews | \o/ | 14:19 |
| mordred | Shrews, pabelanger, rcarrillocruz: I added you all as collaborators with push access to emonty/kazoo - in case there is review feedback and one of you feels like responding to it | 14:23 |
| pabelanger | sure | 14:23 |
| rcarrillocruz | sure | 14:23 |
| pabelanger | we already have SSL keys intree for zuul (gearman) we could use that to setup SSL to zookeeper too for unit tests | 14:24 |
| *** swest has joined #zuul | 14:40 | |
| *** elyezer has joined #zuul | 14:45 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Include list of commits as Zuul variable https://review.openstack.org/539975 | 14:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix missing min_avail_mem in memory governor https://review.openstack.org/539976 | 14:51 |
| tobiash | corvus, pabelanger: this fixes the memory governer... ^ | 14:53 |
| *** swest has quit IRC | 15:01 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Add connection-port to provider diskimage https://review.openstack.org/535686 | 15:06 |
| tobiash | mordred: added the default port mapping ^ | 15:07 |
| *** swest has joined #zuul | 15:15 | |
| mordred | tobiash: both look great | 15:22 |
| tobiash | :) | 15:25 |
| corvus | tobiash: replied on 539707 | 15:29 |
| tobiash | corvus: I see | 15:30 |
| tobiash | +3 | 15:30 |
| pabelanger | corvus: left +2 on 538039, if you wanted to monitor merging. Otherwise, I can +3 too | 15:34 |
| *** myoung|off is now known as myoung | 15:47 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add available RAM to statsd https://review.openstack.org/539707 | 15:54 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 15:54 |
| *** hughsaunders has joined #zuul | 16:04 | |
| openstackgerrit | Merged openstack-infra/zuul master: Fix missing min_avail_mem in memory governor https://review.openstack.org/539976 | 16:05 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 16:10 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 16:20 |
| corvus | SpamapS, mordred: can you re-review https://review.openstack.org/539356 after a test fix ? | 16:27 |
| mordred | corvus: yup! | 16:27 |
| corvus | kklimonda: can you re-propose https://review.openstack.org/#/c/515169/ to the master branch please? | 16:29 |
| *** bhavik1 has joined #zuul | 16:30 | |
| corvus | mhu: ^ can you look over kklimonda's patch and see whether you should build your current work off of that, once it re-appears? | 16:30 |
| *** bhavik1 has quit IRC | 16:36 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zone-zuul-ci.org master: Add records for web site https://review.openstack.org/540029 | 16:36 |
| mhu | corvus, yep kklimonda's patch does the job better, hit me up when it's reproposed and I'll abandon mine | 16:40 |
| openstackgerrit | Merged openstack-infra/zuul master: Zuul: Remove project name https://review.openstack.org/539853 | 16:40 |
| kklimonda | corvus: thanks for reminding me - I'll reopen it for master shortly | 16:43 |
| kklimonda | well, re-propose | 16:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Merger: Stop storing zuul references https://review.openstack.org/540033 | 16:51 |
| mhu | kklimonda, please add me as a reviewer so I know to abandon mine | 16:52 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Allow specifying node's TTL when holding https://review.openstack.org/539493 | 16:52 |
| *** sshnaidm|mtg is now known as sshnaidm|rover | 16:53 | |
| Shrews | mhu: i'm confused as to the purpose of 539493 now? | 16:55 |
| mhu | Shrews, I removed the CLI part, left the logic so that it can be used with zuul | 16:58 |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Add records for web site https://review.openstack.org/540029 | 16:58 |
| Shrews | mhu: can you just combine that with the parent? the commit message makes no sense now | 16:58 |
| mhu | Shrews, sure | 16:59 |
| Shrews | mhu: thanks | 16:59 |
| openstackgerrit | Krzysztof Klimonda proposed openstack-infra/zuul master: Support autoholding nodes for specific changes/refs https://review.openstack.org/540035 | 17:01 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 17:05 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 17:06 |
| kklimonda | would plugging zuulv3 into multiple gerrit instances work as expected? | 17:08 |
| *** harlowja has joined #zuul | 17:09 | |
| pabelanger | it should, zuulv2 supported it | 17:11 |
| corvus | pabelanger: can you review https://review.openstack.org/535540 ? | 17:13 |
| pabelanger | sure | 17:14 |
| Shrews | corvus: We've discussed this long ago, but what are your thoughts on breaking the nodepool/zk.py file out into a separate python-zuullib project? We could even do it post-3.0 if you'd prefer, but we're starting to see folks try to code to ZK itself instead of the API we've developed for zuul-nodepool. | 17:23 |
| Shrews | For example, https://github.com/ansible/ansible/pull/35602 | 17:26 |
| corvus | Shrews: i think that's a good idea. i think there are some parts of the nodepool zk api that i'd like to discuss changing, but they are mostly related to images, which we can probably consider internal for a bit longer. so maybe, post-3.0, we can start on the library, just targeting requests and nodes. i think we're pretty close to considering that api somewhat stable. the main thing i'd worry about is | 17:26 |
| corvus | calcifying things before they are ready. so hopefully we can have a nice 0.X period with the library before we consider it very final. | 17:26 |
| Shrews | corvus: yeah, totally. We're going to need a period of probably not-trivial work to coalesce the duplicated code within nodepool and zuul, too. | 17:28 |
| corvus | Shrews, rcarrillocruz, odyssey4me: regarding that PR, i don't think we're at a point where we can promising internal API stability -- will you be able to iterate and keep up with any changes? | 17:28 |
| Shrews | I tend to like your model objects better in the way they handle attributes | 17:28 |
| rcarrillocruz | yeah, totally | 17:28 |
| corvus | Shrews: indeed. there are subtle but important differences there, so resolving those first gets us our first pass at abstraction. :) | 17:29 |
| corvus | or, i guess our second pass. but our first public pass. :) | 17:29 |
| rcarrillocruz | wow | 17:30 |
| rcarrillocruz | https://review.openstack.org/#/c/535551/1 | 17:30 |
| rcarrillocruz | didn't know the plugin driver autoload landed | 17:31 |
| pabelanger | rcarrillocruz: exciting right | 17:31 |
| rcarrillocruz | that's awesome | 17:31 |
| rcarrillocruz | like | 17:31 |
| rcarrillocruz | wow | 17:31 |
| Shrews | rcarrillocruz: so did static node support! | 17:31 |
| rcarrillocruz | geez, what a fest! | 17:31 |
| openstackgerrit | Merged openstack-infra/zuul master: requirements: remove paramiko <2.0 cap https://review.openstack.org/535540 | 17:32 |
| corvus | rcarrillocruz: to be clear, the plugin interface is not external yet -- that's just for internal drivers right now. | 17:41 |
| corvus | (but we'll stabilize it, and support external drivers eventually -- though, of course, we want a bunch of internal drivers anyway (aws, etc), so this lets us start doing that) | 17:42 |
| rcarrillocruz | ah right, that's why i'm surprised, i kind of remembered you -2 it and saying 'let's stabilize this before making it real' or along those liens | 17:42 |
| rcarrillocruz | right | 17:42 |
| corvus | we probably won't land more drivers before v3.0 release, but folks can start coding them and getting them ready so we can merge them afterwords | 17:43 |
| rcarrillocruz | yas! | 17:43 |
| rcarrillocruz | like, i'm just talking with other folks about writing a Cisco VIRL driver | 17:44 |
| rcarrillocruz | the way we test cisco today, i sideload VIRL VMs onto RDO | 17:44 |
| rcarrillocruz | but given that VIRL has autoupdate and other goodies, if i can just write a thing that talks natively to it the better | 17:44 |
| corvus | rcarrillocruz: oh that sounds cool | 17:46 |
| corvus | tobiash: thanks for the -1/abandon work! | 17:47 |
| corvus | i think we may have gotten through everything? | 17:47 |
| corvus | i think all of the non-WIP patches for zuul and nodepool may now be current | 17:47 |
| corvus | i think maybe i'll send an email to zuul-discuss about that, so that if folks have been only looking at the most recent changes, that now they can review all of the open changes | 17:49 |
| openstackgerrit | Merged openstack-infra/nodepool master: Remove the hold command https://review.openstack.org/540002 | 17:49 |
| corvus | https://zuul-ci.org/ exists | 17:54 |
| rcarrillocruz | WOOTZ! | 17:55 |
| rcarrillocruz | congrats folks, looking really great! | 17:56 |
| corvus | that's at http://git.openstack.org/cgit/openstack-infra/zuul-website/tree/ if anyone wants to start making it better (pls ;) | 17:56 |
| corvus | i'll send an email about that too | 17:56 |
| *** dkranz has quit IRC | 17:59 | |
| *** jpena is now known as jpena|off | 18:07 | |
| mordred | corvus: \o/ | 18:10 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Update page title https://review.openstack.org/540076 | 18:13 |
| pabelanger | +2 | 18:14 |
| *** dkranz has joined #zuul | 18:14 | |
| openstackgerrit | Merged openstack-infra/zuul master: Add noop job description https://review.openstack.org/535534 | 18:16 |
| openstackgerrit | Merged openstack-infra/zuul master: pep8: ignore E124 rule https://review.openstack.org/535536 | 18:16 |
| *** openstackgerrit has quit IRC | 18:18 | |
| *** hashar has joined #zuul | 18:20 | |
| *** hashar has quit IRC | 18:21 | |
| *** openstackgerrit has joined #zuul | 18:24 | |
| openstackgerrit | Merged openstack-infra/zuul master: scheduler: better handle format status error https://review.openstack.org/535539 | 18:24 |
| fungi | for some reason i can't seem to keep my editor from appending a stray carriage return to the footer line in the page index | 18:25 |
| corvus | fungi: http://logs.openstack.org/76/540076/1/check/zuul-website-build/86edd2b/html/ looks good :) | 18:25 |
| corvus | fungi: i think the whole thing may have crlf. maybe we should do a mass crlf->lf conversions | 18:26 |
| fungi | guessing emacs thinks html should have crlf eol markers | 18:26 |
| fungi | oh, or the arcana template came that way? | 18:26 |
| corvus | fungi: yeah, i think that may be it. and i didn't really notice because...emacs | 18:27 |
| fungi | i'll do a mass convert before i edit further and make that a separate change | 18:27 |
| tobiash | corvus: I did my best to judge correctly | 18:27 |
| fungi | i'll also fix the trailing whitespace and handful of mixed tab+space indents (there aren't many) | 18:28 |
| *** harlowja has quit IRC | 18:29 | |
| fungi | yeesh, some of the javascript mixes lines with crlf and lf too | 18:30 |
| fungi | this is a mess | 18:30 |
| corvus | yay there's stuff to do! | 18:30 |
| fungi | 10 files besides the index which have at least some crlf eol | 18:30 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Update page title https://review.openstack.org/540076 | 18:34 |
| corvus | all of the website build and publish jobs are zero-node jobs, btw (they only run on the executors). so updates should be relatively fast. the publish job for the title update is running now. | 18:41 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Convert Arcana files from DOS to UNIX line endings https://review.openstack.org/540082 | 18:41 |
| corvus | fungi: and your title change is live :) | 18:42 |
| fungi | i saw! | 18:42 |
| fungi | so fast | 18:42 |
| fungi | i rebased the line endings change on it | 18:43 |
| fungi | before pushing up, which is why it took a few extra minutes to redo some bits | 18:43 |
| corvus | it's only on a RW volume right now, so we'll slow it down by another 5min when we switch it to RO replicas | 18:43 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul-website master: Link page source from the index footer https://review.openstack.org/540083 | 18:44 |
| *** sshnaidm|rover is now known as sshnaidm|off | 18:59 | |
| *** sshnaidm|off has quit IRC | 19:05 | |
| *** harlowja has joined #zuul | 19:09 | |
| *** harlowja_ has joined #zuul | 19:11 | |
| *** harlowja has quit IRC | 19:14 | |
| *** rlandy is now known as rlandy|brb | 19:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't use run_lock in executor's merger https://review.openstack.org/540112 | 19:52 |
| corvus | that's an easy resolution to point #2 from yesterday's slowdown | 19:53 |
| *** rlandy|brb is now known as rlandy | 19:59 | |
| corvus | 20:02 < openstackgerrit> James E. Blair proposed openstack-infra/gear master: Make workers admin command py3 safe https://review.openstack.org/540116 | 20:02 |
| corvus | and that's point #3 | 20:02 |
| pabelanger | yay | 20:06 |
| *** dkranz has quit IRC | 20:22 | |
| openstackgerrit | Merged openstack-infra/zuul master: Merger: Stop storing zuul references https://review.openstack.org/540033 | 20:25 |
| openstackgerrit | Merged openstack-infra/zuul master: Merger: retry network operations https://review.openstack.org/539356 | 20:33 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not submit min-ready requests with min-ready 0 https://review.openstack.org/540133 | 20:45 |
| *** jimi|ansible has joined #zuul | 20:55 | |
| jimi|ansible | w00t! | 20:55 |
| jimi|ansible | i had no idea this channel existed before :) | 20:55 |
| Shrews | jimi|ansible: IT WAS A SEKRET! | 20:55 |
| Shrews | welcome | 20:56 |
| pabelanger | o/ | 20:57 |
| jimi|ansible | did the website just go up today? or has it been up for a while? | 20:58 |
| Shrews | jimi|ansible: today: http://lists.zuul-ci.org/pipermail/zuul-discuss/2018-February/000021.html | 21:00 |
| jimi|ansible | +1 | 21:01 |
| SpamapS | hrm.. one thing that stinks about using label instead of review for sending things into the gate on github is that labels aren't cleared on push | 21:06 |
| *** sshnaidm|off has joined #zuul | 21:09 | |
| *** jpena|off is now known as jpena | 21:18 | |
| corvus | SpamapS: what's the downside to using review? | 21:22 |
| *** dkranz has joined #zuul | 21:29 | |
| mordred | jimi|ansible: for your viewing pleasure - http://zuul.openstack.org/stream.html?uuid=2becb411740a474cb4ac243f7f10e889&logfile=console.log is the streaming output of a zuul job currently running on an ansible change | 21:30 |
| SpamapS | corvus: can't self Approve | 21:30 |
| jimi|ansible | mordred: yeah we were talking about it on slack :) | 21:31 |
| mordred | jimi|ansible: :) | 21:31 |
| SpamapS | corvus: have a lot of repos managed by teams of 2 / 3 people.. they sometimes can't wait for an approver. | 21:31 |
| rcarrillocruz | hah, inception ftw | 21:31 |
| SpamapS | I have thought about writing a dumb bot | 21:31 |
| mordred | jimi|ansible: bear in mind - it still may not pass - still debugging the job content a little bit ... but fingers crossed :) | 21:31 |
| jimi|ansible | is it still kind of hackish like when we talked in SF a while back? | 21:31 |
| SpamapS | that just takes things tagged 'self-approve' and reviews them as Approved | 21:32 |
| SpamapS | and then removes the self-approve tag. | 21:32 |
| mordred | SpamapS: why not just have your zuul clear approval tags in your check pipeline? | 21:32 |
| mordred | jimi|ansible: which thing? the streaming? or the running of things? the streaming is still a bit hackish, but I should hopefully have that fixed and no-longer hackish in the next few weeks | 21:33 |
| mordred | jimi|ansible: running things themselves is totally legit - we just keep finding fun corner-case assumptions- ansible using the 'devel' branch has helped us find a few interesting issues :) | 21:34 |
| jimi|ansible | streaming | 21:34 |
| SpamapS | mordred: I don't believe it has that ability (label isn't an action AFAICT) but that might be worth adding. | 21:35 |
| SpamapS | I may also have missed it | 21:35 |
| mordred | jimi|ansible: woot! it worked - and by 'it worked' I mean - it caught a bug in the PR! | 21:36 |
| mordred | jimi|ansible: http://logs.openstack.org/74/20974/86beef0891f5c2a7576ec79d26aa15c7ee29ed57/third-party-check/shade-ansible-devel-functional-devstack/2becb41/ara/result/2df44881-0b20-4d99-b0ff-0e904593439d/ | 21:36 |
| jimi|ansible | :D | 21:37 |
| mordred | SpamapS: https://docs.openstack.org/infra/zuul/admin/drivers/github.html#attr-pipeline.<reporter>.<github source>.unlabel | 21:39 |
| SpamapS | mordred: oh neat! | 21:43 |
| SpamapS | that would in fact work. | 21:43 |
| mordred | \o/ | 21:43 |
| SpamapS | Can't do it in check on success though.. since that would remove the label something needs to go into gate. But I _can_ remove it on fail | 21:44 |
| SpamapS | which should at least prevent some dumbness | 21:44 |
| SpamapS | I guess I could remove it on start | 21:46 |
| SpamapS | if I can somehow say "remove it on start caused by a new pull request only" | 21:47 |
| SpamapS | cause I don't want to remove it on recheck | 21:47 |
| corvus | SpamapS: you can make a pipeline just for that | 21:48 |
| mordred | corvus: the last ansible change properly reported failure to the PR ... but doesn't seem tobe showing up in the dashboard build history | 21:49 |
| mordred | corvus: I tried looking for mysql issues in the debug.log - but grepping for 'mysql' ... well, gets the phonebook due to jobs with mysql in the title | 21:50 |
| corvus | mordred: there should be a log line about reporting | 21:50 |
| mordred | corvus: any idea off the top of your head Ishould grep for - reporter perhaps? | 21:50 |
| corvus | mordred: yeah "Report" and the change id | 21:50 |
| SpamapS | corvus: ah good point.. a little "clean stuff up" pipeline. :) | 21:50 |
| SpamapS | I like it | 21:51 |
| SpamapS | for that matter I could also make a pipeline that informs users of the label when they submit reviews | 21:51 |
| SpamapS | so many ideas.. so little time | 21:52 |
| SpamapS | oh boy.. I am creating a job that has 44 required-projects. | 21:52 |
| *** jpena is now known as jpena|off | 21:52 | |
| SpamapS | may the git gods forgive my executors. | 21:52 |
| mordred | corvus: WELP - turns out it's an easy issue | 21:57 |
| *** threestrands has joined #zuul | 21:59 | |
| *** threestrands has quit IRC | 22:00 | |
| *** threestrands has joined #zuul | 22:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Executor: Don't start too many jobs at once https://review.openstack.org/540162 | 22:09 |
| corvus | tobiash, dmsimard|afk, pabelanger, clarkb, SpamapS: ^ i think that (or something like it) will help the governors work better (i think we need it, or something like it, for the current system, and any future system such as what tobiash is suggesting) | 22:13 |
| SpamapS | corvus: Indeed, +2'd already. I think we can get that responsiveness even better, but this is a good start. | 22:16 |
| openstackgerrit | Merged openstack-infra/nodepool master: Do not submit min-ready requests with min-ready 0 https://review.openstack.org/540133 | 22:19 |
| mordred | corvus: patch lgtm - I left the +A off in case you want anyone else to look at it | 22:25 |
| corvus | thanks, i'm still thinking about a test... and it probably needs a doc update too for the statsd addition. | 22:28 |
| corvus | hrm, do we have any tests for any of the governor functions? | 22:35 |
| SpamapS | I thought I had tests | 23:01 |
| SpamapS | but.. memory.. bad | 23:01 |
| corvus | i've got tests for ram and cpu written, working on the slow start test now | 23:14 |
| *** rlandy is now known as rlandy|biab | 23:26 | |
| corvus | SpamapS: do you happen to know what C gearmand does when a worker is sleeping, then registers for a job which is in the queue? does it do nothing (ie, wait for the worker to also send GRAB_JOB, or does it send NOOP to prompt the worker that it may be able to get a job which it just registered for?) | 23:33 |
| SpamapS | corvus: it should send a NOOP | 23:36 |
| SpamapS | I haven't confirmed that | 23:36 |
| SpamapS | but that is what I'd expect. | 23:36 |
| SpamapS | Because the whole "wake workers up" thing is an independent task that is triggered via a queue IIRC. | 23:36 |
| corvus | ok. we're going to need another gear release :) | 23:37 |
| SpamapS | Hrm here's a thing we might want to loosen on the file restrictions. I have a thing where I want to re-use the zuul inventory as-is, but zuul won't let me because it is outside the work dir. | 23:38 |
| SpamapS | corvus: :-P | 23:38 |
| corvus | we're not noticing this right now because on a not-busy system, the load isn't high enough for us to unregister execute:execute. and on a busy system, there's always a new job being added to cause a noop. | 23:39 |
| corvus | but as we make the executor unregister/register more often, it's more likely that we may have to wait a bit of extra time for a noop to come along. unless we fix geard to send noop after registration. | 23:40 |
| SpamapS | I also think it is perfectly valid for a sleeping worker to send GRAB_JOB before NOOP. | 23:40 |
| SpamapS | the sleep is just a signal "wake me up if you have work", but I don't think you have to wait for the NOOP at that point. | 23:40 |
| corvus | SpamapS: yeah, i may do that as an interim fix | 23:40 |
| corvus | since we can land that in the executor for now, then update geard, then remove it | 23:41 |
| SpamapS | right | 23:41 |
| corvus | SpamapS: i think the inventory thing is mainly so that jobs can't write to it. i think at this point, secrets are in their own pseudo-file, so that's probably not a concern. | 23:42 |
| corvus | SpamapS: maybe we could write an unused copy of the inventory in the work dir? | 23:43 |
| SpamapS | corvus: I'm looking at a simpler way to get at it, since it's already all in memory anyway. | 23:43 |
| SpamapS | and I actually may want to massage it. | 23:43 |
| corvus | SpamapS: ah yeah, maybe what you need is in hostvars | 23:43 |
| SpamapS | Basically I need to translate zuul's inventory into a kolla-ansible inventory. | 23:43 |
| SpamapS | Was just shortcutting | 23:44 |
| SpamapS | http://paste.openstack.org/show/658366/ | 23:44 |
| SpamapS | does not work ;) | 23:44 |
| clarkb | secrets are passed in on the command line iirc | 23:44 |
| SpamapS | Yeah secrets are on cli | 23:44 |
| SpamapS | but this is an untrusted context anyway | 23:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Executor: Don't start too many jobs at once https://review.openstack.org/540162 | 23:55 |
| corvus | that includes tests for all the governor stuff, and docs for the new statsd metric | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!