| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: gerrit: add support for report only connection https://review.openstack.org/568216 | 00:06 |
|---|---|---|
| *** threestrands has quit IRC | 01:00 | |
| jhesketh | corvus: the main reason I placed my WIP driver webhook patch where I did as I was playing around with an alternative to your patch with the intention of possibly squashing them together | 01:46 |
| jhesketh | the approach in 568028 felt hacky and not quite correct tbh | 01:47 |
| jhesketh | but I know it's because you want to get the cherrypy stuff in so if that's urgent enough to warrant this as an interim solution then sure | 01:48 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 03:08 |
| dmsimard|off | corvus: the mqtt reporter from tristanC has two +2s, does it need anything else ? I'd love to enable that to publish messages on firehose. | 03:34 |
| dmsimard|off | review is https://review.openstack.org/#/c/535543/ | 03:34 |
| *** pcaruana has joined #zuul | 04:31 | |
| SpamapS | Shrews: are you coming to YVR? I'd really like to find a quiet corner and show you how our nodepool gets locked up if you are. | 04:39 |
| SpamapS | corvus: or maybe you'd like to try your hand too? or really anybody | 04:39 |
| SpamapS | it just gets.. stuck. | 04:39 |
| SpamapS | and then unsticks itself | 04:39 |
| tristanC | SpamapS: perhaps your executor is too busy to complete node requests? | 04:53 |
| tobiash | SpamapS: I'll be there and happy to help | 05:12 |
| *** snapiri has joined #zuul | 05:58 | |
| SpamapS | tristanC: no this is all nodepool | 06:33 |
| SpamapS | It gets stuck with many requests and many nodes ready, and it does nothing | 06:33 |
| SpamapS | until it does.. many many minutes later | 06:33 |
| tristanC | but are there jobs running when this happen? | 06:36 |
| tristanC | or even merge job keeping the executor busy | 06:37 |
| SpamapS | tristanC: yes | 06:39 |
| SpamapS | a few running, and a few queued | 06:39 |
| SpamapS | tristanC: my load is so tiny | 06:40 |
| SpamapS | I run on a single 16GB VM. | 06:40 |
| SpamapS | with 8 vcpu's | 06:40 |
| SpamapS | The queued ones are the ones that are frustrating | 06:40 |
| tristanC | SpamapS: iirc, nodepool may appear to be stuck when it may be the executor simply too busy to accept node requests | 06:42 |
| tristanC | in statsd, you can verify that the executors queues are empty | 06:42 |
| SpamapS | I don't have statsd. | 06:42 |
| SpamapS | But it's a plausible theory | 06:43 |
| tristanC | especially if executor are doing merger jobs too | 06:44 |
| SpamapS | It's just not that busy | 06:47 |
| SpamapS | but yeah maybe things are queueing up in there | 06:47 |
| SpamapS | Been meaning to get a statsd going | 06:47 |
| tristanC | yeah, that would be useful :-) | 06:47 |
| *** sshnaidm has joined #zuul | 07:09 | |
| *** pcaruana has quit IRC | 07:17 | |
| *** pcaruana has joined #zuul | 07:17 | |
| *** pcaruana has quit IRC | 07:18 | |
| *** sshnaidm is now known as sshnaidm|rover | 07:21 | |
| *** pcaruana has joined #zuul | 07:23 | |
| tobiash | SpamapS: in order to check if tristanC is right you might also want to look for governor related messages in the zuul-executor as it throttles itself under high load or memory pressure | 07:26 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Support merged as requirement in github driver https://review.openstack.org/568488 | 07:39 |
| *** ssbarnea_ has joined #zuul | 08:34 | |
| *** xinliang has quit IRC | 09:51 | |
| *** xinliang has joined #zuul | 10:04 | |
| *** xinliang has joined #zuul | 10:04 | |
| *** rlandy has joined #zuul | 12:33 | |
| *** electrofelix has joined #zuul | 12:46 | |
| *** dkranz has joined #zuul | 12:52 | |
| *** jesusaur has quit IRC | 12:57 | |
| dmsimard|off | a question for ARA users - would you like it if the task tab defaulted to filtering out 'ok' and 'skipped' tasks ? So that only changed/failed/unreachable showed. This filter could be cleared -- it'd be just a default value for the search box, basically. | 13:03 |
| *** jesusaur has joined #zuul | 13:05 | |
| *** pwhalen has quit IRC | 13:41 | |
| *** pwhalen has joined #zuul | 13:43 | |
| *** pwhalen has joined #zuul | 13:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Replace use of aiohttp with cherrypy https://review.openstack.org/567959 | 13:56 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Convert streaming unit test to ws4py and remove aiohttp https://review.openstack.org/568335 | 13:56 |
| Shrews | rcarrillocruz: congrats on the new bundle of joy/poop! | 14:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Replace use of aiohttp with cherrypy https://review.openstack.org/567959 | 14:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Convert streaming unit test to ws4py and remove aiohttp https://review.openstack.org/568335 | 14:13 |
| *** gtema has joined #zuul | 14:20 | |
| rcarrillocruz | hehe, thx sir | 14:22 |
| *** pwhalen has quit IRC | 14:29 | |
| *** pwhalen has joined #zuul | 14:31 | |
| mordred | corvus: with the WIP off, I'm assuming those are ready for review now yeah? | 14:38 |
| corvus | mordred: i thought so, but there's still a non-trivial test failure; it'll need at least one more revision | 14:42 |
| mordred | corvus: kk. I left a note on the first patch for you | 14:45 |
| corvus | mordred: replied! | 14:48 |
| corvus | we should probably call that thing the DriverRegistry instead of ConnectionRegistry | 14:49 |
| mordred | corvus: nod. and yes- I thnk we should rename that -it's unsettling to read : | 14:51 |
| *** pwhalen has quit IRC | 14:56 | |
| *** pwhalen has joined #zuul | 14:58 | |
| *** sshnaidm|rover is now known as sshnaidm|bbl | 15:31 | |
| *** bhavik1 has joined #zuul | 15:43 | |
| *** snapiri has quit IRC | 15:43 | |
| Shrews | how interesting that someone implemented a kubernetes driver for nodepool and didn't bother to tell us :) | 15:58 |
| pabelanger | oh? Where did you see that | 16:04 |
| Shrews | pabelanger: https://www.openstack.org/summit/vancouver-2018/summit-schedule/events/21177/devops-implementation-for-openstack-on-kubernetes | 16:07 |
| SpamapS | tobiash: no governor firing. | 16:09 |
| SpamapS | Also I don't think the executor would be the issue.. the nodes aren't even claimed.. they're ready/unlocked ... so how would the executor cause that? | 16:09 |
| SpamapS | Also if i restart nodepool-launcher, the requests suddenly get satisfied. | 16:10 |
| pabelanger | Shrews: look at that, hopefully they decide to work upstream on the open spec | 16:11 |
| corvus | we can chat with them next week :) | 16:11 |
| fungi | huh... anybody (besides me) noticed yet that the release notes aren't in any sane order? https://zuul-ci.org/docs/zuul/releasenotes.html | 16:12 |
| fungi | wonder if that's some quirk of reno we're not aware of | 16:13 |
| pabelanger | corvus: ++ | 16:13 |
| corvus | fungi: it's fixed in reno, lets see if it's released | 16:13 |
| fungi | aha, glad to know i'm not the first to trip over it | 16:14 |
| corvus | fungi: released yesterday; so next doc build should fix it | 16:14 |
| fungi | excellent timing | 16:14 |
| *** gtema has quit IRC | 16:19 | |
| tobiash | SpamapS: that definitely sounds like nodepool is the problem | 16:20 |
| tobiash | Maybe the handlers are paused because of some reason | 16:21 |
| SpamapS | I do wonder if it's something weird like a deadlock that gets resolved by a timeout, quietly. | 16:21 |
| corvus | SpamapS: i'd add lots of debug statements, then upstream the ones used to solve the problem | 16:22 |
| SpamapS | corvus: yeah, that's what I hope to find some time to do. | 16:23 |
| SpamapS | Usually I just bounce nodepool launcher and move on with my day. | 16:23 |
| SpamapS | But I dislike the icky feeling of windows admin sickness it brings. ;) | 16:23 |
| *** pcaruana has quit IRC | 16:33 | |
| *** dkranz has quit IRC | 16:50 | |
| *** dkranz has joined #zuul | 16:56 | |
| *** acozine1 has joined #zuul | 17:01 | |
| *** dkranz has quit IRC | 17:04 | |
| *** yolanda_ has quit IRC | 17:09 | |
| *** bhavik1 has quit IRC | 17:09 | |
| *** yolanda_ has joined #zuul | 17:20 | |
| dmsimard|off | corvus: not sure if you saw my (admittedly) late ping last night about whether or not we could merge https://review.openstack.org/#/c/535543/ | 17:25 |
| tobiash | corvus: I've posted a question on 568028 | 17:38 |
| openstackgerrit | Merged openstack-infra/zuul-sphinx master: Add logo to docs https://review.openstack.org/566413 | 17:53 |
| openstackgerrit | Merged openstack-infra/zuul-website master: Clarify usage of Ansible for users https://review.openstack.org/567640 | 17:58 |
| *** gtema has joined #zuul | 18:05 | |
| openstackgerrit | Merged openstack-infra/zuul master: dont wait infinitely for the connection to zuul_console server https://review.openstack.org/567861 | 18:09 |
| *** dkranz has joined #zuul | 18:12 | |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zone-zuul-ci.org master: Add zuulci.org typo domain https://review.openstack.org/568661 | 18:21 |
| fungi | review topic:zuulci.org includes the related openstack-infra/system-config changes | 18:24 |
| *** elyezer has quit IRC | 18:38 | |
| *** elyezer has joined #zuul | 18:40 | |
| *** elyezer has quit IRC | 18:43 | |
| *** elyezer has joined #zuul | 18:47 | |
| *** dmsimard|off is now known as dmsimard | 18:58 | |
| *** acozine1 has quit IRC | 19:35 | |
| *** gtema has quit IRC | 19:37 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: WIP: Status branch protection checking for github https://review.openstack.org/535680 | 19:38 |
| corvus | dmsimard: i expect it's ready to go; i'm personally not keen on landing big changes this week. i'm swamped with summit prep. | 20:07 |
| dmsimard | corvus: fair, thanks | 20:08 |
| corvus | fungi: dns changes lgtm! | 20:12 |
| fungi | corvus: thanks, didn't know if you would be okay with adding more domains in the zones tree within that repo | 20:15 |
| corvus | fungi: yeah, i think that makes sense in this case | 20:15 |
| corvus | fungi: (though, if infra starts hosting non-zuul related domains, i think they should get their own repo(s)) | 20:16 |
| openstackgerrit | Merged openstack-infra/zone-zuul-ci.org master: Add zuulci.org typo domain https://review.openstack.org/568661 | 20:16 |
| fungi | it seemed like a reasonable compromise since the only domain we really care about there is the one for which the repo is named, and the other one is just a husk | 20:16 |
| corvus | fungi: i mean, apparently we thought ahead and named the directory "zones" :) | 20:18 |
| fungi | i saw! | 20:18 |
| *** dkranz has quit IRC | 20:23 | |
| *** sshnaidm|bbl is now known as sshnaidm|rover | 20:42 | |
| *** ssbarnea_ has quit IRC | 21:36 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Make imagesAvailable() part of the driver API https://review.openstack.org/568702 | 21:39 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Use ProviderConfig iface to validate labels https://review.openstack.org/568703 | 21:39 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Simplify driver API https://review.openstack.org/568704 | 21:39 |
| Shrews | that last review (704) doesn't pass tests just yet, but i'm pretty happy with the decoupling it does there | 21:39 |
| Shrews | corvus: ^^^ | 21:39 |
| Shrews | need to fix up the static driver poll() method before the tests will pass | 21:42 |
| *** dmsimard is now known as dmsimard|off | 21:43 | |
| Shrews | these things might also be release note worthy | 21:46 |
| * Shrews EODs | 21:46 | |
| *** dkranz has joined #zuul | 21:51 | |
| *** myoung|ruck is now known as myoung|ruck|afk | 21:53 | |
| mordred | corvus, Shrews: patch to openstacksdk that is fixing a problem that PROBABLY isn't affecting nodepool in any meaningful way - but does have an impact on yaml loading globally: https://review.openstack.org/568705 | 22:11 |
| *** pwhalen has quit IRC | 22:38 | |
| *** pwhalen has joined #zuul | 22:42 | |
| *** pwhalen has joined #zuul | 22:42 | |
| SpamapS | Shrews: corvus I think my nodepool issues may be related to I/O load on the VM where I have everthing running slowing down ZK. I'm going to move to a dedicated ZK. | 22:49 |
| *** ssbarnea_ has joined #zuul | 23:29 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!