| jamielennox | i'm wondering on the remaining value of keeping the connections something handled by zuul core vs an implementation detail of a driver | 00:00 |
|---|---|---|
| jamielennox | and whether that patch is a complete proposal or a step on the way to a larger refactor | 00:01 |
| jeblair | jamielennox: i'd say it's a structurally complete proposal -- the thing i'm hand-wavey on is whether the api should be duck-typed, or abstract base classes, or straight up inheritance... these things i expect us to settle on in a followup change. the thing i'd like to nail down here is the relationships between all those classes -- exactly the kind of question you asked. :) | 00:09 |
| jeblair | jamielennox: to the queston, i think there are a few things that lead me here.... | 00:10 |
| jeblair | 1) connections are defined in zuul.conf, so it seemed reasonable for them to be a first class object that the scheduler knows about (and creates/destroys as needed on reconfiguration) | 00:10 |
| jeblair | 2) the reporters/triggers/etc are likely to be created/destroyed far more often as pipeline configurations change, whereas connections are long lived, so keeping those separate made sense. | 00:12 |
| jamielennox | yea, connection reconfiguration i hadn't seen a way around yet, re zuul.conf that wouldn't seem to be a big problem for now because you have a 1-1 relationship for driver->connection with the same name so the transition case is easy there | 00:13 |
| jeblair | jamielennox: but i think i see what you're getting at... if we pass in zuul.conf to driver initialization, it can set up the connections on its own, and 'driver.getTrigger(pipeline_config)' would then have enough information to fetch the right connection from its own internal connection registry | 00:14 |
| jamielennox | right, and really reconfiguration of triggers/sources can be done by basically dropping all existing triggers/sources and asking for a new one from the driver with new config | 00:15 |
| jeblair | jamielennox: right | 00:16 |
| jamielennox | so looking at the connectionregistry we do: | 00:16 |
| jamielennox | # def getSource(self, connection_name): | 00:16 |
| jamielennox | # connection = self.connections[connection_name] | 00:16 |
| jamielennox | # return connection.driver.getSource(connection) | 00:16 |
| jamielennox | (which i commented to see where it was used) | 00:16 |
| jamielennox | which would seem to be a backwards relationship to me | 00:17 |
| jeblair | well, connectionregistry is awkwardly named at this point :) | 00:17 |
| jamielennox | it seems to mostly be config and connection_name is coming from layout.yaml/zuul.yaml | 00:17 |
| jamielennox | yea | 00:18 |
| jamielennox | but at this point do you think the name in zuul.yaml refers to a connection or a driver? | 00:18 |
| jeblair | oh but that's a good point i didn't think of | 00:18 |
| jeblair | it refers to a connection, unless the driver is connectionless, in which case it's the driver | 00:18 |
| jeblair | so 'openstack_gerrit' from the gerrit driver; 'zuul' from the (connectionless) zuul driver | 00:19 |
| jamielennox | so if a driver is named and has one getConnection method how does the driver name come to differ from the connection name? | 00:19 |
| jeblair | it zuul.conf you say "[connection openstack_gerrit]\n driver=gerrit" | 00:20 |
| jamielennox | oh, derp, gerrit as a driver can connect to multiple different gerrit instances with different names | 00:20 |
| jamielennox | right | 00:20 |
| jeblair | yeah, so having connections as something zuul knows about lets it look up the driver based on the connection specified in a trigger, in that code snippet you pasted | 00:21 |
| jamielennox | so why do we instantiate a driver as an object with different connections rather than create an Instance of the Gerrit driver per connection? | 00:22 |
| jamielennox | i guess that's harder to manage when doing a reconfigure | 00:22 |
| jeblair | that's to facilitate things like the zuul and timer triggers which maintain truly global state (other drivers might end up having some global state too) | 00:23 |
| jeblair | (example, the timer trigger wants to have a single apscheduler instance, even across tenants, that is never destroyed) | 00:24 |
| jeblair | in a nutshell, the approach gives us global state, connection state, and pipeline state | 00:25 |
| jamielennox | do tenants play into this? can a driver/connection exist within only a tenant? | 00:25 |
| jeblair | connections are currently global across all tenants | 00:26 |
| jeblair | continuing the example the timer driver is tenant aware and knows to remove all the timer events for a tenant if it is reconfigured | 00:27 |
| jamielennox | ok, that clears up a few things i was thinking about incorrectly - thanks | 00:29 |
| jamielennox | i'll keep having a poke through that review | 00:29 |
| jeblair | jamielennox: thanks! this was helpful :) | 00:30 |
| jamielennox | yea, i see what you mean, the inter-relationship between connections and triggers, sources is a bit funny now | 00:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add FakeNodepool test fixture https://review.openstack.org/413296 | 00:47 |
| jeblair | first zuul <-> nodepool interaction :) | 00:48 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add FakeNodepool test fixture https://review.openstack.org/413296 | 00:49 |
| *** harlowja has joined #zuul | 01:06 | |
| *** harlowja has quit IRC | 01:44 | |
| Shrews | jeblair: i'm feeling like maybe we should just have zuul import nodepool and use the zk.py from there | 03:41 |
| Shrews | maintaining that code in two places will quickly become tiresome | 03:42 |
| *** abregman has joined #zuul | 06:16 | |
| *** willthames has quit IRC | 07:33 | |
| *** openstackgerrit has quit IRC | 07:48 | |
| *** pleia2_ has joined #zuul | 08:09 | |
| *** pleia2 has quit IRC | 08:16 | |
| *** mmedvede has quit IRC | 08:16 | |
| *** abregman is now known as abregman|mtg | 08:20 | |
| *** mmedvede has joined #zuul | 08:23 | |
| *** abregman|mtg has quit IRC | 08:24 | |
| *** mmedvede has quit IRC | 08:50 | |
| *** mmedvede has joined #zuul | 09:16 | |
| *** abregman has joined #zuul | 09:27 | |
| *** herlo has quit IRC | 09:29 | |
| *** bhavik1 has joined #zuul | 09:35 | |
| *** bhavik1 has quit IRC | 09:39 | |
| *** abregman has quit IRC | 09:42 | |
| *** abregman has joined #zuul | 09:44 | |
| *** openstackgerrit has joined #zuul | 10:27 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add glean libs-from-git install https://review.openstack.org/413546 | 10:27 |
| *** abregman is now known as abregman|mtg | 10:57 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Add glean libs-from-git install https://review.openstack.org/413546 | 10:58 |
| *** persia has quit IRC | 11:07 | |
| *** persia has joined #zuul | 11:09 | |
| *** hashar has joined #zuul | 12:06 | |
| *** abregman|mtg is now known as abregman | 12:51 | |
| *** pleia2_ is now known as pleia2 | 13:54 | |
| *** abregman is now known as abregman|intervi | 13:57 | |
| *** abregman|intervi has quit IRC | 13:57 | |
| dmsimard | Hey zuul people, I'm going to be working on ARA during the holidays so I'm collecting wishlist/ideas/comments -- I've already started prototyping a UI refactor in case you want to have a look at where I might be going: https://www.youtube.com/watch?v=h3vY87_EWHw | 14:18 |
| *** abregman has joined #zuul | 15:19 | |
| *** herlo has joined #zuul | 15:22 | |
| *** herlo has joined #zuul | 15:22 | |
| *** jamielennox is now known as jamielennox|away | 15:46 | |
| *** bhavik1 has joined #zuul | 15:46 | |
| *** openstackgerrit_ has joined #zuul | 15:48 | |
| *** openstackgerrit_ has quit IRC | 15:49 | |
| *** jamielennox|away is now known as jamielennox | 15:54 | |
| *** openstack has quit IRC | 17:02 | |
| *** openstack has joined #zuul | 17:07 | |
| *** abregman has quit IRC | 17:25 | |
| *** bhavik1 has quit IRC | 17:50 | |
| *** hogepodge has quit IRC | 18:46 | |
| *** hogepodge has joined #zuul | 19:12 | |
| *** abregman has joined #zuul | 19:23 | |
| * SpamapS wishes the diff command had a '--ignore-timestamps' mode | 19:33 | |
| * SpamapS makes one with python and vimdiff | 19:43 | |
| SpamapS | adam_g: did I see you saying something about the conflict pipeline yesterday? | 20:00 |
| adam_g | SpamapS: maybe? | 20:00 |
| adam_g | SpamapS: one sec, distracted | 20:02 |
| SpamapS | adam_g: np | 20:02 |
| Shrews | jeblair: ah, kazoo has an API that automatically re-registers watches for you. seems to work nicely in simple testing | 20:02 |
| SpamapS | Indeed, I'd expect most zk libs to do that. | 20:04 |
| SpamapS | as long as you have record of users asking for something... keep doing it. | 20:04 |
| adam_g | SpamapS: theres https://review.openstack.org/409376 | 20:05 |
| adam_g | SpamapS: and https://review.openstack.org/411467 deals with conflicts as well | 20:06 |
| SpamapS | adam_g: k, I am working on a test that seems to depend on that stuff, so I may rebase on one or both of those. | 20:09 |
| adam_g | SpamapS: which one? | 20:11 |
| adam_g | SpamapS: those kind of put me down a rabbit hole i have yet to come out of | 20:12 |
| *** Shuo has joined #zuul | 20:12 | |
| SpamapS | adam_g: hah yeah | 20:16 |
| SpamapS | adam_g: test_dependent_behind_dequeue | 20:16 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul: Re-enable test_dependent_behind_dequeue https://review.openstack.org/413768 | 20:22 |
| SpamapS | adam_g: ^ fails because of missing jobs in conflict pipeline | 20:24 |
| adam_g | SpamapS: one min ill poke at it.i think in one of my patches i moved relevant config into a different config fixutre, and grouped merge-related tests into a class that uses that | 20:28 |
| SpamapS | adam_g: this one is just counting jobs that ran in the common single-tenant config .. and right now there are just missing pipelines. | 20:29 |
| SpamapS | adam_g: so if you're not going to add those back to the common config then I can just adjust my counts. :) | 20:29 |
| adam_g | SpamapS: huh, that one test you pushed seems to pass here? | 20:46 |
| adam_g | oh wait | 20:47 |
| adam_g | i ran the wrong one heh | 20:47 |
| adam_g | SpamapS: where does this test use the conflict pipeline you mentioned? | 20:51 |
| SpamapS | adam_g: when I run it in master, it runs some stuff in conflict | 21:05 |
| SpamapS | I think | 21:05 |
| adam_g | SpamapS: so that test shouldnt need any of the merge conflict fixes i was adding. 'A.fail_merge = True' is a different merge failure, just gerrit basically saying the thing failed to merge on that side | 21:05 |
| adam_g | SpamapS: i think the issue is, the v2 test was using tests/fixtures/layout.yml and the one you enabeld is using the single-tenant one, and the projects in there dont have the same jobs configured, so the counts will be off | 21:05 |
| adam_g | it looks like its doing the right thing otherwise, i think | 21:06 |
| SpamapS | adam_g: yeah, I suppose I could just add some of the tests back in. | 21:06 |
| adam_g | SpamapS: that may screw with counts on other tests tho | 21:07 |
| SpamapS | true | 21:07 |
| SpamapS | and agreed, it looks like it does the right thing in most places | 21:07 |
| adam_g | i just quickly added the missing jobs in but still get the ABORTED off by 3 (instead of 6), maybe i missed something | 21:07 |
| SpamapS | There's such a mountain of logs | 21:08 |
| SpamapS | I wrote some python one-liners and am using vimdiff to try and make sense of the changes | 21:08 |
| adam_g | i usually just redirect to a file somewhere if i dont have a breakpoint to stop it first | 21:10 |
| jeblair | Shrews: yeah, i recall we discussed possibly making it a library later, or just having zuul import nodepool.zk as you suggest might be an option. at the moment, i don't want to sweat it too much because i *think* there might be very little overlap. (it still might be a good idea, even in that case, but i think it lowers the urgency some) | 21:12 |
| jeblair | Shrews: re watches -- that's cool, i think we'll be in a position to try them out pretty soon in the zuul code | 21:13 |
| *** Shuo has quit IRC | 21:30 | |
| openstackgerrit | Merged openstack-infra/nodepool: Add glean libs-from-git install https://review.openstack.org/413546 | 21:35 |
| *** abregman has quit IRC | 21:43 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add FakeNodepool test fixture https://review.openstack.org/413296 | 22:04 |
| jeblair | i just discovered the zookeeper admin protocol -- telnet to the client port and type text, just like gearman :) | 22:10 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul: Add FakeNodepool test fixture https://review.openstack.org/413296 | 22:17 |
| *** vikk has joined #zuul | 22:46 | |
| *** vikk has left #zuul | 22:47 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool: Switch devstack job to Xenial builds https://review.openstack.org/413799 | 22:51 |
| *** saneax-_-|AFK is now known as saneax | 23:09 | |
| jamielennox | so in the 0.4 nodepool we removed the image-update command, is there an equivalent? | 23:16 |
| clarkb | jamielennox: image build and image upload I think | 23:20 |
| clarkb | update was a carry over from snapshot building that got mangled to deal with local builds | 23:21 |
| jamielennox | clarkb: there's not an image-upload i can see | 23:22 |
| jamielennox | basically i need a way to manually re-trigger an image-build/upload because the current image is bad | 23:22 |
| clarkb | indeed, I guess the idea is once an image is built it should be automatically uploaded? | 23:24 |
| clarkb | so I think you can image-delete any bad images | 23:24 |
| clarkb | and image-build new ones | 23:24 |
| clarkb | and then uploads will be autoperformed aggressively until successful | 23:24 |
| jamielennox | ok, well build is in progress so i'll check how uploads go | 23:28 |
| jamielennox | thanks | 23:28 |
| jamielennox | seems very agressively uploaded, thanks clarkb | 23:33 |
| pabelanger | I've just been using dib-image-delete ubuntu-xenial | 23:36 |
| pabelanger | that will delete all uploads, dib on disk, then kick off a new build and upload | 23:36 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool: [WIP] Reduce min-ram for ubuntu images for devstack testing https://review.openstack.org/413807 | 23:40 |
| SpamapS | adam_g: yes, I redirected to a file, but it was just too much to process with human power alone | 23:41 |
| SpamapS | adam_g: so I stripped the timestamps, and ran vimdiff. ANd it's actually _REALLY_ helpful. | 23:41 |
| *** willthames has joined #zuul | 23:45 | |
| *** hashar has quit IRC | 23:50 | |
Generated by irclog2html.py 2.14.0 by Marius Gedminas - find it at mg.pov.lt!