| mordred | jamielennox, clarkb: re: collapse/expand - intent is to be able to provide more than one view so that grep works well, but also so that 'normal' use can see 'important' bits easily - I'm very behind on actually writing up a quick doc on that | 00:51 |
|---|---|---|
| * mordred is doing talk in a few minutes, sorry missed the meeting | 00:52 | |
| * mordred has very long flight in a few hours - plans to write roll-out plan, build log and zuul-web thoughts that he owes everyone | 00:55 | |
| *** harlowja has quit IRC | 01:13 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Abstract Nodepool request handling code https://review.openstack.org/468748 | 02:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Abstract Nodepool provider management code https://review.openstack.org/468749 | 02:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Collect request handling implementation in an OpenStack driver https://review.openstack.org/468750 | 02:34 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Extend Nodepool configuration syntax to support multiple drivers https://review.openstack.org/468751 | 02:34 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 02:34 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 02:38 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Remove FakeProvider getClient monkey-patch https://review.openstack.org/475131 | 04:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Fix tenant include example https://review.openstack.org/486872 | 05:33 |
| *** amoralej|off is now known as amoralej | 06:55 | |
| *** isaacb has joined #zuul | 07:38 | |
| *** jhesketh has joined #zuul | 08:16 | |
| *** jhesketh_ has joined #zuul | 08:22 | |
| *** hashar has joined #zuul | 08:23 | |
| *** jhesketh has quit IRC | 08:25 | |
| *** jhesketh_ is now known as jhesketh | 08:30 | |
| *** jasondotstar has quit IRC | 08:32 | |
| *** jasondotstar has joined #zuul | 08:40 | |
| *** openstackgerrit has quit IRC | 10:17 | |
| *** jkilpatr has quit IRC | 10:29 | |
| *** jkilpatr has joined #zuul | 10:59 | |
| *** isaacb has quit IRC | 11:34 | |
| *** hashar is now known as hasharLunch | 11:44 | |
| *** isaacb has joined #zuul | 11:48 | |
| *** isaacb has quit IRC | 12:50 | |
| *** hasharLunch is now known as hashar | 12:53 | |
| *** isaacb has joined #zuul | 12:56 | |
| Shrews | tristanC: i'm confused as to why nodepool/builder.py is showing up in 468749 when there are no changes there | 13:00 |
| tristanC | Shrews: oh, it must be my emacs to set mode +x when file has a shebang | 13:01 |
| Shrews | tristanC: also, why s/getImage/labelReady/ ? I don't think that adds any clarity in terms of an openstack provider | 13:08 |
| Shrews | oh, it's part of the Provider api | 13:09 |
| Shrews | nm, i see now | 13:09 |
| Shrews | tristanC: I left what may be a confusing comment on 468749 regarding ProviderManagers. Hopefully I've had enough coffee that it makes sense. | 13:29 |
| Shrews | tristanC: tl;dr ProviderManager provides some protections against having more than 1 object for a provider (storing the objects within the config itself). I forget the historical reasons for that. Perhaps jeblair can remind us. | 13:30 |
| Shrews | i think it may have had something to do with resource allocation and a single view into that? | 13:31 |
| Shrews | tristanC: it's also possible i'm wrong and haven't seen something :) | 13:34 |
| *** amoralej is now known as amoralej|lunch | 13:34 | |
| Shrews | the logic is confusing | 13:35 |
| tristanC | Shrews: that code was indeed confusing, hence my attempt at naming things just 'Provider' and keeping the staticmethod only in the ProviderManager class. Though I could totally have missed the automatic config update thing you mentioned. I'll have a closer look tomorrow | 13:35 |
| Shrews | but would appreciate others helping decipher it to either confirm or deny it | 13:35 |
| tristanC | thanks for having a look! | 13:36 |
| Shrews | tristanC: sure. i'll continue looking at it. i'm not sure i'm correct that it is indeed broken, but i at least want to express my worry about it | 13:38 |
| Shrews | tristanC: oh, i think i was, indeed, incorrect. yay! | 13:47 |
| leifmadsen | emacs?! | 13:48 |
| leifmadsen | :) | 13:48 |
| tristanC | leifmadsen: with evil though, I switch back recently :) | 13:53 |
| *** hashar has quit IRC | 13:56 | |
| Shrews | jeblair: I've reviewed tristanC's nodepool driver abstraction down to (but not including) the static driver review. I need to reset my brain before digging into the static driver one. I hurt it a bit on the providermanager stuff. :) | 14:01 |
| Shrews | jeblair: i didn't want to +A any of them without you having a peek | 14:01 |
| jeblair | Shrews: thanks, i'll take a look today | 14:05 |
| *** hashar has joined #zuul | 14:26 | |
| *** openstackgerrit has joined #zuul | 14:27 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Allow loading additional variables file for site config https://review.openstack.org/447734 | 14:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement a static driver for Nodepool https://review.openstack.org/468624 | 14:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Extend Nodepool configuration syntax to support multiple drivers https://review.openstack.org/468751 | 14:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Collect request handling implementation in an OpenStack driver https://review.openstack.org/468750 | 14:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Abstract Nodepool provider management code https://review.openstack.org/468749 | 14:27 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Implement an OpenContainer driver https://review.openstack.org/468753 | 14:27 |
| *** amoralej|lunch is now known as amoralej | 14:32 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create tox_environment_defaults variable for tox based jobs https://review.openstack.org/486679 | 14:44 |
| pabelanger | tobiash: jeblair: I've removed tox_environment_defaults from the docs for now | 14:44 |
| leifmadsen | tristanC: :) | 14:45 |
| jeblair | pabelanger: ok. easy enough to add back later if we want. | 14:45 |
| pabelanger | jeblair: agree | 14:45 |
| *** isaacb has quit IRC | 14:52 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Move subunit processing into fetch-testr-output https://review.openstack.org/485840 | 15:00 |
| *** dkranz has joined #zuul | 15:05 | |
| jeblair | tristanC: ooh, i like the fake provider becoming a driver :) | 15:23 |
| *** hashar has quit IRC | 15:32 | |
| tristanC | jeblair: there is also https://review.openstack.org/468750 you may appreciate too :) | 15:36 |
| tristanC | then, regarding the static/oci driver, what's really missing is proper configuration management, where the config module needs to somehow delegate provider setting to the driver | 15:39 |
| tristanC | for example, right now there needs to be at least one cloud defined, otherwise the os_client_config.OpenStackConfig() call fails | 15:40 |
| tristanC | and similarly, labels <-> diskimage relation needs to removed | 15:40 |
| SpamapS | software is one of the few areas of my life where I like fakes :) | 15:46 |
| pabelanger | looking at flights out of Ottawa, thinking I might travel into PTG on saturday to avoid late night hacking, early sunday flight | 15:56 |
| pabelanger | still about 6.5 hours of travel for me | 15:57 |
| SpamapS | darn.. I was wrong.. executor-diskaccountant still has to be whitelisted I guess. :-P | 15:57 |
| * SpamapS wonders if the .stop() method should join() | 15:57 | |
| pabelanger | https://review.openstack.org/486228/ could use a +3, fixes tox logging on multinode jobs | 15:58 |
| pabelanger | https://review.openstack.org/#/q/topic:tox_environment_defaults also ready too | 15:58 |
| pabelanger | clarkb: ^ if you want to re-review | 15:58 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Support multi node jobs for tox logs https://review.openstack.org/486228 | 16:00 |
| *** dkranz has quit IRC | 16:07 | |
| *** dkranz_ has joined #zuul | 16:07 | |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 16:15 |
| SpamapS | jeblair: ^ This approach to shutting down threads might be viable for some of the other whitelists. | 16:19 |
| SpamapS | especially watchdog | 16:19 |
| jeblair | SpamapS: *nod* | 16:20 |
| SpamapS | derp | 16:23 |
| SpamapS | I need a git review pre-hook to run pep8 | 16:23 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 16:24 |
| SpamapS | hmmm | 16:24 |
| SpamapS | the du threads spit a lot onto stderr | 16:24 |
| SpamapS | I think we might want to devnull it | 16:24 |
| SpamapS | 2017-07-25 16:20:54.797845 | ubuntu-xenial | du: cannot access '/tmp/tmptjqhd1h_/zuul-test/18d6989c61654634ae0b935043987bd8/trusted/project_0/github.com/common-config/.git/packed-refs.lock': No such file or directory | 16:25 |
| SpamapS | yeah | 16:25 |
| jeblair | SpamapS: yeah, that's pretty boring :) | 16:25 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 16:26 |
| SpamapS | I can't imagine we'd ever really care about the stderrs.. until we have a stuck du or something, then we'll be enraged | 16:26 |
| SpamapS | but look at that, py3 has a new shiny: subprocess.DEVNULL | 16:26 |
| jeblair | that is shiny | 16:26 |
| SpamapS | dear py3: I don't know how we ever lived without you. Don't tell py2, but... I think you're the only one for me now. | 16:27 |
| * Shrews feels uncomfortable | 16:29 | |
| SpamapS | Shrews: you and py3 have been hanging out doing async things... don't judge. | 16:34 |
| SpamapS | btw, the web streaming is so nice to have | 16:35 |
| SpamapS | like really | 16:35 |
| SpamapS | amazing job everyone who worked on it | 16:35 |
| Shrews | definitely a good group effort on that | 16:36 |
| SpamapS | 2017-07-25 16:36:31.920904 | ubuntu-xenial | PASSED (id=0, skips=7) | 16:37 |
| SpamapS | we should probably circle back on those 7 skips now | 16:37 |
| SpamapS | IIRC they're all complicated | 16:37 |
| SpamapS | jeblair: I think I"m going to declare 2000773 dead (since it's almost unusable from storyboard) and we should make a story for every skip left. | 16:38 |
| SpamapS | like it basically crashes my browser | 16:38 |
| jeblair | Shrews, tristanC: i +3 all the changes that Shrews has +2d, which are the refactor changes. i left a -1 on the static node driver change. | 16:38 |
| jeblair | SpamapS: no objection -- though what about 1 more story with 7 tasks? | 16:39 |
| SpamapS | jeblair: Probably easier to manage that way. +1 | 16:40 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Abstract Nodepool request handling code https://review.openstack.org/468748 | 16:41 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Abstract Nodepool provider management code https://review.openstack.org/468749 | 16:41 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Collect request handling implementation in an OpenStack driver https://review.openstack.org/468750 | 16:41 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Extend Nodepool configuration syntax to support multiple drivers https://review.openstack.org/468751 | 16:42 |
| jeblair | SpamapS: the default disk limit is half the size of the nova repo :) | 16:53 |
| SpamapS | jeblair: I figured that was an overly conservative limit :) | 16:53 |
| jeblair | of course, most of the git repos should be hard links, so they kinda sorta don't count. but i don't know if there's an easy way for us to exclude those. | 16:54 |
| SpamapS | du will count a hard link against a dir | 16:54 |
| SpamapS | but only once IIRC | 16:55 |
| jeblair | SpamapS: heh, so the first nova job loses | 16:56 |
| jeblair | $ du -h --max-depth=1 . | 16:56 |
| jeblair | 199M./nova2 | 16:56 |
| jeblair | 36M./nova3 | 16:56 |
| jeblair | 235M. | 16:56 |
| jeblair | nova2 and nova3 are both hard-link clones of ../nova | 16:56 |
| SpamapS | http://paste.openstack.org/show/616469 | 16:56 |
| jeblair | so basically, nova2 gets the full accounting, and nova3 gets only the non-hardlink stuff | 16:56 |
| SpamapS | yeah | 16:57 |
| SpamapS | that's interesting | 16:57 |
| SpamapS | we can do -l | 16:57 |
| SpamapS | which counts them every time | 16:57 |
| jeblair | adding "-l" will do the full accounting for both | 16:57 |
| jeblair | which is at least consistent, but if i had a choice, i'd love the opposite -- get 36M for both. | 16:57 |
| SpamapS | probably more fair this way | 16:57 |
| SpamapS | rather | 16:57 |
| SpamapS | probably more fair with -l | 16:57 |
| jeblair | SpamapS: oh! idea! | 16:58 |
| jeblair | SpamapS: what if we du'd the merger cache first, then the jobdir? | 16:58 |
| SpamapS | yeah we can probably do that | 16:58 |
| SpamapS | du takes a list | 16:58 |
| jeblair | SpamapS: we'll ignore the merger cache output, but du will have assigned all the hardlink sizes to it | 16:58 |
| SpamapS | http://paste.openstack.org/show/616470 | 16:59 |
| SpamapS | works | 16:59 |
| jeblair | SpamapS: yep. i got the same with my local test too | 16:59 |
| SpamapS | so we can just make DiskAccountant accept a list of cache dirs to du first and throw away | 16:59 |
| jeblair | SpamapS: yeah. we only need to provide one though -- ExecutorServer.merge_root | 17:00 |
| SpamapS | this thing we're doing btw, it's called "The opposite of how systemd happened" | 17:00 |
| SpamapS | jeblair: k | 17:01 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix tenant include example https://review.openstack.org/486872 | 17:10 |
| SpamapS | jeblair: mmm this has been a rather fun coding exercise. | 17:23 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 17:23 |
| jeblair | SpamapS: left 2 questions on there. one important. :) | 17:33 |
| pabelanger | I cannot remember, did we say we wanted the 'Zero tests were run. At least one test should have been run; exit 1' logic in our zuul-jobs tox role? | 17:34 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Simplify _deleteLocalBuild parameters https://review.openstack.org/484413 | 17:36 |
| SpamapS | jeblair: you're right on your question. Forgot that max-depth=1 means we'll get everything under it too. That said, I think it would just try to cancel jobs that are unique==reponame .. so just a waste of time. ;) | 17:36 |
| jeblair | pabelanger: i think if we are sure we're running unit tests, yes? so only for py27 py35 jobs | 17:36 |
| SpamapS | jeblair: also do you think maybe we should still raise the default higher than 100MB? | 17:37 |
| jeblair | SpamapS: yeah, probably enough time+error messages to make it worth special casing :) | 17:37 |
| pabelanger | jeblair: what is our story with unittest playbooks? When would not use them for tox? | 17:38 |
| jeblair | SpamapS: i'm torn on the default. now that we have hardlinks out, it seems a little more reasonable, but i bet any serious use is going to need to be a lot higher. maybe 250 or so? | 17:38 |
| pabelanger | I don't think I understand there purpose | 17:38 |
| jeblair | pabelanger: i don't think i understand the question | 17:39 |
| pabelanger | jeblair: why to we have unittest playbooks today? What would go into that over the tox playbooks? | 17:39 |
| jeblair | pabelanger: i have no idea? | 17:41 |
| pabelanger | okay, will follow up with mordred on that | 17:41 |
| jeblair | pabelanger: tox can be used for running things other than unit tests | 17:41 |
| jeblair | pabelanger: and, of course, unit tests can be run without otx | 17:41 |
| jeblair | tox | 17:41 |
| pabelanger | jeblair: right, today our tox job has a parent of unittests, so it is currently a hard dependency | 17:42 |
| pabelanger | atm revoke-sudo and test-setup are within unittest, so that makes a little sense | 17:43 |
| jeblair | pabelanger: it seems like putting the zero-tests check in unittests/post is a good place, yeah? | 17:43 |
| pabelanger | jeblair: ya, that's what I am thinking too | 17:44 |
| jeblair | pabelanger: there's already fetch-testr output there. that seems similar. | 17:44 |
| pabelanger | agree | 17:44 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 17:44 |
| SpamapS | jeblair: I like 250.. let me real quick raise that | 17:45 |
| openstackgerrit | Clint 'SpamapS' Byrum proposed openstack-infra/zuul feature/zuulv3: Monitor job root and kill over limit jobs https://review.openstack.org/485902 | 17:45 |
| SpamapS | That means 1GB per 4 jobs.. and I kind of expect we'll have executors running in the "tens of jobs" area | 17:45 |
| jeblair | SpamapS: we think our v2.5 capacity is 150 jobs per launcher | 17:46 |
| jeblair | on 8gb vms | 17:47 |
| jeblair | SpamapS: those come with 80G ephemeral drives (which we're not actually using right now, but could). if the numbers hold, that gives us > 500MB per job. of course, we should be able to oversubscribe. | 17:49 |
| SpamapS | Yeah should be oversubscribing a lot really. | 17:50 |
| jeblair | i very much doubt we're going to be able to run *more* jobs on an executor in v3 than v2.5, so these seem like a fairly good match. | 17:50 |
| SpamapS | Since it would be relatively hard to maliciously overload the disk, without also just overloading the entire system itself. | 17:51 |
| SpamapS | jeblair: is there nothing in v2.5 preventing launcher overload btw? | 17:52 |
| jeblair | pabelanger, clarkb: ^ we should probably mount the ephemeral disks on our ze servers and put the merger and job root there. :) | 17:52 |
| SpamapS | like, if there are jobs to run, the launchers just keep taking them, right? | 17:53 |
| jeblair | SpamapS: nope. it's uh, part of our experimental protocol for determining launcher capacity. :) | 17:53 |
| pabelanger | jeblair: clarkb: Agree, I missed that step when we launched ze01.o.o | 17:53 |
| SpamapS | jeblair: a bold experiment | 17:53 |
| jeblair | pabelanger: well, it wasn't important for v2.5, and it won't really be important for v3 until we turn on the firehose. :) | 17:53 |
| clarkb | yup nothing in the job roots should need to persist so the ephemeral disk should be fine | 17:54 |
| *** harlowja has joined #zuul | 17:55 | |
| *** robled has joined #zuul | 18:07 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change jobroot_dir to job_dir in executor config https://review.openstack.org/487165 | 18:10 |
| jeblair | Shrews, mordred: ^ it's *that* bikeshed again :) | 18:11 |
| jeblair | i tried to write the docs for that, and that's the only thing that looked right to me | 18:11 |
| jeblair | (aside from changing everything else to match (ie, "merge_root, jobdir_root, state_root", etc.)) | 18:12 |
| jeblair | SpamapS, pabelanger: ^ figured we should document this little nugget of info :) | 18:12 |
| Shrews | jeblair: awesome. i look forward to changing it again soon! | 18:13 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Change jobroot_dir to job_dir in executor config https://review.openstack.org/487165 | 18:13 |
| jeblair | Shrews: i just know the perfect shade of fuscia is out there | 18:14 |
| Shrews | i love that there is also a jobdir_root | 18:16 |
| Shrews | (just for the little extra splash of color) | 18:16 |
| jeblair | yeah, i still think our sanity is secondary to consistency in the config file. so most users don't need to know there's something called "jobdir_root" in zuul internals. | 18:18 |
| jeblair | but that's because i consider my own sanity a lost cause. | 18:18 |
| Shrews | jeblair: i suppose something like jobbase_dir isn't any better | 18:22 |
| mordred | jeblair: wow.I'm not sure there IS the perfect shade of fuscia :) | 18:31 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add zuulv3 jobs for nodepool https://review.openstack.org/487169 | 18:34 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Jobs should default to voting https://review.openstack.org/487172 | 18:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Make tox-cover job non-voting https://review.openstack.org/487173 | 18:39 |
| pabelanger | jeblair: ^ what is the process of creating a variant job that is non-voting? I assume ^ is invalid syntax since I didn't parent tox-cover | 18:41 |
| pabelanger | jeblair: that would mean, the job would need to be called tox-cover-nv, right? | 18:41 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-cover-nv job https://review.openstack.org/487173 | 18:43 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-cover-nv job https://review.openstack.org/487173 | 18:45 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Make tox-cover job non-voting https://review.openstack.org/487173 | 18:46 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Create tox-cover-nv job https://review.openstack.org/487173 | 18:47 |
| *** amoralej is now known as amoralej|off | 18:53 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Ensure tox-cover is non-voting https://review.openstack.org/487173 | 18:54 |
| leifmadsen | when the documentation explains the zuul-executor, it says it reads the configuration from the git repository, but it's not really specified what git repository, and how to configure which one to read from | 18:54 |
| leifmadsen | any tips? (I'm updating documentation while going through a setup) | 18:54 |
| leifmadsen | oh man, maybe I just need to read a little more carefully, that's what `main.yml` provides I guess | 18:55 |
| leifmadsen | I'll try and clean it up to make it slightly more clear | 18:55 |
| mordred | leifmadsen: yah - main.yml tells the scheduler which repositories to manage | 18:55 |
| mordred | leifmadsen: after that point, all config is read from those repositories | 18:56 |
| pabelanger | jeblair: okay, I figured it out. Had to re-read zuulv3 spec again | 18:57 |
| mordred | leifmadsen: (don't know if you've seen our config in case looking at a working example would be helpful for clarification - and thanks for working on docs updates!) | 18:57 |
| leifmadsen | mordred: link to the working repo would be useful I think | 18:58 |
| mordred | leifmadsen: http://git.openstack.org/cgit/openstack-infra/project-config/tree/zuul/main.yaml is our main.yaml - and obviously from there the zuul.yaml and .zuul.yaml filesin the repos is where the fun is | 18:58 |
| leifmadsen | trying to attack it from a "I have no fucking clue what I'm doing!" viewpoint :D | 18:58 |
| leifmadsen | also I want to add examples for GitHub events because that's what I'm most interested in | 18:58 |
| leifmadsen | cool thanks much | 18:59 |
| leifmadsen | I'll be back with more questions later :) | 18:59 |
| mordred | leifmadsen: ++ | 18:59 |
| leifmadsen | ok yes, so this makes slightly more sense now | 18:59 |
| mordred | leifmadsen: I find thatviewpoint to be an excellent viewpoint to start with - and is VERY useful :) | 19:00 |
| leifmadsen | starting at the start is usually the best way to start :) | 19:00 |
| mordred | leifmadsen: and now I have a song in my hea | 19:02 |
| mordred | head | 19:02 |
| leifmadsen | you're welcome | 19:02 |
| leifmadsen | :) | 19:02 |
| leifmadsen | Start it up! | 19:02 |
| leifmadsen | what is the difference between openstack-infra/zuul-jobs and openstack-infra/openstack-zuul-jobs ? | 19:28 |
| leifmadsen | oh, I guess the former is used by zuul itself to instantiate an environment for a test, and then the latter is used by jobs themselves on the node when executing? | 19:29 |
| mordred | leifmadsen: nope | 19:37 |
| leifmadsen | so many repos | 19:38 |
| mordred | leifmadsen: zuul-jobs are jobs intended to be used by anybody | 19:38 |
| mordred | leifmadsen: so other zuul installations should add openstack-infra/zuul-jobs to their main.yaml to get the standard library of jobs | 19:38 |
| mordred | leifmadsen: openstack-infra/openstack-zuul-jobs contains jobs that are openstack specific but general forour use and expected to be used widely across ourdifferent projects | 19:39 |
| leifmadsen | ah ok | 19:39 |
| leifmadsen | so I want to basically add openstack-infra/zuul-jobs to my untrusted-projects, then probably shadow my own project-config? | 19:39 |
| pabelanger | leifmadsen: right. You should be able to use the git connection driver for zuul-jobs | 19:41 |
| leifmadsen | right | 19:41 |
| leifmadsen | yea, should point to the github mirror no problem I assume | 19:42 |
| pabelanger | then setup foobar-zuul-jobs for your specific variants / shadow jobs | 19:42 |
| leifmadsen | right right | 19:42 |
| leifmadsen | which would be the openstack-zuul-jobs equiv | 19:42 |
| leifmadsen | and those are really just the roles/playbooks being used by the ansible run, but it looks like I define the pipelines and jobs primarily in my project-config/ repo? | 19:43 |
| pabelanger | leifmadsen: right, project-config is our trusted repo | 19:43 |
| mordred | leifmadsen: yup. you might not even need a foobar-zuul-jobs if you don't have a need for a ton of shared jobs amongst your other repos | 19:47 |
| leifmadsen | right, at this point it looks like I don't :) | 19:47 |
| mordred | (we should probably hav ea "so you have one repo and you want to use zuul" "so you have a few repos" and "zomg you have a ton of repos, here's some best practice suggetsions" | 19:47 |
| leifmadsen | indeed | 19:53 |
| leifmadsen | I'm thinking of even starting with updating the repo README to make it more clear wtf they are for :) | 19:53 |
| jeblair | leifmadsen, mordred: well, the *idea* is to use the git driver, but i'm not sure that will work for a remote repo right now. you may just need to clone zuul-jobs to your host and point at the local repo for the moment (and then update it manually), or use the gerrit driver. | 20:03 |
| jeblair | it shouldn't be too hard to finish up the git driver. i just don't think it's quite there yet. | 20:03 |
| SpamapS | Hrm | 20:03 |
| * SpamapS starts spraying unfiltered thoughts about github PRs and zuul... | 20:04 | |
| SpamapS | https://github.com/gearman/gearmand/pull/133 | 20:04 |
| SpamapS | so that's a PR on gearmand, which is managed by BonnyCI's github+zuul2.5 thingy | 20:04 |
| SpamapS | If I approve that, it will land a merge commit with all those tiny commits | 20:04 |
| SpamapS | I'm just wondering if I actually want that. | 20:04 |
| SpamapS | or if Zuul can help me... or is this just Github making my life harder. | 20:05 |
| * SpamapS turns filter back on | 20:05 | |
| clarkb | if zuul grows the ability to push directly back rather than relying on $system you could also have it auto squash. But yes I think this is one of the massive failures with github's review/PR system. It encourages dirty history and sorting that out requires work | 20:06 |
| jeblair | SpamapS: if you don't, i don't think it's zuul's job to change that. :) | 20:06 |
| clarkb | basically because reviews happen on branches which get merged rather than discrete changes you end up with ^ | 20:07 |
| SpamapS | yep | 20:07 |
| jeblair | clarkb: i agree -- though some argue that's more correct history :) | 20:07 |
| SpamapS | and a lot of people do just hit the squash button after reviewing. | 20:07 |
| SpamapS | jeblair: that's the bzr way of thinking ;) | 20:07 |
| SpamapS | all history is good history if you hide it deep enough in merges | 20:08 |
| clarkb | right gerrit tracks that history in the review system rather than the "pristine" working copy of a repo (which I quite like) | 20:08 |
| tobiash | leifmadsen, mordred: tried the git driver last week and it didn't work yet (throws some not implemented exception) | 20:09 |
| SpamapS | on a side note... This is being done because Travis CI started automatically running shellcheck against all shell scripts it sees you trying to run. | 20:09 |
| SpamapS | which seems like a really weird thing to do. | 20:09 |
| mordred | SpamapS: I imagine zuul could (and does?) expose the option for your pipeline of using squash when merging things? | 20:10 |
| tobiash | leifmadsen, mordred: I used a local mirror for that (had to shadow the base job for my deployment) | 20:10 |
| jeblair | anyway, yeah, auto-squash is *conceivable*. it certainly shouldn't be on by default. and i think you'll probably have to, erm, convince me over beers it's a good idea to even include. :) | 20:10 |
| clarkb | Shrews: jeblair fwiw zuul did just install successfully on python 3.4 so it gets at least that far :) I'm going to get v2.5 going first though as I should be able to get that running really quickly then dig into v3 | 20:10 |
| jeblair | clarkb: cool. i think Shrews is right, if you stay away from the web stuff, should be fine. | 20:10 |
| mordred | jeblair: well, I think it's a standard github feature and not including support for telling zuul to tell github to do it when it tells github to merge a repo would be a very weird choice for us to make | 20:11 |
| pabelanger | mordred: have time to review: https://review.openstack.org/#/c/486679/ our NOSE settings for tox jobs | 20:11 |
| mordred | pabelanger: yup! | 20:11 |
| jeblair | mordred: and here i thought SpamapS was asking for something that didn't exist. i clearly don't know what i'm talking about so i'll just go back to work now. :) | 20:11 |
| pabelanger | mordred: that should lead you to https://review.openstack.org/#/c/483987/ | 20:11 |
| clarkb | mordred: github PRs can be auto swaushed now? | 20:12 |
| mordred | jeblair: he may have been - and I agree that in the world of zuul merging and pushing rather than clicking a merge button I'd be hard pressed to think a squash option for zuul would be a good idea | 20:12 |
| mordred | clarkb: yes | 20:12 |
| mordred | clarkb: there is a "merge and squash" option when you decide to merge a PR | 20:12 |
| clarkb | nice | 20:12 |
| SpamapS | the thing is | 20:13 |
| clarkb | I remember the dark days of constantly being asked to curate a series by hand | 20:13 |
| SpamapS | I don't think I'd always want squash | 20:13 |
| jeblair | clarkb: when you have a sec, can you take a look at https://review.openstack.org/485875 and its 16 children? i don't need a detailed review from you if you don't have time, but i don't want the chance series to surprise you. | 20:13 |
| jeblair | clarkb: s/chance/change/ | 20:13 |
| mordred | SpamapS: if you don't always want it then I think you're stuck rebasing/curating - and yes, I thnk this is a github-makes-things-harder thing :) | 20:13 |
| SpamapS | mordred: me too :-/ | 20:13 |
| mordred | SpamapS: also, hopefully soon you'll be able to ditch travis for gearman | 20:13 |
| mordred | :) | 20:14 |
| SpamapS | for zuul but yeah | 20:14 |
| SpamapS | :) | 20:14 |
| SpamapS | or | 20:14 |
| SpamapS | oh yeah | 20:14 |
| SpamapS | I know what you meant | 20:14 |
| jeblair | SpamapS: i doubt you're doing much if anything with ZUUL_ env vars atm, but if you are, please also see 485875 and children | 20:14 |
| SpamapS | words | 20:14 |
| SpamapS | hard | 20:14 |
| SpamapS | jeblair: will do | 20:14 |
| clarkb | jeblair: looks like its basically just removing env vars in favor of ansible vars? then I guess we add a "v2.5 compatible shell task" that injects the old env vars back in using ansible if we find we rely on them somewhere? | 20:14 |
| mordred | pabelanger: YES!!!! excellent | 20:14 |
| jeblair | clarkb: exactly. i expect that to be part of the automatic translation we do | 20:15 |
| clarkb | ok I need second lunch because all I had before was vegetables so I am still hungry. Will poke at it when I get back | 20:15 |
| clarkb | but generally the replace with compatible sehll task runner thing seems like its fine for filling that transitional need | 20:16 |
| jeblair | clarkb: cool, thx | 20:16 |
| pabelanger | mordred: great! | 20:17 |
| jeblair | i'm going to tag and release zuul-sphinx | 20:17 |
| jeblair | what with us not having had a release yet | 20:17 |
| jeblair | 0.1.0? | 20:18 |
| mordred | jeblair: ++ | 20:19 |
| jeblair | mordred, pabelanger: do you think it might be possible to have tox based jobs automatically install secondary repos from source in ~/src/... ? | 20:20 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Create tox_environment_defaults variable for tox based jobs https://review.openstack.org/486679 | 20:20 |
| pabelanger | jeblair: could you give an example? | 20:21 |
| jeblair | mordred, pabelanger: for example, since we want zuul-jobs docs to use zuul-sphinx, rather than having the docs job install sphinx from pypi, it installs it from src. so we could get automatic depends-on behavior. | 20:21 |
| jeblair | mordred, pabelanger: this would also apply to things like neutron driver unit tests, etc. | 20:21 |
| mordred | jeblair: I thnk doing that would be hard and get into a weird place | 20:22 |
| jeblair | mordred: i have an idea -- what if we made another requirements file with all of the zuul-prepared repos, and installed that reqs file first? | 20:23 |
| mordred | jeblair: I mena - I suppose it could do a for repo in ~/src ; .tox/{env}/bin/pip install -e $repo | 20:23 |
| jeblair | mordred: or that, yeah | 20:23 |
| jeblair | mordred: (doesn't have to be "-e" | 20:23 |
| pabelanger | okay, I understand now. | 20:24 |
| mordred | yah - it's kind of overriding tox at that point though - which makes me wonder if maybe a non-tox based 'run unittests by installing direcetly" job might be better | 20:24 |
| jeblair | mordred: true, though i'm wondering if this could be the standard job. | 20:25 |
| pabelanger | If you could shadow tox/pre, it would be easy to pip install something | 20:25 |
| *** jkilpatr has quit IRC | 20:25 | |
| mordred | jeblair: I mean, it's not SPER unsafe since most unittests jobs won't have an extra repo ... EXCEPT openstack/requirements | 20:26 |
| pabelanger | I guess something like: http://paste.openstack.org/show/616497/ wouldn't work | 20:26 |
| jeblair | mordred: i think fundamentally what would be nice is this: it's currently easy to say "here's all the packages i depend on, install them from releases for the unit tests". but there is not a good way to say "here are the packages i depend on, install (some of) them from branch tip or later because we are closely related". i would like that to be easy. :) | 20:27 |
| mordred | jeblair: I strongly agree with the desire | 20:27 |
| jeblair | pabelanger: i think that might work? | 20:27 |
| mordred | jeblair: mostly quibbling/thinking about impl details and ramifications of such | 20:27 |
| jeblair | mordred: ++ | 20:27 |
| jeblair | mordred: this is total brainstorming. :) | 20:27 |
| mordred | I'm concerned about openstack/requirements | 20:28 |
| jeblair | mordred: yeah. doing it site-local lets us special case that. harder if we generalize it. | 20:28 |
| jeblair | mordred: oh, we could compare with the contents of the main project's (test-)requirements file(s) | 20:28 |
| mordred | well - we could parameterize the special casesimilar to the tox_environment | 20:28 |
| mordred | jeblair: requirements files don'tnecessarily match the git repo - we'd have to read setup.cfg files in the repos | 20:29 |
| jeblair | mordred: that too. if we have to list the repos twice on the job (once in 'repos' and once in 'vars), it won't be the worst thing in the world | 20:29 |
| mordred | which is doable, of course | 20:29 |
| mordred | jeblair: I was mostly thining we could special case requirements to be skipped | 20:29 |
| jeblair | mordred: oh yeah i see | 20:29 |
| mordred | like a "ignore repos" list on the baes job or something | 20:29 |
| mordred | and we could set _that_ on tox-py35-constraints | 20:30 |
| mordred | since that's where openstack/requirements gets added and messes things up | 20:30 |
| jeblair | ya | 20:30 |
| mordred | jeblair: I like it - we might want to write a tiny python module todo the "look for repos in src and match them to requirements.txt" | 20:31 |
| jeblair | okay, i don't want to derail pre-ptg stuff, so this should probably happen later if we do it. but sounds like it may be desirable and we have some leads on approach. | 20:31 |
| mordred | jeblair: ++ | 20:31 |
| jeblair | i'll write a story | 20:31 |
| mordred | once peopel can declare a second repo for their job easily, I think this becomes the 'obvious' expectation of how such a job would operate | 20:32 |
| jeblair | mordred: ++ | 20:32 |
| pabelanger | mordred: 484027 is ready to land too, adds shade to zuulv3.o.o | 20:33 |
| jeblair | "you wrote the code that checks out the repo, and you wrote the code that runs the job, but the job you wrote ignores the repo you checked out?" :) | 20:33 |
| jeblair | (is what they will ask us) | 20:33 |
| pabelanger | https://review.openstack.org/487172/ is another review to bikeshed, makes all our jobs in zuul-jobs voting by default | 20:34 |
| mordred | jeblair: yup | 20:34 |
| pabelanger | lastly, https://review.openstack.org/485840 refactors our subunit handling | 20:35 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Jobs should default to voting https://review.openstack.org/487172 | 20:36 |
| *** jkilpatr has joined #zuul | 20:56 | |
| *** hashar has joined #zuul | 21:13 | |
| *** dkranz_ has quit IRC | 21:16 | |
| SpamapS | Alright, story #2000773 is officially closed and will be off the progress board when the next cron job runs. https://storyboard.openstack.org/#!/story/2001134 is the new "re-enable tests" story with the 7 tasks that are left. | 21:27 |
| SpamapS | jeblair: I'm satisfied with 485875, but I'm hesitant to block approve the stack given its potential to break people. Should we let it hang out there for a while? | 21:46 |
| jeblair | SpamapS: i think we're good at this point. but i can +a tomorrow. | 21:58 |
| clarkb | I'm starting to look at zuul config for v3 against review-dev.o.o and notice that zuul.conf has key value pairs with empty values | 22:05 |
| clarkb | I assume that means they fall back to defaults? | 22:05 |
| clarkb | or are they becoming literal empty configs? | 22:06 |
| SpamapS | ask config parser | 22:09 |
| SpamapS | actually I think they do end up as defaults | 22:09 |
| SpamapS | jeblair: ok, your comfort with that makes me more comfortable. :) +A'd | 22:12 |
| jeblair | clarkb: i suspect those are all accidents | 22:12 |
| SpamapS | and yeah, "something=" is very ambiguous no matter where it appears | 22:13 |
| jeblair | probably puppet needs more "ifs" sprinkled | 22:13 |
| *** maxamillion has quit IRC | 22:15 | |
| clarkb | jeblair: doesn't look like the git connection type is documented as a zuul tenant config source? | 22:25 |
| *** hashar has quit IRC | 22:25 | |
| jeblair | clarkb: correct; it's not finished | 22:26 |
| jeblair | clarkb: tobiash said he tried it the other day and had problems, it may have bitrotted a bit | 22:27 |
| jeblair | clarkb: if it doesn't work for you, i'll jump on it since that's a task for *two* priority efforts :) | 22:27 |
| clarkb | I guess its trivial to merge a change to repo in review-dev | 22:27 |
| clarkb | (to add a .zuul.yaml) | 22:27 |
| jeblair | clarkb: yeah, maybe that's easiest to get moving | 22:28 |
| clarkb | looking at code it appears that I can use the git source without a connection? is that a correct interpretation of the intent at least? | 22:29 |
| clarkb | I'll probably give it a go really quick just because | 22:29 |
| jeblair | clarkb: http://paste.openstack.org/show/616503/ is what i have in zuul.conf from last time i used it | 22:30 |
| jeblair | clarkb: http://paste.openstack.org/show/616504/ | 22:31 |
| jeblair | clarkb: course, we changed that entry in main.yaml to 'config-projects' since then :) | 22:32 |
| *** maxamillion has joined #zuul | 22:32 | |
| clarkb | ya I found that :) it is documented that way | 22:36 |
| clarkb | jeblair: another piece of feedback, zk is apparently required, zuul-scheduler exits (at least when run as non daemon) when it cannot connect to zk. May want to update https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/quick-start.html#install-zuul | 22:44 |
| jeblair | clarkb: ah, thanks. | 22:45 |
| jeblair | clarkb: you can probably just apt-get install zookeeper and run with localhost real quick for tihs | 22:45 |
| jeblair | clarkb: oh you know what? we haven't special-cased noop for nodes yet | 22:46 |
| jeblair | clarkb: it still actually needs a nodepool in order to fulfill the noop request | 22:46 |
| jeblair | clarkb: it should work (and be a fun test) to point it at the "prod" v3 zookeeper, but i think that'll need a firewall change | 22:47 |
| jeblair | clarkb: actually, i'm not 100% sure of that. i know we haven't special cased empty nodesets, but it may be that noop doesn't need a nodeset. | 22:48 |
| clarkb | ok I'll start with a local zk for now | 22:48 |
| clarkb | jeblair: it submitted merger:cat jobs for the git connection at least | 22:55 |
| clarkb | also I just realized the way I set this up I have two projects with the same name... I don't expect that to quite workl | 22:56 |
| clarkb | (in one tenant) | 22:56 |
| *** xinliang has quit IRC | 22:56 | |
| jeblair | clarkb: if they're under different connections, it should work :) | 23:04 |
| jeblair | clarkb: (you may need to fully qualify their names in some cases, but zuul should tell you) | 23:04 |
| clarkb | jeblair: it didn't look like the git source was working anyways so I collapsed into one list entry and merged a change for zuul.yaml | 23:10 |
| clarkb | but good to know | 23:10 |
| *** xinliang has joined #zuul | 23:10 | |
| *** xinliang has quit IRC | 23:10 | |
| *** xinliang has joined #zuul | 23:10 | |
| jeblair | clarkb: k | 23:10 |
| clarkb | jeblair: in the logs I see http://paste.openstack.org/show/616506/ but doesn't appear to trigger any jobs | 23:11 |
| clarkb | (or complain about lack of nodes or missing job def) | 23:11 |
| clarkb | I'm assuming that is because I don't have an executor and it would handle all that, eg noop isn't quite magical enough to happen just in the scheduler? | 23:11 |
| clarkb | (though I would kind of expect gearman logs about attempting to run a job and failing or something like that) | 23:12 |
| clarkb | oh there is a clue, TenantParser is waiting for merger:cat job on my trusted config to finish | 23:18 |
| clarkb | I don't need to run a separate merger do I? | 23:18 |
| clarkb | in any case that gives me a breadcrumb to follow | 23:18 |
| clarkb | http://paste.openstack.org/show/616507/ is what I appear to eventually get maybe due to a gearman timeout? | 23:21 |
| clarkb | in any case that is likely the problem | 23:21 |
| jeblair | clarkb: yeah, you need either an executor or merger in order to load the config | 23:22 |
| clarkb | oh right the executor is the merger now if no explicit merger | 23:22 |
| clarkb | ok easiest thing to try is run a merger | 23:22 |
| jeblair | that should really be a special error message :) | 23:22 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use default sphinx theme and index attributes https://review.openstack.org/487239 | 23:23 |
| jeblair | mordred: ^ i think i have everything needed for that change to render a sample. that is approximately what we discussed earlier about having full path names for attributes. | 23:23 |
| jeblair | mordred: it doesn't do "pipeline.trigger[].event" but more like "pipeline.event". but it's a start. :) | 23:25 |
| jeblair | mordred: it doesn't do "pipeline.trigger[].event" but more like "pipeline.trigger.event". but it's a start. :) [corrected] | 23:25 |
| jeblair | i only updated a couple of attributes in that change, just to demonstrate the framework | 23:26 |
| clarkb | and with that sorted out it is definitely submitting node requests to my local zk and then getting mad that nothing is servicing them | 23:30 |
| clarkb | or at least nothing is servicing that request. So now question is would it be faster to make noop magical again or just use production nodepool? | 23:31 |
| clarkb | I've got to go do a grocery run now so that will likely be tomorrow morning's fun | 23:31 |
| clarkb | everything on the front end of the zuul gerrit event procssing seems to work fine though | 23:32 |
| clarkb | I've gone ahead and stopped all zuul related services (including zk) on review-dev now | 23:34 |
| clarkb | will pick this up in the morning | 23:34 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 23:38 |
| jeblair | clarkb: i think it might be that easy ^ ? | 23:38 |
| jeblair | mordred: http://docs-draft.openstack.org/39/487239/1/check/gate-zuul-docs-ubuntu-xenial/6c7ede3//doc/build/html/user/config.html#attr-name | 23:39 |
| jeblair | mordred: there's a deep link to the thing i'm changing | 23:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use default sphinx theme and index attributes https://review.openstack.org/487239 | 23:42 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool feature/zuulv3: Remove FakeProvider getClient monkey-patch https://review.openstack.org/475131 | 23:48 |
| tristanC | Great, thank you for the review of the nodepool-driver refactor! | 23:51 |
| jeblair | tristanC: \o/ glad that's in; hopefully that'll make adding the new drivers easier :) | 23:52 |
| jeblair | mordred: http://docs-draft.openstack.org/39/487239/2/check/gate-zuul-docs-ubuntu-xenial/fa68dbe//doc/build/html/user/config.html#attr-name link to updated build which fixes a typo | 23:53 |
| *** deep-book-gk_ has joined #zuul | 23:55 | |
| *** deep-book-gk_ has left #zuul | 23:57 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!