| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 00:01 |
|---|---|---|
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix null ref in github tests https://review.openstack.org/490257 | 00:01 |
| pabelanger | Oh, I see the issue now | 00:01 |
| jeblair | jlk: can you take a look at 490257 and verify that's correct? | 00:01 |
| pabelanger | http://paste.openstack.org/show/617352/ | 00:01 |
| pabelanger | jeblair: ^ do you know why that is? Is that because of shadow jobs? | 00:02 |
| jeblair | pabelanger: oh yeah, you can't inherit from later jobs | 00:02 |
| pabelanger | Ah, darn | 00:02 |
| jlk | jeblair: looking | 00:02 |
| pabelanger | jeblair: guess we should revert for now and come up with a new plan | 00:03 |
| jeblair | that's something i'd like to fix, but is not trivial. ie, i doubt it'll happen pre-ptg. | 00:03 |
| pabelanger | k | 00:03 |
| jeblair | pabelanger: did zuul +1 the patch to add that? | 00:04 |
| pabelanger | looking | 00:04 |
| pabelanger | jeblair: no, didn't leave anything | 00:04 |
| pabelanger | so, should see why that was | 00:05 |
| jeblair | ++ | 00:05 |
| pabelanger | remote: https://review.openstack.org/490258 Revert "Create publish-openstack-python-tarball trusted job" | 00:05 |
| pabelanger | until we land that zuulv3.o.o is down :( | 00:05 |
| pabelanger | looking at logs now | 00:05 |
| pabelanger | jeblair: looks like dynamic reload worked properly, and project-config isn't in check pipeline so no reason for zuul to +1 | 00:07 |
| pabelanger | guess we should conf error on that | 00:08 |
| jeblair | pabelanger: oh, well if we don't report on anything, that makes sense. :) | 00:08 |
| jeblair | pabelanger: should probably add it to check | 00:08 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix github dependent pipeline with merge https://review.openstack.org/489925 | 00:11 |
| pabelanger | jeblair: k | 00:13 |
| jlk | mordred: Well good news, Ansible upstream (jimi) confirmed that Python urllib2 doesn't allow a redirect to file:/// like that. | 00:23 |
| *** http_GK1wmSU has joined #zuul | 00:25 | |
| *** http_GK1wmSU has left #zuul | 00:27 | |
| jlk | mordred: https://github.com/python/cpython/blob/master/Lib/urllib/request.py#L710-L717 | 00:33 |
| pabelanger | okay, zuulv3.o.o working again | 01:22 |
| pabelanger | now EOD | 01:22 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Make timer tests less racy https://review.openstack.org/490208 | 05:30 |
| *** robcresswell has quit IRC | 05:41 | |
| *** amoralej|off is now known as amoralej | 07:28 | |
| *** robcresswell has joined #zuul | 07:41 | |
| *** hashar has joined #zuul | 08:12 | |
| *** openstackgerrit has quit IRC | 08:18 | |
| *** electrofelix has joined #zuul | 08:45 | |
| tristanC | pabelanger: jeblair: it's ok, worst case we just have to touch /etc/lsb-release | 08:58 |
| tristanC | though tobiash suggested to improve the --ro-bind management by checking if the source file exist in the first place, like it's done L182 for nsswitch.conf | 08:59 |
| *** hashar has quit IRC | 09:30 | |
| *** hashar has joined #zuul | 09:32 | |
| *** openstackgerrit has joined #zuul | 10:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490134 | 10:21 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove status handling from FakeGithubConnection https://review.openstack.org/490413 | 10:21 |
| *** jkilpatr has quit IRC | 10:44 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove status handling from FakeGithubConnection https://review.openstack.org/490413 | 10:54 |
| *** amoralej is now known as amoralej|lunch | 11:05 | |
| *** jkilpatr has joined #zuul | 11:17 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add zuul:var directive and role https://review.openstack.org/490246 | 11:21 |
| mordred | jlk: oh good! | 12:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 12:04 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow use of the uri module from localhost https://review.openstack.org/490215 | 12:04 |
| tobiash | mordred: what is rtfd? | 12:06 |
| *** amoralej|lunch is now known as amoralej | 12:09 | |
| tobiash | mordred: I've a comment/question on 490215 | 12:11 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some lookup plugin tests https://review.openstack.org/454396 | 12:12 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Don't ignore inexistent jobs in config https://review.openstack.org/488758 | 12:13 |
| *** dkranz_ has joined #zuul | 12:20 | |
| mordred | tobiash: readthedocs.org - service that builds and hosts python documentation | 12:25 |
| tobiash | tobiash: ah | 12:25 |
| mordred | tobiash: we do this for some of our projects currently: http://git.openstack.org/cgit/openstack-infra/project-config/tree/jenkins/jobs/hooks.yaml | 12:25 |
| mordred | and so I kindof went down a rabbit hole yesterday thinking that we can translate that into a job that doesn't need a node :) | 12:26 |
| tobiash | yay | 12:26 |
| mordred | oh - good point in that comment | 12:26 |
| mordred | I'm working on getting a test written - I'll do that in the next rev | 12:27 |
| Shrews | fungi: you have made me feel old today, displaying my membership ID among the others in your email from yesterday | 13:49 |
| Shrews | at least mordred seems a bit older based on that number :-P | 13:49 |
| Shrews | ooh, and jeblair | 13:50 |
| tobiash | looks like I can feel pretty young... | 14:14 |
| fungi | Shrews: it's a ranking of how imporant everyone is. hence tons of people ahead of me | 14:20 |
| pabelanger | mordred: jeblair: so, I am at a lose how to move forward with the publish-openstack-python-tarball now. If we cannot parent to zuul-jobs, we have no way currently for a config-project to access any existing jobs. | 14:29 |
| jeblair | pabelanger: i have two thoughts about that: 1) if we need to be able to forward-parent, i think i may be able to do that with a few days work. however.... | 14:33 |
| jeblair | pabelanger: 2) i'm not sure we actually want to parent popt onto tox-tarball. i think that may actually be backwards. | 14:33 |
| jeblair | pabelanger: https://etherpad.openstack.org/p/0BO00tBBUi | 14:34 |
| pabelanger | jeblair: there is tox-tarball-post, which handles the sync to executor | 14:39 |
| * mordred looking | 14:41 | |
| jeblair | pabelanger: okay, the current contents of that etherpad are the job hierarchy -- job names in bold, and the roles in italics | 14:42 |
| jeblair | i simplified some things, but you should get the idea | 14:42 |
| jeblair | and it shows the nesting structure | 14:42 |
| jeblair | the main point is -- by having the publish-openstack-python-tarball (which i'm going to call popt now) last means that we copy the artifacts to tarballs.o.o *before* we fetch them from the host, which obviously won't work. | 14:43 |
| pabelanger | jeblair: ya, that is the rough idea of the job | 14:43 |
| pabelanger | Hmm, let me think. because you are missing tox-tarball-post ATM | 14:44 |
| jeblair | pabelanger: no that's represented by "fetch artifacts" | 14:44 |
| jeblair | that's from tox/tarball-post | 14:44 |
| jeblair | "Collect tarball artifacts." is actually the task name | 14:44 |
| pabelanger | ah, did I mess up the order of the post-run | 14:45 |
| jeblair | pabelanger: no, you can't change that because it's inherited | 14:45 |
| jeblair | so what i'm saying is this is not the best inheritance structure for this sort of thing | 14:45 |
| pabelanger | Right, I was assuming the parent post-run would run before openstack-python-tarball post-run | 14:46 |
| jeblair | publishing an artifact is *more fundamental* than building it, so it should be earlier in the hierarchy (if one were to use inheritance) | 14:46 |
| jeblair | pabelanger: no, they nest | 14:46 |
| jeblair | it's possible we're overusing inheritance a bit here | 14:46 |
| jeblair | and instead we should just make a new job that invokes the same roles | 14:47 |
| pabelanger | Ya, then the job isn't designed properly | 14:47 |
| mordred | yah - I was just starting to have the same thought | 14:47 |
| mordred | I keep getting in this hole myself... | 14:47 |
| mordred | I want to make _everytihng_ about jobs | 14:47 |
| mordred | but roles are actualy the basic unit of reuse | 14:47 |
| mordred | it's worth noting as well, even though we have not yet done it in any of ours - that roles can declare dependencies on each other too | 14:48 |
| jeblair | ya. so i'll mock up 2 more things in the etherpad now. first -- this structure but with a different inheritance hierarchy | 14:48 |
| jeblair | and then a third option which is just a new job with roles | 14:48 |
| jeblair | the alternate hierarchy one is impractical because of course we can't parent unittests to publish-openstack-tarball. | 14:50 |
| jeblair | er, i'm going to mock up 3 things :) | 14:54 |
| jeblair | so there's a 'new job' option where we just do everything with roles | 14:55 |
| jeblair | and a 'hybrid' option which has a generic tarball publishing job as the parent of the python tarball publishing job. | 14:55 |
| jeblair | the difference between the two, i think, is whether we want to tell folks, "if you need to upload something to tarballs.o.o" do we say "use this job as the parent" or "use this role in a post playbook" | 14:56 |
| jeblair | i think i just answered the question myself | 14:56 |
| pabelanger | Okay, the part I am still trying to loop my brain around, is with all 3 examples, do not use the playbooks in zuul_jobs/tox/pre, run, post, correct? | 14:57 |
| mordred | jeblair: (btw, I'm taking over your lookup plugin test patch) | 14:58 |
| jeblair | actually no, i haven't. i honestly think either might work almost as well for folks writing the sorts of jobs which publish to tarballs. | 14:58 |
| jeblair | mordred: ack, thanks. i didn't get a chance to look at current failures yesterday. lemme know if you have questions. | 14:58 |
| jeblair | pabelanger: that's a good question. | 14:59 |
| mordred | I got it - I'm going to push up a not-quite-finished stack - I reparented my patches on your patch, have fixed the issue (I think) in your patch - and will actually add a test here in just a sec | 14:59 |
| mordred | jeblair: I also nerd-sniped myself into writing a sphinx domain ... | 14:59 |
| jeblair | pabelanger: my inclination would be to try to avoid reusing those playbooks, because roles provide a cleaner API | 14:59 |
| jeblair | pabelanger: so some of those playbooks may need to be made into roles if we want to do that | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 15:00 |
| jeblair | pabelanger: though, it's certainly true we can just list a bunch of playbooks from "other jobs". | 15:00 |
| jeblair | mordred: what do you think ^? | 15:00 |
| mordred | yes, I agree - for now our playbooks contain next to no actual logic - even though it's a slightly higher amount of boilerplate files, I think we shouldn't shy away from multiple playbooks | 15:02 |
| mordred | and in this case, it's likely much better | 15:02 |
| jeblair | mordred: so, avoid using playbooks from "other jobs" and use roles instead? | 15:02 |
| mordred | yah | 15:02 |
| jeblair | k | 15:02 |
| jeblair | mordred: what's the sphinx domain? | 15:02 |
| mordred | let's do that until we can't | 15:02 |
| mordred | jeblair: it's in https://review.openstack.org/490215 - I added one for being able to link to an upstream ansible module | 15:03 |
| pabelanger | Ya, that means we shoulding be building up jobs from zuul-jobs, instead creating new jobs from the same roles | 15:03 |
| pabelanger | shouldn't* | 15:03 |
| pabelanger | and new jobs that new parent to zuul-jobs, should only be setting up variables? | 15:04 |
| mordred | pabelanger: right. well - except for things where we actually want the inheritance properties - like inherting from base or unittests | 15:04 |
| pabelanger | okay, so for example: openstack-doc-build would be how we process | 15:05 |
| pabelanger | proceed* | 15:05 |
| pabelanger | it has roles: from zuul-jobs | 15:05 |
| pabelanger | but its own playbooks | 15:05 |
| pabelanger | however, parents to tox-docs | 15:05 |
| pabelanger | Hmm | 15:06 |
| jeblair | indeed, i think another thing pointing us in the direction of inheritance being wrong for this case is that we can actually drop some of the roles we are doing for unittests jobs (eg -- fetch testr output, and maybe test_setup.sh) | 15:06 |
| pabelanger | tox-docs is in zuul-jobs | 15:06 |
| mordred | jeblair: yah | 15:06 |
| pabelanger | jeblair: right, I didn't think about that part yet | 15:06 |
| pabelanger | however, we could control that logic using variables | 15:07 |
| pabelanger | okay, let me see about writing a new playbook using the roles | 15:07 |
| jeblair | pabelanger: that sounds good -- but do we need to talk about openstack-doc-build? i'm not sure the question you asked above | 15:07 |
| mordred | jeblair, pabelanger: fwiw, this is about the level of complexity I was imagining we were going to hit when working on these tox jobs - so I'm glad we've hit it | 15:07 |
| jeblair | mordred: :) | 15:08 |
| mordred | my hunch is that these will wind p being the most difficult and complex jobs in the entire system | 15:08 |
| jeblair | mordred: agreed. i actually think devstack will be simpler :) | 15:09 |
| pabelanger | jeblair: sure, just your comment avoid reusing playbooks and to use roles | 15:09 |
| pabelanger | openstack-doc-build today parents to tox-docs, which works in that context because we don't need to add any post-run logic | 15:09 |
| pabelanger | so, there are places where parent to a zuul-jobs still makes sense | 15:09 |
| pabelanger | however, if openstack-doc-build was trusted, I don't think it would work, right? | 15:10 |
| pabelanger | since tox-docs loads after project-config | 15:10 |
| jeblair | pabelanger: i think openstack-doc-build parents tox-docs mostly to override the run playbook since "our" docs build command is different. that seems like a pretty good use of inheritance | 15:11 |
| jeblair | pabelanger: so yeah, it's not trying to do any fussy post playbook stuff :) | 15:11 |
| pabelanger | ya, so post-runs are pussy, because they run before the parent | 15:12 |
| pabelanger | fussy* | 15:12 |
| pabelanger | wow, such a typo | 15:12 |
| pabelanger | sorry about that | 15:13 |
| *** hashar has quit IRC | 15:34 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 15:50 |
| jeblair | mordred: i like that ansible sphinx role -- i've been thinking we may want something similar to link to job docs in zuul-jobs | 15:58 |
| jeblair | mordred: (from other repos) | 15:58 |
| pabelanger | remote: https://review.openstack.org/490576 WIP: Create openstack-python-tarball job | 16:00 |
| pabelanger | mordred: jeblair: ^ before I start refactoring, do you mind looking at the flow of the new job?^ Most of that code is copypaste of zuul-jobs playbooks | 16:00 |
| pabelanger | that we likely need to refactor into roles | 16:01 |
| jeblair | fungi, Shrews: wow, my foundation membership number is only one off from my lwn account number. | 16:01 |
| jeblair | fungi: any chance we can bump #82? :) | 16:01 |
| pabelanger | mordred: something like configure-mirrors, should that move into our base job? | 16:04 |
| jeblair | pabelanger: configure-mirrors in base sounds good to me | 16:05 |
| jeblair | pabelanger: and i think we have site-local vars now, so you can do that TODO in the process | 16:06 |
| fungi | jeblair: we might be able to switch you to 82... that profile is a 404 page in the member directory now ;) | 16:08 |
| jeblair | pabelanger: and yeah, i think that flow looks correct -- those are the things we should refactor into roles. most of those look like they should be 1:1 playbook:role | 16:09 |
| jeblair | fungi: lol, most important task! what could go wrong?! | 16:09 |
| pabelanger | jeblair: ya, I almost wonder if we could parent to unittests. But the issues are does test-setup run and, do we intall bindep with test profile | 16:09 |
| pabelanger | but, that would be part of the forward -parent stuff you mentioned before | 16:10 |
| pabelanger | because that playbook in unittest, is just roles | 16:10 |
| jeblair | yeah, i don't think we need test setup or testr output, both of which are unittest. | 16:11 |
| jeblair | and later on, if mordred has his way with the pti, we won't need bindep either | 16:11 |
| Shrews | jeblair: what is the purpose of the :value: sphinx markup used in the docs? That seems to be impossible to google | 16:15 |
| Shrews | jeblair: wondering if there was a reason :ref: wasn't used instead | 16:15 |
| jeblair | Shrews: i made it up | 16:15 |
| Shrews | oh. how does that work? | 16:16 |
| jeblair | Shrews: it's a role to cross-reference zuul "value" directives | 16:16 |
| jeblair | Shrews: there are a few (up to 3 now) things in the documentation that i made custom directives for so i can control how they are displayed, and ensure they have targets for cross referencing | 16:16 |
| jeblair | Shrews: i don't think they automatically end up in anything that would normally get picked up by a :ref: role (but i could be wrong about that, and it's sphinx, so even if i'm right, we probably could still make it happen) | 16:18 |
| pabelanger | jeblair: do you happen to have some reading material on our local-site variables? Looking to see how we did that | 16:18 |
| jeblair | Shrews: so i also made role equivalents, so if you want to reference a zuul:value or zuul:attr definition, you can use the corresponding :zuul:value: or :zuul:attr: role, and it will look it up based on the full "pathname" of the thing. | 16:19 |
| jeblair | Shrews: having our own roles for that also lets us customize how they are represented as well (for instance, i think we may want to change the :zuul:value: role to only display the 'short' name of the value, based on how it's actually getting used in the text) | 16:19 |
| jeblair | Shrews: i was thinking it might be time for a 'docs' page in the developer guide :) | 16:20 |
| Shrews | jeblair: ok. i'm going through the most recent docs and have a few small tweaks to put up. One is add a reference link at the first mention of the independent pipeline (similar to the dependent pipeline), but i noticed the use of :value: for that one. Not sure which to use | 16:20 |
| jeblair | pabelanger: https://docs.openstack.org/infra/zuul/feature/zuulv3/admin/components.html#id7 search for 'variables' (sorry i haven't gotten to that page yet to add link targets) | 16:20 |
| jeblair | Shrews: cool; it may be easier to look at that after you push it up | 16:22 |
| Shrews | jeblair: ok | 16:22 |
| jeblair | Shrews: *hopefully* :value: is appropriate for that one too | 16:22 |
| pabelanger | jeblair: great, thanks | 16:25 |
| *** electrofelix has quit IRC | 16:26 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 16:46 |
| jeblair | pabelanger, mordred, fungi: our pkcs secret can hold 3760 bits or 470 bytes. | 16:58 |
| jeblair | i'll work on the usenet solution now. :) | 16:58 |
| fungi | jeblair: there's our "chunk size" now! | 16:58 |
| fungi | we just need grouping and ordering and we're set | 16:58 |
| Shrews | hah, usenet. can we call the solution alt.zuul.secrets? | 17:11 |
| jeblair | Shrews: lol | 17:12 |
| fungi | Shrews: indeed, uuencoded and split binary posting to newsgroups was the first thing that design brought to my mind as well | 17:22 |
| fungi | i have some fond memories of following instructions to write a very simple uudecoder in edlin so i could uudecode the uudecode utility ;) | 17:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 17:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 17:32 |
| jeblair | fungi: the pkcs1-oaep overhead length is 42 bytes (4096-3760 bits == 42 bytes). :) | 17:34 |
| * fungi can't disagree with that math | 17:34 | |
| dmsimard | oi, I just tagged and released ARA 0.14.0, nothing should explode.. but just in case, letting you know. It usually takes a bit for the pypi mirrors to pick it up. | 17:54 |
| pabelanger | jeblair: mordred: should we introduce zuul_site as a prefix for local-site varaibles to be used in zuul-jobs? | 17:56 |
| jeblair | pabelanger: sounds reasonable | 17:56 |
| pabelanger | remote: https://review.openstack.org/490586 Move configure-mirror role into base-test/pre | 17:57 |
| pabelanger | is an example^ | 17:58 |
| pabelanger | guess it could be a dict too | 17:58 |
| jeblair | yeah. i kind of like deep structures. | 18:01 |
| jeblair | but don't have strong feelings on this. | 18:02 |
| *** amoralej is now known as amoralej|off | 18:04 | |
| pabelanger | same | 18:07 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Support longer pkcs1-oaep secrets https://review.openstack.org/490620 | 18:22 |
| jeblair | fungi, pabelanger, mordred, SpamapS: ^ | 18:23 |
| pabelanger | jeblair: nice | 18:26 |
| fungi | jeblair: i suppose that does also mean that an attacker could guess at the approximate size of our signing keyring, modulo 3760 bits. not especially risky for this case anyway | 18:27 |
| jeblair | fungi: easier for them to read the docs, i'd wager ;) | 18:28 |
| fungi | indeed | 18:28 |
| jeblair | (then again, maybe not! it's so hard to convince people to read docs! why should attackers be any different!) | 18:29 |
| fungi | i certainly get a nonzero number of excited "security researchers" who want to make sure i know that we have <random web service x> running on the internet for all to see | 18:30 |
| fungi | or that we have web servers accidentally configured to display system logs (oh my!) | 18:30 |
| jeblair | "that's why they call it Internet" | 18:30 |
| fungi | "look, i found that someone has accidentally exposed your logserver to the internet, and it's full of all sorts of sensitive syslog data: https://logs.openstack.org/..." | 18:32 |
| fungi | that's always a fun one to try to explain to people who sometimes haven't even heard of free software and never imagined that there are communities of people who write and run software by coordinating over the open internet | 18:33 |
| fungi | then again, it's also an opportunity for me to practice my sales pitch ;) | 18:34 |
| mordred | jeblair, pabelanger: I also like deep structurs/dicts - but also don't have strong preferences | 18:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Remove mirror_domain from configure-mirror https://review.openstack.org/490625 | 18:50 |
| jlk | tobiash: so question on your protected branch PR, why is it y'all went with branch+pr within the same repo instead of the traditional fork+pr method? (I'm genuinely curious) | 18:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Rename mirror_host to mirror_fqdn in configure-mirrors https://review.openstack.org/490628 | 18:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove getProjectBranches from FakeGithubConnection https://review.openstack.org/490043 | 18:56 |
| tobiash | jlk: I'm new to github and we'll get one soon. So I'm currently in progress of investigating how to work effectively with github | 19:06 |
| tobiash | jlk: currently we're on gerrit with a 50+ tightly coupled (c++, compiled together in a single workspace) | 19:07 |
| tobiash | jlk: so currently most people think that managing such a beast would be easier with the shared repo model under a organization | 19:08 |
| jlk | single repo, lots of sub folders? | 19:09 |
| tobiash | jlk: I'm not sure about this, but I want to be open and don't want to force the projects into the fork+pull model because the CI system I'm running doesn't support shared repos | 19:09 |
| jlk | or lots of repos, but no forks of those repos? | 19:09 |
| tobiash | jlk: sure, that would be my preference, but there are modules in there which have different visibilities for employees and also externals work directly on the codebase... | 19:10 |
| tobiash | jlk: so more political stuff inhibits throwing everything just into one repo | 19:11 |
| tobiash | jlk: so currently we have lots of repos managed with git-repo | 19:12 |
| jlk | Protected branches makes (no forks) more manageable. Before that it was really messy since everybody would need write access to the repo. I'm curious how well it will work out for you to do it all within the same canonical repo(s) instead of forks. | 19:12 |
| tobiash | it will be fun telling the people that git-repo won't work with github... | 19:12 |
| jlk | I agree that we (zuul) should support that work model. It's just not one I'm all that familiar with. | 19:12 |
| tobiash | jlk: I personally think that this will get messy with the branches | 19:13 |
| jlk | I do too | 19:13 |
| jlk | You'd probably want to namespace them somehow | 19:13 |
| jlk | but that doesn't help as much as one would like | 19:13 |
| tobiash | jlk: but I'm not in the position to decide that, but just give (hopefully) good advices | 19:13 |
| jlk | one advantage is that people can "fix up" somebody else's pull request. Doing that is much harder in the fork+pr model. | 19:14 |
| jlk | dunno if your team uses that at all | 19:14 |
| tobiash | we probably have to modularize the codebase more with stable interfaces between (that's my advice, answer is that's difficult/impossible) | 19:14 |
| tobiash | but this is the process of learning | 19:15 |
| jlk | I wish google wasn't still single-repo. People wind up pointing at them and saying "But it works for google!" | 19:15 |
| tobiash | single company repo would probably also not work with git | 19:15 |
| tobiash | google probably has its own proprietary stuff there | 19:16 |
| jlk | nah, it's git+gerrit | 19:16 |
| tobiash | with a single repo? | 19:16 |
| tobiash | TB sized repo? | 19:16 |
| jlk | yeah I don't know how big it is off hand. | 19:17 |
| jlk | https://cacm.acm.org/magazines/2016/7/204032-why-google-stores-billions-of-lines-of-code-in-a-single-repository/fulltext | 19:17 |
| jlk | hrm, that's not git. I'm confused now | 19:18 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove mirror_domain from configure-mirror https://review.openstack.org/490625 | 19:19 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Rename mirror_host to mirror_fqdn in configure-mirrors https://review.openstack.org/490628 | 19:20 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Optionally limit github to protected branches https://review.openstack.org/490134 | 19:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Remove status handling from FakeGithubConnection https://review.openstack.org/490413 | 19:23 |
| Shrews | jeblair: So, while reading the v3 User guide, I keep finding myself over and over wishing I had a complete example up front so that the pieces of the guide fit together better. | 19:23 |
| Shrews | for example, we current have this sentence: "The actual tasks that are run by the job appear in the playbook for that job while the attributes that appear in the Zuul configuration specify information about when, where, and how the job should be run." | 19:24 |
| Shrews | and i'm thinking, "yeah, that's great, but it doesn't make much sense without an example" | 19:25 |
| Shrews | I mean, I personally understand it, but even I had to go look at current examples to get a better picture of what it meant | 19:26 |
| jeblair | Shrews: yeah, i agree. i think there's lots of room for more examples. i have a couple of questions (which may just take trying things out to answer): should we add examples to an appendix and cross-reference them a lot? that lets us have lots of examples ranging in complexity together which can be helpful. should we put simpler examples at the starts of sections (some sections do this already)? should we do both? should the zuul-jobs repo serve as ... | 19:28 |
| jeblair | ... external examples? or should we only occasionally mention zuul-jobs (ie, it's an *additional* resource)? | 19:28 |
| Shrews | So my thinking was that if we conjured up the simplest example for a fake project, then broke it down piecemeal later (even if it doesn't make sense up front), it would help with seeing how the pieces fit as they are discussed | 19:28 |
| jeblair | Shrews: i think that's a great idea, let's try it | 19:29 |
| pabelanger | Shrews: ++ | 19:29 |
| jlk | I concur | 19:30 |
| Shrews | jeblair: if the example could initially leave out reference zuul-jobs and such (but we can introduce it later in a more complicated example), i think that would be better | 19:30 |
| jeblair | Shrews: agreed | 19:30 |
| jeblair | my *guess* as to what we'll end up with is maybe what Shrews suggests in the main narrative, then an appendix with lots more examples (here's how to override a branch, etc), and carefully selected references to zuul jobs | 19:31 |
| jeblair | but i'm really open :) | 19:31 |
| Shrews | jeblair: i think that sounds wonderful | 19:31 |
| jeblair | cool | 19:31 |
| jeblair | Shrews: i'm basically done with my syntax updates in the user guide, so go crazy | 19:32 |
| jeblair | Shrews: the updates i have left to do are mostly in the admin guide | 19:32 |
| jeblair | (i mention this in relation to likely merge conflicts) | 19:33 |
| * jeblair forages for food now | 19:33 | |
| Shrews | jeblair: ok. one of the reasons i'm reading the docs now is because I sorely lag behind in understanding how the config stuff works. I can try to come up with the simplest example, but I'll likely need guidance on it. | 19:33 |
| Shrews | i can poke and prod pabelanger for most of that, maybe :) | 19:34 |
| jeblair | Shrews: cool, i think that's great and we're at the place where we need that. i will be happy to help, both to expand the number of people who grok this, and i may answer some questions in the form of a doc patch. :) | 19:35 |
| tobiash | and some possibly in a bugfix ;) | 19:36 |
| pabelanger | Shrews: for sure | 19:37 |
| jlk | jeblair: re PTG, the plan is to do the cut over on Saturday evening? Should we plan to be there in person, or remotely and then fly in Sunday? Just trying to get a feel of what / where / when I could be the most useful. | 19:42 |
| Shrews | pabelanger: we allow use of the ansible command module, yeah? If so, we could start the simple example with running "command: tox -e pep8" directly, then later use that to introduce the zuul-jobs tox-pep8 role | 19:45 |
| Shrews | err, job rather | 19:47 |
| jlk | We allow command module when using a remote node | 19:48 |
| pabelanger | Shrews: Yup, you could have the untrusted playbook run that, but not on localhost. We limit that. | 19:48 |
| pabelanger | so your playbook could be hosts: ubuntu-xenail to make that clear | 19:49 |
| pabelanger | with a ubuntu-xenial nodeset | 19:49 |
| Shrews | yeah | 19:49 |
| pabelanger | I would avoid using all for the moment | 19:49 |
| pabelanger | and be specific | 19:50 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Doc improvements on user guide https://review.openstack.org/490640 | 19:56 |
| Shrews | getting some minor things out of the way first ^^^ | 19:56 |
| *** harlowja has quit IRC | 19:57 | |
| jeblair | jlk: plan is to do trial cutovers sat and sunday evenings, then final cutover monday morning. that's to accomodate various travel/meeting schedules. so traveling saturday or sunday should be fine (people are doing both) | 20:21 |
| jeblair | jlk: there will be online coordination for at least the sat/sunday trials, so location less important | 20:22 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Doc improvements on user guide https://review.openstack.org/490640 | 20:22 |
| pabelanger | jlk: I plan to be in denver on Saturday | 20:23 |
| pabelanger | and working from hotel bar :) | 20:23 |
| jeblair | or under hotel bar | 20:23 |
| SpamapS | jeblair: oh neat! longer secrets! | 20:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create fetch-tox-output role https://review.openstack.org/490643 | 20:38 |
| pabelanger | jeblair: mordred: where did we end up on namespacing of roles? so^ create fetch-tox-output, but thinking we might want to refer to it as zuul-jobs.fetch-tox-output in project config, to better indicate which project the role is in. | 20:42 |
| pabelanger | project config playbook* | 20:42 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove status handling from FakeGithubConnection https://review.openstack.org/490413 | 20:44 |
| *** jkilpatr has quit IRC | 20:48 | |
| jeblair | pabelanger: i don't recall us deciding to do anything about namespacing roles. | 20:54 |
| *** dkranz_ has quit IRC | 20:55 | |
| *** harlowja has joined #zuul | 21:15 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: add initial documentation style guide https://review.openstack.org/490654 | 21:20 |
| jlk | Okay, I'll plan for Saturday -> Thursday. | 21:29 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 21:32 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 21:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 21:42 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 21:42 |
| mordred | jeblair: ^^ I'm getting closer on that I thnk :) | 21:44 |
| mordred | jeblair: also - I have now learned that the default delimiter for csvfile lookup plugin is ... | 21:44 |
| mordred | wait for it | 21:44 |
| mordred | TAB | 21:44 |
| clarkb | mordred: thats because c is for char not comma | 21:46 |
| clarkb | it makes no sense to me either | 21:46 |
| mordred | clarkb: it's ... I mean ... | 21:46 |
| jeblair | mordred: did you try "import csv as tsv" ? ;) | 21:46 |
| mordred | jeblair: ;) | 21:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: convert some component config to zuul directives https://review.openstack.org/490662 | 21:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 22:01 |
| mordred | jeblair: if the debug things I just added don't help - I may need your help to figure out how to get the ansible logging output | 22:01 |
| mordred | because "2017-08-03 21:51:28,933 zuul.AnsibleJob DEBUG [build: 1f13838b572f487caf5d83590fdf50e0] Ansible complete, result RESULT_NORMAL code 2" isn't super helpful ;) | 22:02 |
| jeblair | mordred: in a pinch, i've set KEEP_TEMPDIRS=1 ttrun ... and then i cd into the jobdir and look at the output there. i think having the test automatically fetch that before the jobdir is destroyed and making a subunit attachment would be cool though. | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat merger and executor config docs https://review.openstack.org/490665 | 22:09 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat web section https://review.openstack.org/490666 | 22:12 |
| jeblair | okay my tab finger is wearing out; time for a break :) | 22:12 |
| mordred | jeblair: yah - that was what I was just starting to try to do ... if I do self.getJobFromHistory() ... | 22:12 |
| mordred | jeblair: I'll have a Job and I also have self.test_root in the test case already ... I'm not 100% sure how to get from that Job to the job-output.txt file | 22:13 |
| mordred | oh | 22:13 |
| mordred | nevermind | 22:13 |
| jeblair | ok | 22:14 |
| mordred | I have ound the thing I need | 22:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Ensure ref-updated jobs run with their ref https://review.openstack.org/488216 | 22:16 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix null ref in github tests https://review.openstack.org/490257 | 22:17 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 22:17 |
| mordred | k. let's see how that goes | 22:17 |
| mordred | jeblair: with self.jobLog was the thing I believe I needed in this case | 22:20 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add project related type hints to gerritconnection https://review.openstack.org/488593 | 22:21 |
| jeblair | mordred: omg i wrote that and forgot about it. | 22:23 |
| jeblair | mordred: sorry! | 22:25 |
| mordred | jeblair: no worries - I'm glad you wrote it! | 22:30 |
| mordred | jeblair: (I believe the test is failing because our use of csvfile is not good - which I think is a result of attempting to reference a path and getting the relative location incorrect) | 22:31 |
| mordred | jeblair: so - basically - it's not failing any of the things it wants to test - so as soon as we can figure out the right path incantation we should be good | 22:31 |
| jeblair | mordred: you mean the test with "file=test.csv" ? | 22:44 |
| jeblair | (and yeah, i think i assumed those would be related to the work root, but i think i now understand that most modules running locally end up running from the playbook dir) | 22:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: add initial documentation style guide https://review.openstack.org/490654 | 22:50 |
| jeblair | mordred: another spurious post_failure (where nothing apparently failed): http://paste.openstack.org/show/617477/ http://logs.openstack.org/65/490665/1/check/tox-docs/bb863fb/ | 22:52 |
| jeblair | mordred: i think we're seeing those often enough we're going to have to dig into it before the ptg; it will be very noticable at production-scale | 22:52 |
| jeblair | Ansible output: b'ERROR! A worker was found in a dead state' | 22:52 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: update format in connections.rst https://review.openstack.org/490672 | 22:55 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Docs: convert some component config to zuul directives https://review.openstack.org/490662 | 23:00 |
| mordred | jeblair: zomg. found it | 23:09 |
| mordred | jeblair: csvfile lookup plugin in ansible does not work in python3 :) | 23:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Add some ansible plugin tests https://review.openstack.org/454396 | 23:11 |
| * mordred is going to be optimistic and think the first patch is going to pass... | 23:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Prevent execution of locally overridden core modules https://review.openstack.org/490216 | 23:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Allow and document use of the uri module from localhost https://review.openstack.org/490215 | 23:12 |
| jeblair | mordred, SpamapS: do you think the 'ERROR! A worker was found in a dead state' error could be related to ssh controlmaster being killed between playbook runs? like maybe it's racing in a way that there's still a socket from the previous playbook at the time the next playbook starts, but is deleted before it's actually used? | 23:29 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat gerrit driver docs https://review.openstack.org/490681 | 23:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: update format in connections.rst https://review.openstack.org/490672 | 23:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat web section https://review.openstack.org/490666 | 23:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat merger and executor config docs https://review.openstack.org/490665 | 23:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Docs: reformat gerrit driver docs https://review.openstack.org/490681 | 23:47 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove state_dir requirement from merger https://review.openstack.org/490682 | 23:48 |
| jeblair | jlk: thanks! ^ :) | 23:48 |
| jlk | hah | 23:49 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!