| mordred | pabelanger: woot! | 00:02 |
|---|---|---|
| mordred | pabelanger: I think you'll be excited :) | 00:02 |
| pabelanger | nice | 00:04 |
| *** jkilpatr has quit IRC | 00:11 | |
| * SpamapS is a slave to tests :-P | 03:02 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: executor: add support for custom ansible_port https://review.openstack.org/468710 | 03:04 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add TenantProjectConfig object https://review.openstack.org/479073 | 03:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Permit config shadowing https://review.openstack.org/479084 | 03:11 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix race in test_queue_rate_limiting_dependent https://review.openstack.org/481824 | 03:18 |
| *** isaacb has joined #zuul | 04:08 | |
| *** isaacb has quit IRC | 04:10 | |
| tobiash | Shrews, jeblair, mordred: let me rephrase to avoid the word abort... | 05:08 |
| tobiash | Shrews, jeblair, mordred: I get this exception every time a client (curl, html client, etc) closes the connection (ctrl+c, close tab) | 05:09 |
| tobiash | Shrews, jeblair, mordred: my intention was to only suppress the exception in this case | 05:09 |
| tobiash | Shrews, jeblair, mordred: now I learned that this cancelederror seems to be a bit more generic than I thought... ;) | 05:10 |
| tobiash | jeblair, mordred: what is the preferred way of synching the zuul and related repos to a local mirror? | 05:15 |
| tobiash | jeblair, mordred: attaching a zuul listening on change merged or just polling once an hour or so? | 05:15 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Add zuul.d configuration split documentation https://review.openstack.org/482405 | 05:20 |
| tobiash | jeblair, mordred: what do you think how the crm114 stuff for log line classification could be incorporated into zuulv3? | 05:42 |
| tobiash | jeblair, mordred: currently (in my v2 setup) I have a service attached to jenkins which is used for log upload and classification (and generating nice looking colored html logs) | 05:43 |
| tobiash | jeblair, mordred: as this crm114 classifying data is something central and single threaded I wonder how this could be included in the zuulv3 setup where the job itself uploads the logs | 05:45 |
| tobiash | jeblair, mordred: sorry for too many questions at once ;) | 05:47 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Prevent loading zuul.(yaml|d) in untrusted-projects https://review.openstack.org/482411 | 06:02 |
| tobiash | mordred: trying to understand this one: https://review.openstack.org/#/c/473966/3/zuul/ansible/library/command.py | 06:48 |
| tobiash | mordred: it breaks every shell/command task while logging nothing except 'Something went horribly wrong during task execution' | 06:50 |
| openstackgerrit | Jamie Lennox proposed openstack-infra/zuul feature/zuulv3: Cleanup on ssh-agent failure https://review.openstack.org/481344 | 06:54 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul feature/zuulv3: Prevent loading zuul.(yaml|d) in untrusted-projects https://review.openstack.org/482411 | 06:55 |
| *** MaciekW has joined #zuul | 07:25 | |
| MaciekW | Hi, is there a way that zuul is not queuing all reviews concerning different branches in gate pipeline? | 07:28 |
| MaciekW | currently I am facing an issue when job from one branch is waiting for job of another branch to finish | 07:28 |
| MaciekW | for the same repository | 07:29 |
| MaciekW | ? | 07:29 |
| SpamapS | MaciekW: is this zuul from master? If so, if the jobs have the same name, they'll share a change queue. | 07:31 |
| SpamapS | MaciekW: you need to name them differently to have different queues. | 07:32 |
| *** openstackgerrit has quit IRC | 07:33 | |
| MaciekW | SpamapS: what do you mean from master? | 07:33 |
| SpamapS | MaciekW: did you just 'git clone' zuul or did you install some other way? | 07:33 |
| SpamapS | if you're using puppet-openstackci then you have "from master" (or more accurately, v2.5) | 07:34 |
| SpamapS | MaciekW: I'm heading to bed, but try giving them different names, like gate-testsuite-branch0 and gate-testsuite-branch1 | 07:34 |
| MaciekW | SpamapS: I have cloned branch 2.5.2 and install using pip | 07:35 |
| MaciekW | SpamamS: ok, I will give it a try thanks | 07:36 |
| *** openstackgerrit has joined #zuul | 08:10 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:10 |
| tobiash | mordred: finally understood what went wrong... ^^ is the fix that works for my deployment | 08:12 |
| tobiash | mordred: with ret=0 (successful task run) it went into 'Something went horribly wrong...' | 08:13 |
| tobiash | mordred: this (or something similar) needs to land before your next executor restart | 08:15 |
| *** isaacb has joined #zuul | 08:23 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:25 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 08:28 |
| tobiash | mordred: now with (indirect) test case ^ | 08:32 |
| *** yolanda has quit IRC | 10:32 | |
| *** yolanda has joined #zuul | 10:33 | |
| *** yolanda_ has joined #zuul | 10:33 | |
| *** yolanda_ has quit IRC | 10:33 | |
| Shrews | tobiash: oh, well i totally misunderstood what you meant by "abort". That's disappointing that occurs, but I think jeblair's suggestion to just have an except clause to catch that exception and ignore it is probably the way to handle it. | 10:34 |
| Shrews | i might dig into aiohttp just to understand why that happens | 10:36 |
| mordred | tobiash: oh - nice catch on the is None - and sorry about that | 11:33 |
| *** jkilpatr has joined #zuul | 11:45 | |
| mordred | tobiash: for logging - I'm planning on doing some of the html transformation like you talk about in the base job | 11:46 |
| mordred | tobiash: we also have a similar log processing service that is central - I believe making sure that we trigger it is a thing we need to solve in the next few weeks | 11:47 |
| mordred | tobiash: in our v2 based system we have a little daemon that listens for 0mq job completed events and when it gets them it puts a job into a gearman so that a worker cna pick it up and process the logs for logstash | 11:49 |
| mordred | tobiash: I'm not sure if we'll wind up adapting that system to listen to an event from zuul (there is a WIP set of patches to add a fedmsg/mqtt reporter to zuul) or whether we'll just have the base job send a notification message after it's uploaded the logs | 11:50 |
| mordred | it's an excellent question :) | 11:50 |
| mordred | tristanC: morning! so - your patch for preventing zuul.yaml in untrusted projects - we actually realized last week that looking for either is quite nice | 11:51 |
| mordred | (mainly this came from reading a line in jeblair's documentation patches) | 11:52 |
| mordred | the distinction we're trying to describe now is that you can use whichever you want- but zuul.yaml is a better choice for repos whose primary purpose in life is describing jobs - lke openstack-infra/zuul-jobs - where the job definitions should be visible and obvious | 11:53 |
| mordred | and .zuul.yaml is a good choice for repos that do other things - like openstack-infra/zuul | 11:53 |
| mordred | jamielennox: +2 with comment on your ssh-add patch | 11:58 |
| mordred | tristanC: oh! I see in your docs patch where the confusion comes from | 11:59 |
| mordred | tristanC: I think instead we should just fix the docs there | 11:59 |
| tristanC | mordred: sure, I'll revert the configloader update | 12:01 |
| mordred | tristanC: when jeblair gets up - I think we should check in with him about the 'only load zuul.yaml from trusted projects' - it seems to me, from reading that paragraph in thedocs, that it's a distinction that may be confusing | 12:03 |
| mordred | but he may also have a good reason for it | 12:03 |
| mordred | tristanC: also - in the meeting yesterday, jamielennox brought up the topic of making the html/javascript portions of zuul-web potentially managed using some more native javascript toolchain | 12:12 |
| mordred | tristanC: jeblair brought up the concern that we want to make sure if we do that such a toolchain is understandable by all the zuul devs - but it's something for us to think about over the next few weeks | 12:13 |
| tristanC | mordred: jeblair: regarding the configloader, as long as it works as documented, i'm fine with it. Currently the only odd thing is that it will load .zuul.yaml from config-projects | 12:14 |
| mordred | tristanC: gotcha. so - I agree, behavior and docs should match | 12:15 |
| tristanC | and trying to fix that, I misread the fact we want zuul.yaml files to be loaded from untrusted-projects, which my last patch broke | 12:15 |
| mordred | tristanC: I'm not sure whether preventing loading .zuul.yaml is the right thing, or updating the docs to allow both in both | 12:15 |
| mordred | tristanC: but I _definitely_ agree that we should fix it in one direction or theother :) | 12:16 |
| tristanC | mordred: jeblair: and regarding the javascript toolchain, I'm not a huge fan of the grunt/npm system | 12:16 |
| mordred | tristanC: good to know - I think zuul developers liking to work with it is important | 12:17 |
| mordred | tristanC: what about only bower? do you think that's useful? | 12:17 |
| tristanC | though it seems like the way to go to pack a collection of js client into a convenient library | 12:17 |
| *** MaciekW has quit IRC | 12:20 | |
| mordred | tristanC: so the way storyboard is split - with a python and a web repo and the web repo using grunt/npm is annoying to work with for you? | 12:21 |
| mordred | (since y'all work with storyboard, I figure you've got a good pov on that) | 12:22 |
| mordred | and that was jeblair's concern - if we used a toolchain that none of us really fully understand that it would make it harder for us to hack on important parts of zuul | 12:23 |
| mordred | oh- apparently bower recommends not using bower | 12:25 |
| tristanC | I have to say I'm having a hard time building the storyboard webclient, either using fedora native tools or the tox venv | 12:26 |
| tristanC | However splitting the code to a 'zuul-webclient' may be convenient in the long term | 12:27 |
| mordred | oh wow. I just learned a LOT of things | 12:29 |
| mordred | tristanC: http://requirejs.org/docs/whyamd.html is worth a quick read | 12:29 |
| mordred | I got to it because bower now tells people to look at yarn and webpack instead - so on the yarn page for jquery: https://yarnpkg.com/en/package/jquery | 12:30 |
| mordred | there is a section on common ways to use jquery | 12:30 |
| tristanC | requirejs looks interesting, at least the app.build.js file doesn't look too complicated | 12:34 |
| mordred | yah - it seems like it's designed to do javascript loading from the internet (since javascript) but with some optional optimization steps | 12:34 |
| mordred | which seems like it's a better design than some of the other systems | 12:34 |
| mordred | since, you know, there's no real specific reason for every website you goto to have its own copy of jquery if maybe require.js in your browser can download and cache it for you | 12:35 |
| *** dkranz has joined #zuul | 12:35 | |
| tristanC | On the other hand, a python-zuulclient would be more fun to start with... :) | 12:39 |
| mordred | :) | 12:39 |
| *** openstackgerrit has quit IRC | 12:47 | |
| lennyb | hi, is there a way for zuul to be triggered by a specific patchset number only or by a specific owner? | 13:15 |
| mordred | lennyb: not to my knowledge - what are you trying to accomplish? | 13:19 |
| *** openstackgerrit has joined #zuul | 13:19 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Handle aborted websocket client connection cleanly https://review.openstack.org/482560 | 13:19 |
| Shrews | tobiash: can you try out that patch ^^^^ ? | 13:20 |
| lennyb | mordred, I am trying to run CI on a specific os-vif patchset/owner. Until this patchset will be merged there is no need to run CI on other commits and commenting | 13:21 |
| lennyb | this will add irrelevant noise and false alarms | 13:21 |
| lennyb | mordred, I know I can filter only a relevant files as a workaround, but I prefer an 'owner' filtering | 13:22 |
| Shrews | jlk: fedora 26 upgrade now gives me docker-compose 1.14.0. will try z8s again a bit later | 13:30 |
| pabelanger | morning! | 13:32 |
| mordred | lennyb: yah - we don't really have any concept of filtering on patchset owner - the problem you're trying to solve becomes better with zuul v3 which is not too far away | 13:59 |
| mordred | lennyb: but for now, I think the best choice you have is to define a job in the experimental pipeline - then you can trigger it by leaving the comment "check experimental" - so you dont have to spam other people with the results until you're read | 13:59 |
| mordred | ready | 13:59 |
| mordred | pabelanger: morning! | 13:59 |
| lennyb | mordred, thanks. | 13:59 |
| mordred | pabelanger: so - biggest news is that most of the jobs have been moved into zuul-jobs | 14:13 |
| mordred | pabelanger: also, the content from the jenkins scripts dir has been directly copied in to them - and I've started refactoring those to be more native ansible instead of having a giant embedded script | 14:14 |
| mordred | pabelanger: I also started getting some of the output of the setup scripts (like the network info) into a report file rather than just as stdout | 14:16 |
| mordred | pabelanger: and finally we realized that all of the playbooks start with /home/zuul as the working directory since that's the home dir of the user - so we can define paths relative to that | 14:16 |
| mordred | pabelanger: in several of the single-repo roles we're defining a zuul_work_dir in defaults/main.yaml - that's so that we can easily run the roles on a different repo | 14:17 |
| mordred | pabelanger: you can see an example of this in the tox-py35-on-zuul job - which is running tox -epy35 on the zuul repo - but on changes to the zuul-jobs repo | 14:19 |
| mordred | pabelanger: that let's us test that changes to zuul-jobs will work when run with another project (there was an issue we ran in to just running tox job on zuul-jobs since zuul-jobs doens't have a test-setup script - the bindep and test-setup were being called out of order) | 14:20 |
| mordred | pabelanger: we've mostly sorted out the issues with logging and debugging things - there are still one or two gotchas, but for the most part errors in job configs can all be debugged through the results returned to the browser | 14:21 |
| mordred | pabelanger: although there is a bug that seems to be happening right now in that changes to zuul seem to be being run with stale versions of the job definitions from zuul-jobs - which jeblair is going to look in to when he wakes up | 14:22 |
| mordred | pabelanger: current todo-list items are finishing getting your configure-mirror role applied (but putting it in zuul-jobs instead of openstack-zuul-roles and attaching it to the unittest base job for now so we can more easily hack on it) | 14:23 |
| mordred | pabelanger: and refactoring run-tarball, run-cover and tox tasks/main.yaml to be ansible and not bash | 14:24 |
| mordred | pabelanger: jlk has offered to help with those tasks - although looking at them there's a non-zero amount of work we need to do in terms of thinking about the openstack-specific vs. general case | 14:25 |
| mordred | so I think it might be nicer to find a different jobs task for jlk to help with since I think you and I are going to need to dig in to a bunch of infra-specific things on those | 14:25 |
| mordred | pabelanger: the bindep role is mostly good- although I do think it could be improved by doing an include: "{{ ansible-pkg-mgr }}.yaml" in the loop and having each one of those have an even smaller shell snippet | 14:26 |
| mordred | pabelanger: I did not use the actual ansible package management modules because the apt module only accepts single package names and bindep returns a list - so my OCD wouldn't let me write code that called apt 30 times | 14:27 |
| mordred | pabelanger: it makes me want to make a patch to upstream apt module to make it accept a list of packages for a one-shot | 14:27 |
| mordred | since I _do_ think that writing those to use the pkg modules will be nicer | 14:28 |
| pabelanger | okay to all of that | 14:31 |
| pabelanger | I'll check out the bindep module, I've already created http://git.openstack.org/cgit/openstack/ansible-role-bindep a while back too. That just gets bindep on to the system, via pip/git/pkgmanager | 14:32 |
| pabelanger | been using it for a while, works well | 14:32 |
| pabelanger | mordred: I'll start looking at zuul-jobs now and see what else needs to be done with configure-mirror | 14:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Split bindep package install into tasks https://review.openstack.org/482584 | 14:39 |
| mordred | pabelanger: neat - so - might very well be a good idea to see if we can combine the two and then just use ansible-role-bindep | 14:40 |
| mordred | pabelanger: the thing I've got written now should be totally re-usable | 14:40 |
| mordred | pabelanger: one main difference is that if it finds it needs to install bindep, it does it in a throwaway virtualenv in /tmp | 14:41 |
| mordred | pabelanger: so that we dont' wind up installing globally | 14:41 |
| mordred | pabelanger: we might want to put that behavior behind a parameter flag for the general role | 14:41 |
| mordred | pabelanger: I started poking at your configure-mirror stuff into zuul-jobs yesterday - let me push up what I've got real quick | 14:41 |
| pabelanger | mordred: excatly, I do the same thing with my bindep role, it exposes to the use how to install bindep, either as global, user or virtualenv: http://git.openstack.org/cgit/openstack/ansible-role-bindep/tree/tasks/install/pip.yaml. I was planning on updating it to support your bindep task from upstream ansible/ansible once that landed too | 14:43 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Add phase and count to variables passed in to playbooks https://review.openstack.org/481942 | 14:43 |
| pabelanger | mordred: it is also gating on centos / fedora / all ubuntu :) | 14:43 |
| mordred | pabelanger: cool :) | 14:44 |
| mordred | pabelanger: well, maybe let's make task-one getting the two synced and the jobs updated to use that role | 14:44 |
| mordred | pabelanger: which means we shold probably add ansible-role-bindep to zuul's main.yaml | 14:45 |
| mordred | pabelanger: oh - I just realized something that might make this harder ... or maybe not ... | 14:45 |
| mordred | pabelanger: if the role installs bindep into a virtualenv, or finds it in a virtualenv - it's going to need to pass ansible_python_interpreter when it runs the bindep module | 14:46 |
| mordred | pabelanger: but - anyway - we should update the PR for the bindep module upstream -but we can also just copy it in to the role for now - and use that as a good way to test that it works | 14:47 |
| mordred | pabelanger: then remove it when the bindep module lands upstream | 14:47 |
| pabelanger | mordred: Ya, I still need to do some updates for that too. Here is the latest version of that logic I am using in ansible-role-zuul: http://git.openstack.org/cgit/openstack/ansible-role-zuul/tree/tasks/install/pip.yaml I need to read up on ansible_python_interpreter stuff you recently found | 14:47 |
| *** openstackgerrit has quit IRC | 14:48 | |
| *** openstackgerrit has joined #zuul | 14:52 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 14:52 |
| mordred | pabelanger: there's an updated version of your openstack-zuul-roles patch for zuul-jobs ^^ | 14:52 |
| mordred | (let's see how it runs! :)) | 14:53 |
| pabelanger | mordred: cool, IIRC, I had it working just before ansiblefest | 14:53 |
| *** isaacb has quit IRC | 14:59 | |
| mordred | pabelanger: I seem to have broken something: https://review.openstack.org/#/c/482593/ | 15:03 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 15:05 |
| mordred | found it | 15:05 |
| pabelanger | cool | 15:07 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add tox-docs jobs https://review.openstack.org/482596 | 15:09 |
| pabelanger | jeblair: ^ is an interesting one. I had to remove our header format because zuul-sphinx would error: SEVERE: Unexpected section title. | 15:09 |
| mordred | pabelanger: SO ... we put that one in openstack-zuul-jobs because how we run docs is pretty weird in opentsack land | 15:09 |
| mordred | pabelanger: oh - bleh | 15:10 |
| pabelanger | mordred: Oh, that patch just runs the gate job. Should it be a different zuul-job? | 15:11 |
| mordred | pabelanger: yah - we have a openstack-doc-build job for openstack-style docs jobs | 15:11 |
| mordred | we should likely update zuul to use that one too | 15:12 |
| pabelanger | k, let me change | 15:12 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Add openstack-doc-build job for documentation testing https://review.openstack.org/482596 | 15:13 |
| pabelanger | Ya, see the issue with zuul_sphinx, we are prepending 4 spaces, which is breaking headers | 15:15 |
| pabelanger | maybe we should stop doing that and just assume the readme as standard rst file | 15:16 |
| *** isaacb has joined #zuul | 15:23 | |
| jlk | mordred: saw backscroll. I'm still happy to review the openstacky jobs. | 15:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: EOL ubuntu-precise for dsvm job https://review.openstack.org/482613 | 15:58 |
| pabelanger | oh, nice. we now have logs on RETRY_LIMIT | 16:13 |
| *** isaacb has quit IRC | 16:16 | |
| jeblair | \o/ :) | 16:17 |
| *** pabelanger has quit IRC | 16:18 | |
| *** pabelanger has joined #zuul | 16:18 | |
| jeblair | pabelanger: yeah, the included docs are inside of an indented block right now, we're not expecting them to have section headings. they should be able to have any other rst though. | 16:18 |
| pabelanger | jeblair: ack, I've updated the RST to remove them for now | 16:19 |
| jeblair | pabelanger: (it would be difficult (but possible) to permit them to have section headings since we have to maintain an ordering across lots of files) | 16:19 |
| pabelanger | ack | 16:20 |
| mordred | good to know that they can't have section headings | 16:25 |
| mordred | pabelanger: so - looking at the source code in ansible, it turns out you CAN give a list of packages to apt | 16:27 |
| mordred | it's just not documented | 16:27 |
| jlk | oh yeah | 16:27 |
| jlk | apt/yum/etc most the packaging modules do magic to turn with_items into a single transaction | 16:27 |
| pabelanger | mordred: Ya, I thought so. I use a list of package for some of existing ansible roles | 16:27 |
| * mordred is making a doc patch | 16:28 | |
| mordred | jlk: with_items can be turned into a single transaction by a module? | 16:29 |
| pabelanger | mordred: btw: why apt directly and not package? Been using package for a while on 3 operating systems and seems okay so far: http://git.openstack.org/cgit/openstack/ansible-role-zuul/tree/tasks/install.yaml#n20 | 16:30 |
| mordred | well - I would have said no reason ... but it turns out package does _not_ squash with_items into a single list by default | 16:34 |
| mordred | but - we can configure it to by defining squash_actions in our ansible.cfg | 16:34 |
| mordred | I will try it out | 16:34 |
| jeblair | mordred: [keep in mind we may not want to set too many exotic options in ansible.cfg] | 16:35 |
| pabelanger | Ah, for tuning. Ya, I haven't done too much of that yet | 16:35 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Add docs for web server component https://review.openstack.org/482630 | 16:42 |
| mordred | jeblair: indeed | 16:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Improve logging on reconfiguration https://review.openstack.org/482635 | 17:01 |
| jeblair | mordred: ^ i'd like to land that and restart everything soon. it looks like there was an error which caused us not to load the new configuration after the unittest post fix landed, but i do not know what the error was. :( | 17:02 |
| jeblair | mordred: with any luck, it's actually frequent and it may happen again. | 17:02 |
| tobiash | Shrews: just tested and it works: http://paste.openstack.org/show/615053/ | 17:03 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 17:03 |
| Shrews | tobiash: awesome! please leave a vote on the change for us | 17:03 |
| Shrews | 482636 is a temporary cure for my OCD'ing on the docs I had to touch for 482630 | 17:04 |
| tobiash | Shrews: I was already typing ;) | 17:04 |
| Shrews | tobiash: sorry i misunderstood the whole "abort" thing in the other review | 17:04 |
| tobiash | Shrews: no worries | 17:05 |
| mordred | jeblair: lgtm | 17:07 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: Cleanup components doc https://review.openstack.org/482636 | 17:08 |
| jeblair | mordred: i'm going to go aheand and +3 the logging change then so it gets in before any more job work lands | 17:17 |
| mordred | jeblair: ++ | 17:17 |
| mordred | pabelanger, jlk: ok - so I have learned many things - the first and most important of which is that one cannot use until with include: - which means the only way we can actually do the "retry install command in a loop" logic is with a shell script | 17:18 |
| mordred | pabelanger, jlk: we can get rid of the retry-install-loop and instead just have the final bindep command throw an error which would cause zuul to retry the job | 17:19 |
| Shrews | mordred: jeblair: before i get to work on the static page handling from zuul-web, should we consider changing its default port from 9000 to 80 before folks start using it? | 17:19 |
| pabelanger | mordred: still catch up on things, but was is the reasoning for the rety logic inside our playbooks, over try once and fail? | 17:19 |
| Shrews | or even 8080, or does it really matter? | 17:20 |
| mordred | pabelanger: I believe the reason for having that loop being package mirrors being temporarily unreachable - and retrying an install is fairly cheap | 17:20 |
| clarkb | Shrews: I wouldn't use a low port to avoid the root required issue that we had/have with finger. | 17:20 |
| mordred | Shrews: I imagine people will use it behind an apache or nginx rather than directly - so I think 8080 or 9000 is fine | 17:20 |
| Shrews | ok, i'll just leave it alone then | 17:21 |
| pabelanger | mordred: Right, bindep does the loop. I cannot remember, do we also do that for package installs in devstack too today? | 17:21 |
| clarkb | pabelanger: if devstack can't install it fails | 17:21 |
| mordred | pabelanger: well - the bindep script in slave_scripts does the loop which is why I kept it here | 17:21 |
| mordred | clarkb: do you have an opinion on whether to keep the loop in the ansible or to fail and let zuul retry the job since it'll be a pre-playbook failure? | 17:22 |
| clarkb | mordred: I think the job more likely to eventually succeed if reschedled on a new node | 17:22 |
| mordred | ++ | 17:23 |
| mordred | yah - I think the retry-on-pre-failures feature makes this better if we just fail | 17:23 |
| clarkb | I expect that the vast majority of those failures will be due to node or region loacl networking blips | 17:23 |
| mordred | yup | 17:23 |
| pabelanger | clarkb: mordred: Right, I am wondering if we should consider making package installs consistant in zuulv3 playbooks. Then we can either write a simple role to handle package installs and have everything just use it over multiple places for rety logic | 17:24 |
| mordred | pabelanger: I'd rather wait before abstracting that out just yet - I'm not sure how many places we're going to need that directly in job playbooks | 17:24 |
| clarkb | pabelanger: the problem with devstack is a significant responsibility of devstack is doing thsoe installs | 17:25 |
| mordred | yah | 17:25 |
| clarkb | pabelanger: so you can't stop doing it there nad it will fail time to time | 17:25 |
| pabelanger | fwiw: I've tend to default to no retry logic in roles, to avoid to much business logic with in them | 17:25 |
| mordred | that's the thing - the things we're testing have the responsibility of installing the packages | 17:25 |
| mordred | pabelanger: ++ | 17:25 |
| mordred | the only place where we really own that responsibility is bindep - but I think having that fail and letting the job get rescheduled is better | 17:25 |
| * mordred makes new version of patch | 17:25 | |
| pabelanger | Ya, that seems right | 17:25 |
| pabelanger | since we likely hit a networking issue anyways | 17:26 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Use package module to install bindep packages https://review.openstack.org/482584 | 17:52 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Simplify bindep logic removing fallback support https://review.openstack.org/482650 | 17:52 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Enable verify_host for synchronize https://review.openstack.org/482655 | 18:13 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 18:38 |
| pabelanger | mordred: pushed up a update to configure-mirrors^ However, I am starting to think we might need to make that more generic. EG: configure-pip-mirrors, configure-apt-mirrors, etc. Otherwise, I can see that role growing very large and complex over time | 18:39 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Improve logging on reconfiguration https://review.openstack.org/482635 | 18:41 |
| mordred | pabelanger: yah- it's also possible that it wants to just be an openstack specific role (in fact, that's quite probable) | 18:55 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Create configure-mirrors role https://review.openstack.org/482593 | 18:56 |
| pabelanger | mordred: right, if openstack specific is zuul-jobs still the place? | 18:56 |
| jlk | o/ | 18:57 |
| jlk | Working in Seattle today, at a fancy bike shop/cafe. Meeting up with other seattle openstackers, like deva and morgan and Anne | 18:57 |
| pabelanger | jlk: nice, have an openstack meetup tonight too | 18:58 |
| mordred | pabelanger: for now - it's easier for us to iterate on it on the unittest base job | 18:58 |
| pabelanger | mordred: understood | 18:58 |
| jlk | This place serves waffles, dis gonna be good. | 18:58 |
| mordred | pabelanger: so -yah - I think I was trying to parameterize it too much - let's go ahead and just say that'll be an opnestack role that lives in openstack-zuul-roles once we're done, yeah? | 18:59 |
| pabelanger | mordred: wfm. We could easily create an apt mirror role from it. That should like in zuul-jobs I think | 19:00 |
| morgan | jlk: i actually wont be making it today, i don't think | 19:00 |
| morgan | sadly | 19:00 |
| tobiash | jeblair: if you're going to restart the zuul-executor you'd need https://review.openstack.org/#/c/482442/ | 19:00 |
| jlk | morgan: oh boo. | 19:00 |
| jlk | I was going to regale you with my story of 140 miles of gravel. | 19:00 |
| mordred | jlk: heya - we were just talking about you in the infra meeting in #openstack-meeting | 19:04 |
| jlk | oh poop | 19:04 |
| jlk | sorry I missed it | 19:04 |
| Shrews | oh dang. /me forgot | 19:04 |
| mordred | jlk: tl;dr - congrats, welcome aboard, and condolences for new responsibility | 19:06 |
| jlk | I... what? | 19:06 |
| jeblair | tobiash: ah thanks. i +3d it and will wait for it to land before restarting. ;) | 19:07 |
| tobiash | :) | 19:07 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Check ret for None in zuul_run_command https://review.openstack.org/482442 | 19:23 |
| jeblair | yay! ^ | 19:47 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Enable verify_host for synchronize https://review.openstack.org/482655 | 19:55 |
| jeblair | tobiash, mordred, clarkb: hrm. i wonder if we could maybe make the gerrit driver smart enough to figure out what the right case is. | 19:56 |
| clarkb | gerrit does let you query the gerrit version at least | 19:57 |
| clarkb | so worst case we could switch based on that maybe? | 19:57 |
| jeblair | maybe we could just look at the approvals required for the change, then heuristically convert the lowercased versions from the config into what gerrit reports for them | 19:57 |
| tobiash | jeblair: that sounds doable, but it feels like a hack | 19:58 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 19:59 |
| tobiash | I'd vote to be consistent with the gerrit project config (==same cased) | 19:59 |
| jeblair | tobiash: *nod* | 20:04 |
| pabelanger | jeblair: mordred: configure-mirror patch passed: https://review.openstack.org/#/c/482593/ | 20:05 |
| mordred | pabelanger: woot! now - did it actually configure mirrors? | 20:06 |
| pabelanger | mordred: YES! | 20:06 |
| mordred | \o/ | 20:06 |
| pabelanger | mordred: I had to update a few things | 20:06 |
| mordred | jeblair: you've restarted the zuul executor recently, right? | 20:07 |
| mordred | cause this: https://review.openstack.org/#/c/481825/ didn't seem to work - I'll need to dig further | 20:08 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 20:10 |
| *** jkilpatr has quit IRC | 20:11 | |
| jeblair | mordred: not yet | 20:12 |
| jeblair | mordred: i'm about to | 20:12 |
| mordred | jeblair: ah - ok. then it's not a problem :) | 20:16 |
| * tobiash EODs | 20:17 | |
| pabelanger | Nice, have chrome opening finger:// URL into termals now | 20:18 |
| * jlk offers up his first +2, on a zuul-jobs change. | 20:18 | |
| pabelanger | Yay | 20:19 |
| jeblair | i restarted zuulv3.openstack.org (scheduler and executor) | 20:23 |
| mordred | jeblair: awesome | 20:25 |
| mordred | pabelanger: you did not git add | 20:26 |
| jeblair | is there a zuul-jobs change ready to land now that we've restarted? :) | 20:27 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using http://keyserver.ubuntu.com for GPG keys https://review.openstack.org/482683 | 20:29 |
| jeblair | or, i should say, a change to a zuul.yaml | 20:29 |
| pabelanger | mordred: ya, fixed now | 20:29 |
| mordred | jeblair: yes | 20:29 |
| pabelanger | mordred: also added full path to be safe | 20:29 |
| mordred | jeblair: oh - no ... one sec | 20:30 |
| mordred | jeblair: I was going to say: https://review.openstack.org/#/c/482593/4 (which could use a +3) | 20:30 |
| mordred | jeblair: https://review.openstack.org/#/c/482596/ | 20:30 |
| mordred | jeblair: adds a job to zuul-jobs | 20:30 |
| *** jkilpatr has joined #zuul | 20:32 | |
| Shrews | jlk: so, docker-compose now does things, but it looks like the running zuul services are borked. seems they all report "ImportError: No module named 'zuul' | 20:41 |
| Shrews | " | 20:41 |
| jlk | well that's odd. | 20:42 |
| jlk | in the container, can you show the mount paths? | 20:42 |
| Shrews | lemme restart | 20:42 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul feature/zuulv3: Avoid using apt-add-repository https://review.openstack.org/482683 | 20:42 |
| jlk | while they're oging on one terminal, you should be able to docker exec into them, like | 20:43 |
| jlk | docker exec -u root -i -t z8s_zuul-scheduler_1 /bin/bash | 20:43 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add openstack-doc-build job for documentation testing https://review.openstack.org/482596 | 20:43 |
| Shrews | jlk: well, no i can't. they are all in a "restarting" state so i can't get a shell | 20:43 |
| jlk | oooh drat | 20:44 |
| jlk | um | 20:44 |
| jlk | hrm. | 20:44 |
| Shrews | zk is running, so yay | 20:44 |
| jlk | what's your execution line look like? | 20:44 |
| Shrews | docker-compose -f docker-compose.yaml -f devel.yaml up zuul-zookeeper zuul-scheduler zuul-executor | 20:44 |
| Shrews | you gave me that a week or so ago. | 20:45 |
| jlk | Did you modify devel.yaml at all to define where your zuul checkout is? | 20:45 |
| Shrews | jlk: no. i set the ZUUL_SRC env variable | 20:46 |
| jlk | let me see if that does what I expect it to. | 20:46 |
| jlk | So I just did ZUUL_SRC=~/src/z8s/ docker-compose -f docker-compose.yaml -f devel.yaml up zuul-scheduler and that broke, because src/z8s is not where zuul is | 20:47 |
| jlk | I wonder if it's not taking it from ENV properly, can you try setting it specifically when you execute? | 20:47 |
| Shrews | jlk: sure. do i need to remove the existing containers? | 20:48 |
| jlk | does "docker ps" show them as running? If not, then I don't think so | 20:48 |
| Shrews | no. they're inactive | 20:48 |
| jeblair | mordred: yay! i have an error! http://paste.openstack.org/show/615072/ | 20:48 |
| jeblair | i will look into that after a short break | 20:49 |
| jlk | Shrews: okay, then trying again should get the mount | 20:49 |
| Shrews | jlk: nope. same. http://paste.openstack.org/show/615073/ | 20:50 |
| Shrews | jlk: do i need to do anything with the python version? | 20:52 |
| jlk | blarg, okay, really fallback, maybe try setting it specifically _in_ devel.yaml | 20:52 |
| jlk | also you might try changing docker-compose.yaml for what the "command" is on one of the containers so that you can get it to stay running, so you can shell into it and see what's going on with mount paths. | 20:53 |
| mordred | jlk: woot! | 20:54 |
| Shrews | jlk: that didn't work either | 20:54 |
| jlk | there _should_ be a file: | 20:55 |
| jlk | cat /srv/zuul/lib/python3.5/site-packages/zuul.egg-link | 20:55 |
| jlk | /zuul | 20:55 |
| jlk | and /zuul should be what gets mounted in, on top of the "installed" zuul. | 20:55 |
| jlk | er, the git checkout of zuul done during the build phase. | 20:56 |
| Shrews | $ cat /srv/zuul/lib/python3.5/site-packages/zuul.egg-link | 20:58 |
| jlk | your client ate the /zuul part I think | 20:58 |
| Shrews | yep | 20:58 |
| jlk | (since it started with a / ) | 20:58 |
| Shrews | but it was /zuul | 20:58 |
| jlk | and what's in /zuul/ ? | 20:58 |
| Shrews | oh | 20:59 |
| Shrews | ls: cannot open directory '/zuul': Permission denied | 20:59 |
| jlk | oooh? | 20:59 |
| jlk | I wonder if there's some selinux fun happening | 20:59 |
| Shrews | $ ls -ld /zuul | 21:00 |
| Shrews | drwxrwxr-x. 11 zuul zuul 4096 Jul 11 16:41 /zuul | 21:00 |
| Shrews | y | 21:00 |
| Shrews | so, i have a 'zuul' user on my local machine with id 1001 | 21:01 |
| jlk | zuul in my container is 1000 | 21:01 |
| Shrews | yeah, just saw. | 21:02 |
| jlk | and I'm user 500 outside the container | 21:02 |
| jlk | but there is also a small VM between me and the container, because Mac. | 21:02 |
| jlk | I honestly don't know what docker on Fedora does to handle user mounted dirs into containers | 21:02 |
| Shrews | whoami says i am zuul, and i own /zuul, yet i cannot read /zuul | 21:03 |
| jlk | I assume if you take out the -f devel.yaml part, it'll work, because you'll have access to the in-container files in /zuul/ | 21:03 |
| jlk | Shrews: I'm guessing there are some selinux failures preventing the access | 21:03 |
| jlk | Shrews: maybe http://www.projectatomic.io/blog/2015/06/using-volumes-with-docker-can-cause-problems-with-selinux/ ? | 21:04 |
| Shrews | jlk: oh, well that got me further. now at: Exception: Unable to locate config file in ['/run/secrets/zuul.conf'] | 21:11 |
| jlk | ah right | 21:11 |
| jlk | so. | 21:11 |
| Shrews | gotta do the same for the z8s directory maybe? | 21:11 |
| jlk | there's a zuul.conf example file in the repo | 21:11 |
| jlk | you'll need to copy it to zuul.conf.secret | 21:12 |
| jlk | and then modify it for your tastes. | 21:12 |
| jlk | same with clouds.yaml(.secret) | 21:12 |
| Shrews | i did that | 21:12 |
| Shrews | not for clouds.yaml, but i'm not starting nodepool | 21:12 |
| jlk | ah, hrm. | 21:13 |
| jlk | maybe it is failing to get that file because of selinux, no idea on that one | 21:13 |
| Shrews | and i didn't really modify the zuul.conf.secret except to comment out the github driver | 21:13 |
| Shrews | lemme do the same command for the z8s dir | 21:13 |
| jlk | https://bugzilla.redhat.com/show_bug.cgi?id=1440389 ? | 21:14 |
| openstack | bugzilla.redhat.com bug 1440389 in docker "Can not create services with secrets in swarm mode" [Unspecified,Closed: errata] - Assigned to amurdaca | 21:14 |
| Shrews | jlk: yup, that gets me running, except it doesn't like that i commented out the github stuff | 21:15 |
| jlk | oh, because a lot of the config is coming from a github repo | 21:16 |
| Shrews | jlk: so i HAVE to give it a github api key? | 21:17 |
| jlk | oh, no. | 21:17 |
| jlk | er. | 21:17 |
| jlk | You shouldn't have to. What was the error you saw/ | 21:17 |
| Shrews | jlk: http://paste.openstack.org/show/615076/ | 21:18 |
| jlk | I think you migh tjust need [connection github]\ndriver = github | 21:18 |
| Shrews | k. i had that commented out | 21:18 |
| Shrews | thought i was being clever | 21:18 |
| jlk | ah I see. | 21:18 |
| jlk | ooooh | 21:19 |
| jlk | it's because of root/etc/zuul/tenant.yaml that file references a connection name of "github" | 21:19 |
| jlk | Maybe I should make this file a secret as well. | 21:20 |
| Shrews | http://paste.openstack.org/show/615078/ | 21:20 |
| jlk | whoh... | 21:21 |
| jlk | well, it's definitely reading content from github | 21:22 |
| jlk | I wonder if that _just_ broke... | 21:22 |
| Shrews | can i blame mordred?? plz oh plz | 21:22 |
| jlk | hrm, it's still working here. | 21:23 |
| Shrews | perhaps it's my local checkout of zuul | 21:23 |
| Shrews | lemme update to HEAD on feature/zuulv3 | 21:23 |
| jlk | I'm wondering where it's finding a reference to "openstack-infra/zuul" | 21:24 |
| Shrews | hrm, no. same error | 21:24 |
| jlk | oh it's in zuul.yaml there | 21:24 |
| jlk | wtaf. | 21:24 |
| jlk | let me update my zuul too | 21:26 |
| jlk | I thikn this is the new config shadowing thing | 21:27 |
| jlk | hahah yeah, I get that now. | 21:27 |
| jlk | okay neat. Hey jeblair | 21:27 |
| Shrews | \o/ | 21:27 |
| mordred | jeblair: looks like people have found your new work! :) | 21:28 |
| jeblair | it's nice to have your work appreciated | 21:28 |
| Shrews | mordred: you are off the hook... this time | 21:28 |
| Shrews | :-P | 21:28 |
| jlk | so the upshot here is that we have to add the referenced project to the tenant config | 21:29 |
| mordred | oh - actually - this IS my fault | 21:29 |
| jeblair | jlk: can you add another layer of tldr? or maybe a link to some config for me to look at? | 21:30 |
| jlk | ah right | 21:30 |
| jlk | so, openstack-infra/zuul-jobs grew a | 21:30 |
| jlk | required-projects: | 21:30 |
| jlk | - name: openstack-infra/zuul | 21:30 |
| mordred | jlk, Shrews: I put a job definition in zuul-jobs that runs jobs on zuul-jobs patches against openstack-infra/zuul | 21:30 |
| jeblair | oh, ha. yes. | 21:30 |
| jlk | for the tox-py35-on-zuul job | 21:30 |
| mordred | but that's bad for reconsuming zuul-jos | 21:30 |
| mordred | yup. that needs to move | 21:30 |
| mordred | sorry - didnt' think of the ramifications of that | 21:30 |
| jeblair | yeah, i guess we should put that in project-config | 21:30 |
| mordred | jeblair: ++ | 21:30 |
| * mordred makes patch real quick | 21:31 | |
| jeblair | neat we learned a thing | 21:31 |
| jlk | Shrews: YOU FOUND A BUG! | 21:31 |
| Shrews | jlk: no, i tripped over a bug while i was lost in the woods | 21:31 |
| jlk | so uh | 21:31 |
| jeblair | we added that because we want to run jobs on zuul-jobs which actually exercise the code in them (it's just 'jobs' repo, so it doesn't have any real unit tests to run, but it has all the code for running unit tests) | 21:31 |
| jlk | to get things working for you right this minute, you could add openstack-infra/zuul to the list of projects in root/etc/zuul/tenant.yaml and rebuild the containers | 21:32 |
| mordred | jeblair: I think I need to put it in the zuul repo | 21:32 |
| jeblair | so the idea is to run unit tests on some random project. ie, zuul itself. using those changes | 21:32 |
| jeblair | mordred: oh this is about to get really tricky | 21:32 |
| mordred | jeblair: if I put it in project-config, project config doesn't have tox-py35 base job | 21:32 |
| mordred | but I guess I can't do that | 21:32 |
| mordred | hrm | 21:32 |
| jlk | feels whiteboardy | 21:32 |
| jeblair | mordred: it has to be a trusted repo in order for us to add it to the project definition. but if it's in a trusted repo, it isn't going to use speculative changes. | 21:33 |
| jeblair | mordred: also your thing is probably an issue. | 21:33 |
| mordred | jeblair: oh - wait - we want to use object filtering for other people using zuul-jobs right? | 21:34 |
| mordred | jeblair: like, they shouldn't be consuming the part where .. nope, nevermind | 21:34 |
| jeblair | mordred: i was hoping to avoid it, but that is a solution to this | 21:34 |
| mordred | jeblair: is it? it's the job definition that's a problem? | 21:34 |
| mordred | jeblair: are those late-binding? | 21:34 |
| jeblair | mordred: you're right. it's only a solution to the project dfn, not the job defn. | 21:34 |
| mordred | Shrews, jlk: yah - if y'all can just add zuul to your tenant.yaml for a minute while we're ponder this | 21:35 |
| Shrews | jlk: rebuilding now | 21:37 |
| Shrews | sorry to cause such heavy pondering | 21:38 |
| mordred | Shrews: no - this is good! this is a lovely edge case :) | 21:38 |
| Shrews | but ya'll love it | 21:38 |
| jlk | Adding that project definitely worked | 21:39 |
| Shrews | your machine is faster than mine | 21:39 |
| jlk | the build should have been really quick since only one file changed | 21:39 |
| jlk | docker cache is a neat thing | 21:39 |
| jeblair | mordred: one option might be to late bind required-projects as you suggest. we could bind it when we run the job instead of on config load. that means we lose detection of a class of config errors. that could be annoying if it were a gate job because you'd have to remove it from the project config before merging a fix. that's not quite a gate wedge, but it's close. | 21:40 |
| jlk | "docker-compose build" was done in a few seconds. | 21:40 |
| Shrews | oh, i rmi'd the z8s images | 21:40 |
| jlk | ah. | 21:40 |
| Shrews | doh | 21:40 |
| Shrews | jlk: yup, zuul is zuuling the thing | 21:41 |
| Shrews | things, even | 21:41 |
| jlk | \-/ | 21:41 |
| jeblair | mordred: we could make that an error only in the case where the job is included in a project-pipeline definition. that feels slightly better, but still not great. | 21:41 |
| Shrews | jlk: many thanks | 21:41 |
| jlk | this was fun! If you'd take a moment at some point and toss things into the README that you ran into, that'd be totes cool | 21:41 |
| Shrews | jlk: like, contribute back and stuff??? | 21:42 |
| jlk | crazy talk | 21:42 |
| Shrews | i mean... geez | 21:42 |
| Shrews | will do. EOD, so tomorrow | 21:42 |
| jeblair | mordred: what if we put it in openstack-zuul-jobs? | 21:43 |
| mordred | jeblair: well - we could - but then we lose the "iterate on jobs and roles in zuul-jobs and test them on other projects" | 21:43 |
| mordred | do be fair - that wasn't really a thing we planned | 21:43 |
| mordred | so it's not like it's a *TERRIBLE* loss - but it is a neat happy feature | 21:44 |
| jeblair | mordred: that's an untrusted repo, so it'll get speculatively executed. it comes after zuul-jobs, so the base job will be defined. we can add it to the project-pipeline in project-config. | 21:44 |
| jeblair | mordred: i'm not sure what loss you're describing | 21:44 |
| mordred | jeblair: oh - you're saying define the job in openstack-zuul-jobs | 21:44 |
| jeblair | mordred: yep | 21:44 |
| mordred | jeblair: but put it in the pipeline in project-config | 21:44 |
| jeblair | yep | 21:44 |
| mordred | yah - I think we can do that | 21:44 |
| jlk | the ultimate goal here though, is that you want a change in zuul-jobs to trigger a test of something _consuming_ zuul-jobs | 21:44 |
| jeblair | jlk: yep | 21:44 |
| mordred | but only in this zuul | 21:45 |
| mordred | we dont' want to make other zuul-jobs consumers need those repos too | 21:45 |
| jlk | and as you said, you can put that project config anywhere (downstream) of zuul-jobs, right? | 21:45 |
| jeblair | jlk: the job definition has to be downstream, but the project config can be in an upstream config repo (eg, openstack-infra/project-config) | 21:46 |
| jeblair | jlk: because jobs are all parsed in one phase, then project definitions in a later phase | 21:46 |
| jlk | oh right, yes | 21:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul-jobs master: Remove tox-py35-on-zuul https://review.openstack.org/482717 | 21:47 |
| jlk | workflowing that one | 21:49 |
| mordred | jeblair: https://review.openstack.org/482719 is the last change | 21:50 |
| jeblair | mordred: +2 on all of them; we should probably +3 them in sequence :) | 21:52 |
| mordred | jeblair: ++ | 21:52 |
| jeblair | oh they have depends on, so they should land in the right order. i'll go ahead and +3 them | 21:53 |
| mordred | \o/ | 21:53 |
| Shrews | jlk: PR waiting for you! | 21:54 |
| jlk | Funny, I wasn't even watching my own project | 21:55 |
| Shrews | jlk: ugh, github renders that horribly. lemme fix | 21:55 |
| jlk | kk | 21:55 |
| jlk | ugh, I should hook up some CI or something. | 21:57 |
| Shrews | jlk: good enough. not pretty, but readable | 21:57 |
| jlk | yeah I haven't really markdowned it yet. | 21:58 |
| Shrews | *now* i will EOD | 21:59 |
| jlk | cheers! | 21:59 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Remove tox-py35-on-zuul https://review.openstack.org/482717 | 22:04 |
| jlk | oh looks like we'll need to grow a zuul-web container | 22:05 |
| jeblair | mordred: i have reproced the config loading error in a unit test | 22:07 |
| mordred | jeblair: that's exciting! | 22:08 |
| mordred | jeblair: also - maybe we should, at some point in the future, add a third-party-ci of zuul from bonny - so that we could get a failure on patches proposed to zuul-job that would introduce the sorts of dependencies that tox-py35-on-tox introduced | 22:10 |
| mordred | since that likely would have failed with the same error in a bonny check pipeline when we proposed the patch | 22:10 |
| jlk | it probably would have. That'll be a fun thing to hook up. | 22:12 |
| jeblair | wfm. we should run that by fungi. | 22:12 |
| fungi | mordred: jeblair: seems fine if it's continuously deployable in such a way that it can report on proposed patches | 22:14 |
| jlk | Remind me again why status_url is in the [webapp] section, but it used only by scheduler? | 22:15 |
| fungi | jlk: optimism? ;) | 22:15 |
| jeblair | fungi: actually, i think the thing we can do sooner than patches on zuul is patches on zuul-jobs. since we're aiming for that to be globally usable right now | 22:16 |
| fungi | oh, right. that's probably relatively trivial to integrate given they're running zuul v3 | 22:16 |
| jeblair | fungi: so bonnyci can speculatively execute them as well as we can, but they'll have a different configuration which will help us identify if we have made an assumption | 22:16 |
| mordred | yah - this would be "attach the running bonnyci service to openstack's gerrit and have it vote verified -1/+1 on changes to zuul-jobs | 22:16 |
| mordred | ++ | 22:16 |
| jeblair | on zuul itself may be useful later, but, honestly, at this point, we're breaking deployment almost daily, so we already know it's going to be red. :) | 22:17 |
| mordred | we're going to wind up with a network of interrelated zuuls all triggering each other | 22:17 |
| fungi | a circle of zuuls | 22:17 |
| jeblair | on zuul-jobs, however, we're ready to start ensuring stability | 22:17 |
| jlk | The circle of after-life | 22:19 |
| jeblair | jlk: yeah, i'd go with fungi's answer on status_url for now. | 22:20 |
| jeblair | aha, i see the bug which is causing us to load configuration out of order | 22:24 |
| jeblair | i'm going to write a fix that is right beneath a comment about how ordering is important | 22:26 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Handle aborted websocket client connection cleanly https://review.openstack.org/482560 | 22:31 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:32 |
| jeblair | mordred, jlk: ^ there we go! | 22:32 |
| jeblair | oh wait, missed a git add | 22:32 |
| jlk | Is there any real difference between "del foo" and "del(foo)" ? | 22:33 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:33 |
| jeblair | mordred, jlk: ^ there we go! (again) | 22:33 |
| jeblair | jlk: i'm not aware of any | 22:34 |
| jlk | interesting gertty bug. If I already reviewed a change (that was subsequently updated), as non-core, then open it as core, I only get options to +/- 1 not 2 nor w. | 22:35 |
| fungi | other than i like function syntax, i'm not aware of any difference | 22:35 |
| fungi | jlk: force refresh the change in gertty (ctrl-r in the default keybinding map) | 22:35 |
| clarkb | that happens with the web ui too | 22:35 |
| jlk | ah | 22:35 |
| fungi | for slightly different reasons with the web ui, but yes strangely similar resuly | 22:36 |
| jlk | The official python docs just use "del foo" | 22:36 |
| fungi | result | 22:36 |
| jlk | (I would prefer del() as well, feels better) | 22:36 |
| fungi | i also used print() syntax for years before it was actually a callable function | 22:36 |
| fungi | because it was interpreted as "print (whatever)" which was still valid for pretty much everything | 22:37 |
| clarkb | del is still a statement in python3 though | 22:37 |
| fungi | you just couldn't pass named parameters to it or anything | 22:37 |
| clarkb | so del foo is more correct I think | 22:37 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add job's project as implicit role project https://review.openstack.org/482726 | 22:37 |
| jeblair | just wanted to push that up so i don't lose it. it should work, just needs docs and tests | 22:37 |
| jlk | oh so is it essentially doing "del (foo)" ? | 22:37 |
| fungi | yep | 22:37 |
| fungi | afaik | 22:37 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Cleanup on ssh-agent failure https://review.openstack.org/481344 | 22:46 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix error loading projects out of order https://review.openstack.org/482725 | 22:56 |
| openstackgerrit | Jesse Keating proposed openstack-infra/zuul feature/zuulv3: Use finishJob funtion to clean up on failure https://review.openstack.org/482731 | 22:57 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Move ssh-agent cleanup into existing exception handler https://review.openstack.org/482735 | 23:01 |
| jeblair | jlk, mordred: an alternate approach ^ | 23:02 |
| jlk | oh | 23:02 |
| jeblair | i think we let the complexity get away from us in bits and pieces; hopefully that makes things a little more easy to see | 23:03 |
| jeblair | speaking of things getting out of hand. i'm about ready to declare the static method experiment in the configloader to be a failure. :) it seemed like a nice idea when we started -- configuration shouldn't require a lot of state. but it turns out that in zuulv3, configuration is all about state. so i think that file has gotten very complex and would greatly benefit from going back to using real object instances. | 23:11 |
| mordred | jeblair: I will not argue with you about code structure in the configloader | 23:12 |
| jlk | Time has gotten out of hand for me, I gotta flee to ride a bike to catch a train to ride a bike to go home. | 23:13 |
| jeblair | i'm probably going to back-burner that idea since we have a lot of other things to deal with at the moment, and it will take a bit to get right. but i think eventually it's going to have a high maintenance cost, and if we run into something that requires significant changes soon, we may want to take the opportunity to refactor. | 23:13 |
| jeblair | jlk: enjoy all your modes of transport! | 23:14 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fail execution if a role repo is a bare collection of roles https://review.openstack.org/482327 | 23:14 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: WIP Add job's project as implicit role project https://review.openstack.org/482726 | 23:15 |
| mordred | jeblair: ++ to that | 23:18 |
| jamielennox | Shrews: there is at least one api that you can't do via integration, it's commented, realistically you can do it publicly but your quota limits are really low on that so we fall back to an api key if one is present | 23:39 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!