| *** harlowja has quit IRC | 01:29 | |
| mordred | jeblair: I think that looks great! | 01:36 |
|---|---|---|
| *** isaacb has joined #zuul | 04:05 | |
| *** isaacb has quit IRC | 04:05 | |
| *** harlowja has joined #zuul | 04:40 | |
| *** deep-book-gk_ has joined #zuul | 04:46 | |
| *** deep-book-gk_ has left #zuul | 04:48 | |
| *** harlowja has quit IRC | 05:39 | |
| *** amoralej|off is now known as amoralej | 06:54 | |
| tobiash | jeblair: yay, looks great :) | 07:24 |
| tobiash | pabelanger: currently trying out the tox-docs job from zuul-jobs in my environment and it fails during configure-mirrors | 08:27 |
| tobiash | pabelanger: that's pulled in currently by the unittests pre playbook with hard coded openstack specific variables | 08:27 |
| *** hashar has joined #zuul | 08:53 | |
| *** ajafo has quit IRC | 09:15 | |
| *** ajafo has joined #zuul | 09:23 | |
| *** jkilpatr has quit IRC | 10:00 | |
| *** jkilpatr has joined #zuul | 10:12 | |
| *** jkilpatr has quit IRC | 10:39 | |
| *** clarkb has quit IRC | 10:45 | |
| *** clarkb has joined #zuul | 10:45 | |
| *** jkilpatr has joined #zuul | 11:12 | |
| *** dkranz_ has joined #zuul | 12:30 | |
| *** hashar has quit IRC | 13:05 | |
| *** hashar has joined #zuul | 13:06 | |
| *** amoralej is now known as amoralej|lunch | 13:59 | |
| dmsimard | pabelanger mentioned a job_output.json file, where are those located ? | 14:17 |
| tobiash | dmsimard: it's part of the job result folder | 14:26 |
| tobiash | dmsimard: e.g. here: http://logs.openstack.org/40/485840/14/check/tox-linters/02644f0/ | 14:27 |
| * dmsimard looks | 14:27 | |
| tobiash | dmsimard: or did you mean on the executor during the build? | 14:28 |
| dmsimard | tobiash: no, that seems to be what he was referring to | 14:28 |
| jeblair | it gets written to <jobdir>/work/logs on the executor host when the job is running -- same place as the console log (job-output.txt). then copied up along with it in a post playbook | 14:28 |
| dmsimard | He'd like to import those in ARA so I was curious what they were | 14:28 |
| dmsimard | I'm still juggling with the idea of import/export "drivers" in ARA so knowing what people want to do is helpful :) | 14:29 |
| dmsimard | There was also the notion of picking up messages from mqtt | 14:29 |
| jeblair | dmsimard: it's *similar* to the json produced by the json output plugin, but it's a list of json blobs, one for each invocation of ansible-playbook. | 14:29 |
| dmsimard | jeblair: yeah, let's call it a "proprietary" format :) | 14:30 |
| jeblair | is importing this into ara the better approach? or just adding the ara callback like we discussed? | 14:31 |
| jeblair | tobiash: looks like configure mirrors is intended to switch on the presence of the variable mirror_host, which is hard-coded in there now | 14:33 |
| jeblair | tobiash: and it's waiting on https://review.openstack.org/447734 site-local vars to make that configurable | 14:34 |
| tobiash | jeblair: which is already +2 from me :) | 14:34 |
| jeblair | pabelanger, mordred, SpamapS: ^ maybe one of you want to +3 that? | 14:34 |
| *** amoralej|lunch is now known as amoralej | 14:38 | |
| dmsimard | jeblair: it depends on the approach, the ara callback works well for standalone reports (statically generated like stackviz) | 14:42 |
| dmsimard | jeblair: if we want to aggregate data (like openstack-health), we need to look at something else | 14:42 |
| dmsimard | jeblair: the callback has support for sending data to a remote relational database (i.e, mysql) but at openstack-infra scale, I fear it might not be very reliable, introduce latency and such | 14:43 |
| dmsimard | so we need some way of importing data asynchronously, mqtt was an option, importing the json can be interesting as well | 14:43 |
| pabelanger | tobiash: Ya, we can work on that today to remove the openstack variable bits | 14:43 |
| pabelanger | tobiash: I think some of that is in our shadow base job now | 14:43 |
| dmsimard | mqtt is interesting because of the real time nature of it, importing the json only makes the data available at the end of the run which is okay too but need to be kept in mind | 14:44 |
| dmsimard | importing the json would be very much like openstack-health current imports the subunit files | 14:44 |
| tobiash | pabelanger: it's not that urgent, currently fighting with openshift | 14:44 |
| pabelanger | Right, was going to see if that was possible, then we shouldn't need anything on executor? | 14:44 |
| pabelanger | tobiash: ack | 14:45 |
| dmsimard | pabelanger: for importing the json ? | 14:45 |
| pabelanger | dmsimard: right, like I said, not sure if possible today | 14:46 |
| dmsimard | pabelanger: there's nothing in ara that would allow you to do that easily right now, it needs at the very least a refactor for introducing the API that will abstract a lot of the logic | 14:46 |
| dmsimard | pabelanger: fwiw I just got confirmation that I'm (at least) going to the PTG, AnsibleFest I don't know yet | 14:46 |
| dmsimard | So we'll get the opportunity to hack on things | 14:46 |
| tobiash | does someone know if there's a possibility to install zuul from source without the git dir? | 14:46 |
| dmsimard | and by the PTG, I should have made good progress towards the API | 14:47 |
| tobiash | openshift doesn;t want to put the .git dir into the build context :( | 14:47 |
| dmsimard | tobiash: with s2i ? | 14:47 |
| jeblair | pabelanger: let's not add those vars to the base job, let's land site-vars instead | 14:47 |
| tobiash | dmsimard: Using docker build with dockerfile | 14:48 |
| jeblair | tobiash: are you having problems because of pbr version stuff? | 14:48 |
| jeblair | tobiash: if so, there's an env variable you can set that pbr will use instead of trying to do git ops | 14:49 |
| tobiash | jeblair: could be (I'm still a setup.py noob) | 14:49 |
| tobiash | jeblair: ah, checking out | 14:49 |
| jeblair | tobiash: https://docs.openstack.org/pbr/latest/user/packagers.html PBR_VERSION | 14:49 |
| * tobiash trying | 14:50 | |
| tobiash | jeblair: yay, works, you saved me hours of digging into a custom builder workaround :) | 14:56 |
| * tobiash is heading home happily | 14:57 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Remove FakeProvider getClient monkey-patch https://review.openstack.org/475131 | 15:09 |
| *** olaph has joined #zuul | 15:13 | |
| jeblair | tobiash, mordred: i was thinking -- maybe instead of using the zuul domain in https://review.openstack.org/487239 i could make a sphinx yaml domain. so we'd do things like yaml:dict yaml:key etc. might work just as well and be generally applicable. | 15:14 |
| *** nt has left #zuul | 15:21 | |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove refspec https://review.openstack.org/485875 | 15:30 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_REFNAME https://review.openstack.org/486232 | 15:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_OLDREV and ZUUL_NEWREV https://review.openstack.org/486233 | 15:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_COMMIT https://review.openstack.org/486235 | 15:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_REF https://review.openstack.org/486236 | 15:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_PATCHSET https://review.openstack.org/486240 | 15:36 |
| dmsimard | MERGE ALL THE THINGS | 15:38 |
| SpamapS | jeblair, pabelanger: How hard would it be to start having zuulv3.openstack.org gate feature/zuulv3? | 15:41 |
| * SpamapS waits for them to tell me it's already doing that | 15:41 | |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Support UUID as builder identifier https://review.openstack.org/484414 | 15:41 |
| jeblair | SpamapS: i think we'd probably just switch both branches (we're not doing much for master right now anyway). and i think we'll do that soon. | 15:42 |
| tobiash | jeblair: sounds good | 15:42 |
| pabelanger | SpamapS: jeblair: ++ | 15:43 |
| tobiash | I'm gating my project documentation with v3 | 15:44 |
| tobiash | works flawlessly so far | 15:44 |
| pabelanger | jeblair: Shrews: we should upgrade nodepool-launcher to python3 today | 15:44 |
| Shrews | https://review.openstack.org/485247 is waiting for another +2 to do just that | 15:45 |
| pabelanger | Ya, lefts roll that out now, I have time to monitor | 15:46 |
| jeblair | oh look at that it just got one | 15:46 |
| pabelanger | \o/ | 15:46 |
| Shrews | pabelanger: there have been some significant code reorg and the UUID feature added to nodepool since last restart. we should watch logs closely at restart | 15:48 |
| Shrews | i'd submit a change to remove the gate-nodepool-python27-ubuntu-xenial nodepool job for the feature branch, but i can never figure out the project-config magic | 16:01 |
| *** tristanC has quit IRC | 16:03 | |
| *** hashar is now known as hasharMeeting | 16:07 | |
| pabelanger | Shrews: sounds good | 16:07 |
| *** harlowja has joined #zuul | 16:15 | |
| SpamapS | mordred: fyi, you are the inspiration for this tweet: https://twitter.com/spamaps/status/890245921685897216 (specifically https://review.openstack.org/485897) | 16:23 |
| Shrews | jeblair: is there an existing zuul test that you could recommend that I use as a model for a new autohold test? | 16:42 |
| Shrews | or is this going to be a "start from scratch" sort of thing? | 16:43 |
| jeblair | Shrews: test_check_queue_failure is the shortest path to a failed job | 16:46 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGE https://review.openstack.org/486241 | 16:47 |
| *** hasharMeeting has quit IRC | 16:48 | |
| *** tristanC has joined #zuul | 16:48 | |
| Shrews | jeblair: thank you | 16:50 |
| jeblair | Shrews: and there's a self.fake_nodepool (instance of tests.base.FakeNodepool) that will probably be useful. it's the stand-in for nodepool itself (ie, the remote application which interfaces over zookeeper). so that might be the place to put a method to query nodes to verify they have hold state in zk, etc. | 16:54 |
| Shrews | neat. this should be an interesting challenge | 16:57 |
| clarkb | jeblair: comment on https://review.openstack.org/#/c/487243/1 do you want to fix that or should I just push a new patchset then deploy on my test setup against review-dev? | 17:00 |
| jeblair | clarkb: oh sorry, i forgot to check in on those test results. i'll fix. | 17:04 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 17:06 |
| jeblair | clarkb: i at least ran test_nodepool tests on it locally this time :) | 17:06 |
| clarkb | thanks, I should have that up and running and testing against gerrit shortly. Need to file a bug against keystone I said I would file yesterday... | 17:07 |
| pabelanger | Jul 26 15:56:42 nl01 puppet-user[7892]: (/Stage[main]/Nodepool/Exec[install_nodepool]) Failed to call refresh: Could not find command '${pip_command}' | 17:11 |
| pabelanger | fixing puppet-nodepool | 17:11 |
| Shrews | pabelanger: oops | 17:13 |
| mordred | SpamapS: yes - I agree with you. fwiw, getting testing plumbed in for that is very high on my agenda | 17:16 |
| mordred | SpamapS: hackig on it currently is ... non-optimal | 17:16 |
| mordred | jeblair: wfm re: general sphinx domain | 17:16 |
| jeblair | mordred, tobiash: i spent about 30m on it and still really like the yaml domain idea, but it's *complex*. so i think i'm going to defer it for a rainy day and continue with the zuul-specific approach for now. hopefully that will help shape ideas. | 17:18 |
| clarkb | jeblair: https://review-dev.openstack.org/#/c/107960/1 that got things further :) but results in NODE_FAILURE | 17:18 |
| jeblair | by 30 minutes i think i might actually mean an hour. time flies when you're sphinxing. | 17:18 |
| clarkb | jeblair: good new is that means we've tested just about everything except for submit | 17:19 |
| jeblair | clarkb: neat, i'll peek at those logs | 17:19 |
| clarkb | also it toally verified -1 | 17:19 |
| clarkb | tobiash: ^ you were seeing that posting the votes wasn't working right? | 17:19 |
| jeblair | clarkb: where's that zuul running? | 17:19 |
| clarkb | jeblair: its in screen on review-dev (screen owned by my user window 0 | 17:19 |
| jeblair | clarkb: i think tobiash's error only occurred during submit? | 17:19 |
| clarkb | jeblair: feel free to attach and scroll the logs in window zero | 17:20 |
| jeblair | clarkb: no disk logs? | 17:20 |
| clarkb | jeblair: no I've just been running foreground because it is easy | 17:20 |
| mordred | jeblair: ++ | 17:21 |
| clarkb | I can rerun with logs writing to disk if you like | 17:21 |
| mordred | (to rainy day) | 17:21 |
| jeblair | clarkb: i'll try screen first | 17:21 |
| jeblair | derp | 17:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Add zuul:value sphinx directive https://review.openstack.org/487530 | 17:24 |
| clarkb | jeblair: its trying to query the nodeset later on? | 17:25 |
| clarkb | (and that fails because it never was in zk?) | 17:25 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 17:26 |
| jeblair | clarkb: no i think that's it ^. it was treating it as a failed node request because "nodepool" did not set the state to fulfilled. | 17:27 |
| clarkb | ah | 17:28 |
| * clarkb retries | 17:28 | |
| jeblair | clarkb: screen is yours | 17:28 |
| jeblair | clarkb: wow i need coffee or something | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Don't request empty nodesets https://review.openstack.org/487243 | 17:30 |
| jeblair | clarkb: ^ | 17:30 |
| clarkb | jeblair: it certainlly didn'y like ps3 :) | 17:30 |
| *** dkranz_ has quit IRC | 17:31 | |
| *** dkranz_ has joined #zuul | 17:32 | |
| clarkb | https://review-dev.openstack.org/#/c/107960/1 that worked | 17:32 |
| jeblair | clarkb: cool, can you try a submit now? | 17:33 |
| tobiash | clarkb, jeblair: error was during handling of successful gate: gerrit accepted the vote but silently ignored the submit | 17:33 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Fix the rendering of item entries https://review.openstack.org/486242 | 17:36 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove ZUUL_CHANGE_IDS https://review.openstack.org/486243 | 17:36 |
| clarkb | tobiash: gotcha | 17:38 |
| clarkb | jeblair: yup I need to make the jenkisn account inactive so that it doesn't interfere then rerun things | 17:38 |
| clarkb | jeblair: but first I'm trying to properly understand why an empty nodeset implies job success os that I can +2 your change | 17:39 |
| jeblair | clarkb: it doesn't -- noop implies success. the nodeset is orthogonal (unless it fails, obvs) | 17:40 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/zuul feature/zuulv3: WIP: Implement autohold https://review.openstack.org/486692 | 17:40 |
| jeblair | clarkb: reason being -- since you can run certain things on an executor without actually needing a node, an empty nodeset job is actually a legit config. | 17:41 |
| jeblair | clarkb: (eg: hitting an rtfd hook. don't need a node for that) | 17:41 |
| clarkb | gotcha | 17:42 |
| clarkb | so we are running the noop job and it is magically successful as before. Problem was we needed to not request nodes whendoing so | 17:42 |
| clarkb | ya I see that now in the executor client | 17:43 |
| jeblair | clarkb: exactly | 17:43 |
| Shrews | jeblair: so, the pieces aren't in place yet to expect the new autohold test in 486692 to succeed, but does the logic st least seem sound there? https://review.openstack.org/#/c/486692/3/tests/unit/test_scheduler.py | 17:45 |
| Shrews | s/st/at/ | 17:45 |
| *** harlowja has quit IRC | 17:46 | |
| Shrews | it bypasses using the client, but i don't see any tests doing something similar | 17:47 |
| clarkb | jeblair: I've approved your change. Now to test submit | 17:47 |
| jeblair | Shrews: looks dead on | 17:47 |
| Shrews | jeblair: shocking! lol, thanks a bunch. now i have something to work from | 17:48 |
| jeblair | Shrews: i think that's the right thing to do in this case -- this way we're not relying on anything in zuul to tell us the node is held. we're actually checking zk. | 17:48 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage status javascript https://review.openstack.org/487538 | 17:51 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate console streaming to webpack/yarn https://review.openstack.org/487539 | 17:51 |
| clarkb | jeblair: tobiash I also notice that the +1 from my zuul didn't end up overwriting the -1 from when it failed on node failure. Gerrit comment says Patch Set 1: Verified+1 Verified-1 | 17:52 |
| clarkb | its psosible we may need to work around new gerrit behavior when setting new votes | 17:52 |
| clarkb | I'm going to go ahead and clear that out for now though and move on to checking submit | 17:52 |
| mordred | clarkb: isn't that a config we need to set in gerrit on repos? did we get that in on the review-dev repo? | 17:52 |
| clarkb | mordred: in this case its not a new patchset its just me voting differently | 17:53 |
| mordred | hrm | 17:53 |
| clarkb | mordred: first I -1'd then recheked and +1'd | 17:53 |
| clarkb | end result was still -1 according to web ui and the comemnt said both :) | 17:53 |
| mordred | clarkb: I certainly hope that's not new intended behavior in new gerrit | 17:53 |
| tobiash | clarkb: ah, could also be such a side effect, now I remembere that clearing votes at start of jobs wasn't working properly | 17:53 |
| clarkb | mordred: ya its possible its related to tobiash's case sensitivity fix | 17:54 |
| mordred | ah. gotcha | 17:54 |
| mordred | *phew* | 17:54 |
| tobiash | clarkb: after applying my fix in my deployment I didn't notice such things anymore | 17:54 |
| clarkb | tobiash: good to know | 17:54 |
| tobiash | just got an idea for a multinode use case... running tox-py35 on 5 nodes in parallel, that could increase the chance of blocking racy tests from landing | 17:55 |
| tobiash | (or prevent any change from landing if a racy test got through) | 17:56 |
| clarkb | I'm going to stop zuul on zuul-dev because marking the user inactive didn't kill the existing ssh connection and I don't want it to dos me (we've seen that in production whee ssh fails) | 17:58 |
| clarkb | does this conflict with anyone else's work? | 17:58 |
| jeblair | tobiash: yeah, we actually did something similar for neutron for a while a couple years ago: we just ran a second job. that idea is easier and more elegant. :) | 17:58 |
| jeblair | clarkb: nope | 17:59 |
| tobiash | mordred: I've put a question in https://review.openstack.org/#/c/487538/1 | 18:00 |
| mordred | tobiash, jeblair: yah - especially since the unittest jobs are deisgned to run on "all" | 18:00 |
| mordred | log collection might be slightly weird ... | 18:00 |
| tobiash | didn't pabelanger already have some patches in that direction? | 18:01 |
| mordred | tobiash: that's an excellent question - in this case I believe putting it in the repo is 'good' because this is an app and not a library | 18:02 |
| mordred | tobiash: the rust folks have a great writeup on this here: http://doc.crates.io/faq.html#why-do-binaries-have-cargolock-in-version-control-but-not-libraries | 18:02 |
| mordred | (obviously slightly different- but similar philosophy) | 18:02 |
| mordred | tobiash: that siad - we can obviously go either way on that | 18:03 |
| * mordred will put that as comment on change too - so it doens't get lost | 18:03 | |
| tobiash | mordred: ah, so it's for fixing dependencies, now I see... | 18:04 |
| tobiash | mordred: then it *should* be part of the repo, otherwise the build will be broken frequently | 18:04 |
| tobiash | mordred: had such issues a few years ago with composer and it was pain in the ass... | 18:04 |
| SpamapS | mordred: Indeed, but everything can't have tests, so we all have to defuse code bombs every day. :-D | 18:08 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate console streaming to webpack/yarn https://review.openstack.org/487539 | 18:09 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage status javascript https://review.openstack.org/487538 | 18:09 |
| tobiash | looks like we were all impatient and didn't wait on the test result of 487243... | 18:11 |
| mordred | tobiash: :) | 18:11 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool feature/zuulv3: Add zuulv3 jobs for nodepool https://review.openstack.org/487169 | 18:12 |
| pabelanger | mordred: Shrews: jeblair: should we pip uninstall nodepool on nl01 first for py27? | 18:12 |
| mordred | pabelanger: probalby not a terrible idea | 18:13 |
| pabelanger | ok, will stop nl01 in a few minutes | 18:13 |
| clarkb | jeblair: turns out I didn't define a gate pipeline, with that now done do I need to restart zuul to pick up new pipelines from trusted configs or did that get addressed? | 18:14 |
| SpamapS | anbody know what this is? http://logs.openstack.org/45/486245/2/check/gate-zuul-python35/33bcbe2/testr_results.html.gz <-- looks to be a racey fail somewhere deep in the bowels... | 18:16 |
| SpamapS | http://logs.openstack.org/45/486245/2/check/gate-zuul-python35/33bcbe2/testr_results.html.gz | 18:16 |
| SpamapS | oops | 18:16 |
| SpamapS | ft2.6: tests.unit.test_executor.TestExecutorRepos.test_periodic_override_StringException | 18:16 |
| jeblair | clarkb: i think that should be fine | 18:16 |
| SpamapS | Have seen it a couple of times now | 18:16 |
| clarkb | ya I now see the reaosn it didn't work is there is an error in my new yaml | 18:17 |
| jeblair | SpamapS: :| i'll try to take a look at it later and see if i can guess what might be happening | 18:17 |
| mordred | Shrews, tobiash: I still don't get tired of watching consoles stream | 18:18 |
| SpamapS | other ones seem to be producing mass fails that make the subunit too big | 18:18 |
| tobiash | :) | 18:19 |
| *** amoralej is now known as amoralej|off | 18:19 | |
| pabelanger | zuul-npm-build o.0 | 18:20 |
| pabelanger | that is new | 18:20 |
| tobiash | clarkb: jlk fixed that in https://review.openstack.org/#/c/483597/ | 18:21 |
| pabelanger | okay, stopping nl01 to preform uninstall / pip3 install | 18:23 |
| clarkb | now running into http://paste.openstack.org/show/616611/ as far as I can tell reading the docs and even reading voluptuous schemas that should be valid? | 18:23 |
| clarkb | but I notice we have event_approvals, require-approvals, and reject-approvals now and event apparovals which I am using are defined differently in the schema, perhaps that is a bug? or I'm just not reading something right | 18:23 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Change jobroot_dir to job_dir in executor config https://review.openstack.org/487165 | 18:24 |
| jeblair | clarkb: i think you need a 'gerrit' after require now | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate console streaming to webpack/yarn https://review.openstack.org/487539 | 18:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage status javascript https://review.openstack.org/487538 | 18:24 |
| tobiash | clarkb: do you have connection specific requirements? | 18:24 |
| clarkb | ah ok | 18:25 |
| jeblair | clarkb: require: gerrit: ...the block currently under require... | 18:25 |
| pabelanger | nl01.o.o restarted | 18:26 |
| clarkb | https://docs.openstack.org/infra/zuul/feature/zuulv3/user/config.html#pipeline doesn't really make that clear | 18:26 |
| pabelanger | nodepool 24711 8.0 0.6 2926712 52744 ? Sl 18:25 0:01 /usr/bin/python3 /usr/local/bin/nodepool-launcher -c /etc/nodepool/nodepool.yaml -l /etc/nodepool/launcher-logging.conf -p /var/run/nodepool-launcher/nodepool-launcher.pid | 18:26 |
| clarkb | it lists approval as a sub item of require | 18:26 |
| mordred | pabelanger: it's new in that patch - because yay zuulv3 we can define jobs along with patches! :) | 18:26 |
| pabelanger | mordred: \o/ Wasn't sure we landed that job or not | 18:27 |
| jeblair | clarkb: yep. i think we forgot to update that. | 18:27 |
| mordred | pabelanger: nope - it's got some issues even so - we should likley not actually land it in that particular form | 18:27 |
| pabelanger | Shrews: py35 running for nl01, so far, so good | 18:27 |
| pabelanger | mordred: Ya, thought we might have converted our exixting npm build JJB stuff | 18:28 |
| mordred | pabelanger: it does uncover an 'interesting' edge-case - which is that that job needs to install some additional package repos - working around it for now, but might be worth us pondering | 18:28 |
| Shrews | pabelanger: cool. watching log now | 18:29 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: Move subunit processing into fetch-testr-output https://review.openstack.org/485840 | 18:29 |
| pabelanger | mordred: tobiash: ^ nip fixed, if you want to re-review and maybe land | 18:30 |
| clarkb | tobiash: and http://paste.openstack.org/show/616614/ must be what you see with the case sensitivity problem? | 18:30 |
| clarkb | tobiash: basically it says all tests succeeded, I'm reporting, then merge failed so item failed | 18:30 |
| tobiash | clarkb: jepp | 18:31 |
| pabelanger | https://review.openstack.org/487173 also makes tox-cover non-voting for zuul, but job is passing so maybe we should just leave it voting? | 18:31 |
| clarkb | ok now to pull in tobiash's patch and give it a go. I have to update my config to be Verified too | 18:31 |
| tobiash | clarkb: and the change probably has now V+2 | 18:31 |
| pabelanger | or, maybe we don't care about coverage on zuul long term. Might be there just for testing | 18:32 |
| clarkb | tobiash: https://review-dev.openstack.org/#/c/107960/1 is the change, it doesn't look like the +2 made it | 18:32 |
| pabelanger | Shrews: cool, 487169 is ready now too | 18:32 |
| openstackgerrit | Merged openstack-infra/zuul-jobs master: Move subunit processing into fetch-testr-output https://review.openstack.org/485840 | 18:34 |
| Shrews | hrm, i'm not finding the uuid file | 18:37 |
| Shrews | oh, not a builder. duh | 18:38 |
| clarkb | tobiash: jeblair I seem to have confirmed that tobiash's change fixes the -1 to +1 problem | 18:39 |
| tobiash | yay | 18:39 |
| clarkb | still getting the it failed to merge error though | 18:40 |
| clarkb | oh that may be coming from my merger not from gerrit submit | 18:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 18:42 |
| clarkb | DEBUG:zuul.Scheduler:Adding merge complete event for build set: <BuildSet item: <QueueItem 0x7f3f4806d2b0 for <Change 0x7f3f4807d2e8 107960,1> in gate> #builds: 0 merge state: PENDING> why would merge state be pending if merge complete event is being added? | 18:42 |
| jeblair | clarkb: state is updated after that gets called (that's an event being added to the queue, it's processed later async) | 18:44 |
| * Shrews will now take a few moments to go through mordred's giant pile | 18:45 | |
| clarkb | oh! I see the problem. I didn't have a gate pipeline so I added that, well I still don't have a gate job >_> | 18:45 |
| mordred | clarkb: haha | 18:45 |
| clarkb | so I'll need to retest things without tobiash's change as well | 18:46 |
| tobiash | mordred: just read the zuul-web section of your mail | 18:52 |
| tobiash | mordred: what I'm asking myself is wouldn't it make sense to use the event ingestor idea someone here had for the webhook -> github driver data? | 18:53 |
| clarkb | woo I think I may have just tripped over a zuul bug http://paste.openstack.org/show/616615/ is the result of pushing up https://review-dev.openstack.org/#/c/107964/ | 18:53 |
| clarkb | jeblair: ^ | 18:53 |
| jeblair | tobiash: that was mordred's idea iirc | 18:53 |
| tobiash | mordred: or is it too early to think about this? | 18:53 |
| jeblair | clarkb: looking | 18:54 |
| mordred | tobiash: so - I think the github patch will largely be the same as what github would need for the ingestor too | 18:54 |
| mordred | tobiash: because of github being webhooks having a scale-out story for it it much easier - and I don't think it's as immediately essential to do the de-duping since load-balancers are pretty good at that sort of thing | 18:54 |
| tobiash | mordred: sure it will work easily for this, just thinking if gearman will stay forever or be replaced at some time completely by zookeeper | 18:55 |
| jeblair | tobiash: replaced eventually. we call that "zuul v4". | 18:55 |
| openstackgerrit | Merged openstack-infra/nodepool feature/zuulv3: Add zuulv3 jobs for nodepool https://review.openstack.org/487169 | 18:56 |
| mordred | tobiash: it's certainly a good question - I know gearman will stay for at least a while, since that'show we're getting data from the scheduler for log streaming ... so I think just using it for this for now is fairly easy and if we do a different approach in the future *should* be easy enough to change | 18:56 |
| mordred | tobiash: but yes - as jeblair says - v4 :) | 18:56 |
| jeblair | clarkb: i verified that copy.deepcopy(re.compile('foo')) errors on 3.4 and 3.5 | 18:57 |
| jeblair | but i'm just assuming that's what happened | 18:57 |
| jeblair | i wish deepcopy told you which key failed | 18:58 |
| jeblair | rather than "this" | 18:58 |
| tobiash | mordred: I'm fine with that, just like the idea of having persistent queues of events,ipc,etc in the future | 18:58 |
| jeblair | it's like a clickbait exception. "You won't believe this error!" | 18:58 |
| clarkb | New in python! this fails! | 18:58 |
| clarkb | jeblair: but legit bug ya? | 18:58 |
| * Shrews clicks on jeblair | 18:59 | |
| jeblair | Shrews: ouch! | 18:59 |
| clarkb | I'll just manually merge the change then restart zuul if I have to | 18:59 |
| mordred | tobiash: ++ | 18:59 |
| jeblair | clarkb: i suspect so, yeah. we may not have any pipeline requirements in the dynamic dependent pipeline addition. | 18:59 |
| jeblair | clarkb: i suspect so, yeah. we may not have any pipeline requirements in the dynamic dependent pipeline addition test. | 18:59 |
| jeblair | left off important word :) | 18:59 |
| clarkb | in this case my regex is in the independent pipeline | 19:00 |
| clarkb | so I'm assuming that is what is causing the problem | 19:00 |
| clarkb | jeblair: do you need anything more from me before I submit manually and restart zuul? | 19:01 |
| jeblair | clarkb: nope that helps, thx | 19:01 |
| Shrews | mordred: do you still see zuul-web acting as the finger protocol gateway too? | 19:02 |
| clarkb | tobiahs's change worked to merge https://review-dev.openstack.org/#/c/107960/1 I'm now going to rollback the case sensitive changes and retest without them to make sure that I wasn't just failing due to lack of gate job | 19:03 |
| mordred | Shrews: I don't see why not - it has to get the same info it needs for finger protocol for websocket streaming - and it's also scale-out able | 19:03 |
| mordred | Shrews: (most of finger protocol gateway code should be fairly similar to the websocket streaming code - just different socket listener, yeah?) | 19:03 |
| Shrews | mordred: yeah | 19:04 |
| mordred | Shrews: I mean - we could ALSO write it as a completely separate daemon if we wanted to | 19:04 |
| tobiash | mordred: maybe it makes sense to be able to start it separately (for e.g. starting, scaling separately in a k8s environment) | 19:04 |
| clarkb | mordred: you'll also potentially need to accomodate for raw bytes rather than text because websockets handle the text aspect more natively | 19:05 |
| mordred | clarkb: ++ | 19:05 |
| mordred | clarkb: actually - I believe we're emitting as utf-8 encoded bytes rather than text already - but I agree, the receiver on the other side of that is going to be less likely to be expecting weirdness | 19:06 |
| mordred | tobiash: nod. I mean, we can keep the shared "here is how you verify stuff from teh scheduler via gearman" in library code and make a little daemon that uses it - it's an interesting trade-off - since adding another daemon will make some deployment forms nicer and others more work | 19:07 |
| tobiash | mordred: ack | 19:08 |
| tobiash | mordred: where would you see a nodepool status page (like w web-'nodepool list' thingy)? | 19:08 |
| Shrews | mordred: there is a slight advantage in a separate thing in that the finger gateway MUST be started as root. so zuul-web would have to do privilege de-escalation and fork things which might interact weirdly with asyncio event loops, but i'm sure it could eventually be figured out | 19:08 |
| tobiash | mordred: in nodepool or zuul-web crawling zookeeper? | 19:08 |
| mordred | tobiash: I think in nodepool - there's folks who want to use nodepool without zuul who would want that | 19:09 |
| tobiash | ok | 19:09 |
| mordred | Shrews: that's a good point | 19:09 |
| mordred | Shrews: I mean - you could start it on a high port and depend on a load-balancer in front of it to run on the low port - but I agree, for the simple-case where one gateway is fine, having it separate limits the code that needs root | 19:10 |
| clarkb | so thats weird, I didn't even get gate jobs triggered without tobiash's change | 19:11 |
| Shrews | mordred: may not be that big of deal since a forked process should have its own event loops. just thinking outloud | 19:11 |
| clarkb | with tobiash's change the gate runs and merges | 19:11 |
| clarkb | so still not seeing a +2 without merge, it just says I've got nothing to do | 19:11 |
| tobiash | clarkb: without you have to change the config casing again | 19:11 |
| clarkb | tobiash: oh do I have to lower case everything in my config again? | 19:11 |
| tobiash | clarkb: yepp | 19:11 |
| clarkb | tobiash: I thought zuul was normalizing everything to lower case though? or is it only normalizing what it receives from gerrit? | 19:12 |
| mordred | Shrews: ++ | 19:12 |
| Shrews | pabelanger: nl01 looks good for now. going to monitoring it after it's been running for an extended period | 19:12 |
| Shrews | s/monitoring/monitor/ | 19:12 |
| clarkb | in any case its an easy switch back to lower case so giving that a go | 19:12 |
| tobiash | clarkb: almost everything: https://review.openstack.org/#/c/469946/4/zuul/driver/gerrit/gerritconnection.py | 19:13 |
| tobiash | clarkb: allow_needs takes the label as defined in the config | 19:13 |
| tobiash | clarkb: that's why without the patch there is no combination possible to get the casing right in the config | 19:14 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate console streaming to webpack/yarn https://review.openstack.org/487539 | 19:18 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Use yarn and webpack to manage status javascript https://review.openstack.org/487538 | 19:18 |
| mordred | TIL that an https apt source without apt-transport-https installed makes apt-get update ***HANG*** | 19:20 |
| mordred | not emit an error - just hang | 19:20 |
| clarkb | error: fatal: Failed to submit 1 change due to the following problems: Change 107965: needs Verified | 19:20 |
| clarkb | tobiash: ^ reproduced I think | 19:21 |
| pabelanger | Shrews: wfm | 19:21 |
| *** harlowja has joined #zuul | 19:21 | |
| clarkb | tobiash: jeblair so ya I think we should get tobiash's change rebased and squashed (to deal with merge conflict and failing tests) then get that in. As this affects our mysql set up as well | 19:23 |
| clarkb | I haven't done all this testing with v2 though. Beginning to wonder if it is even necessary | 19:24 |
| clarkb | I don't expect v3 and v2 are very different in this gerrit code | 19:25 |
| *** harlowja has quit IRC | 19:25 | |
| mordred | clarkb: yah - I think that part of the gerrit code is mostly the same | 19:29 |
| tobiash | clarkb: the changes are identical (except tests), but the code locations moved to different files/functions | 19:29 |
| *** jkilpatr has quit IRC | 19:46 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul feature/zuulv3: Migrate console streaming to webpack/yarn https://review.openstack.org/487539 | 19:49 |
| clarkb | tobiash: would you be willing to rebase and squash the two changes on the v3 branch? | 19:55 |
| tobiash | clarkb: almost done... | 19:55 |
| clarkb | ah cool | 19:55 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul feature/zuulv3: Case sensitive label matching https://review.openstack.org/469946 | 19:56 |
| tobiash | clarkb: here it is ^ | 19:56 |
| tobiash | (hopefully with working tests) | 19:56 |
| jeblair | google searches for "sphinx" and "index" are :( [sphinx is also a fulltext search engine] | 19:56 |
| tobiash | did run whole py35 suite, but test_idle seemed had a race once | 19:57 |
| jeblair | i should say, sphinx is a fulltext *document* search engine | 19:57 |
| jeblair | so seriously, major overlap here | 19:57 |
| tobiash | jeblair: test_timer seems to be racy | 20:01 |
| tobiash | jeblair: just ran it a few times and got another fail | 20:02 |
| tobiash | s/test_timer/test_idle | 20:03 |
| tobiash | clarkb: yay, +1 from jenkins | 20:04 |
| jeblair | tobiash: timeout or otherwise? | 20:06 |
| tobiash | jeblair: console text rush for more than a minute and ending with http://paste.openstack.org/show/616622/ | 20:08 |
| tobiash | jeblair: possibly some crash in python | 20:10 |
| mordred | why is did the pep8 job post_failure :( | 20:12 |
| tobiash | mordred: some fetch failure http://paste.openstack.org/show/616625/ | 20:14 |
| tobiash | jeblair: did you encounter such stuff already? http://paste.openstack.org/show/616623/ | 20:14 |
| tobiash | jeblair: that's dominating my console when test_idle fails | 20:15 |
| mordred | tobiash: that is, unfortunately, a "normal" error | 20:16 |
| tobiash | mordred: ? | 20:16 |
| mordred | tobiash: the fetch failure - it's not actually a failure - it's tox/pip being confused by zuul having a 'remote' configured but that remote not being a URL | 20:19 |
| tobiash | mordred: then that seems to be something during/after gzipping the console log | 20:22 |
| tobiash | mordred: can that fail if we archive it and at the same time write to it? | 20:22 |
| mordred | if so I think it would fail every job - since we should be writing to and archiving every time | 20:23 |
| jeblair | tobiash: ah yeah, that's some extra gc debugging info. it's being falsely triggered by a preceding error | 20:25 |
| tobiash | mordred: don't know if that could race: https://github.com/openstack-infra/zuul-jobs/blob/master/roles/upload-logs/tasks/main.yaml#L27 | 20:25 |
| jeblair | tobiash: we could probably drop that, i think we got what it was debugging under control | 20:26 |
| jeblair | i'll make a patch later | 20:26 |
| tobiash | jeblair: ok | 20:26 |
| tobiash | mordred: maybe the answer can be found in executor debug logs (at least which task failed) | 20:27 |
| mordred | tobiash: sigh. I had looked in the SCHEDULER logs and didn't see anything. I will now go look in the executor like a sane person | 20:32 |
| *** hashar has joined #zuul | 20:32 | |
| mordred | nope. still nothing | 20:34 |
| tobiash | mordred: maybe next time | 20:38 |
| mordred | tobiash: yah - I don't like things falling in to a hole like that | 20:39 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Provide nicer index entries for config items https://review.openstack.org/487580 | 20:45 |
| *** dkranz_ has quit IRC | 20:45 | |
| tobiash | eod now | 20:45 |
| tobiash | cya | 20:45 |
| jeblair | tobiash: night! | 20:45 |
| openstackgerrit | Merged openstack-infra/zuul feature/zuulv3: Remove state_dir from setMountsMap https://review.openstack.org/486766 | 20:59 |
| *** jkilpatr has joined #zuul | 21:17 | |
| *** hashar has quit IRC | 21:17 | |
| clarkb | tobiash: jeblair I left a comment on the category case sensitivity change. I think we can/should remove the last use of normalization just to avoid any ambiguity around what is an acceptable value | 21:37 |
| clarkb | mordred: I'm confused by your big thoughts email, are you suggesting we backport web changes to master? | 21:40 |
| clarkb | mordred: if not I'm not sure I'd worry too much about the backward compatibility issues | 21:41 |
| clarkb | (it is a feature branch afterall and we can spam people beforehand) | 21:41 |
| jeblair | clarkb: my understanding is that we go ahead and do this before we release 3.0. | 21:54 |
| jeblair | clarkb: ie, the transition to this is gnarley. we already know the transition from 2->3 is gnarley, so doing this then won't make a difference. doing it *after* 3.0 would be annoying for all. | 21:55 |
| clarkb | jeblair: right but on which branch? CDing relatively stable master has different concerns than CDing not stable v3 branch | 21:55 |
| pabelanger | jeblair: mordred: github credentials now live on zuulv3.o.o | 21:55 |
| pabelanger | I'm guessing we need to restart zuul to pick up settings | 21:56 |
| pabelanger | or reload config | 21:56 |
| pabelanger | will have to do that in the morning, about to head off to docker meetup this evening | 21:56 |
| clarkb | jeblair: if we merge v3 and webapp changes at once into master and tag 3.0 I think thats fine is what I'm trying to get at | 21:57 |
| jeblair | clarkb: tbh, i don't know that we've worked out exactly when we would switch back to master (before, or at v3.0 release). but my guess is still on the feature branch. | 21:57 |
| clarkb | but merging webapp before we merge v3 is where it becomes tricky for consumers | 21:57 |
| jeblair | clarkb: i don't think mordred is suggesting anything go into the master branch before v3 is merged into the master branch | 21:58 |
| clarkb | gotcha | 21:58 |
| mordred | clarkb: yah - this is all v3 related | 21:58 |
| * jeblair sighs with relief | 21:59 | |
| clarkb | I interpreted your concern of breaking all the people as implying you wanted to update master | 21:59 |
| mordred | gotcha - nope, I just want to avoid breaking new v3 users once we release v3 | 21:59 |
| mordred | and since there's no good graceful way for us to release that shift for CD users, I think we should get it done before we 'release v3' | 22:00 |
| *** harlowja has joined #zuul | 22:00 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Provide nicer index entries for config items https://review.openstack.org/487580 | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Use zuul config sphinx directives for pipeline https://review.openstack.org/487604 | 22:03 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove email-filter requirement https://review.openstack.org/487607 | 22:07 |
| clarkb | I'd be curious to see your response to my comments in https://review.openstack.org/#/c/469946/5 | 22:26 |
| clarkb | er jeblair ^ | 22:26 |
| clarkb | I'm going to put tobiash's patch back in place on my test setup and test the you already have a verified +1, you recheck with a workflow +1, does it gate behavior. Newer gerrit should fix that for us I think (but we might have to configure it to do so) | 22:35 |
| jeblair | clarkb: yeah, i think removing normalize category is probably a good idea. that way approval requirements must look the same as reporter actions. | 22:36 |
| jeblair | clarkb: (that's what you were getting at, right?) | 22:37 |
| clarkb | jeblair: ya | 22:37 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 22:37 |
| clarkb | woot new gerrit does indeed fix the preexisting vote isn't seen in the event stream problem we have with 2.11 | 22:38 |
| clarkb | things are looking good with the case sensitivity fix in place | 22:39 |
| clarkb | those were the two items I knew we needed to test. Any other scenarios that check edge cases in gerrit/zuul behavior? | 22:39 |
| jeblair | can't think of any right now, but i have sphinx brain | 22:40 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 22:49 |
| openstackgerrit | Paul Belanger proposed openstack-infra/zuul-jobs master: WIP: Simplify run tox task https://review.openstack.org/487551 | 23:01 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Cleanup pipeline requirements https://review.openstack.org/487618 | 23:11 |
| jeblair | clarkb: there's your requirements doc fix ^ | 23:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove extra GC debug info https://review.openstack.org/487622 | 23:39 |
| clarkb | jeblair: reviewing ^ and in zuul/driver/gerrit/gerrittrigger.py there is approval, require-approval, and reject-approval. Is approval now redundant as well? | 23:39 |
| clarkb | I've also posted comments directly to the change where things are in the files edited | 23:39 |
| clarkb | oh nice removing the gc debugging, I take it that means the problem was sorted out (I don't remember) | 23:40 |
| jeblair | clarkb: hasn't shown up except in a false positive in a while | 23:42 |
| clarkb | jeblair: and commented on the GC change | 23:43 |
| jeblair | clarkb: approval is "this event is the addition of an approval" require-approval is "regardless of what this event is, in order for this event to trigger, there must also be an approval" | 23:44 |
| jeblair | that's an area of docs that still needs revising | 23:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul feature/zuulv3: Remove extra GC debug info https://review.openstack.org/487622 | 23:46 |
| jeblair | clarkb: thanks, fixed ^ | 23:46 |
| clarkb | reading the docs they take both of the same info so it is really confusing as to how they are different | 23:46 |
| clarkb | maybe require-approval can refer to approvals by name? | 23:47 |
| clarkb | that changes things a bit though and may not be worth the headache to update | 23:47 |
| jeblair | clarkb: when you say refer by name, you mean change the code or change the docs? | 23:51 |
| clarkb | jeblair: the code | 23:51 |
| clarkb | but I may be misunderstanding the relationship between them | 23:51 |
| jeblair | clarkb: ah, they might be completely different | 23:52 |
| jeblair | clarkb: here's our prod config: http://paste.openstack.org/show/616639/ | 23:53 |
| clarkb | ah, maybe we call it require-state and approval? | 23:53 |
| jeblair | clarkb: so that works out to "if someone adds a workflow+1, only consider adding it to the check pipeline if jenkins has already left a -2 on it" | 23:54 |
| jeblair | clarkb: 'approval' is the gerrit word for 'vote', that's why we use it. | 23:55 |
| clarkb | anyways, I update my review on that change, I think you did miss a spot unrelated to this thing | 23:55 |
| clarkb | (and tried to explain the reasoning) | 23:55 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!