| openstackgerrit | Merged openstack-infra/zuul-jobs master: Add flag for turning off successful job logs https://review.openstack.org/537986 | 00:01 |
|---|---|---|
| corvus | mordred: ^ | 00:02 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 00:14 |
| *** rcarrillocruz has quit IRC | 00:15 | |
| *** rlandy|biab is now known as rlandy | 00:50 | |
| *** JasonCL has quit IRC | 00:52 | |
| tristanC | corvus: maybe with https://stackoverflow.com/questions/824349/modify-the-url-without-reloading-the-page ? | 00:53 |
| *** JasonCL has joined #zuul | 00:53 | |
| *** JasonCL has quit IRC | 00:58 | |
| *** JasonCL has joined #zuul | 00:59 | |
| tristanC | corvus: also an ui update could also enable stacking the filters, it's possible to do project=zuul&project=zuul-jobs for example | 01:03 |
| corvus | tristanC: yeah, that sounds fine -- i think this is a fine thing to update the history with too (ie, the back button takes you to the previous query) | 01:04 |
| corvus | tristanC: oh, neat, i didn't know about that url trick | 01:05 |
| tristanC | corvus: is the js makefile thing in the scope of the 3.0 release? | 01:05 |
| corvus | tristanC: yeah, mordred's tooling patch is (so we have an automatable way to deploy the static stuff) | 01:06 |
| corvus | tristanC: i think it's really close too -- last iteration only had a few things that needed fixing | 01:06 |
| tristanC | corvus: ok good, then i'll rebase my changes on top of it | 01:06 |
| corvus | tristanC: last i looked it hadn't been re-proposed to master | 01:07 |
| tristanC | corvus: fwiw i've been working on some d3 representations, here is a preview of the job graph: http://38.145.32.112/zuul/local/jobs.html (see the 'toggle graph' button on top right) | 01:07 |
| corvus | tristanC: that's cool! | 01:08 |
| tristanC | corvus: similarly for project pipeline, this one is fun to play with: http://38.145.32.112/zuul/local/project.html?project_name=zuul-distgit | 01:08 |
| corvus | tristanC: i bet it's not going to scale to big systems though, so it may be worth thinking about how to keep that managable. like maybe rather than drawing the whole system, when you're on a job, draw only its graph. | 01:09 |
| tristanC | corvus: definitely, it's just experiment with d3 to see what could we do with the rest data | 01:10 |
| corvus | tristanC: ++ it has potential :) | 01:11 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 01:14 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 01:23 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 01:24 |
| *** yolanda has quit IRC | 01:30 | |
| *** JasonCL has quit IRC | 02:01 | |
| *** harlowja has quit IRC | 02:16 | |
| *** JasonCL has joined #zuul | 02:31 | |
| *** JasonCL has quit IRC | 02:37 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Use yarn and webpack to manage zuul-web javascript https://review.openstack.org/538099 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add babel transpiling enabling use of ES6 features https://review.openstack.org/538125 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add StandardJS linting and analysis https://review.openstack.org/538126 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix source_url handling for jobs view https://review.openstack.org/538127 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Fix StandardJS warnings and turn them to errors https://review.openstack.org/538128 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Add bundle analysis to the lint target https://review.openstack.org/538129 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Inject url endpoint information https://review.openstack.org/538130 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Make bundle of build web content https://review.openstack.org/538131 | 03:10 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: Remove use strict https://review.openstack.org/538132 | 03:10 |
| *** JasonCL has joined #zuul | 03:37 | |
| *** JasonCL has quit IRC | 03:47 | |
| *** JasonCL has joined #zuul | 04:02 | |
| *** JasonCL has quit IRC | 04:06 | |
| *** JasonCL has joined #zuul | 04:39 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add /{tenant}/jobs/{job_name} route https://review.openstack.org/535545 | 04:41 |
| *** JasonCL has quit IRC | 04:43 | |
| *** harlowja has joined #zuul | 04:51 | |
| *** JasonCL has joined #zuul | 04:56 | |
| *** JasonCL has quit IRC | 05:00 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add jobs graph rendering https://review.openstack.org/537869 | 05:17 |
| *** JasonCL has joined #zuul | 05:26 | |
| *** JasonCL has quit IRC | 05:31 | |
| *** JasonCL has joined #zuul | 05:44 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: /{tenant}/projects.json routes https://review.openstack.org/537870 | 05:49 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul master: web: add project pipeline rendering https://review.openstack.org/537871 | 05:49 |
| *** JasonCL has quit IRC | 05:55 | |
| *** JasonCL has joined #zuul | 06:07 | |
| *** JasonCL has quit IRC | 06:12 | |
| *** JasonCL has joined #zuul | 06:29 | |
| *** JasonCL has quit IRC | 06:33 | |
| *** JasonCL has joined #zuul | 06:43 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Fix cross source dependencies for github https://review.openstack.org/537428 | 06:44 |
| *** JasonCL has quit IRC | 06:48 | |
| tobiash | corvus: I'm currently rethinking the parent-change-enqueued and neededby check for github apps | 06:49 |
| tobiash | http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/driver/github/githubconnection.py#n730 | 06:49 |
| tobiash | corvus: what do you think about making that tenant scoped? | 06:49 |
| tobiash | corvus: that should make that operation much cheaper in multi tenant environments like ours | 06:50 |
| tobiash | corvus: and I think sacrificing cross-tenant parent-change-enqueued events would be ok in comparison to not being able to use parent-change-enqueued | 06:51 |
| *** rlandy has quit IRC | 06:57 | |
| *** harlowja has quit IRC | 07:23 | |
| *** JasonCL has joined #zuul | 07:24 | |
| *** JasonCL has quit IRC | 07:29 | |
| *** rcarrillocruz has joined #zuul | 07:47 | |
| *** elyezer has quit IRC | 08:01 | |
| *** elyezer has joined #zuul | 08:13 | |
| *** JasonCL has joined #zuul | 08:19 | |
| *** JasonCL has quit IRC | 08:23 | |
| *** JasonCL has joined #zuul | 08:40 | |
| *** jpena|off is now known as jpena | 08:43 | |
| *** JasonCL has quit IRC | 08:44 | |
| *** JasonCL has joined #zuul | 09:02 | |
| *** JasonCL has quit IRC | 09:07 | |
| *** Cibo has quit IRC | 09:16 | |
| *** electrofelix has joined #zuul | 09:53 | |
| *** JasonCL has joined #zuul | 09:56 | |
| *** JasonCL has quit IRC | 10:03 | |
| *** Cibo has joined #zuul | 10:18 | |
| *** JasonCL has joined #zuul | 10:57 | |
| *** JasonCL has quit IRC | 11:06 | |
| *** sshnaidm|off has quit IRC | 11:11 | |
| *** sshnaidm has joined #zuul | 11:12 | |
| *** JasonCL has joined #zuul | 11:16 | |
| *** jkilpatr has quit IRC | 11:24 | |
| *** JasonCL has quit IRC | 11:25 | |
| *** JasonCL has joined #zuul | 11:33 | |
| *** JasonCL has quit IRC | 11:39 | |
| *** jkilpatr has joined #zuul | 11:59 | |
| *** elyezer has quit IRC | 12:24 | |
| *** elyezer has joined #zuul | 12:37 | |
| *** jpena is now known as jpena|lunch | 12:39 | |
| *** JasonCL has joined #zuul | 12:53 | |
| *** JasonCL has quit IRC | 13:00 | |
| *** elyezer has quit IRC | 13:03 | |
| *** elyezer has joined #zuul | 13:05 | |
| dmsimard | tobiash, tristanC: created another zuul-web story https://storyboard.openstack.org/#!/story/2001502 -- "Additional build filters for date and status" | 13:05 |
| *** JasonCL has joined #zuul | 13:08 | |
| *** rlandy has joined #zuul | 13:34 | |
| *** jpena|lunch is now known as jpena | 13:45 | |
| *** dmellado has joined #zuul | 13:59 | |
| tobiash | yay http header case conversions | 14:06 |
| tobiash | github webhook tests work, reality not... | 14:06 |
| tobiash | looks like every http framework messes with header casing and each does it differently | 14:20 |
| *** yolanda has joined #zuul | 14:54 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Split out erasing from 'info' command into 'erase' https://review.openstack.org/537663 | 14:57 |
| Shrews | oh the joys of tox on a Friday: [Errno 25] Inappropriate ioctl for device | 15:31 |
| dmsimard | tobiash: btw we merged the fixes for zuul jobs log uploading now | 15:33 |
| dmsimard | tobiash: is everything ok ? | 15:33 |
| tobiash | dmsimard: thanks | 15:33 |
| tobiash | I personally was not affected as we work with a local mirror | 15:34 |
| tobiash | but perhaps other people like tristanC work directly with upstream zuul-jobs | 15:34 |
| mordred | tobiash: SOOO ... | 15:35 |
| corvus | at the very least, it meant your couldn't update your local mirror | 15:35 |
| dmsimard | Ok, we'll try to be more careful | 15:35 |
| dmsimard | Thanks for calling us out on it :) | 15:35 |
| tobiash | corvus: yeah, we hadn't time to automate that yet ;) | 15:35 |
| corvus | tobiash: when you say tenant scoped for needed-by -- you mean: get the installations for all projects in the tenant and search with all of them? i think that's okay. | 15:36 |
| corvus | tobiash: also, i had a conversation with mordred a while ago about the idea of making the change cache persistent (eg, in zookeeper, maybe?) and then potentially relying on that for that query. that's a long ways off, but i consider anything we do now for this to be a temporary compromise. | 15:38 |
| mordred | ++ | 15:38 |
| tobiash | corvus: Yes I mean that | 15:39 |
| tobiash | hrm, now I have a really working webhook over zuul-web and I cannot get that patch through our proxy :/ | 15:41 |
| dmsimard | the zuul autokeep feature I suggested would be quite useful right about now.. | 15:45 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 15:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 15:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 15:48 |
| * tobiash shakes fist at mandatory authenticating proxy | 15:48 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add web server URL tests https://review.openstack.org/538291 | 15:50 |
| corvus | tristanC: if you're taking over mordred's yarn/webpack changes -- we need that test ^ to pass before we can merge it. it currently doesn't, which means that we're not able to actually serve the static content at some urls. | 15:51 |
| corvus | tristanC: we also need to update that change to include the submount rewrite you use in softwarefactory | 15:52 |
| tristanC | corvus: mordred: i didn't meant to take over the yarn thing, i just restore the patch to rebase the new dashboard pages. i may have some time to have a look at it next week | 15:53 |
| tristanC | corvus: what do you mean by subomunt rewrite? | 15:53 |
| corvus | tristanC: can you point me at your rewrite rules again? | 15:54 |
| corvus | (i know you did this 2 weeks ago, but i lost track, sorry) | 15:54 |
| Shrews | would someone like to +3 tobiash's change here: https://review.openstack.org/537776 It's going to conflict with a change I want to get in, and it's and easy review | 15:55 |
| Shrews | an* easy review | 15:55 |
| corvus | Shrews: will do | 15:55 |
| Shrews | corvus: thank you | 15:55 |
| tristanC | corvus: the current apache configuration is https://softwarefactory-project.io/r/gitweb?p=software-factory/sf-config.git;a=blob;f=ansible/roles/sf-gateway/templates/gateway.common.j2;hb=HEAD#l166 | 15:55 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add web server URL tests https://review.openstack.org/538291 | 15:57 |
| corvus | tristanC: there we go ^ take a look at the last class - TestSuburl | 15:58 |
| tristanC | corvus: though this apache configuration we'll be mostly rewritten to accomodate the new bundled js stuff, either still hosted on the gateway, or just entirely proxy passed to zuul-web | 15:59 |
| corvus | tristanC: right -- that test should verify that if you entirely proxy to zuul-web, the bundled js will actually be served. | 16:00 |
| corvus | right now, i believe there's a bug that will cause at least some of those tests to fail. | 16:01 |
| mordred | tristanC, corvus: yah - I'll jump on fixing that stack next week | 16:01 |
| corvus | mordred: cool. that test should be ready to go -- at least, it has a really basic check for all 3 ways of hosting | 16:02 |
| tristanC | corvus: mordred: thanks, i'll test this out more thorougly next week | 16:02 |
| *** myoung|pto has quit IRC | 16:09 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Default max pool resources to math.inf https://review.openstack.org/537776 | 16:09 |
| tristanC | corvus: fwiw i used this temporary configuration to plug the npm dev server https://softwarefactory-project.io/paste/show/956/ , not sure if it's already documented/supported | 16:10 |
| dmsimard | mordred: before I go on and fix what I think is a bug.. where did you retrieve that exit code 4 from ? https://github.com/openstack-infra/zuul/blob/master/zuul/executor/server.py#L1415 | 16:12 |
| dmsimard | mordred: looks to me like exit 4 is about unreachable hosts https://github.com/ansible/ansible/blob/stable-2.3/lib/ansible/executor/task_queue_manager.py#L62-L67 | 16:12 |
| *** myoung has joined #zuul | 16:13 | |
| mordred | dmsimard: I *believe* that was from experience? | 16:13 |
| mordred | dmsimard: yah - just tested ... an unparseable yaml returns 4 | 16:14 |
| dmsimard | o_O | 16:14 |
| mordred | dmsimard: http://paste.openstack.org/show/653664/ | 16:15 |
| mordred | dmsimard: it's certainly possible/likely that they're re-using an exit code | 16:15 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Do not attempt to handle requests when disabled https://review.openstack.org/537954 | 16:17 |
| mordred | dmsimard: I have also verified a 4 returned from an invalid host | 16:17 |
| mordred | dmsimard: http://paste.openstack.org/show/653665/ | 16:17 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Update docs to reflect newer Python requirement. https://review.openstack.org/538296 | 16:22 |
| Shrews | oopsie in the docs ^^^ | 16:23 |
| dmsimard | mordred: re-using the same exit codes for different things is bad | 16:30 |
| dmsimard | incidentally, I was investigating a playbook that failed with exit code 2 which isn't handled by the executor | 16:30 |
| mordred | dmsimard: oh, fascinating | 16:42 |
| dmsimard | mordred: yeah, this one: http://paste.openstack.org/raw/653651/ | 16:45 |
| pabelanger | I also had no luck trying to ignore_errors for UNREACHABLE, and confirm just returns -4 from ansible-playbook | 16:45 |
| dmsimard | mordred: I have nothing to work with in the logs, that's like all I have. I have no idea what failed and why it failed -- it's a release job so even requeuing it (especially without the zuul inventory file) is not trivial | 16:45 |
| pabelanger | dmsimard: which executor? | 16:48 |
| pabelanger | looks like exit code 2 | 16:48 |
| pabelanger | then we failed to upload logs, | 16:49 |
| dmsimard | pabelanger: yeah but exit code 2 isn't handled here: https://github.com/openstack-infra/zuul/blob/master/zuul/executor/server.py#L1401-L1445 | 16:54 |
| dmsimard | and exit code 2 is ambiguous -- I asked #ansible-devel about it and they didn't really manage to explain it every well :) | 16:55 |
| pabelanger | right, exit 2 usually something failed on node, maybe executor | 16:56 |
| pabelanger | would have to see what the playbook did | 16:56 |
| dmsimard | pabelanger: it's a313a980d05f4526aba5128189f45cd7 on ze05, I was investigating for dhellmann due to a release failure: http://lists.openstack.org/pipermail/release-job-failures/2018-January/000733.html | 17:01 |
| pabelanger | k, lets move back to openstack-infra | 17:03 |
| dmsimard | sure | 17:03 |
| *** harlowja has joined #zuul | 17:10 | |
| *** jpena is now known as jpena|off | 17:30 | |
| *** jkilpatr has quit IRC | 17:56 | |
| tobiash | Shrews: any idea how I fix a node which is ready but locked? | 18:00 |
| tobiash | I have one which is in that state since three hours | 18:00 |
| *** yolanda has quit IRC | 18:01 | |
| *** rlandy is now known as rlandy|brb | 18:02 | |
| *** elyezer has quit IRC | 18:03 | |
| *** elyezer has joined #zuul | 18:15 | |
| *** Cibo has quit IRC | 18:19 | |
| corvus | tobiash: first thing is probably figure out who has it locked. telnet to zookeeper port 2181 and type 'dump' | 18:21 |
| *** Cibo has joined #zuul | 18:21 | |
| corvus | tobiash: it will show you ephemeral nodes grouped by connection, so look for the lock you're interested in, and also see what else it has locked | 18:22 |
| tobiash | is it possible that a launcher picks its old session on startup? | 18:22 |
| corvus | tobiash: it should not | 18:22 |
| corvus | here's an example from our system: http://paste.openstack.org/show/653773/ | 18:23 |
| tobiash | earlier today I had this issue and a launcher restart released it | 18:23 |
| corvus | (you can see that's obviously the connection for nl03, and it has a lock on several nodes) | 18:23 |
| tobiash | corvus: thanks for this tip, that's cool | 18:24 |
| tobiash | unfortunately I already deleted the lock in the zkCli which released it | 18:24 |
| tobiash | but next time I'll check the dump | 18:25 |
| corvus | yeah, that may help us find the bug | 18:25 |
| tobiash | btw, I had zookeeper problems throughout the week due to slow disk io | 18:27 |
| tobiash | now I changed zookeeper to 5 replicas on tmpfs and now it's stable :) | 18:27 |
| *** rlandy|brb is now known as rlandy | 18:28 | |
| *** Cibo has quit IRC | 18:28 | |
| *** Cibo has joined #zuul | 18:29 | |
| tobiash | what I observed is that if an fsync takes several seconds (15) to complete (e.g. due to overloaded storage) it can kick nodepool out of the session | 18:29 |
| *** electrofelix has quit IRC | 18:30 | |
| corvus | tobiash: oh interesting. there may be tunables for that, but perhaps it's better to do what you did (since the tunables would just make the system very slow) | 18:30 |
| tobiash | that's what I also thought | 18:30 |
| tobiash | so my takeaway for this is a few more replicas more to better cope with node crashes | 18:32 |
| tobiash | draw back is if you have a full system shutdown you have to rebuild all images | 18:32 |
| tobiash | but that takes us currently at most 1-2 hours so I'll take this risk | 18:32 |
| *** weshay is now known as weshay|ruck|brb | 18:43 | |
| *** harlowja has quit IRC | 19:14 | |
| *** jkilpatr has joined #zuul | 19:26 | |
| *** weshay|ruck|brb is now known as weshay|ruck | 19:27 | |
| *** harlowja has joined #zuul | 19:46 | |
| *** harlowja_ has joined #zuul | 19:58 | |
| *** harlowja has quit IRC | 20:00 | |
| Shrews | tobiash: besides what corvus said, the first thing is to always check the allocated_to value of the node for the request #, then understand what's happening to that request by looking at the debug logs | 20:34 |
| Shrews | tobiash: i've found that there's usually a valid reason for it | 20:35 |
| Shrews | paused request handler is the usual culprit | 20:36 |
| Shrews | if that's not it, then the next likely culprit is an unhandled exception causing chaos in our logic | 20:38 |
| *** jkilpatr has quit IRC | 20:57 | |
| corvus | mordred, tobiash, Shrews: i'd appreciate a review on https://review.openstack.org/537655 when you get a chance. gets the ansible reporting moving. | 21:22 |
| Shrews | corvus: lgtm | 21:27 |
| mordred | corvus: bother. I totally reviewed through that already and didn't actually leave a vote | 21:27 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 21:44 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul-website master: Add initial website content and jobs https://review.openstack.org/538039 | 21:45 |
| *** rlandy has quit IRC | 21:49 | |
| openstackgerrit | Merged openstack-infra/zuul master: Use override-checkout to select jobs https://review.openstack.org/537655 | 21:49 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Don't override allowed-projects https://review.openstack.org/538353 | 22:29 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!