| *** goern has quit IRC | 00:56 | |
| dmsimard | I really like where the UI is going | 02:37 |
|---|---|---|
| dmsimard | live status, searchable jobs, builds, descriptions | 02:37 |
| dmsimard | ++ zuul-web | 02:37 |
| dmsimard | happy to see the progress bar padding made it haha | 02:39 |
| dmsimard | a meager contribution | 02:39 |
| *** bhavikdbavishi has joined #zuul | 02:47 | |
| *** pwhalen has quit IRC | 04:25 | |
| *** bhavikdbavishi1 has joined #zuul | 05:47 | |
| *** evrardjp_ has joined #zuul | 05:51 | |
| *** jpenag has joined #zuul | 05:53 | |
| *** ianw_ has joined #zuul | 05:54 | |
| *** bhavikdbavishi has quit IRC | 05:55 | |
| *** Diabelko has quit IRC | 05:55 | |
| *** gothicmindfood has quit IRC | 05:55 | |
| *** jpena|off has quit IRC | 05:55 | |
| *** evrardjp has quit IRC | 05:55 | |
| *** chkumar|off has quit IRC | 05:55 | |
| *** ianw has quit IRC | 05:55 | |
| *** jlvillal has quit IRC | 05:55 | |
| *** aluria has quit IRC | 05:55 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 05:55 | |
| *** ianw_ is now known as ianw | 05:55 | |
| *** Diabelko has joined #zuul | 06:03 | |
| *** chandankumar has joined #zuul | 06:06 | |
| *** bhavikdbavishi has quit IRC | 06:48 | |
| *** evrardjp_ is now known as evrardjp | 07:37 | |
| *** goern has joined #zuul | 07:42 | |
| *** hashar has joined #zuul | 07:54 | |
| *** SotK has joined #zuul | 08:06 | |
| *** panda|off is now known as panda | 08:32 | |
| *** rfolco|rover has quit IRC | 08:35 | |
| *** electrofelix has joined #zuul | 09:58 | |
| *** ssbarnea has joined #zuul | 10:12 | |
| *** bhavikdbavishi has joined #zuul | 10:28 | |
| *** bhavikdbavishi has quit IRC | 10:32 | |
| *** bhavikdbavishi has joined #zuul | 10:34 | |
| *** jpenag is now known as jpena | 10:36 | |
| *** bhavikdbavishi has quit IRC | 10:38 | |
| *** ssbarnea has quit IRC | 10:49 | |
| *** AJaeger_ has joined #zuul | 10:57 | |
| *** AJaeger has quit IRC | 10:59 | |
| *** jpena is now known as jpena|lunch | 11:01 | |
| *** EmilienM is now known as EvilienM | 11:24 | |
| *** panda is now known as panda|lunch | 11:29 | |
| *** hashar is now known as hasharAway | 11:31 | |
| *** jpena|lunch is now known as jpena | 11:57 | |
| *** panda|lunch is now known as panda | 12:24 | |
| *** rfolco|rover has joined #zuul | 12:34 | |
| *** rlandy has joined #zuul | 12:36 | |
| openstackgerrit | Simon Westphahl proposed openstack-infra/zuul master: Use branch for grouping in supercedent manager https://review.openstack.org/613335 | 12:44 |
| *** AJaeger_ is now known as AJaeger | 13:18 | |
| dmsimard | I created a short video demo of the Zuul web interface and the workflow/interaction with ara for troubleshooting jobs: https://www.youtube.com/watch?v=Rg4vdc0ptgA | 13:28 |
| dmsimard | I wanted to start out with a gif but it ended up being too long eh | 13:28 |
| *** chandankumar is now known as chkumar|off | 13:33 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: DNM Link to change page from status panel https://review.openstack.org/613593 | 14:10 |
| mordred | corvus, tristanC: ^^ I tried that locally- and I think it's a terrible idea after having looked at it, but I thought I'd push it up so you could look at it too | 14:10 |
| mordred | corvus, tristanC: after having poked/looked - I think I've convinced myself that if we link to the change page from code review it's enough. if we do link to it from the status page, I think adding a little icon in the change panel that looks like an expand/full-screen icon would be he way to go - but I think that might get busy | 14:12 |
| fungi | corvus: is the idea about basing priority on project-group to alleviate the situation where a dependent pipeline has three changes from one project followed by one change from another project and we end up prioritizing jobs for the changes in position 1 and 4 but then the change in position 4 is still stuck waiting for the changes in position 2 and 3 to get tested anyway before it can merge (possibly even | 14:17 |
| fungi | further delaying 4 when 3 eventually gets tested and failed and so 4 has to be retested anyway)? | 14:17 |
| fungi | it seems like performance wise for dependent pipelines it would make sense to prioritize node requests for the first change(ish) in each queue, then the second in each queue, and so on, but maybe i'm misunderstanding the proposal somewhat | 14:19 |
| mordred | fungi: I think, based on my reading, that the last thing you said (prioritize node requests for first change) is what basing priority on project-group (as determined by membership in shared queue) would be achieving | 14:22 |
| fungi | yeah, i thought so too. just not sure why it was mentioned as a "we might even want to..." | 14:22 |
| corvus | mordred, fungi: yes. | 14:23 |
| fungi | seems like if we don't the result would be worse than what we have already | 14:23 |
| corvus | fungi: mostly because i haven't fully thought out all of the different tweaks we might need in the algorithm(s). i was mostly focused on a mechanism that could do something hand-wavey like this. i think you make a compelling case for why, in fact, at least in a dependent pipeline, we need to do exactly that. :) | 14:24 |
| corvus | maybe the result too is that the different pipeline managers behave slightly differently wrt priority. dependent does that, independent is strictly per-project, etc. | 14:24 |
| fungi | okay, cool. just wanted to be sure i wasn't misunderstanding. for independent pipelines i agree a per-project round-robin ought to work out fine | 14:24 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: quick-start: add a note about github https://review.openstack.org/613398 | 14:25 |
| mordred | corvus: fixed the sphinx issue in that ^^ | 14:26 |
| corvus | mordred: re the change link -- i think your expand-icon idea may be worth exploring... (i haven't seen the result of the DNM change you pushed up yet, but i will look when it's available) | 14:27 |
| corvus | mordred: i certainly do understand the potential ui problems with linking to the change page and agree it warrants careful consideration :) | 14:27 |
| corvus | mordred: maybe putting it in the top-right part of the box (so it's not near the change number) will help indicate it's something that will "embiggen the box" rather than link to the change itself. | 14:29 |
| Shrews | corvus: i don't think you'd have to add a new priority field to the znode. just alter the precedence value from, say, 200, to 200-<priority>. | 14:33 |
| Shrews | nodepool wouldn't need to change at all then | 14:33 |
| Shrews | except for caching things | 14:33 |
| Shrews | (at least, i don't *think* it'd have to change for that. it just does a reverse sort of that field, iirc) | 14:35 |
| corvus | Shrews: right, but the priority needs to be able to change (possibly many times) after the request is created. | 14:37 |
| tristanC | mordred: i also agree, from the main status page, using filter is more efficient than loading a dedicated per change status page. it seems like the only use for such page is to be referenced from code review system. | 14:37 |
| Shrews | oh, hrm | 14:37 |
| mordred | corvus: ya - Im mostly concerned about number of pixels the icon would take up | 14:37 |
| corvus | tristanC: see my comment on the parent change for why we should try to add it. | 14:37 |
| mordred | corvus: but that was what I was thinking ... upper right corner of box | 14:38 |
| fungi | the former network engineer in me wonders if we can learn anything from traditional fair queuing and flow control algorithms for network traffic which could apply to the zuul problem space... | 14:38 |
| * fungi should reacquaint himself with the altq implementations in *bsd | 14:40 | |
| corvus | mordred: looks like https://review.openstack.org/613161 is passing if you want to get that moving again | 14:48 |
| corvus | oh sweet, http://logs.openstack.org/27/613027/4/check/zuul-quick-start/113c721/container_logs/ works | 14:49 |
| mordred | corvus: I was just reviewing that in fact | 14:49 |
| mordred | corvus: I think once that lands we can revert revert the pep8 patch - and add a flag to it so that people can disable it | 14:49 |
| *** ssbarnea|bkp2 has quit IRC | 14:51 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Add the process environment to zuul.conf parser https://review.openstack.org/612824 | 14:51 |
| corvus | mordred: ya, also we should have something copy the zuul return file into the logs... either in the pep8 role or maybe the base job? i looked and i think it would be awkward to have zuul_return itself do that. | 14:52 |
| corvus | mordred: (or, temporarily, have the pep8 role call zuul_return twice -- you can pass a path argument, so the second invocation could write it into the log dir) | 14:53 |
| mordred | corvus: maybe that's a good thing to do at the start | 14:54 |
| Shrews | tobiash: that test failure in the node cache change is interesting | 14:54 |
| corvus | SpamapS: ^ we should get logs on 612824 now -- though also see my comment on that about adding unit test coverage | 14:54 |
| tobiash | Shrews: I saw that this morning and I think that's two test races I need to address but didn't have time today to do so | 14:55 |
| corvus | fyi -- i'm about to be afk until wednesday | 14:55 |
| corvus | oh, i wonder if 'docker logs' emits things to stdout *and* stderr | 14:57 |
| corvus | because http://logs.openstack.org/27/613027/4/check/zuul-quick-start/113c721/container_logs/gerrit.log looks really sparse | 14:58 |
| corvus | so maybe that needs to be >file 2>&1 | 14:58 |
| *** ssbarnea has joined #zuul | 15:00 | |
| Shrews | tobiash: i might poke at it a bit after lunch | 15:00 |
| *** hasharAway is now known as hashar | 15:12 | |
| SpamapS | corvus: Indeed, I was kind of feeling bad about not having unit test coverage. ;) | 15:52 |
| *** gothicmindfood has joined #zuul | 15:54 | |
| tobiash | Shrews: the test_node_caching fail is because I was just too stupid to correctly use iterate_timeout | 16:13 |
| tobiash | Shrews: and the second one is a test race | 16:13 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 16:20 |
| tobiash | that should fix the test_node_caching ^ | 16:20 |
| logan- | when a with_items is used in a zuul playbook it seems like the console log only receives output for the first item, subsequent items just show "ok: Item: <item> Runtime: 0:03:36.089373" -- has anyone else seen this? I am running a somewhat older sha of Zuul (July-ish) so this might have been addressed later on | 16:20 |
| tobiash | logan-: yes I see this too in our master++ deployment, but didn't have time yet to do a deeper analysis | 16:22 |
| tobiash | logan-: that's likely a bug in the zuul_stream callback plugin | 16:22 |
| logan- | tobiash: thanks, agreed regarding the callback. ara report shows the full output | 16:26 |
| mordred | item iteration in callbacks is ... fun | 16:27 |
| dmsimard | mordred: it's not | 16:27 |
| dmsimard | :D | 16:27 |
| *** hashar is now known as hasharAway | 16:28 | |
| tobiash | dmsimard: you're expert on this, maybe you see what's wrong ;) | 16:29 |
| mordred | dmsimard: yah. by "fun" I did, in fact, mean "not fun" | 16:29 |
| dmsimard | logan-: have a link to an example ? | 16:30 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 16:31 |
| tobiash | Shrews: and I think that should resolve that test race ^ | 16:31 |
| dmsimard | tobiash: I can look, I'm biased but I don't use the actual job-output.txt file very much :P | 16:35 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 16:41 |
| Shrews | tobiash: :) | 16:42 |
| tobiash | Shrews, corvus: I just crafted a test case that creates 200 node requests and waits until all are fulfilled. On master that took 174s, with node caching 49s on my laptop with local zk on tmpfs | 16:53 |
| Shrews | \o/ | 16:56 |
| tobiash | Shrews, corvus: I have a further idea to optimize, we currently run updateNodeStats on every create and delete of a server which is pretty costly even if cached. What do you think about doing that asynchronously with a rate limit to let;s say every 10s? | 16:58 |
| tobiash | that actually costs 30s in my test case (the 49s was with deactivated updateNodeStats). With functional updateNodeStats the cached time was 79s | 17:00 |
| logan- | dmsimard: no sorry, I don't. its on an internal repo. but it seems easy to repro.. just with_items over a command task and you'll see output from the first command but not the rest | 17:00 |
| Shrews | tobiash: i'd be fine with that change | 17:03 |
| tobiash | Shrews: currently this is done by each thread that launches or deletes a node. Would you move that to a stats thread of each provider and just notify it on node creation/deletion? | 17:05 |
| *** jpena is now known as jpena|off | 17:06 | |
| Shrews | tobiash: i might consider something a bit simpler. have the reporter cache results until X seconds have passed, then report to statsd | 17:14 |
| Shrews | but i haven't thought that all the way through | 17:15 |
| tobiash | Shrews: you mean adding a global cache per provider in the stats reporter? | 17:15 |
| Shrews | tobiash: yeah | 17:15 |
| tobiash | ok, that is simple and would work | 17:16 |
| tobiash | I'm just unsure if we want to introduce a global var for that | 17:16 |
| Shrews | i always prefer the simpler approach... unless it doesn't work :) | 17:17 |
| tobiash | but on the other side I don't want to add the nodepool object to every nodelauncher | 17:17 |
| Shrews | it doesn't have to be global. could be a class attr | 17:17 |
| tobiash | does that work? It's just a base class of nodedeleter and launcher | 17:18 |
| Shrews | part of StatsReporter | 17:18 |
| tobiash | will test that | 17:18 |
| Shrews | it'd have to be a class attribute, not instance attr | 17:19 |
| tobiash | Shrews: the only problem I see with this approach is that if we hit the cache and then don't have an event for a long time we end up with wrong stats | 17:20 |
| tobiash | Shrews: so we might need to kick off a thread that updates the stats after 10s | 17:21 |
| tobiash | hrm, that might get more complicated than I thought | 17:22 |
| tobiash | I'll try something | 17:23 |
| *** electrofelix has quit IRC | 17:32 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Cleanup down ports https://review.openstack.org/609829 | 17:41 |
| *** toabctl has quit IRC | 19:19 | |
| *** hasharAway is now known as hashar | 19:20 | |
| *** toabctl has joined #zuul | 19:21 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Small script to scrape Zuul job cpu usage https://review.openstack.org/613674 | 19:35 |
| *** j^2 has quit IRC | 19:40 | |
| openstackgerrit | Merged openstack-infra/zuul master: quick-start: add a note about github https://review.openstack.org/613398 | 19:42 |
| openstackgerrit | Jeremy Stanley proposed openstack-infra/zuul master: Add reenqueue utility https://review.openstack.org/613676 | 19:44 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Support node caching in the nodeIterator https://review.openstack.org/604648 | 20:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats https://review.openstack.org/613680 | 20:03 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Rate limit updateNodeStats https://review.openstack.org/613680 | 20:09 |
| tobiash | Shrews: fixed some further test races and implemented the nodestats rate limit ^ | 20:10 |
| *** openstackstatus has quit IRC | 20:42 | |
| *** openstack has joined #zuul | 20:46 | |
| *** ChanServ sets mode: +o openstack | 20:46 | |
| *** hashar has quit IRC | 20:57 | |
| *** pwhalen has joined #zuul | 21:20 | |
| *** jimi|ansible has quit IRC | 21:41 | |
| *** rlandy has quit IRC | 21:41 | |
| *** pcaruana has quit IRC | 23:10 | |
| *** jesusaur has quit IRC | 23:45 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!