| openstackgerrit | Merged openstack-infra/zuul-jobs master: Fix noqa warning https://review.openstack.org/613785 | 00:10 |
|---|---|---|
| *** dmsimard has quit IRC | 00:28 | |
| openstackgerrit | Merged openstack-infra/nodepool master: Cleanup down ports https://review.openstack.org/609829 | 01:12 |
| *** bhavikdbavishi has joined #zuul | 01:36 | |
| *** bhavikdbavishi has quit IRC | 02:08 | |
| *** rcarrillocruz has quit IRC | 02:43 | |
| *** bhavikdbavishi has joined #zuul | 03:11 | |
| *** pwhalen has joined #zuul | 03:13 | |
| *** dmsimard has joined #zuul | 04:27 | |
| *** dmsimard has quit IRC | 04:45 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Logs stats for nodepool automated cleanup https://review.openstack.org/614074 | 04:57 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Logs stats for nodepool automated cleanup https://review.openstack.org/614074 | 05:01 |
| *** dmsimard has joined #zuul | 05:10 | |
| tobiash | clarkb, mordred: https://review.openstack.org/613674 looks tailored a bit to the openstack ecosystem. Shall this really be part of zuul or rather live in project-config? | 05:16 |
| clarkb | tobiash ya I think everything but that is generally useful | 05:19 |
| clarkb | and if we assume a generic grouping method we could maybe feed in a "projects" file that isnt openstack specific? | 05:20 |
| clarkb | fwiw it totally works if that file isnt present but the output may say too mamy openstacks | 05:20 |
| clarkb | feel free to -1 and I'll see if I can clean it up to be more general | 05:20 |
| clarkb | mostly I think zuul needs something like that for people to better understand usage and the openstack bits can be cleaned up | 05:23 |
| tristanC | clarkb: it seems like we could build such report from the build endpoint instead, but it is still missing nodeset information | 05:28 |
| clarkb | tristanC: correct. This was a quicker easy step that corvus pushed up | 05:29 |
| tobiash | clarkb: yes, I'd really like that to have in zuul, I just stumbled over the openstackiness ;) | 05:29 |
| tristanC | another informations that may be useful is the nodesets/labels used by each project | 05:32 |
| tobiash | clarkb: I'm fine feeding in a projects file as a followup. Regarding the openstack comment, I think this can be cleaned up quickly :) | 05:36 |
| *** pcaruana has joined #zuul | 05:36 | |
| tobiash | clarkb: a further think I stumbled accross is that this script makes a difference between project and repo where zuul itself uses these terms as synonyms | 05:37 |
| tobiash | this can be quite confusing and might need more explanation in the script | 05:37 |
| clarkb | yes I even stumbled over that myself abit if you look at older patchsets. Basically tries to show a differemce between logical collections of wokr and repo specific work | 05:41 |
| *** pcaruana has quit IRC | 05:47 | |
| *** mnaser has quit IRC | 06:16 | |
| *** mnaser has joined #zuul | 06:17 | |
| *** smyers has quit IRC | 06:18 | |
| *** smyers has joined #zuul | 06:18 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Add request handler timer to stats https://review.openstack.org/614079 | 06:31 |
| tobiash | Shrews, ianw, corvus: what do you think about this metric? ^ | 06:31 |
| tobiash | if you think that is valuable I'll add documentation and release notes | 06:31 |
| *** quiquell|off is now known as quiquell | 06:34 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Logs stats for nodepool automated cleanup https://review.openstack.org/614074 | 07:17 |
| ianw | tobiash: lgtm | 07:23 |
| tobiash | :) | 07:23 |
| *** themroc has joined #zuul | 07:43 | |
| *** pcaruana has joined #zuul | 07:57 | |
| *** pcaruana is now known as pcaruana|elisa| | 07:59 | |
| *** quiquell has quit IRC | 08:12 | |
| *** jpena|off is now known as jpena | 08:57 | |
| *** sshnaidm|bbl is now known as sshnaidm|off | 09:02 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: WIP: Add request handler timer to stats https://review.openstack.org/614079 | 09:19 |
| *** hashar has joined #zuul | 09:26 | |
| *** bhavikdbavishi has quit IRC | 10:20 | |
| *** electrofelix has joined #zuul | 11:04 | |
| *** bhavikdbavishi has joined #zuul | 11:14 | |
| *** nilashishc has joined #zuul | 11:22 | |
| *** hashar is now known as hasharLunch | 11:28 | |
| *** bhavikdbavishi has quit IRC | 11:35 | |
| tobiash | Shrews, corvus: the cleanup ports feature in nodepool had severe side effects in our production environment. It cleaned up *all* ports regardless of the state. (maybe the reported state of the cloud was wrong) | 11:38 |
| tobiash | Shrews: corvus: I think we need to make this optional and disabled by default | 11:38 |
| *** bhavikdbavishi has joined #zuul | 11:41 | |
| *** jesusaur has quit IRC | 11:56 | |
| *** jpena is now known as jpena|lunch | 11:57 | |
| *** jesusaur has joined #zuul | 12:01 | |
| *** bhavikdbavishi has quit IRC | 12:07 | |
| tobiash | Shrews: corvus: mordred: when running the port list the filters don't work: https://git.zuul-ci.org/cgit/nodepool/tree/nodepool/driver/openstack/provider.py#n432 | 12:13 |
| *** jesusaur has quit IRC | 12:13 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Revert "Cleanup down ports" https://review.openstack.org/614198 | 12:14 |
| tobiash | Shrews, mordred, corvus: I don't know if you want to fix forward or first revert that patch so here is the revert. Feel free to ignore if you see the problem and want a fix instead ^ | 12:15 |
| tobiash | mordred: how are the filters intended to use in list_ports of openstacksdk? | 12:23 |
| tobiash | local testing shows that this has no effect | 12:24 |
| tobiash | mordred: it seems to be a bug in openstacksdk | 12:27 |
| tobiash | mordred: 'port = cloud._list_ports({'status': 'DOWN'})' works | 12:28 |
| tobiash | mordred: while 'port = cloud.list_ports(filters={'status': 'DOWN'})' ignores the filter | 12:28 |
| tobiash | mordred: I suspect the problem is here: http://git.openstack.org/cgit/openstack/openstacksdk/tree/openstack/cloud/openstackcloud.py#n1706 | 12:29 |
| *** rlandy has joined #zuul | 12:35 | |
| *** bhavikdbavishi has joined #zuul | 12:39 | |
| *** jesusaur has joined #zuul | 12:41 | |
| jhesketh | mordred, clarkb: I've made some progress on the API and tooling required for running local jobs. I pushed this up before I left on vacation for most of October. I'm back as of this week and will be resuming work on it: https://review.openstack.org/#/q/status:open+project:openstack-infra/zuul+branch:master+topic:freeze_job | 12:54 |
| tobiash | infra-root: just to be sure: the port cleanup problem was *not* caused by my cloud but by a bug in openstacksdk. Thus you should not update your nodepool-launchers until this problem is fixed. | 12:58 |
| fungi | thanks for the heads up tobiash! | 12:59 |
| fungi | tobiash: do you have a link to the openstacksdk story with details? | 12:59 |
| tobiash | there is no story yet | 12:59 |
| tobiash | just a code pointer to mordred 20 lines up in the backlog ;) | 12:59 |
| tobiash | fungi: I'll create a story | 13:01 |
| *** hasharLunch is now known as hashar | 13:01 | |
| fungi | that line was last changed 2016-08-30 | 13:03 |
| tobiash | fungi: that might have something to do with caching | 13:05 |
| fungi | but yeah, definitely looks like it clears all filters passed to the method if the first conditional doesn't match | 13:05 |
| tobiash | fungi, mordred: story for openstacksdk: https://storyboard.openstack.org/#!/story/2004207 | 13:06 |
| *** jpena|lunch is now known as jpena | 13:12 | |
| tobiash | fungi: you seem to use the same caching so it's likely that openstack-infra would run into the same issue | 13:12 |
| *** bhavikdbavishi has quit IRC | 13:12 | |
| Shrews | tobiash: eek. i think we should revert, then we can add it back later with updated requirements | 13:27 |
| mordred | tobiash: oh sigh | 13:28 |
| Shrews | tis a silly bug | 13:28 |
| Shrews | tobiash: but kudos for trying it in production! :) | 13:29 |
| Shrews | mordred: fungi: one of you care to +3 that revert (https://review.openstack.org/614198)? | 13:34 |
| mordred | done | 13:35 |
| mordred | Shrews: I'm working on sdk fix | 13:36 |
| Shrews | mordred: cool thx. is it possible to create ports in the DOWN state? would be good to have a func test for that | 13:38 |
| mordred | Shrews: I do not know. we could try being evil and just editing the neutron database behind the scenes... | 13:38 |
| * Shrews checks api guide | 13:38 | |
| mordred | y | 13:40 |
| mordred | gah | 13:40 |
| Shrews | not seeing a way to do it | 13:41 |
| mordred | Shrews: https://review.openstack.org/614213 Filter ports in list_ports when batching is in effect | 13:42 |
| Shrews | mordred: i dont think filters=None is going to work in _filter_list | 13:45 |
| mordred | no? do we need to pass it a {}? | 13:45 |
| Shrews | mordred: oh, nm. it will | 13:45 |
| Shrews | if not filters: return data | 13:45 |
| Shrews | cool | 13:45 |
| mordred | oh cool | 13:46 |
| mordred | yay! it's almost like we thought of something | 13:46 |
| openstackgerrit | Merged openstack-infra/nodepool master: Revert "Cleanup down ports" https://review.openstack.org/614198 | 14:04 |
| *** nilashishc has quit IRC | 14:21 | |
| *** pcaruana|elisa| has quit IRC | 14:33 | |
| mordred | tobiash: https://review.openstack.org/#/c/614213 I believe should fix the error you hit | 14:41 |
| tobiash | mordred: thanks | 14:42 |
| tobiash | mordred: thank god my nodepool was operating in a separate tenant | 14:44 |
| tobiash | mordred: in my dev environment this is not the case. That would have committed suicide... | 14:44 |
| *** pcaruana|elisa| has joined #zuul | 14:45 | |
| mordred | tobiash: yah - it's a really bad bug ... I'm *VERY* glad it didn't cause more damage | 14:45 |
| tobiash | well, it killed 300 jobs... | 14:45 |
| tobiash | but it was just jobs | 14:46 |
| tobiash | so could have been worse | 14:46 |
| mordred | ++ | 14:46 |
| openstackgerrit | Benoît Bayszczak proposed openstack-infra/zuul master: Disable Nodepool nodes lock for SKIPPED jobs https://review.openstack.org/613261 | 14:47 |
| Shrews | tobiash: sorry for the mess | 14:47 |
| tobiash | Shrews: that wasn't foreseeable | 14:48 |
| tobiash | Shrews: and in fact I approved this change ;) | 14:49 |
| *** bbayszczak has joined #zuul | 14:52 | |
| *** ianychoi_ is now known as ianychoi | 15:07 | |
| *** nilashishc has joined #zuul | 15:10 | |
| *** swest has quit IRC | 15:18 | |
| *** pcaruana|elisa| has quit IRC | 15:35 | |
| *** pcaruana|elisa| has joined #zuul | 15:50 | |
| *** pcaruana|elisa| has quit IRC | 16:00 | |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Small script to scrape Zuul job node usage https://review.openstack.org/613674 | 16:10 |
| clarkb | tobiash: ^ I think that is generally useable by anyone now | 16:10 |
| clarkb | tobiash: can you look it over and make sure I didn't miss anything? | 16:10 |
| clarkb | rather than make it an argument to the script I was thinking just do the right thing if the data is there. But I think current ps still prints the header which may be confusing. Lets not print the header if there is no logical grouping data | 16:16 |
| openstackgerrit | Clark Boylan proposed openstack-infra/zuul master: Small script to scrape Zuul job node usage https://review.openstack.org/613674 | 16:18 |
| *** themroc has quit IRC | 16:41 | |
| *** hashar is now known as hasharAway | 16:55 | |
| openstackgerrit | Fabien Boucher proposed openstack-infra/zuul master: WIP - Pagure driver https://review.openstack.org/604404 | 17:05 |
| *** nilashishc has quit IRC | 17:21 | |
| *** dmsimard has quit IRC | 17:24 | |
| *** panda is now known as panda|off | 17:37 | |
| tobiash | clarkb: will look later, out for dinner atm | 17:44 |
| *** electrofelix has quit IRC | 17:50 | |
| *** bbayszczak has quit IRC | 17:53 | |
| ssbarnea | i think found a bug in zuul where "files" (include) does not take precedence over "irrelevant-files" (exclude), see https://review.openstack.org/#/c/614242/ | 17:56 |
| ssbarnea | why is this a problem: because it does not allow me to override default excludes from parent by adding explicit includes at lower level. | 17:57 |
| ssbarnea | for example rsync respects this, by applying includes after excludes, not before. | 17:57 |
| ssbarnea | am I wrong? | 17:57 |
| *** chandankumar is now known as chkumar|off | 18:05 | |
| *** jpena is now known as jpena|off | 18:10 | |
| Shrews | ssbarnea: i don't files is supposed to take precedence over irrelevant-files. You can override 'files' from parent jobs, but I don't think using both is recommended. | 18:19 |
| Shrews | unless we've added a feature i'm not aware of | 18:20 |
| ssbarnea | Shrews: i can explain why. At parent I have a long list of ignores, but for specific job i wan to remove two of these ignores and run it. with current implementation i am forced to copy the entire list of ignores form parent to child job. | 18:21 |
| ssbarnea | this means that any change to parent would count as "a bug" in child job, because nobody would ever bother to copyi them to children. | 18:21 |
| Shrews | yeah, i think you just have to copy the entire list (minus what you don't want) | 18:21 |
| ssbarnea | Shrews: you are right, this is the only way to do it now. But if include / excluded would work as mentioned, it could be possible to do it like i described | 18:22 |
| ssbarnea | this is why most of the tools i know that allow for both includes and excludes are following this logic: apply blacklist filter and whilelist after. | 18:23 |
| Shrews | that's a feature request (not a bug), and i'm not sure how zuul would know how to handle that | 18:23 |
| ssbarnea | otherwise the include does not really work. | 18:23 |
| ssbarnea | yeah it fist the "fug" description, being a bug and a feature, depending how you look at it :D | 18:24 |
| Shrews | ssbarnea: in your example review, how is zuul supposed to know what order to process 'files' and 'irrelevant-files'? Order matters there | 18:24 |
| ssbarnea | am i wrong to assume that ".*" is the implicit behavior? | 18:25 |
| ssbarnea | i can write a bug for it with explanation, maybe even find some examples from other tools. | 18:26 |
| fungi | ssbarnea: you may want to read http://lists.openstack.org/pipermail/openstack-dev/2018-June/131304.html | 18:26 |
| fungi | the present behavior is not the original behavior, and has been discussed at length before settling on it | 18:26 |
| Shrews | fungi: ah, yes. good memory | 18:27 |
| fungi | suffice to say, a lot of thought has gone into the way it's worknig now | 18:27 |
| fungi | the alternatives have down-sides of their own | 18:28 |
| ssbarnea | ... time to block a change before is merged, time to copy/paste config. | 18:28 |
| ssbarnea | in fact my case has nothing to do with attribute inheritance, is about documenting the behaviour if a file matches both patterns: files and irrelevant-files. | 18:30 |
| ssbarnea | i do not question the lack of merge on inheritance, i can fully understand why this was at least controversial if not really bad. | 18:30 |
| fungi | we talked about making files and irrelevant-files mutually exclusive, but now i don't remember if that ever happened (i guess it didn't?) | 18:40 |
| fungi | i have a feeling part of why files is applied before irrelevant-files is because zuul has to decide whether a job (or variant) is going to be run for a given event so needs to check files if specified, and then can optionally decide not to run based on irrelevant-files? | 18:44 |
| fungi | okay, initial implementation of the current behavior was https://review.openstack.org/571745 | 18:54 |
| fungi | ssbarnea: so is your concern purely over order in which files and irrelevant-files are considered when deciding whether a job is run, or over being forced to replace the lists rather than being able to append to them in a child/variant? | 18:58 |
| ssbarnea | fungi: my concern was only about order. | 18:59 |
| fungi | i don't think we discussed the former much (other than to say that combining files and irrelevant-files in the same job was a strange choice with undefined results) | 18:59 |
| ssbarnea | i find the inheritance behavior ok, expected. | 18:59 |
| ssbarnea | in this case i will later file a feature request and document the reasoning. not sure if is easy to implement. i do have a workaround for the moment. | 19:00 |
| ssbarnea | and thanks to everyone for the insights on this. | 19:00 |
| ianw | tobiash: ahh, i think dib functional testing was trying to tell us that the ports thing wasn't working too ... http://logs.openstack.org/47/614047/2/check/nodepool-functional-py35-ubuntu-src/83d0c4e/ara-report/ | 19:21 |
| *** dmsimard has joined #zuul | 19:23 | |
| tobiash | ouch | 19:24 |
| *** pcaruana|elisa| has joined #zuul | 20:12 | |
| *** _ari_ has quit IRC | 20:13 | |
| *** pcaruana|elisa| has quit IRC | 20:32 | |
| *** dmsimard has quit IRC | 21:56 | |
| *** dmsimard has joined #zuul | 22:38 | |
| *** pbrobinson has quit IRC | 22:54 | |
| *** pbrobinson has joined #zuul | 22:59 | |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Logs stats for nodepool automated cleanup https://review.openstack.org/614074 | 23:27 |
| openstackgerrit | Ian Wienand proposed openstack-infra/nodepool master: Revert "Revert "Cleanup down ports"" https://review.openstack.org/614370 | 23:27 |
| *** rlandy has quit IRC | 23:46 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!