| *** robcresswell has quit IRC | 00:12 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Prune stale branches from mergers https://review.openstack.org/536632 | 00:22 |
|---|---|---|
| corvus | mordred: ^ | 00:23 |
| *** jkilpatr has quit IRC | 00:28 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Clarify uniqueness of some config items https://review.openstack.org/536634 | 00:30 |
| corvus | SpamapS: there's your other comment ^ | 00:30 |
| SpamapS | corvus: ty.. thought maybe I was just being dumb | 00:34 |
| corvus | SpamapS: no quite the opposite. i had a myopic worldview :) | 00:35 |
| openstackgerrit | Merged openstack-infra/zuul master: Handle secrets in branches https://review.openstack.org/535501 | 00:50 |
| openstackgerrit | Merged openstack-infra/zuul master: Handle nodesets in branches https://review.openstack.org/535502 | 00:50 |
| openstackgerrit | Merged openstack-infra/zuul master: Normalize semaphore branch handling https://review.openstack.org/535503 | 00:50 |
| openstackgerrit | Merged openstack-infra/zuul master: Fix a copy-pasta in a comment https://review.openstack.org/536620 | 00:54 |
| openstackgerrit | Merged openstack-infra/zuul master: Add change information to Build Completed log message https://review.openstack.org/536629 | 00:54 |
| mordred | corvus: woot! that's exciting | 01:22 |
| mordred | corvus: zuul does not agree with your attempt to alter it | 01:23 |
| mordred | corvus: I should be clear to get rid of the bong python-openstacksdk repos from the mergers and executors now, yeah? | 01:23 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove shade_inner_exceptions https://review.openstack.org/536636 | 01:25 |
| *** elyezer has quit IRC | 02:04 | |
| openstackgerrit | liusheng proposed openstack-infra/zuul master: Fix AttributeError when handle periodic job with github driver https://review.openstack.org/536645 | 02:06 |
| *** yolanda has joined #zuul | 02:09 | |
| *** elyezer has joined #zuul | 02:09 | |
| *** robcresswell has joined #zuul | 02:09 | |
| *** elyezer has quit IRC | 02:25 | |
| *** harlowja has quit IRC | 02:27 | |
| *** elyezer has joined #zuul | 02:38 | |
| *** elyezer has quit IRC | 03:11 | |
| *** elyezer has joined #zuul | 03:14 | |
| *** rlandy has quit IRC | 03:43 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add linters job and role https://review.openstack.org/530682 | 03:53 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add linters job and role https://review.openstack.org/530682 | 04:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-lint job https://review.openstack.org/532083 | 04:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-review job https://review.openstack.org/535223 | 04:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-import-to-galaxy job https://review.openstack.org/532084 | 04:04 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/zuul-jobs master: Add ansible-spec job https://review.openstack.org/532085 | 04:04 |
| *** yolanda has quit IRC | 04:32 | |
| *** elyezer has quit IRC | 04:34 | |
| *** elyezer has joined #zuul | 04:36 | |
| *** bhavik1 has joined #zuul | 04:49 | |
| *** bhavik1 has quit IRC | 04:52 | |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a static driver for Nodepool https://review.openstack.org/535553 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor run_handler to be generic https://review.openstack.org/535554 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Refactor NodeLauncher to be generic https://review.openstack.org/535555 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an OpenContainer driver https://review.openstack.org/535556 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement a Kubernetes driver https://review.openstack.org/535557 | 05:08 |
| openstackgerrit | Tristan Cacqueray proposed openstack-infra/nodepool master: Implement an Amazon EC2 driver https://review.openstack.org/535558 | 05:08 |
| *** harlowja has joined #zuul | 05:12 | |
| *** harlowja has quit IRC | 05:13 | |
| *** dtruong2 has joined #zuul | 05:57 | |
| *** zhuli has quit IRC | 06:28 | |
| *** zhuli has joined #zuul | 06:28 | |
| AJaeger | SpamapS: we can add a redirect in openstack-manuals - that's the global redirect file, folder www | 06:50 |
| *** elyezer has quit IRC | 06:56 | |
| *** elyezer has joined #zuul | 06:59 | |
| *** dtruong2 has quit IRC | 07:09 | |
| *** elyezer has quit IRC | 07:14 | |
| *** elyezer has joined #zuul | 07:18 | |
| *** elyezer has quit IRC | 07:39 | |
| *** elyezer has joined #zuul | 07:51 | |
| *** Cibo is now known as Cibo- | 07:58 | |
| *** dkranz has joined #zuul | 08:21 | |
| *** dkranz has quit IRC | 08:26 | |
| *** jpena|off is now known as jpena | 08:49 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 09:04 |
| *** Cibo has joined #zuul | 09:21 | |
| *** Cibo- has quit IRC | 09:21 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 09:27 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 09:27 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 09:31 |
| *** xinliang has quit IRC | 09:34 | |
| *** xinliang has joined #zuul | 09:46 | |
| *** xinliang has quit IRC | 09:46 | |
| *** xinliang has joined #zuul | 09:46 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 10:05 |
| tobiash | corvus, jlk: this should be functional now and with a test via zuul-web ^^ | 10:05 |
| *** Cibo has quit IRC | 10:15 | |
| *** Cibo has joined #zuul | 10:16 | |
| *** elyezer has quit IRC | 10:55 | |
| *** elyezer has joined #zuul | 10:58 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/nodepool master: Clean held nodes automatically after configurable timeout https://review.openstack.org/536295 | 11:00 |
| *** elyezer has quit IRC | 11:05 | |
| *** elyezer has joined #zuul | 11:08 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 11:19 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 11:34 |
| tobiash | corvus: not sure if we already can remove the webapp after the webhook move but I think this was the last missing piece ^^ | 11:37 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 11:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 11:41 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 11:41 |
| *** jkilpatr has joined #zuul | 12:12 | |
| *** bramwelt has quit IRC | 12:22 | |
| *** bramwelt has joined #zuul | 12:25 | |
| *** jpena is now known as jpena|lunch | 12:33 | |
| *** hashar has joined #zuul | 12:56 | |
| *** yolanda has joined #zuul | 13:05 | |
| *** sshnaidm|off is now known as sshnaidm | 13:09 | |
| *** electrofelix has joined #zuul | 13:25 | |
| *** jpena|lunch is now known as jpena | 13:29 | |
| *** rlandy has joined #zuul | 13:34 | |
| *** dkranz has joined #zuul | 13:47 | |
| *** elyezer has quit IRC | 13:49 | |
| *** elyezer has joined #zuul | 13:51 | |
| *** Wei_Liu has quit IRC | 13:59 | |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 14:22 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move github webhook from webapp to zuul-web https://review.openstack.org/535711 | 14:29 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Move status_url from webapp to web section https://review.openstack.org/536773 | 14:29 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/zuul master: Remove webapp https://review.openstack.org/536780 | 14:29 |
| *** myoung is now known as myoung|taxi | 14:29 | |
| mordred | tobiash: \o/ | 14:31 |
| tobiash | I like removing code :) | 14:32 |
| mordred | tobiash: lgtm - there's a change in the first patch to the default value for listen, is that a thing we want to do? | 14:33 |
| mordred | tobiash: also, I like removing code too :) | 14:33 |
| tobiash | mordred: already removed this | 14:33 |
| tobiash | this was part of the initial change I took over | 14:33 |
| mordred | oh - wow, you're quick | 14:33 |
| tobiash | I think if we want to change that it should be its own change | 14:34 |
| mordred | agree | 14:34 |
| mordred | tobiash: I'm sad about your revert signal handling issue - I'll be interested to learn what the issue is there | 14:35 |
| tobiash | I've given up on this for now | 14:35 |
| tobiash | maybe it has something to do with the abstract base class | 14:35 |
| tobiash | but haven't been able to find the reason for this in two hours of debugging | 14:36 |
| tobiash | so I've given up on this for now | 14:36 |
| openstackgerrit | Merged openstack-infra/zuul master: Revert "Register term_handler for all zuul apps" https://review.openstack.org/536376 | 14:46 |
| tobiash | mordred: +2 with comment on https://review.openstack.org/#/c/534431/10 | 14:47 |
| *** yolanda has quit IRC | 14:56 | |
| *** hashar has quit IRC | 14:59 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove shade_inner_exceptions https://review.openstack.org/536636 | 15:00 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Set app_name and app_version https://review.openstack.org/536854 | 15:00 |
| *** elyezer has quit IRC | 15:08 | |
| *** elyezer has joined #zuul | 15:09 | |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Convert from legacy to native devstack job https://review.openstack.org/535899 | 15:16 |
| *** hashar has joined #zuul | 15:18 | |
| corvus | the power is out here, so i need to shut things down; hopefully will be back soon | 15:30 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add job for publishing docs and tarballs https://review.openstack.org/536860 | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Add reno releasenotes support https://review.openstack.org/536861 | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Switch to stestr https://review.openstack.org/536862 | 15:33 |
| openstackgerrit | Monty Taylor proposed openstack-infra/nodepool master: Remove --no-subunit-trace argument https://review.openstack.org/536863 | 15:33 |
| *** corvus has quit IRC | 15:34 | |
| dmsimard | What would be a good forum for discussing if certain roles are a good idea or not ? Even as core reviewer sometimes I feel my opinion might not be representative of everyone | 15:35 |
| dmsimard | Zuul meetings ? Zuul mailing list ? Something else ? | 15:35 |
| Shrews | roles for which repo(s)? | 15:37 |
| dmsimard | zuul-jobs | 15:39 |
| Shrews | i'd suggest #zuul first, zuul ML for longer debate. just my opinion though | 15:40 |
| AJaeger | dmsimard: I agree, looking at some of the additions I totally miss why those are useful. | 15:40 |
| AJaeger | dmsimard: they might be perfectly fine, I just don't get it yet ;) | 15:41 |
| dmsimard | AJaeger: well, if someone's contributing it it's because there is a use case and it's important to take the time to understand those use cases | 15:41 |
| AJaeger | dmsimard: if the commit message is just "Add X" ;( | 15:41 |
| AJaeger | dmsimard: then I cannot really understand the use case | 15:41 |
| pabelanger | I man, we could first add them to openstack-zuul-jobs, see how they work and promote up into zuul-jobs | 15:42 |
| *** corvus has joined #zuul | 15:42 | |
| dmsimard | https://review.openstack.org/#/c/535803/ is an example of where I don't know where I stand -- but blocking it's inclusion will simply make it proliferate elsewhere outside zuul-jobs anyway because users have that kind of freedom.. so might as well maintain it and make sure it doesn't get out of control ? I dunno | 15:43 |
| dmsimard | Or other times, some roles feels like they should not be roles but something ran by zuul-executor proper | 15:43 |
| dmsimard | i.e, https://review.openstack.org/#/c/514489/ | 15:44 |
| corvus | dmsimard: i think everything should be handled by ansible. that's why really fundamental things, like 'set up git repos' and 'install logging' are done in ansible. basically, we have a configurable remote execution framework, so let's use it. however, we should supply the basics to people who want to run zuul so they don't have to invent everything themselves -- thus the zuul-base-jobs repo. | 15:46 |
| *** myoung|taxi is now known as myoung | 15:47 | |
| dmsimard | corvus: there's a fine line, though, is there not ? Like how we added the setup task before the ansible playbooks start running | 15:48 |
| dmsimard | It's still Ansible but it's within the executor code | 15:48 |
| corvus | dmsimard: that wasn't to perform a task, that was solely so that we could run ansible more reliable later. it doesn't do anything that a deployer would want to change, so it doesn't need any additional configuration. | 15:49 |
| dmsimard | "doesn't need any configuration" is something I can work with to make informed decisions :) | 15:50 |
| corvus | dmsimard: ++ | 15:51 |
| dmsimard | Back to the SSH keys in https://review.openstack.org/#/c/535803/, we've historically discouraged (and turned down) reviews that wanted to add SSH keys to running nodes -- the role is meant to be used with nodepool/zuul hold (in a world where tenants/operators might be able to control zuul/nodepool remotely with an API, for example) | 15:52 |
| dmsimard | For example pabelanger and I blocked https://review.openstack.org/#/c/526187/ in fairly recent history | 15:53 |
| dmsimard | But with Zuul v3, people are going to be able to do everything they want anyway -- it involves trusting the users and zuul v3 was built especially to empower the users so I guess we might as well land it and make it convenient for people to use ? | 15:54 |
| dmsimard | If they foot shoot, it'll be on them I guess | 15:55 |
| corvus | dmsimard: yeah, whether and how we use this in openstack is open to discussion. but i think you're right, someone somewhere in the world is going to write that anyway, so we should land it in zuul-jobs so that the folks who want to use it can have the best version of that. i'm not sure we should add the *playbook* to the default base job though; or if we do, that may be a case where i could be convinced | 15:56 |
| corvus | that to enable the feature at all, an admin has to set an executor variable. | 15:56 |
| corvus | Discussion of the project gating system Zuul | Docs: http://docs.openstack.org/infra/zuul/ | Source: https://git.openstack.org/cgit/openstack-infra/zuul/ | V3 development: http://git.openstack.org/cgit/opensta | 15:57 |
| corvus | 15:46 < corvus> dmsimard: i think everything should be handled by ansible. that's why really fundamental things, like 'set up git repos' and 'install logging' are done in ansible. basically, we have a | 15:57 |
| corvus | configurable remote execution framework, so let's use it. however, we should supply the basics to people who want to run zuul so they don't have to invent everything themselves -- thus the | 15:57 |
| corvus | zuul-base-jobs repo. | 15:57 |
| corvus | 15:47 -!- myoung|taxi is now known as myoung | 15:57 |
| corvus | 15:48 < dmsimard> corvus: there's a fine line, though, is there not ? Like how we added the setup task before the ansible playbooks start running | 15:57 |
| corvus | 15:48 < dmsimard> It's still Ansible but it's within the executor code | 15:57 |
| corvus | 15:49 < corvus> dmsimard: that wasn't to perform a task, that was solely so that we could run ansible more reliable later. it doesn't do anything that a deployer would want to change, so it doesn't need any | 15:57 |
| corvus | additional configuration. | 15:57 |
| corvus | 15:50 < dmsimard> "doesn't need any configuration" is something I can work with to make informed decisions :) | 15:57 |
| corvus | 15:51 < corvus> dmsimard: ++ | 15:57 |
| corvus | 15:52 < dmsimard> Back to the SSH keys in https://review.openstack.org/#/c/535803/, we've historically discouraged (and turned down) reviews that wanted to add SSH keys to running nodes -- the role is meant to | 15:57 |
| corvus | be used with nodepool/zuul hold (in a world where tenants/operators might be able to control zuul/nodepool remotely with an API, for example) | 15:57 |
| corvus | 15:53 < dmsimard> For example pabelanger and I blocked https://review.openstack.org/#/c/526187/ in fairly recent history | 15:57 |
| corvus | i meant to paste: https://docs.openstack.org/infra/zuul/admin/components.html#attr-executor.variables | 15:57 |
| Shrews | O.o | 15:57 |
| dmsimard | oops :( | 15:57 |
| Shrews | /kickban corvus | 15:57 |
| Shrews | :) | 15:58 |
| corvus | #undo | 15:58 |
| dmsimard | With the amount of pings in there, sigyn would have most likely k-lined corvus lol | 15:58 |
| dmsimard | That would've been screenshot worthy | 15:58 |
| corvus | i hope she's not that naive | 15:59 |
| corvus | i only pinged you and me :) | 15:59 |
| dmsimard | In all seriousness, I think the algorithm checks if ops are being pinged mindlessly | 15:59 |
| corvus | dmsimard: left review on 803 | 16:03 |
| *** openstackgerrit has quit IRC | 16:03 | |
| *** dkranz has quit IRC | 16:10 | |
| *** openstackgerrit has joined #zuul | 16:12 | |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Use reno for release notes https://review.openstack.org/262597 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove python-keystoneclient from test-requirements https://review.openstack.org/536879 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove pep8 and pyflakes from test-requirements https://review.openstack.org/536880 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Update docs to use sphinx-build and doc/requirements https://review.openstack.org/536881 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Switch to stestr https://review.openstack.org/536882 | 16:12 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Remove --no-subunit-trace argument https://review.openstack.org/536883 | 16:12 |
| dmsimard | corvus: yeah, we actually don't have any playbooks in zuul-jobs | 16:30 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Convert from legacy to native devstack job https://review.openstack.org/535899 | 16:48 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: WIP: Convert from legacy to native devstack job https://review.openstack.org/535899 | 16:49 |
| *** hashar has quit IRC | 16:53 | |
| *** jkilpatr_ has joined #zuul | 16:58 | |
| *** jkilpatr has quit IRC | 17:00 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul-jobs master: role: Inject public keys in case of failure https://review.openstack.org/535803 | 17:03 |
| *** jkilpatr_ has quit IRC | 17:12 | |
| *** yolanda has joined #zuul | 17:17 | |
| dmsimard | ok I'm going to bikeshed here... plural javascripts? http://git.openstack.org/cgit/openstack-infra/zuul/tree/zuul/web/static/javascripts :D | 17:19 |
| corvus | dmsimard: that may change with mordred's tooling stack | 17:19 |
| dmsimard | ok! I was trying to understand how the zuul-web dashboard was fetching it's data but I know nothing about angular/js. Was mostly trying to understand if we queried an API or if we queried the database directly | 17:20 |
| corvus | dmsimard: javascript uses http api served by zuul-web which contacts database | 17:21 |
| dmsimard | corvus: that's what I managed to understand too, I'll search to see if that API documented somewhere. Is it publicly consumable ? | 17:22 |
| *** dkranz has joined #zuul | 17:22 | |
| corvus | dmsimard: consider it a preview. we should start documenting it, but with big flashy warnings that say it may change. i'd like us to stabilize it and formally support it after the 3.0 release. | 17:23 |
| dmsimard | corvus: okay so it's meant to be public eventually ? | 17:24 |
| corvus | yep | 17:24 |
| corvus | (so probably anything you start doing with the api now, you'll be able to do in the future; we just might need to change 'how'. but probably not 'what'.) | 17:25 |
| dmsimard | My use case was to expose data from sql reporter (i.e, what zuul-web's dashboard is doing) but in a more consumable format fit for things like metrics, graphs, etc. | 17:25 |
| corvus | dmsimard: maybe the dashboard should have graphs? | 17:25 |
| dmsimard | corvus: yeah, that's actually what I discussed with sshnaidm -- we got some interesting data out of review.rdo's zuul but at the same time there's an overlap with the statsd/graphite metrics. | 17:27 |
| pabelanger | dmsimard: I'd love to see how we could move the tripleo dashboards for CI upstream. Either into zuul-dashboard or openstack-health | 17:28 |
| mordred | dmsimard: are there more things we should be emitting into graphite? | 17:30 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Prune stale branches from mergers https://review.openstack.org/536632 | 17:30 |
| corvus | mordred: ^ the test failures were not substantial, that should be gtg now | 17:31 |
| dmsimard | mordred: I don't know. Both graphite/statsd and sql reporter implementations are optional -- they probably serve different purposes but they also hold some of the same metrics users might be interested in (success/failure, time start, time end, duration) | 17:32 |
| corvus | i think that (post 3.0) we may want to consider making sql non-optional. | 17:32 |
| corvus | perhaps in conjunction with supporting sqlite | 17:32 |
| mordred | corvus: +2 with comment typo nit that can be fixed later or just ignored | 17:33 |
| corvus | anyway, put that on the table when thinking about these questions :) | 17:33 |
| mordred | dmsimard, corvus: I actually think we can keep them optional and have the dashboard DTRT - but I need to write down my thoughts on that and send them out | 17:33 |
| corvus | mordred: okay. post 3.0? :) | 17:34 |
| mordred | corvus: but I'd also be fine with sql reporter being non-optional, since AIO installs could always just use sqlite (although we'd probably want to add sqlite testing if we wanted to suggest that for AIO) | 17:34 |
| mordred | corvus: totally | 17:34 |
| corvus | mordred: oh tobiash had a comment too; i'll see if i can address both and revise | 17:34 |
| tobiash | mordred: first we would need sqlite fixing ;) | 17:36 |
| dmsimard | sqlite is a bit tricky, I was bitten in ara and there's literally no way of fixing it | 17:36 |
| dmsimard | Something about unnamed foreign keys/indexes which makes it impossible to act on them in SQL migrations | 17:36 |
| tobiash | Some of the migration scripts are not compatible with sqlite | 17:37 |
| mordred | yah. we'd have to fix the migration scripts ... | 17:37 |
| pabelanger | tobiash: I'm trying to debug a quota issues for CPUs, does the following make sense? http://paste.openstack.org/show/650991/ | 17:37 |
| mordred | dmsimard: my suggestion is to just skip the foreign keys if the driver is sqlite - the data integrity is all coming via the sqlalchemy layer app anyway | 17:38 |
| pabelanger | trying to understand why cores would be inf | 17:38 |
| dmsimard | mordred: you're the pro so I won't argue, I was taught that foreign keys and indexes are usually good things :P | 17:38 |
| tobiash | pabelanger: we have two kinds of quota | 17:39 |
| tobiash | The quota of the cloud and the quota of the pool (set by nb config) | 17:40 |
| mordred | dmsimard: yah. they are 'good things' from a data model perspective. however, my opinion is that if there is an app-level data model that understands the foreign key constraints, it's more scalable to not define them in the database itself and just use the relationships in the model layer - processing and enforcing the keys in the db layer isn't free, and the db is the hardest piece of the puzzle to scale out | 17:40 |
| mordred | dmsimard: **indexes** on the other hand are essential | 17:40 |
| tobiash | The lowest wins per value | 17:40 |
| pabelanger | tobiash: okay, that helps | 17:41 |
| pabelanger | for some reason we are launching more instances that cores allows | 17:41 |
| pabelanger | I would think quota of -8 cores is not good | 17:41 |
| tobiash | pabelanger: the predicted quota should be the quota remaining if the node we want to launch would be launched | 17:42 |
| tobiash | So a predicted negative quota should prevent the node launch and pause the provider | 17:43 |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Prune stale branches from mergers https://review.openstack.org/536632 | 17:43 |
| corvus | tobiash, mordred: ^ should address both comments | 17:43 |
| pabelanger | tobiash: okay, let me see what is happening in quota when we do launch a failure | 17:44 |
| tobiash | pabelanger: what I currently don't know is id nodes in deleting state are counted towards the quota | 17:46 |
| tobiash | If not there could be a race between nodes not deleted yet and new nodes | 17:46 |
| tobiash | That could be worth checking | 17:46 |
| *** yolanda has quit IRC | 17:47 | |
| corvus | mnaser: ^ do you know off hand? | 17:47 |
| pabelanger | tobiash: interesting, let me check for that | 17:48 |
| openstackgerrit | Merged openstack-infra/zuul master: Clarify uniqueness of some config items https://review.openstack.org/536634 | 17:49 |
| mnaser | corvus: im not really sure (assuming you're pinging re quotas?) | 17:49 |
| tobiash | corvus: actually I meant quota accounting in nodepool itself | 17:49 |
| mnaser | i've never seen a negative quota happen but arent those numbers calculated by nodepool? | 17:50 |
| mordred | WELLLLLLL | 17:50 |
| mordred | checking real quick - but I believe we trim nodes in DELETING state from the list at the shade layer | 17:50 |
| mordred | nope. nevermind. I was thinking about images I think | 17:51 |
| mordred | yup. we filter images that are in deleted state | 17:52 |
| * mordred goes back to his hole | 17:52 | |
| pabelanger | tobiash: so, here is what I am seeing on failure: http://paste.openstack.org/show/650995/ | 17:52 |
| corvus | mnaser: yes, i think the question is: does nova count deleting instances toward quota usage | 17:52 |
| tobiash | pabelanger: are these failures because of overquota? | 17:54 |
| pabelanger | tobiash: yes, for cores | 17:56 |
| pabelanger | http://paste.openstack.org/show/650997/ | 17:56 |
| pabelanger | i think that shows the issue | 17:56 |
| pabelanger | from when we get a node request, to quota check, to failure | 17:56 |
| tobiash | ok, so it invalidates the cache, after that it should have current values | 17:57 |
| tobiash | are there some nodes in weird state? | 17:57 |
| * tobiash rereads nodepool source | 17:57 | |
| pabelanger | tobiash: where does it check quota again before launching 2nd attempt? | 17:58 |
| mnaser | corvus: a cloud that might not have this? https://github.com/openstack/nova/blob/stable/pike/releasenotes/notes/recheck-quota-conf-043a5d6057b33282.yaml | 17:58 |
| corvus | fascinating | 17:59 |
| tobiash | pabelanger: it's in the openstack provider in estimateNodepoolQuota | 17:59 |
| Shrews | ah, the consistency of openstack clouds | 17:59 |
| tobiash | pabelanger: this calculates the quota nodepool has available | 17:59 |
| corvus | tobiash, mnaser: so the answer is "maybe"? :) | 18:00 |
| mnaser | i think so | 18:00 |
| mnaser | i think "maybe" is a good answer :P | 18:00 |
| tobiash | pabelanger: and estimatedNodepoolQuotaUsed calculates the quota used by the nodes nodepool manages | 18:02 |
| tobiash | http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/provider.py#n209 | 18:03 |
| tobiash | so I suspect the error here or in a mismatch between getquota and internal quota accounting in nova | 18:03 |
| pabelanger | tobiash: Hmm, is it possible that once we invalidating quota cache, because 2nd launch in _launch() function, we don't get the refreshed quota? | 18:04 |
| tobiash | we had such errors also in our private cloud where getquota tells me there is plenty free, but nova tells me overquota on launch attempts | 18:04 |
| pabelanger | tobiash: yah, I am having troubling seeing how quota is updated once we are in http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/handler.py#n214 | 18:07 |
| tobiash | pabelanger: that's completely handled in http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/provider.py#n166 | 18:08 |
| tobiash | so the quota available to nodepool as a whole is cached | 18:09 |
| tobiash | not the free quota | 18:09 |
| tobiash | and that is calculated by absolute quota of the cloud minus all vms not belonging to nodepool | 18:10 |
| tobiash | pabelanger: can you check what shade's get_compute_limits call returns and then manually launch a node? | 18:11 |
| pabelanger | tobiash: okay, it is possible I'm not understanding something, but once we are in NodeLauncher(), we try 3 times to launch a node. If I understand properly, we check quota before we enter NodeLauncher(), is if first launch attemts fails, I'm not seeing how we call out to quota again to get latest info. We'd just try 2 more times and fail, because we are still over quota. | 18:12 |
| pabelanger | wouldn't an option be, if launch 1 fails due to quota, exit right way and skip other 2 attempts? | 18:12 |
| tobiash | pabelanger: it should check it here: http://git.openstack.org/cgit/openstack-infra/nodepool/tree/nodepool/driver/openstack/handler.py#n487 | 18:14 |
| pabelanger | tobiash: right, line 524 is when we enter NodeLauncher(), and try up to 3 times to bring the node online? | 18:15 |
| tobiash | pabelanger: hrm, maybe it doesn't pause the handler when hitting overquota | 18:15 |
| pabelanger | no, i don't believe so, we try right again to launch 2nd and 3rd attempts before failing | 18:17 |
| *** jkilpatr has joined #zuul | 18:20 | |
| pabelanger | Yah, it doesn't look like we can pause the request once NodeLauncher() is running, once before we call it | 18:21 |
| tobiash | pabelanger: ah now I understand, there we're already one level deeper than the handler | 18:21 |
| pabelanger | tobiash: Yah, sorry. That is what I was trying to explain :) | 18:21 |
| tobiash | hrm, that could get complicated | 18:22 |
| tobiash | we would need to pause the handler and then wait until we have quota again | 18:22 |
| tobiash | I thought the handler will try to launch again | 18:23 |
| tobiash | then it would handle it correctly | 18:23 |
| pabelanger | okay, so that's basically what I was seeing when looking at code | 18:24 |
| tobiash | pabelanger: what about when hitting overquota we abort and just delete the zk node? | 18:24 |
| pabelanger | tobiash: maybe? is there a downside to that? | 18:25 |
| tobiash | then I think the normal handler should take care of relaunching it | 18:25 |
| pabelanger | right | 18:25 |
| pabelanger | I think it would be safe to say, if we get overquota on first launch, it is possible 2nd and 3rd would also fail | 18:25 |
| pabelanger | unless we build in pausing / update quota logic | 18:25 |
| tobiash | it's even likely as we don't wait | 18:26 |
| pabelanger | correct | 18:26 |
| tobiash | so I think it's worth trying to just abort and delete the znode to pretend nothing happened | 18:26 |
| pabelanger | tobiash: did you want to propose patch to do that and see if corvus and Shrews are okay with it? | 18:26 |
| tobiash | k, I'll try | 18:27 |
| *** yolanda has joined #zuul | 18:29 | |
| jlk | hey all, quick question re PTG. I am going to be there for at least Sunday and Monday. On Monday I'd really like to have some time to present the GitHub internal CI/CD use case and spend some time exploring whether or not Zuul could be a useful tool to use in that way. | 18:29 |
| jlk | corvus: mordred ^^ | 18:30 |
| *** dtruong has quit IRC | 18:34 | |
| tobiash | jlk: cool, zuuling github itself would be a major break through | 18:38 |
| dmsimard | jlk: that would be super neat | 18:39 |
| * dmsimard super sad not to be attending ptg | 18:39 | |
| jlk | full disclosure, the CI folks inside of GItHub are interested in EXPERIMENTING with zuul. Not necessarily to make use of it. No promises. That said, the folks I've talked to ARE familiar with Zuul (2.x) and have ran Zuul before at HP. I'm bringing them up to speed on v3 too. | 18:40 |
| dmsimard | so our diabolical plan to infiltrate other organizations to spread zuul is working | 18:41 |
| jlk | lol | 18:41 |
| dmsimard | jlk: spoiler https://twitter.com/dmsimard/status/951982512116551681 :P | 18:42 |
| jlk | wait, what? | 18:42 |
| dmsimard | jlk: it's a joke :( | 18:43 |
| jlk | ah | 18:43 |
| dmsimard | well, not a joke but a coincidence I mean | 18:43 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 18:47 |
| tobiash | pabelanger, corvus, Shrews: I think that should fix pabelanger's quota issues ^^ | 18:48 |
| *** jpena is now known as jpena|off | 18:48 | |
| tobiash | however it's too late for me to think about how to test this | 18:48 |
| tobiash | so I'll add a test tomorrow hopefully | 18:48 |
| openstackgerrit | Tobias Henkel proposed openstack-infra/nodepool master: Fix relaunch attempts when hitting quota errors https://review.openstack.org/536930 | 18:50 |
| pabelanger | tobiash: thanks, will look shortly | 18:50 |
| *** jkilpatr has quit IRC | 18:59 | |
| *** myoung is now known as myoung|biab | 19:00 | |
| corvus | jlk: that's cool! :) i'm happy to help when that would be useful | 19:02 |
| jlk | corvus: I would specifically like to get it on the agenda for Monday, because my wife plans to drag me up to Belfast on Tuesday. | 19:04 |
| corvus | jlk: oh good to know | 19:06 |
| *** JasonCL has joined #zuul | 19:06 | |
| pabelanger | tobiash: we might want to see if we can test 536930, but suspect it will be a little tricky | 19:08 |
| tobiash | pabelanger: I think we maybe could be having a large cache ttl, reducing the quota of the fake cloud and requesting a node | 19:10 |
| corvus | jlk: looks like ptg brainstorming is happening here: https://etherpad.openstack.org/p/infra-rocky-ptg | 19:10 |
| tobiash | my theory is that without the patch the request fails and with the patch it pauses | 19:11 |
| corvus | jlk: we may have to set aside an informal time monday for that | 19:11 |
| jlk | okay | 19:11 |
| tobiash | jlk: I'd be happy to join this discussion | 19:12 |
| mordred | corvus, jlk: I have openstack board meeting on monday (yay for overlap) but I can get a braindump from corvus (and possibly you too) | 19:16 |
| openstackgerrit | David Shrewsbury proposed openstack-infra/nodepool master: Convert from legacy to native devstack job https://review.openstack.org/535899 | 19:16 |
| jlk | mordred: sure. I suspect it'll be an ongoing conversation if/when GitHub starts to experiment. | 19:17 |
| mordred | jlk: also, I seem to have a few different bay-area thigns on my plate in the not-too-distant future ... if it would be helpful for me to meet with wenlock whilst I'm in the general area I'd be happy to | 19:17 |
| corvus | mordred: wow, wasn't there like a bunch of pushback on that for that reason last time? | 19:17 |
| pabelanger | jlk: that is exciting | 19:17 |
| jlk | I believe a desired outcome is agreement that the workflow is something that Zuul could support, and a willingness to have a follow up meeting with our folks when the time gets closer. | 19:17 |
| jlk | mordred: that is indeed one of the desired outcomes, meetings. Because everybody loves meetings. | 19:18 |
| mordred | jlk: \o/ | 19:19 |
| openstackgerrit | Honza Pokorny proposed openstack-infra/zuul-jobs master: Add always_use_sha option to version-from-git role https://review.openstack.org/536938 | 19:21 |
| openstackgerrit | Honza Pokorny proposed openstack-infra/zuul-jobs master: Add always_use_sha option to version-from-git role https://review.openstack.org/536938 | 19:26 |
| *** elyezer has quit IRC | 19:28 | |
| *** elyezer has joined #zuul | 19:30 | |
| *** harlowja has joined #zuul | 19:34 | |
| *** sshnaidm has quit IRC | 19:39 | |
| openstackgerrit | Merged openstack-infra/zuul master: Prune stale branches from mergers https://review.openstack.org/536632 | 19:59 |
| *** electrofelix has quit IRC | 20:01 | |
| *** sshnaidm has joined #zuul | 20:11 | |
| *** myoung|biab is now known as myoung | 20:23 | |
| *** jkilpatr has joined #zuul | 20:39 | |
| pabelanger | Shrews: do you mind adding https://review.openstack.org/536930/ to your reviews, wanted to see if you think that approach will work | 20:41 |
| Shrews | pabelanger: certainly | 20:42 |
| Shrews | pabelanger: commented | 20:55 |
| pabelanger | Shrews: thanks, I'm working on seeing if we can actually test that via a unit test | 20:58 |
| *** elyezer has quit IRC | 21:37 | |
| *** elyezer has joined #zuul | 21:41 | |
| *** rlandy is now known as rlandy|bbl | 22:13 | |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 22:22 |
| *** elyezer has quit IRC | 22:41 | |
| openstackgerrit | James E. Blair proposed openstack-infra/zuul master: Fall back on parent jobs from default branch https://review.openstack.org/536998 | 22:42 |
| *** elyezer has joined #zuul | 22:43 | |
| *** elyezer has quit IRC | 22:52 | |
| *** elyezer has joined #zuul | 22:53 | |
| corvus | mordred: ^ i've implemented suggestion #1 from my email, but in doing so, i realized that the problem doesn't only appear in inheritance. in digging back into irc history, i belatedly noticed that, indeed, you ran into the problem without inheritance first, then fixed it at that layer, before running into the problem with inheritance. | 23:09 |
| corvus | mordred: i sent a followup email to zuul-discuss about it, because i think we may want to consider that case as well. | 23:09 |
| *** dkranz has quit IRC | 23:21 | |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Remove fedora-25 from nodepool functional testing https://review.openstack.org/537004 | 23:28 |
| openstackgerrit | Paul Belanger proposed openstack-infra/nodepool master: Switch to fedora-27 for nodepool functional testing https://review.openstack.org/537005 | 23:28 |
| mordred | corvus: awesome. I'm good at finding problems!!! | 23:35 |
| openstackgerrit | Matthieu Huin proposed openstack-infra/zuul master: [WIP] zuul autohold: allow filtering per commit https://review.openstack.org/536993 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Register .json routes with and without suffix https://review.openstack.org/537010 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add /info and /{tenant}/info route to zuul-web https://review.openstack.org/537011 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add support for configuring graphite_url https://review.openstack.org/537012 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Move WebInfo config processing into zuul.model https://review.openstack.org/537013 | 23:46 |
| openstackgerrit | Monty Taylor proposed openstack-infra/zuul master: Add facility for plugins to register web routes https://review.openstack.org/537014 | 23:47 |
| mordred | corvus: wow. yup. my brain cannot fully process the ramifications of that email at this point in the day | 23:50 |
| corvus | mordred: that's okay, i probably won't write #2 today anyway; i need to digest it as well. | 23:50 |
| mordred | corvus: in other news, I made you a nice pile of changes related to zuul-web that I know you're going to fully enjoy reviewing | 23:51 |
| tristanC | dmsimard: zuul-web rest api is documented in https://review.openstack.org/#/c/535541/ | 23:52 |
| mordred | tristanC: I'd love feedback from you on the stack I just pushed up | 23:53 |
| *** rlandy|bbl is now known as rlandy | 23:54 | |
| mordred | tristanC: oh. hrm. your patch is showing a hole in build-sphinx-docs | 23:57 |
| mordred | tristanC: I think I have a todo comment for it somewhere ... but the tl;dr is that we need to implement the tox-install-siblings logic for build-openstack-sphinx too | 23:58 |
| mordred | tristanC: I'll try to remember to get that done first thing in the morning, because I love the arbitrary depends-on you've got there :) | 23:59 |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!