| SpamapS | corvus:neat! thanks | 00:11 |
|---|---|---|
| fungi | corvus: i agree that sounds like it could save a lot of idle time for both the builder and devstack | 00:16 |
| *** mattw4 has quit IRC | 00:19 | |
| *** mattw4 has joined #zuul | 00:22 | |
| *** mattw4 has quit IRC | 00:47 | |
| *** michael-beaver has quit IRC | 01:39 | |
| *** mattw4 has joined #zuul | 01:57 | |
| openstackgerrit | Ian Wienand proposed zuul/zuul-jobs master: Add reboot-host role https://review.opendev.org/666748 | 01:58 |
| *** zer0c00l_ has joined #zuul | 02:28 | |
| zer0c00l_ | I have a cluster with nova-network, keystoneauth/nodepool seems to be having trouble talking to it | 02:29 |
| zer0c00l_ | http://paste.openstack.org/show/753241/ | 02:29 |
| *** kevinluuuuu has quit IRC | 02:32 | |
| zer0c00l_ | Seems like cleanupLeakedPorts eventually leads to keystoneauth's service_discovery network endpoint and fails | 02:32 |
| zer0c00l_ | This is one of those old clusters still running nova-net :( | 02:33 |
| *** bhavikdbavishi has joined #zuul | 03:24 | |
| *** bhavikdbavishi1 has joined #zuul | 03:34 | |
| *** bhavikdbavishi has quit IRC | 03:36 | |
| *** bhavikdbavishi1 is now known as bhavikdbavishi | 03:36 | |
| *** mattw4 has quit IRC | 04:39 | |
| *** migi has quit IRC | 04:44 | |
| *** mhu has quit IRC | 04:44 | |
| *** mhu has joined #zuul | 04:44 | |
| *** ianw is now known as ianw_pto | 04:44 | |
| *** pcaruana has joined #zuul | 04:45 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Filter out unprotected branches from builds if excluded https://review.opendev.org/666664 | 05:48 |
| *** gtema has joined #zuul | 06:15 | |
| flaper87 | tobiash: I'm trying to install a python package in zuul's python env. I can't find what's the right python env I should install this package on T_T | 06:25 |
| flaper87 | Should I just use the pip ansible module and delegate to localhost? | 06:25 |
| tobiash | flaper87: you need one in the ansible context? | 06:28 |
| flaper87 | tobiash: yes | 06:31 |
| tobiash | flaper87: there are some variables you can set during installation time to add additional packages to the ansible venvs: https://zuul-ci.org/docs/zuul/admin/installation.html?highlight=ansible_extra_packages#ansible | 06:33 |
| flaper87 | tobiash: a-ha, interesting. I assume this can be set in the pod | 06:34 |
| flaper87 | and that zuul-ansible-manage will run at startup | 06:34 |
| flaper87 | thanks | 06:34 |
| tobiash | if you use the official images zuul-manage-ansible has been run during image creation | 06:35 |
| tobiash | so you probably need to either add another layer that calls it again in the image or add your own startup script that runs it during startup | 06:36 |
| flaper87 | perfect, thanks | 06:37 |
| *** gtema has quit IRC | 06:44 | |
| flaper87 | tobiash: that worked, thanks! | 06:46 |
| *** saneax has joined #zuul | 06:46 | |
| flaper87 | do you know what user is used to run ansible? The same as the zuul process? | 06:46 |
| *** saneax has quit IRC | 07:08 | |
| tobiash | yes, should be | 07:09 |
| tobiash | mhu: I've posted a question on https://review.opendev.org/576907 | 07:09 |
| *** gtema has joined #zuul | 07:11 | |
| *** mhu has quit IRC | 07:27 | |
| *** mhu has joined #zuul | 07:27 | |
| openstackgerrit | Mark Meyer proposed zuul/zuul master: Extend event reporting https://review.opendev.org/662134 | 07:36 |
| *** jpena|off is now known as jpena | 07:37 | |
| *** saneax has joined #zuul | 07:51 | |
| *** hashar has joined #zuul | 08:09 | |
| *** jangutter has joined #zuul | 08:24 | |
| *** hashar has quit IRC | 08:53 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 08:56 |
| *** hashar has joined #zuul | 08:57 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 09:03 |
| *** spsurya has joined #zuul | 10:07 | |
| *** gtema has quit IRC | 10:11 | |
| *** gtema_ has joined #zuul | 10:11 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 10:38 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 10:39 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 10:40 |
| *** jpena is now known as jpena|lunch | 11:26 | |
| *** pwhalen has quit IRC | 11:41 | |
| *** gtema_ has quit IRC | 11:54 | |
| *** gtema_ has joined #zuul | 11:55 | |
| *** EmilienM is now known as EvilienM | 11:59 | |
| *** gtema_ has quit IRC | 12:13 | |
| *** gtema_ has joined #zuul | 12:26 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 12:35 |
| *** rlandy has joined #zuul | 12:36 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 12:36 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 12:36 |
| *** jamielennox has quit IRC | 12:41 | |
| *** michael-beaver has joined #zuul | 12:55 | |
| *** jpena|lunch is now known as jpena | 12:59 | |
| *** rfolco has joined #zuul | 13:02 | |
| fungi | so... we've (opendev) got a report from one of our openstack providers that nodepool seems to be "cleaning up" newly allocated ports while waiting for the associated instances to be scheduled... the theory is that the request causes neutron to allocate a port in a "down" state, but if it takes >3 minutes for the corresponding instance to be scheduled to a host and created then nodepool assumes the port is | 13:10 |
| fungi | leaked and deletes it. does this seem likely to anyone more versed in https://review.opendev.org/609829 and the related changes which followed it? | 13:10 |
| *** gtema_ has quit IRC | 13:20 | |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: web: add tenant and project scoped, JWT-protected actions https://review.opendev.org/576907 | 13:21 |
| *** jeliu_ has joined #zuul | 13:29 | |
| *** gtema_ has joined #zuul | 13:32 | |
| *** bhavikdbavishi has quit IRC | 13:43 | |
| tobiash | fungi: afaik it cleans them after 7 minutes | 13:44 |
| tobiash | Which might not be enough in some cases | 13:44 |
| tobiash | Oh it is 3 minutes | 13:46 |
| tobiash | we should probably increase that interval | 13:47 |
| *** jamesmcarthur has joined #zuul | 13:58 | |
| fungi | i guess we don't have any convenient way to be able to tell if a port is preallocated and still waiting on an instance request which is in progress | 13:58 |
| fungi | mordred probably can explain all the terribleness happening with that | 13:59 |
| flaper87 | Is there a built-in auth management in zuul-web? | 14:03 |
| flaper87 | if not I'd prolly configure basic auth on nginx and just skip auth for the webhook endpoint (not sure if that's even possible) | 14:03 |
| corvus | flaper87: no built-in auth; using a web server auth module is the way to go | 14:11 |
| *** jamesmcarthur has quit IRC | 14:18 | |
| flaper87 | corvus: roger that | 14:25 |
| tobiash | fungi: that's correct, thats why we sweep every three minutes over all down ports and clean up the ones that we already know | 14:26 |
| clarkb | tobiash: fungi: does that rely on a 3 minute timer or an age field on the port | 14:27 |
| clarkb | I wonder if we can sweep often but delay deletion until port age is much older | 14:28 |
| tobiash | it's a 3 minute periodic task in nodepool itself | 14:28 |
| clarkb | (30 minutes maybe?) | 14:28 |
| flaper87 | corvus: what would be the endpoint to leave out of auth so that github can send webhooks? | 14:29 |
| flaper87 | I guess the one I put in the app | 14:29 |
| flaper87 | dumb question | 14:29 |
| tobiash | yepp ;) | 14:30 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Allow operator to generate auth tokens through the CLI https://review.opendev.org/636197 | 14:31 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Zuul CLI: allow access via REST https://review.opendev.org/636315 | 14:33 |
| corvus | clarkb, tobiash, fungi: yeah, it's a sort of mark/sweep -- it checks every 3m and the port has to be down the current check and prev check. | 14:36 |
| tobiash | corvus: I think port leaks are typically not that dramatically so we could increase that to let's say 10 minutes? | 14:37 |
| corvus | k, i'll make a patch | 14:38 |
| fungi | thanks corvus! | 14:39 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: Increase port cleanup interval https://review.opendev.org/666852 | 14:40 |
| openstackgerrit | Matthieu Huin proposed zuul/zuul master: Add Authorization Rules configuration https://review.opendev.org/639855 | 14:42 |
| *** jangutter has quit IRC | 15:09 | |
| *** hashar has quit IRC | 15:14 | |
| clarkb | catching up, it appears we made a zuul release. Did we end up reverting the parallel github api requests or fixing that properly? | 15:15 |
| tobiash | Parallel is still reverted, reimplementing is on my todo list | 15:17 |
| clarkb | ok no rush. I'm just catching up on the status of a few of the things I was following before taking most of the week off :) | 15:19 |
| *** hashar has joined #zuul | 15:26 | |
| tobiash | But we still landed a bunch of improvements to the github driver | 15:28 |
| *** saneax has quit IRC | 15:33 | |
| *** saneax has joined #zuul | 15:33 | |
| *** saneax has quit IRC | 15:45 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: DNM: fail devstack jobs https://review.opendev.org/666880 | 15:46 |
| *** clarkb has quit IRC | 15:53 | |
| *** jamesmcarthur has joined #zuul | 15:56 | |
| *** jpena is now known as jpena|off | 16:01 | |
| openstackgerrit | Merged zuul/nodepool master: Increase port cleanup interval https://review.opendev.org/666852 | 16:01 |
| *** clarkb has joined #zuul | 16:05 | |
| *** panda is now known as panda-pto | 16:08 | |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add command processor to zuul-web https://review.opendev.org/666307 | 16:11 |
| openstackgerrit | Tobias Henkel proposed zuul/zuul master: Add repl server for debug purposes https://review.opendev.org/579962 | 16:12 |
| *** mattw4 has joined #zuul | 16:21 | |
| *** mgoddard has quit IRC | 16:24 | |
| *** mgoddard has joined #zuul | 16:24 | |
| *** spsurya has quit IRC | 16:24 | |
| *** mattw4 has quit IRC | 16:27 | |
| *** mattw4 has joined #zuul | 16:28 | |
| *** pwhalen has joined #zuul | 16:31 | |
| SpamapS | has anyone ever asked for or worked on a way to change the hard coded "roles/" path to something one can set on the job configuration? | 16:42 |
| SpamapS | I have a giant pile of roles in a sub-directory because monorepo... but I want to use them from another repo. | 16:42 |
| SpamapS | was thinking roles: - {zuul: {name: Project/Name path: auto/ansible/roles}} | 16:43 |
| corvus | SpamapS: we're trying to be compatible with ansible here | 16:43 |
| *** jamesmcarthur has quit IRC | 16:44 | |
| SpamapS | It's a bit inflexible though. The default is fine, but for me.. I'm having to extract all the roles from the monorepo into a 3rd repo... | 16:44 |
| SpamapS | anyway, have to run afk.. but something to ponder. | 16:44 |
| fungi | maybe if this corresponded to a new feature of upstream ansible where it could be configured to look for additional role paths? | 16:49 |
| corvus | SpamapS: the end result is that those roles will be easier to share if they aren't buried deep in a repo, and sharing is the ultimate goal. basically, you can put roles anywhere you want in a repo as long as they're used for playbooks in that repo, but if you want to share those roles with other repos, we take cues from ansible about how to do that, so we don't invent zuul-specific ways of sharing roles. | 16:49 |
| corvus | so the current system mimics what you see on galaxy. mazer has more advanced ideas about collections of roles, perhaps there is opportunity there. | 16:49 |
| corvus | fungi: you can tell ansible to look anywhere for roles, so this is more about following the lead of galaxy and mazer regarding sharing roles | 16:50 |
| fungi | ahh | 16:51 |
| *** jeliu_ has quit IRC | 16:55 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 17:07 |
| *** jamesmcarthur has joined #zuul | 17:07 | |
| dmsimard | SpamapS: haven't tested this with Zuul but Ansible roles can be called by their path relative to the root of ANSIBLE_ROLES_PATH | 17:09 |
| dmsimard | so, for example, you might have roles nested like "roles/infra/install-docker", "roles/dev/bootstrap" | 17:10 |
| dmsimard | but then, the roles as written in your playbooks must be called "infra/install-docker", "dev/bootstrap" | 17:10 |
| dmsimard | whereas you might otherwise have had "roles/install-docker", "roles/bootstrap" and called them as "install-docker" and "bootstrap" | 17:11 |
| *** jamesmcarthur has quit IRC | 17:23 | |
| *** igordc has joined #zuul | 17:25 | |
| *** hashar has quit IRC | 17:25 | |
| mattw4 | Does anyone know why the executor container would have an old version of my (untrusted) jobs repo? I thought it refreshed its copy in /var/lib/zuul/executor-git on every job run. | 17:51 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 17:53 |
| *** jeliu_ has joined #zuul | 18:15 | |
| SpamapS | dmsimard: that's interesting, but the path I have is {repo}/auto/ansible/roles | 18:16 |
| SpamapS | And I couldn't care less about Ansible's norms for my purposes, as these roles will never be shared with anybody but us (the ones we share with the world are at https://github.com/GoodMoney/goodmoney-zuul-roles), but I appreciate the desire to align. I | 18:17 |
| SpamapS | The other option is a symlink from roles -> auto/ansible/roles in the repo where I want them to live. | 18:18 |
| SpamapS | but being monorepo-ish... paths matter a lot.. so I don't do it lightly. | 18:18 |
| SpamapS | Would much rather that I can just specify the sub-path. | 18:19 |
| *** jeliu_ has quit IRC | 18:20 | |
| *** jeliu_ has joined #zuul | 18:21 | |
| pabelanger | collections is the new hotness, I believe there is more flexability there | 18:26 |
| SpamapS | Why isn't there flexibility in roles path though? You're already building a roles path with x/y/z/roles:a/b/c/roles | 18:26 |
| SpamapS | Seems a bit rigid to enforce that it has to be {repo}/roles | 18:27 |
| SpamapS | Also, just from my perspective.. I don't find roles all that sharable outside limited contexts like zuul jobs. :-P | 18:27 |
| *** gtema_ has quit IRC | 18:29 | |
| SpamapS | I'll propose a patch, and we can discuss from there. :) | 18:38 |
| corvus | mattw4: the content in /var/lib/zuul/executor-git isn't really meant to be directly used; it may have the latest content without actually checking it out. what ultimately matters is what it puts into the build directory. you can see the refs and shas that it checks out for the build in the executor debug log if there's a question. | 18:58 |
| fungi | yeah, the date of the checkout in the /var/lib/zuul/executor-git tree is likely an artifact of when it was cloned and nothing more | 19:00 |
| fungi | i wonder if that could just be replaced by a bare git repo with no worktree? | 19:00 |
| corvus | fungi: we need to perform merge ops | 19:04 |
| corvus | that requires a working tree | 19:05 |
| corvus | but we don't make any attempt to leave it in a sensible state. if you cd into it and do a 'git log', you might see the current tip of master, or you might see the result of a speculative merge commit of a 3 month old change on a stable branch. | 19:06 |
| fungi | okay, so it'll be left in whatever state was required by the last merge (which could involve relatively ancient commits) | 19:14 |
| fungi | i suppose we *could* divorce the worktree to some other temporary path separate from the gitdir and clean it up with something along the lines of the `git worktree remove` command | 19:16 |
| fungi | once it's state is not in use | 19:17 |
| fungi | er, its | 19:17 |
| openstackgerrit | Arun S A G proposed zuul/nodepool master: Clouds running nova-network won't have public network endpoints https://review.opendev.org/666905 | 19:18 |
| *** EvilienM is now known as EmilienM | 19:39 | |
| *** gtema_ has joined #zuul | 19:45 | |
| mattw4 | corvus: Thanks for the response. I figured the executor-git dir was some kind of staging dir, but at least it had my latest commit. What I don't understand is why my new job, defined in that latest commit, is not available in the Zuul jobs listing page at <zuul_ip>:9000/t/example_tenant/jobs. Any idea why that listing would be stale? | 20:17 |
| corvus | mattw4: the executor should notice changes to zuul jobs in repos it knows about. it should receive the event from the merge, decide that it might contain a config update, and reload the config for the tenant. you can check the scheduler logs to see if that happened or if something went wrong. | 20:20 |
| mattw4 | will do corvus, thanks again! | 20:20 |
| corvus | mattw4: if there was an error in the config, it may be staying with the old one. if it just missed the event, you can run "zuul-scheduler full-reconfigure" to force it to reload. | 20:21 |
| *** rfolco has quit IRC | 20:25 | |
| fungi | also if it missed that event due to a fluke, then the next event for that tenant should catch it up | 20:26 |
| mattw4 | corvus, fungi: I'm seeing my "recheck" trigger fly by in the main log so I think it's catching the trigger, but I'm stumped as to why my jobs listing doesn't have my newly defined jobs | 20:28 |
| fungi | mattw4: when you visit the status page in a web browser, do you see a small "bell" icon in the top-right corner? | 20:30 |
| fungi | (that only appears if there are configuration errors detected by the scheduler) | 20:32 |
| mattw4 | fungi: yeah, there are a lot of them :/ Where is the correct place to fix the "Unknown project opendev.org/openstack/devstack" error? I have it listed in my tenant config as an untrusted project | 20:32 |
| corvus | mattw4: that's the right way to fix that, did you do a full reconfiguration or restart after adding it? | 20:33 |
| mattw4 | corvus: I have tried both | 20:34 |
| corvus | mattw4: it may be worth reading the scheduler log when it starts to see if it mentions anything about opendev.org/openstack/devstack | 20:35 |
| mattw4 | corvus: is it correct to list projects in tenant config as you have above^? i.e. opendev.org/openstack/devstack instead of openstack/devstack ? | 20:36 |
| fungi | is opendev.org the name of your opendev connection? | 20:36 |
| corvus | mattw4: in the tenant config, it should just be 'openstack/devstack', but underneath a connection to opendev | 20:36 |
| mattw4 | fungi: yep | 20:36 |
| corvus | mattw4: just like https://opendev.org/openstack/project-config/src/branch/master/zuul/main.yaml#L79 | 20:37 |
| mattw4 | corvus: yeah, that's ithe pattern I'm using. Checking reference you linked... | 20:37 |
| *** gtema_ has quit IRC | 20:44 | |
| mattw4 | in my job definitions, should I preface my required-projects with their connection name? e.g. is "required-projects: - opendev.org/openstack/devstack" correct or should I drop the leading opendev.org? | 20:45 |
| fungi | i think you need to specify the connection name unless they're listed in the same connection | 20:52 |
| mattw4 | thanks fungi | 20:56 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:02 |
| *** jeliu_ has quit IRC | 21:12 | |
| *** pcaruana has quit IRC | 21:16 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 21:16 |
| *** openstackgerrit has quit IRC | 21:33 | |
| *** persia_ is now known as persia | 21:45 | |
| *** openstackgerrit has joined #zuul | 22:06 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:06 |
| mattw4 | Does anyone know why, if I have projects included in my tenant config and they are showing up in the Zuul web page "projects" list, why am I getting config errors like "Unknown project opendev.org/openstack/devstack"?? I'm really stumped on this one | 22:16 |
| mattw4 | It seems like my issue above^ should be solved with the correct "include:" in tenant config. | 22:16 |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 22:41 |
| corvus | mattw4: if you click on the devstack project in the projects list, what is the name you see on the resulting page? | 22:42 |
| corvus | mattw4: eg where it says "opendev.org/airship/airship-in-a-bottle" on this page: http://zuul.openstack.org/project/airship/airship-in-a-bottle | 22:43 |
| mattw4 | corvus: when I click on openstack/devstack in the projects list, it opens a page with the header 'review.opendev.org/openstack/devstack', but the page is empty | 22:45 |
| corvus | mattw4: i think you're missing the "canonical_hostname" setting in your zuul.conf file: https://zuul-ci.org/docs/zuul/admin/drivers/gerrit.html#attr-%3Cgerrit%20connection%3E.canonical_hostname | 22:45 |
| mattw4 | corvus: I sure am! Is that a new setting? I don't remember seeing it a few weeks ago when I set up my Zuul | 22:46 |
| corvus | mattw4: set canonical_hostname=opendev.org for that connection... | 22:46 |
| mattw4 | corvus: I should stop and start the containers after that, right? I never remember when I need to restart... | 22:47 |
| corvus | mattw4: it's not new, but it's not usually needed. it is needed for opendev. | 22:47 |
| corvus | mattw4: yes, that requires a restart of the scheduler | 22:47 |
| corvus | mattw4: zuul currently thinks the name of that project is "review.opendev.org/openstack/devstack" after the change and restart, it will know it as "opendev.org/openstack/devstack" so it should match | 22:48 |
| mattw4 | corvus: gotcha...I think I'm still repairing configuration from the domain switchover. | 22:48 |
| mattw4 | that makes a lot of sense with the issues I've been seeing today | 22:49 |
| corvus | well, the same problem would have happened before, just would have been "review.openstack.org" vs "git.openstack.org" | 22:49 |
| mattw4 | Thanks corvus! The project listings seem "complete" like the airship example you sent me! | 22:52 |
| *** rlandy has quit IRC | 23:00 | |
| *** igordc has quit IRC | 23:02 | |
| openstackgerrit | James E. Blair proposed zuul/nodepool master: WIP: new devstack-based functional job https://review.opendev.org/665023 | 23:29 |
| *** jamesmcarthur has joined #zuul | 23:59 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!