| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 909269: Add some github configuration deprecations https://review.opendev.org/c/zuul/zuul/+/909269 | 00:37 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 909269: Add some github configuration deprecations https://review.opendev.org/c/zuul/zuul/+/909269 | 00:37 | |
| -@gerrit:opendev.org- Samuel Jan Surovka proposed: [zuul/nodepool] 908579: Add a new metric, for handleable requests per provider https://review.opendev.org/c/zuul/nodepool/+/908579 | 12:09 | |
| @sjal:matrix.org | how do you guys debug nodes built by nodepool that are stuck in 'building' after being launched by the launcher | 12:59 |
|---|---|---|
| @sjal:matrix.org | I know that you can test the images beforehand and that you can run it and check but I'm just wondering | 13:05 |
| -@gerrit:opendev.org- Samuel Jan Surovka proposed: [zuul/nodepool] 908579: Add a new metric, for handleable requests per provider https://review.opendev.org/c/zuul/nodepool/+/908579 | 13:14 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 910739: Only use latest proposed config for project-branch https://review.opendev.org/c/zuul/zuul/+/910739 | 14:02 | |
| -@gerrit:opendev.org- Simon Westphahl proposed: [zuul/zuul] 910746: Only return the latest config for project-branch https://review.opendev.org/c/zuul/zuul/+/910746 | 14:11 | |
| @clarkb:matrix.org | > <@sjal:matrix.org> I know that you can test the images beforehand and that you can run it and check but I'm just wondering | 14:53 |
| For openstack at least nodepool should try to capture a console log for failed boots. I'll usually start there to see if there was anything obviously wrong with the boot process (no networking etc). | ||
| The next debug step is to manually boot the image and see if I can reproduce the problem and then debug from there. | ||
| @sjal:matrix.org | > <@clarkb:matrix.org> For openstack at least nodepool should try to capture a console log for failed boots. I'll usually start there to see if there was anything obviously wrong with the boot process (no networking etc). | 15:06 |
| > | ||
| > The next debug step is to manually boot the image and see if I can reproduce the problem and then debug from there. | ||
| yeah I'm referring to problems with public cloud, I don't really know how to debug those problems better than just booting the image, thanks anyway | ||
| @sjal:matrix.org | maybe I'll come up with something now that I'm having a bigger impact on what's going on with our Zuul | 15:07 |
| @clarkb:matrix.org | Whether or not the cloud is public doesn't really affect this much | 15:07 |
| @sjal:matrix.org | I mean I don't really have anything regarding the failed boots, they are just building forever | 15:07 |
| @sjal:matrix.org | I'm not really sure how it works so it switches from building to ready | 15:08 |
| @clarkb:matrix.org | Nodepool waits for the cloud API to report a ready state then it begins to poll for ssh connectivity (for ssh based nodes anyway) and collects ssh host key info. Once that is done the node should be marked ready within nodepool. | 15:09 |
| @clarkb:matrix.org | If things are stuck for a long time in this state you may be able to inspect the stuck nodes directly | 15:10 |
| @clarkb:matrix.org | Collect console logs or look for cloud reported errors | 15:10 |
| @sjal:matrix.org | damn, now that I think about it probably the user changed or something | 15:10 |
| @clarkb:matrix.org | I don't think it actually tries to make a full ssh connection with auth. Just checks ssh is listening and reports host keys back. | 15:12 |
| @clarkb:matrix.org | But there are ready scripts you can specify which may do extra checking like that | 15:12 |
| -@gerrit:opendev.org- Zuul merged on behalf of James E. Blair https://matrix.to/#/@jim:acmegating.com: [zuul/zuul] 909269: Add some github configuration deprecations https://review.opendev.org/c/zuul/zuul/+/909269 | 18:54 | |
| -@gerrit:opendev.org- James E. Blair https://matrix.to/#/@jim:acmegating.com proposed: [zuul/zuul] 910856: Fix stack_dump_handler test https://review.opendev.org/c/zuul/zuul/+/910856 | 22:55 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!