| rpittau | good morning ironic! o/ | 07:08 |
|---|---|---|
| arne_wiebalck | Good morning rpittau o/ | 07:16 |
| rpittau | hey arne_wiebalck :) | 07:19 |
| queensly[m] | Good morning | 07:30 |
| abongale | Good morning Ironic! | 08:19 |
| AmarachiOrdor[m] | Good Morning Ironic! | 08:22 |
| dtantsur | cid: amazing job with cherrypy. Looks like a viable path forward? | 10:19 |
| dtantsur | not to distract you from that, but we can consider https://docs.pylonsproject.org/projects/waitress/en/stable/ as a fallback option probably | 10:22 |
| cid | dtantsur: Yeah, cherrypy seems to work with the minimalist amount of changes yet. | 10:32 |
| cid | A fallback could make things a lot better, in a setup where either of cherrypy or waitress is the primary server. | 10:33 |
| dtantsur | True. And on a brief glance, it seems to support the things we need, even unix sockets | 10:33 |
| dtantsur | if cherrypy works and can be added to requirements, we don't need a fallback option really | 10:34 |
| dtantsur | cid: note that cherrypy uses https://github.com/cherrypy/cheroot internally. I wonder if cheroot is sufficient for our goals | 10:34 |
| dtantsur | (it has fewer dependencies) | 10:35 |
| dtantsur | here is an old example: https://stackoverflow.com/questions/55366395/how-to-run-a-flask-app-on-cherrypy-wsgi-server-cheroot-using-https | 10:37 |
| dtantsur | (they're not rich on docs) | 10:37 |
| cid | That would be interesting to find out, and cheroot already exists in upper-constraints. | 10:41 |
| dtantsur | oh, does it? that's a strong argument as well | 10:41 |
| cid | Yea | 10:42 |
| cid | I'm thinking pushing a patch with just cheroot to see CI's reaction. | 10:42 |
| dtantsur | https://cheroot.cherrypy.dev/en/latest/pkg/cheroot.wsgi/ is the relevant part for us, combine it with https://cheroot.cherrypy.dev/en/latest/pkg/cheroot.server/#cheroot.server.HTTPServer.ssl_adapter for TLS | 10:42 |
| dtantsur | cid++ | 10:42 |
| dtantsur | I don't expect surprises, it's the library behind cherrypy after all | 10:42 |
| cid | Right! | 10:43 |
| dtantsur | I cannot find it in global-requirements, but it is already in upper-constraints (I'm curious why) | 10:44 |
| cid | Maybe it's a dependency in some other library | 10:45 |
| * cid looking | 10:46 | |
| dtantsur | Yeah. It's still an easier argument to add it to global-requirements when it's already used by something, even if indirectly. | 10:46 |
| cid | ++, we'll also be getting rid of a layer of abstraction (which is not necessarily good or bad) | 10:47 |
| dtantsur | One thing we need to be careful about, *regardless* of which server we choose. We're not in the real threading zone, so we need to check if IPA possible uses any shared global state without locking. | 10:48 |
| dtantsur | It shouldn't (green threads are also threads), but it's worth an extra look. | 10:48 |
| frickler | does this ring a bell for anyone? | 11:12 |
| frickler | Job metalsmith-integration-glance-netboot-cirros-direct not defined - https://zuul.opendev.org/t/openstack/config-errors?project=openstack%2Fopenstacksdk&skip=0 | 11:12 |
| dtantsur | frickler: mmm, that's an ANCIENT job | 11:14 |
| dtantsur | as in: removed 4 years ago https://opendev.org/openstack/metalsmith/commit/048ddb57440abf8dc6875363035485309db09ab5 | 11:15 |
| dtantsur | frickler: with metalsmith losing in importance here, it's probably fair to remove all jobs based on it from the SDK | 11:16 |
| frickler | dtantsur: ok, that was kind of my implicit question: drop completely or replace with something more modern? but I'm fine with the former | 11:25 |
| frickler | oh, wait, there was https://review.opendev.org/c/openstack/openstacksdk/+/945308 already for master and 2025.1, does it make sense to backport? | 11:27 |
| frickler | cc rpittau since you approved the above | 11:28 |
| iurygregory | good morning ironic o/ | 11:29 |
| dtantsur | frickler: you can backport it for consistency then | 11:36 |
| dtantsur | but it's also fine if you decide to drop metalsmith jobs | 11:37 |
| dtantsur | morning iurygregory | 11:37 |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 11:44 |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 11:48 |
| *** dking is now known as Guest14182 | 11:56 | |
| *** Guest14182 is now known as dking | 11:56 | |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 12:04 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add shared image support https://review.opendev.org/c/openstack/ironic/+/947115 | 12:12 |
| rpittau | frickler: as dtantsur wrote, either is ok, what's simplest is better :) | 12:16 |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 12:23 |
| dking | Would any folks like to comment on a strange issue I'm seeing with a node. I have a server that I've just re-provisioned, but it's failing inspection. It had worked previously, and I manually cleaned up all the left over interface files on the web server. The inspection is opening up the heartbeat callback and waiting for it, and I can even reach it from my Ironic API pod with curl. I see a log entry saying that it's going to | 12:28 |
| dking | ry to _check_status, but it just eventually times out. Anybody have some thoughts? | 12:28 |
| dking | Oh, I just am now seeing in the logs: Heartbeat from node ... in unsupported provision state inspect failed, not taking any action. What could it be thinking the state is in ? | 12:33 |
| opendevreview | Jay Faulkner proposed openstack/ironic-python-agent master: SCIENCE: Remove *ALL THE EVENTLET* https://review.opendev.org/c/openstack/ironic-python-agent/+/947744 | 12:34 |
| dtantsur | dking: this all is not really descriptive. Inspection failed for some reason, then IPA moved into heartbeating, which Ironic is rejecting. | 12:43 |
| dtantsur | All consequences of the initial inspection failure | 12:43 |
| dking | dtantsur: The provision state message looks like it was later, after the inspection failed (and because of it). But I don't see in the logs how to know why it wasn't connecting. It's like I'm not seeing an attempt to connect. | 12:45 |
| dtantsur | To you have IPA logs? The hint is before heartbeating started | 12:48 |
| opendevreview | Merged openstack/networking-baremetal master: Drop redundant allowlist_externals https://review.opendev.org/c/openstack/networking-baremetal/+/947692 | 12:49 |
| opendevreview | Merged openstack/bifrost master: Add notes to provide more clarity for bifrost installation https://review.opendev.org/c/openstack/bifrost/+/946603 | 12:51 |
| dking | dtantsur: I do have the logs. I see "Calling to inspector to check status of node..." and "Successfully released shared lock for checking hardware inspection status on node..." | 12:51 |
| dtantsur | dking: this is ironic, not IPA | 12:51 |
| dtantsur | (there might be further hints in the Inspector service logs) | 12:51 |
| dking | dtantsur: Good point. I think that in my instance (using CAPI/Metal3), the logs for the API and Ironic might be combined, but inspector is separate and I didn't check. | 12:53 |
| dtantsur | Metal3 dropped inspector some time ago, so it's only Ironic. But I don't know how old your setup is. | 12:53 |
| dtantsur | and I'm talking about IPA, not API :) | 12:54 |
| dtantsur | The ramdisk. | 12:54 |
| opendevreview | Merged openstack/ironic master: Release notes title to "unreleased" for in-progress https://review.opendev.org/c/openstack/ironic/+/946933 | 12:55 |
| dking | dtantsur: We're fairly old, I think. We're at ironic:capm3-v1.4.2 for the Ironic image we're running. I'll have to look for IPA's version. I have it version locked rather older also. | 12:55 |
| dking | dtantsur: IPA is at 9.12.0 | 13:05 |
| * TheJulia tries to wake up | 13:06 | |
| dking | dtantsur: However, I don't think that either of these have been updated from what has been working. | 13:06 |
| dtantsur | early 2024 | 13:06 |
| dtantsur | 2024.2 | 13:06 |
| dtantsur | dking: we can guess for a long time, but if you manage to SSH into the running ramdisk and check IPA logs, it will probably tell you the answer | 13:07 |
| dking | dtantsur: Well, I mean we haven't updated our versions of these recently. I need to get on that. | 13:07 |
| dking | dtantsur: I'm in the ramdisk. It's running, and I can watch the logs. As far as it's concerned, it posted a callback for the heartbeat and is waiting for instructions. | 13:07 |
| dtantsur | dking: yep, your problem is before any heartbeating | 13:08 |
| dking | It already finished all of the collectors, etc. | 13:08 |
| dking | Oh, I didn't check for any other errors since it went to heartbeat. One moment. | 13:08 |
| dking | dtantsur: Welp. I see it now. It was an issue with a collector update. I greatly appreciate the help. | 13:09 |
| dtantsur | cool, np | 13:10 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Coverage for neutron with automated cleaning https://review.opendev.org/c/openstack/ironic/+/947535 | 13:22 |
| opendevreview | Ayo Edwin Kayode proposed openstack/bifrost master: Fix RST formatting in testenv.rst to resolve build error https://review.opendev.org/c/openstack/bifrost/+/947830 | 13:23 |
| Ayo[m] | rpittau: I just made the correction on the doc, please review | 13:33 |
| opendevreview | Merged openstack/networking-baremetal master: Fix failing genconfig target https://review.opendev.org/c/openstack/networking-baremetal/+/947691 | 13:33 |
| dking | dtantsur: On reflection, what I think threw me off is that usually if there's a collector error, that error shows up in the API call show the node status. Usually it says that there's an error and gives the error. This time, it said that there was a timeout. | 13:55 |
| dking | I'm wondering what caused that. | 13:56 |
| opendevreview | Verification of a change to openstack/ironic master failed: CI: Coverage for neutron with automated cleaning https://review.opendev.org/c/openstack/ironic/+/947535 | 14:13 |
| opendevreview | Queensly Kyerewaa Acheampongmaa proposed openstack/bifrost master: Clarify testenv and install usage order in testenv.rst https://review.opendev.org/c/openstack/bifrost/+/946116 | 14:24 |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 14:56 |
| TheJulia | Has anyone spent any time looking at the snmp job? https://zuul.opendev.org/t/openstack/builds?job_name=ironic-tempest-ramdisk-bios-snmp-pxe | 16:09 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add shared image support https://review.opendev.org/c/openstack/ironic/+/947115 | 16:15 |
| dtantsur | ugh, pretty red | 16:15 |
| JayF | TheJulia: IMO we should land https://review.opendev.org/c/openstack/ironic/+/946843 then remove that job or make it -nv | 16:23 |
| JayF | because we shouldn't explicitly break the snmp job, but it's an unmaintained driver and we shouldn't go outta our way for CI for it either | 16:24 |
| JayF | ugh that's 2-for-2 on Length too long: 8851389 failures | 16:24 |
| TheJulia | so... | 16:26 |
| TheJulia | https://www.irccloud.com/pastebin/puz6ezo5/ | 16:26 |
| JayF | ugh | 16:29 |
| JayF | I didn't know we had runtime deps on their mirrors for non-ipa-src jobs | 16:29 |
| TheJulia | ugh | 16:33 |
| TheJulia | so, mirrors.dotsrc. are offline | 16:33 |
| TheJulia | well, is | 16:33 |
| TheJulia | https://dotsrc.org/ | 16:33 |
| TheJulia | disk enclosure failure | 16:33 |
| opendevreview | cid proposed openstack/ironic-python-agent master: WIP: Eventlet Removal- WSGI server https://review.opendev.org/c/openstack/ironic-python-agent/+/946091 | 16:36 |
| opendevreview | Julia Kreger proposed openstack/ironic master: ci: mark ramdisk tests non-voting due to mirror failure https://review.opendev.org/c/openstack/ironic/+/947854 | 16:38 |
| dtantsur | ssl.SSLError: [SSL] record layer failure (_ssl.c:2651) | 16:40 |
| dtantsur | bloody eventlet, when are finally getting rid of you? | 16:40 |
| TheJulia | hopefully making progress this cycle.... | 16:41 |
| dtantsur | TheJulia: IPA is nearly ready, cid has done some amazing research job | 16:41 |
| TheJulia | i know :) | 16:41 |
| dtantsur | I looked at Ironic today, and while there are more actions to take there, I think the same approach will be more than actionable. | 16:42 |
| dtantsur | Now, why is bloody JSON RPC even using TLS at all? | 16:42 |
| TheJulia | because secrets can pass over it | 16:43 |
| dtantsur | You're answering the question "why SHOULD JSON RPC use TLS" :) | 16:44 |
| dtantsur | It's a valid question, and you're giving a valid answer | 16:44 |
| dtantsur | but I definitely did not code TLS support for JSON RPC in ironic-standalone-operator | 16:45 |
| * dtantsur reminds everyone that computers were a mistake | 16:45 | |
| cardoe | yep. big EMP. start over. | 17:41 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: Patch configdrive metadata https://review.opendev.org/c/openstack/ironic/+/946677 | 17:42 |
| TheJulia | heh | 19:19 |
| opendevreview | Dr. Jens Harbott proposed openstack/ironic master: Update some docs https://review.opendev.org/c/openstack/ironic/+/947880 | 21:23 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: Trivial: Fix spelling issue in configuration desc https://review.opendev.org/c/openstack/ironic/+/947881 | 21:40 |
| JayF | rpittau: dtantsur: One of you around for a quick question today? If not, we need to sync up early tomorrow morning. | 21:47 |
| TheJulia | its super late for them | 22:22 |
| JayF | I figured, was an opportunistic ask. | 22:37 |
| iurygregory | JayF, does it has to be rpittau and dtantsur only? | 23:12 |
| iurygregory | not sure if I will have the answer, but happy to help | 23:12 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!