| JayF | James is a long time OpenStacker! It's goood to hear about that. That's where I'm from :D | 01:14 |
|---|---|---|
| JayF | thank you for the service steps julia, I'll review tomorrow | 01:14 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Add get_service_steps logic to the agent https://review.opendev.org/c/openstack/ironic-python-agent/+/890864 | 01:18 |
| opendevreview | Tianqi Xiao proposed openstack/networking-generic-switch master: Add package definition for the tempest_plugin module https://review.opendev.org/c/openstack/networking-generic-switch/+/890865 | 01:23 |
| JayF | Hmm. re: my devstack troubles, I have a lot of angry ovsdb_monitor log messages in q-svc logs saying there are no hosts in the hash ring | 02:35 |
| rpittau | good morning ironic! o/ | 07:00 |
| rpittau | JayF mmalchuk, I expect to see some improvements soon on Bifrost CI, the new package is ready but not published yet, you can follow the progress here https://bugs.launchpad.net/neutron/+bug/2029335 | 07:03 |
| mmalchuk | rpittau so, CI still broken? | 07:30 |
| mmalchuk | morning \o | 07:30 |
| rpittau | mmalchuk: unfortunately it is still broken | 07:36 |
| * mmalchuk sad | 07:36 | |
| rpittau | mmalchuk: there's a lot of pressure for that to be verified, lot of bug reports, so I expect the fixed package to show very soon, the new build is actually already available | 07:37 |
| rpittau | https://kojihub.stream.centos.org/koji/buildinfo?buildID=35705 | 07:37 |
| mmalchuk | rpittau and there need a time to sync mirrors when released | 07:38 |
| rpittau | yeah | 07:38 |
| mmalchuk | so not today? | 07:38 |
| rpittau | not sure :/ | 07:40 |
| mmalchuk | :/ | 07:40 |
| kubajj | Good morning Ironic o/ | 07:53 |
| rpittau | hey kubajj :) | 08:40 |
| arne_wiebalck | Good morning rpittau, kubajj and Ironic o/ | 10:14 |
| rpittau | hey arne_wiebalck :) | 10:14 |
| kubajj | Hi arne_wiebalck | 10:15 |
| *** toheeb__ is now known as toheeb | 11:39 | |
| iurygregory | good morning Ironic | 11:44 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [trivial] Remove todo note that was already done https://review.opendev.org/c/openstack/ironic/+/890912 | 12:33 |
| rpittau | hey iurygregory :) | 12:33 |
| iurygregory | o/ | 12:34 |
| arne_wiebalck | hey iurygregory o/ | 13:26 |
| TheJulia | good morning | 13:37 |
| rpittau | hey TheJulia :) | 13:48 |
| iurygregory | hey hey arne_wiebalck o/ | 13:57 |
| mmalchuk | rpittau any CI updates? | 13:57 |
| rpittau | mmalchuk: nope :/ | 13:57 |
| mmalchuk | sad | 13:57 |
| mmalchuk | hi TheJulia | 13:57 |
| iurygregory | good morning TheJulia =) | 13:57 |
| rpittau | I'm waiting for the new systemd pkg to show up here https://mirror.rackspace.com/centos-stream/9-stream/BaseOS/x86_64/os/Packages/, or at least here https://mirror.stream.centos.org/9-stream/BaseOS/x86_64/os/Packages/ | 13:57 |
| mmalchuk | it already built? bot not in place? | 14:02 |
| rpittau | the pkg has not been tagged for release yet https://kojihub.stream.centos.org/koji/buildinfo?buildID=35705 | 14:04 |
| dtantsur | it's possible that they tag things on certain days.. | 14:10 |
| TheJulia | or they had to push async updates which might have bumped things for a day | 14:13 |
| TheJulia | the new intel cpu register vulnerability comes to mind | 14:13 |
| TheJulia | I guess it is good I went on vacation last week.... | 14:30 |
| rpittau | vacation is always good :) | 14:32 |
| mmalchuk | its bad to back from vacation) | 14:41 |
| mmalchuk | don't want to work)) | 14:42 |
| JayF | TheJulia: this is your morning reminder to try to carve out some time to test sharding | 14:43 |
| JayF | I'm not going to be around around until this afternoon. I have a medical appointment. | 14:43 |
| TheJulia | JayF: ack! | 14:44 |
| TheJulia | JayF: test or review what was done :) | 15:05 |
| opendevreview | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Add get_service_steps logic to the agent https://review.opendev.org/c/openstack/ironic-python-agent/+/890864 | 15:44 |
| rpittau | good night! o/ | 15:53 |
| opendevreview | Alex Welsh proposed openstack/bifrost master: Replace Cirros references with generic descriptors https://review.opendev.org/c/openstack/bifrost/+/883943 | 16:12 |
| JayF | TheJulia: yes 😁 | 16:16 |
| TheJulia | k | 16:17 |
| dking | JayF: Could you tell me how to have a collector run again inside of the IPA image after it has run and failed? I would like to be able to update code locally and run it exactly as it would be run. | 16:31 |
| dking | Previously, I had just been opening up a python REPL and loading the collector package and running that, but I think that there is some difference happening when inside of the ironic-python-agent. | 16:32 |
| dking | Oh, I just saw the above. Would anybody else happen to know how to do that? | 16:33 |
| NobodyCam | Good morning Ironic Folks | 16:40 |
| NobodyCam | off the all question, is sushy release tied to openstack release, I.e. Can one run latest sushy with ussuri ironic? | 16:41 |
| TheJulia | So... technically no, realistically, there may be a break since ussuri was ... what sushy 2.x ? | 16:47 |
| TheJulia | Just not tested is maybe the best answer | 16:47 |
| NobodyCam | +++ Thank you | 17:08 |
| dking | Is anybody currently here terribly familiar with the ironic-python-agent code? | 17:11 |
| JayF | I'm here now | 17:11 |
| JayF | but I'm not sure I understand your original ask at 1631 | 17:11 |
| dking | Currently, I'm attempting to modify my custom collector locally and re-run it inside of ironic-python-agent without having to build a new image, PXE again, etc. | 17:12 |
| JayF | oooh | 17:13 |
| dking | I know how to run the custom collector directly through the python REPL, but I am seeing an error that I can only seem to replicate so far in the logs from the initial ironic-python-agent run of the collector. | 17:13 |
| JayF | yeah python repl is probably a reasonable approach to that | 17:13 |
| JayF | can you share the err log? | 17:14 |
| JayF | there used to be a developer mode for IPA - standalone - but that was removed when we added agent token because it was at the intersection of security-risky and technically-hard | 17:14 |
| dking | Unfortunately, the error message itself isn't terribly verbose: Aug 9 12:12:45 localhost ironic-python-agent[1437]: 2023-08-09 12:12:45.249 1437 ERROR ironic_python_agent.utils [-] collector ark failed: not all arguments converted during string formatting#033[00m | 17:16 |
| JayF | oh, that *is* potentially useful | 17:16 |
| JayF | can you share the code in question? | 17:16 |
| JayF | that's going to be something weird in a format string | 17:17 |
| JayF | I think | 17:17 |
| JayF | you are likely missing an int() or a str() somewhere in a logging call, f string, or .format() call | 17:17 |
| dking | That would seem to imply that there's some string formatting issue, and maybe a missing variable. I suspect, though I can't confirm it yet, that it has something to do with eventlet and how I tried to patch my code to work around some issues with proxy URLs. | 17:18 |
| JayF | Yeah if you show me the code I strongly suspect I could spot it | 17:19 |
| dking | This is probably going to get rather involved, unfortunately. Would it be preferable to move this into a direct message? | 17:19 |
| JayF | I prefer in public so others can learn; if the code in question is private I can take a cursory look but won't be able to look in depth | 17:19 |
| dking | Okay, I'm good with that. First, what's the pastebin URL again? | 17:20 |
| dking | I think that I can limit what I share enough that it won't be a problem. | 17:20 |
| JayF | you can use any pastebin you want, I usually use gist.github.com I think there's an openstack one though | 17:22 |
| JayF | yeah https://paste.openstack.org | 17:22 |
| dking | Oh, I just managed to add in some extra debugging, and I found what I suspected. I'm having trouble connecting to a URL (in particular, using pynetbox) which is using a proxy URL. The error message above is really just secondary, and probably related to how I log an error somewhere. | 17:25 |
| JayF | that sounds like it fits | 17:25 |
| JayF | I'll note that if you're experiencing a bug around proxy support, reporting it upstream might save you a patch :) | 17:25 |
| dking | So, the context here is that I'm running code that had previously worked in a new environment where it requires the specific requests to use a proxy URL (set with the `https_proxy` environment variable). This works fine for curl and even inside python using `requests.get(...)`. The problem occurs when I try to use pynetbox. | 17:26 |
| JayF | ah, so basically your custom collector introduces a library that doesn't respect proxy | 17:30 |
| JayF | I would strongly suggest trying to find a different library. What are you using pynetbox for, exactly? | 17:30 |
| dking | When I try to use pynetbox, the URLs gave an issue in Python's ssl.py calling the same super() over 400 times before giving up. The "fix" for this issue was to run `eventlet.monkey_patch()` before loading pynetbox. This works great inside of the REPL where it was failing too, but for some reason, when it is run inside of ironic-python-agent, I am now seeing a connection error. | 17:30 |
| dking | I do some magic updating the server and NetBox using data from both. | 17:31 |
| dking | With pynetbox by itself, there's no problem, even inside the running node. It only seems to happen when pynetbox is run in ironic-python-agent. Also, I see that the "fix" involved the module eventlet, which is a dependency of ironic-python-agent, probably rewriting the connection code somewhere. | 17:32 |
| JayF | It sounds like you're in the neighborhood | 17:33 |
| JayF | eventlet 10000% for sure does monkey patch python modules to make them more green | 17:34 |
| JayF | including ssl, socket, etc | 17:34 |
| JayF | that can break some modules, depending on what they are doing | 17:34 |
| dking | Is there a way to "ungreen" that stuff? I'm guessing it has something to do with SSL, and in this case, particularly with using a proxy URL. | 17:37 |
| JayF | No; and you don't want that even if you think you do -- imagine how much I/O IPA does -- reading/writing images, etc | 17:38 |
| dking | I'm even open to a dirty fix. | 17:38 |
| JayF | if you ungreen it it turns into a giant freeze pop from all the I/O :D | 17:38 |
| JayF | the fix is answer the question of what pynetbox gets you | 17:38 |
| JayF | and find a library, or implement natively, whatever value it's giving you | 17:38 |
| JayF | because not supporting http proxys is a smell to start with, which makes me suspect the eventlet incompatability will not be easy to find (I've literally spent multiple-weeks chasing this kind of issue in IPA in the past) | 17:39 |
| dking | Well, pynetbox supports proxies. Again, it's only a problem when running pynetbox from ironic-python-agent. | 17:45 |
| JayF | You still have never addressed the primary question | 17:47 |
| JayF | > answer the question of what pynetbox gets you | 17:47 |
| JayF | literally, say it out loud | 17:47 |
| JayF | we might suggest a different library :D | 17:47 |
| dking | It gets met he ability to easily and concisely collect data from NetBox and update data from NetBox. | 17:48 |
| dking | If you know of something else, that might be nice. | 17:48 |
| JayF | oh, netbox is a thing? | 17:49 |
| JayF | oh yikes | 17:49 |
| dking | Oh, yeah. It's a service to manage your infrastructure. I suppose that I just assumed you'd be familiar with it. | 17:50 |
| JayF | I have no real answer for you then :( It's likely going to take engineering work to figure out the incompatability | 17:50 |
| JayF | If I were in your shoes I'd do the following: | 17:50 |
| JayF | 1) Try to make a minimal python test case to reproduce. Like eventlet monkey patch, setup the proxy, demonstrate the failure, and file it as an issue with upstream netbox | 17:50 |
| dking | Yeah, probably digging into what exactly IPA does with eventlet, or what eventlet does to ssl.py, etc. And then try to figure out what pynetbox actually needs from it. | 17:50 |
| JayF | no, going that direction will not likely give you anything but pain (IPA<>Eventlet) | 17:51 |
| JayF | looking at how eventlet breaks the tool is useful | 17:51 |
| JayF | but I think the minimal reproducer is a prereq to getting there | 17:51 |
| JayF | but 2) I'd work around by shipping a separate python script which I'd shell out to | 17:51 |
| JayF | and/or shipping a netbox CLI and shelling out to it to get your work done | 17:52 |
| dking | yeah, I might put in an ugly temporary path where the "collector" just calls out to a shell script that runs the actual collector. Because it runs fine. | 17:54 |
| dking | JayF: So, in the meantime, is there a way for me to cause the ironic-python-agent to re-run the collector? | 18:12 |
| JayF | You'd have to re-inspect via Ironic API, I believe | 18:14 |
| JayF | Inspection is not my strength :) | 18:14 |
| * TheJulia is under a rock today, sorry | 18:37 | |
| dking | JayF: So.... does ironic-python-agent pick up environment variables from /etc/environment? | 18:40 |
| JayF | it runs outta a systemd unit | 18:43 |
| JayF | which usually doesn't load /etc/environment on most distros | 18:44 |
| dking | That was my problem currently. However, I still think I needed eventlet.monkey_patch() to get this far. | 18:44 |
| dking | Okay, so a separate issue, still regarding inspection, though. I finally got that node to run the collector. Now, it's in state "inpsect wait", the last logs in IPA show that it is listening, and in the ironic API logs, I see that it's making POST requests with 202 responses, but that's been going for a while, and I suspect that it's going to timeout. Another node already did. | 19:20 |
| dking | Could it be that it's having trouble powering off the node? | 19:21 |
| TheJulia | possibly | 19:55 |
| TheJulia | so inspector will try to ask ironic to power it of I think | 19:56 |
| TheJulia | but there is a weird path in there I don't quite remember | 19:56 |
| dking | Okay. I think I got it. It was just an error I introduced. Still, I would be interested in knowing how to troubleshoot that part of the process better in the future. | 19:59 |
| TheJulia | Without asking more questions I won’t be able to provide an easy answer to that :) | 20:07 |
| TheJulia | Anyway, need to go get a passport photo taken | 20:07 |
| JayF | I will say, some of it is just hard when you're troubleshooting actual-hardware things | 20:07 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!