| opendevreview | Julia Kreger proposed openstack/ironic master: ci: stabilize ironic-standalone-redfish https://review.opendev.org/c/openstack/ironic/+/954303 | 01:20 |
|---|---|---|
| rpittau | good morning ironic! o/ | 06:32 |
| opendevreview | Jacob Anders proposed openstack/ironic master: [WIP] Skip initial reboot to IPA when updating firmware out-of-band https://review.opendev.org/c/openstack/ironic/+/954311 | 06:32 |
| queensly[m] | Good morning | 07:07 |
| dtantsur | TheJulia: good question. Initial boot logging can be pretty useful, e.g. when debugging something around NetworkManager | 07:18 |
| Continuity | Morning all. Has anyone had any experience with fast-track-deploy | 09:27 |
| dtantsur | Continuity: we use it by default in metal3, it's also used in bifrost. If your question is in the context of Nova though, there are way fewer people who tried it. | 11:26 |
| Continuity | dtantsur: thanks, I guess ill be off to do some work on the bleeding edge... and you know why they call it the bleeding edge... | 11:55 |
| Continuity | From all the cuts you get :D | 11:55 |
| opendevreview | Takashi Kajinami proposed openstack/networking-generic-switch master: etcd: Discover api version automatically https://review.opendev.org/c/openstack/networking-generic-switch/+/954331 | 12:01 |
| iurygregory | good morning ironic | 12:10 |
| iurygregory | CI is sad today? <eyes> | 12:16 |
| iurygregory | metal3-integration seems unhappy, but seems like a centos issue because nothing provides the versions we want | 12:19 |
| rpittau | that's unfortunately still centos repos shenanigans impacting us | 12:19 |
| iurygregory | =( | 12:21 |
| rpittau | and well others too :/ | 12:21 |
| iurygregory | pep8 is going crazy also lol | 12:21 |
| iurygregory | ERROR: No matching distribution found for pbr>=2.0.0 | 12:22 |
| rpittau | mmm that's weird | 12:22 |
| iurygregory | https://zuul.opendev.org/t/openstack/build/e388682264654e3992b6b3aeb5f7cd70 a lot | 12:22 |
| iurygregory | before the recheck the job was green XD | 12:23 |
| dtantsur | Does anyone understand the logic that informed this ordering? https://opendev.org/openstack/ironic/src/branch/master/ironic/conductor/steps.py#L34-L41 | 12:24 |
| dtantsur | It means, among other things, that in-band steps from IPA override BIOS interface steps but do not override firmware interface ones. | 12:24 |
| rpittau | iurygregory: that's the error, but the root cause looks like an issue contacting pypi, or better the pypi mirror | 12:25 |
| iurygregory | rpittau, yeah | 12:25 |
| dtantsur | (in case these ever clash, which is a bad idea on its own) | 12:25 |
| iurygregory | dtantsur, wait what? :eyes: | 12:25 |
| dtantsur | iurygregory: steps are collected in the order of these priorities | 12:25 |
| iurygregory | yup | 12:25 |
| iurygregory | but we don't have in-band steps for firmware interface in IPA, so yeah it wouldn't override | 12:26 |
| dtantsur | The more I think about it, the more I'm convinced that I lied to JayF back at the PTG, and overriding steps does not actually work reliably | 12:27 |
| dtantsur | iurygregory: I'm being hypothetical here. The context is the change that janders is working on. | 12:27 |
| iurygregory | oh ok! | 12:27 |
| dtantsur | If overriding steps does not actually work for servicing/cleaning, we're safe to make certain assumptions. | 12:27 |
| iurygregory | I was missing some context <sorry> | 12:27 |
| dtantsur | No worries, I'm struggling to put thoughts into words today | 12:28 |
| iurygregory | same, I only had 5hrs of sleep, so I'm a bit slow | 12:28 |
| * dtantsur is now struggling to understand how software RAID works at all | 12:30 | |
| iurygregory | tldr; magic :D | 12:30 |
| * iurygregory needs more coffee | 12:30 | |
| dtantsur | We might be relying on accident here.. | 12:31 |
| opendevreview | Merged openstack/bifrost master: bug: drop baremetal introspection mention https://review.opendev.org/c/openstack/bifrost/+/946726 | 12:48 |
| TheJulia | good morning | 13:04 |
| TheJulia | dtantsur: magic in the agent :) | 13:04 |
| guilhermesp | hello team! Question: is it possible to use ubuntu-signed DIB element for ipa so I can clean a system with secure boot enabled? | 13:07 |
| iurygregory | I think it should be possible, but I never used ubuntu-signed .. | 13:09 |
| rpittau | guilhermesp: you'll have to include the ubuntu-signed DIB element when using ipa-builder | 13:17 |
| rpittau | guilhermesp: there's asepcific option called "extra-elements" | 13:17 |
| rpittau | errr "extra-element" | 13:18 |
| guilhermesp | rpittau: thanks! trying to get hte idea here... becasue it seems ubuntu-signed is already cooked? https://docs.openstack.org/diskimage-builder/latest/elements/ubuntu-signed/README.html | 13:23 |
| guilhermesp | tried that: DIB_REPOLOCATION_ironic_python_agent=/root/dib/IPA-path/ironic-python-agent DIB_REPOREF_ironic_python_agent=master DIB_RELEASE=ubuntu ironic-python-agent-builder -o test-ipa-ubuntu-signed ubuntu-signed ( becasue i have to add a pr by dtantsur for software raid ) | 13:23 |
| rpittau | guilhermesp: you need the -e option of ipa-builder | 13:25 |
| rpittau | here -> https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html | 13:26 |
| TheJulia | guilhermesp: are you using PXE or virtual media? | 13:51 |
| guilhermesp | pxe TheJulia | 13:52 |
| guilhermesp | i tried injeecting those | 13:53 |
| guilhermesp | https://www.irccloud.com/pastebin/d8wgGZxl/ | 13:53 |
| guilhermesp | but somehow the clean process failed and i could not see logs from it, then the node rebooted and could not pxe boot anymore since secureboot was enabled | 13:54 |
| dtantsur | TheJulia: morning! Do we have a way for a user to provide agent token? And if no, is it something you'd fine acceptable? | 13:55 |
| dtantsur | Context: I have frequent cases of "something happened, and now the token is invalid", which are hard to debug. If we could embed the token into the ISO, it would hopefully be better. | 13:56 |
| dtantsur | And we cannot use Ironic's embedding process since it does not know how to work with CoreOS images | 13:57 |
| TheJulia | guilhermesp: are you using a signed PXE loader, i.e. grub, or self signed ipxe binary? | 13:59 |
| TheJulia | with your own key management | 13:59 |
| TheJulia | dtantsur: eek... uhhhhhh | 13:59 |
| TheJulia | dtantsur: uhhhhhhhh | 13:59 |
| TheJulia | dtantsur: give me a few to ponder this one | 13:59 |
| TheJulia | (make more coffee) | 14:00 |
| dtantsur | sure | 14:00 |
| dtantsur | Separately, I need to pick your brain on threads, periodics and long-running tasks in the post-eventlet world. Whenever you have time and spoons. | 14:01 |
| TheJulia | I should have time after the board meeting, but that means in like 3 hours | 14:02 |
| dtantsur | May be a bit late. I can drop you an email instead and copy cid (if I know your email) and whoever is interested in eventlet. | 14:16 |
| TheJulia | I also have time tomorrow morning, fwiw | 14:16 |
| dtantsur | TheJulia: Wednesdays are usually pretty busy for me. I assume 1500 UTC is a bit early? | 14:17 |
| dtantsur | In any case, an email should give you food for thoughts | 14:17 |
| TheJulia | cool cool | 14:18 |
| TheJulia | uhh | 14:18 |
| TheJulia | I already have someone who has asked for 1500 UTC | 14:18 |
| TheJulia | unfortuantely | 14:18 |
| dtantsur | No worries, let's do it asynchronously | 14:19 |
| dtantsur | TheJulia: these are my expanded thoughts about the agent token: https://bugs.launchpad.net/ironic/+bug/2116188 | 14:26 |
| TheJulia | Okay, I'll try to take a look later today | 14:27 |
| opendevreview | Takashi Kajinami proposed openstack/networking-generic-switch master: etcd: Discover api version automatically https://review.opendev.org/c/openstack/networking-generic-switch/+/954331 | 14:33 |
| dtantsur | Thought dump sent too (you'll probably be happy I was not doing the same on a call - it's messy) | 14:38 |
| kubajj | Hello, we recently came across an issue with the efibootmgr (it took us a while to pinpoint it there). In order to shorten the time to solve the issue, we are planning to include a debug logging of the efibootmgr output before and after the efi_utils._run_efibootmgr runs. I have 2 questions: 1. should I commit this upstream? 2. Is there a convention on how to log blobs of text spanning multiple lines? (I could loop over the | 14:44 |
| kubajj | lines, but am not sure) | 14:44 |
| dtantsur | kubajj: how long is it? If it's like 10 lines, our convention is to just log it. | 14:47 |
| dtantsur | (we actually log pretty long things in IPA - check any CI output) | 14:47 |
| guilhermesp | TheJulia: its a ipxe selfsigned binary i believe. | 14:49 |
| kubajj | dtantsur: I think on a reasonable deployment, it could fit into 10 lines (I tested it on a node we use for QA - i.e. deploy every hour, and it was ~50 lines :D ) | 14:49 |
| dtantsur | kubajj: do you need it in runtime? Another option is to add it to the logs collection. | 14:49 |
| kubajj | dtantsur: nope, just for debugging after | 14:50 |
| dtantsur | kubajj: then add your command to https://opendev.org/openstack/ironic-python-agent/src/branch/master/ironic_python_agent/hardware.py#L3447-L3461 | 14:50 |
| dtantsur | try to enable the maximum verbosity: the output will be in a separate file | 14:51 |
| dtantsur | I'd consider it a very useful addition for upstream IPA btw | 14:51 |
| kubajj | dtantsur: ah, that's neat, where does it store these logs? | 14:52 |
| dtantsur | kubajj: IPA sends them to Ironic, which stores them according to https://opendev.org/openstack/ironic/src/branch/master/ironic/conf/agent.py#L63-L99 | 14:53 |
| dtantsur | docs: https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#retrieving-logs-from-the-deploy-ramdisk | 14:54 |
| kubajj | dtantsur: ok, thanks, I'll test it out and then create a change | 14:54 |
| dtantsur | cool | 14:54 |
| kubajj | dtantsur: I understand now, TIL how the other files other than the journal get created | 14:57 |
| opendevreview | Takashi Kajinami proposed openstack/networking-generic-switch master: Document implicit requirement of etcd3gw backend https://review.opendev.org/c/openstack/networking-generic-switch/+/954351 | 15:06 |
| opendevreview | Verification of a change to openstack/ironic master failed: Handle unresponsive BMC during Firmware Updates https://review.opendev.org/c/openstack/ironic/+/938108 | 15:17 |
| iurygregory | I'm wondering for how long the centos repo will be causing problems =( | 15:47 |
| Sandzwerg[m] | <TheJulia> "sandzwerg: based on what you'..." <- I made some progress. It seems the ubuntu would boot after all it took just like reaaaaally long (in the order of multiple minutes). I can provide a complete log from the sol with grub debug enabled but I saw a ubuntu login screen at the end. That would be really cool, just trying to recreate it. The deployment itself failed still because of a timeout but I didn't watch and have | 16:13 |
| Sandzwerg[m] | currently no way to access the IPA to see what's going on so if secure boot works that would be next. If booting the kernel takes so long that would be odd and something I'd need to get fixed before I can roll it out for productive use cases. What are other people using who are working with secure boot? CentOS? Something completely different? | 16:13 |
| dtantsur | RHEL/CoreOS for us | 16:14 |
| Sandzwerg[m] | Which version? And do you build that with DIB or something else? | 16:15 |
| dtantsur | Sandzwerg[m]: we use official (downstream) CoreOS images for both IPA and final instances. Versions tend to be quite recent and match OpenShift versions we're working with. We don't choose versions ourselves. | 16:19 |
| Sandzwerg[m] | dtantsur are these public available? | 16:26 |
| dtantsur | Sandzwerg[m]: I don't think so :( | 16:26 |
| Sandzwerg[m] | <Sandzwerg[m]> "I made some progress. It seems..." <- boot time over all from "server create" to "I see ubuntu kernel messeages is about 17mins and for the "node powers on and dos virtualmedia" it's probably around 2-3mins or so? | 16:30 |
| stephenfin | cid: The ironic gating job in SDK is permafailing due to a failing inspection rule test. I assume something changed here recently? | 16:30 |
| stephenfin | https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack-ironic&project=openstack/openstacksdk | 16:30 |
| Sandzwerg[m] | dtantsur: to bad | 16:31 |
| stephenfin | Any chance you'd be able to put together a fix if so? | 16:31 |
| * TheJulia exits board meeting | 16:36 | |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Redfish Firmware Interface - NIC Support https://review.opendev.org/c/openstack/ironic/+/953394 | 16:36 |
| TheJulia | dtantsur: yeah, I only scratched the surface of your email and I'll try to reply today. | 16:37 |
| dtantsur | thx! | 16:37 |
| TheJulia | kubajj: I *thought* we already log that, but logging around that is worthwhile if there is an issue. Often UEFI troubleshooting is a pain so all for more logging | 16:38 |
| cid | stephenfin, looking... | 16:40 |
| TheJulia | guilhermesp: okay, so I don't know about ipxe in signed mode, I also don't know if it does next step enforcement or not. I'm not sure it does, actually. One thing which is super interesting is some vendors have explicitly given up on Secure Booting PXE operations because PXE is super insecure anyhow, so internally you may apply settings, then when PXE is requested, the vendor might temporarily disable and then re-enable. | 16:40 |
| TheJulia | We don't have good insight into that, unfortunately. | 16:40 |
| stephenfin | cid: tl;dr: the API was returning 'loop' before but no longer appears to be doing so | 16:40 |
| cid | Ahhhh, yep. | 16:41 |
| TheJulia | guilhermesp: Sounds like if it is still in PXE boot post reboot but then failing, that your vendor is doing something along those lines. | 16:41 |
| TheJulia | iurygregory: speaking of vendors and run, whats the latest on idrac10, trying to figure out what (if anything) I should plan on us doing. | 16:41 |
| iurygregory | TheJulia, let me grab the latest info I have 1min | 16:44 |
| TheJulia | Sandzwerg[m]: WOW, that is crazy. Turning off "nofb" might help. It sounds like it might be some sort of network bandwidth issue in the load process. You said vmedia, so I'd actually try to extend the timeout. A thing to double check is the BMC link rate/speed, if its set to 100M or 1G, is a huge variable. | 16:45 |
| iurygregory | TheJulia, https://paste.opendev.org/show/bGtLlkKmFPqucHacchLV/ this was the latest error we saw when QE was testing, dtantsur gave the idea to include 404 as a retry code in https://opendev.org/openstack/sushy/src/branch/master/sushy/oem/dell/asynchronous.py#L46 , I'm getting back to this today to see if I can have a downstream build for QE with this change, to see if helps | 16:49 |
| TheJulia | Sandzwerg[m]: What your describing sounds very much like the link rate is constrained and there may also be additional latency. Might be worth investigating that path for any issues/constraints. When I was working with etingof early on, we did some early tests with the image backhauled across vpn connections and ended up getting to watch the boot process crawl along for like 45 minutes. A larger kernel/ramdisk that you'd | 16:54 |
| TheJulia | end up with will take ~1 minute in perfect conditions to transfer over the wire to even be in ram for it to boot. | 16:54 |
| TheJulia | iurygregory: I'll want to see what I can get backported, I just don't entirely understand what is broken where at this point and what we need to to do project wise at the moment. | 16:55 |
| iurygregory | TheJulia, so far the fixes are all in sushy | 16:55 |
| iurygregory | we will need them in sushy-oem-idrac for old versions of ironic | 16:56 |
| iurygregory | https://review.opendev.org/c/openstack/sushy/+/950694 | 16:56 |
| iurygregory | after this fix we are hitting the problem I've shared above that can be related to the asynchronous.py | 16:57 |
| TheJulia | iurygregory: okay, if we can detail it out, or at least have it detailed out, it would be super helpful (also because I'm context switching a lot righ tnow) | 16:57 |
| iurygregory | I will add to the etherpad about idrac10 issues | 16:58 |
| TheJulia | cid: I guess stephenfin's item is related to the fix which got merged 1-2 weeks ago?! | 17:04 |
| dtantsur | TheJulia: re replacing local calls for rpc_transport==None: how bad will it be to use JSON RPC on localhost (with a temporary password)? | 17:05 |
| dtantsur | I've considered unix sockets, but the client side of them is quite sad in both python-requests and keystoneauth | 17:05 |
| TheJulia | dtantsur: I'd be cool with that, I was sort of suspecting that might be the path anyhow | 17:06 |
| dtantsur | right | 17:06 |
| TheJulia | .... I'd likely consider a random... on startup password if configured | 17:06 |
| cid | TheJulia, stephenfin. Yeah. The fix should be up in a jiffy :D | 17:06 |
| TheJulia | which could then be shared/established early on and run with | 17:06 |
| dtantsur | TheJulia: yeah, I want to generate a long random password on start-up. | 17:06 |
| TheJulia | dtantsur: sgtm | 17:07 |
| dtantsur | TheJulia: is it safe to assume I don't need TLS in this case or am I being naive? :) | 17:07 |
| TheJulia | dtantsur: if your binding to localhost, it stays inside the host, so... Its likely fine | 17:07 |
| TheJulia | anyone who does anything like "iptables -t nat -I PREROUTING -j DNAT --to 127.0.0.1:8080" deserves what they get... | 17:08 |
| dtantsur | :D | 17:08 |
| TheJulia | That sounds worse than I was thinking, but it gets the point across | 17:08 |
| TheJulia | we can't be expected to guard rail that | 17:08 |
| TheJulia | since it is such an unexpected/unreasonable thing to really do | 17:09 |
| TheJulia | so as long as the config loading locks it in, it should be okay | 17:09 |
| TheJulia | *one* thought I had was order of operations wise, I expect we *may* need to break the metal3-integration and bifrost jobs for a relatively short period of time, so I suspect whatever order *works* for us, we could just stack everything, get to the confidence level we want and potentially toggle jobs (with upfront communication) and all | 17:10 |
| dtantsur | TheJulia: let's see when we get there. I hope we can have some sort of a smooth upgrade, but it may be wishful thinking | 17:11 |
| TheJulia | I *fully* expect once you do this, we might see some fun with locking and sqlite3 (as we've discussed) as well, so we might want to just get everything stacked up and just figure out the path to merge the things in whatever path makes sense | 17:11 |
| dtantsur | right | 17:11 |
| TheJulia | yeah, I think we can get to smooth, but at a minimum we may have a day or two where we know a job is going to be very broken | 17:11 |
| dtantsur | Mmmm, googing confirms my suspicion that tcpdump will work on loopback | 17:12 |
| TheJulia | one step at a time | 17:12 |
| TheJulia | oh, yes, it very much does :) | 17:12 |
| TheJulia | *and* the MTU is 64k | 17:12 |
| dtantsur | Mmm, okay, I think I'll generate a temporary TLS certificate as well. You're never too safe, right? | 17:13 |
| TheJulia | Well... hmm... errr.... | 17:13 |
| TheJulia | https://tenor.com/view/castle-nathan-fillion-richard-castle-talk-talking-gif-5154792 <-- exactly what I'm doing | 17:13 |
| dtantsur | I was imagining a cat (speaking tongues as they do), but this works as well | 17:14 |
| Sandzwerg[m] | <TheJulia> "Sandzwerg: What your describing..." <- That's a good idea, I'll investigate that tomorrow. I think it's 1G but I'm not sure. And apparently swift is quiet slow for this and the ubuntu image is ~700mb so much much bigger than our older images ( we used a second stage there to circumvent some of the issues). maybe also rebuilding it as ubuntu-minimal might help. Thanks for the pointers | 17:17 |
| TheJulia | Sandzwerg[m]: the other thing is firmware blobs which get included. Even upstream right now with centos, our compressed image size is like 506MB for just the ramdisk, which uncompresses to like 833MB, and of that ~250MB is just firmware blobs for variou shardware. Like... in centos, the Mellanox Spectrum series of images are present in the ramdisk and that alone is 90MB and if anyone is installing centos on switches, I | 17:19 |
| TheJulia | really want to meet them. | 17:19 |
| Sandzwerg[m] | Ha, centos on switches would be interesting indeed. On the other hand I was so surprised when I noticed that arista switches are based on an ancient fedora version (I think it was 18 or something?) | 17:20 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Stop short-cutting to local manager with all-in-one processes https://review.opendev.org/c/openstack/ironic/+/954363 | 17:21 |
| * TheJulia twitches | 17:21 | |
| TheJulia | (that was w/r/t arista) | 17:21 |
| Sandzwerg[m] | Apparently by now it's only veeeery losley based on that, but they apparently started by forking fedora. it was quiet a WTF moment when I found the version file when poking on a switch | 17:22 |
| Sandzwerg[m] | *they | 17:22 |
| TheJulia | wow | 17:23 |
| TheJulia | I'm not really that surprised actually | 17:23 |
| TheJulia | A good example is SONiC is based on a fork of debian | 17:23 |
| Sandzwerg[m] | TIL | 17:27 |
| Sandzwerg[m] | https://en.wikipedia.org/wiki/Arista_Networks#Extensible_Operating_System it's even in the wikipedia | 17:28 |
| TheJulia | TIL! | 17:29 |
| opendevreview | Chris Krelle proposed openstack/ironic master: update Jinja2 to address CVE-2024-2383 https://review.opendev.org/c/openstack/ironic/+/953902 | 18:36 |
| NobodyCam | sorry that took so long to update. | 18:38 |
| guilhermesp | thanks TheJulia yeah... in this case im dealing with some lenovo gear here and ended up getting a https://pubs.lenovo.com/sr650/FQXSFPU0023G after enabling secure boot which is not so flexible to automatically switch things... i got this warning there even disabling secure boot and apparently lenovo needs to reset secureboot keys, so yeah kinda of cumbersome | 18:58 |
| TheJulia | NobodyCam: no worries | 18:58 |
| TheJulia | guilhermesp: Sigh, yeah, lenovo. | 18:59 |
| TheJulia | guilhermesp: I even forgot they will do that. SR650s in particular are... weird. | 18:59 |
| TheJulia | A bunch of our EFI record management logic is due to the SR650s because you get only *one* write per boot, otherwise it reverts UEFI records/settings to last known good. | 18:59 |
| cid | Talking about taking long to update. TheJulia, stephenfin, I bet the fix to https://zuul.opendev.org/t/openstack/builds?job_name=openstacksdk-functional-devstack-ironic&project=openstack/openstacksdk is likely very tiny. But I have been starring at it for a while now and I don't see how the last patch failing that test. | 19:00 |
| TheJulia | I, too, was looking at that error oddly. I was wondering if the error message was odd, and if the logged text was swapped in terms o fwhat was receieved and expected | 19:02 |
| cid | It's like it's working in reverse. The field is now optional and tests, you want to fail now? | 19:03 |
| cid | There's no where the loop field is added in the functional test, and it seems the API is returning correctly too. | 19:06 |
| cid | I will just look at it tomorrow. | 19:07 |
| TheJulia | I think the test is codified in the sdk | 19:07 |
| TheJulia | and that is part of it | 19:07 |
| TheJulia | I too, looked at it and was like "what?!" and then realized it was something in the sdk | 19:08 |
| guilhermesp | TheJulia: er i figure that would be complicated. All fleet will be composed by those sr650s... but there is just one specific use case which users will need secure boot, so im just thinking of, besides getting a signed OS to deploy, just ask ppl to run redfish commands to enable secure boot one node is ACTIVE and before releasing it, disable secure boot lol not very practical but i guess it is what it is | 19:23 |
| guilhermesp | s/one node/when a node/ | 19:24 |
| TheJulia | iurygregory: much appreciated | 20:32 |
| * TheJulia blinks | 20:32 | |
| TheJulia | wow the log jumped | 20:32 |
| TheJulia | guilhermesp: well, does the existing knobs not work for setting it by default? Because it sounds like they disable it for PXE, but maybe they don't re-enable it? | 20:33 |
| TheJulia | Depending on your version, maybe a service step could be whipped up (like whipped cream) to apply it after the fact. | 20:34 |
| opendevreview | Verification of a change to openstack/ironic master failed: Handle unresponsive BMC during Firmware Updates https://review.opendev.org/c/openstack/ironic/+/938108 | 20:45 |
| TheJulia | cid: so I sort of got on this subject (and this doesn't need an immediate reply, I'm only mentioning it here because I mentioned it in the email back to yourself and dtantsur). I'm wondering how we might want to load test our concurrency. I'm thinking the easy path may be to spin up a bunch of fake sushy services I'm thinking something locally so we can explicitly stress a few aspects under "normal" circumstances, I'm | 20:58 |
| TheJulia | just not sure of any other good ways. Have you managed to put any time into that subject yet? also, if anyone else has any thoughts in this area, it would be good to get them out into discussion | 20:58 |
| opendevreview | Verification of a change to openstack/ironic master failed: Handle unresponsive BMC during Firmware Updates https://review.opendev.org/c/openstack/ironic/+/938108 | 23:45 |
Generated by irclog2html.py 4.0.0 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!