| kamlesh6808c | Good morning Ironic! | 05:22 |
|---|---|---|
| arne_wiebalck | Good morning kamlesh6808c and Ironic! | 06:54 |
| janders | good morning arne_wiebalck o/ | 06:57 |
| janders | ajya looking at potential fix to the verfy_steps issue for older iDRACs, I noticed that the WSMAN code has some exception handling while the Redfish one doesn't: | 07:08 |
| janders | RF: https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/drac/management.py#L619 | 07:08 |
| janders | WSM: https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/drac/management.py#L802 | 07:09 |
| janders | also, looking at the sushy_oem_idrac code - maybe we just need to check if this URL exists before calling it - WDYT: https://opendev.org/openstack/sushy-oem-idrac/src/branch/master/sushy_oem_idrac/resources/manager/job_service.py#L53 | 07:13 |
| janders | reminds me of a fix I made for another vendor: https://opendev.org/openstack/sushy/src/branch/master/sushy/resources/system/system.py#L286 | 07:14 |
| janders | would that be a good approach (also CC rpittau iurygregory dtantsur as we were chatting about this last week) | 07:15 |
| ajya | Hi janders , I think for WS-Man it is for general error handling if smth goes wrong, not to handle cases where it's not supported. Could add similar to redfish part, but what is the current exception being raised? | 07:20 |
| ajya | Could it be enough to `except` that in Ironic and re-raise with more info? | 07:21 |
| rpittau | good morning ironic! o/ | 08:08 |
| arne_wiebalck | hey janders ajya rpittau o/ | 08:27 |
| rpittau | hey arne_wiebalck :) | 08:28 |
| janders | ajya had to step out for a bit, back now, finding the exception | 08:42 |
| ftarasenko | gm ironic! I have some problems with software raid in W, can you point me if it was fixed or I can help with fix if not. | 08:47 |
| ftarasenko | Currently, after provision, I have no rd.md.uuid in kernel cmdline, so I need to do mdadm --assemble --scan in dracut after server reboot (Centos Stream) | 08:47 |
| ftarasenko | After that, I add rd.md.uuid to kernel cmdline and everything works and survives after reboot. | 08:48 |
| ftarasenko | Second problem is /boot/efi mounted to /dev/md126p1 instead of /dev/md127, so I need to copy grubenv file to md127 device | 08:49 |
| arne_wiebalck | ftarasenko: we use `rd.auto` in our stream images, that seems to be sufficient | 08:53 |
| arne_wiebalck | `rd.auto` as part of the kernel cmdline I mean | 08:54 |
| ftarasenko | arne_wiebalck: sounds good, but why we do not set this option by ipa? | 08:54 |
| arne_wiebalck | ftarasenko: this is inside the image you deploy, so no need to set anything in the IPA :) | 08:55 |
| arne_wiebalck | ftarasenko: I think there is some code to set the UUID, it could probably be done in a similar way | 08:55 |
| ftarasenko | arne_wiebalck: So I need to build image with rd.auto and use it everywhere (as I remember, it does not break boot if we do not have software raid)? | 08:56 |
| arne_wiebalck | ftarasenko: that is correct, we have this on all our images, with and without raid | 08:56 |
| ftarasenko | arne_wiebalck: nice idea, I'll try now, thank you. But I still think that IPA need to add this option while creating software raid. | 08:58 |
| ftarasenko | arne_wiebalck: Have you seen problem with mounts? When /boot/efi mounted to /dev/md126p1 instead of /dev/md127 ? | 08:58 |
| arne_wiebalck | ftarasenko: happy to receive patches to IPA :) | 08:59 |
| arne_wiebalck | ftarasenko: the mount issue is that there are multiple EFI devices | 08:59 |
| arne_wiebalck | ftarasenko: the one which comes from the image plus the ones Ironic creates | 09:00 |
| arne_wiebalck | ftarasenko: there should be a patch to remove the one from the image, so the new one is mounted ... let me check | 09:00 |
| arne_wiebalck | ftarasenko: https://github.com/openstack/ironic-python-agent/blame/master/ironic_python_agent/raid_utils.py#L354-L355 | 09:06 |
| arne_wiebalck | ftarasenko: this will remove the source EFI partition, so that the new one is mounted | 09:06 |
| arne_wiebalck | ftarasenko: should be in W as well: https://github.com/openstack/ironic-python-agent/blob/stable/wallaby/ironic_python_agent/extensions/image.py#L540-L541 | 09:07 |
| arne_wiebalck | ftarasenko: probably as a backport, so you may want to check if you have it | 09:07 |
| ftarasenko | arne_wiebalck: Thought that I have fresh W IPA image, but I'll recheck. Thank you for your assistance. | 09:08 |
| ftarasenko | rd.auto=1 works well, already started rebuild of my image. | 09:09 |
| arne_wiebalck | ftarasenko: well, it could be something else of course, but the lines I pointed you to are supposed to address the issue you see | 09:09 |
| arne_wiebalck | ftarasenko: nice | 09:09 |
| ftarasenko | Command: cp /dev/md127p1 /dev/md0 | 09:11 |
| ftarasenko | cp: error writing '/dev/md0': No space left on device\n | 09:12 |
| ftarasenko | Strange, it fails for unknown reason. | 09:12 |
| ftarasenko | sorry, not this problem. Just an error in logs | 09:13 |
| dmellado | morning ironic | 09:14 |
| arne_wiebalck | rpittau: do we have a recipe how to build arm images with IPAB? or is that self-evident? :-D | 09:19 |
| rpittau | arne_wiebalck: mmmmm build on arm hardware? :P | 09:20 |
| arne_wiebalck | rpittau: heh | 09:20 |
| arne_wiebalck | rpittau: cross-building is an option, or not at all? | 09:21 |
| rpittau | arne_wiebalck: it can be done using the target architecture option | 09:23 |
| rpittau | only with DIB though | 09:23 |
| rpittau | there's a reference here https://docs.openstack.org/ironic-python-agent-builder/latest/admin/dib.html#with-diskimage-builder | 09:24 |
| dtantsur | morning folks! How are CI gods today? | 09:46 |
| dtantsur | do we have a DIB regression https://e9bcf39a4c4d20634a76-e4a3bc1e570c5531ddd029ae927c00a3.ssl.cf2.rackcdn.com/845788/1/check/ipa-tempest-bios-ipmi-direct-src/ffd7732/job-output.txt > | 09:50 |
| dtantsur | ? | 09:50 |
| dtantsur | hmm, after checking DIB patches: this is a CS9 regression (yay!), should be fixed already | 10:03 |
| arne_wiebalck | rpittau: thanks, we will give it a try :) | 10:05 |
| rpittau | np | 10:05 |
| rpittau | dtantsur: still missing new swiftclient to fix grenade :/ | 10:05 |
| dtantsur | rpittau: is the swift team aware of that? | 10:09 |
| rpittau | there were comments in the fix patch, but haven't reach to them directly | 10:10 |
| dtantsur | rpittau: is there something we can do to work around it in Yoga? | 10:13 |
| dtantsur | like, tell swift to accept old hashes, dunno | 10:13 |
| janders | https://review.opendev.org/c/openstack/devstack/+/845675 rpittau would this still be potentially useful? | 10:15 |
| rpittau | dtantsur: I proposed a backport of the fix in devstack ^ | 10:15 |
| rpittau | of the workaround, not fix | 10:15 |
| dtantsur | rpittau: I suggest we do a dirty thing and apply it in ironic yoga :) | 10:20 |
| dtantsur | this can be merged like today.. | 10:20 |
| rpittau | heh didn't really want to do that, but at this point..... :/ | 10:23 |
| dtantsur | yep\ | 10:23 |
| iurygregory | good morning Ironic | 10:51 |
| janders | hey iurygregory | 11:15 |
| janders | back on local time? :) | 11:16 |
| iurygregory | yeah =) | 11:17 |
| arne_wiebalck | Dear Ironicers, let me briefly introduce kubajj to you: he joined the CERN Ironic team last week will be working on the Ironic safeguards we discussed some weeks ago. | 11:37 |
| dtantsur | welcome kubajj! | 11:37 |
| kubajj | Hey everyone | 11:38 |
| arne_wiebalck | ... he joined last week and already has questions I cannot answer :-D | 11:38 |
| dtantsur | a solid start :) | 11:38 |
| arne_wiebalck | heh | 11:39 |
| nightmare_unreal | welcome kubajj | 11:43 |
| kubajj | I am using devstack to try to modify the cleaning process so that a user can specify some drives that should not be touched. (So far, I am still in the part of learning how to use devstack.) And I was wondering, how does the Ironic Python Agent development work in devstack? For conductor and api I can simply modify the files in /opt/stack and restart the appropriate service, but what is the process for IPA? | 11:47 |
| dtantsur | I wonder if this is documented somewhere.. | 11:59 |
| rpittau | hey kubajj welcome :) | 12:00 |
| * dtantsur writes some docs | 12:00 | |
| janders | Hey kubajj, welcome! | 12:01 |
| janders | IIRC, there was a way to point https://opendev.org/openstack/diskimage-builder to a particular in-flight change | 12:01 |
| janders | but it's been a while since I've tried that, not sure about the details | 12:01 |
| dtantsur | I'm writing a guide, I do it quite often | 12:04 |
| janders | dtantsur that's great, thank you! | 12:05 |
| kubajj | Thank you dtantsur | 12:05 |
| rpittau | dtantsur: for the swift-related issue, I was thinking to add the change here https://github.com/openstack/ironic/blob/stable/yoga/devstack/lib/ironic#L1520 since it's related to tempurls, wdyt? | 12:07 |
| rpittau | mmm we probably need to restart swift though | 12:08 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-python-agent-builder master: Document building an image with patches https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/846609 | 12:09 |
| dtantsur | kubajj, janders, IIRC ^^^ | 12:09 |
| dtantsur | testing welcome | 12:09 |
| dtantsur | rpittau: I don't remember the service start-up ordering in devstack, to be honest.. | 12:09 |
| iurygregory | welcome kubajj o/ | 12:12 |
| rpittau | I think ironic is last in any case, maybe we can check if swift is running or not, I'll give that a try | 12:13 |
| opendevreview | Riccardo Pittau proposed openstack/ironic stable/yoga: Add workaround for sha1 support removal from Swift https://review.opendev.org/c/openstack/ironic/+/846610 | 12:22 |
| rpittau | it looks like all services are configured before starting, so this ^ should work | 12:23 |
| dtantsur | is anyone looking into https://etherpad.opendev.org/p/Fix_Horizon_Plugins_With_Angularjs_v1.8.2.2 already? | 12:35 |
| rpittau | not me, didn't even see that :/ | 12:36 |
| * dtantsur asks for help on the ML | 12:38 | |
| opendevreview | Merged openstack/sushy-tools master: Remove support for Python 3.6 and 3.7 https://review.opendev.org/c/openstack/sushy-tools/+/845914 | 12:41 |
| iurygregory | I missed that thread on the ML... (it would be good if they tagged the projects but ok...) | 12:51 |
| dtantsur | yeah, I have everything with the word "ironic" come to my inbox | 12:55 |
| dtantsur | HTTP GET https://<snip>/redfish/v1/Systems/1/EthernetInterfaces/DE07A001 returned code 500. iLO.0.10.ExtendedInfo: See @Message.ExtendedInfo for more information. Extended information: [{'MessageArgs': ['Unexpected error (00,4E,34)'], 'MessageId': 'iLO.2.14.InternalErrorWithParam'}] | 13:46 |
| dtantsur | anyone ever seen anything like that? I see stendulker is not online.. | 13:46 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Fix discovering WWN/serial for devicemapper devices https://review.opendev.org/c/openstack/ironic-python-agent/+/845788 | 13:56 |
| dtantsur | next, something fishy in https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_019/845788/1/gate/ipa-tempest-bios-ipmi-direct-src/019aa78/controller/logs/ironic-bm-logs/node-1_console_2022-06-20-13:49:43_log.txt | 14:00 |
| dtantsur | at some point IPA loses network connectivity (ironic cannot reach it either) | 14:00 |
| dtantsur | I'm also curious why IPA uses the ironic's public URL | 14:05 |
| dtantsur | meanwhile: https://support.hpe.com/hpesc/public/docDisplay?docId=a00123948en_us&docLocale=en_US #thisisfine | 14:10 |
| *** mat_fechner is now known as matfechner | 14:14 | |
| rpittau | "may Return a 500 HTTP Error" I like the subtle chaos of that statement | 14:33 |
| rpittau | I wonder if grenade is smart enough so that I can test the swift workaround with a depends-on, let's try | 14:37 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [DNM] Test grenade swift workaround https://review.opendev.org/c/openstack/ironic/+/846639 | 14:39 |
| iurygregory | seems like today is Holiday in the US... | 14:42 |
| rpittau | it is indeed | 14:42 |
| * iurygregory wondering if we will have quorum for the meeting | 14:42 | |
| iurygregory | probably worth a quick meeting anyway | 14:43 |
| rpittau | only time will tell.... | 14:43 |
| iurygregory | just to sync on patches that needs review | 14:43 |
| iurygregory | ouch "may Return 500" not good =( | 14:43 |
| rpittau | I don't think my workaround is working | 14:48 |
| opendevreview | Riccardo Pittau proposed openstack/ironic stable/yoga: Add workaround for sha1 support removal from Swift https://review.opendev.org/c/openstack/ironic/+/846610 | 14:54 |
| rpittau | I think we don't enabled tls-proxy always with tempurls, probablyt need a different if-then entry | 14:54 |
| iurygregory | #startmeeting ironic | 15:00 |
| opendevmeet | Meeting started Mon Jun 20 15:00:00 2022 UTC and is due to finish in 60 minutes. The chair is iurygregory. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:00 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:00 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:00 |
| iurygregory | Hello ironicers, welcome to our weekly meeting | 15:00 |
| iurygregory | o/ | 15:00 |
| erbarr | o/ | 15:00 |
| ajya | o/ | 15:00 |
| matfechner | o/ | 15:01 |
| iurygregory | I don't think we will have quorum, so let's try a quick meeting so people can add patches for review =) | 15:01 |
| smachi | o/ | 15:01 |
| rpittau | o/ | 15:01 |
| iurygregory | #topic Announcements / Reminder | 15:01 |
| iurygregory | #info OIS Berlin 2022 playlist is available on youtube | 15:02 |
| iurygregory | #link https://www.youtube.com/playlist?list=PLKqaoAnDyfgqUDjoZhZnDKF5aJrLMnm9E | 15:02 |
| iurygregory | Does anyone have something to add for Announcements / Reminder ? | 15:03 |
| iurygregory | ok, moving on | 15:04 |
| iurygregory | no action items so skipping next topic, since we don't have everyone let's skip subteam status | 15:05 |
| iurygregory | #topic Deciding on priorities for the coming week | 15:05 |
| iurygregory | #link https://review.opendev.org/q/status:open+hashtag:ironic-week-prio | 15:05 |
| iurygregory | Does anyone have patches we should try to take a look this week? | 15:05 |
| ajya | can this be added - https://review.opendev.org/c/openstack/ironic/+/842979 ? | 15:05 |
| snanda | Can these be added https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/843590 and https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/838462 | 15:05 |
| rpittau | I guess this one if it works https://review.opendev.org/c/openstack/ironic/+/846610 | 15:05 |
| snanda | Also please review below patch: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/845577 | 15:05 |
| JayF | I'd note we probably need to go the other direction as well -- removing old patches from ironic-week-prio tag | 15:06 |
| JayF | we have several in there that are not in a mergable state and are months old | 15:06 |
| dtantsur | ++ | 15:06 |
| rpittau | yep | 15:06 |
| iurygregory | ajya, sure (feel free to add it) | 15:06 |
| smachi | also please review below patch : https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/845356 | 15:07 |
| iurygregory | smachi, they are ok to add I would say =) | 15:08 |
| iurygregory | rpittau, right the one to fix the swift problem ++ | 15:08 |
| rpittau | heh hopefully will have the new swiftclient soon | 15:08 |
| iurygregory | JayF, agree, I will do a clean-up this week | 15:08 |
| JayF | thank you | 15:09 |
| JayF | The other question related to priorities; when reviewing I've noticed an extremely large number of gate/check failures | 15:09 |
| JayF | is there any kind of event going on? Have we just passed some threshold of flappiness in tests passing? | 15:09 |
| JayF | I try to avoid recheck-spam unless I know for sure it's not an issue we have any control over; but I'm not sure how much folks have looked yet | 15:10 |
| smachi | <iurygregory> ok | 15:10 |
| iurygregory | I haven't really looked at CI (got back from Summit last week), I'm aware about the problem with swift only | 15:10 |
| rpittau | the CI in ironic at the moment is broken because of the grenade job | 15:10 |
| JayF | I don't know what issue you're talking about, and nothing is documented in the whiteboard | 15:10 |
| rpittau | swift removed support for sha1 and we need a new compatible swiftclient | 15:10 |
| JayF | Is that not the place to look for that info anymore? | 15:11 |
| JayF | ah | 15:11 |
| iurygregory | JayF, right the CI information and status says we are Green | 15:11 |
| rpittau | we are very red :/ | 15:12 |
| iurygregory | ouch >.< | 15:13 |
| iurygregory | I will see if I can spin up a devstack to test things | 15:13 |
| iurygregory | the py36 rant was complicated during past weeks | 15:14 |
| opendevreview | Merged openstack/ironic stable/train: CI: Removing ironic job queue https://review.opendev.org/c/openstack/ironic/+/843516 | 15:15 |
| rpittau | there are some patches to remove support for py36 once and for all, I'll tag them into priorities | 15:16 |
| iurygregory | I will try to look at the state of other jobs to see if there is something really broken or if is just related to infra and update the whiteboard | 15:16 |
| iurygregory | rpittau, thank you! | 15:16 |
| iurygregory | I think that's all for today ... | 15:17 |
| iurygregory | the topic I have for discussion I will see if I can bring up next week (in case we have more people in the meeting =) ) | 15:18 |
| iurygregory | thanks everyone! | 15:20 |
| iurygregory | #endmeeting | 15:21 |
| opendevmeet | Meeting ended Mon Jun 20 15:21:00 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:21 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-06-20-15.00.html | 15:21 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-06-20-15.00.txt | 15:21 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-06-20-15.00.log.html | 15:21 |
| dtantsur | JayF: check the scrollback before the meeting, I was complaining about a curious error in the CI | 15:24 |
| JayF | dtantsur: I don't really have time to point at this today; I spent some time Friday trying to review and look at CI and I got a little frustrated b/c it was clear something was going on but it wasn't documented | 15:25 |
| JayF | dtantsur: If there's a bug or something, or you wanna email it to me, I can look at it when I have more time | 15:25 |
| dtantsur | I haven't filed anything (and don't have any energy left for today) | 15:25 |
| dtantsur | I have a feeling our CI is dying from 100 paper cuts | 15:25 |
| dtantsur | on this positive note, I'm going to lie down | 15:26 |
| dtantsur | see you tomorrow o/ | 15:26 |
| JayF | dtantsur: that's kinda the impression I got. If we could write down the location of one or two of those paper cuts, that's the perfect thing for me to spike on with my limited time for Ironic :) | 15:26 |
| rpittau | going offline too for today, have a great rest of the day everyone! o/ | 15:26 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Remove importlib-metadata from requirements https://review.opendev.org/c/openstack/ironic-python-agent/+/846782 | 15:33 |
| rpittau | ^ we really need this :) | 15:33 |
| rpittau | ok leaving for real now | 15:33 |
| rpittau | o/ | 15:33 |
| TheJulia | sigh, I need to do some board stuff this week (although today is a holiday), and can likely help look at CI stuffs Thursday/Friday | 16:04 |
| TheJulia | just fwiw | 16:05 |
| opendevreview | Merged openstack/bifrost master: Remove support for Python 3.6 and 3.7 https://review.opendev.org/c/openstack/bifrost/+/846154 | 18:52 |
| opendevreview | Verification of a change to openstack/ironic-python-agent master failed: Remove importlib-metadata from requirements https://review.opendev.org/c/openstack/ironic-python-agent/+/846782 | 19:38 |
| opendevreview | Merged openstack/virtualbmc master: Remove support for Python 3.6 and 3.7 https://review.opendev.org/c/openstack/virtualbmc/+/845915 | 19:44 |
| janders | good morning Ironic o/ | 21:27 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!