| iurygregory_ | good morning Ironic o/ | 06:28 |
|---|---|---|
| *** iurygregory_ is now known as iurygregory | 06:28 | |
| *** rpittau|afk is now known as rpittau | 07:24 | |
| rpittau | good morning ironic! o/ | 07:24 |
| iurygregory | good morning rpittau o/ | 07:25 |
| rpittau | heu iurygregory :) | 07:25 |
| rpittau | or even hey | 07:25 |
| iurygregory | heu sounds nice also :D | 07:26 |
| dtantsur | morning ironic | 07:53 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Use native Centos 8 Stream for dib images https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/805957 | 08:25 |
| opendevreview | Mark Goddard proposed openstack/bifrost stable/wallaby: Use stable/wallaby branch to install dependencies https://review.opendev.org/c/openstack/bifrost/+/806104 | 08:27 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Add uWSGI role and use systemd instead of emperor mode https://review.opendev.org/c/openstack/bifrost/+/806106 | 08:31 |
| iurygregory | good morning dtantsur | 08:32 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Redfish RAID: Use identity instead of durable_name https://review.opendev.org/c/openstack/ironic/+/805866 | 08:43 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add RAID mode conversion for idrac-redfish https://review.opendev.org/c/openstack/ironic/+/804112 | 08:49 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: consolidate uWSGI config, drop non-existing plugin https://review.opendev.org/c/openstack/bifrost/+/806121 | 09:09 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: consolidate uWSGI config, drop non-existing plugin https://review.opendev.org/c/openstack/bifrost/+/806121 | 09:11 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Add uWSGI role and use systemd instead of emperor mode https://review.opendev.org/c/openstack/bifrost/+/806106 | 09:13 |
| iurygregory | timeu, jungleboyj hey o/ since you are Lenovo experts, have you ever tested redfish virtual media with Lenovo SR650 (ThinkSystem) - Type 7X06 ? we are having some weird issues because the iso generated by ironic causes a Kernel Panic https://paste.opendev.org/show/808333/ , I'm wondering if there is something specific that we need to set in the XClarity controller (any required configuration or something). Thanks! | 09:14 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Allow configuring enabled raid interfaces https://review.opendev.org/c/openstack/bifrost/+/806123 | 09:24 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: deprecate a separate admin service https://review.opendev.org/c/openstack/bifrost/+/806125 | 09:31 |
| *** sshnaidm|afk is now known as sshnaidm | 09:38 | |
| timeu | iurygregory | 09:41 |
| timeu | yes | 09:41 |
| timeu | had the same issue, was due to the size of the IPA | 09:41 |
| iurygregory | hummm | 09:41 |
| timeu | which caused an error when loading the ramdisk | 09:41 |
| timeu | and ended up in a kernel panic | 09:42 |
| timeu | because it couldn't load the root filesystem | 09:42 |
| timeu | we had to swithc tot he upstream victoria IPA images that are smaller | 09:42 |
| iurygregory | WOW | 09:42 |
| timeu | however we tested this on a SR530 | 09:42 |
| timeu | but I believe it might be the same issue | 09:42 |
| timeu | the redhat upstream IPA images are too big as it seems | 09:42 |
| iurygregory | yeah .-. | 09:42 |
| timeu | sec I have a screenshot of the console | 09:43 |
| dtantsur | oookay, now we have an example of something that will get worse if we switch to Stream.. | 09:43 |
| iurygregory | :D | 09:43 |
| * iurygregory wondering if there is a possible workaround =X | 09:43 | |
| * iurygregory will cry trying to solve the Lenovo bug now | 09:44 | |
| timeu | we tested this on the latest XCC/UEFI firmware | 09:44 |
| timeu | we get the error " can't allocate initrd" | 09:44 |
| iurygregory | I think they are using the latest XCC | 09:44 |
| iurygregory | 6.6 or something | 09:44 |
| timeu | and then a kernel panic with VFS unable to mount root fs on unknown-block | 09:44 |
| timeu | they should try the https://tarballs.opendev.org/openstack/ironic-python-agent/dib/files/ipa-centos8-stable-ussuri.kernel | 09:45 |
| iurygregory | time to buy more beer (I think I will need) | 09:45 |
| timeu | ussuri centos8 ipa images and see if that fixes their issue ;-) | 09:45 |
| timeu | hehe yeah or vodka ;-) | 09:45 |
| iurygregory | ++ | 09:45 |
| iurygregory | vodka is always good | 09:45 |
| dtantsur | maybe we should indeed publish both Debian (smaller) images and Stream (larger)? | 09:46 |
| iurygregory | this would probably solve for upstream | 09:46 |
| timeu | our 200 lenovo nodes would appreciate this :-p | 09:46 |
| timeu | I documented some of the Lenovo issues here btw: https://storyboard.openstack.org/#!/story/2008852 | 09:46 |
| dtantsur | rpittau: ^^ | 09:46 |
| iurygregory | oh nice! | 09:47 |
| timeu | eventually once we have played around more with ironic, I can make a PR for the docs regarding the caveats with the Lenovo servers (we have SR530,SR630,SR650,SD950,SR850, SR670) so almost their entire fleet to test ;-) | 09:49 |
| dtantsur | would be awesome | 09:50 |
| iurygregory | ++ | 09:50 |
| rpittau | mmmm I guess we'll have to tweak a lot the centos8-stream images, they're currently ~200MB bigger (not compressed) | 09:59 |
| * iurygregory brain stopped working | 09:59 | |
| rpittau | timeu: do you happen to know the size limit of the ISO ? | 10:01 |
| iurygregory | I know the iso downstream we generated was 466MB | 10:02 |
| rpittau | so we're probably around 400 MB | 10:03 |
| iurygregory | using wallaby | 10:03 |
| rpittau | dtantsur: I agree publishing both would help, and we already have debian jobs voting, so it doesn't change too much, it's just a matter of adding the publishing job | 10:04 |
| rpittau | and of course in the latest patches the debian jobs are failing :D | 10:06 |
| iurygregory | rpittau, round 3 ? :D | 10:06 |
| rpittau | heh round 3 not finished yet | 10:07 |
| opendevreview | Merged openstack/bifrost stable/wallaby: Use stable/wallaby branch to install dependencies https://review.opendev.org/c/openstack/bifrost/+/806104 | 10:32 |
| jeeva | okay i get that i need a sperate PXE boot VLAN for IRONIC, but what must the "production" VLAN be in correlation to openstack ? | 10:59 |
| jeeva | because kolla-ansible only wants these 3 network settings: | 11:00 |
| jeeva | ironic_dnsmasq_dhcp_range: "192.168.5.100,192.168.5.110" | 11:01 |
| jeeva | ironic_cleaning_network: "public1" | 11:01 |
| jeeva | ironic_dnsmasq_default_gateway: 192.168.5.1 | 11:01 |
| jeeva | like the same network as my mellanox_openstack_vm-to-vm ? | 11:03 |
| dtantsur | mgoddard: ^^ | 11:12 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Allow configuring enabled raid interfaces https://review.opendev.org/c/openstack/bifrost/+/806123 | 11:12 |
| mgoddard | jeeva: what do you mean by a production VLAN? | 11:13 |
| jeeva | trying to wrap my head around creating an ironic setup | 11:24 |
| jeeva | the network part | 11:24 |
| jeeva | it takes about a cleaning network & PXE boot network, but how does the IPMI network comes into play ? | 11:26 |
| mgoddard | jeeva: kolla doesn't care about the IPMI network, the controllers just need to be able to access it | 11:28 |
| jeeva | ok cool | 11:29 |
| jeeva | thanks | 11:29 |
| rpittau | yep, debian is broken | 11:44 |
| jeeva | does the OS controllers need to be exposed on the same PXE-VLAN ? or just the cleaning network | 11:44 |
| dtantsur | iurygregory: do you maybe have a second for https://review.opendev.org/c/openstack/sushy/+/805367 ? | 11:45 |
| dtantsur | apparently, there was a wrong deadline for non-client libraries, and we now have a chance of sneaking more stuff in | 11:45 |
| rpittau | oh cool | 11:45 |
| iurygregory | dtantsur, sure! | 11:45 |
| dtantsur | I've asked to hold https://review.opendev.org/c/openstack/releases/+/806115 until this merges | 11:45 |
| iurygregory | tks dtantsur | 11:51 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Add support for being dhcp relay target https://review.opendev.org/c/openstack/bifrost/+/804482 | 12:00 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: copy bifrost logs to the log directory https://review.opendev.org/c/openstack/bifrost/+/806155 | 12:07 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: collect keystone information https://review.opendev.org/c/openstack/bifrost/+/806156 | 12:07 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: deprecate a separate admin service https://review.opendev.org/c/openstack/bifrost/+/806125 | 12:07 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: deprecate the separate admin service https://review.opendev.org/c/openstack/bifrost/+/806125 | 12:07 |
| opendevreview | Michal Nasiadka proposed openstack/bifrost master: Add support for being dhcp relay target https://review.opendev.org/c/openstack/bifrost/+/804482 | 12:32 |
| opendevreview | Merged openstack/sushy master: Support credentials for VirtualMedia URI https://review.opendev.org/c/openstack/sushy/+/805367 | 12:46 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: copy bifrost logs to the log directory https://review.opendev.org/c/openstack/bifrost/+/806155 | 13:00 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: collect keystone information https://review.opendev.org/c/openstack/bifrost/+/806156 | 13:00 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: deprecate the separate admin service https://review.opendev.org/c/openstack/bifrost/+/806125 | 13:01 |
| viks__ | hi, what is the reason for the below warning in ironic-conductor.log: | 13:09 |
| viks__ | WARNING ironic.common.nova [req-3d717ae0-f11a-4763-acc4-606d0d9bd7aa - - - - -] Could not connect to Nova to send a power notification, please check configuration. An auth plugin is required to determine endpoint URL: MissingAuthPlugin: An auth plugin is required to determine endpoint URL | 13:09 |
| viks__ | i have `[service_catelog]` section set in `ironic.conf` | 13:10 |
| timeu | rpittau: I think it's around 500 MB. At least the rhel train IPA images where around 600 MB which was too big. | 13:22 |
| iurygregory | enrr =( | 13:22 |
| rpittau | mmm | 13:22 |
| timeu | ok I checked the train rhel image is 538MB which didn't work | 13:23 |
| iurygregory | I think it's probably less considering https://paste.opendev.org/show/808347/ | 13:23 |
| timeu | so 538 MB for the rhel one and 338 MB for the ussuri upstream one | 13:25 |
| dtantsur | interesting, I think the RDO upstream images are smaller | 13:26 |
| dtantsur | the master version is clearly under 500M: https://images.rdoproject.org/centos8/master/rdo_trunk/current-tripleo/ | 13:26 |
| dtantsur | even smaller for train: https://images.rdoproject.org/train/rdo_trunk/current-tripleo/ | 13:27 |
| dtantsur | I wonder why RHEL ended up with 500+, it should be more or less the same as RDO | 13:27 |
| dtantsur | viks__: it's a different section, probably [nova] | 13:27 |
| viks__ | dtantsur: ok.. thanks | 13:28 |
| dtantsur | iurygregory: I'm pondering how we can have a two-stage loader for IPA. sounds doable, can be annoying in practice, of course | 13:28 |
| iurygregory | dtantsur, yeah it would be interesting | 13:29 |
| dtantsur | timeu: this initramfs size problem, was it with iPXE or virtual media? | 13:38 |
| timeu | dtantsur: only with virtual media. iPXE worked fine with the bigger IPA image | 14:05 |
| dtantsur | ack, this is useful | 14:05 |
| timeu | both with UEFI btw | 14:06 |
| iurygregory | good to know :D | 14:06 |
| timeu | maybe the virtual media slot of the Lenovo machines silently truncates the iso | 14:07 |
| timeu | but in the Lenovo BMC web interface it detects the correct size of the iso | 14:07 |
| iurygregory | redfish + iPXE? or did you use the xclarity driver from ironic? | 14:07 |
| iurygregory | or even ipmi =) | 14:08 |
| timeu | virtual media was with redfish | 14:08 |
| timeu | iPXE I tried with both I think | 14:08 |
| timeu | ipmi and redfish | 14:08 |
| iurygregory | nice! | 14:09 |
| dtantsur | okay, at least iPXE works | 14:09 |
| * iurygregory now can at least provide some good news in the bz \o/ | 14:10 | |
| timeu | is the bz public ? | 14:10 |
| iurygregory | nope =( | 14:11 |
| timeu | so redfish and ipmi + iPXE definately works with Lenovo machines. Virtual media has the image size limitation | 14:12 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: copy bifrost logs to the log directory https://review.opendev.org/c/openstack/bifrost/+/806155 | 14:13 |
| iurygregory | my brain started working fine again after reading this hehe :D | 14:13 |
| timeu | sorry for the confusing sentence :-p | 14:14 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: CI: collect keystone information https://review.opendev.org/c/openstack/bifrost/+/806156 | 14:15 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Keystone: deprecate the separate admin service https://review.opendev.org/c/openstack/bifrost/+/806125 | 14:15 |
| dtantsur | too many rebases.. | 14:15 |
| rpittau | seems like the debian image is broken on DIB, some repo issue, hopefullt this https://review.opendev.org/c/openstack/diskimage-builder/+/806188 should fix it | 14:17 |
| dtantsur | sigh | 14:17 |
| rpittau | they dont test debian stable in the CI by default | 14:18 |
| rpittau | so anything can happen \o/ | 14:18 |
| TheJulia | timeu: say if a storyboard item was noting that there are not usable virtual media slots due to a size limitation inside the BMC, then I woudl happilly send that over to some folks I know at lenovo so maybe it could get put on their engineering backlog to fix...e ventually | 14:20 |
| TheJulia | eventually | 14:20 |
| timeu | TheJulia: sorry I am not sure I can follow. The Lenovo nodes have different kinds of virtual media slots. If you use an old firmware the 2 (RODC1 and RODC2) are listed first and they don't support mounting via HTTP(s) and thus virtual media deployment via ironic fails. Upgrading the firmware changes the sort order and the EXTx slots come first. Those support mounting ISOs | 14:27 |
| timeu | via http(s), however they seem to only support mounting ISOs up to ~ 400 MB (there is no error, but deployment fails). Should I update the storyboard to make this clearer ? | 14:27 |
| TheJulia | timeu: Yes, update the storyboard and I'll send it over to some folks at Lenovo | 14:27 |
| dtantsur | timeu: unrelated, but do RODC* advertise supporting CD type? otherwise I'm not sure why ironic picks them. | 14:28 |
| TheJulia | specifically the size limitation being key :) | 14:28 |
| TheJulia | dtantsur: they may support access via something like CIFS | 14:28 |
| timeu | The MediaTypes for all virtual media slots seems to be the same ("CD, DVD, Floppy, USBStick"). There is actually an issue on the Lenovo github org: https://github.com/lenovo/python-redfish-lenovo/issues/21 where I brought this up | 14:29 |
| dtantsur | ahhh. maybe we should check TranfserProtocol as well | 14:29 |
| dtantsur | "TransferProtocolType" and "TransferMethod" properties are not supported yet, but per your voices, development team plan to support them on future release". sigh. | 14:30 |
| TheJulia | yeaaaah | 14:30 |
| dtantsur | maybe we need an explicit exception for vendor == 'lenovo' and 'RODC1 in name? | 14:30 |
| dtantsur | 'RODC' in name | 14:30 |
| iurygregory | so we can only choose 'RODC' devices? | 14:32 |
| iurygregory | when using Lenovo? | 14:32 |
| timeu | RODC won't work | 14:34 |
| timeu | so we need to exclude them | 14:34 |
| timeu | for Lenovo | 14:34 |
| iurygregory | oh ok =) | 14:34 |
| iurygregory | 10 slots for vmedia (I'm wondering why so many hehehe) | 14:34 |
| TheJulia | Likely different customer use cases | 14:35 |
| timeu | TheJulia I updated the storyboard ("Update2") and tried to make it a bit clearer. Is this ok ? | 14:36 |
| TheJulia | timeu: much appreciated | 14:37 |
| dtantsur | TheJulia: maybe you can check my memory: do I recall it right that keystone no longer requires a separate admin API on port 35357? (re https://review.opendev.org/c/openstack/bifrost/+/806125) | 14:52 |
| TheJulia | dtantsur: I honestly don't remember | 14:53 |
| dtantsur | ack, no worries | 14:53 |
| dtantsur | I can throw a quick patch disabling it to see what breaks | 14:53 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: DNM remove the keystone admin endpont https://review.opendev.org/c/openstack/bifrost/+/806207 | 14:54 |
| opendevreview | Felix Huettner proposed openstack/python-ironicclient master: Validate path when passing a config-drive https://review.opendev.org/c/openstack/python-ironicclient/+/806209 | 15:31 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Use native Centos 8 Stream for dib images https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/805957 | 15:51 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add clear foreign config to idrac-redfish RAID https://review.opendev.org/c/openstack/ironic/+/804848 | 15:55 |
| rpittau | so I tested it locally and https://review.opendev.org/c/openstack/diskimage-builder/+/806188 fixes debian-minimal in ipa-builder (and dib in general) | 16:00 |
| rpittau | see ya tomorrow! o/ | 16:02 |
| *** rpittau is now known as rpittau|afk | 16:02 | |
| TheJulia | iurygregory: timeu: w/r/t lenovo gear, the word I'm getting back is that we do need to exclude RODC, it sounds like they are for updating the BMC.... | 16:26 |
| iurygregory | yeah, but the limitation about the size happens in the other vmedia slots (I think) | 16:27 |
| TheJulia | iurygregory: that also sounds like a known issue | 16:29 |
| iurygregory | :D | 16:29 |
| TheJulia | RemoteX are apparently web browser injected | 16:30 |
| TheJulia | trying to clarify size stuff | 16:31 |
| iurygregory | and probably there is no workaround for the size (I think) | 16:32 |
| TheJulia | stripping every ounce of firmware out might do it | 16:35 |
| TheJulia | but at what cost to the telecoms of the wordl | 16:35 |
| TheJulia | wordl | 16:35 |
| TheJulia | gah | 16:35 |
| TheJulia | world | 16:35 |
| iurygregory | I know they were using BMC 6.00 / UEFI 2.90 | 16:37 |
| iurygregory | I saw 6.60 on the lenovo page, so I asked them to update and try again, but the problem was the same | 16:38 |
| TheJulia | iurygregory: so it seems like it is an outstanding question on the size limitation | 16:43 |
| dtantsur | I tried imagining how a 2-stage IPA would work.. but that's not trivial. | 16:43 |
| dtantsur | IPA itself is not the largest thing we ship, splitting it away doesn't bring us much. | 16:43 |
| dtantsur | We can of course revive the idea of a container... | 16:43 |
| dtantsur | Pretty much the same thing I did for CoreOS IPA, but without CoreOS :) | 16:43 |
| TheJulia | the conundrum is the firmware assets really | 16:47 |
| TheJulia | that stuff is already compressed, it is huge | 16:47 |
| dtantsur | there is enough heavy stuff without them too | 16:47 |
| TheJulia | and generally not really needed except in the fewer less common cases, but yeah | 16:48 |
| TheJulia | Yeah, some stuff that we can't cleanly rip out because of odd dependency chains that seem nonsensical | 16:48 |
| dtantsur | if we use a container with IPA, we can do without Python even | 16:48 |
| dtantsur | (in the 1st stage, I mean) | 16:48 |
| TheJulia | yeah | 16:48 |
| dtantsur | this is the container we build: https://github.com/openshift/ironic-agent-image | 16:49 |
| dtantsur | the last image I built was around 450 MiB | 16:50 |
| opendevreview | Dmitry Tantsur proposed openstack/sushy-tools master: [WIP] Support uploading certificates for virtual media https://review.opendev.org/c/openstack/sushy-tools/+/806253 | 17:19 |
| dtantsur | see you tomorrow folks | 17:19 |
| sam_z1 | hey i noticed the list of power states contains "power_off"/"reboot" but also "soft_power_off"/"soft_reboot"-- what's the difference between the two? | 19:18 |
| *** sshnaidm is now known as sshnaidm|afk | 19:32 | |
| TheJulia | sam_z1: power off/reboot/power on are absolute actions | 20:03 |
| TheJulia | soft actions are "hi, will you please kindly power off or power on or reboot | 20:03 |
| sam_z1 | oh so as a node power state "soft power off" would be "someone asked me to shut myself off"? | 20:09 |
| TheJulia | sam_z1: exactly | 20:13 |
| sam_z1 | cool cool! | 20:15 |
| sam_z1 | ty! | 20:15 |
| janders | good morning Ironic o/ | 22:41 |
| janders | back to the living after pfizer-1 | 22:49 |
| stevebaker | janders: welcome back | 23:11 |
| janders | thank you stevebaker :) | 23:12 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!