| opendevreview | Verification of a change to openstack/ironic master failed: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 01:20 |
|---|---|---|
| *** pmannidi is now known as pmannidi|brb | 04:00 | |
| *** pmannidi|brb is now known as pmannidi | 04:19 | |
| opendevreview | Merged openstack/ironic master: require_exclusive_lock: log traceback that lead to an error https://review.opendev.org/c/openstack/ironic/+/810615 | 05:31 |
| opendevreview | Merged openstack/ironic master: Refactor: move base_iso handling from create_boot_iso https://review.opendev.org/c/openstack/ironic/+/810076 | 05:33 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 05:47 |
| iurygregory | good morning janders and Ironic o/ | 06:10 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 06:22 |
| rpittau | good morning ironic! o/ | 06:59 |
| iurygregory | morning rpittau o/ | 07:15 |
| rpittau | hey iurygregory :) | 07:15 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/xena: Fix iDRAC configuration mold docs https://review.opendev.org/c/openstack/ironic/+/811495 | 07:29 |
| opendevreview | Aija Jauntēva proposed openstack/ironic stable/wallaby: Add iDRAC configuration mold docs https://review.opendev.org/c/openstack/ironic/+/811496 | 07:30 |
| *** pmannidi is now known as pmannidi|AFK | 07:38 | |
| janders | good morning iurygregory rpittau and Ironic o/ | 07:44 |
| rpittau | hey janders :) | 07:44 |
| iurygregory | janders, o/ | 07:44 |
| dtantsur | TheJulia: ack re molteniron | 07:48 |
| dtantsur | good morning ironic | 07:48 |
| rpittau | hey dtantsur :) | 07:48 |
| iurygregory | morning dtantsur | 07:48 |
| dtantsur | ayoung: welcome! | 07:48 |
| iurygregory | anything very urgent we need to take a look? | 07:49 |
| * dtantsur no idea | 07:49 | |
| janders | hey dtantsur o/ | 07:50 |
| rpittau | the issue wit the CI should be fixed, so maybe some rechecks are in order ? | 07:52 |
| iurygregory | rpittau, ack | 07:52 |
| deke | Hi | 07:59 |
| deke | What is the current status of Xilinx drivers for Ironic? | 07:59 |
| iurygregory | humm I don't recall any Xilinx driver in Ironic | 08:04 |
| deke | sorry I actually meant cyborg | 08:05 |
| deke | wrong channel | 08:05 |
| iurygregory | oh no worries deke =) | 08:05 |
| rpittau | still something to do with metal :P | 08:06 |
| deke | close enough haha | 08:06 |
| deke | while I'm here though, quick question | 08:07 |
| iurygregory | go ahead =) | 08:07 |
| deke | has there been any discussion about integration between ironic and MAAS? | 08:07 |
| dtantsur | deke: I don't think anyone has ever expressed any interest in that | 08:08 |
| iurygregory | not that I'm aware | 08:08 |
| dtantsur | I given the past rhetoric of Canonical.. I doubt it | 08:08 |
| deke | If we deploy Openstack with juju on MAAS, it seems silly to also then deploy ironic on top of that when MAAS is already there doing baremetal provisioning. | 08:09 |
| deke | just a thought | 08:09 |
| deke | what rhetoric do you mean? | 08:09 |
| dtantsur | deke: there have been discouraging public statements, let me leave that at this | 08:09 |
| dtantsur | if you already have bare metal provisioning, why you need ironic? | 08:10 |
| deke | well, for end users of the cloud who don't have access to MAAS | 08:10 |
| deke | Is the relationship between canonical and openstack rocky? I was unaware of this | 08:11 |
| dtantsur | I wouldn't put it like that. But there is no cooperation between the MAAS team and Ironic, as far as I'm aware. | 08:11 |
| deke | Say I have an end user who needs a dedicated baremetal server orchestrated via the horizon dashboard. Right now, my only option is to deploy ironic and enable that for them | 08:11 |
| dtantsur | deke: what you need in theory, is a Nova driver for MAAS, not Ironic integration | 08:12 |
| deke | yea that sounds more accurate | 08:12 |
| rpittau | or install ironic with juju | 08:12 |
| * dtantsur -> doctor appt, bbl | 08:12 | |
| rpittau | and keep maas and ironic separated | 08:12 |
| deke | do they play nice? | 08:12 |
| deke | if I have a baremetal machine setup with MAAS and then I use ironic to deploy something else, will MAAS pick it back up afterward no problem? I have not tested this | 08:13 |
| rpittau | never tried it :) | 08:13 |
| rpittau | but I'm pretty sure you will find some guide on that | 08:13 |
| deke | interesting | 08:14 |
| deke | thanks guys! | 08:14 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/x/sushy-oem-idrac/+/811611 | 08:27 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/x/sushy-oem-idrac/+/811612 | 08:27 |
| opendevreview | Riccardo Pittau proposed x/sushy-oem-idrac master: Add Python3 yoga unit tests https://review.opendev.org/c/x/sushy-oem-idrac/+/811613 | 08:33 |
| opendevreview | Aija Jauntēva proposed x/sushy-oem-idrac master: Add Python3 Yoga unit tests https://review.opendev.org/c/x/sushy-oem-idrac/+/811614 | 08:34 |
| ajya | uff | 08:34 |
| rpittau | d'oh | 08:34 |
| rpittau | I will abandon mine, it's ok | 08:35 |
| ajya | ok, thanks, was a bit too slow :) now just don't abandon mine | 08:35 |
| ajya | (I'll do the same for python-dracclient just in case) | 08:36 |
| rpittau | ok! :) | 08:36 |
| mgoddard | dtantsur: hi. Regarding https://storyboard.openstack.org/#!/story/1737947, we generally use the dnsmasq filter these days. Unsure if the iptables filter is still affected | 08:47 |
| opendevreview | Merged x/sushy-oem-idrac stable/xena: Update .gitreview for stable/xena https://review.opendev.org/c/x/sushy-oem-idrac/+/811611 | 08:58 |
| opendevreview | Merged x/sushy-oem-idrac stable/xena: Update TOX_CONSTRAINTS_FILE for stable/xena https://review.opendev.org/c/x/sushy-oem-idrac/+/811612 | 09:00 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 09:19 |
| dtantsur | mgoddard: so, does it work okay with the dnsmasq filter? | 10:00 |
| mgoddard | dtantsur: yes, seems to AFAIK | 10:01 |
| dtantsur | thanks! | 10:01 |
| dtantsur | deke: you have to be aware of potential DHCP conflicts | 10:01 |
| opendevreview | Merged openstack/ironic master: PXE: avoid trailing dots in exception messages https://review.opendev.org/c/openstack/ironic/+/810828 | 10:30 |
| opendevreview | Derek Higgins proposed openstack/ironic master: Exit ipxe script if enable_netboot_fallback failed https://review.opendev.org/c/openstack/ironic/+/811710 | 10:42 |
| opendevreview | Derek Higgins proposed openstack/ironic master: Exit ipxe script if enable_netboot_fallback failed https://review.opendev.org/c/openstack/ironic/+/811710 | 10:48 |
| opendevreview | Verification of a change to openstack/ironic master failed: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 10:52 |
| janders | see you tomorrow Ironic o/ | 12:01 |
| dtantsur | wow, what a thunderstorm we have here | 13:02 |
| ayoung | Can the CLI output be made to show a nested column value? LIke, to get the ipmi address of a node, I need driver_info.ipmi_address. -c driver_info works, but it gives the whole dictionary. I know I can pull a value out of there using jq, but was wondering if there is some format that allows the CLI to do that in a shorter format? | 13:29 |
| ayoung | I mean this is not that bad | 13:30 |
| ayoung | for UUID in `openstack baremetal node list --provision-state "clean failed" -f value -c UUID` ; do openstack baremetal node show $UUID -f json | jq -r '.driver_info | .ipmi_address' ;done | 13:30 |
| dtantsur | this looks about right | 13:35 |
| TheJulia | dtantsur: Well, fall storms are a thing there right? | 13:35 |
| dtantsur | TheJulia: pretty much, although usually without thunder | 13:35 |
| dtantsur | also good morning TheJulia | 13:35 |
| TheJulia | ayoung: pathces welcome? | 13:35 |
| iurygregory | good morning TheJulia =) | 13:37 |
| TheJulia | dtantsur: well handled w/r/t the maas questions | 13:39 |
| dtantsur | :) | 13:39 |
| TheJulia | Pity there were no keynote video links | 13:39 |
| TheJulia | but I think the worst actually didn't get posted by OIF staff | 13:40 |
| dtantsur | heh, yeah, I didn't want to start the whole archaeology | 13:41 |
| TheJulia | yeah | 13:41 |
| TheJulia | a waste of energy really | 13:41 |
| ayoung | TheJulia, I'd actually prefer there not be a way to do it, so I can feel justified relying on my preloaded jq know-how. | 13:41 |
| ayoung | is there a write up on what happens during the cleaning process? How to debug etc? | 13:44 |
| TheJulia | yeah, quick searching did not yield a video | 13:44 |
| TheJulia | ayoung: https://docs.openstack.org/ironic/latest/admin/cleaning.html by default erase_devices or erase_devices_metadata based upon what your settings are | 13:45 |
| TheJulia | at least, a stock deploy | 13:45 |
| ayoung | to be clear, we get an automated if we do: openstack baremetal node maintenance unset ${i} | 13:46 |
| ayoung | openstack baremetal node manage ${i} | 13:46 |
| ayoung | openstack baremetal node provide ${i} | 13:46 |
| ayoung | right? | 13:46 |
| TheJulia | are you in maintenance mode right now? | 13:47 |
| TheJulia | the latter to commands, yes. Between provide and the node becoming available, automated cleaning is run | 13:48 |
| TheJulia | dtantsur: https://storyboard.openstack.org/#!/story/list?status=active&project_group_id=75&tags=rfe <-- down to 195 | 13:48 |
| dtantsur | \o/ | 13:48 |
| dtantsur | very nice | 13:48 |
| ayoung | so if I do a provide (from manage) I ge an automated clean. Is there a way to list the steps that are (or will be) performed? | 13:48 |
| TheJulia | ayoung: it is generated dynamically from the hardware managers loaded in IPA. The in-built default includes erase_devices and erase_devices_metadata and global configuration knobs in your conductor(s) ironic.conf files | 13:50 |
| TheJulia | which allows you to tune if they run at all or which one to run | 13:50 |
| TheJulia | obviously scrubbing disks can take a while | 13:50 |
| TheJulia | metadata is easy | 13:50 |
| ayoung | So there would be no way to know until it is actually run, as it needs to PXE boot the "clean me" Kernel image, which loads IPA? | 13:51 |
| dtantsur | yep | 13:53 |
| TheJulia | yup | 13:54 |
| TheJulia | there have been various proposals to make an api to figure it out | 13:55 |
| opendevreview | Merged openstack/ironic master: Use an ImageCache for provided boot/deploy ISO images https://review.opendev.org/c/openstack/ironic/+/810241 | 13:55 |
| TheJulia | but... given the hardware manager nature, and the demands of "i want it absolutes right now", we reached a conflict over operational consistency and continuity of workloads because the proposals were basically ending up in kill the running workload if queried and boot the ramdisk to find out because caching was unacceptable. | 13:57 |
| ayoung | API is a step beyond where I need. I just need to get smart on what would be done right now, if I have full access to everything | 13:59 |
| ayoung | like, can I restart a clean from a 'clean wait' state? | 13:59 |
| ayoung | The requested action "manage" can not be performed on node "5411e7e8-8113-42d6-a966-8cacd1554039" while it is in state "clean wait". (HTTP 400) | 13:59 |
| dtantsur | ayoung: you need to "abort" it first | 13:59 |
| TheJulia | and you may be doing a full erase_deviecs | 14:00 |
| ayoung | yeah but how would I know? | 14:00 |
| TheJulia | if you look at the entire node detail you should see clean_steps | 14:00 |
| ayoung | I think I can attach to the machine via ipmi SOL | 14:00 |
| dtantsur | ayoung: the current step is in the 'clean_step' field | 14:01 |
| ayoung | driver_internal_info | 14:02 |
| ayoung | openstack baremetal node show 5411e7e8-8113-42d6-a966-8cacd1554039 -f json | jq '.driver_internal_info | .agent_cached_clean_steps' | 14:03 |
| ayoung | that is hte "plan" I take it? | 14:03 |
| dtantsur | ayoung: I think during the process you have just "clean_steps" which are the steps that will be run | 14:04 |
| ayoung | "clean_step": {}, | 14:05 |
| dtantsur | this means no step is running now | 14:05 |
| ayoung | so I assume that this means it is waiting for PXE to finish | 14:06 |
| ayoung | And a call back from IPA on the node | 14:06 |
| dtantsur | correct | 14:06 |
| ayoung | this is good stuff | 14:07 |
| ayoung | OK let me see if I can get something off the node | 14:07 |
| ayoung | SOL is just hanging. How do I abort the process to kick it off again? | 14:07 |
| dtantsur | ayoung: baremetal node abort <node> | 14:08 |
| ayoung | k | 14:08 |
| dtantsur | you may have to configure where the serial console goes via the kernel params | 14:08 |
| dtantsur | I don't remember if we have docs for that... | 14:08 |
| ayoung | openstack baremetal node show 5411e7e8-8113-42d6-a966-8cacd1554039 -c provision_state | 14:08 |
| ayoung | +-----------------+--------------+ | 14:08 |
| ayoung | | Field | Value | | 14:08 |
| ayoung | +-----------------+--------------+ | 14:08 |
| ayoung | | provision_state | clean failed | | 14:08 |
| ayoung | +-----------------+--------------+ | 14:08 |
| dtantsur | you may also try runtime SSH configuration via something like https://docs.openstack.org/diskimage-builder/latest/elements/dynamic-login/README.html | 14:08 |
| ayoung | koolio | 14:08 |
| ayoung | so I can do the three above operations to kick it off again... | 14:09 |
| TheJulia | ayoung: yes, just it woudl likely help to do what dmitry notes with your ipa image and/or look at the logs uploaded to the conductor | 14:16 |
| TheJulia | the logs *should* yield why | 14:16 |
| dtantsur | if it's a PXE timeout, there will be no logs uploaded | 14:17 |
| TheJulia | ++ | 14:17 |
| TheJulia | and things like SOL would be useful then | 14:17 |
| ayoung | OK...logs. I have the server side logs, but should there be an ipa log on the node? I can get in via SOL | 14:32 |
| ayoung | debian instance is running, ipa is running | 14:32 |
| TheJulia | ayoung: so if uploaded, /var/log/ironic/deploy_logs I think | 14:32 |
| TheJulia | if it never heartbeated and just timed out | 14:32 |
| ayoung | on the server? | 14:32 |
| TheJulia | Like...after 10 minutes, then that would mean network connectivity issues *or* the ramdisk didn't boot | 14:33 |
| ayoung | NO ip address on the node | 14:33 |
| TheJulia | ayoung: on the conductor, on the ipa image, it gets written out to the system logger so journald or whatever file for the service | 14:33 |
| TheJulia | ayoung: \o/ does your OS support your nics? | 14:33 |
| TheJulia | how was the ramdisk built is another question | 14:34 |
| dtantsur | ayoung: how did you build your IPA image? | 14:34 |
| dtantsur | I think debian does not DHCP by default, I needed to add the dhcp-all-interfaces element (or simple-init as an alternative) | 14:34 |
| ayoung | running dhclient gets me an ip address | 14:35 |
| dtantsur | yep, it sounds like nothing DHCPs. on RH systems NetworkManager tries to do something (sometimes wrongly), on Debian there is nothing by default. | 14:35 |
| ayoung | and in conductor 2021-09-29 07:35:09.402 6 INFO ironic.conductor.task_manager [req-1af9106c-9850-4ab2-93a3-811f758b3b94 - - - - -] Node 5411e7e8-8113-42d6-a966-8cacd1554039 moved to provision state "available" from state "cleaning"; target provision state is "None" | 14:35 |
| ayoung | I did not build the IPA image. I can ask the person who did | 14:36 |
| dtantsur | ++ | 14:36 |
| TheJulia | ayoung: that seems like successful cleaning | 14:36 |
| TheJulia | conductor log wise | 14:36 |
| dtantsur | yeah, probably DHCP was the issue | 14:36 |
| ayoung | And now SOL is locked, guessing it is powered down | 14:37 |
| dtantsur | should be, unless you disabled it explicitly | 14:37 |
| ayoung | | 5411e7e8-8113-42d6-a966-8cacd1554039 | jade05-r097 | None | power off | available | 14:38 |
| ayoung | OK, these are the kind of problems we like to have | 14:38 |
| dtantsur | okay, I need to leave now.. and I don't feel like going outside at this moment :( | 14:56 |
| TheJulia | dtantsur: :( | 14:59 |
| dtantsur | okay, the thunderstorm seems to relax a bit, see you o/ | 15:12 |
| rpittau | bye dtantsur | 15:12 |
| rpittau | good night! o/ | 16:12 |
| TheJulia | o/ | 16:13 |
| stevebaker | morning | 19:49 |
| TheJulia | good morning! | 19:51 |
| TheJulia | stevebaker: are you available tomorrow around this time. Well, starting in like 23 hours 38 minutes ? | 19:52 |
| stevebaker | TheJulia: I certainly am | 19:56 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Yoga: Change default boot mode to uefi https://review.opendev.org/c/openstack/ironic/+/810249 | 20:03 |
| TheJulia | stevebaker: excellent, I'll send a meeting invite! | 20:03 |
| TheJulia | and ^^^ was ouchy | 20:03 |
| TheJulia | the patch that was | 20:03 |
| TheJulia | sent! | 20:05 |
| TheJulia | stevebaker: fwiw, your not on downstream irc :) | 20:15 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Remove debian packages file for devstack https://review.opendev.org/c/openstack/ironic/+/810314 | 20:52 |
| TheJulia | just a rebase since the main bindep file should work | 20:52 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Yoga: Change default boot mode to uefi https://review.opendev.org/c/openstack/ironic/+/810249 | 21:20 |
| opendevreview | Julia Kreger proposed openstack/ironic master: DNM/WIP - Add FIPS jobs https://review.opendev.org/c/openstack/ironic/+/797739 | 21:43 |
| * TheJulia goes and exercises | 21:47 | |
| *** pmannidi|AFK is now known as pmannidi | 23:58 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!