Wednesday, 2022-08-17

* TheJulia goes and tries to figure out dinner00:14
TheJuliaJayF: https://d73732804f9d700e903b-8e240bc4e64280e0dda3f4c38c5a562b.ssl.cf1.rackcdn.com/835917/13/check/ironic-standalone-redfish/4804a02/controller/logs/ironic-bm-logs/node-2_no_ansi_2022-08-17-00%3A00%3A59_log.txt may amuse you02:50
TheJuliadtantsur: thoughts on passing webserver validation down, since the stock template has no concept02:50
TheJuliaWe’ve got the global switch, we could double stack it easily I think02:59
TheJuliaOr we might be able to find a way to get it to just accept it….02:59
TheJuliaDunno02:59
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Fix releasenotes before release 21.0  https://review.opendev.org/c/openstack/ironic/+/85338703:39
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Ironic Release 21.0  https://review.opendev.org/c/openstack/ironic/+/85338803:40
stevebaker[m]arne_wiebalck: hey I'm not seeing any new SIG video in the dropbox03:46
ajyaGood morning Ironic06:31
ajyaTheJulia: I do more or less06:31
rpittaugood morning ironic! o/07:27
adam-rozmangood morning ironic!07:28
* rpittau E_TOO_MANY_MESSAGES07:29
jandersgood morning ajya rpittau adam-rozman and Ironic o/07:47
rpittauhey janders :)07:48
adam-rozmanhey07:48
kubajjgood morning everybody!07:51
ajyahey janders , rpittau, adam-rozman, kubajj 08:08
rpittauhey ajya :)08:13
dtantsurTheJulia: sorry, I might me missing the context of the question, cannot pick it up from the scrollback.09:16
cboucharTheJulia, Jayf: cbouchar is in Yoga.09:55
iurygregorygood morning Ironic10:18
opendevreviewDmitry Tantsur proposed openstack/sushy stable/yoga: Do not send compression as acceptable encoding  https://review.opendev.org/c/openstack/sushy/+/85343110:35
opendevreviewDmitry Tantsur proposed openstack/sushy stable/xena: Do not send compression as acceptable encoding  https://review.opendev.org/c/openstack/sushy/+/85343210:35
waleedmhi guys, now the default mode in itrospection is eufi, I need to do the intrpspection with bios, what should I change? i changed the default_mode_mode in ironic.conf to bios, but when i start the introspection and it got the IP address I get connection reset on the ilo for the node 10:38
waleedmanyone could help with that ?10:39
iurygregorywaleedm, this would probably help https://docs.openstack.org/ironic/latest/install/advanced.html#boot-mode-support10:40
iurygregorywhen you create the node you can specify the boot-mode it should use (if not set it will use the default from ironic)10:41
waleedmyeah I already set it to bios also10:42
opendevreviewMerged openstack/ironic master: Fix releasenotes before release 21.0  https://review.opendev.org/c/openstack/ironic/+/85338710:45
kamlesh6808cHey Iurygregory and Ironic!10:49
kamlesh6808cIurygregory : Would you please help to second review for https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/84160110:50
iurygregorykamlesh6808c, will do later today o/10:51
kamlesh6808cIurygregory : ok Thanks !10:51
opendevreviewSteven Relf proposed openstack/networking-generic-switch master: Implements basic functionality for use with Pluribus switches.  https://review.opendev.org/c/openstack/networking-generic-switch/+/85342710:58
opendevreviewSteven Relf proposed openstack/networking-generic-switch master: Implements basic functionality for use with Pluribus switches.  https://review.opendev.org/c/openstack/networking-generic-switch/+/85342711:16
kubajjdtantsur: TheJulia: I discussed the volume_name-inspector problem with Arne and what neither of us understood is what is happening with the root_disk in inspector. Is that the problem?11:17
opendevreviewVanou Ishii proposed openstack/ironic stable/yoga: Fix iRMC driver to use certification file in HTTPS  https://review.opendev.org/c/openstack/ironic/+/85279711:52
opendevreviewMerged openstack/ironic master: Ironic Release 21.0  https://review.opendev.org/c/openstack/ironic/+/85338812:04
opendevreviewMerged openstack/ironic master: Do not reboot into nowhere after BIOS settings with fast-track  https://review.opendev.org/c/openstack/ironic/+/85317712:05
opendevreviewDmitry Tantsur proposed openstack/ironic bugfix/20.2: Do not reboot into nowhere after BIOS settings with fast-track  https://review.opendev.org/c/openstack/ironic/+/85343412:09
opendevreviewDmitry Tantsur proposed openstack/ironic stable/yoga: Do not reboot into nowhere after BIOS settings with fast-track  https://review.opendev.org/c/openstack/ironic/+/85343512:09
opendevreviewMerged openstack/ironic-tempest-plugin master: Add iDRAC RAID cleaning steps tests  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/84160112:19
iurygregorydtantsur, going to update the ironic release now o/12:30
dtantsur++12:33
*** matt__ is now known as matfechner12:45
kubajjBtw, is there any problem with ironic-inspector-grenade?12:46
opendevreviewMerged openstack/ironic-python-agent master: Improve function list_block_devices_check_skip_list  https://review.opendev.org/c/openstack/ironic-python-agent/+/85328412:47
opendevreviewSteven Relf proposed openstack/networking-generic-switch master: Implements basic functionality for use with Pluribus switches.  https://review.opendev.org/c/openstack/networking-generic-switch/+/85342713:03
opendevreviewMerged openstack/networking-baremetal master: Add netconf-openconfig device driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83532413:13
iurygregoryrpittau, dtantsur when you have time https://review.opendev.org/c/openstack/releases/+/853390 =D13:19
rpittauiurygregory: lgtm thanks!13:20
iurygregoryrpittau, inspector also needs a +1 https://review.opendev.org/c/openstack/releases/+/853386 (I've updated to 11.0 since we dropped py36/3713:22
rpittaudone13:22
iurygregoryty13:23
rpiosoGood morning, ironic13:44
rpiosoarne_wiebalck: Is a recording of the July Bare Metal SIG meeting, ""Bare Metal Kubernetes at G-Research", available?13:46
iurygregoryrpioso, afaik is not ready yet13:47
rpiosoiurygregory: Thank you13:47
iurygregoryyw13:48
opendevreviewHarald Jensås proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC Test Concept  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/84242714:20
TheJuliadtantsur: so the tl;dr regarding kickstart, is to test it with any ease and not huge images being stored forever, involves the use of stock upstream images. Well, those images don't know about our CI job SSL certs, and the curl calls don't presently have any way of "don't verify the cert"14:24
TheJuliaAnd I'm not sure there is a way to explicitly really inject a preferred cert into the ramdisk upon start by default, so I'm wondering if we just extend the ssl cert verification flag down to the anaconda driver to add a -k flag to the curl command lines if appropriately set14:26
opendevreviewMerged openstack/networking-baremetal master: Add LACP support to Netconf OpenConfig driver  https://review.opendev.org/c/openstack/networking-baremetal/+/83537814:26
TheJuliathat way we can have a job to test it with the one off setting, and explicitly note the requirements of use around x509 certificate validity14:26
ftarasenkoHi Ironic! Is there any option to run import_configuration for idrac-redfish with automated cleaning? If yes, can you point me how to configure import_configuration_location for every node?14:37
iurygregoryTheJulia, I would be ok with this approach to test the anaconda driver in CI14:38
TheJuliaftarasenko: afaik, it was intended as purely a manual cleaning operation14:39
TheJuliaiurygregory: ack, I think it is a... weird and frustrating thing at the same time, but FQDNs and real certs are important()14:39
iurygregoryTheJulia, yeah it's frustrating indeed, but having a coverage would be a big win in general so we would catch problems earlier in some cases  14:41
TheJulia++14:41
ftarasenkoTheJulia: Tnx! I have request to set server's BIOS settings to predefined values after each customer, who can manually change some values, so it's interesting to run this operation automatically.  14:41
dtantsurftarasenko: I guess providing the configuration location may be harder14:46
dtantsurTheJulia: IPA allows passing a CA via virtual media, I think that's it. We use insecure for all our jobs and bifrost.14:47
dtantsurso I don't see huge issues with using -k in anaconda14:47
ftarasenkodtantsur: I think it's possible to use node's name or serial number as variable for configuration location. But for now I'd go to figure out why it fails to import previously exported config with SYS055: Import of Server Configuration Profile operation completed with errors.) 14:50
dtantsur"Operation failed successfully" :)14:51
JayFTheJulia: was that just a bad ks file? re: the log failure from last night15:00
JayFrpioso: yes, on G-R youtube15:00
JayFrpioso: lemme find it for you15:00
TheJuliadtantsur: ack, then I think this is reasonable to do. patch will be up in a few minutes15:01
TheJuliaJayF: basically the anaconda driver failed because it couldn't heartbeat as the ssl cert didn't fail.15:01
JayFrpioso: actually no, we don't have it posted, I thought we did but it was something else :) 15:01
TheJuliaerr15:01
TheJuliadidn't validate to be more precise15:01
TheJuliabut otherwise, it 'almost worked'15:01
JayFsomeone go say something nice to that CA cert15:01
JayFit needs validation15:01
TheJulia++15:01
TheJulia"Are X.509 certificates actually cats?"15:02
dtantsur:D15:03
dtantsurafter spending this morning fighting with httpd, I have a certain aversion to TLS :)15:03
opendevreviewJakub Jelinek proposed openstack/ironic-python-agent master: Create RAIDs with volume name  https://review.opendev.org/c/openstack/ironic-python-agent/+/85318215:11
arne_wiebalckrpioso: not yet, we're working on ... actually stevebaker[m] is doing all the work, I am just saying it :)15:21
opendevreviewJulia Kreger proposed openstack/ironic master: Add kickstart template 'url' option  https://review.opendev.org/c/openstack/ironic/+/85336815:35
opendevreviewJulia Kreger proposed openstack/ironic master: CI: anaconda: permit tls certificate validation bypass  https://review.opendev.org/c/openstack/ironic/+/85351015:35
TheJuliawhoops, pep815:40
rpiosoJayF, arne_wiebalck, stevebaker[m]: Thank you! I'm looking forward to it.15:45
rpittaubye everyone, see you tomorrow! o/15:54
opendevreviewJulia Kreger proposed openstack/ironic master: CI: anaconda: permit tls certificate validation bypass  https://review.opendev.org/c/openstack/ironic/+/85351015:55
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Changes to support Anaconda CI jobs  https://review.opendev.org/c/openstack/ironic/+/84958715:55
opendevreviewMerged openstack/ironic stable/yoga: Do not reboot into nowhere after BIOS settings with fast-track  https://review.opendev.org/c/openstack/ironic/+/85343515:59
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Initial tempest test idea anaconda deploy  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/83591716:00
arne_wiebalckbye everyone o/16:15
opendevreviewJulia Kreger proposed openstack/ironic master: Fix ilo boot interface order  https://review.opendev.org/c/openstack/ironic/+/85103916:50
opendevreviewJulia Kreger proposed openstack/ironic master: Allow project scoped admins to create/delete nodes  https://review.opendev.org/c/openstack/ironic/+/85279416:53
TheJuliahttps://review.opendev.org/c/openstack/ironic/+/850259 wants reviews so it can take over the world^W^W^W^Wfix all the problems17:00
JayFit's in my queue17:16
arozmanHi is there a way to specify IPA kernel parameters on a per node bases?17:29
JayFThat's sorta an x/y question, most IPA kernel params can be configured in other ways17:33
JayFWhat exactly are you trying to do?17:33
arozmanI am working on some IPA feature that can be enabled by kernel parameter but even in a cluster of nodes that is managed by a singe ironic deployments some nodes needs this feature when I boot IPA to inspect and provision some don't17:35
arozman*single ironic deployment17:36
JayFWorking on a feature, so something new?17:37
arozmanyes, I can show you https://github.com/Nordix/ironic-python-agent/commit/364331c9cb3ed64ead77ca15d4bcf3546d7a368d it is a feature that handles possible errors that happen in udev in relation to disk serial numbers17:38
JayFYou want general comments on that code as well? You've got a misspelling in there at first glance (aggressive has two gs)17:39
JayFarozman: I'm wondering why you're making this conditional at all17:39
JayFI suspect you could do something like check the contents of SERIAL or SERIAL_SHORT for some form of validity17:40
JayFif they don't seem valid, then fetch other potential serials as well17:40
JayFthis is an example of the sort of thing we really try not to make conditional if we can help it b/c we want to get the better behavior on any systmes with the issue17:40
arozmanI have noticed multiple times that ID_SERIAL_SHORT and ID_SERIAL are not reliable I had 2 situations I had cases when te WWN number was in ID_SERIAL_SHORT and things like that. So I thought that what if I collect all SERIAL variables for a device under the serial property, I have to be compatible with all hard disk and wierd controllers like HPSA so I don't even know what sort of SERIAL variables can 17:43
arozmanbe in udev. 17:43
arozmanI had this issues apear for different controller and disk combos with different udev/systemd versions17:44
arozmanSo I just wanted to raise the scope of the serial detection as I am also forced by downstream needs to use only serialNumber for now. 17:46
arozmanAnd I thought I make it conditional on a kernel param but  I got the info that it would be better if it would be possible to enable and disable this feature on a per node basis so this is the story17:47
JayFTheJulia: re: redfish sessions; it's accurate that the session_key never goes outside of ram?17:50
TheJuliaJayF: that is correct17:50
JayFarozman: yeah, if you want to disable/enable selectively per node, you'll need glue code in Ironic to enable that too17:51
JayFarozman: I strongly suggest you work towards an upstream feature to collate all possible serial number values though, I think there's general value in this and figuring out how to make it work without a toggle :)17:51
TheJuliabut hashed it just in case someone dumps the ram out17:51
JayFTheJulia: well, I'm thinking... salt17:51
JayFTheJulia: trying to talk myself out of it, but I think we should salt it17:51
JayFTheJulia: SHA512 really only has one weakness: rainbow tables17:52
TheJuliatrue, I guess we could save the start of the process and use that as the salt17:52
TheJuliaor as the start for the salt to be more precise17:52
TheJuliasince it is just in process session deduplication17:52
TheJulia... and tracking.17:52
JayFeven a static salt value is fine for the theoretical attack vector17:53
JayFhmm17:53
JayFactually17:53
JayFdynamic salt makes sense b/c otherwise you could, based on session key, figure out what nodes had identical passwords17:53
JayFand that *is* a vuln17:53
TheJuliasalt using a bmc address?17:54
arozmanJayF: That is my plan to upstream it eventually for sure.17:54
JayFTheJulia++ that'd be great17:54
TheJulialoosing hexdigest is a pain18:31
JayFFYI, got something Ironic-related going into Nova -> https://review.opendev.org/c/openstack/nova/+/853529 (skipping live migration testing, which can be flakey, for ironic virt driver changes)18:38
* JayF 's backport-everything-train is broken down at the nova gate station18:38
JayFlol18:38
TheJuliawell... there are things that could be said there...18:38
opendevreviewJulia Kreger proposed openstack/ironic master: Redfish: Consider password part of the session cache  https://review.opendev.org/c/openstack/ironic/+/85025918:49
TheJuliaJayF: ^18:49
JayF:eyes:18:49
JayF+218:52
TheJulia\o/19:42
opendevreviewJulia Kreger proposed openstack/ironic master: Add kickstart template 'url' option  https://review.opendev.org/c/openstack/ironic/+/85336820:07
opendevreviewJulia Kreger proposed openstack/ironic master: CI: anaconda: permit tls certificate validation bypass  https://review.opendev.org/c/openstack/ironic/+/85351020:07
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Changes to support Anaconda CI jobs  https://review.opendev.org/c/openstack/ironic/+/84958720:07
JayFfyi https://review.opendev.org/c/openstack/nova/+/800873 landed in nova stable/victoria, sending back to ussuri via https://review.opendev.org/c/openstack/nova/+/85354020:34
JayFtldr > [ironic] Minimize window for a resource provider to be lost20:34
TheJuliamuch appreciated20:39
JayFlike I said, backport party20:39
JayFif there's anything you know of other than that chain of ks patches that I'm going thru rebase hell on, lmk20:39
JayFso looks like 834709 might depend on pieces from https://github.com/openstack/ironic/commit/a813c769e8d27803e87d46280db84607fdfce15321:05
JayFwhich makes me wonder if we need to backport that set of changes, too, or just pull in the vars we need21:06
JayFI'm stopping here for the day so I can look at this with fresh eyes to make sure I'm not breaking anything21:06
JayFOk, that's not the actual problem21:55
JayFhttps://github.com/openstack/ironic/commit/1d6441cc347cfe984721f34ebb0cd64fd9e4d876 is the change that made node be passed into the deploy_utils validate_image_properties method21:55
JayFI'm pretty sure the backport gets into "unbackportable" without node being passed into that method, but that dep looks too heavy to backport21:56
JayFso I think we might be SOL? 21:56
JayFTheJulia: Let me summarize that stream of conciousness: the change I'm backporting (https://review.opendev.org/c/openstack/ironic/+/834709) needs the refactor done in https://review.opendev.org/c/openstack/ironic/+/797875 in order to have any chance of applying without having a significant rewrite21:58
JayFbut I don't really think it's justifyable to backport 79787521:58
JayFtl;dr: I think the best course of action may be that this line of kickstart-improvement-changes may not be backportable21:59
TheJuliaJayF: I think that is fair21:59
TheJuliaI think to yoga would be nice, although not a hard requirement21:59
JayFI am trying to yoga21:59
JayFthat is a description of why it can't backport TO YOGA21:59
TheJuliain Zed is like... perfection()21:59
TheJuliaI know :)21:59
TheJuliaperfection in the end is better than what owuld be nice :)22:00
JayFJust annoyed I literally lost half my day on this22:00
JayFbut poop happens22:00
TheJuliaoh noes22:00
TheJulia:(22:00
TheJuliaWell, thanks for looking into it!22:00
JayFThanks for doing it originally :P 22:01
TheJuliahttps://31842ea8f5f53197e266-38c03e7d9b567653f6179f4fd88f1080.ssl.cf1.rackcdn.com/835917/14/check/ironic-standalone-redfish/83b9de8/controller/logs/ironic-bm-logs/node-0_no_ansi_2022-08-17-21%3A52%3A24_log.txt <--- gaaaaaaah22:04
TheJuliaI think tempest timed it out22:04
TheJuliasoooooooooooooooooooo close22:04
TheJuliaso so so very close22:04
TheJuliaslow() though22:04
JayFThe dev cycles on that downstream were fun. Especially on hardware in labs that had UEFI debugging of some kind enabled and took a half hour to reboot lol22:06
JayFTheJulia: I wonder if you could point it at one of the RHEL-like things that Infra mirrors now (rocky 8?) and see if that is happier22:07
JayFTheJulia: working off infra mirrors for this would probably make a big diff22:07
TheJuliaoh, right, that is not the mirror url22:07
TheJuliaDOH!22:07
JayF2204-1870 ... you spent about 334 seconds downloading22:08
TheJuliafungi: what is the proper in-ci mirror for centos9 22:08
TheJuliayeah22:08
TheJuliait is going to vary based upon a few different factors too :\22:08
JayFmy other question is, you're providing no package selections22:09
JayFI wonder if there's less packages than none specified22:09
JayFe.g. a minimal set smaller than whatever the defaults are22:09
fungiTheJulia: there is configuration preinstalled onto centos-9-stream nodes at job start to point them to the mirror url for the provider in which they're running, so it varies depending on which provider the node was booted in22:12
TheJuliafungi: is there any way to look that up from a devstack job?22:12
TheJuliaJayF: i think it is *supposed* to just do a minimal install, but I'm ont 100% sure22:13
fungii expect so. have a specific build result? i can look at what's archived for it22:13
JayFfungi: we want to use the mirror /as an input to testing our kickstart driver/, so it needs to be programatic for addition to CI22:13
JayFTheJulia: Looking at the package names it logged there, I think it's reasonable to assume that's minimal 22:13
TheJuliafungi: the base is not centos for the job, it happens to be firing up centos as a VM since it has *all* of the bits pre-built on mirrors to do kickstarting as well22:14
TheJulia(they just hide the special file from the file list... as it is named something like .manifest22:14
TheJulia)22:14
fungiahh, in that case we do provide metadata in ansible variables which can be used to construct the mirror urls22:14
TheJuliaoh, cool then22:15
TheJuliaI can look in the vars22:15
JayFTheJulia: we don't test that we create a *bootable* machine, right?22:15
JayFTheJulia: we just want to ensure the install completes and fires the callback?22:15
TheJuliaJayF: just that it *boots*22:15
JayF(is there a callback in the upstream version?)22:15
TheJuliayes22:15
* TheJulia is about to upload fresh docs22:15
JayFTheJulia: then I'd be really curious to see if there's a way to force it to take an empty list or an invalid list too short for a base OS22:16
JayFTheJulia: because who cares if it boots22:16
TheJuliathere is some truth to this22:16
TheJuliawith ramdisk, I htink we just ping it and I think the idea hwere was the same22:16
TheJuliamaybe exclude linux-firmware22:16
TheJulia... work... laptop... vm... making computer hotter than daystar22:17
TheJuliaTIL kickstart is a super overused term22:18
opendevreviewJulia Kreger proposed openstack/ironic master: Add docs for VmWare deployment  https://review.opendev.org/c/openstack/ironic/+/85354922:18
TheJuliaw/r/t ^22:18
JayFTheJulia: talk to your founders about stealing the name from motorcycles :P 22:20
TheJuliablah22:21
* TheJulia once had a kickstart22:21
TheJuliafungi: in zuul anisble job vars, or hidden someplacel else?22:21
fungiTheJulia: this template has some examples which are probably relevant but the mirror_fqdn var you see there gets set from zuul_site_mirror_fqdn which is substituted at build start by the executor to correspond to the provider: https://opendev.org/opendev/base-jobs/src/branch/master/roles/mirror-info/templates/mirror_info.sh.j222:21
fungiclarkb may be able to better answer the question in #opendev (i'm theoretically on vacation this week)22:22
TheJuliafungi: thanks! go back to the execution of ./vacation22:23
TheJulia:)22:23
TheJulia(or.... crashing kerbals or whatever :)22:23
JayFfungi: if your bouncer isn't on vaca, you aren't22:23
TheJulia)22:23
JayFfungi: have a good one o/22:23
fungimany thanks!22:23
TheJuliaI think I'm going to go to the post office... and then go jump in a swimming pool becuase it is absurd hot outside22:23
fungi(and yeah, i'm not very good at "vacation")22:24
opendevreviewSteve Baker proposed openstack/sushy-tools master: Add SecureBoot support to the emulator, all drivers  https://review.opendev.org/c/openstack/sushy-tools/+/84226622:33
opendevreviewJulia Kreger proposed openstack/ironic master: Add docs for VMware deployment  https://review.opendev.org/c/openstack/ironic/+/85354922:45

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!