Friday, 2024-09-20

rpittaugood morning ironic! happy friday! o/07:20
rpittaudtantsur: about bifrost on noble, apparently noble drops linuxefi in favor of the generic linux 08:54
rpittau  * New upstream version, 2.12~rc108:54
rpittau  * build-efi-images: Drop linuxefi, using new loaders now08:54
rpittauI double-checked the presence of linuxefi.mod and it's been indeed removed since mantic08:54
rpittauI guess this impacts us deeply08:54
*** jroll01 is now known as jroll009:00
opendevreviewMerged openstack/ironic-tempest-plugin master: The adoption test should not depend on the iscsi deploy  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92702910:28
masgharHello ironic! For inspection rules, cid has laid the groundwork for CRUD here:  https://review.opendev.org/c/openstack/ironic/+/918303 Reviews are very welcome!10:56
masgharAlso, do you think it would be better to add the rules logic in a different patch (for easier reviews, for example)? (This one itself is pretty large)10:58
dtantsurrpittau: it is a huge problem, going well beyond bifrost, yes11:58
dtantsurwe need to check if grub has any fallback mechanism where we could try one set of commands and fall back to the other11:59
dtantsurmasghar: we generally try to avoid landing API without a backend (because any Ironic commit has to be fully functional)12:00
dtantsurin this case, maybe too few people actually care if we land them close to each other12:00
dtantsurideally, we should land the rule engine first, then the API. but I can agree that it's less convenient for testing.12:01
rpittaudtantsur: this is probably what we're looking for https://www.gnu.org/software/grub/manual/legacy/Booting-fallback-systems.html12:46
dtantsurthat could work, good catch!12:56
rpittauprobably mention that during the meeting on monday, noble support is a community goal for next cycle so we'll have to act on it13:00
dtantsurindeed13:00
dtantsurwanna file a bug?13:01
rpittauyep, will do13:02
rpittauhere https://bugs.launchpad.net/ironic/+bug/208130513:08
rpittauI need to skip now, we'll discuss on Monday, but I guess I'll see some talks before that :)13:09
dtantsursure, thank you13:10
opendevreviewDmitry Tantsur proposed openstack/bifrost stable/2024.1: Fix missed occurrence of SETUPTOOLS_USE_DISTUTILS  https://review.opendev.org/c/openstack/bifrost/+/93002413:13
opendevreviewMerged openstack/ironic-specs master: Support for hardware that cannot be powered off  https://review.opendev.org/c/openstack/ironic-specs/+/92665413:15
opendevreviewDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Provide consistent spelling of the microversion header  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93002813:28
dtantsurYour Friday facepalm ^^^13:28
opendevreviewDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Check inspection data and abortion in the standalone tests  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92792813:29
opendevreviewOpenStack Release Bot proposed openstack/bifrost stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/bifrost/+/93003113:50
opendevreviewOpenStack Release Bot proposed openstack/bifrost stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/bifrost/+/93003213:50
opendevreviewOpenStack Release Bot proposed openstack/bifrost master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/bifrost/+/93003313:50
opendevreviewOpenStack Release Bot proposed openstack/ironic-inspector stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-inspector/+/93003413:50
opendevreviewOpenStack Release Bot proposed openstack/ironic-inspector stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-inspector/+/93003513:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-inspector master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic-inspector/+/93003613:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-prometheus-exporter stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/93003713:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-prometheus-exporter stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/93003813:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-prometheus-exporter master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/93003913:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent-builder stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/93004013:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent-builder stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/93004113:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent-builder master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/93004213:51
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent/+/93004313:52
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent/+/93004413:52
opendevreviewOpenStack Release Bot proposed openstack/ironic-python-agent master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent/+/93004513:52
opendevreviewOpenStack Release Bot proposed openstack/ironic-ui stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-ui/+/93004613:52
opendevreviewOpenStack Release Bot proposed openstack/ironic-ui stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-ui/+/93004713:52
opendevreviewOpenStack Release Bot proposed openstack/ironic-ui master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic-ui/+/93004813:52
opendevreviewOpenStack Release Bot proposed openstack/ironic stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic/+/93004913:53
opendevreviewOpenStack Release Bot proposed openstack/ironic stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic/+/93005013:53
opendevreviewOpenStack Release Bot proposed openstack/ironic master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/ironic/+/93005113:53
opendevreviewOpenStack Release Bot proposed openstack/networking-baremetal stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/networking-baremetal/+/93005213:53
opendevreviewOpenStack Release Bot proposed openstack/networking-baremetal stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/networking-baremetal/+/93005313:54
opendevreviewOpenStack Release Bot proposed openstack/networking-baremetal master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/networking-baremetal/+/93005413:54
opendevreviewOpenStack Release Bot proposed openstack/networking-generic-switch stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/networking-generic-switch/+/93005513:54
opendevreviewOpenStack Release Bot proposed openstack/networking-generic-switch stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/networking-generic-switch/+/93005613:54
opendevreviewOpenStack Release Bot proposed openstack/networking-generic-switch master: Update master for stable/2024.2  https://review.opendev.org/c/openstack/networking-generic-switch/+/93005713:54
JayFI'm going to go grab some breakfast but if they're not landed by then, I'm going to go through and single core land all of the rote updates for Branch cutting. If someone wants to toss plus twos on them in the intervening 15-20 minutes that'd be cool14:12
cardoedtantsur: so I saw your comment around the DHCP and metal3. Something I was thinking about was writing another dhcp driver or more so a wrapper of the existing dnsmasq one for now. But defining some CRDs around what we need to transmit to dnsmasq and storing those in k8s directly from the conductor. Then getting to the point of not having ironic write the dnsmaq file directly but a controller doing it for us.14:58
cardoeThe CRDs would become like the RPC method to communicate that out.14:58
cardoeIf that would be something interesting to metal3, I'd be happy to collaborate.15:01
JayFHow would one set a firmware setting under bios/oem? e.g.: https://support.hpe.com/hpesc/public/docDisplay?docId=a00094361en_us&docLocale=en_US lists redfish/v1/systems/1/bios/oem/hpe/kmsconfig/ as a location with settings we can enable15:10
JayFbut I can't make heads or tails of if it's possible to get our BiosInterface to set these; I'm thinking no right now but not certain15:11
cardoeJayF: you cannot15:15
JayFcardoe: can you help me understand why?\15:15
JayFthis is a bios setting to operators, even if redfish doesn't think so, so we need to hook it up for em15:16
cardoeI absolutely agree.15:16
cardoeCome to my PTG session... "sushy-oem-idrac future and a future for other oems"15:16
JayFcan you help me understand the shape of the problem as someone who doesn't have access to hardware?15:16
cardoeYep.15:16
cardoeSo sushy essentially only talks to the non-OEM endpoints15:16
cardoeSo Ironic only exposes those.15:17
JayFfun15:17
cardoeFor a couple of things, the idrac driver overrides an interface. Which then uses the sushy-oem-idrac library, which talks to the OEM interface and provides that functionality.15:17
JayFyeah, that fits with my mental model and what I got from sushy, cool15:18
cardoeSo what I'd like for us to do is just merge sushy-oem-idrac into sushy proper and have the oem in there.15:18
JayFyeah, but I need ilo oem stuff :D15:18
JayFso we can do both15:19
JayFcardoe: do you have any workarounds? any tricks for downstream patch to do it?15:19
cardoeBut today it's a different interface still. You need to load and use the oem interfaces that sushy has15:19
TheJuliaWhy can’t folks just standardize or use the standard models…15:19
JayFe.g. sushy.do_specific_thing(blah)15:19
cardoeJayF: yeah 15:19
cardoeTheJulia: so in a bunch of cases the vendors provided the feature before DMTF standardized it. They're then slow to update.15:20
cardoeJayF: So where we use sushy, we monkey patch the hell out of it.15:20
JayFTheJulia: I told my co-worker downstream that if stuff like this didn't exist, ironic probably wouldn't either lol15:20
JayFcardoe: that's a really bad answer :( 15:20
cardoeI agree with you.15:21
cardoeThat's why I wanna come up with a solution that the upstream community is happy with.15:21
TheJuliacardoe: I know, or intentionally implement different modeling15:21
TheJuliaBelow the standardization which it is incompatible15:21
cardoeSo like we have sushy-rax. Which depends on sushy-oem-idrac and also has some ilo stuff in it. You use sushy-rax instead of sushy.15:22
JayFcardoe: is that a public fork?15:22
cardoeAnd when you try to hit the standard sushy interface, if the hardware needs it we just add that in15:22
cardoeIt's not.15:22
cardoeI can probably make it public.15:23
JayFI doubt we'd use it directly, but it would be inspirational for both upstream and downstream ideas15:23
cardoeYeah so my goal is for it to die.15:23
cardoe-ETIME :/15:24
cardoehttps://bugs.launchpad.net/sushy/+bug/2075979 so that bug for example.15:25
JayFI'm going to link that under sushy-oem-idrac15:25
JayFfor ptg topic15:25
JayFthat's a useful contextual bug15:25
cardoeSo even when I got Dell to fix the firmware15:26
cardoeman I scattered brained today sorry.15:26
JayFYeah, I think some of our headache here is that I suspect we'll be touching settings that may not make sense or even exist in most hardware in the world, even of similar brands15:26
cardoeSo the firmware fix is there's not NIC.Slot.1-1 and NIC.Slot.1-1-1. It's just NIC.Slot.1-115:26
JayFso I need a path to ensure we can, preferably without patching the world, have a little more flexibility in OOB stuff like we do for IB stuff15:27
cardoehttps://opendev.org/openstack/sushy/src/commit/cd94fc00b69827b488fee15367e6b69bbf09bea1/sushy/resources/system/ethernet_interface.py#L59 that function which is used by the in-band inspector only sees 2 NICs on my hardware15:27
cardoeNot the 6 that exist.15:27
JayFespecially for people who have the experience of "I used to do X in a hardware manager in-band using OEM tools" to "hey look Ironic provides an interface ... but it can't do $5_percent_of_things" and it's not as trivially pluggable15:27
cardoeSo in sushy-rax, that summary function is monkey patched15:28
cardoeIt extends out and uses https://review.opendev.org/c/openstack/sushy/+/92783815:28
cardoeTo query not just the EthernetInterfaces but Port interfaces that are of type=Ethernet15:28
cardoeTo provide me all the data.15:29
cardoeProbably a bad example cause of two endpoints.15:29
cardoeYour BIOS stuff might be a better example.15:29
JayFwell, the thing that made me hrm about that15:30
JayFis it's not a /oem on the root15:30
JayFit's /bios/oem15:30
cardoehttps://opendev.org/openstack/ironic/src/commit/5857ca0cd5bc4949f77b9f3b6c3e0d7ab97f6b88/ironic/drivers/modules/redfish/bios.py#L8215:30
JayFI don't know redfish well enough to know if that's meaningful but it seems like at least partially in the right place?15:30
cardoeSo that's another spot we've monkey patched15:31
cardoeSo we always load an OEM Manager at the top level.15:31
cardoeIt might be a no-op.15:31
JayFI'd love to know what actual url that translates to for most gear15:32
JayFso I could get my downstream to see if the setting is in there 15:32
JayFhmmm15:32
cardoeBut that line of code fetches not just system.bios.attributes but it fetches system.oem.bios.attributes15:32
JayFso it might just work?15:32
cardoeI can set those iLO settings locally.15:32
JayFthat is system.oem.bios -- mine would be system.bios.oem?15:32
cardoeWell that's the attribute structure that sushy is making15:36
cardoeoh yes sorry it's system.bios.oem.attributes15:36
JayFso you're saying system.bios.oem.attributes had to be patched in for those settings to work15:37
JayFI'd /really really/ love to see that patched code15:37
cardoeBUT the attribute field is coming from base.BIOSInterface15:37
cardoeSo there's ILOBios(base.BIOSInterface)15:37
cardoeSo sushy creates an instance of RedfishBIOS at system.bios15:39
cardoeWe create an instance of ILOBios at system.bios.oem15:40
cardoeIf the ILO OEM extension was applied to that connection I think?15:40
JayFcan you just see if you can share the code? It's OSS and I know, at least when I was there, it was pretty easy to get a 'yes' to that for patched oss15:41
cardoeopenstack baremetal node bios setting list against a DL380 Gen10 machine shows "KeyManagementType" as an available option.15:41
cardoeYeah15:41
JayFcardoe: bonus points if the patched version you're running happens to be xena :P15:42
cardoeuh it's older than that15:48
cardoethe problem is that this code is now more like sushy's cousin15:48
cardoeCause the common ancestor according to git is 668c40dec4cb65d658aeb6b22e2009aac963bc8b15:49
cardoeand the ironic using it I think is train15:49
JayFI'd imagine so15:49
JayFI am not under the impression I'm ever going to like, use it directly15:49
JayFbut use it as inspiration for understanding the problem and helping my downstream15:49
dtantsurcardoe: I've definitely seen a k8s controller for dnsmasq, although seemingly abandoned.15:56
dtantsurLet me see if my browsing history retains it..15:57
dtantsurcardoe: yeah https://github.com/aenix-io/dnsmasq-controller15:57
dtantsurit won't solve the HA issues, I suspect, nor the stability problems15:57
cardoeyeah I have too. But I was thinking more for generically DHCP options for a device instead of something specifically managing dnsmasq15:57
cardoeSo that another backend could eventually serve it up15:58
cardoeJayF: requested to change visibility...15:58
JayFthank you :D15:59
dtantsurcardoe: we could invent something for metal3 but I'd rather get it all the way to Ironic so that more community can benefit from it.16:00
JayFI mean, isn't kea dhcp backend pretty much ideal for kubernetes?16:04
JayFI thought it was a cluster-friendly server, since you can have a centralized backend store16:04
dtantsurProbably? All I say is that I'd rather not have anything metal3-specific.16:05
dtantsurEspecially since the metal3 team is also small and overloaded..16:05
JayFI always feel like the more we put in ironic the better it is for everyone16:05
JayFtalking about both openstack and metal3-related features16:05
JayFit's like, you can help one community or the other, but if you put it in ironic directly, both sides benefit16:05
* dtantsur nods16:06
JayFnot always an option but when it is, it's awesome16:06
JayFI try to convey that feel by helping folks in slack, although my hit rate on "stuff that's different in metal3" is pretty bad lol16:06
dtantsurwelll.. there are VERY few people who understand both Ironic and Metal3 well. One hand is more than enough to count.16:07
JayFYep, and by folks like you, riccardo, adam, and hopefully ma little bit of me, we can pull em a little closer16:08
dtantsur++16:08
JayFdoes metal3 intersect at all with project mercury, in terms of use case?16:10
JayFI know almost nothing about networking in the k8s world16:10
dtantsurit very well can in the future16:10
JayFone of the things I put on that PTG topic16:10
dtantsurwe have our own IPAM now but it's pretty limited16:10
JayFis that we need like, a 5 year vision if we want people to invest more in that16:10
JayFif I can tell the story of where we're going, it's a lot easier to get the time to invest it16:11
JayFbecause -- at least for some of my use cases -- we'll need more than just project mercury to overhaul an architecture16:11
TheJuliaWe also need to show people a viable path which is really not the same old same old in terms of vendor ml2 plugin lockin pain16:11
TheJuliabut yeah16:11
* TheJulia returns to pto by hunting through electrical schematics to find a short circuit16:12
JayFthe theme for the PTG gonna be "make the present better by designing the future" :D 16:12
TheJulia+++++16:12
JayFwe're going to prove you /can/ have your cake and eat it too16:12
JayFbecause we can't let the clouds that exist starve as we're getting there16:12
TheJulia++++16:15
TheJuliademos, many demos16:15
* JayF now finally getting around to approving those branch cut passwords lol16:22
JayFs/passwords/patchsets/16:22
JayFI always love approving these, it's like a mini-release-celebration in my office :D 16:36
dtantsurironic-tempest-plugin is absolutely cursed16:37
JayF:( that inspector change still flailing (I left the typo because it fits lol)16:40
opendevreviewDmitry Tantsur proposed openstack/ironic-tempest-plugin master: Check inspection data and abortion in the standalone tests  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/92792816:43
opendevreviewMerged openstack/networking-generic-switch stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/networking-generic-switch/+/93005516:44
dtantsurSome silly fixes for your review pleasure on Friday afternoon: https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/930028 https://review.opendev.org/c/openstack/ironic-python-agent/+/929766 https://review.opendev.org/c/openstack/bifrost/+/92889616:45
JayF+2 all three, I already was looking at the first16:46
opendevreviewMerged openstack/ironic-ui stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-ui/+/93004616:47
opendevreviewMerged openstack/ironic-ui stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-ui/+/93004716:47
opendevreviewMerged openstack/networking-generic-switch stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/networking-generic-switch/+/93005616:48
opendevreviewMerged openstack/ironic-python-agent stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent/+/93004316:50
opendevreviewMerged openstack/ironic-python-agent stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent/+/93004416:54
opendevreviewMerged openstack/networking-baremetal stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/networking-baremetal/+/93005216:55
opendevreviewMerged openstack/networking-baremetal stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/networking-baremetal/+/93005316:55
opendevreviewMerged openstack/ironic stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic/+/93004917:01
opendevreviewMerged openstack/ironic stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic/+/93005017:01
opendevreviewMerged openstack/ironic-inspector stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-inspector/+/93003417:01
opendevreviewMerged openstack/ironic-inspector stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-inspector/+/93003517:01
opendevreviewMerged openstack/ironic-python-agent-builder stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/93004017:06
opendevreviewMerged openstack/ironic-python-agent-builder stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/93004117:06
opendevreviewMerged openstack/ironic-prometheus-exporter stable/2024.2: Update .gitreview for stable/2024.2  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/93003717:08
opendevreviewMerged openstack/ironic-prometheus-exporter stable/2024.2: Update TOX_CONSTRAINTS_FILE for stable/2024.2  https://review.opendev.org/c/openstack/ironic-prometheus-exporter/+/93003817:08
JayFdtantsur: https://review.opendev.org/c/openstack/ironic-python-agent/+/928779 is conflicting in scope, if not in exact code, with changes I've got in flight18:19
JayFdtantsur: I'm going to operate under the assumption you're OK with me pulling that into my chain so it is happier18:20
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Remove use of ironic_lib i18n module  https://review.opendev.org/c/openstack/ironic-python-agent/+/93008018:20
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Migrate to oslo.utils-based format_inspector  https://review.opendev.org/c/openstack/ironic-python-agent/+/92846318:29
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Cleanup usage of imported-from-ironic-lib disk_utils  https://review.opendev.org/c/openstack/ironic-python-agent/+/92846618:29
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Remove use of ironic_lib i18n module  https://review.opendev.org/c/openstack/ironic-python-agent/+/93008018:29
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Migrate more trivial code from ironic-lib  https://review.opendev.org/c/openstack/ironic-python-agent/+/92877918:29
JayF^^ part of a 4 patch series, migrate to oslo.utils format_inspector, refactor/cleanup disk_utils usage, remove use of ironic_lib i18n module (we had ~10 strings with _() in IPA, and it's never translated -- we should do it proper or not at all), and the rebased version of dtantsur's patch18:30
JayFI suspect that first patch might need to be hrm'd at given our experience with raw vs gpt returns18:31

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!