Thursday, 2024-02-29

opendevreviewVerification of a change to openstack/ironic master failed: Add a reserved workers pool (5% by default)  https://review.opendev.org/c/openstack/ironic/+/90280101:02
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.1: stable-only: pin proliantutils/scciclient to prevent break  https://review.opendev.org/c/openstack/ironic/+/91052801:05
opendevreviewVerification of a change to openstack/ironic master failed: Detect ilo6 and redirect to redfish  https://review.opendev.org/c/openstack/ironic/+/88829701:07
opendevreviewVerification of a change to openstack/ironic stable/2023.2 failed: ci: Source install dnsmasq-2.87  https://review.opendev.org/c/openstack/ironic/+/91044401:07
*** tkajinam is now known as Guest134901:24
fungii take it the new dnsmasq 2.90 in jammy wasn't any better?01:27
tonybJayF: my phone had a notification from you requesting an autohold I can't actually find the request anywhere ... did that get taken care of?01:28
TheJuliafungi: not great, could be resource contention issues cropping up as well01:28
fungiah, okay01:28
fungitonyb: i set one, lemme check if it caught anything01:29
tonybfungi: Okay cool.  Just saw the request but I wasn't on my laptop until now01:30
TheJuliaI ran some rechecks about an hour ago since there was minimal gate activity just to see if we could get it sorted (and some stable branch stuff sorted as well. It should give us a data point anyhow01:30
fungiJayF: ssh root@173.231.255.10201:32
fungiTheJulia: ^ i added your key too just in case01:33
TheJuliaokay, I won't be able to dig until tomorrow01:38
TheJuliathanks!01:38
fungii doubt there's any hurry01:38
fungijust wanted to add it before i knocked off for the evening so i don't need to remember in the morning ;)01:39
opendevreviewfrankming proposed openstack/ironic master: Fix iscsi url generate method for ipxe  https://review.opendev.org/c/openstack/ironic/+/91030002:24
opendevreviewfrankming proposed openstack/ironic master: Fix iscsi url generate method for ipxe  https://review.opendev.org/c/openstack/ironic/+/91030002:43
opendevreviewMerged openstack/ironic master: [ci] Temporarily disable standalone job voting  https://review.opendev.org/c/openstack/ironic/+/91054804:17
opendevreviewOpenStack Proposal Bot proposed openstack/ironic-inspector master: Imported Translations from Zanata  https://review.opendev.org/c/openstack/ironic-inspector/+/90693504:31
opendevreviewKaifeng Wang proposed openstack/python-ironicclient master: Client support port name  https://review.opendev.org/c/openstack/python-ironicclient/+/89606705:37
opendevreviewMerged openstack/ironic master: Add a reserved workers pool (5% by default)  https://review.opendev.org/c/openstack/ironic/+/90280107:22
dtantsurwow, so much scrollback07:59
rpittaugood morning ironic! o/08:00
rpittauIndeed!08:00
*** elodilles_pto is now known as elodilles08:29
rpittauJayF: re: https://review.opendev.org/c/openstack/ironic/+/908955  yep and done08:29
opendevreviewSven Kieske proposed openstack/networking-baremetal master: don't force amqp_auto_delete for quorum queues  https://review.opendev.org/c/openstack/networking-baremetal/+/90399509:04
opendevreviewAdam Rozman proposed openstack/ironic-lib master: improve disk I/O error handling during cleanup  https://review.opendev.org/c/openstack/ironic-lib/+/90888909:17
opendevreviewAdam Rozman proposed openstack/ironic-lib master: improve disk I/O error handling during cleanup  https://review.opendev.org/c/openstack/ironic-lib/+/90888909:20
opendevreviewMerged openstack/ironic master: Multiple CI updates/improvements  https://review.opendev.org/c/openstack/ironic/+/90895509:47
opendevreviewDmitry Tantsur proposed openstack/ironic master: Trivial: include vmedia API in the reference  https://review.opendev.org/c/openstack/ironic/+/91058509:51
dtantsurrpittau: ^^09:51
opendevreviewMerged openstack/ironic master: Trivial: include vmedia API in the reference  https://review.opendev.org/c/openstack/ironic/+/91058510:30
opendevreviewMerged openstack/python-ironicclient master: Add missing commands to the documentation  https://review.opendev.org/c/openstack/python-ironicclient/+/91054010:32
rpittaumetalsmith legacy is broken?!?11:04
iurygregorygood morning Ironic, I'm back o/11:28
rpittauwelcome back iurygregory :)11:37
iurygregorytks rpittau o/11:38
*** tosky_ is now known as tosky11:48
rpittauJayF: we forgot a small thing https://review.opendev.org/c/openstack/releases/+/91063415:02
TheJuliaproject config?15:03
rpittauchange from storyboard to launchpad :)15:15
opendevreviewRiccardo Pittau proposed openstack/metalsmith master: [DNM] TEST CI  https://review.opendev.org/c/openstack/metalsmith/+/91063615:22
rpittau^ trying to see if the metalsmith legacy job is really foobar15:22
dtantsurI really wonder why Ironic API does not support lists the way the rest of the world does (fields=a&fields=b&...)15:43
dtantsurand if I fix that, does it have to be hidden behind a microversion?15:46
dtantsuras much as I am an API purist, I tend to think no because we already have code in the wild expecting this behavior (e.g. gophercloud)15:47
opendevreviewMerged openstack/ironic-python-agent-builder master: Update ipmitool version to 1.8.19  https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/91034416:05
opendevreviewDmitry Tantsur proposed openstack/ironic master: Support more standard way of passing lists via query strings  https://review.opendev.org/c/openstack/ironic/+/91064216:08
JayFrpittau: ack, that's a good find and explains weird behavior I've seen16:13
JayFdtantsur: I am not against your change, to be clear, but I see ?x=a,b,c, more often than x=a&x=b&x=c16:14
JayFI think it's maybe a flavoring/era thing16:14
dtantsurMaybe? I just did a quick googling, and found the latter more common16:14
dtantsurOur own framework happily converts it to a Python list (and then our validation rejects it)16:14
opendevreviewVerification of a change to openstack/ironic master failed: Don't import sushy conditionally, it's a requirement  https://review.opendev.org/c/openstack/ironic/+/91052916:24
dtantsurmetalsmith strikes again. not worth rechecking right rpittau?16:25
TheJuliaMetalsmith's CI is broken now?16:28
dtantsurthe legacy job apparently16:28
* dtantsur hasn't looked into it16:28
TheJuliaI'm still on the "I'd like to just wrap up metalsmith" boat16:31
opendevreviewJay Faulkner proposed openstack/ironic master: Fix multiple assignment of redfish_system_id during node creation  https://review.opendev.org/c/openstack/ironic/+/90985116:33
dtantsurTheJulia: not that I object long-term, but chances are high you're saying "I'd like to just wrap up testing local boot with BIOS". Because chances are high that's the cause of the failure.16:33
TheJuliaDo we still see bios booting outside of legacy hardware being leveraged?16:35
dtantsurWell, I literally had a conversation with a Metal3 contributor a few days ago. He's preparing a new quick start guide and explicitly uses legacy boot because of the problems with his ilo4 machines.16:35
TheJulia...16:35
TheJuliaso he is using legacy machiens16:36
TheJuliamachines16:36
TheJulianot new machines, machines.... from before 2017?16:36
TheJuliailo4 == HPE Gen8 and Gen916:36
dtantsurHow many people still do? :)16:37
dtantsurmeanwhile, IPA does not start during cleaning https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_b9e/910529/1/gate/metalsmith-integration-glance-centos9-legacy/b9e6fe0/controller/ironic-bm-logs/node-0_console_2024-02-29-16:20:27.log16:37
TheJuliaQ4 2017 is when Gen10 was released16:37
dtantsurI wonder if it's a regression because of TinyCore 1516:37
dtantsurInterestingly, it's a cleaning *after* deployment, so most of the test has worked.16:37
dtantsurudhcpc: received DHCP NAK16:38
TheJuliacould it just be CI load?16:38
rpittaudtantsur, TheJulia, JayF, I'm testing the metalsmith legacy job here https://review.opendev.org/c/openstack/metalsmith/+/91063616:38
rpittauI won't be online tomorrow so please keep an eye on that16:38
dtantsurfor all IP addresses it tries16:38
dtantsurIt's really, really weird. Nothing like that happens during the successful run.16:40
dtantsurFeb 29 16:08:54.668903 np0036916950 dnsmasq-dhcp[115505]: DHCPDISCOVER(tap9be56843-96) 52:54:00:ab:1b:1116:41
dtantsurFeb 29 16:08:54.668916 np0036916950 dnsmasq-dhcp[114964]: DHCPDISCOVER(tap9be56843-96) 52:54:00:ab:1b:1116:41
dtantsurFeb 29 16:08:54.668925 np0036916950 dnsmasq-dhcp[115505]: DHCPOFFER(tap9be56843-96) 10.1.0.58 52:54:00:ab:1b:1116:41
dtantsurFeb 29 16:08:54.668940 np0036916950 dnsmasq-dhcp[114964]: DHCPOFFER(tap9be56843-96) 10.1.0.41 52:54:00:ab:1b:1116:41
dtantsurCould someone explain me, how and why neutron ends up allocation two addresses to the same MAC?16:41
dtantsurFeb 29 16:08:57.705457 np0036916950 dnsmasq-dhcp[114964]: DHCPREQUEST(tap9be56843-96) 10.1.0.58 52:54:00:ab:1b:1116:42
dtantsurFeb 29 16:08:57.705484 np0036916950 dnsmasq-dhcp[114964]: DHCPNAK(tap9be56843-96) 10.1.0.58 52:54:00:ab:1b:11 address not available16:42
rpittaubye everyone, see you on monday! o/16:44
dtantsurWise choice :)16:44
dtantsurOh fun. I think it's a race between our DHCP code and the built-in udhcp.16:48
dtantsurls: cannot access '/var/run/udhcpc*.pid': No such file or directory16:48
dtantsurand then udhcp actually starts while we're waiting 30 seconds16:48
dtantsurhttps://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa/build_files/bootlocal.sh#L41-L5316:50
dtantsurTheJulia: do you still remember that ^^ part of tinyipa?16:51
dtantsurI don't think this race explains the behavior between udhcp and dnsmasq tbh16:52
dtantsurMmm, eth0 and eth1 actually being the same thing? Maybe?16:54
* TheJulia blinks16:54
TheJuliaI sort of remember the race16:55
JayFcongratulations to rpittau on being the PTL of Ironic for Dalmation/2024.2 cycle16:55
dtantsurwooohoooo \o/16:55
JayFnominations are over, nobody dared oppose him 16:55
JayF;)16:55
dtantsurJayF: welcome to free life :D16:55
TheJuliawell, he is not entirely free, yet16:56
TheJuliaso multiple addresses is sort of reminiscent ipv6 + neutron code16:57
TheJuliabut engaging rescue woud sort of explain things16:57
TheJuliaone not being removed in time16:57
TheJuliaoh... noes16:57
dtantsurehhhmmmmmmm this code is apart of rescue https://opendev.org/openstack/ironic-python-agent/commit/71c89c99fa1c66765c4d478fb9cffdf2503ab53d16:57
dtantsurwut16:57
dtantsurWUT16:57
TheJuliadnsmasq must be holding on to the old config16:58
TheJulia... I mean, it is *not* instant16:58
dtantsuryeah, but why does it give away 2 IPv4 addresses to the same MAC?16:59
dtantsurand why does it NOT happen normally?17:00
* dtantsur has questions and zero clues17:00
TheJuliadid the agent not start?17:00
dtantsurconfigure_dhcp_network happens before17:00
TheJuliawhere before line 34?17:01
TheJulia /etc/init.d/dhcp.sh gets run17:03
dtantsurhttps://opendev.org/openstack/ironic-python-agent-builder/src/branch/master/tinyipa/build_files/bootlocal.sh#L65-L7817:03
* dtantsur needs to do some errands17:05
TheJuliaoh, I was looking at the older one17:05
* TheJulia wonders if udhcpd ever released to the background, ornot17:07
TheJuliaor not17:07
opendevreviewJulia Kreger proposed openstack/ironic master: docs: troubleshooting addition for sync period  https://review.opendev.org/c/openstack/ironic/+/90834217:23
TheJuliacan I get reviews on https://review.opendev.org/c/openstack/ironic/+/910528 which fixes 2023.117:23
TheJuliait will also likely need further backporting to older branches17:24
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.2: ci: multiple fixes  https://review.opendev.org/c/openstack/ironic/+/91065117:30
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.1: stable-only: pin proliantutils/scciclient to prevent break  https://review.opendev.org/c/openstack/ironic/+/91052818:28
TheJuliaJayF: made standalone jobs non-voting...18:29
TheJuliaon ^18:29
TheJuliamanually trying to see if I can get core files to appear18:50
sean-k-mooneythe nova patches for the shard support are now merged just an fyi18:51
sean-k-mooneyso we just need to merge the first ironic ci patch then we can merge the nova ci patch18:51
TheJuliacool, I think I took a look at it yesterday and was good with it18:51
opendevreviewJay Faulkner proposed openstack/ironic master: [CI] Support for running with shards  https://review.opendev.org/c/openstack/ironic/+/89446018:51
TheJuliaokay, that answers that!18:52
JayFHad to remove the depends-on18:52
JayFshould be good if the gate is good18:52
sean-k-mooneynova is in the default "integrated" queue for gate ironic is in the "ironic" queue18:52
sean-k-mooneyso depends on works for check but not gate18:52
sean-k-mooneyso ya once the ironic one is landed ill drop the depend on in nova and reappove it18:53
TheJuliaJayF: approved the change18:55
sean-k-mooneyhttps://github.com/openstack/ironic/blob/78b6f00af8350c286d92420012a0b09fdb54ff32/ironic/cmd/__init__.py#L2218:57
sean-k-mooneyby the way JayF said ye are having dnsmasq issues18:57
sean-k-mooneywe removed this workaroudn in nova a few releases ago18:58
TheJuliayes, we're seeing jobs fail with dhcp interactions and dnsmasq just going *poof* deep inside of our heaviest jobs18:58
sean-k-mooneyhttps://github.com/openstack/nova/commit/fe1ebe69f358cbed62434da3f1537a94390324bb18:58
sean-k-mooneyok so its proably not related to ^ then18:58
TheJuliayeah, unrelated18:59
sean-k-mooneybut i think you could also remove that workaround in the fugure just an fyi18:59
TheJuliaI really wish it was that simple as related to dns right now :)18:59
TheJulia.... which is why one of my terminals is in devstack node re-running tempest tests right now18:59
TheJulia"come on tests, fail so I can see if a core file appears!18:59
TheJulia"18:59
sean-k-mooneyTheJulia: i know there used to be a gap with the extra dhcpu options required for baremental and ovn19:00
TheJuliasean-k-mooney: still some gaps, some cases things just go sideways, for example if grub is used19:01
sean-k-mooneybut maybe you coudl remvoe the use fo dnsmasq in those jobs next cycle if that has been adresseed?19:01
TheJuliawe've got a whole doc on "known issues"19:01
sean-k-mooneyack19:01
sean-k-mooneydo you knwo what we are doing for that in the new installer? are we just going to supprot neutron dncp on a dedicated edpm host somewhere19:01
TheJuliaIt is likely worthwhile to try with a standalone job though...19:01
opendevreviewJay Faulkner proposed openstack/ironic master: Fix multiple assignment of redfish_system_id during node creation  https://review.opendev.org/c/openstack/ironic/+/90985119:01
JayFoh no19:02
TheJuliasean-k-mooney: neutron dhcp on a networking host if needed19:02
opendevreviewJay Faulkner proposed openstack/ironic master: [DNM/Science] 𝆕 Green is the color of my DNS 𝆕  https://review.opendev.org/c/openstack/ironic/+/91065719:02
JayFsean-k-mooney: ^ some science, just for you19:02
sean-k-mooney:)19:02
TheJuliasean-k-mooney: ovn dhcp *does* work in some cases, just not perfectly19:02
JayFthe devstack patch I just accidentally pushed is redy to review as well, fwiw19:02
TheJuliaand again, goes back to that list of known issues19:03
sean-k-mooneyJayF: the orignal issues was fixed in 201519:03
TheJuliadoh! didn't fail *but* we have another tempest test bug I've found19:03
JayFsean-k-mooney: I believe it19:03
* TheJulia goes and files a new bug19:03
JayFsean-k-mooney: I can't wait for the more satisfying removal19:03
JayFoh btw19:03
JayFadamcarthur5 is working on IPA eventlet in the same vein you did for nova19:03
JayFtrying to find ways to break our dependency without doing a full migration19:04
JayFsince we don't know what that looks like yet19:04
JayFjust unspider the arms of eventlet from places it can be extracted from lol19:04
sean-k-mooneyya rodolfo wanted to chat about this too19:04
sean-k-mooneyi think we need ot get concensous still that thsi is somehting we are going to do and how19:04
JayFWe really,really need people to go participate in tthat gerrit thread19:05
adamcarthur5I have half an idea in my head. Not so sure how it will play out - hopefully I'll have an update soon. 19:05
JayFbut I think there are several tc-members who are trying to punt it to PTG19:05
sean-k-mooneybtu we can hopefully make some progress in the ptg and do some of it in dalmaition19:05
sean-k-mooneyJayF: i can try an loop back to the doc again in a few days19:05
sean-k-mooneyi have not looked at it for a while19:05
sean-k-mooneybut ya my goal was to push all the eventlet code into one cornder of the code base19:06
TheJuliaI'm surprised nobody has reported https://bugs.launchpad.net/ironic/+bug/2055456 before19:06
sean-k-mooneywhere we could then deal with it in a more tageted way 19:06
sean-k-mooneyTheJulia: well without using rally/browbeat or somethign like that19:07
sean-k-mooneyits proably not done that often unless ye have a grenade type job19:07
TheJuliayeah, nobody has interest in that with ironic19:08
TheJuliaThe grenade stuff is highly focused too, this is just a single scenario job leaking stuffs19:08
sean-k-mooneywell when it comes to baremetal provisioing i stongly susect the time is spent mostly not in ironic19:08
TheJuliayeah, most of it spent waiting for things to actually boot or move over a wire19:09
TheJuliaand physics, and people run away19:09
TheJulia"Lets talk about the speed of light in a medium!"19:09
TheJulia"or waveform in copper wire"19:09
sean-k-mooneyi have expriced the "my <insert vendor here> server takes 30 mins to post any time i update a bios settign tyep issues ebfore" not fun19:10
* TheJulia shivers19:10
TheJuliayeah, that is a thing19:10
sean-k-mooneymandatory memory training is just as enjoyable as mandaroy coperate training19:11
sean-k-mooneythat actully what happened in that specific case. updating a bios seeting cause the firmware to need to do full memory training for the ram agian19:12
JayFI suggested to my downstream that when they get complaints about boot speed, to do a time-taken chart19:13
JayFso people can see the tiny sliver of "ironic doing stuff" beside the giant sliver of "these machines take 900 years to boot"19:13
TheJuliaI swear I didn't add 300 petabytes of ram to the machine19:14
TheJuliagrrr, no core :(19:36
TheJuliadnsmasq is definitely crashing19:58
TheJuliahttps://paste.openstack.org/show/823336/19:58
TheJuliawe've got a bug it seems19:58
TheJuliahttps://paste.opendev.org/show/823337/20:04
TheJuliaThose are 2.8720:05
TheJuliarepeating with 2.9020:13
TheJulia2.90 == https://paste.openstack.org/show/823338/20:18
TheJuliasame result20:18
JayFso dnsmasq is freeing a bad pointer, somewhere20:19
JayFer, null(?)20:19
TheJuliadouble free is my guess, kernel is not logging it20:20
TheJuliabut only one in the output, we'd have to dig at the code to really know20:22
TheJuliarunning with 2.86, and waiting with strace20:23
TheJuliaand already20:24
TheJuliawritev(2, [{iov_base="munmap_chunk(): invalid pointer", iov_len=31}, {iov_base="\n", iov_len=1}], 2) = 3220:24
TheJuliainterestingly, not free20:28
JayFsean-k-mooney: I was scanning scrollback and half-reading, and I saw "mandatory memory training" and all the sudden had a vision of hundreds of red hat employees getting lectured about the superiority of rust in light of the recent USGOV declaration of "please be memory safe"20:29
* TheJulia imagines log4j appearing with an opinion20:29
TheJuliatrying 2.83, next I guess 2.80 most likely20:30
TheJuliathe auto-restart neutron has likely hides this but I bet this is not unique by baremetal20:31
TheJulia2.83 is seeming solid20:47
TheJuliaI bet it appears in 2.8620:47
TheJuliaI think this is the same dnsmasq bug just presenting less visibly20:47
TheJuliaso 2.85 seems happ so far21:14
TheJulia2.86test5 == writev(2, [{iov_base="free(): invalid pointer", iov_len=23}, {iov_base="\n", iov_len=1}], 2) = 2421:32
TheJulia 21:32
TheJulia2.86test1 also fails21:40
TheJuliaI think i found where the issue came from21:51
TheJuliahttps://thekelleys.org.uk/gitweb/?p=dnsmasq.git;a=commit;h=10d8b5f001a34ff46b3a72575f3af64b065f863721:51
TheJuliaI'm goign to re-run this twice most likely which means it will take a while21:53
*** jph6 is now known as jph22:24
TheJuliaupdate posted to https://bugs.launchpad.net/ironic/+bug/202675722:53
opendevreviewJulia Kreger proposed openstack/ironic master: ci: pin CI to dnsmasq 2.85  https://review.opendev.org/c/openstack/ironic/+/91066822:55
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.2: ci: pin CI to dnsmasq 2.85  https://review.opendev.org/c/openstack/ironic/+/91066922:58
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.1: ci: pin CI to dnsmasq 2.85  https://review.opendev.org/c/openstack/ironic/+/91067923:06
opendevreviewMerged openstack/ironic master: [CI] Support for running with shards  https://review.opendev.org/c/openstack/ironic/+/89446023:54

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!