Thursday, 2021-05-20

*** openstack has joined #openstack-ironic00:22
*** ChanServ sets mode: +o openstack00:22
buhmanhave there been any freenode-related discussions in the last day? I'm not sure which mailing list that would appear on.00:48
*** mgoddard has quit IRC00:56
stevebakerbuhman: openstack-discuss has a big thread00:58
buhmanthanks!00:58
stevebakerJayF: so unmasking tmp.mount made my LVM mount issues just go away, yay!00:59
*** mgoddard has joined #openstack-ironic01:04
*** rloo has quit IRC01:23
* stevebaker searches for OFTC and gets http://www.oftc.org.nz/ "We are a tramping club, based in Christchurch, New Zealand, for trampers over forty years old."01:25
stevebakersign me up, I qualify!01:25
TheJuliaI just wish there was consensus01:37
TheJuliaI also should realize that there won't be consensus01:37
TheJuliaWhat if we were to migrate to >1 locations?01:37
*** iurygregory has quit IRC01:48
*** ricolin_ is now known as ricolin02:05
buhmanI recovered my oftc nick, so I'm ready for anything02:20
*** ricolin has quit IRC02:27
*** ricolin_ has joined #openstack-ironic02:31
*** gyee has quit IRC02:31
*** ricolin_ is now known as ricolin02:32
janderswarning for those on FC3302:35
jandersthere is a suspected breaking sssd update02:35
jandersit locked me out this morning, fixed now, I hear more people are seeing it02:36
jandersif you're on FC33 and are thinking of updating, maybe hold off a bit02:36
*** ricolin has quit IRC02:39
*** uzumaki has joined #openstack-ironic02:56
*** pmannidi has joined #openstack-ironic03:29
*** pmannidi has quit IRC03:29
*** pmannidi has joined #openstack-ironic03:29
*** pmannidi has quit IRC03:34
*** ociuhandu has joined #openstack-ironic03:47
*** ociuhandu has quit IRC03:51
*** ricolin has joined #openstack-ironic03:56
*** k_mouza has joined #openstack-ironic04:16
*** k_mouza has quit IRC04:21
*** mkrai has joined #openstack-ironic04:46
*** jawad_axd has joined #openstack-ironic05:01
*** mkrai_ has joined #openstack-ironic05:22
*** mkrai has quit IRC05:25
*** iurygregory has joined #openstack-ironic06:28
iurygregorygood morning ironic o/06:29
arne_wiebalckGood morning iurygregory and Ironic!06:45
iurygregorymorning arne_wiebalck o/06:46
*** lucasagomes has joined #openstack-ironic07:05
*** mkrai_ has quit IRC07:06
*** rpittau|afk is now known as rpittau07:24
rpittaugood morning ironic! o/07:24
openstackgerritVerification of a change to openstack/ironic failed: Rename redfish_deploy_iso -> deploy_iso  https://review.opendev.org/c/openstack/ironic/+/79001607:26
*** uzumaki has quit IRC07:27
iurygregorymorning rpittau o/07:32
rpittauhey iurygregory :)07:32
*** donnyd has quit IRC07:34
*** gouthamr has quit IRC07:34
*** jungleboyj has quit IRC07:35
*** erbarr has quit IRC07:35
*** mnaser has quit IRC07:36
*** guilhermesp has quit IRC07:36
*** vkmc has quit IRC07:36
*** gmann has quit IRC07:36
*** bbezak has quit IRC07:36
*** bbezak has joined #openstack-ironic07:37
*** jungleboyj has joined #openstack-ironic07:37
*** gouthamr has joined #openstack-ironic07:37
*** mnaser has joined #openstack-ironic07:37
*** vkmc has joined #openstack-ironic07:37
*** erbarr has joined #openstack-ironic07:38
*** donnyd has joined #openstack-ironic07:38
*** guilhermesp has joined #openstack-ironic07:38
*** gmann has joined #openstack-ironic07:38
openstackgerritHitesh Kumar proposed openstack/ironic-ui master: Migrate from testr to stestr  https://review.opendev.org/c/openstack/ironic-ui/+/79232407:39
*** tosky has joined #openstack-ironic07:46
*** rcernin has quit IRC07:52
*** ociuhandu has joined #openstack-ironic07:53
*** dougsz has joined #openstack-ironic08:00
*** dtantsur|afk is now known as dtantsur08:03
*** derekh has joined #openstack-ironic08:07
*** pmannidi has joined #openstack-ironic08:19
jandershey arne_wiebalck iurygregory rpittau08:25
rpittauhey janders :)08:25
arne_wiebalckhey janders o/08:25
jandersquick one: to boot via virtualmedia/cdrom on Dell/HP do I use legacy or UEFI boot?08:25
iurygregoryhey janders o/08:25
dtantsurjanders: in theory - either08:26
rpittaujanders: both should work08:26
jandersthanks dtantsur rpittau08:26
*** paras333 has joined #openstack-ironic08:31
*** ociuhandu has quit IRC08:34
jandersyou have likely seen my earlier message, but if youre on FC33 and haven't updated for few days, may be worth being careful with the current sssd upgrades - they locked me out of my system due to breaking sssd cache08:46
dtantsurugh08:47
iurygregoryO.o08:47
iurygregorywow08:47
jandersnot sure if FC34 is impacted - who knows08:47
iurygregoryI'm using F34 for a while,08:47
iurygregory1month I think08:48
jandersif you're updating, see if there are outstanding sssd pacakages and we can confirm if it's the same version as in fc3308:48
iurygregoryno problems here :D (but normally I do updates weekly)08:48
jandersit is also possible it's somehow to do with my specific setup, but the way errors started popping up during the upgrade (and it kept going) so not too impressed08:49
jandersall fixed now after some hacking though :)08:49
jandersI'll probably do a fresh FC34 install soon, too, just to clear all the outstanding hacks :)08:49
dtantsuris there any way to check if you're affected?08:51
dtantsurlike using su?08:51
jandersyeah sudo broke for me right after the upgrade I think08:51
jandersalthough I rebooted prior to trying so not 100% if it is immediate08:51
dtantsurworks for me for now08:51
jandersthat is a good sign08:52
dtantsuris there a bugzilla to follow?08:52
jandersI have to admit I haven't looked08:52
jandersjust pinged IT guys to see if there is any known sssd backend issue08:52
rpittauI'm also using FC34 now and didn't have issues so far08:52
janderssounds like an incentive for me to jump to 34 :)08:52
rpittauI actually saw an improvement with wifi and memory consumption08:53
jandershttp://pastebin.test.redhat.com/96535208:53
jandersthat was the errors that killed me (and sssd versions in question)08:53
jandersit would be curious how FC34 versions compare - same? much newer?08:54
rpittauoh wow08:54
rpittauwell kernel is 5.11.20 so looks the same08:54
janderslooks like I went 2.4.2 > 2.508:54
rpittauah nvm, it's 300 though08:55
rpittausssd is 2.4.208:55
rpittauat least so far08:55
jandersif 2.5 comes out, there is a small chance of There Being Dragons (TM)08:56
jandersthey might eat your sssd cache08:56
rpittaudo not feed the dragons08:56
jandersno08:57
jandersthey get used to being fed and things get really nasty08:57
jandersfrom your experience, how responsive (or not responsive) are vMedia installs?08:59
*** k_mouza has joined #openstack-ironic08:59
janderstrying to get things going off the https attached vMedia - nodes boot but takes much much longer to boot than what I am used to08:59
jandershowever08:59
*** k_mouza has quit IRC09:00
jandersbenchmarking did show average speeds (2,5,20MB/s)09:00
jandersso not sure if this is normal or kind of slow?09:00
jandersI only really booted servers from physically connected USB drives like this09:00
janders(or pxe)09:00
*** k_mouza has joined #openstack-ironic09:00
jandersHP took 2-3 minutes till I saw the kernel boot messages09:02
janderslooks scary but if it's 5MB/s and 700MB iso, that's not too crazy09:02
jandersgetting a Dell going now09:02
openstackgerritDmitry Tantsur proposed openstack/ironic master: Trivial: comment why we don't check retired in allocations  https://review.opendev.org/c/openstack/ironic/+/79233509:03
dtantsurjanders: I've heard about 7 minutes even09:03
jandersdtantsur thank you, that's reassuring. Is this still workable for what we need it for?09:04
*** rcernin has joined #openstack-ironic09:04
dtantsurit's annoying but workable09:04
iurygregoryI was about to say the 7min scenario hehe09:04
janders(HPC types would run away screaming, but we tend to pick on things that no one else cares about :) )09:04
jandersat what stage can I claim success? If the node boots of https-backed image and the CentOS installer starts, is that good enough?09:04
janderss/of/off09:05
jandersor would we want a successful install?09:05
*** ociuhandu has joined #openstack-ironic09:05
openstackgerritMerged openstack/ironic master: Retrieve BIOS registry from sushy  https://review.opendev.org/c/openstack/ironic/+/79195609:08
*** ociuhandu has quit IRC09:14
*** uzumaki has joined #openstack-ironic09:15
*** ociuhandu has joined #openstack-ironic09:29
*** ociuhandu has quit IRC09:30
*** ociuhandu has joined #openstack-ironic09:31
*** rcernin has quit IRC09:38
*** mkrai has joined #openstack-ironic09:44
*** rcernin has joined #openstack-ironic09:53
*** mkowalski_ has joined #openstack-ironic09:54
dtantsurjanders: it's probably fine at this point10:04
jandersgreat, thank you dtantsur10:18
openstackgerritMerged openstack/ironic master: Rename redfish_deploy_iso -> deploy_iso  https://review.opendev.org/c/openstack/ironic/+/79001610:25
jandersis there any specific setting I need on to be able to boot an R640 off virtual media?10:29
jandersit's attached, I can read contents when booted to the OS, but doesn't show up in boot device list (either for permanent or one off boot menu)10:30
jandersresetting the iDRAC now as I've been DDOSing it for the last week, but I have a sneaking suspicion it may be me doing something wrong10:31
*** pmannidi has quit IRC10:32
*** pmannidi has joined #openstack-ironic10:33
*** ociuhandu has quit IRC10:34
*** ociuhandu has joined #openstack-ironic10:34
*** mkrai has quit IRC10:37
*** ociuhandu has quit IRC10:39
dtantsurjanders: yeah, this is why we have a separate idrac-redfish-virtual-media: it requires on OEM call10:52
jandersright!10:55
jandersI got it to boot via UEFI one-shot boot menu after iDRAC reset10:56
jandersbut got hit by I/O errors10:56
iurygregoryif anyone has some spare spoons to look at https://review.opendev.org/c/openstack/sushy/+/792053 I would appreciate :D10:56
rpittauthere is no spoon10:56
jandersdo you think it may be related to the lack of this call, or likely a separate problem?10:56
dtantsurjanders: dunno about I/O errors10:56
jandersI'll reboot it into the OS and copy the whole vMedia to judge if the connection is solid10:56
dtantsursounds suspicious10:56
jandersyeah10:56
iurygregoryrpittau, hehe10:57
jandersive got another box running of the same URL no worries10:57
dtantsuriurygregory: I'll bookmark it, but probably won't grok in one go10:57
jandersbut who knows, maybe they are in differnet locations (latency etc)10:57
iurygregorydtantsur, np o/ I appreciate =)10:58
*** ociuhandu has joined #openstack-ironic11:04
*** ociuhandu has quit IRC11:16
*** bburns has quit IRC11:21
*** bburns has joined #openstack-ironic11:23
*** ociuhandu has joined #openstack-ironic11:28
*** ociuhandu has quit IRC11:35
*** ociuhandu has joined #openstack-ironic11:39
*** alexmcleod has quit IRC11:50
*** alexmcleod has joined #openstack-ironic11:53
*** k_mouza_ has joined #openstack-ironic11:59
*** k_mouza has quit IRC12:03
*** rcernin has quit IRC12:14
openstackgerritMerged openstack/ironic master: Trivial: comment why we don't check retired in allocations  https://review.opendev.org/c/openstack/ironic/+/79233512:24
*** lbragstad_ has quit IRC12:24
*** lbragstad has joined #openstack-ironic12:31
*** lbragstad has quit IRC12:32
*** lbragstad has joined #openstack-ironic12:33
TheJuliagood morning13:05
rpittaugood morning TheJulia :)13:06
TheJuliajanders: fyi, supermicros run a hybrid bios so there is less of a delineation between the two modes as there is with hpe/dell13:07
*** uzumaki has quit IRC13:09
*** ociuhandu has quit IRC13:11
*** paras333 has quit IRC13:14
cennGood morning TheJulia, rpittau13:16
rpittauhey cenn :)13:16
cennhey!13:16
TheJuliagood morning cenn13:16
TheJuliaOut of curiosity, have we noticed any instability with the bfv job? or anything with hashrings going sideways? https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0c8/792275/1/check/ironic-tempest-bfv/0c8ccd5/controller/logs/screen-ir-cond.txt just seems bizzar13:17
TheJulialooks like neutron also hit a hiccup on the same job https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0c8/792275/1/check/ironic-tempest-bfv/0c8ccd5/controller/logs/screen-q-svc.txt13:18
TheJuliaheh, 50 second gap in logs and then the errors appear13:19
* TheJulia guesses vm got cpu throttled to molasses speeds13:19
dtantsurhi cenn!13:19
cennhi dtantsur :) good afternoon.13:20
*** ociuhandu has joined #openstack-ironic13:23
*** tzumainn has joined #openstack-ironic13:28
*** ociuhandu has quit IRC13:28
*** rloo has joined #openstack-ironic13:33
*** ociuhandu has joined #openstack-ironic13:34
*** jawad_axd has quit IRC13:43
*** ociuhandu has quit IRC13:45
*** ociuhandu has joined #openstack-ironic13:45
*** ociuhandu has quit IRC13:50
*** ociuhandu has joined #openstack-ironic13:56
*** jawad_axd has joined #openstack-ironic14:00
*** jawad_axd has quit IRC14:06
*** k_mouza_ has quit IRC14:13
*** k_mouza has joined #openstack-ironic14:14
*** bburns has quit IRC14:14
*** bburns has joined #openstack-ironic14:16
TheJulia54 failing unit tests, take one down, pass it around, 48 failing unit tests14:19
*** jawad_axd has joined #openstack-ironic14:20
iurygregory\o/14:24
dtantsurnot too bad14:24
iurygregoryyeah14:24
TheJuliamgoddard: you around, have a question w/r/t traits for you14:25
mgoddardTheJulia: not a good time, try again later :)14:25
TheJuliamgoddard: ack14:25
TheJuliaI'm at the point where I'm getting stupidly cryptic errors from the api :)  I guess that is a good or very bad sign14:25
*** uzumaki has joined #openstack-ironic14:38
*** ociuhandu has quit IRC14:40
*** ociuhandu has joined #openstack-ironic14:41
arne_wiebalckWhen reprocessing stored introspection data, how would an action:fail actually be reflected? Would the node state change, would the fault or last_error fields be set, sth else?14:41
*** jawad_axd has quit IRC14:44
*** jawad_axd has joined #openstack-ironic14:44
*** jawad_axd has quit IRC14:44
*** ociuhandu has quit IRC14:46
*** ociuhandu has joined #openstack-ironic14:46
dtantsurarne_wiebalck: only introspection status15:00
arne_wiebalckdtantsur: thanks ... what is the introspection status?15:02
* arne_wiebalck is trying to figure out if the reprocessing actually does something in our deployment :)15:03
dtantsurarne_wiebalck: ironic-inspector has its own API, including listing/getting introspection statuses per node15:03
arne_wiebalckdtantsur: oh, ok, is there client support to query this?15:04
dtantsurarne_wiebalck: baremetal introspection list / show <node>15:04
JayFFYI; I will not be at SPUC for US time tomorrow. It overlaps with a (rare) companywide Friday meeting.15:06
arne_wiebalckdtantsur: boom! that was it, ty! (list + grep for UUID show the failed rule)15:07
*** lucasagomes has quit IRC15:08
arne_wiebalckdtantsur: (there is no show from what I see)15:08
dtantsurarne_wiebalck: mm, yeah, it's called $ baremetal introspection status15:09
dtantsurfor some reason15:09
* arne_wiebalck owes dtantsur a beer as he looked at the list of available commands twice and did not see this15:11
arne_wiebalckdtantsur: thanks!15:11
arne_wiebalckdtantsur: and it works: reprocessing fails the rule and flags the node15:11
*** uzumaki has quit IRC15:24
*** sdanni has joined #openstack-ironic15:25
TheJuliaJayF: fun :(15:28
TheJuliaso I figured out how I broke the api15:28
iurygregoryI hope you broke the API locally ...15:29
TheJuliawe have default list views that do reconciliation *in* the API and still pass around whole objects, or at least expect it.15:29
TheJuliathese do have lists of fields, but they don't match the objects so things go *kaboom*15:30
iurygregoryouch15:30
TheJuliaat least, they go kaboom when sqlalchemy has not returned a giant list of all node data15:30
NobodyCamTop of the morning Ironic folks15:34
iurygregorymorning NobodyCam15:35
NobodyCammorning iurygregory15:35
rpittauhey NobodyCam :)15:38
NobodyCamhey hey rpittau good Morning15:38
openstackgerritJulia Kreger proposed openstack/ironic master: Only return the requested fields  https://review.opendev.org/c/openstack/ironic/+/79227415:42
openstackgerritJulia Kreger proposed openstack/ironic master: Hand object a list of fields  https://review.opendev.org/c/openstack/ironic/+/79227515:42
*** ociuhandu has quit IRC15:46
rpittaugood night! o/15:50
*** rpittau is now known as rpittau|afk15:50
*** dtantsur is now known as dtantsur|afk16:23
arne_wiebalckdtantsur: TheJulia: I continue the burn-in journey with disks and I have noted some thoughts in task 42384 of https://storyboard.openstack.org/#!/story/2007523. The options are in decreasing order of my preference. Let me know if you have any comments or ideas :)16:26
*** ociuhandu has joined #openstack-ironic16:30
*** dougsz has quit IRC16:35
*** ociuhandu has quit IRC16:36
arne_wiebalckbye everyone o/17:02
*** derekh has quit IRC17:08
TheJuliatake one down, pass it around, about a thousand tests failing17:28
TheJuliaq:q17:29
openstackgerritIury Gregory Melo Ferreira proposed openstack/sushy master: EventService support  https://review.opendev.org/c/openstack/sushy/+/79205318:23
iurygregorybfournie, tks for the review!18:24
bfournieiurygregory: no problem, thank you too!18:24
TheJuliahmm, down to 7 failed tests18:32
*** dustinc has joined #openstack-ironic19:28
*** ociuhandu has joined #openstack-ironic19:55
*** ociuhandu has quit IRC20:00
*** jawad_axd has joined #openstack-ironic20:01
*** jawad_axd has quit IRC20:06
*** rcernin has joined #openstack-ironic20:07
*** mkowalski has quit IRC20:09
*** mkowalski has joined #openstack-ironic20:09
*** rcernin has quit IRC20:13
*** rcernin has joined #openstack-ironic20:50
*** dustinc has quit IRC21:38
*** rcernin has quit IRC21:49
NobodyCamHumm we don't pay attention to `TARGET_BRANCH` in our Devstack plugin :(21:56
*** dansmith has quit IRC22:02
openstackgerritJulia Kreger proposed openstack/ironic master: Only return the requested fields  https://review.opendev.org/c/openstack/ironic/+/79227422:02
openstackgerritJulia Kreger proposed openstack/ironic master: Hand object a list of fields  https://review.opendev.org/c/openstack/ironic/+/79227522:02
openstackgerritJulia Kreger proposed openstack/ironic master: WIP: API to pass fields to node object list  https://review.opendev.org/c/openstack/ironic/+/79229622:02
TheJuliaNobodyCam: what are you trying to do?22:02
NobodyCamdeploy stable/queens for testing22:03
NobodyCamit was a easy workaround22:03
TheJulialikely not in our plugins because nobody develops on older branches22:04
TheJuliabut we do test on it22:04
NobodyCamgot dependency error, changed to ironic folder, did `git check stable/queens`, reran stack.sh22:04
TheJuliayup22:04
TheJuliathat would do it22:04
TheJuliabecause zuul does the proper checkouts for us22:04
NobodyCam;p22:05
NobodyCamhehehe22:05
NobodyCamhow's your day going22:05
*** dansmith has joined #openstack-ironic22:06
TheJuliaeh, my day is going okay.22:12
TheJuliafighting with api performance issues22:12
NobodyCam:) oh the good fight!22:13
TheJuliaheh22:13
TheJuliaIt is a complex and interwoven fight of technical debt22:14
NobodyCamyea. years of build up22:16
TheJuliaAlso questioning preconceptions is necessary22:18
TheJuliaI've looked at a few things and gone "why in the world did we do that?!"22:18
*** rloo has quit IRC22:19
*** paras333_ has joined #openstack-ironic22:23
*** dansmith has quit IRC22:25
*** dansmith has joined #openstack-ironic22:25
*** paras333_ has quit IRC22:27
*** rcernin has joined #openstack-ironic22:42
*** ociuhandu has joined #openstack-ironic23:14
openstackgerritJulia Kreger proposed openstack/ironic master: Secure RBAC - Efficent node santiziation  https://review.opendev.org/c/openstack/ironic/+/79014223:14
TheJuliastevebaker: so I realized with ^, I could just strip the node first.23:15
TheJuliawhich means it is disjointed from object work23:15
*** ociuhandu has quit IRC23:18
*** k_mouza has quit IRC23:26
*** tosky has quit IRC23:52
*** pmannidi has quit IRC23:54
*** sdanni has quit IRC23:55
*** pmannidi has joined #openstack-ironic23:57

Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!