Tuesday, 2023-12-19

rpittaugood morning ironic! o/08:25
dtantsurJayF: Partly commented on Mastodon but 1) Stream is not more *-walled than Ironic (we also don't publish source RPMs, do we?), 2) eventlet is only in Stream 9+, before that it was in RDO/OSP (your link clearly has openstack-xena in it), 3) I suspect only OSP got patched (yep, that's on us).08:51
dtantsurJayF: I believe what we did (at least that's what I proposed) is removing this line https://github.com/eventlet/eventlet/blob/v0.33.3/eventlet/green/ssl.py#L10809:32
rpittaudtantsur: in the failing dhcp bifrost job there is no dnsmasq.d config dir https://8b2692338bc36f15b700-3cea8240e395cd42011245fbdc9f1a59.ssl.cf5.rackcdn.com/903755/2/check/bifrost-integration-dhcp-centos-9/6f12f85/logs/dnsmasq_config/index.html09:42
rpittauapparently the dir is lost at some point09:42
dtantsurO_____O09:42
dtantsurIs it present in the revert?\09:42
rpittauyep09:43
dtantsuroh ffs09:43
rpittauthe Setup DHCP for nodes step never happens09:43
dtantsuris it skipped or just.. absent?09:43
rpittaustill have no idea why09:43
rpittauit's absent09:43
dtantsurokay, so maybe the new collection does not set some variables we rely on09:43
dtantsurthat's a lead at least09:43
rpittauyeah09:43
rpittauI'll keep digging09:44
dtantsurit's only conditional on inventory_dhcp though..09:44
rpittauoO09:44
rpittauwould be good to merge https://review.opendev.org/c/openstack/bifrost/+/903137 to remove some noise TheJulia JayF iurygregory please when you have a moment check that, thanks!09:45
dtantsurhooooold on09:46
* dtantsur slowly blinks09:46
dtantsurah09:47
dtantsurOOOOOOOOH09:47
dtantsurokay, I got it. patch incoming, you'll see what is happening :)09:47
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Configure static DHCP before automated cleaning  https://review.opendev.org/c/openstack/bifrost/+/90393910:04
dtantsurrpittau: ^^^10:04
rpittauoh!10:11
rpittausubtle10:12
rpittaufrom a glance it looks good10:13
opendevreviewMerged openstack/ironic master: Fixes Raid creation in iLO6 and other BMC with latest schema  https://review.opendev.org/c/openstack/ironic/+/90274410:33
opendevreviewDmitry Tantsur proposed openstack/ironic-lib master: Compatibility with zeroconf 0.129.0  https://review.opendev.org/c/openstack/ironic-lib/+/90381510:50
dtantsurzigo: ^^^10:50
zigodtantsur: Thanks ! :)10:58
zigodtantsur: So, with your version of the patch, it's going to work with all versions of zeroconf ?10:59
iurygregorygood morning Ironic11:02
dtantsurzigo: that's my expectation11:11
dtantsurERROR: Could not find a version that satisfies the requirement ansible<8,>=711:15
dtantsurI hope it's transient...11:15
dtantsurrpittau: btw is there a reason we didn't migrate straight to ansible 9?11:17
dtantsur(meanwhile, one DHCP job passed!)11:17
rpittaudtantsur: the only reason is to not skip a version 11:22
rpittauI was actually planning to go up, but one by one11:22
dtantsurah, okay11:22
iurygregoryrpittau, +W11:25
rpittauI'll add a patch for that on top of the dhcp fix11:25
rpittauwondering if we'll see the disk space issue again 11:26
dtantsuryep, we're seeing it11:26
dtantsurI'm checking if we can reduce the testing VM number..11:26
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Slightly reduce the disk size when testing DHCP  https://review.opendev.org/c/openstack/bifrost/+/90394811:28
dtantsurrpittau: let's see ^^11:28
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Update ansible linters to current version used  https://review.opendev.org/c/openstack/bifrost/+/90394911:33
dtantsurrpittau: the other "fun" issue we have is https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_c41/903939/1/check/bifrost-integration-dibipa-debian-centos-9/c4186e5/job-output.txt11:34
dtantsurI'm absolutely puzzled by it. Short of just adding retries to Ironic, I have no other ideas.11:34
rpittauoh snap, do we want to wait for the next ansible update? I can propose the patch and just leave it there instead of adding to the chain11:34
dtantsurwhy? we can totally do the update, I don't think the ansible version has anything to do with these problems11:35
dtantsurjust make sure the CentOS DHCP job passes :)11:35
rpittauyeah, I just thought to not add more stuff on top, but I guess it's not really going to interfere11:37
rpittauthanks iurygregory btw just saw it :)11:38
iurygregoryyw =)11:38
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x  https://review.opendev.org/c/openstack/bifrost/+/90395011:40
rpittaulet's see how all goes11:40
opendevreviewDmitry Tantsur proposed openstack/bifrost master: CI: don't send the raw command to disable the boot timeout  https://review.opendev.org/c/openstack/bifrost/+/90395111:47
dtantsurrpittau: another guess ^^11:47
rpittaummmm ok11:50
rpittaulol ansible linters is not connected to the ansible version, I always forget about that :/11:52
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Update ansible linters to current version used  https://review.opendev.org/c/openstack/bifrost/+/90394911:53
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x  https://review.opendev.org/c/openstack/bifrost/+/90395011:53
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x  https://review.opendev.org/c/openstack/bifrost/+/90395011:55
opendevreviewRiccardo Pittau proposed openstack/bifrost master: Uplift default Ansible version to 8.x  https://review.opendev.org/c/openstack/bifrost/+/90395011:55
dtantsurrpittau: make sure to check my comment in case it gets hidden by the updates11:55
rpittauyeah jsut saw it11:55
dtantsurrpittau: I have a nasty suspicion. The jobs failing with pip errors are on focal. I wonder if it's because of Python version and your bump of Ansible :(12:02
rpittauI was looking at focal too12:04
rpittauand I think you're right12:04
dtantsuryeah, I'm afraid so12:04
dtantsurWe need to either go back to older ansible or drop focal12:04
rpittaufocal is supported until April 2025, but it's now more than 3 years old12:07
rpittauI'm more on dropping it than keeping an old unsupported ansible, but we need more opinions I guess12:07
dtantsurrpittau: ansible 6 is EOL?12:08
rpittauFYI ansible 7 support has ended 6 months ago12:08
rpittauansible 6 1 year aog12:09
dtantsurOuch. Then yes, so long Focal. Should I do the change or do you want to?12:09
rpittauI can take care of it, but after lunch :)12:10
dtantsurEnjoy then, I'll prepare the patch in the meantime12:11
opendevreviewDmitry Tantsur proposed openstack/bifrost master: Drop support for Ubuntu Focal  https://review.opendev.org/c/openstack/bifrost/+/90397512:18
dtantsurrpittau: ^^12:18
dtantsurNisha_Agarwal: hi, do you know anything about HPE Compute Scale-Up server 3200?12:21
dtantsurSomeone asks me if its "partitions" support Redfish properly. They get The attribute Actions is missing - is it the issue you mentioned yesterday?12:22
Nisha_Agarwaldtantsur, yes12:30
dtantsurNisha_Agarwal: ah, I see. Do you have a launchpad bug for it yet?12:30
Nisha_Agarwalthe HPE Compute Scale-Up servers are partionable servers 12:31
Nisha_Agarwaland each partition serve as a baremetal server12:31
Nisha_AgarwalSo when we enroll it in ironic we enroll the partition12:31
Nisha_Agarwaldtantsur, No. We are planning to raise bugzilla. 12:32
Nisha_AgarwalWe can raise launchpad bug id as well12:32
dtantsurNisha_Agarwal: could you file an upstream bug too?12:32
dtantsurYeah, yes please. It's not just a problem for RH OSP.12:32
Nisha_AgarwalYes sure .12:33
Nisha_AgarwalFor more details you can refer to  https://github.com/HewlettPackard/sdflexutils/wiki and https://github.com/HewlettPackard/sdflex-ironic-driver/wiki12:34
Nisha_AgarwalThe HPE Compute Scale-Up server 3200 is a next series of these servers12:35
Nisha_Agarwaland they are already certified with redfish driver in RHOSP 1612:35
dtantsurack, thanks12:35
dtantsurhave you had a chance to test them with the most recent Ironic, not just what OSP has?12:35
Nisha_AgarwalWe tested Scale-Up 3200 without RHOSp with Bobcat release and that worked fine(obviously in non-containerized environment)12:36
dtantsurah, I see. so maybe an upstream bug does not make much sense..12:37
Nisha_Agarwaldtantsur, but this is a timing issue and a day-one bug in sushy12:39
dtantsurYeah, I guess what I'm curious is whether the bug is present in the master version of sushy12:39
Nisha_Agarwalsession auth-token expires and is an invalid token before the GET call is done and even after receiving the 401 status the error response is sent to the caller12:40
Nisha_AgarwalIt is there12:40
dtantsurOkay, clear now, thanks!12:40
Nisha_Agarwalwe hit it less in latest sushy , but it is there12:40
Nisha_Agarwaldtantsur, refer to the pdb logs here12:41
Nisha_Agarwalhttps://paste.openstack.org/show/bKBx58BeU1dYEyXPnJdJ/12:41
dtantsurNisha_Agarwal: we definitely have re-authentication code, I wonder what goes wrong there12:46
Nisha_Agarwaldtantsur, Yes it gets the auth token when it gets the access error....but after that when it attempts the request call, that time the session auth-token is expired or you can say it becomes invalid12:48
dtantsurNisha_Agarwal: is it valid for milliseconds? Oo12:48
Nisha_Agarwalyes if we use curl and get the auth token, and use that session auth token for further curl requests, the token is valid for very long12:49
dtantsurthen why does it expire for us?12:49
Nisha_Agarwalso probably something to do with the library which is used by sushy12:49
Nisha_Agarwalor probably sushy needs to allow to set the session timeout12:50
dtantsurI cannot see how a library will be related..12:50
dtantsurdo you set the timeout when curl'ing?12:50
Nisha_Agarwalcurl -gikx "" -X POST -H 'Content-Type: application/json' -d '{"UserName":"admin","Password":"password"}' https://<ip>/redfish/v1/SessionService/Sessions/12:53
fricklerdtantsur: Nisha_Agarwal: not sure how your actual error looks like, but maybe https://bugs.launchpad.net/openstacksdk/+bug/2046645 is related? (just matching on "auth token" as keyword)12:53
Nisha_Agarwalwe use this command line manually when we test the servers manually12:53
Nisha_Agarwalit doesnt look related...12:55
opendevreviewMerged openstack/bifrost master: Use fully qualified name for ipaddr filter  https://review.opendev.org/c/openstack/bifrost/+/90313712:56
opendevreviewDmitry Tantsur proposed openstack/sushy master: Handle exceptions after re-authentication  https://review.opendev.org/c/openstack/sushy/+/90397912:57
dtantsurNisha_Agarwal: this won't help your problem ^^^ but will at least make the actual error clear12:57
Nisha_Agarwaldtantsur, ok we will apply this and see...12:58
dtantsurfrickler: no, I'm afraid it's a different area of code13:00
Nisha_Agarwaldtantsur, the issue whch u mentioned above https://bugs.launchpad.net/openstacksdk/+bug/2046645 can cause failure for sushy HTTP calls as well?13:04
dtantsurNisha_Agarwal: we don't use openstacksdk in any way in sushy13:32
Nisha_Agarwalhmmm13:34
dtantsurI'd start with checking if the re-authentication code actually works correctly13:36
dtantsuri.e. that it does not accidentally reuse the stale token13:36
Nisha_Agarwaldtantsur, i picked up ur above patch and see that the sushy NoValidSession error13:37
Nisha_Agarwalobviously in pdb :)13:37
dtantsurokay, that's an improvement :)13:37
Nisha_Agarwalyes 13:37
dtantsurNisha_Agarwal: how do you reproduce the bug? Does it just happen immediately for you when using sushy?13:38
Nisha_Agarwaldtantsur, we will file bugzilla and launchpad bug both13:38
Nisha_AgarwalYes we hit this issue 5 out 7 times in 3.7.6 sushy version13:39
Nisha_Agarwalwithout pdb13:39
dtantsurNisha_Agarwal: and on recent sushy how often?13:40
Nisha_Agarwalin latest sushy 4.7.0 also same frequency13:40
Nisha_Agarwalits the matter of luck is what we saw13:40
Nisha_Agarwalcan we configure the timeout ?13:41
Nisha_Agarwalis it possible?13:41
dtantsurNisha_Agarwal: the timeout is on the server side, not on the ours..13:42
opendevreviewDmitry Tantsur proposed openstack/sushy master: Add release version to release notes  https://review.opendev.org/c/openstack/sushy/+/90398413:42
Nisha_Agarwaldtantsur, ok13:43
dtantsurNisha_Agarwal: that's all we have in sushy: https://opendev.org/openstack/sushy/src/branch/master/sushy/resources/sessionservice/sessionservice.py#L111-L13013:44
Nisha_Agarwaldtantsur,  i know...i have seen the code already13:44
Nisha_Agarwaldtantsur, what i observed is that https://opendev.org/openstack/sushy/src/branch/master/sushy/connector.py#L155-L18013:46
Nisha_Agarwalfirst time it gets the token when it is not having valid session13:46
Nisha_Agarwalbut next time it doesnt get it again13:47
Nisha_Agarwalso with ur patch now it fails there as an invalid session13:47
dtantsurOkay, so is it fair to say that the authentication simply does not work?13:47
Nisha_Agarwalsometimes it works, and sometimes it is an invalid session token13:48
Nisha_Agarwaldtantsur, i will have to take half an hr break13:48
Nisha_Agarwalwill connect back after half an hr13:48
dtantsursure, I'm here for the time being13:49
opendevreviewMerged openstack/ironic master: Documentation: tuning worker threads  https://review.opendev.org/c/openstack/ironic/+/90278313:52
opendevreviewDmitry Tantsur proposed openstack/virtualbmc master: Handle any error from libvirt operations  https://review.opendev.org/c/openstack/virtualbmc/+/83510413:53
dtantsurrpittau: I wonder if ^^^ gives us some clarity13:54
opendevreviewDmitry Tantsur proposed openstack/bifrost master: DNM test the CI  https://review.opendev.org/c/openstack/bifrost/+/90367713:55
opendevreviewDmitry Tantsur proposed openstack/bifrost master: DNM test the CI  https://review.opendev.org/c/openstack/bifrost/+/90367713:59
*** dviroel_ is now known as dviroel14:23
samcat116networking-baremetal bugs go on Launchpad correcT?14:38
samcat116The launchpad description says to file them on storyboard still14:39
iurygregorysamcat116, launchpad =) 14:43
iurygregorywe probably need to update the description cc JayF ^14:44
JayFI'm pretty sure anyone in ironic drivers can do that, but I'll look at it when I get to a PC if it's not done by then14:44
dtantsurJayF, samcat116, iurygregory, fixed14:46
opendevreviewSven Kieske proposed openstack/networking-baremetal master: don't force amqp_auto_delete for quorum queues  https://review.opendev.org/c/openstack/networking-baremetal/+/90399515:29
rpittausee you tomorrow! o/16:10
opendevreviewSven Kieske proposed openstack/networking-baremetal master: don't force amqp_auto_delete for quorum queues  https://review.opendev.org/c/openstack/networking-baremetal/+/90399516:50
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: [WIP] Support several API and Inspector URLs  https://review.opendev.org/c/openstack/ironic-python-agent/+/90399917:01
dtantsuriurygregory: have a minute? I've noticed that inspector retry tests in IPA take a long time (22 seconds for the longest), could you take a look why?17:04
dtantsurI suspect the change to exponential retries could cause it, but I haven't dived in17:05
iurygregorydtantsur, sure will look at it17:06
iurygregorywe can probably mock things to be 1s I think17:06
JayFfyi https://blueprints.launchpad.net/nova/+spec/ironic-guest-metadata was approved in the nova meeting today \o/17:16
JayFshould be the piece we need to enable automatic_lessee for Ironic17:16
JayFerm, for nova-integrated Ironi17:16
dtantsurmore instance_info.....17:16
JayFdtantsur: I think of it like ... get all the remaining metadata we want in there, then lock down the schema like we did for port.local_link_connection17:17
dtantsurThe biggest thing that upsets me is how all this further complicates the future deployment API17:18
JayFSo I did say, explicitly, Ironic cores had to approve this and if not we'd have to revisit in nova, so it's not a predone thing :)17:19
JayFbut it's not unreasonable to think that a deployment API needs a place for guest metadata of some kind17:19
dtantsurI have no objections to instance name (will be Deployment name), user_id/project_id17:19
dtantsurFlavor name.. ugh.. IP addresses UGGGHHHHH17:19
JayFflavor is a longtime operator ask17:20
JayFfor people using integrated ironic/nova17:20
JayFIP addresses I am also on the fence for17:20
dtantsurI'm curious what case they have in mind17:20
JayFthey who?17:20
dtantsuroperators that ask17:20
dtantsurfor flavor name in instance_info17:20
JayF"What flavor did someone request when booting this?"17:20
dtantsurWhy cannot they look that up in the instance?17:20
JayFIn the cases I've personally wanted it, I wanted it in the ironic notification tbh17:21
JayFthe other asks; I don't know17:21
dtantsurEsp. since flavor name is super hairy: it can be deleted or completely changed in-between17:21
JayFoooh that's a pretty good point17:21
dtantsurThis is, by the way, why newer Nova microversions don't expose flavor name17:21
* dtantsur knows that because rust-openstack17:21
JayFheh17:21
JayFdtantsur: I can be +1 to dropping flavor name with that context17:23
dtantsurI can imagine use cases for IP addresses, but it's also really scary. Then we're one step away from standalone users wanting to set them, and we're down a rabbit hole..17:23
dtantsurYeah. If nova no longer exposes flavor name, we probably shouldn't either.17:23
iurygregorydtantsur, before: Ran: 33 tests in 22.6581 sec. after: Ran: 33 tests in 4.0256 sec. (tox -e py3 -- test_inspector)17:24
JayFthe hell did you fix, holy cow!17:25
dtantsur\o/17:25
iurygregoryjust adding something to configure the max time in the exponetial 17:25
iurygregorylol17:25
iurygregoryand setting to 1s in the tests lol17:25
JayFbwahaha17:25
JayFwhy do our tests take 1 hour 13 seconds?17:25
JayFHey boss, I commented out that "sleep(1h)"17:26
iurygregory:D17:26
iurygregory@mock.patch.object(inspector, '_RETRY_WAIT_MAX', 1) *MAGIC*17:26
JayFdtantsur: thanks for that feedback, sounds like you saved me from a footgun17:26
dtantsursure thing :)17:26
JayFdtantsur: updated to indicate flavor uuid, per sean mooney's feedback, and updated with a note that we can basically send *less* fields if we want when implementing17:27
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic-python-agent master: Fix inspector retries to not take a long time  https://review.opendev.org/c/openstack/ironic-python-agent/+/90402117:30
iurygregorydone =)17:30
iurygregorybrb 1-117:31
dtantsurthanks!17:32
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: Support several API and Inspector URLs  https://review.opendev.org/c/openstack/ironic-python-agent/+/90399917:59
opendevreviewDmitry Tantsur proposed openstack/ironic-python-agent master: Make inspection URL optional if the collectors are provided  https://review.opendev.org/c/openstack/ironic-python-agent/+/90402618:32
opendevreviewMerged openstack/ironic-python-agent master: Fix inspector retries to not take a long time  https://review.opendev.org/c/openstack/ironic-python-agent/+/90402121:41
JayFA gentoo dev pointed me at a python upgrader tool, supposed to upgrade code to the latest python versions you tell it to, remove old idioms, etc21:53
JayFCurious to see if I can get some usefulness out of it21:54
iurygregoryinteresting21:55
iurygregoryif it updates unit tests / code to work on the right python version you can apply to eventlet lol21:56
iurygregorysorry, couldn't resist to think how this would play on it21:56
JayFoh, you read my mind, essentially21:57
JayFthere was a conversation about the ethics of pulling versions from setup.cfg without pulling compat code21:57
JayF(mainly the argument was, if it half-works, someone could assume it fully works)21:57
iurygregoryyeah21:58
JayFbut it was pointed out to me by Eli (meson dev) that tools exist -- like `ruff` which do some of that work for you, so I'm curious and digging in now, using Ironic/IPA as examples b/c I'm more familar21:58
JayFbut if the tool works well a PR to eventlet is 100% the end goal21:58
JayFalthough the more I think about it, we have like, 100 openstack repos21:58
JayFgetting a config that works for a tool like this could be extremely useful21:58
iurygregoryyeah21:59
iurygregoryagree ++21:59
opendevreviewJay Faulkner proposed openstack/ironic-python-agent master: Change code to >=python3.8 compatability  https://review.opendev.org/c/openstack/ironic-python-agent/+/90404122:29
JayFignore ^, just science as mentioned22:43
JayFand it looks like ruff would have to be put in as a linter (like you'd use a tool like black) to be generally useful for us in most cases22:43
JayFhttps://github.com/astral-sh/ruff/issues/9203 at least I got to generate an upstream bug by proxy lol22:49
tonybJayF: So ruff can be told to "reformat" code to be compatible with a minimum python version (via target-version?)22:59
JayFyes22:59
JayFMy assessment is: it's pretty awesome but also pretty immature, as seen by the above issue22:59
JayFand that it is written with the assumption you put it in CI and run it, similar to how you would a tool like black23:00
JayFso it's going into my toolbox with two use cases:23:00
iurygregoryhow it knew you used in the IPA code base? O.o23:00
iurygregoryWOW23:00
JayF1) Like Eli used it here (the person in Gentoo community who showed it to me) to do eventlet https://github.com/eventlet/eventlet/pull/85323:00
JayFwhich needed a LOT of heavy lifting so the manual work around the automation was worth it23:01
JayFiurygregory: Eli is a friend from another community; the person who showed me the tool and who I griped to when it did that LOL23:01
tonybJayF: Okay. If you could share a "quick'n'easy" here's how you run ruff on $project that'd be really cool.23:01
iurygregoryJayF, oh23:01
tonybUnderstanding it's new and results will vary23:01
JayFtonyb: what project you thinking of?23:01
JayFtonyb: let me PR a config up for you as a demo23:01
JayFas like a tox target, not even PR'ing the results23:01
tonybJayF: I'd probably start with something I know like requirements23:03
JayFbasics are:23:03
JayF (1) add config to .ruff.toml or pyproject.toml in root of repo (https://docs.astral.sh/ruff/configuration/)23:03
JayF (2) run `ruff check` from the root of the project23:03
JayF (3) run `ruff check --fix` from the root to fix the stuff; and `ruff check --fix --unsafe-fixes` if you're feeling spicy23:03
JayFtonyb: my .ruff.toml for that run was basically just: `target-version = "py38"\n [lint]\n select = ["UP"] \n ignore = ["UP031"]`23:04
tonybJayF: Perfect23:05
JayFI had to fix expected for like, 7 unit tests23:08
JayFwhere it removed the mode from open(/blah, r) and didn't remove it from some of our mocks23:08
JayFI've wanted to do an autoformatter proposal for ironic for a while, this tool might push me over the edge  to do it if I can get a loose consensus before I do most of the work23:09
JayFespecially if you like the tool too and would consider adoption23:09
JayFI'll note it is written in rust; I know some people dislike that for irrational (or rational-to-them reasons :P)23:10
tonybI'll play with it and see where it takes us.23:11

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!