Wednesday, 2024-11-20

opendevreviewcid proposed openstack/ironic master: Fix double JSON encoding of error message  https://review.opendev.org/c/openstack/ironic/+/93179506:53
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Test double encoding of error message  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93574006:53
opendevreviewcid proposed openstack/ironic-tempest-plugin master: Fix test to not expect double-JSON-encoded errs  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93254406:53
rpittaugood morning ironic! o/07:37
opendevreviewMerged openstack/tenks stable/2.0: Remove deprecated use of include  https://review.opendev.org/c/openstack/tenks/+/93562408:09
opendevreviewOpenStack Release Bot proposed openstack/tenks stable/2.0: Update .gitreview for stable/2.0  https://review.opendev.org/c/openstack/tenks/+/92715908:31
opendevreviewVerification of a change to openstack/ironic master failed: Use specific fix-commit from dnsmasq  https://review.opendev.org/c/openstack/ironic/+/93569908:57
opendevreviewPierre Riteau proposed openstack/tenks stable/2.0: Update TOX_CONSTRAINTS_FILE for stable/2.0  https://review.opendev.org/c/openstack/tenks/+/92716013:53
lajoskatonaHi, sorry what is the zoom meeting passcode? for some reason zoom stopped me at this point to join....15:10
luisfdezLink in etherpad: https://etherpad.opendev.org/p/ironic-networking15:11
lajoskatonaluisfdez: after I copied the link to zoom it asks me for passcode15:13
luisfdez67449515:15
dtantsurlajoskatona: you sure you did not miss any characters?15:15
dtantsurah, this15:15
lajoskatonadtantsur: it's working now, thanks15:18
shermanmTheJulia: thanks for putting this together! If you've got links to where I should look for writing up a reference architecture, could you shoot an email to shermanm@uchicago.edu?  I'm on conference wifi so I'll probably miss IRC messages16:15
JayFSomeone tell shermanm about irccloud :P 16:19
TheJuliauhoh, we have his email address now <insert evil laugh here>16:19
TheJuliaI think the networking call was at least semi-useful16:27
opendevreviewVerification of a change to openstack/ironic master failed: Use specific fix-commit from dnsmasq  https://review.opendev.org/c/openstack/ironic/+/93569916:27
rpittaugood night! o/17:13
JayF\o17:15
opendevreviewPierre Riteau proposed openstack/tenks stable/2.0: Update TOX_CONSTRAINTS_FILE for stable/2.0  https://review.opendev.org/c/openstack/tenks/+/92716018:17
cardoegah there was an ironic networking call?18:32
opendevreviewcid proposed openstack/ironic master: Fix double JSON encoding of error message  https://review.opendev.org/c/openstack/ironic/+/93179518:44
JayFhttps://bugs.launchpad.net/ironic-ui/+bug/2089029 uh-oh.19:02
JayFLooks like our run of "ironic-ui just works so we keep it alive" is coming to an end.19:02
opendevreviewMerged openstack/ironic master: Use specific fix-commit from dnsmasq  https://review.opendev.org/c/openstack/ironic/+/93569919:14
JayFSo, I'm looking at helping cid with getting kea testable in devstack, and I'm extremely confused about something: I'm trying to determine if and where the dnsmasq dhcp provider is tested in Ironic.19:27
JayFFor ironic-standalone job, we do not set IRONIC_DHCP_PROVIDER, which means it falls back to the default of neutron, which doesn't seem like it should work for ironic standalone ... but then looking in the logs, we run q-dhcp even in standalone19:28
JayFtl;dr I'm fairly convinced there are no devstack-based jobs that use/test dnsmasq dhcp interface19:28
JayFis this correct? am I missing something?19:29
JayFI'm digging through the bifrost jobs, and it looks almost like the jobs that test dhcp are testing ironic-inspector dhcp, and not ironic-proper19:33
JayFIf someone could spare a few minutes sometime this week or next to help cid and I unravel what is and is not being tested in terms of DHCP, it'd be very helpful to our next steps.19:33
opendevreviewAdam McArthur proposed openstack/ironic master: api: Introduce new mechanism for API versioning  https://review.opendev.org/c/openstack/ironic/+/92891919:48
opendevreviewAdam McArthur proposed openstack/ironic master: api: Add schema validation framework  https://review.opendev.org/c/openstack/ironic/+/92892019:48
opendevreviewAdam McArthur proposed openstack/ironic master: WIP: api: Add schema for allocations API  https://review.opendev.org/c/openstack/ironic/+/92892119:48
opendevreviewAdam McArthur proposed openstack/ironic-tempest-plugin master: Testing bad microversions on v1/allocations  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93582719:58
opendevreviewAdam McArthur proposed openstack/ironic-tempest-plugin master: Testing bad microversions on v1/allocations  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93582719:59
JayFWe've had some metal3-integration jobs failing due to docker hub rate limits; I was going to propose we mark it non-voting but I wanna see if https://review.opendev.org/c/openstack/project-config/+/935725 helps first (cc adamcarthur5 )22:08
TheJuliaJayF: I guess we could amend the read policy to a matter of do we redact conductors on a reply or not22:12
TheJuliabut that is a solid sign of minimal use22:12
JayF*blink*22:13
JayFI think I've lost context on whatever you're asking?22:13
JayFunless it's about ironic-ui and I just can't make the leap you're making22:13
TheJuliaironic-ui trying to get the list of drivers in the ironic deployment22:13
JayFah, I see22:13
JayFit's trying to do validation it can't do?22:13
TheJuliaits trying to make a pick list for the user22:13
TheJuliabut it can't get the data because with the new model we went and dialed that access back because it provides strucutral insight into the deployments, including conductor hostnames if memory serves22:14
JayFDo we have any way to limit a given project to a specific conductor group or driver set?22:14
TheJulianot really22:14
TheJuliathat would also require upfront mapping22:15
JayFif not; it seems like a good fix for this (and this is probably what you meant) would be a microversion where we just return available drivers and no other information (redact the conductors)22:15
JayFI think I've caught up to you22:15
JayFlol22:15
JayFcid: ^ that might be an RFE you could attach to that bug we triaged during your 1:122:16
JayFcid: if so, it'd be a good first change for your friend who wanted to get involved :) 22:17
TheJuliawe can basically change the overall role and make it admin only or something for the conductors22:17
TheJuliaor system access only22:17
TheJuliathe shifts in meaning in the overall rbac model don't help us here at all22:17
JayFmy thought was: system scope manager/admin -> you get unredacted conductors22:18
JayFproject scope manager/admin -> you get redacted conductors but driver names22:18
JayFI thought that's what you were suggesting?22:18
JayFand service ... well I don't really get the idea of service scope so probably treat like admin :D 22:18
dkingI've been a bit out of the loop lately, but I just went to build a new image, and I'm getting import errors from ironic_python_agent. It's a change that happened since last week.22:18
JayFlogs?22:19
dkingSorry to pop in the middle of the other conversation. I accidentally hit send.22:19
cidJayF: Anything RBAC may not be a good first issue, from experience22:19
JayFyou would likely know better than most :)22:20
cidI'd like to think so :D. I will take a closer look later.22:20
JayFeither way, one of us three should summarize this conversation ... essentially we /could/ allow less-privledged users to see drivers without seeing other valuable system information (like conductor hostnames)22:21
cid++22:21
JayF that would at least let us move the bug outta ironic-ui into something more maintained lol22:24
dkingWhile folks are still here, if I'm not interrupting and If anybody has a moment, were there any changes recently that folks know about which were likely to cause issues with disk_utils or qemu_img in ironic_python_agent?22:25
dkingThe first error, from where I had a version locked to 9.12.0 was: ImportError: cannot import name 'qemu_img' from 'ironic_lib' After removing the version lock, at version 9.14.0, I get a traceback shortly after the same import ending in: oslo_config.cfg.NoSuchOptError: no such option disk_utils in group [DEFAULT]22:27
TheJuliaJayF: that was what I was thinking22:30
JayFdking: I suspect you're mixing versions of IPA/Ironic-lib22:31
TheJuliacid: a test or two can do amazing things. On your original chagne you mirrored some of the owner matching logic, it is just extra context you didn't have. Now you have it :)22:31
JayFdking: are you patching anything? or building from git rather than from pypi? This is what happens when you have an older ironic-lib and newer IPA (or vice versa)22:31
JayFdking: this  change happened around the qemu CVE issue22:31
JayF(we know this has to work in the happy path because otherwise CI wouldn't have passed)22:32
opendevreviewAdam McArthur proposed openstack/ironic-tempest-plugin master: Testing bad microversions on v1/allocations  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/93582722:32
JayFTheJulia: cid: TBH I hadn't tied the comment and that regression together :) you can't write code in a complex system without writing bugs sometimes... I tell myself this daily22:32
dkingJayF: That could be a consideration for when I build the images, but I'm also getting the error when I even try to run my unit tests. I only list ironic-python-agent in my requirements, and that package is what installs the others.22:33
JayFAre you using the proper constraints file?22:34
cidTheJulia, I do now for real (around that at least).22:34
JayFe.g. https://github.com/openstack/ironic-python-agent/blob/stable/2024.2/tox.ini#L1622:34
dkingI am using Python 3.9, which is what ironic-python-agent-builder was installing when I last checked, which is old and maybe things are finally using a newer python version now?22:34
JayFthat's what ensures you get the supported versions of the things22:35
dkingJayF: Where would I get the official file? That might help.22:35
JayFsee the linked code :)22:35
JayFjust use the same URL tox is using, adjusted for your branch22:35
JayFBTW; you should do this for /all openstack deliverables/ if you want to be in the tested path :)22:36
JayFand if you're talking about 2024.2, https://governance.openstack.org/tc/reference/runtimes/2024.2.html python 3.9 is still supported22:36
dkingIs that still the default that ironic-python-agent-builder uses for the image?22:40
dkingI'm testing with the constraints file now. I build for my tests inside a container, which helps with the cleanup.22:40
JayFI think for our image we use the default python that's installed for that distro22:44
JayFbut imbw22:44
JayFwhich is likely py3.10, but again, it should work on any supported, and we unit tested that version from 3.9->3.1122:44
dkingJayF: Well, I did find some things I had versioned locked that needed to be let up, but then even with the constraints file I'm still getting the same error.22:53
JayFwhat exact upstream branch are you using?22:54
JayFI guess it's possible if you're on an old enough branch we could have something that broke out from under us22:54
JayFand you're not using any kind of downstream fork? 22:55
dkingCurrently, I'm in a docker container (python:3.9) running tox, and in my requirements.txt I have: ironic-python-agent, oslo_log, oslo_concurrency, pynetbox, and a few other things that shouldn't relate to Ironic. I've added the constraints file. The error comes when I'm running python -m unittest. I'm not using any local branches, except for my own code which imports those things.22:58
JayFpython -m unittest is not how to run Ironic tests :)22:59
JayFI don't know if that's /the problem/ but it may muddy the waters22:59
JayF`stestr run` is the method we support/test23:00
JayFIf you're modifying the requirements.txt I suspect that's more likely the root cause though, but without exact output and knowledge of what exact code you're running this problem is ~impossible to troubleshoot23:01
JayF(and given my experience from running unit tests in my IDE -- which uses unittest module -- sometimes the ordering gets funky which I would believe could cause this)23:01
dkingI think I may have not communicated clearly. I'm building my own hardware manager. It fails during the unit tests (and also on the built IPA image). The same build worked last week. I re-ran my previously passing CI to confirm it wasn't from a local change.23:02
JayFYou've still not indicated what version you're working agains.t23:03
dkingCurrently, when tox does a pip install of ironic-python-agent, it's getting 9.14.023:04
JayFand you're using the 2024.2 constraints?23:06
TheJuliadking: latest from whatever is in pip?23:06
JayFhttps://opendev.org/openstack/releases/src/commit/ea50baad3a49f6c2eb2fede09f2d7b64c3221c94/deliverables/dalmatian/ironic-python-agent.yaml#L2223:07
JayFit's literally not changed a lick :/23:07
JayFsince 2024.223:07
JayFwhich is sad for it's own reasons because we need to push a release of that, there's a regression fix in the repo that's not in the release23:07
* JayF trying HEAD of stable/2024.2 w/ tox -epy3923:08
dkingTheJulia: Correct, whatever pip installs. It's currently installing 9.14.0, from what it looks like.23:09
JayFRan: 1027 tests in 4.8322 sec.23:09
JayFfrom HEAD stable/2024.2, newly created environment23:09
JayFall passing23:09
JayFIs it possible you're doing multiple install calls?23:09
dkingWell, that's good to know. 23:10
JayF`pip install -c constraints.txt package` `pip install -c constraints.txt package2` will not behave the same as doing them all on one line23:10
JayFbecause the second run will happily clobber versions the first one wants23:10
dkingI suppose that is possible that the same requirements.txt is called twice.23:10
JayFyep23:10
JayFI suspect that may be somewhat at the heart of it23:11
JayFpip is extremely finicky about version installing23:11
TheJuliayeah, so it is pulling in latest ironic lib released a week ago which has things removed23:11
TheJuliaso basically latest IPA release doesn't work with latest ironic-lib release as-is23:12
TheJuliaand that is all pip handling versioning, outside of development23:12
JayFI'm confused as to how that's not breaking ipa unit tests in my test, then23:12
JayFdid we cut a bugfix release of ironic-lib without cutting a bugfix ipa?23:12
TheJuliabecause when you run unit tests, it apply constraint logic23:12
TheJuliawhich is *only* known to the runner23:12
TheJulia*not* pypi23:12
TheJuliaor the ipa package itself23:12
TheJuliaJayF: sure looks like it...23:13
JayFugh, looks like they automatically cut one for ironic-lib23:13
JayFhttps://opendev.org/openstack/releases/commit/2d4e54aede724bc8aded603bdda839ada459af8023:14
dkingAre you saying that my issue might be legit?23:14
JayFthat should've been nack'd by us, and should not be done in the future23:14
JayFdking: I'm saying my original claim; you're not following constraints, is proven right :)23:14
JayFdking: you're pulling in epoxy ironic-lib alongside dalmation ipa23:14
JayFwhich is no bueno, and probably is somewhat our fault given this release scenario23:14
JayFrpittau: https://review.opendev.org/c/openstack/releases/+/934598 please do not +1 ironic-lib releases without accompanying ironic/ipa releases, and we should ask releases team to stop making these milestone ones for ironic-lib23:15
TheJuliaI need to get going, but yeah... it is our fault. This is also why when ripping stuff from libraries, we tend to encourage a complete cycle for deprecation23:15
JayFand we need to cut an IPA and Ironic release ASAP to unbreak people23:15
JayFI don't think we can pull releases, can we?23:15
JayFThat's the real solution: that ironic-lib needs to be pulled23:15
TheJuliawe might be able to23:15
TheJuliayeah23:16
TheJuliaputting laptop in bag23:16
dkingThank you all very much. Let me know if that gets pulled upstream.23:16
JayFyeah, but do be careful to use the constraints23:17
JayFI'm glad you found this but there are times this can happen and it not be bugged23:18
dkingJayF: Yeah, it's helpful for them to be set. I've added them now. Thank you.23:22
JayFI was very, very confused because we never release ironic-lib without IPA/Ironic23:22
JayFand your error mimicked something I had wrestled against (and resolved) REPEATEDLY during the CVE shenanigans23:22

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!