| opendevreview | cid proposed openstack/ironic master: [WIP] Add inspection rules https://review.opendev.org/c/openstack/ironic/+/918303 | 01:21 |
|---|---|---|
| opendevreview | OpenStack Proposal Bot proposed openstack/ironic-inspector master: Imported Translations from Zanata https://review.opendev.org/c/openstack/ironic-inspector/+/927701 | 02:18 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 09:29 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 12:34 |
| TheJulia | good morning | 12:59 |
| guilhermesp | hello ppl! i was wondering if you can help me with a operational issue with ironic caracal down here.. | 13:21 |
| guilhermesp | so i have a cloud which is a mix of regular qemu kvms and 2 baremetals. The baremetals are all good in available state. We should expect them, once in available state, to tell nova-compute-ironic and then the resource tracker tells placement the node is a resource provider, right? | 13:22 |
| TheJulia | Good morning! | 13:25 |
| TheJulia | So slightly different: nova-compute services (ideally, only *one* since hash ring use is deprecated and I think gets removed ?next? cycle in favor of shard key usage) with the ironic driver should poll ironic and then update the resource tracker *and* about the node's resource_class so it can be scheduled upon. | 13:27 |
| TheJulia | If you don't see the the resource_class populated in placement, I'd start looking at the nova-compute service logs to see what they are seeing, but then again, you might just be missing something silly like the resource_class value on the baremetal | 13:28 |
| guilhermesp | mornings Julia! Yeah... ive been trying to get something meaningful in nova logs ( enabled debug as well ) but nothing relevant. I have set up the resource class ( called baremetal and flavor uses CUSTOM_BAREMETAL='1') and all but.. even me creating the resource class manually in placement still returns no valid hosts, and scheduler logs saying that the resource class doesnt exist. Pretty weird | 13:31 |
| guilhermesp | thats the overcloud, undercloud runs ironic too, and it works flawless | 13:31 |
| TheJulia | guilhermesp: so crazy question, can the account nova runs with *see* the nodes? | 13:33 |
| TheJulia | undercloud/overcloud? Are you running some sort of caracal based tripleo? | 13:33 |
| guilhermesp | nope, its a vexxhost/atmosphere deployment. We have undercloud there which doesnt setup nova cells since we just want to provision baremetals there, and the overcloud we want both kvms and baremetals... | 13:35 |
| guilhermesp | yeah let me check if we have implied roles for reader role, im pretty sure we had | 13:35 |
| TheJulia | guilhermesp: you have to keep in mind, a project scoped admin in ironic can *only* see nodes it is permitted to see or that it's project created | 13:36 |
| TheJulia | we have that whole extra owner/lessee model on top | 13:36 |
| TheJulia | (which yes, means one could actually use the same ironic for both realistically if they could back the auth) | 13:37 |
| TheJulia | Hey! PTG idea: Diverse auth backends?!? | 13:37 |
| TheJulia | The one ironic to rule them all plan | 13:37 |
| guilhermesp | yeah i do see errors in ironic-api of issues for permissions to list nodes. But i was talking to rico lin and he said it was something related to openstack-exporter since the requests were returning 200 and right after that, a permission issue. Something like "rejecting authoriaztion: baremetal:node_list:all is disallowed by policy. It tries with both admin and ironic users | 13:40 |
| TheJulia | guilhermesp: the permission model is nested, so permissions to just get a blanket list of everything is reserved for system scoped accounts | 13:41 |
| TheJulia | It then falls to a project scoped filter which applies the user's project to the query to match the owner and lessee fields in the db | 13:42 |
| TheJulia | so your basically always going to see an error on project scoped requests, but they are non-fatal, it is just oslo.policy being loud logging wise in ways which help for other requests which fail | 13:42 |
| TheJulia | but, that is a substantial hint to what is going on | 13:42 |
| guilhermesp | gotcha TheJulia . k let me try digging more on nova-compute logs, i will let you all know the outcome if find the issue here.. it could be something really silly yeah, but i mean... using ironic is pretty simple, just make the node available, set a resource class, set a flavor and thats it :P | 13:46 |
| TheJulia | guilhermesp: please share that irc log with rico :) | 13:46 |
| guilhermesp | yup i did that 5 minutes ago | 13:47 |
| TheJulia | cool cool | 13:47 |
| guilhermesp | btw, i was talking to iury yesterday | 13:47 |
| TheJulia | so one thing with nova-compute is it should independently log the number of nodes found | 13:47 |
| guilhermesp | he asked for some stickers in indianapolis :P | 13:47 |
| guilhermesp | i will be there | 13:47 |
| TheJulia | Okay, I'll actually try to drive down to my home this weekend and see if I can find the stickers | 13:47 |
| * TheJulia keeps forgetting | 13:47 | |
| TheJulia | The last few months hav ebeen crazy for me | 13:48 |
| TheJulia | I'm trying to come up with talk titles for OpenInfra Days NA 2025, and I'm thinking "CVEs, why your upstream team needs to plan for them." | 13:49 |
| TheJulia | And talk about how one ripples into your capacity if you have downstream processes | 13:49 |
| TheJulia | OpenInfra Days NA 2025 will be colocated with SCALE | 13:52 |
| opendevreview | Doug Goldstein proposed openstack/networking-baremetal master: avoid attribute error on bad password or config https://review.opendev.org/c/openstack/networking-baremetal/+/929650 | 14:05 |
| cardoe | As I missed the container convo from yesterday. :/ | 14:07 |
| cardoe | I would definitely be interested / involved / invested in container building. | 14:07 |
| TheJulia | cardoe: I missed it to, but I *do* wonder, what level of container building? I know some folks are highly preferring to just use the kolla built containers, or at least its base model and tooling. I know others (who are still asleep and will be for a few hours) might prefer that to change to something $else, but there is that whole ship momentum behind it already | 14:08 |
| cardoe | So comically OpenStack has 2 official projects to build containers for OpenStack components. | 14:09 |
| cardoe | kolla and loci | 14:09 |
| cardoe | I use both... :( | 14:10 |
| TheJulia | yeah, that is slightly problematic | 14:10 |
| cardoe | metal3 uses neither | 14:11 |
| cardoe | vexxhost has a fork of loci | 14:11 |
| TheJulia | metal3 is not an openstack project | 14:11 |
| cardoe | Sure. I'm just speaking from what I know of operators. | 14:11 |
| TheJulia | ahh, yeah | 14:11 |
| cardoe | kolla is fairly opinionated. It's got its own entry point and specific paths that things need to be at. They aren't necessarily Kubernetes friendly cause a lot of assumptions are made about --privileged being passed. | 14:13 |
| TheJulia | Yeah, and I think as folks want to make tighter containers, they might be looking to re-evaluate that as well | 14:14 |
| cardoe | loci on the other hand is used by OSH and is pretty minimalist. But building them involves runes and magical IPs and I think you need to catch a leprechaun every few weeks otherwise it stops building. | 14:14 |
| TheJulia | hmm, that seems... concerning at the same time | 14:15 |
| TheJulia | from the pipeline aspect *and* magical IPs | 14:15 |
| TheJulia | since IPs are always the most infrastucture operator opinionated piece which has burned my team time after time after time | 14:16 |
| cardoe | So loci is a tighter container for sure. It's just building a container of the openstack/requirements repo and stuffing all the wheels there. Then runs an nginx at a specific IP on a docker network that's expected to serve up those wheels to the other builds. | 14:16 |
| TheJulia | I literally had a customer case where they are demanding the ability to just have a duplicate running ipv6 conflict and we're like "nope, no, not going to happen" | 14:16 |
| TheJulia | so very close to upstream then | 14:17 |
| TheJulia | which is a nightmare for packagers | 14:17 |
| TheJulia | since you have to work that requirements tree heavily if you just use it | 14:17 |
| cardoe | I love OSH and I hate OSH. Because it ignores every projects best practices and tries to learn them on its own. | 14:17 |
| cardoe | So dealing with packagers demands around changes has it's own overlay system. | 14:18 |
| cardoe | I do think there's an answer for having a minimalist build that matches what packagers would want and could be extended and could be consumed by kolla. | 14:19 |
| cardoe | I think loci is closer to that minimal environment. But would require folks to come together. | 14:20 |
| cardoe | For us, my rough sketched out plan (except I lost the person I was gonna task with this) was to start with loci. And then layer what kolla wants on top. | 14:20 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-specs master: Support for hardware that cannot be powered off https://review.opendev.org/c/openstack/ironic-specs/+/926654 | 14:40 |
| dtantsur | JayF: quite a rework ^^ | 14:40 |
| JayF | heh you must have raced my review | 14:48 |
| JayF | yeah I found that literally a minute after you updated it | 14:48 |
| JayF | was afraid I re-reviewed an old version but timing was just perfect | 14:49 |
| cardoe | just to confirm if someone else knows, the dnsmasq backend doesn't send signals correct? | 14:49 |
| JayF | what does that question mean? | 14:52 |
| JayF | we HUP dnsmasq on config update | 14:52 |
| JayF | that's where the bug lies in it where it sometimes goes kaput | 14:52 |
| cardoe | https://opendev.org/openstack/ironic/src/commit/f082c24d2a672836dead33eae86e610a7a2afecf/ironic/dhcp/dnsmasq.py#L135 that seems to imply no more SIGHUP? | 15:14 |
| JayF | hmm, I'm surprised by that but I know that's been used+tested so it's just something I didn't know | 15:16 |
| *** dking is now known as Guest3738 | 15:19 | |
| *** Guest3738 is now known as dking | 15:21 | |
| dking | I have a cleaning fail because I had a node reboot, and since it had been in power on for a long time, it got a new DHCP lease, and now it fails because agent_url and callback_url do not match. Are there some good solutions to force the cleaning to fail or proceed? | 15:23 |
| dking | I can't change the DHCP lease at this time without introducing some undesirable risk in production. I cannot simply abort the cleaning because the API refuses to do so in the cleaning state. | 15:24 |
| JayF | it sounds like you didn't restart cleaning properly then? | 15:24 |
| JayF | what's the state it was in, what'd you run? | 15:24 |
| JayF | ironic should regenerate those bits at restart of cleaning, so it's strange to hear old values causing a headache | 15:25 |
| dking | JayF: Correct. The cleaning wasn't able to proceed. The cleaning was hung due to an error (which should be fixed now), but since the server was rebooted in the middle of cleaning, it won't proceed because of the new IP. | 15:26 |
| JayF | so what state is/was the node in? | 15:26 |
| dking | My main goal right now is to abort the cleaning process and start over, but the timeout seems to be very long. | 15:26 |
| JayF | btw at :30 I turn into a pumpkin | 15:26 |
| JayF | what state is it in right now? | 15:27 |
| JayF | provision_state to be explicit | 15:27 |
| dking | Before cleaning? I'm not sure. Either manageable or available, probably the later. It is now in "cleaning". | 15:27 |
| dking | Not clean wait, but "cleaning". | 15:28 |
| JayF | okay, if/when it gets to cleanwait you should be able to call set-provision-state abort | 15:28 |
| JayF | maybe even try it now but it's unlikely | 15:28 |
| JayF | only other way to knock it loose outside of the DB would be to restart the conductor handling it | 15:28 |
| JayF | (or, as you observed, wait for timeout) | 15:28 |
| dking | Yeah, abort won't work now. I can't even set it to maintenance. And if it made it any further, I'd probably be happy. | 15:28 |
| JayF | maintenance would do the opposite you want to | 15:29 |
| JayF | it'd keep it in cleaning indef, because we'd ignore heartbeats iirc | 15:29 |
| JayF | I gotta go record a podcast, sorry, won't be back for like 90 o/ | 15:29 |
| dking | Well, sometimes, I've seen it fail after coming out of mainteance. | 15:29 |
| dking | ...but fortunately, I think that it just cycled. Maybe I scared it by coming into chat. :) | 15:30 |
| * TheJulia suspects "work from outside" is up next for today | 15:50 | |
| * TheJulia is only semi paying attention to IRC at the moment, too many asks/pings | 15:53 | |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: WIP: start working on Ubuntu 24.04 support https://review.opendev.org/c/openstack/bifrost/+/928895 | 15:55 |
| JayF | those are always fun :) I got to learn F# is apparently a whitespace based language too | 16:25 |
| * TheJulia twitches | 16:29 | |
| TheJulia | GIIITTTT | 16:50 |
| * JayF playing a game of "SCM" or "country rage" | 16:52 | |
| * JayF 's dog understands "git" as a command now lol | 16:53 | |
| JayF | (for those unaware, it's like "get out" except you just say "GIT"; and no, you don't add an "er done" at the end) | 16:53 |
| TheJulia | SCM | 16:55 |
| TheJulia | toats scm | 16:55 |
| TheJulia | it ate a commit somehow | 16:55 |
| TheJulia | and I found it in the prior commit | 16:55 |
| jssfr | git reflog to the rescue? | 16:55 |
| TheJulia | which I found squahsed in on a prior commit | 16:55 |
| TheJulia | I figured it out and am fixing it, but... blaaarg | 16:56 |
| TheJulia | Join us tonight for our news at 11 story: git rebase gone wrong | 16:56 |
| TheJulia | "When rebases go sideways" | 16:58 |
| JayF | git add -p can be helpful for that situation | 16:59 |
| JayF | I assume you know but if not :) | 17:00 |
| JayF | accidentally git commit --amend when I don't mean to amend is a common failure for me lol | 17:00 |
| jssfr | oh yeah, especially during rebases | 17:00 |
| jssfr | when I marked a commit for an edit and I don't notice I get thrown into conflict resolution before that, bonus if rerere resolves the conflict | 17:01 |
| TheJulia | what was weird was I was in a rebase conflict resolution before it happened and suddenly git was like "no, your not" and I was just flabbergasted | 17:02 |
| TheJulia | I must have though I was on another folder and amended a commit by accident | 17:02 |
| TheJulia | and yeah | 17:02 |
| cardoe | https://docs.openstack.org/releasenotes/ironic/unreleased.html looking at the release notes... should we be consistent in [section]/field as the syntax? | 17:33 |
| JayF | cardoe: I would +1 a patch that did that; although I'll note outside of release notes we can reference directly to the config file | 17:42 |
| TheJulia | consistency is key, at the same time historical style has varried a lot to [section]key | 18:13 |
| TheJulia | since / is not part of the key name it is just a human formed delimiter anyhow | 18:13 |
| TheJulia | not to be confused with human form replicators | 18:14 |
| opendevreview | Doug Goldstein proposed openstack/sushy master: add pyproject.toml to support pip 23.1 https://review.opendev.org/c/openstack/sushy/+/929693 | 18:14 |
| cardoe | TheJulia: yeah I'm more just curious what the syntax should be and maybe making a check for that. | 18:50 |
| cardoe | TheJulia: should we also rebase https://review.opendev.org/c/openstack/ironic/+/927518 ? | 18:50 |
| cardoe | I didn't know if you were happy with it. | 18:50 |
| TheJulia | rebase at will! | 18:53 |
| TheJulia | Rebase parties encouraged | 18:55 |
| TheJulia | They are like Lightswitch raves | 18:55 |
| * TheJulia might have dated herself with that reference | 19:12 | |
| JayF | https://www.youtube.com/watch?v=GtQpThwWQtQ | 19:18 |
| JayF | wait, that cuts off right before the line :( | 19:19 |
| JayF | oh well, the idea is still there, and the song is the best part anyway | 19:19 |
| JayF | TheJulia: you wanna feel dated? The first time I watched this was on the strongbad email video podcast on my **ZUNE 30G** | 19:19 |
| TheJulia | Your zune ran flash? | 19:24 |
| TheJulia | I'm impressed | 19:25 |
| cardoe | TheJulia: I just don't want to be grounded... | 19:25 |
| TheJulia | it suddenly occurred to me that strongbad was clearly a software engineer | 19:25 |
| TheJulia | Earth Grounds are the best kind of grounds. Warm and Fuzzy without the electrons flowing. | 19:26 |
| TheJulia | #electricaljoke | 19:26 |
| cardoe | I will tell that one to my electrician next time I see him. | 19:27 |
| JayF | TheJulia: they did video versions of them forthe video cast | 19:27 |
| cardoe | JayF: https://review.opendev.org/c/openstack/sushy/+/929693 I'm cheating and doing the easier one. | 19:27 |
| JayF | TheJulia: they'd even put a little clip at the end of the stuff you'd get in the flash from clicking the random stuff | 19:27 |
| JayF | not sure if that was first run or some republish anyway, I was sorta obsessed at that point in time with finding video podcasts because of how cool it was to just have a bunch of funny videos in my pocket (who would've thought how far we'd come) | 19:28 |
| TheJulia | JayF: nice | 19:28 |
| cardoe | So now stupid gerrit question... how do I make a branch that has another commit that depends on that change and the two commits are local without gerrit complaining at me to only submit one commit and squash them? | 19:28 |
| cardoe | You know... like a real branch with multiple commits. | 19:28 |
| TheJulia | you do it locally and stack them | 19:29 |
| TheJulia | just like you realistically would with gerrit | 19:29 |
| clarkb | fwiw gerrit shouldn't complain to only submit one and squash anything | 19:29 |
| TheJulia | when you then next run "git review", it will reconcile the difference and restack upstream patches | 19:29 |
| TheJulia | or you can do things like squash/fixup where appropriate | 19:30 |
| cardoe | I swear last time I tried it complained that I can only submit 1 commit in a review. | 19:30 |
| clarkb | git review warns you when you are pushing more than one commit because people have historically accidentally pushed up hundreds of changes. So git review is saying "heres a list of commits you are about to push make sure you intend to do so" | 19:30 |
| cardoe | I'm LKML koolaid wrt to commits. Each one should stand on its own and the series tells a story. | 19:30 |
| clarkb | but it just wants you to confirm that you intend on pushing more than one to avoid accidents. | 19:31 |
| clarkb | ya thats exactly how gerrit operates | 19:31 |
| JayF | that's mostly fair, but when doing a large feature it can be nice to stack 'em | 19:31 |
| clarkb | you're just using git push to an ssh/http server instead of git am email stuff | 19:31 |
| JayF | cardoe: we kinda are similar, fwiw, because we have every commit has to pass CI | 19:32 |
| JayF | cardoe: so while you might merge an interface that's unused, we should *never* have the HEAD of master nonworking | 19:32 |
| * JayF points at the word should for emphasis | 19:33 | |
| cardoe | I agree with that. | 19:34 |
| JayF | of course, master was broken w/r/t qemu security for 10 years, so ymmv :D | 19:38 |
| JayF | we do the best we can tho | 19:38 |
| TheJulia | cardoe: There just needs to be change-id values and then it can submit them all | 19:59 |
| TheJulia | and then magic happens | 19:59 |
| TheJulia | magic() | 19:59 |
| opendevreview | Doug Goldstein proposed openstack/sushy master: switch to pre-commit for style checks https://review.opendev.org/c/openstack/sushy/+/929704 | 20:07 |
| cardoe | Well I'll be. | 20:08 |
| cardoe | Anyway, hopefully okay changes. Those errors that I had to clean up in sushy happen for me locally always so not really sure how that passes in Zuul today. | 20:09 |
| cardoe | The big changes are after that. Like just bringing in ironic's pre-commit results in 110 files needing a change. | 20:12 |
| JayF | 👀 | 20:13 |
| TheJulia | brraaains | 20:57 |
| * TheJulia doubles JayF's surprise | 20:58 | |
| cardoe | JayF: I'll sling it but you might cringe. I'll fix get these 2 landed. | 21:11 |
| JayF | context? | 21:11 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 21:12 |
| cardoe | JayF: the sushy pre-commit / flake8 / etc. | 21:27 |
| JayF | ah, yeah, it seems fine to me | 21:27 |
| JayF | well, it's failing CI for what appears to be config reasons | 21:38 |
| JayF | but the general shape is okay | 21:38 |
| opendevreview | Doug Goldstein proposed openstack/sushy master: switch to pre-commit for style checks https://review.opendev.org/c/openstack/sushy/+/929704 | 22:08 |
| opendevreview | cid proposed openstack/ironic-tempest-plugin master: Add tempest tests for runbooks https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/928958 | 23:29 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!