| rpittau | good morning ironic! o/ | 06:52 |
|---|---|---|
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Remove outdated API version information from the enrollment docs https://review.opendev.org/c/openstack/ironic/+/879428 | 07:58 |
| dtantsur | TheJulia, JayF, I did a pass on ironic lp bugs. Also revived sushy, sushy-tools, ironic-python-agent, ironic-inspector, bifrost, python-ironicclient | 08:29 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/877786 | 08:32 |
| adam_metal3 | hi does any one of you have network_data.json for Ironic that sets up bonding between interfaces ? or do you know about some online example that I could looke at | 09:48 |
| jssfr | we use `network-config` instead of network_data.json, which works with the nocloud cloud-init driver (if that's the correct name) and supports (a subset of) netplan, including bonds | 09:53 |
| adam_metal3 | some context: I am using simple init + glean to provide some network config for IPA in a dhcpless environment, thus I am using glean but folks downstream are experiencing issues with bonding | 10:04 |
| adam_metal3 | and I am no sure we are specifying the bond correctly, so I am looking for some example that supposed to work | 10:05 |
| jssfr | ah right, then what I wrote doesn't help :) | 10:14 |
| iurygregory | good morning Ironic | 11:24 |
| iurygregory | adam_metal3, so the only doc I remember related to dhcp less would be https://docs.openstack.org/ironic/latest/admin/dhcp-less.html | 11:26 |
| iurygregory | the docs from glean doesn't seem very helpful https://docs.openstack.org/infra/glean/index.html | 11:27 |
| iurygregory | but we can ask the infra team if they know more about using it to set-up bonding | 11:27 |
| adam_metal3 | iurygregory, thanks sadly this has no info about the bond interfaces, the only resource about the bond interfaces that helped somewhat was this: https://docs.openstack.org/nova/latest/_downloads/9119ca7ac90aa2990e762c08baea3a36/network_data.json | 11:28 |
| iurygregory | yeah =( | 11:29 |
| adam_metal3 | I am trying to understand glean source code ATM, I hope I will be able to figure out something from that | 11:30 |
| adam_metal3 | iurygregory,jssfr, Thanks for trying to help anyways! | 11:34 |
| * jssfr gets flashbacks to understanding how nocloud cloud-init works | 11:34 | |
| jssfr | I don't envy you, good luck! | 11:34 |
| iurygregory | adam_metal3, np! good luck | 11:34 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Run metal3 integration with ironic-image from source https://review.opendev.org/c/openstack/ironic/+/879162 | 12:09 |
| opendevreview | Merged openstack/ironic master: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/877786 | 12:35 |
| TheJulia | good morning | 13:38 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/2023.1: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879407 | 13:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/21.3: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879408 | 13:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic bugfix/21.2: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879409 | 13:40 |
| TheJulia | dtantsur: thanks, I was trying to avoid the sub-projects for a little while until :) | 13:40 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/zed: Add error logging on lookup failures in the API https://review.opendev.org/c/openstack/ironic/+/879470 | 13:40 |
| dtantsur | TheJulia: I hope I did not stomp on your shoes then :) | 13:40 |
| dtantsur | The reason I did that was some bugs that reference several projects, e.g. sushy-tools | 13:41 |
| dtantsur | you cannot mark a bug as fixed/invalid for a disabled project | 13:41 |
| TheJulia | ahh, yeah | 13:41 |
| TheJulia | I was hoping for us to have a discussion regarding the subproject bug tracking, but yeah, I noticed that as well | 13:41 |
| dtantsur | We can still have it! If we decide that e.g. sushy-tools should be tracked in sushy, we can archive all bugs and disable it again | 13:42 |
| TheJulia | yeah | 13:42 |
| dtantsur | TheJulia: what would be your final proposal? | 13:42 |
| * TheJulia tries to wake up | 13:42 | |
| * dtantsur hopes to disable ironic-inspector in the not too remote future :) | 13:43 | |
| TheJulia | I was thinking anything ironic*, we just funnel into just ironic's lp bugs | 13:43 |
| dtantsur | also good morning TheJulia. Feeling better this week? | 13:43 |
| TheJulia | much better this week! Thanks! How about you? | 13:43 |
| dtantsur | not better at all :( actually slightly worse today.. | 13:43 |
| dtantsur | a counter-argument could be tracking work items. if e.g. I need to fix something in ironic and IPA, I can track the bug against two projects. | 13:44 |
| dtantsur | if we only have ironic, the bug will be closed once the first patch merges | 13:44 |
| dtantsur | (or we have to use Partial-Bug on both, then it won't be closed) | 13:44 |
| dtantsur | I don't feel strongly, just trying to analyse all aspects of it | 13:45 |
| TheJulia | eek, worse! Get some rest! | 13:45 |
| dtantsur | one more meeting.. (the same as you in 15 minutes) | 13:45 |
| TheJulia | I'm kind of a partial bug fan, tbh | 13:45 |
| dtantsur | yep, we could do it, and we could just have regular bug health check to close actually fixed or no more valid bugs | 13:46 |
| TheJulia | yeah | 13:48 |
| TheJulia | I'm definitely thinking less things to track == better in terms of entirely different projects to go look at bugs, and that kind of aligns with shutting down ironic-inspector's tracking | 13:48 |
| TheJulia | bifrost, sushy, they are different enough it just makes sense to keep them as separate I think | 13:48 |
| dtantsur | what about clients? | 13:49 |
| TheJulia | python-ironic-inspector-client would take the same fate as ironic-inspector I think | 13:49 |
| TheJulia | python-ironicclient I think we would just roll in | 13:50 |
| dtantsur | I assume we cannot track subtasks in lp like we do in sb? | 13:50 |
| TheJulia | no, we cannot | 13:51 |
| TheJulia | grenade has like a 40% fail rate right now :( | 13:52 |
| TheJulia | 500 on openstack server create, yay | 13:53 |
| * TheJulia goes and gets glasses to begin the nova debugging | 13:53 | |
| dtantsur | that's something new. previously, floating IPs were not pingable | 13:53 |
| TheJulia | wheeeee | 13:54 |
| TheJulia | HTTP exception thrown: Flavor baremetal could not be found. | 14:01 |
| dtantsur | TheJulia: I think this one is a red herring. The result of openstackclient trying to call /servers/<name>, which Nova does not support | 14:02 |
| * dtantsur has gone through the same thought process once :D | 14:02 | |
| dtantsur | s/servers/flavors/, but you can the idea | 14:02 |
| TheJulia | GET /compute/v2.1/flavors/baremetal | 14:04 |
| opendevreview | Julia Kreger proposed openstack/sushy stable/2023.1: Add TransferProtocolType for any general error that mentions it https://review.opendev.org/c/openstack/sushy/+/879473 | 14:18 |
| TheJulia | iurygregory: I don't know what branches you need that on | 14:18 |
| iurygregory | TheJulia, from OCP side we can take care | 14:19 |
| TheJulia | so if other branches, by all means push buttons and just tag me on reviews | 14:19 |
| iurygregory | ack | 14:20 |
| TheJulia | has anyone raised the grenade failures with nova yet? | 14:56 |
| TheJulia | specifically, it does seem like the flavor is not found for specifically the server create | 14:56 |
| dtantsur | What I observed (may be different), was the server creation actually succeeding (despite some ugly errors in the logs) | 15:06 |
| JayF | adam_metal3: so, I've done bonding+vlans via network_data.json. Unfortunately it was with a downstream patch to cloud-init so I'm unsure what things work upstream now and what ones don't. | 15:08 |
| JayF | adam_metal3: I'd make sure to ask the question, not only "how do I represent this as JSON?" but also "am I sure bonding is supported in glean/cloud-init" | 15:08 |
| clarkb | glean supports it on some platforms at least. But I'm not sure if it does for all of them | 15:09 |
| JayF | ack | 15:12 |
| TheJulia | JayF: I think cloud-init may have finally added support, but it has been ages | 15:13 |
| TheJulia | w/r/t grenade, it does appear that we can't find the record in the db. | 15:13 |
| opendevreview | Merged openstack/sushy master: Add TransferProtocolType for any general error that mentions it https://review.opendev.org/c/openstack/sushy/+/878208 | 15:14 |
| adam_metal3 | JayF, Thanks for the insight, I have checked the source code, Glean expects the bonded interfaces, parses the data and there is the logic to produce network-script outputs, so I assume it is in the scope of the tool | 15:25 |
| TheJulia | network-script outputs for what os? | 15:26 |
| TheJulia | nova things it is a red herring or tempest is just running with the wrong scope, but the record should have been found in the db... soooo. yeah. | 16:03 |
| *** iurygregory_ is now known as iurygregory | 16:24 | |
| iurygregory | TheJulia, dtantsur was looking at the failures (maybe they are not the same) he found it was related to floating IP not being pingable | 16:28 |
| iurygregory | when I checked yesterday all I could find was the problem to create the baremetal flavor | 16:28 |
| TheJulia | yeah, the instance never gets created yesterday | 16:28 |
| TheJulia | the create gets a 200, the immediate turnaround request to get it gets a 404 | 16:28 |
| TheJulia | and that 404 is coming out of the db | 16:29 |
| JayF | could the mysql changes be to blame? maybe things taking too long to get into the db? | 16:29 |
| JayF | https://github.com/openstack/devstack/commit/7567359755a105e7278bbf97541332f28228b87d | 16:29 |
| JayF | timing seems right | 16:29 |
| TheJulia | timing wise, nothing in the file since 2022 | 16:31 |
| TheJulia | like... a year | 16:31 |
| JayF | huh? | 16:31 |
| TheJulia | nova/objects/flavor.py | 16:31 |
| TheJulia | that devstack change breaking flavors would be... epicly bad | 16:32 |
| JayF | I don't disagree, but I'm in "think of anything that's changed that could cause that" | 16:32 |
| * TheJulia tries to figure out why https://review.opendev.org/c/openstack/ironic-lib/+/879211 breaks so orribly | 16:32 | |
| JayF | and I land on things like that | 16:32 |
| iurygregory | https://zuul.opendev.org/t/openstack/builds?job_name=ironic-grenade&project=openstack%2Fironic&branch=master&result=FAILURE&skip=0 | 16:36 |
| JayF | iurygregory: https://github.com/openstack/devstack/commit/7567359755a105e7278bbf97541332f28228b87d roughly maps to when it started failing more frequently | 16:37 |
| * JayF smells a rat | 16:37 | |
| iurygregory | yeah | 16:37 |
| iurygregory | agree | 16:37 |
| JayF | you gonna push a test-disable PR or do you want me to? | 16:37 |
| JayF | I don't really have time to troubleshoot it too much further today, gotta go heads down downstream again | 16:38 |
| iurygregory | so maybe we are a bit lucky in some cases since is not 100% of the time it failed | 16:38 |
| JayF | which is exactly how a borderline performance issue would present | 16:38 |
| iurygregory | I can push a revert to test against our changes | 16:38 |
| iurygregory | to see if helps | 16:38 |
| JayF | you just need to unset that config | 16:39 |
| iurygregory | oh right | 16:39 |
| JayF | MYSQL_REDUCE_MEMORY: false | 16:39 |
| iurygregory | yeah, let me add that to our ironic-base job | 16:39 |
| JayF | Honestly, I think it's like a 25% or less chance to actually fix things | 16:39 |
| JayF | but it's too easy to try not to try :) | 16:39 |
| iurygregory | or maybe even to just grenade job | 16:39 |
| JayF | I was thinking grenade job, but wasn't going to dictate implementation :D | 16:39 |
| iurygregory | ++ let me push a patch | 16:40 |
| JayF | iurygregory: another thing to consider perhaps -> the uwsgi migration stuff | 16:44 |
| iurygregory | did this happen already? O.o | 16:45 |
| iurygregory | hummm | 16:45 |
| iurygregory | I saw some discussions in the list only | 16:46 |
| JayF | https://lists.openstack.org/pipermail/openstack-discuss/2023-April/033173.html I just saw this | 16:46 |
| JayF | I do not know if devstack is configured to use this currently, but it's also one of those things that could be breaky without being a hard break | 16:46 |
| iurygregory | oh wow | 16:46 |
| iurygregory | yeah | 16:46 |
| iurygregory | going to push a change to set MYSQL_REDUCE_MEMORY: false, and another one to only run grenade job so we can recheck to see if helps | 16:47 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Disable MYSQL_REDUCE_MEMORY for grenade https://review.opendev.org/c/openstack/ironic/+/879494 | 16:48 |
| JayF | ++ thank you | 16:48 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: [DNM] Run only grenade job with MYSQL_REDUCE_MEMORY https://review.opendev.org/c/openstack/ironic/+/879495 | 16:49 |
| opendevreview | Julia Kreger proposed openstack/ironic-lib master: Add jsonrpc client port capability https://review.opendev.org/c/openstack/ironic-lib/+/879211 | 17:01 |
| TheJulia | yay for silly human mistakes | 17:01 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Fix requests calls with timeouts https://review.opendev.org/c/openstack/ironic/+/879497 | 17:38 |
| opendevreview | Julia Kreger proposed openstack/ironic master: Make bandit voting on check gate https://review.opendev.org/c/openstack/ironic/+/879498 | 17:38 |
| JayF | iurygregory: https://zuul.opendev.org/t/openstack/build/0e86492412874c699d52915bd9cfa218 first one passed | 17:51 |
| opendevreview | Merged openstack/sushy stable/2023.1: Add TransferProtocolType for any general error that mentions it https://review.opendev.org/c/openstack/sushy/+/879473 | 18:03 |
| *** tzumainn__ is now known as tzumainn | 18:54 | |
| Sandzwerg[m] | <adam_metal3> "hi does any one of you have..." <- Ping me tomorrow and I can provide an example | 19:05 |
| opendevreview | Julia Kreger proposed openstack/ironic-specs master: Add service steps framework https://review.opendev.org/c/openstack/ironic-specs/+/872349 | 19:14 |
| iurygregory | JayF, yeah, actually 2 since we have https://review.opendev.org/c/openstack/ironic/+/879495/ | 19:53 |
| iurygregory | will attempt some more rechecks to see, but I think this will be our approach | 19:53 |
| * JayF mentioned in -qa, and someone pointed out it's default off | 19:57 | |
| JayF | and I misread it the first time | 19:57 |
| JayF | so that's probably "luck" that it passed 2x (or unluck) | 19:58 |
| gmann | here but plan is to enable it soon so advance testing will be good to know potential issue if any https://github.com/openstack/devstack/blob/b10c06027273d125f2b8cd14d4b19737dfb94b94/stackrc#L207 | 20:12 |
| JayF | Yeah I think it's a classic red herring | 20:12 |
| JayF | have a bug that shows up 30-50% of the time | 20:12 |
| JayF | think you have it fixed because you didn't hit it a few times in a row | 20:12 |
| * TheJulia lays down for a little bit to allow migraine meds to kick in | 20:17 | |
| iurygregory | well, 3 success in grenade in the patch .-. (maybe I'm lucky) | 20:55 |
| JayF | I hope this against-the-odds success sticks with me when I vacation in Vegas next week lol | 21:03 |
| TheJulia | vegas sounds fun | 21:14 |
| JayF | Yeah, christmas gift for mom+brother was a trip to Vegas. We're executing it next week during Vanessa's week off so she can be home taking care of animals | 21:15 |
| TheJulia | That is awesome | 21:21 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!