| arne_wiebalck | Good morning, Ironic! | 06:36 |
|---|---|---|
| rpittau | good morning ironic! o/ | 06:53 |
| arne_wiebalck | hey rpittau o/ | 07:02 |
| kubajj | Good morning arne_wiebalck rpittau | 08:58 |
| rpittau | hey kubajj :) | 08:58 |
| opendevreview | Nisha Agarwal proposed openstack/ironic master: Fixes Secureboot with Anaconda deploy https://review.opendev.org/c/openstack/ironic/+/860820 | 10:06 |
| opendevreview | Nisha Agarwal proposed openstack/ironic master: Enables boot modes switching with Anaconda deploy for ilo driver https://review.opendev.org/c/openstack/ironic/+/860821 | 10:28 |
| iurygregory | good morning Ironic | 11:09 |
| rpittau | hey iurygregory :) | 11:22 |
| iurygregory | rpittau, welcome back o/ | 11:22 |
| rpittau | thanks :) | 11:22 |
| arne_wiebalck | hey kubajj o/ | 11:41 |
| TheJulia | good morning | 13:16 |
| rpittau | good morning TheJulia :) | 13:18 |
| rpittau | is the ptg calendar up-to-date? we have sessions only tuesday and wednesday? | 13:34 |
| TheJulia | Dunno, I think it might be | 13:46 |
| iurygregory | good morning TheJulia =) | 13:47 |
| iurygregory | rpittau, yeah sounds about right.. not sure if we will need more time | 13:47 |
| rpittau | ok, thanks, I thought we had sessions mon-thu :) | 13:48 |
| iurygregory | in the past we used to have this, but we didn't have many topics this time | 13:48 |
| TheJulia | Also, to be fair, we shoudl take some time and be in other group's sessions at times | 13:49 |
| TheJulia | like.. Monday, I'm going to be in enviro-sus | 13:49 |
| TheJulia | and then possibly diversity/edge | 13:49 |
| TheJulia | *somehow* | 13:49 |
| JayF | rpittau: there's also a session at 1400 UTC on Monday that is a joint session with Nova and probably booked under Nova | 14:12 |
| JayF | To talk about Nova driver issues | 14:13 |
| rpittau | JayF: thanks! | 14:13 |
| TheJulia | yay multi-booked | 14:15 |
| TheJulia | :( | 14:15 |
| TheJulia | story of my life | 14:15 |
| * TheJulia wants to print a leeloo dallas multipass for multibooked self | 14:18 | |
| JayF | 5 minute warning for Ironic meeting | 14:56 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: Use declarative reader/writer with enginefacade https://review.opendev.org/c/openstack/ironic-inspector/+/463768 | 14:58 |
| iurygregory | o/ | 15:00 |
| JayF | #startmeeting ironic | 15:01 |
| opendevmeet | Meeting started Mon Oct 10 15:01:05 2022 UTC and is due to finish in 60 minutes. The chair is JayF. Information about MeetBot at http://wiki.debian.org/MeetBot. | 15:01 |
| opendevmeet | Useful Commands: #action #agreed #help #info #idea #link #topic #startvote. | 15:01 |
| opendevmeet | The meeting name has been set to 'ironic' | 15:01 |
| iurygregory | o/ | 15:01 |
| matfechner | o/ | 15:01 |
| JayF | Good morning, who all is here today | 15:01 |
| TheJulia | o/ | 15:01 |
| rpittau | o/ | 15:01 |
| kubajj | o/ | 15:01 |
| JayF | #topic Announcements/Reminder | 15:02 |
| JayF | #note Meeting next week pre-emptied by PTG; please participate in PTG and our next meeting is 10/24 | 15:02 |
| JayF | #note PTG sessions are scheduled. Check schedule at https://ptg.opendev.org/ptg.html for Ironic and Baremetal SIG Hour meetings. Additionally we are participating in a Nova session, Monday at 1400 UTC, to discuss our Nova driver. | 15:03 |
| opendevreview | Julia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep https://review.opendev.org/c/openstack/ironic-inspector/+/860731 | 15:03 |
| JayF | Does anyone else have an announcement that wasn't already on the agenda? | 15:03 |
| arne_wiebalck | o/ | 15:03 |
| erbarr | o/ | 15:03 |
| JayF | Moving on. | 15:04 |
| JayF | We had no action items last meeting to review; so skipping that agenda item | 15:04 |
| JayF | Subteam status report review usually goes here; but I'm not sure we have too many subteams running right now? We should likely re-evaluate what gets reported here at PTG | 15:04 |
| JayF | #link https://wiki.openstack.org/wiki/Meetings/Ironic is the meeting agenda, if anyone needs the link | 15:05 |
| JayF | I'm going to give a couple of minutes for folks to look at the status reports in the whiteboard, and say something here if there is a discussion item | 15:05 |
| JayF | #link https://etherpad.opendev.org/p/IronicWhiteBoard subteam status reports | 15:06 |
| TheJulia | I've updated the items I've been working on | 15:07 |
| JayF | Thank you :) It'll be exciting to see the DB stuff land | 15:07 |
| JayF | #topic Stable branch status & decisionmaking | 15:07 |
| JayF | I put an item on the agenda to talk about branch status for our EM branches; and discuss plans for moving forward | 15:08 |
| JayF | Currently wallaby, xena, yoga, and zed are all maintained branches | 15:08 |
| JayF | we also maintain queens, rocky, stein, train, ussuri, and victoria to some degree for extended maintenance | 15:08 |
| JayF | We had agreed last cycle to retire queens, rocky, and stein but they were never actually retired -- are folks still on board for these to go away? | 15:08 |
| TheJulia | I generally care about all the way down to Train, but if nobody else cares about Train, I'm happy to leave it be | 15:09 |
| JayF | #link https://releases.openstack.org/ official branch support status and terminology | 15:09 |
| rpittau | I stop caring at wallaby :P | 15:09 |
| JayF | I have no problem with maintaining back to Train downstream; I just suspect we need to get a little more direct in ensuring that stable branches still work with CI | 15:09 |
| JayF | e.g. a periodic run we can check and report on at the meeting, or something else to track CI status | 15:10 |
| JayF | Less busy projects have had a problem with the older brnaches bit-rotting because they get so few things backported | 15:10 |
| TheJulia | we are though | 15:10 |
| JayF | Runinng a periodic job? | 15:10 |
| TheJulia | it is just we have a finite amount of resources and CI can break at anytime | 15:10 |
| TheJulia | nope, we have no periodics afaik | 15:10 |
| TheJulia | nor would we actually be able to pay attention to it short of directly triggering and watching it | 15:11 |
| TheJulia | well, I should be specific | 15:11 |
| TheJulia | I wouldn't be able to | 15:11 |
| TheJulia | if I'm the only one that cares, then we can kill the t, u, v branches. | 15:11 |
| JayF | Lets talk specifics instead; With the recent need to migrate out "Queue" directives from zuul, nobody else was stepping up to get it done and we had dozens of branches broken and showing up in the broken zuul report | 15:12 |
| JayF | reducing that matrix is one of the only strategies I can think of to reduce that load | 15:12 |
| JayF | and/or ensuring we fix things as they break instead of having a periodic painful CI migration | 15:12 |
| TheJulia | I agree, the queue fixes I previously put up took months to make any review progress | 15:13 |
| TheJulia | unfortunately, I thought we had them all, and we didn't | 15:13 |
| JayF | Yeah; there's a real cost to keeping these up and we don't seem to be doing a good job of making sure that work happens ad-hoc | 15:13 |
| JayF | Either in not reviewing changes, not pushing them up, etc | 15:14 |
| TheJulia | yup | 15:14 |
| TheJulia | or they fall of change dashboards because there are too many open changes | 15:14 |
| TheJulia | well, individual's dashboards | 15:14 |
| JayF | So maybe we can change the approach; should we reduce the testing load for TUV? | 15:14 |
| TheJulia | I think we should | 15:14 |
| JayF | e.g. if we retired q,r,s and set t,u,v to unit testing and linting only | 15:15 |
| JayF | that would reduce our testing matrix by probably hundreds of tests | 15:15 |
| JayF | across all Ironic projects | 15:15 |
| iurygregory | humm unit testing would be enough? | 15:15 |
| TheJulia | I'd keep the more reliable integration jobs there, I can propose a change to cut down the integration jobs | 15:15 |
| iurygregory | I'm ok with the approach to reduce | 15:15 |
| iurygregory | but some integration jobs would be good | 15:15 |
| iurygregory | probably not all of them | 15:15 |
| JayF | As far as OpenStack/TC policy is concerned; we have to run unit tests and linting on stable branches | 15:15 |
| TheJulia | we've done it before, just haven't done it for victoria and older, recently | 15:15 |
| JayF | everything else is up to us | 15:15 |
| JayF | TheJulia: so you said you have interest in back to T, do you expect that to be heavy backports or just security items or what? | 15:16 |
| JayF | that might be informational to iurygregory's comment about if we should keep some integration jobs | 15:17 |
| TheJulia | mainly security or super awful bug | 15:17 |
| TheJulia | Basically where we were with queens two years ago | 15:17 |
| JayF | That reinforces my thought that unit tests, linting, and trusting folks who backport past W to manually test or validate the changes | 15:18 |
| JayF | is enough | 15:18 |
| TheJulia | I'd still wants some integration testing on train, fwiw | 15:18 |
| TheJulia | but we can dial a lot of it back I think | 15:18 |
| JayF | Do you mind taking the action to do that dialing back in V,U,T | 15:19 |
| JayF | and I'll take the action to retire q,r,s? | 15:19 |
| JayF | or at least, let me know what we want to keep around | 15:19 |
| TheJulia | I can do so | 15:19 |
| TheJulia | well, dialing htings back that is | 15:19 |
| JayF | #action JayF to officially announce and retire Queens, Rocky, Stein EM branches from Ironic projects | 15:19 |
| JayF | #action TheJulia to reduce CI jobs for Train, Ussuri, and Victoria | 15:19 |
| JayF | Thank you :-) I'll update the thread I started on the mailing list with this outcome. | 15:20 |
| JayF | I will also note; I'm still working to get releases cut from all maintained ironic branches | 15:20 |
| JayF | #topic Priorities for the coming week | 15:20 |
| JayF | As always, please tag any patches that need priority review with #ironic-week-prio | 15:21 |
| rpittau | I still have some patches from last week so I'm not adding more | 15:21 |
| JayF | I'd like to ask that my series of stable-only patches to remove the packaging requirement and update releaase notes get some attention; I'll be tagging those in there | 15:21 |
| JayF | rpittau: after the meeting i'll take a look thru the list :) | 15:21 |
| rpittau | thanks JayF :) | 15:21 |
| JayF | rpittau: I'll note I have a comment on one of those (the bugfix branch job pin) if you wanna take a look at it | 15:22 |
| rpittau | sure | 15:22 |
| JayF | Giving folks a couple more minutes to talk about priority patches they might have before moving on | 15:22 |
| JayF | #topic Baremetal SIG | 15:26 |
| JayF | #note Next Baremetal SIG will be at the PTG, serving as Ironic's operator hour | 15:26 |
| JayF | There's nothing else to mention about BM SIG, afaik. | 15:27 |
| arne_wiebalck | no ... any more feedback? | 15:27 |
| arne_wiebalck | on the mail I mean? | 15:27 |
| arne_wiebalck | I have not seen any | 15:27 |
| JayF | I haven't seen any :( | 15:27 |
| JayF | I presume the best place to get input might be at the operator hour | 15:27 |
| JayF | especially if we have operator participation there | 15:27 |
| arne_wiebalck | the question is if we continue to chase community members to give presentations | 15:27 |
| JayF | I think the answer to that has to be no? At least not at the current monthly cadence | 15:28 |
| TheJulia | It is a good question to ask, and I would tweet that out with an note indicating that we've secheduled an operator hour | 15:28 |
| arne_wiebalck | the etherpad still has a couple of potential topics | 15:28 |
| TheJulia | fwiw, Operators have often indicated the best means of engagement is Twitter, not any mailing list | 15:28 |
| JayF | TheJulia: I've heard; I'm working with spotz and fungi to try and open that line of comms up for Ironic. | 15:29 |
| JayF | TheJulia: for instance, I'll get them to mention our EM branch retirement | 15:29 |
| JayF | arne_wiebalck: I'll volunteer to do a talk for next BM SIG outside of PTG; that'll at least take us a step further down the road | 15:29 |
| * JayF can talk about IPA and hardware managers | 15:29 | |
| arne_wiebalck | JayF: awesome, thanks! | 15:30 |
| JayF | #action JayF to connect with ops meetup twitter to plug (and Call-for-talks) for BM SIG after PTG | 15:30 |
| JayF | arne_wiebalck: anything else we wanted to hit on this topic before moving on? | 15:30 |
| fungi | yeah, the suggestion is for someone like JayF to tweet something, and then we can retweet or whatever with more widely followed twitter handles like openstack or osops | 15:30 |
| TheJulia | JayF: when I was ptl, I would just tweet at the twitter accounts with whatever my ask happened to be to retweet :) | 15:31 |
| arne_wiebalck | JayF: nothing else for the SIG | 15:31 |
| * TheJulia still does this actually... | 15:31 | |
| JayF | Chairwoman, former PTL, social media professional ;) | 15:31 |
| JayF | lol | 15:31 |
| JayF | Alright, moving on, we're almost done | 15:31 |
| * TheJulia begins building a trebuchet | 15:31 | |
| JayF | There are no RFEs for review. As a reminder, add RFEs to review to the meeting agenda if you have any. | 15:32 |
| JayF | #topic Open Discussion | 15:32 |
| JayF | any topics for open discussion today? | 15:32 |
| TheJulia | nothing from me | 15:33 |
| JayF | gonna give folks until :35 | 15:33 |
| * TheJulia awaits patiently to get the link for the commit message she has typed out... | 15:34 | |
| JayF | I'll run next meeting, I should be here. | 15:35 |
| JayF | #endmeeting | 15:35 |
| opendevmeet | Meeting ended Mon Oct 10 15:35:05 2022 UTC. Information about MeetBot at http://wiki.debian.org/MeetBot . (v 0.1.4) | 15:35 |
| opendevmeet | Minutes: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-10-10-15.01.html | 15:35 |
| opendevmeet | Minutes (text): https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-10-10-15.01.txt | 15:35 |
| opendevmeet | Log: https://meetings.opendev.org/meetings/ironic/2022/ironic.2022-10-10-15.01.log.html | 15:35 |
| rpittau | thanks JayF | 15:35 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/victoria: CI: Reduce overall test load https://review.opendev.org/c/openstack/ironic/+/860851 | 15:35 |
| rpittau | now I guess I need to review the pin as metalsmith jobs look very broken | 15:35 |
| TheJulia | I guess it has been a few weeks since I've looked at metalsmith :( | 15:36 |
| rpittau | I don't think the problem is on metalsmith side, probably needs to pin openstacksdk | 15:36 |
| TheJulia | to <1.0.0 ? | 15:36 |
| * TheJulia imagines a mordred throwing things | 15:37 | |
| rpittau | it says, I qquote: "To utilize this module, the installed version of the openstacksdk library MUST be <=0.98.999." | 15:37 |
| * TheJulia imagines mordred would exhaust wet kittens | 15:37 | |
| rpittau | lol | 15:38 |
| * TheJulia wonders how he is doing | 15:38 | |
| rpittau | let's see if I have enough oxygen to fix that now | 15:38 |
| TheJulia | just don't dive too deeply, or you'll need to use a different gas mixture | 15:39 |
| JayF | TheJulia: we didn't specify in the meeting; but are you going to get a patch like that up for ipa, ir-lib, inspector, bifrost, etc etc? | 15:39 |
| Nisha_Agarwal | 15:39 | |
| Nisha_Agarwal | TheJulia hi...I have raised 4 patches in total for Anaconda deploy support issues | 15:39 |
| TheJulia | JayF: yeah, anticipate I will, just want to see if anyone is going to table flip on ironic first | 15:39 |
| JayF | I'm waiting for jobs there | 15:40 |
| TheJulia | well, the fact it is running is a good sign, syntax wise :) | 15:40 |
| TheJulia | Nisha_Agarwal: I've not had time to review code really in the last two weeks, I should be able to this week | 15:41 |
| rpittau | ok mmmm so *maybe* it is related to metalsmith | 15:42 |
| * JayF is about to review all #ironic-week-prio patches, if Nisha_Agarwal wants to tag those patches it'll get into the list | 15:42 | |
| Nisha_Agarwal | TheJulia, Thanks....Just to add here i have raised patch for supporting secure boot also with Anaconda | 15:42 |
| Nisha_Agarwal | JayF, How do we add tag? | 15:42 |
| Nisha_Agarwal | i can do that | 15:42 |
| JayF | Top left of the change -> click "Show All" -> under hashtags field, hit the pencil to edit -> add ironic-week-prio | 15:43 |
| JayF | this is for items that are ready for review | 15:43 |
| Nisha_Agarwal | JayF, ok let me see... :) | 15:44 |
| Nisha_Agarwal | The patches are tested and working...i will address the comments recvd on them | 15:44 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/ussuri: CI: Reduce overall test load https://review.opendev.org/c/openstack/ironic/+/860852 | 15:46 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/train: CI: Reduce overall test load https://review.opendev.org/c/openstack/ironic/+/860853 | 15:50 |
| Nisha_Agarwal | JayF, i added for all 4 patches | 15:51 |
| JayF | nice, hopefully that'll help you get more review attention on it :) | 15:51 |
| JayF | waiting for TheJulia to review something can take a while, she's always got a lot to do | 15:51 |
| Nisha_Agarwal | JayF, Yes thanks | 15:52 |
| Nisha_Agarwal | review comments will also help to refine it further | 15:52 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/wallaby: Suppress Bandit B509 in snmp.py https://review.opendev.org/c/openstack/ironic/+/860847 | 15:53 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/victoria: Suppress Bandit B509 in snmp.py https://review.opendev.org/c/openstack/ironic/+/860848 | 15:53 |
| TheJulia | whee for bandit changes going backwards in branches :( | 15:53 |
| TheJulia | Nisha_Agarwal: hey, out of curiosity have you looked at python 3.9 compatability for proliantutils? | 15:54 |
| TheJulia | We've gotten reports of compatability issues :( | 15:54 |
| Nisha_Agarwal | TheJulia, havent checked it | 15:55 |
| Nisha_Agarwal | TheJulia, i will check that | 15:55 |
| opendevreview | Riccardo Pittau proposed openstack/metalsmith master: [DNM] TEST CI https://review.opendev.org/c/openstack/metalsmith/+/860854 | 15:55 |
| rpittau | bye all o/ | 15:56 |
| JayF | o/ | 15:56 |
| TheJulia | Nisha_Agarwal: thanks, my appologies if bugzilla.redhat.com is emailing you about it | 15:56 |
| Nisha_Agarwal | TheJulia, i dont see any mails from bugzilla | 15:58 |
| TheJulia | hmm, pity | 15:59 |
| TheJulia | okay | 15:59 |
| Nisha_Agarwal | TheJulia, can u point me to one such issue.... | 15:59 |
| TheJulia | hmm, I'm not seeing it off hand, but I know stevebaker[m] would be good to chat with. He is in New Zealand and his workday will start in ~3 hours | 16:01 |
| Nisha_Agarwal | TheJulia, ok | 16:04 |
| TheJulia | I think the tl'dr is tox -epy39 fails or something like that, but I'm not sure and going and hunting it down would take a while | 16:06 |
| Nisha_Agarwal | TheJulia, The latest patch on proliantutils which was done almost a month back shows that py39 was success | 16:09 |
| Nisha_Agarwal | https://review.opendev.org/c/x/proliantutils/+/855646 | 16:09 |
| Nisha_Agarwal | Are u sure u have latest proliantutils? | 16:10 |
| TheJulia | I think it was on a stable branch where it was expected to be supported, but honestly I just don't remember | 16:17 |
| opendevreview | Adam Rozman proposed openstack/ironic-python-agent master: prioritize lsblk as a source of device serials https://review.opendev.org/c/openstack/ironic-python-agent/+/855866 | 16:32 |
| Nisha_Agarwal | k | 16:32 |
| JayF | TheJulia: should ironic-standalone-anaconda be expected to pass (it's n-v right now) | 16:39 |
| TheJulia | it needs the tempest test to actually merge | 16:40 |
| TheJulia | then it should be good to go | 16:40 |
| JayF | https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/854031 | 16:40 |
| JayF | ? | 16:40 |
| TheJulia | yup | 16:40 |
| JayF | ack; I'd love to get that in before landing Nisha_Agarwal's fixes | 16:40 |
| TheJulia | tempest is the chicken before the egg case :( | 16:41 |
| TheJulia | ++ | 16:41 |
| JayF | iurygregory: around? Got a sec to land https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/854031 ? | 16:41 |
| frickler | maybe someone can have a look at the testing template change for antelope https://review.opendev.org/c/openstack/ironic/+/859040 , py310 seems to be passing now, the failures look unrelated but I don't know enough ironic to be sure | 16:42 |
| JayF | frickler: I'll look, ty for the pointer | 16:43 |
| frickler | would be good to get that landed soonish to avoid outdated testing | 16:43 |
| JayF | That's a real unit test failure on 310 | 16:43 |
| JayF | Oh, it ran again and passed, I see it | 16:44 |
| * JayF misses the latest-CI-run-at-the-top-of-zuul | 16:44 | |
| JayF | I think > metalsmith-integration-glance-centos8-legacy might be failing ~everywhere? | 16:44 |
| JayF | or at least flakey ~everywhere | 16:45 |
| JayF | > To utilize this module, the installed version of the openstacksdk library MUST be <=0.98.999. | 16:45 |
| JayF | heh, yep, that's what rpittau was working on I think | 16:45 |
| Nisha_Agarwal | TheJulia, JayF just a suggestion for the anaconda CI test....if we have the ISO downloaded and mounted on the webserver and then given that as the URL, then the gate will work end-to-end and the node will move to active within some 10-15 min | 16:56 |
| TheJulia | that time is your local test correct? | 16:57 |
| Nisha_Agarwal | Yes | 16:58 |
| TheJulia | I'm not sure we want to download the multigigabyte iso image to a test node | 16:58 |
| Nisha_Agarwal | hmm can it be downloaded once and kept mounted in some webserver which is local to CI | 16:58 |
| TheJulia | Infra mirror can't take the additional artifacts | 16:59 |
| Nisha_Agarwal | infra is the central webserver for CI? | 17:00 |
| TheJulia | OpenDev CI is distributed among many cloud operators | 17:00 |
| TheJulia | and the maintain a AFS mirror cache which is exceeding 80% of space utilized already | 17:00 |
| Nisha_Agarwal | hmmm | 17:01 |
| iurygregory | JayF, looking now | 17:01 |
| JayF | WDYT about making that metalsmith job non-voting while we troubleshoot it? | 17:15 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: CI: Temporarily make metalsmith job non-voting https://review.opendev.org/c/openstack/ironic/+/860861 | 17:24 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: CI: Temporarily make metalsmith job non-voting https://review.opendev.org/c/openstack/ironic/+/860861 | 17:35 |
| JayF | dtantsur: iurygregory: rpittau: You all are still documented as release liasons, I believe. They may want you to vote on https://review.opendev.org/c/openstack/releases/+/860499 (if you don't want to be release liasons anymore; lmk and I can update the docs) | 17:42 |
| iurygregory | JayF, will add to take a look later today | 17:43 |
| JayF | ack; it's just queens eol stuff | 17:43 |
| iurygregory | ack o/ | 17:43 |
| JayF | That chat message is, as much as anything else, confirming you all want to remain as liasons :) | 17:43 |
| iurygregory | JayF, from my side I would like to continue as a liason =) | 18:01 |
| JayF | :D | 18:02 |
| iurygregory | yeah, metalsmith CI is broken | 18:05 |
| iurygregory | something related to openstacksdk... | 18:05 |
| iurygregory | going to try to bump the requirements to see how it goes | 18:07 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/metalsmith master: DNM - Testing CI https://review.opendev.org/c/openstack/metalsmith/+/860865 | 18:08 |
| opendevreview | Merged openstack/ironic-tempest-plugin master: Tempest test for anaconda deploy https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/854031 | 18:11 |
| JayF | iurygregory: that's not going to fix; it blows up early with a specific warning like it's explicitly checking for openstacksdk <1 | 18:15 |
| JayF | iurygregory: so I think we'll have to find that warning + remove it, too? | 18:15 |
| iurygregory | oh right is <0.98 | 18:16 |
| iurygregory | the funny thing is figure out from where this message comes from lol | 18:18 |
| JayF | Yeah, I was going to go hunting for that here shortly lol | 18:18 |
| JayF | honestly I need like, a metalsmith 101 class at some point | 18:18 |
| JayF | I am not sure I know much beyond the 'elevator pitch' for it | 18:19 |
| * TheJulia is starting to get a migraine thanks to inspector's functional tests | 18:19 | |
| iurygregory | https://github.com/openstack/metalsmith/blob/f87484776b2df6fa1010cdf85750ef49235955ea/metalsmith_ansible/roles/metalsmith_deployment/tasks/main.yml#L30-L37 | 18:19 |
| stevebaker[m] | good morning | 19:21 |
| TheJulia | good morning | 19:28 |
| JayF | o/ | 19:59 |
| iurygregory | stevebaker[m], morning, since you more expertise in metalsmith, do you know about this error https://zuul.opendev.org/t/openstack/build/2051a54acffa4e8aa96966050c65a771 ? So far I was only able to discover it comes from https://github.com/openstack/metalsmith/blob/f87484776b2df6fa1010cdf85750ef49235955ea/metalsmith_ansible/roles/metalsmith_deployment/tasks/main.yml#L30-L37 but nothing else .-. | 20:07 |
| JayF | iurygregory: I am at a stopping point on my other items, I can dig with you now | 20:09 |
| JayF | I bet it's the ansible module https://github.com/openstack/metalsmith/blob/f87484776b2df6fa1010cdf85750ef49235955ea/metalsmith_ansible/ansible_plugins/modules/metalsmith_instances.py#L22 | 20:11 |
| JayF | that on import is going kaboom | 20:11 |
| iurygregory | ok, so this is the openstack ansible module | 20:12 |
| iurygregory | maybe we should specify the version we need to install... | 20:12 |
| JayF | I'm curious if the cutover to default newer ansible is what broke this | 20:12 |
| iurygregory | I do remember some problems related to the openstacksdk + openstack ansible collections... | 20:13 |
| JayF | fungi has been warning us about it in TC meetings, and the timing seems too good to be a coincidence | 20:13 |
| iurygregory | hummm | 20:13 |
| iurygregory | can be.. | 20:13 |
| iurygregory | first failure was on oct3 but not with same error | 20:13 |
| iurygregory | problem started on 2022-10-07 | 20:14 |
| iurygregory | https://zuul.opendev.org/t/openstack/builds?job_name=metalsmith-integration-glance-centos8-legacy&project=openstack/ironic | 20:14 |
| JayF | iurygregory: https://lists.opendev.org/pipermail/service-announce/2022-October/000046.html | 20:15 |
| JayF | YUP | 20:15 |
| stevebaker[m] | Nisha_Agarwal: hey are you still about? | 20:15 |
| JayF | so I think this means we can workaround it for now | 20:16 |
| JayF | iurygregory: so you keep digging root-cause-fixes, I'm going to put up a "try the older ansible version" fix | 20:16 |
| fungi | JayF: iurygregory: https://review.opendev.org/860474 updated the default ansible version to 6 at 2022-10-06 16:31:21 utc | 20:16 |
| JayF | Yeah, that timing 100% lines up with our metalsmith failures | 20:17 |
| JayF | I'm going to do a manual set of `ansible_version` to see if it fixes the jobs | 20:17 |
| JayF | to buy us time to get the new ansible support in place | 20:17 |
| fungi | you can override the ansible version in jobs back to 5 for now, but we'll soon be upgrading to a version of zuul which lacks support for ansible 5 | 20:17 |
| fungi | there will be more announcements of course | 20:18 |
| fungi | also i recommend following the service-announce@lists.opendev.org mailing list, where we try to make these things more visible | 20:18 |
| opendevreview | Jay Faulkner proposed openstack/metalsmith master: CI: Force ansible to version 5 https://review.opendev.org/c/openstack/metalsmith/+/860874 | 20:19 |
| fungi | the final announcement about the default update to 6 can be found here if you need to reference it: https://lists.opendev.org/pipermail/service-announce/2022-October/000046.html | 20:19 |
| JayF | fungi: tbh, I saw the notices, just didn't have awareness this was likely to break an ironic job, and this is a project folks don't work on much it didn't get tesetd | 20:19 |
| JayF | yeah I linked that to iury above^^^ | 20:19 |
| JayF | fungi: (Metalsmith, I mean, isn't worked on much)\ | 20:20 |
| fungi | also we usually don't update the default until the new version has been available in zuul for a while, so while i know it's not easy to find time to try things out, there is always the possibility of tipping jobs up to the newer ansible in a test change before the default changes, if there's anything you're particularly worried about | 20:21 |
| opendevreview | Jay Faulkner proposed openstack/ironic master: DNM: CI TESTING FOR METALSMITH https://review.opendev.org/c/openstack/ironic/+/860876 | 20:21 |
| JayF | fungi: I was saying this morning I honestly know very little about metalsmith outside of the "elevator pitch" for it. So I'm going thru metalsmith 101 right now trying to unbreak the gate :) | 20:22 |
| fungi | zuul has been lagging behind on ansible support updates because we had some security-impacting rearchitecture needed in order to support more recent ansible versions, but hopefully that churn is almost done (we'll be caught up when ansible 5 support is dropped) | 20:22 |
| iurygregory | ok nice! | 20:23 |
| JayF | So we still need to have someone look at making ansible 6 work with metalsmith | 20:23 |
| JayF | Do we have an SME for metalsmith? | 20:23 |
| iurygregory | not sure what SME stands for .-. | 20:24 |
| JayF | subject matter expert | 20:24 |
| iurygregory | ok | 20:24 |
| iurygregory | I would say Dmitry, Steve and Harald | 20:26 |
| fungi | also the first mate on captain hook's pirate ship from peter pan | 20:26 |
| fungi | nevermind, that was smee | 20:27 |
| iurygregory | lol :D | 20:27 |
| TheJulia | fungi: but now I have to watch Hook... | 20:34 |
| jrosser_ | imho what is happening here is that the version of the openstack collection in ansible 6 has firm ideas about what version of openstack-sdk it wants, and that job installs whatever openstack-sdk upper-constraints has. those two things are incompatible. | 20:34 |
| jrosser_ | it's probably something to do with the ansible task running in whatever python environment got set up in the job with a recent version of openstack-sdk, rather than a python environment set up like the one ansible is installed into | 20:38 |
| JayF | Hm. That would imply our metalsmith testing, generally, is invalid then | 21:13 |
| JayF | if it's not running in the test environment / with the test ansible, right? | 21:13 |
| JayF | and instead is using the ansible install that manages CI | 21:13 |
| stevebaker[m] | Nisha_Agarwal: I've added a comment to the bz regarding proliantutils stable/xena pip failures. Let me know if the issue should be raised elsewhere https://bugzilla.redhat.com/show_bug.cgi?id=2122161 | 21:30 |
| stevebaker[m] | Nisha_Agarwal: on stable/xena, there is a dependency conflict for sushy==4.3.0 vs sushy<4 | 21:31 |
| stevebaker[m] | JayF: Those jobs have never installed a test-env ansible, so in that respect they're working as intended. However this is not the first time there have been problems caused by using the CI ansible | 21:53 |
| JayF | stevebaker[m]: ack | 22:01 |
| * TheJulia as battles inspector locking up trying to use autocommit | 22:02 | |
| JayF | stevebaker[m]: TheJulia: Metalsmith gate fix https://review.opendev.org/c/openstack/metalsmith/+/860874 | 22:02 |
| JayF | iurygregory: ^ | 22:02 |
| JayF | you can see ironic ci happy here -> https://review.opendev.org/c/openstack/ironic/+/860876 | 22:03 |
| stevebaker[m] | +2 | 22:03 |
| JayF | stevebaker[m]: single cores can land CI fixes ;) | 22:03 |
| stevebaker[m] | yes they can | 22:06 |
| JayF | \o/ | 22:06 |
| JayF | ty | 22:07 |
| stevebaker[m] | JayF: is the issue with ansible 6 that openstacksdk isn't installed anymore, because the (reduced) core modules don't use it? | 22:08 |
| JayF | stevebaker[m]: the jobs are failing explicitly when importing ansible.[whatever].openstack | 22:08 |
| JayF | because it detects a too-new openstacksdk | 22:08 |
| JayF | it doesn't even /try/ to run | 22:08 |
| JayF | stevebaker[m]: https://zuul.opendev.org/t/openstack/build/2051a54acffa4e8aa96966050c65a771 | 22:08 |
| JayF | > To utilize this module, the installed version of the openstacksdk library MUST be <=0.98.999. | 22:08 |
| JayF | So that is worked around now; but Zuul support for Ansible 5 is going away so we *do* have to fix it long term. | 22:09 |
| opendevreview | Jay Faulkner proposed openstack/ironic bugfix/19.0: Redfish: Consider password part of the session cache https://review.opendev.org/c/openstack/ironic/+/860891 | 22:09 |
| JayF | TheJulia: https://review.opendev.org/c/openstack/ironic/+/860853 the failure on this (your CI change for train) is weird, and maybe valid? I can't tell... | 22:11 |
| JayF | TheJulia: not urgent; but wanted to call your attention to it looking "real" (if it's just broken environmentally I can try to find time to look if you need) | 22:12 |
| TheJulia | w/r/t zuul versions tracking ansible versions, it will move fairly swiftly AIUI. | 22:12 |
| hjensas | JayF: stevebaker[m]: I ran into this ansible issue as well, it needs 2.x.x release of the ansible openstack.cloud collection but that has not been released yet I think. See: https://opendev.org/openstack/ansible-collections-openstack#breaking-backward-compatibility-warning | 22:12 |
| stevebaker[m] | Maybe all it needs is changing the import path to from ansible_collections.openstack.cloud.plugins.module_utils.openstack | 22:12 |
| JayF | I am almost certainly the worst possible person to run this fix; despite my devops background I know nearly zero about ansible | 22:13 |
| TheJulia | JayF: looks like something zuul... :\ | 22:13 |
| JayF | TheJulia: ack; I'm just going to recheck and see if it repeats then | 22:13 |
| *** rcastillo_ is now known as rcastillo | 23:30 | |
| opendevreview | Merged openstack/metalsmith master: CI: Force ansible to version 5 https://review.opendev.org/c/openstack/metalsmith/+/860874 | 23:30 |
| TheJulia | well, I know where inspector is going sideways in functional tests for the db | 23:39 |
| TheJulia | why... remains elusive :( | 23:39 |
| TheJulia | a fresh brain tomorrow and a revisit on sqlalchemy drivers is likely going to occur | 23:39 |
| JayF | If you want to post it and write down what you know about the problem, I can take a look in the morning as well. | 23:55 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!