Friday, 2023-05-19

TheJuliaso I'm thinking we should just remove the test00:03
TheJuliawhich sounds awful00:03
TheJulia*but* these tests rely upon nothing triggering the object to get saved in the stack00:03
TheJuliaand obviously, that doesn't work so well in unit tests00:03
TheJuliabetter idea!00:05
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Try to isolate test failures in neutron vif logic  https://review.opendev.org/c/openstack/ironic/+/88359800:17
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Disable mysql counters for grenade  https://review.opendev.org/c/openstack/ironic/+/88273100:27
TheJuliaflat out disable for grenade run00:28
opendevreviewJulia Kreger proposed openstack/ironic-tempest-plugin master: CI: Advance stable jobs to 2023.1  https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/88360000:58
opendevreviewMerged openstack/ironic master: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88298502:21
opendevreviewIury Gregory Melo Ferreira proposed openstack/ironic master: Add DB model for Firmware  https://review.opendev.org/c/openstack/ironic/+/88303103:28
iurygregorygoing to test with Depends-On with the mysql counters to see03:29
opendevreviewVerification of a change to openstack/ironic master failed: Fix self_owned_node policy check  https://review.opendev.org/c/openstack/ironic/+/88259704:02
*** dmellado90 is now known as dmellado905:09
opendevreviewMerged openstack/ironic master: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88228509:25
MikeCTZAmorning ironic (well still morning here anyway)09:31
opendevreviewFabian Wiesel proposed openstack/sushy master: Handle session-uri in body  https://review.opendev.org/c/openstack/sushy/+/86796909:45
opendevreviewLana Kaleif proposed openstack/ironic bugfix/21.0: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88357210:31
opendevreviewLana Kaleif proposed openstack/ironic bugfix/21.2: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88357310:32
opendevreviewLana Kaleif proposed openstack/ironic bugfix/21.3: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88357410:32
opendevreviewLana Kaleif proposed openstack/ironic stable/2023.1: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88357510:32
opendevreviewLana Kaleif proposed openstack/ironic stable/zed: Fix anaconda stage2_id loading from image properties  https://review.opendev.org/c/openstack/ironic/+/88357610:33
opendevreviewVerification of a change to openstack/ironic master failed: Fix self_owned_node policy check  https://review.opendev.org/c/openstack/ironic/+/88259711:20
iurygregorygood morning Ironic11:51
TheJuliagood morning!13:34
TheJuliaiurygregory: https://review.opendev.org/c/openstack/ironic/+/88273113:36
TheJuliait wants a +2+A :)13:37
iurygregoryTheJulia, sure13:43
opendevreviewJulia Kreger proposed openstack/ironic stable/2023.1: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88357713:45
opendevreviewJulia Kreger proposed openstack/ironic stable/zed: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88357813:45
opendevreviewJulia Kreger proposed openstack/ironic stable/yoga: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88357913:45
opendevreviewJulia Kreger proposed openstack/ironic stable/xena: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88358013:45
opendevreviewJulia Kreger proposed openstack/ironic stable/wallaby: Fix Cinder Integration fallout from CVE-2023-2088  https://review.opendev.org/c/openstack/ironic/+/88358113:45
TheJuliawheeee13:46
TheJuliaclean all the way13:46
* TheJulia dances13:46
iurygregorybackport to all branches \o/13:46
TheJuliasigh "last_error": "Deploy step deploy.deploy failed with ReadTimeoutError: HTTPConnectionPool(host=\'tinycorelinux.net\', port=80): Read timed out.."14:08
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Change tinycore URL  https://review.opendev.org/c/openstack/ironic/+/88367614:22
TheJuliaiurygregory: ^^^ should make ironic-standalone-redfish happier14:22
TheJuliasince it uses it for virtual media attachment/ramdisk testing14:22
iurygregoryouch >.<, ack14:44
iurygregorylooking14:44
TheJuliahjensas: ... did you find anything further on that CI job failing on somehow not supporting vfat ?!?14:45
iurygregoryfunny thing to look at friday, understand why inspection was finished at 2023-05-16 14:45:02.741 but at 2023-05-16 15:24:12.730 inspector is saying Node a157ca55-7a07-4fcf-ae25-5e7604c270f6 is not active, its provision state is inspect failed 14:53
iurygregoryyay for friday...14:54
hjensasTheJulia: there was an error before it, permissions I think on writing the image. I had a meeting so did'nt dig deeper. - job seems to pass on the recheck :-/14:55
TheJuliahjensas: yeah, looks like /bin/sh was missing14:57
TheJuliainside of the image14:57
TheJuliajust.. bizzar14:57
iurygregoryhumm it's ok if the nic for pxe boot is bonded?15:10
opendevreviewVerification of a change to openstack/ironic master failed: CI: Disable mysql counters for grenade  https://review.opendev.org/c/openstack/ironic/+/88273115:17
TheJuliaiurygregory: hold one moment15:25
NobodyCamGood Morning Ironic folks. Thank you for the pointers yesterday. and of course....15:25
NobodyCamTGIF!15:25
TheJuliaiurygregory: https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#why-does-x-issue-occur-when-i-am-using-lacp-bonding-with-ipxe15:27
TheJuliaTranslation: pain is on this path, but it *can* be done15:27
TheJuliawheee https://review.opendev.org/c/openstack/ironic/+/88273115:31
* TheJulia is staritng to wonder if we need to just add a skip login option to tempest15:32
iurygregoryTheJulia, ouch >.< the funny thing is that it has ipv4 and ipv6 for the mac https://paste.opendev.org/show/bmaidOPjNegF78TmBwa9/15:33
iurygregoryI don't think this is a good idea .-.15:33
iurygregoryspecially for the ipmi scenario...15:33
TheJuliawell15:34
TheJuliaso those are unique IPs15:34
TheJuliafirst is a global v4, second is a link layer only v615:34
TheJuliaeach interface can have any number of IPs15:35
TheJuliadmitry and I have a disagreement in ?ipa? regarding type of v6 addressing actually15:35
TheJuliabecause I don't want us to record temporary addresses because it creates risk of non-working addresses 15:36
iurygregoryhummm15:39
iurygregoryI'm going to check ramdisk logs to see if I can find something15:39
iurygregorybecause it really doesn't make sense that inspector reported success and finished, but in 40min it was showing that could not find a node with the attributes .-.15:40
opendevreviewVerification of a change to openstack/ironic master failed: CI: Disable mysql counters for grenade  https://review.opendev.org/c/openstack/ironic/+/88273115:40
TheJuliaiurygregory: coudl conflicting tempest tests be running together?15:45
iurygregoryin https://review.opendev.org/c/openstack/ironic/+/882731 ?15:46
TheJuliathe machine never booted15:47
iurygregorywoot15:47
TheJuliait was unreachable, and rebooted into network boot, since this is snmp based, the machine never found dhcp15:48
TheJuliabest guess, neutron dhcp stalled out15:48
iurygregorywe changed a few things related to snmp 15:48
TheJuliatrue15:48
iurygregorywondering if there is a problem in the new packages we added 15:48
iurygregoryor missing something15:48
TheJuliaThey passed once, the console log showing it never dhcpe'ed seems telling to me15:48
iurygregoryhttps://i.redd.it/ybsfu8ntavc51.png15:49
iurygregoryDNS Haiku15:50
iurygregory<315:50
iurygregorygoing to grab lunch, brb15:50
TheJuliawe've alway ssort of seen dhcp sometimes not be up to date in time15:50
TheJuliajust... now it seems way worse since jammy15:50
opendevreviewJulia Kreger proposed openstack/metalsmith master: Reduce retry timeout for Metalsmith CI  https://review.opendev.org/c/openstack/metalsmith/+/88368316:16
opendevreviewMerged openstack/ironic master: CI: Try to isolate test failures in neutron vif logic  https://review.opendev.org/c/openstack/ironic/+/88359816:18
opendevreviewJulia Kreger proposed openstack/ironic master: CI: Move metalsmith to non-voting  https://review.opendev.org/c/openstack/ironic/+/88368416:26
*** melwitt_ is now known as melwitt17:11
opendevreviewMerged openstack/ironic master: [iRMC] Fix parse_driver_info bug enforcing SNMP v3 under FIPS mode  https://review.opendev.org/c/openstack/ironic/+/88135817:42
JayFfungi: fyi, https://review.opendev.org/c/openstack/ironic/+/882985 cinder CVE fix ironic fixes landed in master. We're backporting them as far as we can. Do you think the OSS{A,N} should be edited to reflect you need to upgrade Ironic, too?18:01
JayFTheJulia: rbac-scope-enforced jobs appear to be failing in 2023.1 and zed with the same error (on your patch)18:03
TheJulianon-voting on master18:03
TheJuliaand we need to land fixes it seems18:03
JayFI was going to ask if it was related to your patch or just broken 18:04
JayFsounds like the answer is "just broken"18:04
TheJuliayeah18:04
TheJuliaunfortunately18:04
JayFI suggest we mark it nv on stable branches, too, so we can unbreak cinder18:04
JayFespecially if that job fails without your change18:04
* JayF pushes a dummy change to test18:04
TheJuliain theory, that might be a waste of time if we can get the fixes merged18:04
TheJuliaCI seems to be getting happier on master18:04
TheJuliabut we may need to backport the rbac fix too18:05
TheJuliaso the tests will work18:05
JayFdo you know which fix right off? I can go digging if needed18:05
TheJuliaso -nv is the fastest route, if you want to backport an nv change, I can approve18:05
TheJuliaI can look, but not right now18:05
JayFI've got he git log open, will see if I can nail it down18:05
fungiJayF: we can add an errata 4 to ossa-2023-003 with links to the ironic patches, propose a change to the openstack/ossa repo and i'll review it18:06
JayFfungi: ack; we'll get it backported thru first, just was curious what the next step was18:07
fungithere's already 3 errata for that ossa, so another isn't going to hurt ;)18:07
TheJuliaJayF: hasn't merged yet on master18:07
JayFTheJulia: ??? https://review.opendev.org/c/openstack/ironic/+/88298518:08
JayFthat is landed on master18:08
JayFor did you mean the rbac fixes18:08
JayFoh you meant the rbac fixes18:08
TheJuliarbac18:08
TheJuliatempest needs a fix too, which is why we should just non-voting the stable branches until we get it sorted18:08
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.1: Make rbac enforced test non-voting for the time being  https://review.opendev.org/c/openstack/ironic/+/88358418:09
opendevreviewJay Faulkner proposed openstack/ironic stable/zed: Make rbac enforced test non-voting for the time being  https://review.opendev.org/c/openstack/ironic/+/88358518:09
TheJuliahttps://review.opendev.org/c/openstack/ironic/+/882597 needs to merge to make tempest test fix possible18:09
JayFack I'll try to usher it thru ci18:09
TheJuliaack18:10
JayFwe also need to backport the mysql grenade fixes, I suspect18:10
* JayF sees a grenade failure on yoga18:10
opendevreviewJay Faulkner proposed openstack/ironic stable/yoga: Make rbac enforced test non-voting for the time being  https://review.opendev.org/c/openstack/ironic/+/88358618:11
opendevreviewJay Faulkner proposed openstack/ironic stable/xena: Make rbac enforced test non-voting for the time being  https://review.opendev.org/c/openstack/ironic/+/88358718:11
JayFTheJulia: https://review.opendev.org/c/openstack/ironic/+/883581 BFV job itself failed on wallaby, volume could not be found :( 18:12
JayFyou're probably not worried about it now, but letting you know since I looked at the logs18:12
* JayF will comment it there18:12
TheJuliaeh, wallaby I knew might be an issue18:15
TheJuliaI'll have to look when I get time, likely sometime next week18:15
JayFit'll likely take us that long to get the ones in front of it landed18:15
TheJuliayup18:15
JayFI wouldn't mind a +2A on all those backports of the -nv18:15
TheJuliasure18:15
JayFand I'll handle ci to get them in18:16
TheJuliawhen I can ctxt switch18:16
JayFjust sometime today, really, it'll take CI a few hours18:16
opendevreviewVerification of a change to openstack/ironic master failed: CI: Change tinycore URL  https://review.opendev.org/c/openstack/ironic/+/88367618:16
JayFI was already thinking of flexing out some time this afternoon, doing that to try and get these landed seems extra good now18:16
opendevreviewJulia Kreger proposed openstack/ironic master: DPU modeling - parent_node DB/Model/API  https://review.opendev.org/c/openstack/ironic/+/88011418:19
opendevreviewJulia Kreger proposed openstack/ironic master: execute on child node support  https://review.opendev.org/c/openstack/ironic/+/88054518:19
opendevreviewJulia Kreger proposed openstack/ironic master: Add hold steps  https://review.opendev.org/c/openstack/ironic/+/87906018:19
JayFaw hell18:19
JayFI think that CI failure on tinycore18:19
JayFis a real, net-new race in unit tests caused by no autocommit18:20
JayF2023-05-19 16:50:51.326653 | ubuntu-focal |     oslo_db.exception.DBError: (sqlite3.InterfaceError) Cursor needed to be reset because of commit/rollback and can no longer be fetched from.18:20
JayFlooks like basically without autocommit you have to catch, handle any errors, then refresh the connection, otherwise next use of that cursor throws18:20
TheJuliayup, patch running tests now18:25
TheJulia(or you know, actually don't let the thing race upfront)18:25
JayFyou already knew about this, already had a patch up? 18:25
JayFwow18:25
TheJuliaeh, in progress18:25
* JayF just trying to keep up18:25
TheJuliawe've see similar things before, I've just tried to avoid changing everything18:26
TheJuliaI'm much more of a "change one thing at a time" person18:26
JayFI'd also suggest if this is the first time we've seen it18:26
JayFwe're operating in the "really really really rare" area 18:26
JayFs/the/one of the/18:26
TheJuliawe've seen a handfull of similar ones before18:26
TheJuliaI went and did weeding on the db access pattern as a result and that is the first in ?2? weeks like that18:26
TheJuliaI do need to get a dev machine setup with py3.1118:27
TheJuliaI'd likely be able to reproduce some of these race condition issues18:28
JayFare you familiar with pyenv? I think it's pyenv...18:28
JayFhttps://realpython.com/intro-to-pyenv/ yeah pyenv18:28
JayFshould be able to give you a working py311 pretty trivially on fedora18:28
JayFI don't use it because it doesn't work so well with gentoo (and gentoo usually has all the pythons that are reasonable in repo)18:28
opendevreviewJulia Kreger proposed openstack/ironic master: CI: DB: Don't return inside of node get wrappers  https://review.opendev.org/c/openstack/ironic/+/88369218:32
TheJuliawheeeeeeeeeeeeeeeeeeeeeeee18:32
TheJuliaJayF: somewhat, just I don't run fedora on this computer18:32
TheJuliaI'm on my rhel machine right now18:32
TheJuliamy next one will be fedora or something else, haven't decided yet18:33
JayFit should still mostly work as long as you get gcc on it18:33
opendevreviewVerification of a change to openstack/ironic master failed: CI: Change tinycore URL  https://review.opendev.org/c/openstack/ironic/+/88367618:33
* TheJulia tosses tommatoes at CI18:33
JayFthat is the ARM bug18:33
JayFwhere it's like, the recheck is rechecking so it tells you LOL FAILED a bonus time18:34
JayFjust to drive you a little more crazy18:34
TheJuliaJayF: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_f6b/883676/1/gate/openstack-tox-py38/f6b76e5/testr_results.html <--- wheeeeeeeee same error18:34
TheJuliaoh18:34
JayFthat's the one I saw, isn't it?18:34
* TheJulia tosses more tomattoes18:35
TheJuliayeah18:35
* TheJulia wonders if a BLT is a good idea for lunch18:35
JayFI'm hungry, but also afraid if I ate a good lunch my already-tired-body would stop braining harder lol18:36
TheJuliaheh18:37
TheJuliaso....18:38
TheJuliaif cinder is broken, it occurs to me we need to no-vote both to fix it18:39
JayFoh hell18:39
JayFso we need to bail all those stable -nv patches18:39
JayFand instead -nv both jobs18:40
TheJuliayup18:42
TheJuliathe downside of being in our position :(18:42
JayFI'll do it18:44
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.1: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88369718:49
opendevreviewJay Faulkner proposed openstack/ironic stable/zed: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88358818:49
opendevreviewJay Faulkner proposed openstack/ironic stable/yoga: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88358918:49
opendevreviewJay Faulkner proposed openstack/ironic stable/xena: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88371018:49
JayFwallaby doesn't need it, it doesn't have an rbac job at all18:50
TheJuliaJayF: syntax errors19:00
JayFwill check19:01
TheJulia: on end of line19:03
JayFoof19:04
opendevreviewMerged openstack/ironic master: CI: Disable mysql counters for grenade  https://review.opendev.org/c/openstack/ironic/+/88273119:04
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.1: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88369719:05
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.1: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88369719:06
JayFI'll wait for CI to at least start before going backport-crazy on that one19:07
opendevreviewJay Faulkner proposed openstack/ironic stable/2023.1: CI: Disable mysql counters for grenade  https://review.opendev.org/c/openstack/ironic/+/88369919:08
TheJuliais 2023.1 also on jammy/python 3.11?19:11
JayFI don't think so; but I think it's still valuable to disable those perf counters on stable branches since we don't look at them19:12
JayFand random perf issues causing grenade failures is not uncommon for us :D19:12
opendevreviewJulia Kreger proposed openstack/ironic master: Enable vendor interfaces to be called as steps  https://review.opendev.org/c/openstack/ironic/+/87908919:13
TheJuliatrue19:14
JayFoh, I forgot about bugfix branches19:15
JayFyou know what, I'm just going to skip them for this19:15
JayFthe plate is too full and once all the fixes are in those can be unwound19:15
JayFunless there's a strong objection19:16
TheJulianone from me19:16
TheJuliaI've done the same, primary points and if they need something they can get it19:16
TheJuliaonce the bulk of the consumer/use consumption point is addressed19:17
opendevreviewJay Faulkner proposed openstack/ironic stable/zed: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88358819:17
JayFI also think it's reasonable to assume19:17
JayFmost standalone users are going to be less concerned about cinder breakages lol19:18
opendevreviewJay Faulkner proposed openstack/ironic stable/yoga: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88358919:18
opendevreviewJay Faulkner proposed openstack/ironic stable/xena: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88371019:18
JayFTheJulia: fyi I'm probably going to bail around 2100 utc (2ish hours) and will poke at these patches over the weekend to try and get as many of them landed as possible19:19
JayFI'm going to draft that ossa errata too19:19
TheJuliaok19:22
TheJuliaJayF: ack, I'm likely bailing around then as well, possible lunch soon() as well19:23
TheJuliaAlso need to work on my summit presentation :(19:23
JayFoh yeah, lunch, I bailed on my trip to get lunch b/c I saw the syntax err message19:23
JayFI should do that now lol19:23
TheJulia++19:25
JayFthose z/y/x backports of the CI job are syntactically correct20:10
JayFif someone wants to at least +2 them (or +2A) I'll make sure they land over the weekend20:11
TheJuliawow, so in october Ironic is going to need a birthday cake20:13
TheJuliaactually birthday wise, we're only 17 days after it's 10th brithday...20:21
JayF10 year anniversary blogpost for ironicbaremetal.org time20:21
JayFlol20:21
TheJulia++20:22
TheJuliaokay, about 30% done, not bad for a friday afternoon with my presentation21:30
opendevreviewMerged openstack/ironic master: CI: Change tinycore URL  https://review.opendev.org/c/openstack/ironic/+/88367622:23
opendevreviewMerged openstack/ironic master: Fix self_owned_node policy check  https://review.opendev.org/c/openstack/ironic/+/88259722:23
opendevreviewVerification of a change to openstack/ironic stable/2023.1 failed: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88369722:24
NobodyCamwow 10 years22:38
opendevreviewMerged openstack/ironic stable/zed: [stable-only] [CI] BFV, RBAC jobs marked non-voting  https://review.opendev.org/c/openstack/ironic/+/88358823:37

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!