Wednesday, 2022-11-16

vanougood morning ironic02:10
arne_wiebalckGood morning Ironic!07:31
*** akahat|ruck is now known as akahat|rover|lunch07:35
*** akahat|rover|lunch is now known as akahat|rover08:28
rpittaugood morning ironic! o/09:15
rpittauFYI switch to jammy is in 2 days, we should be ok, but please keep an eye on https://review.opendev.org/c/openstack/ironic/+/863873 more info at https://etherpad.opendev.org/p/migrate-to-jammy09:37
dtantsurrpittau: I'm fine with keeping an eye on that patch, but I suspect you meant another one ;)09:52
rpittauwellll.... yeah..... :/09:55
rpittauthis is the one! https://review.opendev.org/c/openstack/ironic/+/86132509:56
opendevreviewArne Wiebalck proposed openstack/ironic master: [doc] Add documentation on SMART test after disk burn-in  https://review.opendev.org/c/openstack/ironic/+/86455012:36
arne_wiebalckrpittau: ^^ (thanks for spotting it!)12:37
arne_wiebalckrpittau: zuul is failing in a network test, this should be unrelated to a string change in the docs :)13:40
rpittauI really hope it is! :)13:41
*** akahat|rover is now known as akahat|ruck|afk14:03
TheJuliaGood morning everyone14:09
opendevreviewVerification of a change to openstack/ironic master failed: [doc] Add documentation on SMART test after disk burn-in  https://review.opendev.org/c/openstack/ironic/+/86455014:11
kubajjHello everyone!14:14
TheJuliagood morning14:15
* TheJulia looks at meeting invite which is now before 6 AM14:18
dtantsurthat's evil14:25
dtantsuralso good morning TheJulia, kubajj 14:25
kubajjHi dtantsur14:25
TheJuliadtantsur: Yeah, I've requested it to be moved back just 15 minutes in the past and my request was ignored... soooo *shrug*14:26
dtantsur:(14:34
opendevreviewMerged openstack/ironic master: [doc] Add documentation on SMART test after disk burn-in  https://review.opendev.org/c/openstack/ironic/+/86455014:45
*** akahat|ruck|afk is now known as akahat|ruck15:00
opendevreviewHarald Jensås proposed openstack/ironic master: Add ports statistics to tools/benchmark scripts  https://review.opendev.org/c/openstack/ironic/+/86476216:33
opendevreviewJulia Kreger proposed openstack/ironic-specs master: Add a shard key  https://review.opendev.org/c/openstack/ironic-specs/+/86180316:34
hjensasTheJulia: https://paste.opendev.org/show/boW6bFAIxuh4bMcYsvse <- the via ports API call pattern improvements looks good so good I think I must have made a mistake?16:35
TheJuliawow!16:36
JayFhjensas: or you fixed like, 8 years of mistakes16:38
TheJulia3533% improvement16:40
hjensasJayF: that would be good. *blink* I can't belive it.16:40
TheJuliathat is absurd16:40
JayFhjensas: you're working on a python project that uses a heavy ORM layer. I'm impressed but not in disbelief.16:41
JayFhjensas: python - the real cause of climate change /s16:41
TheJuliaheh16:41
TheJuliaYeah, the loop/iteration causes a lot of small db queries and that adds up to be a ton over time16:42
TheJuliaso definitely not in land of disbelief, but quite impressed :)16:42
TheJuliahjensas: this is blog worthy, fwiw16:43
JayFhjensas: this is summit-worthy16:45
JayFhjensas: TheJulia: finding performance improvements in frequently hit API endpoints16:45
opendevreviewJulia Kreger proposed openstack/ironic-specs master: Add Ironic work items for 2023.1  https://review.opendev.org/c/openstack/ironic-specs/+/86253816:45
JayFthere's your title, go write the paper16:46
TheJuliaActually yes, it *is* summit worthy16:46
TheJuliai broke some minds in berlin talking about the ports because people just didn't seem to grok just how much a performance impact spreading out that loop creates16:46
JayFseriously; about node and ports16:46
JayFI bet there are other projects with similar benefits to make16:46
TheJuliaI did w/r/t nodes, but there is even another piece in nodes which can be improved, chassis_uuid....16:47
TheJuliaand that would be awesome to revisit16:47
JayFheh16:47
TheJuliabtw, JayF, I rebased the work items, clean minus the index, re-applied my +216:48
JayFI figured; ty16:48
TheJulianp16:49
TheJuliaI also fixed the shard key validation error16:49
rpittaugood night! o/17:05
TheJuliagoodnight rpittau 17:07
TheJuliahttps://review.opendev.org/c/openstack/ironic-specs/+/861803 wants reviews so it is not sitting off in a corner feeling like an abandoned spec17:10
* TheJulia wonders if we can corgi-gif specs....17:11
TheJuliastevebaker[m]: umm.. w/r/t the failure on :\ https://review.opendev.org/c/openstack/ironic-python-agent/+/862656 18:02
stevebaker[m]Hmm? 18:05
JayFTheJulia: recheck that before digging18:07
JayFTheJulia: I suspect some of those jobs may be fixed by gmann's patch to tag our branches for ironic-tempest-plugin properly18:07
TheJuliaahh18:07
TheJuliait failed with the dib build though18:08
TheJulianot tempest18:08
JayFaw hell18:08
opendevreviewMerged openstack/ironic-lib stable/victoria: CI: Various fixes  https://review.opendev.org/c/openstack/ironic-lib/+/86017518:12
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: Use declarative reader/writer with enginefacade  https://review.opendev.org/c/openstack/ironic-inspector/+/46376818:38
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep  https://review.opendev.org/c/openstack/ironic-inspector/+/86073118:38
TheJuliahjensas: and wow, those are barebones ports objects, no llc data18:40
TheJuliahjensas: https://review.opendev.org/c/openstack/ironic/+/862933 is awesome, thanks!18:48
TheJuliakubajj: o/ so, I'm not sure why your second patch is failing19:01
hjensasTheJulia: thanks, I'll look for other places where we could use the same pattern.19:02
kubajjTheJulia: I am not really sure either. I did not change much.19:02
TheJuliathings *look* like it seems to be working, but I guess I'm wondering what the overall goal of the second change is in the grand scheme of things. Maybe just bridge/duplicate until the rest of the code is moved in?19:03
TheJuliahjensas: ++ chassis_uuid ;)19:03
hjensasTheJulia: yes, probably portgoup and allocations as well.19:04
TheJulia++19:04
* TheJulia is feeling really low energy as the day is going on19:05
TheJuliakubajj: Anyhow, I would consier continuing with the other data model stuffs so you at least have that foundation laid. That one in particular, I'm wondering if it is transient, or if we're swallowing errors in an exception catch someplace19:06
TheJuliakubajj: in other words, debug logging might help19:06
JayFkubajj: also, if you wanna put an early review on https://review.opendev.org/c/openstack/ironic/+/864236 I wouldn't hate it19:06
* TheJulia runs to the market around the corner to get something to boost the energy.19:07
* JayF needs to unanger the linter19:07
opendevreviewMerged openstack/bifrost master: Switch bifrost to use the dnsmasq DHCP driver  https://review.opendev.org/c/openstack/bifrost/+/85168219:24
opendevreviewMerged openstack/ironic-python-agent stable/yoga: Drop python2 from bindep.txt  https://review.opendev.org/c/openstack/ironic-python-agent/+/86265319:24
TheJuliajoy linter fun19:25
opendevreviewMerged openstack/ironic-python-agent stable/xena: Drop python2 from bindep.txt  https://review.opendev.org/c/openstack/ironic-python-agent/+/86265419:30
TheJuliais review suddenly super slow?19:31
TheJuliafor anyone else that is19:31
JayFthat link opened pretty quickly for me ^19:31
stevebaker[m]TheJulia: ok I think that failure will stop with newer diskimage-builder, I'll recheck https://review.opendev.org/c/openstack/requirements/+/86265119:33
TheJuliaack19:33
TheJuliaoh wow it is just crawling for me19:39
TheJuliasome stuff works just fine19:39
TheJuliaonly other slow thing was bugzilla... and arguably... that can be normal19:39
opendevreviewHarald Jensås proposed openstack/ironic master: Use association_proxy for port groups node_uuid  https://review.opendev.org/c/openstack/ironic/+/86478119:43
TheJuliaJayF: I htink if you fix the import order and take care of the pep8 stuffs, you'll be good to go19:45
TheJuliaAlso, raised a compound index question19:46
JayFdo I need to test the object layer stuff?19:46
TheJuliait may not matter in reality19:46
JayFI have that as a todo but looking at the tests not sure there's anything to test19:46
TheJuliaI'd just have a test that ensures we get a list from the db19:46
TheJuliayou'll be adding more to ensure it works as you go19:46
JayFack; will-do19:47
JayFhow close is kubajj's first patch to landing?19:47
JayFI guess it shouldn't matter; none of this should merge until the Ironic+Nova specs are seen as OK19:47
TheJuliaI +2'ed it19:47
TheJuliaI think it is good to go from a data storage point of view19:48
JayFso if I review it, you think it's OK to land that piece now?19:48
TheJulianormally we would want to see more, but I think he is going to end up with ~3 more tables in the grand scheme of things19:48
TheJuliayeah, I think so19:48
JayFeven though bugfix releases are cut very, very soon?19:48
TheJuliait wouldn't be harmful19:48
TheJuliayeah, I think it is fine19:48
TheJuliaI don't see us needing to be hesitant for bugfixes since we can always backport to them to fix them if there is a huge issue19:49
opendevreviewMerged openstack/ironic-python-agent stable/wallaby: Drop python2 from bindep.txt  https://review.opendev.org/c/openstack/ironic-python-agent/+/86265519:49
JayFthat's mostly my thought; but with my oper hat on I'd probably be mildly steamed at having to run a ~useless migration19:49
kubajjJayF: what is a shard?19:56
TheJuliayou have to run it on any upgrade, at least that is the standard pattern19:56
TheJuliathat being said, I do want to go ahead and make us do auto-schema upgrades19:56
TheJuliaor at least, support doing it19:56
JayFkubajj: so sharding is a generic term for when you split a big group of things into smaller groups to make them easy to handle; for Ironic; we're adding a shard_key which external services (like nova or networking-baremetal) can use to shard with assistance from Ironic's info20:01
JayFkubajj: how we're doing that for Ironic is specified here https://review.opendev.org/c/openstack/ironic-specs/+/86180320:01
JayFkubajj: so, as described in that document, the shard table will be used to hold rough, cached information about how many nodes per shard20:02
JayFkubajj: essentially a cached output of `SELECT count(*), shard_key from nodes group by shard_key;` 20:03
JayF(that sql may be bad, it's off the top of the head)20:03
opendevreviewSteve Baker proposed openstack/bifrost master: Copy shim and grub into tftp and http directories  https://review.opendev.org/c/openstack/bifrost/+/84924720:06
opendevreviewSteve Baker proposed openstack/bifrost master: Remove enable_uefi_ipxe  https://review.opendev.org/c/openstack/bifrost/+/84924820:06
opendevreviewSteve Baker proposed openstack/bifrost master: Support PXE network boot with grub  https://review.opendev.org/c/openstack/bifrost/+/80722020:06
opendevreviewSteve Baker proposed openstack/bifrost master: Restore discovery for dnsmasq dhcp provider  https://review.opendev.org/c/openstack/bifrost/+/86478720:36
opendevreviewMerged openstack/ironic master: Implements node inventory: database  https://review.opendev.org/c/openstack/ironic/+/86256921:12
hjensasTheJulia: this test - https://opendev.org/openstack/ironic/src/branch/master/ironic/tests/unit/api/controllers/v1/test_node.py#L60321:54
hjensasWe don't allow deleting a chassis if there are nodes - https://opendev.org/openstack/ironic/src/branch/master/ironic/db/sqlalchemy/api.py#L122221:54
TheJuliathere is a possible race condition if your mid-transaction on one21:55
TheJuliaor if you update one and not the other21:55
TheJuliait is stupid weird, but turned out to be a legit sort of issue21:56
hjensasok, the actual user pattern would be unset chassis on node(s) then delete chassis?21:57
TheJuliaI think actually delete node and chassis and be in-flight on the nodes table21:58
TheJuliathe tl;dr is the test can get deleted in the end if a join is used21:58
hjensasok, yeah seems this could only happen with the _get_chassis_uuid pattern in the api controller.21:59
TheJuliayup22:00
TheJuliaspecifically more possible if you have a read replica in the mix22:00
TheJuliait hurt my head when I came across the bug because I think someone actuallly had logs of it happening22:01
opendevreviewJay Faulkner proposed openstack/ironic master: WIP: DB implementation of shards / shard_key  https://review.opendev.org/c/openstack/ironic/+/86423622:09
TheJuliawell https://zuul.opendev.org/t/openstack/build/8f87a555cf9c45fdb87b6966558440c1 feels new22:12
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: Use declarative reader/writer with enginefacade  https://review.opendev.org/c/openstack/ironic-inspector/+/46376822:32
TheJulialets see if that gets us back to 90%...22:33
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep  https://review.opendev.org/c/openstack/ironic-inspector/+/86073122:38
TheJuliamuch better22:39
opendevreviewJulia Kreger proposed openstack/ironic-inspector master: SQLAlchemy 2.0 prep  https://review.opendev.org/c/openstack/ironic-inspector/+/86073122:57
opendevreviewJay Faulkner proposed openstack/ironic master: DB & Object layer for of shards & node.shard_key  https://review.opendev.org/c/openstack/ironic/+/86423623:28
JayF^^ should be code-good, but can't land until the spec docs (and arguably shouldn't until the nova spec lands)23:30
JayFs/docs/lands/23:32

Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!