| *** ice45 has quit IRC | 00:04 | |
| *** lmcgann has quit IRC | 00:11 | |
| *** k_mouza has joined #openstack-ironic | 00:27 | |
| *** k_mouza has quit IRC | 00:32 | |
| *** gyee has quit IRC | 00:32 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic master: WIP: Add additional node indexes https://review.opendev.org/c/openstack/ironic/+/788625 | 00:49 |
|---|---|---|
| openstackgerrit | Julia Kreger proposed openstack/ironic master: Add additional node indexes https://review.opendev.org/c/openstack/ironic/+/788625 | 00:58 |
| *** rcernin has quit IRC | 02:35 | |
| *** rcernin has joined #openstack-ironic | 02:42 | |
| openstackgerrit | xinliang proposed openstack/ironic-python-agent-builder master: Fix build error on aarch64 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/788388 | 03:07 |
| *** tzumainn has quit IRC | 03:12 | |
| *** mkrai has joined #openstack-ironic | 03:36 | |
| *** mkrai has quit IRC | 03:55 | |
| *** rcernin has quit IRC | 03:59 | |
| *** rcernin has joined #openstack-ironic | 04:00 | |
| TheJulia | arne_wiebalck: so I think I get what is going on with why the transactions are occuring | 04:12 |
| TheJulia | and it is rooted in the use of orm. Realistically it is not that bad except it adds another exchange, the conundrum is that the there is python side latency when it converts the response to objects which adds a layer where we basically double the time it takes to download the result cursor from the database server to render it. | 04:14 |
| TheJulia | the orm code also always runs joins by default which makes sense for objects, but not for things like getting the result set of nodes back out to nova. | 04:15 |
| TheJulia | unfortunately, nova wants the trait list as well, which is in a separate table. | 04:15 |
| TheJulia | :\ | 04:15 |
| erbarr | is there an issue with config drive and ussuri? I've stacked twice today and tried to run tempest and it fails very quickly with an error on n-cond-cell1 that has, "Exception: a bytes like object is required, not 'str'\n" | 04:54 |
| TheJulia | sounds like your trying to use python2 ? | 04:59 |
| TheJulia | erbarr: ^^^ | 04:59 |
| TheJulia | arne_wiebalck: and JayF might care about this as well. So the *list* out of ironic, when fields are defined, actually has a ton of extra work that is done to it which adds to the processing time. it _looks_ like we could base everything on the columns we know we want, get the required extra fields, hydrate that internal object with it (yes, we can actually do that because the way our code is written) (Yes, | 05:01 |
| TheJulia | I've already done this locally), and let the api code handle that... which I *suspect* would shred the amount of time that the actual object and reconversions take | 05:01 |
| *** openstackgerrit has quit IRC | 05:01 | |
| TheJulia | the downside, is the traits join for newer version users, and somewhere we do drop the version column but expect it to magically re-appear in the api object handling code | 05:01 |
| TheJulia | once I figure that out, it *looks* like I might be able to have the api spit out selected field nodes much faster | 05:02 |
| erbarr | TheJulia: i got USE_PYTHON3=True and PYTHON3_VERSION=3.6 in local.conf, same on train and train is fine | 05:02 |
| TheJulia | erbarr: but your saying nova is spitting this error out? | 05:03 |
| erbarr | yea, on ussuri | 05:03 |
| erbarr | that's still bionic, right? | 05:04 |
| * TheJulia shrugs | 05:04 | |
| TheJulia | support is declared by python version, not distribution | 05:04 |
| TheJulia | well, distribution version | 05:05 |
| TheJulia | erbarr: what tempest test is causing this failure? | 05:05 |
| erbarr | basic ops | 05:05 |
| TheJulia | which basic ops | 05:05 |
| erbarr | both wholedisk and partition boot | 05:06 |
| TheJulia | Baremetal Basic Ops, or Basic basic ops | 05:06 |
| TheJulia | ahh | 05:06 |
| erbarr | baremetal | 05:06 |
| TheJulia | so likely there is an incompatability somewhere in there since tempest tracks master | 05:06 |
| * TheJulia wonders how this didn't come up ages ago | 05:07 | |
| erbarr | yea, i guess i'll try to see if i can even just do a deploy and no tempest | 05:07 |
| TheJulia | erbarr: Unfortuantely I've been working since like 6 AM, so I need to call it a day. I'd dump as much info as you can into an item on storyboard | 05:07 |
| erbarr | that's a tomorrow thing tho, is late lol | 05:07 |
| TheJulia | erbarr: you could also checkout an older tempest plugin most likely | 05:08 |
| TheJulia | since they are versioned and released | 05:08 |
| erbarr | okay, thanks! have a good night! | 05:08 |
| TheJulia | iurygregory: fyi ^^^ | 05:08 |
| *** maratg has joined #openstack-ironic | 05:20 | |
| *** maratg has quit IRC | 05:22 | |
| arne_wiebalck | Good morning, ironic! | 06:47 |
| iurygregory | TheJulia, ack I will look in a few | 06:51 |
| iurygregory | good morning arne_wiebalck and Ironic o/ | 06:51 |
| arne_wiebalck | hey, good morning iurygregory o/ | 06:59 |
| iurygregory | o/ | 07:00 |
| *** rcernin has quit IRC | 07:04 | |
| *** ociuhandu has joined #openstack-ironic | 07:06 | |
| *** ociuhandu has quit IRC | 07:07 | |
| *** ociuhandu has joined #openstack-ironic | 07:07 | |
| *** ociuhandu has quit IRC | 07:21 | |
| *** ociuhandu has joined #openstack-ironic | 07:22 | |
| *** ociuhandu has quit IRC | 07:27 | |
| *** rpittau|afk is now known as rpittau | 07:27 | |
| rpittau | good morning ironic! o/ | 07:27 |
| iurygregory | morning rpittau o/ | 07:28 |
| rpittau | hey iurygregory :) | 07:29 |
| iurygregory | erbarr, can you confirm the ironic-tempest-plugin version in train and ussuri? looking at CI both are tracking master... I'm wondering why we don't see the error in our Ci gates .-. | 07:30 |
| *** ociuhandu has joined #openstack-ironic | 07:34 | |
| *** dtantsur|afk is now known as dtantsur | 07:40 | |
| * dtantsur rubs eyes | 07:40 | |
| dtantsur | good morning | 07:40 |
| iurygregory | morning dtantsur o/ | 07:41 |
| iurygregory | how did the meetup go? =) | 07:41 |
| rpittau | good morning dtantsur :) | 07:41 |
| dtantsur | iurygregory: really well, we talked for 3.5 hrs :) | 07:41 |
| dtantsur | I'll try to summarize the main finding in a blog post once I wake up | 07:42 |
| iurygregory | (I hope 3.5 was the whole time slot - no extra time) | 07:42 |
| iurygregory | take your time =) | 07:42 |
| iurygregory | great job =) | 07:42 |
| dtantsur | it was 3 hrs + extra time :) | 07:42 |
| *** tosky has joined #openstack-ironic | 07:44 | |
| *** openstackgerrit has joined #openstack-ironic | 07:44 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent stable/wallaby: Always fall back to sysrq when power off fails https://review.opendev.org/c/openstack/ironic-python-agent/+/788602 | 07:44 |
| arne_wiebalck | dtantsur: very much looking forward to such a post! | 07:56 |
| openstackgerrit | Verification of a change to openstack/ironic failed: Remove a pause before cleaning when fast-tracking https://review.opendev.org/c/openstack/ironic/+/788450 | 08:02 |
| *** lucasagomes has joined #openstack-ironic | 08:05 | |
| *** ociuhandu has quit IRC | 08:11 | |
| *** ociuhandu has joined #openstack-ironic | 08:12 | |
| *** ociuhandu has quit IRC | 08:18 | |
| *** vmud213 has joined #openstack-ironic | 08:41 | |
| *** ociuhandu has joined #openstack-ironic | 08:50 | |
| *** ociuhandu has quit IRC | 08:56 | |
| *** k_mouza has joined #openstack-ironic | 09:01 | |
| openstackgerrit | xinliang proposed openstack/ironic-python-agent-builder master: Fix build error on aarch64 https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/788388 | 09:16 |
| *** ociuhandu has joined #openstack-ironic | 09:19 | |
| dtantsur | TheJulia: hi! why are https://docs.openstack.org/ironic/latest/admin/node-multitenancy.html and https://docs.openstack.org/ironic/latest/admin/secure-rbac.html two different documents? | 09:24 |
| *** ociuhandu has quit IRC | 09:24 | |
| *** bnemec has quit IRC | 09:27 | |
| *** ociuhandu has joined #openstack-ironic | 09:27 | |
| *** bnemec has joined #openstack-ironic | 09:28 | |
| *** k_mouza has quit IRC | 09:32 | |
| *** derekh has joined #openstack-ironic | 09:32 | |
| *** ociuhandu has quit IRC | 09:35 | |
| *** ociuhandu_ has joined #openstack-ironic | 09:35 | |
| *** k_mouza has joined #openstack-ironic | 09:39 | |
| *** zzzeek has quit IRC | 09:42 | |
| *** zzzeek has joined #openstack-ironic | 09:43 | |
| *** ociuhandu_ has quit IRC | 09:44 | |
| *** k_mouza has quit IRC | 09:44 | |
| *** k_mouza has joined #openstack-ironic | 09:47 | |
| dtantsur | arne_wiebalck: I remember there were a lot of discussions re inventorization and ironic. has it led anywhere? | 10:09 |
| arne_wiebalck | dtantsur: for our deployment, yes :) | 10:09 |
| arne_wiebalck | dtantsur: we send inspection data to S3, extract it from there, and (will) add this into OpenDCIM | 10:10 |
| arne_wiebalck | dtantsur: with manual inspection on active nodes, we should (in theory) be able to also update this information | 10:11 |
| dtantsur | I see | 10:12 |
| dtantsur | the S3 part, is it done through a hook? | 10:12 |
| *** ociuhandu has joined #openstack-ironic | 10:14 | |
| *** zzzeek has quit IRC | 10:33 | |
| *** zzzeek has joined #openstack-ironic | 10:34 | |
| *** vmud213 has quit IRC | 10:44 | |
| *** akahat is now known as akahat|ruck | 10:50 | |
| *** iurygregory has quit IRC | 10:54 | |
| *** iurygregory has joined #openstack-ironic | 10:58 | |
| arne_wiebalck | dtantsur: no, this the swift backend | 11:21 |
| arne_wiebalck | dtantsur: store_data = swift (but we do not have swift deployed and use Ceph RGW which is swift compliant) | 11:22 |
| dtantsur | I see | 11:27 |
| *** k_mouza has quit IRC | 11:29 | |
| *** k_mouza has joined #openstack-ironic | 11:29 | |
| openstackgerrit | Merged openstack/ironic-python-agent stable/wallaby: Always fall back to sysrq when power off fails https://review.opendev.org/c/openstack/ironic-python-agent/+/788602 | 11:38 |
| *** ociuhandu has quit IRC | 11:41 | |
| *** ociuhandu has joined #openstack-ironic | 11:42 | |
| *** ociuhandu has quit IRC | 11:47 | |
| *** ociuhandu has joined #openstack-ironic | 11:48 | |
| *** rh-jelabarre has joined #openstack-ironic | 11:52 | |
| *** ociuhandu has quit IRC | 11:53 | |
| openstackgerrit | Wenping Song proposed openstack/ironic-specs master: setup.cfg: Replace dashes with underscores https://review.opendev.org/c/openstack/ironic-specs/+/788717 | 11:56 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic master: Do not mask configdrive when executing in-band deploy steps https://review.opendev.org/c/openstack/ironic/+/788718 | 11:56 |
| *** ociuhandu has joined #openstack-ironic | 12:14 | |
| *** ociuhandu has quit IRC | 12:20 | |
| *** ociuhandu has joined #openstack-ironic | 12:33 | |
| *** k_mouza has quit IRC | 12:34 | |
| TheJulia | dtantsur: multitenancy was regarding networking I thought | 12:36 |
| dtantsur | TheJulia: no, this is specifically about ownership | 12:37 |
| dtantsur | would be good to consolidate these two | 12:37 |
| TheJulia | yeah, in that case likely, one file will still exist and point to the other most likely | 12:37 |
| dtantsur | and good morning | 12:37 |
| TheJulia | good morning | 12:37 |
| TheJulia | https://storyboard.openstack.org/#!/story/2008869 | 12:40 |
| * TheJulia tries to wake up | 12:42 | |
| *** k_mouza has joined #openstack-ironic | 12:43 | |
| *** ociuhandu has quit IRC | 12:44 | |
| *** ociuhandu has joined #openstack-ironic | 12:45 | |
| dtantsur | I'm still trying, no luck | 12:48 |
| TheJulia | heh | 12:48 |
| TheJulia | I have a podcast to jump on and 2 meetings I need to wake up for | 12:48 |
| dtantsur | podcast, nice! | 12:48 |
| * dtantsur spend 3.5 hrs on a meetup yersterday | 12:48 | |
| TheJulia | I ended up working like a 16 hour day... sqlalchemy sucked me in | 12:49 |
| dtantsur | yay, sqlalchemy | 12:49 |
| janders | see you tomorrow Ironic o/ | 12:52 |
| rpittau | bye janders o/ | 12:52 |
| iurygregory | bye janders o/ | 12:53 |
| *** k_mouza has quit IRC | 12:53 | |
| iurygregory | good morning TheJulia =) | 12:53 |
| *** mkrai has joined #openstack-ironic | 12:55 | |
| TheJulia | dtantsur: i replied to you rindex change comment. You won't like two replies most likely, but I agree with the doc stuffs. Hopefully it will make sense. FWIW, local testing, w/o the maintenance column, it was 0.04 where as with 0.01 seconds for the cursor to return | 12:55 |
| TheJulia | which led to "why does sqlalchemy take so long to return results | 12:55 |
| TheJulia | and then much sadness | 12:55 |
| TheJulia | much much much sadness | 12:55 |
| dtantsur | TheJulia: how many nodes did you use when testing maintenance? how many of them had maintenance=True? | 12:56 |
| TheJulia | dtantsur: 115k nodes, It think it was 10% in maintenance randomly | 12:57 |
| TheJulia | I can go check a little later | 12:57 |
| dtantsur | TheJulia: have you warmed up caches first? | 12:58 |
| TheJulia | that was with warmed cache | 12:58 |
| dtantsur | "It keeps it in memory instead of spread across the disk." doesn't seem the correct explanation to me | 12:58 |
| dtantsur | the whole table probably fits in RAM, unless you limited it on purpose | 12:58 |
| *** ociuhandu_ has joined #openstack-ironic | 12:58 | |
| TheJulia | it is the tl;dr because the index maps get consulted before columns and htier contents | 12:58 |
| dtantsur | yeah, but the more indexes we have, the more space they take in memory | 12:58 |
| TheJulia | because the query optimization stuff looks for what may be the most restrictive indexes to apply first | 12:58 |
| dtantsur | is it possible to evaluate how much spaces the indexes take for your 115k nodes? | 12:59 |
| TheJulia | dtantsur: should be, but that will be later | 12:59 |
| dtantsur | "the most restrictive index" is actually a very good explanation, I haven't thought about it | 12:59 |
| dtantsur | but I'd still check the size difference this change introduces | 12:59 |
| TheJulia | mysql and postgres have different names for the logic | 12:59 |
| TheJulia | but both basically look for the most restrictive and reorder the query to go from most restrictive to leasts to pair down the rows that need manual evaluation | 13:00 |
| TheJulia | sure | 13:00 |
| TheJulia | easy to do, just need to get caffinated, ddressed, and ready to podcast | 13:00 |
| *** ociuhandu has quit IRC | 13:01 | |
| *** k_mouza has joined #openstack-ironic | 13:01 | |
| *** rloo has joined #openstack-ironic | 13:01 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Provide an option to not cache bootable iso ramdisks https://review.opendev.org/c/openstack/ironic/+/788734 | 13:03 |
| rpittau | uh jess forgot the release note.... | 13:04 |
| rpittau | blah I forgot another thing... ffff | 13:11 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Provide an option to not cache bootable iso ramdisks https://review.opendev.org/c/openstack/ironic/+/788734 | 13:23 |
| *** zzzeek has quit IRC | 13:34 | |
| *** zzzeek has joined #openstack-ironic | 13:37 | |
| *** zzzeek has quit IRC | 13:40 | |
| *** zzzeek has joined #openstack-ironic | 13:41 | |
| *** k_mouza has quit IRC | 13:42 | |
| *** zzzeek has quit IRC | 13:42 | |
| *** zzzeek has joined #openstack-ironic | 13:43 | |
| *** tzumainn has joined #openstack-ironic | 13:44 | |
| *** ociuhandu_ has quit IRC | 13:49 | |
| *** ociuhandu has joined #openstack-ironic | 13:50 | |
| openstackgerrit | Zane Bitter proposed openstack/ironic-python-agent master: Fix missing data in log messages https://review.opendev.org/c/openstack/ironic-python-agent/+/788586 | 13:56 |
| openstackgerrit | Zane Bitter proposed openstack/ironic-python-agent master: unit tests: Capture log output https://review.opendev.org/c/openstack/ironic-python-agent/+/788587 | 13:56 |
| *** ociuhandu has quit IRC | 13:59 | |
| *** ociuhandu has joined #openstack-ironic | 14:01 | |
| *** ociuhandu has quit IRC | 14:06 | |
| *** mkrai has quit IRC | 14:07 | |
| *** ociuhandu has joined #openstack-ironic | 14:09 | |
| *** ociuhandu has quit IRC | 14:12 | |
| *** ociuhandu has joined #openstack-ironic | 14:13 | |
| *** ociuhandu has quit IRC | 14:13 | |
| *** ociuhandu has joined #openstack-ironic | 14:14 | |
| *** ociuhandu has quit IRC | 14:19 | |
| erbarr | iurygregory: i know the plugin I'm checking out master and passing the location to local.conf, i don't believe I'm setting any knobs to change tempest version, what's a sure way to check? | 14:23 |
| *** ociuhandu has joined #openstack-ironic | 14:33 | |
| *** ociuhandu has quit IRC | 14:38 | |
| JayF | TheJulia: ack, thanks for the summary. Basically what I expected :( We spent a huge amount of time trying to optimize around/with sqlalchemy at $lastjob | 14:41 |
| iurygregory | erbarr, there shouldn't be a difference in our plugin itself (since you are using master), I'm wondering if is something in our code train vs ussuri... | 14:43 |
| iurygregory | you had a chance to test victoria? | 14:43 |
| TheJulia | I have this feeling we could likely cut on the selected column queries, like those used by nova... substantially | 14:43 |
| JayF | ++ | 14:43 |
| iurygregory | TheJulia, the corgi in the PTG recap was amazing :D | 14:43 |
| TheJulia | it was a huge derailing and it sounds like someone's home alarm is going off too | 14:44 |
| rpittau | mmm PTG recap ? | 14:44 |
| TheJulia | openinfra live | 14:44 |
| *** k_mouza has joined #openstack-ironic | 14:44 | |
| rpittau | oh | 14:45 |
| erbarr | iurygregory, yea from master to train only ussuri had an issue | 14:45 |
| rpittau | I missed it of course :/ | 14:45 |
| TheJulia | There was a corgi objection | 14:45 |
| TheJulia | "grrrr Woof Woof" | 14:45 |
| iurygregory | hehe yeah | 14:45 |
| iurygregory | erbarr, weird, we are probably missing some backport I think =( | 14:45 |
| rpittau | :d | 14:45 |
| erbarr | iurygregory, idk, "openstack server create" isn't working either so seems like a nova thing to me but idk that much | 14:46 |
| iurygregory | can be =) | 14:48 |
| *** paras333_ has joined #openstack-ironic | 14:58 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Provide an option to not cache bootable iso ramdisks https://review.opendev.org/c/openstack/ironic/+/788734 | 15:02 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Provide an option to not cache bootable iso ramdisks https://review.opendev.org/c/openstack/ironic/+/788734 | 15:07 |
| *** zzzeek has quit IRC | 15:09 | |
| *** zzzeek has joined #openstack-ironic | 15:10 | |
| *** ociuhandu has joined #openstack-ironic | 15:21 | |
| rpittau | dtantsur: I think centos7/python2 compatibility is gone on ipa-builder | 15:31 |
| dtantsur | :( | 15:32 |
| dtantsur | why? | 15:32 |
| rpittau | I'm not 100% sure, but looking at recent CI logs I can see the test failing as it can't find pbr>=2 | 15:33 |
| rpittau | for python 2 | 15:33 |
| rpittau | https://274c506cda779b308daa-70ebae5b6c1b3c9f5a56fab857f079c1.ssl.cf1.rackcdn.com/788388/3/check/ironic-python-agent-check-image-dib-centos7/2767920/job-output.txt | 15:33 |
| dtantsur | Oo | 15:33 |
| dtantsur | rpittau: maybe we need update pip further? | 15:33 |
| iurygregory | WOOT | 15:33 |
| rpittau | mmmm | 15:33 |
| dtantsur | there have definitely been pbr 2.x releases for python 2 | 15:33 |
| rpittau | same pip on centos8 | 15:34 |
| rpittau | although python 3 there | 15:34 |
| rpittau | this is the error distutils.errors.DistutilsError: Could not find suitable distribution for Requirement.parse('pbr>=2.0.0') | 15:34 |
| dtantsur | I seem to recall something about pip not falling back to an older version if the new version is not Python 2 compatible | 15:34 |
| dtantsur | maybe we need to try explicitly installing 'pbr>=2.0.0,<3.0' first? | 15:34 |
| rpittau | I can give it a try with pip >=20 | 15:34 |
| rpittau | or that, yeah | 15:35 |
| rpittau | that's on ipa side though, not sure if we can change pbr versions there now | 15:36 |
| dtantsur | rpittau: it's not about changing, it's a hack we can insert in IPA-builder | 15:36 |
| dtantsur | actually maybe not <3, maybe <5 | 15:37 |
| dtantsur | pbr is at 5.6.0 now | 15:37 |
| rpittau | ok, so modify upper-constraints on the fly ? | 15:38 |
| dtantsur | I don't think we're getting to upper-constraints at this point | 15:38 |
| openstackgerrit | Merged openstack/ironic stable/wallaby: Remove a pause before cleaning when fast-tracking https://review.opendev.org/c/openstack/ironic/+/788450 | 15:38 |
| dtantsur | this is probably triggered by setup.py | 15:38 |
| rpittau | we are, the error is triggered by /opt/ironic-python-agent/bin/pip install -c /tmp/requirements/upper-constraints.txt /tmp/ironic-python-agent | 15:39 |
| dtantsur | rpittau: look at the traceback, it's not pip | 15:39 |
| dtantsur | rpittau: we may solve it by installing requirements separately first btw | 15:40 |
| dtantsur | so start with -r /tmo/ironic-python-agent/requirements.txt | 15:40 |
| rpittau | oh ok, I got confused by the command | 15:40 |
| rpittau | requirements first should work, I'll do a local test first | 15:44 |
| dtantsur | I think we're ready for another ironic wallaby release | 15:44 |
| rpittau | oh the patch merged | 15:45 |
| rpittau | do we want to wait for https://review.opendev.org/c/openstack/ironic/+/788446 ? | 15:46 |
| *** gyee has joined #openstack-ironic | 15:52 | |
| *** ociuhandu_ has joined #openstack-ironic | 15:58 | |
| *** ociuhandu has quit IRC | 16:02 | |
| *** lucasagomes has quit IRC | 16:03 | |
| *** ociuhandu_ has quit IRC | 16:03 | |
| *** ociuhandu has joined #openstack-ironic | 16:23 | |
| *** uzumaki has joined #openstack-ironic | 16:24 | |
| *** ociuhandu has quit IRC | 16:27 | |
| *** zzzeek has quit IRC | 16:37 | |
| *** zzzeek has joined #openstack-ironic | 16:39 | |
| dtantsur | just docs, not necessary | 16:49 |
| *** dtantsur is now known as dtantsur|afk | 16:49 | |
| rpittau | ack | 16:49 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] fix centos7 job https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/788804 | 16:49 |
| *** iurygregory has quit IRC | 16:58 | |
| rpittau | see ya tomorrow folks! o/ | 16:59 |
| *** rpittau is now known as rpittau|afk | 16:59 | |
| *** derekh has quit IRC | 17:01 | |
| *** k_mouza has quit IRC | 17:16 | |
| *** k_mouza has joined #openstack-ironic | 17:17 | |
| TheJulia | dtantsur: so size with no data on the index seems not to even register as really measurable change. I did have about 75k unique owners and 22.5k unique lesses and that did cause the indexes on my data set to be at ~50 MB instead of 39MB, but it didn't even blink otherwise. Of course, there is white space in that from the tables churning (like my db was 53 MB and index 59MB before repacking), but I also had | 17:19 |
| TheJulia | multiple indexes and a couple test composites as well. I think part of it is the changes seem to be fitting with-in the preallocated space buffer on tables/indexes | 17:19 |
| *** k_mouza has quit IRC | 17:22 | |
| *** iurygregory has joined #openstack-ironic | 17:31 | |
| *** iurygregory has quit IRC | 17:31 | |
| *** zzzeek has quit IRC | 17:32 | |
| openstackgerrit | Merged openstack/ironic-python-agent master: Fix missing data in log messages https://review.opendev.org/c/openstack/ironic-python-agent/+/788586 | 17:36 |
| openstackgerrit | Merged openstack/ironic stable/wallaby: Docs: dhcp-less works with Glean 1.19.0 https://review.opendev.org/c/openstack/ironic/+/788446 | 17:36 |
| openstackgerrit | Merged openstack/ironic-python-agent master: unit tests: Capture log output https://review.opendev.org/c/openstack/ironic-python-agent/+/788587 | 17:36 |
| *** zzzeek has joined #openstack-ironic | 17:36 | |
| *** iurygregory has joined #openstack-ironic | 17:38 | |
| openstackgerrit | Arne Wiebalck proposed openstack/ironic-python-agent master: Burn-in: Add CPU step https://review.opendev.org/c/openstack/ironic-python-agent/+/788211 | 17:48 |
| arne_wiebalck | bye everyone o/ | 17:52 |
| iurygregory | bye arne_wiebalck o/ | 17:57 |
| *** MahnoorAsghar has joined #openstack-ironic | 18:01 | |
| arne_wiebalck | TheJulia: iurygregory: first burn-in step https://review.opendev.org/c/openstack/ironic-python-agent/+/788211, let me know in case you have major issues with this as I plan to add the others (for mem, disk, net) in a simialr way. TIA! | 18:07 |
| arne_wiebalck | rpittau|afk: and dtantsur|afk had an initial look and did not disapprove :) | 18:07 |
| * arne_wiebalck really calls it a day now :) | 18:08 | |
| iurygregory | ack =) | 18:08 |
| NobodyCam | Good Morning Ironic folks. | 18:12 |
| *** uzumaki has quit IRC | 18:25 | |
| *** zzzeek has quit IRC | 18:41 | |
| *** zzzeek has joined #openstack-ironic | 18:42 | |
| openstackgerrit | Zane Bitter proposed openstack/ironic-python-agent master: Fix getting memory size in some lshw output https://review.opendev.org/c/openstack/ironic-python-agent/+/788588 | 18:44 |
| TheJulia | good morning NobodyCam | 18:44 |
| NobodyCam | :) o/ TheJulia | 18:47 |
| iurygregory | morning NobodyCam o/ | 18:54 |
| NobodyCam | morning iurygregory | 18:55 |
| NobodyCam | how goes the day | 18:55 |
| iurygregory | waiting for my dinner while looking at terraform logs \o/ | 18:57 |
| iurygregory | and ironic logs :D | 18:57 |
| NobodyCam | oh fun! | 18:58 |
| iurygregory | yup, trying to make instance_info/capabilities show up when deploying nodes :D | 18:58 |
| *** k_mouza has joined #openstack-ironic | 19:17 | |
| *** k_mouza has quit IRC | 19:23 | |
| TheJulia | show up where? | 19:30 |
| iurygregory | TheJulia, terraform wasn't setting the instance_info/capabilities on the node, so I'm trying to fix that =) | 19:36 |
| *** frigo has joined #openstack-ironic | 19:39 | |
| TheJulia | oh joy | 19:41 |
| iurygregory | the best part is that terraform doesn't accept nested dicts :D and capabilities is a dict .. *BOOM* :D | 19:43 |
| *** ociuhandu has joined #openstack-ironic | 19:47 | |
| *** paras333_ has quit IRC | 19:52 | |
| *** ociuhandu has quit IRC | 19:54 | |
| frigo | hello Ironic! out of curiosity, you are working with the "terraform-provider-ironic" from metal3? | 19:57 |
| iurygregory | frigo, at the moment I'm =) | 19:58 |
| iurygregory | not metal3 but openshift-metal3 =) | 19:58 |
| iurygregory | https://github.com/openshift-metal3/terraform-provider-ironic | 19:58 |
| frigo | yes ok:) I tried it just a few days ago | 19:58 |
| frigo | if you manage to have a "make install" that works with recent versions of terraform that'd be just great | 19:59 |
| frigo | installing custom provider in terraform seems painful | 19:59 |
| frigo | (maybe I should do have done that myself now that I am thinking) | 20:00 |
| iurygregory | I've never installed directly (either using openshift-installer or dev-scripts) =) | 20:00 |
| iurygregory | I'm hacking the source code to make the changes hehe | 20:01 |
| iurygregory | to see the results | 20:01 |
| *** jobewan has quit IRC | 20:03 | |
| frigo | :) good luck | 20:05 |
| iurygregory | tks =D | 20:05 |
| *** cenne|out is now known as cenne | 20:32 | |
| cenne | hello | 20:34 |
| cenne | Sorry for the extra delay. Wasn't feeling well after I returned. Am fine now. I should have sent an email sooner. my bad. | 20:36 |
| cenne | I'll try not to let that happen again. | 20:36 |
| cenne | yy | 20:36 |
| cenne | hi iurygregory: nice seeing you guys again. | 20:37 |
| cenne | good day everyone! | 20:37 |
| *** frigo has quit IRC | 20:51 | |
| stevebaker | morning | 20:57 |
| *** rcernin has joined #openstack-ironic | 21:44 | |
| *** rcernin has quit IRC | 21:56 | |
| *** rcernin has joined #openstack-ironic | 22:02 | |
| *** rcernin has quit IRC | 22:37 | |
| janders | good morning stevebaker | 22:55 |
| janders | (and Ironic) | 22:55 |
| TheJulia | So.... I basically tried to rewire so we don't expand the full models on selected column queries. Basically looks like a wash performance change wise. maybe 5% faster, but that seems to vary a lot. | 23:07 |
| stevebaker | TheJulia: Is it possible to express the improvement in terms of reduced query count? | 23:15 |
| TheJulia | I think we would need a way to do a bulk query that is not restricted to 1000 nodes | 23:16 |
| TheJulia | That *should* return in a relatively sane speed | 23:16 |
| TheJulia | but the existing query to get thousands of nodes keeps rerunning the query just ends up moving the cursor | 23:17 |
| TheJulia | which is *not* nice to the DB unless there is an index precisely matching the query pattern | 23:17 |
| TheJulia | The way sqlalchemy works with the way we've got everything coded.... | 23:18 |
| TheJulia | The entire result set gets shipped to the client | 23:18 |
| *** k_mouza has joined #openstack-ironic | 23:18 | |
| TheJulia | that gets converted to objects | 23:18 |
| stevebaker | right | 23:18 |
| *** MahnoorAsghar has quit IRC | 23:18 | |
| TheJulia | and you can guess what happens there | 23:18 |
| *** rcernin has joined #openstack-ironic | 23:19 | |
| stevebaker | yep | 23:19 |
| TheJulia | Likely we need to slim that down to just columns, but it won't make it *much* faster... ~5% when I ran a bulk query (and I've not solved joining traits in so that may be skewing it that 5%) | 23:19 |
| TheJulia | Hmmm... 59k in 624 seconds so far | 23:20 |
| TheJulia | I'm guessing thermal throttling, the fan is running harder and it is a little warmer in here | 23:20 |
| stevebaker | how make fake nodes are you doing this against? | 23:20 |
| TheJulia | 115k | 23:21 |
| TheJulia | if we were to take default pagination to 10k nodes, it would cut the query load down subtantially | 23:22 |
| *** k_mouza has quit IRC | 23:22 | |
| stevebaker | o_O | 23:23 |
| TheJulia | Yeah, this is going to end up being more like 10% slower on this second run | 23:23 |
| TheJulia | :( | 23:23 |
| * TheJulia wins at making ironic slower | 23:23 | |
| stevebaker | TheJulia: how about a dedicated API call which has an implementation that doesn't use sqlalchemy or rpc object abstractions? | 23:27 |
| TheJulia | That would be *much* faster. We wouldn't be able to ensure versioning on the same level we do today | 23:28 |
| TheJulia | But we don't have object upgrade code all over the place, so it should be fine and navigable | 23:28 |
| TheJulia | And sqlalchemy really is fine, it is the orm models that bite us | 23:29 |
| TheJulia | In theory, a dedicated endpoint could also immediately begin streaming back results | 23:32 |
| TheJulia | But that might be a little hard to do with the assembly of the api as is | 23:32 |
| TheJulia | I should likely dump my hacked code to a change and upload it | 23:33 |
| TheJulia | So it is 2C warmer in the office today than it was yesterday.... | 23:37 |
| TheJulia | So yay thermal throttling. | 23:37 |
| *** tosky has quit IRC | 23:40 | |
| stevebaker | yeah may as well put it up as a review, for a bit more context | 23:42 |
| *** k_mouza has joined #openstack-ironic | 23:57 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!