| opendevreview | Julia Kreger proposed openstack/ironic master: WIP: idea of cleaning up username/password for drivers https://review.opendev.org/c/openstack/ironic/+/801406 | 00:34 |
|---|---|---|
| TheJulia | wheee | 00:41 |
| TheJulia | iurygregory: I stubbed out some days on ethercalc, since the foundation is asking for this stuff to be answered by the 21st of this month. | 01:03 |
| TheJulia | only... 3 months... (wtf) | 01:03 |
| TheJulia | https://ethercalc.openstack.org/8tum5yl1bx43 | 01:04 |
| opendevreview | Merged openstack/metalsmith master: Replace deprecated import of ABCs from collections https://review.opendev.org/c/openstack/metalsmith/+/801089 | 05:05 |
| opendevreview | Jonas Schäfer proposed openstack/ironic-python-agent master: Expose BMC MAC address in inventory data https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 | 05:52 |
| jssfr | iurygregory, done! | 05:52 |
| opendevreview | Jonas Schäfer proposed openstack/ironic-python-agent master: Expose BMC MAC address in inventory data https://review.opendev.org/c/openstack/ironic-python-agent/+/801260 | 05:55 |
| arne_wiebalck | Good morning jssfr and Ironic! | 06:15 |
| jssfr | Good Morning! :) | 06:17 |
| iurygregory | TheJulia, ack I will take a look | 06:21 |
| iurygregory | good morning arne_wiebalck jssfr and Ironic o/ | 06:21 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 06:23 |
| arne_wiebalck | hey iurygregory o/ | 06:23 |
| *** rpittau|afk is now known as rpittau | 07:06 | |
| rpittau | good morning ironic! o/ | 07:07 |
| iurygregory | morning rpittau | 07:08 |
| iurygregory | o/ | 07:08 |
| rpittau | hey iurygregory :) | 07:08 |
| iurygregory | rpittau, if you have a chance can you test locally https://review.opendev.org/c/openstack/ironic/+/801064 I see some weird errors in tox -e cover and tox -e py38 .-. | 07:10 |
| iurygregory | the errors in py38 are a bit different locally .-. and in a function I didn't touch hehe | 07:10 |
| rpittau | sure | 07:12 |
| rpittau | as soon as I wake up :P | 07:12 |
| iurygregory | yeah :D | 07:12 |
| rpittau | mmmmm | 07:14 |
| iurygregory | try running more than once the tests hehe | 07:15 |
| iurygregory | stendulker, hey o/ should I check if is an URL or just URI? does destination in newer versions supports something that its not URLs? | 07:25 |
| iurygregory | also, good morning/afternoon =) | 07:25 |
| stendulker | just URI should be good. URL could to tough I suppose | 07:26 |
| stendulker | iurygregory good morning :) | 07:26 |
| iurygregory | ack =) | 07:26 |
| stendulker | good morning arne_wiebalck, rpittau | 07:26 |
| rpittau | hey stendulker :) | 07:26 |
| iurygregory | if is an "invalid URI" the BMC will fail so it should be fine | 07:26 |
| stendulker | yes | 07:27 |
| arne_wiebalck | Good morning stendulker o/ | 07:27 |
| opendevreview | Arne Wiebalck proposed openstack/ironic master: Retry getting the power state of a node in fast-track mode https://review.opendev.org/c/openstack/ironic/+/801425 | 08:12 |
| arne_wiebalck | TheJulia: dtantsur: this ^^ fixes the weird if branch issue (which is not that mysterious after all) on our nodes. | 08:14 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 08:18 |
| opendevreview | Merged openstack/ironic stable/wallaby: Redfish: Skip non-RAID controllers for RAID https://review.opendev.org/c/openstack/ironic/+/800932 | 08:25 |
| opendevreview | Merged openstack/ironic master: Fix handling driver_info[agent_verify_ca] == False https://review.opendev.org/c/openstack/ironic/+/797669 | 08:25 |
| iurygregory | rpittau, you had any luck testing locally? | 08:26 |
| rpittau | iurygregory: I can reproduce the issue locally, looks like something messed up with mock | 08:26 |
| iurygregory | locally to me is a bit random like 50% of the time it fails/works | 08:27 |
| iurygregory | =( | 08:27 |
| iurygregory | for the py38, cover doesn't fail at all (not sure why CI is blowing up) | 08:27 |
| rpittau | I can reproduce locally with both py36 and py38 | 08:28 |
| rpittau | the failing tests are random | 08:28 |
| rpittau | but the fail reason is the same | 08:28 |
| dtantsur | morning ironic | 08:29 |
| rpittau | good morning dtantsur :) | 08:29 |
| dtantsur | arne_wiebalck: mmmm, I need to think about it a bit | 08:29 |
| dtantsur | I wonder if we should use the cached node.power_state rather than going to the BMC directly | 08:29 |
| dtantsur | especially since agent_is_alive sort of duplicates the power check | 08:29 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Test lower-constraints https://review.opendev.org/c/openstack/ironic/+/801350 | 08:30 |
| iurygregory | morning dtantsur | 08:35 |
| iurygregory | rpittau, weird 36 doesn't fail for me =( | 08:35 |
| arne_wiebalck | dtantsur: hmm, yeah | 08:40 |
| arne_wiebalck | dtantsur: also unclear why an immediate retry works | 08:42 |
| dtantsur | a network glitch? | 08:42 |
| arne_wiebalck | dtantsur: too reproducible | 08:42 |
| arne_wiebalck | dtantsur: this is not a one-off effect, it happens every time | 08:43 |
| dtantsur | if you're burning-in network at that point, chances are high your saturating it | 08:43 |
| arne_wiebalck | dtantsur: the management network is separate | 08:43 |
| dtantsur | hmm | 08:43 |
| arne_wiebalck | dtantsur: and also this is CPU burn-in | 08:43 |
| arne_wiebalck | dtantsur: TheJulia was also suggesting side-channeling and the BMC being somehow impacted by the CPU, but all should be completely separate hardware-wise | 08:45 |
| arne_wiebalck | dtantsur: I wonder why we go to the BMC in the first place ... node cache potentially outdated ? | 08:46 |
| dtantsur | likely. e.g. a node has just been powered off, the agent liveness check has not been updated. | 08:47 |
| dtantsur | (although if it's powered off through ironic, it should be reflected on the node) | 08:48 |
| arne_wiebalck | dtantsur: while I agree we should not scatter retries everywhere (rather than understanding what is happening), this one may be hard to understand, it is retried once only, and only under certain circumstances (and it seems to fix the issue on real hardware) | 08:52 |
| iurygregory | dtantsur, fyi I've left a comment in https://review.opendev.org/c/openstack/bifrost/+/799837 | 08:52 |
| dtantsur | iurygregory: a good question, but we don't include wsman implementations anywhere else | 08:52 |
| * arne_wiebalck is, despite arguing, still open for suggestions :) | 08:52 | |
| dtantsur | iurygregory: so if we want to do it, it should be a separate patch (and given that Dell moves away from WSMAN, I'm somewhat unsure about it) | 08:53 |
| * arne_wiebalck will go on holidays for 2.5 weeks, so there is time to reflect | 08:53 | |
| iurygregory | dtantsur, hummm gotcha because I do see 3 interfaces idrac idrac-wsman and idrac-redfish | 08:54 |
| dtantsur | iurygregory: yep. idrac is an alias of idrac-wsman (I think) | 08:54 |
| iurygregory | Vendor passthru interface 'idrac' is deprecated and may be removed in a future release. Use 'idrac-wsman' instead. | 08:55 |
| iurygregory | yeah | 08:55 |
| iurygregory | I will +2 +W the bifrost patch =) | 08:57 |
| iurygregory | tks! | 08:57 |
| dtantsur | thank you! | 08:57 |
| opendevreview | Dmitry Tantsur proposed openstack/metalsmith stable/wallaby: Replace deprecated import of ABCs from collections https://review.opendev.org/c/openstack/metalsmith/+/801325 | 08:57 |
| * iurygregory goes back to fight random unit tests failures =( | 08:58 | |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] Test latest ansible https://review.opendev.org/c/openstack/bifrost/+/800440 | 09:05 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: [DNM] Test latest ansible https://review.opendev.org/c/openstack/bifrost/+/800440 | 09:06 |
| iurygregory | rpittau, one thing I've noticed if I comment the unit tests test_get_event_service test_get_event_service_error I don't see any failures .-. but I don't see anything strange in the tests I've created | 09:06 |
| rpittau | iurygregory: yeah, I don't think the problem is *in* the tests you added, they probably just exposed something that was wrong before | 09:07 |
| iurygregory | *magic* | 09:07 |
| * iurygregory has no idea on how to fix =( | 09:08 | |
| rpittau | iurygregory: I'm having another look, can't promise anything but so far it looks like the issue seems to be in the redfish_utils._get_connection mock | 09:19 |
| iurygregory | hummm | 09:22 |
| iurygregory | all the tests I saw failures are the ones using @mock.patch.object(sushy, 'Sushy', autospec=True) | 09:23 |
| iurygregory | @mock.patch('ironic.drivers.modules.redfish.utils.' | 09:23 |
| iurygregory | 'SessionCache._sessions', {}) | 09:23 |
| opendevreview | Merged openstack/ironic-python-agent master: Force immediate NTP time sync with chronyd at IPA startup https://review.opendev.org/c/openstack/ironic-python-agent/+/801032 | 09:39 |
| opendevreview | Merged openstack/bifrost master: Allow configuring enabled vendor interfaces https://review.opendev.org/c/openstack/bifrost/+/799837 | 09:59 |
| janders__ | I am hitting an issue while testing the verify steps / iDRAC reset patch ( https://review.opendev.org/c/openstack/ironic/+/800001 ). Logs: | 10:30 |
| janders__ | https://paste.opendev.org/show/807587/ | 10:30 |
| janders__ | specifically: No extensions found for "manager" under namespace "sushy.resources.manager.oems" | 10:31 |
| iurygregory | probably missing sushy-oem-idrac in your system | 10:31 |
| iurygregory | =) | 10:31 |
| janders__ | iurygregory: it was missing initially but I added it and it's still giving me a hard time | 10:31 |
| iurygregory | but this should only be necessary if you need redfish... | 10:32 |
| iurygregory | hummm | 10:32 |
| janders__ | yes, I am running idrac-redfish management driver | 10:32 |
| janders__ | let me try to stop everything and make sure Ironic stopped and restarted correctly before I start asking more questions | 10:32 |
| iurygregory | yeah I'm wondering if you can try to load the manager in the venv | 10:33 |
| iurygregory | trying some python code to see | 10:33 |
| iurygregory | janders__, https://opendev.org/x/sushy-oem-idrac/src/branch/master/README.md#example-use | 10:34 |
| iurygregory | maybe checking if oe_manager returns a correct object in your venv | 10:35 |
| dtantsur | if ironic is in a venv, make sure you're installing sushy-oem-idrac in the same venv | 10:36 |
| dtantsur | and make sure to restart ironic-conductor | 10:37 |
| janders__ | I will kick it once more to be sure-sure it picked it up | 10:37 |
| janders__ | https://paste.opendev.org/show/807588/ - should be in the same venv as bifrost stuff | 10:38 |
| janders__ | (I installed it with "pip install --user sushy-oem-idrac" while having bifrost venv active) | 10:39 |
| janders__ | that's the correct way, right? | 10:39 |
| janders__ | okay I'm 100% sure all Ironic services have now been restarted after installing sushy-oem-idrac, still same issue | 10:45 |
| dtantsur | janders__: not exactly. --user forces installation into ~/.share/local | 10:47 |
| dtantsur | just drop --user (and don't use sudo) | 10:47 |
| janders__ | ok! that may be the problem.. thank you dtantsur! | 10:47 |
| janders__ | trying this now | 10:47 |
| janders__ | do we typically use any particular flavour of idrac ( idrac-redfish or idrac-wsman ) for vmedia based installs ( downstream )? | 10:52 |
| janders__ | dtantsur: this worked, got me a step closer ("on to the next problem") | 10:59 |
| janders__ | thanks! | 10:59 |
| dtantsur | janders__: we only use the redfish flavor downstream. wsman doesn't have virtual media support. | 10:59 |
| janders__ | dtantsur: that's good, at least I am testing the right one :) | 11:00 |
| janders__ | https://paste.opendev.org/show/807589/ | 11:02 |
| janders__ | the next problem is : Error: Manager 3256444f-c0c7-3880-5210-00544c4c4544: The attribute Links/Oem/Dell/DelliDRACCardService is missing from the resource /redfish/v1/Managers/iDRAC.Embedded.1 execute_oem_manager_method /opt/stack/bifrost/lib64/python3.6/site-packages/ironic/drivers/modules/drac/utils.py:109#033[00m | 11:02 |
| janders__ | Jul 20 10:58:21 u640-u04 ironic-conductor[3705175]: 2021-07-20 10:58:21.601 3705175 ERROR ironic.drivers.modules.drac.utils [req-2225886c-de23-43c2-a097-8e21fdd3d5cc bifrost_user - - - -] In system 4c4c4544-0054-5210-8038-c7c04f445632 for node f8fd9093-3fe3-4444-96bc-647626112188 all managers failed: reset iDRAC. Errors: ['Manager 3256444f-c0c7-3880-5210-00544c4c4544: The attribute Links/Oem/Dell/DelliDRACCardServic | 11:02 |
| janders__ | e is missing from the resource /redfish/v1/Managers/iDRAC.Embedded.1'] | 11:02 |
| janders__ | oops! sorry, this didn't look that long in paste :P | 11:02 |
| janders__ | I have a sneaking suspicion I should check if there are iDRAC updates for that machine | 11:03 |
| dtantsur | yep | 11:03 |
| dtantsur | that's not something we can fix, I suspect | 11:03 |
| janders__ | and testing with wsman is probably not worth the time, right?? | 11:04 |
| janders__ | (even if it works, it won't implement the functionality we need) | 11:04 |
| dtantsur | well.. wsman has the reset call, but not virtual media | 11:06 |
| janders__ | if my thinking is correct, we would need to change drivers in-flight then | 11:06 |
| janders__ | (strictly speaking, just the management interface I think) | 11:06 |
| janders__ | BIOS on the machine is 1.6.13 and the latest is 2.11.2 | 11:07 |
| janders__ | I think I need to sort this out before going any further, with a bit of luck that may be it | 11:07 |
| janders__ | it's probably still the one it shipped with looking at the discrepancy | 11:08 |
| dtantsur | yeah, start with an upgrade of both BIOS and iDRAC | 11:13 |
| * dtantsur brb, groceries | 11:14 | |
| iurygregory | janders__, if you need a machine with latest FW etc for vmedia let me know | 11:22 |
| iurygregory | if you don't need to deploy ofc =) | 11:22 |
| janders__ | thank you iurygregory | 11:22 |
| janders__ | I will try to fix -u05 ( I might accidentally do bad things to it - or at least drop power few times) | 11:23 |
| janders__ | should that fail I will let you know :) | 11:23 |
| iurygregory | ack o/ | 11:23 |
| iurygregory | https://review.opendev.org/c/openstack/ironic/+/801064 still this weird issues with py38 and cover (╯°□°)╯︵ ┻━┻ | 11:28 |
| opendevreview | Derek Higgins proposed openstack/ironic stable/wallaby: Allow reboot to hard disk following iso ramdisk deploy. https://review.opendev.org/c/openstack/ironic/+/801343 | 11:56 |
| janders__ | dtantsur: with new BIOS/iDRAC it gets heaps further | 12:05 |
| janders__ | thanks for your hints | 12:06 |
| janders__ | trouble I see is it gets right to manageable before completing known_good_state - it should be in verifying state | 12:06 |
| janders__ | I will need to track down why tomorrow | 12:07 |
| janders__ | looks like a problem with verify steps code this time | 12:07 |
| iurygregory | congrats janders__ \o/ | 12:13 |
| janders_ | thank you iurygregory | 12:16 |
| janders_ | it was one of the messier FW upgrades actually but all good now | 12:17 |
| iurygregory | awesome | 12:21 |
| iurygregory | rpittau, I'm wondering if is something in the third_party_driver_mocks... https://7427588983fdfe8dc6c2-2bc8542270d98932adf49f2123132d6c.ssl.cf1.rackcdn.com/801064/10/check/openstack-tox-py38/e58904f/testr_results.html | 12:23 |
| rpittau | iurygregory: isn't that using sushy ? | 12:25 |
| iurygregory | it does have informations for sushy and other drivers | 12:25 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Install sushy-oem-idrac for idrac-redfish-virtual-media https://review.opendev.org/c/openstack/bifrost/+/801457 | 12:38 |
| dtantsur | janders_, iurygregory, related ^^ | 12:38 |
| iurygregory | dtantsur, nice! | 12:38 |
| dtantsur | testing welcome, I don't think it's enabled by default in the CI | 12:40 |
| dtantsur | okay, I've tested it locally, it works | 12:55 |
| janders__ | dtantsur - this is great | 12:56 |
| iurygregory | I would test but my mind is completely focused in trying to fix the unit tests in py38 =( | 12:56 |
| dtantsur | sure, no worries | 12:57 |
| janders__ | see you tomorrow Ironic o/ | 12:58 |
| janders__ | thanks for all your help today, much appreciated | 12:59 |
| rpittau | bye janders__ :) | 12:59 |
| opendevreview | Dmitry Tantsur proposed openstack/bifrost master: Support ilo5 hardware type https://review.opendev.org/c/openstack/bifrost/+/801465 | 13:03 |
| TheJulia | good morning | 13:06 |
| iurygregory | good morning TheJulia =) | 13:07 |
| dtantsur | morning TheJulia | 13:07 |
| iurygregory | TheJulia, I'm wondering if you already answered the survey https://openinfrafoundation.formstack.com/forms/oct2021_vptg_survey =) | 13:08 |
| iurygregory | also in the ethercalc I see 7 sessions (last PTG we had 9) so we could sync with APAC timezone | 13:09 |
| TheJulia | iurygregory: I did | 13:13 |
| TheJulia | iurygregory: I won't be able to cover apac time zones, that week I'll be UTC-4 | 13:13 |
| iurygregory | I see! | 13:13 |
| iurygregory | since I'm not in Brazil I also can't cover the APAC timezone | 13:14 |
| iurygregory | maybe next year I can XD | 13:14 |
| dtantsur | we have Jacob and Steve :) and Jay from the west coast. | 13:19 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: Prepare 19.0 and clean up release notes https://review.opendev.org/c/openstack/ironic/+/801469 | 13:32 |
| opendevreview | Merged openstack/ironic master: Set glance limit for baremetal friendly images https://review.opendev.org/c/openstack/ironic/+/801365 | 13:48 |
| opendevreview | Merged openstack/ironic-specs master: Add attestation interface spec https://review.opendev.org/c/openstack/ironic-specs/+/576718 | 14:07 |
| iurygregory | I've updated the doodle for the midcycle with more options tks arne_wiebalck =) | 14:18 |
| TheJulia | you mean more than the 60? | 14:19 |
| iurygregory | more 1hrs slots for EU/US timezones | 14:19 |
| iurygregory | so we can choose =) | 14:20 |
| iurygregory | if anyone is expert in mock and has some ideas on how I can fix the py38 failures I would appreciate :D https://review.opendev.org/c/openstack/ironic/+/801064 | 14:20 |
| arne_wiebalck | TheJulia: it was more about adding filling some 1-hour holes and offering 2pm UTC as well (which I think is still ok for you, no?) | 14:22 |
| arne_wiebalck | TheJulia: to be a little more aligned with working hours :) | 14:23 |
| TheJulia | Yeah, 2pm UTC is doable for me | 14:23 |
| TheJulia | Updated my reply | 14:27 |
| arne_wiebalck | TheJulia: dtantsur: I was looking a little more into the fast-track ipmi power issue today. The return from ipmitool on the "chassis power status" command is an empty string, which Ironic translates into "error". I checked in the ipmitool code and this can happen when ipmitool is unable to get the power status. And indeed stderr reports difficulties to establish a session ... which immediately after works (as I | 14:42 |
| arne_wiebalck | mentioned) ... erm, yeah. | 14:42 |
| dtantsur | arne_wiebalck: we are supposed to retry establishing a session. I assume we don't? | 14:42 |
| JayF | Sounds like there might be a fix in our ipmitool driver to ignore empty string there | 14:42 |
| arne_wiebalck | I will go on holidays, but one of my colleagues will keep having a look :-D | 14:42 |
| JayF | hmmm... version of ipmitool issue? | 14:43 |
| JayF | but that shouldn't be an issue, if you have that same conductor working on other nodes | 14:43 |
| arne_wiebalck | JayF: I do. | 14:43 |
| arne_wiebalck | dtantsur: hmm, it really looks like right after the initial call, it works on the 2nd, like a wake-up call. | 14:44 |
| dtantsur | arne_wiebalck: it sounds like you need a more robust retry logic in the ipmitool driver tbh | 14:44 |
| dtantsur | there are other places where we use driver.power.get_power_state | 14:44 |
| arne_wiebalck | I *think* I have seen log entries on power sync where it sometimes also fails initially, but I never paid too much attention. There, we retry. | 14:45 |
| TheJulia | I've seen similar things in some logs customers have posted to us. Where the bmc *really* doesn't like returning power status reliably | 14:46 |
| TheJulia | which causes some churn in our db and lots of logging | 14:46 |
| dtantsur | moar retries \o/ | 14:47 |
| TheJulia | Problem is, we log 2-3 lines every single time regardless | 14:47 |
| arne_wiebalck | I see "During sync_power_state, could not get power state for node d5e506a8-d99c-4973-bb1a-a82bdce53797, attempt 1 of 10. Error: Power driver returned ERROR state while trying to sync power state.." | 14:47 |
| TheJulia | That is the sync power state loop itself if memory serves | 14:48 |
| dtantsur | yep. so you hit the power sync retries. arguably, you shouldn't. | 14:48 |
| arne_wiebalck | I do not see this with higher number of attempts, at least not often. | 14:48 |
| arne_wiebalck | (quickly looking through my dashboard) | 14:48 |
| TheJulia | so 3-ish attempts might keep get_power_state happy, we would just need to trace back the sources to make sure it is more than just the sync powers state loop | 14:49 |
| dtantsur | we do have retries in ipmitool, the question is why they don't catch arne_wiebalck's case | 14:49 |
| TheJulia | ++, there is some logical reason there if memory serves | 14:50 |
| arne_wiebalck | I guess someone else needed the power sync retries, otherwise we would not have them :) | 14:51 |
| TheJulia | well | 14:52 |
| arne_wiebalck | (or maybe we had them before we had ipmitool retries) | 14:52 |
| TheJulia | there is a whole logic pattern with ipmitool that brings forth a migraine | 14:52 |
| TheJulia | and it is a huge context switch | 14:52 |
| arne_wiebalck | don't do it, I am about to leave, we can pick this up once I am back in 2.5 weeks | 14:53 |
| TheJulia | but basically everything should be able to get retried for ipmitool | 14:53 |
| TheJulia | arne_wiebalck: ack, dd if=/dev/zero of=/path/to/ipmitool/driver/reasons | 14:53 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 14:53 |
| iurygregory | ok this should probably fix the random failures .-. | 14:54 |
| iurygregory | if fails again I will blame my computer... 20 runs 0 failures locally (╯°□°)╯︵ ┻━┻ | 14:54 |
| TheJulia | iurygregory: oh no... not one of those... again | 14:55 |
| iurygregory | well 2 days trying to fight Mock is not funny =D | 14:55 |
| TheJulia | no, it is not | 14:56 |
| TheJulia | I think our sqlalchemy gremlin has mostly disappeared | 14:57 |
| * TheJulia votes we now rename it "sqlalchemy corgi" | 14:57 | |
| iurygregory | ++ | 14:57 |
| TheJulia | "sqlalchemy corgi give you disbelieving side-eye for your database change" | 14:57 |
| iurygregory | amazing! | 14:57 |
| TheJulia | I don't remember where the last one, but I remember spending like a week on trying to get rid of it | 14:58 |
| TheJulia | iurygregory: cross-linked the pads and added a few ideas | 14:58 |
| iurygregory | TheJulia, ack | 14:58 |
| iurygregory | funny the tests failed again -.-' | 15:07 |
| dtantsur | RIP Iury's table | 15:07 |
| iurygregory | I can't do anything now because I'm attending a meeting | 15:08 |
| iurygregory | maybe after... | 15:09 |
| iurygregory | on the bright side.. tox-cover didn't fail | 15:11 |
| TheJulia | added a few more items to the ptg etherpad | 15:13 |
| iurygregory | time for more coffee to fight against Mock... | 15:17 |
| arne_wiebalck | TheJulia: lol | 15:17 |
| iurygregory | CI you must be kidding with me....Build succeeded (ARM64 pipeline) py38,py39... | but check pipeline fails with py36,py38,py39 | 15:20 |
| iurygregory | ¯\_(ツ)_/¯ | 15:20 |
| TheJulia | heh | 15:22 |
| iurygregory | mock is Nondeterministic <3 | 15:24 |
| arne_wiebalck | bye everyone, see you in 2.5 weeks o/ | 15:30 |
| rpittau | arne_wiebalck: have a great vacation! | 15:30 |
| dtantsur | arne_wiebalck: enjoy! | 15:30 |
| iurygregory | bye arne_wiebalck, enjoy the time off =D | 15:32 |
| * iurygregory won't be able to summon arne_wiebalck in the weekly meeting | 15:33 | |
| TheJulia | iurygregory: are all these tests in the same class? | 15:33 |
| iurygregory | TheJulia, yeah, the thing is after I added test_get_event_service and test_get_event_service_error this random failures started to pop-up | 15:35 |
| TheJulia | iurygregory: put them in another class :) | 15:35 |
| iurygregory | will do that | 15:35 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] Test lower-constraints https://review.opendev.org/c/openstack/ironic/+/801350 | 15:36 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 15:43 |
| * iurygregory watches zuul running the tests | 15:46 | |
| rpittau | and zuul watches you back | 15:47 |
| iurygregory | it will after the job finishes... | 15:48 |
| opendevreview | Will Szumski proposed openstack/bifrost master: Allow HTTP headers to be customised for IPA image download https://review.opendev.org/c/openstack/bifrost/+/798825 | 15:56 |
| iurygregory | TheJulia, I moved my tests to a new class but the random tests are still failing https://6f8ef474bc864f2ec7a2-1f2089e8a25dfa4acbcd47153d041690.ssl.cf5.rackcdn.com/801064/12/check/openstack-tox-py36/f39631a/testr_results.html | 15:57 |
| iurygregory | =( | 15:57 |
| iurygregory | zuul successfully watches iurygregory back | 15:57 |
| TheJulia | so the driver internally caches right? | 15:58 |
| iurygregory | yeah | 16:00 |
| iurygregory | I think there is some wrong in ironic.tests.unit.drivers.third_party_driver_mocks | 16:01 |
| iurygregory | but not idea what it can be | 16:01 |
| TheJulia | well | 16:05 |
| TheJulia | so... the caching is, for all intents and purposes, crossing the test boundary which is why I'm saying separate parent classes to isolate it down because the runner invokes the test classes separately. Within one class, one test can stomp on the other so even while code and operating mechanics is fine, it is basically due to the order of execution in the runner's execution of the class's tests. | 16:06 |
| TheJulia | CI having different performance profile likely results in different run pattern | 16:07 |
| iurygregory | I will move all tests with cache to another class to see | 16:07 |
| iurygregory | would it make sense? | 16:07 |
| iurygregory | ok I've split tests using SessionCache mock and not using | 16:20 |
| TheJulia | iurygregory: okay, you may just be shifting it around, *but* worth to see | 16:30 |
| TheJulia | err, worth it | 16:30 |
| iurygregory | you mean the order of the tests in the class? | 16:30 |
| TheJulia | no | 16:33 |
| TheJulia | this might just be easier to explain | 16:33 |
| TheJulia | iurygregory: https://meetpad.opendev.org/ironic | 16:35 |
| iurygregory | going | 16:35 |
| rpittau | good night! o/ | 16:45 |
| *** rpittau is now known as rpittau|afk | 16:45 | |
| cenne | gn rpittau|afk | 16:46 |
| cenne | hey all. Can api.request.version be `None`? | 16:47 |
| cenne | I am assuming it's an integer, should I be defensive and work around possible null values? | 16:47 |
| opendevreview | cenne proposed openstack/ironic master: [WIP] Add api endpoint for setting boot_mode on managed node https://review.opendev.org/c/openstack/ironic/+/800084 | 16:49 |
| JayF | Hmm. I don't know, but we can check to see if existing code guards for it | 16:50 |
| cenne | Hmm. The existing code seems not to do that. (Atleast the ~5 instances I checked) | 16:53 |
| JayF | cenne: https://github.com/openstack/ironic/blob/master/ironic/api/controllers/v1/utils.py#L500 doesn't look like it handles None | 16:54 |
| JayF | so I think you can follow that same pattern; if we discover it's wrong we'll have to fix it everywhere anyway :) | 16:54 |
| cenne | But i do get errors in test, when I am not passing the api version num. | 16:55 |
| cenne | Okay, looking into that now. Thanks! | 16:55 |
| cenne | Oh yeah. That example doesn't handle None. Saw some of those. I seem to be getting errors though. I'll look again into what's actually happening. | 16:58 |
| JayF | I'd suggest testing one of those endpoints | 16:59 |
| JayF | and seeing how it reacts | 16:59 |
| JayF | when given that | 16:59 |
| TheJulia | so... if I remember correctly if it is none, assumes 1.0 | 17:02 |
| JayF | I guess no review jam? | 17:03 |
| TheJulia | no :\ | 17:04 |
| JayF | Should we even pretend it exists anymore/ | 17:04 |
| TheJulia | I dunno | 17:04 |
| TheJulia | it is killing me because I've now got conflicting meetings | 17:05 |
| JayF | You gotta manage your calendar/life, but if we have an upstream meeting scheduled during NA hours that nobody, including the PTL, is gonna attend we shoudl just kill it | 17:06 |
| TheJulia | Yeah. I think it worked well towards end of cycle where we had a lot of stuff to churn through | 17:06 |
| TheJulia | but yeah | 17:06 |
| cenne | TheJulia: Yup. | 17:10 |
| cenne | Hmm, so giving no version defaults to the lowest one. 🤷. okay.. | 17:10 |
| cenne | wasn't None. So changing my test: (be explicit with version num) | 17:10 |
| JayF | cenne: our API didn't always have microversions; so by definition if you don't pass a version header you're using the original 1.0 and not 1.whatever we support now | 17:11 |
| cenne | it's defaulting to 1.1 with me, but same diff. It's taking the min one from range. | 17:11 |
| JayF | if that makes sense? microversion unaware clients get the oldest api possible | 17:11 |
| cenne | Ahh. right. okay. Will stick to explicitly mentioning them. | 17:11 |
| cenne | > if that makes sense? microversion unaware clients get the oldest api possible | 17:12 |
| cenne | thats so sad :( | 17:12 |
| cenne | poor clients. | 17:15 |
| JayF | lol | 17:16 |
| opendevreview | cenne proposed openstack/ironic master: [WIP] Add api endpoint for setting boot_mode on managed node https://review.opendev.org/c/openstack/ironic/+/800084 | 17:36 |
| dtantsur | cenne: haha, you have a point | 17:42 |
| dtantsur | have a good evening folks | 17:42 |
| JayF | o/ | 17:44 |
| TheJulia | cenne: I have repeatedly expressed this... and it is quite sad :( | 17:44 |
| * TheJulia is finally away from whatever meeting that was | 17:44 | |
| cenne | o/ dtantsur | 17:50 |
| iurygregory | TheJulia, seems like it worked (locally at least...) I've split the tests in 3 classes so far it worked... (let's see how CI will react) https://paste.opendev.org/show/807600/ | 19:07 |
| TheJulia | iurygregory: well, i hope :) | 19:07 |
| opendevreview | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add vendor_passthru method for subscriptions https://review.opendev.org/c/openstack/ironic/+/801064 | 19:08 |
| iurygregory | fingers crossed | 19:08 |
| * iurygregory watches zuul | 19:09 | |
| iurygregory | openstack-tox-py36 is running... | 19:10 |
| iurygregory | openstack-tox-py38 running... | 19:15 |
| TheJulia | py36 passed | 19:17 |
| iurygregory | yeah | 19:17 |
| iurygregory | now looking at poy38.. | 19:17 |
| TheJulia | it is still in db setup | 19:18 |
| iurygregory | yup I'm watching the console XD | 19:18 |
| iurygregory | TheJulia, I owe you a beer =) | 19:23 |
| TheJulia | \o/ | 19:23 |
| iurygregory | 2 days trying to figure out this omg D: | 19:23 |
| TheJulia | Look at it this way, next time you see it, you'll know what is happening and what to do | 19:28 |
| iurygregory | exactly! :D | 19:30 |
| iurygregory | we are always learning =) | 19:30 |
| iurygregory | now I don't need to stay awake till midnight trying to solve unit tests problems \o/ | 19:32 |
| iurygregory | time to watch some anime \o/ | 19:33 |
| cenne | :) | 20:01 |
| opendevreview | cenne proposed openstack/ironic master: [WIP] Add api endpoint for setting boot_mode on managed node https://review.opendev.org/c/openstack/ironic/+/800084 | 20:36 |
| eandersson | Does anyone happen to have an CLI example of setting BIOS setting during the clean step? | 20:48 |
| eandersson | As an example how would this translate to a CLI command https://docs.openstack.org/ironic/latest/admin/drivers/idrac.html#example ? | 20:50 |
| eandersson | Is it just something like this? | 20:53 |
| eandersson | > openstack baremetal node clean --clean-steps '[{"interface": "bios", "step": "apply_configuration", "args": {"settings": [{"name": "ProcVirtualization", "value": "Enabled"}]}}]' <node> | 20:53 |
| JayF | that would run manual cleaning with those arguments | 20:55 |
| JayF | getting it to run with those arguments via automated cleaning; IDK how to do that if it's not documented, sorry :( | 20:56 |
| eandersson | Yea - I was just wondering. I see --target-raid-config, but I don't see a way to do it for BIOS. | 20:57 |
| JayF | yeah; honestly I don't even know if it's possilbe | 20:57 |
| JayF | but the command you did it specifically for manual cleaning | 20:57 |
| eandersson | Yea - I kinda feel like there should be an equilvement to --target-raid-config, --target-bios-config | 20:58 |
| eandersson | for BIOS settings | 20:58 |
| iurygregory | yay zuul +1 \o/ | 20:58 |
| JayF | it's also not impossible it exists in api but not yet in cli | 20:58 |
| JayF | I'm really unfamiliar with the bios interface so most of this is informed guessing, outside of `baremetal node clean` being a trigger for manual cleaning | 20:58 |
| iurygregory | there is an API to set BIOS settings | 20:58 |
| iurygregory | I don't think we have CLI support cc bfournie | 20:59 |
| eandersson | Probably not. I don't see it in the code at first glance at least. | 20:59 |
| iurygregory | oh seems like is only to get the values | 21:01 |
| iurygregory | https://docs.openstack.org/api-ref/baremetal/ | 21:02 |
| bfournie | iurygregory: you can set bios settings through clean steps | 21:02 |
| JayF | bfournie: so him running manual cleaning with those steps will persist those settings in automated cleaning? | 21:02 |
| iurygregory | I knew there was a way to set hehe :D | 21:03 |
| JayF | bfournie: I think eandersson figured out how to do it in manual clean; I was trying to help him figure out how to do it in automated cleanign | 21:03 |
| bfournie | JayF: so you want to do it in automated cleaning instead? | 21:04 |
| iurygregory | JayF, according to our docs "The desired BIOS settings are applied during manual cleaning." | 21:04 |
| JayF | I think eandersson does :) I don't know for sure, I'm just proxying for him b/c I was trying to help ^^^^ | 21:04 |
| iurygregory | yeah, I don't think it's possible (not sure if we tested the scenario) | 21:05 |
| iurygregory | https://docs.openstack.org/ironic/latest/admin/bios.html | 21:05 |
| bfournie | iurygregory: yeah that's the doc, I've always done it with manual cleaning | 21:05 |
| TheJulia | It was always intended to just be manual and not automatic | 21:08 |
| JayF | eandersson: ^ there's your answer. Manual cleaning only :) | 21:08 |
| TheJulia | in part, because some vendors need additional reboots when rolling out settings | 21:26 |
| TheJulia | fwiw, I don't think there is an objection to supporting it in automatic mode | 21:26 |
| eandersson | Awesome. Thanks! | 22:16 |
| rpioso | eandersson: FWIW, the manual cleaning approach to configuring BIOS settings is supported by the idrac driver -- https://docs.openstack.org/ironic/latest/admin/drivers/idrac.html#bios-interface. Both WS-Man and Redfish BIOS hardware interface implementations are available, idrac-wsman and idrac-redfish, respectively. | 22:42 |
| opendevreview | Merged openstack/metalsmith stable/wallaby: Replace deprecated import of ABCs from collections https://review.opendev.org/c/openstack/metalsmith/+/801325 | 23:18 |
| eandersson | Thanks rpioso | 23:31 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!