| *** Goneri has quit IRC | 00:27 | |
| *** livelace has quit IRC | 00:36 | |
| *** tkajinam_ is now known as tkajinam | 01:04 | |
| *** rcernin has quit IRC | 01:14 | |
| *** rcernin has joined #openstack-ironic | 01:15 | |
| *** gyee has quit IRC | 01:48 | |
| *** hjensas has quit IRC | 02:25 | |
| *** bfournie has joined #openstack-ironic | 02:27 | |
| *** cdearborn has quit IRC | 02:55 | |
| *** janders has quit IRC | 02:56 | |
| *** janders has joined #openstack-ironic | 02:57 | |
| *** mkrai has joined #openstack-ironic | 03:02 | |
| *** rh-jelabarre has quit IRC | 03:16 | |
| *** dsneddon has quit IRC | 03:45 | |
| *** ijw has quit IRC | 04:07 | |
| *** ijw has joined #openstack-ironic | 04:08 | |
| *** ijw has quit IRC | 04:10 | |
| *** ijw has joined #openstack-ironic | 04:10 | |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 04:16 |
|---|---|---|
| *** rcernin has quit IRC | 04:32 | |
| stevebaker | spam alert | 04:55 |
| * stevebaker cranks the klaxon | 04:55 | |
| stevebaker | or not, git review is stalled | 04:59 |
| *** Lucas_Gray has joined #openstack-ironic | 05:03 | |
| *** rcernin has joined #openstack-ironic | 05:03 | |
| openstackgerrit | Steve Baker proposed openstack/ironic master: New argument validate decorator https://review.opendev.org/742795 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Add status_code argument to @method.expose https://review.opendev.org/743677 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Add @method.body decorator https://review.opendev.org/743678 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Duplicate trait validation with jsonschema https://review.opendev.org/745778 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert allocation endpoint to plain JSON https://review.opendev.org/743679 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert bios endpoint to plain JSON https://review.opendev.org/745475 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert chassis endpoint to plain JSON https://review.opendev.org/745476 | 05:03 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert conductor endpoint to plain JSON https://review.opendev.org/745779 | 05:04 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert deploy_templates endpoint to plain JSON https://review.opendev.org/745780 | 05:04 |
| *** tzumainn has quit IRC | 05:04 | |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert drivers endpoint to plain JSON https://review.opendev.org/741865 | 05:04 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert event endpoint to plain JSON https://review.opendev.org/746623 | 05:04 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: WIP Convert node endpoint to plain JSON https://review.opendev.org/747820 | 05:04 |
| *** SuiongNg|ITRI_TW has joined #openstack-ironic | 05:11 | |
| *** gudrutis1 has joined #openstack-ironic | 05:11 | |
| *** gudrutis has quit IRC | 05:14 | |
| *** gudrutis1 is now known as gudrutis | 05:14 | |
| *** uzumaki has joined #openstack-ironic | 05:17 | |
| *** gudrutis5 has joined #openstack-ironic | 05:24 | |
| *** gudrutis has quit IRC | 05:27 | |
| *** gudrutis5 is now known as gudrutis | 05:27 | |
| *** rcernin has quit IRC | 05:41 | |
| *** Lucas_Gray has quit IRC | 05:44 | |
| *** ricolin_ has joined #openstack-ironic | 06:04 | |
| *** ricolin_ has quit IRC | 06:11 | |
| arne_wiebalck | Good morning, ironic! | 06:15 |
| uzumaki | bonjour arne_wiebalck o/ comment vas-tu? | 06:28 |
| janders | good morning arne_wiebalck uzumaki | 06:29 |
| janders | how are we? | 06:29 |
| arne_wiebalck | hey uzumaki janders o/ | 06:30 |
| arne_wiebalck | how are things on the other side of the planet? | 06:30 |
| *** ijw has quit IRC | 06:31 | |
| *** noama has joined #openstack-ironic | 06:31 | |
| noama | hi dtantsur|afk could you backport https://review.opendev.org/#/c/743498/ to ussuri? | 06:32 |
| patchbot | patch 743498 - ironic-inspector - Relax extra_hardware data validation by default (MERGED) - 2 patch sets | 06:32 |
| rpittau | good morning ironic! o/ | 06:41 |
| rpittau | noama: you can also propose a backport :) | 06:42 |
| noama | it think it should include 2 patches | 06:42 |
| rpittau | noama: what's the other one ? | 06:43 |
| openstackgerrit | Noam Angel proposed openstack/ironic-inspector stable/ussuri: extra_hardware: stop storing raw data in swift https://review.opendev.org/747840 | 06:43 |
| arne_wiebalck | are these eligible for backporting? | 06:44 |
| *** ijw has joined #openstack-ironic | 06:44 | |
| rpittau | it's what I was checking, one is a fix, but this one looks like a change | 06:44 |
| rpittau | it does change an internal logic though | 06:45 |
| noama | extra_hardware is missing in ussuri | 06:48 |
| *** ijw has quit IRC | 06:49 | |
| rpittau | yeah, I remember the issue | 06:50 |
| noama | what do you suggest rpittau would it be better just to backport https://review.opendev.org/#/c/743498/? | 06:50 |
| patchbot | patch 743498 - ironic-inspector - Relax extra_hardware data validation by default (MERGED) - 2 patch sets | 06:50 |
| janders | good morning rpittau o/ | 06:50 |
| janders | arne_wiebalck things are good, thanks for asking. The days are finally getting a bit longer! :) | 06:51 |
| rpittau | noama: that looks like the actual fix | 06:51 |
| rpittau | hey janders :) | 06:51 |
| noama | ok | 06:51 |
| noama | ill try that | 06:51 |
| *** mkrai has quit IRC | 06:52 | |
| *** mkrai_ has joined #openstack-ironic | 06:52 | |
| *** rcernin has joined #openstack-ironic | 06:56 | |
| rpittau | janders: s/retest/recheck :) | 06:57 |
| *** rcernin has quit IRC | 07:03 | |
| janders | rpittau done! I was swearing at Zuul being "slow", now I see why LOL | 07:04 |
| janders | off for a walk, back soon | 07:04 |
| *** rcernin has joined #openstack-ironic | 07:05 | |
| *** rcernin has quit IRC | 07:14 | |
| *** ijw has joined #openstack-ironic | 07:15 | |
| *** ijw has quit IRC | 07:20 | |
| *** iurygregory has joined #openstack-ironic | 07:21 | |
| iurygregory | good morning Ironic! | 07:22 |
| *** dtantsur|afk is now known as dtantsur | 07:26 | |
| dtantsur | morning ironic | 07:26 |
| dtantsur | noama: I agree with rpittau, only the fix should be backported (you'll need to resolve the conflicts manually, unfortunately) | 07:27 |
| iurygregory | morning dtantsur | 07:29 |
| dtantsur | noama: if you're struggling with resolving the conflicts, feel free to ping me | 07:29 |
| *** tosky has joined #openstack-ironic | 07:36 | |
| *** xinliang has joined #openstack-ironic | 07:36 | |
| *** Qianbiao has joined #openstack-ironic | 07:36 | |
| *** dougsz has joined #openstack-ironic | 07:39 | |
| *** Qianbiao has quit IRC | 07:40 | |
| *** Qianbiao has joined #openstack-ironic | 07:41 | |
| dtantsur | folks, could you check https://review.opendev.org/#/c/747413/ please? it demoes nicely that the direct deploy can be used in a low RAM situation without swift | 07:45 |
| patchbot | patch 747413 - metalsmith - CI: use the direct deploy by default - 3 patch sets | 07:45 |
| *** ijw has joined #openstack-ironic | 07:47 | |
| iurygregory | I will add to my list | 07:49 |
| *** ijw has quit IRC | 07:52 | |
| * iurygregory grabs more coffee | 07:56 | |
| openstackgerrit | Merged openstack/ironic master: Enforce autospec in common tests https://review.opendev.org/747767 | 08:00 |
| *** xinliang has quit IRC | 08:03 | |
| openstackgerrit | Riccardo Pittau proposed openstack/metalsmith master: [WIP] Use Centos 8 https://review.opendev.org/737278 | 08:11 |
| *** janders6 has joined #openstack-ironic | 08:11 | |
| openstackgerrit | Merged openstack/bifrost master: ansible-lint: fix error 204 (Lines should be no longer than 160 chars) https://review.opendev.org/747692 | 08:13 |
| *** lucasagomes has joined #openstack-ironic | 08:13 | |
| *** janders has quit IRC | 08:13 | |
| *** janders6 is now known as janders | 08:13 | |
| *** ijw has joined #openstack-ironic | 08:18 | |
| *** ijw has quit IRC | 08:23 | |
| iurygregory | dtantsur, merge conflict https://review.opendev.org/#/c/747434/ =( | 08:27 |
| patchbot | patch 747434 - bifrost - Remove support for non-venv deployments - 2 patch sets | 08:27 |
| rpittau | probably the lint patches | 08:28 |
| iurygregory | yeah | 08:28 |
| openstackgerrit | Merged openstack/bifrost master: ansible-lint: fix errors 303, 305, 306 https://review.opendev.org/747694 | 08:29 |
| noama | hi dtantsur https://review.opendev.org/#/c/743498/ seems much to me to backport it to ussuri | 08:30 |
| patchbot | patch 743498 - ironic-inspector - Relax extra_hardware data validation by default (MERGED) - 2 patch sets | 08:30 |
| *** uzumaki has quit IRC | 08:34 | |
| *** derekh has joined #openstack-ironic | 08:41 | |
| rpittau | if anyone has a moment, this is quite important to prevent future pip-related cataclysms https://review.opendev.org/747218 | 08:44 |
| patchbot | patch 747218 - ironic-python-agent-builder - Pin pip version to install in tinyipa images - 2 patch sets | 08:44 |
| openstackgerrit | Merged openstack/ironic bugfix/15.2: Ansible deploy - Ignore invalid devices https://review.opendev.org/747729 | 08:45 |
| openstackgerrit | Merged openstack/ironic bugfix/15.1: Ansible deploy - Ignore invalid devices https://review.opendev.org/747730 | 08:45 |
| SuiongNg|ITRI_TW | Hi everyone, I encountered a problem while deploying node in UEFI mode. IPA seems to be able to correctly create a vfat partiton (for ESP?) but fails to install grub. By contrast, if the node is deployed in leagcy mode, no vfat partition is created and grub is correctly installed. | 08:45 |
| SuiongNg|ITRI_TW | last_error says "grub-install: error: install device isn't specified" | 08:46 |
| rpittau | SuiongNg|ITRI_TW: which version of IPA/ironic? do you have ironic-python-agent and ironic-conductor logs maybe ? | 08:46 |
| SuiongNg|ITRI_TW | one minute. | 08:47 |
| *** ijw has joined #openstack-ironic | 08:49 | |
| janders | rpittau: Zuul: Magic word? janders: recheck :) | 08:51 |
| SuiongNg|ITRI_TW | http://paste.openstack.org/show/797110/ | 08:51 |
| rpittau | :D | 08:52 |
| rpittau | janders: you need a rebase, zuul tricked you :) | 08:52 |
| SuiongNg|ITRI_TW | I wasn't able to retrieve IPA log because the node power itself off immediatedly. | 08:53 |
| *** ijw has quit IRC | 08:54 | |
| *** k_mouza has joined #openstack-ironic | 08:57 | |
| janders | rpittau last time I needed a rebase I got "in merge conflict" message and some details, now it's just "cannot merge" - what made the difference this time? | 08:57 |
| *** uzumaki has joined #openstack-ironic | 08:58 | |
| rpittau | SuiongNg|ITRI_TW: it seems grub is missing the target for the installation for some reason, hard to say why without ipa logs, you could try pausing IPA setting the node to maintenance, also be sure to have debug enabled | 08:59 |
| rpittau | janders: I believe it's the same, if you pass the mouse cursor on the "Cannot Merge" message you will see the reason | 09:00 |
| arne_wiebalck | SuiongNg|ITRI_TW: is this with software RAID? | 09:00 |
| dtantsur | noama: sorry, what do you mean? You don't think it should be backported? Or you're struggling with conflicts? | 09:01 |
| janders | rpittau ACK. I bet I'm in conflict with dtantsur ansible linter work as I patched some horribly long lines indeed. | 09:01 |
| SuiongNg|ITRI_TW | arne_wiebalck: hardware raid (LSI card) | 09:02 |
| dtantsur | yeah, the lint patches conflict with everything.. haven't thought about it, sorry | 09:02 |
| dtantsur | janders: you get 'in merge conflict' when anything happens with your patch (zuul is running, you post an update, etc) | 09:03 |
| dtantsur | otherwise zuul will or will not let you know explicitly | 09:03 |
| dtantsur | not sure why exactly | 09:03 |
| janders | No worries, I will fix it. It's much nicer to work with the updated code - great work dtantsur :) | 09:04 |
| openstackgerrit | Merged openstack/bifrost master: Remove env-vars https://review.opendev.org/747668 | 09:04 |
| SuiongNg|ITRI_TW | rpittau: So I have to first put the node in deployment mode. And set it to maintenance as soon as it enter "deploying" state ? | 09:04 |
| SuiongNg|ITRI_TW | arne_wiebalck: btw, it also fails on another node without any hardware raid card. | 09:07 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Remove support for non-venv deployments https://review.opendev.org/747434 | 09:08 |
| dtantsur | iurygregory: ^^ | 09:08 |
| dtantsur | janders: btw there is a (currently ignored) lint for missing mode in file operations | 09:08 |
| dtantsur | you may even take the task of removing it | 09:08 |
| arne_wiebalck | SuiongNg|ITRI_TW: I am little puzzled on why it creates a partition ... shouldn't this come from the image? Or are you using partition images? | 09:09 |
| dtantsur | (although probably in a separate patch because this one has to be backported) | 09:09 |
| arne_wiebalck | Does the white paper link on openstack.org/baremetal work for anyone? It seems to redirect me to the very same page .. | 09:11 |
| SuiongNg|ITRI_TW | arne_wiebalck: I'm not sure. I didn't specify the ramdisk and kernel. However, the image is generate by disk-image-create with baremetal element. Yet from my understanding, this "baremetal" element only extract the kernel and ramdisk, which means is doesn't alter the qcow partition image. | 09:13 |
| iurygregory | arne_wiebalck, when you click "Read It Now"? | 09:14 |
| arne_wiebalck | iurygregory: does that work for you? | 09:14 |
| iurygregory | arne_wiebalck, nope | 09:14 |
| arne_wiebalck | iurygregory: that is what I mean | 09:14 |
| arne_wiebalck | iurygregory: :) | 09:14 |
| iurygregory | trouble for us =) | 09:14 |
| arne_wiebalck | the stats on clicking the link will go up as everyone tries twice :-D | 09:15 |
| janders | will grab some dinner, back in half an hour | 09:15 |
| iurygregory | ouch, in the ironic website we re-direct to openstack.org/baremetal hehe | 09:16 |
| *** mkrai_ has quit IRC | 09:18 | |
| *** mkrai__ has joined #openstack-ironic | 09:18 | |
| arne_wiebalck | I will contact the foundation. | 09:19 |
| iurygregory | arne_wiebalck, ++ | 09:19 |
| *** ianychoi__ has joined #openstack-ironic | 09:21 | |
| *** SuiongNg|ITRI_TW has quit IRC | 09:21 | |
| *** ijw has joined #openstack-ironic | 09:21 | |
| *** SuiongNg|ITRI_TW has joined #openstack-ironic | 09:22 | |
| *** ianychoi_ has quit IRC | 09:24 | |
| *** belmoreira has joined #openstack-ironic | 09:25 | |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic master: Add tempest default_boot_option and altflavor https://review.opendev.org/741886 | 09:26 |
| *** ijw has quit IRC | 09:26 | |
| *** hjensas has joined #openstack-ironic | 09:29 | |
| *** k_mouza has quit IRC | 09:36 | |
| *** k_mouza has joined #openstack-ironic | 09:45 | |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: ansible-lint: fix error 206 https://review.opendev.org/747883 | 09:49 |
| openstackgerrit | Merged openstack/ironic stable/train: Ansible deploy - Ignore invalid devices https://review.opendev.org/747577 | 09:49 |
| openstackgerrit | Merged openstack/metalsmith master: CI: use the direct deploy by default https://review.opendev.org/747413 | 09:51 |
| *** livelace has joined #openstack-ironic | 09:53 | |
| *** ijw has joined #openstack-ironic | 09:53 | |
| openstackgerrit | Merged openstack/python-ironic-inspector-client master: Provide a clear error message when trying to access ironic (not inspector) https://review.opendev.org/746956 | 09:53 |
| *** ijw has quit IRC | 09:57 | |
| dtantsur | JayF: started https://storyboard.openstack.org/#!/story/2008057 for client certificates, mostly a copy paste for now | 09:57 |
| *** uzumaki has quit IRC | 10:03 | |
| *** NobodyCam has quit IRC | 10:14 | |
| *** buhman has quit IRC | 10:14 | |
| janders | do I need to do anything special in the commit message while finishing the rebase? | 10:14 |
| janders | or just leave as-is and submit the new revision for review? | 10:15 |
| *** livelace has quit IRC | 10:15 | |
| iurygregory | janders, no need =) | 10:15 |
| iurygregory | just git rebase -i master | 10:15 |
| *** johnsom has quit IRC | 10:16 | |
| iurygregory | and git review =D | 10:16 |
| *** NobodyCam has joined #openstack-ironic | 10:16 | |
| *** buhman has joined #openstack-ironic | 10:16 | |
| openstackgerrit | Merged openstack/ironic bugfix/15.2: Ensure in-band deploy steps are present in time for fast-track deployments https://review.opendev.org/747737 | 10:16 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 10:17 |
| *** johnsom has joined #openstack-ironic | 10:17 | |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 10:18 |
| janders | oops I somehow missed the merging markers | 10:19 |
| janders | cleaning up now | 10:19 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 10:21 |
| dtantsur | janders: a release note please! | 10:21 |
| janders | dtantsur ACK will do | 10:21 |
| dtantsur | backportable bug fixes need release notes that explain the fix | 10:21 |
| dtantsur | (and features need release notes that introduce the features) | 10:21 |
| janders | I think I already broke World Record for number of revisions needed for adding a bunch of chmods :) one revision more will make the record stand longer | 10:22 |
| dtantsur | heh | 10:22 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Remove support for the legacy CSV format https://review.opendev.org/747893 | 10:22 |
| janders | apologies for the spam | 10:22 |
| dtantsur | janders: I still remember the review that added IPA support in ironic. It reached 100 revisions, and gerrit started choking on it :D | 10:23 |
| janders | LOL | 10:23 |
| dtantsur | so another review was created and reached probably 50 revisions more | 10:23 |
| janders | that is DDoS in style | 10:23 |
| dtantsur | fun times :) | 10:23 |
| janders | hats off | 10:23 |
| janders | breaking stuff for fun and profit as they say | 10:23 |
| *** ijw has joined #openstack-ironic | 10:24 | |
| openstackgerrit | Merged openstack/ironic-python-agent-builder master: Pin pip version to install in tinyipa images https://review.opendev.org/747218 | 10:26 |
| *** k_mouza has quit IRC | 10:26 | |
| openstackgerrit | Verification of a change to openstack/ironic failed: Ansible deploy - Ignore invalid devices https://review.opendev.org/747576 | 10:26 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Use packaged iPXE ROMs for openSUSE https://review.opendev.org/747894 | 10:27 |
| *** k_mouza has joined #openstack-ironic | 10:28 | |
| *** ijw has quit IRC | 10:29 | |
| janders | dtantsur does the release note for the bifrost/get_url/chmod fix only need "fixes" section? | 10:30 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Use packaged iPXE ROMs for openSUSE https://review.opendev.org/747894 | 10:31 |
| dtantsur | janders: well, we can also add an upgrade section to highlight the fact that we'll change ownership and ACL | 10:31 |
| janders | dtantsur thank you | 10:32 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Use packaged iPXE ROMs for openSUSE https://review.opendev.org/747894 | 10:35 |
| dtantsur | I'm also good at updating a trivial patch too often.. | 10:35 |
| *** livelace has joined #openstack-ironic | 10:35 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Trivial: stop using deprecated HTTP basic options https://review.opendev.org/747897 | 10:40 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 10:47 |
| janders | hopefully I got ^^^ right :) | 10:48 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Remove support for OpenStack MetaData version 2012-08-10 https://review.opendev.org/747900 | 10:49 |
| dtantsur | so nice to remove code | 10:49 |
| *** pas-ha has quit IRC | 10:49 | |
| *** ildikov has quit IRC | 10:49 | |
| dtantsur | janders: let's also do owner/group ironic:ironic? | 10:49 |
| dtantsur | janders: and we tend to use present tense in release notes | 10:50 |
| *** sri_ has quit IRC | 10:50 | |
| dtantsur | imagine saying "This version" and adding your note | 10:50 |
| dtantsur | okay, this can be used with past tense as well :) but we tend to use present | 10:50 |
| *** rpioso has quit IRC | 10:50 | |
| janders | dtantsur adding user/group now (good idea!) - and will fix up the tenses while I'm at it | 10:52 |
| dtantsur | janders: added a few other comments | 10:52 |
| janders | dtantsur ACK. Would you like me to switch all the files I'm changing to modern syntax (multi-line and ":" instead of "=") or just the tasks I'm updating? | 10:54 |
| janders | happy to do both but it feels like that might be better off in a separate change | 10:54 |
| *** ijw has joined #openstack-ironic | 10:56 | |
| *** Qianbiao has quit IRC | 10:57 | |
| dtantsur | janders: just the tasks you're updating | 10:57 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Stop supporting storing introspection data in nginx https://review.opendev.org/747901 | 10:57 |
| *** rcernin has joined #openstack-ironic | 10:57 | |
| dtantsur | I'd prefer we update them gradually every time we have to touch a task | 10:57 |
| janders | dtantsur ACK, on it | 10:59 |
| *** rpioso has joined #openstack-ironic | 10:59 | |
| *** ijw has quit IRC | 11:00 | |
| *** sri_ has joined #openstack-ironic | 11:01 | |
| *** pas-ha has joined #openstack-ironic | 11:01 | |
| ajya | dtantsur: did you get any volunteers for deploy_steps RFE? If not, I could volunteer, though in the next cycle | 11:03 |
| dtantsur | ajya: I don't remember.. I think I didn't. Thank you, I'll keep that in mind, although I'd really prefer this to be finished this cycle. | 11:03 |
| janders | dtantsur are you happy with hardcoding owner:group ironic:ironic or do we need a new var for this? I see it hardcoded a fair bit in the code but I thought I will check in case we want to move away from this. | 11:04 |
| dtantsur | janders: I'm happy with hardcoding it | 11:04 |
| janders | dtantsur ACK | 11:04 |
| dtantsur | I personally am against having too many options | 11:04 |
| dtantsur | when they don't add a visible benefit for operators | 11:05 |
| janders | yeah flexibility for the sake of flexibility is an average idea and a reliable source of distractions :) | 11:05 |
| * dtantsur has spent this morning removing options from bifrost | 11:06 | |
| ajya | dtantsur: no time left in this cycle for me, so can't commit to it now. Let's see if anyone volunteers for this cycle :) | 11:07 |
| dtantsur | sure, understandable | 11:08 |
| *** ildikov has joined #openstack-ironic | 11:10 | |
| *** dougsz has quit IRC | 11:11 | |
| dtantsur | "File '/lib/systemd/system/uwsgi.service' created with default permissions '600'. The previous default was '666'. Specify 'mode' to avoid this warning." | 11:13 |
| dtantsur | we're getting tons of these warnings now >_< | 11:13 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 11:14 |
| janders | ^^^ has a fair bit of syntax tweaking, so will rerun tests on host07, hence -1/WIP | 11:15 |
| dtantsur | janders: left come comments before you test | 11:16 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Stop supporting storing introspection data in nginx https://review.opendev.org/747901 | 11:16 |
| dtantsur | you're obsoleting my previous fixes, so they can be removed | 11:16 |
| janders | dtantsur got it | 11:17 |
| dtantsur | janders: and please pay attention to trailing spaces | 11:17 |
| dtantsur | if you use vim, you can make it highlight them | 11:17 |
| ajya | dtantsur: looked again at that flavor-deploy template thing and deploy steps not being picked up from flavor/traits. Could it be because I'm using baremetal API and not nova API to create nodes? | 11:17 |
| dtantsur | janders: like https://github.com/dtantsur/config/blob/master/vimrc#L72-L73 | 11:17 |
| dtantsur | ajya: if you're not using nova API, flavors are not used | 11:18 |
| dtantsur | ironic is completely unaware of flavors | 11:18 |
| ajya | dtantsur: should docs be updated for this use case? | 11:18 |
| janders | dtantsur totally stealing this, thanks! :) | 11:19 |
| janders | (it looks I realised half-way breaking up the oldschool long-line calls leaves a ton of whitespaces behind, shame I forgot to go back and clean the previous ones) | 11:19 |
| janders | well, the new vim config will sort this out for the future | 11:20 |
| *** mkrai__ has quit IRC | 11:24 | |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 11:24 |
| *** ijw has joined #openstack-ironic | 11:27 | |
| *** mkrai has joined #openstack-ironic | 11:28 | |
| *** trident has quit IRC | 11:31 | |
| *** ijw has quit IRC | 11:32 | |
| janders | ^^^ (test running) | 11:32 |
| openstackgerrit | Merged openstack/metalsmith master: Move metalsmith_instances from tripleo-ansible https://review.opendev.org/737887 | 11:38 |
| *** Lucas_Gray has joined #openstack-ironic | 11:41 | |
| dtantsur | ajya: I think we do cover the standalone case separately? | 11:43 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Authentication JSON RPC even in no-auth mode https://review.opendev.org/747897 | 11:48 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: Remove support for OpenStack MetaData version 2012-08-10 https://review.opendev.org/747900 | 11:51 |
| openstackgerrit | Jacob Anders proposed openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 11:56 |
| *** rh-jelabarre has joined #openstack-ironic | 11:58 | |
| *** ijw has joined #openstack-ironic | 11:59 | |
| openstackgerrit | Merged openstack/ironic master: Move redfish-virtual-media to the back of supported_boot_interfaces https://review.opendev.org/747430 | 12:00 |
| *** ijw has quit IRC | 12:03 | |
| shadower | re: https://storyboard.openstack.org/#!/story/2008038 -- is the out of bound inspection basically whatever we can get by just asking the BMC? | 12:06 |
| shadower | (trying to undrestand what the difference between out-of-bound and in-bound is) | 12:06 |
| dtantsur | shadower: nit: ouf-of-band. otherwise, yes | 12:06 |
| shadower | lol yes, sorry | 12:07 |
| dtantsur | shadower: https://docs.openstack.org/ironic/latest/admin/inspection.html may help | 12:07 |
| shadower | dtantsur: thanks! | 12:07 |
| * dtantsur dives back into the ansible's TLS support | 12:07 | |
| *** hoonetorg has quit IRC | 12:08 | |
| ajya | dtantsur: I would expect this mentioned in this section https://docs.openstack.org/ironic/latest/admin/node-deployment.html#deploy-templates as it contains info how to override default deploy steps. It only has section about Compute service. | 12:10 |
| *** Goneri has joined #openstack-ironic | 12:11 | |
| dtantsur | ajya: good point | 12:11 |
| ajya | dtantsur: is it documented somewhere else? Then could link from here. | 12:12 |
| dtantsur | ajya: not sure, maybe not | 12:12 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Build centos8 on centos8 https://review.opendev.org/746901 | 12:14 |
| ajya | dtantsur: I can add that small section if that's the valid solution I'm doing - adding deploy template name to trait and instance info manually | 12:14 |
| ajya | or maybe that needs to added to https://docs.openstack.org/ironic/latest/install/standalone.html and linked | 12:16 |
| *** thiagop has joined #openstack-ironic | 12:18 | |
| *** hoonetorg has joined #openstack-ironic | 12:21 | |
| janders | okay, lab testing of https://review.opendev.org/#/c/747815/ succeeded, the instances PXE booted correctly | 12:21 |
| patchbot | patch 747815 - bifrost - Explicitly set permissions on /httpboot contents - 8 patch sets | 12:21 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: Move zuul config in zuul.d https://review.opendev.org/747914 | 12:22 |
| janders | dtantsur would you be happy if I apply rpittau recommendation here: https://review.opendev.org/#/c/747815/8/releasenotes/notes/releasenote-341a5eebe6168aea.yaml and merge the two sections into one "fixes" section? | 12:23 |
| patchbot | patch 747815 - bifrost - Explicitly set permissions on /httpboot contents - 8 patch sets | 12:23 |
| dtantsur | janders, rpittau, it IS an upgrade impact if operators rely on the files having specific owners or permissions | 12:23 |
| dtantsur | dunno, no hard feelings either way | 12:23 |
| rpittau | mmm | 12:23 |
| janders | rpittau - should I change it or leave it? | 12:24 |
| rpittau | wouldn't that have impact on the backport then ? | 12:24 |
| rpittau | dtantsur: ^ | 12:25 |
| *** Qianbiao has joined #openstack-ironic | 12:27 | |
| dtantsur | not necessary | 12:27 |
| rpittau | ok | 12:28 |
| dtantsur | I mean, release notes themselves don't impact backporting. the change does. | 12:28 |
| rpittau | yes, of course, I guess I'm just being too strict here | 12:29 |
| rpittau | janders: I removed the -1 | 12:29 |
| janders | thank you rpittau dtantsur | 12:30 |
| *** ijw has joined #openstack-ironic | 12:30 | |
| janders | my test succeeded as well - let's see what Zuul does, hopefully the tests pass without sacrificing a marshmallow :) | 12:30 |
| rpittau | lol | 12:31 |
| janders | and with this I will call it a night, thanks for all your help and I shall see you tomorrow o/ | 12:31 |
| janders | have a good day all | 12:31 |
| rpittau | bye janders o/ | 12:31 |
| *** ijw has quit IRC | 12:35 | |
| *** hoonetorg has quit IRC | 12:41 | |
| *** mkrai has quit IRC | 12:45 | |
| *** chenhaw has quit IRC | 12:45 | |
| *** hoonetorg has joined #openstack-ironic | 12:54 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 12:56 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 12:59 |
| * dtantsur expects a lot of red on ^^^ | 12:59 | |
| *** bnemec has joined #openstack-ironic | 13:00 | |
| *** vdrok has quit IRC | 13:02 | |
| *** ijw has joined #openstack-ironic | 13:02 | |
| *** trident has joined #openstack-ironic | 13:02 | |
| *** vdrok has joined #openstack-ironic | 13:02 | |
| *** rloo has joined #openstack-ironic | 13:03 | |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: ansible-lint: fix error 206 https://review.opendev.org/747925 | 13:04 |
| openstackgerrit | Riccardo Pittau proposed openstack/bifrost master: ansible-lint: fix error 504 https://review.opendev.org/747925 | 13:05 |
| *** ijw has quit IRC | 13:07 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 13:17 |
| openstackgerrit | Riccardo Pittau proposed openstack/networking-baremetal master: Add missing keystoneauth1 to requirements https://review.opendev.org/747936 | 13:30 |
| openstackgerrit | Riccardo Pittau proposed openstack/networking-baremetal master: Add missing keystoneauth1 and oslo.service to requirements https://review.opendev.org/747936 | 13:32 |
| TheJulia | good morning | 13:33 |
| rpittau | good morning TheJulia :) | 13:33 |
| * TheJulia tries to weigh up | 13:34 | |
| TheJulia | wake | 13:34 |
| TheJulia | jeeze | 13:34 |
| dtantsur | morning TheJulia | 13:36 |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy master: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/747951 | 13:40 |
| dtantsur | TheJulia: lovely, isn't it? ^^^ | 13:40 |
| TheJulia | dtantsur: too early to add whiskey into my coffee that I'm waking up to | 13:40 |
| dtantsur | likely | 13:41 |
| TheJulia | especially when I also have a board meeting this morning | 13:42 |
| * TheJulia takes out the corgi overlord | 13:43 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [WIP] Use tinycore 11 to build tinyipa https://review.opendev.org/711647 | 13:47 |
| *** tzumainn has joined #openstack-ironic | 13:48 | |
| guilhermesp | hi team, just sharing as i raised up the question yesterday about custom python-agent: users were not exporting the ELEMENTS_PATH and they were using dib vm element to create ramdisk images.... that was the trick | 13:48 |
| openstackgerrit | Julia Kreger proposed openstack/sushy stable/ussuri: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/747956 | 13:52 |
| *** weshay is now known as weshay|interview | 13:54 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 13:54 |
| dtantsur | TheJulia: when you cherry-pick unmerged patches via UI, the cherry-picked line is missing | 13:56 |
| openstackgerrit | Verification of a change to openstack/bifrost failed: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 13:58 |
| *** gyee has joined #openstack-ironic | 14:03 | |
| TheJulia | gah :( | 14:04 |
| TheJulia | I can re-pick them later | 14:04 |
| dtantsur | rpioso: a question for you from both iDRAC and the redfish standard standpoint: is it expected for /redfish/v1/SessionService to be itself authenticated? | 14:05 |
| iurygregory | good morning TheJulia =) | 14:05 |
| dtantsur | rpioso: in theory, we have to discovery the URL to use to create sessions by GET /redfish/v1/SessionService (this is how the standard is supposed to work) | 14:06 |
| dtantsur | but I see that this endpoint is authenticated on at least some Dell machines | 14:06 |
| dtantsur | sushy works around it by guessing /redfish/v1/SessionService/Sessions, but that's against the spirit of the standard | 14:06 |
| *** ijw has joined #openstack-ironic | 14:06 | |
| *** ijw has quit IRC | 14:07 | |
| *** ijw has joined #openstack-ironic | 14:07 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/sushy master: Less scary warning when GET /redfish/v1/SessionService fails https://review.opendev.org/747960 | 14:11 |
| dtantsur | patch related ^^^ | 14:11 |
| *** Lucas_Gray has quit IRC | 14:13 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 14:14 |
| JayF | If we can freely change lower-constraints.txt and requirements.txt in IPA repo, what does `requirements-check` check for? | 14:16 |
| Qianbiao | dtantsur i thinks yes. To visit session service, you should be authenticated. | 14:17 |
| dtantsur | Qianbiao: yeah, but you're using it for authentication :) | 14:17 |
| dtantsur | how can you be authenticated before authentication? | 14:17 |
| Qianbiao | dtantsur no. | 14:17 |
| Qianbiao | In ibmc, https://github.com/IamFive/python-ibmcclient/blob/master/ibmc_client/connector.py#L116-L133 | 14:18 |
| Qianbiao | POST /Sessions with your username and password to get a token. | 14:18 |
| dtantsur | Qianbiao: how do you know the '%s/Sessions' bit? why not '%s/MyFancySessions' | 14:18 |
| dtantsur | ? | 14:18 |
| Qianbiao | If user want to know which session is current session. | 14:18 |
| Qianbiao | he can visit /SessionService/sessions to list all session, and one of session will have a flag identify that is yours | 14:19 |
| TheJulia | Does the spec even permit an entirely different Session path? | 14:19 |
| dtantsur | TheJulia: I don't think the spec defines *any* paths | 14:19 |
| dtantsur | it explicitly prefers consumers to walk up from the root | 14:19 |
| TheJulia | eh, true | 14:20 |
| TheJulia | and the root endpoint is _supposed_ to enumerate endpoints | 14:20 |
| TheJulia | but not everyone does | 14:20 |
| dtantsur | hence my question to Qianbiao: how do I know that I need /redfish/v1/SessionService/Sessions not /redfish/v1/SessionService/ImBeingFancy/Sessions? | 14:20 |
| dtantsur | I mean, I hope nobody does THAT, but the standard seems to allow it.. | 14:20 |
| Qianbiao | i see. you want to get path from root element? | 14:21 |
| dtantsur | Qianbiao: I don't want that (I actually don't like it), but the standard seems to suggest it | 14:21 |
| dtantsur | so I'm curious what the vendors think. I guess you expect the path to be static? | 14:21 |
| Qianbiao | Yes | 14:22 |
| Qianbiao | it is documented we need to visit it that way :) | 14:22 |
| Qianbiao | ibmc provide a path to sessions. | 14:22 |
| Qianbiao | { | 14:22 |
| Qianbiao | "@odata.context": "/redfish/v1/$metadata#SessionService", | 14:22 |
| Qianbiao | "@odata.id": "/redfish/v1/SessionService", | 14:22 |
| Qianbiao | "@odata.type": "#SessionService.v1_0_2.SessionService", | 14:22 |
| Qianbiao | "Id": "SessionService", | 14:22 |
| Qianbiao | "Name": "Session Service", | 14:22 |
| Qianbiao | "SessionTimeout": 600, | 14:22 |
| Qianbiao | "Sessions": { | 14:22 |
| Qianbiao | "@odata.id": "/redfish/v1/SessionService/Sessions" | 14:23 |
| Qianbiao | }, | 14:23 |
| Qianbiao | "Oem": { | 14:23 |
| Qianbiao | "Huawei": { | 14:23 |
| Qianbiao | "WebSessionTimeoutMinutes": 480, | 14:23 |
| Qianbiao | "WebSessionMode": "Shared" | 14:23 |
| Qianbiao | } | 14:23 |
| Qianbiao | } | 14:23 |
| Qianbiao | } | 14:23 |
| rpittau | Qianbiao: please don't paste long texts here, use a paste service | 14:23 |
| Qianbiao | ok sorry. | 14:23 |
| dtantsur | mmm, sure, but to see this document you need authentication, no? | 14:24 |
| Qianbiao | Yes, get a token or basic auth | 14:24 |
| dtantsur | so before authentication I have to just know it from the documentation, right? | 14:24 |
| Qianbiao | You need to get the auth endpoint. in ibmc is POST /Sessions | 14:24 |
| rpittau | JayF: mainly compatibility with global requirements | 14:25 |
| JayF | rpittau: so what does that mean, exactly? I have a patch that needs to modify requirements/lower-constraints | 14:26 |
| JayF | https://review.opendev.org/#/c/746774/ | 14:28 |
| patchbot | patch 746774 - ironic-python-agent - Eventlet should be monkey patched as early as poss... - 9 patch sets | 14:28 |
| rpioso | dtantsur: Yes, it is expected that /redfish/v1/SessionService be authenticated. | 14:31 |
| dtantsur | okay, I wonder if we should update sushy to stop even trying to get there before authentication.. | 14:31 |
| rpioso | dtantsur: We should :-) | 14:33 |
| rpioso | dtantsur: What's needed is available from the Service Root. Looking ... | 14:33 |
| Qianbiao | dtantsur if the sessions path is the only thing you want to get. | 14:34 |
| Qianbiao | you can get it from /redfish/v1 root | 14:34 |
| *** thiagop has quit IRC | 14:34 | |
| Qianbiao | it has the path to sessions | 14:34 |
| *** ijw_ has joined #openstack-ironic | 14:34 | |
| dtantsur | could someone paste an example to paste.openstack.org please? | 14:35 |
| Qianbiao | trying. its very slow to visit paste.openstack.org here | 14:35 |
| Qianbiao | http://paste.openstack.org/show/797131/ | 14:35 |
| rpioso | dtantsur: If memory serves, the Links > Sessions property can be used to create a session. | 14:36 |
| dtantsur | sweeet! is it a standard feature? | 14:36 |
| Qianbiao | at least ibmc yes. | 14:37 |
| Qianbiao | not sure about standard | 14:37 |
| jroll | JayF: it means it makes sure lower-constraints are in global-requirements, don't conflict with blacklisted versions, etc afaik | 14:37 |
| rpioso | dtantsur: https://www.dmtf.org/sites/default/files/standards/documents/DSP0266_1.11.0.pdf, p. 158 | 14:38 |
| JayF | jroll: for https://review.opendev.org/#/c/746774/ should I only be modifying lower-constraints.txt, and not requirements.txt? | 14:38 |
| patchbot | patch 746774 - ironic-python-agent - Eventlet should be monkey patched as early as poss... - 9 patch sets | 14:38 |
| rpioso | "Services shall authenticate all write requests to Redfish resources. For example: • POST, except to the Sessions resource collection for authentication" | 14:38 |
| *** ijw has quit IRC | 14:38 | |
| JayF | jroll: the lwoer-constraints.txt are contained within requirements, but I'm pretty sure IPA wouldn't actually run with the older version, so I changed requirements too | 14:38 |
| rpioso | dtantsur: And p. 159 discusses it. | 14:39 |
| dtantsur | rpioso, Qianbiao, many thanks! I've filed https://storyboard.openstack.org/#!/story/2008059 to track it (I won't have time for it myself soon). | 14:39 |
| jroll | JayF: good question. I think requirements.txt has to match global but I'm not 100% | 14:40 |
| rpioso | dtantsur: You're welcome :-) | 14:40 |
| jroll | JayF: also I misspoke on check-requirements, I believe that just checks requirements.txt and not lower-constraints (but maybe both) | 14:40 |
| dtantsur | jroll, JayF, lower-constraints.txt has to match requirements.txt. global-requirements no longer have low caps. | 14:40 |
| Qianbiao | JayF iurygregory I remember i post a bug about eventlet too. | 14:40 |
| dtantsur | jroll: check-requirements checks both | 14:40 |
| Qianbiao | Not sure where it moved to last. | 14:40 |
| JayF | dtantsur: match, how? | 14:40 |
| JayF | dtantsur: e.g. requirements.txt lower cap has to be the same as the lower cap in lower-constraints.txt? | 14:41 |
| Qianbiao | dtantsur you are welcome. | 14:41 |
| dtantsur | JayF: if requirements.txt has foo>=x.y, lower-constraints.txt must have foo===x.y | 14:41 |
| rpioso | dtantsur: And, yes, it's standard. sushy, not so much ;-) | 14:41 |
| JayF | dtantsur: ack, so changing both was correct? Or am I going to have angry global-requirements problems now? | 14:41 |
| dtantsur | rpioso: exactly, hence the ugly warning (and my confusion) | 14:41 |
| * JayF doesn't understand why IPA needs to deal with any of this given it never co-installs | 14:41 | |
| * jroll wonders what's the point of lower-constraints if it has to match the bottom of requirements.txt | 14:42 | |
| dtantsur | JayF: you have to change both. the tricky part will be figuring it which new versions you're pulling by modifying lower-constraints | 14:42 |
| JayF | dtantsur: I already did that | 14:42 |
| JayF | dtantsur: and reflected those changes in lower-c and requirements | 14:42 |
| dtantsur | jroll: the point is to be able to test with lower constraints. just requirements.txt won't give you that, pip is not that smart. | 14:42 |
| JayF | dtantsur: new oslo.service needed new oslo.utils + eventlet, and that's where the chain stopped | 14:42 |
| Qianbiao | JayF https://storyboard.openstack.org/#!/story/2007553 | 14:43 |
| Qianbiao | it's 3rd party lib eventlet issue too. | 14:43 |
| JayF | Qianbiao: yeah, that code is completely incompatible with eventlet. | 14:44 |
| JayF | Qianbiao: my changes are in IPA though, not Ironic, even though that bug is eventlet related | 14:44 |
| JayF | that's directly, 100% incompatible with eventlet code | 14:44 |
| JayF | we probably shouldn't be using pyinotify if it requires select.poll() | 14:44 |
| JayF | ah, as dtantsur said in your bug, it's a oslo_log bug/incompatibility, you'll have to file it with them | 14:45 |
| *** rcernin has quit IRC | 14:45 | |
| Qianbiao | yes | 14:46 |
| *** mkrai has joined #openstack-ironic | 14:49 | |
| rpittau | JayF: AFAICS the patch looks ok for the requirements | 14:50 |
| JayF | rpittau: thanks :D I appreciate the sanity check. Last time I had to touch requirements was when you had to get a committee to approve raising a requirement b/c they were shared across all openstack projects | 14:51 |
| *** chenhaw has joined #openstack-ironic | 14:51 | |
| *** vmud213 has joined #openstack-ironic | 14:53 | |
| rpittau | JayF: to prevent too much sanity loss, you can run the requirements-check locally using the tools in the requirements repo, it's what I usually do :) | 14:53 |
| JayF | rpittau: ah, I was trying to figure that out, didn't realize it was in a separate repo | 14:54 |
| rpittau | JayF: https://opendev.org/openstack/requirements | 14:54 |
| JayF | ack | 14:55 |
| *** Qianbiao is now known as qianbiao|afk | 14:59 | |
| *** qianbiao|afk has quit IRC | 15:04 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 15:06 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 15:15 |
| openstackgerrit | Merged openstack/bifrost master: Explicitly set permissions on /httpboot contents https://review.opendev.org/747815 | 15:21 |
| dtantsur | janders: when you're back, could you backport ^^ to stable/ussuri and bugfix/8.3? | 15:22 |
| dtantsur | there are merge conflicts, like because of lint changes | 15:22 |
| *** livelace has quit IRC | 15:26 | |
| *** livelace has joined #openstack-ironic | 15:26 | |
| *** belmoreira has quit IRC | 15:29 | |
| dtantsur | SSLError(CertificateError("hostname '192.168.122.1' doesn't match '192.168.122.1'",)) | 15:32 |
| dtantsur | technology suchs | 15:32 |
| iurygregory | wow | 15:38 |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 15:40 |
| *** jtomasek has joined #openstack-ironic | 15:41 | |
| openstackgerrit | Merged openstack/sushy master: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/747951 | 15:44 |
| openstackgerrit | Shivanand Tendulker proposed openstack/ironic master: Enforce autospec in ilo tests https://review.opendev.org/747983 | 15:53 |
| *** sshnaidm is now known as sshnaidm|afk | 15:59 | |
| *** jtomasek has quit IRC | 16:03 | |
| *** rloo has quit IRC | 16:04 | |
| *** weshay|interview is now known as weshay | 16:04 | |
| *** rloo has joined #openstack-ironic | 16:04 | |
| *** lucasagomes has quit IRC | 16:06 | |
| *** rloo has quit IRC | 16:07 | |
| *** rloo has joined #openstack-ironic | 16:08 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/bifrost master: [WIP] TLS support for API services https://review.opendev.org/747921 | 16:18 |
| *** dtantsur is now known as dtantsur|afk | 16:19 | |
| dtantsur|afk | o/ | 16:19 |
| *** rloo has quit IRC | 16:25 | |
| *** rloo has joined #openstack-ironic | 16:25 | |
| *** k_mouza has quit IRC | 16:26 | |
| *** bnemec has quit IRC | 16:30 | |
| *** bnemec has joined #openstack-ironic | 16:35 | |
| *** noama has quit IRC | 16:35 | |
| *** rh-jelabarre has quit IRC | 16:43 | |
| *** rh-jelabarre has joined #openstack-ironic | 16:43 | |
| openstackgerrit | vinay kumar muddu proposed openstack/ironic master: [WIP]: Adds ilo-uefi-https boot interface to ilo5 https://review.opendev.org/745501 | 16:44 |
| JayF | Can anyone offer insight in troubleshooting the metalsmith-integration-ipa-src job? | 16:46 |
| JayF | zuul is indicating a failure but I'm having trouble even seeing where/how it failed in the logs | 16:47 |
| *** vmud213 has quit IRC | 16:49 | |
| TheJulia | JayF: got a link to a job log? | 16:51 |
| *** lmcgann has joined #openstack-ironic | 16:52 | |
| JayF | https://bb909fa5d52a05ca950d-f24a7834eba09db97966c05c7e428413.ssl.cf2.rackcdn.com/746774/9/check/metalsmith-integration-ipa-src/cbdd365/ | 16:52 |
| JayF | I think I found something, where ansible said failed to ssh into the instance | 16:52 |
| TheJulia | hmmmm | 16:53 |
| TheJulia | yeah | 16:53 |
| JayF | I don't know what "the instance" is in this case? The actual test node? The BM node deployed into? | 16:53 |
| TheJulia | https://bb909fa5d52a05ca950d-f24a7834eba09db97966c05c7e428413.ssl.cf2.rackcdn.com/746774/9/check/metalsmith-integration-ipa-src/cbdd365/controller/ironic-bm-logs/node-0_console.log | 16:53 |
| TheJulia | yeah, the test node | 16:54 |
| * TheJulia wonders what is causing that | 16:54 | |
| JayF | It's hard to imagine my change could cause that kind of failure, especially when it is succeeding (so far) in the other tests | 16:55 |
| JayF | (only one still running is the inspector tempest tests) | 16:55 |
| JayF | I did notice that metalsmith was flipped to direct deploy for CI yesterday | 16:56 |
| JayF | Hard to imagine that happening in such a way to succeed CI for that commit but fail for ipa-src jobs | 16:56 |
| lmcgann | Hello, I am an engineer from the red hat research team. I am working to integrate Keylime into ironic. Would anybody be able to point me to any resources where I could begin to look into how this would be done. | 16:56 |
| JayF | I don't know what keylime is, but our API documentation is here https://docs.openstack.org/api-ref/baremetal/ | 16:57 |
| *** thiagop has joined #openstack-ironic | 16:58 | |
| TheJulia | the logs don't seem to jive either | 16:59 |
| TheJulia | hmm | 16:59 |
| TheJulia | o/ lmcgann | 16:59 |
| JayF | Yeah; I mean the only way IPA could cause that failure would be if it didn't complete writing the image before returning success to Ironic | 17:00 |
| JayF | which seems borderline-impossible | 17:00 |
| lmcgann | my bad https://keylime.dev/ | 17:00 |
| TheJulia | lmcgann: define integrate is likely a good start. larsks had a initial proposal of wrap ironic's launch by using keylime to start ironic, but then the downside of that would be is that we would never identify failed boots or failed hardware, things would just hang for operators introducing another reason to have to look at the console | 17:00 |
| lmcgann | I don't quite know either. I am new to both of these projects and am trying to get my bearings right now | 17:01 |
| TheJulia | lmcgann: ahh, welcome then! | 17:01 |
| openstackgerrit | Merged openstack/sushy stable/ussuri: Do not log passwords and auth tokens when using SessionService https://review.opendev.org/747956 | 17:02 |
| TheJulia | JayF: was this failing before metalsmith switched to direct? | 17:02 |
| JayF | metalsmith job succeeded on this patch 8/22 | 17:03 |
| patchbot | No data found for patch 8 | 17:03 |
| JayF | which I think was pre the direct change | 17:03 |
| TheJulia | JayF: hmmmmmmm | 17:04 |
| JayF | (and 8/23, it seems) | 17:04 |
| TheJulia | lmcgann: well, how can we best help provide you bearings ? | 17:04 |
| lmcgann | I don't even know what I don't know yet :) I think I just need a more fundamental understanding of ironic and keylime to begin asking the right questions. | 17:06 |
| JayF | Perhaps starting with reading the OpenStack contributor docs, going through that setup, and getting Ironic setup in devstack may help you learn the moving parts | 17:06 |
| *** mkrai has quit IRC | 17:07 | |
| lmcgann | Is this something that has any work done for it already? | 17:08 |
| lmcgann | I may have jumped the gun and came here a bit early, I apologize | 17:08 |
| JayF | Saying hello is always welcome :) | 17:10 |
| JayF | TheJulia: My hunch is to recheck it for that job failure, would you disagree? | 17:10 |
| TheJulia | lmcgann: I wrote a security spec a while back that made not traction that might be an integraiton point, let me find you that link | 17:11 |
| TheJulia | lmcgann: no, your fine :) | 17:11 |
| TheJulia | JayF: yeah, I'd give that a try | 17:11 |
| TheJulia | I'm kind of at a loss why the job failed, tbh | 17:11 |
| TheJulia | well | 17:11 |
| TheJulia | I know why, but not how it got to that state | 17:11 |
| TheJulia | lmcgann: give me a minute to finish cooking a snack and I'll dig up that link and send it on over | 17:11 |
| JayF | the "why" being "bad OS on disk"? | 17:11 |
| JayF | or did you get something further than that? | 17:12 |
| lmcgann | no rush | 17:12 |
| *** derekh has quit IRC | 17:18 | |
| TheJulia | JayF: maybe | 17:21 |
| TheJulia | the incontinuity between the the logs and the results is remarkable though | 17:21 |
| TheJulia | incongruity | 17:21 |
| TheJulia | lmcgann: so, https://review.opendev.org/#/c/576718/ | 17:22 |
| patchbot | patch 576718 - ironic-specs - Add security interface spec - 3 patch sets | 17:22 |
| TheJulia | which amazingly enough had +1s | 17:22 |
| TheJulia | lmcgann: would it be helpful for me to kind of walk through how ironic sees itself in the universe and how it is leveraged? | 17:22 |
| TheJulia | hmm... new dev vm seems... frozen :( | 17:23 |
| openstackgerrit | Vishal Manchanda proposed openstack/ironic-ui master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744708 | 17:23 |
| lmcgann | TheJulia: thank you. I have a meeting right now but I will look at this in a bit | 17:27 |
| TheJulia | lmcgann: Okay. I'm in us-pacific if none of your cohorts have mentioned it, and I am meeting free the rest of the day aside from a inspector checking out some electrical work that was done at my place between 1-2 this afternoon | 17:28 |
| openstackgerrit | Merged openstack/ironic stable/ussuri: Ansible deploy - Ignore invalid devices https://review.opendev.org/747576 | 17:32 |
| *** derekh has joined #openstack-ironic | 17:33 | |
| *** dsneddon has joined #openstack-ironic | 17:36 | |
| *** bnemec has quit IRC | 17:50 | |
| *** derekh has quit IRC | 17:52 | |
| *** bnemec has joined #openstack-ironic | 17:53 | |
| *** k_mouza has joined #openstack-ironic | 17:53 | |
| *** k_mouza has quit IRC | 18:02 | |
| TheJulia | *sigh* | 18:11 |
| ajya | TheJulia: a quick question, did you have a chance to look at proposed solution for storing configuration data securely for import/export configuration steps? Would it be possible to get high level OK/NOK? | 18:14 |
| * TheJulia grumbles about queens branch | 18:14 | |
| TheJulia | ajya: I have not had time to get back to that spec. I did see chatter about a translation table to link to files on disk which is worrisome and also doesn't really scale. I guess I don't understand why we're not using other services and storing pointers/references to the files, or maybe the conundrum is where we have the dividing line | 18:16 |
| openstackgerrit | Aija Jaunteva proposed openstack/ironic-specs master: [WIP] System configuration within whole clean or deploy step https://review.opendev.org/740721 | 18:20 |
| ajya | TheJulia: storing pointers/references where? | 18:21 |
| TheJulia | just chatter I saw in irc | 18:21 |
| TheJulia | well | 18:22 |
| ajya | idea being that table link are the pointers. The files themselves are stored externally | 18:22 |
| TheJulia | pointers to something in glance that the user supplied | 18:22 |
| TheJulia | local file storage is not a great idea, and is actually unusable in the metal3 case | 18:22 |
| TheJulia | yeah | 18:22 |
| openstackgerrit | Ruby Loo proposed openstack/ironic master: driver_internal_info in provision notifications https://review.opendev.org/747996 | 18:22 |
| ajya | TheJulia: initial version of that was taken from virtual media boot, but, ok, virtual media boot has different requirements and users don't create/touch those files, so this wasn't good fit | 18:25 |
| ajya | Isn't glance only for images? Not custom files? | 18:26 |
| * TheJulia the files for virtual media boot get rebuilt | 18:26 | |
| TheJulia | err | 18:26 |
| TheJulia | the files for virtual media get/can be rebuilt | 18:26 |
| TheJulia | configuration at a point in time is just that | 18:26 |
| TheJulia | So I'm not entirely sure, and we ideally don't want to hardcode something on only using glance, so either in a central database and be accessible/downloadable, or the client's responsibility becomes to handle the push/pull of the file | 18:28 |
| * TheJulia does a got py2 unit tests working dance | 18:28 | |
| ajya | TheJulia: is there similar functionality already somewhere that I can take a look at? | 18:36 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/queens: Fix for failure in cleaning https://review.opendev.org/747288 | 18:37 |
| TheJulia | in which regards? | 18:37 |
| TheJulia | or what specifically? I ask because that is a very broad question | 18:38 |
| TheJulia | realistically, how large are the files we're talking about? | 18:38 |
| ajya | to store custom user files securely | 18:38 |
| ajya | for idrac could be 500KB max, let's say 1MB | 18:38 |
| ajya | to add some buffer | 18:38 |
| TheJulia | why don't we just make a database table that stores the file blobs in a field? | 18:39 |
| TheJulia | that allows the conductor to survive failure conditions | 18:39 |
| ajya | if that's acceptable, I don't mind :) | 18:39 |
| TheJulia | if an API user can download the file, then metal3 migh tbe able to support it. Keep in mind in the metal3 case their mysql is ephemeral | 18:39 |
| TheJulia | it is kind of what we do with configdrive | 18:40 |
| ajya | ok, I'll look into it | 18:40 |
| TheJulia | just as a separate table would be ideal since it keeps things cleanly separated | 18:40 |
| ajya | then it could be new DB table and API to List, Create, Update, Delete that data? | 18:40 |
| TheJulia | likely, it is fine if there are reasonable restrictions around that | 18:41 |
| ajya | restrictions based on owner? | 18:41 |
| TheJulia | owner/project could be a concern, y es | 18:41 |
| TheJulia | yes | 18:41 |
| TheJulia | possibly, but I was thinking more "does it make sense to permit an API consumer to update it? | 18:42 |
| TheJulia | or update an etry | 18:42 |
| TheJulia | or is it upload/download and list only | 18:42 |
| TheJulia | I guess that comes down to how would an API user use it | 18:42 |
| openstackgerrit | Julia Kreger proposed openstack/ironic stable/queens: Retries and timeout for IPA command https://review.opendev.org/747271 | 18:43 |
| TheJulia | JayF: somehow I lost an underscore, but after fighting a centos7 vm today, I finally have an environment where I can run the unit tests locally \o/ | 18:44 |
| ajya | ok, I'll detail this in the spec and then can discuss if some improvements necessary. Thanks TheJulia | 18:44 |
| TheJulia | ajya: awesome, thanks! | 18:44 |
| TheJulia | sorry to be so troublesome, there are just many considerations to keep in mind | 18:44 |
| ajya | no worries, learned something new | 18:45 |
| *** sri_ has quit IRC | 18:48 | |
| *** sri_ has joined #openstack-ironic | 18:49 | |
| *** pas-ha has quit IRC | 18:50 | |
| *** ildikov has quit IRC | 18:50 | |
| *** ildikov has joined #openstack-ironic | 18:53 | |
| *** pas-ha has joined #openstack-ironic | 18:53 | |
| JayF | TheJulia: metalsmith failed in an identical way. | 18:57 |
| JayF | TheJulia: suggestions for next steps? I'm tempted to see if the CI passes on an essentially noop change | 18:57 |
| * JayF suspects the change-metalsmith-to-direct change is the breaker | 18:57 | |
| * iurygregory hides because he put +2 on the change | 18:59 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: DO NOT MERGE: testing metalsmith job https://review.opendev.org/748015 | 18:59 |
| JayF | eh, tbh it's kinda messed up if it's only one-way gated | 18:59 |
| JayF | but we'll see, my change is eventlet related, and I don't disallow the possibility of crazy crap happening, so I'm just running this to validate | 19:00 |
| iurygregory | you could also test the metalsmith on ipa adding a job and updating the config for it | 19:01 |
| iurygregory | I can provide guidance if you want to set a job to iscsi to see both details | 19:01 |
| iurygregory | results* | 19:01 |
| JayF | TheJulia: interesting. in one of the failed metalsmith jobs, it looks like the node is rebooted while it's downloading image: https://zuul.opendev.org/t/openstack/build/cbdd365ed34649b68bce9aa0a9634196/log/controller/ironic-bm-logs/node-0_no_ansi_2020-08-25-15:28:35.log | 19:10 |
| TheJulia | JayF: that makes more sense :\ | 19:12 |
| JayF | but how is it possible that's happening? | 19:12 |
| JayF | that itself does not make sesne | 19:12 |
| JayF | *sense | 19:12 |
| JayF | unless a deploy is getitng cancelled halfway or something | 19:12 |
| JayF | I am not familiar with metalsmith at all, I don't even know what it does, but if it was randomly rebooting nodes that seems a little nutty | 19:13 |
| JayF | and bluntly it's hard to imagine my change causing this behavior, and not *also* seeing failures in the other integration jobs | 19:14 |
| JayF | I'm tempted to just completely punt on that until I see if metalsmith fails on my noop change | 19:18 |
| JayF | but even if it doesn't; I'm not sure I can ID what's different about the metalsmith job to cause it to fail with my changes | 19:18 |
| TheJulia | Yeah, I'm mentally at the same place. Kind of confused why and not sure what exactly is occuring | 19:24 |
| JayF | I mean, changing eventlet things + causing $weirdStuff to happen isn't a hard set of dots to connect. | 19:25 |
| JayF | I just have no idea whatsoever what'd be different about the metalsmith job. | 19:25 |
| JayF | I don't even know what metalsmith /does/. | 19:25 |
| TheJulia | simple scheduling/node matching for deployment basically | 19:26 |
| jroll | metalsmith is like nova but if it was made for bare metal and ran on the user machine | 19:26 |
| JayF | well right now my | 19:26 |
| JayF | *their CI is breaking me, so I hate it /s | 19:26 |
| TheJulia | lol | 19:26 |
| JayF | lol | 19:26 |
| iurygregory | poor JayF =( | 19:27 |
| TheJulia | lets see what the noop change says | 19:27 |
| *** ijw_ has quit IRC | 19:27 | |
| JayF | iurygregory: tbh I only find it this obnoxious because I didn't realize how dusty my upstream-ci-troubleshooting knowledge had become in 2-3 years away | 19:27 |
| *** ijw has joined #openstack-ironic | 19:28 | |
| iurygregory | JayF, one advice I can give you... don't look at grenade jobs=) | 19:28 |
| JayF | that is one thing that has /not/ changed over the years | 19:29 |
| JayF | you just have to make sure to get really quiet anytime someone talks about upgrade testing, to avoid accidentally volunteering yourself :P | 19:29 |
| iurygregory | hehehe well grenade works <3 | 19:29 |
| iurygregory | but don't ask about multinode =D | 19:29 |
| jroll | TIL page up/down doesn't work on these zuul logs :| | 19:30 |
| iurygregory | this would be a conversation where you would need some alcohol | 19:30 |
| * TheJulia looks at clock, and determines it is too early for that... | 19:31 | |
| TheJulia | at least, for me | 19:31 |
| iurygregory | hehe | 19:31 |
| JayF | I hvae always thought the worst jobs in tech invovled CI, QE, or ops | 19:31 |
| JayF | little control, high visibility, high crunch | 19:31 |
| iurygregory | hehehe | 19:32 |
| *** ijw has quit IRC | 19:32 | |
| * TheJulia suspects tonight is a margharita night | 19:32 | |
| *** ijw has joined #openstack-ironic | 19:34 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Update the cache if we don't have a root device hint https://review.opendev.org/747072 | 19:34 |
| *** ijw has quit IRC | 19:39 | |
| clarkb | jroll: its a bug with the patternfly 4 upgrade | 19:40 |
| jroll | gotcha | 19:41 |
| clarkb | there are a couple different hacks to fix it proposed but I think people are struggling with which is the least hacky fix (or maybe one landed and we just have to restart the container /me double checks) | 19:41 |
| jroll | thanks. I'll take reporting that off my list :) | 19:41 |
| clarkb | jroll: it has to do with how the error window fly out thing grabs "focus" and then never gives it back aiui | 19:41 |
| clarkb | https://review.opendev.org/#/c/743917/ oh that has some new info in it and it is navbar not drawer related | 19:43 |
| patchbot | patch 743917 - zuul/zuul - PF4: Use a ref to focus main content on initial pa... - 2 patch sets | 19:43 |
| larsks | TheJulia: do you know who's been working with ironic in the context of the openshift baremetal installer? | 20:00 |
| *** ijw has joined #openstack-ironic | 20:06 | |
| *** JamesBenson has quit IRC | 20:10 | |
| *** ijw has quit IRC | 20:11 | |
| *** k_mouza has joined #openstack-ironic | 20:13 | |
| *** janders4 has joined #openstack-ironic | 20:18 | |
| *** k_mouza has quit IRC | 20:18 | |
| *** janders has quit IRC | 20:19 | |
| *** janders4 is now known as janders | 20:19 | |
| *** Lucas_Gray has joined #openstack-ironic | 20:21 | |
| *** zaneb has quit IRC | 20:21 | |
| JayF | TheJulia: metalsmith job passes on a noop change :C | 20:24 |
| *** thiagop has quit IRC | 20:32 | |
| *** thiagop has joined #openstack-ironic | 20:36 | |
| *** ijw has joined #openstack-ironic | 20:37 | |
| TheJulia | larsks: a number of people, is there a specific area your looking at that I can point you towards? | 20:38 |
| TheJulia | JayF: got a link to the logs? | 20:39 |
| larsks | TheJulia: maybe! It looks like inspection is running multiple times, resulting in the installer failing ("Port ac:1f:6b:f5:3e:f2 already exists..."). | 20:39 |
| JayF | https://56506052d281c5f34ae9-d0226936c9cadcaed5032c3f3c093f07.ssl.cf5.rackcdn.com/748015/1/check/metalsmith-integration-ipa-src/0926c27/ is the passing one | 20:40 |
| larsks | Wondering who might be familiar with the orchestration of things at this stage of the installer. | 20:40 |
| JayF | I'm digging for differences | 20:40 |
| JayF | they're both rackcdn, so "CI shenanigans based on datacenter/provider" seems unlikely | 20:40 |
| JayF | it's using tinyipa, as are many other jobs, so "not building ipa the same way" seems unlikely | 20:40 |
| TheJulia | larsks: so that uses a mode called fast track so it can go through discovery and there is explicit introspection in certian cases/versions | 20:41 |
| JayF | I'm afraid it may be actually exposing a bug, but without knowing if the metalsmith job is doing something notably different, I can't really isolate it | 20:41 |
| JayF | https://zuul.opendev.org/t/openstack/build/dd4067f4fa5344c6ac0c5c47d4ec6f88 is a failing example; the most recent one | 20:41 |
| larsks | TheJulia: Since it's starting with a an empty database (ironic runs in the "bootstrap vm" which is ephemeral and exists only for the duration of the installer), an error like this seems to be a symptom of something else going wrong earlier in the install. | 20:42 |
| *** ijw has quit IRC | 20:42 | |
| TheJulia | larsks: where exactly are you seeing the error and what are you asking it to do in terms of the deployment? | 20:43 |
| *** ijw has joined #openstack-ironic | 20:43 | |
| *** lmcgann has quit IRC | 20:43 | |
| larsks | TheJulia: the error is cropping up while running a baremetal install. I'm not asking it to do anything :). I'm just trying to figure out if this is something someone has seen before and might know what's causing it. That's why I'm hoping to find someone who's worked on the use of ironic by the installer... | 20:44 |
| TheJulia | so, your likely best bet is someone on the openshift baremetal installer team | 20:45 |
| TheJulia | dtantsur|afk might be a good person to talk to when he is around tomorrow | 20:45 |
| larsks | Awesome, thanks. | 20:46 |
| TheJulia | i think I've seen it, but there are version specific changes between when we started on that path and now | 20:46 |
| JayF | TheJulia: hmm, AFAICT, the metalsmith job is using a CentOS7 image (for the image), whereas all of our other jobs are using cirros | 20:50 |
| JayF | that's the only meaningful difference I can identify | 20:50 |
| TheJulia | hmmm | 20:51 |
| JayF | also IRONIC_VM_COUNT=1 instead of IRONIC_VM_COUNT=2, but I don't believe that should matter | 20:52 |
| JayF | although I'm surprised at that difference | 20:52 |
| TheJulia | nope... redfish or ipmi? | 20:52 |
| JayF | ipmi | 20:53 |
| JayF | I have a hunch I'm going to dig: if the http[] streaming URL for the centos image is https, while the cirros one is http | 20:53 |
| JayF | that'd be a meaningful difference that could be impacted by my changes | 20:53 |
| JayF | nah, I don't think so, because they're both coming from glance | 20:54 |
| TheJulia | you know, I did notice the metalsmith job is very much https | 20:54 |
| JayF | > 2020-08-25 18:16:41.411 595 INFO ironic_python_agent.extensions.standby [-] Attempting to download image from http://10.209.64.38:3928/agent_images/052857ee-027c-4eb9-a23e-2407ca75a9f5[00m | 20:56 |
| JayF | so it's not downloading the image http | 20:56 |
| JayF | > ipa-api-url=https://10.209.64.38/baremetal | 20:56 |
| JayF | ironic is https though, checking a passing job to see if it's https or http | 20:57 |
| JayF | > ipa-api-url=https://10.176.194.171/baremetal passing job has an https ironic as well | 20:58 |
| JayF | (ipa-tempest-bios-ipmi-direct-src job passing on my change) | 20:58 |
| *** thiagop has quit IRC | 21:09 | |
| JayF | TheJulia: either we're missing logs from the "broken" agent, or it's freezing solid when asked to download the image | 21:10 |
| JayF | I just can't nail down *what is different* to figure out why it's freezing up there | 21:11 |
| JayF | (versus the other jobs) | 21:11 |
| TheJulia | hmmm | 21:11 |
| TheJulia | freezing solid would explain some of it | 21:11 |
| JayF | I mean, that's the only explanation, and it's a "common" failure mode for eventlet stuff | 21:12 |
| TheJulia | :( | 21:15 |
| JayF | the size of the image has to be the answer, right? | 21:15 |
| JayF | ngl, I'm tempted to throw an `os=False` into the eventlet monkey patch, since that's what's in Ironic | 21:16 |
| JayF | I just wish I had an idea of /why/ | 21:16 |
| openstackgerrit | Steve Baker proposed openstack/metalsmith master: Capture metalsmith python logging https://review.opendev.org/748031 | 21:18 |
| TheJulia | stevebaker: o/ | 21:18 |
| stevebaker | TheJulia: hey | 21:18 |
| TheJulia | JayF: might work | 21:18 |
| TheJulia | worth a try! | 21:18 |
| JayF | I mean, don't like making changes without understanding what's going on... | 21:19 |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 21:23 |
| TheJulia | Same :( | 21:24 |
| JayF | looking at what eventlet patches, it's all the os.read/fdread etc stuff which would be used in our ImageDownloader class | 21:25 |
| JayF | which is where the deadlock appears to happen | 21:25 |
| JayF | so I gave it a shot | 21:25 |
| openstackgerrit | Merged openstack/ironic master: Decouple the ISO creation logic from redfish https://review.opendev.org/739174 | 21:34 |
| openstackgerrit | Steve Baker proposed openstack/metalsmith master: Capture metalsmith python logging https://review.opendev.org/748031 | 21:40 |
| openstackgerrit | Steve Baker proposed openstack/metalsmith master: Document metalsmith_instances https://review.opendev.org/738131 | 21:40 |
| openstackgerrit | Steve Baker proposed openstack/metalsmith master: metalsmith_deployment role switch to metalsmith_instances https://review.opendev.org/737888 | 21:40 |
| *** rcernin has joined #openstack-ironic | 22:00 | |
| *** rcernin has quit IRC | 22:01 | |
| *** rcernin has joined #openstack-ironic | 22:02 | |
| *** JamesBenson has joined #openstack-ironic | 22:03 | |
| *** bnemec has quit IRC | 22:14 | |
| *** JamesBenson has quit IRC | 22:17 | |
| TheJulia | JayF: hmm... | 22:17 |
| JayF | did you find something compelling? | 22:18 |
| *** rh-jelabarre has quit IRC | 22:21 | |
| *** rh-jelabarre has joined #openstack-ironic | 22:21 | |
| *** zaneb has joined #openstack-ironic | 22:22 | |
| janders | good morning Ironic o/ | 22:26 |
| *** livelace has quit IRC | 22:36 | |
| TheJulia | JayF: just interesting that it blows up there | 22:36 |
| TheJulia | I'm working on yet another uefi bootloader fix | 22:36 |
| JayF | "blows up" would be a better failure case | 22:52 |
| JayF | "goes out for a cheeseburger and never returns" is more what we're dealing with :C | 22:52 |
| TheJulia | that does sound like eventlet sadly | 22:52 |
| TheJulia | if only ipa could log "MMMM CHEEZBURGR | 22:53 |
| TheJulia | " | 22:53 |
| JayF | I've never cheered more for a job to fail than when I did the clean test | 22:53 |
| JayF | lol | 22:53 |
| JayF | hmm. You know what, I was about to say, the monkey_patch didn't break it, because it passed before I updated requirements/constraints | 22:53 |
| JayF | BUT that was also simultaneous with the change to it doing direct | 22:54 |
| JayF | so that metalsmith job might have saved us a metric crapton of WTF-troubleshooting when victoria got released | 22:54 |
| JayF | and people started using IPA with real-world sized images | 22:54 |
| TheJulia | yup | 22:54 |
| JayF | I hope I'm about to get to the bottom of it though. Frankly, enabling SSL causing the whole thing to cease up to me is a sign we had some other, preexisting bug in IPA | 22:55 |
| JayF | and hopefully this is making progress to fixing it. | 22:55 |
| *** JamesBenson has joined #openstack-ironic | 22:56 | |
| TheJulia | well, we're trying to stream through it with the read/write on raw transfers which is what I believe the metalsmith ends up running | 22:57 |
| JayF | which is why I'm hoping the os=False has a positive impact | 22:58 |
| TheJulia | the cirros image might not be, or as you suggested, it just may be the size | 22:58 |
| JayF | it's been about 1:30, and I'm about to turn into a pumpkin, so I'm going to check zuul status to see if there's a result on that job | 22:58 |
| TheJulia | anything on the current zuul job? | 22:58 |
| JayF | https://7a08a3030f2403a359fd-4034ad4bc806fe95a8503cbe82cd73fb.ssl.cf2.rackcdn.com/746774/10/check/metalsmith-integration-ipa-src/894a516/ fao;ire | 22:58 |
| JayF | *failure | 22:58 |
| JayF | appears to still deadlock in the same location | 22:59 |
| JayF | I'm tempted to go the other direction -- test locally to see what `module=True` I need to pass to monkey patch to make SSL work, and just go that way | 22:59 |
| TheJulia | :( | 22:59 |
| JayF | but I think that's just dodging the bug instead of solving it | 22:59 |
| TheJulia | socket, realistically | 22:59 |
| TheJulia | I hope it is not socket that is causing this | 22:59 |
| JayF | socket is what I'm going to have to patch to make ssl work, almost certainly | 23:00 |
| TheJulia | yeah | 23:00 |
| *** JamesBenson has quit IRC | 23:01 | |
| clarkb | is IPA doing any other work during the image download? could it just be synchronous? | 23:02 |
| JayF | IPA is always running a heartbeater in the background | 23:02 |
| JayF | regardless of what else it's doing, it heartbeats every "n" seconds | 23:03 |
| JayF | to rewind all the way back to the original symptom I'm trying to address: when I enable use_ssl in the oslo.service wsgi server (like is done here: https://review.opendev.org/#/c/747193/ ) the API does not send a response. | 23:03 |
| patchbot | patch 747193 - ironic-python-agent - If listen_tls is true, enable TLS on wsgi server - 5 patch sets | 23:03 |
| *** tosky has quit IRC | 23:04 | |
| JayF | I found that putting an `eventlet.sleep(0)` deep in the eventlet wsgi server code (I could find exactly what line, if you're interested), fixes that behavior | 23:04 |
| JayF | also, monkey_patching earlier in IPA, as I do here https://review.opendev.org/#/c/746774/, also resolved that symptom -- but appears to have a giant amount of knock-on effects upon testing. | 23:04 |
| patchbot | patch 746774 - ironic-python-agent - Eventlet should be monkey patched as early as poss... - 10 patch sets | 23:04 |
| JayF | tl;dr: All I wanted was a little SSL, and I'm in eventlet hell | 23:05 |
| clarkb | I'm mostly asking because download 500MB image isn't going to go any faster because you're using greenthreads and not "real" threads | 23:05 |
| JayF | the main reason it's streamed is due to memory pressure, not for speed | 23:05 |
| clarkb | but if other stuff is going on then untangling a synchronous thread from the other stuff could be pain | 23:06 |
| TheJulia | yeah, people have a desire to try and stream gigabytes through megabytes of ram | 23:06 |
| TheJulia | the heartbeat uses select... I wonder if that is where thigns are deadlocking | 23:06 |
| JayF | that's my suspicion, but I'm not sure how else to achieve our goals there without using a select | 23:07 |
| JayF | and honestly, a select.select(timeout) should *not* be deadlocking, should it? with an eventlet-patched select? | 23:07 |
| JayF | I guess I should read that code. | 23:07 |
| TheJulia | yeah, I should have brain cells that I can help look at this tomorrow | 23:08 |
| JayF | Maybe even see how a naive heartbeater (using time-based heartbeats, like we used to) would do under test | 23:08 |
| TheJulia | I did a bunch of research today on the image elements being used so I should be able to speak authoratively to my other headache tomorrow | 23:08 |
| JayF | that's obviously not the solution, but it'd be a clear pointer to what is deadlocking | 23:08 |
| TheJulia | yeah | 23:10 |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: WIP: Support partition image efi contents https://review.opendev.org/748049 | 23:15 |
| TheJulia | iurygregory: Please take a look at ^^^ tomorrow for me. I basically think we need to be able to support partition images with efi contents embedded | 23:15 |
| JayF | I wonder if the solution to this is more along the lines of, still use a select to trigger heartbeats, but don't have it on such a large timeout, and measure time for the periodic heartbeats some other way | 23:16 |
| JayF | e.g. a secondary thread which feeds a forced_heartbeat in on [interval] | 23:16 |
| JayF | IDK; I'm going to turn into a pumpkin now, maybe the solution will come to me in a dream | 23:16 |
| TheJulia | JayF: pumpkin++ | 23:16 |
| TheJulia | goodnight! | 23:16 |
| TheJulia | speaking of becoming a pumpkin, I think I'm going to do the same | 23:17 |
| JayF | have a good one o/ | 23:17 |
| TheJulia | you too! | 23:18 |
| *** JamesBenson has joined #openstack-ironic | 23:37 | |
| *** JamesBenson has quit IRC | 23:42 | |
| *** uzumaki has joined #openstack-ironic | 23:46 | |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!