| TheJulia | bdodd: Question for you. I'm looking at a report of issues with reauthenticating sessions with a redfish bmc, and wondering if you've heard of or are aware of any issues | 00:12 |
|---|---|---|
| bdodd | TheJulia I'm not aware of any session re-authentication issues. | 00:14 |
| TheJulia | bdodd: ack, I looked through the code and the only thing I can think is that the bmc is just flat out rejecting the new session as well | 00:15 |
| TheJulia | I need to add more debug logging or something though | 00:16 |
| bdodd | Could be. I'm just taking a look at what the spec says for session lifetime, etc. | 00:16 |
| TheJulia | I think the bmc in question ships with a default super-low session lifetime in the range of minutes | 00:18 |
| bdodd | TheJulia It looks like the sessions terminates if there is no activity for longer than the session timeout period. | 00:18 |
| TheJulia | And that is kind of what looksl ike is happening, and then another forbidden seems to get logged in that entire sequence of events to retry | 00:19 |
| bdodd | If it does time out the session is terminated. A new session would need to be created in that case. | 00:19 |
| TheJulia | Yeah, unfortunately our logging in all of that is extremely sparse | 00:20 |
| TheJulia | read: empty | 00:20 |
| *** rloo has quit IRC | 00:24 | |
| bdodd | TheJulia That's too bad. Sounds like more debug logging is the next best step. | 00:24 |
| TheJulia | Yeah :( | 00:26 |
| *** janders has quit IRC | 00:28 | |
| *** janders has joined #openstack-ironic | 00:42 | |
| openstackgerrit | Steve Baker proposed openstack/ironic master: New argument validate decorator https://review.opendev.org/742795 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Add status_code argument to @method.expose https://review.opendev.org/743677 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Add @method.body decorator https://review.opendev.org/743678 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Duplicate trait validation with jsonschema https://review.opendev.org/745778 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert allocation endpoint to plain JSON https://review.opendev.org/743679 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert bios endpoint to plain JSON https://review.opendev.org/745475 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert chassis endpoint to plain JSON https://review.opendev.org/745476 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert conductor endpoint to plain JSON https://review.opendev.org/745779 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert deploy_templates endpoint to plain JSON https://review.opendev.org/745780 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert drivers endpoint to plain JSON https://review.opendev.org/741865 | 00:55 |
| openstackgerrit | Steve Baker proposed openstack/ironic master: Convert event endpoint to plain JSON https://review.opendev.org/746623 | 00:55 |
| stevebaker | spam alert | 00:55 |
| *** ricolin has quit IRC | 01:17 | |
| *** alexmcleod has quit IRC | 01:24 | |
| *** alexmcleod has joined #openstack-ironic | 01:25 | |
| *** cdearborn has quit IRC | 01:29 | |
| *** gyee has quit IRC | 01:49 | |
| *** tzumainn has quit IRC | 01:52 | |
| *** ijw has quit IRC | 02:16 | |
| *** ijw has joined #openstack-ironic | 02:25 | |
| *** mkrai has joined #openstack-ironic | 02:25 | |
| *** mkowalski has quit IRC | 03:29 | |
| *** ricolin has joined #openstack-ironic | 03:33 | |
| *** mkowalski has joined #openstack-ironic | 03:36 | |
| openstackgerrit | Xinliang Liu proposed openstack/ironic master: Add uefi pxe iscsi boot https://review.opendev.org/746628 | 03:37 |
| openstackgerrit | Xinliang Liu proposed openstack/ironic master: Add uefi pxe iscsi boot https://review.opendev.org/746628 | 03:39 |
| *** xinliang has joined #openstack-ironic | 03:41 | |
| *** mkrai has quit IRC | 03:47 | |
| *** mkrai has joined #openstack-ironic | 03:55 | |
| *** xinliang has quit IRC | 04:04 | |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 04:27 |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 04:49 |
| *** gudrutis has joined #openstack-ironic | 04:50 | |
| *** gudrutis0 has quit IRC | 04:52 | |
| *** strigazi has quit IRC | 04:52 | |
| *** strigazi has joined #openstack-ironic | 04:54 | |
| *** gudrutis9 has joined #openstack-ironic | 05:08 | |
| *** abdysn has joined #openstack-ironic | 05:11 | |
| *** strigazi has quit IRC | 05:12 | |
| *** gudrutis has quit IRC | 05:12 | |
| *** gudrutis9 is now known as gudrutis | 05:12 | |
| *** janders0 has joined #openstack-ironic | 05:21 | |
| *** janders has quit IRC | 05:21 | |
| *** janders has joined #openstack-ironic | 05:21 | |
| *** gudrutis5 has joined #openstack-ironic | 05:23 | |
| *** gudrutis has quit IRC | 05:23 | |
| *** gudrutis5 is now known as gudrutis | 05:23 | |
| *** ijw has quit IRC | 05:25 | |
| *** ijw has joined #openstack-ironic | 05:25 | |
| *** ijw has quit IRC | 05:30 | |
| *** ijw has joined #openstack-ironic | 05:33 | |
| *** ijw has quit IRC | 05:37 | |
| *** ociuhandu has joined #openstack-ironic | 05:43 | |
| *** ociuhandu has quit IRC | 05:47 | |
| *** gudrutis has quit IRC | 05:52 | |
| *** belmoreira has joined #openstack-ironic | 05:56 | |
| *** gudrutis has joined #openstack-ironic | 05:57 | |
| *** ijw has joined #openstack-ironic | 06:04 | |
| *** ijw has quit IRC | 06:09 | |
| *** xinliang has joined #openstack-ironic | 06:25 | |
| *** ijw has joined #openstack-ironic | 06:36 | |
| *** ijw has quit IRC | 06:40 | |
| openstackgerrit | Nisha Agarwal proposed openstack/ironic master: Adds few of the security dashboard parameters to capabilities https://review.opendev.org/746416 | 06:42 |
| *** rcernin has quit IRC | 07:02 | |
| *** rcernin has joined #openstack-ironic | 07:03 | |
| *** ijw has joined #openstack-ironic | 07:07 | |
| *** ijw has quit IRC | 07:11 | |
| *** ijw has joined #openstack-ironic | 07:38 | |
| *** ijw has quit IRC | 07:43 | |
| rpittau | good morning ironic! o/ | 07:56 |
| gudrutis | hello ironic | 07:57 |
| iurygregory | good morning rpittau gudrutis and Ironic o/ | 07:58 |
| janders | good morning rpittau | 07:58 |
| janders | o/ | 07:58 |
| janders | hey iurygregory | 07:58 |
| iurygregory | hey janders o/ | 07:58 |
| rpittau | hey iurygregory janders gudrutis :) | 07:58 |
| *** ociuhandu has joined #openstack-ironic | 08:06 | |
| *** ijw has joined #openstack-ironic | 08:09 | |
| *** ociuhandu has quit IRC | 08:11 | |
| *** lucasagomes has joined #openstack-ironic | 08:12 | |
| *** mkowalski has quit IRC | 08:12 | |
| *** Lucas_Gray has joined #openstack-ironic | 08:13 | |
| *** ijw has quit IRC | 08:14 | |
| *** mkowalski has joined #openstack-ironic | 08:21 | |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 08:24 |
| *** Qianbiao has joined #openstack-ironic | 08:32 | |
| *** rcernin has quit IRC | 08:34 | |
| *** priteau has joined #openstack-ironic | 08:35 | |
| *** ociuhandu has joined #openstack-ironic | 08:37 | |
| *** ijw has joined #openstack-ironic | 08:40 | |
| *** ijw has quit IRC | 08:45 | |
| *** k_mouza has joined #openstack-ironic | 08:55 | |
| iurygregory | It's funny that 3/4 jobs works with privsep on ironic-lib =( | 08:58 |
| * iurygregory cries | 08:58 | |
| janders | I've been looking through my clean-steps patch with iurygregory and this came up: https://review.opendev.org/#/c/744117/8/ironic/conf/conductor.py@276 | 08:59 |
| patchbot | patch 744117 - ironic - [WIP]Generic way to configure clean step priorites - 8 patch sets | 08:59 |
| janders | what do you think? Is it better to use None or empty dict as the default config value? | 09:00 |
| *** ijw has joined #openstack-ironic | 09:00 | |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 09:05 |
| openstackgerrit | Jacob Anders proposed openstack/ironic master: [WIP]Generic way to configure clean step priorites https://review.opendev.org/744117 | 09:07 |
| rpittau | janders: IMHO empty dict, in python they're evaluated as False anyway | 09:19 |
| janders | thanks rpittau :) | 09:19 |
| janders | you are right (I tested this earlier out of curiosity ) | 09:19 |
| rpittau | :) | 09:20 |
| iurygregory | =D | 09:20 |
| rpittau | iurygregory: I promise I'll get to your privsep patches today | 09:23 |
| iurygregory | rpittau, no worries I just have the feeling we will probably need something on ironic first... | 09:23 |
| iurygregory | since I'm getting "Could not link image ... " | 09:24 |
| *** mkrai_ has joined #openstack-ironic | 09:26 | |
| *** mkrai has quit IRC | 09:29 | |
| janders | i'm again trying to figure out how to use bifrost developer mode to fetch a specific gerrit patch | 09:33 |
| janders | do you guys have any experience with this? | 09:33 |
| *** ijw has quit IRC | 09:34 | |
| iurygregory | janders, I think you need to set the git folder of the repository and have source_install true | 09:37 |
| iurygregory | for example, by default ironic folder is on /opt/stack/ironic so you would need to apply the patch there and run bifrost | 09:39 |
| janders | is source_install a flag passed to the install step? | 09:40 |
| iurygregory | or maybe change the ironic_git_url... | 09:40 |
| iurygregory | https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-ironic-install/defaults/main.yml | 09:40 |
| janders | I tried what you described without any additional config and bifrost happily cloned the latest master over the top of my fetched patch | 09:40 |
| janders | so I think I need to get the options you referenced just right | 09:41 |
| iurygregory | source install https://opendev.org/openstack/bifrost/src/branch/master/playbooks/roles/bifrost-pip-install/defaults/main.yml | 09:41 |
| *** ijw has joined #openstack-ironic | 09:41 | |
| janders | this looks good | 09:42 |
| janders | iurygregory do you think if I change this and reinstall, this will make bifrost use ironic source in /opt/stack/ironic? | 09:43 |
| janders | or do I do this *and* set the git_url as well? | 09:43 |
| * janders is grepping for ironic_git_url | 09:43 | |
| iurygregory | I'm still trying to fully understand hehe | 09:44 |
| janders | I will try run a little test and report back | 09:44 |
| janders | I think this looks good: | 09:45 |
| janders | ./doc/source/install/offline-install.rst: ironic_git_url: file:///vagrant/git/ironic | 09:45 |
| janders | I think it could be used to point to a fetched patch | 09:45 |
| * janders is opening this doc in the web browser :) | 09:45 | |
| janders | https://opendev.org/openstack/bifrost/src/branch/master/doc/source/install/offline-install.rst | 09:47 |
| janders | maybe just a local ironic_git_url override will suffice | 09:47 |
| iurygregory | yeah | 09:47 |
| janders | I will try that... iurygregory thanks heaps for the pointer | 09:47 |
| iurygregory | fingers crossed here =) | 09:48 |
| janders | will report back | 09:48 |
| *** tkajinam has quit IRC | 09:58 | |
| *** janders0 has joined #openstack-ironic | 10:03 | |
| *** janders has quit IRC | 10:06 | |
| *** janders0 is now known as janders | 10:06 | |
| janders | not having a lot of luck so far | 10:07 |
| janders | my override got ignored | 10:07 |
| janders | I've got some more clues, still working on it | 10:08 |
| *** ijw has quit IRC | 10:14 | |
| iurygregory | ok | 10:16 |
| *** ociuhandu has quit IRC | 10:17 | |
| iurygregory | rpittau, do you have any idea on the failure in the privsep patch? | 10:17 |
| *** ociuhandu has joined #openstack-ironic | 10:18 | |
| iurygregory | the problem seems to me when we try to execute the os.link https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/image_cache.py#L168 | 10:18 |
| *** xinliang has quit IRC | 10:28 | |
| *** Qianbiao has quit IRC | 10:30 | |
| janders | iurygregory rpittau this seems promising | 10:30 |
| janders | http://paste.openstack.org/show/796903/ | 10:31 |
| janders | (context: getting bifrost to use a specific patch) | 10:31 |
| janders | there is probably a slicker way of doing this but hey this is a start | 10:31 |
| *** Qianbiao has joined #openstack-ironic | 10:31 | |
| janders | I haven't had any luck without disabling force_update_repos | 10:31 |
| *** Qianbiao has quit IRC | 10:32 | |
| janders | it was overriding my checkout anyway | 10:32 |
| *** Qianbiao has joined #openstack-ironic | 10:32 | |
| iurygregory | I'm wondering if you dont need the developer_mode set | 10:33 |
| iurygregory | doing this testenv worksfine with the changes? | 10:34 |
| janders | I'm just waiting for the second install to finish | 10:34 |
| janders | but first one had the code from my patch | 10:34 |
| iurygregory | cool | 10:35 |
| janders | (bifrost) [janders@host15 bifrost]$ grep CONF.conductor.clean_step_priority_overrides /opt/stack/bifrost/lib/python3.6/site-packages/ironic/conductor/steps.py | 10:37 |
| janders | if CONF.conductor.clean_step_priority_overrides: | 10:37 |
| janders | it's alive | 10:37 |
| janders | now - time for a quick test | 10:37 |
| iurygregory | alive?! | 10:37 |
| iurygregory | run! | 10:37 |
| iurygregory | sudo shutdown -P now | 10:37 |
| janders | haha :) | 10:38 |
| iurygregory | I couldn't resist sorry =) | 10:38 |
| janders | boom! | 10:40 |
| janders | didn't get through enrollment | 10:40 |
| janders | http://paste.openstack.org/show/796904/ | 10:40 |
| * janders is scratching his head | 10:41 | |
| * janders is rerunning with -vvv | 10:42 | |
| *** ijw has joined #openstack-ironic | 10:42 | |
| janders | http://paste.openstack.org/show/796905/ | 10:44 |
| janders | still puzzled | 10:44 |
| iurygregory | Authorization required is a bit strange | 10:44 |
| iurygregory | '-' | 10:44 |
| janders | I suppose I can rollback the patch and retry see if it's related, it shouldn't be... I haven't spotted anything interesting in the logs | 10:45 |
| janders | time to make late dinner - depending how long this takes the authorization problem might be tomorrow's problem | 10:45 |
| iurygregory | it shouldn't be related... | 10:46 |
| *** ijw has quit IRC | 10:47 | |
| *** mkrai_ has quit IRC | 11:05 | |
| *** Qianbiao has quit IRC | 11:06 | |
| *** ijw has joined #openstack-ironic | 11:14 | |
| *** ijw has quit IRC | 11:19 | |
| janders | Zuul is happy with the revisions of my patch - great | 11:28 |
| janders | I shall figure out what is not working in my lab setup tomorrow | 11:29 |
| janders | see you later Ironic o/ | 11:29 |
| *** Lucas_Gray has quit IRC | 11:32 | |
| *** ijw has joined #openstack-ironic | 11:45 | |
| *** ijw has quit IRC | 11:50 | |
| *** Qianbiao has joined #openstack-ironic | 12:00 | |
| *** ijw has joined #openstack-ironic | 12:16 | |
| *** zzzeek has quit IRC | 12:19 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: Use ubuntu focal for ipa src https://review.opendev.org/725794 | 12:20 |
| *** ijw has quit IRC | 12:21 | |
| rpittau | iurygregory: where are you seeing that error ? | 12:22 |
| iurygregory | rpittau, https://zuul.opendev.org/t/openstack/build/18b0f829505c474d8035206b934a4ec5/log/controller/logs/screen-ir-cond.txt | 12:22 |
| *** zzzeek has joined #openstack-ironic | 12:23 | |
| rpittau | iurygregory: ok, did you see my comment in the patch ? | 12:23 |
| iurygregory | rpittau, about having the 2 CAPS you mentioned? | 12:23 |
| rpittau | yeah | 12:23 |
| iurygregory | you think that would solve? | 12:23 |
| iurygregory | or "it's necessary" ? | 12:24 |
| rpittau | iurygregory: well that is the way to have the context as close as possible to sudo rights, so I would start from there anyway | 12:25 |
| iurygregory | rpittau, I had in the past, Julia was a bit worried since it was too open (at least I understood like that) | 12:27 |
| rpittau | iurygregory: this could be a little overwhelming, but can give a lot of knowledge in understanding how privsep works: https://man7.org/linux/man-pages/man7/capabilities.7.html | 12:27 |
| iurygregory | rpittau, yeah I have this page open | 12:28 |
| rpittau | ok, cool | 12:28 |
| iurygregory | https://review.opendev.org/#/c/745536/4..6/ironic_lib/privsep/__init__.py it was on patch set 4 | 12:28 |
| patchbot | patch 745536 - ironic-lib - Add initial privsep structure - 6 patch sets | 12:28 |
| rpittau | iurygregory: if you look at nova nd cinder implementation, they both adopt the caps I told you | 12:29 |
| iurygregory | rpittau, yup! I was looking when I started the implementation | 12:29 |
| iurygregory | thats why my initial patch had hehe | 12:30 |
| rpittau | consider that qemu and dd require kind of high privileges | 12:31 |
| iurygregory | the error seems to be because we executed qemu with privilege and os.link is not executed with privilege... | 12:34 |
| iurygregory | th interesting part is that it only happened in one job | 12:34 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-lib master: Add initial privsep structure https://review.opendev.org/745536 | 12:35 |
| rpittau | I wonder why we hard link instead of soft | 12:37 |
| *** Lucas_Gray has joined #openstack-ironic | 12:42 | |
| rpittau | mmm ok maybe because otherwise the cache doesn't get updated | 12:43 |
| *** ijw has joined #openstack-ironic | 12:47 | |
| *** Goneri has joined #openstack-ironic | 12:50 | |
| *** tzumainn has joined #openstack-ironic | 12:52 | |
| *** ijw has quit IRC | 12:52 | |
| openstackgerrit | Vishal Manchanda proposed openstack/ironic-ui master: [goal] Migrate testing to ubuntu focal https://review.opendev.org/744708 | 12:52 |
| *** rloo has joined #openstack-ironic | 12:54 | |
| *** priteau has quit IRC | 13:06 | |
| *** Qianbiao has quit IRC | 13:07 | |
| *** Qianbiao has joined #openstack-ironic | 13:07 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-inspector master: Replace retrying with tenacity https://review.opendev.org/745667 | 13:11 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-inspector master: Replace retrying with tenacity https://review.opendev.org/745667 | 13:11 |
| *** ijw has joined #openstack-ironic | 13:15 | |
| TheJulia | good morning | 13:28 |
| iurygregory | good morning TheJulia =) | 13:28 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-inspector stable/ussuri: Fix URL parsing to enable redfish_address matching https://review.opendev.org/746718 | 13:31 |
| *** ijw has quit IRC | 13:31 | |
| rpittau | good morning TheJulia :) | 13:31 |
| *** ijw has joined #openstack-ironic | 13:31 | |
| iurygregory | rpittau, same error =) | 13:39 |
| rpittau | mmmmm | 13:39 |
| iurygregory | https://storage.bhs.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_45c/745536/7/check/ironic-lib-wholedisk-bios-ipmi-iscsi-src/45c90fc/ | 13:40 |
| iurygregory | zuul hasn't finished the other jobs yet | 13:40 |
| iurygregory | if the flavor and the baremetal node has specific property capabilites is there something we also need to pass for the nova instance? | 13:45 |
| *** ijw has quit IRC | 13:45 | |
| *** tkajinam has joined #openstack-ironic | 13:50 | |
| *** k_mouza has quit IRC | 13:51 | |
| *** k_mouza has joined #openstack-ironic | 13:53 | |
| *** ntt_ has joined #openstack-ironic | 13:59 | |
| *** abdysn has quit IRC | 14:04 | |
| *** ntt_ has quit IRC | 14:04 | |
| rpittau | it *could* be that the qemu-img filter in ironic-lib conflicts with the one in ironic | 14:05 |
| rpittau | actually I'm not sure we need the filter in ironic | 14:08 |
| rpittau | all the qemu-img used in images.py come from disk_utils under ironic-lib | 14:09 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic master: Remove qemu-img rootwrap filter https://review.opendev.org/746731 | 14:13 |
| rpittau | let's see if zuul likes it | 14:13 |
| iurygregory | yeah it comes from ironic-lib, the only think weird is the link falling | 14:14 |
| rpittau | that could be an issue with the owner of the final image | 14:15 |
| iurygregory | yeah | 14:16 |
| rpittau | if the patch passes we can try rebasing yours on that, see if the conflict goes away | 14:16 |
| iurygregory | yup =D | 14:16 |
| iurygregory | If zuul is happy I will push with Depends-On on yours | 14:16 |
| rpittau | onw thing you could try is adding some logs, like ownership of the images/dirs for example | 14:16 |
| rpittau | oh yeah, depends-on, not rebase :) | 14:17 |
| *** hjensas is now known as hjensas|afk | 14:29 | |
| *** ijw has joined #openstack-ironic | 14:37 | |
| openstackgerrit | Merged openstack/ironic stable/train: DRAC: Drives conversion from raid to jbod https://review.opendev.org/738508 | 14:37 |
| *** Qianbiao has quit IRC | 14:47 | |
| *** Qianbiao has joined #openstack-ironic | 14:47 | |
| *** mkrai has joined #openstack-ironic | 14:57 | |
| *** belmoreira has quit IRC | 14:57 | |
| *** priteau has joined #openstack-ironic | 15:04 | |
| *** mkrai has quit IRC | 15:10 | |
| *** mkrai_ has joined #openstack-ironic | 15:10 | |
| *** gyee has joined #openstack-ironic | 15:16 | |
| *** ijw_ has joined #openstack-ironic | 15:21 | |
| *** ijw has quit IRC | 15:24 | |
| dking | Is there a way to have all the clean steps executed manually that would be executed by the automated clean process, without having to send each in "clean_steps"? | 15:25 |
| *** Qianbiao has quit IRC | 15:44 | |
| *** lucasagomes has quit IRC | 16:02 | |
| *** hjensas|afk is now known as hjensas | 16:17 | |
| *** ricolin has quit IRC | 16:19 | |
| TheJulia | I feel like there is supposed to be | 16:24 |
| TheJulia | but maybe not | 16:24 |
| openstackgerrit | Iury Gregory Melo Ferreira proposed openstack/ironic-lib master: Add initial privsep structure https://review.opendev.org/745536 | 16:25 |
| *** ricolin has joined #openstack-ironic | 16:29 | |
| JayF | TheJulia: is there a reason I'm unaware of we don't explicitly call `eventlet.monkey_patch()` early in IPA's lifetime, and instead wait until a library that needs it does it? I think this is the root-cause of my SSL bug, but wanted to ensure nobody was aware of a specific reason it wouldn't be. | 16:30 |
| JayF | TBH I also wonder if that is part of the performance stuff that was done recently to change eventlet.sleep(0) -> eventlet.sleep(0.1) to prevent IPA eating up CPU, but that's more hopeful than understanding | 16:31 |
| JayF | I'm talking a code change as "simple" as https://gist.github.com/jayofdoom/6e0826e5644cff7dc919ef21cfc388d7 | 16:34 |
| *** mkrai_ has quit IRC | 16:34 | |
| *** Lucas_Gray has quit IRC | 16:43 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: [WIP] Prepare to move ipa-builder jobs https://review.opendev.org/746766 | 16:48 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent-builder master: [DNM] Test CI https://review.opendev.org/746767 | 16:49 |
| openstackgerrit | Julia Kreger proposed openstack/sushy master: Remove auth token header completely when error occurs https://review.opendev.org/746768 | 16:57 |
| TheJulia | bdodd: ^^^ | 16:58 |
| TheJulia | JayF: eventlet doesn't play well with one of the other libraries | 16:58 |
| JayF | TheJulia: do you know which one or have any details? This is almost guaranteed to be a real bug at a certain level and I'd like to get an upstreamable fix for it | 16:59 |
| TheJulia | JayF: interesting... because I think it was the wsgi code that things were going sideways on | 16:59 |
| JayF | oslo.service 100% monkey_patches eventlet, just too late to save us from ssl errors | 16:59 |
| JayF | so I am highly suspect that has underlying issues, except perhaps tranient bugs in eventlet (believe me, I've read a metric crapton of wsgi/ssl/eventlet bugs in the last 3 days) | 17:00 |
| JayF | *transient | 17:00 |
| TheJulia | so socket would hang... I wonder if it was the same issue with ssl when we tried to monkey patch there. I also wonder if it was the greendns code too.... | 17:01 |
| bnemec | You should definitely monkey patch as early as possible. Preferrably at a top-level __init__.py so nothing else gets imported first. | 17:01 |
| TheJulia | if the unit tests pass, then I say toss it up | 17:01 |
| JayF | bnemec: that's what my proposed patch would do | 17:02 |
| JayF | TheJulia: do you want me to do that /as part of/ a feature to expose SSL? Or independently? | 17:02 |
| TheJulia | JayF: independent please | 17:02 |
| JayF | ack; will-do | 17:02 |
| *** ociuhandu has quit IRC | 17:03 | |
| *** k_mouza has quit IRC | 17:04 | |
| TheJulia | I don't remember where anymore but there was some fundimental incompatability and it is really bothering me now | 17:08 |
| JayF | I wonder if it was the old built-in wsgi server which is gone now. | 17:14 |
| JayF | Either way, eventlet is there, and we need to use it properly, and fix incompatibilites if they exist. | 17:14 |
| JayF | Rather than relying on being lucky that we're monkey_patching late (via libraries) and things aren't breaking | 17:15 |
| *** mkrai has joined #openstack-ironic | 17:15 | |
| *** ociuhandu has joined #openstack-ironic | 17:17 | |
| *** ociuhandu has quit IRC | 17:22 | |
| openstackgerrit | Jay Faulkner proposed openstack/ironic-python-agent master: Eventlet should be monkey patched as early as possible https://review.opendev.org/746774 | 17:24 |
| JayF | TheJulia: I think rloo unearthed it, I'm going to adjust my patch to take this into account https://opendev.org/openstack/ironic/commit/556b1d0871d01ab17715955bef566c99bcf6bedc | 17:37 |
| * JayF also wonders if we need os=false in IPA, like we have in Ironic | 17:41 | |
| JayF | tempted to cargo cult that since we'd already have `os` imported there... | 17:41 |
| bdodd | TheJulia I reviewed the re-authentication changes. LGTM | 17:43 |
| *** mkrai has quit IRC | 17:54 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-lib master: Additional testing for Metal3 string cases https://review.opendev.org/746781 | 17:59 |
| *** trandles has joined #openstack-ironic | 18:37 | |
| *** johnhaan has quit IRC | 19:00 | |
| *** arne_wiebalck has quit IRC | 19:01 | |
| *** rajinir has quit IRC | 19:01 | |
| *** johnhaan has joined #openstack-ironic | 19:08 | |
| *** rajinir has joined #openstack-ironic | 19:08 | |
| *** arne_wiebalck has joined #openstack-ironic | 19:09 | |
| *** k_mouza has joined #openstack-ironic | 19:14 | |
| *** k_mouza has quit IRC | 19:17 | |
| *** dustinc has joined #openstack-ironic | 19:39 | |
| *** priteau has quit IRC | 19:44 | |
| *** ociuhandu has joined #openstack-ironic | 19:58 | |
| *** ociuhandu has quit IRC | 20:02 | |
| *** zbitter has joined #openstack-ironic | 20:04 | |
| *** zaneb has quit IRC | 20:04 | |
| *** hjensas has quit IRC | 20:11 | |
| openstackgerrit | Julia Kreger proposed openstack/ironic-python-agent master: Add basic retries for inspection https://review.opendev.org/745598 | 20:28 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-python-agent bugfix/6.3: Update .gitreview for bugfix/6.3 https://review.opendev.org/746794 | 20:42 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic-python-agent bugfix/6.3: Update TOX_CONSTRAINTS_FILE for bugfix/6.3 https://review.opendev.org/746795 | 20:42 |
| stevebaker | morning | 20:46 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic bugfix/15.2: Update .gitreview for bugfix/15.2 https://review.opendev.org/746796 | 20:47 |
| openstackgerrit | OpenStack Release Bot proposed openstack/ironic bugfix/15.2: Update TOX_CONSTRAINTS_FILE for bugfix/15.2 https://review.opendev.org/746797 | 20:47 |
| *** Lucas_Gray has joined #openstack-ironic | 21:03 | |
| TheJulia | good morning stevebaker | 21:03 |
| *** hjensas has joined #openstack-ironic | 21:07 | |
| JayF | TheJulia: if you're interested, found the eventlet "catch". select.poll doesn't exist in eventlet, which is how we time heartbeats in IPA today | 21:16 |
| JayF | TheJulia: I'm highly suspicious that could be a cause of the high cpu usage that https://opendev.org/openstack/ironic-python-agent/commit/952489020ea36aced3737079fa17eff2e37b4da7 worked around | 21:16 |
| JayF | I have two paths I can go: | 21:16 |
| JayF | 1) monkey_patch(select=false) | 21:16 |
| JayF | 2) try to find a replacement for select.poll there -- eventlet has a few options. | 21:16 |
| JayF | I think I'll timebox #2 and fallback to #1 if needed | 21:17 |
| TheJulia | I wonder if tenacity could replace it | 21:17 |
| JayF | Hmm, that's not one I had considered, I'll keep that in mind | 21:18 |
| *** rcernin has joined #openstack-ironic | 21:18 | |
| TheJulia | and I too was curious about the high cpu usage. There IS a case where it is possible if memory serves, but blarg | 21:18 |
| JayF | I was more wondering why it's just not a pure oslo.concurrency periodic task | 21:18 |
| TheJulia | good question | 21:18 |
| JayF | the move to select.poll was to keep it from using wall-clock-time https://opendev.org/openstack/ironic-python-agent/commit/4f57590b2e41863770b764a8b84b8b81830ad577 | 21:19 |
| JayF | because when we were resetting clocks, heartbeats started getting missed because the wall-time changed | 21:19 |
| TheJulia | *sigh* | 21:20 |
| iurygregory | TheJulia, if I want to set the capabilite of a node that was created would be http://paste.openstack.org/show/796926/ right? | 21:20 |
| TheJulia | iurygregory: what are you still doing up? | 21:21 |
| iurygregory | it's only 23 | 21:21 |
| JayF | I'm not her but that looks right :) | 21:21 |
| JayF | 23 is beer:thirty, not work:thirty :D | 21:21 |
| TheJulia | iurygregory: heh | 21:21 |
| TheJulia | It does look right | 21:21 |
| TheJulia | And what JayF said! | 21:21 |
| iurygregory | hehehe | 21:24 |
| iurygregory | well yesterday I had 5 beers | 21:25 |
| iurygregory | to forget about the problems since I didn't sleep from sunday to monday | 21:25 |
| iurygregory | and I had headache | 21:25 |
| iurygregory | today I'm like 85% better | 21:25 |
| iurygregory | seems like tempest didn't like the patch to update the node capabilities | 21:31 |
| TheJulia | how so? | 21:33 |
| iurygregory | tempest doesn't show any attempt to update the node setting boot_option... | 21:34 |
| iurygregory | some type of magic since it's tempest hehehe | 21:34 |
| TheJulia | hmm | 21:35 |
| *** rcernin has quit IRC | 21:35 | |
| TheJulia | verified that there is no PATCH operation in the api log? | 21:35 |
| iurygregory | let me check | 21:35 |
| iurygregory | nope .-. | 21:37 |
| TheJulia | iurygregory: link to patch? | 21:37 |
| iurygregory | https://review.opendev.org/#/c/741886/ | 21:37 |
| patchbot | patch 741886 - ironic - Add tempest default_boot_option and altflavor - 9 patch sets | 21:37 |
| TheJulia | where are you not seeing it run | 21:41 |
| TheJulia | meaning like what job? | 21:41 |
| iurygregory | all jobs with nova that failed | 21:41 |
| iurygregory | I was debugging on https://zuul.opendev.org/t/openstack/build/65e4cb538b7149d48f0942e8639b9bb3 | 21:44 |
| TheJulia | hmm | 21:46 |
| TheJulia | hmm | 21:48 |
| iurygregory | according to sean on nova I need to set the capabilities before creating the nova instance o.o | 21:48 |
| iurygregory | like on line 183 on tempest https://review.opendev.org/#/c/741445/23/ironic_tempest_plugin/tests/scenario/baremetal_manager.py@190 | 21:48 |
| patchbot | patch 741445 - ironic-tempest-plugin - Update basic_ops - 23 patch sets | 21:48 |
| *** rcernin has joined #openstack-ironic | 21:52 | |
| iurygregory | ok, there is probably a better way to do it, since we set the cababilities with https://github.com/openstack/ironic/blob/master/devstack/lib/ironic#L2293-L2307 | 21:52 |
| TheJulia | your setting capabiliteis way too late | 21:52 |
| iurygregory | yeah | 21:52 |
| TheJulia | your doing it after create_server | 21:52 |
| TheJulia | if you want nova to pick it up at all, it needs to be well before | 21:53 |
| iurygregory | other way I see now is doing something on devstack/lib/ironic... | 21:53 |
| iurygregory | we can do some magic and say "node0 will have this" "node1 will have this other option" | 21:53 |
| TheJulia | you likely have to since you don't know what node you'll be scheduled on | 21:54 |
| TheJulia | your trying to influence it and the resulting behavior | 21:54 |
| iurygregory | yeah, but since we will have > 2 nodes it should be fine | 21:54 |
| TheJulia | as long as we don't create a condition where we need 4 nodes | 21:55 |
| iurygregory | I will update things tomorrow morning | 21:55 |
| iurygregory | and see how it goes =) | 21:55 |
| iurygregory | almost midnight and now at least a have a plan on how to make it work \o/ | 21:56 |
| openstackgerrit | Merged openstack/ironic-python-agent-builder master: Update name of ipa job https://review.opendev.org/743043 | 21:57 |
| TheJulia | goodnight | 21:57 |
| TheJulia | I'm going to go exercise | 21:57 |
| iurygregory | ++ | 21:57 |
| *** rcernin has quit IRC | 21:57 | |
| *** rcernin has joined #openstack-ironic | 22:09 | |
| *** rcernin has quit IRC | 22:09 | |
| *** dking has quit IRC | 22:24 | |
| *** rcernin has joined #openstack-ironic | 22:33 | |
| *** pas-ha has quit IRC | 22:42 | |
| *** sri_ has quit IRC | 22:42 | |
| *** ildikov has quit IRC | 22:43 | |
| *** rpioso has quit IRC | 22:43 | |
| *** trandles has quit IRC | 22:47 | |
| *** sri_ has joined #openstack-ironic | 22:51 | |
| *** rpioso has joined #openstack-ironic | 22:53 | |
| *** ildikov has joined #openstack-ironic | 22:55 | |
| *** pas-ha has joined #openstack-ironic | 22:55 | |
| *** Lucas_Gray has quit IRC | 23:05 | |
| *** JamesBenson has joined #openstack-ironic | 23:10 | |
| *** hjensas has quit IRC | 23:33 | |
| janders | good morning Ironic o/ | 23:33 |
| stevebaker | janders: good morning! | 23:37 |
| janders | hey stevebaker how is it going? | 23:40 |
| stevebaker | janders: pretty good thanks, and you? | 23:40 |
| janders | I'm good, thank you! :) Slowly getting back to my outdoor routine, which is great. | 23:41 |
| stevebaker | I went for the first run in 3 weeks yesterday, which was real nice | 23:43 |
| janders | great! :) | 23:46 |
| janders | do you run for fun, or are you working towards some goal and/or event? | 23:47 |
| janders | I used to do duathlons and triathlons before fires and then COVID shut me down, so hoping to get back to that | 23:48 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!