| iurygregory | good morning Ironic o/ | 06:05 |
|---|---|---|
| arne_wiebalck | Good morning iurygregory and Ironic! | 06:35 |
| iurygregory | morning arne_wiebalck o/ | 06:36 |
| stendulker | Good morning arne_wiebalck, iurygregory o/ | 06:56 |
| iurygregory | morning stendulker o/ | 06:56 |
| arne_wiebalck | Good morning stendulker o/ | 06:56 |
| *** rpittau|afk is now known as rpittau | 07:07 | |
| rpittau | good morning ironic! o/ | 07:07 |
| iurygregory | morning rpittau o/ | 07:07 |
| rpittau | hey iurygregory :) | 07:07 |
| janders | hey iurygregory arne_wiebalck stendulker rpittau and Ironic o/ | 07:19 |
| rpittau | hey janders :) | 07:19 |
| stendulker | morning janders o/ | 07:19 |
| iurygregory | hey janders o/ | 07:19 |
| dtantsur | good morning folks | 10:08 |
| rpittau | good morning dtantsur :) | 10:12 |
| janders | hey dtantsur | 10:34 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Allow ramdisk_image_download_source in instance_info for ramdisk deploy https://review.opendev.org/c/openstack/ironic/+/797508 | 10:53 |
| dtantsur | rpittau: fixed the typo ^^^ | 10:53 |
| rpittau | ok :D | 10:53 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix ramdisk boot option handling https://review.opendev.org/c/openstack/ironic/+/797517 | 10:59 |
| janders | see you tomorrow Ironic o/ | 12:04 |
| TheJulia | good morning | 12:57 |
| rpittau | good morning TheJulia :) | 13:31 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 13:32 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Scoped RBAC Devstack Plugin support https://review.opendev.org/c/openstack/ironic/+/778957 | 13:37 |
| TheJulia | good morning rpittau | 13:37 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 14:04 |
| dtantsur | morning TheJulia | 14:06 |
| TheJulia | The fun of working on the tempest plugin... | 14:08 |
| TheJulia | all of the waiting :( | 14:08 |
| TheJulia | https://imgflip.com/i/5e9kh9 | 14:12 |
| arne_wiebalck | For the TLS issue I mentioned, I summarized obserbvations here for the record: https://storyboard.openstack.org/#!/story/2009004 | 14:12 |
| arne_wiebalck | I tried a couple of things but was not able to get to the bottom of it. | 14:12 |
| * arne_wiebalck will either disable TLS for the IPA and learn to live with the error in the logs ... | 14:14 | |
| arne_wiebalck | s/and/or | 14:15 |
| TheJulia | I wonder if the ssl library is loading once, and only refreshing certificate caches after failure? | 14:20 |
| dtantsur | arne_wiebalck: could you try $ openssl s_client -showcerts -connect IPA_IP:9999 | 14:30 |
| dtantsur | this should give you detailed information about the generated certificate | 14:30 |
| arne_wiebalck | dtantsur: let me try ... | 14:30 |
| dtantsur | we have a strange problem with TLS on victoria as well, but it is not solved by retrying | 14:30 |
| arne_wiebalck | :( | 14:31 |
| arne_wiebalck | TheJulia: the IPA API is started only after the cert is generated, should there be some SSL caching before? | 14:33 |
| dtantsur | and the certificate is verified on the ironic side, not the IPA side | 14:34 |
| TheJulia | arne_wiebalck: I think it is likely there could be on library load | 14:36 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 14:36 |
| TheJulia | or the file could be open on an old file handler and the failure could close it out? | 14:37 |
| arne_wiebalck | Hmm ... I removed the file between runs, but if sth has it still open ... | 14:40 |
| arne_wiebalck | dtantsur: should I look for anything specific in the output of the command you suggested? | 14:41 |
| arne_wiebalck | dtantsur: I see "Verify return code: 18 (self signed certificate)" | 14:41 |
| arne_wiebalck | dtantsur: but this makes sense | 14:41 |
| dtantsur | arne_wiebalck: it should have the IP address in the extended fields IIRC | 14:41 |
| arne_wiebalck | There is no fallback like: "failure, ok: drop TLS" ... is there? | 14:41 |
| dtantsur | btw have you tried with 'curl' and providing the certificate path? | 14:42 |
| arne_wiebalck | such a fallback would also perfectly match the pattern :-D | 14:43 |
| arne_wiebalck | dtantsur: no, I have not | 14:43 |
| dtantsur | try it with maximum verbosity, it can also be quite detailed | 14:48 |
| dtantsur | TheJulia may be on to something with the caching idea | 14:49 |
| dtantsur | I'm trying to understand from the requests code if the TLS parameters may be cached in the pool | 14:49 |
| TheJulia | if there is a pre-existing handler, then it will have the old contents until the file is closed and reopened | 14:50 |
| dtantsur | I don't think it's related to the file, but rather to a connection | 14:50 |
| dtantsur | connections are reused very explicitly | 14:50 |
| TheJulia | oh! | 14:50 |
| TheJulia | yes, that could also be | 14:50 |
| dtantsur | on the other hand, requests seem to update the connection with new parameters each time | 14:52 |
| arne_wiebalck | one more data point: it does not happen all the time | 14:55 |
| arne_wiebalck | I removed the cert and restarted conductor: no error | 14:55 |
| * arne_wiebalck will repeat that exercise | 14:55 | |
| arne_wiebalck | this would fit the caching/reuse idea | 14:56 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Fix handling driver_info[agent_verify_ca] == False https://review.opendev.org/c/openstack/ironic/+/797669 | 15:00 |
| dtantsur | not exactly related, just noticed while browsing the code ^^^ | 15:00 |
| dtantsur | I wonder if driver_info == False should take priority over the auto-generated certificate in driver_internal_info | 15:00 |
| dtantsur | okay, lemme try something around caching clients | 15:04 |
| arne_wiebalck | TheJulia: you meant caching on the client (conductor) side, right? | 15:06 |
| TheJulia | yes | 15:06 |
| ygk_12345 | Hi all | 15:06 |
| TheJulia | hello ygk_12345 | 15:07 |
| arne_wiebalck | sorry, ignore my comment about the IPA API then | 15:07 |
| JayF | dtantsur: arne_wiebalck: I wonder, with metrics we had to ensure we were late loading some stuff so the config actually took effect. Makes me wonder if there could be a similar race for TLS things... | 15:07 |
| JayF | That's all highly dusty knowledge though so it may not work that way anymore | 15:07 |
| ygk_12345 | Does ironic bare metal nodes support tenant networks in open stack ? | 15:07 |
| TheJulia | ygk_12345: you need a neutron ml2 driver and the baremetal node network_interface set to neutron. | 15:07 |
| TheJulia | ygk_12345: give me a minute, I'll get you a link | 15:07 |
| ygk_12345 | Sure please | 15:07 |
| TheJulia | https://docs.openstack.org/ironic/latest/install/configure-tenant-networks.html and https://docs.openstack.org/ironic/latest/admin/multitenancy.html | 15:10 |
| TheJulia | ygk_12345: ^^^ | 15:10 |
| ygk_12345 | TheJulia, thank you | 15:11 |
| TheJulia | That should provide the context and information requirements your looking for. https://docs.openstack.org/ironic/latest/admin/multitenancy.html#neutron-network-interface is the most vital step outside of ironic | 15:11 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: [WIP] Cache AgentClient on Task, not globally https://review.opendev.org/c/openstack/ironic/+/797674 | 15:11 |
| dtantsur | arne_wiebalck: do you think you can try this patch and see if it changes anything ^^^ | 15:11 |
| arne_wiebalck | JayF: or the conductor uses the crt file for the node it had from before and only considers to check if the cert changed on a verification error? | 15:11 |
| arne_wiebalck | dtantsur: sure! | 15:12 |
| arne_wiebalck | I removed the file and restarted the conductor now before cleaning: no SSL error. | 15:14 |
| arne_wiebalck | I repeated this three times, no SSL error. | 15:14 |
| arne_wiebalck | Really looks like re-use/caching more than a race. | 15:15 |
| dtantsur | okay, my patch should remove connection caching between task executions | 15:17 |
| dtantsur | which may make performance somewhat worse, but I'm out of better ideas for now | 15:18 |
| arne_wiebalck | dtantsur: thanks, I will need a moment to backport this to victoria | 15:21 |
| dtantsur | that's going to have a few conflcits.. | 15:22 |
| arne_wiebalck | yep, on all files :) | 15:22 |
| arne_wiebalck | nah, 3/4 :) | 15:22 |
| dtantsur | I can try doing it locally if you need me to | 15:22 |
| dtantsur | I assume you're using direct deploy? | 15:22 |
| arne_wiebalck | yes | 15:23 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic stable/victoria: [WIP] Cache AgentClient on Task, not globally https://review.opendev.org/c/openstack/ironic/+/797679 | 15:27 |
| dtantsur | arne_wiebalck: this should be victoria for you ^^^ | 15:27 |
| arne_wiebalck | dtantsur: wow, thanks, I will pull that in! | 15:28 |
| dtantsur | I cannot promise I haven't made a silly mistake, don't deploy on prod :) | 15:28 |
| TheJulia | stevebaker: so I was looking at puppet-ironic, and it looks like the knob for ipxe or not with inspector is pxe_transfer_protocol | 15:29 |
| dtantsur | I think this ^^^ is correct | 15:29 |
| arne_wiebalck | dtantsur: ha ha ha ... will try not to :-D | 15:29 |
| TheJulia | dtantsur: do you know if ipxe_enabled influences it, or if tripleo could be using ipxe_enabled in undercloud.conf to infer it ? | 15:29 |
| arne_wiebalck | dtantsur: applies w/o conflicts | 15:29 |
| dtantsur | TheJulia: the latter sounds about right. but note that I haven't seen this code in many months | 15:30 |
| TheJulia | dtantsur: ack, thanks stevebaker ^^ | 15:30 |
| dtantsur | previously ipxe_enabled would change both this and the ironic's ipxe_enabled | 15:30 |
| TheJulia | yeah | 15:31 |
| arne_wiebalck | dtantsur: pipeline to build test rpms is running ... | 15:32 |
| dtantsur | nice | 15:33 |
| TheJulia | stevebaker: https://review.opendev.org/c/openstack/puppet-ironic/+/797683 <-- in kind of whipping this usp, it looks like templates/inspector_dnsmasq_tftp.erb has no equilvelant transport or option for EFI booting. It does seem that there is an syslinux.efi binary out there for pxelinux.0's uefi replacement. https://wiki.syslinux.org/wiki/index.php?title=PXELINUX#UEFI | 15:37 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 16:07 |
| rpittau | good night! o/ | 16:08 |
| *** rpittau is now known as rpittau|afk | 16:08 | |
| dtantsur | our boot interface validation still makes me want to cry | 16:14 |
| dtantsur | arne_wiebalck: FYI our problem downstream was indeed clock skew :D | 16:26 |
| dtantsur | I wonder if we should add validation that clocks match between IPA and ironic | 16:26 |
| dtantsur | maybe on lookup | 16:26 |
| arne_wiebalck | dtantsur: heh ... I was thinking after hours of testing various things: what if dtantsur's first thought about clock skew is true and I did not listen carefully enough :-D | 16:33 |
| * arne_wiebalck is testing the new RPM since 30 mins ... | 16:34 | |
| *** sshnaidm is now known as sshnaidm|afk | 16:35 | |
| dtantsur | have a good night folks | 16:48 |
| arne_wiebalck | dtantsur: I am not able to reproduce the SSL error | 16:49 |
| arne_wiebalck | so far | 16:49 |
| TheJulia | goodnight dtantsur | 16:52 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: WIP: Secure RBAC support https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 17:13 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Use get_service_clients framework with basic Secure RBAC https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 18:34 |
| arne_wiebalck | bye everyone o/ | 18:50 |
| opendevreview | Ade Lee proposed openstack/ironic master: DNM/WIP - Add FIPS jobs https://review.opendev.org/c/openstack/ironic/+/797739 | 18:55 |
| opendevreview | Ade Lee proposed openstack/ironic-python-agent master: WIP/DNM: Add FIPS jobs https://review.opendev.org/c/openstack/ironic-python-agent/+/797741 | 19:08 |
| opendevreview | Ade Lee proposed openstack/ironic-python-agent-builder master: DNM/WIP - Add FIPS job https://review.opendev.org/c/openstack/ironic-python-agent-builder/+/797743 | 19:24 |
| opendevreview | Julia Kreger proposed openstack/ironic-tempest-plugin master: Use get_service_clients framework with basic Secure RBAC https://review.opendev.org/c/openstack/ironic-tempest-plugin/+/797521 | 20:32 |
| TheJulia | and so, no syslinux.efi. Looks like it was transiently into rawhide, fedora 32, and fedora-33, but didn't go any further. | 21:10 |
| TheJulia | stevebaker: ^^^ which leads to the only available option of including ipxe (easyish I guess) or grub2 for network booting (hey, arm compatible with the right binary) | 21:12 |
| stevebaker | TheJulia: I see | 21:14 |
| TheJulia | we know the grub pattern+binaries. We have a job in CI for it. | 21:18 |
| TheJulia | on a plus side, it looks like ironic-tempest-plugin is semi-ready for enforced scopes | 21:22 |
| TheJulia | at least, with ironic. | 21:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: WIP Scoped RBAC Devstack Plugin support https://review.opendev.org/c/openstack/ironic/+/778957 | 21:33 |
| TheJulia | hmm, no dice on standalone test jobs yet | 21:42 |
| TheJulia | but likely closer | 21:42 |
| * TheJulia calls it a day | 21:52 | |
| janders | good morning Ironic o/ | 23:45 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!