| *** mat_fechner is now known as matfechner | 04:24 | |
| kamlesh6808c | Good morning Ironic ! | 04:25 |
|---|---|---|
| arne_wiebalck | Good morning kamlesh6808c and Ironic! | 06:56 |
| arne_wiebalck | stevebaker[m]: thanks for picking this up! | 06:56 |
| rpittau | good morning ironic! o/ | 07:41 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 07:50 |
| rpittau | can I please get a quick review on https://review.opendev.org/c/openstack/ironic-python-agent/+/841220 ? | 07:51 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 08:09 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 08:58 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 08:59 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 09:20 |
| rpittau | dtantsur: https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 is green, not without some effort | 09:57 |
| dtantsur | rpittau: even more effort required, sorry. with this IPA is also checked out at stable/yoga, which means we're not testing patches | 11:11 |
| iurygregory | good morning Ironic | 11:11 |
| dtantsur | I wonder if we need required-projects with IPA explicitly set to bugfix/whatever | 11:11 |
| dtantsur | morning iurygregory | 11:11 |
| rpittau | so we need to override by project, not by job :/ | 11:11 |
| rpittau | yeah | 11:12 |
| rpittau | hey iurygregory :) | 11:12 |
| dtantsur | rpittau: given that we only care about requirements and IPA, it may be easier to do by project | 11:12 |
| rpittau | ok, I'll have a look after lunch | 11:12 |
| dtantsur | thx! | 11:12 |
| rpioso | Good morning, ironic :) | 12:31 |
| rpioso | arne_wiebalck: Thank you! That's consistent with the reply I got about remote presentation. | 12:31 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 12:49 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 13:08 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:11 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:20 |
| rpittau | I'm the king of typos.... | 13:20 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:26 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:28 |
| TheJulia | good morning | 13:47 |
| rpittau | good morning TheJulia :) | 13:51 |
| rpittau | dtantsur: can you please double-check when you have a moment? https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:52 |
| rpittau | TheJulia: I've updated the multipath patch https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 I think it's ready now | 13:53 |
| iurygregory | rpittau, wow re https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 13:59 |
| rpittau | heh.... | 13:59 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.4: Use Yoga constraints for bugfix/8.4 https://review.opendev.org/c/openstack/ironic-python-agent/+/841290 | 14:26 |
| arne_wiebalck | rpioso: ++ | 14:32 |
| ayoung | Where would I drop an SSL cert in the ipa image in order to get a successful SSL (TLS) connection back during the clean stage? | 14:38 |
| ayoung | I am assuming in the initrd somewhere | 14:38 |
| TheJulia | ayoung: locally signed endpoint ssl certifiacates? | 14:45 |
| ayoung | TheJulia, I think so, yes | 14:45 |
| ayoung | I installed via bifrost with enable TLS = true so whatever that does | 14:46 |
| ayoung | The IPA image generation is x86 specific still and I want to test before working on making that aarch64 enabled | 14:46 |
| dtantsur | ayoung: by default IPA is configured with insecure=True so you don't have to bother about this too much | 14:52 |
| dtantsur | unless you're messing with configuration | 14:52 |
| TheJulia | That call is also just the heartbeat | 14:53 |
| dtantsur | (yes, bifrost generates self-signed certificates) | 14:53 |
| TheJulia | so potentially acceptable opsec risk | 14:53 |
| ayoung | well, the ipa instances are sticking around in clean wait | 14:53 |
| dtantsur | ayoung: can you check the generated kernel params (in the httpboot directory, location depends on version) | 14:54 |
| ayoung | pretty sure TLS is the only difference between the one I had working a week or two ago and now | 14:54 |
| dtantsur | we have a CI job with TLS enabled | 14:54 |
| ayoung | ok, so in httpboot, where do I look> | 14:55 |
| ayoung | ? | 14:55 |
| dtantsur | there should be a weirdly named script, just grep for "^kernel"? | 14:55 |
| dtantsur | probably in pxelinux.cfg/<MAC> | 14:56 |
| ayoung | just the sha | 14:56 |
| ayoung | boot.ipxe ? | 14:56 |
| dtantsur | this is the generic one, you need the per-node conf | 14:56 |
| ayoung | ah...already gone...let me reclean | 14:57 |
| dtantsur | $ sudo grep kernel /var/lib/ironic/httpboot/pxelinux.cfg/52-54-00-eb-42-e6 | head -1 | 14:57 |
| TheJulia | yeah, it will be in /httpboot/<node-uuid>/config if memory serves | 14:57 |
| dtantsur | kernel http://192.168.122.1:8080//4e41df61-84b1-5856-bfb6-6b5f2cd3dd11/deploy_kernel selinux=0 troubleshoot=0 text nofb nomodeset systemd.journald.forward_to_console=yes console=ttyS0 ipa-insecure=1 ipa-debug=1 ipa-api-url=http://192.168.122.1:6385 ipa-global-request-id=req-5db4d02d-4589-4db4-b3f3-68f7b15c1e33 BOOTIF=${mac} initrd=deploy_ramdisk || goto retry | 14:57 |
| dtantsur | or that | 14:57 |
| TheJulia | the pxelinux.cfg files are just links | 14:57 |
| dtantsur | one of them is a symlink to the other | 14:57 |
| ayoung | I think it actually just worked... | 14:58 |
| ayoung | the node was in avaiable...hrm... | 14:58 |
| dtantsur | I have a guess | 14:58 |
| dtantsur | is it possible that the node had the agent running when you enabled TLS? | 14:59 |
| dtantsur | e.g. from previous cleanings? | 14:59 |
| ayoung | No, it rebooted | 14:59 |
| dtantsur | I mean, when you had the error? | 14:59 |
| dtantsur | that it rebooted now may be the reason it started working :) | 15:00 |
| ayoung | I'll keep playing see what data I can generate | 15:00 |
| * dtantsur is trying to have a meeting from the balcony | 15:02 | |
| TheJulia | balcony++ ? | 15:02 |
| TheJulia | ayoung: why did they reboot? | 15:02 |
| TheJulia | what triggered it? | 15:02 |
| TheJulia | ayoung: if an un-expected reboot occurs, your agent token can't be retrieved again | 15:03 |
| ayoung | enP4p3s0u1u3c2 who came up with this naming scheme? Can whomever sits closest to that person deliver a swift kick to the shins for me? | 15:18 |
| ayoung | TheJulia, just the initila PXE boot kicked off by the clean, I think | 15:18 |
| ayoung | its seems to be working ok but not consistently | 15:18 |
| ayoung | which might be hardware, as it is a new cluster | 15:18 |
| ayoung | and cluster is only half the word | 15:19 |
| dtantsur | when you retry cleaning after a failure, a reboot is always done, even in fast-track mode | 15:20 |
| ayoung | OK. But this was not after a failure. I triggered the clean manually | 15:21 |
| ayoung | openstack baremetal node clean mystique14-r116 --clean-steps '[{"interface": "deploy", "step": "erase_devices"}]' | 15:21 |
| ayoung | I rally was testing whether the PXE boot stages works, which it does | 15:21 |
| ayoung | gonna try a full deploy | 15:21 |
| ayoung | once I see if this clean succeeds or fails, but it looks good on one node so far | 15:22 |
| TheJulia | okay | 15:22 |
| ayoung | so...node in clean wait state, sol on the node shows it at the debian login prompt | 15:35 |
| ayoung | cleaned 2 nodes. one is in the manageable state now, the other in clean wait | 15:36 |
| ayoung | not exactly success, not quite failure. Too early for whisky, too late for coffee | 15:36 |
| dtantsur | exactly right for coffee with whisky? | 15:37 |
| TheJulia | could it still be cleaning? | 15:38 |
| TheJulia | could the block device be... damaged? | 15:38 |
| dtantsur | erase_devices can take as long as it wants if there is no hardware-assisted secure erase | 15:48 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.3: Use Yoga constraints for bugfix/8.3 https://review.opendev.org/c/openstack/ironic-python-agent/+/841291 | 15:49 |
| TheJulia | Indeed | 15:49 |
| TheJulia | also, we default to a single thread | 15:49 |
| * TheJulia has a change... someplace... to make it 4 threads | 15:49 | |
| TheJulia | Here change... when I finally get back to code | 15:49 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.1: Use Xena constraints for bugfix/8.1 https://review.opendev.org/c/openstack/ironic-python-agent/+/841293 | 15:54 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.1: Use Xena constraints for bugfix/8.1 https://review.opendev.org/c/openstack/ironic-python-agent/+/841293 | 15:54 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.3: Use Yoga constraints for bugfix/8.3 https://review.opendev.org/c/openstack/ironic-python-agent/+/841291 | 15:55 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent bugfix/8.3: Use Yoga constraints for bugfix/8.3 https://review.opendev.org/c/openstack/ironic-python-agent/+/841291 | 15:56 |
| ayoung | I have a cluster of 14 machines. So different machines get different results, and yet they are all supposed to be identical. The joys of new hardware | 15:59 |
| opendevreview | Riccardo Pittau proposed openstack/ironic-python-agent master: Multipath Hardware path handling https://review.opendev.org/c/openstack/ironic-python-agent/+/837039 | 16:04 |
| rpittau | good night! o/ | 16:12 |
| opendevreview | Merged openstack/ironic-python-agent master: The Python 3.6 and Python 3.7Support has been dropped since yaga https://review.opendev.org/c/openstack/ironic-python-agent/+/841073 | 16:12 |
| iurygregory | typo :D | 16:17 |
| dtantsur | iurygregory: not necessarily: https://en.wikipedia.org/wiki/Baba_Yaga | 16:27 |
| iurygregory | LOL | 16:29 |
| iurygregory | I wasn't expecting that | 16:29 |
| * dtantsur is happy to help | 16:29 | |
| TheJulia | Oh wow | 17:02 |
| JayF | I just keep hearing the Weird Al song except "Y-A-G-A Yaaahhhggaaaa" | 17:12 |
| TheJulia | I keep thinking of angry women | 17:24 |
| * TheJulia feels just exhausted sending a bunch of emails | 17:37 | |
| opendevreview | Merged openstack/ironic-python-agent stable/yoga: Collect a full lsblk output in the ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/840937 | 17:44 |
| opendevreview | Merged openstack/ironic-python-agent bugfix/8.6: Collect a full lsblk output in the ramdisk logs https://review.opendev.org/c/openstack/ironic-python-agent/+/840938 | 17:44 |
| opendevreview | Verification of a change to openstack/ironic-python-agent stable/xena failed: Use a pre-defined partition UUID to detect configdrive on GPT https://review.opendev.org/c/openstack/ironic-python-agent/+/840348 | 17:44 |
| ayoung | last_error: 'Node failed to start the first cleaning step: Connection to agent failed: | 18:20 |
| ayoung | Failed to connect to the agent running on node af8ebf65-50cf-4aad-8799-8e5678d2574e | 18:20 |
| ayoung | for invoking command clean.get_clean_steps. Error: HTTPSConnectionPool(host=''192.168.116.47'', | 18:20 |
| ayoung | port=9999): Max retries exceeded with url: /v1/commands/?wait=true&agent_token=nKxmsok4Whd2ko9Ra5T9slzhGaEfsPUz_JznjIJK3Vw | 18:20 |
| ayoung | (Caused by SSLError(SSLCertVerificationError(1, ''[SSL: CERTIFICATE_VERIFY_FAILED] | 18:20 |
| ayoung | certificate verify failed: certificate is not yet valid (_ssl.c:1131)'')))' | 18:20 |
| ayoung | that is what I saw before. About Half the nodes failed with errors like this | 18:20 |
| TheJulia | clocks? | 18:26 |
| TheJulia | do you have a time server? | 18:26 |
| ayoung | Not that I know of | 18:27 |
| TheJulia | so, I suspect it might be that your system local clocks are just not consistent | 18:27 |
| TheJulia | we have logic in the agent to force the clock to update someplace | 18:28 |
| ayoung | OK, I can handle that. And I can get a clock server if needs be | 18:28 |
| TheJulia | https://github.com/openstack/ironic-python-agent/blob/fcb65cae18f4a6b4b05fb70677e2fa114e0558a9/releasenotes/notes/set-clock-prior-to-poweroff-af6ec210aad8b45a.yaml | 18:28 |
| TheJulia | we also save the time at power-off | 18:28 |
| NobodyCam | good afternoon Ironic folks | 19:05 |
| NobodyCam | crazy question did deploy steps work in ussuri? | 19:06 |
| TheJulia | NobodyCam: that is a great question | 19:20 |
| NobodyCam | LOL | 19:20 |
| NobodyCam | docs seem to be all over the place | 19:21 |
| TheJulia | well, clean steps were a thing, and deploy steps were almost a thing between releases | 19:21 |
| NobodyCam | yea I saw this: `Starting with the Victoria release cycle, deployment can be customized similarly to cleaning. ` | 19:22 |
| TheJulia | Yeah, I was just looking at code suggesting victoria as well | 19:22 |
| NobodyCam | so there are deploy templates | 19:23 |
| NobodyCam | `Starting with the Stein release, with Bare Metal API version 1.55, deploy templates offer a way to define a set of one or more deploy steps to be executed with particular sets of arguments and priorities.` | 19:23 |
| NobodyCam | not sure I've ever used the deploy templates | 19:24 |
| TheJulia | oh jeeze I'm trying to remember how that maps through | 19:26 |
| NobodyCam | LOL | 19:26 |
| TheJulia | so mgoddard has a video someplace demonstrating deploy time raid | 19:26 |
| NobodyCam | oh | 19:26 |
| NobodyCam | happen to know if a customer deploy template over writes the default or is added to them? | 19:28 |
| NobodyCam | *custom* | 19:28 |
| TheJulia | I have no idea | 19:30 |
| NobodyCam | we'll find out ! | 19:32 |
| NobodyCam | hehehee | 19:32 |
| NobodyCam | well I guess if I read the doc it would tell me: `During deployment, if any of the traits in a node’s instance_info.traits field match the name of a deploy template, then the steps from that deploy template will be added to the list of steps to be executed by the node.` | 19:33 |
| TheJulia | That sounds right! | 19:54 |
| TheJulia | cryptic crash of the day https://b514d6c133582c0af7f3-9355d865b880bd099576064727df95b2.ssl.cf2.rackcdn.com/841275/2/check/ipa-tempest-uefi-redfish-vmedia-src/d0afc64/controller/logs/ironic-bm-logs/node-1_console_2022-05-10-23%3A02%3A29_log.txt | 20:54 |
| opendevreview | Julia Kreger proposed openstack/networking-generic-switch master: CI: use pre-existing ssh key on multinode jobs https://review.opendev.org/c/openstack/networking-generic-switch/+/841265 | 21:00 |
| stevebaker[m] | arne_wiebalck, iurygregory : Hey the SIG video is live https://www.youtube.com/watch?v=_K-aPdKnt1Y | 22:35 |
| NobodyCam | now to figure out how to correctly add a step: `Validation of deploy steps from deploy templates matching this node's instance traits failed. Matching deploy templates: CUSTOM_ZACK_DEPLOY_STEP. Errors: node does not support this deploy step` | 22:46 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!