| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default. https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 07:55 |
|---|---|---|
| opendevreview | Matt Crees proposed openstack/kolla stable/2023.1: Move apt_prefs pins to respective images https://review.opendev.org/c/openstack/kolla/+/918973 | 08:16 |
| opendevreview | Matt Crees proposed openstack/kolla stable/2023.1: Build RabbitMQ 3.11, 3.12 and 3.13 in 2023.1 https://review.opendev.org/c/openstack/kolla/+/918974 | 08:16 |
| opendevreview | Will Szumski proposed openstack/kolla-ansible master: WIP: Try overriding docker collection https://review.opendev.org/c/openstack/kolla-ansible/+/924043 | 08:57 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Use community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 09:04 |
| mnasiadka | jovial: any idea why it's not failing in kolla-ansible CI? :) | 09:08 |
| jovial | mnasiadka, I'll take a look. I've not fully investigated that yet. Possibly you are pinning requests in your kolla-ansible venv? | 09:20 |
| mnasiadka | No - we're not using anything from community.docker :) | 09:20 |
| mnasiadka | (only in the kolla publish jobs) | 09:21 |
| jovial | Hit it on: "Fetch fluentd Docker image labels" | 09:22 |
| jovial | https://github.com/openstack/kolla-ansible/blob/362cf2bbe324b16d7a7f63db4dac7026d70c567e/ansible/roles/common/tasks/config.yml#L58 | 09:22 |
| jovial | Here is the log: https://45ab50c71482ae2fd55d-f5b371aed546eed3ac03c77e9adc0c03.ssl.cf5.rackcdn.com/923978/1/check/kayobe-overcloud-rocky9/a5af761/primary/ansible/overcloud-deploy | 09:23 |
| mnasiadka | is that master? | 09:26 |
| jovial | In theory, yes | 09:27 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Use community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 09:35 |
| jovial | mnasiadka, Do you think it would be better just to pin requests and wait for the ansible bump to bring in a new community.docker? | 09:38 |
| mnasiadka | I'm not convinced we're pinning in k-a | 09:39 |
| mnasiadka | so question why it works in k-a CI :) | 09:39 |
| jovial | I'll do a bit more digging there now :) | 09:40 |
| sylvr | Hello #openstack-kolla ! | 09:50 |
| jovial | Morning sylvr | 09:51 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Pin requests to workaround bug in community.docker https://review.opendev.org/c/openstack/kayobe/+/920269 | 09:51 |
| sylvr | jovial: Do you have any news about the regression on the IPMI driver I told you about ? The datacenter is powered on again and I can run tests on my nodes ! | 09:52 |
| sylvr | On my deployment I can see that nodes are registered into Bifrost but their missing all their driver_info (no IPMI cred' at all) | 09:58 |
| jovial | sylvr: Can you do a baremetal introspection data save <node> on one of them. I'm curious to see if it collected the info, but failed to set the property on the node. | 09:59 |
| sylvr | jovial : Introspection data not found for node 23b422ae-2184-405e-99b9-4ea3fd402851, processed=True | 10:00 |
| sylvr | I had to build my own IPA (using kayobe) to enable devuser on the nodes, this is unrelated right? | 10:01 |
| jovial | sylvr, building your own IPA shouldn't hurt | 10:02 |
| jovial | Let me check the defaults for Ironic storing the introspection data | 10:02 |
| sylvr | btw, node are stuck in enroll state (as far as I understand, this is "normal" as their driver isn't properly configured) | 10:03 |
| jovial | Correct, it won't let you transition to manageable if it fails validation | 10:04 |
| jovial | sylvr: This is the relevant option: https://github.com/openstack/bifrost/blob/stable/2024.1/playbooks/roles/bifrost-ironic-install/templates/ironic-inspector.conf.j2#L88 | 10:05 |
| jovial | Could you check ironic-inspector.conf inside your bifrost container? | 10:05 |
| sylvr | yep, the line is present | 10:07 |
| jovial | And `baremetal introspection status <node>` doesn't show an error? | 10:08 |
| sylvr | jovial : Could not find node 23b422ae-2184-405e-99b9-4ea3fd402851 in cache | 10:08 |
| jovial | I'm wondering if inspector errored out for some reason when processing the introspection data | 10:08 |
| jovial | funky | 10:09 |
| jovial | Could you check for errors in the ironic-inspector logs? | 10:09 |
| sylvr | maybe I should power cycle my nodes ? | 10:09 |
| sylvr | yep! | 10:09 |
| jovial | Also see if there are any messages when you grep for the node uuid in the logs | 10:09 |
| sylvr | on the seed there's nothing in /var/lob/ironic-inspector/ramdisk/ | 10:10 |
| jovial | Any did you explicitly register the nodes in bifrost via baremetal node create? | 10:11 |
| jovial | and* | 10:11 |
| sylvr | nop | 10:11 |
| sylvr | the plan is to use autodiscovery | 10:11 |
| jovial | Yeah, I thought so. Just wanted to check my assumptions. | 10:11 |
| sylvr | I know that sometimes there's issue when redeploying the seed-bifrost | 10:11 |
| jovial | I would delete that node and get it to PXE again. Just see if you get the same behaviour. | 10:12 |
| sylvr | and I know that my nodes are sometimes a bit capricious and I have to log on the webgui of the iDRAC to reset to PXEboot | 10:12 |
| sylvr | on it right now | 10:12 |
| jovial | sylvr, Also check: /var/log/kolla/ironic-inspector/ironic-inspector.log | 10:14 |
| sylvr | jovial : the logs : https://pastebin.com/gJvdgPN7 | 10:17 |
| PrzemekK | Do You know when new kolla images for glance will be updated to 28.1.0 ? Currentlly cat /glance-base-source/glance-28.0.2.dev27/glance/async_/flows/plugins/image_conversion.py | grep -i 'QCOW images with data-file set are not allowed' 'QCOW images with data-file set are not allowed') patch for 2024.1 https://launchpadlibrarian.net/737116834/glance-2024.1-2059809.patch | 10:18 |
| sylvr | jovial : hum... it seems I have nodes in duplicate | 10:21 |
| sylvr | if I delete them all and then power cycle them, they should register themself again correctly right ? | 10:21 |
| jovial | slyvr, yep, that should work | 10:29 |
| sylvr | jovial : my node register themself into bifrost, but nothing in the cache and still no ramdisk logs | 10:41 |
| jovial | sylvr: Any errors in /var/log/kolla/ironic-inspector/ironic-inspector.log? | 10:51 |
| sylvr | jovial : yes (sample : https://pastebin.com/6mD7NrPi ) | 10:53 |
| sylvr | jovial : https://pastebin.com/0ekcTuK3 (INFO & DEBUG) | 10:54 |
| kevko | anybody to allow https://review.opendev.org/c/openstack/kolla-ansible/+/923859 backport to be merged ? | 10:56 |
| kevko | mnasiadka ^^ frickler ^^ | 10:56 |
| sylvr | jovial : saw that in a node logs : Jul 12 06:35:34 localhost ironic-python-agent[1362]: 2024-07-12 06:35:34.243 1362 WARNING root [-] Executable 'biosdevname' not found: FileNotFoundError: [Errno 2] No such file or directory: 'biosdevname' | 11:10 |
| sylvr | it seems that the node introspect, enroll and have the heartbeat OK, but not data is saved from the introspection | 11:14 |
| sylvr | no data* | 11:15 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default. https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 11:37 |
| sylvr | jovial: well the good news is that if manually provided with the correct credentials, the node can be managed using ironic | 11:38 |
| jovial | slyvr: That sounds like a good workaround. There is definitely a bug somewhere though :D | 11:42 |
| jovial | I didn't see anything problematic in the logs | 11:43 |
| sylvr | jovial: what else can I do found this bug ? can it be a configuration issue ? | 11:46 |
| sylvr | to find* | 11:46 |
| jovial | Its hard to say why it isn't storing the introspection data without seeing an obvious in the logs. The other thing you can try is setting `always_store_ramdisk_logs` in ironic-inspector.conf and debug=True. Would have restart the bifrost container for those changes to be picked up. | 11:50 |
| jovial | It seems to me that the data is being sent back, but inspector is failing to process it properly | 11:51 |
| sylvr | I pushed my current config to my repo | 11:52 |
| sylvr | is it necessary if I can directly log on the ramdisk ? | 11:52 |
| jovial | sylvr: No, just checking journalctl -u ironic-python-agent is probably enough. Did we get any errors on the client side? | 11:53 |
| sylvr | yes | 11:53 |
| sylvr | Jul 12 06:35:34 localhost ironic-python-agent[1362]: 2024-07-12 06:35:34.243 1362 WARNING root [-] Executable 'biosdevname' not found: FileNotFoundError: [Errno 2] No such file or directory: 'biosdevname' | 11:53 |
| jovial | That one can largely be ignored, but you could install that package in your IPA image | 11:54 |
| sylvr | Jul 12 06:35:04 localhost.localdomain ironic-python-agent[1362]: 2024-07-12 06:35:00.636 1362 DEBUG root [-] Get driver for interface lo fail | 11:55 |
| sylvr | ed. Error: Unexpected error while running command. | 11:55 |
| sylvr | Jul 12 06:35:04 localhost.localdomain ironic-python-agent[1362]: 2024-07-12 06:35:00.648 1362 DEBUG root [-] No iscsi connection detected. Sk | 11:55 |
| sylvr | ipping iscsi. Error: Unexpected error while running command. | 11:55 |
| sylvr | Jul 12 06:35:06 localhost.localdomain ironic-python-agent[1362]: 2024-07-12 06:35:00.879 1362 DEBUG ironic_lib.utils [-] Command failed: Unex | 11:55 |
| sylvr | pected error while running command. | 11:55 |
| sylvr | Jul 12 06:35:06 localhost.localdomain ironic-python-agent[1362]: 2024-07-12 06:35:00.897 1362 WARNING root [-] Attempted to determine if mult | 11:55 |
| sylvr | ipath tools were present. Not detected. Error recorded: [Errno 2] No such file or directory: 'multipath': FileNotFoundError: [Errno 2] No suc | 11:55 |
| sylvr | h file or directory: 'multipath' | 11:55 |
| jovial | Those errors aren't fatal. It is just trying to collect as much information as possible. Of course if you are using multipathing, you may wish to install those packages | 11:56 |
| jovial | You are more looking for an error after it posts the data to Ironic | 11:57 |
| sylvr | well, the IPAs seem clean, let's go check the ironic server then :D | 11:58 |
| jovial | Yeah, I think debug logs could help for that | 11:59 |
| sylvr | jovial : well /var/log doesn't contain the ironic directory on my seed, journalctl -xeu ironic return no logs (just start up service) | 12:02 |
| jovial | slyvr: its /var/log/kolla that you need to be looking in | 12:07 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Use community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 12:10 |
| sylvr | jovial: got confused as there is a /var/log/ironic-inspector directory ^^ yeah there's lot of error, I'm preparing the pastebin with the full logs or just the error ? (grep -i error) | 12:10 |
| sylvr | jovial : grep -i error /var/log/kolla/ironic/ironic.log : https://pastebin.com/2U3H4irD | 12:12 |
| jovial | Hmm, nothing particularly alarming | 12:14 |
| jovial | Do you transport_url=fake:// in your ironic-inspector.conf | 12:15 |
| sylvr | yes | 12:16 |
| sylvr | written the same | 12:17 |
| sylvr | forgot to ask, do you want me to send the /etc/ironic-inspector/inspector.conf ? | 12:17 |
| jovial | I'm mostly out of ideas. You have more luck asking on the mailing list or in #openstack-ironic | 12:18 |
| sylvr | going to the IRC, thanks for everything, so the issue seems to be that the ironic server is unable to store the data from the inspection right ? | 12:19 |
| jovial | Yeah, from what I can tell, and it might be more obvious with the full inspector logs, it is receiving the data posted from ironic-python-agent but failing to fully process it and store it | 12:21 |
| sylvr | okay, thanks, I'll keep you in touch with this issue | 12:21 |
| jovial | Cheers, it would be good to update the bug you posted if you get it resolved. Worth commenting with the workaround of manually setting ipmi_address too. | 12:22 |
| jovial | sylvr: No errors like this one:https://github.com/openstack/ironic-inspector/blob/e96ee4b5ffc44c70ad601a7029c9cad20e93dc4b/ironic_inspector/process.py#L176 ? | 12:23 |
| sylvr | jovial: doesn't seems so, I'll update the bug report | 12:26 |
| opendevreview | Will Szumski proposed openstack/kayobe master: WIP: Use community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 12:50 |
| jovial | thanks :) | 12:50 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla master: Add ovn-bgp-agent / FRR / Horizon BGPVPN dashboard https://review.opendev.org/c/openstack/kolla/+/891617 | 13:18 |
| opendevreview | Maximilian Sesterhenn proposed openstack/kolla-ansible master: WIP: Add ovn-bgp-agent / FRR / neutron-bgpvpn / Horizon BGPVPN dashboard support https://review.opendev.org/c/openstack/kolla-ansible/+/891622 | 13:22 |
| jovial | mnasiadka, To answer your earlier question, I found k-a has this: https://github.com/openstack/kolla-ansible/blob/master/requirements-core.yml and in CI, we install ansible-core only, so trigger this installation of these requirements with kolla-ansible install-deps | 14:41 |
| jovial | Was think that we should use install-deps in kayobe | 14:41 |
| jovial | thinking* | 14:41 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Use explicit version of community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 15:17 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Pin requests to workaround bug in community.docker https://review.opendev.org/c/openstack/kayobe/+/920269 | 15:40 |
| opendevreview | Petr Slavchenkov proposed openstack/kolla-ansible stable/2023.1: Add ca_path for module uri in opensearch role https://review.opendev.org/c/openstack/kolla-ansible/+/924080 | 16:21 |
| opendevreview | Petr Slavchenkov proposed openstack/kolla-ansible stable/2023.1: Add ca_path for module uri in opensearch role https://review.opendev.org/c/openstack/kolla-ansible/+/924080 | 16:28 |
| opendevreview | Will Szumski proposed openstack/kayobe master: Use explicit version of community.docker https://review.opendev.org/c/openstack/kayobe/+/923978 | 17:53 |
| kevko | jovial: i think we should add ansible>= some_version into kolla-ansible | 18:07 |
| kevko | i really don't know why we don't have this in requirements | 18:07 |
| kevko | as kolla-ansible has ansible also in the name :D | 18:07 |
| jovial | Its certainly a bit odd not to install it in requirements. I guess it does allow you to choose between ansible-core and full fat ansible. | 18:11 |
| jovial | But could we not just have ansible-core in there? Since that is a dependency of ansible? | 18:11 |
| kevko | jovial: i think you need to have ansible installed always .. | 18:26 |
| kevko | jovial: ansible-core is not too much | 18:27 |
| kevko | *not enough | 18:27 |
| kevko | jovial: for example git module is from ansible not ansible-core , for example used in ansible/roles/nova-cell/tasks/clone.yml << | 18:32 |
| opendevreview | Rafal Lewandowski proposed openstack/kolla-ansible master: [WIP] Enable ML2/OVN and distributed FIP by default. https://review.opendev.org/c/openstack/kolla-ansible/+/904959 | 22:19 |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!