| holtgrewe | Good morning. Is it possible to have the ironic inspector to create port groups for me? | 06:27 |
|---|---|---|
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add more sources to redfish firmware upgrade https://review.opendev.org/c/openstack/ironic/+/822781 | 07:35 |
| opendevreview | Aija Jauntēva proposed openstack/ironic master: Add known issue for iDRAC Swift firmware update https://review.opendev.org/c/openstack/ironic/+/822782 | 07:37 |
| zigo | Hi there! Is there anyone that knows what's going on with this bug? | 08:22 |
| zigo | https://bugs.debian.org/1002407 | 08:22 |
| zigo | It's been filled together with many others by Lucas Nussbaum after a full rebuild of the Debian archive. | 08:23 |
| zigo | There's 44 similar unit test errors when I build the package. | 08:23 |
| rpittau | good morning ironic! o/ | 08:34 |
| rpittau | zigo: that's due to Pyth on3.10 | 08:35 |
| rpittau | zigo: a fix is in https://review.opendev.org/c/openstack/ironic/+/821370 | 08:35 |
| zigo | rpittau: Ah, thanks, trying ! | 08:36 |
| dmellado | morning all ;) | 08:58 |
| dmellado | rpittau: o/ | 08:59 |
| rpittau | hey dmellado :) | 08:59 |
| dmellado | do you happen to know how can I set the image to have a predefined ipv4? | 08:59 |
| dmellado | when booting it all goes well, but at some point it fails to get a lease from dhcp | 09:00 |
| dmellado | I thought thatit would get ipv4_address from the json file, but maybe I'm missing something | 09:00 |
| rpittau | dmellado: which image? | 09:03 |
| zigo | rpittau: I tried backporting your patch to Xena, and still got 2 unit test failures: | 09:04 |
| zigo | https://paste.opendev.org/show/811825/ | 09:04 |
| zigo | Could you help me with these 2 ? | 09:05 |
| rpittau | zigo: let me check | 09:05 |
| zigo | Or maybe I can just blacklist them ... if it's only a problem with tests, and Ironic otherwise works well, that is fine too. | 09:06 |
| rpittau | zigo: we can fix that, although I'm not sure xena is supposed to run with python 3.10 :) | 09:06 |
| zigo | The 2nd one that fails in what I patched, is probably because I had to manually rebase the tests because it wouldn't apply. | 09:06 |
| rpittau | yeah, there' something else missing there | 09:07 |
| zigo | rpittau: No choice, Python 3.10 is in Debian Unstable ... :) | 09:07 |
| rpittau | ah kk | 09:07 |
| zigo | rpittau: I think it's fine, I'll just blacklist the 2 broken unit tests, since it's only a problem with tests and not the rest of everything. Don't bother ... :) | 09:07 |
| rpittau | zigo: sounds good, we're waiting to have the py3.10 test up and running to verify we got everything https://review.opendev.org/c/openstack/openstack-zuul-jobs/+/821863 | 09:08 |
| zigo | Good, good. | 09:08 |
| dmellado | rpittau: centos8 | 09:09 |
| zigo | The infra is always lagging a bit behind what we do in Debian, so I'm always getting this kind of stuff in my face. Though no way to fix that, it's a chicken and egg problem... :) | 09:09 |
| zigo | ie: py 3.10 must be there in Debian, so it can reach Ubuntu, and then be tested in OpenStack infra. | 09:09 |
| rpittau | dmellado: ok, so is that an ipa-ramdisk? I would need a bit more context, although it sounds like the issue is on the dhcp server config itself | 09:09 |
| rpittau | zigo: I get it, and yes py3.10 is close to be made available for focal, hopefully | 09:10 |
| rpittau | dmellado: or network, would be great to see the logs | 09:11 |
| zigo | FYI, Doko (ie: Mathias Klose, the Python interpreter package maintainer for Canonical) always do his work in Debian Unstable first... which is why I'm affected before everyone else! :) | 09:12 |
| rpittau | well that must be fun :D | 09:13 |
| zigo | You want fun? ~25 failed to build reported bugs against all my packages after Python 3.10 upgrade and a Debian archive full rebuild... :P | 09:16 |
| zigo | I already managed to close 9 of them yesterday... and slowly recovering ... | 09:16 |
| zigo | Many collections -> collections.abc fixup. | 09:16 |
| zigo | Soon going after https://bugs.debian.org/1002415 (looks like a testtool upgrade fun as well... likely easy to fix). | 09:17 |
| rpittau | I think we got all of the collections thing change, luckily that got some deprecations warnings | 09:19 |
| dmellado | rpittau: so, which logs would you like to see? | 09:20 |
| dmellado | I was checking the idrac viewer | 09:21 |
| dmellado | let me paste you my config around and I'll explain, hopefully xD | 09:21 |
| rpittau | alright :) | 09:21 |
| dmellado | https://paste.openstack.org/show/811827/ | 09:23 |
| opendevreview | Riccardo Pittau proposed openstack/ironic master: [WIP] replace mkisofs with pycdlib https://review.opendev.org/c/openstack/ironic/+/819121 | 09:23 |
| dmellado | so, those are the settings from group/vars/target and the baremetal.json | 09:23 |
| dmellado | I can get to the ipmi | 09:24 |
| dmellado | it reboots, gets on pxe mode and boot | 09:24 |
| dmellado | I had to change from tinyipa to normal ipa as per a kernel panic | 09:24 |
| dmellado | then it loadas boot.ipxe just fine | 09:24 |
| dmellado | kernel | 09:24 |
| dmellado | and then it gets to the ramdisk, loads | 09:24 |
| dmellado | and gets stuck on the dhcp | 09:24 |
| dmellado | let me try to boot and capture some screen from the idrac | 09:24 |
| dmellado | unless you tell me that there's some log that I could fetch | 09:25 |
| dmellado | I was checking conductor and api logs | 09:25 |
| dmellado | and didn't see anything really relevant | 09:25 |
| rpittau | ok so you're building your ipa ramdisk on the fly with centos 8 stream, that looks correct at a glance | 09:25 |
| dmellado | on provisioning state | 09:25 |
| rpittau | dmellado: no ideally you would want the console logs from the ipa-ramdisk boot | 09:25 |
| dmellado | it says deploying failed | 09:25 |
| dmellado | and if I get to show | 09:26 |
| dmellado | oh, wait, the console says it's there? | 09:28 |
| dmellado | but why does ironic say it failed? | 09:28 |
| dmellado | xD | 09:28 |
| rpittau | you should have the reason in the ironic logs | 09:28 |
| dmellado | https://paste.openstack.org/show/bXt4VcllSx2fYVqu9lxM/ | 09:29 |
| dmellado | which ironic logs should I search for? | 09:29 |
| rpittau | check conductor first | 09:31 |
| dmellado | hmm I don't see anything that triggers an ERROR | 09:34 |
| dmellado | also, IIUC my ssh should be picked from baremetal/localhost group_vars? | 09:35 |
| dmellado | or do I need to put that as well on target? | 09:35 |
| dmellado | Dec 23 03:34:56 jumphost2.dfwt5g.lab ironic-conductor[3472035]: 2021-12-23 03:34:56.906 3472035 INFO ironic.conductor.task_manager [req-87b06b53-cc08-4793-b6db-27337e8cd68b bifrost_user - - - -] Node ef4bbe4a-3ecd-4bae-888e-52c406707206 moved to provision state "wait call-back" from state "deploying"; target provision state is "active" | 09:36 |
| dmellado | it gets there, now let's see | 09:36 |
| dmellado | rpittau: so, captured stufff | 09:52 |
| dmellado | ironic python agent unable to connect to http://192.167.122.1:8082/deployment_image.qcow2 blah (which is my http server) | 09:52 |
| dmellado | but the image did get deployed | 09:53 |
| dmellado | it failed to start dhcp interface ens7f0 | 09:53 |
| dmellado | should I assume that somehow pxe worked but then boom? | 09:53 |
| dmellado | I see a centos stream 8 asking for login if I just hit enter | 09:53 |
| rpittau | dmellado: yes, the pxe definitely worked because the server booted, then dhclient failed for some reason, you need to understand why | 09:55 |
| rpittau | are you sure the centos8 login is not from the ipa ramdisk? if it failed starting dhcp the ipa probably failed as well | 09:55 |
| dmellado | oh, I see | 09:55 |
| dmellado | the ipa is based on centos8 now | 09:55 |
| rpittau | I'm actually dealing with a similar issue on a debian ipa ramdisk | 09:55 |
| dmellado | should be the ipa then, as I can't even login | 09:55 |
| dmellado | do you have any default login for the ipa so I can check? | 09:55 |
| rpittau | yep, so probablt the same issue I'm facing :) | 09:56 |
| dmellado | does it also fail on dhcp? | 09:56 |
| rpittau | well dhcp seems to never start | 09:56 |
| rpittau | I'm still troubleshooting | 09:56 |
| dmellado | on the client, I assume | 09:56 |
| rpittau | there is no default login for ipa | 09:56 |
| dmellado | sigh | 09:56 |
| dmellado | rpittau: let me know if I can somehow help | 09:56 |
| rpittau | you need to rebuild the ipa ramdisk with the devuser element or manually edit the ramdisk | 09:57 |
| rpittau | dmellado: https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#patching-the-deploy-ramdisk | 09:57 |
| dmellado | rpittau: as I said these days, I feel like getting there and learning lots xD | 09:57 |
| rpittau | well that's good! :D | 09:57 |
| dmellado | but I have to say that I thought this was going to be much more straightforward lol | 09:59 |
| dmellado | rpittau: do you have a bug filled out for the issue so I can track? | 10:00 |
| rpittau | dmellado: we don't, I guess I should create one as we have already fixed another related issue :) | 10:01 |
| dmellado | awesome, thanks! | 10:04 |
| rpittau | dmellado: can you try and get the ipa logs please ? just want to see if the error is the same | 10:06 |
| dmellado | rpittau: sure, how would I get those? I got two crappy screenshots... | 10:07 |
| dmellado | can send that to you over email, but it kinda sucks xD | 10:07 |
| rpittau | dmellado: https://docs.openstack.org/ironic/latest/admin/troubleshooting.html#retrieving-logs-from-the-deploy-ramdisk | 10:08 |
| opendevreview | Riccardo Pittau proposed openstack/bifrost master: Install pip package in dib based images https://review.opendev.org/c/openstack/bifrost/+/821223 | 10:09 |
| dmellado | d'oh someone screwed up the access to the lab.... | 10:10 |
| rpittau | \o/ | 10:10 |
| dmellado | rpittau: sent that to you over gchat | 10:12 |
| dmellado | but these are CRAPPY screenshots | 10:12 |
| rpittau | dmellado: yeah, impossible to say what's wrong with dhcp with that :/ | 10:15 |
| rpittau | my suggestion si to patch the ramdisk and restart the node | 10:15 |
| rpittau | add a user and login via console | 10:15 |
| rpittau | since the node is not getting an ip you're also not getting the logs from ipa :/ | 10:16 |
| dmellado | yeah, I'll do that and see, sigh | 10:27 |
| dmellado | I'll check the bifrost bugs to check your as well | 10:28 |
| dmellado | brb | 10:28 |
| opendevreview | Merged openstack/bifrost master: Change the default image to a DIB-built one https://review.opendev.org/c/openstack/bifrost/+/822751 | 10:39 |
| janders | I'm off for the year! Merry Christmas to all of you who are celebrating and Happy New Year 2022 to all! See you next year Ironic o/ | 12:11 |
| rpittau | see ya tomorrow o/ | 16:27 |
| *** akahat|ruck is now known as akahat|out | 17:57 | |
| *** sshnaidm is now known as sshnaidm|afk | 21:57 | |
Generated by irclog2html.py 2.17.3 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!