| opendevreview | Merged openstack/ironic stable/xena: Explicitly pin upper sushy version https://review.opendev.org/c/openstack/ironic/+/820923 | 00:27 |
|---|---|---|
| opendevreview | Steve Baker proposed openstack/bifrost master: Use distro conf loader_file_paths to copy ipxe/grub files https://review.opendev.org/c/openstack/bifrost/+/818709 | 00:54 |
| opendevreview | Steve Baker proposed openstack/bifrost master: Don't create ironic_tftp_master_path directory, change default https://review.opendev.org/c/openstack/bifrost/+/820311 | 00:54 |
| opendevreview | Steve Baker proposed openstack/bifrost master: Support PXE network boot with grub https://review.opendev.org/c/openstack/bifrost/+/807220 | 00:54 |
| opendevreview | Julia Kreger proposed openstack/ironic bugfix/18.1: Explicitly pin upper sushy version https://review.opendev.org/c/openstack/ironic/+/820927 | 02:44 |
| * TheJulia yawns | 02:51 | |
| arne_wiebalck | Good morning, Ironic! | 07:16 |
| jingvar | good morning | 07:22 |
| rpittau | good morning ironic! o/ | 07:42 |
| opendevreview | Verification of a change to openstack/ironic master failed: All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 07:51 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add "none" RPC transport that disables the RPC bus https://review.opendev.org/c/openstack/ironic/+/820036 | 07:51 |
| opendevreview | Verification of a change to openstack/ironic master failed: All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 08:13 |
| dtantsur | morning ironic | 08:18 |
| dtantsur | today is another cold day, send hugs | 08:33 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add "none" RPC transport that disables the RPC bus https://review.opendev.org/c/openstack/ironic/+/820036 | 08:36 |
| dtantsur | openstack --os-cloud port delete 3ef922de-3223-43d4-a725-13de8595226c | 08:42 |
| dtantsur | 2021-12-07 19:05:41.862 | Cloud port was not found. | 08:42 |
| dtantsur | wtf | 08:42 |
| dtantsur | TheJulia: I think we need a default for OS_CLOUD | 08:42 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: devstack: provide a default for OS_CLOUD https://review.opendev.org/c/openstack/ironic/+/820985 | 08:46 |
| dtantsur | TheJulia, stevebaker[m] ^^ | 08:46 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: devstack: provide a default for OS_CLOUD https://review.opendev.org/c/openstack/ironic/+/820985 | 08:49 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic-inspector master: DNM test the CI https://review.opendev.org/c/openstack/ironic-inspector/+/820987 | 08:50 |
| opendevreview | Merged openstack/sushy stable/train: Raise an AccessError with SessionService init https://review.opendev.org/c/openstack/sushy/+/820603 | 09:01 |
| janders | good morning arne_wiebalck jingvar rpittau dtantsur and Ironic o/ | 09:26 |
| rpittau | hey janders :) | 09:26 |
| dtantsur | This is a fun one, rings any bells? https://zuul.opendev.org/t/openstack/build/d08dcb22f70c4848965e6831f007464d/log/controller/logs/screen-ir-cond.txt#1234 | 09:28 |
| arne_wiebalck | hey janders o/ | 09:38 |
| opendevreview | Merged openstack/sushy stable/ussuri: Raise an AccessError with SessionService init https://review.opendev.org/c/openstack/sushy/+/820602 | 09:39 |
| jingvar | pain https://github.com/canonical/cloud-init/blob/main/cloudinit/sources/DataSourceNoCloud.py#L119-L129 | 09:41 |
| dtantsur | folks, please review https://review.opendev.org/c/openstack/ironic/+/820985/ it's needed for the Inspector CI | 10:09 |
| opendevreview | Derek Higgins proposed openstack/ironic-python-agent master: IPA instructs qemu-img to write image zeros to disk. https://review.opendev.org/c/openstack/ironic-python-agent/+/820997 | 10:10 |
| mnasiadka | Hello | 11:10 |
| mnasiadka | Another weird MD raid deployment issue - it seems partx -a has some problem - https://paste.opendev.org/show/811536/ | 11:11 |
| mnasiadka | And since that fails, the EFI partition doesn't get created (outside of MD) | 11:11 |
| mnasiadka | arne_wiebalck: ^^ - thoughts? | 11:11 |
| dtantsur | Oo | 11:11 |
| dtantsur | software RAID will never start being not funny | 11:11 |
| mnasiadka | not funny is HPE delivering a ,,RAID controller'' that works only in legacy BIOS mode (hence probably a fakeraid) ;) | 11:12 |
| dtantsur | wow | 11:16 |
| arne_wiebalck | mnasiadka: off hand, I have no idea ... any chance for you to log into that box and run things by hand to see if we get some more information? | 11:17 |
| arne_wiebalck | mnasiadka: mind you, we are using that patch with md raid | 11:18 |
| arne_wiebalck | mnasiadka: I guess this is reproducible for you on more than 1 box? | 11:18 |
| mnasiadka | arne_wiebalck: I'm logged in after reboot (to IPA) - there is free space for EFI partition on the hdd - I can create everything by hand - so a bit weird | 11:18 |
| mnasiadka | arne_wiebalck: well, currently it's just 1 box - but undeploy and deploy ran multiple times and ends up like that. | 11:19 |
| arne_wiebalck | mnasiadka: and partx -a also works then? | 11:19 |
| arne_wiebalck | mnasiadka: manually, I mean | 11:19 |
| mnasiadka | I was sure it does, but it doesn't - fails the same way | 11:19 |
| mnasiadka | but only on one of the two disks in raid1/md | 11:20 |
| mnasiadka | and now it started working, hmm | 11:21 |
| mnasiadka | I'll fix this one by hand and see if it happens anytime soon... | 11:21 |
| arne_wiebalck | mnasiadka: what do you mean by "on one of the two" ? | 11:22 |
| mnasiadka | arne_wiebalck: argh, I tried partx on the physical disk - but yes, for md0 it still fails the same way. | 11:23 |
| arne_wiebalck | mnasiadka: I guess that is good :) | 11:24 |
| mnasiadka | with -v - it says device or resource busy | 11:25 |
| mnasiadka | partx -u although works | 11:28 |
| dtantsur | I suspect that's because `partx -u` doesn't do much | 11:29 |
| mnasiadka | https://review.opendev.org/c/openstack/ironic-python-agent/+/816941/1/ironic_python_agent/extensions/image.py | 11:30 |
| mnasiadka | here's an explanation | 11:30 |
| mnasiadka | ok then, but still it fails re-adding the partitions, because "Device or resource is busy" | 11:31 |
| arne_wiebalck | mnasiadka: is that since the device is an md device? | 11:39 |
| mnasiadka | tried other working hosts, and it seems partx -a fails on them as well - so that's probably a valid assumption | 11:41 |
| mnasiadka | wonder how did it get part this section and set up EFI partitions - or maybe it's unrelated. | 11:41 |
| mnasiadka | but there's nothing else after that in journal uploaded to Ironic server... | 11:42 |
| mnasiadka | Is there a good way to stop Ironic from rebooting a server after deploying? | 11:43 |
| dtantsur | there should be, I think | 11:43 |
| dtantsur | mnasiadka: [deploy]power_off_after_deploy_failure | 11:44 |
| mnasiadka | but it's not failing :D | 11:45 |
| mnasiadka | the state is active and it's rebooting | 11:45 |
| mnasiadka | into IPA again (since it can't boot from local disk, no EFI partitions visible) | 11:45 |
| dtantsur | Oo | 11:48 |
| janders | shooting from the hip: 1) use fake power driver (just for this test) and 2) change IPMI creds at the right time maybe? | 11:49 |
| janders | both ugly but maybe workable? | 11:49 |
| janders | (I think I remember doing things like this for troubleshooting back in the day :) ) | 11:50 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Allow enabling fast-track per node https://review.opendev.org/c/openstack/ironic/+/821010 | 11:56 |
| opendevreview | Derek Higgins proposed openstack/ironic-python-agent master: Instruct qemu-img to write image zeros to disk. https://review.opendev.org/c/openstack/ironic-python-agent/+/820997 | 12:44 |
| opendevreview | Merged openstack/ironic master: devstack: provide a default for OS_CLOUD https://review.opendev.org/c/openstack/ironic/+/820985 | 12:49 |
| dtantsur | we're seeing increasing cases of a new (?) problem in the CI: deployed nodes refuse to boot from disk: https://9eed2832462b93d99d63-9a2c0e6b238d9eb37d7a57d0c5074dee.ssl.cf1.rackcdn.com/819620/11/gate/ironic-standalone-redfish/930f51b/controller/logs/ironic-bm-logs/node-2_console_log.txt | 12:58 |
| dtantsur | (it's a good question why we even use netboot with whole disk images) | 12:59 |
| *** outbrito_ is now known as outbrito | 13:02 | |
| opendevreview | Verification of a change to openstack/ironic master failed: All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 13:09 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add "none" RPC transport that disables the RPC bus https://review.opendev.org/c/openstack/ironic/+/820036 | 13:09 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Allow enabling fast-track per node https://review.opendev.org/c/openstack/ironic/+/821010 | 13:23 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: Allow enabling fast-track per node https://review.opendev.org/c/openstack/ironic/+/821010 | 13:27 |
| opendevreview | Dmitry Tantsur proposed openstack/ironic master: WIP automatically configure interfaces https://review.opendev.org/c/openstack/ironic/+/820909 | 13:28 |
| opendevreview | Verification of a change to openstack/ironic master failed: All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 13:28 |
| opendevreview | Verification of a change to openstack/ironic master failed: Add "none" RPC transport that disables the RPC bus https://review.opendev.org/c/openstack/ironic/+/820036 | 13:28 |
| TheJulia | dtantsur: that is just weird... | 15:00 |
| dtantsur | the weird world is weird | 15:00 |
| dtantsur | good morning TheJulia | 15:00 |
| opendevreview | Riccardo Pittau proposed openstack/sushy master: [trivial] fix Xena release versions https://review.opendev.org/c/openstack/sushy/+/821081 | 15:02 |
| dtantsur | we're seeing an increased number of CI failures, at least some of them are PXE-related | 15:24 |
| dtantsur | ronic] Failed to bind port bf00128d-fb1f-4b6e-bafb-b7e40dfcc65c on host 692792d5-980b-4e37-9bfe-2bd06ccaf08b for vnic_type baremetal using segments [{'id': '8e5ad010-b8f6-451f-916f-4e3c33fc203d', 'network_type': 'vxlan', 'physical_network': None, 'segmentation_id': 310, 'network_id': 'be35c3a8-79ad-4bd9-b696-08d147e73654'}] | 15:26 |
| dtantsur | well, of course | 15:26 |
| dtantsur | Refusing to bind due to unsupported vnic_type: baremetal | 15:27 |
| opendevreview | Merged openstack/ironic-inspector master: Add support for state selector in the list introspection https://review.opendev.org/c/openstack/ironic-inspector/+/807578 | 15:31 |
| dtantsur | okay, we've got something to release for ironic-inspector :) | 15:31 |
| rpittau | \o/ | 15:42 |
| rpittau | dtantsur: I don't see anything else for inspector at the moment, I'm going to prepare the release | 15:52 |
| opendevreview | Derek Higgins proposed openstack/ironic-python-agent master: Instruct qemu-img to write image zeros to disk. https://review.opendev.org/c/openstack/ironic-python-agent/+/820997 | 15:56 |
| TheJulia | unsupported vnic type, what?!? | 16:06 |
| TheJulia | dtantsur: do you have an example where you saw unsupported vnic type ? | 16:09 |
| TheJulia | dtantsur: so regex queries in kibana are not yielding anything, a pointer would be helpful w/r/t vnic binding | 16:25 |
| opendevreview | Merged openstack/ironic master: All-in-one Ironic service with a local RPC bus https://review.opendev.org/c/openstack/ironic/+/819620 | 16:35 |
| opendevreview | Arne Wiebalck proposed openstack/ironic-python-agent master: Burn-in: Add options for named log files https://review.opendev.org/c/openstack/ironic-python-agent/+/821098 | 16:45 |
| opendevreview | Arne Wiebalck proposed openstack/ironic-python-agent master: Burn-in: Add options for named log files https://review.opendev.org/c/openstack/ironic-python-agent/+/820032 | 16:49 |
| TheJulia | hmm oom-killer in ci | 17:08 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: reduce api worker processes to 1 https://review.opendev.org/c/openstack/ironic/+/821103 | 17:20 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/xena: CI: reduce api worker processes to 1 https://review.opendev.org/c/openstack/ironic/+/821037 | 17:21 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/xena: Remove redfish cache entry upon errors https://review.opendev.org/c/openstack/ironic/+/820588 | 17:22 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Disable neutron metadata service https://review.opendev.org/c/openstack/ironic/+/821105 | 17:24 |
| jingvar | guys! can you help? | 17:34 |
| jingvar | TASK [bifrost-ironic-install : Validate interface broadcast addresses] | 17:34 |
| jingvar | fatal: [127.0.0.1]: FAILED! => {"changed": false, "msg": "Interface pxe broadcast incoherence 10.228.18.255/24 vs False/False\n"} | 17:34 |
| jingvar | : pxe@mgmt inet 10.228.18.24/24 scope global pxe | 17:35 |
| jingvar | what is wrong? | 17:35 |
| rpittau | good night! o/ | 17:36 |
| jingvar | my falt | 17:38 |
| TheJulia | jingvar: broadcast variable issue? | 17:38 |
| jingvar | I didnt update env ips | 17:39 |
| jingvar | but I have another question with cloud init and netpaln | 17:40 |
| TheJulia | no idea w/r/t cloud-init | 17:40 |
| jingvar | https://paste.opendev.org/show/811556/ | 17:44 |
| jingvar | cloud init reder it to netplan, but up only pxe interface | 17:44 |
| arne_wiebalck | bye everyone o/ | 17:48 |
| TheJulia | dtantsur: do you remember any issues where the agent kind of looses it's mind in the exchange with the conductor https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_417/820934/1/check/ironic-standalone/4179e39/controller/logs/ironic-bm-logs/node-0_console_log.txt ? | 17:53 |
| TheJulia | jingvar: sorry, I never really use cloud-init. Maybe someone else here does | 17:54 |
| TheJulia | dtantsur: any ideas w/r/t butfix/18.1 https://zuul.opendev.org/t/openstack/build/b8524a50851e4e78958b8d2b115b96cf would be appreciated | 17:56 |
| TheJulia | works just fine elsewhere | 17:56 |
| TheJulia | Wheeeee: - Failed: 6066 | 18:01 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/train: Remove redfish cache entry upon errors https://review.opendev.org/c/openstack/ironic/+/820937 | 18:34 |
| opendevreview | Merged openstack/ironic master: Add "none" RPC transport that disables the RPC bus https://review.opendev.org/c/openstack/ironic/+/820036 | 18:57 |
| NobodyCam | Good Morning OpenStack Folks! | 19:04 |
| NobodyCam | Happy Hump Day | 19:04 |
| TheJulia | good morning NobodyCam | 19:16 |
| opendevreview | Merged openstack/ironic master: Use test_with_retry to get the tap device name https://review.opendev.org/c/openstack/ironic/+/818703 | 19:27 |
| opendevreview | Merged openstack/ironic-python-agent master: Prepare for bugfix release https://review.opendev.org/c/openstack/ironic-python-agent/+/820489 | 19:27 |
| opendevreview | Julia Kreger proposed openstack/ironic stable/xena: CI: Lower test VM memory by 200MB https://review.opendev.org/c/openstack/ironic/+/821120 | 19:37 |
| TheJulia | So I think the oom-killer is making us sad | 19:48 |
| TheJulia | in the last 7 days, oom-killer has struck ironic 48 times | 19:53 |
| TheJulia | The only project which exceeds that is Horizon | 19:54 |
| stevebaker[m] | good morning | 20:01 |
| TheJulia | o/ stevebaker[m] | 20:03 |
| * TheJulia wonders if there are any oom-killer gifs out there | 20:03 | |
| TheJulia | https://memegenerator.net/img/instances/24021474.jpg | 20:06 |
| * TheJulia steps away while waiting on CI | 20:08 | |
| *** bodgix3 is now known as bodgix | 20:10 | |
| *** dking is now known as Guest8016 | 20:43 | |
| *** Guest8016 is now known as dking | 20:44 | |
| dking | If I wanted to fill a fake instance of Inspector with fake data for testing, how might I go about doing that? What are the other minimal services that must be running, and how much functionality must they have? | 20:46 |
| dking | I know that I'll at least need the db. I'm assuming also a working Ironic-API with its own db with fake data, and maybe Ironic-Conductor? | 20:47 |
| TheJulia | dking: yeah, ironic-conductor if an object needs to be created | 20:50 |
| dking | I'm trying to figure that one out. I tried running /continue, but it tells me that "Resource temporarily unavailable". Of course it is, it's not a real server. I just want to put data in and test getting it back out. Is there something it needs from the Conductor? | 20:52 |
| dking | I'm trying to find the minimal fake that I can do for testing purposes. | 21:24 |
| stevebaker[m] | Alright, I'm off to a work xmas thing. See y'all tomorrow | 21:58 |
| TheJulia | stevebaker[m]: enjoy! | 21:58 |
| stevebaker[m] | will do :) | 21:59 |
| opendevreview | Julia Kreger proposed openstack/ironic master: CI: Lower test VM memory by 200MB https://review.opendev.org/c/openstack/ironic/+/821047 | 22:43 |
Generated by irclog2html.py 2.17.2 by Marius Gedminas - find it at https://mg.pov.lt/irclog2html/!
