| *** ociuhandu has quit IRC | 00:06 | |
| *** ociuhandu has joined #openstack-ironic | 00:42 | |
| *** ociuhandu has quit IRC | 00:58 | |
| *** rloo has quit IRC | 01:15 | |
| *** goldyfruit_ has joined #openstack-ironic | 01:17 | |
| *** ociuhandu has joined #openstack-ironic | 01:23 | |
| *** Lucas_Gray has quit IRC | 01:37 | |
| *** ociuhandu has quit IRC | 01:40 | |
| *** ociuhandu has joined #openstack-ironic | 01:51 | |
| kaifeng | re discussion on L3 spec above, it seems we are in a crossroad of dhcp-less deployment and dhcp-less deployment/instance | 02:01 |
|---|---|---|
| kaifeng | i think that will be an important scope definition needs to be determined first. | 02:01 |
| kaifeng | i didn't advocate for portgroup support, but rather leaving a room for furture enhancement. | 02:02 |
| kaifeng | besides we don't have ngs support for that, though we have a modified ngs to do the switch config. | 02:04 |
| kaifeng | write our custom tooling is fine, it doesn't need too much work as long as we are doing dhcp-less deployment, in which case we don't need to apply the network config persistently to the system. | 02:06 |
| kaifeng | but if we want to support dhcp-less deployment/instance, then we need to consider various configuration methods in different distros. | 02:07 |
| kaifeng | in such case, common tools like cloudinit would be a better solution, which in return force use to adopt a compatible format of network config. | 02:10 |
| kaifeng | s/use/us/ | 02:11 |
| *** ociuhandu has quit IRC | 02:17 | |
| *** rh-jelabarre has quit IRC | 02:28 | |
| *** ociuhandu has joined #openstack-ironic | 02:49 | |
| *** yedongcan has joined #openstack-ironic | 02:53 | |
| *** ociuhandu has quit IRC | 03:09 | |
| *** ociuhandu has joined #openstack-ironic | 03:10 | |
| *** ociuhandu has quit IRC | 03:14 | |
| *** ociuhandu has joined #openstack-ironic | 03:16 | |
| *** ociuhandu has quit IRC | 03:21 | |
| *** ricolin has joined #openstack-ironic | 03:39 | |
| *** ociuhandu has joined #openstack-ironic | 03:42 | |
| *** ociuhandu has quit IRC | 03:51 | |
| *** yedongcan has quit IRC | 03:53 | |
| *** dtantsur|afk has quit IRC | 04:07 | |
| *** stevebaker has quit IRC | 04:07 | |
| *** dtantsur has joined #openstack-ironic | 04:08 | |
| *** stevebaker has joined #openstack-ironic | 04:08 | |
| *** ijw has quit IRC | 04:13 | |
| *** ijw has joined #openstack-ironic | 04:14 | |
| *** dtantsur has quit IRC | 04:17 | |
| *** dtantsur has joined #openstack-ironic | 04:22 | |
| *** dtantsur has quit IRC | 04:30 | |
| *** dtantsur has joined #openstack-ironic | 04:30 | |
| *** dtantsur has quit IRC | 04:37 | |
| *** stevebaker has quit IRC | 04:38 | |
| *** dtantsur has joined #openstack-ironic | 04:39 | |
| *** stevebaker has joined #openstack-ironic | 04:40 | |
| *** dtantsur has quit IRC | 04:43 | |
| *** threestrands has joined #openstack-ironic | 04:43 | |
| *** dtantsur has joined #openstack-ironic | 04:44 | |
| *** dtantsur has quit IRC | 04:49 | |
| *** dtantsur has joined #openstack-ironic | 04:49 | |
| *** ociuhandu has joined #openstack-ironic | 05:09 | |
| *** ociuhandu has quit IRC | 05:14 | |
| *** stevebaker has quit IRC | 05:18 | |
| *** stevebaker has joined #openstack-ironic | 05:18 | |
| *** dtantsur has quit IRC | 05:19 | |
| *** stevebaker has quit IRC | 05:23 | |
| *** stevebaker has joined #openstack-ironic | 05:26 | |
| *** dtantsur has joined #openstack-ironic | 05:26 | |
| *** dtantsur has quit IRC | 05:31 | |
| *** dtantsur has joined #openstack-ironic | 05:31 | |
| *** stevebaker has quit IRC | 05:31 | |
| *** stevebaker has joined #openstack-ironic | 05:36 | |
| *** stevebaker has quit IRC | 05:41 | |
| *** stevebaker has joined #openstack-ironic | 05:42 | |
| *** stevebaker has quit IRC | 05:48 | |
| *** spotz has quit IRC | 05:48 | |
| *** kfox1111 has quit IRC | 05:48 | |
| *** JayF has quit IRC | 05:48 | |
| *** mgoddard has quit IRC | 05:51 | |
| *** mgoddard has joined #openstack-ironic | 05:53 | |
| *** spotz has joined #openstack-ironic | 05:54 | |
| *** kfox1111 has joined #openstack-ironic | 05:54 | |
| *** JayF has joined #openstack-ironic | 05:54 | |
| *** mrda has quit IRC | 05:55 | |
| *** ociuhandu has joined #openstack-ironic | 06:01 | |
| *** stevebaker has joined #openstack-ironic | 06:02 | |
| *** rcernin has quit IRC | 06:06 | |
| *** dtantsur has quit IRC | 06:09 | |
| *** dtantsur has joined #openstack-ironic | 06:10 | |
| *** stevebaker has quit IRC | 06:10 | |
| *** stevebaker has joined #openstack-ironic | 06:11 | |
| *** ociuhandu has quit IRC | 06:14 | |
| *** bobmel has quit IRC | 06:27 | |
| *** bobmel has joined #openstack-ironic | 06:28 | |
| *** dtantsur has quit IRC | 06:46 | |
| *** dtantsur has joined #openstack-ironic | 06:46 | |
| *** stevebaker has quit IRC | 06:47 | |
| *** ociuhandu has joined #openstack-ironic | 06:47 | |
| *** stevebaker has joined #openstack-ironic | 06:47 | |
| *** threestrands has quit IRC | 06:48 | |
| openstackgerrit | Merged openstack/ironic master: Add timeout when querying agent's command statuses https://review.opendev.org/696072 | 07:09 |
| openstackgerrit | Vladyslav Drok proposed openstack/ironic stable/train: Add timeout when querying agent's command statuses https://review.opendev.org/696458 | 07:11 |
| *** rcernin has joined #openstack-ironic | 07:17 | |
| *** jtomasek has quit IRC | 07:28 | |
| *** jtomasek has joined #openstack-ironic | 07:29 | |
| arne_wiebalck | Good morning, ironic! | 07:31 |
| kaifeng | good morning, arne_wiebalck o/ | 07:38 |
| arne_wiebalck | hey kaifeng o/ | 07:38 |
| *** ociuhandu has quit IRC | 07:45 | |
| *** pcaruana has joined #openstack-ironic | 07:49 | |
| *** pcaruana has quit IRC | 07:53 | |
| *** pcaruana has joined #openstack-ironic | 07:56 | |
| *** jtomasek has quit IRC | 08:20 | |
| *** tesseract has joined #openstack-ironic | 08:22 | |
| *** tssurya has joined #openstack-ironic | 08:29 | |
| openstackgerrit | Arne Wiebalck proposed openstack/ironic-inspector master: Get details when listing ports upon inspection of active nodes https://review.opendev.org/696133 | 08:30 |
| *** pcaruana has quit IRC | 08:33 | |
| *** pradip_k has joined #openstack-ironic | 08:37 | |
| *** xXraphXx has joined #openstack-ironic | 08:44 | |
| *** pradip_k has quit IRC | 08:44 | |
| *** rpittau|afk is now known as rpittau | 08:54 | |
| rpittau | good morning ironic! o/ | 08:54 |
| kaifeng | good morning rpittau o/ | 08:59 |
| rpittau | hey kaifeng :) | 08:59 |
| *** mbeierl has quit IRC | 09:03 | |
| *** mbeierl has joined #openstack-ironic | 09:05 | |
| *** lucasagomes has joined #openstack-ironic | 09:15 | |
| *** dougsz has joined #openstack-ironic | 09:15 | |
| *** alexmcleod has joined #openstack-ironic | 09:23 | |
| *** alexmcleod is now known as alexmcleod|bbl | 09:24 | |
| *** iurygregory has joined #openstack-ironic | 09:26 | |
| iurygregory | good morning o/ | 09:26 |
| *** pcaruana has joined #openstack-ironic | 09:26 | |
| kaifeng | good morning iurygregory o/ | 09:27 |
| * kaifeng hunts for supper | 09:28 | |
| *** hjensas has joined #openstack-ironic | 09:29 | |
| * etingof wishes kaifeng a fruitful hunt! | 09:32 | |
| iurygregory | kaifeng, o/ | 09:34 |
| *** derekh has joined #openstack-ironic | 09:36 | |
| *** alexmcleod|bbl is now known as alexmcleod | 09:37 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic stable/stein: Add timeout when querying agent's command statuses https://review.opendev.org/696484 | 09:39 |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic stable/rocky: Add timeout when querying agent's command statuses https://review.opendev.org/696485 | 09:39 |
| *** rcernin has quit IRC | 09:40 | |
| *** rcernin has joined #openstack-ironic | 09:42 | |
| *** rcernin has quit IRC | 09:48 | |
| openstackgerrit | Arne Wiebalck proposed openstack/ironic-inspector master: Get details when listing ports upon inspection of active nodes https://review.opendev.org/696133 | 09:50 |
| *** pcaruana has quit IRC | 09:51 | |
| *** bobmel has quit IRC | 10:01 | |
| openstackgerrit | Merged openstack/ironic-ui master: Drop python 2.7 support and testing https://review.opendev.org/695801 | 10:10 |
| *** hamzy_ has joined #openstack-ironic | 10:11 | |
| *** hamzy has quit IRC | 10:13 | |
| iurygregory | team quick question configdrive always use cloud-init? | 10:29 |
| *** jtomasek has joined #openstack-ironic | 10:36 | |
| *** rcernin has joined #openstack-ironic | 10:40 | |
| rpittau | iurygregory: not necessarily | 10:41 |
| *** khansa has joined #openstack-ironic | 10:49 | |
| kaifeng | iurygregory: we only used cloudinit, if there are better alternatives please recommend ;) | 10:50 |
| iurygregory | kaifeng, if you are using standalone and the OS doesn't support cloud-init who runs the script? https://docs.openstack.org/ironic/train/install/advanced.html#local-boot | 10:51 |
| kaifeng | etingof: thanks :) now i feel recharged | 10:51 |
| kaifeng | iurygregory: we currently doesn't have standalone usecase :( | 10:53 |
| iurygregory | kaifeng, https://docs.openstack.org/ironic/train/install/configdrive.html#when-used-standalone ? O.o | 10:53 |
| kaifeng | but this is not necessary when you can build your own instance image | 10:53 |
| kaifeng | iurygregory: the doc here is talking about configdrive usage with or without compute service | 10:57 |
| kaifeng | if you don't use configdrive, then you can bypass that part ;) | 10:57 |
| iurygregory | kaifeng, gotcha | 10:57 |
| dtantsur | morning ironic | 10:58 |
| * dtantsur immediately goes to lunch | 10:58 | |
| *** bobmel has joined #openstack-ironic | 11:02 | |
| etingof | apparently, ipa-induced OS reboot does count as libvirt domain reboot | 11:03 |
| etingof | s/does/does not/ | 11:04 |
| *** bobmel has quit IRC | 11:04 | |
| kaifeng | etingof: reboot issued in the ipa? | 11:04 |
| etingof | yes | 11:04 |
| kaifeng | i think it's a soft reboot, if you have xml modification, it will not take effect | 11:04 |
| *** bobmel has joined #openstack-ironic | 11:04 | |
| etingof | exactly | 11:05 |
| kaifeng | destroy/start required | 11:05 |
| etingof | can I tell ironic to play harder with the node? | 11:05 |
| etingof | that is, power-cycle upon deploy | 11:05 |
| kaifeng | funny i just browsed an oob reboot path after deployment | 11:05 |
| *** bobmel has quit IRC | 11:06 | |
| etingof | yeah, me too | 11:06 |
| etingof | but I am not sure I encountered oob path yet, is it there? | 11:07 |
| kaifeng | there is an conf option or driver option i remember | 11:08 |
| kaifeng | not sure if it's still there because the code i checked is still ocata version | 11:08 |
| etingof | what's also confusing, that libvirt changes its power state briefly during soft reboot | 11:11 |
| kaifeng | https://opendev.org/openstack/ironic/src/branch/master/ironic/drivers/modules/agent_base_vendor.py#L698 | 11:11 |
| kaifeng | still there ^ | 11:11 |
| rpittau | https://docs.openstack.org/ironic/latest/admin/drivers/ipa.html at the bottom ? | 11:11 |
| rpittau | etingof kaifeng ^ | 11:12 |
| etingof | yes! thanks! | 11:13 |
| *** sshnaidm|afk is now known as sshnaidm | 11:13 | |
| kaifeng | i didn't know we have a doc /o\ | 11:13 |
| rpittau | lol | 11:13 |
| * rpittau too much sun outside, can't resist | 11:14 | |
| *** rpittau is now known as rpittau|bbl | 11:14 | |
| *** hwoarang has joined #openstack-ironic | 11:17 | |
| openstackgerrit | Merged openstack/ironic-python-agent master: RAID 5/6 https://review.opendev.org/694705 | 11:18 |
| mgoddard | hi arne_wiebalck, FYI I'm about to make a start on the agent deploy steps spec. | 11:24 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Make redfish CI jobs pulling sushy-tools from git https://review.opendev.org/693866 | 11:24 |
| mgoddard | not sure we have a story for it yet :/ | 11:24 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Add a CI job to legacy boot over Redfish virtual media https://review.opendev.org/693017 | 11:24 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Add a CI job to UEFI boot over Redfish virtual media https://review.opendev.org/693867 | 11:24 |
| *** pcaruana has joined #openstack-ironic | 11:26 | |
| etingof | libvirt is evil - why does it pretend to flip power on soft reboot... | 11:27 |
| *** khansa has quit IRC | 11:35 | |
| dtantsur | to make our life funnier, of course :) | 11:41 |
| *** ociuhandu has joined #openstack-ironic | 11:46 | |
| *** tssurya has quit IRC | 11:47 | |
| *** khansa has joined #openstack-ironic | 11:49 | |
| dtantsur | hjensas: hi, around? I think your IPv6 DIB patch breaks CentOS (and RHEL?) 8 building | 11:49 |
| hjensas | dtantsur: hi, I replied on the review. RHEL8 should be ok as long as the openstack repos are available. | 11:49 |
| dtantsur | well, this breaks source builds. we need to fix it or revert, unfortunately.. | 11:50 |
| *** ociuhandu has quit IRC | 11:50 | |
| hjensas | dtantsur: With yum it installs all it can, and warn about missing packages. With dnf we need to tell dnf to behave like yum. i.e `--setopt=strict=0` | 11:51 |
| openstackgerrit | khansa proposed openstack/sushy master: Add OEM extension example script https://review.opendev.org/689835 | 11:51 |
| dtantsur | hjensas: right, but what should we do now? we cannot fix it on ironic side. | 11:51 |
| dtantsur | do you suggest using an explicit dnf command instead of pkg-map? | 11:51 |
| hjensas | dtantsur: I'd suggest changing diskimage builder to use `--setopt=strict=0` with the pkg-map? | 11:52 |
| dtantsur | hjensas: I'm fine with that, but I'm not a DIB person | 11:52 |
| hjensas | dtantsur: that would make dib work the same with yum and dnf afaict. | 11:52 |
| hjensas | dtantsur: I will take a look at DIB to see if I can figure it out. | 11:53 |
| *** tssurya has joined #openstack-ironic | 11:53 | |
| dtantsur | thanks! | 11:53 |
| dtantsur | it also looks like dhcp-all-interfaces doesn't work any more in the CI: https://a9680179c3e6feb2c4f2-4bde758ed39109c7f8278d34d2b54de4.ssl.cf5.rackcdn.com/696379/2/check/ipa-tempest-wholedisk-bios-ipmi-direct-dib-src/4232b8f/controller/logs/ironic-bm-logs/node-0_no_ansi_2019-11-27-19%3A19%3A35_log.txt.gz | 11:53 |
| dtantsur | hjensas: is it possible that https://review.opendev.org/#/c/692110/13/diskimage_builder/elements/dhcp-all-interfaces/install.d/50-dhcp-all-interfaces@24 breaks DHCP? | 11:55 |
| patchbot | patch 692110 - diskimage-builder - Add IPv6 support in dhcp-all-interfaces (MERGED) - 13 patch sets | 11:55 |
| hjensas | dtantsur: ouch, if it does'nt respond in 30 seconds yes. If we leave the retry it will do 30seconds * 30 retry's before moving on to ipv6. (because initscripts are stupid ...) | 11:56 |
| dtantsur | hmmm | 11:57 |
| hjensas | dtantsur: we could bump DIB_DHCP_TIMEOUT or look at how to get systemd to retry the ifup $DEVICE? | 11:58 |
| dtantsur | I think we've already bumped it to a minute | 11:59 |
| *** kaifeng is now known as kaifeng|afk | 12:03 | |
| kaifeng|afk | good night o/ | 12:03 |
| dtantsur | hjensas: do you want to propose a DIB patch for strict=0 or should I? | 12:07 |
| hjensas | dtantsur: I think if we use Restart=always, StartLimitInterval=N, StartLimitBurst=N, RestartSec=N in diskimage_builder/elements/dhcp-all-interfaces/install.d/dhcp-interface@.service we could get it to do retries without relying on dhclient to do the retries which is holding us up for timeout*retries before initscripts tries ipv6. | 12:08 |
| hjensas | dtantsur: if you see where to put it go ahead. | 12:09 |
| dtantsur | hjensas: I'm not sure where, I got distracted by the CentOS 7 timeout issue | 12:09 |
| hjensas | dtantsur: me as well. | 12:09 |
| dtantsur | I think we need Restart=on-failure, no? | 12:10 |
| openstackgerrit | raphael.glon proposed openstack/ironic master: Feature: finalize_rescue, send sha512 hashed password, not clear ones https://review.opendev.org/695649 | 12:12 |
| hjensas | dtantsur: here probably - diskimage_builder/elements/yum/bin/install-packages:137 - I will write a patch. | 12:13 |
| dtantsur | thx! I'll hack together something to try Restart=on-failure in the CI | 12:14 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent-builder master: DNM try Restart=on-failure for dhcp-all-interfaces https://review.opendev.org/696530 | 12:19 |
| *** ociuhandu has joined #openstack-ironic | 12:20 | |
| hjensas | dtantsur: DIB patch, https://review.opendev.org/696532, I think this should do it. | 12:32 |
| patchbot | patch 696532 - diskimage-builder - dnf - Don't error on missing packages - 3 patch sets | 12:32 |
| *** ociuhandu has quit IRC | 12:38 | |
| *** ociuhandu has joined #openstack-ironic | 12:41 | |
| openstackgerrit | Stephen Finucane proposed openstack/ironic master: Remove references to 'firewall_driver' https://review.opendev.org/696535 | 12:44 |
| *** S4ren has joined #openstack-ironic | 12:45 | |
| *** ociuhandu has quit IRC | 12:45 | |
| dtantsur | thanks hjensas! | 12:48 |
| *** rcernin has quit IRC | 12:54 | |
| *** ociuhandu has joined #openstack-ironic | 12:57 | |
| *** ociuhandu has quit IRC | 13:03 | |
| etingof | kaifeng|afk, I think libvirt in devstack would benefit from <on_reboot>destroy</> domain option | 13:03 |
| *** rcernin has joined #openstack-ironic | 13:08 | |
| dtantsur | etingof: is it a real option or you just wish it would exist? | 13:11 |
| *** ajya[m] has quit IRC | 13:15 | |
| *** csmart has quit IRC | 13:15 | |
| *** mkowalski has quit IRC | 13:15 | |
| *** goldyfruit_ has quit IRC | 13:19 | |
| etingof | dtantsur, very real! | 13:23 |
| dtantsur | etingof: then just add it? :) | 13:23 |
| etingof | absolutely! | 13:24 |
| *** ociuhandu has joined #openstack-ironic | 13:31 | |
| *** rloo has joined #openstack-ironic | 13:35 | |
| dtantsur | etingof: reading the docs, I'm afraid it will power off *instead of* rebooting | 13:37 |
| *** ociuhandu has quit IRC | 13:37 | |
| dtantsur | hmm, although in another place it seems to be what you expect. let's try | 13:38 |
| dtantsur | but https://libvirt.org/formatdomain.html seems to claim that the VM won't start automatically with <on_reboot>destroy | 13:41 |
| dtantsur | etingof: hmm, another thing: we seem to do power off, not reboot from inside IPA. | 13:43 |
| dtantsur | (which is logical, we need to do a few actions while the node is off) | 13:43 |
| etingof | hmmm | 13:43 |
| etingof | if we do poweroff from guest, that should lead to destroy | 13:44 |
| dtantsur | exactly | 13:44 |
| etingof | but it does not | 13:44 |
| dtantsur | do we have libvirt logs in that CI job? | 13:44 |
| etingof | or at least pending domain config changes still not applied | 13:44 |
| etingof | we do | 13:45 |
| etingof | although I am using my local devstack | 13:45 |
| * dtantsur checks | 13:45 | |
| *** ociuhandu has joined #openstack-ironic | 13:46 | |
| etingof | perhaps the domain is always destroyed -- Append log message (vm='node-0' message='2019-11-28 12:08:41.755+0000: shutting down, reason=destroyed | 13:49 |
| *** belmoreira has quit IRC | 13:49 | |
| etingof | I do not see any other reason when shutting down | 13:49 |
| dtantsur | yep, seems destroyed 4 times: https://6b97d39e57495a65f197-dfb7731ce6789292a31228a6fdf28206.ssl.cf5.rackcdn.com/693017/15/check/ironic-tempest-ipa-partition-bios-redfish-vmedia/5073853/controller/logs/libvirt/qemu/node-0_log.txt.gz | 13:49 |
| etingof | however that's not really important - I think I know what the new problem with that job is | 13:50 |
| etingof | still, even if ipa does not do reboot, domain destruction on reboot seems to make sense anyway... | 13:51 |
| etingof | whenever reboot happens | 13:51 |
| *** ociuhandu has quit IRC | 13:51 | |
| *** belmoreira has joined #openstack-ironic | 13:51 | |
| etingof | on the other hand, whatever ipa does, pending domain config is never applied | 13:55 |
| *** pcaruana has quit IRC | 13:55 | |
| dtantsur | yep | 13:55 |
| dtantsur | and I think in the CI job it uses hard power off anyway (not sure why) | 13:56 |
| * etingof needs stare at the green screen for some more | 13:56 | |
| dtantsur | ah. because you configured that :) https://review.opendev.org/#/c/693017/15/devstack/lib/ironic | 13:56 |
| patchbot | patch 693017 - ironic - Add a CI job to legacy boot over Redfish virtual m... - 15 patch sets | 13:56 |
| etingof | right, that's in my last patch | 13:56 |
| etingof | but before that, guest poweroff did not apply domain config - that's puzzling | 13:57 |
| dtantsur | etingof: I guess it should be reverted since it does not help | 13:58 |
| *** rpittau|bbl is now known as rpittau | 13:58 | |
| dtantsur | (and it's not something we should use in production by default) | 13:58 |
| etingof | dtantsur, I suspect it helped and revealed the next problem \o/ | 13:59 |
| dtantsur | etingof: no, it still boots from CD/DVD | 13:59 |
| dtantsur | and we shouldn't rely on it if possible | 13:59 |
| dtantsur | otherwise we end up with a (yet another) non-production option in the CI | 13:59 |
| etingof | so my theory now is that libvirt domain has the right instance image in virtual CD | 14:00 |
| etingof | however boot device is not properly set on CD, so the default applied which is previously deployed hdd... | 14:01 |
| *** rcernin has quit IRC | 14:01 | |
| etingof | well, that's what I think I see in devstack | 14:01 |
| etingof | I will experiment some more to see if that oob option is really necessary | 14:02 |
| etingof | hmm, but why do you say it still boot from CD? is it set to do local boot? | 14:03 |
| *** ociuhandu has joined #openstack-ironic | 14:05 | |
| dtantsur | etingof: the latest CI run failed the same way as the previous one: the node booted from virtual media instead of local disk | 14:10 |
| *** ociuhandu has quit IRC | 14:10 | |
| dtantsur | so I don't think it depends on how we power the node off | 14:10 |
| dtantsur | I rather suspect something is wrong with changing the boot device and/or removing the virtual media | 14:11 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent-builder master: DNM try Restart=on-failure for dhcp-all-interfaces https://review.opendev.org/696530 | 14:14 |
| *** ociuhandu has joined #openstack-ironic | 14:14 | |
| etingof | dtantsur, hold on, is it a local boot? | 14:16 |
| dtantsur | etingof: mmm, right, netboot | 14:16 |
| dtantsur | which means that booting a CD is probably correct, but booting IPA is not :) | 14:16 |
| etingof | dtantsur, no, should be booted from CD | 14:16 |
| *** ricolin_ has joined #openstack-ironic | 14:16 | |
| etingof | right | 14:16 |
| mgoddard | about to try to describe the ironic deployment process in words for my spec. New cup of tea required | 14:17 |
| dtantsur | okay, correcting myself: it boots IPA instead of the final instance. same as the previous run with in-band power off. | 14:17 |
| dtantsur | mgoddard: heh, good luck :) | 14:17 |
| etingof | that's puzzling indeed. but power cycle worked out | 14:19 |
| *** ociuhandu has quit IRC | 14:19 | |
| *** ricolin has quit IRC | 14:19 | |
| dtantsur | yeah, it's not a problem. changing the CD apparently is | 14:20 |
| * etingof is on that | 14:21 | |
| *** hjensas has quit IRC | 14:21 | |
| *** belmoreira has quit IRC | 14:21 | |
| *** belmoreira has joined #openstack-ironic | 14:24 | |
| dtantsur | ouch, TheJulia, it seems that some of our jobs have turned into IPv6 jobs themselves: https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_3b6/688411/2/check/ironic-standalone-ipa-src/3b61f33/controller/logs/ironic-bm-logs/node-1_console_log.txt.gz | 14:26 |
| iurygregory | dtantsur, this a bad thing? | 14:27 |
| dtantsur | iurygregory: well, it doesn't work in the CI yet. so kinda bad :) | 14:27 |
| *** ricolin_ has quit IRC | 14:28 | |
| iurygregory | dtantsur, ouch! | 14:28 |
| *** belmoreira has quit IRC | 14:30 | |
| *** belmoreira has joined #openstack-ironic | 14:40 | |
| *** gudrutis has joined #openstack-ironic | 14:40 | |
| *** ricolin has joined #openstack-ironic | 14:40 | |
| *** goldyfruit_ has joined #openstack-ironic | 14:54 | |
| *** belmoreira has quit IRC | 14:55 | |
| *** belmoreira has joined #openstack-ironic | 14:58 | |
| *** goldyfruit___ has joined #openstack-ironic | 14:59 | |
| *** goldyfruit_ has quit IRC | 15:01 | |
| *** belmoreira has quit IRC | 15:02 | |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Add a CI job to legacy boot over Redfish virtual media https://review.opendev.org/693017 | 15:07 |
| *** hjensas has joined #openstack-ironic | 15:09 | |
| *** csmart has joined #openstack-ironic | 15:11 | |
| *** mkowalski has joined #openstack-ironic | 15:11 | |
| *** ajya[m] has joined #openstack-ironic | 15:11 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-lib master: Updating testing doc to py3 https://review.opendev.org/696593 | 15:14 |
| *** hamzy_ is now known as hamzy | 15:15 | |
| TheJulia | dtantsur: I had a feeling we were getting extra v6 stuff someplace... | 15:25 |
| TheJulia | :( | 15:25 |
| dtantsur | heh | 15:25 |
| dtantsur | if you don't get IPv6, IPv6 gets YOU | 15:25 |
| iurygregory | lol | 15:25 |
| TheJulia | I hear someone saying we will use ipv6 or we will use ipv6 | 15:26 |
| dtantsur | exactly | 15:26 |
| * dtantsur relocates home | 15:27 | |
| *** dtantsur is now known as dtantsur|brb | 15:27 | |
| *** bobmel has joined #openstack-ironic | 15:31 | |
| rpittau | I was wondering, now that we're dropping support for Python 2.7 (yay!), is it too early to start maybe removing some 'six' usage? | 15:31 |
| *** belmoreira has joined #openstack-ironic | 15:31 | |
| *** bobmel has quit IRC | 15:33 | |
| iurygregory | rpittau, we can start I would say | 15:35 |
| * iurygregory it's on my plans after finishing other priorities | 15:35 | |
| *** belmoreira has quit IRC | 15:39 | |
| rpittau | iurygregory: yeah, my concern is that removing six will definitely break any kind of compatibility with Python <= 3.x | 15:43 |
| rpittau | iurygregory: I kind of started removing six from some places as at the moment I don't have a lot on my plate, upstream at least :) | 15:43 |
| iurygregory | rpittau, but Ussuri will be python3.6+ so it's fine =) | 15:44 |
| iurygregory | rpittau me too XD | 15:44 |
| *** ociuhandu has joined #openstack-ironic | 15:46 | |
| *** ociuhandu has quit IRC | 15:52 | |
| openstackgerrit | Merged openstack/ironic master: Make redfish CI jobs pulling sushy-tools from git https://review.opendev.org/693866 | 15:54 |
| *** bobmel has joined #openstack-ironic | 15:54 | |
| *** bobmel has quit IRC | 15:56 | |
| *** khansa has quit IRC | 15:57 | |
| *** belmoreira has joined #openstack-ironic | 15:58 | |
| *** dtantsur|brb is now known as dtantsur | 15:58 | |
| dtantsur | iurygregory, rpittau, a few patches are already on the way that break this compatibility | 15:59 |
| dtantsur | namely, ironic-lib mdns-v6 and netaddr removal | 15:59 |
| iurygregory | dtantsur, cool! | 15:59 |
| rpittau | dtantsur: yeah, I was reviewing some recent changes not only in ironic and it seems we stop caring entirely about Python 2.7 (finally) | 16:00 |
| *** belmoreira has quit IRC | 16:00 | |
| dtantsur | well, that was the goal, right? :) | 16:00 |
| *** bobmel has joined #openstack-ironic | 16:01 | |
| rpittau | oh I'm not complaining :D | 16:02 |
| dtantsur | :) | 16:02 |
| *** belmoreira has joined #openstack-ironic | 16:04 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: Stop using six library https://review.opendev.org/696609 | 16:15 |
| *** bobmel has quit IRC | 16:16 | |
| *** bobmel has joined #openstack-ironic | 16:17 | |
| dtantsur | hjensas: "dhcp-interface@eth0.service has Restart= setting other than no, which isn't allowed for Type=oneshot services. Refusing." sigh | 16:18 |
| *** belmoreira has quit IRC | 16:18 | |
| *** belmoreira has joined #openstack-ironic | 16:19 | |
| rpioso | Good morning, ironicers | 16:20 |
| dtantsur | morning rpioso | 16:20 |
| rpittau | good morning rpioso :) | 16:20 |
| rpioso | dtantsur, rpittau: o/ | 16:20 |
| rpioso | Have you recently deployed devstack? And on bionic? | 16:20 |
| rpittau | yes! | 16:21 |
| rpittau | well, define recently | 16:21 |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent-builder master: DNM try Restart=on-failure for dhcp-all-interfaces https://review.opendev.org/696530 | 16:21 |
| rpioso | This week | 16:21 |
| *** ociuhandu has joined #openstack-ironic | 16:22 | |
| rpittau | yes, at the beginning of the week actually | 16:22 |
| rpioso | I've been trying to redeploy it. A month old deployment encounters a failure during node deployment. The conductors command to the ramdisk agent to soft power off times out after a long time. I'm hoping that has been addressed. | 16:23 |
| *** ociuhandu has quit IRC | 16:23 | |
| openstackgerrit | Dmitry Tantsur proposed openstack/ironic-python-agent-builder master: DNM try Restart=on-failure for dhcp-all-interfaces https://review.opendev.org/696530 | 16:24 |
| rpittau | rpioso: I didn't notice that issue, but it's true that I was testing something specific, so my deployment could have been different from yours | 16:24 |
| rpioso | With USE_PYTHON3 not set in local.conf, update_clouds_yaml.py fails, because no yaml package can be found. | 16:25 |
| dtantsur | well, this ^^ is a generic devstack problem | 16:25 |
| rpittau | yep | 16:25 |
| dtantsur | (probably related to the fact that people are dropping python 2 support) | 16:25 |
| *** ricolin_ has joined #openstack-ironic | 16:25 | |
| rpittau | rpioso: did you update an environment or redeployed from scracth? Cause recently everyone started dropping python 2 and if you don't redeploy there might be some conflicts | 16:26 |
| rpittau | s/some/a lot/ | 16:26 |
| *** hjensas has quit IRC | 16:27 | |
| rpioso | I set USE_PYTHON3=True and PYTHON3_VERSION=3. Also edit that script's first line to #!/usr/bin/env python3 | 16:27 |
| *** ricolin has quit IRC | 16:28 | |
| rpioso | rpittau: It redeployed from scratch. I revert to a snapshot pre-deployment. | 16:28 |
| rpioso | dtantsur: Any insight into the soft power off issue I've been wrestling with over the past couple of weeks? | 16:29 |
| rpioso | rpittau: Are those settings and that script change the remedy? | 16:32 |
| dtantsur | rpioso: not really, I haven't seen anything alike. | 16:33 |
| * rpioso was surprised yaml couldn't be found | 16:33 | |
| *** lucasagomes has quit IRC | 16:33 | |
| rpittau | rpioso: I do have USE_PYTHON3="True" set in my local.conf, that's it | 16:34 |
| rpioso | dtantsur: Also, agent logging is not displayed on the console. When I previously deployed from an RDO undercloud, lots of logging appeared, but that was over a year ago. | 16:35 |
| rpioso | rpittau: Thank you! I've kicked off another deployment. Will see how it goes :-) | 16:35 |
| *** belmoreira has quit IRC | 16:36 | |
| rpioso | dtantsur: Is there a knob to turn on console log output? | 16:36 |
| dtantsur | it may be related to what is set in pxe_append_params | 16:37 |
| dtantsur | rpioso: you may want to play with IRONIC_TTY_DEV devstack variable. I'm not sure where the current value is coming from. | 16:40 |
| dtantsur | I suspect it's required for the CI, but may differ for local testing | 16:41 |
| rpioso | dtantsur: pxe_append_params is set to the ironic default in ironic.conf. | 16:42 |
| dtantsur | hmm | 16:42 |
| rpioso | dtantsur: Btw, I'm testing against real hw. | 16:42 |
| dtantsur | I *think* the default ironic value should be fine (unlike what devstack is trying to do) | 16:43 |
| *** tssurya has quit IRC | 16:44 | |
| rpioso | dtantsur: Well, it's not set in devstack's local.conf. Perhaps the plugin changes it. | 16:44 |
| dtantsur | yeah, devstack has a different default | 16:44 |
| dtantsur | but maybe only for VM case | 16:44 |
| rpioso | dtantsur: I'll look into that more after the deployment finishes. | 16:45 |
| dtantsur | rpioso: what pxe_append_params in your ironic.conf? | 16:45 |
| *** pcaruana has joined #openstack-ironic | 16:45 | |
| rpioso | dtantsur: Can't look now. A deployment is under way. | 16:45 |
| rpioso | If memory serves, it was set, though. Oh wait! I have an ironic.conf stashed away from earlier. | 16:46 |
| rpioso | dtantsur: pxe_append_params = nofb nomodeset vga=normal console=ttyS0 | 16:47 |
| *** tesseract has quit IRC | 16:48 | |
| dtantsur | rpioso: this "console" bit is probably hurting you. It comes from IRONIC_TTY_DEV variable and may not match what the hardware needs | 16:48 |
| dtantsur | I'd recommend maybe removing it? | 16:48 |
| rpioso | dtantsur: Will do! | 16:48 |
| * rpioso doesn't like pain ;-) | 16:48 | |
| *** iurygregory has quit IRC | 16:49 | |
| rpioso | dtantsur: Do you know if over the past year or so the ramdisk agent has changed how it soft power offs the server? What I'm seeing, cdearborn also has been seeing. AND, our 3d party CI sees it, too. | 16:50 |
| rpioso | dtantsur: To the casual observer, the deployment appears to go well and succeed. The node is active and no last error is set. | 16:52 |
| dtantsur | rpioso: I don't recall any changes to that bit. | 16:52 |
| rpioso | dtantsur: However, when it boots into the deployed instance OS, the server tries to PXE boot first. After that times out, it proceeds to the disk. | 16:53 |
| rpioso | dtantsur: Stranger still, in the hw's BIOS setup (tough name), the OS on the disk isn't listed as a UEFI boot source. | 16:54 |
| *** ociuhandu has joined #openstack-ironic | 16:54 | |
| rpioso | dtantsur: This has been reproduced on different machines. And models. Firmware and BIOS is up-to-date. | 16:56 |
| dtantsur | rpioso: which driver are we talking about? with IPMI we don't have much power over boot order. | 16:56 |
| * dtantsur has to go really soon | 16:57 | |
| rpioso | dtantsur: redfish and idrac WS-Man. | 16:57 |
| rpioso | dtantsur: No IPMI | 16:57 |
| rpioso | dtantsur: Haven't tried IPMI | 16:58 |
| *** ricolin_ is now known as ricolin | 16:58 | |
| dtantsur | rpioso: I'd look into hardware side to be honest (and try IPMI just to be sure). | 16:58 |
| rpioso | dtantsur: Okie doke. Thanks for your assistance. | 16:58 |
| * rpioso got a successful devstack deployment \o/ | 16:58 | |
| dtantsur | np, I don't think I've helped much :) | 16:58 |
| *** dtantsur is now known as dtantsur|afk | 16:59 | |
| dtantsur|afk | g'night | 16:59 |
| rpioso | dtantsur|afk: The pxe_append_params pointer will hopefully enable me to gain more insights. | 16:59 |
| *** hjensas has joined #openstack-ironic | 16:59 | |
| rpioso | dtantsur|afk: o/ | 16:59 |
| *** ociuhandu has quit IRC | 17:00 | |
| rpioso | rpittau: Thanks, again! | 17:02 |
| rpittau | rpioso: anytime :) | 17:02 |
| *** jtomasek has quit IRC | 17:04 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-inspector master: Stop using six library https://review.opendev.org/696617 | 17:36 |
| rpittau | good night! o/ | 17:36 |
| *** rpittau is now known as rpittau|afk | 17:36 | |
| goldyfruit___ | Guys, is there a way to get the node ID during the discovery ? | 17:41 |
| goldyfruit___ | I guess it's totally up to Ironic API to generate the UUID | 17:43 |
| goldyfruit___ | But maybe there is a way to retrieve this value during the discovery | 17:43 |
| goldyfruit___ | It seems that the response from the /v1/continue is the node UUID | 17:44 |
| openstackgerrit | Mark Goddard proposed openstack/ironic-specs master: Add in-band deploy steps spec https://review.opendev.org/696619 | 17:46 |
| mgoddard | arne_wiebalck: WIP, but feel free to take a look | 17:46 |
| *** S4r3n has joined #openstack-ironic | 17:48 | |
| openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: [WIP] Bring libvirt domain down prior to any change https://review.opendev.org/696620 | 17:49 |
| *** S4ren has quit IRC | 17:52 | |
| *** derekh has quit IRC | 17:59 | |
| *** dougsz has quit IRC | 18:00 | |
| *** ricolin has quit IRC | 18:01 | |
| *** S4r3n has quit IRC | 18:30 | |
| *** ianychoi_ has quit IRC | 18:30 | |
| *** bobmel has quit IRC | 18:42 | |
| openstackgerrit | Riccardo Pittau proposed openstack/ironic-python-agent master: Stop using six library https://review.opendev.org/696609 | 18:45 |
| openstackgerrit | Ilya Etingof proposed openstack/sushy-tools master: Always log via Flask application https://review.opendev.org/693639 | 18:46 |
| openstackgerrit | Ilya Etingof proposed openstack/ironic master: Add a CI job to legacy boot over Redfish virtual media https://review.opendev.org/693017 | 18:47 |
| *** lennyb has quit IRC | 18:57 | |
| *** goldyfruit_ has joined #openstack-ironic | 19:04 | |
| *** ociuhandu has joined #openstack-ironic | 19:06 | |
| *** goldyfruit___ has quit IRC | 19:07 | |
| *** ociuhandu has quit IRC | 19:10 | |
| *** lennyb has joined #openstack-ironic | 19:12 | |
| *** lennyb has quit IRC | 19:21 | |
| *** bobmel has joined #openstack-ironic | 19:30 | |
| *** ociuhandu has joined #openstack-ironic | 19:43 | |
| *** ociuhandu has quit IRC | 19:48 | |
| *** ijw has quit IRC | 20:02 | |
| *** ijw has joined #openstack-ironic | 20:03 | |
| *** ociuhandu has joined #openstack-ironic | 20:03 | |
| *** ociuhandu has quit IRC | 20:10 | |
| *** pcaruana has quit IRC | 20:13 | |
| *** kfox1111 has quit IRC | 20:23 | |
| *** mrda has joined #openstack-ironic | 20:37 | |
| *** ociuhandu has joined #openstack-ironic | 20:52 | |
| *** hoonetorg has quit IRC | 20:59 | |
| *** ociuhandu has quit IRC | 20:59 | |
| *** ociuhandu has joined #openstack-ironic | 21:06 | |
| *** michchap has quit IRC | 21:07 | |
| *** alexmcleod has quit IRC | 21:10 | |
| *** ociuhandu has quit IRC | 21:12 | |
| *** hoonetorg has joined #openstack-ironic | 21:16 | |
| *** sshnaidm is now known as sshnaidm|afk | 21:22 | |
| *** goldyfruit___ has joined #openstack-ironic | 21:27 | |
| *** goldyfruit_ has quit IRC | 21:30 | |
| arne_wiebalck | Thanks for letting me know mgoddard! Let me know if I can help. (I'm off for the rest of this week, though). | 21:35 |
| goldyfruit___ | From API documentation when IPA post the collected data it should returns the node UUID but I don't find it into the IPA ramdisk logs | 21:45 |
| goldyfruit___ | https://docs.openstack.org/api-ref/baremetal-introspection/index.html?expanded=ramdisk-callback-detail | 21:45 |
| goldyfruit___ | The call_inspector function: https://github.com/openstack/ironic-python-agent/blob/master/ironic_python_agent/inspector.py#L114-L134 | 21:45 |
| goldyfruit___ | I guess the resp.json() should contains the uuid response | 21:45 |
| *** ociuhandu has joined #openstack-ironic | 21:55 | |
| *** ociuhandu has quit IRC | 21:57 | |
| *** ociuhandu has joined #openstack-ironic | 21:57 | |
| *** Goneri has quit IRC | 22:01 | |
| rpioso | dtantsur, etingof, TheJulia: I've obtained console output of ramdisk IPA failure to execute the soft power off conductor command at the end of deployment. It's a JPEG file. May I share it with you, and, if so, how? | 22:14 |
| rpioso | dtantsur|afk: ^^^ | 22:15 |
| rpioso | dtantsur|afk, etingof, TheJulia: It appears the 'poweroff' command isn't present -- "ERROR root OSError: [Error 2] No such file or directory". What!? | 22:19 |
| *** Goneri has joined #openstack-ironic | 22:40 | |
| *** rcernin has joined #openstack-ironic | 22:41 | |
| *** ociuhandu has quit IRC | 23:00 | |
| *** adrianc has quit IRC | 23:03 | |
| *** ociuhandu has joined #openstack-ironic | 23:30 | |
| *** ociuhandu has quit IRC | 23:35 | |
| *** goldyfruit___ has quit IRC | 23:39 | |
Generated by irclog2html.py 2.15.3 by Marius Gedminas - find it at mg.pov.lt!